Misconfigured customer SLO triggers continuous Lambda backfill, exhausting shared Lambda capacity
In early August 2022, Honeycomb's SLO measuring trigger run latency began alarming. Engineers initially attributed the issue to known prior reports of customer telemetry with future timestamps that pulled trigger queries into cold storage and onto AWS Lambda. The Incident Commander had been monitoring the future-timestamps issue, and that framing dominated the response. An engineer with fresh context later traced the bulk of Lambda usage to a single misconfigured customer SLO whose SLI never returned valid results, causing Basset to assume a backfill was needed every minute and to relaunch a 60-day cold-storage scan repeatedly. Fixing the SLO on the customer's behalf stopped the bleeding.
9hNot disclosed affectedcustomers depending on triggers and SLO alerting; query performance degraded
background jobscascading failuremanual actionsla breach