Convex outage, 4 March 2026
high error rates on customer backends
- Started
- Resolved
- 4 Mar 2026, 18:28 UTC (after 1 h)
- Impact
- Critical
- Affected
- Free & Starter
Last update from Convex
Convex had downtime for some customers caused by an anomalous load spike that pushed an internal service into a load-shedding mode, which then triggered an unexpected panic in a caching library we were using. Specifically the load triggered two queue management algorithms: CoDel which proactively drops requests to keep queues small, and adaptive-LIFO which dequeues in reverse order to avoid wasting time on old requests. These are both rather subtle algorithms that large services use to avoid congestion collapse under high load or attack. The panic in the caching library was just a bug that depended on both these algorithms simultaneously. We've made some changes as a result of this inciden...
Is Convex down now?
This page is the record of one past incident. For the live status, see Is Convex down right now? - it reads the official status page every 2 minutes.