Svix outage, 24 August 2023

Elevated response times in the US region for a small percentage of the requests

Started
Resolved
24 Aug 2023, 16:00 UTC
Impact
Minor

Last update from Svix

There were elevated response times in the US region for a small percentage of the requests. The issue was due to a memory leak in a new third party library we integrated with, which caused some nodes to operate with constrained resources (heavy CPU/MEM usage). This fixed itself automatically by killing the affected nodes (each one as it exhausted its resources over the span of a few minutes) and starting new ones instead. The team has also deployed a fix to revert the inclusion of the new library pending further investigation on why this was happening, and why it wasn't detected in the staging environment.

24 Aug 2023, 16:00 UTC

Read the full incident on the official Svix status page

Is Svix down now?

This page is the record of one past incident. For the live status, see Is Svix down right now? - it reads the official status page every 2 minutes.

Other Svix incidents

  1. Elevated error rate in US region

  2. [Analytics] Delayed message listing in dashboard, portal, and list-message APIs

  3. Dashboard logins failing

  4. Elevated latency in US region

  5. [EU] elevated latency and errors for a small fraction of API requests

  6. [EU] Elevated response times and error rates affecting some customers.

  7. [usw2] very small fraction of API errors (UNAFFECTED: delivery and create-message, only supporting API affected)

  8. USW2: increased errors in the region due to AWS connectivity issues

All recorded Svix incidents