Mezmo outage, 18 February 2022

Ingestion of new logs - for Syslog only - is intermittently failing

Started
Resolved
18 Feb 2022, 02:15 UTC (after 34 min)
Impact
Major
Affected
Log Ingestion (Syslog)

Last update from Mezmo

**Dates:** Start Time: Wednesday, February 17, 2022, at 20:56 UTC End Time: Thursday, February 18, 2022, at 02:15 UTC Duration: 5:19:00 ‌ **What happened:** The ingestion of new logs to our Syslog endpoint was intermittently failing. ‌ **Why it happened:** We made a code change to the area of our service \(Syslog Forwarder\) that handles the ingestion of logs sent by Syslog and inadvertently changed how memory is managed. Routine memory garbage collection stopped and memory usage increased on the pods that accept newly submitted log lines over Syslog. Eventually, the increase in memory caused the pods to crash. Any log lines held on those pods were lost and never ingested....

1 Mar 2022, 20:21 UTC

Read the full incident on the official Mezmo status page

Is Mezmo down now?

This page is the record of one past incident. For the live status, see Is Mezmo down right now? - it reads the official status page every 2 minutes.

Other Mezmo incidents

  1. Impact of AWS Issues

  2. Search performance issues

  3. Problems with Search

  4. Issue with Search

  5. Intermittent issues

  6. Log availability and search performance temporarily degraded

  7. Issues accessing logs in boards and screens

  8. Intermittent Pipeline Service Degradation

All recorded Mezmo incidents