**Title** Google Chat customers are experiencing issues loading Chat, creating DMs (Direct Messages) and Chat Spaces. **Description** We are experiencing an iss...

What caused it: Google Chat's root cause analysis

# Incident Report Summary Google Chat users experienced errors when creating new direct messages (DMs) and chat spaces for a total duration of 50 minutes between 21:05 US/Pacific to 21:55 US/Pacific on Tuesday November 4, 2025. At its peak, the issue impacted approximately 0.3% of users. Affected users also observed increased latency in message delivery and issues with initial application loading and opening chat rooms. To our Google Chat users who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Root Cause The incident was triggered by database lock contention and hotspotting caused by a large volume of automated background jobs for database maintenance. These resource-intensive tasks created a bottleneck. This resulted in a cascading failure where other unrelated operations that relied on database resources were delayed, causing yet more operations to be delayed in a similar way. This resulted in the errors that users experienced when trying to perform actions like creating new conversations. Remediation and Prevention Google engineers were alerted to the incident through a series of monitoring alerts at 21:06 US/Pacific on Tuesday November 4, 2025 for high error rates on our internal chat services, and immediately started to investigate. The primary mitigation action was to temporarily halt the queue that processes these background cleanup jobs at 21:45 US/Pacific. This action immediately reduced the load and relieved hotspotting on the database, allowing normal operations to resume. A more specific control was later introduced to disable only the problematic background task on November 5, 2025 at 17:34 US/Pacific. We apologize for the disruption this incident caused. We are committed to learning from this event and implementing changes to prevent similar issues in the future. The following preventative measures are planned and will be implemented: - Improve Monitoring: Enhance monitoring to quickly identify the specific background tasks causing database contention. - Improve Resilience: Improve and document procedures for our engineering teams to mitigate similar incidents more quickly, such as by temporarily pausing or slowing down non-essential background tasks. - Fix Underlying Cause: Implement a fix to prevent root cause from occurring again by reducing the excessive resource...

Read the full postmortem from Google Chat