Google Workspace Status Dashboard
Incident affecting Gmail
Incident began at 2022-12-10 13:29 and ended at 2022-12-10 17:52 (times are in Coordinated Universal Time (UTC)).
Date | Time | Description | |
---|---|---|---|
| Dec 16, 2022 | 10:49 PM UTC | Incident ReportBackgroundGmail's delivery system is responsible for routing, classifying, and enhancing emails. Emails are placed in a queue and are processed by a set of backend services (e.g. spam classification). If everything goes well, the mail delivery system sends the emails. However, when a failure occurs, the system keeps the emails in the queue and retries delivering them later. SummaryOn 10 December 2022 at 05:29 PT, some Gmail users observed email delivery delays of 60 minutes on average, as well as email bounce backs (i.e. senders received a delivery failure notification). Root CauseAn email campaign from a single domain triggered a latent bug in one of the dependencies of the email delivery service, which crashed and then became overloaded. These emails contained dynamic mail content triggering service instance failures. Due to this failure and a latent bug validation feature configuration, 80% of tasks crashed, which overloaded the remaining tasks. Remediation and PreventionOn 10 December 2022 at 05:29 PT, Google Engineering was alerted to an issue via internal monitoring that Gmail delivery was impacted. Google Engineering began investigating and identified the following actions to mitigate the issue:
The backlog of emails were then able to to be processed without any additional issues. Google is committed preventing a repeat of this issue in the future and is completing the following actions:
|
| Dec 12, 2022 | 7:18 PM UTC | Mini-Incident ReportWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 10 December 2022 05:29 Incident End: 10 December 2022 09:52 Duration: 4 hours and 23 minutes Affected Services and Features: Gmail Regions/Zones: Global Description: Gmail users experienced delays in email delivery and bounce back of emails globally for a period of 4 hours and 23 minutes. From preliminary analysis, the root cause of the issue was a latent bug in the message delivery logic. Customer Impact: Customers may have experienced delays in delivery or bounce back of their emails during the duration of the impact. |
| Dec 10, 2022 | 6:27 PM UTC | The issue with gmail is now fully mitigated. All the backlog of undelivered messages have been cleared and mail services are back to normal. Thanks for your patience while we resolve this issue. |
| Dec 10, 2022 | 5:28 PM UTC | Mitigation is currently underway and email delivery is no longer failing. However, the Google Engineering team is now working through the backlog of undelivered messages and expect all messages to be delivered in the next few hours. We will provide an update by Saturday, 2022-12-10 10:30 US/Pacific with current details. |
| Dec 10, 2022 | 4:16 PM UTC | Mitigation is currently underway and email delivery is no longer failing. However, the Google Engineering team is now working through the backlog of undelivered messages and expect all messages to be delivered in the next few hours. We will provide more information by Saturday, 2022-12-10 10:00 PT |
| Dec 10, 2022 | 3:35 PM UTC | We are experiencing an issue with Gmail. Users may experience delays in email delivery. Our engineering team continues to investigate the issue. We will provide an update by Saturday, 2022-12-10 08:30 US/Pacific with current details. |
- Times are listed in Coordinated Universal Time (UTC)