Google Workspace Status Dashboard
Incident affecting Gmail
Incident began at 2023-01-26 14:20 and ended at 2023-01-26 16:08 (times are in Coordinated Universal Time (UTC)).
Date | Time | Description | |
---|---|---|---|
| Feb 2, 2023 | 10:18 PM UTC | Incident ReportBackgroundGoogle’s back end SMTP Gmail delivery system allows customers accessing Gmail through third party email clients to send and receive email. To verify the identity of users, our SMTP delivery system uses an internal authentication framework to validate their account credentials before they are allowed to send email. SummaryOn Thursday, 26 January 2023 at 06:20 PT, some customers using third party email clients may have experienced errors and/or retry latency when attempting to send mail through our SMTP Gmail delivery system for a period of 1 hour and 48 minutes. We sincerely apologize to our Google Workspace customers who were impacted during this disruption. This is not the level of quality and reliability we strive to offer you. We have conducted an internal investigation and are taking immediate steps to improve the resiliency of our service. Root CauseAfter a detailed investigation, Engineers determined this issue was caused by memory limitations caused by unanticipated organic growth in one of the back end databases used by our internal authentication framework. This resulted in errors and/or latency experienced by customers using 3rd party email clients as they attempted to send emails to Gmail endpoint. Remediation and PreventionGoogle engineers were alerted to this issue through automated monitoring for errors and immediately began an investigation. At 06:45 PT, engineers determined that components of the authentication framework had run out of memory and started to scale up instances at 07:45 PT. The issue was completely resolved by 08:08 PT. Google is committed preventing a repeat of this issue in the future and is completing the following actions:
We appreciate your patience and apologize again for the impact to your organization. We thank you for your business. Detailed Description of ImpactOn Thursday, 26 January 2023 between 06:20 and 08:08 PT, third party email clients connecting to Gmail’s SMTP services would have experienced errors or latency when attempting to send mail. Depending on the third party email client being used, this could have resulted in errors propagated to the end user or retries. |
| Jan 27, 2023 | 4:00 PM UTC | Mini Incident ReportWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 26 January 2023 06:20 Incident End: 26 January 2023 08:08 Duration: 1 hour, 23 minutes Affected Services and Features: Gmail Regions/Zones: Global Description: Gmail experienced mailing list errors and message delivery failures, due to the back end management system unavailability. Google will be completing a full Incident Report in the following days that will provide a detailed root cause. Customer Impact:
|
| Jan 26, 2023 | 4:07 PM UTC | The problem with Gmail has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. |
| Jan 26, 2023 | 4:06 PM UTC | Our team is continuing to investigate this issue. We will provide an update by Jan 26, 2023, 5:00 PM UTC with more information about this problem. Thank you for your patience. The affected users are able to access Gmail, but are seeing error messages, high latency, and/or other unexpected behavior. Delayed mailing list messages No workaround at this time |
| Jan 26, 2023 | 4:03 PM UTC | We're investigating reports of an issue with Gmail. We will provide more information shortly. Delayed mailing list messages |
- Times are listed in Coordinated Universal Time (UTC)