Incident Report

Summary

On 17 June 2024, a subset of our customers using Google Workspace products (Google Chat, Google Meet, Google Drive, Google Calendar, Google Cloud Directory and Google Docs) experienced elevated error rates and latency of varying impact between the period of 6 hours and 40 minutes between 09:06 and 15:46 US/Pacific.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The root cause is a latent issue that caused contention in getting access in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns.

Remediation and Prevention

Google engineers were alerted to the issue from our internal monitoring at 09:21 US/Pacific on 17 June 2024 and immediately started working on the incident. Once the nature and scope of the issue was identified, our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.

Google is committed preventing a repeat of this issue in the future and is completing the following actions :

• Change the underlying design that caused the contention in the cache so that the issue does recur.
• Implement mechanisms that facilitate quicker application of changes and detection of contention, to allow faster remediation.
• Update Google Chat to be more resilient to profile information unavailability.

Detailed Description of Impact

On Monday, 17 June 2024, six of our Workspace products were impacted by varying degrees between 09:06 and 15:46 US/Pacific. The details and timeframes for the impact to each product have been outlined below.

Google Calendar : Impacted users saw email addresses instead of names while using Calendar for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.

Google Chat : Impacted users experienced intermittent issues loading chat, opening chat rooms, and sending messages for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.

Google Meet : Impacted users sparsely experienced latency or failures while joining scheduled meetings over a period of 4 hours, between 11:00 to 15:00 US/Pacific on 17 June 2024.

Google Docs : Impacted users encountered the error "Name loading" in comments or other areas where authors’ names were expected for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.

Google Drive : Impacted users experienced elevated errors/latency while using third party APIs, for a duration of 6 hours 30 minutes from 09:06 to 15:36 US/Pacific on 17 June 2024.

Google Cloud Directory : Affected customers were unable to access the “User” section in the admin console. The initial dashboard displayed the error "Data can't be fetched due to system” for a period of 6 hours 11 minutes from 09:05 to 15:16 US/Pacific on 17 June 2024.

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 17 June, 2024 09:23

Incident End: 17 June, 2024 15:46

Duration: 6 hours, 23 minutes

Affected Services and Features: Google Calendar, Google Chat, Google Docs, Google Drive & Google Meet.

Regions/Zones: Global

Description:

Multiple Google Workspace products experienced elevated error rates and latency. From preliminary analysis, the root cause is a latent issue that caused lock contention in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns. Our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.

Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

• Google Chat - Affected users would have experienced intermittent latency and errors while loading chats, accessing the chat spaces and sending messages.
• Google Meet - Affected users would have experienced latency or failures while joining meetings.
• Google Drive - Affected users would have experienced latency/errors while using the third party API.
• Google Docs - Affected users would have experienced latency loading comments on the document. Some comments would have author names missing.
• Google Calendar - Affected users would have experienced elevated error rates while using Calendar.

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

The issue with Google Calendar, Google Chat, Google Docs, Google Drive, Google Meet has been resolved for all affected users as of Monday, 2024-06-17 15:27 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

The mitigation applied previously did not yield results as expected.

Our Engineering team continues to investigate and ascertain a mitigation strategy to fix all known issues

We will provide more information by Monday, 2024-06-17 16:30 US/Pacific.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-17 14:45 US/Pacific.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.