Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Apps Script

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Jun 13, 2024 6:34 PM UTC

Incident Report

Summary

On Tuesday, 11 June 2024, Apps Script experienced increased error rates and users may have experienced intermittent issues in script creation and execution for a duration of 37 minutes. In addition to this, any of the Workspace services that internally use Apps Script may have encountered issues during the period of impact. We sincerely apologize to all Apps Script users. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

One of the scripts received an unexpectedly high number of the queries per second (QPS) which resulted in hotspotting for our database service (a database hotspot is created because a single server receives most or all of the writes, which overloads that one server). This led to deadline exceeded errors, thereby increasing the error rate in our remote procedure calls (RPCs) and leading to the script creation and execution issues.

Remediation and Prevention

Google engineers were alerted to the outage by the internal monitoring systems that sent multiple alerts starting 06:25 US/Pacific (approximately 15 mins after issue started) and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly enforced limits on read actions for the affected script and then blocked the script which led to returning database hotspotting and error rate in our RPCs back to their original values.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Enable caching on read replicas for App Info to reduce the number of requests being sent to our database service.
  • Adding limits and alerts for all RPCs to ensure a limit on the QPS that one add on can send.
  • Reduce the number of read replicas to avoid loading App Info on multiple replicas.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Tuesday, 11 June 2024 from 06:04 US/Pacific to 06:41 US/Pacific Apps Script experienced increased error rates (error rate in our RPCs increased to ~20% whilst the expected error rate is ~1.3%.

Apps Script

Affected users experienced intermittent issues in script creation and execution for a duration of their app script applications receiving DEADLINE_EXCEEDED error status.
Jun 11, 2024 5:48 PM UTC

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 11 June 2024 06:04

Incident End: 11 June 2024 06:41

Duration: 37 minutes

Affected Services and Features:

Apps Script

Regions/Zones: Global

Description:

Apps Script experienced increased error rates for a duration of 37 minutes. From preliminary analysis, the root cause of the issue was due to an unexpected increase in traffic related to a third-party add-on that resulted in resource contention.

Engineers temporarily blocked the third-party add-on that was the cause of the increased traffic to mitigate the issue.

Customer Impact:

Customers may have received errors when attempting to load or run Apps Scripts.
Jun 11, 2024 3:20 PM UTC The problem with Apps Script has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
Jun 11, 2024 3:18 PM UTC

Summary: Global: Elevated App Script Failures

Description: We experienced an issue with Apps Script beginning at Tuesday, 2024-06-11 06:04 to 06:41 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-06-11 08:40 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers impacted by this issue may have observed 500 errors and were unable to load / run App Scripts.

Workaround: None at this time.