Google Workspace Status Dashboard
Incident affecting Apps Script
Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).
Date | Time | Description | |
---|---|---|---|
| Jun 13, 2024 | 6:34 PM UTC | Incident ReportSummaryOn Tuesday, 11 June 2024, Apps Script experienced increased error rates and users may have experienced intermittent issues in script creation and execution for a duration of 37 minutes. In addition to this, any of the Workspace services that internally use Apps Script may have encountered issues during the period of impact. We sincerely apologize to all Apps Script users. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Root CauseOne of the scripts received an unexpectedly high number of the queries per second (QPS) which resulted in hotspotting for our database service (a database hotspot is created because a single server receives most or all of the writes, which overloads that one server). This led to deadline exceeded errors, thereby increasing the error rate in our remote procedure calls (RPCs) and leading to the script creation and execution issues. Remediation and PreventionGoogle engineers were alerted to the outage by the internal monitoring systems that sent multiple alerts starting 06:25 US/Pacific (approximately 15 mins after issue started) and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly enforced limits on read actions for the affected script and then blocked the script which led to returning database hotspotting and error rate in our RPCs back to their original values. Google is committed to preventing a repeat of this issue in the future and is completing the following actions:
Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business. Detailed Description of ImpactOn Tuesday, 11 June 2024 from 06:04 US/Pacific to 06:41 US/Pacific Apps Script experienced increased error rates (error rate in our RPCs increased to ~20% whilst the expected error rate is ~1.3%. Apps ScriptAffected users experienced intermittent issues in script creation and execution for a duration of their app script applications receiving DEADLINE_EXCEEDED error status. |
| Jun 11, 2024 | 5:48 PM UTC | Mini Incident ReportWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 11 June 2024 06:04 Incident End: 11 June 2024 06:41 Duration: 37 minutes Affected Services and Features: Apps Script Regions/Zones: Global Description: Apps Script experienced increased error rates for a duration of 37 minutes. From preliminary analysis, the root cause of the issue was due to an unexpected increase in traffic related to a third-party add-on that resulted in resource contention. Engineers temporarily blocked the third-party add-on that was the cause of the increased traffic to mitigate the issue. Customer Impact: Customers may have received errors when attempting to load or run Apps Scripts. |
| Jun 11, 2024 | 3:20 PM UTC | The problem with Apps Script has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. |
| Jun 11, 2024 | 3:18 PM UTC | Summary: Global: Elevated App Script Failures Description: We experienced an issue with Apps Script beginning at Tuesday, 2024-06-11 06:04 to 06:41 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2024-06-11 08:40 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers impacted by this issue may have observed 500 errors and were unable to load / run App Scripts. Workaround: None at this time. |
- Times are listed in Coordinated Universal Time (UTC)