Incident Report

Summary

On 24 May 2024 at 16:20, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. A subset of users experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through multiple Google Workspace products including Google Docs, Google Drive, Google Sheets for a period of 4 hours, 21 minutes.

Root Cause

In ongoing efforts to ensure very strict security controls for Google Drive, our engineers performed changes to remove unused system-level access to Google Drive; i.e., stale Access Control List (ACLs) groups that are no longer in use.

Upon performing the change to remove the unused ACLs groups for Google Drive API, this triggered an inadvertent removal of access groups that are still in use. This deletion of the access groups caused a failure in loading the UI.

Google engineers mitigated the issue quickly by rolling back the change.

Remediation and Prevention

Google engineers were alerted to the outage via internal alerts and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers rolled back the changes and all the access removed this change was restored. While the rollback was still in progress Google engineers were able to partially mitigate the issue with a configuration change in a security policy.

The following changes will be made to ensure that this type of issue does not happen again in the future:

• Augment access control safety checks to prevent the inadvertent removal of in-use access, even in complex access inheritance scenarios.
• Review and redesign Drives access groups to remove nested ACLs (and prevent adding new nested ACLs). This will reduce the complexity of our access control graph and enable easier detection of potential problems.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On 24 May 2024, from 16:20 to 20:41, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. As a result, a subset of users of multiple Google Workspace products like Google Docs, Google Drive, Google Sheets experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through the products.

Google Drive

A small subset of Google Drive users experienced failure in accessing Drive Admin Console. As a result, the impacted users were unable to manage their shared drives on 24 May 2024 between 16:20 and 20:02.

Google Docs, Sheets(Apps script)

AppsScript UI was inaccessible for editing but AppsScript execution was unaffected starting at 16:12 to 20:41.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 May, 2024 16:20

Incident End: 24 May, 2024 20:41

Duration: 4 hours, 21 minutes

Affected Services and Features: Apps Script usage from Google Drive, Google Docs, Google Sheets

Regions/Zones: Global

Description:

The user interface for Apps Script was failing to load for a duration of 4 hours, 21 minutes.

Based on the preliminary analysis the root cause of the issue is a configuration change. The change to the access control list that inadvertently removed access for some processes, has been updated to normalcy.

Google will complete a full incident report in the following days that will provide more details on the root cause.

Customer Impact: The impacted users were unable to develop, manage and execute their script using the Apps Script editor UI. Users would have encountered similar errors while trying to open Apps Script from any of the workspace products as well. However, there was no impact on other executions that did not originate from UI like executions from published scripts, triggers etc.

Summary:

Multiple Google Workspace products experienced failures in loading Apps Script editor UI.

Description:

The issues with Apps Script editor UI for multiple Google Workspace products have been resolved for all affected users as of Friday, 2024-05-24 20:41 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary:

Multiple Google Workspace products are experiencing failures in loading Apps Script editor UI.

Description:

We are experiencing issues with Apps Script editor UI for multiple Google Workspace products beginning on Friday, 2024-05-24 16:20 US/Pacific.

The users trying to access the Apps Script UI may experience failures in loading the Apps Script UI.

Our engineering team has rolled out the fix for all the impacted users related to Apps Script UI issue and we started observing the service recovery.

We will provide an update by Friday, 2024-05-24 22:30 US/Pacific with current details.

The Google Drive UI service degradation has been resolved as of Friday, 2024-05-24 20:02 US/Pacific.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Multiple Google workspace users would experience errors or failures while loading the Apps Script UI. As a result, the impacted users are unable to develop, manage and execute their script using the Apps Script editor UI. However, there is no impact on other execution of deployed scripts.

Workaround:

None at this time.

Summary:

Multiple Google Workspace products are experiencing failures in loading Apps Script editor UI.

Description:

We are experiencing issues with Apps Script editor UI for multiple Google Workspace products beginning on Friday, 2024-05-24 16:20 US/Pacific.

The users trying to access the Apps Script UI may experience failures in loading the Apps Script UI.

Our engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.

We will provide an update by Friday, 2024-05-24 22:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Multiple Google workspace users would experience errors or failures while loading the Apps Script UI. As a result, the impacted users are unable to access or manage their script within the Apps Script. However, there is no impact on the execution of scripts.

Google Drive - a small subset of users may experience service degradation with Drive UI.

Workaround:

None at this time.

Summary:

Google Drive users are unable to load Apps Script UI.

Description:

We are experiencing an issue with Google Drive beginning on Friday, 2024-05-24 16:20 US/Pacific.

Our engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.

We will provide an update by Friday, 2024-05-24 21:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Customers may not be able to load Drive UI, they would encounter error messages while trying to do so. . This issue may also impact users trying to access Apps Script UI.

Workaround:

None at this time.