Incident Report

Summary:

On Monday, 18 July 2022 at 08:30 US/Pacific, Google Cloud and Google Workspace Support customers were unable to attach files to new and existing support cases via the Cloud Console and Admin Console for a period of 5 hours and 30 minutes.

To our support customers who were impacted during this outage, we sincerely apologize, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause:

Google Cloud and Google Workspace Support customers have the ability to attach files to new or existing support cases. During the initial phase of the outage, some users experienced intermittent failures while uploading files.

During a recent effort to improve the reliability of the Cloud Support backend service, a configuration change was made to our Global Service Load Balancer (GSLB), which resulted in unavailability of the file upload service. Part of this change included migrating the servers to new addresses; however, the server that handles file attachments was not ready for the migration. The configuration change inadvertently made the file upload service used by the backend service inaccessible, resulting in file uploads failing.

Remediation and Prevention:

Once the nature and scope of the issue became clear, Google engineers quickly disabled the file attachment feature while they worked on mitigation efforts. The issue was fully mitigated on Monday, 18 July 2022 at 14:00 US/Pacific, once the GSLB configuration change was rolled back.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We are committed to the following actions to prevent this from happening again:

• Improve monitoring and alerting to detect file attachment errors.
• Improve our load balancer emergency change procedures based on known best practices within Google to reduce mitigation time.
• Improve automation testing to detect invalid configuration changes in staging environments.
• Improve internal rollout procedure for load balancer changes in order to detect rollouts that have unanticipated impact and prevent them from proceeding.

We would like to apologize for the length and severity of this incident.

Detailed Description of Impact:

Service(s) Affected:

Google Cloud Console

• 100% of users could not upload files to new or existing support cases. Support case creation was not impacted.

Google Admin Console

• 100% of users could not upload files to new or existing support cases. Support case creation was not impacted.

Mini-IR (Preliminary Incident Report)

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 18 July 2022 08:30

Incident End: 18 July 2022 14:00

Duration: 5 hours, 30 minutes

Affected Services and Features:

Google Cloud Support - Cloud Console Google Workspace Support - Admin Console

Regions/Zones: Global

Description: Google Cloud Support and Google Admin Console customers were unable to attach files to new and existing support cases for a period of 5 hours and 30 minutes. From preliminary analysis, the root cause of the issue was due to an inadvertent change to our Global Service Load Balancer (GSLB), which resulted in nonavailability of the upload service.

Customer Impact:

• Google Cloud Support: Customers were unable to attach files to new or existing support cases.
• Google Workspace Support: Customers were unable to attach files to new or existing support cases.

Google Admin Console customers are unable to attach files to new and existing support cases.