INCIDENT REPORT

Summary:

On Monday, 25 April 2022 at 02:04 US/Pacific, some Google Meet Livestream view requests experienced an error message globally for a period of 2 days, 11 hours, and 30 minutes. Affected customers received the message “Waiting for stream to begin, Please stand by..." instead of the expected livestream. Customers were informed on our Public Status Dashboard (PSD) that they would be able to work around the issue by removing “stream” from the root of the livestream URL.

To our Google Meet customers whose livestreams were impacted during this service disruption, we sincerely apologize – this is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause:

Google Cloud Load Balancing is a collection of software and services that load balance traffic across Google properties. There are two main components: a control plane and a data plane. The control plane provides programming to the data plane on how to handle requests. A key component of the data plane is the Google Front End (GFE).

The GFE is an HTTP/TCP reverse proxy which is used to serve requests to many Google properties including Search, Ads, Workspace (Gmail, Chat, Meet, Docs, Drive, etc.), and many Cloud APIs. Updates are regularly rolled out to GFEs, typically via configuration flags, starting with canary GFEs and gradually expanding to production globally.

Google routinely deploys configuration changes to improve the performance and stability of GFE. On Thursday 21 April 2022, a GFE configuration change was rolled out to reduce complexity. GFE configuration rollouts are completed in waves and don’t immediately take effect.

The rollout updated the GFE scope for stream.meet.google.com but missed a required configuration mapping piece. This caused attempts to view livestreams that went through the affected GFEs to fail to load a static content URL with a 404 error. This resulted in the viewer seeing the following message instead of the expected livestream: "Waiting for stream to begin, Please stand by..."

Remediation and Prevention:

Google engineers were alerted to the issue through a customer support case on 26 April 2022 at 12:38 US/Pacific and immediately started an investigation.

Once the nature and scope of the issue became clear, engineers identified the root cause as the recent GFE configuration change and rolled it back. The configuration containing the rollback completed its push to the affected GFEs at 13:34 US/Pacific, and the issue was fully resolved at that time.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We are taking the following immediate steps to prevent this or similar issues from happening again:

• We reverted the configuration change on all GFEs to prevent a recurrence.
• We will implement regression tests of GFE configurations on Livestream service before rolling to production.
• We will improve monitoring to detect and notify when there are increased errors when accessing Meet services.

We sincerely apologize for the length and severity of this incident.

Detailed Description of Impact:

On Monday, 25 April 2022 from 02:04 to 27 April 2022 13:34 US/Pacific:

Google Meet Livestream

Some Google Meet Livestream customers experienced a "Waiting for stream to begin, Please stand by..." error message instead of the expected livestream. Customers at that time were able to workaround the issue by removing “stream” from the root of the livestream URL.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 25 April 2022 02:04

Incident End: 27 April 2022 13:34

Duration: 2 days, 11 hours, 30 minutes

Affected Services and Features:

Google Meet - Livestream Viewing

Regions/Zones: Global

Description:

Up to 30% of Google Meet Livestream view requests received an error message globally for a period of 2 days, 11 hours, 30 minutes. From preliminary analysis, the root cause was due to a recent GFE (Google Front End) configuration rollout. Impacted customers were able to work around the issue by removing “stream.” from the root of their GVC livestream URL.

Customer Impact:

Impacted customers would have experienced the following:

• Customers may have observed a "Waiting for stream to begin, Please stand by..." message even though the live stream already started.