Incident Report

Summary

Starting on 17 April 2023 at 12:04, some Google Meet Hardware devices failed to launch the Meet app and the devices displayed a “Waiting for application window” message for a duration of 7 days, 19 hours and 34 minutes. To our Google Meet Hardware customers who were affected we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The device requires a healthy network state and a fully initialized Meet application in order to launch. When the affected devices started up after a reboot, certain conditions caused a network check to fail. Under these circumstances, the device displayed a “Waiting for application window” message indefinitely. Devices that were automatically upgraded to the next version of ChromeOS were more susceptible to these conditions.

Remediation and Prevention

Google engineers were able to identify a fix and deploy to the beta fleet of production on 21 April 2023 at 07:24 US/Pacific (within 24 hours of reproducing the issue). Google engineers instructed affected customers, who wanted to expedite receiving the fix, to move their devices to the Beta release channel. The incident was mitigated after the fix was released to the stable channel on 25 April 2023 at 07:38 US/Pacific.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future, by completing the following actions :

• Fix the underlying issue in the application launcher to prevent it from getting in a stuck state
• Improve logging, monitoring, and alerting for the fleet so that issues can be identified quickly

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

Beginning from 17 April 2023 at 12:04 US/Pacific, multiple customers of Google Meet Hardware devices, with Operating System (OS) versions greater than 14816.84.1 but lower than 14816.84.35, began to encounter issues in device startup following the upgrade.

At its peak, Google engineers were able to identify that around ~5000 devices had stopped, which caused the admins to receive an “offline device” alert for each of those devices.

These devices were frozen with “Waiting for application window” and would eventually go to sleep if left idle. The devices needed to be rebooted manually (sometimes with mitigation steps involving disconnecting the ethernet cable) or waiting for the device to reboot during a maintenance window.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 17 April 2023 at 09:00

Incident End: 25 April 2023 at 07:38

Duration: 7 days, 22 hours, 38 minutes

Affected Services and Features:

Google Meet

Regions/Zones: Global

Description:

After an Operating System (OS) update, some Google Meet Hardware units were unable to boot into Meet reliably for a duration of 7 days, 22 hours and 38 minutes.

From preliminary analysis, the root cause of the issue was a latent issue that manifested due to specific network conditions on customer networks following an upgrade to the OS. Affected OS versions were those greater than 14816.84.1 but lower than 14816.84.35.

Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

Impacted Google Meet hardware devices failed to launch the Meet app. Devices displayed a “Waiting for application window” message that never resolved, eventually went to sleep, and were shown as offline in the Google Admin Console. Some customers had to perform a manual reboot (occasionally several times), to overcome this issue.

Additional details:

Customers were provided with a temporary workaround with specific steps to overcome the issue. Google engineers also released a fix to the Beta channel on 21 April 2023 at 07:24 US/Pacific, while the workaround was being fully tested and rolled out to Stable fleet.

Our engineering team has completed the mitigation, and the fix has been deployed to 100% of the Stable release channel as of Tuesday, 2023-04-25 07:38 US/Pacific. We expect that after devices have been updated, subsequent reboots of the device should not experience the issue.

Customers on the Stable release channel will automatically receive this update. However, devices must be rebooted for the changes to take effect. This may take up to 7 days, depending on the maintenance window.

We kindly ask customers who have switched their rooms to the Beta channel to switch them back to the Stable channel.

For more information on the best practices and how to change between Beta and Stable channels is available here : https://support.google.com/a/answer/4618201?hl=en&ref_topic=3341450&visit_id=638176138740742021-1958539996&rd=1#settings&zippy=%2Cchange-settings-for-all-devices

We apologize to all who were affected by the disruption

Mitigation work is currently underway by our engineering team. The fix has been deployed to 50% of the Stable release channel as of Monday, 2023-04-24. We plan to continue the rollout to 100% of the fleet on Tuesday 2023-04-25 18:00 US/Pacific after we have ensured a successful roll out to 50%. After devices have been updated, subsequent reboots of the device should not experience the issue.

Customers will not need to take additional measures to receive this update as long as they remain on the Stable release channel.

We request that any customers that did opt to switch their rooms to the Beta channel should switch them back to Stable channel to ensure the devices remain on the stable release track.

More information on how to change between Beta and Stable channels is available here : https://support.google.com/a/answer/4618201?hl=en&ref_topic=3341450&visit_id=638176138740742021-1958539996&rd=1#settings&zippy=%2Cchange-settings-for-all-devices

Our engineering team has also identified a workaround for devices that are not able to get past the "Waiting for application window" screen after a reboot. Please see the workaround for details.

We will provide more information by Tuesday, 2023-04-25 18:30 US/Pacific

We apologize to all who are affected by the disruption

Workaround :

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue.

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

The fix has been deployed to 10% of the Stable release channel on Friday, 2023-04-21. We plan to continue the rollout to the rest of the fleet on Monday, 2023-04-24 after we have ensured a successful roll out to 10%. After devices have been updated, subsequent reboots of the device will not experience the issue.

Customers will not be required to take additional steps to receive this update as long as the devices remain on the Stable release channel. We request that any customers that did opt to switch their rooms to the Beta channel should switch them back to Stable channel to ensure the devices remain on the stable release track.

More information on how to change between Beta and Stable channels is available here : https://support.google.com/a/answer/4618201?hl=en&ref_topic=3341450&visit_id=638176138740742021-1958539996&rd=1#settings&zippy=%2Cchange-settings-for-all-devices

We will provide more information by Monday, 2023-04-24 12:00 US/Pacific.

We apologize to all who are affected by the disruption

Workaround :

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue.

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

The fix has now been deployed to the Beta release channel and available to customers. Customers wishing to expedite the update can switch impacted rooms to the Beta release channel which typically takes 2-4 hours to propagate after which time the subsequent reboots of the device should not experience the issue.

We plan to release this fix to the Stable release channel the week of April 24th after we have ensured a successful roll out to Beta.

We request any customers that opt to switch their rooms to the Beta channel should switch them back to Stable channel after the update to ensure the devices remain on the stable release track.

More information on how to change between Beta and Stable channels is available here : https://support.google.com/a/answer/4618201?hl=en&ref_topic=3341450&visit_id=638176138740742021-1958539996&rd=1#settings&zippy=%2Cchange-settings-for-all-devices

We will provide more information by Friday, 2023-04-21 14:00 US/Pacific

Workaround :

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue.

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

Mitigation work is currently underway by our engineering team. We have a fix that is currently being tested by our team, and plan to release it to the Beta channel by Thursday, 2023-04-20 18:00 US/Pacific. Customers wishing to expedite the update can switch impacted rooms to the Beta release channel. This update typically takes 2-4 hours to update after the switch to the Beta release channel, and would require a restart after this time period.

We plan to release this fix to the Stable release channel the week of April 24th after we have ensured a successful roll out to Beta.

We request any customers who opt to switch their rooms to the Beta channel to switch them back to Stable channel after the update, to ensure the devices remain back on the stable release process

More information on how to change between Beta and Stable channels is available here : https://support.google.com/a/answer/4618201?hl=en&ref_topic=3341450&visit_id=638176138740742021-1958539996&rd=1#settings&zippy=%2Cchange-settings-for-all-devices

Our engineering team has also identified a workaround for devices that are not able to get past the "Waiting for application window" screen after a reboot. Please refer to the workaround section for details.

We will provide more information by Thursday, 2023-04-20 18:00 US/Pacific

We apologize to all who are affected by the disruption

Workaround :

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue.

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

Mitigation work is currently underway by our engineering team. We have a fix that is currently being tested by our team, and plan to release it to the Beta channel by the end of the day. Customers will be able to switch any critical rooms that need to receive the update to the Beta release channel.

We request any customers that do opt to switch their rooms to the Beta channel to switch them back to Stable after the update, thereby ensuring they remain back on the stable release process.

We plan to release this fix to the Stable release channel the week of April 24th after we have ensured a successful roll out to Beta.

Our engineering team has also identified a workaround for devices that are not able to get past the "Waiting for application window" screen after a reboot. Please refer to the workaround section for details.

We will provide more information by Thursday, 2023-04-20 18:00 US/Pacific.

Workaround

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue.

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

Mitigation work is currently underway by our engineering team. A fix has been developed and it is currently being validated.

We do not have an ETA for mitigation at this point.

Our engineering team has also identified a workaround for devices that are not able to get past the "Waiting for application window" screen after a reboot. Please refer to the workaround section for details.

We will provide more information by Thursday, 2023-04-20 09:00 US/Pacific.

We apologize to all who are affected by the disruption

Workaround

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue. Steps:

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

Our engineering team investigation has revealed that the root cause of the issue is related to sensitivity of network availability during bootup and are currently working on a fix for the retry logic.

Our engineering team has also identified a workaround for devices that are not able to get past the "Waiting for application window" screen after a reboot. Please refer to the workaround section for details.

We will provide an update by Thursday, 2023-04-20 06:00 US/Pacific with current details.

We apologize to all who are affected by the disruption

Workaround

Customers affected with this issue may try the following steps for mitigation if the reboot does not fix the issue. Steps:

1. Unplug network cable on the Meet Hardware device.
2. Reboot the device.
3. Wait until it shows the network connectivity screen (this is the screen that prompts the user to select a wifi connection).
4. Plug in network cable back into the Meet hardware device.
5. Device should recover after this step is completed.

Google Meet Hardware devices are displaying the "Waiting for application window" notification upon reboot.

Our engineering team believes the issue is related to local network availability post reboot following an OS update and has determined that stable network availability is needed during the initial boot post OS update. Further investigation is required to mitigate the issue under all network conditions.

Our engineering team is actively working on verifying the root-cause and resolving the issue. Meanwhile, we have also paused the pending OS updates and expect this to not impact any more devices.

We will provide an update by Wednesday, 2023-04-19 15:30 US/Pacific with current details.

We apologize to all who are affected by the disruption

Workaround : Customers may try rebooting their devices to see if this resolves the issue

Google Meet Hardware devices are displaying the "Waiting for application window" notification upon reboot.

Our engineering team believes the issue is related to networking issue surfacing post rebooting following an OS update and has determined that further investigation is required to mitigate the issue.

Our engineering team is actively working on root-causing and resolving the issue.

We will provide an update by Wednesday, 2023-04-19 13:30 US/Pacific with current details.

We apologize to all who are affected by the disruption

Workaround : Customers may try rebooting their devices to see if this resolves the issue

Chromebox for Meet devices are displaying the "Waiting for application window" notification upon reboot.

Our engineering team believes some devices may have gone to sleep due to Chrome OS policies and has determined that further investigation is required to mitigate the issue.

Our engineering team is actively working on root-causing and resolving the issue.

We will provide an update by Wednesday, 2023-04-19 11:30 US/Pacific with current details.

We apologize to all who are affected by the disruption

Workaround : Customers may try rebooting their devices to see if this resolves the issue

Our engineering team believes some devices may have gone to sleep due to Chrome OS policies and has determined that further investigation is required to mitigate the issue.

We will provide an update by Wednesday, 2023-04-19 10:30 US/Pacific with current details.

Chromebox for Meet devices are displaying the "Waiting for application window" notification upon reboot.

Workaround : Customers may try rebooting their devices to see if this resolves the issue