Incident Report

SUMMARY:

From 06 July 2022 20:36 to 07 July 2022 00:08 US/Pacific, newly delivered Gmail messages received excessive banner notifications. Affected messages displayed an informational banner notification with the text “Do you want to continue receiving messages from this sender?” These notifications were removed from all affected messages by 09 July 2022 13:31.

We understand that this issue may have been confusing to our valued customers and users, and we apologize to those who were affected. We have completed an internal investigation and are taking immediate steps to prevent this issue from recurring.

ROOT CAUSE:

While making updates to Gmail’s configuration, Google engineers inadvertently released a change more broadly than intended, which triggered the excess banners.

REMEDIATION AND PREVENTION:

Google engineers were alerted to the issue on 06 July 2022 at 23:02 US/Pacific and quickly identified the problematic configuration. Google engineers reverted this configuration on 07 July 2022 00:08, which prevented the banner from being added to additional messages. They then retroactively removed the banner from all affected messages, finishing by 09 July 2022 13:31.

Google is committed to quickly and continually improving our technology and operations to enhance the user experience. We would like to apologize for the length and severity of this incident. We are taking the following steps to prevent a recurrence and improve our service’s reliability in the future:

• We are improving our alerting systems to help us respond to issues more quickly and effectively.
• We are improving our release procedures to ensure that all changes are rolled out incrementally.
• We are improving our internal tools so that we can produce more timely mitigations.

Mini Incident

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 06 July 2022 20:36

Incident End: 09 July 2022 13:31

Duration: 2 days 16, hours, 55 minutes

Affected Services and Features:

Gmail

Regions/Zones:

Global

Description:

Gmail users were shown redundant banner notifications from 06 July 2022 20:36 to 09 July 2022 13:31 for emails delivered in their inbox timestamped from 06 July 2022 20:36 through 07 July 2022 00:08. Affected users would have been shown a banner at the top of emails received during that time, asking them if they want to continue getting emails from those senders. From preliminary analysis, the root cause of the issue was a code change in a Gmail subsystem that incorrectly determined which messages to display the banner for.

Customer Impact:

Affected users were getting redundant banner notifications on some emails asking if they want to continue getting emails from some senders. There was no negative impact if the users interacted with the banner

We are experiencing an issue with Gmail wherein emails delivered between 2022-07-06 20:36 PDT - 2022-07-07 00:08 PDT have a redundant banner notification in the users' inbox.

We have confirmed through further investigation that this issue with enterprise customers has been resolved while Gmail consumer users still can be affected. New emails delivered post the impact time also will not have any banner.

Our engineering team has identified the root cause, prevented any further affected messages, and continues to work on fixing the emails that have this incorrect banner.

We will provide an update by Saturday, 2022-07-09 17:00 US/Pacific with current details.

We have confirmed through further investigation that this issue affects both Gmail consumers and Enterprise customers. New emails delivered post the impact time will not have any banner.

Our engineering team has identified the root cause, prevented any further affected messages, and continues to work on removing the labels from the emails that have this incorrect banner.

We will provide an update by Saturday, 2022-07-09 11:00 US/Pacific with current details.

We have confirmed through further investigation that this issue affects both Gmail consumers and Enterprise customers. New emails delivered post the impact time will not have any banner.

Our engineering team has identified the root cause, prevented any further affected messages, and continues to work on removing the labels from the emails that have this incorrect banner.

We will provide an update by Friday, 2022-07-08 12:00 US/Pacific with current details.

We are experiencing an issue with Gmail wherein emails delivered between 2022-07-06 20:36 PDT - 2022-07-07 00:08 PDT have a redundant banner notification in the users' inbox.

Known impact is limited to Gmail consumers and not to Enterprise customers. New emails delivered post the impact time also will not have any banner.

Users will see redundant banner notifications on emails delivered in their inbox between the impacted time frame, asking them if they want to continue getting emails from those senders. Users may have to click 'yes' or 'no' on each banner displayed.

Our engineering team has identified the root cause, prevented any further affected messages, and continues to work on removing the labels from the emails that have this incorrect banner.

We will provide an update by Friday, 2022-07-08 12:00 US/Pacific with current details.

We are experiencing an issue with Gmail wherein emails delivered between 2022-07-06 20:36 PDT - 2022-07-07 00:08 PDT has a redundant banner notification in the users' inbox.

Impact is limited to Gmail consumers and not to Enterprise customers. Also, new emails delivered post the impacted time shall not have this incorrect banner.

Users, however, will see redundant banner notifications on emails delivered in their inbox between the impacted time frame, asking them if they want to continue getting emails from those senders. Users may have to click 'yes' or 'no' on each of the banner displayed to clear them off.

While our engineering team continues to work with utmost priority on removing the labels from the emails that has this incorrect banner, we also confirm that users have been receiving emails, and they are not marked as spam or phishing.

We will provide an update by Thursday, 2022-07-07 16:00 US/Pacific with current details.

We are experiencing an issue with Gmail wherein emails delivered between 2022-07-06 20:36 PDT - 2022-07-07 00:08 PDT has a redundant banner notification in the users' inbox.

Impact is limited to Gmail consumers and not to Enterprise customers. New emails delivered post the impact time will not have this incorrect banner.

Our engineering team continues to work on removing the labels from the emails that has this incorrect banner.

We will provide an update by Thursday, 2022-07-07 13:30 US/Pacific with current details.

We are experiencing an issue with Gmail beginning at Thursday, 2022-07-06 20:36 US/Pacific. Impact is limited to Gmail consumers and not to Enterprise customers who are paying for Workspace.

Users will be receiving redundant banner notifications on delivered emails in their inbox asking them if they want to continue getting emails from some senders.

Users might have to click 'yes' or 'no' on each of the banner displayed.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2022-07-07 11:35 US/Pacific with current details.