SUMMARY

On 25 August 2021 at 09:40 US/Pacific Google Sites Classic and Google Sites New experienced intermittent elevated 404 errors and/or pages failed to load for a duration of 12 hours and 20 minutes. During the disruption up to 13.7% of users were impacted, the 500’s response rate increased from 0.01% to 0.3%, and the 400’s response rate increased from 6% to 14%. 20% of users accessing sites.google.com got served intermittent 404 during this period.This impacted users who typically navigate to sites.google.com and expect to be forwarded to sites.google.com/new (the domain homescreen), but they instead received 404s. Additionally, users who tried to visit their domain's landing page or sites.google.com/classic also received 404 responses. During the incident users were able to access the home screen by simply refreshing the page and some users who visited sites.google.com/new directly were able to access the home screen without issue. We have conducted an internal investigation and are taking steps to improve our service.

ROOT CAUSE

In 2018, a redirect was added to the Google Sites homepage to make Google Site /new the default unless it was specifically disabled on a Google Workspace Domain. The trigger was the addition of a new cleanup workflow which detected sites with obsolete domain ownership IDs, and soft deleted them. The root cause of this issue was due to the soft deletion, causing issues with the logic that handles the loading sites.google.com homepage.

REMEDIATION AND PREVENTION

We appreciate your patience and apologize again for the impact on your organization. Google became aware of the issue via an automated alert from a monitoring system that identified an increase in errors causing user impact starting on Wednesday, 25 August 2021 09:40 US/Pacific. The Google Sites team immediately began investigating and after inspecting logs, were able to identify that the issue was caused by a misconfiguration of a running clean up job. Next, they tested and deployed a rollback which fully restored service on 25 August 2021 23:39 and the error rate was reduced to its normal range.

Google is committed to preventing recurrence of this issue. The following action items are currently being implemented:

• Remove Google Sites redirect logic from Classic to New as this dependency is obsolete.

• Improve monitoring to more easily gauge end-user impact

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact on your organization. We thank you for your business.

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 25 August 2021 09:40 US/Pacific

Incident End: 25 August 2021 22:00 US/Pacific

Duration: 14 hours, 12 minutes

Affected Services and Features:

• Google Sites Classic
• Google Sites New

Description: Google Sites Classic and the new Google Sites experienced intermittent elevated 404 errors and/or pages failing to load for a duration of 14 hours and 12 minutes. During the disruption up to 13.7% of users were impacted, the 500’s response rate increased from 0.01% to 0.3%, and the 400’s response rate increased from 6% to 14%. The root cause was identified as a misconfigured cleanup process. Completing a rollback restored the affected sites, and the error rate returned to normal.

Customer Impact:

• Users visiting sites.google.com sometimes received a 404 error when they should have been redirected to the new Sites Homescreen.
• Some users received a 404 error when they visited the Classic Sites landing page.