Incident Report

Summary

On 17 April 2023 at 11:43 US/Pacific, some Google Docs and Google Slides users experienced issues with actions involving images. During this time, users creating new documents may have received ‘unavailability’ errors when attempting to upload images and ‘500’ errors when creating new documents from templates that included images.

This is not the level of quality and reliability we strive to offer users, and we took immediate steps to remediate the issue and prevent it from happening again.

Root Cause

A recent change to a backend storage policy triggered a validation failure in the system which caused some users to receive error messages when attempting the following actions:

• Creating a presentation based on a template which contains images/thumbnails
• Copying a document with images to their own account
• Inserting embedded images into any Editor

Remediation and Prevention

On Monday, 17 April 2023 at 11:43 US/Pacific, Google engineers were alerted to an issue by internal monitoring systems and immediately started an investigation. On 17 April 2023 at 13:54 US/Pacific, Google engineers mitigated the issue by rolling back the backend storage policy change. Users were advised that images uploaded while the issue was ongoing would remain broken and needed to be reinserted.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Optimize the validation process and add automation for validation of directory policies for Drive with Docs and Slides to prevent a recurrence.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 17 April 2023 11:43

Incident End: 17 April 2023 13:47

Duration: 2 hours, 4 minutes

Affected Services and Features:

Google Docs, Google Slides

Regions/Zones: Global

Description:

Google Docs and Google Slides experienced issues with actions involving images for a duration of 2 hours, 4 minutes. From preliminary analysis, the root cause of the issue was a recent change to a back-end storage policy. Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

Users may have experienced issues when performing actions involving images, including:

• Uploaded images failing to render and displaying an exclamation mark
• 500 errors when creating new documents from templates that included images

There was no impact to already existing Docs or Slides.

Additional details:

The issue was mitigated by rolling back the back-end storage policy changes at 13:47 PDT. Images uploaded while the issue was ongoing would remain broken and need to be reinserted.