Incident began at 2024-03-05 15:25 and ended at 2024-03-05 17:22 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 05 March, 2024 07:25

Incident End: 05 March, 2024 09:22

Duration: 1 hour, 57 minutes

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

Gmail experienced 0.23% UI error rates for 61 minutes, and 1 to 5 minute email delays for just under two hours.

From preliminary analysis, the trigger of the issue was a sudden and unexpected increase in authentication traffic which was caused by user re-login attempts following device restarts. The device restarts were a result of users attempting to mitigate a third party application issue outside of Google’s infrastructure. This additional load led to elevated errors in our backend authentication service, which in turn impacted Gmail.

Google engineers responded promptly to the increased error rates and mitigated the issue by adding compute resources to handle the additional load. By 08:26 AM US/Pacific, Gmail UI error rates were largely resolved, and the error rate returned to baseline by 08:51. Delivery delays continued until 09:22 at which point the impact to Gmail was completely mitigated.

Customer Impact:

• Affected customers would have noticed an “oops” error message in the Gmail UI. Customers encountered these errors for various actions on Gmail, including login.

• Some customers would have experienced delays with email delivery ranging between one and five minutes.

Affected products: Gmail

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 16:51 (times are in Coordinated Universal Time (UTC)).

We saw a surge in traffic starting around 7:25 AM Pacific Time on March 5, and we scaled up our systems to serve the additional load.

Affected products: Gmail

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 16:51 (times are in Coordinated Universal Time (UTC)).

Our team is continuing to investigate this issue. We will provide an update by Mar 5, 2024, 5:15 PM UTC with more information about this problem. Thank you for your patience.

Symptoms:

Some Gmail users are experiencing elevated error rates while performing various actions and may also see delays in email delivery

Affected products: Gmail

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 16:51 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Gmail. We will provide more information shortly.

Affected products: Gmail

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
• Improving the process for blocking problematic Apps Script projects

Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Calendar

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
• Improving the process for blocking problematic Apps Script projects

Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Sheets

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
• Improving the process for blocking problematic Apps Script projects

Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Forms

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
• Improving the process for blocking problematic Apps Script projects

Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Docs

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Chat

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: ChromeOS

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Calendar

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Drive

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Meet

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Tasks

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Docs

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Cloud Search

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Voice

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

Between 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, multiple Google Workspace products experienced propagation delays for enterprise customer policy changes for a duration of 3 days, 3 hours, and 30 minutes.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google Workspace products expose settings through the Admin Console that Customer Admins can group together to create policies that govern how their users interact with Workspace products and services.

When a Customer Admin makes changes to policy settings in Admin Console, those changes are processed by a compilation server, which computes effective settings for the applicable entities. This process includes optimizations to avoid the repeated processing of multiple changes that affect the same set of entities.

On 13 February 2024 at approximately 07:30 PT, an unusual pattern of policy changes triggered a latent issue where compilation optimizations did not work as intended, resulting in increased resource consumption, eventually leading to resource exhaustion. The diversion of resources away from useful work caused an increasing number of compilations to accumulate in the queue and consume resources, further exacerbating the latency.

Remediation and Prevention

Google engineers were alerted to the issue via internal alerting on 13 February 2024 at 14:27 PT and immediately started an investigation. Engineers rolled back recent binary changes for the compilation process as a potential mitigation on 14 February 2024 at 15:35 PT, however this did not improve latency.

Engineers continued to attempt further mitigation steps throughout the day on 15 February 2024, including pausing non-essential processing. Once the cause of the issue was identified at 20:23 PT, engineers put in place temporary measures to increase available memory for compilation while keeping non-essential processing paused, and tuned parameters that drive compilation optimizations. These temporary mitigations allowed compilation processing to begin to recover.

Compilation processing returned to pre-outage levels by 16 February 2024, at 11:00 PT.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• We have fixed the underlying issue that caused compilation optimizations to not work as intended as of 21 February 2024.
• Introduce additional optimizations to improve compilation queue processing to better handle scenarios where a large number of changes affect the same set of entities.

Detailed Description of Impact

Between 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, Google Workspace policy updates initiated by customer admins via Admin Console would have experienced delays in propagation.

Affected products: Gmail

Incident began at 2024-02-28 12:23 and ended at 2024-02-28 13:41 (times are in Coordinated Universal Time (UTC)).

The problem with Google Voice has been resolved. After further investigation, we believe the issue existed outside of Google's infrastructure. No further updates will be added to this dashboard. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.

Affected products: Google Voice

Incident began at 2024-02-28 12:23 and ended at 2024-02-28 13:41 (times are in Coordinated Universal Time (UTC)).

We are continuing to investigate this issue. We will provide an update by Feb 28, 2024, 4:00 PM UTC detailing when we expect to resolve the problem. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.

• Placing and receiving the calls that are external to the caller may fail in France.
• The calls within the customer company network will work.

Affected products: Google Voice