Google Apps Status

Subscribe to Google Apps Status feed
Updated: 1 hour 56 min ago

RESOLVED: **Summary:**...

13 March, 2024 - 14:26

Incident began at 2024-03-06 22:32 and ended at 2024-03-06 23:29 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Wednesday, 06 March 2024, Google Workspace users experienced elevated errors while accessing user or device setting pages within the Chrome Admin Console for a duration of 1 hour, 27 minutes.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google Workspace Admin Console uses a backend authentication service to grant privileged access to internal accounts. As part of ongoing efforts to migrate the account authorization workflow to a new backend authentication service, Google engineers identified and cleaned up invalid account role assignments in the configuration.

During this clean up effort, engineers identified a privilege which was undefined in the application manifest and removed it from the configuration. Upon further investigation, engineering confirmed the privilege was mapped to an access control setting used by the Admin Console for Chrome. This change resulted in users being unable to access Chrome user and device settings page.

Remediation and Prevention

Google engineers were alerted to the outage via an internal monitoring alert on Wednesday, 06 March 2024 at 15:04 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers initiated a rollback of the configuration change at 15:36 and the incident was fully resolved at 15:59 once the rollback completed.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Improve procedural documentation on how to safely cleanup access control settings from authorization configuration.
  • Complete thorough investigation of privilege usage for the Admin Console to prevent further issues of this type.
  • Identify and register a new owner for this privilege in the application manifest and create permission mapping for faster verification in the future.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Wednesday, 06 March 2024 from 14:32 to 15:59 US/Pacific, Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console.

Affected products: Admin Console

Categories: IT News

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

12 March, 2024 - 12:52

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 17:22 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Tuesday, 5 March 2024, starting at 07:25 US/Pacific, 0.23% of Gmail UI requests were failing for a duration of 61 minutes and 1.1% of email messages experienced up to five minute delays for a duration of 1 hour, 57 minutes.

Root Cause

This report includes a revised version of root cause for the service issue Gmail customers experienced on 5 March 2024.

After further investigation, we identified that a sudden and unexpected surge of email traffic initiated the issue, rather than an increase in authentication traffic as shared in the previous report.

Gmail frontend services HTTP, POP, or IMAP use an internal lookup service to route the requests to the specific backend instances that contain the user data. When this internal lookup service fails, users experience issues with Gmail UI interactions.

The same internal lookup service is also consulted during email delivery. When the lookup requests fail during email delivery, the requests are retried with an exponential backoff and this would materialize as delayed emails from an end-user perspective.

On 5 March 2024 at 07:25 US/Pacific, Gmail began experiencing a sudden and unexpected increase in real user traffic that coincided with a third-party service outage outside of Google’s infrastructure. At the time of the traffic increase, the internal lookup service in some geo locations was overloaded, which resulted in errors for a small fraction of Gmail UI requests and delayed email message delivery.

Remediation and Prevention

Google engineers were alerted to the increase in traffic via internal systems on 5 March 2024 at 07:32 US/Pacific and immediately started an investigation. Once the scope and nature of the issue was identified, at 08:11 US/Pacific, Google engineers redirected traffic away from the affected locations. This immediately stopped the Gmail UI errors, and prevented delivery delays for new messages.

By 09:22 US/Pacific the backlog of emails in the delivery pipeline had cleared completely.

If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Enhance Gmail’s lookup server traffic surge protection layer to be able to better handle these kinds of traffic surges.
Detailed Description of Impact

On 5 March 2024, between 07:25 and 09:22 US/Pacific, Gmail experienced increased error rates and delivery delays. During this time:

  • Approximately 0.23% of requests in the Gmail UI resulted in an error message. Affected customers encountered these errors for various actions on Gmail, including login.
  • Approximately 1.1% of email messages experienced delivery delays ranging between one and five minutes.

Affected products: Gmail

Categories: IT News

RESOLVED: **Summary:**...

7 March, 2024 - 13:06

Incident began at 2024-03-06 22:32 and ended at 2024-03-07 00:29 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 6 March 2023 14:32

Incident End: 6 March 2023 16:29

Duration: 1 hours, 57 minutes

Affected Services and Features:

  • Admin Console Pages for Chrome user settings
  • Chrome device settings
  • Chrome devices

Regions/Zones: Global

Description:

Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console. From preliminary analysis, the root cause of the issue is a change to the access control setting in a backend authentication service used by the Admin Console. This change resulted in users being unable to access the page.

Our Engineering team was alerted by our internal monitoring systems and swiftly mitigated the issue by rolling back the change to the access control settings.

This impacted users ability to view the current user and device policies, change or edit the existing policies and their ability to create new policies.

Google will complete a full Incident Report (IR) in the following days that will provide a detailed root cause.

Customer Impact:

  • The impacted customers encountered a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console.
  • Additionally, customers were unable to view or change any Chrome policies.

Affected products: Admin Console

Categories: IT News

RESOLVED: **Summary:**...

6 March, 2024 - 16:39

Incident began at 2024-03-06 22:50 and ended at 2024-03-07 00:29 (times are in Coordinated Universal Time (UTC)).

Summary:

Admin Console experienced elevated errors with the setting pages.

Description:

The issue with Admin Console has been resolved for all affected users as of Wednesday, 2024-03-06 16:29 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Symptoms:

The impacted customers would have encountered a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers were unable to view or change any policies.

Workaround:

None at this time...

Affected products: Admin Console

Categories: IT News

UPDATE: **Summary:**...

6 March, 2024 - 16:05

Incident began at 2024-03-06 10:50 (times are in Coordinated Universal Time (UTC)).

Summary:

Admin Console is experiencing elevated errors with the Chrome setting pages.

Description:

Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2024-03-06 16:15 US/Pacific.

We will provide more information by Wednesday, 2024-03-06 16:45 US/Pacific.

Customer Symptoms:

The impacted customers would encounter a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers would not be able to view or change any policies.

Workaround:

None at this time.

Affected products: Admin Console

Categories: IT News

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

5 March, 2024 - 16:19

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 17:22 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 05 March, 2024 07:25

Incident End: 05 March, 2024 09:22

Duration: 1 hour, 57 minutes

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

Gmail experienced 0.23% UI error rates for 61 minutes, and 1 to 5 minute email delays for just under two hours.

From preliminary analysis, the trigger of the issue was a sudden and unexpected increase in authentication traffic which was caused by user re-login attempts following device restarts. The device restarts were a result of users attempting to mitigate a third party application issue outside of Google’s infrastructure. This additional load led to elevated errors in our backend authentication service, which in turn impacted Gmail.

Google engineers responded promptly to the increased error rates and mitigated the issue by adding compute resources to handle the additional load. By 08:26 AM US/Pacific, Gmail UI error rates were largely resolved, and the error rate returned to baseline by 08:51. Delivery delays continued until 09:22 at which point the impact to Gmail was completely mitigated.

Customer Impact:

  • Affected customers would have noticed an “oops” error message in the Gmail UI. Customers encountered these errors for various actions on Gmail, including login.

  • Some customers would have experienced delays with email delivery ranging between one and five minutes.

Affected products: Gmail

Categories: IT News

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

5 March, 2024 - 09:34

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 16:51 (times are in Coordinated Universal Time (UTC)).

We saw a surge in traffic starting around 7:25 AM Pacific Time on March 5, and we scaled up our systems to serve the additional load.

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

5 March, 2024 - 08:35

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 16:51 (times are in Coordinated Universal Time (UTC)).

Our team is continuing to investigate this issue. We will provide an update by Mar 5, 2024, 5:15 PM UTC with more information about this problem. Thank you for your patience.

Symptoms:

Some Gmail users are experiencing elevated error rates while performing various actions and may also see delays in email delivery

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

5 March, 2024 - 08:25

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 16:51 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Gmail. We will provide more information shortly.

Affected products: Gmail

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

29 February, 2024 - 15:46

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

  • Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
  • Improving the process for blocking problematic Apps Script projects
Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Calendar

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

29 February, 2024 - 15:45

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

  • Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
  • Improving the process for blocking problematic Apps Script projects
Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Sheets

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

29 February, 2024 - 15:45

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

  • Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
  • Improving the process for blocking problematic Apps Script projects
Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Forms

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

29 February, 2024 - 15:44

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.

This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.

Remediation and Prevention

Google engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

  • Adding additional system limits to prevent a single Apps Script project from creating overload scenarios
  • Improving the process for blocking problematic Apps Script projects
Detailed Description of Impact

On Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:

Apps Script

Apps Script encountered errors during the creation or loading of scripts.

Google Calendar

A subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.

Google Forms, Google sheets and Google docs

Custom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.

Affected products: Google Docs

Categories: IT News

RESOLVED: We are investigating issues with Google Chat starting at 07:23 PT on 13-February-2024. There is no workaround at this time.

28 February, 2024 - 09:35

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Chat

Categories: IT News

RESOLVED: We are investigating issues with ChromeOS starting at 07:23 PT on 13-February-2024

28 February, 2024 - 09:34

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: ChromeOS

Categories: IT News

RESOLVED: We are investigating issues with Google Calendar starting at 07:23 PT on 13-February-2024. There is no workaround at this time

28 February, 2024 - 09:34

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Calendar

Categories: IT News

RESOLVED: We are investigating issues with Google Drive starting at 07:23 PT on 13-February-2024. There is no workaround at this time.

28 February, 2024 - 09:34

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Drive

Categories: IT News

RESOLVED: We are investigating issues with Google Meet starting at 07:23 PT on 13-February-2024. There is no workaround at this time

28 February, 2024 - 09:34

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Meet

Categories: IT News

RESOLVED: We are investigating issues with Google Tasks starting at 07:23 PT on 13-February-2024. There is no workaround at this time.

28 February, 2024 - 09:33

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Tasks

Categories: IT News

RESOLVED: We are investigating issues with Google Docs starting at 07:23 PT on 13-February-2024. There is no workaround at this time

28 February, 2024 - 09:32

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Docs

Categories: IT News