RESOLVED: We are investigating issues with Google Cloud Search starting at 07:23 PT on 13-February-2024. There is no workaround at this time.

Google Apps Status - 28 February, 2024 - 09:31

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Cloud Search

Categories: IT News

RESOLVED: We are investigating issues with Google Voice starting at 07:23 PT on 13-February-2024. There is no workaround at this time.

Google Apps Status - 28 February, 2024 - 09:30

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

A detailed incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV

Affected products: Google Voice

Categories: IT News

RESOLVED: We are investigating issues with Gmail starting at 07:23 PT on 13-February-2024. There is no workaround at this time

Google Apps Status - 28 February, 2024 - 09:27

Incident began at 2024-02-13 15:30 and ended at 2024-02-16 19:00 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

Between 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, multiple Google Workspace products experienced propagation delays for enterprise customer policy changes for a duration of 3 days, 3 hours, and 30 minutes.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google Workspace products expose settings through the Admin Console that Customer Admins can group together to create policies that govern how their users interact with Workspace products and services.

When a Customer Admin makes changes to policy settings in Admin Console, those changes are processed by a compilation server, which computes effective settings for the applicable entities. This process includes optimizations to avoid the repeated processing of multiple changes that affect the same set of entities.

On 13 February 2024 at approximately 07:30 PT, an unusual pattern of policy changes triggered a latent issue where compilation optimizations did not work as intended, resulting in increased resource consumption, eventually leading to resource exhaustion. The diversion of resources away from useful work caused an increasing number of compilations to accumulate in the queue and consume resources, further exacerbating the latency.

Remediation and Prevention

Google engineers were alerted to the issue via internal alerting on 13 February 2024 at 14:27 PT and immediately started an investigation. Engineers rolled back recent binary changes for the compilation process as a potential mitigation on 14 February 2024 at 15:35 PT, however this did not improve latency.

Engineers continued to attempt further mitigation steps throughout the day on 15 February 2024, including pausing non-essential processing. Once the cause of the issue was identified at 20:23 PT, engineers put in place temporary measures to increase available memory for compilation while keeping non-essential processing paused, and tuned parameters that drive compilation optimizations. These temporary mitigations allowed compilation processing to begin to recover.

Compilation processing returned to pre-outage levels by 16 February 2024, at 11:00 PT.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • We have fixed the underlying issue that caused compilation optimizations to not work as intended as of 21 February 2024.
  • Introduce additional optimizations to improve compilation queue processing to better handle scenarios where a large number of changes affect the same set of entities.
Detailed Description of Impact

Between 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, Google Workspace policy updates initiated by customer admins via Admin Console would have experienced delays in propagation.

Affected products: Gmail

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Voice. We will provide more information shortly.

Google Apps Status - 28 February, 2024 - 09:23

Incident began at 2024-02-28 12:23 and ended at 2024-02-28 13:41 (times are in Coordinated Universal Time (UTC)).

The problem with Google Voice has been resolved. After further investigation, we believe the issue existed outside of Google's infrastructure. No further updates will be added to this dashboard. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.

Affected products: Google Voice

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Voice. We will provide more information shortly.

Google Apps Status - 28 February, 2024 - 06:44

Incident began at 2024-02-28 12:23 and ended at 2024-02-28 13:41 (times are in Coordinated Universal Time (UTC)).

We are continuing to investigate this issue. We will provide an update by Feb 28, 2024, 4:00 PM UTC detailing when we expect to resolve the problem. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.
  • Placing and receiving the calls that are external to the caller may fail in France.
  • The calls within the customer company network will work.

Affected products: Google Voice

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Voice. We will provide more information shortly.

Google Apps Status - 28 February, 2024 - 05:56

Incident began at 2024-02-28 12:23 (times are in Coordinated Universal Time (UTC)).

Our team is continuing to investigate this issue. We will provide an update by Feb 28, 2024, 3:00 PM UTC with more information about this problem. Thank you for your patience. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.

Placing and receiving calls may fail in France.

Affected products: Google Voice

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Voice. We will provide more information shortly.

Google Apps Status - 28 February, 2024 - 05:09

Incident began at 2024-02-28 12:23 (times are in Coordinated Universal Time (UTC)).

We're aware of a problem with Google Voice affecting a significant subset of users. We will provide an update by Feb 28, 2024, 2:00 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.

Placing and receiving calls may fail in France.

Affected products: Google Voice

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Voice. We will provide more information shortly.

Google Apps Status - 28 February, 2024 - 05:08

Incident began at 2024-02-28 12:23 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Voice. We will provide more information shortly.

Placing and receiving calls may fail in France.

Affected products: Google Voice

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 15:33

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard. https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6

Affected products: Google Calendar

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 15:32

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6

Affected products: Google Sheets

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 15:31

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6

Affected products: Google Docs

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 15:27

Incident began at 2024-02-26 09:42 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 26 Feb, 2024 1:42

Incident End: 26 Feb, 2024 13:02

Duration: 11 hours, 20 minutes

Affected Services and Features:

Workspace product integrations with Apps Script

Regions/Zones: Global

Description:

Multiple Google Workspace products experienced intermittent failures while attempting various operational actions for the incident’s duration of 11 hours, 20 minutes. From preliminary analysis, the root cause of the issue appears to be due to a queued backlog which led to the performance delays. Google will complete a full Incident Report in the following days that will provide a detailed root cause.

Customer Impact:

During the incident time frame customers may have experienced the following product impact:

  • Apps Script: Apps Script is returning errors on creation or loading of scripts
  • Google Calendar: Calendar 3rd party video conferencing integrations are impacted
  • Google Forms: Custom Functions using Apps Script could be impacted
  • Google Sheets, Google Docs: Add-ons which are built on Apps Script in Google Sheets and Google Docs can also be impacted

Other Workspace product integrations with Apps Script may have also encountered elevated error rate. The impact includes any third party integrations to Apps Script.

Affected products: Google Forms

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 13:27

Incident began at 2024-02-26 08:11 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.

We received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

Affected products: Google Calendar

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 13:26

Incident began at 2024-02-26 08:11 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.

We received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

Affected products: Google Sheets

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 13:25

Incident began at 2024-02-26 08:11 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.

We received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

Affected products: Google Docs

Categories: IT News

RESOLVED: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 13:21

Incident began at 2024-02-26 09:38 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.

We received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

Affected products: Google Forms

Categories: IT News

UPDATE: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 12:21

Incident began at 2024-02-26 09:38 and ended at 2024-02-26 21:02 (times are in Coordinated Universal Time (UTC)).

We believe the issue is mostly resolved for impacted services. Our engineers have implemented a new mitigation to prevent recurrence of this issue and are showing signs of recovery.

We are investigating a few reports of triggers/API related errors which may be impacting a small number of users.

We will provide an update by Monday, 2024-02-26 13:20 US/Pacific with current details.

Affected products: Google Forms

Categories: IT News

UPDATE: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 11:09

Incident began at 2024-02-26 09:38 (times are in Coordinated Universal Time (UTC)).

We believe the issue is mostly resolved for impacted services. Our engineers have implemented a new mitigation to prevent recurrence of this issue in the future which is showing signs of improvement.

We will provide an update by Monday, 2024-02-26 12:10 US/Pacific with current details.

Affected products: Google Forms

Categories: IT News

UPDATE: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 10:08

Incident began at 2024-02-26 09:38 (times are in Coordinated Universal Time (UTC)).

We believe the issue is mostly resolved for impacted services. Mitigation work is ongoing and our engineers are continuing to work on preventing similar issues in the future.

We will provide an update by Monday, 2024-02-26 10:30 US/Pacific with current details.

Affected products: Google Forms

Categories: IT News

UPDATE: We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific....

Google Apps Status - 26 February, 2024 - 09:36

Incident began at 2024-02-26 08:11 (times are in Coordinated Universal Time (UTC)).

We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.

For further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6

Affected products: Google Calendar

Categories: IT News