Feed aggregator

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 09:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

None at this time...

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-03 19:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

None at this time...

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-03 17:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

None at this time...

Affected products: ChromeOS

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:41 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On 24 May 2024 at 16:20, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. A subset of users experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through multiple Google Workspace products including Google Docs, Google Drive, Google Sheets for a period of 4 hours, 21 minutes.

Root Cause

In ongoing efforts to ensure very strict security controls for Google Drive, our engineers performed changes to remove unused system-level access to Google Drive; i.e., stale Access Control List (ACLs) groups that are no longer in use.

Upon performing the change to remove the unused ACLs groups for Google Drive API, this triggered an inadvertent removal of access groups that are still in use. This deletion of the access groups caused a failure in loading the UI.

Google engineers mitigated the issue quickly by rolling back the change.

Remediation and Prevention

Google engineers were alerted to the outage via internal alerts and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers rolled back the changes and all the access removed this change was restored. While the rollback was still in progress Google engineers were able to partially mitigate the issue with a configuration change in a security policy.

The following changes will be made to ensure that this type of issue does not happen again in the future:

• Augment access control safety checks to prevent the inadvertent removal of in-use access, even in complex access inheritance scenarios.
• Review and redesign Drives access groups to remove nested ACLs (and prevent adding new nested ACLs). This will reduce the complexity of our access control graph and enable easier detection of potential problems.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On 24 May 2024, from 16:20 to 20:41, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. As a result, a subset of users of multiple Google Workspace products like Google Docs, Google Drive, Google Sheets experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through the products.

Google Drive

A small subset of Google Drive users experienced failure in accessing Drive Admin Console. As a result, the impacted users were unable to manage their shared drives on 24 May 2024 between 16:20 and 20:02.

Google Docs, Sheets(Apps script)

AppsScript UI was inaccessible for editing but AppsScript execution was unaffected starting at 16:12 to 20:41.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 and ended at 2024-05-30 17:11 (times are in Coordinated Universal Time (UTC)).

We expect to resolve the problem affecting a majority of users of Google Drive at May 30, 2024, 7:30 PM UTC. Please note that this time frame is an estimate and may change.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 and ended at 2024-05-30 17:11 (times are in Coordinated Universal Time (UTC)).

We have determined that this is an isolated issue that will be corrected as part of our ongoing maintenance activities.

The incident with Google Drive was initially triggered by our internal monitoring systems.

Upon further investigation, our engineering teams has confirmed that the scope is very limited and there was no impact to the users of Google Drive.

If you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 and ended at 2024-05-30 17:11 (times are in Coordinated Universal Time (UTC)).

The problem with Google Drive has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Drive. We will provide more information shortly.

Customer might see 5XX errors while using Google Drive.

Affected products: Google Drive

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 May, 2024 16:20

Incident End: 24 May, 2024 20:41

Duration: 4 hours, 21 minutes

Affected Services and Features: Apps Script usage from Google Drive, Google Docs, Google Sheets

Regions/Zones: Global

Description:

The user interface for Apps Script was failing to load for a duration of 4 hours, 21 minutes.

Based on the preliminary analysis the root cause of the issue is a configuration change. The change to the access control list that inadvertently removed access for some processes, has been updated to normalcy.

Google will complete a full incident report in the following days that will provide more details on the root cause.

Customer Impact: The impacted users were unable to develop, manage and execute their script using the Apps Script editor UI. Users would have encountered similar errors while trying to open Apps Script from any of the workspace products as well. However, there was no impact on other executions that did not originate from UI like executions from published scripts, triggers etc.

Affected products: Google Drive

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

For more service information please follow

https://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG

Affected products: Google Docs

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

For more service information please follow

https://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG

Affected products: Google Sheets

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Google Workspace products experienced failures in loading Apps Script editor UI.

Description:

The issues with Apps Script editor UI for multiple Google Workspace products have been resolved for all affected users as of Friday, 2024-05-24 20:41 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: Google Drive

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Google Workspace products are experiencing failures in loading Apps Script editor UI.

Description:

We are experiencing issues with Apps Script editor UI for multiple Google Workspace products beginning on Friday, 2024-05-24 16:20 US/Pacific.

The users trying to access the Apps Script UI may experience failures in loading the Apps Script UI.

Our engineering team has rolled out the fix for all the impacted users related to Apps Script UI issue and we started observing the service recovery.

We will provide an update by Friday, 2024-05-24 22:30 US/Pacific with current details.

The Google Drive UI service degradation has been resolved as of Friday, 2024-05-24 20:02 US/Pacific.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Multiple Google workspace users would experience errors or failures while loading the Apps Script UI. As a result, the impacted users are unable to develop, manage and execute their script using the Apps Script editor UI. However, there is no impact on other execution of deployed scripts.

Workaround:

None at this time.

Affected products: Google Drive

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

For more service information please follow

https://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG

Affected products: Google Docs

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

For more service information please follow

https://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG

Affected products: Google Sheets

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Google Workspace products are experiencing failures in loading Apps Script editor UI.

Description:

We are experiencing issues with Apps Script editor UI for multiple Google Workspace products beginning on Friday, 2024-05-24 16:20 US/Pacific.

The users trying to access the Apps Script UI may experience failures in loading the Apps Script UI.

Our engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.

We will provide an update by Friday, 2024-05-24 22:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Multiple Google workspace users would experience errors or failures while loading the Apps Script UI. As a result, the impacted users are unable to access or manage their script within the Apps Script. However, there is no impact on the execution of scripts.

Google Drive - a small subset of users may experience service degradation with Drive UI.

Workaround:

None at this time.

Affected products: Google Drive

Incident began at 2024-05-24 11:20 (times are in Coordinated Universal Time (UTC)).

Summary:

Google Drive users are unable to load Apps Script UI.

Description:

We are experiencing an issue with Google Drive beginning on Friday, 2024-05-24 16:20 US/Pacific.

Our engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.

We will provide an update by Friday, 2024-05-24 21:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Customers may not be able to load Drive UI, they would encounter error messages while trying to do so. . This issue may also impact users trying to access Apps Script UI.

Workaround:

None at this time.

Affected products: Google Drive

Incident began at 2024-05-02 15:53 and ended at 2024-05-02 19:02 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.omg

(All Times US/Pacific)

Incident Start: 2 May 2024 08:53

Incident End: 2 May 2024 12:02

Duration: 3 hours, 9 minutes

Affected Services and Features:

Google Meet Broadcast via YouTube Live

Regions/Zones: Global

Description:

Google Meet experienced streaming playback issues for Meet Broadcast via YouTube Live for 3 hours, 9 minutes. From preliminary analysis, the root cause of the issue was a recent rollout for YouTube Live that inadvertently impacted Meet Broadcast.

Engineers rolled back the change to mitigate the issue. For active streams that were affected by this issue, the host would need to restart the stream for the mitigation to take effect.

Customer Impact:

• Users attempting to join a live stream that was using Meet Broadcast via YouTube Live would have been stuck at the message “Waiting for streaming to begin. Please stand by.”
• Live streams utilizing Meet Live were not affected.

Affected products: Google Meet

Incident began at 2024-05-02 15:53 and ended at 2024-05-02 19:02 (times are in Coordinated Universal Time (UTC)).

The issue with Google Meet has been resolved for all affected users.

For new streams, there is no problem now. For broken streams, the stream host can stop and restart the stream to mitigate the issue.

We thank you for your patience while we worked on resolving the issue.

Affected products: Google Meet

Incident began at 2024-05-02 15:53 and ended at 2024-05-02 19:02 (times are in Coordinated Universal Time (UTC)).

Google Meet customers using Broadcast via YouTube Live are experiencing playback issues beginning at Thursday, 2024-05-02 08:53 US/Pacific.

Our engineering team continues to investigate the issue.

We apologize to all who are affected by the disruption.

Affected products: Google Meet