Feed aggregator

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

The problem with Apps Script has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Apps Script

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Summary: Global: Elevated App Script Failures

Description: We experienced an issue with Apps Script beginning at Tuesday, 2024-06-11 06:04 to 06:41 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-06-11 08:40 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers impacted by this issue may have observed 500 errors and were unable to load / run App Scripts.

Workaround: None at this time.

Affected products: Apps Script

Incident began at 2024-06-03 18:36 and ended at 2024-06-04 23:00 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 3 June 2024 11:36

Incident End: 4 June 2024 16:00

Duration: 1 Day, 4 Hours and 24 Minutes

Affected Services and Features:

ChromeOS

Regions/Zones: Global

Description:

A subset of Google ChromeOS users experienced persistent sign-in failures when attempting to access their locked devices on ChromeOS version 125 using ephemeral mode for the period of 1 Day, 4 Hours and 24 Minutes.

Based on the preliminary analysis the root cause of the sign-in failure was a conflict between a recent local password feature release and the ephemeral mode authentication process on ChromeOS. This conflict inadvertently terminated the authentication session prematurely, preventing users from logging into their devices.

To mitigate the issue and to restore normal functionality, Google engineers manually disabled the problematic feature flag. Subsequent verification confirmed successful login attempts for the affected users.

Customer Impact:

The impacted users of ChromeOS version M125 with the "DeviceEphemeralUsersEnabled" device policy activated would have experienced persistent login failures when attempting to access their locked devices. This issue was specifically observed when utilizing ephemeral mode and prevented users from successfully logging back into their ChromeOS devices.

Additional details:

The issue has been mitigated as of 4 June 2024 at 16:00. Google strongly recommends that all impacted users upgrade to ChromeOS version 125.0.6422.140 for enhanced stability.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 and ended at 2024-06-04 23:00 (times are in Coordinated Universal Time (UTC)).

Summary

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description

The issue with ChromeOS has been resolved for all affected users as of Tuesday, 2024-06-04 16:00 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 and ended at 2024-06-04 23:00 (times are in Coordinated Universal Time (UTC)).

SUMMARY

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

DESCRIPTION

Our engineering team is continuing to work towards mitigation of this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 19:00 US/Pacific.

CUSTOMER SYMPTOMS

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

WORKAROUND

ChromeOS customers to use https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock and https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Our engineering team is continuing to work towards mitigation of this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 17:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

ChromeOS customers to use

https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock

and

https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Our engineering team is continuing to work towards mitigation of this issue. We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 13:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

ChromeOS customers to use

https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock

and

https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2024-06-04 09:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

ChromeOS customers to use https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock and https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 09:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

None at this time...

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-03 19:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

None at this time...

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-03 17:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

None at this time...

Affected products: ChromeOS

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:41 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On 24 May 2024 at 16:20, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. A subset of users experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through multiple Google Workspace products including Google Docs, Google Drive, Google Sheets for a period of 4 hours, 21 minutes.

Root Cause

In ongoing efforts to ensure very strict security controls for Google Drive, our engineers performed changes to remove unused system-level access to Google Drive; i.e., stale Access Control List (ACLs) groups that are no longer in use.

Upon performing the change to remove the unused ACLs groups for Google Drive API, this triggered an inadvertent removal of access groups that are still in use. This deletion of the access groups caused a failure in loading the UI.

Google engineers mitigated the issue quickly by rolling back the change.

Remediation and Prevention

Google engineers were alerted to the outage via internal alerts and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers rolled back the changes and all the access removed this change was restored. While the rollback was still in progress Google engineers were able to partially mitigate the issue with a configuration change in a security policy.

The following changes will be made to ensure that this type of issue does not happen again in the future:

• Augment access control safety checks to prevent the inadvertent removal of in-use access, even in complex access inheritance scenarios.
• Review and redesign Drives access groups to remove nested ACLs (and prevent adding new nested ACLs). This will reduce the complexity of our access control graph and enable easier detection of potential problems.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On 24 May 2024, from 16:20 to 20:41, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. As a result, a subset of users of multiple Google Workspace products like Google Docs, Google Drive, Google Sheets experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through the products.

Google Drive

A small subset of Google Drive users experienced failure in accessing Drive Admin Console. As a result, the impacted users were unable to manage their shared drives on 24 May 2024 between 16:20 and 20:02.

Google Docs, Sheets(Apps script)

AppsScript UI was inaccessible for editing but AppsScript execution was unaffected starting at 16:12 to 20:41.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 and ended at 2024-05-30 17:11 (times are in Coordinated Universal Time (UTC)).

We expect to resolve the problem affecting a majority of users of Google Drive at May 30, 2024, 7:30 PM UTC. Please note that this time frame is an estimate and may change.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 and ended at 2024-05-30 17:11 (times are in Coordinated Universal Time (UTC)).

We have determined that this is an isolated issue that will be corrected as part of our ongoing maintenance activities.

The incident with Google Drive was initially triggered by our internal monitoring systems.

Upon further investigation, our engineering teams has confirmed that the scope is very limited and there was no impact to the users of Google Drive.

If you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 and ended at 2024-05-30 17:11 (times are in Coordinated Universal Time (UTC)).

The problem with Google Drive has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Google Drive

Incident began at 2024-05-30 14:23 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Drive. We will provide more information shortly.

Customer might see 5XX errors while using Google Drive.

Affected products: Google Drive

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 May, 2024 16:20

Incident End: 24 May, 2024 20:41

Duration: 4 hours, 21 minutes

Affected Services and Features: Apps Script usage from Google Drive, Google Docs, Google Sheets

Regions/Zones: Global

Description:

The user interface for Apps Script was failing to load for a duration of 4 hours, 21 minutes.

Based on the preliminary analysis the root cause of the issue is a configuration change. The change to the access control list that inadvertently removed access for some processes, has been updated to normalcy.

Google will complete a full incident report in the following days that will provide more details on the root cause.

Customer Impact: The impacted users were unable to develop, manage and execute their script using the Apps Script editor UI. Users would have encountered similar errors while trying to open Apps Script from any of the workspace products as well. However, there was no impact on other executions that did not originate from UI like executions from published scripts, triggers etc.

Affected products: Google Drive

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

For more service information please follow

https://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG

Affected products: Google Docs

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

For more service information please follow

https://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG

Affected products: Google Sheets

Incident began at 2024-05-24 23:20 and ended at 2024-05-25 03:20 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Google Workspace products experienced failures in loading Apps Script editor UI.

Description:

The issues with Apps Script editor UI for multiple Google Workspace products have been resolved for all affected users as of Friday, 2024-05-24 20:41 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: Google Drive