Feed aggregator

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Chat

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Drive

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Calendar

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Meet

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-17 14:45 US/Pacific.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Meet

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Calendar

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Drive

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Chat

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Tuesday, 11 June 2024, Apps Script experienced increased error rates and users may have experienced intermittent issues in script creation and execution for a duration of 37 minutes. In addition to this, any of the Workspace services that internally use Apps Script may have encountered issues during the period of impact. We sincerely apologize to all Apps Script users. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

One of the scripts received an unexpectedly high number of the queries per second (QPS) which resulted in hotspotting for our database service (a database hotspot is created because a single server receives most or all of the writes, which overloads that one server). This led to deadline exceeded errors, thereby increasing the error rate in our remote procedure calls (RPCs) and leading to the script creation and execution issues.

Remediation and Prevention

Google engineers were alerted to the outage by the internal monitoring systems that sent multiple alerts starting 06:25 US/Pacific (approximately 15 mins after issue started) and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly enforced limits on read actions for the affected script and then blocked the script which led to returning database hotspotting and error rate in our RPCs back to their original values.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Enable caching on read replicas for App Info to reduce the number of requests being sent to our database service.
• Adding limits and alerts for all RPCs to ensure a limit on the QPS that one add on can send.
• Reduce the number of read replicas to avoid loading App Info on multiple replicas.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Tuesday, 11 June 2024 from 06:04 US/Pacific to 06:41 US/Pacific Apps Script experienced increased error rates (error rate in our RPCs increased to ~20% whilst the expected error rate is ~1.3%.

Apps Script

Affected users experienced intermittent issues in script creation and execution for a duration of their app script applications receiving DEADLINE_EXCEEDED error status.

Affected products: Apps Script

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 11 June 2024 06:04

Incident End: 11 June 2024 06:41

Duration: 37 minutes

Affected Services and Features:

Apps Script

Regions/Zones: Global

Description:

Apps Script experienced increased error rates for a duration of 37 minutes. From preliminary analysis, the root cause of the issue was due to an unexpected increase in traffic related to a third-party add-on that resulted in resource contention.

Engineers temporarily blocked the third-party add-on that was the cause of the increased traffic to mitigate the issue.

Customer Impact:

Customers may have received errors when attempting to load or run Apps Scripts.

Affected products: Apps Script

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

The problem with Apps Script has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Apps Script

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Summary: Global: Elevated App Script Failures

Description: We experienced an issue with Apps Script beginning at Tuesday, 2024-06-11 06:04 to 06:41 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-06-11 08:40 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers impacted by this issue may have observed 500 errors and were unable to load / run App Scripts.

Workaround: None at this time.

Affected products: Apps Script

Incident began at 2024-06-03 18:36 and ended at 2024-06-04 23:00 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 3 June 2024 11:36

Incident End: 4 June 2024 16:00

Duration: 1 Day, 4 Hours and 24 Minutes

Affected Services and Features:

ChromeOS

Regions/Zones: Global

Description:

A subset of Google ChromeOS users experienced persistent sign-in failures when attempting to access their locked devices on ChromeOS version 125 using ephemeral mode for the period of 1 Day, 4 Hours and 24 Minutes.

Based on the preliminary analysis the root cause of the sign-in failure was a conflict between a recent local password feature release and the ephemeral mode authentication process on ChromeOS. This conflict inadvertently terminated the authentication session prematurely, preventing users from logging into their devices.

To mitigate the issue and to restore normal functionality, Google engineers manually disabled the problematic feature flag. Subsequent verification confirmed successful login attempts for the affected users.

Customer Impact:

The impacted users of ChromeOS version M125 with the "DeviceEphemeralUsersEnabled" device policy activated would have experienced persistent login failures when attempting to access their locked devices. This issue was specifically observed when utilizing ephemeral mode and prevented users from successfully logging back into their ChromeOS devices.

Additional details:

The issue has been mitigated as of 4 June 2024 at 16:00. Google strongly recommends that all impacted users upgrade to ChromeOS version 125.0.6422.140 for enhanced stability.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 and ended at 2024-06-04 23:00 (times are in Coordinated Universal Time (UTC)).

Summary

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description

The issue with ChromeOS has been resolved for all affected users as of Tuesday, 2024-06-04 16:00 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 and ended at 2024-06-04 23:00 (times are in Coordinated Universal Time (UTC)).

SUMMARY

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

DESCRIPTION

Our engineering team is continuing to work towards mitigation of this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 19:00 US/Pacific.

CUSTOMER SYMPTOMS

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

WORKAROUND

ChromeOS customers to use https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock and https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Our engineering team is continuing to work towards mitigation of this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 17:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

ChromeOS customers to use

https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock

and

https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Our engineering team is continuing to work towards mitigation of this issue. We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-06-04 13:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

ChromeOS customers to use

https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock

and

https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS

Incident began at 2024-06-03 18:36 (times are in Coordinated Universal Time (UTC)).

Summary:

ChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.

Description:

Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2024-06-04 09:00 US/Pacific.

Diagnosis:

Affected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.

Workaround:

ChromeOS customers to use https://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock and https://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction policies to prevent occasional screen locking.

Affected products: ChromeOS