Feed aggregator

Incident began at 2024-07-23 07:00 and ended at 2024-07-25 23:00 (times are in Coordinated Universal Time (UTC)).

Summary: Poly Studio X devices experienced unexpected deprovisioning and might require manual re-enrollment.

Description: Poly Studio X devices running Google Meet experienced unexpected deprovisioning between Tuesday, 2024-11-23 to Thursday, 2024-11-25.

Google engineers rolled out a software fix to mitigate the issue, however if the device had already got impacted and got to the activation code screen, they would require manual re-enrollment. Please refer to the workaround section for more information.

Diagnosis: The affected customer may encounter Poly Studio X devices offline/deprovisioned.

Workaround: Customer Action required: Each affected device has to be manually re-enrolled in the Admin Console using the displayed activation code, as part of the “Enroll your device (non-ChromeOS devices) steps. Following the re-enrollment, the devices will be reconfigured to restore their previous settings and its functionality.

Affected products: Google Meet

Incident began at 2024-07-23 07:00 and ended at 2024-07-25 23:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Poly Studio X devices experienced unexpected deprovisioning and might require manual re-enrollment.

Description:

Poly Studio X devices running Google Meet experienced unexpected deprovisioning between Tuesday, 2024-11-23 to Thursday, 2024-11-25.

Google engineers rolled out a software fix to mitigate the issue, however if the device had already got impacted and got to the activation code screen, they would require manual re-enrollment. Please refer to the workaround section for more information.

Diagnosis: The affected customer may encounter Poly Studio X devices offline/deprovisioned.

Workaround:

Customer Action required: Each affected device has to be manually re-enrolled in the Admin Console using the displayed activation code, as part of the “Enroll your device (non-ChromeOS devices) steps. Following the re-enrollment, the devices will be reconfigured to restore their previous settings and its functionality.

Affected products: Google Meet

Incident began at 2024-07-24 22:36 and ended at 2024-07-25 16:27 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 July, 2024 15:36

Incident End: 25 July, 2024 09:27

Duration: 17 hours, 51 minutes

Affected Services and Features:

Chrome Browser

Regions/Zones: Global

Description:

A significant number of users were unable to find or save passwords in the password manager for Chrome browser, for a period of 17 hours, 51 minutes.

From the preliminary analysis the root cause of the issue is a change in product behavior without proper feature guard. Google engineers mitigated the issue by deploying a fix.

Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

Impacted users were unable to find passwords in Chrome's password manager. Users can save passwords, however it was not visible to them. The impact was limited to the M127 version of Chrome Browser on the Windows platform.

Approximately 2% of users out of the 25% of the entire user base where the configuration change was rolled out, experienced this issue.

Additional details:

While an interim workaround was provided during the incident, after the fix was fully rolled out, users are being advised to restart their Chrome browser to ensure that the fix takes effect.

Affected products: Chrome Browser

Incident began at 2024-07-17 17:53 and ended at 2024-07-25 16:27 (times are in Coordinated Universal Time (UTC)).

The issue with Chrome Browser has been mitigated for all affected users as of Thursday, 2024-07-25 09:27 US/Pacific.

Users are advised to restart their Chrome Browser to ensure that the fix takes effect.

We thank you for your patience while we worked on resolving the issue.

Affected products: Chrome Browser

Incident began at 2024-07-17 17:53 (times are in Coordinated Universal Time (UTC)).

SUMMARY: Chrome Browser Password Manager Issue

We are experiencing an issue with Chrome Browser where impacted users may be unable to find or save passwords in Chrome's Password Manager. This impacts users on M127 version of Chrome Browser.

Our Engineering team has identified the cause of the issue and are actively working on deploying a fix for all users.

We will continue to monitor the situation as the fix is rolled out to ensure a smooth resolution.

We will provide more information by Thursday, 2024-07-25 09:45 US/Pacific.

WORKAROUND:

Users currently impacted by this can try the following:

Launch Chrome with " --enable-features=SkipUndecryptablePasswords" command line flag.

1. Locate Chrome shortcut on desktop. If you don't have it, go to chrome://settings/manageProfile and at the bottom enable "Create desktop shortcut".

2. Fully exit Chrome

3. Go to desktop and right click the shortcut then click Properties.

4. To the target field paste the following string to the end " --enable-features=SkipUndecryptablePasswords".

5. Close the dialog with OK and launch Chrome using this shortcut.

Affected products: Chrome Browser

Incident began at 2024-07-16 17:40 and ended at 2024-07-16 18:34 (times are in Coordinated Universal Time (UTC)).

Summary:

We have received reports of an issue with Gmail.

Description:

Upon further investigation, our engineering teams believe that the scope is very limited and/or no customers were impacted.

If you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved. No further updates will be provided here.

Affected products: Gmail

Incident began at 2024-07-16 17:40 and ended at 2024-07-16 18:34 (times are in Coordinated Universal Time (UTC)).

Summary:

We have received reports of an issue with Gmail.

Description:

Upon further investigation, our engineering teams believe that the scope is very limited and/or no customers were impacted.

If you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved. No further updates will be provided here.

Affected products: Gmail

Incident began at 2024-07-16 18:18 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Gmail. We will provide more information shortly.

Affected products: Gmail

Incident began at 2024-06-17 16:06 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Meet

Incident began at 2024-06-17 16:06 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Drive

Incident began at 2024-06-17 16:06 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Chat

Incident began at 2024-06-17 16:06 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Calendar

Incident began at 2024-06-17 16:06 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On 17 June 2024, a subset of our customers using Google Workspace products (Google Chat, Google Meet, Google Drive, Google Calendar, Google Cloud Directory and Google Docs) experienced elevated error rates and latency of varying impact between the period of 6 hours and 40 minutes between 09:06 and 15:46 US/Pacific.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The root cause is a latent issue that caused contention in getting access in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns.

Remediation and Prevention

Google engineers were alerted to the issue from our internal monitoring at 09:21 US/Pacific on 17 June 2024 and immediately started working on the incident. Once the nature and scope of the issue was identified, our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.

Google is committed preventing a repeat of this issue in the future and is completing the following actions :

• Change the underlying design that caused the contention in the cache so that the issue does recur.
• Implement mechanisms that facilitate quicker application of changes and detection of contention, to allow faster remediation.
• Update Google Chat to be more resilient to profile information unavailability.

Detailed Description of Impact

On Monday, 17 June 2024, six of our Workspace products were impacted by varying degrees between 09:06 and 15:46 US/Pacific. The details and timeframes for the impact to each product have been outlined below.

Google Calendar : Impacted users saw email addresses instead of names while using Calendar for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.

Google Chat : Impacted users experienced intermittent issues loading chat, opening chat rooms, and sending messages for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.

Google Meet : Impacted users sparsely experienced latency or failures while joining scheduled meetings over a period of 4 hours, between 11:00 to 15:00 US/Pacific on 17 June 2024.

Google Docs : Impacted users encountered the error "Name loading" in comments or other areas where authors’ names were expected for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.

Google Drive : Impacted users experienced elevated errors/latency while using third party APIs, for a duration of 6 hours 30 minutes from 09:06 to 15:36 US/Pacific on 17 June 2024.

Google Cloud Directory : Affected customers were unable to access the “User” section in the admin console. The initial dashboard displayed the error "Data can't be fetched due to system” for a period of 6 hours 11 minutes from 09:05 to 15:16 US/Pacific on 17 June 2024.

Affected products: Google Docs

Incident began at 2024-06-17 16:23 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Chat

Incident began at 2024-06-17 16:23 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Drive

Incident began at 2024-06-17 16:23 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Calendar

Incident began at 2024-06-17 16:23 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Meet

Incident began at 2024-06-17 16:23 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 17 June, 2024 09:23

Incident End: 17 June, 2024 15:46

Duration: 6 hours, 23 minutes

Affected Services and Features: Google Calendar, Google Chat, Google Docs, Google Drive & Google Meet.

Regions/Zones: Global

Description:

Multiple Google Workspace products experienced elevated error rates and latency. From preliminary analysis, the root cause is a latent issue that caused lock contention in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns. Our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.

Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

• Google Chat - Affected users would have experienced intermittent latency and errors while loading chats, accessing the chat spaces and sending messages.
• Google Meet - Affected users would have experienced latency or failures while joining meetings.
• Google Drive - Affected users would have experienced latency/errors while using the third party API.
• Google Docs - Affected users would have experienced latency loading comments on the document. Some comments would have author names missing.
• Google Calendar - Affected users would have experienced elevated error rates while using Calendar.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 and ended at 2024-06-17 22:27 (times are in Coordinated Universal Time (UTC)).

The issue with Google Chat is mitigated. Please find the details in the below link.

https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Chat

Incident began at 2024-06-17 16:00 and ended at 2024-06-17 22:27 (times are in Coordinated Universal Time (UTC)).

The issue with Google Drive is mitigated. Please find the details in the below link.

https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Drive