Subscribe to Google Apps Status feed
Updated: 1 hour 10 min ago

RESOLVED: We're aware of a problem with Google Meet affecting a significant subset of users. We will provide an update by Oct 31, 2022, 2:00 AM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate...

30 November, 2022 - 11:04

Incident began at 2022-10-10 22:18 and ended at 2022-11-07 15:20 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On 10 October 2022 at 15:18 PT, Google was alerted to a subset of users experiencing issues with Duo phone number registration with our Google Meet iOS application. Once the root cause was identified, Google pushed out an update to the App Store on 4 November 2022. We believe most users (66%) updated their app by 7 November 2022. To our Google Workspace customers who were impacted during this outage, we sincerely apologize. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause

The database version for our Google Meet backend infrastructure uses a specific library to complete the database operations to register or reregister a Duo calling account. An issue with the error-handling mechanism in the library resulted in the backend operations not functioning as intended. This caused affected users to be unable to register or reregister phone numbers, preventing the use of calling features, on the Meet iOS app.

Remediation and Prevention

Google engineers were alerted to the situation on 10 October 2022 at 15:18 and immediately started an investigation. Engineers reviewed the Google Meet for iOS app’s feedback logs and identified the root cause as related to registration workflow. Code improvements were identified to handle the app error seen when a user tries to register their account. The fix was deployed in a new version of the iOS app, which was rolled out to the App Store by 4 November 2022. At least 66% of users upgraded to the new version by 7 November 2022.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Deploy additional client logging to Meet to report registration-related issues, so we can respond in a timely manner.
  • Deploy additional monitoring for registration errors reported by the client to more quickly alert us if these types of events happen in the future.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Detailed Description of Impact

From 10 October 2022 at 15:18 to 7 November 2022 at 08:20, Google Meet iOS app users may have encountered an “unable to add calling” error when trying to register or reregister their Duo calling account, which resulted in phone numbers not registering as intended.

Affected products: Google Meet

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Docs. We will provide more information shortly.

23 November, 2022 - 10:08

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Docs

Categories: IT News

RESOLVED: We're investigating reports of an issue with Admin Console. We will provide more information shortly.

23 November, 2022 - 10:08

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Admin Console

Categories: IT News

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

23 November, 2022 - 10:07

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Gmail

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Cloud Search. We will provide more information shortly.

23 November, 2022 - 10:07

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Cloud Search

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

23 November, 2022 - 10:06

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Chat

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Voice. We will provide more information shortly.

23 November, 2022 - 10:05

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Voice

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Drive. We will provide more information shortly.

23 November, 2022 - 10:05

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Drive

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Meet. We will provide more information shortly.

23 November, 2022 - 10:04

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Meet

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Tasks. We will provide more information shortly.

23 November, 2022 - 10:03

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Tasks

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Groups. We will provide more information shortly.

23 November, 2022 - 10:03

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Groups

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

23 November, 2022 - 10:00

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:28 (times are in Coordinated Universal Time (UTC)).

A combined GCP/GWS Incident Report has been published on our Cloud Status Dashboard for this Incident - https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Calendar

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Tasks. We will provide more information shortly. The affected users are able to access Google Tasks, but may not have access to the most recent data.

22 November, 2022 - 10:38

Incident began at 2022-11-21 23:15 and ended at 2022-11-22 03:00 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 21 November 2022 15:15

Incident End: 21 November 2022 19:00

Duration: 3 hours, 45 minutes

Affected Services and Features:

  • Google Calendar
  • Google Tasks

Regions/Zones: Global

Description:

Google Calendar and Google Tasks experienced an issue with mismatch in the newly created task information across multiple devices due to a Google Tasks service sync issue for 3 hours and 45 minutes.

Initial investigation revealed that Google Tasks service received an unusual inflow of new requests on its private API which serves Calendar Web and Task services and that might have caused the sync issue. Google engineering performed a rollback of a recent code change to mitigate the issue.

Customer Impact:

Newly created tasks appeared in the Google Calendar grid of the same device via which those tasks were created, however those tasks were not shown in the calendar grid of other devices though the users were logged in.

Affected products: Google Tasks

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Tasks. We will provide more information shortly. The affected users are able to access Google Tasks, but may not have access to the most recent data.

22 November, 2022 - 10:38

Incident began at 2022-11-21 23:15 and ended at 2022-11-22 03:00 (times are in Coordinated Universal Time (UTC)).

The problem with Google Tasks has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

The issue with Google Calendar has been resolved for all affected users as of Monday, 2022-11-21 19:34 US/Pacific. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue.

Affected products: Google Tasks

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Tasks. We will provide more information shortly. The affected users are able to access Google Tasks, but may not have access to the most recent data.

22 November, 2022 - 10:37

Incident began at 2022-11-21 23:15 and ended at 2022-11-22 03:00 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Tasks. We will provide more information shortly. The affected users are able to access Google Tasks, but may not have access to the most recent data.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2022-11-21 20:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Affected products: Google Tasks

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are able to access Google Calendar, but may not have access to the most recent data.

22 November, 2022 - 10:35

Incident began at 2022-11-21 23:15 and ended at 2022-11-22 03:00 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 21 November 2022 15:15

Incident End: 21 November 2022 19:00

Duration: 3 hours, 45 minutes

Affected Services and Features:

  • Google Calendar
  • Google Tasks

Regions/Zones: Global

Description:

Google Calendar and Google Tasks experienced an issue with mismatch in the newly created task information across multiple devices due to a Google Tasks service sync issue for 3 hours and 45 minutes.

Initial investigation revealed that Google Tasks service received an unusual inflow of new requests on its private API which serves Calendar Web and Task services and that might have caused the sync issue. Google engineering performed a rollback of a recent code change to mitigate the issue.

Customer Impact:

Newly created tasks appeared in the Google Calendar grid of the same device via which those tasks were created, however those tasks were not shown in the calendar grid of other devices though the users were logged in.

Affected products: Google Calendar

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are able to access Google Calendar, but may not have access to the most recent data.

21 November, 2022 - 19:57

Incident began at 2022-11-21 23:15 and ended at 2022-11-22 03:34 (times are in Coordinated Universal Time (UTC)).

The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

The issue with Google Calendar has been resolved for all affected users as of Monday, 2022-11-21 19:34 US/Pacific. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue.

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are able to access Google Calendar, but may not have access to the most recent data.

21 November, 2022 - 19:55

Incident began at 2022-11-21 23:15 and ended at 2022-11-22 03:34 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are able to access Google Calendar, but may not have access to the most recent data.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2022-11-21 20:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Affected products: Google Calendar

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

14 November, 2022 - 22:20

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:27 (times are in Coordinated Universal Time (UTC)).

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Google Workspace Impact Start: 14 November 2022 10:50

Google Workspace Impact End : 14 November 2022 11:27

Duration: 37 minutes

Google Cloud Platform Impact Start: 14 November 2022 10:50

Google Cloud Platform Impact End: 14 November 2022 11:38

Duration: 48 minutes

For more information about incident impact details, please refer: https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Chat

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Cloud Search. We will provide more information shortly.

14 November, 2022 - 22:20

Incident began at 2022-11-14 18:50 and ended at 2022-11-14 19:27 (times are in Coordinated Universal Time (UTC)).

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Google Workspace Impact Start: 14 November 2022 10:50

Google Workspace Impact End : 14 November 2022 11:27

Duration: 37 minutes

Google Cloud Platform Impact Start: 14 November 2022 10:50

Google Cloud Platform Impact End: 14 November 2022 11:38

Duration: 48 minutes

For more information about incident impact details, please refer: https://status.cloud.google.com/incidents/rjV26MkG41n1YSAWL97S

Affected products: Google Cloud Search

Categories: IT News