Google Apps Status

Subscribe to Google Apps Status feed
Updated: 3 min 47 sec ago

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

20 April, 2024 - 13:11

Incident began at 2024-04-19 14:12 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Gmail

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

20 April, 2024 - 13:10

Incident began at 2024-04-19 14:12 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Google Chat

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

20 April, 2024 - 13:08

Incident began at 2024-04-19 14:12 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

19 April, 2024 - 15:52

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Further communication will be provided on the below link.

https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

19 April, 2024 - 15:52

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Further communication will be provided on the below link.

https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Affected products: Google Chat

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

19 April, 2024 - 15:48

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Summary:

Gmail , Chat and Calendar customers experiencing issues with Contact sharing

Description:

Our Engineering team has identified the root cause of the issue and is deploying the appropriate mitigations.

The issue is expected to be resolved by Saturday 2024-04-20 13:00 US/Pacific.

We will provide more information by Saturday 2024-04-20 13:00 US/Pacific.

Diagnosis:

Customers for Gmail, Chat, Calendar may experience "Contact Sharing" disabled. Users may not see an address list in Gmail, but will still be able to send email.

Workaround:

None at this time.

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

19 April, 2024 - 14:08

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Summary:

Gmail , Chat and Calendar customers experiencing issues with Contact sharing

Description:

Our Engineering team has identified the root cause of the issue and rolled back a recent change.

Our engineers have started to deploy mitigations to affected customers, but will take some time to fully propagate.

We do not have an ETA for full resolution at this point.

We will provide more information by Friday, 2024-04-19 16:30 US/Pacific.

Diagnosis:

Customers for Gmail, Chat, Calendar may experience "Contact Sharing" disabled. Users may not see an address list in Gmail, but will still be able to send email.

Workaround:

None at this time.

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

19 April, 2024 - 14:06

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Further communication will be provided on the below link.

https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

19 April, 2024 - 14:05

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Further communication will be provided on the below link.

https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Affected products: Google Chat

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

19 April, 2024 - 12:36

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Our Engineering team has identified the root cause of the issue and rolled back a recent change.

Our engineers are taking additional repair actions for the rollback to take effect for the impacted customers.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2024-04-19 14:00 US/Pacific.

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

19 April, 2024 - 12:36

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Our Engineering team has identified the root cause of the issue and rolled back a recent change.

Our engineers are taking additional repair actions for the rollback to take effect for the impacted customers.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2024-04-19 14:00 US/Pacific.

Affected products: Google Chat

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

19 April, 2024 - 12:36

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Our Engineering team has identified the root cause of the issue and rolled back a recent change.

Our engineers are taking additional repair actions for the rollback to take effect for the impacted customers.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2024-04-19 14:00 US/Pacific.

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

19 April, 2024 - 11:33

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Engineering team potentially identified the root cause and working on mitigation.

The mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

19 April, 2024 - 11:33

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Engineering team potentially identified the root cause and working on mitigation.

The mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.

Affected products: Google Chat

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

19 April, 2024 - 11:32

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

Engineering team potentially identified the root cause and working on mitigation.

The mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

19 April, 2024 - 11:23

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

Google Calendar customers may experience "Contact Sharing" disabled.

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

19 April, 2024 - 11:22

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Chat. We will provide more information shortly.

Google Chat customers may experience "Contact Sharing" disabled.

Affected products: Google Chat

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

19 April, 2024 - 11:21

Incident began at 2024-04-19 14:12 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Gmail. We will provide more information shortly.

Gmail customers may experience "Contact Sharing" disabled. Users will not see an address list in Gmail, but will still be able to send email.

Affected products: Gmail

Categories: IT News

RESOLVED: **Summary:**...

13 March, 2024 - 14:26

Incident began at 2024-03-06 22:32 and ended at 2024-03-06 23:29 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Wednesday, 06 March 2024, Google Workspace users experienced elevated errors while accessing user or device setting pages within the Chrome Admin Console for a duration of 1 hour, 27 minutes.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google Workspace Admin Console uses a backend authentication service to grant privileged access to internal accounts. As part of ongoing efforts to migrate the account authorization workflow to a new backend authentication service, Google engineers identified and cleaned up invalid account role assignments in the configuration.

During this clean up effort, engineers identified a privilege which was undefined in the application manifest and removed it from the configuration. Upon further investigation, engineering confirmed the privilege was mapped to an access control setting used by the Admin Console for Chrome. This change resulted in users being unable to access Chrome user and device settings page.

Remediation and Prevention

Google engineers were alerted to the outage via an internal monitoring alert on Wednesday, 06 March 2024 at 15:04 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers initiated a rollback of the configuration change at 15:36 and the incident was fully resolved at 15:59 once the rollback completed.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Improve procedural documentation on how to safely cleanup access control settings from authorization configuration.
  • Complete thorough investigation of privilege usage for the Admin Console to prevent further issues of this type.
  • Identify and register a new owner for this privilege in the application manifest and create permission mapping for faster verification in the future.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Wednesday, 06 March 2024 from 14:32 to 15:59 US/Pacific, Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console.

Affected products: Admin Console

Categories: IT News

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

12 March, 2024 - 12:52

Incident began at 2024-03-05 15:25 and ended at 2024-03-05 17:22 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Tuesday, 5 March 2024, starting at 07:25 US/Pacific, 0.23% of Gmail UI requests were failing for a duration of 61 minutes and 1.1% of email messages experienced up to five minute delays for a duration of 1 hour, 57 minutes.

Root Cause

This report includes a revised version of root cause for the service issue Gmail customers experienced on 5 March 2024.

After further investigation, we identified that a sudden and unexpected surge of email traffic initiated the issue, rather than an increase in authentication traffic as shared in the previous report.

Gmail frontend services HTTP, POP, or IMAP use an internal lookup service to route the requests to the specific backend instances that contain the user data. When this internal lookup service fails, users experience issues with Gmail UI interactions.

The same internal lookup service is also consulted during email delivery. When the lookup requests fail during email delivery, the requests are retried with an exponential backoff and this would materialize as delayed emails from an end-user perspective.

On 5 March 2024 at 07:25 US/Pacific, Gmail began experiencing a sudden and unexpected increase in real user traffic that coincided with a third-party service outage outside of Google’s infrastructure. At the time of the traffic increase, the internal lookup service in some geo locations was overloaded, which resulted in errors for a small fraction of Gmail UI requests and delayed email message delivery.

Remediation and Prevention

Google engineers were alerted to the increase in traffic via internal systems on 5 March 2024 at 07:32 US/Pacific and immediately started an investigation. Once the scope and nature of the issue was identified, at 08:11 US/Pacific, Google engineers redirected traffic away from the affected locations. This immediately stopped the Gmail UI errors, and prevented delivery delays for new messages.

By 09:22 US/Pacific the backlog of emails in the delivery pipeline had cleared completely.

If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Enhance Gmail’s lookup server traffic surge protection layer to be able to better handle these kinds of traffic surges.
Detailed Description of Impact

On 5 March 2024, between 07:25 and 09:22 US/Pacific, Gmail experienced increased error rates and delivery delays. During this time:

  • Approximately 0.23% of requests in the Gmail UI resulted in an error message. Affected customers encountered these errors for various actions on Gmail, including login.
  • Approximately 1.1% of email messages experienced delivery delays ranging between one and five minutes.

Affected products: Gmail

Categories: IT News