RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

Google Apps Status - 8 December, 2023 - 03:23

Incident began at 2023-12-07 19:00 and ended at 2023-12-07 20:32 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues.

If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 7 December 2023 11:00

Incident End: 7 December 2023 12:32

Duration: 1 hours, 32 minutes

Affected Services and Features:

Sign-up and log-in for parts of unauthenticated traffic on Google Workspace and Google Cloud products.

Regions/Zones: us-east (partial impact)

Description:

Starting on 7 December 2023 at 11:00, users trying to sign up or login to Google Workspace products or Google Cloud Console services in the us-east region experienced intermittent issues for a duration of 1 hour and 32 minutes.

From our preliminary investigation, this was likely caused by performance issues on the sign up and login frontend servers in one of the datacenters in the affected region. To mitigate the impact, our engineering team rerouted traffic and optimized server distribution. Service was restored on 7 December 2023 at 12:32.

Google will complete a detailed Incident Report in the following days that will provide a full root cause.

Customer Impact:

Some unauthenticated users in parts of the us-east region intermittently experienced difficulties accessing the impacted products. These users saw 502 errors when trying to sign up, login, or re-authenticate services.

Affected products: Gmail

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:59

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

During the issue, some users were unable to access Google Calendar services.

Affected products: Google Calendar

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Docs. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:57

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Google Docs has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

During the issue, some users were unable to access Google Docs services.

Affected products: Google Docs

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Forms. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:56

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Google Forms has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

During the issue, some users were unable to access Google Forms services.

Affected products: Google Forms

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Drive. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:55

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Google Drive has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

During the issue, some users were unable to access Google Drive services.

Affected products: Google Drive

Categories: IT News

RESOLVED: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:53

Incident began at 2023-12-07 19:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

During the issue, some users were unable to access Google Chat services.

Affected products: Google Chat

Categories: IT News

RESOLVED: We're investigating reports of an issue with Classroom. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:53

Incident began at 2023-12-07 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Classroom has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

During the issue, some users were unable to access Classroom services.

Affected products: Classroom

Categories: IT News

RESOLVED: We're investigating reports of an issue with Gmail. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:52

Incident began at 2023-12-07 19:17 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Gmail

Categories: IT News

UPDATE: We're investigating reports of an issue with Classroom. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:46

Incident began at 2023-12-07 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Classroom. We will provide more information shortly.

Affected products: Classroom

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Chat. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:41

Incident began at 2023-12-07 19:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Chat. We will provide more information shortly.

Affected products: Google Chat

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:41

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

Affected products: Google Calendar

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Docs. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:40

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Docs. We will provide more information shortly.

Affected products: Google Docs

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Forms. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:40

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Forms. We will provide more information shortly.

Affected products: Google Forms

Categories: IT News

UPDATE: We're investigating reports of an issue with Google Drive. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:38

Incident began at 2023-07-12 18:00 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Drive. We will provide more information shortly.

Affected products: Google Drive

Categories: IT News

UPDATE: We're investigating reports of an issue with Gmail. We will provide more information shortly.

Google Apps Status - 7 December, 2023 - 12:37

Incident began at 2023-12-07 19:17 and ended at 2023-12-07 20:49 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Gmail. We will provide more information shortly.

Gmail users are unable to login. Users may see errors in the Google login page.

Users can retry their login attempts if they see errors.

Affected products: Gmail

Categories: IT News

RESOLVED: We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific....

Google Apps Status - 6 December, 2023 - 16:19

Incident began at 2023-12-01 18:52 and ended at 2023-12-01 20:20 (times are in Coordinated Universal Time (UTC)).

The Incident Report is now available on the Google Workspace Status Dashboard : https://www.google.com/appsstatus/dashboard/incidents/Hjq8CpkhWroicw7R1wSh

Affected products: Gmail

Categories: IT News

RESOLVED: We are experiencing an issue with Gmail beginning on Thursday, 2023-11-30 08:30 US/Pacific....

Google Apps Status - 6 December, 2023 - 16:15

Incident began at 2023-11-30 16:30 and ended at 2023-11-30 21:15 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

Gmail’s core message processing system experienced delays for email delivery on Thursday, 30 November 2023 and Friday, 01 December 2023, for a cumulative duration of 6 hours and 23 minutes. To our Gmail customers whose email delivery was impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google runs a core message processing system that is responsible for delivering, monitoring, queuing, and receiving email messages. The root cause of both incidents was an atypical pattern of email traffic, resulting in high resource utilization in this system and delays in email delivery.

Remediation and Prevention

On both occasions, Google engineers were alerted to the delays via monitoring alerts at 08:39 US/Pacific on 30 November and again at 10:57 US/Pacific on 01 December.

We immediately began mitigation efforts, including increasing capacity for the impacted regions. We have also implemented reasonable limits to prevent such an atypical traffic pattern from recurring.

We apologize for the recurrence and severity of this incident and are taking the following step to improve reliability in the future:

  • Implement rate limit mechanisms in our systems.
Detailed Description of Impact

On Thursday, 30 November between 08:30 and 13:15 US/Pacific, and on Friday, 01 December between 10:52 and 12:30 US/Pacific, Gmail users experienced email delivery delays for at least 5 minutes. During this period, users may also have observed an increase in rejected emails.

Affected products: Gmail

Categories: IT News

RESOLVED: We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific....

Google Apps Status - 1 December, 2023 - 12:36

Incident began at 2023-12-01 18:52 and ended at 2023-12-01 20:20 (times are in Coordinated Universal Time (UTC)).

The issue with Gmail has been resolved for all affected users as of Friday, 2023-12-01 12:20 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Affected products: Gmail

Categories: IT News

UPDATE: We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific....

Google Apps Status - 1 December, 2023 - 12:16

Incident began at 2023-12-01 18:52 and ended at 2023-12-01 20:20 (times are in Coordinated Universal Time (UTC)).

Mitigation work is currently underway by our engineering team. Our engineers are seeing positive signs of recovery.

However we are actively monitoring to ensure the systems are fully recovered.

We will provide more information by Friday, 2023-12-01 13:00 US/Pacific.

Customer Symptoms : Customers may experience delays in their email delivery. Customers may not be required to re-send the affected messages unless they bounce back. Customers with messages traversing in our US-based servers may be impacted.

Workaround : No action required from Customers unless the email that has bounced back.

Affected products: Gmail

Categories: IT News

UPDATE: We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific....

Google Apps Status - 1 December, 2023 - 12:06

Incident began at 2023-12-01 18:52 and ended at 2023-12-01 20:20 (times are in Coordinated Universal Time (UTC)).

We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2023-12-01 12:45 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms : Customers may experience delays in their email delivery. Customers may not be required to re-send the affected messages unless they bounce back. Customers with messages traversing in our US-based servers may be impacted.

Workaround : No action required from Customers unless the email that has bounced back.

Affected products: Gmail

Categories: IT News