IT News

Incident began at 2024-06-17 16:00 and ended at 2024-06-17 22:27 (times are in Coordinated Universal Time (UTC)).

The issue with Google Calendar is mitigated. Please find the details in the below link.

https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Calendar

Incident began at 2024-06-17 16:00 and ended at 2024-06-17 22:27 (times are in Coordinated Universal Time (UTC)).

The issue with Google Meet is mitigated. Please find the details in the below link.

https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Meet

Incident began at 2024-06-17 16:00 and ended at 2024-06-17 22:27 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

The issue with Google Calendar, Google Chat, Google Docs, Google Drive, Google Meet has been resolved for all affected users as of Monday, 2024-06-17 15:27 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Chat

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Drive

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Calendar

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Meet

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

The mitigation applied previously did not yield results as expected.

Our Engineering team continues to investigate and ascertain a mitigation strategy to fix all known issues

We will provide more information by Monday, 2024-06-17 16:30 US/Pacific.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Chat

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Drive

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Calendar

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an ongoing issue. Our engineers are working towards mitigation.

Please find more updates in the below link. https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT

Affected products: Google Meet

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2024-06-17 14:45 US/Pacific.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Docs

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Meet

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Calendar

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Drive

Incident began at 2024-06-17 16:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Multiple Workspace products are experiencing elevated error rates and latency

Description:

We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.

Our engineering team has identified the root cause of the issue and are currently working on a mitigation.

We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

• Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.

• Some Google Meet users are having issues joining the meetings.

• Some Google Drive users are experiencing elevated latencies and error rates

• Some Google Docs users are experiencing issues where they see "Name loading" in comments.

Workaround:

None at this time.

Affected products: Google Chat

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On Tuesday, 11 June 2024, Apps Script experienced increased error rates and users may have experienced intermittent issues in script creation and execution for a duration of 37 minutes. In addition to this, any of the Workspace services that internally use Apps Script may have encountered issues during the period of impact. We sincerely apologize to all Apps Script users. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

One of the scripts received an unexpectedly high number of the queries per second (QPS) which resulted in hotspotting for our database service (a database hotspot is created because a single server receives most or all of the writes, which overloads that one server). This led to deadline exceeded errors, thereby increasing the error rate in our remote procedure calls (RPCs) and leading to the script creation and execution issues.

Remediation and Prevention

Google engineers were alerted to the outage by the internal monitoring systems that sent multiple alerts starting 06:25 US/Pacific (approximately 15 mins after issue started) and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly enforced limits on read actions for the affected script and then blocked the script which led to returning database hotspotting and error rate in our RPCs back to their original values.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Enable caching on read replicas for App Info to reduce the number of requests being sent to our database service.
• Adding limits and alerts for all RPCs to ensure a limit on the QPS that one add on can send.
• Reduce the number of read replicas to avoid loading App Info on multiple replicas.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Tuesday, 11 June 2024 from 06:04 US/Pacific to 06:41 US/Pacific Apps Script experienced increased error rates (error rate in our RPCs increased to ~20% whilst the expected error rate is ~1.3%.

Apps Script

Affected users experienced intermittent issues in script creation and execution for a duration of their app script applications receiving DEADLINE_EXCEEDED error status.

Affected products: Apps Script

Incident began at 2024-06-11 13:04 and ended at 2024-06-11 13:41 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 11 June 2024 06:04

Incident End: 11 June 2024 06:41

Duration: 37 minutes

Affected Services and Features:

Apps Script

Regions/Zones: Global

Description:

Apps Script experienced increased error rates for a duration of 37 minutes. From preliminary analysis, the root cause of the issue was due to an unexpected increase in traffic related to a third-party add-on that resulted in resource contention.

Engineers temporarily blocked the third-party add-on that was the cause of the increased traffic to mitigate the issue.

Customer Impact:

Customers may have received errors when attempting to load or run Apps Scripts.

Affected products: Apps Script