Google Apps Status

Incident began at 2024-05-02 15:53 and ended at 2024-05-02 19:02 (times are in Coordinated Universal Time (UTC)).

We're investigating reports of an issue with Google Meet. We will provide more information shortly.

Affected products: Google Meet

Incident began at 2024-04-30 00:00 and ended at 2024-05-01 22:55 (times are in Coordinated Universal Time (UTC)).

Summary:

Gmail customers are currently unable to send unsigned messages to Yahoo.

Description:

The issue has been resolved for all affected users as of Wednesday, 2024-05-01 15:55 US/Pacific.

There was a problem external to Google which has now been resolved.

We thank you for your patience while we worked on resolving the issue.

Affected products: Gmail

Incident began at 2024-04-30 00:00 (times are in Coordinated Universal Time (UTC)).

Summary

Gmail customers are currently unable to send unsigned messages to Yahoo.

Description:

A small subset of Gmail customers are experiencing difficulty sending messages to Yahoo.

Our engineering team continues to work on mitigation.

Customer messages may be bounced with the below error:

"This mail has been blocked because the sender is unauthenticated. Yahoo requires all senders to authenticate with either SPF or DKIM."

If you're seeing this error message, please consider setting up SPF (https://support.google.com/a/answer/33786) or DKIM (https://support.google.com/a/answer/174124) for your sending domain(s).

Note that Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM) records may take some time to become fully propagated in DNS and therefore the issue may persist for a little while longer after setting these records up.

We will provide an update by Wednesday, 2024-05-01 20:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Affected products: Gmail

Incident began at 2024-04-30 00:00 (times are in Coordinated Universal Time (UTC)).

Gmail customers are currently unable to send unsigned messages to Yahoo.

Our engineering team continues to investigate.

Customer messages sent to Yahoo/AOL recipients may be bounced with the error "This mail has been blocked because the sender is unauthenticated. Yahoo requires all senders to authenticate with either SPF or DKIM." If you're seeing this error message, please set up SPF (https://support.google.com/a/answer/33786) or DKIM (https://support.google.com/a/answer/174124) for your sending domain(s). For DKIM, you should set up a record for your own sending domain. You should not rely on the default DKIM key applied by Google (*.gappssmtp.com) to avoid this error.

Note that SPF and DKIM records may take some time to become fully propagated in DNS and therefore the issue may persist for a little while longer after setting these records up.

We apologize to all who are affected by the disruption.

Affected products: Gmail

Incident began at 2024-04-30 00:00 (times are in Coordinated Universal Time (UTC)).

We are aware of an issue affecting Gmail users sending emails to some email addresses. We are currently investigating.

Affected products: Gmail

Incident began at 2024-04-18 21:40 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

An incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/D6dP4yHWc2m93ZhpLYKw

Affected products: Google Calendar

Incident began at 2024-04-18 21:40 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

An incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/D6dP4yHWc2m93ZhpLYKw

Affected products: Gmail

Incident began at 2024-04-18 21:40 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

Incident Report Summary

On 18 April 2024, Gmail, Google Calendar and Google Chat experienced issues with the ‘Contact Sharing’ feature for a duration of 1 day, 21 hours 43 minutes. To our Google Workspace customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

An internal code update unintentionally altered the default values of a setting that interacts with the current contact sharing system. This change resulted in a misconfiguration of directory settings, impacting the Contact Sharing feature within Google Workspace services like Gmail, Chat, and Calendar.

Remediation and Prevention

We were alerted to the outage via a support case on April 19 at 7:12 US/Pacific and promptly initiated an investigation. The investigation confirmed that the issue originated from a code change deployed on April 18 at 14:40 US/Pacific. Upon troubleshooting, we immediately halted and rolled back the code change, preventing a wider impact on Google Workspace services.

We then focused on repairing the issue for affected customers.

We apologize for the length and severity of this incident. Google is committed to preventing recurrence by implementing the following action items:

Engineering team implemented release controls to prevent code changes that alter the default values from being deployed to the production environment. Engineering team to add test coverage and verify code rollouts like these so issues like this can be detected early. Engineering team to set up alerts for unexpected spikes in compilation requests during automated syncs, as this could indicate a potential problem with data changes.

Detailed Description of Impact

On April 18 at 14:40 US/Pacific an internal update unexpectedly turned off the "Contact Sharing" feature for Google Workspace users. This resulted in features like autocompleting email addresses and seeing contact details in Gmail, Chat, and Calendar not working as expected.

Known user-facing impacts are as follows:

Gmail

Affected users may not have seen a contact list in Gmail. Sending emails were not affected. Autocomplete for individual and group addresses entered by users was affected. Hovercards, also known as profile information cards that show up when users pointed to or tapped on someone’s profile image did not work.

Google Contacts

Impacted users could not find profile information in Google Contacts.

Google Calendar

Searching for users, and intelligent meeting room suggestions in Google Calendar were affected.

Affected products: Google Chat

Incident began at 2024-04-24 15:00 and ended at 2024-04-24 21:17 (times are in Coordinated Universal Time (UTC)).

The issue with users creating unauthenticated support cases for account recovery help using toolbox has been resolved for all affected users as of Wednesday, 2024-04-24 14:17 US/Pacific.

From our preliminary analysis, the root cause of the issue is a recent roll out. Our engineers rolled back to last stable version hence mitigating the issue.

We thank you for your patience while we worked on resolving the issue.

Affected products: Google Workspace Support

Incident began at 2024-04-24 15:00 and ended at 2024-04-24 21:17 (times are in Coordinated Universal Time (UTC)).

Summary:

Workspace Super Admins trying to open unauthenticated support cases for account recovery help using toolbox are encountering 500 errors and unable to open support cases for account recovery.

We will provide an update by Wednesday, 2024-04--24 14:45 US/Pacific with current details.

Workaround:

Customers having issues opening the account recovery cases can use https://support.google.com/a/contact/recovery_form to contact support until the issue is resolved.

Affected products: Google Workspace Support

Incident began at 2024-04-22 11:10 and ended at 2024-04-23 16:50 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 22 April 2023 04:10

Incident End: 23 April 2023 09:50

Duration: 1 day, 5 hours, 40 mins

Affected Services and Features: Classroom

Regions/Zones: us-east1

Description:

Google Classroom experienced an increased error rate in Brazil (for the State of Paraná) due to hotspotting (overloading) in the product’s database for a duration of 1 day, 5 hours, 40 mins. From preliminary analysis, the root cause of hotspotting was a software bug, which we’ve fixed.

Customer Impact:

A subset of users in the state of Paraná, Brazil had encountered intermittent errors while accessing Google Classroom. To the best of our knowledge, no other customers were impacted.

Affected products: Classroom

Incident began at 2024-04-22 12:00 and ended at 2024-04-23 18:02 (times are in Coordinated Universal Time (UTC)).

The issue with Classroom has been resolved for all affected users as of Tuesday, 2024-04-23 11:02 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: Classroom

Incident began at 2024-04-22 12:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Google Classroom is experiencing an increased error rate in Brazil (for State of Paraná)

Description:

Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Tuesday, 2024-04-23 12:30 US/Pacific.

We will provide more information by Tuesday, 2024-04-23 12:45 US/Pacific.

Customer Symptoms:

A subset of users in the state of Parana, Brazil would encounter intermittent error while accessing Google Classroom. However, the impact is only limited to the state of Parana, no impact to customers to the rest of the world.

Workaround:

None at this time.

Affected products: Classroom

Incident began at 2024-04-22 12:00 (times are in Coordinated Universal Time (UTC)).

Summary:

Google Classroom is experiencing an increased error rate in Brazil (for State of Paraná)

Description:

We are experiencing an intermittent issue with Google Classroom.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-04-23 09:00 US/Pacific with current details.

Customer Symptoms:

A subset of users in the state of Parana, Brazil would encounter intermittent error while accessing Google Classroom. However, the impact is only limited to the state of Parana, no impact to customers to the rest of the world.

Workaround:

None at this time.

Affected products: Classroom

Incident began at 2024-04-18 21:40 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Affected products: Google Calendar

Incident began at 2024-04-18 21:40 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Affected products: Google Chat

Incident began at 2024-04-18 21:40 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 18 April 2024 14:40

Incident End: 20 April 2024 12:23

Duration: 1 day, 21 hours, 43 minutes

Affected Services and Features: Gmail, Google Calendar, Google Chat

Regions/Zones: Global

Description:

Gmail, Google Calendar and Google Chat experienced issues with “Contact Sharing” for a duration of 1 day, 21 hours, 43 minutes.

From the preliminary analysis, the root cause of the issue was a rollout that led to an unexpected change to the “Directory Settings” policy causing “Contact Sharing” to be disabled for users.

Google engineers mitigated the issue on 20 April 2024 at 12:23 US/Pacific by reverting to a previous stable release and took additional actions to re-enable "Contact Sharing" for all affected users.

Google will complete a full IR in the following days that will provide a detailed root cause.

Customer Impact:

• Affected users may not have seen a contact list in Gmail, but would still have been able to send mail.

• Affected users could not find profile information in Google Contacts and other Google services.

• Autocomplete of the individual and group addresses as users entered them was affected.

• Searching for users and intelligent meeting room suggestions in Google Calendar was affected.

• Profile information cards that opened when users pointed to or tapped on someone’s profile image were impacted as well.

Affected products: Gmail

Incident began at 2024-04-19 14:12 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Gmail

Incident began at 2024-04-19 14:12 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Google Chat

Incident began at 2024-04-19 14:12 and ended at 2024-04-20 19:23 (times are in Coordinated Universal Time (UTC)).

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Affected products: Google Calendar