Following one of the longer cross-service outages for Google in recent memory, the search and software giant sent out an apology and explanation for today's occurrences. According to the Official Google Blog, an internal system that sends out configuration information for systems beyond it encountered a software bug that sent out incorrect commands to several areas.
It only took from 10:55am PT when the bug was first seen to 11:02am when users began seeing massive outages in Gmail, Google+, Drive and other services. Roughly 12 minutes later while engineers were still in the process of figuring out what was happening, the initial system that sent out the bad information had self-corrected and began to properly configure other systems. Google claims nearly all users' services were back up and running by 11:30, which seems to be consistent with the general consensus among users.
As you would expect, the post gives some detail on what's being done to prevent this from happening in the future. More checks are being put in place so that improper configurations, if they are generated by bugs, aren't so easily sent out to other systems. Additionally, Google plans to improved targeted searching for issues during service failures.
Earlier today, most Google users who use logged-in services like Gmail, Google+, Calendar and Documents found they were unable to access those services for approximately 25 minutes. For about 10 percent of users, the problem persisted for as much as 30 minutes longer. Whether the effect was brief or lasted the better part of an hour, please accept our apologies—we strive to make all of Google's services available and fast for you, all the time, and we missed the mark today.
The issue has been resolved, and we're now focused on correcting the bug that caused the outage, as well as putting more checks and monitors in place to ensure that this kind of problem doesn't happen again. If you're interested in the technical explanation for what occurred and how it was fixed, read on.
At 10:55 a.m. PST this morning, an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users' requests for their data to be ignored, and those services, in turn, generated errors. Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google's Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time. By 11:30 a.m. the correct configuration was live everywhere and almost all users' service was restored.
With services once again working normally, our work is now focused on (a) removing the source of failure that caused today's outage, and (b) speeding up recovery when a problem does occur. We'll be taking the following steps in the next few days:
1. Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug.
2. Adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption.
3. Adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.
Posted by Ben Treynor, VP Engineering
Google explains reasons behind 30-minute service outage
