On Twitter, @awscloud, AWS’s verified feed, explained the situation in a series of six tweets that began and ended on Tuesday. As of mid-morning on Wednesday, @awscloud hadn’t Tweeted a word in about 22 hours.
It initially explained what had happened as S3, its Simple Storage Service, “experiencing high error rates,” and said the company was “working hard on recovering.”
Slightly later Tweets about the outage confirmed an issue with an Amazon dashboard not changing color was related; and the fact that S3 continued to experience high error rates in the US-East-1 region had impacted “some other AWS services.”
Not every news organization reacted so calmly. A somewhat alarmist headline by The Register, a United Kingdom IT news website, called the incident “so bad Amazon couldn’t get into its own dashboard to warn the world.”
“Essentially, S3 buckets in the US-East-1 region in northern Virginia, US, became inaccessible at about 0945 PST,” The Register reported. “Software, from web apps to smartphone applications, relying on this cloud-based storage quickly broke, taking out a sizable chunk of the internet as we know it.”
The actual recovery reportedly began about three hours later on Tuesday, or around 1 p.m. Pacific time, and is believed to have been fully resolved roughly an hour after that.
Websites that went “wobbly,” as the Register put it, reportedly included Quora, Slack, Adobe’s cloud, and even Yahoo Mail. Other life forms like SiriusXM, the Hey You coffee-ordering app, and Internet of Things devices that relied on S3 were also reported to have had issues.
In @awscloud’s final tweet on the issue, a sense of finality was expressed:
For S3, we believe we understand root cause and are working hard at repairing. Future updates across all services will be on dashboard. — Amazon Web Services (@awscloud) February 28, 2017
TechRepublic’s Brandon Vigliarolo pointed out that S3 has been around since 2006, and noted that downtimes are rare in the public cloud. He quoted Forrester Research’s Dave Bartoletti, a public cloud analyst, who also sought to tamp down concern.
“S3 has consistently outperformed the four nines they shoot for, year over year,” Bartoletti told TechRepublic, referring to AWS’s goal of having S3 up 99.99 percent of the time.
Four states so far — California, Colorado, Minnesota and Oregon — have signed Criminal Justice Information Services (CJIS) agreements with AWS to give their law enforcement access to its cloud. The move reflects a stronger public-sector focus by AWS, which last year announced another version of its City on a Cloud Innovation Challenge.
A source familiar with the situation told Government Technology on Wednesday that AWS government cloud storage, including CJIS material, isn’t housed in US-East-1, so it wasn’t affected.
An official at the Colorado Bureau of Investigation said via email that state operations — which build in their own redundancy — were unaffected by the outage.
The state’s CJIS Compliance Officer Ted DeRosa confirmed to Government Technology there was no disruption to its Colorado Crime Information Center (CCIC) access.
If any state agency’s CCIC-connected computerized or records systems had been housed on AWS and had gone down, DeRosa said, “they would move immediately to the redundant user interface Colorado Bureau of Investigation offers at no cost to the agency, which is hosted at CBI.”
It remains unclear whether law enforcement in California, Minnesota or Oregon was affected by the break in service, as officials in California and Oregon did not respond to requests for comment, and Jill Oliveira, public information officer for the Minnesota Bureau of Criminal Apprehension, said the agency had no comment.
