One errant keystroke to blame

It turns out that it was a simple typo that caused the Tuesday internet meltdown that affected thousands of websites.

Amazon’s Simple Storage Service (S3) — a cloud storage provider dubbed the backbone of the internet because more than 150,000 websites, including giants like Netflix and Pinterest, use it to store video, data and other content — went down for more than four hours, taking many of its clients down with it.

Amazon said on Thursday that it has gotten to the bottom of what caused the blackout: a small typing error.

“Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended,” the Seattle-based company said in a statement.

Amazon explained that its team was looking into an issue that caused its S3 billing system to run slowly. To figure out what caused the slowdown, Amazon allowed its staffers to take a small number of its servers offline — a move that wouldn’t have been a big deal because the few selected servers don’t host crucial operational mechanisms.

But an Amazon employee accidentally entered a typo into the command to take down the servers, and a whole lot more were knocked offline. At least two of those mistakenly shuttered servers contained vital systems: One held information about the location of East Coast customers’ data and the other supported storage for the massive number of files that S3 hosts.

To get the mistakenly-dismantled servers back online, Amazon restarted the systems, a process that took more than four hours and left many S3-supported websites in the dark.

A blackout about every two years

Since 150,714 websites use S3, the consequences of the blackout were far-reaching.

News site Business Insider, GIF-maker Giphy, question-and-answer site Quora, blog platform Medium and newsletter provider Sailthru all had glitches. Teams couldn’t share files in Slack, the popular workplace communication tool. Even some smart light bulbs and thermostats went down.

S3 is notoriously reliable: Outage-tracking company CloudHarmony reports near-perfect availability ratings for the website and Amazon itself guarantees “99.999999999% durability.”

Read more: Fox8