Ten of the Strangest Data Center Outages update from June 2014

Every once in a while, utility power goes out and the backup systems fail, or a technician makes a mistake, and the data center goes down. While outages have become less frequent, as the industry’s practices continuously improve, things still occasionally go wrong. But sometimes there are also instances when something strange and completely unexpected causes the dreaded unplanned data center downtime.

Here is a list of some of the strangest data center downtime causes we've seen:

The Leap Second Bug

A leap second is a one-second adjustment that is occasionally applied to Universal Time to account for variations in the earth’s rotation speed. The addition of a single second to the world’s atomic clocks caused problems for a number of IT systems in 2012, when several popular web sites, including LinkedIn, Reddit, Mozilla and The Pirate Bay, went down. In Australia, 400 Qantas flights were delayed by two hours as the airline had to switch to manual check-ins.

Squirrel takes down Yahoo's Santa Clara data center

Squirrels taking a data center down isn’t actually all that rare. They chew everything, including all of those important wires we use to transfer communications. In 2010, “A frying squirrel took out half of our Santa Clara data center,” said Mike Christian, who runs business continuity for Yahoo, during a keynote at the O’Reilly Velocity conference.

Migration highway

Moving servers can be a tricky business. NaviSite (now owned by  Time Warner) acquired a hosting provider called Alabanza in 2007 and was moving customer accounts from Alabanza’s main data center in Baltimore to a facility in Andover, Massachusetts.

They literally unplugged the servers, put them on a truck, and drove the servers for over 420 miles. Many websites hosted by Alabanza were reportedly offline for as long as the drive and re-installation work took.

Another move-related problem occurred a few months earlier, when Hostway moved ValueWeb servers from Miami to Tampa. Hostway said later that more than 500 servers suffered hardware failures when they were restarted in the new facility.

Ship drops anchor on the Internet

Massive undersea cables carry traffic from continent to continent. These cables are durable, considering where they reside. However, there has been at least one instance where a ship dropped its anchor on one of them. There was a plague of undersea cable cuts in 2008, and while not necessarily a data center outage, it did cause downtime for some regions.

“Every wall is a door.” – Ralph Waldo Emerson

Nianet, a Danish ISP went down when thieves cut a hole in its Taastrup data center's walls to sneak in and steal stuff. They walked off with a bunch of networking cards, according to news reports. How the thieves were able to cut through a data center wall, and why they did so just to steal networking cards remains a mystery.

Careful where you throw your cigarette butt

At least one data center downtime incident was sparked by smoldering mulch. The Perth iX data center in western Australia shut down for an hour after its VESDA(Very Early Smoke Detection Apparatus) system detected smoke in the data center. The cause was identified to be a smoldering mulch-filled garden bed alongside the outside wall of the facility, most likely lit by a burning cigarette butt.

Keep on truckin’

In 2007, Rackspace, a company with a phenominal uptime record, suffered an outage for several hours after a a truck drove into a power transformer, which exploded.

The backup power tried to kick in, but two chillers failed to start. It took down some of the biggest sites on the internet at the time.

Czech-mate, Internet

In 2009, a single errant BGP announcement by an unknown Czech ISP created brief outages at several large hosting companies. Czech provider Supronet “single-handedly caused a global Internet meltdown for upwards of an hour,” said Renesys in a report.

Is my server down? Yes, down at the pawn shop

In 2007, two masked men broke into a Chicago data center and stole a bunch of computer equipment. The data center belonged to an old hosting company called C I Host. The company was eventually acquired and the name doesn’t exist anymore.

Some reported the lone employee working the night of the robbery was tazered and some reported he was pistol-whipped. Around 20 servers were stolen, bringing down a bunch of websites for good.

Following the robbery, there was rampant speculation as to what happened. Some say the robbers cut into the facility with a high-powered saw (which did happen to Nianet years later). C I Host said that the two men hid in the mechanical closet. There was some speculation that the company staged the robbery in order to commit fraud, but this was unsubstantiated.

But perhaps the strangest thing about it was that it wasn’t even the first time it happened.

“One of the biggest mistakes is that people are talking about four robberies,” the CEO told theWHIR at the time. “A robbery means than property has been seized through violence or intimidation. C I Host has technically only been robbed twice in two years. The other two were break-ins where things were stolen, but not robberies.”

Superstorm Sandy

It’s hard to argue for a stranger event than Superstorm Sandy. The freak, once in a lifetime (we hope) storm caused havoc in New York. Data center outages it caused led to some amazing stories, like the Bucket Brigade.

Storms usually die down by the time they reach that far north. However, as Sandy moved north, it took on some extra-tropical characteristics and grew to enormous dimensions. A major deviation of the high-altitude jet stream made the storm take a sharp left towards the coast. And it hit at high tide. The moon was full, so it was an even higher tide than normal.

This type of storm has never been recorded in the Northeast, making it an even freakier occurrence than a squirrel frying out your power, a mulch fire or a truck swerving off the road. There are some things you just can’t predict in this world.

Did we miss one? We’d love to hear from you.