New York Data Centers Battle Back from Storm Damage update from November 2012

verizon-sandy_flood

The flooded lobby at the Verizon site at 140 West Street during the height of the storm surge Monday night. (Photo:Verizon)

As Thursday dawned on lower Manhattan, the city's battered data centers continued their recovery efforts. Wednesday was a day of fast-moving events, as some facilities that were down came back online, and some that were up went down. Several of the hardest-hit facilities were hoping to be back in action Thursday, as Con Edison gradually begins to restore power to some neighborhoods below 34th street. Here's an update of the status of Manhattan data centers affected by SuperStorm Sandy:

Verizon Business

The extent of the damage to Verizon Business facilities became more widely understood Wednesday, and was described in detail in Thursday's  Wall Street Journal.

"The company's headquarters - a key communications hub just north of the World Trade Center site at 140 West Street - was in a state of crisis not seen since the 9/11 attacks, which partially destroyed the building," The Journal reported. "Mud still covered parts of the ornate lobby. Down below, 3½ floors of the building's five-level basement were still submerged, the brackish water sloshing up the walls of the stairwell. Verizon employees said Monday night's storm surge was so powerful that it breached the protective plugs that surround cables coming into the building. As a result, water flooded the critical basement "cable vault" that takes in communications cables and directs them to switching gear upstairs, which wasn't damaged."

Verizon published a photo of the flooding at 140 West on its blog, showing at least three feet of water filling the building's lobby.

"Record storm surge and extensive flooding in the area forced water into some lobbies and basements of Verizon facilities," the company said. "Verizon has been able to quickly secure pumping equipment and began the process of removing water from our sites.   Because critical communications equipment (such as voice switches, data equipment and routers) are located on high floors in our buildings, they were not damaged by floodwater.  But the water did damage some of backup power equipment such as switch gear, generators and fuel pumps, causing some repair delays.

"Technical restoral efforts are complicated in many areas by hundreds of closed roads, persistently high water, mandatory evacuations that limit access, continued snow, downed trees and other debris," the company added. "In many cases, Verizon must wait for power companies to restore their facilities and declare the area safe before the company’s technicians can move in to restore services."

Internap: 75 Broad Street

The flooded basement at 75 Broad Street is still being pumped out, with 15 feet of water remaining as of late Wednesday afternoon. Meanwhile, data center tenants at the building are coming up with their own workarounds.

"Currently, power and connectivity have been restored at our 75 Broad facility," Internap reported in an update. The colocation provider powered back up at 11:20 p.m. Tuesday. "All customers should be up and operational at this time, working as expected on generator power. We have over 40 hours of fuel onsite with a second truck waiting to refuel the current tank as necessary."

Meanwhile, the Peer 1 data center at 75 Broad was being kept online by a bucket brigade of staff and customers who are manually transporting diesel fuel up 17 stories from the street to the rooftop generator.

"We have a few hundred gallons of spare fuel on the roof along with a full tank, which will take us well into the nigh," reported Squarespace, a Peer 1 customer that has assisted with the recovery effort "On top of teams from Peer1, Fog Creek, and Squarespace, we were able to hire additional help from Brooklyn and Queens. We are cautiously optimistic that fuel will be running to the roof by end of day tomorrow or the day after."

Atlantic Metro

Late Wednesday Atlantic Metro reported progress as it sought to recover from flood damage and power outages at several of its data centers. The company has been keeping customers posted with regular updates on Twitter:

LGA1 (325 Hudson Street) – A failed fuel pump was replaced Wednesday, allowing the generator to be restarted. Then the generator failed and needed replacement parts. “325 Hudson St now has building generator working for life safety," the company said early Thursday. "Datacenter power and network connectivity has been 100% restored. Customers are again welcome to gain access."

LGA4 (121 Varick Street) – There was "significant flooding" in the generator room at this site. “The Cummins Generator team has determined that there is unsalvageable damage to the current genset," the company reported Wednesday evening. "We've secured a temporary rollup generator for LGA4 that is currently set to arrive in Manhattan (Thursday).”

Datagram

Datagram reported late Wednesday that is nearly finished pumping water out of its flooded basement at 33 Whitehall in lower Manhattan, but cannot energize the building without a go-ahead from Con Edison, which doesn't appear imminent due to the widespread water damage in the neighborhood. Datagram says a 2 megawatt mobile backup generator is en route to the site and should arrive Thursday afternoon (Nov. 1). The company, which hosts high-traffic sites including Gawker and Buzzfeed, has been offline since the storm surge arrived Monday evening.

Datagram has set up a liveblog to keep customers posted on developments.

111 8th Avenue

There have been several brief generator-related outages at 11 8th Avenue (the Google building) in Chelsea, which is home to dozens of telecom companies' equipment.

Internap reported that its facility at 111 8th experienced an outage Tuesday night when a building-fed fuel system malfunctioed. "When the issue occurred, the fuel pumps could not provide diesel to the rooftop generators, causing them to stop supplying power to our UPS system," Internap reported on its blog. "Once battery backup was exhausted, our infrastructure lost power. The incident caused a loss of IP connectivity for several service points until power to our P-NAP® was restored. At this point, we have connectivity restored for all customers and power restored to the majority of our data center customers. We continue to work with vendors to bring the entire site back online."

Equinix also reported some issues at 111 8th. "Early (Wednesday) morning, our NY9 site experienced a failed generator that impacted service to several customers," the company said in an update late Wednesday. " Repairs were made and service was restored within one hour and 15 minutes."

Intergate.Manhattan

Sabey's Intergate.Manhattan project at 375 Pearl Street was on the edge of the Zone A flood zone, but was spared flood damage. “The building itself came through the storm perfectly,” John Sabey, president of Sabey Data Centers, told the Puget Sound Business Journal. "The neighborhood around Sabey’s high-rise, meanwhile, was without power, he added.

But the storm will delay the opening of the first phase of the project, Sabey said. The plan was to open in January. “This storm will set that back a couple of weeks,” he said.

For a recap of events Monday and Tuesday, see Massive Flooding Damages Several NYC Data Centers.