Diesel 'Bucket Brigade' Keeps Peer 1 Online at 75 Broad update from November 2012

squarespace-75broad-bucket

Teams from Squarespace fill buckets with diesel fuel to haul them up 17 stories to the generator keeping the data center online. Staff from Peer 1, Squarespace and Fog Creek Software have formed this unusual Internet bucket brigade. (Photo via Squarespace)

A determined team of employees of Peer 1 Hosting, blog host Squarespace and Fog Creek Software have formed a "bucket brigade" to relay 5-gallon buckets of diesel fuel up 17 flights of stairs at 75 Broad Street to refuel a generator providing emergency power to the Peer 1 data center.

It is a decidedly low-tech, brute-force solution to the challenges at 75 Broad, the hobbled data center building in Lower Manhattan. Strong arms and strong backs are replacing fuel pumps that normally bring diesel from tanks in the basement to the upper floors of the building, where the backup generators are providing emergency power to keep the servers online in the Peer 1 data center. Those pumps shut down yesterday when the storm surge from Hurricane Sandy flooded the basement levels of 75 Broad, rendering the pumps unusable.

Peer 1 initially planned for a "controlled shutdown" at 10:45 a.m. While the generators chugged on past that point, the company and its customers developed an alternate plan -

""We have 25 people stationed throughout the building to relay more fuel to the roof once it arrives," Squarespace reported on its status blog. "The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts, joined with the building, can see us through. ... "Spirits are strong and everyone from Peer1, Fog Creek, and Squarespace is working together."

Squarespace acknowledges the effort is "not sustainable" over the long haul. Both Peer 1 and Internap are working on long-term solutions should the pumping and repairs require extended time. But the bucket brigade reflects an unusual physical embodiment of the depth of commitment to keep data centers online, come what may.

UPDATE: As of 1 p.m. Eastern the generator's day tankw as full, allowing the bucket brigade to take a 90-minute break. But little progress was being made on long-term resolutions. "The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough," Squarespace reports. "They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply."

Perhaps more problematic is the stats report on the flooding. "The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point."