Diesel: The Lifeblood of the Recovery Effort update from November 2012

sentinel-generator

An example of a diesel backup generator in a New Jersey data center. (Photo: Rich Miller)

Much of the Internet is currently running on diesel fuel and priority service contracts. Emergency backup generators powered by diesel are helping many East Coast data center providers weather Superstorm Sandy. The importance of diesel in the post-Sandy economy is hard to overstate. In the wake of Sandy, diesel is the lifeblood of lower Manhattan, where generators are currently providing virtually all the area's electricity.

Data centers often tout their ability to operate for days, if not weeks, on diesel reserves stored on site in case of disaster. There are two potential problems that can arise when a data center relies on diesel generation to get it through a disaster. The first is running out of diesel fuel. While data centers are preferred customers and have contracts in place, there are others ahead in the line such as hospitals and water plants during disasters. There's also the challenge of fuel contamination, which is closely tied to maintenance practices.

Issues and Complications

First, let's start with a high-level primer on diesel generators. Data centers spend millions of dollar on their emergency power so they may continue to run during events like  Sandy. Should power go out, first a UPS kicks in while the powerhouse of emergency power, the diesel generators, get warmed up. A diesel generator is a combination of diesel engine and electrical generator, and is used in lieu of the power grid during a utility failure. Once the generators are running, a data center is still not totally in the clear. Issues and complications can arise when switching back and forth to emergency power, as well as while the generators are running.

As we’re seeing with Sandy, an extended outage may result in some sites burning through their initial reserves. That's what happened to Internap, for instance, which ran out of diesel at 75 Broad and had to seek out a refueling arrangement and workaround. Flooding submerged and destroyed the building's diesel pumps, further complicating the problem. Peer 1 Hosting, on the other hand, reported that its diesel was lasting longer than expected and has extended its generator life by using a human bucket brigade in place of fuel pumps.

The switch over to generators isn’t always smooth, as issues can arise during and after the switchover, as illustrated by a series of outages in 2010 in which automatic trasnfer switches failed to handle the transition properly. However, most data centers seamlessly switched over during this storm.

Supply and Demand

During extended disasters, the supply of diesel can be strained. Although data centers support many critical communications services, they aren't alone in needing priority access to fuel. NYU Langone Medical Center’s backup generator failed, forcing a mass evacuation of patients. So there are higher priorities, and in a serious situation like this one, there is a chance that immediate supply might run low, straining even those with preferred contract. Reuters recently noted that low stocks of some diesel distillates have affected pricing this fall.

Additional diesel is delivered by drop trailers and iso tanks; however, these vehicles need to be able to reach the delivery site. In the cases of extreme flooding, even when having a preferred contract, the issue of getting the fuel to the data center can be an issue. That's a particular challenge in Lower Manhattan, where tunnel access is limited and some roadways remain flooded or are blocked by stranded vehicles.

Hidden Threat: Fuel Contamination

While the supply of fuel is the most critical consideration, another unforeseen problem is fuel contamination. Diesel reserves often sit for long periods of time, which is the opposite of the optimal scenario, in which diesel fuel is used within two to three weeks of leaving the refinery. Because of this, oil companies are not compelled to produce a diesel fuel meant for long-term storage.

Due to tighter regulation and economic concerns, fuel is becoming more unstable and contaminable, industry sources say. Should a disaster strike, there is a chance that diesel reserves are old, leading to premature shutdown of generators. For this reason, data center providers should test fuel samples on an annual basis, and refresh their reserves to prevent this. There are also treatments, such as adding biocides to control microbial growth, chemical additives, and filtering to remove water and sediment that builds up over time.

The NFPA 110 refers to diesel fuel "storage life" of 1.5 to 2 years. "Tanks should be sized so that the fuel is consumed within the storage life, or provision should be made to replace stale fuel with fresh fuel," the standard recommends. The current refining process means that an even more conservative standard might be needed.

An additional precaution is testing the generators on a regular basis to see if they can handle the load at partial and/or full capacity. The generator itself needs to be brought to full load on a regular basis to prevent damage to it. While testing is integral, there have been issues in the past stemming from failed testing.

So data center providers need complex backup power systems including massive diesel generators, intelligent diesel fuel management and reserves on site, and priority service contracts. Even then, some problems can’t be avoided – especially if your generators are in the basement in a flood zone.