Power Outage Knocks DreamHost Customers Offline update from March 2013

Web hosting provider DreamHost experienced an extended outages when power systems failed at its data center in Irvine, Calif. The incident created hours of downtime across Tuesday and Wednesday for many of DreamHost's more than 350,000 customers, who host 1.2 million blogs, websites and apps with the company.

The problems started at 3 pm Pacific time on Tuesday, when the uninterruptible power supply (UPS) system failed suddenly at the Irvine facility operated by DreamHost's data center provider, Alchemy Communications. When the UPS systems died, the emergency back generators also failed to start properly.

"The power failure lasted just a few minutes, however it created a number of major issues with our network and systems in the Irvine DC that took many hours for our operations teams to recover from," wrote DreamHost CEO Simon Anderson on the DreamHost status blog ." Not the least of which was the loss of several critical pieces of networking hardware which did not survive the power event."

Second UPS Issue Prompts More Downtime

Anderson said Alchemy has a "good track record" in maintaining uptime at the Irvine facility, but may have been conducting unannounced UPS maintenance. At 4:30 a.m. Pacific on Tuesday, the UPS systems failed again.

"This resulted in another complete power outage and another intense period of reboots, restores and system checks from our team," said Anderson. "The time to restore most services in the wake of this second power outage was much quicker, mainly because there were no resulting hardware failures and we had learned from the first failure. Alchemy has opted to run the Irvine DC on generators until the UPS issues are fully identified and resolved."

Anderson apologized to customers for the outage and said that DreamHost would offer service credits to affected customers.

" I fully recognize that any disruption to services can affect important production environments and projects," the CEO said. "Our team will work diligently to ensure that we mitigate the power issues going forward, including a full audit of all facilities that house DreamHost customer data. We will learn from this event and continuously improve our operations and services."

DreamHost has been involved in a number of high-profile outages over the years, but those incidents never seemed to slow the growth of the company, which has focused on affordable web hosting accounts.

Last year DreamHost expanded its data center footprint to the East Coast, adding a presence in Ashburn, Virginia as part of a broader effort to improve its reliability and boost network performance.