Skype: Overloaded Servers Caused Ripple Effect update from December 2010

Skype has published a detailed explanation of the causes of last week's massive outage for the Internet telephony service. The root cause: a cluster of servers became overloaded, causing congestion and capacity problems that were exacerbated by a bug in the latest version of Skype's Windows software client.

As a result, the problems rippled through Skype's peer-to-peer infrastructure, taking out many of the supernodes that are key to the smooth operation of Skype's network. A supernode acts like a directory,helping to establish connections between Skype clients and creating local clusters typically of several hundred peer nodes.

"Although Skype staff responded quickly to disable the overloaded servers and to eliminate client requests to them, a significant number of supernodes had already failed," writes Skype's CIO Lars Rabbe. "Once a supernode has failed, even when restarted, it takes some time to become available as a resource to the P2P network again. As a result, the P2P network was left with 25–30% fewer supernodes than normal. This caused a disproportionate load on the remaining available supernodes."

Rabbe also outlined the steps Skype will take to prevent a recurrence, which focused primarily on better testing of software updates and improved bug detection processes. There also may be additional spending on core infrastructure. "We will keep under constant review the capacity of our core systems that support the Skype user base, and continue to invest in both capacity and resilience of these systems," Rabbe wrote. "An investment program we initiated a year ago has significantly increased our capacity already and more investment is planned for 2011."

Read Post Mortem on the Skype Outage for details.