Record World Cup Traffic Slams Twitter update from June 2010

The 2010 World Cup has lived up to its billing as a watershed event for the Internet, with web traffic from early game easily exceeding previous records from Election Night 2008. The leading casualty has been Twitter, where Tweetstorms during  World Cup games have exposed deeper scalability challenges.

Traffic spikes over the past week have overwhelmed Twitter's internal network capacity, and the performance problems are likely to continue until a long-term fix is found, the company said last night, meaning outages and maintenance downtime are likely to continue through the remainder of the World Cup.

"From a site stability and service outage perspective, it's been Twitter's worst month since last October," Twitter's Sean Garrett wrote on the company blog. "We're working through tweaks to our system in order to provide greater stability at a time when we're facing record traffic. We have long-term solutions that we are working towards, but in the meantime, we are making real-time adjustments so that we can grow our capacity and avoid outages during the World Cup."

Problems Run Deeper
Twitter has been turning off features during traffic spikes to ease the load. On Tuesday, for example, Twitter's trend tracking was turned off during the Brazil-North Korea game. But those tweaks are addressing symptoms of larger scalability problems.

"As we go through this process, we have uncovered unexpected deeper issues and have even caused inadvertent downtime as a result of our attempts to make changes," wrote Garrett, who said Twitter had fully expected capacity challenges during the World Cup. "What we didn't anticipate was some of the complexities that have been inherent in fixing and optimizing our systems before and during the event."

Traffic Hits All-Time High
Web traffic on Internet news sites hit a record 12.1 million visitors per minute on June 11 during the opening games of the World Cup, according to Akamai's Net Usage Index.  That easily exceeded the previous record 8.6 million visitors per minute on Nov. 4, 2008 as Barack Obama was elected president. Yesterday's match between Brazil and North Korea hit 10.6 million visitors, suggesting that record traffic may be a daily event through the World Cup finale on Sunday, July 11.

That traffic has surfaced structural problems with the design of Twitter's internal network, the engineering team wrote last Friday. Jean-Paul Cozzatti reported that the company put two fast-growing, high-bandwidth components on the same segment of its internal network, which wasn't adequately monitored.

In response, Twitter has doubled its internal network capacity and improved its load balancing and network monitoring. That still hasn't been enough to avert performance problems, and there are more outages ahead as Twitter seeks to address its challenges.

"Over the next two weeks, we may perform relatively short planned maintenance on the site," said Garrett. "During this time, the service will likely be taken down. We will not perform this work during World Cup games, and we will provide advance notification."

The long-term fix? Twitter has announced plans to operate its own data center. We're likely to hear more about Twitter's problems and plans at next week's Velocity 2010 conference, where Twitter's John Adams is scheduled to present.

Here’s more of our coverage of Twitter’s infrastructure: