Microsoft is Upgrading Hotmail's Storage Infrastructure

Microsoft this week described a major update to the storage infrastructure for its Hotmail web-based email service, in which it will be moving away from RAID and to a scheme called JBOD, or "just a bunch of disks."

"Hotmail’s storage system supports over one billion mailboxes and hundreds of petabytes of data (one petabyte is a million gigabytes, or a million billion bytes)," Hotmail architect Kristof Roomp wrote in a blog post describing the changes. "The system services hundreds of thousands of simultaneous transactions from across the world [and] just like the rest of Hotmail, our storage system is built using Microsoft technology, including Windows Server and Microsoft SQL Server."

According to Roomp, Microsoft has recently begun a major storage upgrade to Hotmail in which it will switch from the technically dated RAID-style storage it's currently using to a new system called JBOD, which he says is essentially a distributed RAID set up, but implemented completely in software.

"The software we developed for the JBOD system monitors the hard drives schedules repair actions, detects failures, and diagnoses repairs, he writes. "If the watchdog detects the failure that it is looking for, it raises an alert, which automatically triggers a repair process. This repair process can range from rebooting a machine or restarting a process, to fixing data corruption or even involving a human if progress can’t be made."

Additionally, Microsoft is moving to SSD (solid state drive) storage because traditional hard drives can no longer meet Hotmail's performance requirements. But because SSD is so expensive--10 to 100 times more than traditional hard disks of the same capacity--Microsoft will be using SSD only for the most performance sensitive operations, while still using larger, traditional hard drives for storing messages.

These Hotmail storage updates will occur over the next couple of months, and Roomp says that 30 million users are already on JBOD today.

Having recently spent some time in one of Microsoft's data centers and discussing Windows Server 8, I'm curious to see when Hotmail--and other Microsoft online services--take advantage of some of the next generation storage technologies I saw there. For example, Microsoft is testing what is essentially a RAM-based storage array that has the same performance advantages over SSD as SSD does over regular hard disks. And Windows Server 8 includes a data deduplication feature that could significantly improve how Microsoft handles data storage going forward.