What You Need to Know About How Windows Server 2008 Developed - 28 Nov 2007

With Windows Server 2008 heading toward first quarter 2008 launch, I sat down with Alex Hinrichs, the Server 2008 project manager, to discuss the development of the product and how changes to it will benefit customers going forward. There’s no doubt about it: Server 2008 is the most customer-driven version of Windows Server yet, and though it’s a major revision that will surely bring with it some compatibility problems in the short term, the long-term benefits are obvious and unassailable. Here’s what you need to know about the development of Server 2008.

Hinrichs Maneuver
Much of the work around Windows Server occurs in Building 43 on the Microsoft campus in Redmond. Alex Hinrichs runs the Windows Server ship room and manages the development of this increasingly complex product line that affects almost every level of Microsoft’s customers. He sets the schedule, defines the processes, and is the point man for any decisions that need to be pushed up the ladder to his superior, Iain McDonald, or Bill Laing, the general manager of Microsoft’s Server Business.

A 12-year Microsoft veteran, Hinrichs was originally hired as a Windows NT program manager. He worked as the release product manager for all Small Business Server (SBS) releases from SBS 4.0 to SBS 2003. On the heels of SBS 2003’s completion, Hinrichs jumped over to Windows Server.

Define Servers by the Roles They Perform
The fundamental change in Windows Server development has been the move to a roles-based management architecture. Microsoft began working on nascent versions of this architecture as far back as Windows 2000, but it wasn’t until Server 2008 that the OS was finally componentized, allowing the management roles inside the product to map both to the underlying architecture and to the product groups working on Windows Server.

“The thing we really hang our hat on is that we’ve had a clear vision for Windows Server 2008 from the beginning,” Hinrichs said. “We talked to our customers and they don’t think of the product in terms of product versions, but rather about the server boxes. In their data center and server rooms, they can point to different machines and say, ‘That’s the domain controller, that’s the file server, that’s the Web server, and that’s DHCP. 'That’s how they think. The problem is, we produce a Swiss Army-knife kind of server product that does a bunch of different things. But customers wanted to define their servers by the roles they performed.” Thus, the roles-based architecture in Windows Server was born.

“This also allows us to better engineer the product from the start,” Hinrichs added. “Roles define things from the beginning, and deep componentization means we can install as little functionality as possible by default and give admins only exactly what they need.” Even Microsoft’s engineering teams, with few exceptions, are organized by roles. “We have general managers and product unit managers whose job, literally, is to manage things like the Terminal Services business, the Active Directory business, the IIS business, and so on,” Hinrichs said. “We engineer the product soup to nuts to make that happen. Again, it’s a very clear focus and vision that helped us scope the product and make the right decisions.”

Manage Complex Product Changes
From a process perspective, Server 2008 has been in development longer than any previous version of Windows Server. To manage such a complex product over many years and not run into the problems that, quite frankly, were rampant with the Windows Vista team, is an achievement. In stark contrast to that of Vista, Server 2008 development was steady, sure, and without controversy. “

The way we get to this point is that we had a clear feature list from the start,” Hinrichs told me. “We locked down the bulk of the features we wanted to include in 2005, and did a final triage on features back in December 2006. Of course one or two things did trickle in over time, but the feature list was locked and loaded at end of 2006.”

By doing so, the Windows Server team was able to hit its milestones as well as an impressive quality bar without having to “churn the code” in response to unexpected feature additions. “We just have so many OS components,” Hinrichs said. “If you make a change to some low-level component, components that are high on the stack can be affected. So you have to lock it early. It reduces the amount of shifted sand.”

This clear vision of a roles-based product lets Microsoft steer a ship that would otherwise be unwieldy. “We can tell a few thousand people [in the Server division] that this is what we are doing, and then they can execute on it,” Hinrichs said. “The scope of our customer base is pretty big, from small businesses to the enterprise. But it’s not as big a challenge as what you see on the client [i.e., Vista] where the scope runs from the consumer all the way up to the enterprise desktop.”

Replay the Positives from Windows 2003
The development of Server 2008 was also affected by lessons Microsoft learned from previous product versions. “What we tried to do with Windows Server 2008 is take the positives from the Windows Server 2003 playbook and replay those,” Hinrichs said. “And then we avoid the mistakes. We know what’s working because customers tell us it’s good.”

The most important lesson was to deploy early and often. Microsoft knew that some of the more “disruptive” server roles-Web server, Active Directory (AD), Network Access Protection (NAP), Terminal Services Gateway- needed to be rock solid before they could be shipped in final form to customers. So these roles got at least 18 months of testing internally at Microsoft and externally with its Technology Adoption Program (TAP) customers. “Over time, we’ve discovered that that’s the magic time period,” Hinrichs noted. “So that’s what we’ve done, and over two years in some cases. We deployed Beta 2 [back in 2005] within Microsoft and externally with 50 customers we watch very closely. They get weekly calls and extra funding and people out to visit them on the road regularly. They were running [Server 2008’s] Active Directory two years ago, just like [Microsoft].”

Continued on Page 2

Feedback from the TAP deployments led to a dramatic improvement in quality as Microsoft fixed bugs related to reliability and usability. “We’d get admins telling us that certain UIs didn’t make sense,” Hinrichs added. “Eventually we got to the point where the Active Directory role was so stable that every Monday we’re updating our Windows development domain with the Monday build. It’s in such good shape. But you can only do that with 18 months of deployment to validate that it’s ready.”

Realistically, Microsoft realizes that even the most stringent beta test process won’t uncover all bugs because some issues simply don’t crop up until you’ve gotten the product out in the real world. “We can’t predict everything,” Hinrichs told me. “So the only way to make sure is to deploy broadly. We built that religion with Windows 2000/2003, and it’s the mantra we live by for 2008.”

Microsoft IIS is another good example. Microsoft has been incredibly aggressive deploying IIS internally and externally, and Microsoft.com has been running on the IIS version in Server 2008 for years now. Microsoft also pushed the IIS Go Live program to customers as early as Beta 3. “The message is simple,” Hinrichs said: “Deploy, deploy, deploy.”

A Compartmentalized Build Process That Flows
Internally, Microsoft has restructured the build process for Server 2008 so that the process, like the product itself, is more compartmentalized. A Main OS build is created every day, as with previous product versions, but the process of getting revisions into that Main build is far more granular than before.

Under the server roles group, for example, you’ll see subgroups such as the AD team, the Terminal Services team, the IIS team, and about 20 others. The developmental lead on the AD team checks in code at the server roles level while inheriting code from above. After that code is ready for broader consumption, it’s checked into a higher branch and consolidated back into the Main development tree. This process is ongoing, obviously, and requires people at each level who can be trusted to monitor the quality and necessity of new code additions.

“There’s code flowing up and down the tree nonstop,” Hinrichs said, “but we maintain high quality at both levels by a set of quality gates. These gates are looking at BVTs [build verification tests], a battery of tests against subbuilds that make sure something the AD guys do doesn’t break other things or prevent other teams from testing. Everything has to work.”

Microsoft also runs code quality tools which look for security bugs, buffer overruns, and anything else that might cause problems. There are also code dependency checks- some 40-odd layers of dependencies between components, Hinrichs told me. “To maintain the componentization of the OS, you have to make sure you aren’t unwittingly breaking dependencies.” The goal of all these tests is to catch these issues far down on the tree so that they affect the smallest possible group of developers. “All these tools run overnight,” Hinrichs said. “And we get status reports in the morning. We can see where different teams are.”

Because of the componentization of the development process with Server 2008, the ship room strategy has also changed since Windows 2003. “It’s more evolved now,” Hinrichs said. “We don’t just have the main ship room. Now we also have seven distributed ship rooms, run by people who meet with the people checking in code below them. They all have daily meetings, as does the main ship room. The main ship room’s agenda is simple: Who in the seven distributed ship rooms is ready to bring code up [the tree into the Main build]?”

While the main ship room is still used for triaging code bugs, many of these bugs are now handled lower in the tree, so the main ship room’s emphasis has changed somewhat. “We communicate what the focus is, the testing we’re doing, but we have to rely on local expertise [lower in the tree],” Hinrichs said. “It’s much more distributed now, with more local ownership. The system is just so big. As you can imagine, the people in the middle tier have awful jobs, awful. They have to work up and down the tree and end up working their butts off. They have over 20 groups below them and me on the top. It’s a very, very tough job.”

Looking Ahead
In future issues of Windows IT Pro and on our SuperSite for Windows, I will continue this behind-the-scenes look at the development of Server 2008. Stay tuned for more information about Microsoft’s internal build process for this very complex product.