Windows Server 2003: The Road To Gold, Part Three: Testing Windows

As the development of Windows 2000 wound down over three years ago, Microsoft was making a transition of another kind: The company's development focus was moving from delivering technology to delivering solutions that met real customer needs. It sounds like an obvious strategy, but consider the ramifications: In the past, Microsoft would determine what features to include in each revision of its products, deliver as many of those features as it could in the time allotted, and then move any dropped features into the next version. Often, the company would tout customer feedback as one of the prime inspirations for the features that were included in each product version, but customer feedback was just one of many criteria that the company considered, and it certainly wasn't the primary one.

One must consider the competitive landscape of the past decade and the markets Microsoft was targeting to understand why this was the case. For much of its history, Microsoft was the fiery upstart, scrambling for market share and paranoid that the Next Big Thing would come along, leaving the company behind. For many of its core markets in the 1990's--desktop operating systems, office productivity software, and workgroup-based corporate computing--Microsoft was primarily concerned with low costs, simplicity, and features, as it sought to outdo competitors such as IBM, Lotus, Novell, WordPerfect Corporation, Borland, Apple, and others. Customer research basically amounted to "more is better": The thought was that most users would compare bulleted lists of features and pick the software that best met their needs. And because Microsoft was often able to undercut the prices of the competition, the choice was usually obvious.

Flash-forward a decade and the computing industry has changed dramatically. Having secured its domination of the desktop operating system and office productivity markets, Microsoft had to make two key changes. First, it had to think up ways to drive sales in those markets where its mature products already dominated. Second, the company had to enter new markets to drive growth. Some of these new markets include handheld computing and other non-PC devices, video games, consumer electronics products, and, of course, the high-end enterprise computing market. Windows NT 4.0, released in 1996, saw some scalability advances, but it wasn't until Windows 2000 that Microsoft made a credible product for the markets that were traditionally served by high-end UNIX boxes. And with Windows Server 2003, the company has finally fulfilled its long-term goal of supplying products that meet the needs of virtually any type of corporation, large or small.

However, the high-end enterprise market runs on a different set of rules than the desktop market or even that of small- and medium-sized businesses, both of which often demand cutting-edge features as soon as possible. In the more mature, slower-moving enterprise market, stability and reliability are valued far above any other criteria, and these operations often employ the same software for tens of years, not two or three years. It was a market with which Microsoft had little experience through the late 1990's. And it caused the company to make dramatic changes across the board. For example, Microsoft has twice extended the support lifecycle of Windows NT 4.0, simply because its customers have been using this technology far longer than the company had originally anticipated. For Windows 2000 and, more obviously, Windows Server 2003, Microsoft understands that customers will likely be running this software for several years to come. Its customer will expect this software to be easy to deploy, administer, and update, of course, but they also expect it to just work, and to just work forever.

Developing simple software that just works is difficult enough, but when it comes to software as complex as Windows Server 2003, with over 50 million lines of code, several different product editions, and an unknowable number of future updates, security fixes, hot-fixes, and service packs, traditional software testing methodology isn't enough. Microsoft realized early on that it could not release this product into the wild, wait for the inevitable problems to arise, and then fix them later, with many customers--as usual--waiting for the Service Pack 1 (SP1) release to start deployment plans. No, Windows Server 2003 would be different, with the Release Candidate 2 (RC2) build considered "final" and the Release To Manufacturing (RTM) or "Gold" release of the product, due April 24, 2003, the equivalent of an SP1 release under the previous development methodologies. Customers should feel confident in rolling out the release candidate, knowing that that the RTM version would be even better, and follow just months later.

Windows Server 2003 would need to be tested in real world situations, and deployed live in the field, and in far larger numbers than in the past, before the final version was made publicly available. But this requires a certain level of trust between Microsoft and its customers, and, for the software giant, a software testing infrastructure that goes far beyond the standard stress tests it employed in the past. And one of the key components of this new testing infrastructure and, indeed, the new Microsoft, is the company's Enterprise Engineering Center (EEC).

Real-world testing in the Microsoft EEC

Itself the result of two years of testing, the Microsoft Enterprise Engineering Center (EEC) opened its doors in April 2002, during the Windows Server 2003 beta. The EEC is the brain-child of George Santino, the Product Unit Manager of Windows Integration Scenario Tests at Microsoft, who wanted to establish better criteria for long term product testing. It's located at Microsoft's Redmond campus, and was designed to let the company's customers duplicate their specific environments in a lab setting and see how various Microsoft software upgrades, migrations, and deployments perform using those company's real-world data and systems. The EEC is made available at no cost to customers.

"We built this facility to allow us to engage with customers while products are still under development," Santino told me, as we walked by a massive bank of servers at the EEC. "We wanted to test products directly with the customers that would deploy them, and understand their network topologies and the issues they're trying to solve. This helps Microsoft make products that are more enterprise-ready for these customers when they ship, and it allows us to engage customers directly." Microsoft benefits from the EEC by ensuring that its products work in the real-world scenarios that, in the past, weren't fully tested until those products were actually released. And customers benefit from one-on-one interaction with the product teams responsible for the products in question.

In many cases, a positive experience at the EEC will result in customers deploying Microsoft solutions in pre-release form. "Customers like jetBlue went off and immediately deployed Windows Server 2003 and Titanium [Exchange 2003] Beta 2 live, in production, after visiting the EEC," Santino told me. "They came here, met with people, tested their environment, and saw that it would work, and be more scalable. We were able to fix issues for them before they left."

In the past, many of these customers might have previously waited until the final product release or, more typically, until the first service pack. "We need to be deployable when products go out to the door," Santino said, "not after SP1, not after QFEs."

The first round of EEC testing involved almost 50 customers, all of whom tested Windows Server 2003 upgrades, migrations, and deployments. Customers typically spend two weeks at the EEC--though the Australian tax office was in for almost six weeks--and Microsoft can host up to five customers at the EEC simultaneously. More important, several problems and 650 design changes were made to Windows Server 2003, thanks to input from EEC attendees. And there have already been a number of repeat customers, such as Intel, Siemens, Chevron-Texaco, and Continental Airlines.

EEC control center montage

Bring on the customers

Physically, the EEC currently includes three enterprise customer labs (a fourth is now being built and a fifth is planned) and a massive server room called the control center that features just about every kind of hardware imaginable. The three labs are filled with IBM, Hewlett-Packard (HP), and Dell equipment, respectively, and are used to host end-user tests. "We let customers come and test their actual environments," Santino told me. "We configure the test environment exactly the same way the customer configures their real-world environment. It uses the same software they use, the exact same hardware, everything. It's their total environment. This helps us find interesting bugs, and we can get developers to come down to the environment, and learn more about what our customers are doing." Once an environment is set up and running in the EEC, Microsoft can perform additional tests, such as migrating from Exchange 5.5 to Exchange 2003, or whatever, build it out, and see what breaks.

One obvious question involves how customers can take part in the EEC. Santino told me that customers come to the EEC from a variety of avenues. "Our account teams in field bring in customers, as do MCS [Microsoft Consulting Service] people, and a number of customers come because they are in the JDP [Joint Deployment Program]. We also have some that come from our hardware partners and other OEMs. Once we get pinged, we love to get 3-4 weeks notice to engage and get information about how to build out the environment, and get all the software installed. But we've done it with just a couple of days notice. We can scurry around and get it done in a few weeks if we have to."

Testing at the EEC isn't confined to a single Microsoft product running against a customer's environment. The company asks customers ahead of time which software products they'd like to test, including their own proprietary solutions, and competitor's software and hardware. "We've got just about every kind of server imaginable in here," Santino says, laughing. "If it's in their environment, it needs to work. Another thing that amazes customers is that they assume we won't test that [non-Microsoft] stuff. But that's not true. If you're using [IBM] DB/2, fine. We will test what they are using in their business."

Also, customers are asked who they want to meet with at Microsoft. "We also have a program manager who essentially lives with them while they're here," Santino said. "We find different bugs because we're working with products that are still under development. But we can get the developer working on that code right here, have him find out what the customer is doing and why. And then he can go back and get a fix before the customer has left the EEC."

While the customers are at the EEC, Microsoft gathers information and writes a business model for each customer, noting how they make money, but also what their issues are, and where their problems lie. "We ask, 'What do you do with SQL Server? Exchange? Tell us more,'" Santino notes. In many cases, the answers are surprising, and Microsoft is discovering that its customers use its products in ways it never imagined.

After the customer is gone

After the customer has left, Microsoft keeps an image of their environment, minus any real-world data. "We keep it around, send out an email to various product teams and tell them, 'we have this environment, how would you like to come and test on it?' The test teams used to just test generic environments, but now they can see how their products react to real-world situations. It's an amazing opportunity." Over time, when a product team is working on various products, they can go through the environment archive and pick the environments that they'd like to test on. "We can build it back up in a couple of days and let them test on it," Santino says.

"Before the EEC, we were limited in how we could test larger engagements, where more products are involved," Santino told me. "It's amazing how these things interact. Most environments we've tested are heterogeneous. For example, Chevron-Texaco was interesting, because they merged two totally different environments. But Barnes & Nobles was almost pure Microsoft, and there were no conflicts."

Customer reaction to the EEC

To help customers see how the EEC works, Microsoft holds regular tours of the facility. "Customers come over from the Executive Briefing Center [on campus] and say, 'we're hearing about how Microsoft listens to customers. Prove it to me.' Well, this is where it's happening. This is where Microsoft works with customers. They can come here, take the tour, see the hardware, and the actual customers doing testing. Ultimately, the EEC helps make products enterprise ready, sure, but it also makes them enterprise credible."

So what's the reaction been like? "Customers say it's been beneficial," Santino said. "They say, 'You guys really do care, you're fixing problems, and I can see it first hand.' They are telling us what they think, and seeing the changes based on their feedback happen in real time. Best of all, there is no cost involved. We're not marketing or sales. It's about shipping higher quality products that the customer can actually use. Ultimately, the EEC will generate sales, sure. But we want to get them in there--at no cost--and we want them to help us understand how they use our stuff. We want real data to stress the products, not a simulation. It has to be real. And the customer knows, if they're going to deploy it, they need to find out if it works now and not later."

Santino says many of the customers who participated in the EEC would live there if they could. "They see the value of coming here and testing, and meeting with developers and program leads. That face time is extremely valuable to them."

Indeed, the customers I've spoken with who have used this facility are uniformly impressed with the quality and depth of the help and expertise they receive during their stay at the EEC. And all of them say they'd come back in the future to test other products. One of those customers, the Kentucky Department of Education, had a particularly interesting story to tell.

The Kentucky Department of Education heads to Redmond

The Kentucky Department of Education (KDE) maintains 1,400 schools in over 175 school districts statewide, a massively distributed computing environment that serves over 600,000 students, and 125,000 educators, staff members, and other employees. Previously, the KDE environment consisted of over 4400 servers, most running Windows NT 4.0, in almost 400 isolated and autonomous NT domains. Obviously, this infrastructure presented a huge problem moving forward to Active Directory (AD), the more modern directory services infrastructure that debuted with Windows 2000. The task of moving this environment forward to one that was more centralized and easily managed fell to Tim Cornett, the Active Directory Architect for KDE, and John Logan, the Exchange Architect for KDE.

"Every district had at least one NT domain, but most had two," Cornett told me. "We have to take these roughly 400 domains and collapse them to 178 domains in one Active Directory forest, and have the entire forest structure up by the end of 2003." When I spoke with Cornett and Logan in February 2003, they had just completed the first pilot program. "The first pilot was here at the Department of Education," Cornett said. "We migrated from NT 4 to Windows Server 2003 RC2." Six school districts piloted in April, and the remaining 170 school districts will migrate between May and November 2003.

Architecturally, the KDE environment will consist of a single AD forest with 178 domains. There will be a root domain, one domain for KDE employees, and one domain for each of the state's 176 school districts. This structure will provide Cornett and his colleagues with the central management that eludes them with the current, scattered, NT 4-based solution.

Why the EEC?

Originally, the KDE worked to upgrade its infrastructure to Windows 2000 on its own. But after spending 18 months evaluating and testing, Cornett realized they'd need some help. The department contacted Microsoft Consulting Services (MCS) to ask about architectural guidance, and hired a full-time technical account manager from Microsoft?s Enterprise Services group. Eventually, the KDE joined Windows Server 2003 Rapid Adoption Program (RAP), which allowed them to begin working with the product early in its development process.

"Microsoft gave us the opportunity to come to the EEC to test our environmnet," Cornett told me. "Because of our size, it was impossible to sit here, throw some servers together and say it works. So we headed to Redmond. Everything was prepared in advance. They pre-built 178 different domains in VMWare sessions using NT 4, on about 40 different servers. And when we arrived, they helped us build the new forest root and work through the upgrade process."

"There were no compatibility issues at all," Cornett told me, "and everything ran really well. They used VMWare GSX Server on Windows 2000. We had the occassional hardware glitch here and there, but overall everything ran really smoothly. The folks there were excellent." While they were at the EEC, the KDE met several times with various product groups at Microsoft. The experience convinced Logan to move the KDE's Exchange infrastructure to Titanium, and it's likely they'll be back at the EEC soon to help facilitate that change. "Our Exchage layout is similar to our NT layout, in that it's complex and spread out," Logan said. "We want to look at greatly collapsing our Exchange topology as well. The Exchange team is very interested in talking to us about that. We're testing Titanium and Outlook [2003] now." Now part of the Rapid Deployment Program (RDP) for Exchange 2003, the KDE will likely roll out a Titanium migration by the end of 2003. The department will also deploy Microsoft Operations Manager (MOM) to help manage the massive infrastructure.

Service, service, service

Cornett and Logan came away highly impressed with the EEC and Windows Server 2003. "Any time there was a Windows Server glitch, within 20 minutes there was a person from the product team that showed up from across campus and started digging into the code," Cornett said. "It was pretty intense. We were there for 5 very long days, though they had offered it for two weeks. But it was right before Christmas. We were there 16-18 hours a day, but they were willing to stay with us 24-7 if we wanted them to. They were excellent. The resources at your disposal are seemingly unlimited. It was pretty impressive."

"Our only regret," Cornett said, "was that we didn't know enough about the lab before we got there. We would have prepared better. We had a plan, but we figured they might have just a few machines, and we'd need to spend days just getting them ready. But when we went there, the number and quality of the machines far exceeded anything we expected. By the end of the first day, we were already at Wednesday on our plan. We could have accomplished more if we knew what resources were available before we went." Indeed, the EEC visit was so impressive, Cornett added, that it helped convert some anti-Microsoft sentiment in the UNIX crowd that went on-sight at the campus. "We took some folks with us that were anti-Microsoft, or extremely skeptical--Solaris guys, mostly. But they came back with a very positive outlook of the product and Microsoft in general."

Cornett said the big concern when they first arrived at the EEC was that Microsoft wouldn't have enough disk space to duplicate their environment, which has an enormous Global Catalog (GC). "They asked us how much space we needed, so we did the math, and it came to about 1 terabyte (TB)," Cornett told me. "On the first day, we left the EEC at 1:00 am on Tuesday, and were back in the lab at 8:30 am that morning. They had run fibre to our room, and given us access to a SAN with two 500 GB drives. We had a need and immediately, it was solved. They said, 'We were here all night but we got it for you.' There was no question or complaint."

Before the KDE left the EEC, Microsoft backed up their environment so that they could put it back online and test it if the department ran into any problems when they did the production rollout. And of course, Microsoft's various product groups can come to the EEC and test their products on the complex KDE environment if desired. "Now they have a simulated environment of our size, minus all the names and other real-world data, of course," Cornett told me. "But they can use it to test, so it's good for both sides."

Meanwhile, back in Kentucky

Now that Cornett, Logan, and the other KDE team members are back in Kentucky, the real work can begin. Before Windows Server 2003 came along, the department was running on everything from aging Pentium 75 servers with 64 MB of RAM to quad Xeon boxes with 4 GB of RAM; all were running Windows NT 4.0. With the upgrade to Windows Server 2003, all of the districts' servers will be dual-processor-capable Dell PowerEdge 2600 machines. During the pilot, each of the servers will be run in single-processor mode to guage processor utilization; if it goes over 40 percent, they'll be upgraded to dual processors. And because they got 6 years out of Windows NT 4.0--it was implemented in Kentucky in 1997--Cornett hopes to get a similar amount of use out of Windows Server 2003. "In the education world we run stuff until its dead," he said. "We're hoping to get a long-term lifespan out of Windows Server 2003 without having to do major upgrading."

Regarding the Exchange migration, Logan told me that the overal goal there was similar to what the KDE is doing with Windows Server 2003: Simplicity of management. "We'd like to collapse our 184 Exchange sites into potentially one routing group, or a few routing groups," he said. "With Titanium, Outlook [2003], and Outlook Web Access (OWA), we will be able to do a significant amount of centralized hosting. We have an infrastructure in place to host all of the districts if we have to. This will help us more easily do backups, perform virus protection, and so on. We really want to centralize the services."

Logan's grand plan is to reduce the current collection of 320 Exchange 5.5 servers scattered around the state to just 20 Exchange Titanium Servers. The only problem with this plan is political, rather than technical, however. Some of the larger school districts might not want their Exchange Servers centrally managed. But for the smaller districts, where there is often just one technical person on staff, responsible for managing, backing up, and protecting Exchange, the centralized management will be quite welcome. There's also a small overall cost savings involved.

One might wonder how a cash-starved educational agency can afford all of this upgrading in this trouble economic times. Logan says it's all about size. "First, each district gets 100 percent matching funds for up to $20 per student per year, so the state matches that, providing a total of $40 per student to spend each year, plus any local funds. That's a large buying power to provide contracts for hardware and software to get it to as low a cost as possible. We get some really good discounts."