Microsoft and Beta Testing

Running beta versions is not for the faint of heart

Tens of millions of lines of code. That's not the amount of work I have to do before the year 2000; it's a measure of the amount of work Microsoft is putting into Windows 2000 (Win2K, formerly Windows NT 5.0). The question users ask me repeatedly about Win2K code is, "How can Microsoft test all the permutations of hardware and software we run into every day on the job?"

The answer is simple: Microsoft can't. Microsoft will test thousands—if not tens of thousands—of hardware and software combinations for Win2K. Undoubtedly, though, your enterprise will have hardware and software combinations that Microsoft hasn't tested, and some of these combinations won't work.

Consider NT 4.0's Service Pack 4 (SP4). Microsoft delayed the release of this service pack for a long time while the company tested, added features to, and retested the product. Yet, when users installed the final version of SP4, quite a few folks ended up with dead systems. Worse than the bugs that manifested themselves obviously were the more insidious problems that didn't show up until weeks later, when a user accessed a particular feature. For example, one of my NT desktop systems ran perfectly with a couple of SP4 betas and release candidates (RCs). Then, with the release code, the system developed a video driver bug. I received a hotfix for the bug, but the experience made me wary of deploying the service pack and ruined my confidence in running unreleased versions of SP4.

How will Microsoft avoid those annoying bugs users discover after the company releases the product? Simple: Microsoft will get users to pay for the privilege of testing the product to find those bugs before the product's final release. On the surface, paying for a new version of software that comes with bugs but without formal support seems foolish, but Microsoft discovered that the prestige of having beta software was enough to get people to fork over cash for the right to tell their friends that they have the beta code.

Win2K Beta
You can get a copy of the Win2K beta code in several ways. You can receive the beta version if you are a major corporate customer or one of Microsoft's beta testers, or you can pay for it. Microsoft isn't officially selling Win2K yet, but you can get it if you're willing to spend money. If you're a Microsoft Developer Network (MSDN) subscriber, you'll receive the Win2K beta with your subscription. If you attend one of Microsoft's Win2K road shows, you can pay $295 to hear about Win2K and receive a copy of the Win2K beta. Finally, you can buy a workstation from a major system vendor, such as Dell, with the Win2K beta preinstalled. However, good luck getting support if you install legacy hardware. It's not that the vendor won't want to provide support—you just might be the first person to run your problem configuration on the vendor's system.

I don't understand why someone would choose to purchase a system with a beta version. Why buy a new system that runs a buggy OS and has almost no vendor support? What happens when your applications don't run? Application vendors probably won't want to spend time helping users running an unsupported OS. However, companies such as Dell are in business to make money, and if those companies are offering a beta configuration, customers must be requesting it. That doesn't mean the idea makes sense, though.

Office 2000 Beta
As Office 2000's release date gets closer (Microsoft plans to ship the software sometime in April 1999), Microsoft has moved into the final stages of the beta cycle. At Comdex, Microsoft announced that the company expected 70,000 users to take advantage of the Corporate Preview Program. Approximately half as many users will sign up for the Consumer Preview Program that Microsoft announced in its Office email newsletter and on its Office Web site (http://www.microsoft.com/office). So, approximately 100,000 users will each send Microsoft $19.95 to test the latest Office 2000 beta. The preview versions will time out in mid-1999. However, you can install them to test your system's Year 2000 (Y2K) compliance if your system clocks are set between December 15, 1999, and March 30, 2000.

If 10 percent of the users who send $19.95 to Microsoft to test the Office 2000 beta actually use the product regularly, and 10 percent of those users actually send reports back to Microsoft, Microsoft will have 1000 beta testers for the preview version of the code. The $2,000,000 that Microsoft collects will probably cover the administrative costs of producing and distributing the CD-ROMs. Microsoft will provide technical support only through a Microsoft newsgroup server, which will probably include a fair amount of peer support. Even if the people enrolled in the preview programs discover only a small number of bugs, these programs are a huge win for Microsoft. The programs let Microsoft truthfully claim that huge numbers of beta testers and end users provided input in the product design cycle. The company can also claim that it provided an opportunity for users to be involved in beta testing.

Beta Testing
I don't want to sound as though I think beta-testing programs are bad. I wrote this article on a system that runs Win2K beta 2 and Office 2000 beta 1. But part of my job is learning as much as possible about new products while they're still in beta. I maintain extra computers on which I perform regular backups, and I'm prepared to do complete OS reinstalls (a task I perform with annoying frequency) at a moment's notice.

Running beta versions is not for the occasional user or the faint of heart. Beta versions make major—and in many cases permanent—changes to the systems you install them on. In addition, beta versions often aren't feature-complete or they lack functionality in crucial areas.

Widespread beta testing is becoming de rigueur in the computer industry. But if you don't need to get involved in beta testing, don't. Testing the latest beta doesn't provide you any benefit other than early exposure to code that might or might not be the same code that the final version of the product ships with. Being able to say you're running the Win2K beta is cool, but the first major system crash you experience might convince you that purchasing the beta wasn't worth the investment. Helping software companies get good products out the door is a great idea, but getting your job done is a more important one.