Do Betas Make Sense Anymore?

You probably read the late-October story wherein Microsoft had to delay Windows Vista's release by a couple of weeks because of the last-minute appearance of an egregious bug. Apparently, under some circumstances, upgrading a Windows XP system to Vista with the almost-released code could cause you to lose everything on your hard disk. I’d say that’s pretty much a picture-in-the-dictionary definition of egregious. Sure, the normal response to the bug is either an involuntary "yikes!" or a deprecating chuckle at Microsoft's expense. However, to me, the big question is, "How did the Vista beta go so long without anyone noticing the bug?” I think I know the answer: Betas don't make sense as testing tools anymore.

Once upon a time, it was easy to be a beta tester, for a couple of reasons. First, OSs and applications were much easier to install. When a beta version of a new DOS came out, you could test much of it by just creating a bootable floppy version of the new OS. Instead of having to blow away a whole system to test the soon-to-be released OS, you could just boot from the floppy and play to your heart's content. Being a beta tester for something like DOS 3.3 or dBase IV simply didn't require the level of involvement that modern software testing does. Heck, as recently as 1994, I took down a server just so that I could use the hardware to test the Windows NT 3.5 beta. It wasn't much of a hardship because rebuilding the existing NT 3.1 "production server" was so simple and quick.

Wait a minute. Wipe a production server clean to use its hardware to test a beta? Sure, and here we come to the second reason why it used to be so easy to play with betas: We just didn't depend as much on microcomputers as we do now. In my case, the NT 3.1 server's sole job was to act as a file and print server for my company, and it was Saturday. Everyone was off for the weekend. Reinstalling NT 3.1 and reloading the backed-up data from a single tape was easy and quick, so where was the harm?

Clearly, things are different today. Taking down a server for any reason for a few hours would probably be a career-limiting move in today's Internet-connected world. And even if our servers weren't on call 24x7, I sincerely doubt that many people would be willing to try beta software on anything except completely disposable hardware. Rebuilding either a server or a workstation is a significant task in today's software world. Add to that the time necessary for even the simplest OS installation, and you can see the reason why a fairly small percentage of the PC-using population even considers testing betas. I was once perfectly willing to rebuild an NT 3.1 file server, but I’m extremely wary of risking my XP workstation and its applications' configurations to a beta OS.

It's no wonder that apparently few test Vista systems were loaded atop existing Windows configurations. Now, take that notion a step further and ask yourself, "What else probably doesn't get tested?" That question makes me wonder about Exchange Server.

I can easily see a curious Exchange administrator finding an old machine or creating a virtual machine (the one thing about the modern world that does favor betas) and setting up a test Active Directory (AD) domain, then—atop it—a test Exchange server in some imaginary organization. It would involve a lot of setup, but anyone with an interest in Exchange would probably find it worthwhile.

But come on, Minasi's Fifth Law of Networks is Everything works fine in small networks, and any test Exchange system is by definition going to look good. The beta rubber doesn’t really hit the road until you plug in an Exchange 12 server into the production network, join it to the domain, and start handling some real email. And, again, who's crazy enough to do that? Well, okay, Microsoft is crazy enough to do that, having lived with Vista on everyone's desktop and Exchange in everyone's mailbox. But, then, there's a profit motive for the company, in that case—a motive that most of us don't have.

Now, I'm not an Exchange guru, but folks that I know who are Exchange gurus are pretty excited about Exchange 12, and lots of other folks are excited about Vista, so I guess this beta-testing stuff is working somehow. I just wonder how much longer it'll be of any bug-stomping value. Time will tell, I suppose. A more cynical author might opine that the main value of public betas was for their marketing value, but I'm not nearly that cynical.