Do Betas Work?

Time for a new approach to quality in software

[Editor's Note: "Do Betas Work?" is Mark's last En Garde column in Windows 2000 Magazine. You can find more of Mark's opinion columns in the free email newsletter Windows 2000 Magazine UPDATE Special Edition. To receive UPDATE Special Edition (look for it on the last Friday of each month), go to http://www.win2000mag.com and subscribe to Windows 2000 Magazine UPDATE.]

Bugs. We hate 'em. Whether you're running software from Microsoft, IBM, Novell, or another vendor, you've probably struggled with serious organizational or project setbacks because of defects in software. You probably heard about the internal Microsoft memo that claimed that Windows 2000 shipped with 63,000 bugs. If Win2K did ship with only 63,000 bugs, Microsoft should take a bow. Given the size of Win2K and standard industry quality expectations, Microsoft could have shipped a product with 10 times as many bugs and not blushed. The software industry considers about 15 bugs per thousand lines of code to be normal. If you believe the rumors that put Win2K at approximately 40 million lines of code, simple math reveals that we could expect 600,000 defects if Microsoft wrote Win2K to "normal" standards.

Software firms try to achieve quality in their products through testing. Some testing involves automated testing scripts that run interim versions of the program under development on a variety of hardware. Another kind of testing is beta testing, by which software firms let their customers see preproduction builds of upcoming software. But most software firms I've spoken with claim that outside beta testers don't find all that many bugs. I recall Frank Artale, then a program manager at Microsoft, reporting in 1996 that outside beta testers found fewer than 1 out of 10 of the bugs uncovered during Windows NT 4.0's testing process.

I suspect that the true value of most betas is marketing: Beta testing involves customers in the development process, so customers feel that they're part of the product's development. During beta testing, the software vendor is receptive to customer suggestions. Because many of those suggestions are often good, the product ends up better overall. Perhaps we ought to stop calling the process of early customer involvement "beta testing" and call it something like "previewing."

Why bother with this linguistic nicety? Because many software quality experts think that the only way to stop shipping products with hundreds or thousands of defects is to stop using testing as the main road to quality. One software quality expert, Watts Humphrey of Carnegie Mellon's Software Quality Institute, puts it this way (in "Comments on Software Quality," http://www.2bguide.com/docs/whsq.html): "No technologically intense industry, other than software, relies exclusively on product testing to remove defects. It has been known for over 20 years that testing is an expensive and ineffective way to eliminate defects from any product, including software."

If we don't use testing to create quality software, what can we use? Software engineers have created many approaches over the years, so I can't do justice to the alternatives here. But most alternatives to testing borrow from more than 50 years of experience in improving industrial processes. Look at the problem this way: What if Boeing adopted a software engineer's approach to designing a new jet? ("So, Smithers, are we ready to ship the new 787?" "Looks good to me, J.B. None of the beta testers have reported any plane crashes in the past month or so.") Instead, Boeing employs a development process that emphasizes design over testing. That way, Boeing doesn't need to build a lot of beta test jets that don't work. Software vendors could benefit from this approach by spending more time designing their code and less time typing it in, running it, and waiting to see whether it blows up.

But can this approach work? Motorola employed a quality design process while it was developing the software for the popular StarTAC cellular phones. The result? In 2MB of code, no more than 24 known defects exist. Perhaps one day a major software vendor will employ nonstandard quality approaches to the development process. One day, perhaps.