Toward More Reliable Software

Public bug lists would make purchases less risky

Microsoft President Steve Ballmer recently commented that the final release of Windows 2000 (Win2K—formerly Windows NT 5.0) will have 40 million lines of code. Forty million lines of code! I'm sure glad I don't have to manage the Win2K project. Compared to the new OS, the program that runs the space shuttle looks as complex as Notepad.

The other day in the shower, I remembered an old statistic: The average commercial software has 15 bugs per 1000 lines of code. This statistic means we can expect Win2K to have about 600,000 bugs. Yes, you read that right. If Microsoft ships Win2K with more than half a million bugs, the OS's quality will be par for the industry.

No product is free from defects, but software could be much less buggy if developers spent more time fixing bugs and less time adding new features. Evil, uncaring people don't run the software industry—quite the contrary. In interviewing many of the big names in the business, I've found that software executives believe that they're doing exactly what customers want. No less a personage than Bill Gates said a few years ago that no one would buy a software upgrade that was simply a bug-free version of a previous release. In classic Gatesian fashion, he called that notion "the stupidest idea I've ever heard."

If bugs are a fixture of the software landscape, what can unwary travelers on the Road Ahead do to simplify the use of buggy products? I have a modest proposal: Why not require software vendors to reveal their lists of known bugs?

You might wonder what bug lists I'm talking about. Well, you probably won't be surprised to learn that vendors don't always wait until they have stomped out the last bug before they ship a software package. Instead, they keep fixing the bugs they know about until a deadline arrives, at which time they release the product, sometimes with thousands of known bugs. Good vendors release buggy products. Bad vendors don't keep bug lists because they don't try to find and fix bugs—a survey of software development firms reported that one in every seven firms ships code without ever testing it!

Vendors are entitled to ship imperfect products (and whether companies that ship imperfect products should stay in business is a topic for another day). However, if a vendor ships a defective product, the company should share its list of known defects with prospective buyers. Software vendors already know about more than 90 percent of the bugs that users report. Have you ever spent a couple of days isolating some odd behavior in a product, only to hear from the vendor's technical support department that they knew about the bug—but they wouldn't acknowledge the problem until you proved it? This lack of communication is tremendously aggravating and time-consuming. A little preventive information from vendors could save administrators much frustration.

But is my proposal realistic? What company would post its products' defect lists for customers to see? Lotus recently listed known bugs for one of its new products. (Bravo, Lotus!) If expecting all companies to be so forthcoming isn't realistic, then the government could require bug disclosure by law. All software vendors would have to comply, so exposing bugs wouldn't be as treacherous for firms; the playing field would be level. Vendors would have strong incentives to lie about their products' bugs, but the government could uncover cheaters.

Consider the computer industry's approach to another honesty issue: software piracy. How can software vendors possibly hope to expose a firm that uses more copies of an application than it buys? The answer's simple: Recruit whistle-blowers. One of the largest software industry associations, the Business Software Alliance (BSA), solicits unhappy employees with a series of ads the association calls its Nail Your Boss campaign. National Public Radio reported that the BSA is even paying employees in England to turn in their employers.

The software industry has had excellent luck recruiting the disgruntled to identify software pirates. A similar campaign could identify vendors who lie about their software's quality. Sauce for the goose....