DeWitt Clauses: Serving the Wrong Master

Creating a benchmark that accurately describes a database's performance characteristics is hard.

People can too easily accidentally or maliciously use benchmarks to misrepresent the true performance characteristics of a database.

Customers might make poor database purchasing and deployment choices if they base their decisions on these inaccurate results.

Although I agree wholeheartedly with the first two points, I don't believe that DeWitt clauses serve customers' best interests. Here are a few arguments to support my position.

1. Extreme Control Issues

Yes, people will accidentally or perhaps purposely publish inaccurate database benchmarks if vendors remove DeWitt clauses from EULAs and allow free testing of their products. Taken to an extreme, however, this line of reasoning can lead vendors of all kinds of products to believe and demand that they must approve any review of their product—to protect the customer from making poorly informed choices, of course. That's clearly extreme. If vendors eliminated DeWitt clauses, I believe the market would quickly become savvy about evaluating the merits of database benchmarks. DBAs would learn how to sift through the useless and misleading information to find the benchmarks that are genuinely valuable. And they would learn how to better run performance tests of their own.

2. Where in the (Real) World?

Today's database-benchmarking offerings don't come remotely close to meeting the needs of real-world database users. Transaction Processing Performance Council (TPC) scores are useful for exploring the theoretical high end of a database's performance. But few of us run applications on 16-, 32-, or 64-CPU systems that cost tens of millions of dollars. TPC and vendor-sponsored benchmarks rarely give us the information we need to tune the 2- or 8-CPU systems that most of us use.

3. Serving the Marketing Master

For better or worse, vendor-sponsored benchmarks typically serve the higher master of marketing goals rather than help customers understand how to best use their systems. Most database users are concerned with making their system run faster—not with comparing the relative performance of one database platform with another, which inevitably is the goal of a vendor-sponsored benchmark such as a TPC-C score.

4. The Advantages of Public Debate and Review

Have you ever conducted a scientific study of what happens to SQL Server performance when you make one tuning or configuration change at a time? Have you fully experimented with changing the number of files in a filegroup to find the optimal setting or bounced back and forth between RAID 0, 1, 5, and 10 to find the optimal I/O settings for your system? Taking the time to run these types of tests in the real world is essentially impossible. Most of us don't even have time to document the systems we're already running. Wouldn't it be wonderful if independent third parties were free to fully explore the ins and outs of database performance tuning by using a public benchmark? Wouldn't the benchmark improve over time, and wouldn't we root out inaccuracies if the benchmark methodology were placed in the public domain for peer review?

5. What Does the Law Say?

Discussions about DeWitt clauses, which are facing legal challenges, might soon be moot. A reader alerted me to a case that the State of New York brought against Network Associates concerning a firewall review that Network World performed. Network Associates claimed the review violated its DeWitt clause and threatened legal action. You can find a summary of the case at http://antivirus.about.com/library/weekly/aa012003b.htm . The short version is that the New York Supreme Court, which said Network Associates' DeWitt clause stifles free speech, prohibited the benchmark disclosure language in the company's existing EULA. Eliot Spitzer, New York Attorney General, had this to say about the case:

"Whether the subject is political debate, debate in the arts and sciences, or debate over what software to buy, we must protect free and open speech from intimidation. The public has a right to information about products, and technologists have a right to create and innovate, so long as they adhere to the law when doing so. After all, we will lose the benefits of life in the Information Age if language contained on Web sites or product packaging prevents citizens from participating in vital exchanges of information."

High-end TPC-C scores serve a necessary and useful purpose, giving us information about how far a database can scale. However, benchmarking of real configurations is sorely underrepresented, with vendors playing a never-ending game of "TPC-C King of the Mountain." I hope vendors will recognize the shortcomings of the current database-benchmarking situation and find ways to meet their customers' needs in the real world. Power to the people! Down with DeWitt!

On a personal note: