All's Fair?

More damn lies about benchmarks

If anything is more suspect than a benchmark, it's the marketing hype that accompanies benchmarks. Microsoft released its official response to Oracle's million-dollar challenge, which stated that Oracle would pay $1 million to anyone who could demonstrate that SQL Server wasn't 100 times slower than Oracle 8i on Query 5 of the TPC-D 1.0 benchmark. Oracle shrewdly stacked the deck in its favor by picking a benchmark that took advantage of an Oracle feature, materialized views. Materialized views essentially perform a preaggregation on the database, permitting OLAP-like performance in a benchmark where OLAP (and therefore Microsoft's OLAP Services) isn't allowed. And Oracle's use of a $10 million Sun Enterprise 10000 server system to achieve its 71.5-second response time for the benchmark didn't hurt.

The irony of benchmarks is that as soon as a vendor posts one, it's fair game. IBM one-upped Oracle and Microsoft with its subsecond benchmark in TPC-D Query 5. IBM used a 32-node cluster of 4-way Netfinity servers (which cost a mere $6 million).

Microsoft issued its response after Oracle's Larry Ellison closed the contest in February. Microsoft explained how it partnered with HP to build a 1TB TPC-D database query that answered the "same business problem" that comprises TPC-D Query 5. The solution consisted of an HP NetServer LXr 8000 with four 450MHz processors, 4GB of RAM, and nine HP NetRAID-3Si disk-array controller cards attached to 560 disk drives running SQL Server 7.0 Enterprise Edition with OLAP Services. Total price: a relatively modest $600,000. The result was an execution time of 1.075 seconds on TPC-D Query 5. Microsoft didn't meet Oracle's deadline, but it met and exceeded the challenge with a solution that cost less than Oracle's.

In the twisted world of benchmarks, Microsoft's response is fair enough. It addresses the perceived intent of the Oracle challenge. Oracle's use of materialized views follows the letter of the benchmark, but the ability to perform preaggregation subverts the benchmark's goal of measuring a database's ad hoc abilities. Microsoft's use of OLAP Services means that this highly publicized response also didn't follow the letter of the TPC-D benchmark. The General Council of the Transaction Processing Performance Council now has recommended that two new benchmarks replace the existing TPC-D benchmark. The new TPC-R benchmark will continue to allow the use of materialized views, whereas the TPC-H benchmark will permit only true ad hoc queries.

Although the Microsoft response used the same TPC-D database and performed the same query, this response wasn't a real TPC-D benchmark and it didn't address the Oracle challenge. But the response does address the notion that SQL Server is slower than Oracle 8i. Two wrongs don't make a right, but it appears that sometimes they cancel each other out.