What If You Had a Benchmark and Nobody Came?

The SQL Server 2008 launch event in Los Angeles, François Ajenstat, a director of product management for SQL Server, showed me some of the impressive benchmark numbers that were generated by the new release. Benchmark testing is pretty much obligatory for a new database release because it enables customers to compare the new release to other database platforms. Let’s take a look at some of SQL Server 2008’s benchmark scores.

As loyal SQL Server customers have come to expect, SQL Server 2008 running on the latest high-powered x64 multi-core hardware set several new benchmark records. Most notably, SQL Server 2008 running on Windows Server 2008 recorded Microsoft’s first-ever result in the 10TB category of the TPC-H decision-support benchmark with a score of 63,000 Query-per-Hour (QphH) running on a 32-proc (64-core) HP Superdome Itanium server. The TPC-H benchmark is very demanding, and only six other TPC-H scores have been recorded in 10 years.

Other significant SQL Server 2008 benchmark scores include a world-record SAP Sales and Distribution (SD) Benchmark score for 4-Socket Industry Standard Blade servers in a three-tier test. In conjunction with Unisys, Microsoft and SQL Server 2008 also set a new standard for extraction, transformation, and loading (ETL) performance by loading 1TB of data in less than 30 minutes.

In addition, Microsoft posted a new set of TPC-E results for SQL Server 2008. SQL Server 2008’s TPC-E results include a score of 1126 tpsE on an Itanium 32-proc (64-core) server. This result is the first TPC-E result on a 64-way server and the first score of more than 1000 tpsE—beating the previous high score by 70 percent. SQL Server also set a new four-socket server high of 479 tpsE on a Xeon 4-proc (16-core) server, which is a 14 percent performance gain over SQL Server 2005 and Windows Server 2003.

However, as François shared SQL Server 2008’s TPC-E benchmark scores with me, I couldn’t help but notice that Microsoft is the only database vendor participating in the TPC-E benchmark. Seeing as the purpose behind the TPC benchmarks is to provide a tool that can be used to fairly compare competing database and hardware platforms, it’s a bit anticlimactic when the results include the scores of only a single database vendor. It’s true that the TPC-E benchmark is fairly new; it was first introduced in March of 2007. However, since that time, Microsoft has been the only database vendor to post TPC-E scores. If you take a look at the Transaction Processing Performance Council’s Web site at www.tpc.org, you’ll see that Oracle and IBM are continuing to submit TPC-C scores.

So what’s the purpose of the TPC-E benchmark? The idea behind the new database benchmark was to more accurately reflect the workloads of today’s applications. The TPC-C benchmark was designed more than 15 years ago and is based on an order-entry-shipment model. One clear indicator of the TPC-C benchmark’s age is its primary measurement: transactions per minute (tpm). Fifteen years ago, a tpm measurement was reasonable. However, today’s high-powered systems are capable of much more. Now it’s more valid to measure transactions per second (tps)— the way it’s done in the TPC-E benchmark. The TPC-E benchmark is designed to reflect the workload of a financial brokerage firm and defines a required mix of sample transactions such as trades, account inquiries, and market research. The benchmark itself is scalable based on the number of customers defined to represent small, medium, and large businesses.

Although the database world could use an updated benchmark, a benchmark without industry support is of limited value. More TPC-E benchmark scores won’t do Microsoft or its customers any good until one of the other major database vendors releases TPC-E benchmark scores. Until then, Microsoft, its partners, and its customers would be better served by continuing to post TPCC scores, which enable customers to compare SQL Server, Oracle, and DB2.