Editorial: Benchmarking Redux

Just when I thought I'd covered the last of the exciting benchmarking news, Microsoft turned in several new Transaction Processing Council (TPC) benchmark scores on a beta version of SQL Server 7.5 (that's the name Microsoft submitted, anyway). First, in traditional TPC-C online transaction processing (OLTP) testing, SQL Server posted a new high-end TPC-C score of 40,670, thanks mainly to the use of the new 8-way Intel Profusion chipset. This new TPC-C number shows that the 8-way Profusion chipset delivers the goods in terms of scalability, netting a 52 percent increase over SQL Server's older 4-way TPC-C score of 26,680. The price of the system that produced this score was $749,208. Although that seems like a lot of money to me, systems from IBM and Sun/Oracle that produced comparable TPC-C scores cost $5,564,844 and $4,723,164, respectively. To no one's surprise, SQL Server holds all 10 of the top 10 price and performance scores.

The other set of interesting SQL Server benchmarking scores came from the new TPC-H benchmark, which replaces the recently retired TPC-D (decision support) benchmark. As the Oracle challenge aptly revealed, the TPC-D benchmark didn't adequately capture the effects of a feature known as materialized views, which provided terrific performance on a TPC-D test but also subverted the ad hoc query performance that the TPC designed the TPC-D test to measure. To handle this type of feature, the TPC retired the TPC-D test in favor of two new benchmarks: the TPC-R for reporting and the TPC-H for ad hoc queries. The TPC-R benchmark allows the use of materialized views whereas TPC-H does not. Like the TPC-D, the new TPC-H benchmark runs a set of queries against a 100GB database. As I write this, TPC has released seven TPC-H results. SQL Server holds positions three through seven, and HP/Informix holds the number one spot with a query-per-hour (QphH) score of 1481 at a cost of $872 per QphH. Sun/Informix holds the number two spot with a QphH mark of 1280 at $816 per QphH. SQL Server was close behind with a score of 1233 QphH at $285 per QphH. In other words, SQL Server delivered 84 percent of the performance of the first-place holder at less than one-third of the price.

TPC scores can be useful because they can give you an idea of the total high-end scalability that you can achieve with a hardware and software platform. But take all benchmark scores—even the ones you like—with a grain of salt because they probably won't reflect your environment. You can find out more about the recent TPC-C benchmarking scores and the new TPC-R and TPC-H benchmarks at http://www .tpc.org.