Microsoft Still the Only Database Vendor Posting TPC-E Scores

Would Michael Phelps’ recent Olympic gold medal extravaganza have been as impressive if no one else was swimming in the pool with him? Probably not. That’s sort of how I feel about OLTP database benchmarks these days. Here’s some seemingly impressive news about SQL Server performance. The TPC-E benchmark from the Transaction Processing Council was released well over a year ago and is the most modern, and realistic, benchmark for OLTP applications ever published by the group. Amazingly enough, SQL Server 2008 holds the world record for the top 10 performance scores for both clustered and non-clustered systems. It also holds the top 10 world records for price/performance. In fact, just this week, Microsoft bested its previous world record. Alas, those data points would be a bit more impressive if anyone else was swimming in the pool.

My last article on TPC-E was more than a year ago. Last August, I talked about Microsoft’s first TPC-E score that was published in July 2007 in my commentary “SQL Server Holds Record for TPC-E Database Benchmark” (http://www.sqlmag.com/Article/ArticleID/96943/sql_server_96943.html). I said “Sane vendors don't post TPC-E scores that make them look bad, but I suspect it's only a matter of time before IBM and Oracle post TPC-E scores for their database products that leapfrog the latest SQL Server scores, which will in turn be bested by Microsoft in the never-ending game of benchmark leapfrog.” Well, it’s been more than a year and no other vendor has come to the party. So what does that mean? Well, one rational answer is that Oracle and IBM have done internal tests, found they don’t beat SQL Server, and therefore declined to post TPC-E scores. It’s also possible that Oracle and IBM simply don’t think the TPC-E is a good, balanced test, and have therefore declined to post TPC-E scores for philosophical reasons.

I haven’t been able to get a formal response from IBM or Oracle about why they haven’t posted any TPC-E scores to date. I’ve sent email messages to their public relations departments and will let you know if I can track down an official answer. However, I’d be grateful to any readers who can pass along a link to Oracle or IBM’s official position on TPC-E. Although I’m lacking a formal answer, the most rational answer is that Oracle and IBM have tried to top Microsoft’s numbers and simply can’t. The big database companies traded benchmark highs for years under TPC-C and the fact that the game of leapfrog stopped cold with TPC-E is telling. To be fair, Oracle and IBM have continued to post new TPC-C scores while Microsoft has eschewed any new TPC-C on SQL Server 2008 in favor of TPC-E. Knowing for sure that Microsoft is soundly beating Oracle and IBM on raw performance and price/performance would be compelling news indeed. Alas, lowly consumers of database technology, such as you and I, are left to guess. The vendors aren’t compelled to release numbers, and in fact, the license agreement you sign with every major database vendor includes language that prevents you, or anyone, from publishing performance numbers without the vendor’s permission.

If you’re wondering whether TPC-E is a better mousetrap for testing database performance, I encourage you to read the article “Transaction Processing Council Launches TPC-E Benchmark” (http://www.itjungle.com/tlb/tlb032007-story03.html). It points out many of the TPC-E improvements that I, and others from the Microsoft world, have talked about before. But the article is commentary published in a newsletter called “The Linux Beacon,” which presumably doesn’t have a pro-Microsoft slant that might be skewing its opinion. A quick Internet search will show you that it’s not just Microsoft-oriented folks who are wondering why Oracle, IBM, and others haven’t jumped into the TPC-E pool. Could it simply be because they’re scared to have the results published?

By the way, I heard this week that Microsoft is launching the SQL Server 2008 Experience next Monday, September 29. I haven’t been able to track down just what the SQL Server 2008 Experience is, but it might be fun to check out the SQL Server home page next Monday or Tuesday