SQL Server 2008 & the TPC-E

While I usually prefer to focus on technical solutions that can help readers solve specific business problems, after reading Microsoft’s latest press announcement about SQL Server 2008 #1 ranking in the TPC-E benchmark I felt compelled to discuss this somewhat misleading announcement.  While SQL Server 2008 certainly did top the TPC-E benchmark, the fact is that no other vendors participate in the TPC-E benchmark. As one of my colleagues put it, they are number one in a field of one.  Woo Hoo! Seriously, this is no accomplishment to be trumpeting to the world.

Don’t get me wrong. I’m one of SQL Server 2008’s best fans.  And no one was more excited than me when SQL Server used to top the TPC-C benchmarks. However, that’s not what this is about. The TPC-C is where Microsoft and SQL Server used to compete. IBM and Oracle are still there. The TPC-E is a totally different benchmark. A couple of years back Microsoft quit issuing TPC-C benchmarks in favor of the new TPC-E benchmark.  However, IBM and Oracle did not follow suit. Nor did anyone else.

Without multiple vendor participation TPC-E numbers are meaningless. They can’t be used to compare SQL Server 2008 against other database platforms (the true purpose behind the TPC in the first place) nor can they even be used to compare SQL Server 2008 against previous version of SQL Server.  If no one besides Microsoft uses TPC-E then there can be no cross database vendor comparisons. It can’t even be used for comparing SQL Server 2008 and the earlier SQL Server 2005 release because they are run on different hardware platforms.

Industry standard benchmarks are about comparing the performance of multiple vendor solutions.  At the best, announcing #1 TPC-E is silly and meaningless – unless Oracle and IBM begin producing TPC-E benchmarks as well. However, all of their latest OLTP tests have been in the TPC-C. At the worst, Microsoft TPC-E announcements are misleading as they imply that SQL Server is better than the other competing enterprise database platforms (isn’t that what it is to be #1?).  But, in fact, that’s not demonstrated by the TPC-E. It makes you wonder if the problem is that SQL Server 2008 can’t post competitive scores in the TPC-C