How Does TPC-C Score?

One dollar can't buy many things any more. A cup of coffee? Doubtful. Heck, most dollar stores don't even sell things for a dollar anymore. But 99 cents does cover the cost of a single transaction--as measured by Price/tpmC--a new world record for price/performance ratios for TPC-C scores. You can see the details of the current TPC-C scores at the Transaction Processing Performance Council site.

Over the years, I've seen the Price/tpmC number come down bit by bit from the hundreds to the teens--and now it's less than a buck. Heck, at this rate, what will happen if Microsoft drives the Price/tpmC to 0, or even into negative territory? Maybe Microsoft will have to start sending customers checks whenever the TPC announces a new score.

A few years ago, I wrote about TPC-C scores all the time. Lately, they don't catch my interest quite as much. A colleague at Microsoft Consulting Services recently sent me an email asking, "Do you think anyone cares anymore about TPC-C numbers?" My short answer was (and yes, this is my favorite all around answer) "It depends."

I'm amazed at the vast improvements in TPC-C performance scores and the price/performance ratios that go along with them. But the upper-end systems that Microsoft and other database vendors use in these tests go well beyond the kind of "big iron" that only a tiny handful of OLTP customers will ever need. Today, I truly believe that any of the major database vendors can meet the performance needs of most database customers. These upper-end TPC-C scores are helpful in justifying a "Yes, SQL Server can do it" decision if you're in an organization that still doubts the enterprise scalability of a Microsoft database. But honestly, I don't see that doubt as a problem these days. And although it's nice to see the Price/tpmC numbers, artificial benchmarks such as the TPC-C measure the workload characteristics of one narrow suite of tests. It's unlikely that your complex, real-world application will exhibit the same performance workload characteristics. So you can't really use TPC-C to predict how much your system will cost.

TPC-C has evolved into a game of leapfrog, with each vendor attempting to best the others by using systems that aren't relevant to the workloads of most database customers. But I suppose it's nice to know all that horsepower is there if you need it. Do customers care about TPC-C scores? As sort of a security blanket, sure. Beyond that, they're mostly useful for columnists looking for something interesting to pontificate about.