Scalability Haze

NT scalability is a critical topic for Windows NT Magazine,and testing scalability is a regular focus for our Lab. Thus, you would thinkthe Lab Guys were thrilled by Microsoft's Scalability Day last May, right?Excitement was high when the Lab Guys flew into Manhattan, but expectations weremodest--after all, Scalability Day had all the earmarks of a well-oiledMicrosoft press event. As things turned out, that's exactly what Scalability Daywas--a series of slick, well-rehearsed speeches and demonstrations on NTscalability. To be honest, I left New York with more questions than answers.This month, I'd like to brush the dust off Scalability Day to reflect on whatattendees saw and, equally important, what attendees didn't see.

The Event
Let's start with the basics. Scalability Day was a two-part event. In partone, Microsoft herded an audience of about 500 press members and industryanalysts into an auditorium and exposed us to a series of speakers--Microsoft'sBill Gates (chairman and CEO), Paul Maritz (group vice president, platforms andapplications), Deborah Willingham (vice president, enterprise customer unit),and SAP's Gunter Tolkmit (vice president, corporate marketing). Each Microsoftspeaker called in additional Microsoft staff to conduct the live demonstrations.The demonstrations were glitzy, MTV-speed presentations designed for theattention-challenged (I'll get to the details of the presentations later).

The second part of Scalability Day was a Microsoft Partner Pavilion whereyou could press the flesh with some of the Solution Partners that Microsoftdeemed worthy. This group was eclectic--vendors ranged from high-end scalabilityhardware manufacturers to financial application software developers. I cannotbegin to guess how Microsoft came up with the invitation list, nor can I guesswhy some vendors weren't invited. Notably absent? IBM comes to mind. (For MarkSmith's view about noticeable absentees, see his July editorial, "ScalabilityPolitics.") The pavilion tour was a decidedly low-key event; much to mydisappointment, no scalability demonstrations took place on the show floor. Forexample, seeing Tandem's 16 * 4 cluster demonstration would have been prettycool, but instead, attendees were supposed to be content looking at a pictureof the 16 * 4 configuration. Sometimes a picture is not worth a thousand words.

Substance or Substance Abuse?
After sifting through the morning speeches and wandering through thepavilion in the afternoon, I realized that the meat of Scalability Day was theseries of live demonstrations that peppered the speeches. The six demonstrationswere

Were all these demonstrations impressive? You bet! They were flashy, withlots of moving images and buzzing counters. Were they substantive? Well, that'san entirely different matter. Let's look at these demonstrations a littlecloser.

TB or Not TB?
At the terabyte demonstration, a database on the Sphinx release of SQLServer contained slightly more than 1TB of data. The bulk of the data was imagesof the Earth taken by satellite cameras. The database contained locationinformation so that you could find an image from geographical input. The lookupapplication was Web-based. You could enter the necessary parameters and receivethe corresponding image. The amount of data was impressive, as was the fastlookup speed, but what did this test really prove?

If you think about the database this test used, the image files ate up mostof the terabyte of disk space and were not processed during any of the front-endlookups. Geographical information was associated with the image files, but theamount of geographical information wasn't close to a terabyte. In the finalanalysis, Microsoft had a fairly small set of indexes pointing to a series oflarge image files. Scanning those indexes quickly and retrieving a file based onthe results is hardly equivalent to reading or writing a terabyte ofinformation.

Was this test completely invalid? No, it wasn't. I was impressed that youcan use a SQL Server database to store a terabyte of data. But the demonstrationapplication--the online lookup of images--is not a hard-core, real-worlddatabase application. A more impressive test would be to fill the SQL Serverdatabase with 1TB of searchable rows and columns. Of course, that kind of testwould execute too slowly for the fast pace of the Scalability Day--a task thattook minutes to process would have seemed snail-like compared to thepace of the show.

Take It to the Bank
The second demonstration was a simulated banking environment capable ofhandling 1 billion ATM transactions per day. As a former application andnetworking consultant, I know that ATM transactions are short and sweet. In manyways, ATM transactions represent the best possible transaction load because notonly are they short, but they involve very few message exchanges between the ATMdevice and the server. If I were going to set up a simulated transactionenvironment to show a high transaction rate, I'd be inclined to use ATMs, too.

The configuration of the server side of the demonstration, however, wasvery interesting. Instead of setting up one database to handle all the ATMs,Microsoft grouped the ATMs into 20 divisions (bank branches) and assigned aseparate SQL Server to each division. Multiple SQL Servers were interlinked viaMicrosoft's Distributed Transaction Coordinator. All things considered, thisdesign is pretty good and would stand up in numerous real-world environments; Igive Microsoft high marks for the server side of this demonstration.

The demonstration showed that under ideal conditions, you can generate 1billion transactions today using NT technology. The problem is that nobody livesin these conditions. In reality, the SQL Servers must process standard bankingtransactions for name and address changes, new accounts, account closings, wiretransfers, and other full-size transactions that sap the potential to deliver 1billion ATM transactions per day. If all you want to do is create an ATMnetwork, you can achieve the advertised rate of 1 billion transactions per day;if you want to run a real banking environment, you will undoubtedly find yourperformance to be less than the advertised rate.

Real World or Ideal World?
Although the demonstrations were impressive, they were designed to run inthe ideal world, not the real world. This trend continued through three otherdemonstrations. The IIS demonstration of 100 million hits per day used staticWeb pages, not dynamic pages. The Exchange server mail demonstration used POP3,not the standard Microsoft mail connection. The Wolfpack failover demonstrationwas well orchestrated and hardly represents the dramatic and unpredictablefailures that can occur in our data centers. In short, the demonstrations madefor a good show, but not for good research data.

Don't get me wrong--I'm not saying that the demonstrations weremeaningless. Taken in the proper context, they show how NT can perform in thebest of times. The problem is, most of us don't live in an ideal world. We alllive in the world where networks clog and break, applications are resource hogs,and our users continue to confound and confuse our best business practices.Trying to map Microsoft's glitzy, MTV-style tests into the dirty, gritty realityof our day-to-day data processing lives is an impossible feat.

However, I will admit that the demonstration of the NT 64-bit Very LargeMemory (VLM) implementation impressed me. This demonstration showed two NT Alphamachines running the same application (a database lookup); only one of themachines was running with VLM enabled. The VLM-enabled system ran theapplication unbelievably faster--I was ready to take out my wallet and invest inan Alpha machine with lots of memory right then and there.

Lasting Memories
Scalability Day reinforced the Lab's plan to test NT scalability usingreal-world tools and analysis techniques. Our tests are not glitzy and don't runat MTV speed. A single test takes us hours or days to conduct. You won't see usperforming these tests on stage or on video because they are, frankly, boring towatch. But they provide us with interesting insights about NT and BackOfficescalability. You get to skip watching the tests and instead can read meaningfulresults in the magazine.