How I Tested

Each of the three vendors markets its system as a departmental-class server intended to satisfy the needs of several hundred users. I used Doculabs' @Bench test and Microsoft's BackOffice Server product suite to test each server's performance. Doculabs' @Bench test is an e-commerce application that simulates a Web bookstore with customers opening accounts, browsing, and buying. (For more information about Doculabs and the @Bench Web Application Server benchmark, see http://www.doculabs.com/.) The version of the test I ran works in conjunction with Quest Software's Benchmark Factory test suite. I ran the Web site under Microsoft IIS. Doculabs' @Bench Active Server Pages (ASP) application used Microsoft SQL Server 7.0 stored procedures to access data services.

Because this version of the Doculabs @Bench test is scalable, I could choose the size of the database to use in the test. The database I generated, including indexes and data, totaled about 2.6GB. The database had five tables: three preloaded and two that test transactions added records to during the test. The Products table, with 7.64 million records, occupied most of the space. The Customer table, with 50,000 records—each about three times the length of a Product record—took up most of the remaining space.

To provide additional load for each server, I installed Microsoft Exchange Server and ran a continuous mail workload (i.e., about eight mail transactions per second—tps) in the background while varying the workload that the Doculabs @Bench test generated against the server. To help ensure repeatability of results, I restored the SQL Server database and Exchange Server's private store and preloaded key data tables into SQL Server's server cache before each test iteration.

I tuned each server identically, setting specific options within the Registry, NIC adapter configuration, SQL Server, and Exchange Server. For a summary of these options, see Web Table 1. Because the performance characteristics of RAID array controllers vary and the performance and configuration of RAID arrays can affect application performance, I accepted each vendor's recommendations. Specifically, I let each vendor determine how to divide the system's drives between RAID arrays, how to configure each RAID array, and where on the RAID arrays to place the Exchange Server and SQL Server log and data files. Therefore, I accepted each vendor's suggestions for disk drive and RAID array configuration and the placement of data sets on logical drives.