Testing Your Web Environment

Microsoft's Web Capacity Analysis Tool makesit easy and efficient

Last month, I explained that bigger is not always better, and that you can buy too much hardware for some Web site environments. I've seen several Web performance tests done on dual Pentium Pro 200 machines, and I haven't been impressed with the performance of my quad Pentium Pro 200 machine. So I decided to run some performance tests of my own. This month, I'll tell you how I tested the performance of my Web environment and show you what the results mean.

I wanted to make the tests as easy as possible, yet informative enough forme to draw some conclusions. So I grabbed the Microsoft Windows NT ServerResource Kit Version 4.0, Supplement 1 and loaded the Microsoft Web CapacityAnalysis Tool (WCAT). WCAT lets you run a simulated workload on a Web server andcomes with several prepared scripts that let you simulate different types ofenvironments.

You can run any of four types of WCAT tests:

You can customize the WCAT test environment to mimic your Web environment.For example, I could have created the scripts required to simulate the WindowsNT Magazine Web environment (i.e., most of the pages dynamically createdwith Cold Fusion). But I wanted to try WCAT right out of the box to check itsease-of-use before I started tweaking it. Besides, because I was just glad tohave access to the 22 machines I needed to run the tests when I needed them, Iwasn't about to mess around with the configuration.

To run WCAT, you need a Web server, a controller server, and some clients.The Web server contains all the files that the WCAT tests request. Thecontroller server is the test administrator. It tells the clients what to do andfor how long. It then monitors the Web server and collects data from the clientsat the end of the test. The client machines produce a load on the Web server.Each client machine can support up to 200 virtual clients.

Using WCAT, I wanted to evaluate the performance differences between a dualPentium Pro 200 Web server and a quad Pentium Pro 200 Web server. To minimizethe number of variables involved, I used one machine to serve as both Webservers and set it up to be two different machines during different stages ofthe testing. The Windows NT Magazine Lab lent me a quad Pentium Pro 200with 512MB of RAM. To level the playing field for all the tests and to simulatepractical configurations, I configured the machine to use just 256MB of RAMduring the tests. Figure 1, page 211, shows my test configuration.

To set the number of processors and the amount of memory without crackingthe case on the machine and yanking out RAM and processors, I used a setting inthe Boot.ini file to specify the configuration I wanted to use. In this file,you can specify the number of processors and the maximum amount of memoryavailable to NT. To do so, you need to add the line

This approach is not suitable for true benchmark testing, but for mypurposes, this configuration worked perfectly.

For the controller server, I used a second quad Pentium Pro 200. You don'tneed to use a machine this big for this task, but the Windows NT MagazineLab already had this system set up as a controller. For the clients, I used 20Pentium 133 machines, each with 32MB of RAM, and added virtual clients to eachof these machines. Finally, I performed the tests on a network made up of 100Mbconnections, because I knew that this many clients would easily saturate a 10Mbnetwork.

Following the WCAT installation directions to the letter, I first installedWCAT on the Web server. Then I installed and configured WCAT on the controllerserver and the 20 client machines. Setting up the tests was easy. However,installing and configuring 20 client machines can get tedious. Fortunately, youhave to set them up only once.

At this point, I tested the WCAT client's ability. I didn't want tooverload the clients and have them become a potential problem or bottleneck, soI performed a few tests to determine how many virtual clients each physicalclient could sustain. For this part of the exercise, I used the ISAPI WCAT testthat I planned to use for the primary testing and kept adding virtual clients toeach physical client until I started to see errors. I discovered that eachclient could sustain 60 virtual clients. So with 20 client machines, I was ableto simulate a maximum of 1200 total clients.

The Test
After you finalize your test configuration, all you must do is decide whichtest to run. I wanted to make the testing realistic (i.e., I wanted a test thatcombined static and dynamic pages), so I ran one of the ISAPI tests. The test Iselected consisted of half dynamic pages and half static pages. I ran the testusing 5 client machines (300 virtual clients), 10 client machines (600 virtualclients), 15 client machines (900 virtual clients), and then 20 client machines(1200 virtual clients). I tested these settings for both the dual Pentium Pro200 and the quad Pentium Pro 200. I considered conducting this test with onlyone Pentium Pro 200 running, but the test choked after I started the 10 clientmachines. After I discovered that this configuration couldn't make the grade, Idecided to stay with the original test configuration--dual versus quad.

With all the clients running and waiting, you only need to edit the testconfiguration file to tell it how many client machines to use, how many virtualclients on each, and how long to run the test (for my tests, the duration was 10minutes). Then, from the command prompt, you simply type

where TESTNAME is the name of the WCAT test you want to use (in my case, thetest name was isapi50). The controller server takes care of the rest and tellsthe clients everything they need to know.

Although WCAT is handy for this type of testing, Microsoft never intendedWCAT to be an official benchmark or an extremely scientific test. Thislimitation doesn't mean you can't get down to some specific numbers about theperformance of your Web site with WCAT; I think you can. However, for mypurposes, I just wanted to run a test with everything equal except the number ofprocessors until the machines choked. That way, I could see what the resultstold me about the benefits of beefing up my Web server hardware.

The Results
So what did the test tell me? I learned that in this environment, a quadPentium Pro 200 does not have a significant advantage over a dual Pentium Pro200. Throughout the tests, the results were almost identical for bothconfigurations. As you see in Graphs 1 and 2, the dual and quad Pentium Pro 200machines have almost identical performance numbers for 300, 600, and 900clients. Both configurations have unacceptable numbers for 1200 clients becauseof several connection errors. This falloff is probably the result of saturatingthe network somewhere between 900 and 1200 clients. But as you can see, bothconfigurations handled the saturation in the same way. The test shows that thedual Pentium Pro 200 machine can do everything the quad Pentium Pro 200 machinecan do up to the point of saturation of the 100Mb network (translation--saveyour money or spend it on several Web servers rather than one big Web server).

WCAT is an excellent tool that makes performing consistent tests a breeze.With the proper environment and this type of tool, you can really fine-tune yourWeb server. I plan to use WCAT a lot in the near future. I've found severalinteresting Registry hacks I want to try out and some tuning suggestions forCold Fusion. With WCAT, you run the test against your Web server, make whatevertweaks you want, and simply run the test again. The results can easily tell youif you're headed in the right direction.