Understanding and Using LoadSim 5.0: Part 1

What is this tool and how does it simulate loads on Exchange Server?

Have you ever wanted to try out some new hardware to see how well it holds up under an Exchange Server load? Or maybe you want to know how many users your current configuration can handle. Microsoft has a tool that can help. This series of four articles will help you understand the Microsoft Exchange Server Load Simulator program, commonly known as LoadSim. I'll explain the major aspects of this powerful and complex tool so that you can start using it quickly.

The first article in the series introduces the LoadSim program and explains what you can use the program for. This article also introduces some key concepts, such as user profiles and the suggested number of users you can simulate with this tool.

The second and third articles will walk you through a sample LoadSim scenario and give an example of how to install, configure, and customize LoadSim. You'll learn how to use the program, and from there you can adapt it to your needs.

The fourth article will tell you how to run LoadSim and present aspects of the data collection and analysis. This article will describe techniques for gathering accurate data and analyzing it. When you finish this series, you will be able to set up LoadSim on your machines and start simulating. See the glossary, "LoadSim Terminology," page 164, for definitions of common terms I'll refer to throughout the series.

Microsoft does not officially support LoadSim. Although the company designed and wrote LoadSim, Microsoft makes the tool available as is. Also, LoadSim 5.0 is not on the Exchange Server 5.0 CD-ROMs, unlike the LoadSim that shipped with Exchange Server 4.0. Microsoft makes LoadSim available, and by the time you read this article, you should be able to access it on Microsoft's Web site at www.microsoft.com/exchange. You'll get the LoadSim program with Messaging API (MAPI) protocol support only—the same as with LoadSim 4.0. LoadSim does not support Internet protocols. Instead, Microsoft recommends that you use the InetLoad utility to create user loads with Internet protocols such as HTTP, Lightweight Directory Access Protocol (LDAP), Network News Transfer Protocol (NNTP), and Post Office Protocol (POP) 3. InetLoad is easy to configure, and it's script based. In the next revision, InetLoad will support IMAP. By the time you read this article, InetLoad should be shipping in Microsoft BackOffice Resource Kit.

Introduction to LoadSim
LoadSim simulates a user load on an Exchange Server computer. If you want to test performance on a system, one of your main concerns is to disturb the system under test as little as possible. In other words, you want to measure the performance of the system under test without affecting its performance. This task sounds easy, but it's challenging in practice. LoadSim makes the task possible.

Suppose you want to simulate 100 simultaneous users on a server. Without LoadSim, you need 100 identically configured Exchange client machines, each running some kind of script that regulates the activity. Then you must have a way to measure all the response times and track each user's activities.

With LoadSim, you can accomplish this 100-user simulation with one relatively powerful Windows NT machine. A 100MHz Pentium with 48MB of RAM, a suitable hard disk, and a good network card will do the job. This requirement hasn't changed much since Microsoft first introduced LoadSim. Fortunately, hardware prices have improved, making good simulations attainable for people who don't have huge hardware labs.

Don't run LoadSim on the Exchange Server machine. Technically, you can do so if you also install the Exchange client on the system running Exchange Server, but running LoadSim on the server puts extra stress on the server and will severely skew your test results. Instead, run LoadSim on a separate machine.

LoadSim Theory
The engineers at Microsoft and those of us who tested LoadSim during its development have invested much time and effort in helping LoadSim address the issues associated with simulating a user load. LoadSim takes full advantage of NT's capabilities to reliably simulate multiple Exchange users. It is a multithreaded, multiprocess program that lets one machine running NT simulate one user or many users. Each LoadSim process has one or more threads, and each thread performs the task of one LoadSim user. Because LoadSim relies on the inner workings of NT to handle running each thread, each thread, or user, gets equal attention from NT. Each simulated user produces the same load on the targeted Exchange server.

LoadSim requires NT Workstation (or Server) 3.51 or later. It uses the Microsoft Exchange Client software for NT to talk to Exchange Server, just like a real Exchange user does. Thus, you must install the Microsoft Exchange Client for NT on the LoadSim client before you get underway. Do not use LoadSim under Windows 95.

By design, LoadSim does not disturb the Microsoft Exchange Server that you're testing. Exchange Server acts as though it is seeing a real user load, and you can measure how it performs. Because the LoadSim client is a separate machine from the Exchange Server computer, and because LoadSim uses the Exchange client software to talk to Exchange Server, the Exchange server treats the LoadSim users as real users.

Other features make LoadSim act like a real Exchange user and minimize the number of performance artifacts that LoadSim causes. You want the load imposed to be as realistic as possible. For example, Microsoft built realistic think times into the LoadSim users. These delays simulate the time users spend typing or reading before they perform a task, such as sending mail or opening a folder. That approach keeps the load from being too predictable or patterned, which would skew the results.

The artificial user names and user loads that LoadSim produces are not in any particular pattern. This approach ensures that the Exchange Server gains no unfair efficiencies by processing names or loads that are similar or follow a pattern. LoadSim doesn't cause predictable patterns of performance, but it does let you generate reproducible results.

What's New in LoadSim 5.0?
If you have used LoadSim with Exchange Server 4.0, you'll notice several enhancements in LoadSim for Exchange Server 5.0. You can still use LoadSim 4.0 against Exchange 5.0 servers with no problem. However, the latest version has an added feature set, new capabilities, and increased load accuracy.

An obvious enhancement is the user interface. Microsoft has done a lot of work in this area. If you are familiar with LoadSim 4.0, you'll find that Microsoft rearranged the menus, dialog boxes, and property sheets. The interface is better organized and more efficient.

To run tests against Exchange Server, LoadSim 5.0 lets you use MAPI as your protocol. The MAPI protocol support for LoadSim mirrors what you get with Exchange Server. Microsoft considered incorporating Internet protocols such as LDAP, NNTP, and POP3 into LoadSim 5.0, and the company planned to add IMAP4. However, an existing tool called InetLoad has support for all the Internet protocols. Plus InetLoad proved to be more flexible than LoadSim in measuring performance with Internet protocols.

Unlike LoadSim 4.0, LoadSim 5.0 no longer requires you to specify both senders and recipients. Now the LoadSim users you specify in the test are both the senders and the recipients, leaving less room for confusion. I'll talk more about this issue in part two of this series.

LoadSim 4.0 had the capability to simultaneously support multiple LoadSim clients using a single LoadSim test setup (.sim file). You just specified the LoadSim clients in the test and the program automatically divided the clients evenly among the LoadSim clients. With LoadSim 5.0, you must configure a separate .sim file for each LoadSim client. This method is less convenient, but the data you produce is no different.

The LoadSim Screen
Screen 1 shows a LoadSim 5.0 display during execution. Two categories appear at the top of the screen: Time Elapsed and Users. Time Elapsed shows how long the test has been running. Users shows how many LoadSim users are logged on from this LoadSim client.

The body of the screen also contains useful information. The first line shows that the Exchange—or MAPI—protocol is loaded. The next two lines represent initialization information. The second line shows that LoadSim is configured to run with a maximum of 25 processes, which means that no matter how many LoadSim users you simulate with this client, only 25 processes will be spawned. Although I don't recommend changing this setting, you can configure it from the Tools, Options menu. On the third line, the SIM file lstest.sim is loaded. This file contains all the simulation settings for the test.

For simplicity, the run shown in Screen 1 simulates only five LoadSim users. The fourth through eighth lines show LoadSim creating messaging profiles for those five users because they weren't already in the Registry. The next time you run LoadSim, this step won't be necessary.

The next five lines show the five LoadSim users successfully logging on to the Exchange server. This display shows up in every run so that you can see the users are logged on. As the users log on, LoadSim increments the number beside the Users field at the top of the screen.

In the final five lines, you see LoadSim user activity. Each line shows the activity detail for a LoadSim user. For example, in the last entry, you see the date and time of the activity, followed by the LoadSim user name and the activity detail. In this entry, at 23:28:41, user arrakis-exch2 browsed the Deleted Items folder for existing messages and found none.

This information tells me I need to run Initialize Database, which I'll cover in part three of the series. If the database is initialized properly, LoadSim won't find empty folders.

By default, LoadSim writes all this data to the loadsim.sum, loadsim.out, and loadsim.log log files. A utility that comes with LoadSim, LSLOG, lets you analyze the scores from a LoadSim log. LSLOG parses all the data that LoadSim collects and spits out the summarized results, which you can use for creating a users-per-server graph. I will cover details about LSLOG in part four of this series.

LoadSim User Profiles
LoadSim works with user profiles. User profiles determine how much load a simulated user places on Exchange Server. LoadSim comes with three default user profiles—Light, Medium, and Heavy—that represent three classes of users in LoadSim.

Table 1, page 166, lists the default LoadSim user profiles and their attributes. Don't let all the attributes and their associated values confuse you. If you don't know what an entry means, don't worry; the meaning will become evident later. You need each attribute to properly simulate an Exchange user. The Approximate Message Traffic values at the bottom of the table are based on the other values in the table. For example, if you increase the amount of new mail sent per day, the calculated numbers go up accordingly.

The three user profiles give you an idea of the different loads you can place on Exchange Server with LoadSim. More important, you can now analyze these user loads and determine which one best fits your organization.

The Heavy profile is most useful as a stress test rather than as a user test. It does not represent a realistic user profile, because the load is so intense. However, if you need to pound on your server to test its limits or to see whether software breaks, the Heavy profile is useful.

The Medium profile most closely reflects users who rely on email, such as a Fortune 500 corporate email user who depends on email as a regular, integral part of corporate communications. The Medium profile is a practical upper limit or worst-case scenario for estimating the number of users Exchange Server can support. In other words, if the Exchange server being tested can support 300 Medium users with acceptable client response time and server performance, you can plan to support approximately 300 real users on that server.

The Light profile most closely reflects users who employ email infrequently or irregularly. A good example is a user at a small company or at a company that has just started using email. The users do not rely heavily on email yet, so their usage pattern is light. Light represents a practical best-case scenario for many circumstances.

Of course, you aren't stuck with the three profiles LoadSim provides. The three profiles exist to get you started and to provide a common frame of reference for comparing sets of LoadSim data. You can configure LoadSim user profiles to mirror your particular users.

How Many Users Can I Simulate?
Your next question might be, "How many users can I simulate?" This question has two answers.

First, the limit to the total number of users you can simulate depends on the number of LoadSim clients you have and the number of users Exchange Server supports. You can, for example, simulate 20,000 users if you have enough LoadSim machines and a powerful enough Exchange Server.

Second, the number of users one LoadSim client can simulate depends on the class of hardware the client is running on and the profile you choose. Because you have to be running NT, you need to factor in that requirement. Table 2 shows some typical hardware recommendations for a LoadSim client. The estimates are conservative, but I don't recommend exceeding these numbers of simulated users per LoadSim client if you want to produce a valid user load on the Exchange Server computer.

Furthermore, when using MAPI, if you plan to rely on the score that a particular LoadSim client produces, do not simulate more than 100 users of any profile on that client. If you stay within the guidelines listed in Table 2, the user load you place on the server will be accurate.

If you attempt more users, however, your score will likely be skewed higher than it should be. The higher score will make Exchange Server appear as though it is performing worse than it really is. The result? Although the score on the LoadSim client might be inaccurate, the Exchange Server performance data remains valid. In fact, this approach is one way to squeeze more simulated users out of fewer machines. I'll cover this concept in detail in part four.

When you simulate a significantly higher number of LoadSim users—for example, 300 or more—on one client, the LoadSim client will eventually bottleneck. In this case, LoadSim will not produce the intended load on the Exchange Server computer. Both the LoadSim score and the Exchange Server performance data will be completely unusable.

What to Look for in a LoadSim Run?
Look for two items from a LoadSim run against Exchange Server: the load placed on Exchange Server and the response time (score) of the LoadSim client. The more complex item is observing the load on Exchange Server.

NT Performance Monitor (PerfMon) is the best tool to accomplish this task. Some PerfMon objects and counters are generic to NT and others are specific to Exchange Server. But all the counters together tell a large part of the story about how your server is faring under the load.

Of course, you don't have to run LoadSim to monitor PerfMon counters, which are also useful for observing a server under a real user load. Some preconfigured PerfMon workspaces are installed with Exchange Server. If you need to keep an eye on things, become familiar with the counters in these workspaces because they might be helpful.

The second task is to analyze the score on the LoadSim client. The score represents LoadSim's main performance metric. The score simply represents the average LoadSim client response time in milliseconds. I'll cover this issue in part four of the series.

I've introduced LoadSim and how you can use it to simulate the load on an Exchange Server. In part two, I'll review the basic steps to configure LoadSim. In part three, I'll describe customizing the LoadSim parameters. In the fourth installment, I'll explain initializing the Exchange database, running the test, and collecting and analyzing the data.