Getting Started with ExMon

The Exchange Server User Monitor (ExMon), announced by the Exchange Server engineering group last April, lets Exchange administrators investigate and respond quickly to user complaints that "the mail system is running slowly." You can use ExMon to monitor the performance of individual Outlook client connections with the Exchange server in near real time and possibly identify heavy resource usage (e.g., excessive CPU consumption). You can also use ExMon to determine the impact of user activity on the Exchange system. ExMon is a great tool for a quick look at overall system performance (as opposed to detailed performance analysis). It relies on the Event Tracing for Windows (ETW) subsystem that Microsoft created to allow low-overhead performance analysis of Windows applications. (You can gain a basic understanding of ETW at http://blogs.msdn.com/matt_pietrek/archive/2004/09/16/230700.aspx.) To get you started with ExMon, I outline the tool's requirements and provide some guidelines about how to install and use the tool.

Installing ExMon
ExMon operates in two modes: data collection and data viewing. To collect data from an Exchange server, you must install ExMon on that server. You can't use ExMon on one system to collect data from a remote system that doesn't have ExMon installed. ExMon works with both Exchange 2000 Server and Exchange Server 2003. However, when installing ExMon on an Exchange 2000 server, you must run Exchange 2000 Service Pack 2 (SP2) or later. On Exchange 2003 systems, ExMon is supported only with Exchange 2003 SP1 or later. If you install ExMon on an Exchange server to capture performance data, you can use it to view the captured data later.

You don't need to install ExMon on an Exchange server if you only want to view ExMon data files. You can install the tool on a Windows server (that isn't running Exchange) and view data files that were captured by ExMon on an Exchange server. There are some requirements and limitations, though. If the ExMon data files were created on a Windows Server 2003 server, you can view the ExMon data only on a Windows 2003 server running ExMon. However, if the ExMon data files were created on a Windows 2000 Server system, you're free to use ExMon running on a Windows 2003 server, a Win2K server, or even a Windows XP workstation to view the data files. ExMon has no specific service pack prerequisites for these OSs, although keeping systems updated with the latest service pack is always a good idea. The version of Exchange on which a data file was created has no bearing on the version of the OS you must use to view the data file.

ExMon is available from the Microsoft Web site at http://www.microsoft.com/exchange/tools/2003.asp. After you download the tool, you will have one Windows Installer (.msi) file named exmon.msi. Double-click the MSI file, and the tool will, by default, be installed to the Program Files Exmon folder on your system drive, although you can change the folder location. ExMon isn't an I/O-intensive application; Microsoft says that CPU consumption on the system under scrutiny should be minimal (allegedly less than 2 percent). However, locating applications on the system volume is never a good idea, so you should consider some other location for the ExMon installation directory. Also, when you run the tool, you can specify a separate location for its data-collection log files. Avoid using any volumes used by Exchange (e.g., volumes that host Exchange database files, transaction log files, or queue directories) to collect data for ExMon because ExMon's I/O activity can affect normal system performance for Exchange files on these volumes.

The installation isn't really an installation as such. Double-clicking the .msi file just unpacks the ExMon utility and three ancillary files. After unpacking the files, you should open a command prompt window and navigate to the directory that holds them. Figure 1 shows the installation directory created by running exmon.msi. The Using_Exmon.doc file provides useful information about how to use the tool.

Before you can actually run the ExMon tool to start gathering and viewing data, you must create two ExMon registry entries, which Table 1 shows. In the command prompt window shown in Figure 1, type the command

exmon.reg 

to create the entries. The Exchange System Attendant service will detect the registry subkey changes within a few minutes (typically within 15 minutes); there's no need to restart any Exchange services or reboot the server.

The RpcEtwTracing registry entry, when enabled, allows Windows to capture performance information using the ETW technology. Although merely enabling the registry entry doesn't affect Exchange system performance, if you aren't currently using ExMon to monitor the system (and don't intend to for some time), Microsoft recommends that you either set the value to 0 or delete the registry subkey. In fact, if you run the Exchange Server Best Practices Analyzer (ExBPA) tool, it will highlight an enabled RpcEtwTracing registry entry as a potential problem because you shouldn't have the system configured to capture tracing information unless you're troubleshooting a performance problem or determining a system performance baseline (tracing puts unnecessary load on the system).

Running ExMon Directly
You can collect data for viewing in ExMon by using ExMon directly, by using System Monitor, or by using command-line tools. Running ExMon directly is straightforward. Double-clicking exmon.exe in Windows Explorer launches the application, and data capture begins immediately with a default reporting interval of 1 minute. Initially, the ExMon window will be blank, but as soon as the first minute has elapsed, the averaged data values from that reporting interval will be displayed, as shown in Figure 2. You can stop data collection at any time by clicking the Stop icon (a black square) on the toolbar (or selecting Stop from the File menu); similarly, you can restart data collection by clicking the Start icon (a right-pointing arrow) on the toolbar or via the File menu.

As performance information is collected, it's logged to a temporary file that's placed in the directory you selected during installation or a directory of your choosing. ExMon generates the filename from the server name and a pseudo-random identifier and gives it an .etl extension (e.g., MARKOV-428b6c81.etl). The .etl extension is used for event trace log files.

You can set the sampling interval to any value between and including 1 and 30 (minutes). ExMon retains the last log file written to the installation directory in addition to the log file to which data is currently being written. At the end of a sampling interval, ExMon opens a new log file for the new data to be collected and processes the collected data from the previous sampling interval. If you have a long (30 minute) sampling interval and your Exchange server is busy, processing the data at the end of the sampling interval might use significant CPU and memory resources on the system. Therefore, you should exercise caution when using ExMon on busy production servers, especially with large sampling intervals. When you exit ExMon, the log files are removed.

When you examine the information displayed in ExMon, don't be alarmed if one user appears to account for a large CPU percentage. The CPU percentage shown is the user's portion of the CPU utilized by the Exchange Store process rather than the user's portion of the total system CPU utilization, so even on a multiprocessor system, the sum of the users' CPU percentage values won't exceed 100. Remember that ExMon provides only a snapshot of resource utilization; for short sampling intervals, it's likely that some users will stand out, especially if they've performed CPU-intensive operations at the time of the sampling interval. For example, user CA M.Loughran accounted for 28 percent of the CPU utilization during the 1-minute sampling period shown in Figure 2. I happen to know that this user was attaching a rather large file to an email message during this sampling period, explaining the high CPU utilization. Over a longer sampling interval, the resource utilization would be much more evenly distributed across users. Over a longer sampling period, very heavy consumers of CPU resources might well warrant some closer scrutiny to determine exactly what operations they were performing.

You might occasionally notice an ExMon results entry that has a blank username and low system resource use; Figure 2 shows such an entry. These entries indicate that ExMon has detected a client logon and has begun reporting on the client's usage but hasn't yet been able to detect a username for the process.

The status line at the bottom of the ExMon window provides additional information, including a Captured field. This value represents the percentage of the Information Store CPU usage captured by ExMon relative to the total amount of Information Store CPU usage. ExMon captures data only for Messaging API (MAPI) clients; other client types such as Outlook Web Access (OWA), POP3, and IMAP4 consume Information Store cycles but aren't processed by ExMon.

Using System Monitor
If you want to record performance information for a period longer than 30 minutes, you can use the standard System Monitor. You could, for example, use System Monitor to record system information from 9:00 A.M. to 5:00 P.M. and schedule this operation to be performed daily. If you take this approach, you can review the performance data log files at your convenience and not necessarily on the production Exchange servers.

To enable System Monitor–based ExMon data collection, start the Performance Monitor console (select Start, Run and type perfmon), expand the Performance Logs and Alerts node, right-click Trace Logs, and select New Log Settings. Enter a name for the log, and click Add to add the Exchange Information Store nonsystem provider. You must then add in the Run As field an account under which the ExMon log can be created (this account must be at least a member of the Administrators built-in group on the Exchange server you want to monitor). You must also click Set Password to set a password. You can set other options such as the start time and end time for the logging operation. Figure 3 shows the settings for my System Monitor–based ExMon log.

Using System Monitor for ExMon operations creates .etl log files in the location you specified in the log properties. Again, you should make sure that this location isn't on the system volume or any other volume that will host Exchange files such as databases and transaction logs.

Determining the extent to which log files will grow is difficult because the size depends so much on user activity, but as an estimate, you could plan for approximately 50KB of log file growth per active user per hour. For example, on my system hosting 283 active Outlook users, the .etl log file reached 5632KB after 30 minutes, growing at a rate of about 40KB per active user per hour. The log file growth rate would be roughly twice as great when running ExMon in direct collection mode because there's a duplication of files in this mode, but when running in System Monitor collection mode, the .etl file grows at roughly the 50KB per user per hour rate.

When a System Monitor collection period has ended, the utility leaves an .etl file that contains the performance data for that period with a name in the format shown in the Current log file name field in Figure 3. To view the information, move the .etl file to the directory that contains exmon.exe and at a command prompt, type

exmon.exe   "" 

where ETL filename represents the path and filename of the .etl log file. The double quotes are optional.

ExMon from the Command Line
The third method of capturing ExMon data is to use the command-line interface. This approach is useful if you want to develop scripts for collecting system performance data. At a command prompt, you can enter the following command (on one line) to start ExMon tracing and create an .etl log file:

tracelog.exe   -start "Exchange Tracing"   -f    -guid  

As above, ETL filename is the path and filename of the .etl log file; GUID filename is the path and filename of the guid.txt file, which by default is in the same directory as exmon.exe.

The TraceLog utility is part of the MicrosoftWindows 2000 Server Resource Kit and can be downloaded from the Microsoft Web site at http://www.microsoft.com/windows2000/tech info/reskit/tools/existing/tracelog-o.asp. To terminate the TraceLog thread and close the .etl file, execute the command

tracelog.exe   -stop "Exchange Tracing" 

The resulting .etl log file, similar to the one that System Monitor generates, can be opened using ExMon, as described earlier.

Although ExMon is a pretty solid and stable utility, I've heard reports of instances in which the data trace remains active after ExMon has crashed (rare) or after the process has been explicitly terminated (more likely). This situation could occur with any of the three modes of ExMon data collection. The symptoms of this situation include a continually growing .etl file and an Unknown Start Trace Error (183) message when you attempt to start ExMon directly. The data-collection thread will continue to run until the .etl file reaches a maximum size of 512MB, at which point, tracing will stop. However, you can terminate the tracing at any time by using the TraceLog -stop switch, as shown above.

The ExMon Experience
ExMon provides performance information only for users who are connected via the MAPI interface (e.g., the Outlook client or the original Exchange Capone client). Performance information for users connected through other protocols (e.g., POP3, IMAP4, OWA, or Outlook Mobile Access—OMA) isn't reported. Mobile Exchange users connected through ActiveSync aren't reported on individually, but ActiveSync effectively acts as a MAPI client; therefore, aggregate performance information can be gleaned for ActiveSync users. Information about server-side MAPI connections (e.g., Exchange System Manager—ESM) isn't processed by ExMon.

Although ExMon will happily operate with Exchange 2003 SP1 or later or Exchange 2000 SP2 or later, you'll get the best results when the environment in which you run ExMon data collection uses Exchange 2003 SP1 on Windows 2003. When you run ExMon (in any data-collection mode) on an Exchange 2000 platform, all clients show a version number of 0.0.0 (as Figure 2 shows). However, when the Exchange platform is Exchange 2003, ExMon displays accurate client version information.

If users are using multiple clients to access their Exchange inbox, all the current client versions and IP addresses for those clients will be displayed. For example, if a user is using Outlook and OWA at the same time, from two systems, to access his or her Inbox, there will be two values in the Client Versions field and two addresses in the Client IP Addresses field. If the user is also a Research In Motion (RIM) BlackBerry user, a third value will be displayed in each of these fields to represent the Black-Berry server connection.

ExMon displays the Avg. Foreground Client Latency (ms) and Max. Foreground Client Latency (ms) counters only when the server is Exchange 2003 SP1 or later and the client is Microsoft Office Outlook 2003 or later. In addition, the By Clientmon view in ExMon, which displays client-specific information (i.e., performance information such as RPC Latency and Succeeded RPC Count rather than information related to the server such as CPU time and CPU percentage), is valid only with Outlook 2003 clients.

You can use one of three views to display ExMon data: By User, By Version, or By Clientmon. By User, which Figure 2 shows, organizes the aggregated data by individual Exchange user. By Version organizes the information based on specific client MAPI version. By Clientmon specifically relates to performance information collected from Outlook 2003 clients.

Running ExMon in live data-collection mode with a short sampling interval (1 or 2 minutes) is useful for troubleshooting performance problems for individual clients and users. However, for longer periods of analysis, you might want to use the System Monitor–based or TraceLog–invoked data collection method. Although you can view the .etl file by using the ExMon utility, you might want to export the data to a more readable or usable format. To convert the .etl log information into comma-separated value (CSV) file format, you can use the command

exmon.exe -SU     

where -SU indicates that user statistics are to be processed, CSV filename represents the path and filename of the intended CSV output file, and ETL filename represents the path and filename of the input .etl log file. You can also use the -SC parameter to process Clientmon statistics and the -SV parameter to process version statistics. You can specify all three parameters in the same command, but you must associate a separate output file with each parameter. The parameters are case sensitive.

Exporting ExMon data to CSV format lets you easily manipulate it by using an application such as Microsoft Excel. And the data that ExMon provides can be used in a variety of ways. For example, if you're migrating clients from one version of Outlook to another, you can create and analyze ExMon CSV files on a regular basis to chart the progress of the migration. And if you have concerns that using Outlook 2003 in Cached Exchange Mode might put the Exchange system under strain at various times of the day (e.g., during early-morning logons), creating CSV data files as the migration proceeds will let you determine which clients cause the most load. If Exchange systems are becoming overloaded, you can determine this trend before it becomes a problem and move users from server to server accordingly. Even if you aren't in the middle of a migration, you can isolate heavy users and move them to dedicated servers.

In addition to Outlook clients showing up in ExMon data, some applications might show up as well. Remember that ExMon reports on all MAPI clients, so applications that use MAPI to work against Exchange data stores (e.g., backup applications) will be evident when you view ExMon data. Applications will typically have a user identity of their own. For example, Figure 2 shows a QSMDaemon user that integrates with Exchange to provide a call-handling system.

Microsoft positions ExMon as a versatile utility for performance investigation and analysis of client behavior, especially given its ability to collect data over the long term via System Monitor and TraceLog. This capability isn't really that new—by using Performance Monitor, we've been able to gather this kind of data on an ongoing basis for some time. But ExMon highlights the specific performance counters that are relevant for this kind of investigation—counters that weren't previously that obvious to most system and Exchange administrators. However, I think the real power of ExMon stems from its live data-collection capability with a short sampling interval. This almost real-time insight into the key performance characteristics of a user's connection is invaluable for problem identification.