Performance Management in Windows

When you think about performance management in a Windows environment, the first thing that comes to mind might be the ubiquitous Performance Monitor tool. Performance Monitor has been the go-to tool for performance information since Windows NT's early days. Alternatively, you might choose Task Manager or Windows Management Instrumentation (WMI), which exposes Performance Monitor data and provides a model for exposing other types of management information related to performance. Here I provide tips about how you can get the most from some old but updated friends—such as Performance Monitor and WMI—and I highlight some new Windows XP tools and explain how you can use them to determine how your systems are performing.

What Is Performance Management?
For many administrators, performance management in Windows means running Performance Monitor and checking CPU utilization, disk busy time, or memory pressures on your servers. Usually a performance problem triggers you to use Performance Monitor—maybe the server is responding slowly or users are unable to access the server. This approach, which I call "performance firefighting," isn't ideal. For effective performance management, you need to track your servers' performance before something goes wrong. Taking a proactive approach to performance management lets you determine what is typical. Armed with this data, you can then determine when things aren't running smoothly and what the cause might be. You can also use proactive performance data to plan for future capacity needs. For example, if you know that your Microsoft IIS-based Web server runs at 60 percent CPU utilization with 200 simultaneous visitors to your site, you can estimate when the server will max out and how many users you can handle when it does. Based on your site's growth rate, you can also determine when you'll need to buy new hardware.

A system's performance depends on several factors, such as CPU utilization, CPU queue length (i.e., how many tasks are waiting for the CPU to service them), disk busy time (i.e., how much of a disk drive's time is spent satisfying requests), disk queue length (i.e., how many requests are waiting for the disk to service them), available physical memory, and network interface utilization. Table 1 outlines the most important Performance Monitor counters to watch in each of these areas. Unfortunately, few counters have a magic limit beyond which you need to be concerned. Memory pressures that result in a low Available Bytes metric can be the result of a poorly written application that leaks memory or a system that's under-equipped for its workload. You need to continually monitor the most important metrics to determine what is typical for the workload on your systems. You should be able to easily spot changes and quickly remedy your performance problems. You also need to watch for trends as your server workloads change. For example, if you're adding new users to a Web server every week, you'll probably notice increases in the counters during heavy use periods. Bottlenecks in system capacity often begin in one area (e.g., memory) and cascade across other system areas, such as CPU and I/O capacity.

Performance Monitor Revisited
Performance Monitor is still your best friend in XP and Windows 2000. The tool has had some improvements since NT 4.0 that provide additional capabilities. In Win2K, a Microsoft Management Console (MMC) snap-in implements Performance Monitor's capabilities. If you start Performance Monitor in XP or Win2K, you get the MMC window that Figure 1 shows.

When you start Performance Monitor in XP, the tool has three preloaded counters: Pages/sec, Avg. Disk Queue Length, and % Processor Time. If you want to start Performance Monitor without any counters, you need to remove the read-only file attribute on the perfmon.msc file in the %systemroot%system32 directory. Go to a command prompt and enter

attrib ­r "perfmon.msc"

Start Performance Monitor, then highlight each counter and click the black X at the top of the System Monitor chart to delete the three counters. Select File, Save to save the changes to the console. To reflag the file as read-only, go to a command prompt and enter

attrib +r "perfmon.msc"

System Monitor
In NT 4.0, Performance Monitor contains a realtime chart, as well as logging and alerting features. XP and Win2K's Performance Monitor separates these functions. In XP and Win2K, the realtime chart is called System Monitor. Below System Monitor, you'll see the Performance Logs and Alerts tool. You can think of the System Monitor control as strictly a performance data­viewing tool. You can view realtime information, but you can't save the data. Click the plus symbol at the top of the System Monitor chart to add performance counters. System Monitor's Performance Logs and Alerts tool collects historical performance data that you can view. To create an MMC tool that includes just the System Monitor tool, open a blank MMC window and select File, Add/Remove snap-in. Select Add, then select ActiveX Control from the list of available snap-ins. Select System Monitor Control from the list of controls.

The Performance Logs and Alerts Tool
System Monitor is useful if you just want to fire up a quick realtime session to show performance counters on local or remote machines. But if you want longer-term persistent logging of performance data, you need to use the Performance Logs and Alerts tool's logging features. The Performance Logs tool lets you store Performance Monitor counter data from multiple servers to a central log file that you can later use System Monitor to view or other tools to manipulate. If you expand the Performance Logs node in MMC, you'll see three categories: Counter Logs, Trace Logs, and Alerts. I focus mainly on Counter Logs. The Alerts tool simply lets you take an action (e.g., send an email or a Net Send message) based on a counter passing a specified threshold value. Alerts is sort of a poor man's system-monitoring tool.

To illustrate how you can use Counter Logs, we need to create a new log session. Right-click the Counter Logs node and select New Log Settings, then give the logging session a unique name. The dialog box that opens, which Figure 2 shows, lets you select the objects and counters that you want to monitor across one or more servers.

The first thing you'll notice when you create a new logging session is that System Monitor creates the performance log for data storage in a folder called C:perflogs on the system running the logging session. You can use the Log Files tab to change this path, but you must first select the objects and counters to log. Click Add Objects to select all the counters for an object. If you want to log only individual counters, click Add Counters. For either option, you can select objects and counters from a local or remote machine.

You can set the frequency with which you'd like to collect data, as Figure 3 shows. If you're collecting long-term data, I recommend using a reasonable interval—especially if you're monitoring many counters on multiple machines. Log files can get large if the interval is too small. Start with every 15 minutes, assess your log file size after a few intervals to estimate how big your log files will grow, and adjust accordingly.

To change the default user ID, enter a new ID in the Run As text box. System Monitor uses the user ID to connect to the machines you're monitoring.

You need sufficient permission to collect Performance Monitor data. The permissions on the HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionPerflib registry subkey control access to Performance Monitor data. Performance Monitor data feeds through this registry subkey into tools such as System Monitor. By default, this subkey grants access to performance data to members of the local Administrators and Server Operators groups, as well as to any user logged on to the local machine. You can adjust permissions on this subkey to restrict which groups or users can access data.

After you set the options for which objects and counters on which servers you want to log, you can use the Log Files tab to adjust the format of the log file that generates. You can use the Schedule tab to change the schedule on which the logging session runs. If you set the session for a timed start or stop, logging starts as soon as you apply the changes to the session. If you set the start as manual, you must right-click the log session in the MMC window to explicitly start logging.

Under the Log Files tab, Performance Monitor logging supports storing log files in several formats in addition to the native binary .blg file format. You can store your data to text-based comma-separated value (CSV) files or even SQL Server tables—which is a new Windows Server 2003 and XP feature. If you choose SQL Server, you must specify a data source and table that you want to use to store the data—but the SQL Server option is useful if you need to collect a large amount of performance data over time.

When you start logging, you'll notice that a 65KB log file writes to the log directory. Each time you stop and start logging, a new log file generates with an incremented number in its name. An interesting new feature in XP and Win2K lets you continue logging performance data even if the Performance Monitor tool isn't running. In NT 4.0, you had to install the Microsoft Windows NT Server 4.0 Resource Kit­based Datalog service to get this unattended monitoring functionality. XP and Win2K include the Performance Logs and Alerts Windows service. This service automatically starts when a logging session starts (and stops when the session stops).

Another XP and Win2K performance-logging improvement is saving logging sessions. In NT 4.0, you can create logging workspaces that store to a binary file. If you want to reuse your selected performance objects and counters in a logging session for a different set of servers, you need to recreate the workspace file for each server or servers you want to monitor. This task is tedious if, for example, you want to distribute a set of logging settings to run locally on a set of servers. In XP and Win2K, all the settings files (i.e., logging and alerting) are implemented as HTML-based files. Thus, you can easily modify a settings file such as the one that Listing 1 shows. The code at callout A in Listing 1 shows the name of the log file to which performance data will be stored during this session. The code at callout B lists the command, script, or batch file that should run when the logging session stops. In this case, no command is specified. The code at callout C starts to list the performance objects and counters being tracked during this session. Listing 1 shows only a portion of a full session file. If you understand each entry, you can easily edit a session file manually. You can change the server name and even the counters you're monitoring, then distribute the settings file to any machine from which you want to perform logging. To create an HTML settings file for a logging session, right-click the session name in Performance Monitor and select Save Settings As.

Command-Line Tools for Performance Logging
You might wonder how to use XP and Win2K's performance-logging capabilities from the command line. I don't know an easy way in Win2K without using resource kit tools, but XP provides a new tool called Logman that lets you start and stop logging sessions and create new logging sessions from the command line. (For a list of XP and Windows 2003 performance-management tools, including Logman, see the sidebar "New Performance-Management Tools.")

The first command that Listing 2 shows creates a new logging session called c:perflogsIISLogging.blg, adds the Process% Processor Time counter for the inetinfo.exe process running on the local server, and sets the collection interval to 15 minutes. The second command starts the logging session. If you start the MMC Performance Monitor tool, you'll see this new logging session in the list of Counter logs.

You can use Logman's ­s option to start logging on a remote machine (e.g., ­s \server1). In such a case, you're actually starting the Performance Logs and Alerts service and running logging on the remote server—not the server on which you ran logman.exe. You can also change the counter path in Listing 2 to \server1 Process(inetinfo)% Processor Time to run logging remotely against server1 from the machine on which you set up logging.

WMIC
The Windows Management Instrumentation Command-line (WMIC) tool is new in Windows 2003 and XP. This tool provides a convenient approach for accessing WMI objects and their properties. (For information about WMIC and other new Windows 2003 and XP performance-management tools, see the sidebar "New Performance-Management Tools.") To start WMIC, go to a command prompt and enter

wmic

The first time you run WMIC, the tool installs itself into the WMI namespace. Although Microsoft intended for WMIC to make using WMI easier, the tool's syntax is quite cryptic. WMIC relies on aliases that represent WMI classes you might need to access frequently. For example, the WMIC alias pagefile is the equivalent of the WMI query Select * from Win32_PageFileUsage. So, if you enter

pagefile

at the WMIC prompt, the tool returns current page-file usage on the system. As you might know, Performance Monitor counters are available through WMI. Unfortunately, WMIC doesn't provide aliases to WMI-based Performance Monitor data. To work around this problem, you can directly call the relevant WMI classes for Performance Monitor counters. For example, if you want to determine the current physical memory utilization on a server, you can go to the WMIC command line and enter

path win32_perfformatteddata_perfos_memory

This statement tells WMIC to return the data held in the WMI class win32_perfformatteddata_perfos_memory. Figure 4 shows part of the command's output. The complete output would include every property associated with the WMI memory object you called. To see only the Available Bytes property, change the command to

path win32_perfformatteddata_perfos_memory get AvailableBytes

You can also call WMIC from the command line, rather than interactively. For example, to view the Available Bytes metric, go to the command line and enter

wmic path win32_perfformatteddata_perfos_memory get AvailableBytes

If you want to return multiple properties, you can simply use a comma to append them (e.g., AvailableBytes,AvailableMBytes,CacheBytes).

You might wonder how to obtain the WMI class names for all the supported Performance Monitor counters. The best method I've found is to download Microsoft's WMI Tools (http://msdn.microsoft.com/code/sample.asp?url=/msdn-files/027/001/566/msdncompositedoc.xml&frame=true) and use the WMI Object Browser tool to view all the classes under Win32_PerfformattedData.

A Word About Trace Logs
Event tracing is a low-level, extensible Windows mechanism for giving you detailed information about the subsystems you're monitoring. Microsoft ships several event-trace providers in the box that relate to metrics such as kernel performance and file-system activity. Windows 2003's version of event tracing also includes a Lightweight Directory Access Protocol (LDAP) provider that lets you trace every LDAP call that enters and exits an Active Directory (AD) server. Event tracing is extensible because application developers can use it as a logging mechanism for tracing events or transactions within their applications. The resulting data can be valuable if you're trying to discover performance bottlenecks in a multi-tiered application environment. Event tracing is currently an untapped Win2K feature—I suspect that Microsoft will put a lot of work into leveraging this facility to provide better and more detailed performance metrics in future Windows applications.

In the meantime, you can use XP and Win2K's existing tools to build a reasonably capable performance-management system for your Windows infrastructure. XP and Win2K's new logging features provide good unattended performance management. Although you can't scale these logging tools to numerous servers, you can use the tools to monitor your most crucial machines.