The Beginner's Guide to Optimizing Windows NT Server - 01 Aug 1997

Here's what you can do to tune NT Server'snetwork I/O, disk I/O, and CPU utilization
Now that you've made a strategic decision on a server that supports yourrequirements today and has room to grow tomorrow, you want it to run at its verybest. In my June article, "The Beginner's Guide to Optimizing Windows NTServer, Part 1," I focused on optimizing Windows NT Server's memorysubsystem. This month, I'll examine tuning NT Server's other subsystems, such asthe network I/O, disk I/O, and CPU. Let's review some basic tuning concepts.

Basic Tuning Strategy
To optimize NT Server's performance, develop baselines for each subsystem.Monitor the system closely to identify bottlenecks and plan for future capacityrequirements. Finding the cause of a bottleneck can be difficult because all theserver's major resources--CPU, memory, disk I/O, network I/O, andapplications--are interrelated. Solving one problem can cause another. Try onechange at a time and compare the results to see whether the change was helpful.Always test your new configuration, and then test it again to be sure changeshaven't adversely affected your server. I will discuss only NT Server's built-inmonitoring tools: Performance Monitor (Perfmon), Network Monitor, and TaskManager.

Tuning the Hardware
Hardware tuning is as important for disk optimization as it is for memoryoptimization. Set your disk adapter's BIOS settings for maximum performance andstability. Be sure you have turned on write-back cache and set the SCSI channelto negotiate for the fastest setting the disks can handle. Many drives supportthe Ultra Fast/Wide SCSI (40MB per second) standard. Many disk adapters sold inthe last year support Ultra Fast/Wide SCSI but were set to the slower Fast/Widesettings of 20MBps, the speed that the typical disk drive supported. Ask yourhardware manufacturer for the most recent BIOS release level and optimum BIOSsettings for disk adapter/disk drive combination.

For network I/O connections, set the adapter appropriately for yournetwork. Many of today's adapters support multiple network speeds, includingautosense mode. In autosense mode, the network adapter tries to automaticallydetermine your network's speed and then sets itself to operate at that speed. Iprefer to set the adapter to the exact speed of the network devices the serverwill be interacting with to ensure the best performance possible. Not allservers and network devices want to play together nicely when you've set them torun at peak throughput settings. Be sure that you are using the most recentdevice drivers for disk and network adapters. You will often see improvedthroughput and efficiency just by using the most recent release. Most hardwaremanufacturers have this information on their Web sites. Microsoft keeps similarinformation on its Web site about NT Server patches or service packs. Thesepatches attempt to fix known problems and occasionally include performanceenhancements. Always test new drivers or patches to guarantee that they operateas advertised.

Network I/O
NT Server's network I/O subsystem becomes a bottleneck when demand fornetwork resources outpaces what NT Server can provide. Clients and other serversystems must be able to connect to NT Server with sufficient bandwidth and lowenough latency to provide adequate response times to support customers'requirements. Therefore, you need to understand what type of workload yourclient systems generate and which key network architecture components are inuse. To determine where your bottlenecks are and how to fix them, you mustunderstand the type of network protocol (e.g., Ethernet, NetBEUI) and physicalnetwork (e.g., Ethernet, Fiber Distributed Data Interface--FDDI) you'reusing.

Perfmon. The two primary tools for sleuthing out networkI/O bottlenecks in NT Server are Perfmon and Network Monitor. You can usePerfmon in logging mode over a period of time to develop a baseline and toanalyze the server's resources. Start Perfmon from Start, Programs,Administrative Tools. To enter logging mode, select View, Log, Edit, Add To Log.The key object to observe is Network Interface. (To access the Network Interfaceobject, first install the Simple Network Management Protocol--SNMP--from ControlPanel, Network, Services; then reboot.) Select the Network Interface object,click Add, and then Done. To begin the logging session, select Options, Log.Enter the name of your log file, specify a sampling interval, and then clickStart Log. If you want to look at data the system is currently monitoring, youneed to start a second copy of Perfmon and select Current Activity. (Perfmonstops logging when you view an active log file.)

To access the data that Perfmon collects in the log file, you need to stoplogging and look at the log file. To stop collecting, select Options, Log, StopLog. To read the log file, click Options, Data From, Log File and enter the nameof the log file. From View, select Chart (or Report or Alert). Now choose Editand Add To Chart and select the counters you want to observe. (For moreinformation about using Perfmon for logging, see Part 1 of this article in theJune issue and Michael D. Reilly, "The Windows NT Performance Monitor,"March 1997.)

Perfmon collects data for each separate physical network adapter instance.Traditionally, the Network Interface object has had two notable counters:

The first step in detecting a network bottleneck is looking for symptoms.Are users complaining about slow downloads from your server? Are applicationsthat involve the network and the particular server you are investigating runningsluggishly? In NT 3.51 the Output Queue Length counter is a good indicator. Avalue of 1 or 2 in the Output Queue Length counter is acceptable. However, ifthis measurement begins increasing (particularly above a value of 3 or 4), yournetwork I/O adapter is waiting on the network and can't keep up with theserver's requests to move data onto the network. If the length of the outputpacket queue is frequently higher than baseline, a bottleneck might beoccurring. In NT 4.0, the help information states that because the NetworkDriver Interface Specification (NDIS) queues the requests, the counter's valueis always 0. However, research and testing show that this statement is false.The Output Queue Length counter provides useful information in NT 4.0.

Observe the Bytes Total/sec counter to see whether NT Server is waiting fora slow client to receive data or waiting on an overloaded network. Theinformation that this counter provides is not useful until you compare it withthe network architecture in use. For example, if you are running a TCP/IP-based10Base-T Ethernet network, it has a theoretical maximum throughput of 1.25MBps.If you take into consideration Ethernet and TCP/IP overhead, the maximum goesdown to roughly 1MBps, and the network rarely achieves that speed. Ethernet usesa Carrier Sense Multiple Access/Collision Detection (CSMA/CD) scheme,which leads to a poor degradation curve under heavier loads. Thus, as thenetwork utilization increases above the 40 percent to 60 percent range, clientson the network begin noticing slower response times to their network requests.These slower response times result from increased collisions on the physicallayer cable, which in turn causes the adapter to retry the transmission after arandom delay.

In a 10Base-T network, if the Bytes Total/sec counter shows a sustainedoutput rate between 400,000 bytes per second and 600,000 bytes per second andyour output queue continues to grow, your network adapter and architecturecombination has become a bottleneck. Don't forget to check the other NT Serverresources--memory, CPU, and disk subsystems--to be sure that they aren'tbecoming bottlenecks.

Network Monitor. NT Server includes a version of theNetwork Monitor tool from Microsoft's Systems Management Server (SMS). With theSMS version, you can monitor any system that has an active network monitoragent. However, NT Server's Network Monitor lets you view network traffic onlywith respect to the server it is running on. As you see in Screen 1, to installNetwork Monitor from the Control Panel, you select Network, Services, Add.Select Network Monitor Tools and Agent, and then reboot your server to finishthe installation. Now you can start Network Monitor from the Start, Programsmenu.

After you've launched Network Monitor, select Capture, Start to beginmonitoring the network. Network Monitor is a powerful addition to NT Server: Youcan use it to debug network problems associated with the server down to thepacket/protocol level.

To detect potential network bottlenecks, Network Monitor places the networkcard(s) into promiscuous mode; that is, the card analyzes every packet movingthrough the server's network interface. Under Network Monitor's default view,you can quickly obtain information such as network utilization, frames persecond, bytes per second, and the originating systems of your network traffic.Beware: Network Monitor can use a fair amount of resources on a busyserver. In my tests, I found Network Monitor consumed approximately 4 percent ofthe CPU resources on a 166MHz Pentium Pro with 512KB of RAM. You set the amountof RAM that Network Monitor uses in the Capture buffer settings. The settingmust be low enough to not use too much memory (RAM) but high enough so thatNetwork Monitor doesn't drop any packets. Typically, a 2MB to 3MB buffer settingis sufficient.

Using Network Monitor under its default settings, as shown in Screen 2,observe whether the value of the % Network Utilization counter is consistentlyabove the 40 percent to 60 percent range. If this range of utilization iscommon, you are using your current adapter close to its maximum capacity, and itis becoming a bottleneck. To alleviate this situation, consider adding anothernetwork adapter to segment your network. This action will add more network I/Obandwidth to your server by physically and logically separating your networksinto two parts. You can activate routing under NT Server through Control Panel,Network, Protocol, Routing to pass data between networks. NT Server is capableof supporting basic routing functions under a light-to-medium load, but thisrouting function will add some overhead to your system.

Tuning Network I/O
Network Monitor earns its keep in this segmentation tuning process. In thedefault view, Network Monitor displays bytes sent and received between theserver and the various systems demanding service. You can use this informationto help you decide where to physically connect the various systems on yournetwork; you want to balance your network load by distributing the more heavilyused systems between your two network segments. Even though segmentationinvolves some new cabling (physical addressing) and logical subnetting,it is a proven technique to optimize your server's network I/O and is relativelyeasy to implement. Once you have used this divide-and-conquer technique,continue to monitor your server's network utilization and output queue to headoff future bottlenecks.

Another technique for optimizing your server's network I/O performance isto bind your network adapter to only those protocols that your network is using.Under Control Panel, Network, Protocol, you can check which protocols arecurrently installed. Removing unnecessary protocols lowers the amount of memorythat NT Server allocates for network I/O and ensures that your network has nounnecessary traffic.

Finally, a popular optimization technique for a TCP/IP-based network is toadjust the TCP/IP window size. The TCP/IP sliding window is a dynamically setbuffer for transmitting packets. You can easily adjust the window size in NT'sRegistry, but the optimal size depends on your network architecture. Typically,finding the best window size for your environment requires several network teststo improve on the default NT Server settings. (For more information about thistopic, see Bill McLaren, "13 Tips for Optimizing Internet InformationServer," April 1997.)

The techniques I've described ensure that the network I/O subsystem on NTServer isn't the network bottleneck. But remember that many facets of anetwork's architecture can cause a network bottleneck. These facets includeclient configuration, application design, the physical network, networkprotocol, and network devices (e.g., routers, hubs). Many tools are on themarket for network analysis, including network sniffers and network managerssuch as Unicenter TNG (Computer Associates), Open View (HP), and Works (Cisco).

Disk I/O
To tune disk I/O performance under NT Server, you can purchase additionaldrives and internal/external RAID units, if the server hardware can supportthem. Short of adding hardware, you can tune the disk subsystem in other ways.In particular, review your Perfmon logs regularly to be sure that you haveevenly distributed the disk subsystem's load. A common source of contention isrunning all applications on the root NT Server disk, which can quickly become abottleneck.

To find out if you have disk bottlenecks, first eliminate the possibilitythat the problem is due to insufficient memory. You can easily confuse a diskbottleneck with paging file activity when you have a memory shortage. To helpdistinguish between disk activity related to the virtual memory manager pagingto disk and applications using the disks, keep the paging file systems onseparate dedicated disks. This technique simplifies using Perfmon to distinguishwhich busy disks are not associated with paging activities. With Perfmon, reviewthe Avg. Disk Queue Length and % Disk Time counters under the LogicalDisk objectfor each disk of interest. See Table 1 for definitions of these counters.(Remember to turn on the disk counters with the diskperf-ye command, as Part 1of this article described.)

A high % Disk Time value is not unusual; a high percentage shows that youare receiving a good return on your disk investment. But beware, % Disk Timeexceeding 60 percent can lead to increased response times from an overly busydisk. A problem can occur when the % Disk Time value is high and the Avg. DiskQueue Length is high or increasing. In this situation, a busy disk is nothandling all of its requests well. When you find a hot spot on a particular disksubsystem, move some of the associated application files from their current disklocation to another disk that is not as heavily loaded. Then continue to monitorthe disk I/O subsystem to ensure that you haven't created a new disk hot spot.

Another technique that helps isolate disk performance problems, improvesperformance, and lowers the head movement rate over the disks is to format onlyone logical drive per physical drive. For example, if you have three diskdrives, create only three logical drives, such as C, D, and E.

To improve the performance of your disk drive subsystem, consider matchingthe file system Allocation Unit Size to the block size of the application youare using. For example, suppose SQL Server is using a 4KB block size. When youformat a file system on a new disk drive, launch Disk Administrator, create thepartition, commit the partition changes, select Format, and then set theAllocation Unit Size to 4096 bytes. Matching the file system block sizes canimprove the efficiency of the disk transfers when you use the application.

For example, if you have four 4KB blocks of data to write to the disk andthe disk is slightly fragmented, you might end up with eight separate 2KB writesto disk on a file system created with a 2KB Allocation Unit Size. When readingthis file, the disk heads subsequently have to move to eight random locations.If you use a 4KB Allocation Unit Size, NT Server has to write to disk fourtimes, and thus the disk heads have to move four separate times to complete aread of the data. Test your particular Allocation Unit Size configuration todetermine your optimum file system layout, because each application and disksubsystem environment is a little different.

Another way to improve your disk I/O subsystem performance is to try not touse it. If you have so much RAM that you need only I/O operations to permanentlysave your data, you might not have a problem. But the more power users haveavailable, the more power they use. As I mentioned in Part 1, on the NT Servermemory subsystem, you can use the Control Panel, Network, Server option to sethow NT Server uses RAM. By appropriately selecting RAM use, you can allocatemore space for the dynamic allocation of the file system cache size. Be awarethat the file system cache competes with other applications for main memory. Anapplication that is hogging memory--which you can observe in Task Manager,Processes, as shown in Screen 3--can lower your file system cache hit rate. (Youcan observe the system cache hit rate in Perfmon, Cache object.) Obviously, howyou allocate memory is a trade-off between better application performance andbetter file system and associated disk I/O performance. You must decide whichgoal is more important for your server.

Tuning CPU
To determine whether NT Server has a CPU bottleneck, first ensure that thesystem doesn't have a memory bottleneck. If the system is pagingexcessively or thrashing because the application or process has insufficientmemory, the system is using CPU cycles to service all the paging transactions.If you don't find a memory shortfall, look for a CPU bottleneck. Use Perfmon toobserve all processor instances and closely review the counters shown inTable2.

A high % Processor Time (e.g., 91 percent) does not mean the system has aprocessor bottleneck. If the CPU is servicing all the NT Server schedulerrequests without building up the Server Work Queues or the Processor QueueLength, the CPU is servicing the processes as fast as it can handle them. Aprocessor bottleneck occurs when the Processor Queue Length is growing; %Processor Time is high; and memory, the network, or disks don't havebottlenecks. Thus, when a CPU bottleneck is occurring, the CPU cannot handle theworkload that NT requires because its CPU is running as fast as it can, butrequests are queued waiting for CPU resources.

One way to diminish processor bottlenecks is to move to a faster CPU, whichis particularly helpful if you have predominantly single-threaded applications.If you have a multiuser system using multithreaded applications, you canpreserve your investment (i.e., not throw out the older CPU when the new onearrives) by adding processors.

You can also tune CPU performance by using Task Manager to identify theprocess that is consuming most of the CPU time and then adjust its priority. Aprocess starts with a base priority level, and its threads can vary two levelshigher or lower than its base. If you have a busy CPU, you can boost a process'spriority level from Task Manager, Processes. Right-click the process, chooseSetPriority, and then select Realtime, High, Normal, or Low. (Be careful whenyou set the processor's priority to Realtime--the process can become selfish andnever release the CPU, possibly making your system unstable.)

By increasing a process's priority, you can ensure that the process willget more CPU time than the other user applications. The priority change takeseffect immediately, but the process is manual; when you stop the application andthen restart it, the application will return to its original priority. To letyou launch applications from the command line at various priority levels, NTServer provides the Start command, which you can place in batch scripts that yourun at server startup or directly from a command prompt. To review the Startcommand's options, enter

start /? | more

at the command prompt. For more information about the Start command, see BobChronister, "Ask Dr. Bob Your NT Questions,", and ChristaAnderson, "Foreground Application Handling in NT 4.0," June 1997.

Know Where You're Starting
Before you try to tune NT Server, you need to establish a baseline and findout where the bottlenecks are. The best way to tune your server is to understandthe server hardware, the NT Server operating environment, your networkarchitecture, and the applications running on your system, because thesecomponents are closely interrelated. Until you know how your system performsover time, you won't know how much you've improved your NT Server's performance.