Diagnosing Problems in IIS

Learn how to troubleshoot IIS 5.0 and IIS 4.0 problems

In "IIS Exception Monitor," November 1999, Ken Spencer wrote about using Microsoft IIS Exception Monitor to diagnose problems in IIS 4.0. IIS-related problems have become more complex, and administrators must know a lot more to diagnose them. This month, I show you several new techniques and tools that you can use to isolate and resolve IIS problems. The techniques I describe are the same techniques I use every day when I'm working with customers to solve their IIS problems. I give you a brief introduction to IIS's architecture, then offer tips, tricks, and things to remember about troubleshooting IIS. (If you want to delve into the core Windows NT information I provide, start with the resources I list in the sidebar "Support Extras," page 2.)

How IIS Works
The IIS WWW services run in processes based on the server settings and OS version. The processes that the WWW services can use are

When you're using tools to monitor a process, keep in mind all the processes that can be involved. Also, remember that by default, IIS 5.0 runs all Web sites in the pooled OOP process.

IIS, ASP, and Threading
IIS uses a pool of threads (i.e., instances of code execution) to handle incoming requests and work on those requests. This pool of threads is called the Asynchronous Thread Queue (ATQ). IIS uses a thread pool because reusing an existing thread is faster than making a new thread each time a new request comes in. The ATQ initially handles all incoming requests. After the initial handling, IIS might send the request to another queue or leave the request in the ATQ. Here's a list of request types and the queue in which they run:

Keep this information in mind when you're troubleshooting a hung system or a nonresponding server. If no ASP files are responding but you can open an .htm page, start looking specifically at which ASP requests are running. If IIS won't serve .htm pages, start your troubleshooting by looking at the whole server instead of just ASP. (ASP might still be the cause of the problem, but you need to get the whole picture first.)

NT Synchronization Objects
IIS relies heavily on NT synchronization objects for efficiency. An NT synchronization object lets a running thread put itself to "sleep" while it waits for information, which is important in a multithreaded application such as IIS. For example, in a multithreaded OS, the OS lets each running thread have a certain amount of time to run, then the OS saves the thread's state and lets the next thread in line run (based on priorities). The OS essentially cycles through all the threads that it's letting run. (Note that this overview is very simplistic. The actual way in which NT selects what thread will run next is much more complex.)

The swapping of one thread for another is called context switching, which NT Performance Monitor can measure. The default amount of time a thread can run varies based on processor type, the OS (e.g., NT Server, NT Workstation), and the priority of the thread. For this example, I use a middle-of-the-road value—2700 microseconds.

A user calls up a Web page that goes to a back-end server and retrieves stock quotes. The Web page then takes that information and sends a page with the requested data back to the client. The time it takes to get the stock information might be about 1 second, which seems like an eternity compared with the 2700 microseconds per context switch. So, instead of having the thread stay in the rotation when you know the thread won't be doing anything for a long time, you can take the thread out of the rotation and let NT tell you when the data you're waiting for has come back. Taking the thread out of rotation allows more time for other active threads to work; it also reduces the number of context switches, which can save time and processing overhead. Synchronization objects perform this task.

Because of NT's ability to use synchronization objects and because IIS is usually retrieving data from somewhere for most requests (and this retrieval is always longer than the context time), you can assume that at any given time, most of IIS's worker threads (both ATQ and ASP) will be out of the rotation, or asleep. Therefore, you can often see a lot of threads on an IIS server but still have low CPU utilization.

IIS, TCP/IP, and Incoming Requests
IIS relies heavily on TCP/IP, and you often see a lot of problems at this level. IIS sets up sockets in TCP/IP to listen for incoming requests, then initiates NT synchronization objects that notify IIS when a request comes in. One of the ATQ threads receives the request, and IIS starts processing it. From a troubleshooting standpoint, you must check the status of the network to make sure that IIS can talk to back-end servers or the clients. You can also use network traces or connection statistics.

Thread Creation and Use
Now, let's look at how IIS manages the ATQ. I don't discuss everything that goes on under the hood, but explain the parts you can affect and monitor. IIS starts with two threads in the ATQ. IIS increases the thread count as more simultaneous requests come in. IIS always tries to keep the thread count at least one thread higher than the number of current requests, assuming that this number doesn't put the thread count over the maximum that IIS allows. The default maximum number of threads is four per processor. (You set this value in the MaxPoolThreads entry in the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesInetInfoParameters registry subkey.)

However, a caveat exists. Certain requests can cause IIS to increase the thread count value above the value of MaxPoolThreads for a short period of time. For example, CGI requests can cause the thread count value to temporarily increase up to 10 threads more than MaxPoolThreads. ISAPIs can also temporarily increase the count value. By default, ISAPI requests can increase the pool size by 2 times the number of megabytes of RAM in the machine with a minimum of 64 threads and a maximum of 256 threads. Although you can use the PoolThreadLimit registry value to override this value, I recommend that you change it only in extreme circumstances because changing the value can adversely affect performance.

Let's look at a few examples that will help illustrate what's happening. Let's say you're running on a one-processor machine with 256MB of RAM. Your default value for the maximum number of threads is 1 * 4, or 4; the value for your absolute thread limit is 2 * 256 = 512, which is greater than 256, so you set the thread count to 256. Assume that in all of these cases, you're at your default two threads and you haven't handled any incoming requests. Now, picture these scenarios:

Remember that this increase in the thread count value is temporary. Therefore, when IIS finishes handling the requests, the thread count value goes back down.

Now, let me throw a wrench in the works. The thread count doesn't go down right away. The value that increases and decreases is MaxPoolThreads. The ATQ uses a hard-coded internal algorithm to create and destroy threads, which it does based partly on the value of MaxPoolThreads. So, in the previous scenarios, all you really did was change this value. IIS then decided to change the actual thread count. When the requests are complete, you lower the value, but IIS doesn't immediately decrease the actual number of threads. In fact, the Time to Live (TTL) of an inactive ATQ thread is 24 hours, which means that if you handle 30 ISAPI requests and IIS increases the thread pool count to 31, for the next 24 hours, you could handle 31 simultaneous .htm requests because the threads already exist.

Because of the long TTL of threads, IIS reuses threads in the pool, starting with the most recently used and working backward. Thus, if you have 31 threads and for the next 24 hours you have at most 16 requests at a time, IIS can terminate 15 threads.

Next Month
In each of the next articles in this troubleshooting series, I'll concentrate on one specific type of problem and how you can locate the problem to resolve it. I'll constantly reference information in this article and also introduce more architecture information as needed. Next month, I'll start showing you how to apply the knowledge you've gained this month to isolating and fixing problems you might encounter.

Note: IIS handles all static requests (e.g., .htm, .jpg) in the inetinfo.exe process regardless of which isolation setting the application housing the requested object has.

Note: All values I discuss here are IIS default settings. To change these settings, please refer to the Microsoft articles "The Art and Science of Web Server Tuning with Internet Information Services 5.0" (http://www.microsoft.com/technet/iis/iis5tune.asp) or "Internet Information Server 4.0 Tuning Parameters for High-Volume Sites" (http://www.microsoft.com/technet/iis/technote/iistun.asp), or go to http://www.microsoft.com/technet/iis/maintain.asp.

Note: The actual thread count isn't what changes but rather the value you use to set the thread count. The value might go up or down, but IIS doesn't immediately change the actual thread count.