IIS Troubleshooting Tips and Tricks from the Field

Over the years, Microsoft IIS has grown to be an exceptionally flexible product, supporting numerous technologies, products, extensions, and configuration options. IIS can do almost anything, from servicing simple static pages to being a fully blown, enterprise-grade reverse proxy server. Like any complex product, though, IIS requires troubleshooting on occasion. Here we'll look at some of the most common challenges that web developers and administrators face when deploying and developing applications for IIS and how to troubleshoot them properly.

Runtime Errors and Exceptions

The most common issues customers encounter when deploying or developing for IIS are runtime errors and exceptions, which include various errors from the 40x and 50x families of HTTP errors. These errors include the common 404 and less-common 400 errors, as well as the dreaded 500 internal server error.

The appearance of these errors varies depending on the IIS server's configuration. The HTTP 500 error, which is typically an application or pipeline error where something failed before the application sent any response, could manifest itself as a simple error in the browser, such as the examples shown in Figure 1.

Figure 1: HTTP 500 Internal Server Error Examples

However, if the error occurs in a managed code .NET application, the error might look like that shown in Figure 2.

Figure 2: IIS Server Configuration Error

The first step in figuring out such errors is a differential diagnosis. This means that we need to deduce from user reports that the site hangs are actually IIS worker process (w3wp.exe) crashes (which are what happens when a web application experiences a major error) or possibly something else, such as a configuration problem of some kind. We can find this information from multiple sources:

Examining the IIS logs is a good first step, as the logs are easy to read and would confirm whether or not the error is really a 500 error. If it is a 500 error, the log line would show the HTTP status code to be 500 0 0, as in Figure 3, instead of the expected 200 code.

Figure 3: HTTP 500 Error as Shown in IIS Log

The next step is to look for more details about the error. It's not uncommon for administrators who see this error to assume that it portends the worst. The error seems to suggest there's a problem with the code, and many system administrators are not trained to read or debug code. In reality, this type of error can be really simple to understand once you realize that the seemingly meaningless XML tags are not the actual error but rather a message telling you that the default configuration prevents IIS from actually showing you what's wrong. (Why so? Because we don't want strangers who visit the site to get details about your code!)

To see what's really going on, we need to configure the server to be more informative. This is done by adding a section to the site's web.config file. The steps are:

Note: The reason we use RemoteOnly here is to display the error only on your local server and not to users trying to visit the site. If you want the error to be displayed publicly, or if you are setting the configuration on a Windows Azure Web Sites server (to which you don't have direct access), you would set the mode to Off.

Now, if you connect to the site again, the error page will show more information, which could potentially solve the riddle right then and there. For example, the error page shown in Figure 4 indicates that the code is trying to populate a variable that wasn't declared.

Figure 4: IIS Server Compilation Error

An alternative way to view this error, if you prefer, is to inspect the Application event log on the system. When an application error such as the one shown in Figure 4 occurs, it will be logged as coming from ASP.NET. In such a situation, the event details would contain the same information, as shown in Figure 5. (The code filename, line, and error details are highlighted in yellow.)

Figure 5: Viewing Error Details in Application Event Log

Another common variation of an internal server error is error number 500.19. This error is also one you might run into when you're using Azure Web Sites and developing advanced applications. The error looks like the page shown in Figure 6.

Figure 6: IIS Invalid Configuration Error

Error 500.19 is actually an entire family of errors pertaining to problems in the website's configuration that's specified in web.config. Figure 6 shows a permission problem, which could happen if the access control over the website's folder is not set correctly and IIS cannot read the configuration.

Another variation of this is error 0x80070021, which happens when a section you are trying to configure in a site's web.config conflicts with a setting set in the higher-level ApplicationHost.config file. For example, on an IIS server, the MIME types for content compression are set by default to disallow a site-level web.config configuration. If you try to add a configuration section to web.config without adjusting the inheritance settings first, you would receive this error.

Yet another variation of error 500.19 is 0x80070032, which happens when a section is missing configuration information. It's not always easy to figure out what you did wrong, especially if your configuration is complex or if you're trying to configure something that doesn't have a lot of examples in the documentation, so for this error, an incremental approach is typically useful.

Although it would be impossible to cover every conceivable variation of the 500 error, the key lesson here is to get IIS to show you the complete details. Once you have those, the accompanying text often will provide all the information you need. If you need more information, you can search for the error code or keywords online and in that way find enough information to pinpoint the actual problem. For other errors, the error codes are usually more direct—see this Microsoft article for a list of the most common codes and additional information about them. Also see this list of Win32 errors, which can also help you learn more about many issues.

Crash and Hang Problems

A process crash happens when there is an error in a web application, which causes an exception, leading to termination of the IIS worker process (w3wp.exe). Unlike a regular application crash, where the user can easily see the application disappearing, with a web-based app there's no such disappearance visible to the naked eye, so things can sometimes get murky. Sometimes, the user might get a Service Unavailable message in the browser (which might be totally unclear and confusing to an average user, but at least it's something to start with). Other times, IIS will start a new w3wp.exe process to service the request. Although the second option seems desirable, as the user doesn't run into errors, in reality, it could be almost as bad as the Service Unavailable error, because unbeknownst to the administrator or application developer, the application might be crashing all the time, providing a slow and unreliable service.

The good news is that the server will log events even if the user doesn't experience a problem, which helps the administrator or developer catch such errors early. Typically crash-related events would be event ID 5009, which indicate a worker-process crash, or event IDs 5010 and 5011, which indicate that the process-activation service could not get a reply from a worker process. When any of these errors occur, they are often accompanied by an event ID 1000, which is for an application crash.

Having a process crash occasionally doesn't always warrant a full investigation. No code is perfect, and if you run into a crash a few times a month, it's usually not the end of the world. IIS recovers and starts a new w3wp.exe process upon the next request that comes in for the application. Naturally, though, if the crash causes data loss, or if it's frequent enough to cause turmoil within your user base, then the investigation typically involves getting a crash dump, which contains a snapshot of the process when the exception occurs.

Examining a crash dump can be a challenge, as the connection between what's in the dump and the root cause isn't always evident. If you are the application developer, an alternative to this is to write your application code to include trace information that might make it possible to troubleshoot directly, or at least help in the process. This would involve adding to each function in your code a call to log actions and key variables into a text file (typically through a dedicated logging function). For example, when you run a loop, you can have the function log a line for each iteration, like this:

fTraceLog("Starting going through aRequestItems array at " & now)For I = 0 to ubound(aRequestItems)     fTraceLog("Request Item " & I & " contains " & aRequestItems(i))     sRequestAction=http:// & cstr(aRequestItems(i)) & "/SetRecords.asp" & iJIDNumbernextfTraceLog("Finished going through aRequestItems array at " & now)

With this code, if there is a problem with the loop, the last item in the trace could potentially reveal what it is. Keep in mind, though, that this sort of tracing could lead to a high strain on the server, so it should normally be turned off until an investigation is required.

The tricky part about crash dumps is that they need analysis, which is traditionally seen as a developer-level task. Fortunately there are some advanced tools that make crash-dump analysis more accessible for non-developers. The free Debug Diagnostic Tool (DebugDiag) contains some pretty powerful automatic analysis tools, which can shed light on a lot of issues without requiring you to look at a single line of code.

To use DebugDiag, you need to create a rule, which will instruct the debugger to detect when IIS crashes and collect the crash dump. To do so, follow these steps:

After you have set up the rule, you need to reproduce the issue, if you can, or wait for IIS to crash again. Once that happens, the rule will cause a creation of a dump, which you can now analyze. To perform the analysis, follow these steps:

Figure 7: DebugDiag Analysis Screen

Once the analysis is finished, DebugDiag generates an HTML report and opens it in your browser automatically. The report lists all the problems found and also provides suggestions for solving them. Based on the problem's severity, the issues found are categorized in three types: Error, Warning, and Information.

Depending on the issue, the report could be a simple read, but it could also leave you more confused about the problem. As with any other type of troubleshooting, you'll need to carefully read the data in the onscreen report to gain a better understanding of the problem. Used properly, search engines can also be helpful. For example, you might see this message displayed in the report:

The application domain is restarting because of reason:- A change was made to the Bin folder or to files in it

Too many application domain restarts are not good for an application. Please refer to the article Lost session variables and appdomain recycles for more details on this issue.

As you can see, this message suggests what the problem is and even provides a link to further reading. Here's another useful message:

The requests start getting queued if all the threads available to process the request are busy. Look at the callstacks of the threads which are processing the request to see why they are running for a long time. Also, look at the ASP.NET Request Queue Report to see the ASP.NET Threadpool settings and the runtime for which the requests are getting queued.

There's More to Learn

Naturally, we cannot cover the entire range of possible issues with IIS here. However, as you use DebugDiag more, you will find that it's exceptionally intuitive and helpful in IIS debugging. For additional information about debugging IIS issues, also see the following articles from Microsoft's knowledgebase can guide you further. These articles also discuss scenarios other than a simple crash:

Richard Marr is a senior Escalation engineer with Microsoft support. He has worked for Microsoft since 2000. During his career, he has supported Microsoft Access, IIS, and ASP.NET and now works with Windows Azure Web Sites. Richard is a graduate of Mount Saint Vincent University in Canada and holds a Bachelor of Business Administration. He lives with his family in Texas.

Wei Zhao is a Shanghai-based Escalation Engineer for Microsoft. In his role, he helps Microsoft's customers design, develop, deploy, and operate enterprise solutions using IIS, ASP.NET, and Windows Communication Foundation (WCF). Most recently, Zhao is also focusing on Windows Azure Web Sites.