3 Steps to Troubleshooting Device Drivers

I’m waiting for an application to start…again. As I stare at the hourglass on my Windows XP desktop, I wonder what is slowing things down this time. I press Ctrl+Shift+Esc to bring up Windows Task Manager and click the Processes tab. I’m ready to kill the offending process, but I find nothing but the usual System Idle Process at the top of the list.

Over time, Windows system performance tends to degrade. Many performance problems can be resolved by eliminating unnecessary programs from startup, defragmenting the hard disk, and cleaning the registry. In my case, however, none of these actions worked. I don’t usually get into desktop support issues but since it was my desktop, I was determined to find and resolve the problem.

When I searched the Internet, I found many people complaining about the same symptoms in blogs and forums, but they didn’t provide any solutions. I got a few clues when some people mentioned that the problem went away after they upgraded a driver or replaced a disk controller. At this point, I concluded that the performance problem was probably related to a driver, but I didn’t know for sure. So, I dug deeper. After wading through some device-driver development articles, I found a gem. The Microsoft TechNet article “Advanced DPCs” (www.microsoft .com/technet/sysinternals/information/ advanceddpcs.mspx) discusses how to diagnose driver performance problems by examining deferred procedure call (DPC) queue activity.

DPCs wreaking havoc on Windows system performance is a common problem in not only workstations but also servers. Knowing how to recognize, diagnose, and solve a DPC problem can save you hours of troubleshooting and possibly save you from having to reinstall or reconfigure the problematic workstation’s or server’s OS.

1. How to Recognize the Problem
To recognize the problem, you need to know about DPCs and how to perform an initial check in Windows Task Manager when you’re experiencing a performance problem. DPCs are a part of the Windows interrupt handling architecture. Interrupt handling consists of two components, both of which are part of a device driver. The first component is the Interrupt Service Routine (ISR), which quickly allows the hardware to get the device to stop interrupting. The driver actually handles the processing around the interrupt later by queuing a worker thread from the created DPC queue object. In short, this means that a driver does its initial work with a hardware device in a DPC. Because both the ISR and DPCs are vendor-provided within the driver for a device, a poorly written kernel-mode driver can have a significant impact on overall system performance.

When you’re experiencing a performance problem, you can use Windows Task Manager to see whether it might be due to DPCs. When a performance problem might be related to DPCs, you’ll see that the CPU usage is high on the Performance tab, but no processes are taking the blame on the Processes tab. You’ll also see that System Idle Process is showing the highest CPU usage on the Processes tab, which is typical for an idle system.

The System Idle Process typically doesn’t significantly affect system performance. However, when DPC problems exist, the kernel will be using a large percentage of the CPU, which can impact system performance. You can find out the kernel’s CPU usage by selecting the Performance tab and choosing the Show Kernel Times option on the View menu. The bottom red portion of the CPU Usage graph shows the CPU time being used by the Windows kernel. The sample CPU Usage graph in Figure 1 reveals that something loaded by the kernel is monopolizing the CPU.

2. How to Diagnose the Problem
A kernel can monopolize the CPU for several reasons, so the next step is to diagnose the problem to determine whether DPCs are in fact causing the high CPU usage and if so, which device is using those DPCs. You can use three tools to drill down and get the details needed for diagnosis:
• Performance Monitor (perfmon.exe). This is a built-in Windows tool.
• Process Explorer (procexp.exe). You can download this free utility from Microsoft at www.microsoft.com/technet/ sysinternals/ProcessesAndThreads/ ProcessExplorer.mspx.
• Kernel Profiling Tool (kernrate.exe). This tool is part of the Microsoft Windows Server 2003 Resource Kit, which you can download at www.microsoft.com/downloads/details.spx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en.

To determine whether DPCs are causing high CPU usage, you can use Performance Monitor or Process Explorer. I find Process Explorer easier to use than Performance Monitor for this purpose, but I’ll cover both in case you prefer to use Performance Monitor.

In Process Explorer, you can quickly and easily see whether DPCs are causing high CPU usage. In the main window, the System Idle Process is broken into three groups: Interrupts, DPCs, and System. Just doubleclick DPCs to bring up the DPCs properties page and select the Performance Graph tab. The top two graphs show the DPCs’ CPU current usage and CPU usage history, as Figure 2, page 53, shows.

Under the Processor ob-ject, Performance Monitor includes a counter named % DPC Time, which tells you the percentage of time a processor is spending on receiving and servicing DPCs. Comparing this percentage to the % Processor Time value reveals what portion of total CPU usage is consumed by a thread in the DPC queue, as Figure 3, page 53, shows. For information about howto use Performance Monitor, see “Performance Management inWindows” (March 2003,InstantDoc ID 37933).

After Process Explorer or Performance Monitor has confirmed that the CPU is being disproportionately consumed by activity in the DPC queue, you can use the Kernel Profiling Tool to determine which driver is causing the problem. This command-line tool lists kernel modules, including kernel-mode device drivers and the percentage of kernel time that they're consuming.

Continue on Page 2

The Kernel Profiling Tool has many arguments you can use, but for our purposes, you can run it without any arguments, with the command

kernrate

After running this command, you firstreceive output like that in Figure 4. Wait forabout 30 seconds or so, then press Ctrl+C.While the Kernel Profiling Tool runs, keepan eye on DPC activity in Process Exploreror Performance Monitor to ensure the CPUusage remains consistently high during thesample.

Figure 5 shows an excerpt from the second part of the sample output. As you can see in the “Results for Kernel Mode” section, the problematic driver is intelppm, the Intel processor driver that’s part of the OS load. This driver throttles the CPU in order to conserve power consumption to extend battery life. The Intel processor driver probably wasn’t the cause of the problem; instead, the problem was likely due an interoperability issue with this driver and other hardware or drivers installed on my system. Regardless, it’s not required on a nonportable computer where battery life is of no concern.

3. How to Solve the Problem
The options for solving a DPC problem are to reinstall, update, or disable the driver that’s causing the problem. Which option you choose depends on the driver. For example, you can’t disable a driver that your system needs or update a driver when no updates exist. In my case, because the Intel processor driver isn’t a required driver for my system, I decided to disable it.

There are several ways you can disable drivers, including through the registry, Device Manager, and Recovery Console (RC). I opted to disable the Intel processor driver through the registry. If you’re interested in using RC or Device Manager to disable a driver, see the articles referenced in the “Learning Path.”

The registry data for the Intel processor driver is in the HKEY_LOCAL_MACHINESYSTEMCurrentControl SetServicesintelppm subkey. To disable this driver, I changed the Start entry from a value of 1 to a value of 4. Table 1 shows the Start values you can use for subkeys under the HKEY_LOCAL_MACHINESYSTEMCurrentControlSet Services key. For more information about how to read and change entries in these subkeys, see the Microsoft article “CurrentControlSetServices Subkey Entries” (support.microsoft.com/kb/ 103000).

As Microsoft continually reminds us, it’s important that you understand how to restore the registry before you edit it. It’s also important to reboot before and after editing the registry. Rebooting before a change is important to ensure a clean Last Known Good Configuration in case you need to restore the registry. (The Last Known Good Configuration includes everything under CurrentControlSet key and is updated after a successful logon.) Rebooting after a change is necessary for the change to take effect.

3 Simple Steps Is All It Takes
If your applications are loading slowly or don’t load at all, the culprit might be a device driver and the DPCs it makes. By using tools such as Windows Task Manager, Performance Monitor, Process Explorer, the Kernel Profiling Tool, and the registry, you can quickly recognize, diagnose, and solve driver performance problems.