Inside NT's Interrupt Handling

NT's interrupt management affects realtime system applicability

One of Windows NT's primary responsibilities is interfacing a computer to its peripheral devices. Like most modern operating systems, NT can dynamically integrate device driver programs to manage devices. Device drivers typically use interrupt signals to communicate with the devices they control. When a device has completed a driver-directed operation or when the device has new data for the driver, the device generates an interrupt signal. Depending on the state of the CPU, either a function within the driver immediately services the device's interrupt, or the CPU queues the interrupt for later servicing.

NT implements interrupt processing differently from many other operating systems, so how NT and NT device drivers process interrupts and how thatprocessing affects other operations can be confusing. For example, systemsprogrammers often ask me how interrupts affect thread scheduling--a naturalquestion because the CPU can receive interrupt signals almost anytime, evenwhile user programs perform ordinary processing. A common misconception is thatNT won't service low-priority interrupt signals while high-priority threads areexecuting time-critical tasks. (For more information about thread priorities andscheduling, see "Inside the Windows NT Scheduler, Part 1", July 1997.)

The way NT handles interrupt processing affects NT's viability as anoperating system for realtime (time-critical) environments (e.g., aircraftguidance systems). NT's rich development environment, user interface, and threadpriority scheme make NT attractive to designers of realtime systems. However,realtime environments require the ability to predict how fast an operatingsystem will react to interrupts; thus, the way NT implements interrupt handlingaffects how suitable it is for realtime applications.

In this column, I'll first provide background information about interruptsand describe NT's Interrupt Request Level (IRQL) architecture. Next, I'llpresent how device drivers register to receive notification of interrupts thattheir devices generate and what device drivers typically do upon notification.Finally, I'll describe the effect of interrupt processing on the NT schedulerand comment on how NT's interrupt processing affects its applicability forrealtime systems.

Devices and Interrupts
All the major hardware architectures that NT runs on haveinterrupt-controller hardware to translate device interrupts into signal levelsthat feed into the CPU. The interrupt controller defines the interrupt priorityscheme: When a device triggers an interrupt of a given priority, the controllermasks (or withholds) from the CPU all interrupts of priority less thanor equal to the device interrupt's priority. Until the CPU signals to theinterrupt controller that it has finished servicing an interrupt, the interruptcontroller pends (or puts on hold) lower priority interrupts but letshigher priority interrupts occur. When the interrupt level on the controllerdrops below an interrupt's priority, the controller lets the interrupt proceedto the CPU.

Device controllers connected to modern CPU buses (such as the popular PCIbus) dynamically determine which interrupts their device will use. When a deviceneeds to inform its driver of an event (such as the availability of new data),it generates an interrupt that the device driver will recognize and acknowledge.The device driver registers an interrupt service routine (ISR) with theoperating system, and the operating system executes the ISR in response to theinterrupt. The ISR's job is to read status information from the device,acknowledge the interrupt, and stop the device from signaling the interrupt.

Interrupt Request Levels
NT manages interrupts by mapping interrupt-controller interrupt levels ontoits hardware-independent table of interrupt levels. The hardware abstractionlayer (HAL--the NT module custom-written for individual interrupt controllers,motherboards, or processor chip-sets) performs the mapping. In a multiprocessorsystem, any processor can receive interrupts, so NT maintains an independentIRQL for each processor. A processor's IRQL represents the level of interruptthat the CPU is currently masking and directly corresponds to interrupts thatthe CPU's interrupt controller masks. Because NT's IRQLs are not tied to anyhardware specification, NT can also map non-hardware interrupt types into itspriority hierarchy. The operating system uses software interrupts primarily toinitiate scheduling operations, such as thread switching, or I/O completionprocessing.

When NT services a hardware interrupt, NT sets the processor's IRQL to thecorresponding mapped value in NT's IRQL table. NT programs the interruptcontroller to mask out lower priority interrupts, and device drivers (as well asNT) can query the IRQL to determine its value. (NT permits some operations onlywhen the IRQL is less than certain values, as we'll see later.)

The size of NT's IRQL table varies among processor architectures (Intel,Alpha, etc.) to better map the interrupt levels that standard interruptcontrollers provide, but interrupt levels that device-driver designers and NT'sdevelopers might find interesting have symbolic names. Table 1 summarizes thesymbolic IRQL names and their corresponding numeric values on Intel and Alphaarchitectures.

In Table 1, the lowest IRQL priority is Passive Level. When aprocessor is at this state, no interrupt processing activity is occurring. Whencode in user applications such as Word and Netscape is executing, the processoris at Passive Level. NT's goal is to return from higher IRQLs to Passive Levelas quickly as possible so that NT can service new interrupts immediately andprograms can get their work done.

The next two IRQLs above Passive Level (APC Level and Dispatch Level) arescheduler-related software interrupt levels. When the system is at APC Level,the executing thread will not receive asynchronous procedure call (APC)requests, which NT commonly uses for I/O cleanup operations. When the systemdecides that a scheduling decision needs to take place (e.g., when a thread'sturn on the CPU ends), it issues a Dispatch Level software interrupt. (I'lldescribe the role of Dispatch Level software interrupts later in the article.)

All IRQLs higher than Dispatch Level relate to hardware interrupts. Asystem's peripheral devices (e.g., disk drives, keyboards, serial ports) havehardware interrupts mapped to IRQLs in the Device Level range. You can see in Table 1 that on Intel processors, the range is 3 through 26, and on Alphamachines, the range is 3 through 4. The fact that such a difference existsbetween the two ranges implies that NT does not really prioritize general deviceinterrupts. Even on Intel processors, where hardware interrupts might havedifferent IRQL values, the assignments are arbitrary.

The IRQLs above Device Level have predefined associations with certaininterrupts. Profile Level relates to the kernel profiling timer, Clock Levelrelates to the system clock tick, IPI Level relates to signals sent from one CPUto another, and Power Level relates to power failure events. NT reserves butdoes not currently use High Level.

Interrupt Objects
Device drivers need a way to tell NT that they want specific functionsexecuted when the processor receives interrupts associated with their devices.To satisfy this need, device drivers register an ISR with the I/O Manager bycalling the IoConnectInterrupt subroutine. The parameters passed toIoConnectInterrupt describe all the attributes of the driver's ISR, includingits address, the interrupt the ISR connects to, and whether other devices canshare the same interrupt.

IoConnectInterrupt initializes an Interrupt Object to store informationabout the interrupt and its connected ISR. IoConnectInterrupt also programs theprocessor's interrupt hardware to point at code that IoConnectInterrupt placesin the Interrupt Object. Thus, when the CPU receives the interrupt, controlimmediately transfers to the code in the Interrupt Object. This code calls NT'sinterrupt servicing helper function, KiInterruptDispatch, which raises theprocessor's IRQL, calls the appropriate ISR, and lowers the IRQL to its previousvalue. KiInterruptDispatch also obtains a spinlock specific to theinterrupt and holds it while the ISR is running. A spinlock is a synchronizationprimitive commonly used in the NT kernel. The spinlock ensures that the ISRwon't execute simultaneously on more than one processor (something that mightcause device-driver writers some grief).

In NT, an ISR usually does nothing more than read a minimal amountof information from the interrupting device and acknowledge to the device thatthe driver has seen the interrupt. In other operating systems, ISRs oftenperform additional duties, such as fully processing an interrupt by readinglarge data buffers from or writing large data buffers to a device. However, oneof NT's goals is to minimize time spent at high IRQLs, so NT postpones mostinterrupt servicing until the IRQL decreases. ISRs request a deferred procedurecall (DPC) to inform the I/O Manager that they have work to do at a lower IRQL.A DPC is another function in the driver that the I/O Manager will call after theISR finishes; the DPC performs most of the interaction with the driver's device.

Figure 1 depicts the typical flow of NT interrupt servicing. A devicecontroller generates an interrupt signal on the processor bus that a processorinterrupt controller handles. The signal causes the CPU to execute the code inthe Interrupt Object registered for the interrupt; the code in turn calls theKiInterruptDispatch helper function. KiInterruptDispatch calls the driver's ISR,which requests a DPC.

NT also has a mechanism to handle interrupts not registered by devicedrivers. During system initialization, NT programs the interrupt controller topoint at the controller's default ISRs. Default ISRs execute special processingwhen the system generates expected interrupts. For example, a page fault ISRmust execute logic for situations in which programs reference virtual memorythat does not have allocated space in the computer's physical memory. Thesesituations might occur when programs interact with a file system to fetch datafrom a paging file or program image, or when programs reference an invalidaddress. NT programs unregistered interrupts to point at ISRs that recognize thesystem has generated an illegal interrupt. Most of these ISRs pop up a bluescreen of death to inform a systems administrator that an illegal interrupthappened.

Deferred Procedure Calls
DPCs are the workhorses of NT's interrupt processing. NT tracks DPCs thesame way it tracks interrupts--in objects. Device drivers usually initialize aDPC Object at the same time they connect to an interrupt. The information adriver must specify includes the address of its DPC function, the processor theDPC function needs to execute on, and the DPC's priority. By default, a DPC willalways execute on the processor the ISR executes on; however, a device driverwriter can override this assignment. In addition, DPCs default to mediumpriority (the choices are low, medium, and high), but a device driver writer canalso control priority.

When an ISR requests a DPC, NT places the specified DPC Object on thetarget processor's DPC queue. If the DPC has low or medium priority, NT placesthe DPC Object at the end of the queue; if the DPC has high priority, NT insertsthe DPC Object at the front of the queue. When the processor's IRQL is about todrop from Dispatch Level to a lower IRQL (APC Level or Passive Level), NTremoves the DPC Objects from the DPC queue. NT ensures that the IRQL remains atDispatch Level and pulls DPC Objects off the queue until the queue is empty(i.e., NT drains the queue), calling each DPC function in turn. Only when thequeue is empty will NT let the IRQL drop below Dispatch Level and let regularthread execution continue.

DPC priorities can affect system behavior another way. NT usuallyinitiates DPC queue draining with a software interrupt whose associated IRQL isDispatch Level. NT generates such an interrupt only if the DPC is directed atthe processor the ISR is requested on, and the DPC has high or medium priority.If the DPC has low priority, the DPC requests the interrupt only if the number of outstanding DPC requests for the processor rises above a threshold, or the number of DPCs requested on the processor within a time window is low. If a DPC is targeted at a CPU different from the one the ISR is running on and the DPC's priority is high, NT immediately signals the target CPU to drain its DPC queue. If the priority is medium or low, the number of DPCs queued on the target processor must exceed a threshold. The system idle thread also drains the DPC queue. Table 2, page 56, summarizes the situations initiating DPC queue draining.

Figure 2 shows a typical sequence of events. First, an ISR requests a DPC,and NT places the DPC Object on the DPC queue of the target processor. Dependingon the DPC priority and the length of the DPC queue, NT generates a DPC softwareinterrupt then or at some later time. When the processor drains the DPC queue,the DPC Object leaves the queue and control transfers to its DPC function, whichcompletes interrupt processing by reading data from (or writing data to) thedevice that originated the interrupt.

Interrupts and Scheduling
The IRQL information in Table 1, page 56, shows that Dispatch Level isassociated with scheduling operations. When the IRQL is at Dispatch Level orhigher, NT masks scheduler software interrupts, which means that NT effectivelyturns off the scheduler. In fact, device drivers (and NT) must not performoperations that require an immediate response by the scheduler when a processoris at an IRQL greater than or equal to Dispatch Level. This restriction includesdoing anything that might indicate to NT that the current thread is giving upthe CPU to wait for some event to occur because that action would cause thescheduler to find a new thread to execute. Another action that demands schedulerintervention is a page fault. When a thread accesses virtual memory thatreferences data in the paging file, NT usually blocks the thread until the datais read. Therefore, at Dispatch Level or higher, NT does not permit access tomemory not locked into the CPU's physical memory. If you've seen the IRQL_NOT_LESS_OR_EQUAL blue screen stop code, you've probably witnessed the effect ofa driver violating these rules.

Disabling the scheduler during interrupt processing has another,less-obvious effect: NT counts the time taken by ISRs and DPC functions againstthe quantum of the thread active at the time the CPU receives an interrupt. Forinstance, suppose Word is executing a spell-check operation, an interrupt comesin from a device, and the device's driver has a DPC that takes all the remainingtime in Word's quantum (and then some). When the CPU's IRQL drops below DispatchLevel, the scheduler may decide to switch to a different thread in anotherapplication, effectively penalizing Word for the interrupt processing. Althoughthis practice sounds unjust, most of the time NT distributes interrupt loadsevenly across the applications in a system.

NT as a Realtime Operating System
Deadline requirements, either hard or soft, characterize realtimeenvironments. Hard realtime systems (e.g., a nuclear power plant control system)have deadlines that the system must meet to avoid catastrophic failures such asloss of equipment or life. Soft realtime systems (e.g., a car's fuel-economyoptimization system) have deadlines that the system can miss, but havetimeliness as a desirable trait. In realtime systems, computers have sensorinput devices and control output devices. The designer of a realtime computersystem must know worst-case delays between the time an input device generates aninterrupt and the time that the device's driver can control the output device torespond. This worst-case analysis must take into account the delays theoperating system introduces, as well as the delays the application and devicedrivers impose.

Because NT does not prioritize device interrupts in any controllable wayand user-level applications execute only when a processor's IRQL is PassiveLevel, NT is not always suitable as a realtime operating system. The system'sdevices and device drivers--not NT--ultimately determine the worst-case delay(the time from when an input device interrupts through when a realtimeapplication processes the input and controls the output device). This factorbecomes a problem when the designer of the realtime system uses off-the-shelfhardware. The designer can have difficulty determining how long everyoff-the-shelf device's ISR or DPC might take in the worst case. Even aftertesting, the designer cannot guarantee that a special case in a live system willnot cause the system to miss an important deadline. Furthermore, the sum of allthe delays a system's DPCs and ISRs can introduce usually far exceeds thetolerance of a time-sensitive environment.

Interrupt Management
Learning about how NT manages interrupts can help clear up confusion abouthow NT's interrupt processing affects thread scheduling and other operations:Thread priorities are essentially independent of interrupt priority levels. Ifyou are a systems programmer, information about NT's interrupt management canhelp you understand how ISRs and DPCs fit into application processing. Finally,if you are considering NT as an operating system for a realtime environment, youcan judge NT's suitability for your situation, or at least recognize some of theissues you need to consider before making a decision.