Configuring DEP to Prevent Malware Attacks - 28 Aug 2008

Q: What's the Data Execution Prevention (DEP) security feature that Microsoft introduced in Windows XP SP2? How can it make my Windows system more resistant to malware attacks?

A: DEP is a buffer overrun defense feature. Buffer overruns are often used by malware to get unauthorized access to a system or to execute its malicious payload on a system. A buffer overrun attack typically writes executable malicious code to a program’s memory buffers and then tricks the program into executing the malicious payload. The execution of maliciously injected code can be prevented by DEP. DEP lets an OS mark memory locations that should contain only data as No eXecute (NX). When an application attempts to execute code from NX-marked memory locations, the OS DEP logic will block the application from doing so.

DEP is supported in Windows Server 2008, Windows Vista, XP SP2, Windows Server 2003 R2, and Windows 2003 SP1. Two versions of DEP are available: hardware-enforced DEP and software-enforced DEP. Hardware-enforced DEP leverages a processor feature that AMD refers to as the NX-bit and Intel refers to as the eXecute Disable (XD) bit. Today, all Intel and AMD CPUs support DEP. However, this hasn't always been the case: in the early days of DEP, AMD supported NX only on its 64-bit processors, and Intel supported XD only on the Itanium processor, EM64T 64-bit processors, and a small number of 32-bit Prescott processors. Software-enforced DEP lets Microsoft provide DEP on 32-bit processor systems that aren't equipped with a NX- or XD-compatible processor. With software-enforced DEP, the processor-level NX- or XD-bit functionality is provided by set of cookies that the Windows OS automatically adds to data objects stored in the system memory.

You can verify whether your system supports hardware- or software-enforced DEP from the DEP configuration settings, which you can access by opening the System Control Panel applet and then selecting Advanced Settings, Performance, Data Execution Prevention. At the bottom of the Data Execution Prevention tab, you'll see a reference to the type of DEP your system supports.

Figure 1 shows the DEP configuration settings on a Vista system. The bottom line reads "Your computer’s processor supports hardware-based DEP." If your system supports software-enforced DEP (meaning your machine doesn't have a NX- or XD-compatible processor) the bottom line will read "Your computer’s processor does not support hardware-based DEP. However, Windows can use DEP software to help prevent some types of attacks."

On XP SP2 and Windows 2003 SP1 and later Windows OSs, DEP is enabled by default. However, DEP doesn't always protect all the programs that are running on your system. The programs that are protected by DEP are defined by DEP’s protection level. DEP supports two protection levels. The first protection level (level 1) protects only the Windows system code and executables and doesn't offer DEP protection for additional Microsoft or third-party applications that run on your system. The second protection level (level 2) protects all executable code that runs on your system. This level provides DEP protection for both Windows system code and the Microsoft or third-party applications that run on your system. Application developers can enable DEP support in their applications by using the /NXCompat compilation switch. By default, Vista and XP SP2 run DEP at protection level 1; Server 2008 and Windows 2003 SP1 run DEP at protection level 2.

Administrators can configure the DEP protection levels from the DEP configuration screen that Figure 1 shows. In Figure 1, DEP is enabled for essential Windows programs and services only—this is DEP protection level 1. To enable DEP protection level 2, select the Turn on DEP for all programs and services except those I select radio button, which is the default setting on Server 2008 and Windows 2003 SP1 systems.

Protection level 2 also lets you exempt certain applications from DEP protection, as Figure 1 shows. This functionality is very important because there are legacy applications that won't run properly when DEP is enabled. (For example, at the time of this writing, Microsoft Word was automatically exempted from DEP.) Before you switch your DEP protection to level 2, you must run an application compatibility test to ensure that all applications will run properly when DEP is enabled. To exempt one of your applications from DEP, you can add the application’s executable to the excluded list by clicking the Add button in the DEP configuration screen.

Another way to exempt an application from DEP is to create a software fix that you distribute to your systems that automatically disables DEP for a given application on those systems. Microsoft refers to this type of software fix as a DisableNX shim. DisableNX shim is described in greater detail in the Microsoft Application Compatibility Toolkit (ACT). ACT also includes the Compatibility Administrator tool, which can help you create the software fix for one of your applications. You can download ACT 5 from http://technet.microsoft.com/en-us/windowsvista/aa905078.aspx.

You can easily verify whether an application is protected by DEP by checking the DEP column of the application’s process in the Windows Task Manager. If you don’t see the Data Execution Prevention column in Task Manager, you can easily add it using the View, Select Columns Task Manager option. When you check the DEP column for the Microsoft Internet Explorer 7.0 executable (iexplore.exe) in the Task Manager, you'll notice that DEP is disabled by default. You can enable DEP protection for IE from the IE Advanced Internet Options by selecting Enable memory protection to help mitigate online attacks in the Security options. (Note that changing this property requires administrator-level permissions.) In IE 8.0, DEP protection will be enabled by default.