File-Healing Utilities Help Prevent DLL Hell and Malicious Code

Systems administrators often know the problems associated with troubleshooting an unstable Windows machine. All Windows OSs contain hundreds of common files (i.e., shared code, such as DLLs) that independent developers can use in their applications. In early Windows OSs, problems often resulted when an installed application overwrote a common file with an earlier version of that file—a situation characterized by the phase DLL Hell. Although Microsoft still has a ways to go, it has offered some type of self-healing mechanism to help prevent DLL Hell since Windows 98, when the company began to digitally sign some of its code. By Windows 2000, Microsoft had digitally signed all its code.

To protect crucial system files from being overwritten, Windows Server 2003, Windows XP, Win2K, Windows Me, and Win98 include file-healing tools that are based on digital code signing. (Windows NT doesn't include any file-healing utilities.) These tools are the System File Checker (SFC), File Signature Verification (signverif.exe), Windows File Protection (WFP), and System File Protection (SFP) utilities. Although not their primary intention, these utilities also prevent malicious code (e.g., viruses) from directly modifying protected system executables. To understand how the file-healing utilities work, you first need to understand the basics of digital code signing.

Digital Code Signing
When Microsoft finalizes code, a well-known mathematical hash algorithm (usually Secure Hash Algorithm-1—SHA-1) is run against the code as part of the code-signing process to generate a unique hash result. Within technologically feasible parameters, only the same code will produce the same hash result. Microsoft stores its code's hash totals in digitally signed security catalogs (i.e., .cat files) that reside in subfolders in the %windir%systemcatroot or %windir%system32catroot directory. In Windows Me, you can double-click each security catalog to discover which files (i.e., entries) each catalog protects. As Figure 1 shows, the Security Catalog tab shows each entry's filename, hash algorithm used, digital fingerprint hash result, expected file location, and file version. Although you can open the security catalogs in Windows 2003, XP, and Win2K, those catalogs store only the hash results of the digitally signed files and not the associated filenames. This seemingly small difference can introduce exploitation opportunities, which I describe later in "The Weaknesses" section.

The largest security catalog—called nt5.cat (in Windows 2003, XP, and Win2K) or system.cat (in Windows Me)—is usually from the OS's original installation. As part of the installation process in Windows Me, the OS copies catalog.cab and renames it system.cat. The OS also updates the WFP security catalog or SFP security catalog to make sure new file versions aren't accidentally replaced with earlier versions. Other applications, hotfixes, and service packs have their own security catalogs.

When Windows needs to verify whether an installed file is a valid version, the OS computes the installed file's hash total and compares that total with the result stored in the digitally signed security catalog. If the hashes don't match, Windows replaces the file with the valid original file or a valid updated file from a Microsoft system patch. If the required security catalog is missing or corrupt, the OS will usually restore the original file just to be safe. The valid copies of files typically reside in a predefined file cache area on the hard disk.

SFC and Signverif.exe
In Win98, Microsoft included the SFC utility to check for corrupted, changed, or deleted system files. If the SFC finds any altered files, it prompts the user to replace the file. The SFC in Win98 is a GUI version. Microsoft includes a command-line version of SFC in Windows 2003, XP (including the Tablet PC Edition), and Win2K, but not in NT or Windows Me. The command-line version of SFC can run file scans at boot time, be scripted, and delete and restock (i.e., purge) original file copies in a file cache area. However, unlike the GUI version, the command-line version can scan only Windows system files.

Windows 2003, XP, Win2K, and Windows Me have the related GUI utility signverif.exe. Like the GUI version of SFC, signverif.exe can scan any file or folder (not just Windows system files) for digital signatures.

Most of the instructions in this section cover running SFC under Win98. SFC runs slightly differently on the other OSs, but people with later OSs don't usually run SFC because those OSs have the automated WFP or SFP file-healing utility. I discuss SFC here as a historical reference and to show you the simple beginnings of today's WFP and SFP (which, from this point, I call xFP when referring to them collectively) mechanisms. This discussion also explains why many of the registry settings for xFP refer to SFC.

SFC isn't automatic. You must launch it from the command line or from the Start menu by selecting Programs, Accessories, System Tools, System Information, then selecting System File Checker from the Tools menu. By default, SFC looks for corrupted system files only. You can configure the utility so that it looks for changed or deleted files; however, most people don't use SFC to look for changed files because SFC flags every file updated since the original installation disks were used. You can also use SFC to extract an individual file from an installation disk (although easier ways exist to install files from a disk). So, why does Microsoft include the command-line version of SFC in Windows 2003, XP, and Win2K? The answer is that the command-line version runs faster than the updated signverif.exe.

By default, SFC checks the following folders:

In these folders, the utility looks at the common executable system files (e.g., .386, .dll, .exe, .com, .ocx). You can add or delete file types and folders as well as have SFC check subfolders.

SFC relies on a verification data file, %windir%default.sfc, for filenames and validity checking information. Every time that you run SFC, SFC gives any prior default.sfc file the extension .sf0. SFC saves the scan results to the sfclog.txt file, which you can review to see what files SFC added and updated. When SFC replaces a file, it puts the replaced file in the %windir%helpdesksfc folder in case the replacement file causes a problem and you need to restore the original file.

If you launch SFC from the command line, you can use the following command-line switches:

Note that if you use the /cachesize=x parameter to reduce the amount of disk space used to store valid restoration files, you must use the /purgecache switch afterward to recover the freed space.

When you install or reinstall Windows 2003, XP, Win2K, or Win98, the OS automatically runs SFC at the end of the GUI-mode setup. This run occurs to prevent third-party installations or unattended setups from incorrectly modifying Windows system files.

SFC is useful, but few people use it. Because SFC doesn't run automatically (except when you install or reinstall the OS), an illegitimate system file could easily be installed, used, and remain indefinitely. For that reason, Microsoft created the automated xFP utility.

The xFP Utility
Windows 2003, XP, and Win2K have WFP, whereas Windows Me has SFP. Interestingly, xFP doesn't prevent crucial system files from being altered (i.e., renamed, modified, or deleted). Instead, xFP restores the original files after a change has been made. In doing so, xFP not only protects crucial system files but can also mitigate the damage from malicious attacks of protected files. For example, if a worm deletes a protected file, xFP undoes the worm's action by restoring the original file. Similarly, if a virus modifies a protected DLL, xFP can stop that virus from spreading by replacing the modified DLL with the original DLL. xFP can even save you a little time during a virus hoax. For example, a popular email virus hoax contains a message telling users to delete jdbgmgr.exe. Because jdbgmgr.exe is a protected file, xFP automatically restores that file if users delete it.

Although WFP and SFP perform the same functions, they work differently under the hood. Let's look at the differences between their work processes, protected files, and weaknesses.

The Work Processes
When a protected file is altered, the xFP monitoring agent, which is always present in memory, creates a directory notification event and awakens other xFP protective mechanisms. In WFP, the monitoring agent is winlogon.exe; in SFP, the monitoring agents are stmgr.exe and vxdmon.vxd. Here's a look at what happens in each process.

WFP process. After winlogon.exe wakes up the WFP protective mechanisms, WFP performs the following steps:

SFP process. When stmgr.exe or vxdmon.vxd notices a file change to a protected directory, it wakes up the SFP protective mechanisms and performs these steps:

If many protected files are altered at once—for example, if the system32 folder is deleted—the user will notice a substantial slowdown while xFP restores the original files. Some users have reported that if overburdened, xFP can be overwhelmed and miss restoring key files. However, I haven't experienced this problem.

The Protected Files
Although WFP and SFP protect many of the same files, differences exist. Strangely, WFP and SFP don't universally protect some files that have common names and functions. For example, debug.exe is a protected file in XP but not in Windows Me.

In Windows 2003, XP, and Win2K, WFP protects all .sys, .dll, .exe, and .ocx files and many of the font files (e.g., .ttf, .fon) installed from the original Windows OS CD-ROM. WFP also protects files with the extensions .cat, .drv, .cpx, .sys, .hlp, .chm, .ax, and .cpl. Most protected files are in the %windir% and %windir%system32 directories and their subdirectories. Some technical documents say that WFP protects more than 3000 files, but WFP protects only about 1300 files on my Win2K installation. Interestingly, WFP doesn't protect several files in the C:dllcache directory, which stores unaltered file copies. The files exist in other areas on the disk, but you can delete the files in the C:dllcache directory without WFP kicking in. Unfortunately, no way exists to alter this default behavior without corrupting or disabling the entire process. As I explain in the Web-exclusive sidebar "Disabling WFP and SFP" (http://www.secadministrator.com, InstantDoc ID 38777), I don't recommend disabling the process.

In Windows Me, SFP protects system files in %windir% and %windir%system32 directories and many of their subdirectories, including fonts, help, inf, and pchealth. SFP also protects key SFP files and some files in C:program filescommon files. The %windir%systemrestorefilelist.xml file lists all the files that SFP will protect and those that are excluded. This file doesn't contain hash results. The C:windowssystemsfpsfpdb.sfp file lists the protected files but assists primarily in revealing which security catalog contains a valid hash result for the altered file.

WFP restores files from the %windir%drivercachei386, %winsys%system32dllcache, or network installation directory (if installed from a network). SFP restores files from either the C:_restore or (if installed from a network) network installation directory. If WFP or SFP can't find the file to restore in one of the directories, the utility prompts an administrative user to insert the original Windows OS installation media.

Although SFP doesn't let you configure cache location or size, WFP does. To change the default location of WFP's dllcache directory, you modify the path in the HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionWinlogonSFCDllCacheDir registry subkey. To modify the amount of disk space devoted to the dllcache directory, you can change the HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionWinlogonSFCQuota subkey. By default, SFCQuota is set to 0xFFFFFFFF, which in effect tells WFP to cache all protected files. Alternatively, you can use sfc.exe's /cachesize=x switch to change the cache size. If Windows runs low on disk space, it automatically stops caching any more WFP files.

The Weaknesses
Unfortunately, xFP has several weaknesses. Because Microsoft designed xFP to stop errant software-installation routines and not to prevent malicious code, several of the following weaknesses deal with xFP's inability to stop intruder attacks rather than with its inability to keep the system from crashing because of an installation error.

Weaknesses in WFP only. When a file is altered, WFP compares the altered file's hash results with the hash results in the security catalog. Because of a programming oversight, WFP determines only whether the altered file has a valid stored hash result—it doesn't determine whether the result belongs to the correct file. For example, from within a DOS window, you can overwrite wscript.exe in %windir%system32 with wordpad.exe from C:program fileswindows ntaccessories. Not only does WFP allow this overwrite, WFP then protects the altered file (i.e., wordpad.exe) and places it in the dllcache folder with the forged name. If you try to delete or rename that file, WFP replaces it with a copy of the forged file. I've often used this trick to remove wscript.exe from systems that didn't need it. I suspect that this weakness is because the WFP security catalogs store only hash signatures and not the associated filenames. Although this programming oversight might not seem dangerous, it can be. For example, suppose you overwrite wordpad.exe with wscript.exe. If you open malicious scripts in wordpad.exe, they can do no harm. If you open malicious scripts in wscript.exe, wscript.exe will automatically run any VBScript or JScript commands.

In December 2002, Forensics.org reported two WFP weaknesses. The first weakness is that WFP always trusts previously signed and trusted code. As a result, an intruder can replace a newer signed file with an older signed file that has known vulnerabilities, then exploit those vulnerabilities. The second weakness is that Windows doesn't do the best job of determining certificate authentications or determining who's considered a trusted certificate root signer. As a result, an intruder can use an untrustworthy certificate to sign a malicious file and make it appear as a legitimate protected file. (However, if you're accepting untrustworthy certificates, you have a lot of other potential vulnerabilities besides WFP.)

Weaknesses in xFP. One weakness in both WFP (XP and Win2K only) and SFP (Windows Me) is that both utilities monitor files only in protected directories, which leaves the Windows OS vulnerable to malicious code that's placed elsewhere. For example, a worm can create a file with a name identical to a protected system file, then strategically place that file in an unprotected application directory. If an application needs an OS DLL to run, it always checks the current directory first before searching OS directories. As a result, in XP and earlier, an illegitimate file can pose as a legitimate file, and xFP won't replace it.

Another mutual weakness is that sometimes the utilities' logging isn't accurate or specific. WFP often doesn't log every file alteration to the System log as it should. For example, if you delete dosapp.fon in the protected %windir%fonts directory, WFP restores the original file but doesn't write a corresponding entry in the System log. In addition, generic WFP event-log messages always report a modification, even if the modification was the result of a delete or rename operation. In the same vein, renaming an SFP-protected file always appears in the event log as a delete operation.

As I mentioned earlier, if many protected files are altered at once, the machine slows down substantially while xFP restores the original files. There are even reports of xFP failing to restore all files under a heavy load.

The following weakness is perhaps the most interesting: If a file becomes corrupted rather than being intentionally altered, it won't set off xFP mechanisms. File corruption is a common OS problem, so you'd expect xFP to fix corrupted files. Could a malicious program corrupt a file rather than alter it and thus let the modification escape detection? The inconsistent nature of xFP makes you wonder whether xFP has some serious underlying problems.

Self-Healing OSs and Hardware
The xFP mechanisms are the forerunners to self-healing OSs and hardware. Security and systems administrators have long talked about the day when computers will recognize and fix, without user intervention, such problems as viruses, system failures, and hardware failures. Self-healing OSs and hardware will be especially helpful in light of new fast-spreading worms, such as the Sapphire (aka Slammer) worm, which infected 90 percent of the Internet's vulnerable hosts within 10 minutes of its release, according to the Cooperative Association for Internet Data Analysis (http://www.caida.org/outreach/papers/2003/sapphire/sapphire.html).

Computer scientists want computers to have an antibiotic-like reaction. When an "infection" or "illness" takes place, the system healers repel the invader or fix the failure. In the Wintel world, IBM has been on the forefront of this technology since the early 1990s, when it started to create a rapid response "immune system" to respond to malicious code outbreaks. The work was overly ambitious for its time, and Symantec ultimately assimilated the system into its Norton AntiVirus software in 1998. IBM's self-healing research led to its current autonomic-computing initiative, which involves making servers and software self-configuring, self-optimizing, self-healing, and self-protecting. (For more information about this technology, go to http://www-1.ibm.com/servers/autonomic.) Sun Microsystems, Microsoft, and Hewlett-Packard (HP) are also working on self-healing systems.

Ounce of Prevention Needed
Although self-healing systems sound great, they're not here yet. So, preventing problems is key. You should set up a multi-tiered defense that includes security mechanisms (e.g., antivirus software, firewalls) and xFP or SFC. If the security mechanisms fail to stop an attack or an application or patch erroneously alters a protected file, xFP and SFC can help protect key system files.