Advanced Patch Management

In light of the endless influx of Internet-based worms, scanning scripts, and intruder attacks, patch management has become a growing concern among network administrators. A security guide, checklist, or hardening document wouldn't be complete without some recommendation to keep up with current hotfixes and service packs. But keeping up with patches isn't always as simple as visiting Windows Update and installing the recommended updates. I recommend automated patch-management solutions for updating workstations and noncrucial servers, but you might want to implement more advanced patch-management techniques to protect high-visibility or high-security servers. The strategies that I discuss in this article might not work for all organizations, but those willing to go the extra mile will benefit from the stability and security of careful patch management.

Check the Signatures
Depending on your server's security policy, when you install hotfixes you might receive a warning that the file doesn't contain a valid signature. However, Microsoft claims that it signs all its hotfixes. So why do you get the warning?

The confusion comes because two types of signatures exist: an Authenticode signature and a Windows Hardware Quality Labs (WHQL) signature. A software developer ensures a file's authenticity by appending an Authenticode signature to the file. Windows doesn't automatically check the Authenticode signature for hotfixes. You can verify the signature by right-clicking the file and selecting Properties. You can also use the Chktrust command-line tool that comes with the Microsoft Platform software development kit (SDK) to verify the Authenticode signature. Chktrust.exe will tell you whether a certificate is valid but won't tell you who owns it.

Microsoft stores the WHQL signature in a separate catalog file to show that the company has tested a driver file for Windows XP or Windows 2000 compatibility. Microsoft includes a WHQL signature for each driver file in a hotfix. The warning appears when the WHQL signature for a driver file doesn't match what's stored in the catalog file.

So if the Authenticode signature verifies the hotfix before installation and the WHQL signature verifies the driver files after you install the hotfix, why does the warning still appear? It appears because the hotfix installs new files with new WHQL signatures. Because the new WHQL signatures aren't available until the installation is finished, Windows can't verify the new signatures until the hotfix is installed. You can verify these signatures by running the Windows File Signature Verification tool (sigverif.exe) after you install the hotfix. If you check the Authenticode signature before you install the hotfix and run Sigverif to check the WHQL signatures after installation, you can safely ignore the invalid-signature warning.

Check the Files
Before you install any hotfix on a production server, you should know which files you'll be installing and which files the hotfix will update. By identifying which files will change, you can spot file-version conflicts and determine which changes a hotfix will make to your server. This step adds to the updating process, but it's worth the effort if it prevents downtime of a crucial server.

To list the files in a hotfix, you can manually extract them by using the /x switch. For example, the following command extracts the files from the q123456.exe hotfix:

q123456.exe /x

The /x switch also has an undocumented feature that lets you extract the files to a specific path. When combined with the /q switch, the file extraction requires no user interaction, which means you can use scripts to automate the file-extraction process. For example, the following command extracts the files from the q123456.exe hotfix to the C:temp directory:

q123456.exe /x:c:temp /q

After you extract the files, you can view them and their properties. To make this task easier, you can use the hfinfo.vbs script that I've created. Because Microsoft packages hotfixes differently in Windows Server 2003, this script will work only in Win2K. (To find the script, go to the Windows & .NET Magazine Web site, enter InstantDoc ID 41980 in the InstantDoc ID box, then click the 41980.zip hotlink.) This script extracts the hotfix files and displays each file's date, time, size, and version. To use hfinfo.vbs, type

cscript.exe hfinfo.vbs 

where HotfixFilename >is the name of the hotfix file. Knowing which files and file versions you're installing will make identifying potential conflicts or other problems easier. But most of all, knowing which files you're about to install on your server is simply a good idea.

Watch the Installation Order
With Windows NT 4.0, installing hotfixes in the proper chronological order is crucial. To improve the hotfix installation process, Win2K and later OSs added more thorough version checking to improve the hotfix installation process. Microsoft has also released the Qchain tool (available at http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=a85c9cfa-e84c-4723-9c28-f66859060f5d), which lets you install multiple hotfixes without rebooting the system after each installation. However, situations still exist in which the hotfix installation order is important.

When a hotfix needs to install a file that's in use, the hotfix writes a registry entry that tells the OS to replace the file at the next reboot. When you install multiple hotfixes, the hotfixes might update the same file multiple times. In this situation, qchain.exe checks to make sure that only the most recent file version appears in the list of hotfixes to be installed. You should always run qchain.exe when you install multiple hotfixes.

But qchain.exe isn't perfect. You might still end up with an older file than expected. For example, some file types, such as .asp or .chm files, don't have version information built into the file. Also, qchain.exe works only with files that are in use that Windows can't replace until rebooting. Because a hotfix might replace some files immediately, qchain.exe will never see them.

To avoid these situations, you should install hotfixes in the proper chronological order whenever possible. However, the proper order isn't always clear. You can't just rely on the security bulletin number or the Knowledge Base article number. Although you can usually trust the hotfix release date, the only way to be absolutely sure of the proper installation order is to check the versions and dates of each file in the hotfix.

Don't Rely Exclusively on Automatic Updates
With Win2K Service Pack 3 (SP3), Microsoft introduced the Automatic Updates service, which automatically watches for, downloads, and optionally installs new Windows updates. This service is a huge step forward for Windows security, but solely relying on it does have some drawbacks.

To begin, not all security bulletins involve installing a patch. For example, Microsoft Security Bulletin MS02-064 (Windows 2000 Default Permissions Could Allow Trojan Horse Program) addresses a problem with the default NTFS permissions of the system root folder, but it doesn't provide a patch. Instead, you must manually fix these permissions on each server. The Automatic Updates service can't help you with this and other hotfixes that require manual intervention.

When you use the Automatic Updates service, neglecting to read Microsoft's security bulletins is an easy habit to get into. These bulletins contain valuable information, including best practices that can avoid or mitigate the problem, even without the patch. If you rely on automated systems to patch your servers, you need to watch the security bulletins for manual fixes.

Finally, the Automatic Updates service can't always handle the complexities of a hotfix. Some hotfixes have specific installation conditions that you must carefully review before installation. If you consider these drawbacks, you'll see that the Automatic Updates service is actually a distribution method and not a replacement for checking security bulletins.

Double-Check the Hotfix Checkers
Even when you use best practices to patch your systems, you might still need to reinstall a hotfix. For example, you might install all the Automatic Updates, then go to Windows Update only to find other fixes available. After installing all those other fixes, you might use the HFNetChk tool and discover even more fixes that you need to install. Taken a step further, you might run qfecheck.exe, which reports whether all hotfixes are installed properly, then run the Microsoft Baseline Security Analyzer (MBSA) and find that a hotfix isn't properly installed.

Why do these situations occur? Part of the problem is that Microsoft uses many different installers for hotfixes. You might notice, for example, that Microsoft Internet Explorer (IE) updates have different icons and file-naming conventions than Win2K hotfixes do. Some products use the Microsoft installer, Windows installer, or the Microsoft Systems Management Server (SMS) installer, whereas others, such as Microsoft Office, use yet another installer. With the release of SP3, even Win2K hotfixes switched from using the hotfix.exe installer to using the update.exe installer. Predicting how each of the many installers will behave in particular circumstances is difficult.

Installation order, file corruption, and even running the System File Checker can cause problems. (For details about System File Checker­related problems, see the Microsoft article "The SFC /SCANNOW Command May Overwrite Hotfix Files" at http://support.microsoft.com/?kbid=814510.) Other problems, such as service pack slipstreaming, human error, incorrect version detection, and limited product coverage, contribute to the inconsistent results.

Another part of the problem is that numerous tools exist for checking installed hotfixes, and each tool has its strengths and weaknesses. Each tool uses different combinations of methods to determine which hotfixes are installed and whether they're installed properly. To determine whether you've installed a hotfix, a tool can check the registry, the file versions, or the file signatures. But many complexities can complicate file and product versioning, so the process still isn't perfect. For example, slipstreaming a service pack to a server might not record a product version in the same way as if you'd installed it after the OS.

My solution is to check my hotfixes, then double-check them again with another tool. Table 1 provides a list of free products that you can use to verify your hotfixes.

Don't Forget Other Updates
When you install hotfixes, overlooking other updates that you need to apply to your systems is unfortunately easy to do. For example, most hotfix-checking tools tell you which hotfixes are required for your current service pack level but don't always inform you that a new service pack is available. Also, be aware that Windows Update covers only Microsoft's main products. Table 2 lists several updates that administrators can often overlook. Many third-party patch-management solutions offer much more comprehensive product coverage than the Microsoft tools do. Table 3 lists several such third-party patch-management offerings.

Know When to Reinstall
Even if you carefully follow the guidelines in this article, you should always reinstall hotfixes in the following situations:

At some point before Microsoft releases a service pack, the company must freeze the code to start the testing process. During that test period, Microsoft will likely fix bugs in the service pack and release new hotfixes. Microsoft will often rerelease these hotfixes (and refer to them as post–service pack hotfixes) when releasing the service pack. After you install a new service pack, reinstalling any post–service pack hotfixes that you had previously installed is a good idea. I also suggest that you regularly check your systems with an automated patch-management system to verify that all hotfixes are current and properly installed.

Keep Up with Fixes
At one time, when Microsoft recommended against installing hotfixes unless they were absolutely necessary. In fact, common practice was not to install any fix if a server was already running properly. But the proliferation of Internet-based attacks has changed that practice.

Although some exceptions exist, the new best practice is to install every hotfix and service pack relevant to your high-visibility or high-security servers. Microsoft expects users to have installed the most current service pack, and some hotfixes won't work with older service packs. Some hotfixes rely on DLLs released with service packs, and not having the right version can result in server instability. Thus, installing every update requires careful testing before full deployment.

Speed is also a concern when applying hotfixes. With some security problems, patching your servers as soon as you've tested the patch for your environment is crucial. If someone really wants to break into your Web site, all they have to do is closely monitor Microsoft's security bulletins and exploit the vulnerability before you patch your servers. The only sure way to counter such exploits is to patch your servers as quickly as possible. In the meantime, be sure to use other tools such as firewalls, Intrusion Detection Systems (IDSs), and the URLScan security tool to block known attacks.

Know Where to Get Help
Microsoft's hotfix articles used to carry a disclaimer that hotfixes weren't supported products to emphasize that the company typically didn't subject hotfixes to the same extensive testing procedures used for service packs. This situation has changed, and Microsoft now provides free support for hotfix-related problems. A complete list of ways to contact Microsoft Product Support Services (PSS) is available at http://support.microsoft.com/default.aspx?scid=fh;enus;cntactms.

Microsoft also provides support through newsgroups. This support option is often helpful because it not only includes feedback from Microsoft but also from the Windows community. Go to http://www.microsoft.com/technet/newsgroups/nodepages/security.asp for a complete list of security-related newsgroups. Microsoft provides a search mechanism for these groups, but you might have better results using Google's Usenet search engine, which includes Microsoft newsgroups.

Other public forums and mailing lists are available for community support. For questions specific to patch management, check out http://www.patchmanagement.org. The two largest mailing lists for Microsoft-related security concerns are FOCUS-MS at http://www.securityfocus.com/archive/88 and NTBugtraq at http://www.ntbugtraq.com. If you have a bad experience with a hotfix, be sure to post to all these public forums so that everyone can benefit from your experience.

As you can see, patch management still has many unresolved concerns and has some way to go before you can completely trust the automated process. The best way to deal with hotfixes is to keep up with them. You might find that an automated solution works best for your organization, or you might choose to keep your own list of fixes to install and be aware of the problems as they come up. Monitor the public forums for known concerns and properly test the fixes before deployment. If you follow all these tips, you can be sure that your crucial servers are as secure and up-to-date as you can make them.