Enterprise Patch Management for Windows

Given the ever-increasing threats from hackers, viruses, and Internet-based worms, patch management has become a crucial component of enterprise security. Patch management is the process of identifying, verifying, downloading, and distributing security updates. Security updates are special hotfixes or software patches that a software publisher releases to address specific security threats. Microsoft has a well-established system for notifying the public about security vulnerabilities and makes patches available at http://www.microsoft.com/technet/security.

Tracking and assessing security threats, then finding and deploying the correct patches for each environment is a constant administrative challenge. Enterprise patch-management software can help streamline patch management, and the growing number of products in this arena is a testament to the need for easier patch management. Although I welcome all the development in this area, the currently available products still have plenty of room for improvement.

My associates and I tested seven patch-management products to determine their suitability for managing a Windows-based enterprise network. These products are not the only patch-management programs available, but they provide a good overview of the field. (For information about a free OS patch-management tool from Microsoft, see "Secure Your Clients with SUS," page 81.) We configured a complete test network (see the sidebar "Setting Up the Test Network," page 46) that reflected many common and some not-so-common configurations that IT departments must work with. We then installed each product to see how it performed.

We began the testing process with the assumption that enterprise patch-management software should meet certain minimum requirements:

In addition to testing for these minimum requirements, we reviewed the following additional features:

Our tests produced no clear winners. No one product works best for all environments. Although some products are clear leaders in the field, each has strengths and weaknesses that might make it appropriate or inappropriate for your network. To determine the products that meet your requirements, you must look at their features. Web Table 1 (http://www.winnetmag.com, InstantDoc ID 40710) lists the products we tested, their features, and a summary of their strengths. Because this technology is rapidly changing, check with the vendors for the most recent product information and updates.

Scanning Flexibility
One significant weakness we found in most of the patch managers is that configuring the system to scan a complex network is difficult. Our test network had a variety of common but sometimes complicated configurations. Most of the products scan a network by using standard Windows protocols and remote registry access to query each system (i.e., they use an agentless system) or by installing on each system an agent that reports to a central station (i.e., they use an agent-based system). Table 1 compares the pros and cons of these two scanning methods.

The agentless products we tested were Ecora Patch Manager, Gravity Storm Software's Service Pack Manager 2000, and Shavlik Technologies' HFNetChkPro. The agent-based products were BigFix Patch Manager, PatchLink Update, and SecurityProfiling's SysUpdate. Only one product, St. Bernard Software's UpdateEXPERT, uses both scanning methods, although many products have future plans for using both methods.

Every product we tested had its quirks, and defining target systems in each product was a tedious and frustrating process. Most of the agentless products offer several methods for adding systems, such as by IP address range, by domain name, by AD OU, or by importing a list of host names in a text file. Ecora Patch Manager and HFNetChkPro provided the most flexibility for adding systems. All agentless products allowed for custom logon credentials for each system or group of systems.

Adding systems to our test network was difficult using any of the agentless products. Identifying all our test systems from different domains and workgroups, often with varying system credentials, was awkward and sometimes tricky. UpdateEXPERT and Ecora Patch Manager had problems seeing all the systems in each domain, HFNetChkPro had problems with conflicting credentials, and Service Pack Manager 2000 wouldn't let me add a particular system because the password was too long to fit into the password field on the credentials screen. All the agentless products had problems adding offline domain members. Perhaps some of these problems will be solved by the time you read this article and many of these problems won't show up on simpler networks, but after struggling with each of the agentless products, I believe that agent-based products might be easier to work with.

The agent-based products sidestepped some of the obstacles of the agentless products, but installing the agents on each system required a significant amount of work. Most of the agent-based products let you push agents to remote clients, but that functionality has the same limitations as agentless product installation: You need remote administrative access to each system. Most of the agent installations prompted for information that would make the agents difficult to mass-deploy using automated methods. For example, UpdateEXPERT requires that you manually enter a serial number when installing the client agent. However, BigFix Patch Manager provides easy installation by building custom client configurations that include everything necessary to connect to the server. One problem that the agent-based products had that agentless products didn't have is that after installation, communication between agents and the central console sometimes broke down.

Patch Detection
After running each patch manager against the network, I was surprised with the inconsistent results. Although we expected some false positives and some false negatives, not one product achieved 100 percent accurate results in every test. HFNetChkPro was the only product that achieved 100 percent accuracy on some tests. But most surprising was that no two products produced the same report and no one product produced the same report twice. Each product had different inaccuracies. To be fair, most of the problems occurred because of the confusing nature of some Microsoft fixes.

Patching Windows is more complicated than most people realize. It's not sufficient to simply replace older files with newer versions; a patch-management system also must take into account what other software is in use and which versions of applications such as Microsoft Internet Explorer (IE), Microsoft Data Access Components (MDAC), and Microsoft XML Core Services (MSXML) are installed to know exactly which file versions to use. To further complicate matters, sometimes a file has a more recent file date than the one installed but an earlier file version. And if you installed Windows with a slipstreamed service pack, you have even more hurdles to surmount.

Service Pack Manager 2000 usually returned the longest list of missing hotfixes, but many of the items were patches that had been superseded or weren't relevant to the current configuration. HFNetChkPro, Ecora Patch Manager, and PatchLink Update consistently produced the most accurate results. The rest of the products had varying levels of accuracy, with an average of 5 to 10 mistakes (i.e., false positives and false negatives) for each system. Since my testing, St. Bernard has added patch-validation support to UpdateExpert 6.1, which should improve the product's accuracy, although I haven't tested it yet.

False positives—incorrectly reporting that a patch is missing—aren't as serious as false negatives, but installing extra hotfixes adds an unnecessary load on the network and might result in file-version conflicts. Nevertheless, I'd rather have a product recommend a fix I don't need than miss one I do need.

False positives can occur for three reasons:

If a product recommends installing a hotfix in any of these scenarios, it's a false positive. HFNetChkPro and SysUpdate reported the fewest false positives.

False negatives—not reporting a missing hotfix—can be serious. When I ran Windows Update on one of my test systems, it overlooked three hotfixes that I needed to install. False negatives are usually the result of not properly detecting installed products. Hotfixes for products such as MSXML, Windows Media Player (WMP), and MDAC threw off many of the patch managers because of the inconsistent ways that these products track product versions. Another reason for false negatives is that the patch manager simply doesn't check for a certain product. You must be familiar with your patch manager's product coverage to ensure that you don't miss important fixes. Because product coverage is constantly changing for all the patch managers, check with the vendor to get the most recent list of supported products. At the time of testing, Ecora Patch Manager, PatchLink Update, and HFNetChkPro had the most comprehensive product coverage.

Patch Deployment
After you determine which patches each system requires, you need an efficient means to deploy the patches. Every product we tested has some method of remote installation, with varying degrees of automation. For example, BigFix Patch Manager lets you automatically install approved patches for existing systems as well as for new systems that you add to the network. Most of the products also have flexible scheduling of patch downloading and installation as well as bandwidth control.

Many hotfixes require a reboot after installation. All the test products give you some form of control over remote rebooting, some more control than others. Ecora Patch Manager provides a snooze feature that lets end users delay reboots after patch installation. SysUpdate provides a similar feature, although you can disable this feature through Group Policy. BigFix Patch Manager has an interesting feature that lists not only systems that need rebooting but also those that require an administrator to log on locally to the system to finish an update operation.

PatchLink Update and BigFix Patch Manager provide the most sophisticated mechanisms for custom patch deployment, including the ability to build custom patches from scratch. This functionality lets you distribute updates for products not supported by the patch manager. UpdateEXPERT also provides some limited capabilities for custom patches.

Patch Information
Microsoft Security Bulletins often address more than just installing patches. Sometimes a Security Bulletin recommends specific best practices for security or recommends manual remediation steps. For example, Microsoft Security Bulletin MS02-064 (Windows 2000 Default Permissions Could Allow Trojan Horse Program) states that the vulnerability requires an administrative procedure rather than a patch; you need to tighten the default permissions on the root directory of the system drive. Win2K Service Pack 4 (SP4) doesn't include a fix, and if you search for Win2K SP4 on Microsoft's Security Bulletin search page (http://www.microsoft.com/technet/security/current.asp), the bulletin doesn't appear. Thus, you might not realize that the problem and fix apply to the SP4 version. Such situations require custom instructions from the patch-management solutions so that administrators can manually address them.

Although most of the products provide at least a summary of each patch and a link to the related Microsoft Security Bulletin, HFNetChkPro provides detailed patch information and detailed threat analysis from both TruSecure and Microsoft. HFNetChkPro also provides cross-references to Common Vulnerabilities and Exposures (CVE—a standardized list of names for known security vulnerabilities and exposures) and BugTraq identifiers.

Product Coverage
The patch-management solutions we evaluated had varying degrees of product coverage: Some cover a broad range of OSs; others concentrate on Microsoft products. Although broad support can be important in many environments, bear in mind that broader isn't always better. If you have a Microsoft-only shop, you might benefit more from a company that has more expertise in patching Microsoft products. PatchLink Update provides the broadest overall product coverage, yet still has respectable coverage of Microsoft products.

Another feature that might be important for your organization is the distribution of non­security-related patches. UpdateEXPERT provides the most coverage of nonsecurity patches, and Service Pack Manager 2000 and PatchLink Update also provide some coverage of these patches. As mentioned earlier, BigFix Patch Manager lets you create custom patches, so you could conceivably use the product to distribute any software or software updates; simply research and add any updates yourself.

If you need support for international languages, be sure to check patch managers carefully before investing in a solution. Few of the products we tested support non-English patches. However, patch manager product coverage continually changes, so these products might add that support at some point.

Application Security
When installing a patch-management solution, the last thing you want is to introduce more security holes to your network. Because a patch manager has a wide reach across an enterprise, it must be secure.

Patch managers must obtain patches and patch information databases from a reliable source and must ensure that they're working with original, unmodified files. Most of the products had strong security features. BigFix Patch Manager uses a public key encryption system for which the company issues its own certificates. Although someone might be able to gain access to the application and view hotfix information, he or she can't take any action without knowing the certificate's credentials. HFNetChkPro uses extra caution by checking not only signatures on the XML files and downloaded patches but also on each of the application's executables.

One problem with agentless patch managers is that Windows doesn't encrypt much of the information queried from network hosts. To properly secure this type of traffic, you need to implement IP Security (IPSec) or some other encryption between the scanning server and network hosts.

Scalability
Scalability is a vital concern for some administrators. The products we tested differed widely in their ability to scale to different environments. When evaluating patch-management solutions, consider the following criteria:

Of all the products tested, BigFix Patch Manager was the most scalable, with PatchLink Update following close behind. BigFix designed Patch Manager with scalability in mind; each console can efficiently handle up to 15,000 clients. BigFix Patch Manager also uses relays to establish multiple patch distribution points across a network. Although the other solutions don't have fixed limits for the number of clients they can support, they're not well suited for handling more than 5000 clients per console; however, you can break up the network into segments and manage each segment with a separate console.

Reporting
BigFix Patch Manager, Service Pack Manager 2000, and SysUpdate had the most flexible and useful reporting options. Most of the others had some reporting features but had limitations on output format, features, or interactivity. BigFix Patch Manager provides a user-friendly Web-based reporting module filled with features such as filtering, custom fields, charting, interactive links, and exporting to Microsoft Excel. Service Pack Manager 2000's template-based reporting provides many of these same features without the Web interface. SysUpdate uses Crystal Decisions' Crystal Reports for its reporting engine, allowing for powerful reporting options if you have access to the Crystal Reports Designer. HFNetChkPro also provided powerful reporting capabilities with flexible report criteria and numerous export formats.

Although not all the products have advanced reporting features, they all provide an export feature so that you can use an external reporting mechanism. And many of the products allow ODBC access to their scan databases, providing further options for custom reporting.

Our lab tests didn't single out one overall winner; some products are simply better suited for certain environments. Consider your requirements for flexibility, accuracy, deployment, product coverage, security, scalability, and reporting and compare them with the feature comparison in Web Table 1. Patch management is an industry still in its infancy, and plenty of room for improvement exists, but we've come a long way from where we were just a few years ago. The number of patch-management solutions is growing, and each solution is growing in features and reliability. The hard part is finding the solution that's right for your environment.