Automating Hotfixes

Secure your Web server more efficiently

The ideal time for a malicious individual to find out whether your organization is vulnerable to attack is shortly after the announcement of a vulnerability. How long does your organization wait before deploying a hotfix to externally facing servers? Given the onslaught of Web site defacements and IIS-specific attacks such as the CodeRed Worm, the need for timely identification and remediation of vulnerable servers has never been clearer.

I've built a basic Windows Script Host (WSH) application that lets you leverage the Microsoft Network Security Hotfix Checker (hfnetchk.exe) command-line tool to not only identify vulnerable servers in your enterprise but also automatically apply necessary hotfixes. Although the application might benefit some small enterprises, it's really targeted at larger environments with at least five IIS servers in the internal or demilitarized zone (DMZ) networks.

Introducing Hfnetchk
Building upon the power of the WSH-based Hfcheck tool, Microsoft recently introduced the Hfnetchk command-line tool, which is available as a free download from http://www.microsoft.com. For more information, see the Microsoft article "Microsoft Network Security Hotfix Checker (Hfnetchk.exe) Tool Is Available" (http://support.microsoft.com/directory/article.asp?id=kb;en-us;q303215). Hfnetchk downloads a signed compressed cabinet format (.cab) file from Microsoft and unpacks it into an XML file containing a comprehensive list of Microsoft security bulletins and associated patches. Then, Hfnetchk scans target machines in the enterprise network, compares installed files with patches currently available, and identifies which patches you need to apply to each target machine to make it current.

Most other tools check only for the existence of a registry entry to determine whether you've applied a hotfix. Hfnetchk both examines the registry and compares the hotfix file date with the system file date. Also, this new tool lets you scan for Windows 2000, Windows NT, IIS, Microsoft SQL Server 2000, SQL Server 7.0 (including Microsoft Data Engine—MSDE), and Microsoft Internet Explorer (IE).

Although Hfnetchk is powerful, it isn't quite the panacea administrators have been seeking for hotfix management. At the time of writing, Hfnetchk's inability to download the CAB file through a proxy server means that, without modification, the tool just isn't ready for use in most enterprise environments. Additionally, Hfnetchk doesn't let you take any action when it finds a missing hotfix, limiting its usefulness to information gathering without automating remediation. However, by using a little scripting know-how, you can transform Hfnetchk into the basis for a complete hotfix-management solution.

3 Steps to Hotfix Management
A complete enterprise hotfix-management solution must perform three basic actions:

Hfnetchk takes care of querying Microsoft for the list of current patches and identifying which target machines need which updates, but it can't download the list through a proxy server or implement the patches. To remedy both of Hfnetchk's limitations, I've developed a sample application suite around Hfnetchk called the Hotfix Identification and Verification Engine (HIVE). Hive.exe does everything except download the files associated with the hotfixes and determine which hotfixes to apply and which to ignore, but I explain that later. You can download the fully functional application from the Code Library on the Windows Web Solutions Web site (http://www.windowswebsolutions.com).

Query and Identify
To use HIVE to automate hotfix installation and verification, you need to maintain on a central server

To facilitate storing these items, hive.exe creates the following folders on the central server:

As the code at callout A in Listing 1 shows, _hive.vbs (which resides in the Bin folder) first looks for the Machines folder in the Hive folder. Then, in the code at callout B, the script enumerates the list of directories in the Machines folder to provide the machine name argument that query.cmd uses. As the code at callout C shows, if the script finds a lastrun.log file in a machine's folder, it moves the file to the Archive folder and renames it to the current date and time so that you can reference it later if need be.

As Web Listing 1 shows, query.cmd is a simple command-line batch file that executes Hfnetchk, redirects output to machinestargetmachinelastrun.log, and uses createbatchinstall.vbs to create the remediation script. (You can access this listing on the Windows Web Solutions Web site.) If your central server connects to the Internet through a proxy server, you don't need to modify query.cmd. However, if your server can connect to the Internet without a proxy server, you'll want to comment out the first call to Hfnetchk and use the second call instead. The second call omits the -x parameter so that Hfnetchk can download the .cab file automatically from Microsoft. Downloading the .cab file on each execution of the Hfnetchk application simplifies ongoing maintenance of the HIVE tool significantly.

Hfnetchk generates a tab-delimited output file for each target machine that createbatchinstall.vbs can then parse. As Figure 1 shows, Hfnetchk returns the machine name and product affected, the MS bulletin number, and the reasons that the machine was identified as not having the patch.

Implement Hotfixes
After Hfnetchk creates the lastrun.log file in each machinestargetmachine folder, createbatchinstall.vbs, which Listing 2 shows, parses the file to find the bulletin numbers required to update the server. In the code at callout A in Listing 2, the script first checks in the Ignore folder for a subfolder named for the current bulletin number. (In addition to routinely placing bulletin numbers in Ignore, you can use Ignore for bulletins about which Hfnetchk returns incorrect installation status information.) If the bulletin number exists as a subfolder of Ignore, the script performs no further processing for it other than to write an entry in machinestargetmachinelastrun.log stating that the bulletin was skipped. If the bulletin number doesn't exist as a subfolder of Ignore, the script checks whether you placed the bulletin in the Hotfixes folder so that HIVE target machines can access it. If the script doesn't find the bulletin in the Hotfixes folder, it creates a subfolder in the TBE folder to let you know that you need to do further work on this hotfix before a scheduled task on the target machines can install it. Thus, the script addresses each hotfix and lets you decide whether to ignore it, create an installation routine for it, or simply postpone working with it.

In the code at callout B, the script makes an entry in installstargetmachine.cmd pointing to a hiveexec.cmd file for each bulletin that it finds in the hotfixesbulletinnumber folder. Hiveexec.cmd is a command file that an administrator manually creates for use with the HIVE application; Web Listing 2 shows a sample hiveexec.cmd file. The sample command file launches the hotfix installation routine with the -z switch for no reboot, -q for quiet mode, and -m for unattended operation.

If createbatchinstall.vbs needs to install more than one hotfix, the code at callout C in Listing 2 adds Microsoft's Qchain tool at the end of the list of patches to avoid a system reboot between each patch application while maintaining the integrity of each patch. Finally, the code at callout D calls reboot.cmd. Web Listing 3 shows a sample reboot.cmd file. The script ends after creating an installation file for each target machine. I explain more later about scheduling a task on each target server to call that server's installation file.

Installing and Administering the HIVE Central Server
After downloading hive.exe to the central server, simply run it to unpack the files to the Hive folder and various subfolders that hive.exe creates on the server. Unless you want to modify each script, don't change the folder structure or location of files; install the Hive folder intact.

To let HIVE scan its first machine (the central server is a good place to start), simply create a new subfolder in the Machines folder, using the target machine's NetBIOS name for the subfolder name. For example, if the central server has the NetBIOS name London, create a new subfolder in Machines called London.

If the central server can access the Internet directly without going through a proxy server, you can edit query.cmd to remove the -x parameter, as I mentioned earlier. Otherwise, you need to perform the actions I outline in the Web sidebar "Downloading the .cab File Through a Proxy Server," which you can access at http://www.windowswebsolutions.com, InstantDoc ID 24269.

For the installation batch files to work properly on the target machines, you must create a share to point to the Hive folder. Also, be sure to assign NTFS permissions to the files in the Hive share so that all target machines will be able to use a Universal Naming Convention (UNC) path to access the share and its files when the machines run their scheduled HIVE task. Using the Control Panel System applet, create a new system variable called HIVE_UNC on the central server and set it to \HIVE_ServerHIVE_Share. In a moment, I'll instruct you to create the same system variable on all the HIVE target machines. This variable lets each server running HIVE locate the central server.

Now that you've created a folder for the central server and you have the XML list of security bulletins to work with, you can execute _hive.vbs. To do so, open a command prompt, change the directory to the Hive folder, and type

hive.cmd

A second window will pop up to run query.cmd. This script in turn calls Hfnetchk. After Hfnetchk has run, you'll see the logstargetmachine.log, machinestargetmachinelastrun.log, and installstargetmachine.cmd files on your central server.

On the first run, no hotfix installation files will be available, so no significant data will exist in the installstargetmachine.cmd file. Look in the TBE folder. Unless the server is completely up-to-date (check the logs for details), Hfnetchk will have created a folder for each hotfix it identified as missing, using the Microsoft bulletin number as the folder name. Go to the Microsoft Hotfix & Security Bulletin Service Web site at http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/default.asp and read each bulletin. If you don't want to apply a patch, simply move its BulletinNumber subfolder from the TBE folder to the Ignore folder to prevent Hfnetchk from placing it in the TBE folder again. If you do want to apply a patch, download the applicable distribution files and create a hiveexec.cmd batch file to run the hotfix silent installation. Then, move the hotfix's BulletinNumber subfolder from the TBE folder to the Hotfixes folder so that each target machine's automated installation routine can install the hotfix when the routine detects that the hotfix is missing.

After engineering each hotfix and placing it in the correct folder, run hive.cmd again to create an installation script for the central server. Look in the Installs folder, and you'll see that a complete list of hotfixes has been generated for the server in the form of BulletinNumber subfolders. You can then runinstallstargetmachine.cmd interactively or through a task scheduled to run when the update can reboot the central server without significant disruption.

Now that the central server preparations are complete and a basic set of hotfixes is available for installation, you can begin adding target machines that you want to update regularly. To add a target machine, simply add in the Machines folder a new subfolder with the NetBIOS name of the target machine. On the target machine, set a system variable for the HIVE_UNC path so that the installation file will work correctly.

Next, create a scheduled task to run HIVE_uncinstallstargetmachine.cmd at a regular interval (at least weekly). Because most hotfixes require a reboot, schedule the task during a period of nonuse. Alternatively, you can manually run the installation file from the central server to maintain better control over when and how the target machine will be rebooted.

After HIVE is working properly, secure the HIVE share such that only a limited group of users has access to it (and thus to the other Web servers). Failure to secure this folder can open a huge security hole because HIVE performs scheduled tasks that run a script. If you want to maintain IIS servers in a DMZ, you need to place a central HIVE server in the DMZ itself to facilitate automated hotfix management without traversing the inner firewall.

Unless each target server is identical in configuration (e.g., a Web farm), you'll need to monitor the TBE folder and move hotfixes to the Ignore or Hotfixes folders on an ongoing basis. Also, you should regularly examine the log files to make sure all the target machines are participating as expected.

Although HIVE isn't a complete solution, it takes you a step closer to fully automated hotfix management. Using Hfnetchk and a bit of scripting know-how, you can reduce your enterprise's risk and increase your efficiency by automating the tedious process of managing hotfixes across multiple servers.