Managing Security Hotfixes

Learn how to manage the security-hotfix process from notification through deployment

Software bugs are an inherent part of the software development process and plague all vendors. Even with the best developers working on a project, the number of software bugs typically increases as the amount of code increases. As a result, malicious users find many opportunities for exploits in OSs such as Windows.

Staying up-to-date with Windows hotfixes can overwhelm an IT department. Hotfixes come out so often that IT feels like a road-maintenance crew that's constantly filling potholes on a strip of heavily traveled highway. The key to keeping your systems safe is to develop an approach to learning about vulnerabilities as they're discovered, determining whether your installation is at risk, assessing the level of risk, deciding when to deploy the hotfixes, and having a plan for deploying them.

Vulnerability Notifications
Because of the frequency of hotfixes, you need a way to keep up with the latest patches. The mainstream media reports only what it considers to be hot stories—widespread propagation of a malicious new worm, for example. But information that helps you protect your systems doesn't sell newspapers or attract sponsors to TV news programs, and you can't rely on the mainstream media to keep you up-to-date.

So where should you turn? Windows & .NET Magazine's Security Administrator Web site (http://www.secadministrator.com) is a great place to start. In addition to requesting the Discoveries and Hot News sections, you can sign up for the free Security UPDATE email newsletter and free email Security Alerts. Another Web site for learning about security exploits is Russ Cooper's NTBugtraq (http://www.ntbugtraq.com), an open mail list for reporting reproducible vulnerabilities. You can get on the list with your own email address or have bulletins sent to a Microsoft Exchange Server public folder. One of my favorite security vulnerability resource Web sites is the SANS Institute's monthly Windows Security Digest (http://www.sans.org/newlook/digests/ntdigest.htm).

Another good resource is Microsoft's HotFix & Security Bulletin Service (http://www.microsoft.com/technet/security/current.asp). At this Web site, you can search for fixes by product and service pack level. Most Security Bulletins include technical details, a list of FAQs, a supporting Knowledge Base article, and the associated patch's URL. Microsoft has also established the Microsoft Security Response Center (MSRC) and designated an internal group to respond to security-related concerns. (For information about the MSRC, see the sidebar "Microsoft's Security Role.")

Evaluating the Vulnerability
After you learn of a security vulnerability, the first thing you need to do is evaluate the problem. Read the information that accompanies the vulnerability notification, and refer to the Web sites I mentioned to understand what the vulnerability involves. When reviewing a vulnerability, consider what the bug is affecting and in which situations the bug will be a problem for you. I strongly suggest that you also refer to sources such as Microsoft TechNet and the Microsoft Developer Network (MSDN) for background on the components affected.

Then, look at the various pieces of your environment to determine whether the vulnerability affects your systems. What affects one company's system might not affect all. This step ensures that you don't waste any time correcting vulnerabilities that don't threaten you. For example, if Microsoft releases a security patch for Microsoft SQL Server and your company doesn't use SQL Server, you don't need to act.

If the vulnerability affects your systems, you need to know to what extent. Addressing Microsoft IIS flaws is more urgent if you have an Internet-connected server than if you have only an internal IIS server. Determine who might take advantage of the bug and how much damage it could do in the worst case.

Often, administrators justify a decision not to promptly apply a hotfix by thinking, "I know and trust all my users. None of them would take advantage of a vulnerability." True, most users won't go out of their way to wreak havoc in your environment. However, there's always the risk posed by curiosity and intrigue as well as the occasional disgruntled worker. You need to rate how serious you consider the security flaw to be for your company. The more serious the vulnerability, the more attention it demands. For example, if you run SQL Server on a server that's exposed to the Internet, patching a Windows 2000 or Windows NT vulnerability should be a high priority.

To help in evaluating vulnerabilities and determining what steps to take, consider creating a security committee. The committee can consist of as many people as you think is appropriate. Ask members from your UNIX, database, and networking groups and from your internal audit teams to play an active role—their experience and expertise can be an asset in determining risk. The entire committee should meet at least once per month; the Windows staff on the committee should meet on a weekly or biweekly basis.

When vulnerabilities surface, the committee can make decisions about which fixes should be applied and how soon. Use the committee as a corporate tool for oversight while letting the individual groups implement the changes. When concerns arise, use the steps I give in this article as a basis for committee decision-making. When the security committee meets, you should also make a point of discussing whether the security review process is working as well as it could.

Now or Later?
After you determine how the vulnerability affects your company, decide how soon you need to take care of the problem. Microsoft typically posts several Security Bulletins per month for Win2K on its security Web site. But do you really need to reboot all of your company's Win2K servers two or three times every month to address security-related concerns? Rebooting Exchange servers and file servers requires downtime and can inconvenience users.

The bottom line, of course, is that you must protect your servers, so the question isn't whether to apply hotfixes, but when to apply them. Remember, too, that although staying up-to-date is important, you won't get a medal simply for being the first administrator to apply a hotfix. To the contrary, Microsoft sometimes recalls or reissues patches because of implementation problems. So, you need to balance all the factors, then determine when to deploy the fixes. If your risk is great, you should apply the patches as soon as you can schedule the necessary downtime. If you determine that the hotfix isn't urgent, you might decide to wait for Microsoft to issue the next OS service pack. Service packs usually include all security hotfixes that Microsoft has issued since the previous service pack.

Mapping the Course
After setting your timetable, you need to determine how to deploy the fix. Microsoft security patches come in a prepackaged, compressed executable file consisting of hotfix.exe, hotfix.inf, and the replacement files. Hotfix.exe installs the patch, and hotfix.inf contains instructions for modifying certain files and registry settings when you apply the fix. Microsoft releases a hotfix in different versions for Win2K and NT. When hotfix.exe runs, it determines whether the service pack that's currently on the system is older than the hotfix you're applying and whether the language is the same. By performing this check, hotfix.exe can determine whether someone installed the patch as part of a previous service pack.

If your server passes both of these checks, hotfix.exe installs the update. If the service pack is newer than the hotfix you're applying, the install process quits. During the update, hotfix.exe creates a registry subkey in Win2K and NT at HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionHotfixQxxxxxx, where Qxxxxxx refers to the Microsoft article associated with the patch. In Win2K, the hotfix also adds the HKEY_LOCAL_MACHINESOFTWAREMicrosoftUpdatesWindows 2000SPxQxxxxxx subkey, where SPx stands for the service pack designation.

When you download patches, you'll find that different types of patches use different naming formats. Some NT patches have an eight-character name that resembles the name of the vulnerability (e.g., tearfixi for the Teardrop Attack). Other patch names use a form that references the relevant Knowledge Base article and hardware platform (e.g., q123456i). The naming scheme that Win2K patches use is the easiest to understand and conveys quite a bit of information. Win2K patch names consist of the article number, the OS, the service pack level that will incorporate the patch, the hardware platform, and the language (e.g.,q294391_w2k_sp3_x86_en.exe).

Although deploying patches manually is common practice, scripting patch deployment is a better solution. You script patches by using hotfix.exe's -m switch. Figure 1 shows the Win2K hotfix.exe switches; the switches for NT are the same. You can also use the -x switch to extract the files and see exactly which files the hotfix updates.

The problem with applying patches is that you usually have to reboot after every patch you apply. Although some servers boot relatively quickly, others take time because of BIOS hardware checks. Rebooting one server a few times might not be a problem, but if you manage hundreds of servers, rebooting each becomes an exhausting process that involves extended downtime. Microsoft addressed this problem in 2001 with the release of Qchain. This utility lets you install multiple hotfixes without rebooting between installations. For more information about how to use Qchain in a script to deploy multiple hotfixes, see the sidebar "Using Qchain."

Deployment
Although you can deploy hotfixes manually, you probably don't want to if you have a lot of servers. If you're looking for a way to avoid visiting every server, consider using Telnet. Win2K has built-in Telnet services; NT administrators can find Telnet on the Microsoft Services for UNIX (SFU) product. You might also want to look at a system that helps manage the process. Microsoft Systems Management Server (SMS) and St. Bernard Software's UpdateEXPERT are both products that provide a framework for automating deployments to your servers. If you have test servers, apply the patch on them first, then test the patch. Managers have little tolerance for untested patches.

No matter how you deploy your hotfix, make the process as easy as possible. You should coordinate all deployments with your security committee and technical staff. As a part of deployment, you should also prepare an exit strategy for a worst-case scenario and make sure that you have current backups of the servers you're upgrading. And always document every step of every change that you make.

Monitoring
After deploying patches, monitor upgraded servers for unusual behavior that the patch installation might be causing. Monitor Microsoft and third-party sites for patch updates, and watch for reports of unanticipated problems that the patch causes. In some cases, you might find that you need to uninstall the patch.

When you install a patch, the hotfix creates uninstall directories in %systemroot%$NtUninstallQxxxxx$, as Figure 2 shows. You can uninstall most patches from the Control Panel Add/Remove Programs applet in Win2K and NT. You can also uninstall patches from the command line for both Win2K and NT by running hotfix.exe with the -y and -m switches from the patch uninstallation directory. The -y switch performs an uninstall and can be used with the -m switch for an unattended uninstallation. A few patches don't have an uninstall procedure.

Taking the Necessary Steps
Microsoft administrators need to devote more time to security, whether it's for servers or workstations. Start by locating timely, reliable sources of information about vulnerabilities, then use those resources regularly to stay up-to-date. Coordinating a patch rollout can be difficult, so start now to develop a plan to ease the process. Then, when you learn of vulnerabilities that might affect you, simply follow your plan.