Solve 4 Common Patch Management Problems

Update management is a task IT professionals approach with the same enthusiasm they usually reserve for a visit to the dentist. Ensuring that computers are up-to-date is tedious, and at the back of every IT professional's mind is the fear that lurking somewhere deep within a newly released patch is code that will cause more problems than it solves. You need to balance thoroughly testing updates before deploying them with the knowledge that code that targets the vulnerabilities these updates address usually appears on the Internet within a week of the update's release. Spend too long pontificating on the adverse impact of applying an update and you'll become vulnerable to the exploits the update protects you against.

In this article, you'll learn about several patch-management–related problems and the steps you can take to mitigate them. The particular annoyances this article covers are as follows:

This article focuses primarily on problems related to managing updates for Microsoft OSs and applications. Managing updates for third-party products without using a tool such as Microsoft System Center Configuration Manager (SCCM) 2007 poses even more challenges.

Determine Which Updates Have Been Deployed

As more computers in organizations become mobile, administrators have more difficulty determining whether a particular update has been deployed on every computer or just on some of them. Back when I worked in first-level support, it was easy to keep track of which updates had been installed because we had to install them manually and would cross computers off a central list as each computer was done. When updates deploy automatically over the network, it's more difficult to track whether updates have deployed successfully, unless you use solutions such as SCCM 2007.

Most organizations use Windows Server Update Services (WSUS) to manage the deployment of OS updates, as well as updates for Microsoft applications. When a configured computer contacts the WSUS server to obtain and download updates, the WSUS server records which updates computers have obtained. Computers can contact the WSUS server according to a schedule or the connection can be initiated manually. The drawback to WSUS is that although it records which updates computers have obtained, it doesn't actually check the client to see if any updates are missing, and it can be hazy on whether the update that was obtained has actually installed correctly.

WSUS knows only about updates it has provided; it has no way to know if an update has been installed in another way. For example, what if a laptop user spends a few weeks away from the office and uses Windows Update through the Internet, rather than WSUS, to keep her computer up-to-date? WSUS doesn't know about these updates because it's aware only of updates it distributes, not updates obtained from other locations. WSUS also knows only about computers that have reported to it. It's possible for WSUS to be completely unaware of computers on your network because, for some reason, those computers have never been able to successfully contact the WSUS server.

There are two free solutions to help with the problem of knowing which updates are installed, so you don't have to manually check each computer to see whether a specific update is installed and you don't have to deploy a solution such as SCCM 2007 that can be a drain on your IT department budget. The first free tool you can use to check computers for missing updates is the Microsoft Baseline Security Analyzer (MBSA), which you can download from Microsoft's website. The latest version, MBSA 2.1.1, supports scanning Windows 7 and Windows Server 2008 R2 computers. You can use MBSA to check for all missing updates from a list published by Microsoft or against the list of updates that you’ve approved on a WSUS server. It would be nice if the MBSA tool's functionality were included with WSUS so that update deployment and checking could occur from a single console, but Microsoft currently has no plans to combine these two products.

Instead of the MBSA tool, you can use Get-Hotfix, a new cmdlet included with Windows PowerShell 2.0. Get-Hotfix lets administrators query computers locally or remotely to determine which hotfixes are installed. Using PowerShell scripting, you can query a list of computers to determine whether an update or a specific list of updates is missing. For example, the following script scans all computers listed in the file computers.txt and adds the names of computers missing the hotfix indicated by the identifier KB974332 to a text file named Missing-KB974332.txt:

get-content computers.txt | foreach
  \{ if (!(get-hotfix -id KB974332
  -computername $_ -ea 0))
  \{ add-content $_
  -path Missing-KB974332.txt \}\}\]

Although the code appears on multiple lines here, you would enter it all on one line. The bit of code -ea 0 sets the error action to silent, preventing the command from producing error text during execution.

Prevent Update Traffic from Saturating WAN Links

Organizations deploy solutions such as WSUS not only to centralize the deployment of updates but also to minimize the amount of update traffic downloaded from the Internet. Rather than 1,000 computers downloading a 100MB update, one WSUS server can download the update and then distribute it to all computers on the network. This process works fine until you take into account branch offices and saturated WAN links. Just as you generally don't want to have all your computers downloading the same large update from the Internet, you don't want to have 100 computers at a branch office all downloading the same update from the head office WSUS server across a low-bandwidth WAN link.

You can configure WSUS so that it hosts only the update approval list and not the update files, forcing WSUS clients to download update files from the Internet, but this means that all WSUS clients, not just those in branch offices, source their update files from Microsoft's update servers. In many organizations, the solution to this problem has been to configure separate WSUS servers at each branch office location, with branch office clients obtaining updates from their local WSUS server. However, adding WSUS servers adds to administrative overhead. It's possible to configure WSUS servers in an upstream/downstream relationship so that updates approved on one server are automatically approved on another, but every server you add to your infrastructure will increase costs in some way.

The solution for branch office computers is to leverage a new technology called BranchCache in conjunction with Windows' existing Background Intelligent Transfer Service (BITS) peer caching functionality. BranchCache is a new feature for computers running Windows Server 2008 R2 and Windows 7 (Enterprise or Ultimate editions). BranchCache lets clients at branch offices share content automatically with each other when they obtain that content from an appropriately configured remote server. BITS peer caching is an existing Windows networking technology that can work in concert with BranchCache to make update transfer across the network more efficient.

You configure BITS and BranchCache through Group Policy. BITS and BranchCache policies are found in the Computer ConfigurationAdministrative TemplatesNetwork node of a Group Policy Object. You can leverage BranchCache with WSUS only if the WSUS role is installed on a computer running Windows Server 2008 R2 and the client computers are running Windows 7 Enterprise or Ultimate edition.

The advantage of using BranchCache and BITS with WSUS is that organizations can use a single WSUS server to deploy updates to head office and branch office networks without saturating branch office WAN links with update traffic. Updates are retrieved across the link by one branch office client, then shared with the other clients at that location. This has the advantage of a local branch office WSUS server without the additional administrative overhead. You can find out more about BranchCache in the Microsoft article "Server Configuration."

Prevent Updates from Interrupting End Users

When it comes to scheduling the deployment of updates, you want to avoid the scenario in which a user who has a document open leaves his computer for a short amount of time and returns to find the computer has restarted itself due to the installation of an automatic update. Users generally want their computers to restart only if they initiate the restart themselves. They really dislike computers that, from their perspective, seem to require restarts on an arbitrary basis.

I once had to spend several hours updating a manager's computer because a previous administrator had allowed the manager to choose whether to accept or reject updates after the manager had lost several hours work due to an unexpected restart caused by the installation of an automatically scheduled update. Needless to say, the manager had declined all future updates, so the computer was several service packs behind where it should have been at that point in time!

Some of the randomness of update installation can be mitigated through configuring update-related Group Policies. The Enabling Windows Update Power Management to automatically wake up the system to install scheduled updates policy, in conjunction with the Configure Automatic Updates policy, lets administrators configure computers to wake from hibernation at a preconfigured time for the installation of updates. This system lets computers wake themselves for the purposes of update installation at 3:00 a.m., for example, when no sane user should have a document open. This method requires that the computer has a BIOS that supports waking from hibernation.

If you choose this method, you should also configure policy so that the default shutdown action is to hibernate the computer rather than to power off the computer. You can accomplish this by blocking end users from being able to power off the computer, then configuring the power settings policy to automatically hibernate the computer after a reasonable period of inactivity.

Test Updates

Every administrator fears applying the update that breaks something. An update that causes so many problems that it necessitates a complete OS reinstall are rare. Today, most updates that fail don't do so in a spectacular and obvious manner. Failures, when they occur, are subtle. Administrators are unlikely to find a fault soon after installing the update on a test computer. People who use the OS or applications in day-to-day situations are more likely to find faults than those who only have a passing familiarity with them.

This situation makes it difficult for administrators to know whether deploying an update will cause a problem. Just because a problem isn't immediately obvious doesn't mean it isn't serious. Vendors have released updates that caused data corruption that wasn't apparent to administrators through typical testing, but end users discovered the trouble two days after the update was rolled out to every computer in the organization.

Administrators need a way for typical end users to test updates without deploying the update to every user in the organization. One solution is to have a group of users that function as update testers and to deploy the update to these testers a week before generally deploying the update more widely across the organization. In theory, the testers will encounter problems before the update is introduced to everyone. Inconveniencing one or two testers is less problematic than inconveniencing everyone. If testers can't find a problem in a week's worth of typical computer use, any problems that the update causes probably won't be serious.

The main difficulty in recruiting testers is that testers need only one bad experience and they might be unwilling to test anymore: Someone who loses a day's work is less likely to volunteer to be a guinea pig in future. Users in the IT department don't make good testers because they rarely use applications in the same way that other employees in the organization do. When assembling a test group, you might need to find a way to reward the users, which probably requires the support of your management. Explain to management why you need a group of reliable testers but that also, from time to time, these testers will lose working time because of something unforeseen happening because of updates on their computers. Better, though, that a small number of users lose time than the whole organization suffers downtime because an update that causes a problem gets deployed to everyone without undergoing any local testing.

Acceptance Is the Key

The main way to reduce the annoyance of the patch-management process is to accept that it will always be necessary and that the best way to deal with it is by being organized. Although patch management will never be something that IT professionals eagerly anticipate, following the advice in this article can reduce these specific annoyances to minor irritations.