Manage Those Pesky Patches

You have to download a patch”—few sentences evoke a louder groan from IT pros. Patch management can consume an inordinate amount of your already limited time. Often patches seem to creep in from nowhere—everything is going swimmingly until a zero-day security flaw is discovered and publicized, leaving the vendor scrambling to provide a patch and you scrambling to test it before deploying it to your production systems.

The fact is that patching and all patch-management tasks (downloading, integration into master images, testing) are a necessary evil in today’s IT landscape. Although most of you if presented with the question of “What about patch management do you find annoying?” would answer “Everything!”, I’ve dealt with some specific patching annoyances and figured out how to make them less annoying. I hope that after reading my tips you’ll be able to cross patching off your list of IT annoyances.

Prior Patch Preparation
The first patching annoyance that I’ll cover is the simple problem of finding out what patches are available and what issues they address. IT pros the world over are intimately familiar with Microsoft’s usual patch release day—the second Tuesday of every month, better known as Patch Tuesday. But there’s no reason to blindly check Windows Update on Patch Tuesday and install everything offered on a test machine and hopefor the best.

At the Microsoft Web page “Microsoft Technical Security Notifications” (www.microsoft.com/technet/security/bulletin/notify.mspx) you can sign up for the Comprehensive Alerts email notification or the RSS feed (or more likely, you’ll want to sign up for all of the alert services on that page). Doing so provides you with early notification about the number of updates and the severity rating of security updates that Microsoft is planning to release on each Patch Tuesday. These bulletins are released the Thursday before Patch Tuesday.

On Patch Tuesday, you’ll receive another notification that provides further details on the released patches, including how to get more information. Sometimes Microsoft will release an out-of-band update to address an exceptionally dangerous security vulnerability. Notifications of these updates are also included in the alert services offered on the page noted above.

Also be aware of what I like to refer to as Stealth Patch Tuesday. Microsoft sometimes releases nonsecurity updates on Tuesdays other than the second Tuesday of the month. This is why you’ll be working on your computer on, say, the fourth Tuesday of the month and see the “Updates are available” balloon notification pop up.

You should also keep the Microsoft Security Response Center (MSRC) blog (blogs.technet.com/msrc/default.aspx) in your arsenal of Microsoft patch planning. Here, members of the MSRC not only reiterate the information provided by the notification service mentioned above, they also offer additional insight into the security patch release process and address problems that occur after Microsoft releases a patch. If there’s a buzz around a particular Microsoft security patch, be it a stability, deployment, or compatibility issue, you can be sure the MSRC team will address it.

Now that you’re prepared for when Patch Tuesday arrives, how do you know if a security vulnerability is already being exploited in the wild? A quick way to check is to examine the last two FAQ answers under the security bulletin in question.

Let’s take bulletin MS07-051 as an example. The bulletin is located at www.microsoft .com/technet/security/bulletin/ms07-051.mspx and the section we’re interested in is under the Vulnerability Information heading. Expand the section containing the CVE number (in this case it’s “Agent Remote Code Execution Vulnerability - CVE-2007-3040”), then expand the last section containing the FAQs. You’re interested in the answers to the last two questions. Skipping to the FAQ section about possible exploits doesn’t mean that you shouldn’t understand and plan to deploy all relevant patches to bring your systems up-to-date; it simply lets you quickly prioritize your patching schedule to first address those issues which can be exploited and cause you the most pain.

Even though I’m focusing on Microsoft, it’s rare to be in a homogeneous IT environme n t these days. So what about security patches for products not developed by Microsoft? For these you can either look on the vendor’s Web site for a similar security or patch notification service or invest some time daily at Secunia (secunia.com) and SecurityFocus (www.securityfocus.com). Better yet, subscribe to their respective RSS feeds that are relevant to the systems you support.

Your New Best Friend: WindowsUpdate.log
Patch Tuesday has come and gone. You’ve tested a patch and are ready to deploy it into production. In many IT environments these days, you’ll do this using Windows Server Update Services (WSUS). Smaller shops and home office users will likely have the Automatic Updates service turned on.

However, sometimes it might appear that WSUS and Windows Update (including the superset, Microsoft Update) aren’t cooperating with one or more computers. You might find that Windows Update also isn’t much help in providing a solution, offering only a generic error message and a cryptic hexadecimal code. So what should you do?

Take a look at WindowsUpdate.log, located in your Windows installation directory (typically C:WINDOWS). One way to do so is to go to Start, Run and type

%windir%windowsupdate.log

You’ll want to search the file for the words FATAL and WARNING, paying careful attention to the lines that immediately precede the FATAL or WARNING message. You’ll also want to note any error codes provided and search on those codes in your favorite Internet search engine and in the Microsoft Support Knowledge Base. (For more information about WindowsUpdate.log, see the Microsoft article, “How to read the Windowsupdate.log file” at support.microsoft.com/kb/902093.)

Continue to next page.

One major annoyance I ran into recently occurred with a new, out-of-the-box machine running Windows Vista Business. The machine correctly received the Group Policy Object (GPO) containing its update settings, including the correct WSUS server. However, the machine refused to download any updates. Searching the WindowsUpdate.log file, then searching the Microsoft Support Knowledge Base for the error code, I found the solution— stop the Automatic Updates service, empty the SoftwareDistribution directory under the Windows installation directory, then restart the Automatic Updates service.

When a Patch Is Too Big
The problem of a patch being too big drove me up a wall. I had purchased an HP server and was eager to get it up and running to test Microsoft Forefront Client Security. The server hardware was running flawlessly, as was Windows. I had no trouble installing many of the base components required for Forefront Client Security, including WSUS 3.0 and Microsoft SQL Server 2005. Before I installed Forefront Client Security itself, I decided to take one more trip to Microsoft Update to ensure that none of the products I had just installed required any updating.

As it turned out, part of the SQL Server 2005 installation included some Microsoft Visual Studio 2005 components. Visual Studio 2005 SP 1 was available, so I fired off the download and installation. The installation failed. I didn’t bother to note the error code, figuring it was a transient error. It was the end of the day so I decided to shut the server down and try again the next morning.

The definition of insanity is performing the same task repeatedly and expecting different results, yet insanely I repeated the same steps the next morning, in hopes of a different outcome. I got the same results, of course. Thinking it might be a problem with Microsoft Update, I downloaded VS SP1 directly from the Microsoft Download Center, bypassing Microsoft Update entirely.

As you might have guessed, installing VS SP1 by this method produced the same result. Having installed VS SP1 several times previously on machines that were far less “clean” than this one, I was perplexed. I did, however, wisely note the error code I had been repeatedly given: Error 1718. File [.msp filename of SP1] was rejected by digital signature policy.

The error code was even more perplexing as I was certain this machine had no policy on it that would prevent this installation. Furthermore, I couldn’t imagine that this file wasn’t correctly signed by Microsoft. Off to the Microsoft Knowledge Base I went, where I found the Microsoft article (support .microsoft.com/kb/925336/en-us) detailing my exact problem and offering a link to the hotfix that corrects it. In short, Windows Installer attempts to load the entire .msp package to verify the digital signature but can’t allocate enough contiguous memory and fails with the error above.

Sometimes a transient error isn’t transient at all. When a patch is annoying you, save yourself a heap of time by taking five minutes to research it before you wildly try to fix the problem.

Update Déjà Vu
I’ve seen another problem occur numerous times, and it’s still as annoying as it was the first time I encountered it. Here’s what happens: You download updates via Windows Update. You restart with a heavy sigh, then return to Windows Update only to find the exact same update offered to you again. Thinking the update didn’t install properly, you install it again and restart. Windows Update continues to offer you the same update. What gives?

In some cases, the update truly isn’t installed. Verify that the update is indeed installed by looking in your update history at the Windows Update site. The update status column should read “successful.”

In other cases, the detection logic that Microsoft provided for this particular update has a problem that is causing Windows Update to think that you haven’t installed the update when you already have. Although you can’t fix this yourself—Microsoft has to correct the detection logic—the update was successfully installed and is working, but Windows Update doesn’t realize it. The MSRC blog is a good first stop if you think you’re having this problem, as members of the MSRC team will almost always drop a note indicating that the detection logic needs to be changed.

Computers Missing in Action
The problem of computers missing in action is only slightly less annoying than the repetition of updates. You’re missing one or more computers in your WSUS console. You know the settings have been correctly deployed with Group Policy as some computers are showing up, but some never appear no matter what you try.

Computers can go missing if a Windows installation wasn’t prepared for image duplication by using Sysprep. Subsequent images therefore have a duplicate SusClientID in the registry. It’s unlikely that you’ll run into this problem on a regular basis and the fix for it is relatively easy. First, stop the Automatic Updates service on the computer experiencing the problem. Then, start the registry editor and navigate to the HKEY_ LOCAL_MACHINESOFTWAREMicrosoft WindowsCurrentVersionWindowsUpdate subkey and delete the PingID, AccountDomainSid and SusClientId entries. Restart the Automatic Updates service, then run the following from a command prompt:

wuauclt.exe /resetauthorization /detectnow

You can also use the /detectnow switch (without /resetauthorization) if you’ve recently approved updates on your WSUS server and want to force a machine to run a detection cycle prior to the next scheduled cycle.

The Long Haul
No piece of software is perfect. Despite the annoyances they cause, patches are here to stay. The trick to managing them is to be informed, stay vigilant, and get up-to-date!