SQL Server DBAs Deserve an Apology

You were right. I was wrong. There, that wasn't so hard to say! In last week's commentary ("After the Slammer," I said that professional administrators should be ashamed for not applying the patch that blocks the SQL Slammer worm, perhaps the fastest spreading worm in Internet history. But reader comments I received this week made me realize that I owe some professional administrators an apology. My comments grossly oversimplified the problem. The reader letters reminded me that applying the latest security hotfix isn't as easy as running a setup during your morning coffee break. Last week, I painted DBAs as the bad guys. I was off the mark.

It's true that Slammer wouldn't have spread as quickly as it did—if at all—if most SQL Servers would have had the patch installed. (Last week's SQL Server Magazine Instant Poll showed that 17 percent of respondents still hadn't installed SQL Server 2000 Service Pack 3 (SP3) or the Slammer patch even after the worm attack.) However, the patch isn't a one-size-fits-all solution.

The first obstacle many DBAs face when Microsoft releases a patch or update is that such updates are typically difficult and time-consuming to install. One reader recalled, "The original patch that Microsoft released did not include an installer. I remember back when I first downloaded and installed this patch. It involved manually copying a set of files into different folders, and then running one or more scripts (it's been a while, so I don't remember all the details). Microsoft re-released the patch later at some point (I don't know when) with an installer. I found this out when I read the bulletin after the Slammer had hit—they specifically state that they had re-released the patch with an installer... Installing the original patch was a pain in the neck, and I am sure many administrators skipped it just for this reason."

A second obstacle many readers describe is that blindly applying patches and updates can create problems instead of solving them. As one reader noted, "Most business organizations have a total environment to protect, with many applications, servers, and desktops that are [either] interconnected or widely unrelated. Microsoft patches in the past have had negative effects on production systems and need to be thoroughly tested and proven before 'professional' technical support people apply them. The Microsoft development mentality is that any errors will be proven in live situations, so frequently the patch is worse than the disease it is designed to prevent."

A third obstacle that has increased in these times of tight budgets is that many IT groups are understaffed and overworked. One reader provided a particularly dramatic example: "We have a network of hundreds of SQL Servers, shortly to grow to almost 2000. On top of all these SQL Servers, we have a mainframe running DB2, dozens of UNIX systems running UDB and Oracle—and just four DBAs. This is a business decision—the company has decided that it will not invest money in the DBA function [because] it does not see sufficient value. Rather, it prefers to take chances over issues like [the Slammer worm]. My colleagues and I have nothing at all to be ashamed of, and I suspect the same applies to many other DBAs in other companies as well."

Applying patches becomes infinitely more complex in an environment that requires 24 x 7 availability and supports dozens, hundreds, or thousands of servers. So although DBAs know about the risks, compelling technical and business reasons sometimes keep them from applying patches.

I oversimplified the problem when I blamed administrators for the spread of Slammer, and I don't want to oversimplify again this week by letting the DBA community off the hook entirely. However, these responses show that DBAs face serious problems in keeping their systems up-to-date. People are choosing to ignore certain classes of best practices—not because they want to or because they're lazy or indifferent but because they see the medicine as worse than the disease.

Fixing these problems requires a new way of looking at the application and management of best practices. And the solution must include input from the technical community, software vendors, companies that use the products, and the members of overworked IT groups.

Ultimately, software must evolve so that applying a patch or adhering to best practices is easier. I'll expand on this topic in an upcoming edition of SQL Server Magazine UPDATE, but I want more feedback from you first. How can software vendors such as Microsoft engineer software to make it easier to adhere to best practices? Is it their responsibility? Do you know of simple ways to solve the problems that the reader excerpts above highlight? Tell me what you think.