Lessons for an AD–to–Windows 2003 Upgrade

Sometimes a product realizes its largest benefits through its simplest improvements. Windows Server 2003 is like that: The OS improvements over Windows 2000 Server are significant enough to justify your time and attention from planning through deployment. Particularly in organizations with a large Active Directory (AD) forest, small improvements to the corporate AD can have broad impact across a global organization. However, there's no question that upgrading your AD infrastructure to Windows 2003 is a major undertaking, regardless of the benefits such an upgrade confers. Before you dive into it, you'll likely need to prove to management why the effort will be worth the cost. Fortunately, Windows 2003 makes that easy, and I can show you why and how. Then, once you've made—and won—your business case and are full speed ahead on planning the upgrade, you'll need to look at some important considerations that aren't covered in the Microsoft documentation.

Proving the Upgrade Case
There's little point to an upgrade program if it doesn't provide a solid return on the effort that performing the upgrade requires. When you're sifting through all of Windows 2003's improvements and new features, focus on what gives you the biggest ROI. Windows 2003 shouldn't be a hard sell because several of its features and improvements can pay for an upgrade by themselves. In my experience, though, these might not be the improvements you'd identify at first glance as being the most valuable. And keep in mind that the full cost-saving effects of some improvements might not be obvious until you're well into the upgrade process. Here are a few of the more important Windows 2003 upgrade business drivers.

Single instance store. The feature that might have the most immediate impact on a large domain controller (DC) infrastructure isn't listed in any features guide or on-server help and is mentioned only in passing in the Microsoft article "How to upgrade Windows 2000 domain controllers to Windows Server 2003" (http://support.microsoft.com/?kbid=325379). This feature is the single instance store, and it improves the way in which AD applies inherited permissions to objects.

In Win2K, when you add an access control entry (ACE) to a container and specify that the ACE apply to that container object and all its child objects, AD adds the ACE to every object. Because each ACE addition causes the security descriptor of every object to grow a little, the cumulative effect is that Ntds.dit (the actual AD database file) grows. In the worst case, one or more container-inheritable ACEs are applied to the very top of the AD logical hierarchy (i.e., the "domain heads") and inherited by all child containers and objects. Here's a simple example: Take the group that contains the user accounts for a domain's administrators and, instead of adding it to the Domain Admins group, explicitly grant it control over DC=yourcompany,DC=com and specify that the new permissions apply to this object and all child objects. The result? The storage requirement for every object in DC=yourcompany,DC=com increases a little, and these increases aggregate, causing Ntds.dit to grow.

In Windows 2003, the single instance store reviews the permissions for existing objects in AD and applies a more efficient reference mechanism to them. The result is an increase in free space within Ntds.dit. In fact, as the single instance store is implemented in the background when an upgraded Win2K DC first starts up the Windows 2003 OS, the active portion of the database can shrink by as much as 40 percent. (Take note that you must perform an offline defragmentation of the AD database to realize the increased free disk space that the single instance store makes possible. For more information about performing such a defragmentation, see the Microsoft article "Performing Offline Defragmentation of the Active Directory Database," at http://support.microsoft.com/?kbid=
232122). You won't realize the business benefit of a smaller Ntds.dit if the file isn't crowding out the free space on your hard disk. But if Windows 2003 reduces the size of the database file (the DC automatically optimizes the size on a daily basis), the smaller, more efficient database rows within Ntds.dit mean the database operates more efficiently, with less data to load into memory. This increased efficiency means you have more disk space without having to purchase more disk space.

Performance improvements. Changes in Windows 2003's memory management and addressing can trigger significant improvements in a DC's performance. A DC is essentially a database server, and a great way to improve performance in a database is to throw more memory at it. The Windows 2003 Local Security Authority Subsystem Service (LSASS) process (which controls the AD database process) is tuned to maximize the database cache; whatever virtual memory is free on the system is used for AD. If you have any Windows 2003 DCs, take a look at the LSASS process—you'll be astounded at its working set size.

This change in memory management fits perfectly with architectural changes in the OS. Windows 2003 is the first Windows OS to take advantage of a 64-bit instruction set and memory addressing. The combination of lifting constraints in the database cache with an essentially limitless memory address space means that AD databases that are too large for the 32-bit memory model can be held entirely in memory if you use a processor that supports 64-bit addressing. AD operations can be performed without paging out to the far slower disk subsystem. The business benefit? Enhanced performance from minimizing disk I/O.

Synchronization improvements. The global catalog (GC) is composed of all objects in the forest but contains only a few of these objects' many attributes. Which attributes go into the GC is determined by membership in what's known as the partial attribute set (PAS). To easily see how the PAS works, launch the Microsoft Management Console (MMC) Schema Management snap-in and, as an example, scroll through the attribute list to the displayName attribute. Look at the attribute's properties, and you'll see that Replicate this attribute to the Global Catalog is checked. In Win2K, changing the membership of the PAS will cause a full synchronization of the GC on every DC in the forest that hosts a GC. If you have a large GC or many of your DCs are GCs, this synchronization can be a significant network and server event that may affect how available the GC is to its users (e.g., Exchange 2000). In contrast, a Windows 2003 GC will replicate only PAS changes to another Windows 2003 GC, which is generally an insignificant event. If replication is a concern in your environment, it's a good idea to place a moratorium on any operation that requires changes to the PAS until after you've upgraded your DC infrastructure to Windows 2003. If you do so, you'll prevent a full synchronization of GCs in your production environment and head off unexpected and potentially disruptive performance degradation.

Dfs improvements. Windows 2003 ushers in some important improvements to Dfs. The first is the ability to host more than one Dfs namespace on a single Dfs server. This can help you save money on hardware but might not have any relationship to your AD upgrade plans.

However, Windows 2003 Dfs also supports full site costing, an intelligent Dfs server selection that essentially tells the system, "If I can't access a Dfs resource in my AD site (or I don't have one), choose a resource in the next closest site as determined by the AD site topology." This is a terrific improvement on the simpler Win2K Dfs locator model, and it's an improvement that doesn't even exist on an XP client's actual DC locator.

To take advantage of Dfs site costing, all the DCs in a domain must be upgraded to Windows 2003. This feature is increasingly popular, and if your AD users see it as a must-have because it helps them construct a more robust and scaleable Dfs namespace, it might become an important driver for you, too.

Security. Windows 2003 is the first major product to come from Redmond since Microsoft increased its focus on security. The OS's security track record is far from perfect but does improve significantly on Win2K. One security improvement that many customers have been waiting for is in the area of constrained delegation, also known in Microsoft documentation as trusted for delegation. Trusted for delegation is a service's ability to impersonate a user or service account to gain the access it needs. This feature is an important facet of multi-tier application design. In Win2K, this delegation is all or nothing—if you trust one account for delegation, all services under the local system account on the target computer are trusted for delegation. If a malware attack compromises the system that's trusted for delegation, a malware service can take advantage of the trust to increase its effect. Constrained delegation is the Windows 2003 improvement to this feature in Win2K, and it lets you specify which services an account can be delegated to. Constrained delegation is a huge security improvement, and one that many developers have been waiting for.

Upgrade or Rebuild?
When you're planning your Windows 2003 migration, one of the key decisions you must make is whether to upgrade your DCs from Win2K to Windows 2003 or rebuild them. Each option has advantages and drawbacks. Rebuilding the DCs ensures that you don't carry forward configuration problems or other errors from Win2K—you're starting with a clean slate. Another advantage of a fresh build over an upgrade is that the server security configuration will be tighter.

Unfortunately, Microsoft won't reveal what the security differences are between an upgrade and a fresh build. You must test your own configurations to see what works and what breaks. I've found that some applications work after an upgrade but break if installed on a fresh build, and vice versa. A sound practice is to run winnt32/checkupgradeonly from the upgrade media to test for application compatibility, but doing so won't catch the small security differences between the OSs that can make or break your application. Another disadvantage is that a fresh build of a DC is a lengthy process compared to an upgrade and requires a fair amount of operator intervention. Even with an automated build, you must demote existing DCs/GCs, rebuild them from scratch, then promote them. Any operator intervention in the automated build process, which you most likely must execute during off hours and in at least some remote locations, significantly increases the process's cost and complexity.

In contrast, an upgrade is a straightforward process. You uninstall applications that aren't Windows 2003—compatible (such as the Windows 2000 Server Resource Kit and support tools), upgrade applications such as backup or antivirus software, and use the improved unattended setup tools for Windows 2003 to perform an automated upgrade. You can prestage the upgrade to the server and execute it remotely. If all goes well, no operator intervention is required. You can monitor the upgrade by remotely tracking the upgrade progress with a tool such as Sysinternals' freeware tool PsList (available from http://www.sysinternals.com).

The drawbacks to upgrading are, of course, the opposite of the fresh build's advantages: You'll carry any problems or misconfigurations forward, and the security configuration is not as tight as a rebuild would provide. (For example, services on an upgraded DC continue to run in the older LocalSystem context, whereas a fresh built server will run services in the more specific Local Service or Network Service contexts.) However, many administrators opt for the upgrade method because of its simplicity and therefore lower cost.

FRS Replication
Before you begin the DC upgrade process, make sure your AD replication is healthy. Log on to a Windows 2003 server in the forest with an administrative account and the Windows 2003 support tools installed and run repadmin /replsummary /errorsonly to get a concise summary of AD replication successes and failures.

What won't be obvious is the health of the SYSVOL File Replication Service (FRS) replica set. FRS keeps data synchronized across a collection of network shares. FRS is often used in conjunction with DFS to synchronize network content. FRS's most important function is keeping the contents of the SYSVOL network share on DCs synchronized. Each DC contains its own copy of SYSVOL, which contains Group Policy files and logon scripts, and will not function without it. FRS ensures that GPOs and logon scripts are consistent across all the DCs in a domain.

Considering how important FRS and SYSVOL are to the health of a domain, it's surprising that very few organizations monitor FRS. Several reasons for this lapse are FRS's complexity, its lack of helpful event log messages, and a dearth of tools that monitor FRS. Little has been done in Windows 2003 to address the first two reasons, but Microsoft has taken a big step to correct the lack of FRS tools. The Ultrasound-Monitoring and Troubleshooting Tool for File Replication Service (available as a free download at http://www.microsoft.com/downloads/details.aspx?FamilyID=61acb9b9-c354-4f98-a823-24cc0da73b50&Display

Lang=en) actively monitors all FRS replica sets you designate. The Microsoft Windows File Replication Service Management Pack for MOM (available as a free download at http://www.microsoft.com/downloads/details.aspx?FamilyID=4ddd3477-d903-429e-ae79-41bd69545ef4&DisplayLang=en) monitors and collects information from the Ultrasound database and reports it to the MOM infrastructure (note that you need SQL Server for a large Ultrasound installation). I strongly recommend that you install FRS hotfix 815473, a roll-up of many individual FRS hotfixes, before you upgrade your DCs to Windows 2003. This hotfix contains the latest released build of Ntfrs.exe, the FRS service. (For information about the hotfix and instructions for downloading, see the Microsoft article "File Replication Service Does Not Log Errors on Sharing Violations," at http://support.microsoft.com/?kbid=815473.)

You should also use the Group Policy Management Console (GPMC) with Service Pack 1 (SP1—available for download at http://www.microsoft.com/downloads/details.aspx?FamilyID=0a6d4c24-8cbd-4b35-9272-dd3cbfc81887&DisplayLang=en) to examine your default domain and default DC GPOs. In versions of Win2K earlier than SP4, the SYSVOL version of the default domain GPO and its AD version can be radically different, although they should be synchronized with each other. This situation can be caused by a race condition in the processes that synchronize these two components of Group Policy. The condition, which occurs only in environments with large numbers of DCs, exists because by default every DC in a domain tries to synchronize the AD and SYSVOL GPO components all the time. A hotfix, which is now part of Win2K SP4 and is described in the Microsoft article "Password policy is not updated when replication occurs from one domain controller to another domain controller" (available at http://support.microsoft.com/?kbid=326112), limits this synchronization process to only the PDC role holder. This creates a slower but much simpler process.

Application Compatibility
One of the most troublesome yet overlooked aspects of a Windows 2003 AD upgrade is application compatibility. I'm not referring to the compatibility of server applications such as backup and antivirus tools. The areas likely to cause more trouble are older applications that use AD for authentication.

Upgrade documentation describes compatibility problems with Microsoft clients earlier than Win2K, such as Windows 95 and Win98. What the documentation doesn't mention, however, are compatibility problems with non-Microsoft clients that don't understand the revised remote procedure call (RPC) protocol, Server Message Block (SMB) signing, and other security improvements that a Windows 2003 DC uses. The Microsoft Application Compatibility Toolkit helps you analyze applications running on Windows-based clients, but it won't run on non-Windows OSs. (To learn more about the Application Compatibility Toolkit, go to http://www.microsoft.com/windowsserver2003/compatible/appcompat.mspx.)

NAS file servers. For example, consider Network Attached Storage (NAS) file servers. A NAS filer is essentially a large box of disk storage with a small, efficient OS optimized for file services. Very popular in engineering environments that require lots of disk space to store designs, filers might not be able to authenticate their users against a Windows 2003 DC. If you encounter this situation, the filer is useless until you figure out how to point it toward a Win2K DC. To avoid the problem, the filer's administrators must upgrade their systems to the latest (presumably, Windows 2003—compatible) OS release. Surprise! Your upgrade project is at the mercy of a collection of departmental server administrators.

Other applications. If your organization has relied on AD for several years, you can't possibly know every application someone might install that uses AD authentication. You have a couple of choices: You can either send out numerous broadcast messages, hoping to locate all the applications you didn't previously know about, or you can find the applications by breaking them after you upgrade.

Most compatibility problems fall into two categories: Commercial applications that haven't been upgraded for quite a while, and homegrown applications. If you're lucky, your commercial applications won't be cause for concern because it's likely that a Windows 2003—compatible upgrade is available. Homegrown applications, however, are a more difficult problem. Often, the original programmer is no longer around, and the person now maintaining the application might or might not understand programming for authentication. Assuming you've become aware of these applications because you've broken them with your upgrades, assign someone on your team who's skilled in this area to stay on call to quickly resolve Windows 2003—related application authentication problems as they appear.

Here's a good idea: Don't upgrade all the DCs in a domain at the same time, at least not until your deployment is well underway. If an application compatibility problem arises in a domain in which you've upgraded all the DCs, you won't have any Win2K DCs left to point the application to while you help upgrade or correct the application. Once the majority of your DCs are upgraded, you'll have a critical mass of Windows 2003 DCs to let you perform long-awaited AD and GC operations for new Microsoft products without causing wholesale GC refreshes. Your schema extensions and GC changes are then more likely to run without a hitch, and the resulting GC refreshes on remaining Win2K GCs will have a relatively low impact.

Directory Information Tree growth on Win2K DCs. But you're not out of the woods yet. You still need to hold off on performing certain schema operations until you've upgraded all the DCs in your forest to Windows 2003. The AD setup portion of Microsoft Live Communications Server (LCS), for example, adds a number of groups to the ACL of the domain heads of your forest. If your DCs are running Windows 2003, this operation is no big deal. But Win2K DCs lack Windows 2003's single instance store feature, so an ACE for these DCs will be added to the security descriptor of every object in your domain. This can result in a massive increase in the size of Ntds.dit on your Win2K DCs. If those DCs don't have enough extra space, they can fall into a low disk space condition that triggers a cycle of crashing, rebooting, and crashing again. Be sure to test your LCS installation in a test forest that has an AD comparable in size to your production AD.

See the Forest and the Trees
Keep in mind that an AD upgrade isn't instantaneous, and in the time you take to evaluate, plan, and deploy upgrades, your organization's business drivers can have changed. What once were simply "nice-to-have" features in the new AD will suddenly become "must-haves," so it's important to work closely with your business customers.

Pay close attention to application compatibility with non-Microsoft clients, and have contingency plans in place in case you break an application. Be sure both your AD and FRS replication environments are healthy before you embark on your upgrades. And of course, make sure you have the latest Win2K hotfixes on your DCs and the latest Windows 2003 hotfixes incorporated into your upgrade or rebuild process. If you plan to use LCS, thoroughly test the LCS installation in a test forest, paying attention to the groups it adds to the logical AD hierarchy.

The payoff? Upgrading your DC infrastructure to Windows 2003 will be a big enabler for your customers and your IT department. Your customers will be taking advantage of Dfs improvements, constrained delegation, and faster deployment of AD-aware applications that require schema changes. Your IT department will like the DC's performance improvements, reduced security exposure, and more efficient network utilization. And you'll enjoy the benefit of having made the bedrock of your Windows environment—the AD forest—a better place to live.