Move Server Wizard War Stories

Just when you think you understand an aspect of Exchange Server, you have an experience that shows that you haven't found all the dark crevices. I thought I understood the Exchange Move Server Wizard (MSW) pretty well. Unfortunately, when I recently moved the server that hosts my mailbox, I found some interesting black holes.

Compaq has three active Exchange organizations: two production environments that the company is gradually bringing together with MSW and an organization that Compaq consultants use to establish best practices for customer deployments. My mailbox is on a server in the latter organization. The consultants' organization, called Digital Equipment, included 11 sites and 15 servers around the world. The company's plan was to use MSW to move all the servers into a new organization called Compaq. However, the migration team (which included several other consultants and me) learned that MSW has several shortcomings—including the inability to recognize clusters—that can make the migration process difficult. Here's what we found and the steps we had to take to complete the move.

Flaws We Knew About
MSW isn't perfect. My team and I knew about several problems in advance, including the following:

Although I knew about all these quirks up front, the fact that MSW doesn't support Exchange running on a Microsoft Cluster Server (MSCS) came as a big surprise.

Cluster Woes
Clusters offer a higher degree of robustness and resilience than a regular Windows NT server offers. However, clusters require specific hardware and the more expensive Enterprise editions of NT and Exchange Server. You also need to know how to set up and maintain clusters, and you might be frustrated by the lack of support for some third-party products that don't work on a cluster.

Dealing with the system Registry is more complicated on a cluster than on a standalone server because the MSCS software makes sure that any Registry changes are available to both nodes in the cluster. MSW makes many Registry changes when it runs, but it doesn't tell MSCS to synchronize the changes. Therefore, MSW doesn't run on a cluster; if you attempt to run MSW, the utility will exit as soon as it detects that it's running on a cluster.

Exchange Server runs on a cluster as a virtual server. The virtual server has a name and IP address that it uses to connect clients and other servers. The solution to the lack of cluster support for MSW is to move Exchange from the cluster to a standalone server. The only problem with this scenario is that Microsoft doesn't support a mechanism to demote a cluster. All you can do is remove the cluster from the network and reinstall NT and Exchange Server on one node to create a standalone server. We made an offline backup of the Exchange Server Directory Store and IS databases before we shut down the cluster. Then, we restored the databases on the standalone server after we installed NT and its service packs. Note that you need to run Isinteg -patch after such a restore to ensure that Exchange adjusts the IS's globally unique IDs (GUIDs) before you restart the IS service.

Preparing for the Wizard
After we demoted the cluster, we prepared to run MSW. We installed another server as the hub server in the new Compaq organization. This server gave us a point to join during the MSW run and let us begin the process of building the new messaging and directory replication connectors necessary to form the new organization. We also exported the contents of public folders to .pst files and imported them into the hub server. You can move public folders that contain static information well in advance of a server move. However, wait as long as possible to move public folders that change daily, such as a folder that accepts an incoming feed from an Internet mailing list.

A crucial part of the process is preparing users. Many users don't understand the intricacies of MAPI profiles. Compaq's user community is very Exchange-literate, but in other situations, consider using an automated tool (e.g., Automated Profile Management's Profile Maker—http://www.autoprof.com) to recreate profiles without user intervention.

Moving the Exchange server affects most dramatically users who work remotely or across low-bandwidth connections. These users rely on offline stores and the OAB to work without a permanent network connection. You must download .ost files and the OAB (using the Tools, Synchronize option) after the server move is complete. This download can take a long time. For example, even though we deleted a lot of messages from the folders, synchronizing a 70MB offline folder took several hours and downloading a 100,000-entry OAB took nearly 2 hours.

A new offline folder synchronizes only the default set of folders (e.g., Inbox, Outbox). If users want to maintain offline replicas, they need to reset the properties of other folders, including Public Folders Favorites.

Running the Wizard
After this extensive preparation, we confidently began to run MSW on the server. MSW begins by performing several checks to ensure that the server is ready to move and to detect potential problems, such as connectors or public folders that the wizard won't move. The operation went well until MSW began to process the set of custom recipients on the server. We told MSW that we wanted to move all the custom recipients and distribution lists (DLs) that the server hosted—a reasonable request. However, we seriously underestimated how long this operation takes.

MSW first exports the details (e.g., display names, email addresses) of the custom recipients. It then switches directories and imports the custom recipients into the directory that the server will use in the new organization. During the move, two further processing steps take place. First, the wizard updates the email addresses of the custom recipients to reflect the site-addressing defaults of the site that the server is moving to.

Second, the wizard inserts the custom recipients into the directory of the target server. The target server is the server in the site that you are moving the Exchange server to; the target server provides details of the new organization. Writing the custom recipients into the target server's directory ensures that the recipients exist after the move, but the process also means that Exchange must replicate the recipients back to the server being moved after it joins the site. The server being moved joins the site with a skeleton directory. Before the server can become a full member of the new site, Exchange must update the server's directory with details of all the objects the site owns.

Replication of custom recipients doesn't occur if you create a new site when you move a server. Hindsight suggests that the best practice is to move servers that host custom recipients first to form new sites that other servers can join afterward.

Processing 99,000 custom recipients takes a long time and generates a lot of replication activity. We hadn't factored this activity into our plan, and we were unpleasantly surprised to find that MSW took more than 12 hours to complete a job that we estimated would take no more than 3 hours.

An alternative to letting MSW process custom recipients is to use the Microsoft Exchange Administrator program to manually export and import the set of custom recipients. The manual approach works well if your DLs include few or no custom recipients. Exporting custom recipients to a Comma Separated Values (CSV) file and reimporting the recipients discards any links to DLs. (DLs include a set of backward links to the recipients in the list.) Because our server supports some large DLs with hundreds of entries, most of which are custom recipients, the export-import approach wouldn't have worked for us.

Processing the IS databases usually takes the longest time during an MSW run. Our databases were about 4GB and took less than an hour to process—a much shorter time than we'd expected.

Afterthoughts
The distributed multimaster nature of the Exchange Server directory means that moving a server to a new site or organization has several ramifications. You must replicate the details of the new site or server to all the other sites in the organization. These details include information about the mailboxes, DLs, custom recipients, and public folders that it hosts. Replication can take a long time, and the organization isn't stable until replication has finished. Replication traffic increases the strain on Message Transfer Agents (MTAs) and connectors and can slow interpersonal email. Therefore, move servers slowly, and let message queues flush completely before running MSW on the next server.

Be extremely careful about data security during MSW operations. Perform a backup before and after MSW completes on every server that the move affects. Besides affecting the server you're moving, the operation affects every server in the site that the server is moving from and every server in the site the server is moving to. If a problem such as a hardware failure occurs, backups will save you from having to restore several servers back to the last known good configuration.

Forewarned Is Forearmed
Without a doubt, MSW does its job. The wizard moves servers successfully every time. As with most other technology, most MSW problems are a result of inadequate preparation and unrealistic expectations. You must prepare for an MSW run, and you must know what the wizard will and won't do. In particular, you must appreciate how the move affects users and prepare them for it.

Because MSW is a utility and not a product, I can't complain too much that MSW doesn't support clusters. However, I'm surprised that Microsoft offers no documented way to remove a server from a cluster to form a standalone system. Reconfiguration is a fact of life, and I can only hope that moving servers and clusters around in the Windows 2000 (Win2K)/Exchange Platinum envronment will be easier.