Moving Mailboxes the Exchange 2010 Way

You probably know that you can't upgrade in-place to Microsoft Exchange Server 2010 from older versions of Exchange, so moving mailboxes is the only way to migrate user data to the new platform. Although the move mailboxes concept isn't new, the implementation in Exchange 2010 incorporates some innovative aspects to ease migrations. This article reviews how the Exchange 2010 move mailbox feature works and shows why the new approach taken by Microsoft makes a real difference.  (See also, "Prioritizing Exchange 2010 mailbox move requests").

Swelling Mailboxes

In all previous versions of Exchange, mailbox moves are executed within the process that initiates the move. In other words, if you initiate a mailbox move with the management console of Exchange 2007 or Exchange 2003, the move continues until the last item has been transferred from the source to the target server. During this process, the administrator can do nothing else with the console and the user is locked out of his or her mailbox because Exchange needs exclusive access to ensure that it can transfer the full mailbox content reliably. This process works and is well understood by administrators, so why change it?

The answer lies in the ever-swelling size of mailboxes. It's OK to disconnect users to move their mailboxes when the standard mailbox quota is 100MB and only a few mailboxes might grow to the giddy heights of 1GB or more. It's quite another matter when the average mailbox quota grows to 1GB and it's common to find mailboxes with quotas of 10GB or more. Indeed, in November 2009, Microsoft announced its intention to provide 25GB mailboxes to subscribers to its hosted version of Exchange ("Global Organizations Choose Microsoft Cloud Applications") in a direct competitive move against Google, who also provides 25GB mailboxes to users of Gmail for business.

It's all very well to assign 25GB of storage to a mailbox as long as that storage never has to move—or if it can be moved without affecting the user. In fact, Microsoft designed Exchange 2010 as a platform to elegantly handle very large mailboxes, and many of the changes developers made to the Exchange Information Store are to reduce the impact on the server of multigigabyte mailboxes that contain tens of thousands of items. Changes have also been made in Microsoft Office Outlook 2007 SP2 and later versions (including Outlook 2010, expected to be released around June 2010) to make sure that Outlook performs much better when asked to access large mailboxes.

The server and clients seem set to support large mailboxes, but as administrators are all too aware, mailbox mobility is a part of regular system management; it's a technique used to move users to new servers or to balance mailbox activity across available servers. In a world of large mailboxes, it simply becomes too slow and administratively unwieldy to disconnect users for long periods to move their mailboxes from server to server. We're moving into a world where the distinction between on-premises and cloud services blurs and companies will expect to be able to move users seamlessly between the two environments. In a nutshell, these are the reasons why the need to disconnect mailboxes before they can be moved has been replaced by a new mechanism involving move requests that are processed by the Mailbox Replication Service (MRS).

Move Requests

A move request states that an administrator wishes to move a specific mailbox from a database to another database. The databases can be on an Exchange 2010 server or on a legacy Exchange 2007 SP2 or Exchange 2003 SP2 server in the same or a different organization. Note that we don't talk about source and target servers anymore. The focus in Exchange 2010 is on databases because databases, not servers, are the management entity in Exchange 2010. As you probably know, Exchange 2010 databases can have multiple copies and can move between Mailbox servers. It doesn't make sense to refer to source and target servers because you don't know which server hosts the currently active copy of the database to which you want to move a mailbox.

You create move requests with Exchange Management Console (EMC) or by using the New-MoveRequest cmdlet with Exchange Management Shell (EMS). In either case, you can select a group of mailboxes and have a separate move request generated for each. You can sort mailboxes by database to make it easy to select a group from a specific database. In Figure 1, you can see I've selected a group of mailboxes to move from one database to another, then clicked New Local Move Request in the Actions pane of EMC, which invokes a wizard that generates the move requests. You can generate a group move like this if the destination database is the same for all mailboxes. If you need to move mailboxes to different databases, you have to go through the wizard separately for each destination to generate the move requests.

A local move request is a move to another database in the same Exchange organization. A remote move request is a move to a database in another Exchange organization, a feature designed to support companies that operate multiple Exchange organizations or who want to migrate from one Exchange organization to another, possibly as a result of a merger or acquisition.

The remaining steps of the wizard ask what to do if corrupt items are found in the source mailboxes. You can skip the entire mailbox and fail the move, or you can let Exchange ignore a certain number of corrupt items. The default option is to skip the mailbox, which seems extreme unless the mailbox is very new. Older versions of Exchange were less exact about item formats, and if you have mailboxes that have been used with Exchange 2003 or Exchange 2000 or have been migrated from a different email system, there's a reasonable chance that some mailboxes will contain a few corrupt items.

If you opt to skip corrupt items and Exchange encounters some during the move, they'll be ignored and you might lose some data, albeit data that can't be read by Exchange 2010. I typically set the value to 5 to let Exchange skip as many as 5 corrupt items during a move. You can check the move logs to find out if any corrupt items were found. Exchange stores details for the last two moves of a mailbox as hidden items in the mailbox's root. As you'll see later, you can access these reports with the Get-MailboxStatistics cmdlet. When everything is ready, click OK and the wizard generates the move requests for each mailbox.

Processing Move Requests

Figure 2 shows the set of move requests generated by the wizard. EMC fetches this information from Active Directory (AD) by searching for all move requests in the domain or whatever recipient scope is set for the recipient configuration section of EMC. Fetching move request data in this manner works OK for small organizations but is likely to be slow in large organizations. However, administrators working in large organizations should probably use EMS to manage move requests because the shell is faster, more flexible, and reveals more information about move requests.

You'll notice in Figure 2 that the move requests have different statuses. A Queued request is one that's waiting for MRS to process it. MRS runs on every Client Access server and regularly polls AD to discover move requests in its site. By default, MRS running on a Client Access server can process up to five mailbox moves concurrently. You can tweak the MRS configuration file to force MRS to process more concurrent moves, but doing so is usually a bad idea because you run the risk of swamping target servers with the workload generated by processing incoming mailbox data. For example, Exchange updates its content indexes as mailbox data arrives on a server so that the full index is available when the mailbox move completes. It's possible there are some edge conditions that create circumstances when the MRS can process data faster, but we don't yet have enough operational experience with MRS in production environments to assess how to tweak it in different conditions.

When a move request is taken off the queue, MRS creates a copy of the mailbox in the target database and begins to transfer items. If the mailbox has an associated archive, the archive is moved too. During this time, move requests have the status Moving. If you select an individual move request and click Properties in the Actions pane, you'll see additional information, such as the size of the mailbox.

After all the data is transferred, MRS pauses to see whether any changes have occurred in the source mailbox since the move started, which is easily done by checking the last update time on the mailbox. If changes have occurred, MRS performs an incremental update to synchronize the two mailbox copies; when this process is complete, it switches the AD pointer for the user's mailbox to the newly moved copy. It's only while the pointer is switched that a user loses mailbox connectivity. Outlook Web App (OWA) users get an error message stating the mailbox is temporarily unavailable. If they wait a few seconds and retry, Exchange will have switched over to the moved mailbox and be able to reconnect them. Desktop Outlook users receive a pop-up message to tell them that they have to exit Outlook completely and restart to connect to their moved mailbox. Microsoft would like to make this process more automatic, but the current Outlook design caches some mailbox information, such as the MAPI identifier, that has to be flushed before Outlook is able to reconnect. The situation persists in Outlook 2010.

When a move is finished, Exchange marks it with a Complete status. If you need to move the mailbox again, you have to delete the move request from EMC by selecting it and clicking Clear Move Request in the Actions pane, as Figure 3 shows. This step isn't intuitive, and Microsoft wants to incorporate some mechanism to clear up completed move requests automatically in future versions of Exchange, but for now you have to do it manually.

Scheduled Moves

EMC doesn't support scheduling move requests to occur in the future. When you create a new move request, EMC places it on the queue and MRS processes it as soon as it's ready. However, there are two ways to create a move request with EMS that allow a certain amount of control over MRS activity: You can use the -Suspend parameter or the -SuspendWhenReadyToComplete parameter. For example, you could use -Suspend as in this command:

New-MoveRequest  -id 'Anil Patel' -Suspend  -TargetDatabase 'IT Department' 

In this instance, the queued request is on record but no data is copied to the target database until you release the move request. You would use -SuspendWhenReadyToComplete similarly:

New-MoveRequest -id 'Anil Patel'  -SuspendWhenReadyToComplete  -TargetDatabase 'IT Department' 

The big difference with this command is that it lets MRS perform the initial mailbox copy. When MRS has copied the mailbox, it puts the move request into a Suspended status and reports that a notional 90 percent of the mailbox has been copied. The last 10 percent is the incremental synchronization that MRS performs when you release the suspended move request.

You might use the first approach to create a large batch of move requests that you can release when system demand is low and the servers that support the target databases can support the additional load created by the moves. You could use the second approach to spread out mailbox moves over the working day and then perform the final switchover to the new mailbox when users are offline. The next time they connect, they access their newly moved mailbox without knowing that a move occurred.

In both cases, the move request is released with the Resume-MoveRequest cmdlet:

Resume-MoveRequest -id 'Anil Patel' 

Knowing What's Happening

You can select a move request in EMC to view its properties to see what's happening as the move progresses, or you can use the Get-MoveRequestStatistics cmdlet to gain a little more insight. For example, to retrieve information about a move request, you pass the identifier for the mailbox and then select the fields that you're interested in. In the following example, I select the mailbox name, current move status, the total size of the items in the mailbox, the count of items, the percentage complete (notional) of the move, the bytes that have been transferred, and the items that have been transferred:

Get-MoveRequestStatistics -id 'Anil Patel' |  Select DisplayName, Status, TotalItemSize,  TotalMailboxItemCount, PercentComplete,  BytesTransferred, ItemsTransferred 

When you run this command, you might notice that there's an apparent discrepancy between the total mailbox size reported and the bytes that are actually transferred during the move. There are a couple of reasons for the difference. Along with the items that you expect to be moved, MRS moves mailbox metadata, such as the items stored in hidden folders that hold user preferences. If the mailbox is located on an Exchange 2010 server, MRS moves the contents of the dumpster (deleted items that are recoverable by users until their retention period expires). This behavior is a change from previous versions in which dumpster contents are ignored during mailbox moves. The change is necessary to support Exchange 2010's discovery search and compliance functionality. It wouldn't be very good if a mailbox move resulted in the deletion of items that might be of interest to a fraud or other type of investigation, so MRS copies the dumpster contents along with the rest of the mailbox, and content indexing captures details of these items so that they are discoverable in the new mailbox. Archive mailboxes are a new feature of Exchange 2010; if a mailbox has an archive, it gets moved too (and clearly, an archive mailbox could be much larger than the primary mailbox and so add to the time required for the overall move to complete).

Experience to date indicates that you can anticipate an Exchange 2010 server to process between 4GB and 6GB of mailbox moves per hour. The exact performance of any configuration will vary depending on the time of day, other system load, and disk performance where content indexing and transaction log generation impose a load on the storage subsystem. In any case, the performance reported by Microsoft and validated in production to date estimates that Exchange 2010 moves mailbox data at least 70 percent faster than Exchange 2007 does.

Move Reports

You can get a detailed rundown on what happens during a mailbox move by generating a complete move report with the Get-MailboxStatistics cmdlet. Here are the commands to generate a full move report:

$MRep = Get-MailboxStatistics -id TR -IncludeMoveReport$MRep.MoveHistory\[0\] |  Export-CSV 'C:TEMPMoveReport.CSV' 

You'll see that we use an object to accept piped output from Get-MailboxStatistics, then select the move history data from the object and export it to a comma-separated value (CSV) file that can be opened with Microsoft Excel or a text editor. Figure 4 shows an edited version of some of the output from this command. You can see how MRS connects to the source and target mailboxes and moves items to seed the target mailbox before pausing to check for any new items that have been created since the move began.

Managing MRS

MRS is the service that moves mailboxes from the source to target databases. Its operation is controlled by a text configuration file called MSExchangeMailboxReplicationService.exe.config, found in the Exchange binaries folder. You can edit the text file with a text editor to alter the operational parameters for MRS. Table 1 shows the MRS parameters and what they control. You have to restart the MRS process to pick up any changes that you make to these properties.

These limits are set on a per-server basis and are regarded as "best effort" limits. In other words, there might be times when the limits are exceeded. For example, two MRS servers could access the same source database to process move requests and together exceed the maximum number of active moves that you have configured the database to support. However, these instances will be exceptions rather than the rule, and normal processing will resume in due course.

Fast and Flexible

Moves Microsoft has revamped mailbox moves extensively in Exchange 2010. The new mechanism is faster and more flexible than ever before and although there are some rough edges (such as lingering completed move requests that clutter up AD), you can expect Microsoft to clean these up in future service packs.