Corrupted Items and Mailbox Moves in Exchange 2010

The words "corrupted mailbox data" tend to make Microsoft Exchange administrators break out in a cold sweat. After all, the first thing that those wordsbring to mind is the possible necessity of a database restore. And we all know how much fun those are.

The TechNet Exchange 2010 Server forum  contains multiple discussions aboutencountering bad or corrupted items during mailbox moves and the resulting fuss when administrators see messages such as Error: This mailbox exceeded the maximum number of corrupted items that were specified for this move request. This error appears whenadministrators run the Get-MoveRequestStatistics cmdlet  to check the status of amailbox move that's in progress. The error can also be seen logged as event 1100 from the source MSExchange Mailbox Replication in the Application eventlog.

Just what causes such problematic items? How worried should you be if you encounter this error? And what can you do about the issue? Read on to find out.

Mailbox Replication Service and Bad Items

Conceptually, the error is easy to understand. The Exchange 2010 Mailbox Replication Service (MRS) runs on Client Access servers and is responsible formoving mailbox contents asynchronously. In other words, MRS moves mailboxes behind the scenes so that users can continue to work until their new mailboxesare available and MRS switches the AD points for the user account to point to the new mailbox location. MRS can deal with Exchange Server 2003, ExchangeServer 2007, Exchange 2010, and Exchange Online (part of Microsoft Office 365), so it's a pretty capable component. As we'll discuss later, MRS can alsoplay a role in keeping your Exchange databases in good health.

MRS processes mailboxes according to move requests. A move request describes the target database and other parameters that influence how MRS should dealwith the mailbox. For example, one of the parameters (SuspendWhenReadyToComplete) for a move request can be used to suspend the move after the mailboxcontents are nearly fully copied. This action allows an administrator to review the move report, decide that everything is OK, and then resume the moverequest to allow MRS to flip the Active Directory (AD) settings, and redirect the user to his or her new mailbox before resuming the move. Anotherparameter indicates the permitted bad item limit -- in other words, the number of corrupted items that MRS can tolerate before completing a mailbox moverequest. In this context, "corrupted" means that MRS determines a property or properties of the item to be incorrect, malformed, or in some other form thatis unacceptable to Exchange.

The default value for the bad item limit is 0, meaning that MRS will tolerate no data loss in a mailbox move. This is sometimes appropriate when dataabsolutely must be retained. However, you might find that you can't move mailboxes at all because MRS continues to encounter corrupted data and fail withthe error message mentioned earlier. For this reason, best practice is to set a reasonable limit for bad items. You should also check mailbox-move reportsafter requests complete, to determine whether the limit is in fact reasonable or if valuable data is being lost. I think anything in the 10-to-20 range isreasonable, but that depends on your circumstances and organizational requirements.

If you decide that a mailbox really must be moved no matter how much data is lost en route, you can set a very high bad item limit, which forces MRS tomove the mailbox at all costs. The maximum setting is 2147483647, but 100 is probably a better starting point. Note that after you specify a bad item limitequal to or higher than 51, you also need to pass the AcceptLargeDataLoss switch parameter to tell MRS that it's acceptable to drop all corrupted items.

Positive Intervention Is Required

If a move request is halted because MRS encounters too many bad items, an administrator can intervene to amend the move-request parameters and get thingsmoving again. Use the Set-MoveRequest cmdlet  to amendthe BadItemLimit parameter and increase the tolerance for bad items until MRS can move the mailbox, and then instruct MRS to resumethe move.

For example, suppose that the move for TRedmond's mailbox has stalled because MRS has encountered too many bad items. The first order of business is toview the move report. Use the Exchange Management Console (EMC) to view the properties of the move request, find the items that are causing trouble, andperhaps figure out whether you need to intervene in the mailbox before continuing. See the Microsoft article "View Move Request Properties" fordetails about how to access a mailbox-move report.

Suppose that the original bad item limit for the move request was 10, and MRS encountered more than 10 corruptions as it copied data from the source to thenew target mailbox. We can scan the mailbox move report to discover how far MRS managed to get before reaching this limit. MRS creates checkpoints on afrequent basis so that it can resume a move request without redoing work; MRS reports this checkpoint status in terms of percentage complete. Thispercentage reports the overall progress of the move and takes into account other processing, such as connecting to source and target servers, setting upthe target mailbox, and preparing to copy. The phase between 25 and 90 percent of the overall move is when copying of mailbox data occurs. If MRSencounters a bad item limit, it will happen at some point in this range.

In our example, suppose that the mailbox-move report indicates that MRS reached 80 percent before stopping the move. The last 10 percent or so of a mailboxmove is used to perform a final incremental synchronization of mailbox contents and to switch the mailbox pointers in AD. At 80 percent, there isn't muchmore mailbox data to copy, so there's a fair chance that MRS can finish copying data from the source mailbox if you increase the bad item count to 20. Todo so, you can run the following commands in the Exchange Management Shell (EMS):

Get-MoveRequest -Identity "TRedmond" |Set-MoveRequest -BadItemLimit 20 |Resume-MoveRequest 

The first command retrieves information about the move request. This information is piped to the second command, which increases the bad item limit to 20and pipes the data to the third command. That final command tells MRS to resume working on the move request. If things happen as we expect, MRS willcomplete the move request and all will be well in the world.

Fools Rush In Where MRS Discards Data

Before you increase bad item limits, you must understand that MRS discards every bad item that it meets; it doesn't copy this data from the source to thetarget mailbox. Thus, a limit of 20 bad items means that MRS can discard as many as 20 discrete pieces of data from a user's mailbox while it is en routeto a new home. This data might be terribly unimportant, as in the case of an old calendar meeting request. Then again, it might be a message that containsan Excel attachment describing the current year's budget. Fortunately, in 99.9999 percent of cases, bad items are extremely corrupted and highly unlikelyto be accessible by a client.

Letting MRS dump a lot of bad data as it moves mailboxes might be deemed a very bad thing. However, bad items are corrupted, and it's good to give themailbox the equivalent of a colonic irrigation from time to time, to remove accumulated debris. Indeed, Microsoft moves mailboxes frequently betweendatabases in its Exchange Online cloud service, both to balance workload across available servers and to ensure that mailboxes stay squeaky clean and don'tintroduce corruption that could compromise the databases' service.

Corrupted items arise through many sources. Bugs are the most obvious and can occur on both the server and its clients. Exchange is now a very matureproduct, and its developers have had years to eliminate potential problems that might introduce corruption at the database level. You're likely to see morecorrupted items when you move mailboxes from an Exchange 2003 server than when moving mailboxes from Exchange 2007 or Exchange 2010.

The root cause of item corruption is more likely to be a client problem than a server issue. A profusion of clients can connect to Exchange, using amultitude of connection methods, including add-on software products that integrate with clients such as Microsoft Outlook. Clashes and corruption can occurwhen multiple clients attempt to access the same item concurrently. Given the number of mobile devices that connect to Exchange, it should be no surprisethat corruptions exist, especially in calendar items.

Based on purely anecdotal evidence in online forums, users of Research In Motion (RIM) BlackBerry devices seem to experience more corrupted calendar itemsthan do users who access Exchange only through Outlook or Microsoft ActiveSync devices. A look through calendar items in a mailbox (use the Outlook listview) often identifies problems such as apparently blank items or items with multiple versions (i.e., conflicts). Items such as these can cause MRSproblems and are best removed before mailbox moves are attempted.

The general rule is that the older the item, the more likely it is to cause a problem. If you, like me, have items in your mailbox that go back to 1996, itshould be no surprise that some item properties that Exchange expects a well-behaved MAPI client to populate might not have been written correctly into thedatabase.

Other Bad Item Limits

Mailbox moves aren't the only work that MRS does. Any request that MRS processes can have a defined bad item limit. Think of MRS as a component thatshuttles data around for Exchange, and you can understand why it implements bad item limits in everything it does. Clearly, there's no point in movingcorrupted data from one place to another. It's best to detect and drop the bad stuff.

For example, if you use the New-MailboxImportRequest cmdlet  to importdata from a PST into an Exchange 2010 or archive mailbox, or you use the New-MailboxExportRequest cmdlet  to exportdata from a mailbox to a PST, you'll find that you can specify bad item limits. The same zero default value is used and you'll need to specify theAcceptLargeDataLoss parameter if you decide to use a bad item limit higher than 51. The New-MailboxRestoreRequest cmdlet,  which Microsoft introducedin Exchange 2010 SP1 as the preferred method to restore a soft-deleted or disconnected mailbox, also supports bad item limits. You might need to use thiscmdlet to restore content to a mailbox from a copy in a backup-recovery database.

Importing data from a PST can be tricky. The PST file format was never designed to cater to the many ways that it's used in practice. Older ANSI-formatPSTs are particularly problematic when it comes to item corruptions, and I often need to set high bad item limits (i.e., greater than 50) to successfullyimport data. The problem appears to be less severe with the newer UNICODE-format PSTs created since Outlook 2003 was introduced. Still, I rarely seeimports of large PSTs (i.e., greater than 1GB) without encountering some corruptions.

Rules Can Hiccup

Many users like to create mailbox rules to help automatically process incoming messages. Users like rule processing so much that they might create morethan 32KB of rules in a mailbox. MRS won't process a mailbox that has more than 32KB of rules. The only solution is to specify the IgnoreRuleLimitErrorsparameter when creating the move request or amending an existing move request with the Set-MoveRequest cmdlet. Unfortunately, this parameter allows MRS tomove the mailbox without moving any of the user's rules. The user must then recreate rules when he or she starts to use the new mailbox. Hopefully, thisissue will be addressed in a future version of Exchange.

The Good News

The good news is that a mailbox that has been successfully moved to Exchange 2010 is clean and free of corrupted items -- at least, according to MRS. Howlong the mailbox remains clean is entirely dependent on the software that accesses and updates the mailbox. Software is getting better too, so fewercorrupted items are being created. The combination of better software and MRS sanitization should minimize bad mailbox items in the future. At least,that's the plan!