Cherish your arbitration mailboxes - just in case!

Arbitration mailboxes made their appearance in Exchange 2010 as a special form of mailbox that is designed to be used by Exchange itself rather than a user. In short, there are times when Exchange needs to stuff data away for one reason or another and it makes sense to use a mailbox for this purpose. After all, mailboxes go in databases and can be protected by high availability, and so on…

The full set of arbitration mailboxes is exposed in all its glory by running the command:

Get-Mailbox –Arbitration | Format-Table Name, DisplayName, Database –AutoSize

You can’t view these mailboxes through the Exchange Administration Center (EAC) as their display is suppressed. Everything you might want to do with an arbitration mailbox has to be done through EMS. The set shown in the screen shot above come from my organization and include a couple of arbitration mailboxes that I created for my own purposes.

By default, the arbitration mailboxes are created automatically in the first mailbox database in an organization and will remain there unless they are moved to another database. The first database might be the best place for these mailboxes.

On the other hand, if the first server that you deploy is under someone’s desk and isn’t well protected in terms of high availability, then there’s a fair chance that it would be a bright idea to move these mailboxes to a more appropriate location. In other words, a database that is well protected through multiple copies and is mounted on a server that is “highly accessible” across the organization. You’ll see what I mean by “highly accessible” in a minute.

Exchange 2010 uses arbitration mailboxes as follows:

Exchange 2013 adds an arbitration mailbox used to hold the files generated for the Offline Address Book (OAB). You can add additional OAB arbitration mailboxes to spread the load of OAB generation and provision to clients across an organization.

And of course, from Exchange 2013 CU11 and Exchange 2016 CU1 on, mailbox anchoring will use arbitration mailboxes if the mailbox belonging to a user who runs an EMS session isn't available.

It’s obvious that if these mailboxes are unavailable for some reason, different parts of Exchange won’t function as well as designed. For example, if a database containing the arbitration mailbox used for OAB generation is offline for some reason, Exchange won’t be able to generate updated OAB files and clients won’t be able to refresh their local copies. If the database holding the arbitration mailbox used for message moderation is offline, messages requiring approval will be delayed. However, it’s true to say that transient interruptions in service that make these mailboxes unavailable will probably not be noticed by users – or even potentially by administrators.

The last type of arbitration mailbox is the one that worries me. The arbitration mailbox named Migration.8f3e7716-2011-43e4-96b1-aba62d229136 is used by the Exchange 2013 mailbox migration service, which runs within the Microsoft.Exchange.ServiceHost.exe process. The migration service is responsible for processing batches of mailbox moves in a more automated framework than creating individual mailbox move requests and submitting them to the Mailbox Replication service (MRS).

Obviously, some method is needed to track the processing performed for migration batches and the migration service used the migration arbitration mailbox for this purpose. The migration service creates a separate record for every mailbox in every batch and then updates those records as MRS processes the individual mailbox moves.

Here’s the nub of the problem: only a single migration arbitration mailbox exists in an Exchange 2013/2016 organization and you cannot create additional arbitration mailboxes to spread the load as you can for the OAB. Thus, the migration arbitration mailbox can be regarded as a potential single point of failure (SPOF) for mailbox moves. If the database containing that mailbox is offline or unavailable for any reason (for instance, the network link to the server currently hosting the active copy of the database is down), then the migration service comes to a complete stop. MRS can do a certain amount of processing behind the scenes to move mailbox content, but migration batches will never complete.

Remember, it’s possible that (in a moment of weakness) you left the migration arbitration mailbox in the database where it was originally created. That’s not such a good tactic unless that database is located on a highly available server that is highly available on an organization-wide basis.

From an engineering perspective, you can argue that a single location to hold information about migration batches is the right approach. For instance, it removes the potential for concurrent multiple attempts to move a mailbox to different databases. It’s also true that some extra engineering effort would be necessary to allow for the load to be spread across multiple arbitration mailboxes (EAC would have to gather data from all mailboxes to present a coherent picture of batch processing across the organization).

Given the use of arbitration mailboxes it makes sense to ensure that they are positioned in highly-available databases to minimize the possibility that they become a SPOF. Time to put the thinking caps on – or, at least, to make sure that you protect and cherish those invisible but all-important arbitration mailboxes.

Follow Tony @12Knocksinna