Recovering from an Exchange Server Crash

About 2 weeks ago, I received a call from a client saying that a remote server in San Francisco had Microsoft Exchange Server databases that wouldn't mount. Lately, this particular server has become unstable and freezes every few weeks. The server had frozen again and the administrator rebooted it. Although the server came up, the Exchange private and public databases refused to mount. Usually when both stores refuse to mount on a server, the problem is server related and not necessarily related to the Exchange databases themselves. But, because this was a remote server and I wanted to get it up and running as quickly as possible, I tried to run Eseutil against both the private and public databases. Unfortunately, the databases didn't mount after running Eseutil. This problem occurred when I was speaking at the WinConnections conference in San Diego. I discussed the situation with the client and the company decided to order a new server to replace the server that was unstable. The server hardware was going to take a few days to arrive, so I scheduled a trip to San Francisco immediately following the conference.

Fortunately, this client has several Exchange servers on its WAN. All of the mailboxes were deleted on the San Francisco server by using the Microsoft Management Console (MMC) Active Directory Users and Computers snap-in, and the user's mailboxes were recreated on a local server in Los Angeles. This setup allowed the remote users to at least send and receive new mail until the new server could be installed. Users took a performance hit because their mail resided on a remote server, but slow mail is better than no mail.

When I arrived, the San Francisco server was installed with Windows Server 2003 and Exchange Server 2003. After running a few tests to verify the server was functioning properly, I used Active Directory Users and Computers to move the San Francisco users' mailboxes from the Los Angeles server to the new server. Fortunately these mailboxes were relatively small because the users had been using the new mailboxes on the Los Angeles server for only a week, so even over the WAN the mailbox move took only about 1 hour. Now that the mail was located on the correct local server, I needed to recover all the messages prior to the Exchange server crash. Originally I had planned to use the Recovery Storage Group feature in Exchange 2003 (the original server was running Exchange 2000, which the feature supports), but because the mailboxes had been deleted and recreated on the Los Angeles server, the new mailboxes had new Global Unique Identifiers (GUIDs), and the original GUID assigned to the San Francisco mailboxes prior to the Exchange crash were gone. As you might know, the mailbox GUIDs must be consistent when using a Recovery Storage Group or you'll receive an error when you try to run ExMerge to export the mailbox information to a .pst file. At this point, I had a couple of options: I could try to get the original Exchange 2000 server up and running again, restore the original database, then merge the information; I could purchase an Exchange Recovery Tool from Ontrack (http://www.ontrack.com/powercontrols/) or Quest software (http://wm.quest.com/products/recoverymanagerexchange/); or I could try to somehow get the information out of the old store to merge it with the new mailbox information. After discussing the situation with the client, we decided on the last option.

Knowing that I needed to merge the new and old mailbox information, I used Exmerge to export all the new mailbox information to .pst files. I then created a Recovery Storage Group on the new server and copied the information store databases from the old server, then mounted and dismounted the store (this process proved that the old database files were OK). This ensured that I had a database that was in a consistent state. Using Exchange System Manager (ESM), I opened the properties of the First Storage Group, Private Information Store Database properties and selected the "Database could be overwritten with a Restore" checkbox and dismounted the store. I renamed the original store databases and copied over the old private information store database files (priv1.edb and priv1.stm) from the Recovery Storage Group directory to the live mdbdata directory on the new server. I made sure that the old store database file names were the same as the new database file names (priv1.edb and priv1.stm). Then I mounted the store databases from the old server on the new server and had all the San Francisco users access their mailboxes. At this point, the mailboxes looked empty because they still had the new GUIDs assigned to their mailboxes in Active Directory (AD). I took this approach to create "dummy" mailboxes on the server so I could delete the dummy mailboxes and reconnect the old mailboxes to the corresponding user in AD via ESM. After all the San Francisco users accessed their mailboxes, I went into the ESM and deleted the empty mailboxes, reconnected the users to the old mailboxes, and modified the rights on the mailbox to ensure that the user and mail administrator had full rights to their old mailboxes. Now users had restored mailboxes in a state just before the server crashed. I ran Exmerge and merged all the new mailbox information from the .pst files I had previously exported into the original mailboxes. Now each user had a complete set of information in their mailboxes, both pre- and post-crash. After AD replication completed, I had to refresh the mail profile on all the workstations by deleting the profile and recreating it. After these steps, the users were able to access their mailboxes with a complete information set. I restored the Public folder database from the old server so users could access the Public folders that previously resided on the old server. Fortunately, no additional Public folders were created after the crash, so I didn't need to worry about merging new Public folder information on the new server. I did run into some Public folder rights concerns, and I had to reassign rights to certain Public folders. Consider using the PFDavAdmin utility to reassign Public folder rights if you have problems assigning rights via the ESM. You can download the PFDavAdmin tool at http://www.microsoft.com/downloads/details.aspx?FamilyID=635be792-d8ad-49e3-ada4-e2422c0ab424&DisplayLang=en. Note that this tool isn't officially supported by Microsoft, so use it at your own risk.

The above process allowed me to restore all the users' mailbox information. Fortunately this remote office had a relatively small number of users (20), so it wasn't too much work to recover the information. The users were happy to get back all their old mail and restore the original mail performance now that they were accessing their mail from a local server.

Tip

With Exchange Server 2003 Service Pack 2 (SP2) and Exchange 2003 Standard Edition, you can now have a mail store of 75GB, up from 16GB in earlier versions.