Restoring an Exchange 5.5 IS Database

Recently, a user informed me that hundreds of items were missing from the Calendar of an Exchange Server 5.5 mailbox. The mailbox's Information Store (IS) was configured to retain deleted items, but I couldn't find any deleted items to recover. The items didn't appear to be in any offline folders store (.ost) or personal folder store (.pst) files, and the users who had access to the mailbox were sure they hadn't moved the items. Eventually, I discovered that someone had performed a hard delete of the missing Calendar items, which weren't recoverable because Deleted Items Recovery had been enabled only for the Deleted Items folder. (See the Microsoft article "XADM: How to Recover Items That Are Not First Transferred to the Deleted Items Folder" at http:// support.microsoft.com/?kbid=178630 for a discussion of this type of situation.) Consequently, I needed to restore the deleted items from backup.

Our Exchange application runs on a Windows 2000 Server machine with a tape backup drive, and I use VERITAS Software's Backup Exec 8.6 with the Agent for Microsoft Exchange to perform online backups of Exchange to tape. I recently had set up a Windows NT Server test server that I could use as a recovery server but didn't want to rebuild it as a Win2K Server machine (I planned to use it as a hot spare while I rebuilt another NT server); furthermore, it didn't have a tape drive. So I decided to determine whether I could restore the IS over the network to the recovery server—even though that server was running a different OS version and service pack from the Exchange production server—without bringing down the production server. I discovered that such a restoration, although it contradicts the Exchange 5.5 disaster-recovery information I've read, is indeed possible.

I tested the technique only on Exchange 5.5 by using Backup Exec; you should be able to use NTBackup to perform a similar recovery, but I didn't test that scenario. Be aware that Microsoft doesn't support the methodology I used to restore the missing Calendar items. In some cases, my methodology actually contradicts Microsoft's documented recommendations and works around some restrictions inherent to restoring Exchange databases to a different server. Also, restoring to a recovery server with a different OS version and service pack configuration increases the time required to restore the IS. Despite these caveats, I believe that this process can benefit Exchange administrators who don't have a spare recovery server that's running the same OS version and service pack as the Exchange server and a matching tape drive. Of course, restoring from backup should always be a last resort, and both systems must run the same Exchange 5.5 service pack.

Preparation
Before I could use Backup Exec to perform an online Exchange database restore, I needed to license the Agent for Microsoft Exchange by installing its serial number on the Backup Exec server. I also needed to install a remote agent serial number on the Backup Exec server to redirect the restore to a different server and make sure that the remote agent was installed and running on the recovery server. Backup Exec requires both a remote agent and the Microsoft Exchange Agent to restore Exchange data over the network to a different server; both options require separate licenses from VERITAS. To install the remote agent, I ran the setupaa.cmd script (in the winntinstallengi386taa folder on the Backup Exec CD-ROM). To start the service without rebooting, at the command prompt I entered

net start   BackupExecAgentAccelerator

Before I began, I needed to have at least one complete and successful online IS backup to restore to the recovery server. Depending on the backup method you use and the size of the backup, a backup could require more than one tape.

Next, I installed Exchange on the recovery server by running setup.exe from the serversetupi386 folder on the Exchange 5.5 installation CD-ROM. I didn't run Setup /R, which is intended for a full restore on the same server or a new server with the same name. I selected the Complete/Custom option and used the Exchange service account for the existing site to install only the Exchange Server and Administrator options. In addition, I used the production server's organization and site names to create a new site. I didn't select Join an existing site when I installed Exchange because I didn't want the recovery server to replicate data with the production server.

I installed Exchange 5.5 Service Pack 4 (SP4) on the recovery server because the restored database version must match the Exchange version running on the recovery server. I stopped all Exchange services except the System Attendant and Directory (the Directory service, which has a dependency on the System Attendant, must be running for Backup Exec to recognize the recovery server as an available network server for the redirected restore). At this stage, the recovery server was ready to receive the restored data.

Restoring the Data
On the production server, I opened the Backup Exec console and selected the backup set that contained the IS I wanted to restore. I selected the IS and clicked Restore Selections. In the Restore Job Properties dialog box, on the Redirection tab, I selected the Redirect Exchange sets check box, then entered the recovery server's name in the Restore to server text box. On the Exchange v5.5 section of the Exchange tab, I cleared the No loss restore (do not delete existing job logs) check box because you can't perform a no-loss restore when you're redirecting the restore to another server (the job logs are server-specific). I made sure that the Restore public folder and Restore private mailboxes check boxes were selected (otherwise, Backup Exec will display an error message). I then clicked Run Now. Because I chose to redirect the restore to a different server, Backup Exec displayed the message that Figure 1 shows. I clicked OK to start the restore process.

Notice that although I restored the IS data to my recovery server, I didn't restore the Exchange directory. Because an online Exchange 5.5 directory backup contains server-name—specific data attributes, Backup Exec (or NTBackup or any other backup tool that can perform an online Exchange 5.5 backup) requires you to restore the directory to the same server or to a new server with the same name. For this reason, you can't redirect an Exchange directory restore over the network to another server. I got around this restriction by using a manual import-and-export procedure that's sufficient for online recovery purposes and which I describe in a moment.

NTBackup also lets you perform an Exchange restore to a different server (see the Restoring an Information Store to a Different Server subhead under the Restoring a Server section of the Exchange 5.5 documentation). However, when restoring to a different OS version and service pack levels, you'll need to clear the Start Services After Restore check box, otherwise the services won't be able to start until you perform an offline defragmentation of the IS databases.

Starting the IS on the Recovery Server
After I completed the online restore, Exchange populated the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesMSExchangeISRestore In Progress registry subkey with values crucial to the restore process (see the Microsoft article "XADM: How the Restore in Progress Registry Key Works" at http://support.microsoft.com/?kbid=200941 for details about these values). Because I wanted to start the IS on the recovery server without any reference to the production server, I used regedt32.exe to edit the subkey's EDB_RstMap registry value so that all four Exchange database file paths pointed to the recovery server.

I then attempted to start the IS, which failed to start because of a service-specific error. After this failed attempt, I checked the Restore In Progress subkey to verify that the EDB Database Recovered registry value was equal to 1, indicating that all the IS database transaction logs were replayed successfully. I checked the Application event log, which indicated that the OS and service pack versions had changed, as Figure 2 shows, and that I should perform an offline defragmentation to rebuild the index, as Figure 3 shows.

To perform the required offline defragmentation, I stopped all Exchange services, opened a command-prompt window, changed to the exchsrvrbin folder on the recovery server, and executed the following commands, in sequence, to defragment the private IS (priv.edb) and public IS (pub.edb) databases, respectively:

eseutil /d /ispriveseutil /d /ispub

Populating the Exchange Directory on the Recovery Server
After I completed the offline defragmentation, I was able to start the IS and other Exchange services. However, when I opened Microsoft Exchange Administrator on the recovery server, I couldn't see any mailboxes because I hadn't yet restored the Exchange directory.

To work around this problem, I opened Exchange Administrator on the production server and chose Tools, Directory Export. I chose the Recipients container, selected all export object types, and chose Include hidden objects. This action created a comma-separated value (CSV) file that contained a copy of the production server's Exchange directory for the Recipients container. To use this copy of the directory on the recovery server, I needed to replace every occurrence of the production server's name with the recovery server's name. I opened the file in Notepad and used Edit and Replace to replace all occurrences, then saved the modified CSV file to the recovery server.

On the recovery server, I opened Exchange Administrator and chose Tools, Directory Import. I clicked Import File and selected the modified CSV file. After the import finished, I was able to view the objects in the Recipients container on the recovery server. This process might log errors because the copy of the directory you're importing is out of sync with the copy of the restored IS, but the process should be adequate to access individual mailboxes that weren't deleted after the backup.

Retrieving the Data on the Recovery Server
To complete the recovery process, I logged on to the recovery server by using the Exchange service account and created an Outlook profile that opened the mailbox from which I wanted to recover the missing Calendar items. I exported the mailbox to a .pst file and imported the file back into the IS on the production server. I could have used the ExMerge tool to copy items from the recovery server directly to the production server, but because I was dealing with only one mailbox, doing so seemed unnecessary.

Know Your Limits
Because Microsoft doesn't support restoring data to a server that has a different OS version or service pack, don't use this method to restore data directly to a production server. Rather, use this method only to access an IS on a recovery server. This process can be slow, depending on the size of the IS and the speed of your server, network, tape drive, and other hardware. In addition, recovering a deleted mailbox is difficult unless you edit the CSV file and manually insert a line that contains the directory data for the deleted mailbox. And, as mentioned previously, manually exporting the directory from the production server and importing it to the recovery server can be problematic. But in an emergency, this solution is workable for recovering otherwise lost information from an IS online backup. For additional information, see "Recommended Resources."