Maintaining Exchange's Information Store

Learn how to tend Exchange's databases and fine-tune your organization's messaging capabilities

Last month I began a two-part introduction to the Microsoft Exchange Information Store. The Information Store comprises the private information database (the private store) and public information database (the public store) in Exchange. I discussed the Information Store databases' internal structure, how they process transactions, and what happens when they experience problems. This month I'll show how you can maintain the Information Store to optimize performance and resilience, two requirements of a database application in a production environment. Specifically, I'll show you how to protect, maintain, compress, and repair the Information Store. Then I'll describe how you can optimize the Information Store's unique dynamic buffer allocation and deleted items recovery features.

Protecting Exchange Databases
Installing the best possible I/O subsystem and then maintaining it properly is fundamental to success with Exchange. The situation is simple: Either you think the Exchange databases are important and take every step to protect them, or you don't. Systems that are well configured and managed experience fewer problems than systems that are poorly configured and managed.

I've never encountered a situation in which a software bug has corrupted the Information Store. I'm not saying the Exchange Server software is perfect. I am saying that hardware problems cause most database corruption. The most common hardware problems causing database corruption are hard disk failures and controller failures. You can take the following actions to prevent hardware problems and safeguard your Exchange databases.

Perform a backup every day. Back up daily, and if possible, make your daily backup a full backup. Full backups will make any restore operations you perform much easier.

Isolate transaction logs from databases. Never keep the Information Store's database transaction logs on the same hard disk that houses the databases. Go further and use separate controllers for the hard disks on which you keep Exchange transaction logs and databases. As I emphasized last month, the Information Store's transaction logs are as important as the databases.

Use RAID data storage. Protect Exchange databases by placing them on a RAID-5 volume with a controller that supports hot-swappable disks, battery backup, and mirroring for write-ahead cache. Safeguard this configuration with a UPS. RAID0+1 is an effective alternative to RAID-5 that delivers better I/O throughput for any given disk configuration. However, there is a cost to using RAID0+1: reduced storage capacity.

Review event logs regularly. Review event logs for database problems every day. Check for disk outages during each daily review.

Practice regular maintenance. Understand and use Exchange's maintenance tools. Schedule time for regular maintenance operations.

Shut down correctly. Never power down a server without first cleanly shutting down the information store service. A graceful shutdown ensures that Exchange can write transactions in the log files to the appropriate .edb file, thereby keeping all databases current. The power-off button is for emergencies only and is not the best or the fastest way to close down a server.

Information Store Online Maintenance
Microsoft designed Exchange Server for continuous operation: You can keep the Information Store running, with users connected as you perform backups and maintenance operations. Exchange's online maintenance feature performs background database housekeeping chores that are vital to the smooth and efficient operation of the Information Store. Microsoft put considerable effort into improving online maintenance in Exchange 5.5. For example, online maintenance in Exchange 5.5 does a better job of defragmenting internal database structures than earlier versions did.

The time at which Exchange performs online maintenance is a server property. Systems administrators usually schedule online maintenance to occur when the load on a server is relatively light, as Screen 1 shows. You can schedule online maintenance so that it's active around the clock, but doing so will tie up some system resources that otherwise could be devoted to users.

The variable and unpredictable nature of messaging transactions in Exchange presents a challenge to online maintenance. Exchange messaging is an unusual database application compared to the uniform transactions of, for example, a traditional financial database. Messages in Exchange come in all shapes and sizes, with and without attachments, and with large and small distribution lists. In addition, when messages arrive in a user's mailbox, they are often quickly deleted. During online maintenance, a number of background threads walk through the leaf nodes of the binary tree within the Information Store, comparing adjacent data pages in order to fill the pages completely. The frequent user deletions that characterize Exchange transactions make filling data pages a difficult task.

To fill data pages, Exchange's online maintenance continually shuffles data from one page to another. To understand how Exchange fills data pages, consider message headers. One message header contains about 400 bytes, and online maintenance can store about 10 headers on one data page. Exchange performs best when it views folders in chronological order, so online maintenance stores message headers close to one another on data pages and in chronological order. As new messages arrive in the database, Exchange inserts the message headers at the end of the binary tree. However, users continually delete items throughout the tree. Over time, user deletions cause data pages to become less full, and if online maintenance lets these deletions continue without adjusting data pages, Exchange performance will steadily degrade. Therefore, online maintenance continually moves headers between adjacent data pages to pack the headers closely together. In addition, if two data pages contain less than the content of a full data page, online maintenance merges them. For example, online maintenance will merge a data page that is 30 percent full with a data page that is 60 percent full to create one data page that is 90 percent full and one data page that is empty. Over time, page merging creates a number of empty pages, known as white space, in the database. White space increases the size of the database, which takes up disk space and slows backups.

Exchange 4.0 and 5.0 perform standard page merging. Exchange 5.5 takes data-page processing further by shuffling individual records from adjacent pages to form data pages that are 100 percent full, which is the most efficient form of storage. Exchange 4.0 and 5.0 can't merge data pages that contain message and attachment content because the data pages' content varies greatly from message to message. Instead, user deletions gradually empty the data pages, which become candidates for reuse only when they are completely empty.

Exchange 5.5 reuses empty data pages more efficiently than earlier versions, and therefore it doesn't expand the size of the database as readily. However, Exchange 5.5 does not arrange pages that contain associated information (for example, grouping all the headers for messages in a large folder) in a contiguous manner, as Exchange 4.0 and 5.0 do.

Currently, no version of Exchange can eliminate white space in the data pages. Microsoft is working to solve this situation, but until the solution appears, you can choose between two methods of managing the size of Exchange databases. The first method is to keep the Information Store on online maintenance around the clock--and therefore allow database white space (and thus the Information Store) to grow gradually. The second method is to take the Information Store offline periodically and run a utility to compress the store (which rebuilds it) and eliminate all white space. Offline compression is the only way you can reduce the size of the Information Store. Compressing the Information Store will create disk space and speed backups.

Compressing the Information Store
Compression is CPU-intensive and makes heavy demands on the disk I/O subsystem. Multiple-CPU systems fare better during compression than other systems fare, but even the fastest system can't process data during compression faster than 2GB per hour. If you have Exchange Server 4.0 or 5.0, consider compressing the Information Store every 3 months. If you have Exchange Server 5.5, you can compress less frequently.

Any reduction in the size of the Information Store by compression will vary from one server to another and depends on how much time has elapsed since the last compression and on the nature of most Exchange user transactions on the server. After you compress the Information Store, the rebuilt Information Store database will usually be smaller than the old one, but in some cases it can be the same size. Typically, a compressed Information Store database is between 25 percent and 35 percent smaller than the original database. However, because Exchange 5.5 reuses deleted data pages more efficiently than Exchange 4.0 or 5.0, a compressed Information Store in Exchange 5.5 will not reduce to the same extent as an Information Store in versions 4.0 and 5.0.

In all versions of Exchange, you'll gain more disk space by compressing the private information store than you will by compressing the public information store, because items are more transient and shorter-lived in the user mailboxes in the private store than the items in the public folders in the public store. The exception to this generalization is a system that directs Network News Transfer Protocol (NNTP) feeds to public folders. The data in NNTP feeds are similar to ordinary email messages and are not retained for long periods.

Compressing with the EDBUTIL and ESEUTIL utilities. You can use the EDBUTIL and ESEUTIL utilities to compress the Information Store. EDBUTIL is a utility in Exchange 4.0 and 5.0, and ESEUTIL is a utility in Exchange 5.5. ESEUTIL is an improved version of EDBUTIL and is multithreaded and able to work on the unlimited store. However, you can't use ESEUTIL for earlier versions of Exchange. Administrators often use the ISINTEG utility with ESEUTIL. ISINTEG checks the integrity of the internal contents of the Information Store, whereas ESEUTIL works with the raw data of the store. You can find information about ISINTEG and ESEUTIL in the Microsoft Knowledge Base (http://www.microsoft.com/kb/support). You can find documentation for ISINTEG and ESEUTIL in the serversupportutils directory on the Exchange CD-ROM.

Both EDBUTIL and ESEUTIL use a temporary database, tempdfrg.edb, during compression of the Information Store. EDBUTIL reads data pages from the source database and writes them into tempdfrg.edb. After EDBUTIL processes all the pages, Exchange renames tempdfrg.edb, which then becomes the new active database. ESEUTIL uses tempdfrg.edb to maintain working information about the Information Store as ESEUTIL compresss and rebuilds the store.

ESEUTIL has to rebuild the Information Store database somewhere; therefore, you must have at least the same amount of disk space available to run ESEUTIL as the original database occupies. For example, if you plan to compress a 1GB Information Store database, you must have at least 1GB of free disk space. You can use the /T switch to rebuild the database off the current disk, if necessary.

Always make a backup before and after you compress the Information Store. You must take a full backup after compression, because when you compress the Information Store, you nullify the Information Store's transaction logs. Therefore, transactions in the logs cannot be read and committed into the new database during compression because the database's internal structures change as they are rebuilt.

To rebuild a database in the Information Store, shut down the appropriate service and invoke EDBUTIL if you're running Exchange 4.0 or 5.0, or invoke ESEUTIL if you're running Exchange 5.5. For example, to compress the private store in Exchange 5.5, invoke ESEUTIL by typing at the command prompt

c:> exchsrvrbineseutil /d /n /ispriv

The /D switch instructs ESEUTIL to rebuild the store, and the /ISPRIV switch identifies the target store.

Specifying /ISPRIV (private store), /ISPUB (public store), or /DS (directory store) forces ESEUTIL to extract the location of the selected database from the system Registry. You can substitute a filename instead of the database specifications (e.g., C:exchsrvrmdbdatapriv.edb). If you do, you must specify the /L qualifier to tell ESEUTIL where the transaction logs are. The /N switch instructs ESEUTIL to output details of what it has done to a file called dfrginfo.txt. If you don't want to rebuild a database (e.g., if you just want to check it for internal errors), you can use the /G switch to scan for logical or physical corruptions.

Repairing a Database in the Information Store
Sometimes hardware corruption can disable a database and force you into repair mode. You can use ESEUTIL to repair a corrupted database in Exchange 5.5. However, before you use ESEUTIL for database repair, try to restore the database with the most recent good backup copy, then roll transactions forward from the transaction logs.

In repair mode, ESEUTIL calculates a checksum for each data page and compares its checksum with the data page's stored checksum. If the checksums match, ESEUTIL accepts the data page and writes it to tempdfrg.edb. ESEUTIL ignores any pages that fail the checksum test. After ESEUTIL has processed all the data pages, a new, clean database is available. However, the data in the pages that failed the checksum comparison are lost. How large an effect this lost data has on users depends on the type of data that is lost. If data pages containing message headers fail the checksum test, the messages on those pages will not be visible to users, and each lost page will affect up to 10 messages. If the lost data pages contain message or attachment content, the messages headers will be visible within folders, but their content will not.

You can often prevent the necessity of making repairs if you regularly check system event logs to pick up warnings that Exchange issues. Investigate event log warnings from the information store service, and look for signs of hardware failure, such as a disk failure in an array.

Should You Enable Write-Back Caching?
Many high-end disks and disk controllers support write-back caching. A write-back cache can increase I/O performance by holding data in memory until the controller can write it to disk. The combination of controller hardware and device drivers disguises the existence of write-back caching from applications. From an application's viewpoint, once the controller signals that data has been committed to disk, the data has been committed to disk. Database applications like Exchange are sensitive to I/O performance, so anything that increases throughput improves performance. However, caching represents a hardware component that must function perfectly to ensure database integrity. Unfortunately, write-back caching, especially when enabled on hard disks, can contribute to database corruption. Most OEMs (e.g., Compaq, Digital Equipment, HP, IBM, and NCR) recognize this weakness and disable write-back caching on hard disks at the factory.

Microsoft advises against the use of write-back caching (the Knowledge Base article Q151789 clearly outlines Microsoft's view). Microsoft's advice is sensible, if you take it in context. You play with fire if you enable write-back caching but then don't protect your system with good battery backup that will ensure data in the cache is always written to disk, even during a power loss. High-end controllers equipped with technologies like Error Correcting Code (ECC), mirroring, and battery backup increase system resilience rather than detract from it. Ideally, battery backup will compensate for power outages that last a couple of days, and the cache and battery need to be mounted on a daughtercard you can remove (with cached data still intact) if the controller fails. Some controllers let you tune the cache for different percentages of read/write activity. If your controller allows this kind of tuning, set the cache for 100 percent writes to gain maximum performance.

If your server has a full-featured controller, you can safely enable write-back caching to take advantage of the extra performance it offers. However, always disable write-back caching on controllers that do not protect the cache against system failure or component breakdown.

Dynamic Buffer Allocation
The Information Store (store.exe) is a multithreaded process. As more users connect to the store, the number of threads grows and memory demands increase. In Exchange 4.0 and 5.0, you can use Performance Wizard to analyze a server's load and then adjust the number of buffers the Information Store uses to manage normal demand. The problem with Performance Wizard is that it facilitates a one-time operation against a static snapshot of system load that is based on historical data. If the load increases or decreases dramatically after you've finished using Performance Wizard, your system must cope with that load. If the load increases, Exchange will make additional memory requests to Windows NT, which results in excessive paging. If the load decreases, the memory that Exchange reserves will not accommodate other applications, which results in increased paging.

Exchange 5.5 has dynamic buffer allocation, a new feature that automatically runs as part of store.exe. Dynamic buffer allocation makes Exchange self-tuning. The Information Store continually measures user load against system load and adjusts the number of buffers it uses accordingly. As the load increases, Exchange requests more buffers from NT. When load diminishes, Exchange releases memory, making it available to other applications.

Servers dedicated to Exchange won't benefit from dynamic buffer allocation as much as will servers that manage multiple high-demand applications. For example, the positive effect of dynamic buffer allocation is more pronounced in small to midsized enterprises in which servers run Exchange, SQL Server, Systems Management Server (SMS), and other applications together. Dynamic buffer allocation will not improve the performance of servers with inadequate memory--such servers will continue to run slowly. However, any attempt to reduce paging will benefit most servers.

Deleted Items Recovery
When users delete items from their mailboxes, Exchange moves the deleted items into the Deleted Items folder. When the retention period for a deleted item elapses, the Information Store's online maintenance removes the item. Exchange immediately removes items from the Information Store that users delete from public folders. In Exchange 4.0 and 5.0, items users delete in error can be reclaimed only after administrators restore the database. As Exchange servers and the size of the Information Store grew, the need for an online restore capability became obvious to Exchange engineers. They provided an answer to the problem of recovering deleted items in Exchange 5.5 with a deleted items recovery feature.

The Exchange 5.5 Information Store includes a new flag (a Messaging API--MAPI--property) that enables soft, or two-phase, deletes. After a soft delete, the deleted item is hidden from user view through a flag setting, but Exchange maintains the item in the store until its retention period elapses. Soft-deleted items are collectively referred to as the deleted items cache. The deleted items cache doesn't exist physically--it's a term that refers to items that are waiting to be removed from the store.

The default item retention period for new or upgraded Exchange 5.5 installations is zero days. Therefore, you must configure both the private and public information stores to activate the deleted items cache on each server within a site. To activate the deleted items cache, you select a store and alter its properties to set an item retention period. Note the Don't permanently delete items until the store has been backed up check box in the user properties window in Screen 2. Selecting this box tells Exchange to keep the contents of the cache until the next backup, even if the backup occurs after the item retention period expires. The logic behind this option is obvious: If an important item is accidentally deleted, you can recover it from the backup tape.

You might decide that the deleted items cache is inappropriate for the servers in your organization (keeping email worries corporate lawyers, who view email as a source of data that potential discovery actions can uncover). Some companies impose strict email retention limits on users and use the Mailbox Cleanup Agent from the Microsoft Exchange Server 4.0 Resource Kit to enforce those limits. But even when mail retention limits are in place, exceptions are possible. Exchange lets administrators set special deleted items retention periods on specific mailboxes. For example, your company might decide that all members of the executive staff need a 30-day rather than a 7-day deleted items retention period. To allow special retention periods, you can set the Deleted items retention time for individual mailboxes on the General tab in the Private Information Store Properties window, as Screen 3 shows.

Recovering deleted items. It's a boon to administrators when users can recover deleted items unaided, but users cannot recover deleted items without the appropriate user interface. Currently, Outlook 8.03, the version distributed with Exchange 5.5, is the only Exchange client that includes deleted item recovery in its user interface. Users with earlier versions of Outlook, Post Office Protocol (POP) 3, Internet Message Access Protocol (IMAP) 4, and Web browser clients cannot recover deleted items without administrator help. To help these users, you can log on to a user's mailbox with Outlook 8.03 to recover items, and this strategy can work for a limited time when you're in the process of upgrading Outlook clients. This strategy is also viable if you have a large population of POP3, IMAP4, or Web clients, but if that's the case, recovering deleted items is probably best handled by the Help desk.

By default, the Outlook 8.03 user interface allows recovery from only the Deleted Items folder, even though the deleted items cache is active for all folders. However, non-MAPI clients can delete items in place, without moving them through the Deleted Items folder. Outlook clients can delete items in place by using Shift+Del. If your system has non-MAPI clients or your Outlook users habitually delete items in place, you might want to enable Outlook to recover items in all folders. You can enable Outlook to recover items from all folders by changing the Registry value DumpsterAlwaysOn to 1 through the Registry setting HKEY_LOCAL_MACHINESOFTWAREMicrosoftExchangeClientOptions. When you change this Registry value, Outlook places items a user recovers from the deleted items cache back into the folder from which the user originally deleted the items. Therefore, Outlook places recovered items from the private store into the Deleted Items folder and places recovered items from the public store into the public folder from which users originally deleted the items. The weakness in this recovery system is that a user might successfully recover an item from the private store only to find that the item is deleted again when the Deleted Items folder empties the next time the user exits Outlook.

Users can recover a deleted item in Outlook by selecting either the Deleted Items folder or the public folder from which they deleted the item, then selecting the Recover Deleted Items option from the Tools menu. The Deleted Items option reveals a list of items in the deleted items cache, as Screen 4, page 167, shows. Users select one or more items from the list and then click the recovery icon to retrieve the items from the cache.

Managing the size of the deleted items cache. The deleted items cache forms part of the public and private stores, and implementing it affects the size of the Information Store database. Exchange doesn't let you limit the cache's size, so it can be difficult to predict how much extra disk space the cache requires. Estimating the size of the deleted items cache depends on factors specific to your installation, such as mail volume and the length of the deleted items retention period. Exchange doesn't count items in the cache against mailbox storage quotas.

An easy way to manage the size of the deleted items cache is to set the deleted items retention period at 7 days. Obviously, if the cache holds messages for 14 days or longer, it will be larger than if the retention period is only 7 days. You can monitor the size of the deleted items cache in several ways. Monitoring the size of the cache will help you assess how much space the deleted items cache is taking in the Information Store. One way to monitor the deleted items cache is to view the Mailbox Resources properties of the private and public information stores. Screen 5 shows the contents of the Mailbox Resources tab in the Private Information Store Properties window.

Another way to monitor the size of the deleted items cache is to view mailbox resources through the Exchange administration program. This method lets you capture the data in a Comma Separated Values (CSV) file that you can later import into Excel or Access.

You can use Performance Monitor (Perfmon) to monitor the size and number of items in the deleted items cache. You can use several of Perfmon's counters and objects:

As Screen 6 shows, Perfmon displays the size of the deleted items cache. The four counters are active, and the highlighted counter shows that 5368 items are in the private information store's deleted items cache. The size of the cache, which is not obvious in Screen 6, is 1.4MB. After you first install Outlook 8.03, the deleted items cache will take time to establish a consistent size. After a month, the number of deleted items, and therefore the size of the cache, will stabilize.

A Little TLC
I hope I've demonstrated in this extended discussion of the Information Store that Exchange is a complex database application that needs care to achieve its full potential. Build a thorough understanding of the unique database structure of the Information Store, maintain the database carefully, and you'll be well on your way to getting the most from Exchange messaging for your organization.