Background Maintenance for Exchange Servers

All databases have internal structures that must be maintained in some way, and the Exchange Server Store is no different. Because Exchange is designed to be highly available with as little downtime as possible, the Store performs as many maintenance operations as possible as online background tasks. These tasks are performed each night to ensure logical consistency within the mailbox and public store databases and to remove unwanted data. Let's review the tasks that the Store undertakes in nightly background maintenance to see why some of these tasks are so important to the efficient operation of an Exchange server.

Night Moves
As Table 1 shows, the Store performs 11 background maintenance tasks each night. If the Store can't complete all the tasks during the allocated time, it finishes the last running task and records where processing stopped so that it can pick up from that point in the next maintenance period. Actually, the Store executes the first 10 tasks listed in Table 1, then calls the Extensible Storage Engine (ESE) database engine to perform the last task: Store defragmentation. The Store must complete at least one of the 10 tasks before calling ESE. In concept, this division of tasks between the Store and ESE is similar to the way in which the Isinteg and Eseutil utilities operate; Isinteg works with the structure of tables and indexes into which the Store assembles raw database pages, and Eseutil deals with database pages, at a lower level.

Obviously, the 11 tasks can generate a huge number of updates to Store contents. The Store captures update details in transaction logs, which explains why more transaction logs are generated during the maintenance window than you can account for from user activity (which is usually low at night) or even public folder replication. You can make backups while most background maintenance tasks are going on. However, backups do clash with background defragmentation, so if you attempt to make a backup while online defragmentation is running, ESE will suspend processing until the backup is finished.

By default, Exchange runs background maintenance between 12:00 midnight and 5:00 a.m. local time. You can set up a separate custom schedule for each server by opening Exchange System Manager (ESM), selecting the database, opening its properties dialog box, then selecting the Database tab. Alternatively, you can create a server policy and apply it to databases on multiple servers at one time, which is obviously the best approach in large Exchange organizations. Web Figure 1 (http://www.windowsitpro.com/microsoftexchangeoutlook, InstantDoc ID 43903) shows a custom schedule set through a server policy. The schedule lets the Store run background maintenance from 11:00 p.m. to 6:00 a.m., Monday through Friday (finishing on Saturday morning), and from 7:00 p.m. on Saturday through 6:00 a.m. on Monday. Creating a long maintenance period over the weekend lets the Store complete even the longest operations during a time when user activity is low. Note that you can create a schedule to let background maintenance run Always, which means that the Store is free to conduct maintenance operations, including online defragmentation, continuously. However, doing so isn't a great idea unless you know that your server has the capacity to accept the load that background maintenance generates—and remember that online defragmentation and backups clash with each other.

Background Tasks
Now that you understand the basics of what happens during background maintenance and when it happens, we can consider some of the details of the processing. Clients such as Outlook make extensive use of ESE's ability to dynamically generate indexes, or views. For example, if you decide that you want to view your Inbox by author rather than by date, Outlook asks the Store for this view. If the Store has previously generated the view and has cached it, the response is especially quick, but the Store can quickly process (through ESE) even a request for a new view. Over time, the Store accumulates many views—each folder in every mailbox can have several views—and this situation isn't desirable because each view occupies space within the database and some views are used only once. To manage views, the Store assigns each view an expiration time and maintains an internal table called the index aging table. When background maintenance runs, the Store scans the index aging table for views that are older than 40 days (8 days on Exchange Server 5.5 systems) and removes any view that has expired. Of course, every time someone accesses a view, its expiration time is reset.

The next task is tombstone maintenance. The Store maintains a list of deleted messages (i.e., messages that users have cleared from their Deleted Items folders) plus a list of tombstones for the deleted messages. (Each deleted message has a tombstone that tracks the actual delete operation for the message.) The tombstone list is especially important for replicated folders, such as public folders, because it provides the Store with a list of message-delete operations that the Store must replicate to every server that holds a replica of the folders. After entries have been successfully replicated, the Store can clear the entries from the tombstone list during background maintenance.

When a user deletes a message, the Store sets a flag to hide the message from the mailbox (i.e., the Store performs a soft delete). The user can retrieve the message from the Deleted Items cache, which clears the deleted flag. During background maintenance, the Store examines every message that has the deleted flag set (the entire contents of the Deleted Items cache) to determine whether the message's retention period has expired. If it has, the message is permanently removed from the Store (i.e., the Store performs a hard delete). You can set retention periods on a per-mailbox, per-store, or per­public folder basis, and you can control this setting globally through a server policy that you set through ESM. Web Figure 2 shows some limits set on a public store—you can see that the deleted-items retention time is 7 days. For example, you could use a public folder to hold content, such as a feed from a Network News Transfer Protocol (NNTP) service, that you want to age out in a systematic way. By aging out the content, you impose a check on how large the folder grows. Much the same processing occurs for deleted messages in public folders—the Store checks for messages that have expired and removes them.

The next check is for deleted public folders that have expired. By default, if you delete a public folder, the Store creates a tombstone for the folder to give the replication mechanism time to propagate the deletion properly to servers that hold replicas. Because replication might not occur quickly in some circumstances, the Store retains tombstones for 180 days by default. If replication hasn't propagated the tombstone in 180 days, your organization is experiencing fundamental replication difficulties and a few erroneous tombstones are the least of your problems. During background maintenance, the Store checks for folder tombstones that have expired and removes a maximum of 500 tombstones per 24-hour period. The Store also checks for deleted mailboxes, which have a default retention period of 30 days, and removes any that have expired. Removing obsolete folders and mailboxes cleans up the internal table structure of the database and increases efficiency.

Public folder replication is thought of as something of a black art within the Exchange community. Public folder conflicts can easily occur; for example, multiple users might modify the same item in different replicas of a public folder or multiple users could attempt to simultaneously save an item in the same replica. When a conflict happens, the Store sends a message to the users that caused the problem to ask them to resolve it. Meanwhile, the Store maintains the different versions of the item. If the users fail to respond, the Store examines each version during background maintenance and resolves the problem, typically by accepting the most recent version. You can programmatically control the resolution process by manipulating the Messaging API (MAPI) PR_RESOLVE_METHOD property on the folder. Most administrators are likely to leave Exchange to its own devices, but you can use a custom form to help resolve the conflicts. See the Microsoft Web page "Resolving Message Conflicts with Forms" (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/exchserv/html/forms_3goj.asp) for more details.

The next task is to update server versions for the public stores. This process updates any version information necessary to maintain the system configuration folder. No great overhead is incurred, and you can't control the process. Background maintenance also checks whether any duplicate site folders exist within public stores in an administrative group and removes any duplicates that it detects.

Note that the secures folders task in Table 1 applies only to public folders that are homed in Exchange 5.5 sites. After you establish a native Exchange Server 2003 or Exchange 2000 Server organization, the need for this task goes away.

The Store uses a single-instance storage model, meaning that it keeps one copy of message content and uses pointers in user mailboxes to the content. The Store tracks the number of mailboxes that have pointers to content through a reference count and physically removes the content from the database when the reference count reaches zero. Background maintenance checks for lingering messages that have a zero reference count and removes up to 50,000 of these items per day. The Store doesn't report how many items it has removed, so a large growth in the number of transaction logs is the only indication that this type of activity is going on. If you observe that the Store is generating more than the typical quantity of logs in the midnight to 6 a.m. period, you should consider running Isinteg against the mailbox store.

Online Defragmentation
Exchange administrators typically perform offline defragmentation—that is, they use Eseutil to rebuild the Exchange database page by page—to recover disk space for the file system. You should have at least 120 percent of the database size available in free space to accommodate the temporary files used during processing, although this figure can vary depending on the degree of fragmentation within the database. If necessary, to find the required disk space, you can redirect the temporary files to a disk that has sufficient free space. You can even run Eseutil on a computer that doesn't have Exchange installed by transferring the necessary program files and databases to that computer (see the Microsoft article "XADM: How to Run Eseutil on a Computer Without Exchange Server" at http://support.microsoft.com/?kbid=244525 for details). Users can't access their mailboxes while the defragmentation proceeds.

The Store runs an online defragmentation process nightly for every database on an Exchange server to "shuffle" pages internally so that linked pages are kept together, data is compressed into the smallest and most efficient number of pages (to reduce I/O), and "white space" (i.e., unused pages) is placed at the end of the database. The process calls the ESE engine to examine individual tables in the Store and attempt to reduce the number of pages that each table occupies. For example, if a message table currently uses 100 pages that are half full, the defragmentation process will aim to compress the table into approximately 50 pages. Online defragmentation doesn't return space to the file system and doesn't reduce the overall size of the database. Even though the database remains the same physical size, online defragmentation contributes to the overall efficiency of the Store because the reduced number of pages means that the Store expends less I/O processing to access information.

Early versions of the online defragmentation process weren't very efficient, which led to recommendations that administrators should rebuild their databases on a regular basis to prevent the databases from becoming too large. In Exchange 5.5 and later versions, the online defragmentation process is much more efficient and effective, so you shouldn't need to perform an offline defragmentation (rebuild) unless you have good reason to suspect that you can recover a large amount of disk space. Note that nightly online defragmentation doesn't affect the streaming store (.stm) file because Exchange doesn't attempt to rearrange the internal structure of this file.

After online defragmentation, the Store reports the amount of unused space in the Application log as event ID 1221 (The database has nnn megabytes of free space after online defragmentation has terminated), as Web Figure 3 shows. In the figure, the Store reports that 1672MB—or roughly 12 percent—of the 14GB database is free space, which is a typical percentage of white space in an Exchange 2003 mailbox server. You can recover this space by using Eseutil to perform an offline defragmentation.

You can check event ID 700 (Information Store: Online defragmentation is beginning a full pass on ) and event ID 701 (Information Store: Online defragmentation has completed a full pass on ) to determine how long online defragmentation takes on your server. Processing the 14GB mailbox store on my quad-CPU 900MHz server with 1GB of memory takes about 4.25 hours, yielding a throughput of roughly 3.25GB per hour. Throughput varies with processor speed, I/O capacity, and system load, including other management operations that might be proceeding at the same time, such as emptying of the Deleted Items cache, defragmentation of other databases, and generation of the Offline Address Book (OAB).

Registry Settings
Exchange wouldn't be Exchange without some registry settings that you can twiddle to change the way things work. Background maintenance has a selection for you to experiment with, if you like playing with fire. You should reset these values only if you have an extremely good reason to do so.

All the registry values are of the REG_DWORD type. They fit into three registry subkeys, as Table 2 shows, where PS indicates the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesMSExchangeISParametersSystem subkey, M indicates a subkey for a mailbox store, and P indicates a subkey for a public store. In versions earlier than Exchange 2000, Microsoft placed all the general registry settings for the Store in the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesMSExchangeISParametersSystem subkey, but because Exchange 2003 and Exchange 2000 support Store partitioning, we now have a subkey for each server with separate subkeys for each store under that server. Web Figure 4 shows the registry hierarchy and settings that govern how the Store deals with a public store on an Exchange 2003 server.

Tracking Background Maintenance
As with any other operation in Exchange, the selected level of diagnostic logging determines how much information the Store writes about maintenance operations into the Application log. You set the level for diagnostic logging by selecting a server in ESM, opening its properties dialog box, and going to the Diagnostic Logging tab, which Web Figure 5shows. To log sufficient details about background maintenance events, you need to adjust the Background Cleanup setting for the MSExchangeIS Public Folder and Mailbox objects from its default logging level of None to Minimum, Medium, or Maximum. I recommend the Maximum setting for background maintenance; it doesn't generate a flood of events (unlike the settings for other features such as public folder replication), and you can always learn something from the extra detail generated at this level. After you're familiar with the different operations and know what to expect, you can reduce the setting back to Minimum.

After you apply the new setting, the Store begins to generate events the next time it runs background maintenance. Table 3, page 13, lists the most common events that you'll see. Except on very large or heavily loaded servers, many of the tasks are completed quickly. The obvious exception is background defragmentation. This intense activity can take many hours, depending on background load, server capacity, and the amount of data to process.

Exchange uses some sophisticated database technology to deliver its rich functionality set to clients. Behind the scenes, the Store does a lot of processing to keep everything aligned and in good shape. Most of the Store's work is automatic and doesn't require administrator intervention, but it's good to know what's going on in the background, just in case.