Dealing with Databases

I recently worked with a customer who operated a mixed Exchange Server 2003 and Exchange Server 5.5 organization and was in the process of migrating the Exchange 5.5 users to the company's single Exchange 2003 Standard Edition Service Pack 1 (SP1) server. This server had a maximum database size of 16GB. (The Enterprise Edition doesn't limit database size in this way, but the Standard Edition does.) During the migration, the Exchange Information Store service shut down because the database exceeded that limit. (Naturally, this occurred just before Microsoft released Exchange 2003 SP2, which raises the limit to 75GB.) Does this scenario sound all too likely (or familiar)? Have you ever felt overwhelmed by the ins and outs of managing Exchange databases? You can take some of the worry out of Exchange administration by improving the way you deal with your Exchange databases.

How Big Is Big?
Most Exchange consultants recommend that Exchange private and public stores be kept below 40GB. However, given the considerable improvement in backup and restore technologies, large organizations with appropriate storage and tape configurations can let their stores grow to between 60GB and 80GB. The reason for the traditional 40GB limit isn't architectural; the limit is operational and is driven by service level agreements (SLAs) and the time needed to recover the databases. Traditional backup and restore processes and technologies, such as direct-attached DLT tapes, have restore rates in the region of 40GB to 60GB per hour. Modern SAN-based environments, which use storage virtualization and state-of-the-art Linear Tape-Open (LTO) tape drives, let you back up or restore Exchange databases at a rate of about 100GB per hour. Large databases in Exchange environments can implement Volume ShadowCopy Services (VSS)-based backups and restores in seconds. (The 40GB limit is still valid for stand-alone Exchange 2003 server configurations that use DAS.)

To Defrag or Not to Defrag
Much confusion exists about whether to perform offline defrags of Exchange 2003 databases, what online defrags offer, and whether running a file-level defrag of disks that host Exchange databases is useful. The type of defrag operation that you perform determines whether the size of the Exchange database file will be reduced. An offline defrag does reduce free space (i.e., white space), but the effects are temporary. The database will grow and reclaim the recovered space in a matter of days.

Exchange 2003's nightly online-defrag process is efficient at rearranging the pages within the Exchange databases to maximize the amount of contiguous white space and allow for optimal performance. This online defrag is enabled by default and is set to take place for each database during the maintenance interval defined on an Exchange database's Properties dialog box, as Figure 1 shows.

During an online defrag, I/O load on the database being processed is heavier than normal. Therefore, if you have a number of databases on your Exchange server, stagger the start times of the maintenance operations by setting a customized schedule for the maintenance interval. Also avoid performing online maintenance during periods when users will be logged on to their mailboxes. Out-of-hours settings are appropriate here, unless the server has users distributed across multiple time zones. If that's the case, take special care to distribute users in different time zones across different databases and different physical disk volumes. Any other I/O-intensive operations on the Exchange server should be scheduled for periods that don't overlap with the online maintenance interval. Obviously, backups fall into this category. Perform database defrags first, then back up those databases. If you start a backup while maintenance is running, maintenance will stop. An online database defrag consists of 18 subtasks. Once a subtask has started, it will run to completion even if the end of the maintenance interval is reached, which means that online database defragmentation can run over the maintenance-interval time window. Any of the subtasks that haven't been processed yet will run during the next online-maintenance period. Depending on the length of the maintenance interval and other I/O activity, a full online defrag of a database can take more than 1 day.

Although online database defrags optimize the internal structure of the database to maximize the amount of contiguous white space, they won't shrink the size of the file. Conversely, performing an offline defrag operation will eliminate all white space in a database. However, Microsoft recommends against running an offline defrag of your databases unless something significant has taken place in your environment. For example, perform an offline defrag when you've moved a large number of users from a database or when you've performed a massive content cleanup on a database. Exchange runs just fine with some white space in its databases. Eliminating all the white space and condensing the database to the minimum size through an offline defrag means that the database will have to grow any time new content is inserted into the database (e.g., whenever a new email is received). As a result, the Exchange Store must create a new file extent to expand the database to store the new message, thereby incurring additional overhead. If white space already exists in the database, as is the case when nightly online defrags are taking place, the new message can be written to the database by using the existing white space.

Figure 2 shows a rather simplified example of database fragmentation against database size in an environment that runs a nightly defrag process. The left-hand side of the diagram shows the effects of an online defrag; the right-hand side of the diagram demonstrates the effect of an offline defrag. Considering first the online defragmentation, at the end of Day 1 (D1) the database is fragmented, but after the defragmentation operation that evening (D1+d), the white space is contiguous. The same process takes place on Day 2 (D2) and so on, such that the actual size of the database never exceeds the maximum extent (Emax).

With the offline defrag process, the database is fragmented at the end of Day 1. After the defrag operation that evening, the white space is removed from the database and the database file size is reduced to extent E1. As new messages arrive into the database on Day 2, the database again becomes fragmented and grows to size Emax. After the offline defrag that night, the white space is again removed and the database file size is reduced to extent E2. This process repeats itself every day, incurring extra overhead because the database file is constantly being extended. Because the database never grows bigger than size Emax, this process is wasted work by the file system.

When the Time Comes to Defrag
Because Exchange I/O operations to databases tend to be random, a large contiguous database file isn't as crucial to good I/O performance as would be the case if large sequential reads and writes were necessary. Still, if the volume that holds your Exchange database has a high level of fragmentation, you should defrag the volume. If you don't use specialized third-party disk-defrag software, you can run the standard Disk Defragmenter tool (under Start, All Programs, Accessories, System Tools).

Be sure not to run a disk-defrag program on a production Exchange server during working hours or while a database is mounted (in which case Windows can't defrag the file). Disk defrags are I/O-intensive operations and will affect the performance of the Exchange system: Exchange I/O operations will take inordinate amounts of time to complete. For the best results, take the following steps to perform a file-system defrag:

Database Changes in Exchange 2003 SP2
Exchange 2003 Enterprise Edition effectively has no upper limits on database size. However, Exchange 2003 Standard Edition has had an upper limit of 16GB on the single permitted Exchange database. This 16GB size limit is based on the actual size of the Exchange Database (EDB) file reported by the file system, so if you're approaching this limit, check the amount of white space in the database to determine whether an offline defrag might considerably reduce the database file size. (Bear in mind that an offline defrag operation that uses Eseutil requires that the volume have additional free space of as much as 120 percent of the database size to hold the temporary files created during the defrag.)

Exchange 2003 SP2 raises the 16GB limit on the Standard Edition database to 75GB. Immediately after installing SP2, the 16GB limit is replaced with an 18GB limit so that the database can't quietly (and unknown to the administrator) grow over time to such a size that actual physical disk space is used up. Microsoft chose the 18GB limit because the smaller environments that tend to run Standard Edition might be running Exchange with limited storage capacity. Exchange begins writing warning messages on the 18GB limit at around 16GB. For more information about the database-size changes in Exchange 2003 SP2, see "In the Know: Exchange 2003 SP2 Database Increases," page 13.

If you aren't running Exchange 2003 SP2 and the combined size of the EDB database and the streaming (STM) file reaches 16GB, the Information Store service will dismount the database so that it can't grow further. Any subsequent attempts to mount the database will fail. To proceed, you must temporarily increase the upper limit of the database size to 17GB, remove content, perform an offline defrag to reduce the size of the file, then reset the size limit back to 16GB. You can't change the upper limit to a value higher than 17GB: This facility is available only for use in emergency circumstances and shouldn't be left enabled during normal operations. Doing so would run the risk of extending the database file size up to 17GB, leaving you with no way to access the database and trim its size. In any event, after the next restart of the Information Store service, the size limit on the database reverts to 16GB.

To enable the 17GB temporary extension, you must create the Temporary DB Size Limit Extension registry entry (of type REG_DWORD, supported on Exchange 2003 and Exchange 2000 Server) under the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesMS ExchangeISServerNamePrivateISIdentifier subkey. Server Name represents the name of your Exchange server and Private IS Identifier is a string beginning with the literal value "Private-" followed by the objectGUID value on the database's Active Directory (AD) object. Set the entry's value to 1 to enable the extension or 0 to disable it.

After you increase the limit, stop the SMTP service so that no new email can be delivered into the database before you remount it. Mount the database, remove content (either by moving mailboxes onto another server or by encouraging your users to clean up mailboxes), then force an online-maintenance operation to clean up deleted items in the database. Last, perform an offline defrag of the database to reduce its size. (For step-by-step instructions of this process, see the Microsoft article "Exchange Server Mailbox Store does not mount when the Mailbox Store database reaches the 16-GB limit" at http://support.microsoft.com/?kbid= 828070.)

Putting It All Together
Exchange 2003's and Exchange 2000's online maintenance all but eliminates the need to perform time-consuming and labor-intensive offline defrag operations. The database size-limit increase in Exchange 2003 SP2 also eases the problems involving earlier versions' 16MB limit. Whichever of these Exchange versions you use, get to know how Exchange databases work, then take advantage of the available database-related registry settings, defrag options, and modern backup and archiving technology to get a handle on database management.

Kieran McCorry (kieran.mccorry@hp .com), based in Ireland, is a principal consultant in HP's Advanced Technology Group and a Microsoft Exchange MVP. His most recent book is Microsoft Exchange Server 2003 Deployment and Migration (Digital Press).