Exchange's Database Technology Helps Ensure Data Integrity

Over the past few weeks, we've covered Exchange Server's database file structure. We spent most of the time discussing how user data is represented in both logical and physical terms in the Property Store and the Streaming Store. This week, I wrap things up with a discussion about how Exchange's Extensible Storage Engine (ESE) uses these files to store user data in a transacted manner.

The main idea behind a transacted storage engine is data integrity. If you were a computer science major in college, you might remember the term ACID (Atomic, Consistent, Isolated, Durable), which refers to properties that a database engine must have to support business-critical applications. Exchange's ESE obeys the absolute rules of ACID by ensuring that its transactions meet the ACID properties. For example, in Exchange, if you want to move a message from your Inbox folder to another folder called Hot Items, ESE must perform several logical operations: Insert the message into the Hot Items folder, delete the message from the Inbox folder, and update tables to reflect the correct number of folder items. At the physical database level, this process is much more complex than these three steps. However, because these operations take place within the boundary of a single transaction, either all or none of the operations must complete for the database to maintain integrity.

When writing to the database files, ESE uses three key mechanisms. The first is the log writer. As transactions occur, ESE stores them in log buffers in memory. Using write-ahead logging (write the log records to disk before updating the database), ESE flushes log buffers to disk as transactions complete and buffers are used up. This approach ensures that transactions are stored to disk as soon as possible in a synchronous operation. ESE uses the Information Store (IS) buffer pool (a pool of 4KB buffers allocated from RAM and managed by Exchange's dynamic buffer allocation algorithm) to cache database pages. As a page is modified, the page is read from disk to RAM and modified in RAM by running transactions. When a page in RAM is modified, the page becomes "dirty," and ESE must flush it to disk as soon as possible. The mechanism used to accomplish this process is the "lazy writer." The lazy (meaning when ESE gets around to it) writer manages the process of getting dirty pages to disk and attempts to balance ESE-generated I/O through a leveling algorithm that tries to flush dirty pages to disk without overloading the disk subsystem. A final ESE mechanism, the version store, is also key to maintaining integrity and performance. The version store gives ESE the ability to track and manage multiple concurrent transaction operations to the same database page, providing the isolation and consistency attributes required for ACID compliance. The in-memory version-store data structure stores database pages as they are modified and ensures that each transaction sees the correct page version. Using these mechanisms (the log writer and log buffers, IS buffers and the lazy writer, and the version store), ESE can safely and reliably commit changes to the database while achieving optimum performance and scalability.

Shut off those propellers! I've taken several weeks to introduce you to ESE. This information is crucial for daily operational tasks such as backup and restore and database management. Knowing the underlying technology is key to deploying reliable systems. If you want more details about Exchange database technology, look at my book "Mission Critical Microsoft Exchange 2000." I hope this series has helped you better understand how Exchange works inside.