Streaming Store: A Key Component of the Exchange 2000 Database

Over the past several weeks of discussing Exchange Server's database technology, I've talked mainly about the Exchange Information Store (IS) Property Store (.edb file). The Property Store is a B+Tree database structure that stores the user data specifically for Messaging API (MAPI) clients and is well-suited to semistructured data. This week, I look at the Streaming Store—the other key file that makes up the Exchange database—as well as the files that the database engine uses to maintain data integrity.

Exchange 2000 introduced the Streaming Store (.stm file), a close companion to the Property Store, as a way to store native MIME content for Internet clients. Before Microsoft introduced the Streaming Store, an Exchange process called IMAIL conversion converted content back and forth between Exchange and Internet formats. Microsoft designed the Streaming Store as a way to avoid the resource-intensive process of converting between the native Internet formats and Exchange's rich text, property-based format. Therefore, MAPI clients don't use the Streaming Store because they use Rich Text Format (RTF). The Streaming Store doesn't have a B+Tree database structure; it uses 4KB pages arranged in clusters of 16 (4KB x 16 = 64KB). This format has more similarities to a file system such as NTFS than to a database. However, this structure is optimal for storing large MIME body parts and accessing them quickly.

Even though the Streaming Store is a companion database file to the Property Store, Exchange's Extensible Storage Engine (ESE) doesn't access the streaming store directly. Instead, the Exchange team developed Exchange Installable File System (ExIFS), a high-performance installable file-system driver that exposes the Streaming Store as a direct file system for applications to access. Thus, any application (from Exchange to Notepad) can directly access the Streaming Store. The M drive you see in Exchange 2000 is the direct representation of this access (you can also map a drive to your Exchange 2000 inbox over the network). As a result of this access architecture, the Exchange database engine accesses the Property Store directly and uses ExIFS to access the Streaming Store like any other client. Some data properties in the Streaming Store get "promoted" to the Property Store, where the data is referenced and integrity is maintained between the two stores.

So, how do the Streaming Store and the Property Store share content, and how does Exchange decide whether to convert content between the two stores? For example, what happens when a MAPI client needs to access content in the Streaming Store? The simple answer is that the content is converted on-demand, but the actual process is much more complex. Exchange 2000 determines content conversion based on certain rules that attempt to ensure that the conversion uses minimal server resources. This approach typically leaves data in the format of the last writer (Internet client or MAPI client). Remember that previous versions of Exchange always converted content to MAPI format regardless of the client type. Exchange 2000 is more intelligent about this process and attempts to minimize resource-intensive conversion operations unless they're absolutely necessary.

The Property and Streaming Stores are the core Exchange 2000 database files, but they need other files to make a complete and reliable database. The other key files are the transaction logs and the checkpoint file. When an Exchange server is running, the database files aren't consistent. A consistent database comprises the contents of the database files (committed transactions) plus the outstanding transactions in the transaction log files. Exchange transaction logs are created in generational sequence and consist of log records stored in 5MB files. The transaction logs are key to Exchange disaster recovery and ensure that the database engine can always roll back or replay transactions if the Exchange server fails.

Last, but not least, is the checkpoint file. The checkpoint file maintains a pointer to the most recent transaction log record in the database. Without the checkpoint file, ESE must replay the entire set of transaction logs to perform a full recovery. When the checkpoint file is present, ESE merely starts with the most recent transaction committed to the database and plays forward from there. The checkpoint file is the key to efficient database recovery but not absolutely required for recovery.

Next week, I conclude our series about the Exchange database engine by discussing how transacted storage works for Exchange. I take a high-level look at how a change to the database makes its way to finality in a committed transaction.