Everything You Ever Wanted to Know about ESE

I've received a lot of questions about Exchange Server's database technology. This topic is a particularly interesting aspect of Exchange for me (it really gets my propeller head spinning); therefore, I thought you might enjoy a series about the history, evolution, and technology behind Exchange's Extensible Storage Engine (ESE).

ESE's roots go deep at Microsoft. Formerly known as Joint Engine Technology (JET), ESE is an evolution of what Microsoft hoped would become the unified storage engine for all Microsoft technologies. Unfortunately, the plan didn't work out that way, but ESE is still a very technically advanced product. Microsoft defines the ESE specification as follows: "ESE is a multi-user ISAM database with full DML (Data Manipulation Language) and DDL (Data Definition Language) capability. ESE allows applications to store records and create indexes to access those records in different ways. It is optimized for very fast storage and retrieval of semi-structured data."

Unlike SQL Server, ESE is not designed to take direct client queries or perform direct transactions. To access ESE in this fashion, you need the Microsoft OLE DB interface that ESE supports.

If you talk with Microsoft developers about their ESE design philosophy, you find they had several things in mind as they've evolved ESE over the years. The most important consideration was that everything ESE does has to be designed and written from the standpoint that the system could crash at any time. ESE needs to be able to survive crashes; it must recover so that the data remains consistent and accessible. Another key design point that we can all see demonstrated daily is that going to disk is expensive. Developers designed ESE with the thought that an I/O to disk to retrieve 4KB of data (the size of an ESE page) is about 200,000 times slower than getting the same data from memory. ESE will consume extra CPU cycles and use most of the machine's memory to avoid going to disk. The ESE database engine is also self-tuning. Wherever possible, the engine regulates its use of log files, memory, and other system resources with minimal administrative interference. Finally, ESE is designed for 24x7 operation, including backup operations and database maintenance. These design points have shaped the Exchange database technology into what it is today—a reliable store for semistructured data.

Exchange's ESE uses a data structure that should be familiar to computer science graduates: the B-Tree database structure. Microsoft borrowed this technology for ESE and has evolved and improved on it for Exchange over the past 5 years. I'll look deeper into Exchange ESE next week and discuss ESE's implementation of B-Tree structures, including indexes, and other database structures and formats. In later columns, I'll discuss details about how ESE performs operations and transactions with user data. By the time I'm finished with this ESE overview series, I hope to give you knowledge that will assist you in your daily Exchange operations.