Making Sense of Benchmarks

Major hardware vendors publish sizing guides to give you an idea of their products' basic performance capabilities. These guides report that during a simulation, a certain configuration supported so many mailboxes for a specific workload. Messaging API (MAPI) Messaging Benchmark (MMB) and MMB2 are the two workloads that vendor benchmarks commonly use.

MMB is an older set of user activities (e.g., create and send messages, process calendar appointments) that generates unrealistic results. For example, you'll find published MMB results that reflect hardware support for tens of thousands of mailboxes. In 2000, Microsoft introduced MMB2 as a more accurate measurement of a user workload in a production environment. Typically, MMB2 supports one-seventh of the mailboxes that MMB supports. For example, an MMB result of 14,000 mailboxes would translate into a much more realistic MMB2 figure of 2000 mailboxes.

Keep in mind that these numbers are for simulated users; human beings might not produce the same results. Humans are unpredictable—think of all those users who happily swap messages containing large attachments. Benchmarks also tend to ignore network overhead, replication activity, and all the other demands for system resources that occur in production environments. Still, when you understand the benchmark game, the generated figures for different configurations can give you a useful starting point for your server designs.