Exchange Server Benchmarks: What Are They Good For?

Vendors love to publish Exchange Server benchmarks, but how useful are these measurements?

For years, hardware vendors have put a lot of effort into running benchmarks for Exchange Server, having Microsoft review and approve the results, then publishing those results. These vendors have several objectives. First, they want to prove that they have expertise with Exchange. Second, they want "bragging rights" for having the highest Exchange benchmark number. And third, they want to offer platform-sizing information to help customers select the right server configuration for a specific user load. These objectives are understandable, but I find that when it comes to buying decisions, many organizations are more concerned with a product's deployment factors (e.g., reliability, disaster recovery, management) than with Exchange benchmarks. Still, benchmarks can play an important role in the evaluation of server sizing and configuration—provided that you keep the benchmarks' limitations in mind.

Historically, the only published benchmarks for Exchange have been Messaging API (MAPI)-based, primarily because MAPI was originally the only client protocol that Exchange supported. Microsoft and hardware vendors have adopted the MAPI Messaging Benchmark (MMB) and, more recently, MMB2 as the de facto Exchange benchmark standards. The most recent version of the Microsoft Load Simulator (LoadSim) tool—which you can dowload at the first URL below—uses MMB2 in an attempt to standardize a workload profile for a typical corporate user who accesses Exchange through a MAPI client. Because the majority of Exchange users utilize MAPI, the MMB standards are relatively sound.

For MAPI environments, Exchange benchmarks serve several useful purposes. First, benchmarks provide a way to compare vendor platforms and these platforms' ability to run Exchange. (Keep in mind, however, that most comparisons demonstrate one or the other vendor's expertise with the benchmarking process more than they do any significant differences between hardware products.) Second, and more important, benchmarks let you differentiate between hardware configurations and their impact on Exchange performance. For example, the difference between four and eight processors or 1GB and 4GB of RAM is evident when you compare benchmark results for these configurations. Third, benchmark results provide some degree of confidence that a particular platform is well tested running Exchange Server in a heavily loaded or high-stress environment.

Still, don't place benchmarks on too high a pedestal when selecting hardware vendors or configurations. You can run into problems when you assume that the standard MMB or MMB2 workload profile matches the workload your Exchange users place on your Exchange servers.

Another shortcoming of the MMB and MMB2 standards is that they simulate a MAPI-only workload. Exchange 2000 Server and Exchange Server 5.5 provide full support for non-MAPI clients, but MAPI benchmarks don't apply if you use client protocols such as IMAP, POP3, HTTP, and SMTP to access Exchange. LoadSim doesn't even offer an option to test non-MAPI protocols. In fact, no real standard exists for simulating Internet mail client workloads against messaging servers (with the possible exception of some little-known work being done by the Standard Performance Evaluation Corporation—SPEC—see the second URL below for details). Microsoft provides the Exchange Stress and Performance (ESP) tool—which you can download at the third URL below—for testing Internet protocols against Exchange, but no standard exists for vendors to use in publishing benchmark results for comparison. Therefore, if you use Internet protocol-based clients to access Exchange, you have little available data to determine proper server sizing and configuration. Organizations such as ISPs and application service providers (ASPs) have a great need for performance and sizing information for Exchange but must conduct their own simulation exercises and benchmarking.

No benchmark standard for Exchange will ever be perfect. Furthermore, with the gradual phasing out of MAPI support in Microsoft products such as Exchange and Outlook, the relevance of the MMB and MMB2 benchmarks is fading. In addition, most organizations are more concerned with other deployment and operational factors that benchmark results don't reflect and that have a more significant effect on Exchange server sizing and configuration. However, today's Exchange clients are still predominantly MAPI-based clients running on Exchange 5.5, so the MMB benchmarks continue to provide value for making system comparisons and determining optimum sizing. Just be sure to use benchmark results with care and keep in mind their limitations as well as their benefits.

LoadSim download
http://www.microsoft.com/exchange/downloads/2000/loadsim.asp

SPEC
http://www.spec.org/osg/mail2001

ESP download
http://www.microsoft.com/exchange/downloads/2000/esp.asp