From ExBPA to ExRAP

A virtuous cycle, according to Wikipedia, is a system of events that includes a feedback loop "in which each iteration of the cycle reinforces the first," and "a virtuous cycle has favorable results." That definition pretty well describes the effect of the Exchange Server Best Practices Analyzer (ExBPA), Microsoft's automated scanning and reporting tool that lets you check your Exchange configuration against current Microsoft best practices. Since Exchange Server 2003's ExBPA launched in September 2004, its scope has continually broadened from approximately 500 best-practices rules and 300 related online articles to 3,000 rules with approximately 1,500 related articles today. The rules are based on best practices derived from customer support data and contributed by Exchange users.

But the tool's growing scope isn't the only favorable result of ExBPA, as I learned in a recent conversation with Jim Lucey, a supportability program manager for Exchange in Microsoft's Customer Services and Support (CSS) organization, and Clarence Satchell, a program manager in Premier Field Engineering. In addition, Jim told me, "ExBPA is a proactive tool, but our feedback to the product group was that customers have a lot of other pain areas: performance, mail flow, disaster recovery scenarios, etc. So the product team has created more reactive rules-based troubleshooting tools built on the ExBPA engine. They also built ExBPA into Exchange 2007, so we have built-in tools for some critical situation areas to help troubleshoot for our customers." (To learn about a couple of these tools, see "The Exchange Performance Troubleshooting Analyzer" at http://www.windowsitpro.com/article/articleid/49524/49524.html.) Another favorable result is that other Microsoft products, including ASP.NET, ISA Server, and SQL Server now also have or are developing best practices tools.

ExRAP
An interesting outgrowth from ExBPA is the Exchange Server Risk Assessment and Health Check Program (ExRAP), which Microsoft started in March 2005 for customers who have a Premier Support contract. Clarence is in charge of ExRAP and told me it's "an onsite engagement that consists of three phases: data collection, analysis and reporting, and remediation. During data collection, we use tools such as ExBPA and an operational survey with IT to gauge whether they're working according to best practices." An example of the questions on the operational survey is: "Do you have documented and communicated escalation matrices for Microsoft Exchange Server outages? (Escalation matrices define escalation paths to higher levels of technical resources and higher levels of management based on duration triggers and severity triggers. For example, a severity 1 outage might be escalated immediately to Tier 3 support, and executive management might be notified after 1 hour.)"

Next, Clarence explained, "From those results, we sendthe data to [Microsoft IT] MSIT and generate a scorecard,or benchmarking against MSIT best practices, to see wherethis customer stands. The support engineer can review theresults and interject information as well, because everybody's environment is different. Once the information hasbeen finalized, we produce two reports for the customer:one for the executives and a detailed report for IT."

Clarence continued, "Our scorecard uses red, yellow,green (high, medium, low risk) indicators to rate dependencies such as networking, disaster recovery, security. Fromthose results, we have a dialog with the customer and doa formal presentation at the end. We explain in detail eachof the issues we found and why they have a certain severityassociated with them."

Finally, Clarence said, his team loops back with the customer to "lay out a remediation plan that shows which findings are most critical and which ones need to be addressedin what timeframe." Microsoft engineers can assist withthe remediation, or customers can fix the problems themselves.

The goal of ExRAP, Clarence says, is "to educate customers about the risk that's inherent in their environment.They may have gone years without any issues at all or theymay have constant issues and don't know what's causedthem. We try to bring all this to light so we can reduce thoserepeatable offenses."

Favorable Results
The most impressive aspect of my conversation with Jim and Clarence was discovering how deeply they care about customers and how eager they are to improve not just Microsoft's customer support, but also Microsoft's products. As Jim put it, "I think customers would be amazed to know how much focus we put into understanding customer pain and making sure product groups in the company understand that pain. So many supportability fixes, solutions, scenarios are addressed in the products. We're really trying to learn from groups that have done it well and make that a best practice for other products struggling in that area." (For more information on ExRAP, see http://www.microsoft.com/technet/itshowcase/content/exchrapissues.mspx.)