Talking Exchange Server with Microsoft's Perry Clarke

Talking to any senior technologist can be a challenge for a journalist (which I'm not). The task of the journalist is to extract something juicy during the interview, while the interviewee has to resist all attempts to make himself open up and instead concentrate on passing the "message du jour." When I took over as HP's security lead in 2003, I received advice from the PR team that I should "always speak in paragraphs" because "you are less likely to be misinterpreted" and "whatever you say will be reported verbatim because it hangs together."

Moving forward 10 years, I find myself talking to Perry Clarke, corporate VP for Microsoft Exchange, and I'm the one attempting to get the big scoop. It's an interesting reversal, especially when faced with someone I've known for years. And Perry definitely doesn't answer questions in fully formed paragraphs. Instead, you get torrents of ideas and opinions, deep views, and reflections—all of which makes interesting listening, although it can be challenging to decipher.

Perry has spent his entire 17-year career at Microsoft working on Exchange Server and is now in charge of every piece of code written for Exchange, both on-premises and cloud (i.e., Office 365). Following Perry's recent "Exchange Server: The Road Ahead" post on the EHLO blog, I canvassed the Exchange community for questions and received a lot of feedback. I also checked out the Microsoft Exchange Improvement Suggestions website. I gathered a lot of ideas for questions—certainly far too many to cover in an hour, especially in the cut-and-thrust of a debate between two people who care deeply about a particular technology and have devoted a fair portion of their careers to that technology. The topics Perry and I eventually managed to cover during our conversation include the following:

I recorded the interview, and I later listened to the recording and wrote this article based on our conversation. Direct quotes from Perry are shown within quotation marks. The rest of the text is my interpretation of what Perry said (and my attempt to put it all into a logical order). Any errors in the article are entirely mine. I hope that you find the debate interesting.

The Current State of Exchange

We began our conversation with the current state of Exchange. I asked Perry if he's happy with where Microsoft is, in light of anecdotal evidence that customers aren't moving to Exchange 2013 as quickly as they moved to previous versions, despite some major engineering advances—perhaps because of fears about poor quality, issues with interoperability, some missing functionality, and reports of poor server performance. I asked whether the current situation is similar to Exchange 2000, when customers held back because although a new architecture was introduced, there was also a lack of compelling new features—meaning that Exchange 5.5 was deemed to be "good enough."

Although you'd expect any product's development supremo to defend his work, Perry surprised me by saying that he is happy with the Exchange product group's position, pointing out that they're in a much better position at this point in the development cycle (a year after shipping a major release) than they were at the equivalent time post–Exchange 2010. A common code base to span on-premises and cloud is in place and working, Exchange Online has taken on millions of new mailboxes, and the data reported from Microsoft's experience of running the service is being factored into decisions about the future development of Exchange. Perry pointed out that a flawed product couldn't possibly cope with the stresses and strains that Exchange Online undergoes. The Exchange 2013 code handles four times as many cloud mailboxes today as it did a year ago, and all of Microsoft's internal metrics report better performance and availability, as well as far fewer alerts than with Exchange 2010.

I demurred on this and pointed out that many of those who attempt to deploy Exchange 2013 on-premises have observed that it requires far more resources (mostly memory) than Exchange 2010 and seems to be more fragile. The loss of functionality such as S/MIME support is also being felt, as is the need to change the administrative model. I did concede that a huge amount of knowledge and experience now exist regarding Exchange 2007 and 2010 deployment, and that it will take time to accumulate the same expertise for Exchange 2013.

Perry also said that on-premises customers get a huge advantage from the fact that the "exact same" code base is now in use for both platforms and that new code is stressed by being run in Office 365 for several weeks before it's released to on-premises customers in a cumulative update. "There isn't a part of the core Exchange Server that is not shipped to our on-premises customers. We don't have a special Exchange that runs in the cloud." Perry said that it should "comfort our on-premises customers that the code that they run is validated by running against millions of mailboxes before they have to deploy it."

I pointed out that Office 365 is a very structured environment that thoroughly exercises many parts of the product by running code at extreme scale but misses a lot of the detail found in customer deployments. For example, Office 365 doesn't permit server-side integrations with third-party products and insists on the use of the latest clients, whereas almost every on-premises deployment will use one or more third-party products alongside Exchange and involve a mix of clients. Perry agreed, but he noted that "the problem [of testing for specific customer environments] has always been there" and said that Microsoft "hasn't taken steps back on what we have done historically to address that issue."

We discussed interoperability with Exchange 2007, an area of functionality that isn't exercised or tested by Exchange Online and one that seems to be causing problems in customer projects. Perry acknowledged that "the n-1 release [Exchange 2007] has always been a challenge to get validation of the interoperability story." He noted that Microsoft still has some Exchange 2007 servers running production workloads and test labs but that customer environments have become "more complex," which makes interoperability testing more difficult. Given the nature of some of the infrastructures deployed into customer sites (including third-party software), it's impossible for Microsoft to test every aspect of interoperability, but the company is "not taking steps back" from the responsibility. Perry also said that the extensibility models (from Exchange 2010 onward) that now exist are much better and should prove to be more robust.

Exchange 2013 introduced a new problem in the need to continually increase the scale (of Exchange Online) and keep the code base moving. Perry said that "getting our cumulative update pipeline to be excellent has been an interesting challenge." Similar problems with quality updates had occurred with previous versions and had to be worked through then. A rigorous exercise was conducted during the past summer to figure out where the problems occurred in updates. Although better testing at scale was being done through deploying new code into Exchange Online, regressions were still creeping through. An example of a recent regression in Exchange 2013 CU3 is when Windows XP clients can't use Internet Explorer 8 to connect to OWA.

In Perry's mind, a simple contract exists between the Exchange team and on-premises customers ("just apply the updates and your lives will get better"), so even a single regression is too much. The kind of regression seen in Exchange 2013 CU3 "shouldn't happen," and even though Microsoft feels confident that it has a handle on the problem and has deployed new checks and testing to improve the code that goes to on-premises customers, the company still needs to work hard to ensure that high-quality updates are delivered in the future. Perry acknowledged that customers won't believe the story of quality in Exchange 2013 until they see a stream of successful updates that demonstrate that fact. He said that internal Microsoft data shows that the company is in better shape to deliver updates than at any time in the past, but he also pointed out that it's difficult to implement the kind of functionality that's demanded by customers in older clients, especially when those clients don't support some of the basic underpinnings required to enable new functionality in components such as OWA.

At no point did Perry attempt to say that everything is perfect with the new code. Indeed, he said that every new product creates some new "bumps in the road" for on-premises customers. But because most customers don't deploy until SP1 comes out, it gives Microsoft the chance to discover what bits are missing or don't work and to fix them in SP1. Perry pointed out that with Exchange 2010 there was a "bigger freak-out with what was missing in OWA [in the RTM release] than this time around" (OWA was rewritten for Exchange 2010 SP1). He said that there had been a huge reaction within and outside Microsoft when Exchange 2010 appeared because of some interoperability problems and other issues but that SP1 made everyone forget the original problems. In his view, Exchange 2013 is better than Exchange 2010 was, and we'll see the same degree of improvement when SP1 appears.

I asked why the same story has played out for many releases of Exchange, in which the RTM release is flawed and SP1 saves the day. Was it a case of Microsoft organizational politics requiring Exchange to ship at an arbitrary date, as in the case when Exchange 2013 RTM appeared alongside the other products in Office Wave 15? Perry vigorously disagreed with this hypothesis. So why did we see so many problems in Exchange 2013 RTM?

It's a complex issue. Perry pointed out that the "feedback loop" from customers to Microsoft is "too slow" because it takes on-premises customers a long time to deploy and test new software. The period between new features being designed, coded, tested, and shipped by Microsoft and when the product is eventually used by the "median customer" is "half a decade." No software development group can wait five years to hear whether they have done a good job.

Perry concludes that the only way of getting feedback at scale is therefore to put a product into the marketplace, which is what happened with Exchange 2013 RTM. The customers who wanted the feature set were able to deploy and use the software successfully, whereas those who needed some things to be fixed had to wait for a release like CU1. Many others will wait for SP1, and some will wait even longer. Perry concluded that this is the "nature of the on-premises game." He also said that "there is no customer on the face of the planet that is capable of doing a [major] Exchange deployment in less than three years [after the previous major release]. Systems are on a half-decade cycle: That's a fact of life."

Some might quibble at this, and the exception will prove the rule—but I think this statement expresses some of the natural frustration that software developers have when their work isn't used as quickly as they feel that it should be. Of course, the fact that Exchange has such a large installed base is both a benefit and a drag because it does take time to modernize the bulk of any base. This isn't a new phenomenon; the same is true for operating systems, and it was also true in the 1980s when the team I worked on at Digital struggled to convince customers to deploy new versions of ALL-IN-1, which was the leading corporate email system at the time. But Perry really values the size of the Exchange installed base because it's a huge asset to Microsoft.

Microsoft's new servicing model for Exchange 2013 is therefore based on two principles. The first is that Microsoft will "never go dark for two or three years," which means that the product will be in a state of constant improvement through updates shipped to customers on an incremental basis. The second is that some method has to be adopted whereby customers can keep pace with the development group by deploying the incremental releases. These principles have resulted in the quarterly cadence that now exists for Exchange 2013 cumulative updates.

In explaining the logic behind cumulative updates, each of which is a full-blown version of Exchange that can be deployed from scratch, Perry reflected that a traditional release is composed of three parts: incremental change to UI, improvements in existing functionality, and increased quality (bug fixes). Any update will have different proportions of the three parts: Some might change the UI dramatically (think of the OWA update in Exchange 2010 SP1); another might focus on increased quality; and another might roll out new features. Updates can also include architectural shifts to take advantage of new or emerging technologies and changes in hardware capabilities (the shift Exchange has made to trade memory for disk I/O is a good example of how Microsoft has used the fact that memory is getting dramatically cheaper to improve performance), as well as the evolution of what the user experience should be in light of new technology and the current environment (mobile computing is an example here).

Perry said that Microsoft's challenge is "how to stream the incremental wins into the update path and put the disruptive stuff into buckets that customers can consume." A major release changes the entire ecosystem around Exchange. Microsoft has to explain to its field and partners (major companies such as HP, as well as partners throughout the world that assist customers to deploy Microsoft technologies) what the major changes are and why they're important, including the "deep thought" that has driven the change. Customers take that information and knowledge about the new software and put it into context with their own business requirements and existing infrastructure to figure out how they can build a deployment strategy that justifies the necessary expenditure for the migration project, including the capital expenditure for new computers. Perry said, "No one is asking us for the disruptive stuff at a faster cadence than we're delivering. If anything, they would like that pace to be slower."

So we end up with a release cycle composed of multiple incremental updates leading up to a major release every three years or so. The shift to the new cadence has caused some problems to emerge that customers see as quality issues slipping through into the updates. Perry's view is that the model is now settling down as Microsoft and customers become accustomed to it, and he is confident that the "incremental stream [of updates] will be much more robust going forward."

Exchange and the Cloud

Getting back to the topic of the cloud and why so much importance is being put on cloud platforms today, Perry and I discussed how Microsoft developed the plan to transform Exchange into a cloud-capable application. Perry related that after the release of Exchange 2003, Microsoft had to chart out a plan to bring Exchange into a state where the team could deal with some of the technical developments that they could predict at the time and fix some of the deep technical and architectural flaws that had been introduced in Exchange 2000. Most of the flaws were addressed in Exchange 2003, but some have taken three releases to move Exchange from a product where components were "tightly coupled and loosely structured" to the situation that exists today where the focus is on "loosely coupled, tightly structured." (The best example of where this change has occurred is in the way that Exchange 2013 insists that servers communicate using a small number of well-defined protocols such as SMTP, Exchange Web Services, and MRSProxy.)

In 2005, the cloud was getting some serious traction and looked as if it would be a huge influence on the future. However, the perceived wisdom was that there was "no way that you could take a product team, code base, and knowledge base [that worked on-premises] and take it forward into the cloud."

In some ways, the challenge of embracing the code paralleled that which faced Microsoft when the company developed Windows NT to break into the enterprise market and move away from its desktop client roots. It seemed like "a blank sheet of paper and a new organization would be required to move forward" to make applications such as Exchange cloud-capable. Perry said that two problems were obvious with such an approach. Internally, Microsoft would have to abandon a lot of code, people, and knowledge. In addition, the company would have to abandon the on-premises installed base (which at the time was well over 100 million mailboxes).

I asked if it didn't seem like the on-premises base felt that they were being abandoned today, given Microsoft's apparent eagerness to move everything to the cloud. Perry acknowledged that this perception might exist, but he pointed out that in 2005 Microsoft had three options:

The first option would have completely ignored a platform that is now quite important and ceded command of the email cloud market to Google. The second option would have screwed the installed base and forced them to continue running software that gradually became more and more obsolete. This is a valid option for a newcomer to the email market and is the path taken by Google when the company developed Gmail. This option would also have caused "panic" within Microsoft as parts of the company struggled to retain customers—an influence that would have compromised the ability of the product team to deliver the new product. The third option is the one that Microsoft took, which with the benefit of hindsight has proven to be the most valuable—even if the resulting transformation in the architecture and code base has created some problems that customers see as flaws and quality issues. Perry believes that Microsoft is getting a handle on these issues as the code becomes more settled and Microsoft gets better at levering the single code base to serve both on-premises and cloud platforms. Some evidence of this is seen in Exchange 2013 CU3, which has had fewer problems reported than RTM, CU1, or CU2.

Perry said that there are few customers today who aren't thinking about whether they should move to the cloud. He did say that "a very large percentage of customers are not moving to the cloud" but that "two years ago when customers were thinking about what to do [for the next big upgrade], the percentage of customers who were considering a cloud strategy was much smaller." He believes that it takes time for companies to crystallize their thoughts around new developments and to figure out when it's the right time for them to adopt a new platform. The fact that Office 365 has delivered a robust and reliable service since June 2011 has helped many customers make up their minds.

If a company doesn't know what to do with the cloud, they can create a test Office 365 domain to kick the tires and figure out what's good and bad about cloud-based systems. It's very easy to run a cloud pilot, compared with the organizational and deployment effort required to run an on-premises pilot of new software. This ease of testing helps IT become more agile and make better decisions.

Given that many customers find it difficult to deploy new software as quickly as Microsoft releases it, Perry and I debated how much of the installed base would end up using Office 365—which means that Microsoft takes care of deploying and managing the software. Would we get to a situation, as predicted by some commentators, in which the complete installed base would move to the cloud, or would a substantial portion remain on-premises?

Perry's view is that even though there are many "perceived thought leaders" (including some inside Microsoft) who predict that 100 percent of customers will be in the cloud, this line of thinking is wrong and many customers have valid operational and business reasons to remain on-premises. Customers are hearing from people who are respected in the market (including market analysts) that everything is going to the cloud, but this doesn't make business sense. In Perry's words, "It's nuts."

Although Perry has been told for his entire 17-year Microsoft career that email is dead, he pointed to the "level of investment that customers are making to do email every fraction of every second of the day" to enable access to email everywhere from multiple devices. This creates "interesting challenges in data security, privacy, and data leakage." Microsoft has had to think through these problems in order to deliver Office 365 at scale, and that experience is, in Perry's view, "deeply valuable for our customers."

Given the current run-rate for Office 365 and the available evidence in the market, Perry and I both agreed that somewhere in the region of 40 percent of the total installed base will likely continue using on-premises software for at least the next several years (in my view, probably much longer than that), which gives Microsoft a pretty big reason to ensure that this software remains highly functional. Perry acknowledged that this is actuality and pointed to the large investment that Microsoft is making to ensure that both on-premises and cloud platforms are supported with great interoperability between the two.

Mobile Computing

One of the big changes that Perry has noted over the past few years is the change in focus of customers from a simple cost per mailbox discussion to how they can make their users more productive. In the past, customers wouldn't talk to Microsoft about new software unless that software could reduce their cost; now they want to know what features exist that will make their employees more productive in every sense of the word. The shift in IT results in a difference in the way that Microsoft thinks about products like Exchange. Part of that shift has been provoked by the explosion in the capability of mobile devices and the networks that allow people to remain perpetually connected to services such as email.

I asked about Exchange ActiveSync (EAS) and its future, because not much seems to have changed in the past few years. Perry made the bold assertion, "I don't think that Apple would have been so successful with the iPhone against RIM in the enterprise if we hadn't created the EAS ecosystem. It's theoretically possible that Apple would have built a great client (for Exchange) using other protocols, but it's unlikely that they would have invested the necessary resources on their own. There's just no way that they [Apple] would have been able to engineer a MAPI client [to connect to Exchange]."

It's certainly fair to say that Exchange ActiveSync (EAS) is the de facto standard for mobile device connectivity to Exchange. Early versions of the iPhone were limited to POP3 and IMAP4 connections, but once Apple had licensed EAS, iPhones certainly proliferated inside large corporations. There's no doubt that the advent of the iPhone, followed by Android and Windows Phone devices all using EAS to connect to Exchange, marked the start of RIM's decline as fewer companies could justify paying for expensive BlackBerry Enterprise Server (BES) licenses when Microsoft provided EAS free from Exchange 2003 SP1 onward.

Is this a case of superb business acumen on the part of Apple to figure out that EAS was the best method of communication, or a case of blessed serendipity when everything came together at the right time to benefit Apple? Perry acknowledged that his statement was an extreme way of making the point but said that there is no question that the evolution of the EAS ecosystem and the standard that was created has helped Apple.

Moving forward, Perry said that the arrival of much more capable mobile devices has made it harder for platforms that use a "sync-based approach" (i.e., ActiveSync) to keep up to date with the evolution of functionality on the server. People want to use the latest features on their new mobile devices, but device manufacturers can't update their clients quickly enough—and the EAS protocol doesn't support a lot of the features that are available to other Exchange clients. Being able to go to a repository such as the iOS app store and download code that exposes the full functionality of Exchange is a "dramatic game changer" that will drive user productivity. Moving the logic from clients to the server gives Microsoft the ability to deliver the "complete user experience" across the whole spectrum of mobile device form factors without the need for multiple vendors to create similar functionality numerous times. Another advantage is that Microsoft can take advantage of features that exist on each platform, such as the Safari browser on iOS. All of this comes together in a very real sense in Outlook Web App for Devices, released for iOS in July 2013.

The natural question then arises regarding whether Microsoft is going to decommit from EAS and concentrate on apps that are published by Microsoft and refreshed on a frequent basis. Perry's emphatic response was that although providing its own mobile clients will be "an increasing part of the story," Microsoft continues to invest heavily in EAS, which has more clients connecting via the protocol than ever before, and he claimed that "in the order of a billion mobile devices ship with EAS on them annually."

Perry also pointed out that Microsoft expends a lot of time to encourage EAS licensees to do the right thing when they implement EAS inside their code and that the Exchange group has also addressed some "mistakes" that have been made in EAS and improved the ability of the server to resist the effect of badly behaved clients. He noted that the sheer size of the EAS ecosystem means that the impact of any bug is "just huge" because it's felt by literally hundreds of millions of users and that Microsoft takes the responsibility of delivering a highly reliable and robust EAS protocol very seriously.

At the same time, Microsoft will dedicate the necessary resources to satisfy the need for highly functional mobile clients that can take advantage of the latest browser technology running on mobile devices. Perry said, "The line has blurred when it comes to primary computing devices," meaning that Microsoft must deliver a good experience on high-end smartphones and tablets. The discussion about how best to provide a great mobile experience continues on an ongoing basis with Apple, Samsung, and other vendors, as well as with Microsoft's own Windows Phone development team. Perry noted that "it's not just email clients anymore; there's also the ability to use Exchange Web Services to interact with the server in interesting ways. The amount of innovation and investment that's going on there is just huge. It's a tremendously exciting space."

My interpretation of what's happening is that EAS will continue as the lowest common denominator for mobile device connectivity to Exchange (albeit capable of enabling highly functional clients) in tandem with a set of browser-based clients that expose more of the server functionality. Microsoft can push new releases of these clients out through app marketplaces far more quickly than they can get features enabled by mobile device vendors through EAS extensions. Given the variety (not always good) of EAS-enabled client applications implemented across Windows Phone, Android, and iOS platforms, I think this is a reasonable approach—but we'll have to wait and see how OWA for Devices delivers over the next few years.

In Closing

Perry and I discussed the need for better email security and end-to-end encryption in light of the PRISM revelations and the requirement to protect the content of messages as they pass between companies. Perry said that the "silliness of SMTP being sent in the clear will be fixed" and that Microsoft will work with partners to resolve the problem in a standards-based way. Although no firm date can be put on when this will happen, it should be reasonably soon. No doubt we will all have to upgrade our software (hopefully through an incremental release) to get this level of protection.

We finished up by talking about the upcoming Microsoft Exchange Conference (MEC), taking place in Austin, Texas, in early April. Perry is looking forward to MEC because he believes that the Exchange team will be able to lay down some big markers to disprove the oft-asserted wisdom of many so-called experts that email is going away sometime soon. He wants to be able to show that a server like Exchange can act as the fulcrum for many forms of natural and effective collaboration. It will be interesting to see how the MEC agenda evolves to fulfill the promise contained in his words. We shall see in time!

Follow Tony @12Knocksinna