Office 365: a kind of dogfood environment for on-premises Exchange

You might have been puzzled by the statement “If you want clues about what’s coming in the next version of Exchange Server, keep an eye on what’s happening in Office 365” contained in Microsoft VP Perry Clarke’s recent attempt to pacify the on-premises community. Where does one look for clues in Office 365? Does Microsoft say where the new bits are? And would you know a new feature if you saw it?

It’s a fact of software engineering life that developers have test-bed systems that are used to validate software builds. Famously, Exchange has had its “dogfood” environment for many years. To reflect today's platforms, the Exchange product group runs both on-premises and a cloud dogfood systems and are diligent at consuming its own dogfood by using these servers to host their mailboxes. The idea, of course, is that developers should be able to recognize problems in their code if bugs surface when the developers use new builds to get work done. Think of it as a massive debugging party.

I view the Office 365 datacenters as the dogfood environment for what eventually will appear in an on-premises Exchange update. This isn’t to say that Office 365 runs beta Exchange Online code because code is validated before it is introduced into “the service”. Rather, Office 365 runs code for weeks or months before the code is packaged and provided to on-premises customers in the form of a cumulative update (or service pack). I see lots of goodness here because a solid shake-down by running code used by millions of cloud mailboxes seems like an excellent way to find bugs. Apart from the joy of stressing code at the edge of the performance envelope, running code in a datacenter is certainly a better way than exclusively relying on the suite of automated tests that are used to test basic functionality and that new code has not introduced any regression, which is what used to happen.

Sometimes Office 365 is not a good test platform, largely because Microsoft doesn’t deploy code onto an Office 365 server in the way that on-premises installations do. For example, automated processes strip servers down to bare metal and then reinstall the operating system (complete with any hot fixes and required patches) and the latest version of applications before they go into production. There’s no notion of applying updates to servers like you might when updating a server from Exchange 2013 CU2 to CU3. I assume that wipe and reload is an easier operation to automate for datacenter operations because it always results in a known environment. And because Office 365 looks nothing like your standard on-premises environment, it follows that some processes executed in an on-premises environment such as the application of security patches do not occur in the same way, which is one of the reasons why the MS13-061 fiasco happened.

It’s also fair to say that Office 365 is a much more regimented implementation than anything you will find on a customer site. Third party software is not installed in the same way as in an on-premises deployment (instead, third party vendors like BlackBerry offer integrations with Office 365) and are therefore not tested against new Exchange on-premises code. The kind of integrations that customers do would not be tolerated and probably won’t work in a datacenter at scale because they are looser and less automated than the playbook Microsoft follows. It therefore follows that you cannot assume that code that works for Office 365 will function perfectly when introduced into the imperfect, non-Microsoft centric, and heterogeneous environments that exist in so many customer deployments. This is why bugs appear in on-premises deployments – even though APIs (like the EMS cmdlets and EWS) can be tested, specific conditions that arise in customer environments have not been tested for during development or not exercised in the datacenter.

Accepting that Office 365 is not a perfect match (or near a perfect match) for any known customer environment, a huge dividend can still be gained in terms of bug identification and elimination by running code in Office 365 for weeks before on-premises customers get hold of it. And processes developed by Microsoft for use in their datacenters do find their way into the on-premises product. Not all of the processes, but certainly good examples exist such as Managed Availability and workload management. Microsoft’s focus on driving I/O down so that JBOD storage is more than a viable option for Exchange has also benefited from the need to operate datacenters at scale and at a low price point.

Some goodness is present in the current arrangement. Office 365 customers get “evergreen” software (even if they don’t want it, or realize that they are getting it), on-premises customers should get higher quality software (a problem recently, but signs are that quality is improving), and Microsoft’s marketing fraternity are able to take the battle to Google because Office 365 can introduce new features at the same rapid cadence that seems to have become the norm for cloud-based email services.

Back to the EHLO post, you might wonder how to track Office 365. It’s difficult, because even though more recent code is running, features might not have been “lit up” (made visible to end users). Thus, at the time of writing, my Office 365 tenant domain is running version 820.5 (run the Get-OrganizationConfig cmdlet and check the value of the AdminDisplayVersion attribute) whereas Exchange 2013 CU3 is version 775.38. According to one of my Microsoft sources, a new Exchange build is produced five times weekly. Given that the version number running in Office 365 is some forty-five versions ahead of CU3, you can speculate that Exchange Online is about nine weeks ahead of its on-premises cousin when this snapshot was taken. The gap flexes over the development cycle but this is a reasonable indication of the gap that exists between the respective software builds. In this instance, Office 365 is running a variant of what will eventually be Exchange 2013 CU4 (or Exchange 2013 SP1), which is what you would expect.

Clues are visible and evidence can be gathered to show that Office 365 does indeed run newer software. But apart from the notes on the Office 365 "service upgrades blog", it is possible that you will see little evidence of new features until they surface in Outlook Web App or the Exchange Administration Center (I am too lazy to go looking for a new EMS cmdlet). No doubt the new features will light up and delight me in due course. Until then, I’m clueless.

Follow Tony @12Knocksinna