Exchange 2013 CU6 hybrid and co-existence bugs cause administrators to despair

If you are a regular reader of this blog, you can’t say that you haven’t been warned to test newly released cumulative updates before you deploy them into production. When I wrote about Exchange 2013 CU6, I said “… every cumulative update has its own quirks that only come to light after the update is exposed to the full light of widespread deployment.” We now know about two significant issues that don’t affect every customer but are serious for those who encounter either flaw.

The first involves the way that Exchange 2013 CU6 and Exchange Online work in hybrid configurations (when some mailboxes on-premises and some in the cloud). We knew that a change made in Office 365 compromised the ability of the Hybrid Configuration Wizard to work properly in CU5 (and all previous versions) and had hoped that CU6 would provide a cure. But as MVP Jeff Guillet explains on his blog, CU6 brings its own set of hybrid imperfections to the table. The core issue is that “With CU6, admins can no longer use the Exchange Admin Center (EAC) to create new Office 365 mailboxes, move mailboxes to Exchange Online, or create In-Place Archive mailboxes.” Phew!

Professionals who do a lot of work with hybrid configurations, such as the splendidly named Michael Van Hybrid, are literally tearing what little remains of their hair out about the problems with hybrid connectivity. Michael is due to cover the topic of hybrid configurations at Exchange Connections in Las Vegas later this month and the last that I heard is that he plans to base his demos on Exchange 2013 SP1 because that contains the last reliable version of HCW. Microsoft is currently investigating and we’ll have to see how this situation unfolds. [Good luck to MVH on his move to the U.S. to take up a new position with ENow Software Inc.]

Update September 1: Microsoft has released a script to fix the bad configuration file. See this post for details.

One problem that customers who operate hybrid environments (or who use Exchange Online Archiving (EOA)) face is that Microsoft expects them to “maintain currency on Cumulative Update releases.” In other words, you’re expected to apply new cumulative updates to on-premises servers to keep them within the supportable window as set by Microsoft (now that CU6 is available, that set is CU5 and CU6). The logic underpinning this requirement is that Exchange Online is being continually refreshed within Office 365 and so it makes sense to make sure that the on-premises servers run approximately the same version. Making such a requirement is all very well until the hybrid connectivity components fail to deliver the goods.

I hear that the root cause of the hybrid issue is some bad settings in a configuration file. According to comments posted on the EHLO blog, Microsoft has a solution "in the works" that appears to be coming soon "even over a holiday weekend."  Getting a fast fix would be a good thing but that does not answer the question as to why a cumulative update was allowed to make its way through a protracted release cycle to get to customers with such a flaw.

The other Exchange 2013 CU6 problem occurs when CU6 is deployed inside an organization that contains Exchange 2007 servers. It seems that the Store worker process for a databases crashes after information is received from an Exchange 2007 mailbox when an attempt is made to access a delegated mailbox on the CU6 server. In turn, the crash causes the database to failover (within a Database Availability Group) or unexpectedly dismount (on a standalone server). Neither outcome is good.

Microsoft has published KB2997209 to describe the problem and a hot fix is available for the dismounting database problem from Microsoft Support. You need to deploy the patch alongside CU6 if you want CU6 to co-exist happily with Exchange 2007. The alternative is to watch your Exchange 2013 databases dismount and failover with ominous regularity, which doesn’t seem like a great scenario.

Issuing essentially new versions of Exchange 2013 every quarter is a difficult task but that’s what Microsoft has set out to achieve in the servicing model that is now in force. Testing is especially problematic because of the size of the product, its complexity, and the number of deployment scenarios that Exchange accommodates. After glitches in CU1 and CU2, Microsoft chose to delay the release of CU3 in an effort to achieve better quality and more reliable updates. It seems like they need to slow things down once more. Despite many protestations that quality is important, the fact that CU5 and now CU6 have been afflicted by what seem to be highly visible bugs brings the question of testing into focus once again.

It’s hard to argue that Microsoft should not have picked up and fixed these problems before they shipped CU6. No doubt cynics will say that Microsoft is only interested in testing the code that operates inside Office 365 and that interoperability with older versions is unimportant. But even if Microsoft was so silly as to cut back on the testing that enables customers to move to the newest version of Exchange, I can’t see how it makes any sense for them to de-emphasize testing for hybrid connectivity. Having a good hybrid story is critical in convincing customers to embrace the cloud and it’s also a major competitive advantage against Google, which doesn’t have the same hybrid capability. But in saying all that, if your code is flawed then customer confidence weakens along with faith in Microsoft’s ability to deliver.

Reporting on a continual stream of flaws in Exchange gives me no satisfaction. I’m sure that seeing all these posts and adverse commentary is very frustrating to the Exchange product group too. The solution is in their hands – better testing, better quality, and better software. Let’s hope that action is taken to assure that this happens soon.

Follow Tony @12Knocksinna