Exchange 2013 SP1 suffers late-breaking bug that affects third-party products

Like many others, I eagerly anticipated the arrival of Exchange 2013 SP1. The first service pack of any Microsoft server application has a special resonance with the installed base, most of whom are reluctant to deploy the RTM version of any software. Far better, the adage goes, to let others blaze a trail to glory. Wise administrators wait until software settles down, bugs found and fixed, features completed and enhanced, and sufficient knowledge exists to represent “best practice” for deployment and operation. By happy coincidence, such a confluence appeared to come together when SP1 made its debut just a week ago.

I was therefore dismayed to read a note from Microsoft to the Exchange MVPs notifying us that a hotfix was to be released to fix a problem with third-party transport extensibility agents that prevented the Transport services restarting following an upgrade to Exchange 2013 SP1, or indeed to prevent the installation of the agents with Exchange 2013 SP1. The fix is now available from Microsoft.

Transport is the one place in Exchange that you can absolutely guarantee a message will pass through on its outbound or inbound journey. That’s why transport rules are so effective in applying things like corporate disclaimers or why Microsoft has used transport rules for its Data Loss Prevention feature. Third parties build agents using the Transport Agents SDK to integrate products that need to interact with messages before or after they leave mailboxes. Anti-malware or anti-spam products are common examples that use transport agents.

The agents depend on .NET Framework code provided by Microsoft to integrate with Exchange. Apparently, a late-breaking fix applied during Exchange 2013 SP1 development caused the issue. A mistake was made in a fix to address a problem (the MS13-061 security hotfix requires the installation media to uninstall) and introduced a badly formatted XML comment line in two assembly redirection policy files. These file allows code to work against different versions of Exchange. The result is that any attempt to load the transport agents into the Global Assembly Cache (GAC) is rejected, which in turn made them unavailable to Exchange.

The problem affects any third-party product that uses transport agents to integrate with Exchange including well-known and widely used products such as TrendMicro Antivirus, Code Two, Symantec Anti-Virus, Exclaimer, and ORF anti-spam. Special credit must go to Peter Kansai of Vamsoft (the developers of ORF), who diagnosed and reported the problem on February 27 in the Exchange development forum. Reports of problems with the transport services in upgrades also surfaced in the comments to the EHLO blog post about SP1. On March 4, Code Two were the first company to provide full details of the required fix when they posted this article.

You won’t experience the problem if you upgrade a server that doesn’t use transport agents.

Fortunately, the fix – as is often the case with irritating problems like this – is simple. Microsoft has developed a PowerShell script to address the problem. Those who use a transport agent can incorporate the script into the installation process for Exchange 2013 SP1 or apply it afterward. The problem has also been corrected in the code base and will therefore not appear in the next cumulative update. I would prefer if Microsoft had rereleased SP1 with an integrated fix but understand that this would have been a slower and more complicated process.

The hopes of software engineering often founder on the twin rocks of expectation and dates. Expectation of new features and a high-quality SP1 release create pressure that is added to by the need to make set dates agreed to within Microsoft and with their partners. In this case, Microsoft actually delayed the release of SP1 to allow for better validation and testing, so the fact that this bug has emerged is especially hard felt by the development group.

Whenever problems like this appear, people leap to the conclusion that Microsoft development processes are incapable of shipping robust and reliable software. It is likely that critics will level four charges against Microsoft:

In summary, a late breaking bug that was reported after SP1 shipped has caused considerable embarrassment and reputational damage to Exchange. The good thing is that the development group has stepped up to the plate with early disclosure and swift response to address the problem. Whether this will have the hoped-for reaction from customers is yet to be seen. In the meantime, I hold to my opinion that Exchange 2013 SP1, flaws and all (there will be more bugs – such is the case with software), is worth deploying.

Microsoft is both blessed and burdened by the size of the Exchange on-premises installed base. The size of the base is a huge strength because of the revenues that it generates. The number of add-on products that run alongside Exchange make the base complex to fully understand and incredibly difficult to move forward. Microsoft’s natural desire for leadership in cloud services needs to be tempered by a little more care for those who elect to remain on-premises.

Follow Tony @12Knocksinna