OWA bug compromises item preservation for litigation and in-place holds

We live in a world of competing social fora where the decision as to where to look for information can become exhausting. Facebook, TechNet forums, Twitter, Google+ and blogs compete for our attention. Duplication confuses and obscures and information always needs to be filtered, assessed, and put into context. It’s a whirl of data.

And then there’s Yammer. Now, I am not known to be an ultra-fan of Microsoft’s enterprise social networking platform but it has its loyal adherents who like it. Despite grand pronouncements from people who might know what plans exist, no great evidence of integration between Yammer and Exchange is yet available. And before anyone leaps to complain, let’s make it clear that the ability to send messages from one product to another is not integration. It’s not even a start.

But while I await a more perfect future, I do keep an eye on the Yammer Exchange IT Pro group, and noticed on August 20 the rather startling discovery by Vasil Michev, an Office 365 support engineer at HP, where he reported that he had been able to use Outlook Web App (OWA) to circumvent Exchange’s litigation hold mechanism (the problem was also discussed in a TechNet forum).

As you might recall, when a mailbox is placed on litigation or comes under the control of an in-place hold, Exchange is supposed to maintain copies of items subject to the hold if the user attempts to delete them from the mailbox. The retained copies are held in the Deletions sub-folder of the Recoverable Items folder away from the prying eyes of the user while still remaining indexed and therefore discoverable through eDiscovery searches.

Initially, Vasil reported that he had been able to move a complete folder from a mailbox that was under litigation hold to a shared mailbox without details of the move being preserved. Once the items are moved to the shared mailbox, the user can simply delete them and clear the Deleted Items folder to remove all traces of the items from the server. In this scenario, the user has full access permission for the shared mailbox, but it’s also possible if the user has delete permission for the target folder in the shared mailbox.

Moving without preservation should not have been possible because moving a folder from one mailbox to another is essentially a question of deleting the items from the source mailbox and recreating them in the target. Those delete operations should have been detected and the items retained but they were not. This is a serious issue for any company that is subject to a legal discovery action where a judge has ordered email to be retained.

The key thing was moving a complete folder because OWA correctly processed the movement of individual items and retained copies in the source mailbox. Vasil reported the problem as occurring in an Office 365 tenant. I checked it out against Exchange 2013 CU6 and discovered the same issue. In this screen shot, you see a folder called “Contoso” that contains 14 items.

I then moved the folder to a shared mailbox and looked again. You can see that the Contoso folder no longer exists in the mailbox but that the count of items in the Deletions sub-folder has not changed (it’s still 18).

The problem affects both Exchange Online (Office 365) and all supported versions of Exchange 2013, including Exchange 2013 CU6.

Vasil also reported that you can grant access to another user to a folder in your mailbox that holds items you want to eliminate and have them delete the folder. Once again, the deleted items are not preserved by the hold.

The key elements that combine in this issue are a) OWA, b) some element of delegation involving either a shared mailbox or delegated access to a user’s own mailbox, and c) deletion of complete folders rather than individual items. As far as I know, there is no known scenario where a user can remove an item within their mailbox and it will not be preserved if the mailbox is on hold.

Vasil reported the problem to Office 365 support. After reproducing that the issue existed for CU6, I let the Exchange product group know that the issue existing for both cloud and on-premises versions on August 22, four days before Microsoft released Exchange 2013 CU6.

Microsoft continues to investigate the root cause and has released KB2996477 to acknowledge the problem. One decision already made was not to hold CU6 because their engineers needed more time for investigation and to decide on an appropriate long-term fix.

Both good and bad exists in Microsoft’s decision. On the one hand, holding CU6 up for an indeterminate period when the code was ready to go and contained important fixes for customers (like the public folder scalability enhancements and the HCW fix) is an unattractive option. On the other, you could argue that fixing a known issue that affects compliance is a bad thing and that Microsoft should have held CU6 to fix this issue.

Holding an update is not a matter of slipping in a quick fix and then releasing the code. That kind of slip-shod engineering would be a recipe for a quality disaster. The different ways that this fault can be exploited mean that Microsoft has to investigate, debug, fix, and test multiple scenarios. Regression testing has to be done against both Exchange Online and on-premises versions to ensure that the fix doesn’t introduce other problems for OWA or any other component. It’s reasonable to anticipate that developing and testing the fix will take several weeks and this delay would affect customers who were waiting for the CU6 fixes.

It’s also fair to point out that no one had noticed the problem until Vasil came along and tested what was happening when OWA moved folders around (why was this not tested by Microsoft?). I guess you could say that everyone assumed that Exchange’s compliance features just worked because the deletion of individual items is done correctly. However, you can equally argue that the news of this problem will come as a complete shock to customers who are currently governed by legal discovery actions and now realize that they cannot guarantee retention of email. That is, of course, unless they have been using Outlook because the problem is not seen in that client or any mobile client.

Microsoft has invested heavily in giving Exchange a wide range of compliance features in the last two releases. It’s disappointing to discover that OWA has had this flaw for some time. Let’s hope that Microsoft fixes the problem fast, first in Exchange Online and then for on-premises customers (probably in CU7).

Update September 15: Microsoft has confirmed that they have fixed the bug for Exchange Online and the fix is now rolling out to Office 365 customers. The on-premises customers will get it later. My guess is still that it will be in Exchange 2013 CU7.

Follow Tony @12Knocksinna