Compliance and hybrid problems loom as Microsoft plans to keep every deleted item in Exchange Online

The Office 365 Roadmap item telling us that Microsoft is "Removing the Deleted Items Retention Period" might cause some brows to furrow, especially when you read the description:

"The default 30-day retention period of deleted items folder on an Exchange Online mailbox will now be removed.  This means the user no longer has to worry about their deleted items folder automatically deleting emails every 30 days, but instead they can choose to empty the folder at their convenience. The admin can set a limit through Exchange Admin Console and PowerShell if they want to set a default limit on the folder."

Update: a Microsoft blog post on this topic is now available.

In fact, what's happening is that Exchange Online is taking a new direction in how items deleted by users are managed. You might not care about this because the information conveyed in the roadmap is definitely in the “quite nerdy” category. Well, very nerdy. But important all the same, if you care about Office 365.

Exchange Online applies the default MRM policy automatically to new mailboxes as they are created. Things are done differently in on-premises Exchange as retention policies must be explicitly applied to mailboxes by an administrator. The idea of applying a retention policy to all mailboxes is that they can be managed by the Managed Folder Assistant (MFA) and don’t become a dumping ground for unwanted old messages.

The default MRM policy contains a set of retention tags that instruct the Managed Folder Assistant (MFA) how to maintain mailboxes by removing or archiving content. Up to now, a retention tag has been used to control the Deleted Items folder by removing items from that folder after they are 30 days old. The items go into the Deletions sub-folder of the Recoverable Items folder from where they can be recovered for a further 14 to 30 days, depending on the deleted item retention period configured for the mailbox.

But here’s the rub. It turns out that many Office 365 users are extraordinarily forgetful and delete items when they shouldn’t. And because Exchange Online runs with the full benefit of native data protection and cares not a jot for backups, once an item is removed from its database following the expiry of the deleted item retention period, it’s gone for good. No amount of huffing and puffing at Microsoft will bring the item back. It is a dead item. Bereft. Gone. Departed. No more.

All of which has led, or so I am told, to a great deal of shouting at Microsoft support personnel, who have the great pleasure of being able to communicate news about the non-recoverable status of the departed items to their owners. And because support staff deserve protection from the words they hear in response to the news, Microsoft has taken the decision to update MFA so that it will no longer clean out the deleted items folder.

Apparently the change is not yet effective within Office 365 and will only become effective four weeks after Microsoft makes a formal announcement (which might be imminent). To experiment with what will happen when the change goes live, I disabled the Deleted Items retention tag in my tenant, left things alone for a few days, and then checked to see what had happened.

A good way to find out what MFA is doing is to run the Export-MailboxDiagnosticLogs cmdlet to check the properties updated when MFA processes a mailbox. In this truncated output, we see that the ElcLastRunDeletedFromRootItemCount property reports that only 53 items have been deleted from mailbox folders. This is a relatively small number of items to remove from a busy mailbox if the Deleted Items folder was being cleared out regularly. 

[PS] C:> Export-MailboxDiagnosticLogs –Identity TRedmond –ExtendedPropertiesRunspaceId  : e75673c0-026d-4e1d-af28-fa0f340778b3MailboxLog  :                               ElcLastRunDeletedFromRootItemCount            53                                ElcLastRunDeletedFromDumpsterItemCount            0                                ElcLastRunArchivedFromRootItemCount            13                                ElcLastRunArchivedFromDumpsterItemCount            154                                ELCLastSuccessTimestamp            16/04/2015 14:02:23          LogName     : ExtendedPropertiesIdentity    : TRedmondIsValid     : TrueObjectState : Unchanged

However, you would have to know the characteristics of a mailbox to be able to understand the information reported by mailbox diagnostics and to understand what the impact of the change will be over time. To get the necessary data, I observed the total number of items in the Deleted Items folder and the Clutter folder over a week (data was taken at the same time every day). I chose Clutter because I have applied a retention tag to this folder to remove items after 30 days, so if items are not accumulating in the folder I know that MFA has processed it. The fact that so many items are directed into Clutter is a pleasing indication that this feature is working well for me.

The data is as follows:

The number of items in the Deleted Items folder increased by 3,014 over the seven days. This might not be unusual if the user deleted a lot of items from their folders over the period to clean things up, but because this is my mailbox, I know that these are just regular deletions. I also know that the number of items in the Deleted Items folder has been in the range 15,000 – 18,000 range for the past two years because it's been kept under control by MFA.

The Deletions folder is a sub-folder under Recoverable Items where items are moved when the Deleted Items folder is emptied or an item is removed from a folder by MFA and is "temporarily recoverable". The fact that its number varies is indicative of the processing of the Clutter folder as the retention action in the tag that governs the Clutter folder moves items into Deletions.

In practical terms, what does this change mean? Well, the most obvious thing is that the contents of the Deleted Items folder will continue to grow unchecked. At the rate that items are accumulating in my Deleted Items folder (431/day), the folder will grow to hold an additional 157,315 items annually, or nearly a million items in six years. Your numbers will vary depending on the traffic flowing into mailboxes, but it's reasonable to assume that email traffic will continue to grow so an increase is more likely than a decrease.

Users probably won’t care about the accumulation – and if they even notice. Items will not be removed from the Deleted Items folder unless the user takes the decision to empty the folder themselves. In which case it’s their fault if they lose anything.

There isn’t much difficulty on a practical level for the service either. Office 365 has plenty of storage available to hold these swelling mailboxes. The issues that come into play are more in the compliance area where companies are concerned that too much information is retained online. If you're in this situation, then you need to look at the retention policies that are in place and decide whether they need to be adjusted. Those who running a hybrid deployment will also have to look at this matter to ensure that the same compliance guidelines are enforced across both platforms.

I think Microsoft is taking this approach because they believe it will reduce support calls. It probably will. Mailboxes are so large these days that users can afford to accumulate deleted items for years before their mailboxes explode and run out of quota. However, mega-large Deleted Items folders will impact OST performance as all these items will be synchronized down to the local cache. Outlook 2013 can control OST synchronization to a certain extent; older clients will suffer.

Apparently the formal reason for the change is that it will increase customer satisfaction. Hmmm.... It's true that not removing deleted items will likely be popular with small tenants but not so much with enterprise customers, who tend to care about aspects like compliance a tad more.  Tenants who disagree with the notion of a never-emptied Deleted Items folder can reverse the change by creating a new retention policy that includes a tag to process the Deleted Items folder and applying that policy to mailboxes. Or rename the Default MRM Policy to let MFA know that it can continue processing the Deleted Items folder.

I believe that items go into Deleted Items for a reason – they’re finished with and can be removed. Keeping deleted items around for 30 days just in case a mistake was made is surely enough for any reasonable individual to make up their mind whether an item is required. At least, I think so, but the evidence indicates otherwise.

Follow Tony @12Knocksinna