Exchange message tracing extended to 90 days in Office 365 - what about the on-premises version?

Listening to a call about the recent decision by Microsoft to support message tracing in Office 365 for up to 90 days, my mind wandered (as it does). Not, I hasten to add, because of any inadequacy on the part of the presenter, but more to consider the nature of message tracing and why we need it at all.

Message tracing (or tracking) has been part of Exchange for a very long time. This is code for “so long that I can’t remember.” I therefore consulted a copy of my book “Microsoft Exchange Server: Planning, Design, and Implementation” covering the wonders of Exchange 4.0, published in August 1996, (the ISBN is 1-55558-162-5 for those interested in trivial pursuit - and I am amazed that copies can still be bought) and found (page 347) the text:

“Users send messages at the drop of a hat, but sometimes those messages don’t get through and when this happens it falls on the head of the system administrator to try and find out just what happened to the message. Was it delivered to someone else? Did a gateway reject the message? Maybe the message is stuck somewhere waiting for a connection to be established to a remote system.”

Times haven’t changed much and I suspect that much the same text could be written to describe the need for a solid message tracking facility in both Exchange 2013 and Exchange Online today. Users still wonder sometimes whether a message really did get through. Gateways are less important now that SMTP is the messaging lingua franca of the Internet, but messages do get stuck sometimes. And users still ask pertinent questions of system administrators when they want to find out what happened to their email.

Exchange 2010 and Exchange 2013 both include the Delivery Reports option in Outlook Web App that is designed to allow users to answer these questions themselves. Delivery Reports use the same message tracking logs as administrator-initiated tracking requests and work well in many circumstances. Delivery Reports also include the ability to search by message subject, a useful feature that is required because that’s how many people recall the particular message that they want to know about.

Delivery Reports are not designed to provide a definitive trace of a message’s path. In fact, to know exactly what happened to a message you might have to consult some additional sources of data such as the SMTP protocol logs on a server in additional to message tracking logs.

In any case, the point is that message tracking has been part of Exchange forever and the more recent versions allow users to do some of the investigative work themselves. Administrators are then left with the more (exciting) task of interrogating the message tracking logs generated on servers whenever the need arises to determine exactly what happened to a message. This might be because mail is not flowing but it might also be because you want to check that a transport rule is having the desired effect on messages. Delivery Reports don’t show this kind of detail and you have to get down and dirty to trace a message path in fine detail.

Getting back to the new feature supported by Office 365, the interesting thing about changing the trace window from 7 to 90 days is the amount of data that might have to be examined to trace a message in that time. If you run a large Office 365 tenant domain that supports thousands of busy users, the number of trace entries could be huge (this TechNet article explains all the entries that can be generated for a message).

Message tracking logs are flat files that are indexed and accessible through the Transport Log Search service. Even so, the amount of data that a search might examine means that a different approach is necessary. Microsoft hasn’t revealed what repository they are using on the back end to store historical message tracking log data. My guess is that SQL is involved and that any data older than 7 days is automatically loaded and stored in an SQL database until its retention period (90 days) expires Then again, seeing it's Exchange that's involved, it might be an ESE database. In any case, a new cmdlet called Start-HistoricalSearch supplies the magic link between Exchange and the data.

By comparison, message tracking logs are retained for a default period of 30 days on an Exchange 2013 server. You can set a different period by running the Set-TransportService cmdlet. For instance, to increase the retention period to 90 days:

Set-TransportService –MessageTrackingLogMaxAge 90:00.00.00

TechNet (which never lies) states that the maximum retention period is 24855.03:14:07 or just over 68 years. This seems a tad excessive to me, but you never know…

The interesting thing about Start-HistoricalSearch is that it supports parameters specifically intended to search for message tracking log entries related to actions performed by transport rules and Data Loss Prevention (DLP) policies (a specialized form of transport rules). I imagine that these features are used to support the DLP analysis reports that Office 365 tenants can access through the Exchange Administration Center (EAC).

[Note to Microsoft: It would be nice if on-premises customers could access DLP analysis reports from EAC too]

Microsoft has amended the “message trace” option in the “mail flow” section of EAC so that it switches UI between searches performed up to 7 days and thereafter. Searches up to a week are performed immediately and the results are returned on-screen. Custom searches that cover a period greater than 7 days force EAC to expose some additional fields to gather information about whether to include routing details in the report and if inbound, outbound, or “all” events are required. Be sure to click on the option to include message events and routing details as otherwise you'll receive a much simpler and less useful report. Finally, you have to enter the email address to receive the report (this has to be an address belonging to one of your domains as known to Office 365).

[Note to Microsoft #2: It would be nice if on-premises customers were able to execute message tracking searches through EAC too. Being forced to run Search-MessageTrackingLog through EMS is a fine way to encourage people to learn PowerShell, but it can be tiresome. I know you had a lot of work to do for Exchange 2013 SP1, but this is the kind of thing that ticks people off.]

After the parameters are gathered, EAC submits the search to run in the background. I think this is reasonable given the amount of data that might have to be analyzed. After a while (which depends on the workload running in the Office 365 datacenter that hosts your tenant domain), the trace completes and the results will be available. You can view the progress of the traces and look at the results from previous traces by clicking the "View pending or completed trace" link.

Alternatively, you can opt for the results of a trace to be sent to an email address. Once the trace completes, the nominated receives a message containing a link to the retrieved data. The information is provided in CSV format and so can be analyzed using a tool such as Excel. It can take a little while to understand the contents of the data returned from a message trace so it's a good idea to practice by reviewing the data from a message trace using well-understood parameters. For example, some messages sent or received by your own mailbox over a small period where you have the messages available and can correlate them against the data. Unless you're used to interpreting message traces, the information contained in the downloaded file is cryptic. Here's a quick taste:

"1/16/2014 7:05:23 PM","10.242.109.11","AM3PR04MB433.eurprd04.prod.outlook.com","10.242.109.11","AM3PR04MB433","08D0D9963A8B8050","","STOREDRIVER","RECEIVE","0","","0248780b-3ec7-4a68-738f-08d0e1050b32","[email protected];michael@[email protected];[email protected];[email protected];[email protected]","To;To;To;To;To","177559","5","","","RE: Death to the List... Long Live Yammer Office365MVP group","[email protected]", "[email protected]","03I:","Originating","b662313f-14fc-43a2-9a7a-d2e27f4f3478","86.43.175.31","10.242.109.11","S:MailboxDatabaseGuid=4bf42761-7b04-4f22-9caa-46fe904b4fee;S:ItemEntryId=00-00-00-00-7E-EC-82-E9-14-DC-7C-4E-B9-2D-68-AF-15-61-67-AB-07-00-5E-F4-2B-B0-2D-CD-9F-4C-AE-D6-E3-A2-F5-48-0A-7D-00-00-00-DA-51-C8-00-00-23-93-68-96-5F-66-C8-40-85-4E-76-6C-EE-82-4F-09-00-00-AA-C9-30-83-00-00;S:DeliveryPriority=Normal"

Opening the data with Excel allows you to filter entries more efficiently and to sort by timestamp, which is a good thing to do when attempting to follow the path of a message. Unfortunately some of the entries returned by a message trace use a timestamp in a different format. Oh well, things can't be perfect. Even with Excel, don't expect the kind of elegant presentation of a message's path as displayed by EAC when a message trace is performed against online data. It would be nice if a program was available to make sense of message trace download files but for now it's up to you to make sense of the trace entries.

Up to 5,000 entries are returned after which the search is truncated, which is a good reason to be as precise as possible when constructing message trace queries. Reports remain available online for 10 days

For those who prefer to use EMS, it’s easy to invoke a historical trace. For example:

Start-HistoricalSearch -ReportTitle "Message Trace Search" -StartDate 1/1/2014 -EndDate 2/1/2014 -ReportType MessageTraceDetail -SenderAddress [email protected] -NotifyAddress [email protected] –DeliveryStatus Delivered

Eighteen years ago we had no concept of the amount of message tracking information that would be generated by user activity and the agents (like those that perform message moderation or execute transport rules) that operate in the message pipeline. We also had no concept of the kind of massive multi-tenant environment that runs across multiple datacenters to enable a cloud service like Office 365. Even so, the basic need to answer user questions about what happened to their email still exists. Isn’t it nice that some things never change?

Follow Tony @12Knocksinna