Delve Analytics lets Office 365 users track (and maybe change) bad email habits

At the recent Dreamforce conference, Microsoft CEO Satya Nadella described Delve Analytics as “a new information discovery and knowledge tool to track your time, because that’s the valuable resource you have.” In other words, Delve Analytics analyzes information extracted from Office 365 to help people work smarter. It's an interesting application that is included in the Office E5 plan launched on December 1. Delve Analytics is also available as an add-on for the E1 and E3 plans for $4/month per user. It's an attempt to bring big data analysis to the world of the office to understand how people interact with each other and use tools such as email.

Another description often trotted out positions Delve Analytics as the equivalent of “Fitbit for the office” because like a health tracker, you gain if you pay attention to what to what you learn. I guess “Microsoft Band 2 for the office” does not trip off the tongue quite so smoothly. Let’s discuss what Microsoft has done in Delve Analytics to gather and analyze data gathered from the activities of Office 365 users.

Microsoft Graph is the Foundation

The first thing to clear up is that Delve Analytics has nothing much to do with “classic Delve”, the application that allows Office 365 users to access and locate information drawn from across the service. Both applications share a common foundation in that they use information drawn from the 40-odd signals gathered to record user activities such as sharing documents and creating meetings in the Microsoft Graph, but the only link between the two comes from the fact that Analytics is accessed as an option through the Delve portal. Because the Graph is the collection point for user activities, it doesn’t matter what client someone uses to interact with Office 365.

The concept underpinning Delve Analytics is that by observing and understanding how someone interacts with the components of Office 365, you can build up a picture of their day-to-day activities. You can also compare how they spend their time against an anonymized set of data drawn from other tenant users. Based on data drawn from across Office 365, Microsoft knows that the average office worker spends up to 20 hours per week working with email while senior managers will be glued to their keyboards for between 40 and 70 hours. Your data might vary!

Instead of attempting to be proscriptive on the subject of what constitutes a good work-life balance, Delve Analytics focuses on providing data for an individual to interpret. You might think that working for twenty hours per week outside the normal working day is a good thing because you need to interact with people in many different parts of the world. I might disagree and avoid any notion of working outside a 9-5 window. We’re both right because it all depends on the individual. If you get your work done one way and I get mine done another, we are both effective. The question then is whether you can be more effective and achieve the same results in a different way.

The privacy issue

All of the data used by Delve Analytics is available to users if they care to look. For example, it is possible (but tedious) to count the number of messages you create and send. You can also figure out how much time is taken to respond to other people within your company by looking at the timestamps in message headers and calculating the difference between when a message was sent and when a response was generated. The same is true when it comes to analyzing the number of meetings you attend, who also attends, the topics discussed, and the outcomes. You can assess whether meetings were effective, ineffective, or just so-so. None of the data used by Delve Analytics is invisible to users; it’s just that computer code and algorithms are so much more precise and consistent about analyzing raw data to arrive at conclusions. And unlike humans, who might remain interested in analyzing email for a day or even a week before they become bored and move on to other more interesting tasks, Delve Analytics is persistent in continuing to analyze the stream of data gathered through user activities.

Apart from checking timestamps and determining whether messages go or come from external or internal correspondents, Delve Analytics accesses no content that might reveal confidential information such as message subjects or bodies. All of the information used is available for all messages, even those that are encrypted or protected with Information Rights Management. In fact, it’s the same information used for other administrative purposes such as message tracking.

It should be obvious that if people don’t make maximum use of Office 365, the value of Delve Analytics is degraded. For instance, if you use Exchange Online but use network file servers instead of SharePoint Online or OneDrive for Business to store shared documents, the data collected in Microsoft Graph are incomplete and won’t represent an accurate picture of workflow within the organization. For this reason, any company considering using Delve Analytics must first assure itself that sufficient data is available in Microsoft Graph and Azure Active Directory (to understand organizational relationships) to make analysis useful. 

Using Analytics

It can take up to six weeks after you purchase and assign the necessary license to an Office 365 account before Delve Analytics becomes available to the user. This is not because Microsoft requires that amount of time to provision (or make available) the software to users. Instead, it’s because some preparatory work needs to be done behind the scenes to provide Delve Analytics with sufficient data. In effect, data is being gathered and analyzed about user habits to populate user dashboards with information that is based on recent activity. It’s much better to be able to show users a fully-populated dashboard that contains relevant and insightful information the first time it is accessed rather than showing a blank slate. Once everything is ready, the “Analytics” link will turn up in the Delve page for enabled users (Figure 1).

Figure 1: The Analytics link appears

The personal dashboard

Delve Analytics uses the data available in the Microsoft Graph to break down the time the user spent in different activities and presents the information in a personal dashboard. The dashboard is divided as follows:

The dashboard suggests goals for the time spent in meetings, working with email, focus hours, and after-hours activity (Figure 2). For example, the goal for email is to spend no more than ten hours per week whereas the goal for after-hours work is zero hours. The goals are customizable to match the ebb and flow of someone’s work. For instance, I set a goal of five hours for after-hours work. In fact, I know I work more than this, but a zero-hour goal was just unrealistic.

Figure 2: A summary of your working week

A good example can be seen in how work done outside the standard day is reported. Today, the notion of a working day is pretty fluid. Some people have a fixed schedule and never contemplate doing anything work-related once they down tools each evening. It’s more common to find “knowledge workers” are active at various times during the day, depending on the makeup of their team and the level of responsibility they have.

For instance, someone who works in an international company where colleagues are based in multiple continents invariably finds that conference calls are scheduled at times outside the traditional working day. It’s just a fact of life that has to be taken into account when you consider the out-of-hours workload reported in the dashboard.  The basic working day is deemed to be 9AM to 5PM, but you can customize this through the Calendar section of Outlook Web App Options.

A sliding six-week window of data is available and comparisons are available to show how time spent on different activities varies over time and against the average for other people  within the company (Figure 3). Remember that the company data is anonymized so you can’t compare yourself against a specific individual.

Figure 3: Comparison against company averages

The tyranny of email

Like many knowledge workers, I spend a lot of time working with email. The data reported by Delve Analytics focuses on how two aspects of email. First, how much time is spend reading and writing messages exchanged with other people in your Office 365 tenant. Second, “email etiquette”, which really means how quickly you respond to messages from other people and how quickly they respond to you. No assessment is made of the quality of the responses as everything is simply measured in terms of time. Unsurprisingly, it is usual to discover that more time is spent dealing with inbound messages than writing new messages and responses. 

Figure 4: My week in email

The way that Delve Analytics calculates the time spent dealing with email also contributes to the expected outcome. A relatively simple algorithm is used to calculate how much time you spend working with email. Instead of attempting to track exactly what happens with a message when it arrives in your Inbox or how much effort you put into composing a message, data based on observations of multiple large organizations indicates that an average of 2.5 minutes is spent dealing with an inbound message and five minutes when composing a message. A message is deemed to be read when its read status changes from “unread” to “read”. This applies no matter what folder the message is stored or client is used. In other words, unread mail processed by rules or the Clutter feature to a non-Inbox folder are also counted.  A new message is deemed to be completed when it is sent. Messages sent to people outside the tenant are counted.

Microsoft is well aware that the current algorithm is basic and can be inaccurate in some circumstances. However, it does work in general and will be refined over time to become smarter and more precise. Many iterations of code tweaking and testing are probably going to be required before the algorithm works well for the majority of users.

The data reported in the email etiquette section is skewed by the fact that the vast majority of my email is sent to external users because their response rate cannot be tracked by Delve Analytics. After all, the software can’t contact Gmail or another mail provider to ask what happened when a message arrived. On-premises users who belong to the same organization via a hybrid connection are also excluded because the Microsoft Graph doesn’t ingest signals from the on-premises Exchange servers.

The data shown for response times is based on “qualified recipients”, which means other users within the same Office 365 tenant who are addressed as TO: or CC: recipients either individually or as part of a distribution or Office 365 group. In addition, the users have to opt-in for their data to be included in Delve Analytics (more on this later).

Statistics about email sent to groups are retained for seven days while those relating to email exchanged between individuals are averaged over 30 days. The longer retention period for individual data is because it is also used to track whether you stay in contact with people who are deemed to be important to you. We’ll discuss this aspect in the “Working Relationships” section below. It can take up to 2 days before new data impacts the statistics reported by Delve Analytics.

Mastering meetings

We all spend too much time in meetings and possibly not to good effect. The calendar is a rich source of data for analytics because it represents blocks of formally scheduled time dedicated to specific topics. But meetings vary from a 1:1 discussion with a direct report or your manager to much larger group meetings whose only benefit is in an opportunity to grab some free coffee and donuts.

The time reported for meetings is based on calendar events that include at least one other participant (external or internal). The logic here is that events that have no other participant are personal. Only meetings marked as “busy” are counted, so meetings that are tentatively scheduled but not accepted are excluded because you probably never devoted any time to these events. Obviously, if you don’t add items to your calendar, those time slices are never analyzed. The only data used to review meetings are the start time, duration, number of attendees, and who attends (used to determine working relationships). Again, all of this information is available in the header of calendar notification messages.

The requirement for at least one other participant to be included means that many events that turn up in calendars are excluded. For example, I commonly create events for airline bookings so that the information is available in the calendar on my mobile device. Those events won’t be counted unless I add someone to the event, such as when my wife joins me on a trip and I add her to the event so that she knows when we’re flying. In the eyes of Delve Analytics, that’s a meeting that gets counted.

Focus time

Delve Analytics defines focus time as a two-hour block in the calendar during working hours that is free of meetings. The idea is that you use focus time to concentrate on specific activities that need to get done, such as creating a budget or working on a presentation. Studies have identified 90 minutes as a good average and the Delve developers decided to use a more aspirational two-hour block. An administrator can’t change this definition.

Like any measurement, focus hours need to be put in personal context. For instance, does a two-hour period spent over a long lunch count as focus time? On the surface, it might because you’re not in meetings or working on email, but on the other hand you might not be too focused during the meal.

After hours

The notion of working from 9AM to 5PM every day sounds archaic to many who work in IT who are accustomed to the need to perform systems maintenance and other tasks once other workers have left for the day. As noted earlier, it’s common for those who work on international teams to need to stretch the working day at either end to accommodate conference calls with other team members.

Delve Analytics measures how much time a user spends on late-night or early-morning activities as measured by meetings or working on email. You can process email during an after-hours meeting and clock up time for all measurements to seem like you’re really putting in the time. As before, the data used here comes from the calendar and email headers.

The importance of data

Delve Analytics can only measure the data that’s available to it. I found that it wasn’t good at interpreting who might be important to me because most of those who I correspond with for business purposes are external to my Office 365 tenant, such as clients and contacts working at companies like Microsoft. This underlines the fact that Delve Analytics is designed for the large enterprise rather than small companies where relationships flex and evolve over time.

Some activities are recorded in the Microsoft Graph but are not yet used by Delve Analytics. For instance, even though Skype for Business has become an increasingly important tool for business communication, the fact that you spend hours talking to an important collaborator is not noted. The time might be blocked off in your diary, but an important piece of the overall jigsaw is missing. Ad-hoc meetings that occur when someone notices that you are free and contacts you with Skype for a discussion are also ignored, even though the likelihood is that such meetings are potentially much more productive and important than those formally scheduled in the calendar.

Working relationships

The “Your Collaboration this week” section of the dashboard (Figure 5) displays information about the other users within your Office 365 tenant who you seem to collaborate with most often. In effective, this is an attempt to measure with whom you spend your working time. The data used are:

The calculation of how many hours you spent with key contacts is then adjusted based on the number of people who attended meetings that you participated in with key contacts. Clearly, a 1:1 meeting provides far more “face time” with someone than when you gather around a conference table with 20 others. In the same way, email exchanged between you and someone else is more personal and important in terms of maintaining contact than if you both receive a message sent to a distribution list where you might delete the message because you consider it unimportant whereas your contact thinks it critical. In this context, losing touch means that you haven’t emailed someone in a while or scheduled a meeting with them. “All caught up” means that I manage to keep in touch with all my important contacts, which seems like a good thing.

Figure 5: Intra-organizational collaboration

When you see “0 hours” reported by Delve Analytics, it essentially means that you respond to the other person immediately – or at least, within a couple of minutes. If someone is listed in the “losing touch” section (Figure 6), you can create a message or schedule a meeting to reconnect. You can also dismiss someone from the list, which is possibly an indication that you don’t value them very much! No external contact shows up here, so the dashboard is really reporting the effectiveness of your internal collaboration. 

Figure 6: The need to catch up

The Delve Analytics Outlook app

In addition to the personal dashboard, Delve Analytics includes an Outlook add-in that I call “MailTips on steroids” (the add-in is really just called “Delve Analytics”). The add-in is a small app that can be used with Outlook Web App, Outlook 2013, and Outlook 2016 to expose some information about messages that you send and those you receive.

Figure 7 illustrates the kind of information you’ll see for a sent message. The basic idea is that you can discover how many recipients have read a message you sent and how long it has taken them to respond over the course of the day. Understanding how recipients respond to messages can lead to insights such as “if I send my weekly report on Friday evening, it is often ignored by some of the recipients, but if I send it at 8AM on Monday, 95% of the recipients read it within an hour”. Some will find this kind of thing a tad creepy, but on the other hand, if you’re concerned about getting messages across within large organizations, it’s useful information to know. As with anything, the most important thing about any action is the outcome. If you get better results from email that is responded to in four hours rather than two, then go with the time that gets the best result.

Figure 7: Analyzing what happened to a sent message

I imagine that some will find the ability to understand what has happened with messages to be more interesting than others. Those who send messages and are happy to wait for a response will hardly consider it valuable to track their progress. On the other hand, it can be very useful to know how a message you deem to be important is being processed by recipients. The downside is that the add-on only works for messages circulated to internal recipients as no data is available for copies delivered to people outside the tenant.

To preserve the anonymity of recipients, analysis is only available for sent messages when they have five or more qualified recipients (you can’t change this threshold). Remember, a qualified recipient is another user within the Office 365 tenant, so at least five qualified recipients have to be present as TO: or CC: recipients for a message before analysis can be performed.

In addition, data is not available for messages for some time after they are received or sent. It would be unreasonable to expect anything else. After all, those who receive messages won’t necessarily process them immediately and it is only after an hour or so that the full picture of how a message was dealt with will emerge.

Interpreting data for received messages

The analysis available for received messages is straightforward and reports many messages have you and the sender exchanged over the last 30 days and how quickly each side responded to the other. To produce the analysis, select a message sent by another user in your Office 365 tenant, and click the Delve Analytics app in the app header. Figure 8 shows how the app running inside Outlook 2016 displays the information. 

Figure 8: Analyzing messages exchanged with a sender

The data used for message analysis excludes messages exchanged in the last 48 hours. In the case shown in Figure 8, two things leap out. First, there’s a large difference in the volume of messages received compared with those sent by the other person (me!). There can be many reasons why this might be so. For example, someone might be responsible for reporting the status of a project daily and the recipient might not always comment back, so again this underlines the importance of the user putting data into context as they are the only person who fully understands the traffic that flows between themselves and another user.

Second, the analysis shows that I am much faster at responding to my correspondent than the leisurely way that they respond to my messages. Some people respond to email very quickly, so much so that they become perpetually interrupt-driven with their life dominated by the “ping” of new mail while others take a more reflective stance and consider the nature of their response or wait for more information to arrive before replying.

Enabling or disabling user access to Analytics

It’s possible that users who have been assigned E5 licenses or the Delve Analytics add-in for E1/E3 plans do not want to have Delve Analytics process the signals generated by their Office 365 activity. For example, you might have a group of legal investigators who need to use an E5 feature (such as Equivio Zoom to analyze eDiscovery results) but don’t want to have their data processed. Or perhaps a department located in a country where the use of software like Delve Analytics needs to be agreed with a body such as a work council or union. In these cases, users can set the privacy mode for Delve Analytics to opt-out from processing and disable the feature. An individual user can do this by clicking the cogwheel icon to access Delve Options, selecting Feature Settings, and then toggle the Delve Analytics option from On to Off (Figure 9).

Figure 9: Delve Analytics settings

Giving users the ability to control their own destiny is good, but providing administrators with a way to programmatically manage things for a single user or a group is even better. Microsoft provides the Get-UserAnalyticsConfig/Set-UserAnalyticsConfig cmdlets for this purpose.

As you’d imagine, the Get-UserAnalyticsConfig cmdlet returns the current configuration for a user.  For example:

Get-UserAnalyticsConfig –Identity ‘[email protected]’ PrivacyMode : opt-inIdentity    :IsValid     : TrueObjectState : New

The identity passed to retrieve information about a user is their User Principal Name (UPN). Oher identities used with mailbox cmdlets like the alias are not accepted. The interesting output is found in the PrivacyMode property, shown here as “opt-in”, meaning that the user is happy to have Delve Analytics process their data.

Three values can be set for PrivacyMode:

To change the setting for a user, run the Set-UserAnalyticsConfig cmdlet. For example, this command excludes the user from Delve Analytics:

Set-UserAnalyticsConfig –Identity [email protected] –PrivacyMode Excluded

Company culture is critical

Delve Analytics depends on the data gathered in the Microsoft Graph. If insufficient data is available, the analytics can’t and don’t work. In effect, this means that Delve Analytics is unlikely to be very useful in small companies where some users opt-in for data analysis. In any case, those who work in small organizations tend to know what’s going on without having to be informed by data analysis. The true opportunity for Delve Analytics lies in large, distributed enterprises who depend on email as a primary method for communications. Come to think of it, Microsoft is exactly the kind of company that can use the kind of insight provided by Delve Analytics.

The big question is how will the kind insight into working habits delivered by Delve Analytics be received by end users. Anyone who’s interested in interpreting and understanding what data means will find value in Delve Analytics, but when it comes to “average user”, I think the answer lies in the culture of the company and the personal attitude of individual users. If you work in a typical American company where great value is put on personal growth and ongoing improvement, the information exposed by Delve Analytics will probably be welcomed as another tool to help people be their very best. The YouTube video posted by Microsoft to explain Delve Analytics reflects this attitude while an infomercial PDF produced by Microsoft is a good starting point for companies who want to introduce the technology internally. Those charged with this task will have to be able to clearly outline why Delve Analytics is being deployed and the personal advantages that its users can gain.

Outside the U.S., Delve Analytics might find a more measured welcome in many international companies, especially in Europe, where personal space is more heavily protected and valued than it in the U.S. Microsoft clearly has a tightrope to walk here when it comes to privacy and to communicate the value of personal analytics.

Summary

The current iteration of Delve Analytics is a V1.0 product. The algorithms use simple calculations and need to be refined and enhanced. In addition, the analysis of work is too email-centric. Over time, the algorithms will improve and data representing other Office 365 workloads will be incorporated to more accurately reflect the working day of more users. Understanding the interaction with people who don’t have accounts in the Office 365 tenant is a bigger challenge but it’s one that has to be undertaken, not least because of the almost mandatory presence of hybrid deployments (spanning both on-premises Exchange and SharePoint) found in the large enterprises that appear to be the natural target for Delve Analytics. To some, Delve Analytics will be like the old-style time and motion studies where overseers monitored the productivity of factory workers with the aim of weeding out the unproductive. That concern might hold water if Delve Analytics generated dashboards for departments or the organization that allowed supervisors to drill down to the individual, but that is not the case. And anyway, the current data set that is analyzed is so incomplete that anyone could plead a case that their work is not incorporated into the analysis.

In a nutshell, the current implementation of Delve Analytics is a pointer to the future rather than a finished product. It’s likely to take several years before we see the full worth of analytics. Companies who base their collaboration on email will be able to derive value from the information that’s now exposed while those who focus on other activities will have to wait. And to make the point again, the fear that Office 365 is introducing big brother functionality is far off the mark. At least, I hope it is!

Follow Tony @12Knocksinna