One of the most talked about and more controversial features of Windows 10 is Cortana, your own personal digital assistant and is often compared to Siri or Google Now. “She” (yes, Cortana is officially female and is referred to as “she” or “her”) lives inside your device and goes with you wherever you go. To truly make the most of her services, Cortana has to learn more about you; she does this by collecting data in a number of ways.
Before we go into how Cortana collects data and exactly what she collects, It is worth remembering that Cortana is a personal assistant, which means she needs to know details about you in order to be of any use. If you don’t share information with Cortana, then the whole aspect of personalisation goes away and you are left with just an assistant, like having a stranger on the street tell you the time or setting static reminders on your smartphone. The assistant is no longer personal, but general. In other words the premise of personalised service to “help make things easier for you and keep you up to date on the things that matter to you” becomes much less relevant, if you don’t share data about yourself.
With that in mind, let us have a look at what is being collected and shared to Microsoft if you want to use Cortana fully.
What Cortana reads in your email
One of the really neat skills of any personal assistant is to offer advice and services that you need, but didn’t initiate the dialogue for. The personal assistant should know enough about you to make educated guesses about what you need to know at a the best possible time.
One of these may surprise some. If Cortana has access to your email, she will identify flight information and ask you if she should track your flights for you. She can then make sure you leave in time to get on the flight, and also tell you if it is delayed. She will also track your parcels without having to access the courier websites. All without you ever interacting with your PC or Phone.
Of course this is dependent on you allowing her to access your emails through the Windows Mail app. Cortana will scan your emails and look for schema.org markup, which contains airline flight reservation data, in the case of tracking your flights.
How Cortana knows where you are: Geo-location
If you allow it, Cortana will track your location from the GPS, Wi-Fi or mobile connection depending on what the device she is on, allows. If she knows where you are, then it is a short step to figure out when you need to leave to get to a flight, and even take traffic conditions into consideration. You can turn off this feature by disabling geo-location services, but that means Cortana won’t have access to your location at all.
Figure 1: Cortana reminds you to leave on time
How Cortana answers your questions: Bing Search
If Cortana can't answer your question or request, she'll default to a Bing search. The search queries are collected by Microsoft to better understand how users are using Cortana and to improve on the results, as well as create a more personalised experience for you. If you already have a Microsoft account, the searches done via Cortana are stored in your Bing search history. She will also upload some of your calendar and contacts to enable personalized speech and typing. You can manage this setting via your Bing Dashboard, where you can also clear the data collected so far.
How Cortana gets social with you: Facebook
If you connect Cortana with your Facebook profile, she will use this to further personalize your experience. However, exactly what data is being collected is not easy to figure out. Microsoft only states that “you can control whether Cortana uses information from your Facebook account for personalization”, but nothing more. If you look at the data Bing is allowed to use, if includes public profile, friends list, news, posts, work places, education, pictures, likes and activities from various segments. Whether Cortana uses all of this is not clear.
Why Cortana keeps a Notebook
The way that Cortana knows how to assist you is through the concept of the Notebook. Just as a human personal assistant would keep a notebook with details of personal things to remember, Cortana keeps all of the things she knows about you in a central location. The particular features that you choose to give the Notebook access to are synchronized to Microsoft Azure, and they can then be used by other Microsoft services to personalize the experience to you. You can also tell her your favorite sports team, a particular stock or add an event category of choice.
You can access the Notebook via the Cortana interface and each section or feature has its own toggle to switch on or off, whether Cortana should use it. For example, you can switch off the restaurant recommendations, meeting reminders or Cortana all together. You can also at any time edit the information you have manually entered.
Figure 2: The Cortana Notebook is packed full
What Can Developers Do With Cortana and Your Data?
Now we know what data Cortana is collecting in general, the question is then: Who can use or even see this data? Consumers may trust Microsoft to some extent with their data, but few would be comfortable with giving developers of third party apps access to it.
Good news! Developers and third party apps have no access to any of your data. It is securely stored in Azure. Third party apps can only work within the sandbox that they are assigned, and that goes for Cortana too. What Cortana enables is a hands free way to interact with your app, before it is launched. This could be a travel app you ask to “show my trip to London” or it could be a background task users can use to get an app to perform. In either case, the data accessible by the app is only the data managed and collected by the specific app. Third party apps will not have access to your personal data stored in the Notebook.
How will Cortana work for you?
I say that with an emphasis on you, because she only knows about you and only exists to make your life better and more productive. If you share the information mentioned above and start talking to Cortana regularly, her true power starts becoming obvious.
Cortana thinks independently
The real power is in the features that don’t require you to request, initiate or take action -- in other words, the features that Cortana does for you while you do other things. These include contextual reminders, such as reminding you to get milk when you are near a supermarket, flight tracking, monitoring traffic conditions for your daily commute and tracking parcels. You can set reminders relating to time, people or event location. For example, ask Cortana “remind me to ask my wife about her mother” and next time you communicate with your wife, Cortana will remind you. It doesn’t matter if you are texting, calling or emailing: a reminder will pop up. And of course Cortana knows who your wife is, because you have put that in the Notebook’s “Inner Circle” section.
Ask Cortana to “remind me to pick up painkillers next time I am at the Wall Drug”, she will keep track of your location and match that with locations of Wall Drug outlets, so when you are near one, she will remind you. It is way beyond a gimmick or “fun” feature. This is truly useful.
Figure 3: Contextual reminder for when you are near a pharmacy
Cortana doesn’t quit your apps on you
If you are using Cortana on Windows 10, she quietly hums along in the background. You can continue writing your Word doc, sending your email or filling in the form on the website, then speak your query aloud to Cortana. She doesn’t get in the way of your current task, but sits in the corner listening for your command. So if you say, “Cortana, please find me all the Excel spreadsheets with the name ‘budget’ in them,” she’ll do that while you continue working on a PowerPoint demonstrating your mastery of the budget process.
Note this only works with speaking to Cortana. If you type the question, then focus is removed from the application you were working within.
Cortana’s a handy search assistant
You can ask Cortana to find files you were working on yesterday or last week or whichever natural language period you can think of. You can also ask to “open the last PowerPoint file I worked on” or simply just ask Cortana to find all files starting with a specific word or letters.
Figure 4: Searching for photos with natural language
Cortana is still very much in her infancy and more and more features will be added and expanded. The best way to get started with Cortana is to enable her on your Windows Phone or Windows 10 PC and follow the prompts. I promise you won’t be disappointed.
