How Microsoft uses "feedback loops" to drive product development

To say that Microsoft has not asked for feedback about their services and products until very recently would be an inaccurate statement. You can go back to the days of private Windows and Office betas to find that Microsoft was interested in feedback from their testers however, that feedback typically came very late in the development cycle.

My first operating system beta was Windows Vista. I was accepted into the program just prior to the big reset of the operating systems development which was announced in September of 2005. Our primary tool for providing feedback to Microsoft was through a web portal called Connect. Other testers would try to reproduce/validate the bugs and we could have discussions around the new builds.

Microsoft also distributed builds of Windows Vista to technical partners and had a Community Technical Preview (CTP) program users could sign up for to receive occasional builds of the OS.

Entry into those types of betas occurred very late in the developmental process and it meant only the most serious show stopping bugs identified through those private testers would show up on Microsoft’s radar for fixes before the software went gold for its Release to Manufacturing (RTM).

There might have been some user interface tweaks that might make it into the later builds after testing but for the most part the shell of the OS was nearly set in stone.

Windows 7 was the first Microsoft operating system to have full blown public previews where testers allowed Microsoft to collect telemetry, it was a mandatory part of gaining access to the preview, so the company could gather info about system usage and related crashes. There were also open channels to Microsoft through a third party website called UserVoice that allowed users to give more specific feedback on items they noticed while using the preview builds of Windows.

The same process was used for Windows 8 and 8.1.

Fast forward to early 2013/2014 timeframe and you will find that nearly every Microsoft service or product was engaging users for feedback through the UserVoice website. It became so prominent that last August I wrote a story over on WindowsObserver.com called Microsoft and UserVoice Feedback Portals.

That story showed that nearly every area of Microsoft was opening themselves up to user feedback in a public forum and it was the beginning of what I dubbed the Microsoft Feedback Economy.

Little did we know that within just a few weeks Microsoft would announce the Windows Insider Program as it prepared to start testing its upcoming Windows 10 operating system.

While UserVoice channels were still available for manually entered feedback, the Insider program was built to maximize the benefit of the telemetry which the Windows team would receive from members of the program. It also introduced the Windows Feedback App which allowed users to submit their own feedback and upvote items submitted by others.

It is a real crowd sourcing process in action with those items getting more upvotes floating to the top of Microsoft’s awareness and allow them to look more closely into the issue.

The company had to build an entire new set of tools to handle the influx of telemetry and feedback from Insiders so engineers and executives could drill down into any area and see what was happening – even if the issue did not have multiple upvotes. No piece of data is left behind and engineers retain access to 100% of the data collected at any time they need access.

The sheer amount of data collected by Microsoft through the Windows Insider Program is boggling to the mind. Initially Microsoft believed they would cap the participation in the program but ultimately opened the gates wide which resulted in over 1.5 million participants by the end of 2014.

In February 2015, Microsoft expanded the program to include the upcoming Windows 10 Mobile operating system. By the time Windows 10 launched in late July of this year the total number of Insiders exceeded 5 million members.

So how does the company approach the Feedback Economy as they develop Windows as a Service with plans to constantly provide updates to the operating system and its first party apps that are available through the Windows Store?

Well last month I had the opportunity to sit down with Samer Sawaya, a Senior Program Manager on the Windows Insider team at Microsoft, to talk about how the company is embracing and using this data to make Windows 10 the version of Windows that you will love using as the company’s CEO, Satya Nadella, has said publicly on several occasions.

--------------------

SuperSite: Thanks for taking the time to talk to us today.

So let’s begin with your background here at Microsoft – how did you get into data analysis?

Samer Sawaya: I started at Microsoft 12 years ago and have been a program manager the entire time. I was initially on the SharePoint team and then moved to the Windows team during the Windows 7 era towards the end of its development cycle. I was then on the Windows 8 device experience team, the Windows 8.1 roaming team and now on the Windows 10 feedback app team.

SuperSite: A big change that recently happened with feedback related to Windows 10 is the shutdown of the UserVoice sites that have been in use for quite a while. The shift is to now receiving feedback primarily through the Feedback App.

Why make that change and what happens to all the data that has been collected through the UserVoice site?

Samer: All of the UserVoice data carried over and is in our backend systems so that engineers can access all of that data. In fact, engineers cannot tell the difference between data collected from UserVoice or the Feedback App itself when they are viewing it in the system. So none of the data is lost at all.

The main motivation in moving to the Feedback App as our main source for user feedback was all about consolidation. We will archive data in the backend and even though it may no longer be visible in the Feedback App it is still available to engineers at all times.

SuperSite: Over the last several years Microsoft has moved from private betas to public previews and now to the Windows Insider Program. It seems that every time a new product was launched for testing or in preview that a feedback channel was established as an almost mandatory step in the process.

Why has this Feedback Economy become so integral to everything Microsoft is doing these days?

Samer: The feedback loop is a very important element of development and we are always planning the way ahead based on that feedback. We know that customers want that ability to provide direct feedback as well so providing a dedicated method, that was the reasoning behind a dedicated app in Windows. It is also why that app is now available to anyone running Windows 10 now that the new OS has been released.

Not only can customers provide their own feedback but they can review what has been submitted and upvote those items. The more upvotes an item gets the quicker it floats into our awareness.

SuperSite: I recently had a chance to meet several members of a group of people called Community Champions. These are members of the Windows team who fulfill special roles when it comes to feedback.

What can you tell us about that group of people?

Samer: The Community Champions are revolutionary in our culture and how we build Windows 10 and it is something we want to spread across the entire organization. Community Champions bring a lot of passion to what they do in the forums and social media, tracking down big issues and casting a broad net to listen to feedback from customers.

Sometimes engineers will surface an issue and ask the Community Champions to search and collect feedback from customers for that item which can then be used to target solutions and fixes.

SuperSite: What is the teams sense of the feedback they are receiving from the community?

Samer: The biggest benefit is those experience at very early stages and the team lives through those pains with the users. Early access with the customer onboard helps the team build on their thought process for building features into the OS.

Getting Insiders feedback so early allows us to make course corrections much faster than in our past development cycles because previews of past OS’s came too late in the cycle to make any meaningful changes.

The new Insider process allows those changes to happen in just days or weeks.

SuperSite: What is the best way for a user to get their feedback heard by the team?

Samer: The first step is to be an Insider and be running the latest builds that are available. Feedback on new builds gets seen very quickly so you can also go into the Feedback App and upvote items plus add an issue if it is not there.

We now provide the ability in the app to share links so you can ask for others to upvote the item and that in turn helps to get that feedback seen even quicker.

SuperSite: Well I appreciate your time today so just one last question and it is a big topic of discussion around Windows as a Service and Windows updates.

Recently it was revealed that there would be additional information provided to Windows 10 users when updates are sent out to reflect what was being updated in the OS with these updates.

How are you all working to improve the feedback loop about what changes are being made to the OS when updates are delivered to the end user via Windows Update?

Samer: Well there is nothing to share at the moment on this but more to come.

SuperSite: That will be very welcomed. Thanks again for taking the time to sit down today.

--------------------

Once embraced, the feedback economy can be a powerful force for change.

However, evaluating that feedback is a critical element of this process and needs to be done with concern for both the emotional connections of the users to these changes, the actual usage of that aspect of the OS and the development cost of implementing the changes.

No doubt Microsoft has a very delicate balance to maintain in this area and it sounds like they are endeavoring to do just that within the Feedback Economy they have created.

But, wait...there's probably more so be sure to follow me on Twitter and Google+.