Changes in the Windows Insider Program with Redstone 4

“The Windows Insiders are the millions who represent the billions.”

Windows Insider Program chief Dona Sarkar regularly uses this phrase to describe the open testing program for Windows 10. The Redmond company considers the testers of early Windows 10 development builds as a group who make the operating system better in two ways: through collective telemetry data and user-specific feedback. In theory, having a large pool of advance testers offering multiple streams of diagnostic feedback should then translate into a more reliable OS when major feature updates are released twice each year.

These testers come from all over the world and participate in the Windows Insider Program for a variety of reasons. Windows Insiders also gain early access to first-party app updates through the development process to provide feedback and testing for those product teams at Microsoft.

This week Sarkar announced some changes to the Windows Insider Program as it relates to early access to new features. This coincided with an email from Windows Executive Vice President Terry Myerson that discussed the same subject.

The biggest news: A new feature, called Sets (for now), will be arriving in a future developer build of Redstone 4. This feature will "make sure that everything related to your task: relevant webpages, research documents, necessary files and applications, is connected and available to you in one click," as part of Microsoft's move toward a seamless computing experience.

The feature is not what has Insiders abuzz. The thing is, not every Windows Insider will get to test-drive the feature. This is a dramatic departure from the equal-access approach Windows Insiders have historically enjoyed.

Sarkar laid it out like this:

“…we’re going to start doing controlled studies with some of new features for Windows 10 we’re working on. We’ve been doing this on a smaller scale – like trying out different Hub icons in Microsoft Edge to see which one people like more. Now we’re going to start doing these with entire features. This means that a smaller percentage of you will initially get certain new features in builds and others may not get these new features for a while, as we compare the usage and satisfaction of these features versus people who don’t. Eventually everyone will get these features—but it just could be awhile. With this approach, it also means that we will make these features broadly available when they are ready, not necessarily tied to the next major feature update of Windows 10.”

Myerson provide the same basic message with a few tweaks:

“Sets represents a new chapter in our collaboration with you, our WIP community. You may have heard Satya talk about our cultural shift as a company of moving towards one that’s more “learn it all”. This is something we have embraced wholeheartedly, and it means we’re going to try some new things. We’ll test, learn and adjust based on your feedback.

With Sets specifically, we’ll introduce a controlled study into WIP so that we can more accurately assess what’s working and what’s not. That means a smaller percentage of you will initially get Sets in a build. It also means that some of you won’t get it at all for a while, as we compare the usage and satisfaction of task switching in Windows for people who have sets versus people who don’t. Eventually everyone will get Sets — but it could be awhile.

With this approach we will continue to ship a ton of new features and updates to the entire WIP community as we build the future of Windows. And like all features we ship into WIP, they will be broadly available when they are ready, not necessarily tied to the next major update.”

Bottom line of both descriptions: Not all Windows Insiders will see this new Sets feature. 

Some Windows Insiders were not having any of the company's shift to smaller, controlled studies of new features, and so they headed to social media to voice their opinions about this new approach. So why express concerns about not having guaranteed access to new features in these development builds? Because one of the perks to being a Windows Insider is having that access before everyone else, and now that is at risk with this new approach by Microsoft.

Here is how I expect things will play out: Those Insiders who get the new feature will be taking screenshots and discussing it on social media while those who did not get the new feature will be getting frustrated because of their lack of access to a new feature. Myerson and Sarkar both used the phrase it could be a while and that leaves some wiggle room for modifying the company's approach to controlled testing among Windows Insiders. I would not expect a specific time frame from them anyway because Insiders would quickly hold Microsoft's feet to the fire if that perceived obligation was not met.

Ultimately, the Windows Insider Program and the company's approach to testing Windows 10 is theirs to own and decide how the process will flow out to participants. It is however, very important for the company to remember that the folks who sign up for the program are enthusiasts, fans, and brand advocates. That means it is in their best interest to only wait a couple of builds before expanding the features availability to other Insiders. In other words, quite a while should not turn into months because the development cycle is only five or six months long.

I reached out to Microsoft and asked if they could expand on the approach which Myerson and Sarkar shared this week and this was the response their press office sent back:

“Without getting overly technical, the nature of a controlled study is that one group has the feature and the other group doesn’t. This allows for more accurate data on the experience – specifically what’s working and what isn’t.”

Honestly, I wish they had gotten a little overly technical. Windows Insiders would have loved a deeper explanation. As "the millions who represent the billions," these are the people most passionately engaged in giving Microsoft the feedback they need to refine an operating system. Shouldn't they get an explanation for why the program has changed in return?

 But, wait...there's probably more so be sure to follow me on Twitter and Google+.