MSN: The Inside Story, Part Two: Fly, Butterfly

The MSN division is housed several blocks away from the main Microsoft campus in Redmond, Washington at a place called Red West. The distance between Red West and the main campus has an interesting effect on both visitors and employees. Though it resembles the rest of Microsoft, and is indeed a part of Microsoft, it is also quite clearly a different place as well. The separation is not just physical, however. It's clear from my many talks with MSN folk that the division has a completely different vibe from its corporate parent. Part of it is the market that MSN plies. Unlike the slow-moving business market that much of Microsoft serves, MSN competes with fast-paced Internet companies and provides services for consumers that expect constant improvements. As a result, MSN is not hobbled by the hesitancy that grips the Windows and Office groups. And the place just seems more exciting as a result. These are people that get work done. And their customers see the results of that work almost immediately.

"David [Cole] and Yusuf [Mehdi] were both from the Windows Division, but this was a time when there was a lot more heat going on," Adam Sohn, the Director of Global Sales and Marketing PR told me. "We had the browser innovations going on, with new releases every week. Both guys had a very deep understanding of how to live in a very dynamic and changing world. And both guys know how to ship software."

According to Sohn, Cole is a super-senior technical guy who understood, perhaps better than anybody, what was possible for PC users if you go and exploit the platform. But MSN was a mess at the time with conflicting strategies and non-integrated products.

"I'll be honest here," Sohn said. "We still haven't completely landed where we want to be. There is always this discussion about business models, right? Are we an access business? A subscription business? An ad-funded business? Sometimes the press wants to force somebody in the industry to make a stand and say, 'this is the answer.' Because nobody's been really able to find it yet."

Reorg

About a year and a half ago, MSN instituted a reorganization of its business in a bid to drive integration. "Our world had grown too siloed," Sohn said. "We had a team building one service over here, another team building another service over here, and a team building a client over here. We had multiple operations teams, multiple security teams, product management was scattered all over the place, and it was hard to actually get anything done."

It wasn't efficient. And efficiency is job one at a division that can't afford to wait two or three years for the next product cycle. The senior leadership, including Mehdi, Cole, and Blake Irving--who runs MSN's communications services division--started to think a little bit more about the user, go figure. "We started thinking about the user first and the technology second," Sohn said. "And this is something we now do very well at MSN. And I think other parts of [Microsoft] are getting there. You know Microsoft. Ooh, a problem: Can we solve it with software? Let's all get in a room and talk about code. Well, that's fun and some great stuff comes out of it, but ... millions and millions of people, especially consumers ... when you think about non-technical people, what experience are they having? You have to start organizing yourself to deliver services for those types of people."

MSN internalized the customer-first mindset much earlier than the rest of Microsoft. And all of the products it's shipped over the past several months--MSN Search, MSN Music, MSN Spaces, MSN Messenger 7, and Hotmail--reflect this. You can see the start of a really cohesive experience across all of these services, but more importantly, across all of the ways in which actual users would use these services.

Put another way, instant messaging is not the goal. The goal is to facilitate communications between people and the people they care about. "Is it only Messenger?" Sohn asked. "How does email relate to that? How about sharing my photos? How about sharing my blog, if I decided to go and do that? What about searching for stuff? Our software powered experiences are now driven by thinking about the relationships we have or the ways in which we consume information. Then, we look at how software can be applied to enhancing that experience. The alternative is sorting of jamming people into the shape of the software applications that we build." That, of course, is how much of Microsoft's software was historically created.

The new MSN organization has three main divisions, or groups, all of which are overseen by Cole. As noted previously, Mehdi runs the MSN Information Services & Merchant Platform. He owns Search, the Music, Shopping, Entertainment and Video, and everything you would think of as information services, like the MSN.com portal and the content channels.

Irving runs the MSN Communication Services and Member Platform group, which oversees tools like Hotmail, Messenger, and Spaces. Because they're running these huge services, they also own the operational elements for most of MSN as well.

MSN also has a marketing group that spans across both of the other groups. MSN marketing is spearheaded by Jane Boulware, who is described as the steward of the MSN brand. "We have this great feedback loop," Sohn told me. "The marketing group gets to live across the whole division. We talk to consumers, we do a lot of consumer research that gets fed back into the groups, and no one is siloed anymore. People are working together in a much more interesting way because everyone gets that there is this larger thing that we are trying to do."

"MSN lives in a world where it's innovate or die," Sohn said. It's been that way for Microsoft for a long time, of course. But MSN has made this statement into a credo of sorts. "It's true here on daily and sometimes hourly basis," Sohn added.

The energy at MSN is palpable. Sohn and Larry Grothaus, a Group Product Manager for MSN, have both recently discussed with me the how working for an entity so rapidly delivering offerings like MSN differs from other areas at Microsoft. "Think about enterprise deployments that might take years, or major platform shifts that are absolutely critical for [Microsoft] to be successful, not to mention the millions of ISVs out there, but at the same time, are more deliberate, slower, and more complex processes," Sohn said. "We're all about providing software powered experiences to consumers and those things can be done on a much more rapid fire basis."

"It feels very much like a scrappy, entrepreneurial, freewheeling thing," Grothaus added. "If you've got a good idea, and you want to go do something, you can get the resources to do it. That's the atmosphere down here."

Down here. Again and again, without perhaps even realizing it, Sohn and Grothaus highlighted differences between MSN--Red West--and the rest of Microsoft. Clearly, no disparagement was meant, and indeed, both were quite deferential whenever talk of the rest of the company came up. But it is perfectly clear that MSN, in some ways, operates outside of the normal Microsoft way of doing things. That independence and separation is a crucial part of the division's success. And MSN's ability to ship product stands in sharp contrast to the never-ending delays facing important products like Longhorn, Visual Studio 2005, and SQL Server 2005.

MSN Search: Answers, not links

Nowhere are the differences at MSN felt more than with MSN Search. Sohn described the atmosphere at Search as similar to that of a start-up company in San Francisco. "You walk down the halls of that place and people are here late, late at night, people are always pushing and challenging each other, and there's a cachet to working on that team," said. "We've got a lot of ridiculously brilliant people down there that I can't even stand to be in meetings with because I know I'm not even worthy of being with these guys."

How important is search? Microsoft Research independently sought out MSN and offered to help. "Think about relevancy ratings, and the kinds of algorithmic math that needs to go on, and how you build out a distributed computing architecture that can handle the query volume and is capable of indexing the Web every single week," Sohn said. "It's at 5 billion documents for us right now. There are just some fun computer science problems there."

Almost two years ago, MSN decided to build its own search engine from scratch. Previously, MSN Search used algorithmic results powered by a company called Inktomi, which was bought by Yahoo! MSN's advertisements were powered by a company called Overture, which was also bought by Yahoo! But Search became popular enough that people were realizing how much it didn't meet their needs. "Google showed the world what was possible if you did some innovation around search," Sohn said. "No one in this division would ever try to take any credit away from those guys. They've done some amazing stuff and they continue to drive innovation."

To compete with search and provide the kind of service that will keep people coming back to its network, MSN needed its own search engine. Otherwise, the division wouldn't be able to compete for search users or develop a lucrative advertising business. So they met with Microsoft co-founder Bill Gates who, surprisingly, immediately approved the idea. "Bill was like, 'yeah, come on, let's go,'" Sohn told me. "It's just one of those things where he was probably already thinking, 'why haven't you guys done this already?'" At the time, Google had been around for several years, and there were various other competitors. MSN clearly had its work cut out for it.

"It became clear to us that this was a place where we had to fix the problem we had caused when we helped the world go digital," Sohn explained. "I realize that's a funny way to say it. But there's been an unbelievable information explosion. You think about all the documents on the Web, or all the documents that you create personally. How do you find things three years later? There is so much information out there. How do we connect people to it?"

"When you think about MSN, you think about this big network of services. In order to make revenue, we have to keep lots of people in our network because we fund it mainly with advertising. So we create a bunch of great services, grow the audience, and then provide great technology for advertisers to connect with that audience in a way that is not intrusive. We don't do pop-ups anymore, we have more clarity in advertising, and we actually scaled back the number of display ads you'll see. We do have a good search advertising business, and we'll continue to invest there."

Ultimately, people just want to find information that is important to them. Again, it's not about the software--search in this case--but rather about the people who use the software. "What are they really looking for? How often does digital commerce start with a search?" Sohn asked. "Lots."

The MSN Search architecture was built from the ground up, which enabled the division to do some unique things. "Everyone wants to compare us with Google, so they do a search here and a search there, and they pick apart the results. Frankly, there are some places where we're as good or better, and there are a bunch of places where we're not. And that's always going to be true. And the day we think we win is when 51 percent of our results are better, and 49 percent of their results are better. But we would obviously want to keep driving that number up, because ... we're Microsoft guys and ... we like big numbers."

Today, the search process is simple but takes too long. Maybe you're typing away at the keyboard, getting work done and you realize you need to look something up. Many times, that thing is a fact. How many MP3 players were sold in 2004? What is the population of France? There is no way to find facts on Google.

And, as it turns out, there is no way to find many facts on MSN Search either, but they're getting there. "We did build the infrastructure to do it," Sohn explained. "We built something we call Instant Answers. We started it with MSN Music and Encarta, because those were two discrete information sources that we had great access to. With Encarta, the innovation we did was we did a bunch of natural language work to help us parse a sentence, and then we did a bunch of database work on the back end that helps us look through the Encarta database, extract things we can label as facts, and then we connect the natural language smarts with the fact list. We've done that for Encarta, and we've done that for MSN Music."

If you're searching on the Web and you type in, "what was the population of France in 1980?", Google will return numerous results. But at that point, you'd enter what Sohn calls the "spelunking phase," where we have to go manually search through all of those results hoping that maybe one of them has the fact you need. More often than not, you'll find articles with related information or something unhelpful.

In MSN Search, this type of query is pushed through the Encarta index and you'll see the answer at the top of the results list. Or you could type in the name of a musical artist, like Sting, and you see which Sting songs are selling well these days. Links will take you over to the MSN Music site and you can listen to song clips if you want, and purchase music. It's a seamless experience.

"Now we take this thing, which I call a mini-architecture, and apply it to other databases," Sohn said, hinting at the future. "Ultimately, we can apply that to the entire Internet. That is impossible to do if we just skin results coming in from another search engine. So what we look at when we think about search is, where are the opportunities to change the game by thinking about what people are really looking for? How do we get the number of abandoned queries down? Answers, not links' has been our rallying cry."

What about subscription-based content? For example, many Web sites, like the Wall Street Journal Online, require you to pay a fee in order to access their content. "We're also looking at this huge set of information behind subscription walls that's not generally available to the public through a search," Grothaus said. "How do you get the partnerships in place so that the Wall Street Journal or Lexis-Nexis is available? Even if you had to pay for it, maybe just a small amount per search." Sohn added, "We're going to be pretty aggressive on that. There will be scenarios where we subscribe to services on behalf of the world. Because it's an ad funded business, we're able to do that. Abstracts are a great idea too. You may have to pay for it, but at least you'll know you've got the right article before you do."

MSN Search also includes an innovative feature called Near Me, that gives location-specific search results. "We have varying levels of success with the Near Me results, and I think our guys would be the first to admit that," Sohn said. "But what's interesting underneath it all is that we went out and built another architecture. When we crawl Web sites, it's actually putting some smarts against the data we're getting back from those Web pages. We add what we call a geo tag to entries in the index. Say I want to find an Acura dealership in Seattle. If I only search for 'Acura' and 'Seattle,' I'm relying on the hope that the Acura dealership in Lynnwood, a city just north of Seattle, is using the word 'Seattle' in their Web site."

"We're really looking at those key places where we can up the game a little bit and continue to drive the stuff forward. At the same time, we know there's a price of admission: We need to be fast, need to be relevant, need to have the right number of documents in our index, and we need to do everything we can to cut back on the number of spam searches."

UI Matters: Windows Desktop Search

MSN's recently released Search Toolbar with Windows Desktop Search takes MSN Search technology to the desktop. "It's nice and fast, and we did a good job of thinking about fit and finish," Sohn said. "We want to be the best ISV that develops on top of Windows. We know how to develop on top of Windows. What's crazy to me is that we built this thing on the Win32 API set. Anyone who's ever been to a PDC (Professional Developers Conference) gets that [information] in their backpack. It's one of our advantages: We're incented to be great platform customers and we understand that UI matters, especially when you're building for the broad consumer user base. You have to think about how the experience will drive success for people."

A lot of people are going to be pleasantly surprised by the new Search Toolbar, which features a new preview pane in the search results window and other features that were added after the public beta, an almost unheard of development with other Microsoft products. "We were really focused on responding to the feedback we got," Grothaus told me. "We don't just clean up bugs during the beta, which is the traditional way to do things. You'll see changes in each version we release."

The new Search Toolbar even lets you designate any search engine. So if you want to use Google to do Web searches, it will do that. You can also choose exactly which toolbars get installed during Setup. Indeed, the entire Setup process is different now. You can choose which folders are indexed, which toolbars are installed, and where to put the search index file.

"This is going to be another great platform," Sohn noted. "We sort of replumbed the notion of a toolbar here, We've got an interesting architecture that we can use to deliver innovations to folks in nice, quick, componentized ways. I think you'll see the toolbar become a way that we can offer very, very cool innovations to people in a low impact way. When something new becomes available, customers can choose to add it or ignore it. There will be quick iterations of the toolbar in the near future. They won't be toolbar replacements but just add-ons people might want. It's a componentized architecture. We've been testing the [delivery mechanism] for the add-ons internally. We can notify you when there are updates."

Grothaus added, "If a new file format comes out next week and becomes really popular, then we can support it quickly. If a corporate customer has a proprietary file format internally, they can write an iFilter extension and then that can be indexed as well."

Regarding the overall search strategy at MSN, Sohn noted that it was a market ripe for innovation. "We're on the cutting edge," he said. "We built our entire [search server farm] on 64-bit Windows, which really enables us to push the platform to the limits. We were able to utilize that architecture to build a very, very cool, high-performance, in some ways self-diagnosing and self-healing infrastructure. Although humans are required. At least for now."

Next: Part Three, Services that Communicate