Windows Storage Foundation (WinFS) Preview

On August 29, 2005, Microsoft surprised analysts, developers, and other onlookers with its surprise release of WinFS Beta 1, the first publicly available version of the company's next-generation data storage engine. To understand why this was a surprise, we must step back a bit in time and examine the development of this technology which, until recently, was as pie-in-the-sky as teleportation or cold fusion.

WinFS genesis

.Back in the early 1990's, Microsoft's NT team was still charged with so-called "next-generation" technologies. One of the team's goals for Cairo, a project that encompassed a number of these next-generation products, was a new relational file system then called Object File System (OFS), which would help manage local and distributed objects using an active storage paradigm. Cairo itself was feature-dense and feature-rich, and while its research-oriented ideals were untenable on the hardware of the day, Microsoft bet for a few years that this would change. It's an approach that hasn't failed it before or since, but with Cairo, it proved fatal. A number of technologies we now use every day had their roots in Cairo, but few emerged as elegantly as originally planned. Some, like the OFS, have yet to emerge.

However, OFS work continued. OFS was supposed to support data aggregation features so, for example, you could open a pseudo-folder that contains views of data in different drives on your computer. If you think this sounds suspiciously like a Windows Vista virtual folders (see my showcase) you're correct. Over a decade after the company first started talking about this feature, it's finally being added to Windows (though I should note, once again, in a less elegant way than originally planned).

OFS would have removed the need to understand where data resided, and it would have done so both locally and remotely. Today, we deal with objects such as file shares on remote servers, which are direct representations of physical locations. OFS was designed to remove the need for users understanding these locations. So instead of a "Data" share on the server, we might have simply accessed a pseudo-folder called "Paul's data" that would have aggregated all of that user's server-based documents into a single view.

Storage+

As the Cairo project fell apart in the mid-1990's, and its object-oriented technologies were spread to different product teams, OFS merged into a project called Storage+. Storage+, along with COM+ and Forms+, formed the foundation for Microsoft's mid-1990's object push with COM (Component Object Model). As I wrote in a 1999showcase about COM+, "Storage+ will do away with the NTFS file system and replace it with a new relational, object-oriented file system based on SQL Server 8.0. Expected to arrive with the next version of Business Windows (post-Windows 2000, think 2003 or 2004), Storage+ will turn the file system into a relational database, speeding searches and providing a higher level of granularity for those searches."

As you can see from this description, Storage+ was an update to OFS, but with an added twist: Now, the file system would be based on relational database technology, an obvious choice for a data store. And since Microsoft had just rewritten SQL Server from scratch (SQL Server 7.0 was all-new technology, not the Sybase code the company had used in previous versions) and was proving to be surprisingly powerful, it seemed like an excellent choice.

Relational File System (RFS)

In 2000, I reported that Windows Blackcomb (then the successor to Windows XP/Windows Server 2003) would feature a new relational database-based file system dubbed Storage+. But plans were already in flux. In an April 2000 report, I had the following to say: "Originally designed as a replacement for the Windows file system, Storage+ was to bring the power of the SQL Server relational database engine--with its rich searching and indexing functionality--to Windows users everywhere. But Storage+ is still in limbo at Microsoft for a variety of reasons. The company had originally hoped to ship the product?currently known as the Relational File System (RFS)?with SQL Server 2000. However, SQL Server 2000 is a relatively minor update (it was originally called SQL Server 7.5) with a short development cycle, and Storage+/RFS couldn't be implemented in the allotted time."

There were other delays. When Oracle announced its platform-independent Internet File System (IFS) in 2000, which was a relational file system based on Oracle database technology, Microsoft had to retool yet again, to match that product's functionality. Because of continual delays with OFS/Storage+/RFS, Microsoft's other product teams have had to resort to other, more tested, data storage technologies. Active Directory (AD) and the Exchange message store were (and continue to be) based on the Jet technology that powers Microsoft Access, for example. So once again, work on this relational file system was pushed back to a future date.

Enter WinFS

By the time Longhorn had usurped Blackcomb's position as the next Windows release--and had cycled between being a minor and major update--Microsoft had settled on its file system storage plan. RFS morphed again, this time to WinFS (Windows Future Storage), and instead of being incorrectly labeled as a new file system, it would be a storage engine built on top of NTFS that would include virtually all of the functionality that was originally promised for OFS a decade ago.

It's hard to pinpoint when Microsoft revealed the existence of WinFS, but suffice to say it is simply the continuation of OFS/Storage+/RFS. In my 2002 showcase, The Road to Windows Longhorn, I still referred to the storage engine as Storage+, but this had changed to WinFS by the 2003 version. In that article, I also revealed that WinFS was not a file system itself, but would instead work on top of NTFS. "The oft-misunderstood Windows Future Storage (WinFS), which will include technology from the 'Yukon' release of SQL Server, is not a file system, Mark Myers told me. Instead, WinFS is a service that runs on top of--and requires--NTFS. 'WinFS sits on top of NTFS,' he said. 'It sits on top of the file system. NTFS will be a requirement.'"

The biggest splash for WinFS, of course was at PDC 2003 (see my show report), when Microsoft began publicly revealing technical details of the technology for the first time. There, Microsoft revealed that Longhorn, the next major Windows version, would be built on three core technology pillars: WinFS, Avalon, and Indigo (which, incidentally, map very nicely to the previous generation technologies Storage+, Forms+, and COM+, respectively). I attended a WinFS session at PDC and was pretty impressed with its capabilities. The big deal, of course, was desktop search, a capability that had always been part of the plan, but was being promoted because of the sudden rise in Internet-based searches. Why can we find information so quickly on the Internet, Microsoft asked rhetorically, when it takes forever to find information on our own PCs? (This is a question Apple CEO Steve Jobs would notably borrow when he talked up Spotlight in Mac OS X Tiger--see my review--a year later.)

Over the next several months, and after a few public Longhorn builds, it was pretty clear that WinFS wasn't ready for prime time. As one might expect, WinFS suffered from huge performance issues and simply bogged down the regular builds. And while WinFS was indeed included with the WinHEC 2004 Longhorn build that Microsoft shipped in May 2004, Microsoft was surprisingly quiet about WinFS at that time. A few months later, we found out why.

In late August 2004, after near three months of silence, Microsoft surprised the world by significantly changing its plans for Longhorn, which was now retooled for a 2006 release. Three key technologies from the next Windows version--Avalon, Indigo, and the WinFX APIs needed to support them--would be ported to Windows XP with Service Pack 2 (SP2), Windows XP Professional x64 Edition, and Windows Server 2003 with SP1. Most critically, one of the core Longhorn features--WinFS--would be removed from Longhorn and would ship at a later date.

Because of this significant change, Microsoft quickly went into damage control mode, noting that the release would still include many features that users had mistakenly tied to WinFS. "Longhorn will still feature a very rich search experience," Microsoft's Greg Sullivan told me at the time. "It's kind of a mistake to equate local search with WinFS. They are different platform elements. So local search will still happen, and we will still deliver a very compelling full text search in Longhorn, alongside new shell capabilities. This will provide some semblance of the 'Find My Stuff' experience. But it won't be the full relational store with deep integration and platform exposure via APIs. For end users, however, it will be the same. We will offer a very compelling user experience for local search inside of Longhorn."

Microsoft also announced at the time that WinFS would not ship for Windows XP or 2003. And with that, I, like many others, figured that WinFS was dead in the water. We'd see a first beta release to coincide with the release of Longhorn--since that's what Microsoft told us--and that was that. Once again, WinFS was relegated to pie-in-the-sky status, and it left our minds completely.

WinFS Beta 1

It shouldn't have. Despite years of delays, WinFS was still on the fast track within Microsoft, and the company spent the next year, literally, readying a little surprise. Yesterday, the company announced the Beta 1 release of WinFS, a release that would ship for Windows XP only, no less. (Future betas will also run on Windows Vista and other modern Microsoft platforms.) What's interesting about all this is that it was virtually a secret. Microsoft never let on that it was actively prepping WinFS for a 2006 beta release. And no one really asked about it. WinFS was just another technology that we all figured we'd eventually see elsewhere. Eventually.

So what is WinFS Beta 1? According to Tom Rizzo, the director of SQL Server product management, the focus hasn't changed. "We're building a rich relational file system for Windows," he told me. "With WinFS, we will provide rich new ways to organize and visualize data. And as a final piece, it's a platform. It's not just for end users: Developers can extend WinFS, integrate their applications with WinFS, synchronize data between their applications and private databases and WinFS, and build support for their own data types into WinFS, using full-featured, managed code APIs [application programming interfaces]."

Installing WinFS Beta 1 on Windows XP

WinFS installs (Figure) like any application does--it requires a very specific (and new) version of the .NET Framework (2.0.50215.322), however, and can only be applied to NTFS-based systems--and then requires a reboot. When you boot back into Windows, you'll notice a new top-level object in your My Computer window (Figure), or, as Microsoft puts it, a new top-level object in the shell namespace. Dubbed WinFS Stores, this object speaks loudly to the pros and cons of this release.

On the pro side, WinFS is real. You can store data in WinFS and, most crucially, access WinFS-based data from any existing (read: legacy) Windows application. "One of the most exciting things about WinFS Beta 1 is that you can access it natively from any Win32 API [applications]. You can browse it with Windows Explorer, where it's seen as a drive in your system. You can save documents from Microsoft Word directly into WinFS. It preserves full backwards compatibility when you move data in and out of WinFS author, last saved time, etc. WinFS works fine with existing applications: Copy and paste, whatever."

On the con side, WinFS feels bolted on. It's not a truly integrated part of the operating system, and until third party developers support WinFS with truly innovative applications, it's unlikely that this technology will be of much interest to anyone.

In any event, for end users, WinFS Beta 1 isn't particularly useful. Beyond the top-level shell object, there's just not a lot to see. Under the WinFS Stores level, you see a starter data stored named DefaultStore (Figure). You can also add your own data stores, via a right-click menu option (Figure). When you create a store, you can give it a name, some comments, and decide on which volume the store will reside (Figure). Curiously, you cannot make a shortcut to the store on your desktop (surely a Beta 1 bug).

When you copy data to a WinFS store, the file copy dialog is non-standard, and not the version you see normally in XP (Figure). And when you attempt to access the store via a Microsoft Office application's Open dialog, the store does not appear in My Computer as you'd expect. Instead, it appears as a share under Network Places (Figure). That said, you can use Explorer to navigate into the store, double-click a document file, and cause the application that owns that document type to launch. At that point, Save As works normally and "sees" the store (as a share).

What's really powerful about WinFS, of course, is the way it can be manipulated by developers. Microsoft includes a Getting Started application (Figure) that will help developers access the WinFS and WinFS API help files and other related documentation. I haven't had time to fully explore WinFS' developer features yet, but it's something I'll definitely do.

Timing and delivery

Rizzo told me that Microsoft shipped the Beta 1 version of WinFS two weeks before PDC 2005 so that developers could spend some time experimenting with it and then provide feedback at PDC and beyond.

But recently, I came across some internal Microsoft documentation that spells out the release schedule for WinFS more fully. WinFS Beta 1 will be followed by at least one Community Technology Preview (CTP) release, which is currently due on February 15, 2006. Then, on May 1, 2006, Microsoft is scheduled to release WinFS Beta 2. Beta 3 is currently scheduled for November 15, 2006, with a Beta 3 Refresh release expected in April 2007. WinFS is currently scheduled for RTM in Q3 2007, well after Longhorn Server is released.

When it's finalized, WinFS will be delivered as an out-of-band update for Windows XP and Windows Vista, in the same way that the .NET Framework is delivered today. And while Microsoft hasn't finalized licensing, WinFS will almost certainly be free. Also, WinFS will probably be included in Longhorn Server, though it could be delivered as an add-on for that system as well.

Finally, WinFS is only a codename. The technology will likely be rebranded as Windows Storage Foundation before it's released.

Conclusions

It's far too early to make any determination about the worth of WinFS. However, I do find it notable that Microsoft has been able to keep its active development--and the surprise Beta 1 release--secret. In a day and age when the software giant is repeatedly criticized for constant delivery delays, the release of WinFS Beta 1 is a pleasant about face. Let's hope Microsoft can continue to surprise us in such positive ways in the months ahead.