The Quest for Practical Data Mining Solutions

If Ponce de Leon had been a database architect, his quest would have been to discover a practical approach to data mining. And he would have found that approach in SQL Server 2000, which is shipping later this summer. Data mining will change the face of database solutions as we know them—maybe not this quarter or this year, but soon. And SQL Server 2000 will emerge as the dominant force in this brave new world of data analysis.

Today's OLAP-based Business Intelligence (BI) tools are wonderfully sophisticated compared to reporting and analysis tools of yore, but they're still woefully inadequate in many important ways. Fundamentally, OLAP presents data in an intuitive way that makes it easy for end users to explore. But at least two basic problems with OLAP hinder users' ability to make decisions based on the data they can access. First, they need to know which questions to ask to get at the relevant data. Second, they need to know how to interpret the results so they can make the right decisions.

The Holy Grail of data mining is a magic answer machine that pores through our data vaults seeking valuable nuggets of information. Once it finds the data we need, the magic machine tells us what's interesting and important about the information—and what to do with it. Unlike OLAP, sophisticated data-mining solutions might know the questions we need to ask and could tell us how to interpret and act upon the answers. Add some arms and a few blinking eyes to such a solution, and you've got something "Space Odyssey 2000"-ish. Along those futuristic lines, a colleague recently shared this quote with me from Warren Bennis, professor at the University of California School of Business: "The factory of the future will have only two employees—a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment."

Will we ever reach that data-mining promise? Maybe. Today, vendors are spinning amazing tales of "push-button" data mining. But don't fall for the hype: Real data mining will continue to be difficult while the technology matures and standardizes. Still, smoke and mirrors always precede the first steps toward turning dream technology into commodity solutions. Data mining is on its way to reality, and its potential benefits are simply too great to ignore. Don't just take my word for it. Feel free to call Gartner Group, International Data Corporation (IDC), or any of those other fancy analyst groups that write all those impressive white papers. They're saying the same thing.

I've run out of space this week, but in an upcoming editorial, I'll justify my assertion that SQL Server is destined to be the dominant data-mining engine and explain how you can start experimenting with Analysis Server 2000 (the part of SQL Server 2000 that includes the OLAP engine and new data-mining algorithms) in a production environment—with little or no risk to your production systems.