Data Mining: Big Brother or Big Business?

It's not every day that a particularly sexy piece of database technology winds up on the front pages of national papers or becomes integral to national and international policy and political debates. But that's exactly what's happening with data mining and related technology.

Most people in the United States, and many folks in the international community are aware of the press surrounding the analysis of phone records in the United States and abroad to search for potential terrorists. I won't state my political opinion here. Instead, I want to focus on the fact that the approach wouldn't be possible without vast data warehouses and data mining.

On the front page of the Washington Post, I've seen several recent articles related to data mining, and USA Today ran the front-page story "Data Miners Dig a Little Deeper" on July 12. The USA Today business section that day was devoted to issues surrounding data mining and privacy. It was an interesting read; I encourage you to check it out if you can find an issue or happen to have a subscription for viewing the article. I'm not singling out Microsoft, but the USA Today article began by using an example based on Hotmail and explained that Microsoft's analysis of Hotmail activity includes the time of day that emails are referenced as well as salary ranges for the ZIP code in question. The article said that Microsoft "knows a florist will pay a premium to have a coupon for roses reach males 30-40, earning good wages, who check their e-mail during lunch hours on Valentines Day."

I don't consider myself a data-mining expert, but I think the technology is exciting and holds huge potential to shape data management over the next decade. So, my first reaction to this news is, "That's really cool! I wish I had more time to become an expert in SQL Server-based data-mining technology." I also tend to be pretty conservative and pro-business, so my second reaction runs along the lines of "That's a really smart thing for Microsoft to do and they have every right to do it."

But I have to admit that the part of me that read "1984" in high school can't help but wonder what the natural evolution of this technology will be. Maybe opt-in clauses in the future will be written in a way that allows "Microsoft Flowers" to pre-ship the $500 Hawaiian orchid that Microsoft knows my wife admired because it's scanning our email and cell-phone traffic. (You've read about Microsoft's "Unified Communications" roadmap, right? Check it out at http://www.microsoft.com/presspass/press/2006/jun06/06-25UCGRoadmapPR.mspx .) Microsoft also knows that I haven't bought her a present yet based on a study of credit-card charges it has access to through a data-sharing agreement with ReallyBigCreditCo. Of course, the opt-in agreement will grant me permission to send the orchid back, but Microsoft's data-mining skills will indicate that I'm much too lazy to actually do so, so pre-shipping is a good investment for them.

Ok, enough with the Orwellian postulations. Data mining does raise some interesting privacy concerns that I suspect will be debated broadly over the next few years. What's in data mining for you today? Data-mining is a part of the data-management world that's poised for huge growth. SQL Server is becoming an increasingly compelling platform for data mining, and I'm sure the SQL Server team understands the importance of continuing this trend over the next decade. And only a relatively small number of people have any significant amount of expertise with Microsoft or any other vendor's data-mining tools today. You don't need data-mining analysis to recognize that this trend should create high demand for skilled data-mining professionals over the next decade.