How businesses need to adapt to actually put their Big Data to work

As companies get better at building up Big Data from across their business, they're running into a surprising challenge: What to do with it all.

One study by Forrester, focused on data in Hadoop installation, found that between 60% and 73% of all data within an enterprise goes unused for business intelligence (BI) and analytics.

There's an explosion of data, said Stephen Baker, chief executive at Attivio. But understanding the value of data is very difficult.

Founded in 2007, Attivio works to do just that, with software that helps businesses analyze their data to make more informed decisions.

In an interview, Baker shared what he saw as some of the key challenges as businesses work to become more data driven.

He said that generally speaking, businesses find their data falling in one of two broad buckets.

The first, (relatively) more manageable bucket is structured data: Financials, databases, and every other source that keeps its data well-defined from the get-go.

That's challenging enough to put to work, but Baker said it pales in comparison to unstructured data.

What knowledge can I get out of PDF reports on this file server? What can I get out of the notes from the field on Salesforce.com, by tapping into the wisdom of my sales team? he said. That's a particularly important question as internal wikis become even less structured, and more business now runs on email and chat.

Important shifts in business can happen on the front lines that the back office might never even hear about if they don't have a way to spot those trends.

Structured data is hard enough, but when you start thinking about the value of insight you can get out of an email, it's much bigger, he said.

The newest breed of corporate data analysis packages have that challenge in mind, helping discern trends in unstructured data and even performing operations like sentiment analysis to help spot red flags that might otherwise be lost.

Baker said that for companies to make the most of the opportunity, they need to find ways to spread access — and context — to the data widely from within the organization.

And that means changing culture, starting with those that traditional have done the data interpretation themselves.

We believe that that basic cornerstone, data as a service, has to be self service, has to be non technical in nature, Baker said. That means that along with raw data, employees need to know where it's from — and who can help provide answers and caveats about it.

If I'm a low level marketing analyst, and I search for product profitability for products A, B, and C, I want to see the data, and see how reliable it is, who has used it, who can I call about who has used [that data] in the past.

The alternative approach — keeping data siloed to just the experts trained in analysis — simply doesn't scale to the challenges facing business today.

Data scientists are a scarce commodity, said Baker. You're not going to find enough people who can code in R or understand SQL statements, so we think about a Google ,etaphor: It's really about using basic search techniques to make the data and information searchable and explorable.

That approach, he said, can help empower people throughout your organization, no matter how much data you collect.