DeveloperWeek Enterprise: Sorting Out Big Data to Empower AI

At last week's DeveloperWeek Enterprise 2022 conference, Victor Shilo, CTO of EastBanc Technologies, gave a keynote that aimed to clear up some of the confusion that can come with trying to make soup out of huge datasets.

"In many cases, big data is a big data swamp," he said in his presentation, "The Big Data Delusion - How to Identify the Right Data to Power AI Systems." The problem, he said, comes from traditional analytical systems and approaches being applied to outsized amounts of data.

For example, an unnamed fintech company that was a customer of EastBanc had huge datasets of its customer data, transactional data, and behavioral data that was cleaned by one team then transferred to another team that enhanced the data. While such an approach may be sufficient, Shilo said it can also slow things down.

The fintech company, he said, wanted a way to use its data to predict which of its customers would be receptive to contact. The trouble was it seemed to be a herculean task under traditional processes. "Their current team looked at the task and estimated the effort would take four, five months to complete," Shilo said. "That's a lot of time."

EastBanc sought to tackle the problem within six weeks, he said. Turning huge data into assets that Shilo called "minimal viable predictions" required thinking backwards and thinking about the operational needs for that data. "You want to focus on the business outcome," he said. "You really want to work with the team facing the customer or who's making the decisions, like sales, and ask them, ‘how we can help?'"

Related:5 Ways to Prevent AI Bias

The problem the fintech company had was the calls it had been making to potential customers were unproductive, Shilo said. "Either the customer didn't pick up the phone or they objected to do to anything for them." He called it a waste of time and money in the long run.

Read the rest of this article on InformationWeek.