Time to Consider AI Costs?

This story originally appeared on No Jitter.

My colleague Matt Vartabedian at No Jitter has been running a series of Q&A articles with technology leaders about AI, and his latest is especially useful. Vartabedian interviewed Cognigy CEO Phillip Heltewig, and got to the heart of the matter when he asked for advice to the enterprise IT folks trying to understand the impact of AI on their organization.

"I'll mention something that I think is being completely disregarded by everyone right now," Heltewig said. "Cost."

That got my attention. Heltewig went on to drive the point home: "The cost of these models can be tremendous. One GPT4 query can … easily cost 10 cents or more. That is essentially what a whole [AI] conversation commonly costs now. If you do five GPT4 queries in a conversation — and that's just for the generative AI — that's 50 cents. That's crazy."

Heltewig suggested that every application doesn't necessarily need the biggest, most powerful large language model (LLM). Here's the analogy he drew: "I have a double master's degree in business and computer science, and if you want to ask me, 'what is 10 plus 5,' using my time to get an answer will cost you $50. Or you can ask my seven-year-old son and he only costs 10 cents or an ice cream.

Related:AI-Assisted Coding: What Software Developers Need to Know

"The point being you do not require the super intelligent model for many of the tasks that generative AI can do. [Call] summarization in GPT 3.5 is really good; you don't need GPT 4 [for that]," he concluded. "You really need to think about which model you want to use because the cost difference is so stark."

Heltewig's perspective deserves to be spread more widely as the conversation about AI matures. We need to demystify AI, at a time when a lot of the technology providers seem to be intent on preserving the mystery, and it's especially important to provide clarity as AI trends toward becoming the next big tech bubble for the industry to inflate. I think one of the big challenges for IT/communications/CX folks with regard to AI is figuring out how to ask the right questions about the AI that's driving whatever feature or function you're interested in implementing. You definitely need to know what security and compliance safeguards are in place. To Heltewig's point above, you need to know about the language models to the extent that you understand how this impacts cost and functionality (including accuracy).

On one level, it's not unlike moving to the cloud. Whatever technology your cloud provider uses may be interesting on a purely technical level, but what matters to customers is that it delivers the service at the agreed-upon (fair) price, securely and in keeping with corporate governance. AI has the potential to mess up your enterprise service if it suffers some kind of problem, but that's just as true of a cloud service going down.

Thinking about cost may be a way to bring a little rationality back to the AI conversation, by talking about metrics and functions, rather than hallucinations and other fanciful ways of describing software failures to perform. Getting familiar with what drives costs in AI-enabled software is probably a worthwhile pursuit over the next few years.