When is a cheaper LLM good enough?

Anthropic and OpenAI both released new lightweight LLMs in the last month. On everyday questions about our own data, Haiku 5.5 cost about one cent per answer, and Sonnet 5.5 cost about 14 cents.

Omni already runs light tasks, like searching the docs or writing field descriptions, on a lower-cost LLM. The question is how far that can extend to real analytics questions. 

So the morning after Haiku 5.5 launched, we ran it and GPT-6 Luna on 110 real questions from our own warehouse. Whether a cheaper LLM was good enough came down to whether the question could be read two ways.

What cheaper LLMs don’t know, they guess #

A finance lead might ask: "Why did net revenue retention fall last quarter, and which customer segments contributed most?" 

Calculating it correctly requires knowing the company's rules. Which customers are included? Does "last quarter" mean fiscal or calendar? If a customer changed segments, where should their revenue count?

An LLM has to answer each of those before it writes a query. In our tests, ambiguity separated easy questions from hard ones more than complexity did. On questions with only one reasonable reading, Luna was right about nine times in ten, at around 4% of Sonnet’s cost. Once a question needed a judgment call about what the asker meant, every cheap model fell off.

The cheaper LLMs tended to guess and present their assumptions as facts. None of Luna’s wrong everyday answers said "may" or "assuming." Sonnet 5.5, on the other hand, was more likely to hedge when it was wrong.

A query can run fine, return a plausible number, and still answer the wrong question. If the explanation is convincing, almost nobody checks. But these choices don’t have to be left up to the LLM.

Settling meaning is the harness’s job #

These are decisions made by the harness: the tools, context, instructions, and feedback systems that surround an LLM and allow it to perform useful work. The LLM is only one part of the system, and improving everything around it can make a substantial difference in how reliably it performs.

At Omni, business definitions live in a semantic model: a shared description of metrics, fields, and how data relates. 

Instead of having the LLM calculate net revenue retention from scratch for each query, it can use an existing definition that builds on customer and revenue fields. The semantic model also organizes data into Topics (curated datasets) with field descriptions, synonyms, and example queries that help the agent find what it needs.

People and agents can add those definitions as they go. Omni suggests new context from corrections made in chat, and the Modeling Agent can draft fields and measures. 

When the harness supplies definitions and established query logic, the LLM has a narrower job: interpret the request and select the right pieces. 

A smarter LLM can’t make up for missing context #

To test that, we took away our semantic model and asked the same questions again. Sonnet 5.5 got half as many right. Haiku 5.5 with the semantic model outscored Sonnet without it, at about a third of the cost.

Of course, there’s a limit. The semantic model can't settle a disagreement that hasn’t been resolved by a person. If your company hasn't decided what "last quarter" means, no LLM will decide it reliably for you. 

But you don’t have to fix those gaps on your own. To help customers prioritize what's important but missing or incomplete, Omni’s AI Hub offers suggestions to improve the semantic model based on real usage, and lets you test changes with evals before pushing them to production.

How we test a new LLM #

We test each new LLM when it comes out, and the checklist below came from what we learned in the last month.

• Compare two kinds of scoring: An LLM judge grades the answer, and a stricter check compares query results with expected values. In one round they disagreed by about 25 points, 93% against 69%, and neither was entirely right. Where they disagree, a human reviews the answer.

• Measure everything: Haiku took about twice as long as Sonnet on everyday questions because it generated much more text. Past 100K tokens of context, its per-token price jumps 5x, which nearly tripled its cost per answer on harder questions. We time the full response, warehouse calls included, and calculate cost from token counts and published rates.

• Record every input: If an agent’s response is “42 accounts,” that doesn’t tell you whether it counted the right customers, or had input truncated by a row limit. Omni’s built-in eval judge reads the agent’s queries and results along with its final answer.

• Run it more than once: One correct answer doesn’t make an LLM reliable, so we ask every question at least twice.

When a cheaper LLM is good enough #

A cheaper LLM is good enough when a question has one reasonable reading. When it needs a judgment call about what the asker meant, a cheaper LLM guesses, and states the guess as fact.

Context won’t turn Haiku into Opus, or Luna into Sol, but it can settle more of those judgment calls before the question reaches the LLM. Every definition in the semantic model is one less thing a cheaper LLM has to guess.

Cheaper LLMs still don’t use context as reliably as smarter ones. Even with the semantic model, Sonnet got more right than Haiku, so for complex analysis we’d still reach for a smarter LLM. New lightweight LLMs keep getting cheaper and smarter, though, and we’ll keep testing each one, pushing on how little intelligence a good answer needs.