Every AI analytics tool can answer a question. The ones that answer it correctly usually also answer it fastest and for the least money, and that's not a coincidence.
There are more ways to ask a warehouse a question in plain language than there have ever been. Every major BI vendor now ships a chat interface. Every warehouse has its own agent: Cortex Analyst on Snowflake, Genie on Databricks. Claude, ChatGPT, and Cursor can all query a database directly once you point them at one. Natural language querying (NLQ for short, typing a question instead of writing SQL) went from a demo trick to a checkbox every vendor ships within about two years.
The pitch was that NLQ makes analytics free. Free of SQL, free of waiting on an analyst, free of learning a dashboard. Team after team is now finding out that free wasn't the right word. The bill just moved. It shows up as a wrong answer, as an agent that takes eight minutes to answer a question a person could answer in thirty seconds, and as a token cost that scales in ways nobody budgeted for.
The instinct is to treat correctness and cost as two separate lines on a chart, where you can pick a cheaper, faster setup and expect it to be a little less careful. That instinct is wrong, and it's wrong for a specific, mechanical reason. The thing that makes an AI agent's answer correct, a governed path to the right table, metric, and join, is the same thing that keeps the answer fast and cheap to produce. Get that path wrong, and you don't trade correctness for speed. You lose both at once.
Key takeaways
AI agents that explore a schema from scratch on every question are expensive for a specific, measurable reason: they have to re-read their own accumulated context at every step, and that cost compounds.
Research from Stanford's Digital Economy Lab and Microsoft found agentic tasks consume roughly 1,000 times more tokens than a single chat exchange, and that spending more tokens doesn't make an answer more likely to be right.
Omni's own benchmark of 100 real questions found the setup with a governed semantic layer was correct more often, stayed consistent across repeat runs, and cost less per correct answer than every alternative it tested, including a coding agent handed the same semantic layer and a short playbook on how to use it.
Handing an agent a semantic layer isn't enough on its own. In Omni's test, a coding agent given the full model and a playbook still hand-wrote its own SQL and ignored a warning in the model, because nothing forced it to use what it had read.
The single number that represents correctness, speed, and cost at once is cost per correct answer. A setup that's technically capable of the right answer but wrong half the time, or wildly inconsistent between runs, loses on this number even if its raw response time looks fine. Even this metric fails to capture the full cost of wrong answers, like the consequences of misguided decisions, or loss of trust in the data team.
Why a wrong answer and an expensive one share a root cause #
An AI agent querying raw tables directly has to do something a person with institutional knowledge doesn't: rediscover the business from scratch, every single time. It lists tables, guesses at which ones matter, infers joins, writes SQL, runs it, and checks whether the result looks plausible. Each of those steps costs tokens. Getting any one of them wrong means backtracking, which costs more tokens, on top of the ones already spent finding the wrong path in the first place.
This isn't specific to analytics. Researchers at Stanford's Digital Economy Lab and Microsoft Research studied this pattern in coding agents, tracing eight frontier models across a standard software engineering benchmark, SWE-bench. They found agentic tasks, meaning a multi-step loop where an agent reads a task, calls a tool, interprets the result, and decides what to do next, consume roughly 1,000 times more tokens than a single chat exchange. The cost sits almost entirely on the input side: an agent doesn't write much more than a chatbot does, but it has to re-read everything it has already done before every new step, the original task plus every tool call and response so far, and that accumulated context grows with each loop.
Two other findings from that study matter here. Running the identical agent on the identical task twice produced token counts that varied by up to 30 times. And spending more tokens didn't make an answer more likely to be right. Correctness tended to peak at a moderate token spend and flatten out after that, meaning extra exploration bought nothing once an agent had already gone down the wrong path.
An AI agent answering business questions against a raw warehouse runs into the identical shape of problem. It isn't reasoning its way to a wrong number because the model is weak. It's paying, in tokens and in wall-clock time, to rediscover a business definition that a person on the data team already knows and never wrote down anywhere the agent could read it.
What that actually looks like on a real warehouse #
Omni ran a benchmark to see how this plays out on real business questions rather than a coding task. The team took 100 real questions its own staff had asked in its production Omni instance and tested five AI setups against the same questions, data, and warehouse: Omni's own AI agent, a leading coding agent given Omni's semantic layer plus a written playbook explaining how to use it, that same coding agent given the semantic layer with no playbook, the coding agent with no semantic layer at all (writing raw SQL from scratch), and a warehouse-native AI agent. The non-Omni products were anonymized, since the point was comparing architectures, not vendors.
Every question had a rubric written in advance, defining the correct answer, what a wrong answer looked like, and what "close but wrong" meant, since a plausible-sounding number that's still incorrect is its own failure mode. Claude Opus judged each response against that rubric, a second Opus pass audited the uncertain calls, and the whole matrix ran twice to check whether results held up on repeat.
Three individual questions from that run show the mechanism plainly.
One question asked which EMEA solutions engineer had the most closed-lost opportunities that fiscal year. The setup with no semantic layer started by listing every database in the warehouse, a list too long to read in one pass, so it read only the first portion and skipped the database holding the actual CRM data. It spent eight minutes and ran 60 separate warehouse queries searching for anything named "region" or "engineer," at one point checking whether users' longitude values were positive as a proxy for being in Europe, then gave up and reported that no EMEA region or SE role existed in the data. That run took 489 seconds, spent 6.1 million tokens, and returned no answer. Omni's agent routed the same question to the relevant Topic (a curated slice of the semantic model), ran one query, and named the correct engineer in 34 seconds on 466,000 tokens.
Another question asked for the top five campaigns by pipeline. The coding agent given the semantic layer and a playbook did the research correctly: it found the right Topic, the canonical pipeline measure, and the right filters, and it even read a comment in the model warning that joining campaigns through the membership table instead of directly to opportunities would inflate the numbers. It stated, in its own words, that it had everything it needed. Then it hand-wrote a query that joined through membership anyway, hit exactly the join fan-out the comment had warned about, and reported a top campaign worth $55.5 billion in pipeline. Omni's agent read the same Topic, ran a single governed query with the join path enforced, and returned the correct number using a third of the tokens and half the time.
A third question, about ARR trends, sent the coding agent without the semantic layer into a demo schema full of synthetic data. It didn't notice the schema was synthetic. It invented its own ARR formula from the demo tables and reported a 71 percent ARR crash driven by fictional companies, then added confident-sounding commentary about a healthy, diversified customer base built entirely on made-up numbers. Omni's agent answered the same question from the real ARR Topic, using about a sixth of the tokens, and never saw the synthetic clutter at all.
Across all 100 questions, Omni reported 95 correct answers. That breaks down to 96% on the easy questions, 97% on medium, and 90% on hard. The next-closest alternative hit 60% accuracy on hard questions. Omni gave the same answer on 96 out of 100 questions across two runs; the next-best alternative changed its answer on 30 of them. Omni answered every question it was asked; the alternatives failed to return any answer, or errored out, 19 times combined. Omni's median response was roughly twice as fast, at a cost of about $1.04 per correct answer against $1.29 for the next-best setup, burning around 670,000 tokens per correct answer against one to six million for the alternatives.
Having a semantic layer isn't the same as enforcing one #
The coding agent that invented a $55.5 billion campaign had read the correct warning in the model beforehand. It knew the right join path. It used the wrong one anyway. That single failure is the most important data point in the whole benchmark, because it rules out the easy explanation.
Before running the test, it would have been reasonable to expect the semantic layer itself to be the deciding factor: hand two agents the same metric definitions, join paths, and field descriptions, and they should converge on the same answers. They didn't. A coding agent holding the full semantic model still trailed Omni by 21 points, because reading a rulebook and following it are two different things. The agent could read the model as a reference, decide it understood the situation, and still write its own SQL from scratch.
That gap is what's worth naming plainly: Omni’s AI harness, meaning the system wrapped around the model that decides not just what it can see, but what it's actually allowed to do with what it sees. A semantic layer that an agent can consult and then override isn't a governed path. It's a suggestion. Omni's harness routes a question to a Topic, has the agent select which fields and filters from that Topic apply, and then compiles that selection into SQL through a query engine, so the agent never hand-writes the query itself. A validation step then checks the result against the original question before it's returned. The agent's job is choosing what it needs, not writing how to get it, and that's the step where every observed failure in the benchmark happened for the setups that skipped it.
The number that captures all three problems at once #
Correctness, speed, and cost usually get reported as three separate scores, and that's part of why the tradeoff looks real when it isn't. A single number folds all three together: cost per correct answer, meaning total spend divided by the number of questions actually answered right. A setup that's fast and cheap but wrong half the time loses on this number. So does a setup that's correct but only after burning enough tokens re-deriving the schema that each right answer costs several dollars.
The academic side of text-to-SQL research reached the same conclusion from a different direction. BIRD, one of the standard benchmarks used to evaluate text-to-SQL systems, originally scored submissions on execution correctness alone, whether the generated query returned the right result. The benchmark's maintainers added a Valid Efficiency Score, since renamed the Reward-based Valid Efficiency Score, specifically because a technically correct query that took far longer to run than it needed to wasn't actually a good answer. Rewarding correctness without accounting for the cost of getting there measures half the problem.
Consistency deserves a place in that number too, even though it doesn't fit neatly into a single dollar figure. An answer that's right on one run and wrong on an identical repeat isn't trustworthy, no matter how cheap either individual run was. The Stanford and Microsoft research found token counts on identical repeated tasks varying by up to 30 times. Omni's benchmark found the effect on the output side of that same instability: the least-grounded setups changed their answer to the same question 30 times across two runs. An agent that can't hold still on repeat questions can't be evaluated on cost or speed in isolation, because neither number means anything if the next run might get a different answer at a different cost.
How to run this benchmark on your own stack #
Any team can test this against its own warehouse rather than taking a vendor's word for it, and the shape of a fair test looks the same regardless of which tools are involved.
Start from real questions your team has actually asked, not generic ones a vendor's demo would handle easily. Write a rubric for each one in advance, with input from whoever actually knows the correct answer, covering what right looks like, what wrong looks like, and what a plausible-but-wrong answer would contain, since that third category is where most of the damage in these tests happens. Run every setup against the same questions and data, and use a genuine enterprise dataset, because synthetic datasets produce synthetic benchmark results. Also, make sure to run the full set more than once, since a single pass can't tell you whether an answer would be consistently correct, or was just correct that one time. Use a strong model as a judge against the rubric, and have a second supervised pass review any verdict the first judge wasn't confident about. Measure cost, tokens, and wall-clock time directly in real time, with safeguards in place to prevent context sharing between agents or prompt caching that skews token counts.
Treat the result as directional rather than universal. A benchmark run against one team's warehouse, schema, and question set won't transfer exactly to another team's data, which is a limitation Omni's own writeup is explicit about. The value isn't in one company's published numbers. It's in running the same structure against your own stack before you commit to one.
Where a lighter-weight setup is still fine #
None of this means every question needs a fully governed harness behind it.
A single, simple question against a small, well-understood table carries little risk either way, since there's not much schema to explore incorrectly and not much room for a wrong join to hide in. A team still in the early stages of building out its semantic model won't see the full benefit of a harness yet, because there's less curated context for the agent to be routed to in the first place, though that's an argument for building the model out, not for skipping it. And a one-off, exploratory question a person plans to double-check by hand anyway carries lower stakes than a number that's going into a board deck or an automated workflow nobody reviews before it acts on the answer.
The determining factor is what happens when the agent is wrong, not how often it happens to be wrong on easy questions. A wrong number that a person catches before it matters is an inconvenience. A wrong number that reaches a dashboard, a customer, or another automated system unreviewed is a different kind of problem, and that's the scenario a governed harness is built for.
FAQ #
What is NLQ, and why did it get so popular? #
NLQ, natural language querying, lets someone type a question in plain language to get an answer instead of having to write SQL or navigate a dashboard. Its popularity has grown quickly because it removes the biggest barrier to self-service analytics: needing to know SQL; but the underlying agent still has to translate that question into a correct query somehow, and how it does that determines whether the answer is trustworthy.
Why do AI agents cost more than expected when answering analytics questions? #
An agent without a governed path to the data has to explore the schema itself on every question: list tables, guess at joins, write SQL, and check the result. Research on agentic workloads found this kind of multi-step loop consumes roughly 1,000 times more tokens than a single chat exchange, because the agent has to re-read its own accumulated context at every step.
Does having a semantic layer guarantee a reliable answer from an AI agent? #
No. In Omni's benchmark, a coding agent given the full semantic model and a written playbook still hand-wrote its own SQL, ignored a documented warning about a join that caused a fan-out, and returned a wrong number anyway. A semantic layer only changes outcomes when the harness around the agent requires it to be used, rather than offering it as optional reference material.
What is a harness, in the context of AI analytics? #
A harness is the system wrapped around an AI model that controls what data and definitions it can see and, more importantly, what it does with them. A harness that compiles an agent's field and filter choices into SQL through a fixed query engine rather than letting the agent write SQL by hand removes the step where most observed failures happen.
What is cost per correct answer, and why does it matter more than accuracy alone? #
Cost per correct answer divides total spend by the number of questions actually answered right, which penalizes both wrong answers and answers that were right but expensive to reach. A setup can look fast or cheap in isolation and still lose on this number if it's wrong often enough, or inconsistent enough between runs, that each reliably correct answer ends up costing more.
How should a team benchmark AI analytics tools before buying one? #
Test every setup under consideration against the same real questions from your own warehouse, with a rubric written by someone who knows the correct answers, and run each test more than once to catch inconsistencies between runs. Measure cost, token usage, and response time alongside correctness in the same test, since a vendor's published benchmark on its own data won't transfer exactly to yours.
Related reading #
Case studies:
How Checkr built a data foundation for AI with structured context: using AI Hub and a repeatable eval loop to cut a real stakeholder question from hours to minutes
Brevo builds its AI analytics foundation with Omni: using a semantic layer to make AI reliable for both internal and customer-facing analytics
More guides from Omni:
Semantic Layer for AI and BI (2026): Why It Matters, How to Choose, and How to Implement It
AI-Powered BI Tools (2026): What's Real, What's Hype, and How to Choose
Every warehouse is different, and a vendor's published benchmark, including this one, is a starting point rather than a guarantee. Bring your hardest, most real questions to a free Omni trial and run the same test against your own data, or get a demo and ask to see the harness, not just the chat window.
