Toward an AI harness that carries its own weight

Frontier LLMs are remarkably intelligent, and that’s reflected in their token prices. Newer releases cost upwards of 5-10x previous generations. Steep rates drive organizations, including us, to seek cheaper alternatives without sacrificing accuracy.

We ran GLM 5.3 Flash, an open-weight LLM that costs $0.50 per million tokens, on our internal AI benchmark. It matched Sonnet’s accuracy yet was 12x cheaper.

TL;DR: I was tasked with benchmarking open-weight LLMs on Omni’s AI harness. I didn’t set out to audit the harness itself, but that’s what the benchmark became. Only after handling emergent issues could the benchmark be trustworthy.

Key Learnings: #

• AI evals show more than just AI behavior: with the right metrics, they can reveal bugs in the software wrapped around the LLM.

• Prompt caching is paramount: providers charge less for the part of a prompt they've already processed, as long as the next request starts the same way. It cut our Sonnet bill by about 75%.

• Catering to each LLM is inefficient: this work warranted a new initiative to make Omni’s agentic harness LLM-agnostic.

Building the benchmark #

I joined Omni as a software engineering summer intern while working on my PhD. My first assignment was to test how open-weight LLMs performed inside of Omni’s AI evaluation framework.

Omni’s agent does not ask the LLM to write SQL from scratch. It routes each question to a Topic (curated dataset in Omni), loads semantic definitions and context, and gives the LLM tools to select fields, filters, and joins. The LLM still has work to do, but it operates inside a structured workflow.

Our default provider did not offer the latest open-weight LLMs. Fireworks AI did, so we began integrating it as another inference provider.

Figure 1. Selecting Fireworks AI as an LLM inference provider. We are working to bring this to customers.

Against hard prompts, Sonnet outscored the open-weight LLMs by a considerable margin. However, Sonnet no-thinking and medium-thinking surpass high-thinking in some prompts. It seemed counterintuitive.

How much of the gap was due to LLM capability, versus assumptions in the harness and the provider path?

The initial eval runs left more questions than answers, so I set out to collect more data. Originally, the eval showed pass/fail, agent turn latency, and query latency. I added a tool usage table to list LLM tool calls with their counts, errors, and latency.

Table 1. Tool calls with counts, errors, and latency. We used the same semantic model throughout testing.

Generate Query was the most called tool and failed 45% of the time, suggesting that Minimax (and our Anthropic LLMs early on) relied on guessing-and-checking queries. If a query didn’t pass our parser, it would make a new one.

Tool counts also revealed that some things were flat-out broken. The Generate forecast and Summarize full result tools had a 100% error rate. Visualize tool errors were 76%.

In early July, we switched our own Omni instance to open-weight LLMs for a weekend. GLM 5.2 went in Thursday night; MiniMax M3 replaced it Friday morning after GLM failed on image inputs.

By Saturday morning, a teammate had posted a screenshot of the agent running the same query nine times, returning zero rows each time, and never changing its approach. Another teammate gave up on an app build because the LLM “just wasn’t following instructions.” We switched back that afternoon.

That was when the experiment stopped feeling academic. People were using the path I was testing.

When the benchmark audited us #

My manager and I set out to audit and fix the path for non-Anthropic (default provider) LLMs.

Given the abnormally high tool failure rates in the open-weight LLMs, one of the first things I fixed was linearizing tool-call assembly and adding a truncation guard. We also addressed some issues with file uploads, error classification, and idle timeouts. This solidified our communication with the LLM provider, so we were figuring out why open-weight LLMs ran so slowly.

Where the time went #

Long wall-clock times did not mean slow decoding. GLM generated tokens about as fast as Sonnet, but the open-weight LLMs made more calls per prompt, and every call read the whole prompt again. By the third call, the prompt often ran from 80k to 150k tokens. Unless the prefix was cached, the provider processed all of it again. The latency question became a prompt-cache question.

A cache bust is an invisible bug. Nothing errors. The answer comes back the same. You just pay five to ten times more for tokens you already sent, and you wait for them.

The only witness is the meter. I learned to read our agent traces and diff the request payloads of consecutive calls byte for byte. Any byte that changed above the cache breakpoint threw away everything after it.

Stabilizing tool descriptions significantly preserved our prompt cache. In my third week at Omni, I opened a pull request to change one tool description for Generate forecast. The description embedded the current list of field names, and that list updated every time a query ran.

Tool descriptions sit at the very top of the prompt, ahead of the system prompt and the whole conversation. When the list changed, the first bytes of the prompt changed, and everything after them was a cache miss. The tool didn’t even have to be called. In one eval run, it was present on 525 of 570 calls, invoked zero times, and lined up with 47 of the 48 full cache busts.

That was the first and largest bust, but not the last.

The rest of the busts #

Our semantic-search tool rewrote its results once the agent settled on a Topic. If the Topic changed mid-conversation, the next call had to reread about 25k tokens, and cache reuse dropped to 69%, whereas neighboring calls used 99%.

The available tool list could change too. Hitting the per-turn query cap removed two tools from the top of the prompt. CreateBranch reordered branches by most recently updated. The query generator placed each user’s name and email inside the cacheable prefix, so every user wrote a separate copy of the same 100k-token Topic context. Each change fragmented or invalidated the cache. None was exclusive to open-weight LLMs.

What caching changed #

On the later benchmark runs, the main-loop cache hit rate was 94% for Sonnet 5, 93% for DeepSeek, and 81% for GLM. For Sonnet, the estimated bill without prompt caching would have been $255.88 instead of the actual $65.15. Caching reduced the run’s estimated cost by about 75%.

Anthropic bills a cache read at one-tenth of the input rate and charges a 25% premium to write the entry. That makes a cache hit on Sonnet especially valuable.

Making the tool description independent of the conversation state solved the issue. The LLM already had the field names in its query results, and the tool returned valid options after rejecting a bad guess. The code change touched one product file and two test files.

What the benchmark said once the harness was fixed #

The benchmark ran the same harness, prompts, and judge, with different LLMs and serving paths.

I integrated GLM 5.3 Flash through Fireworks AI and compared it with Sonnet 5, Omni’s default LLM, served through Bedrock. DeepSeek V4 Flash provided a second lower-cost comparison.

We evaluated the LLMs using a 100-prompt test set of varying difficulty, judged by the same LLM, run twice for a total of 200 runs. The prompt set was composed of 30% easy prompts, 40% medium prompts, and 30% hard prompts. Production traffic skews toward easier prompts, so this prompt set was deliberately weighted to stress the LLMs. We set thinking effort to low and enabled our validation agent, which checks each response for wrong dates, row limits, and mental math.

Table 2. Examples of easy, medium, and hard prompts. 

The first prompt is easy because all the key terms are clearly stated in the prompt; there is nothing to infer. In the medium prompt, the LLM has to write both conditions and combine them with OR. This prompt is tricky because a filter on the date alone would silently drop every deal with no next step at all. The hard prompt calls for an average window function on a CTE. Other hard prompts include ambiguous prompt language that calls on the LLM to infer what the user is asking.

Sonnet 5 served as the judge, issuing a pass or fail after reviewing the tool calls, generated query, query results, and final response. It checked query correctness, calculations, requested fields and dates, and whether the final answer addressed the question’s intent.

Price and performance #

In Figure 2, the open-weight LLMs show slightly better accuracy than Sonnet 5. The prompt set was deliberately hard and featured an LLM judge by Sonnet 5 itself. So the scores say more about the benchmark’s difficulty than the product.

Figure 2. Correct responses by LLM and prompt difficulty.

Table 3. Reliability and total cost for the tested configurations.

One GLM 5.3 Flash failure came on an easy prompt: “Which accounts opened an urgent support ticket in the last 30 days?” GLM 5.3 picked the wrong Topic field that excluded “Urgent” tickets, thus returning nothing. While the query ran fine, GLM 5.3 got confused over semantically similar Topic names. Most of GLM’s failures looked like this.

We observe Sonnet 5’s failure point on a hard prompt: “How many new-business opps did we create last week, and what’s the most we’ve ever created in a single week?”. Sonnet delivers the correct query. However, the dates were off by one week, resulting in a failure. Other Sonnet failures omitted a requested column or number, or selected the wrong field.

Where the 12× cost came from #

Sonnet 5 returned more tokens, but the price difference is largely influenced by the LLM rates.

The workloads also look similar. Table 4 counts how many main-loop iterations the LLM made and tools the LLM called. Both GLM and Sonnet ran a median of 4 main-loop iterations, the same number as tool calls. Only DeepSeek V4 Flash had a higher tool call count. GLM's worst run took 30 iterations and DeepSeek's 74, against Sonnet's 17. A handful of prompts sent the open-weight LLMs into a spiral; they adopt a “guess-and-check” strategy more than Sonnet 5.

Table 4. Main-loop iteration and tool call median and max counts.

Median prompt tokens per question were 466k for Sonnet 5, 313k for GLM 5.3 Flash, and 395k for DeepSeek V4 Flash. Median output tokens per prompt were 1,015 for GLM, 4,180 for DeepSeek, and 2,096 for Sonnet.

Table 5. Share of average output tokens per prompt, with average token counts in parentheses.

Sonnet 5 spends about 26% of its tokens on hidden thinking. You pay for those; you never see them. GLM 5.3-flash and DeepSeek V4-flash put more of their budget into visible thinking and tool-call arguments. GLM guesses and checks; you can watch it between tool calls. Sonnet thinks once, privately, and answers. Across all categories, Sonnet 5 is more verbose, returning an average of 3,294 tokens per prompt against GLM’s 1,727.

Each LLMs’ tokenizer affects costs as well. GLM counts 4.1 characters per token, DeepSeek counts 3.9, and Sonnet 5 counts 2.7. Sonnet 5 counts about 50% more than the two open-weight LLMs across the same bytes.

Figure 3. Output tokens by LLM.

When calculating Sonnet 5’s tokens billed at GLM, the total comes out to $6.23. This suggests the LLM’s pricing largely influences the total costs. Token output differences are secondary. The 12x increase from Sonnet can be attributed to the prices in Table 6. Fireworks doesn't bill cache writes but doesn't guarantee TTL either; Anthropic charges $19.10 and does. Another sizable difference lies in the output rates, with Sonnet 5 charging $10 / M tokens and GLM being 20 times cheaper.

Table 6. The table shows prices per million tokens.

Limitations and future directions #

DeepSeek V4 Flash is a reminder that open-weight LLMs are still an experiment. In the worst traces, it endlessly looped on a failing query, retrying and failing again without changing its approach. A lower token rate does not help if the agent can’t break the loop.

I started this internship trying to benchmark LLMs, and ended up benchmarking everything around them, too. A big lesson was not to trust that the measured differences were due solely to the LLMs themselves. It’s important to check things like whether the prompt prefix is byte-identical across turns; or when the LLM seems to ignore an instruction, whether it’s actually ignoring it, or if the instruction was impossible to follow.

Once those issues were resolved, the benchmark I was assigned to run could finally go forward as planned. GLM 5.3 Flash matched Sonnet 5 on our benchmark at roughly one-twelfth the cost. My advice to another team would be simpler: Cache hits. Prompt caching, prompt caching, prompt caching.