The leaderboard is lying to you

The setup

Every eval headline you've read this month compresses a probability distribution into a single number, then ranks products by it. That's fine for marketing. It's useless for the decision you actually face: which model do I point at my terminal, and what will it cost me per unit of work?

So we measured that instead. Four frontier models, the full Terminal-Bench hard suite (240 tasks: builds, debugging, sysadmin forensics, data wrangling), five seeds per model, same machine image, same harness, no retries hidden in the harness. Everything ran fully autonomous, yes, with --dangerously-skip-permissions, inside disposable containers. The blog is named after the methodology.

for seed in 1 2 3 4 5; do
  harbor run terminal-bench@hard \
    --agent claude-code \
    --model claude-fable-5 \
    --seed "$seed" \
    --max-turns 80 \
    --out "runs/fable-5/seed-$seed.jsonl"
done

jq -s 'map(select(.status=="solved")) | length' runs/fable-5/*.jsonl

What the single number says

Here is the table everyone wants, with the columns nobody publishes next to it: standard deviation across seeds, mean cost per attempted task, and mean cost per solved task, the number that actually matters.

Our runs, July 2026, n = 5 seeds per model. Treat the absolute numbers as one lab's sample, not gospel; the structure is the point.

The chart the leaderboard won't show you

Plot solve rate against cost per solved task and the tidy ranking falls apart. Fable 5 is clearly the capability frontier. But Sonnet 5 solves a task for a quarter of Fable's price, and if your tasks are retryable, "cheap model, three attempts" beats "expensive model, one attempt" on 40% of this suite.

Any leaderboard delta smaller than your run-to-run variance is noise wearing a suit.

Variance is the story

Haiku 4.5's five seeds landed between 31% and 44%. That's a 13-point spread from the same model on the same tasks. When a launch post claims a 3-point win over a rival from a single run with an unspecified harness, you are not looking at a measurement. You are looking at a coin that was flipped once.

The spread shrinks as capability rises (Fable 5 held a ±2.1 band), which suggests something real: frontier models don't just solve more tasks, they solve them more repeatably. For agents you run unattended, that second property is arguably worth more than the first.

The harness is a confound

Same model, same tasks, three different agent harnesses: we saw solve rates move by up to 9 points depending on scaffold. Tool-call ergonomics, context compaction policy, retry-on-error defaults: all of it is "the model's score" as far as the leaderboard is concerned.

{
  "max_consecutive_tool_errors": 3,
  "on_error": "summarize_and_retry"
}

What to actually do

• Buying capability? Fable 5. The frontier is real, and the low variance compounds when tasks chain.
• Buying throughput? Sonnet 5, with retries. Best $/solve on anything retryable.
• Buying triage? Haiku 4.5 as a first pass, escalate failures upward. Our two-tier cascade solved 61% at $0.71 per solve.
• Publishing evals? Report seeds, spread, cost, and harness, or you're publishing vibes with axes.

The leaderboard isn't wrong. It's just answering a question nobody asked. The question that matters is a distribution over cost, capability and variance. You can only see it if you run the models yourself. Preferably with the flag this blog is named after.