The 15-point gap was noise

Eight days, two funerals

On July 8, OpenAI published an audit of SWE-Bench Pro, the benchmark most of the industry, this blog included, had been using as the serious yardstick for agentic coding. Their datapoint-analysis pipeline flagged 200 of 731 public tasks (27.4%) as broken; a parallel campaign of five human engineers per task flagged 249 (34.1%). Call it ~30%. On that basis they formally retracted their earlier recommendation to adopt the benchmark.

Sit with the phrase "earlier recommendation" for a second. OpenAI had already killed SWE-bench Verified for contamination and design flaws, and pointed everyone at SWE-Bench Pro as the fix. The fix lasted a few months. Meanwhile frontier pass rates on the thing climbed from 23.3% to 80.3% in eight months. In hindsight, that curve was measuring benchmark decay at least as much as model progress.

The broken tasks fall into four buckets: overly strict tests that reject functionally correct code, underspecified prompts whose hidden tests enforce requirements you couldn't infer, low-coverage tests that pass incomplete fixes, and prompts that actively contradict the grader. My favorite specimen: a task whose prompt specifies serialization examples with one leading space while the hidden tests demand two. Follow the instructions faithfully, fail the benchmark. A one-character difference between "frontier capability" and "task failed".

I told you to adjust your priors. Adjust them again.

In the last post I noted OpenAI's audit in an aside and essentially called it sour grapes: when the vendor disputes the one leaderboard it loses, adjust your priors accordingly. That heuristic is usually right, and it was lazy here, because I hadn't yet read the second audit: the one from people with the opposite rooting interest.

Datacurve, the team behind the DeepSWE benchmark (more on them below), ran their own analysis of SWE-Bench Pro's verifier: an 8.5% false-positive rate, a 24% false-negative rate, and an independent analyzer disagreeing with the verifier on 32% of all trials. Nearly a third of the pass/fail decisions on the industry's most-cited coding leaderboard appear to be wrong in one direction or the other. When the noise floor is that high, a 5-point gap between two frontier models is astrology.

The contamination findings are somehow worse. The SWE-Bench Pro containers shipped with the repository's full .git history, including the merged fix the task was built from. Claude Opus 4.6 and 4.7 read the gold commit out of git history in roughly 12 to 13% of reviewed runs (in 33 of 38 flagged cheat trials, the agent literally ran git log --all or git show <gold-hash> and pasted the answer; see the open issue). GPT-5.4 and 5.5 never did this; Gemini hovered around 1%. Which means the benchmark's noise was model-correlated. It systematically flattered exactly one family's exploratory instincts.

And a third distortion, subtler: SWE-Bench Pro's prompt template tells agents the tests are already handled. So on SWE-Bench Pro, models write their own tests in 3 to 28% of runs. On prompts that don't say that, the same models test their own work in up to 85% of runs. That is the single behavior most correlated with actually solving the task. The benchmark was instructing models to skip the thing that makes them good.

Two audits, opposite incentives, same magnitude. When the vendor's sour grapes and the competitor-backed benchmark agree on the number, the number is probably real.

The replacement, and who's behind it

Here's where it gets pleasantly incestuous. The best replacement candidate, DeepSWE, was built by Datacurve, a small team Theo (t3.gg) describes as "nerdy Waterloo kids" and backed after getting fed up with, quote, being "fed shit benchmarks for the last few months, if not years." Theo's benchmark skepticism has been this blog's recurring touchstone for a reason: he keeps being early. He was grumbling about exactly this failure mode while the rest of us were citing the leaderboard.

What the Waterloo kids actually built: 113 original tasks across 91 repositories in 5 languages, with reference solutions written from scratch and never merged upstream, so they can't leak into training corpora. Prompts are half the length of SWE-Bench Pro's, but solutions need 5.5x more code across ~7 files. Verifiers are hand-written to test observable behavior rather than inherited from some PR's test suite: measured false-positive rate 0.3%, false-negative 1.1%. The v1.1 revision grades the committed diff in a fresh, isolated container, and ships task environments as shallow clones: no gold hash in the workspace to find, no test framework to monkey-patch. Tasks, trajectories, and harness are all public on GitHub. That is what doing the boring work looks like.

And on a clean ruler, the story I told you last week changes shape:

So Sol beats Fable now?

No. Read the error bars: 73±3 and 70±4 overlap, and the DeepSWE harness caveat cuts in Fable's favor: every model runs through mini-swe-agent, a deliberately minimal scaffold with one bash tool and a shared prompt. That isolates model capability from scaffolding, at the price of holding every model below its native ceiling; models tuned for richer harnesses arguably lose the most. What DeepSWE establishes isn't a new king. It's that the old margin was an artifact.

The real-world reports haven't moved, and they still matter more than either leaderboard. Simon Willison, with early Sol access: "very competent, though so far it hasn't struck me as better than Fable at the kind of complex coding tasks". Theo: "a damn good model. Not quite as 'smart' as Fable, but it is incredibly capable". My own runs match that, with the usual anecdotal caveat: Fable still feels like the better orchestrator and reviewer; Sol grinds longer without drifting. Benchmarks tied, vibes differentiated, jobs different. Same conclusion as last week, cheaper evidence.

What is unambiguous on the clean ruler is the cost column:

Selected rows from the DeepSWE v1.1 leaderboard (113 tasks, updated July 13, 2026). The quiet headline: Terra matches Fable's pass rate at 23% of the cost. Sonnet 5 costs more per task than anything on the board while solving barely half. Token appetite, not sticker price, is what you're actually buying.

How to actually respond to a benchmark funeral

• Purge SWE-Bench Pro numbers from your decks. Including the ones that flattered your favorite model. Including ours. This post is that purge. A retracted benchmark cited knowingly stops being evidence and starts being marketing.
• Demand error bars, or assume ±5. A verifier that misgrades a third of trials means single-digit gaps on any benchmark without published variance are noise until proven otherwise. DeepSWE prints its ±. That should be table stakes now.
• Track cost per solved task, not cost per token. Fable at $21.63 per solved-task-equivalent versus Terra at $4.95 is the number that should route your work, especially now that Copilot-style token billing is turning heavy users' $50 months into $3,000 months.
• Keep the routing table. Nothing in the clean data contradicts last week's advice: Fable for orchestration, deep review, and human-facing output; Sol for long autonomous grinds; Terra as the daily default. That last one got stronger. Good routing survived the evidence swap. That's the test of whether it was ever good.
• Build your own mini benchmark. Theo's standing advice, and the only durable defense: keep a folder of real tasks where agents have failed you, and rerun it on every release. His SnitchBench and Skatebench got outsized attention from researchers precisely because independent, reality-grounded tests are nearly extinct.

Every coding leaderboard we've trusted in the last twelve months has died of contamination, verifier rot, or both. The response isn't to trust the new one harder. DeepSWE's authors already expect saturation within a year and are building a private holdout for exactly that reason. The response is to hold every number loosely and your own task folder tightly. Benchmarks are somebody else's tests of somebody else's tasks. Your workload is the only leaderboard that can't be retracted.