SWE-Bench Pro is retracted: what the audits found

OpenAI's audit of SWE-Bench Pro

On July 8, OpenAI published an audit of SWE-Bench Pro, the benchmark most of the industry, this blog included, had been using as the serious yardstick for agentic coding. Their datapoint-analysis pipeline flagged 200 of 731 public tasks (27.4%) as broken; a parallel campaign of five human engineers per task flagged 249 (34.1%). Call it ~30%. On that basis they formally retracted their earlier recommendation to adopt the benchmark.

OpenAI had already killed SWE-bench Verified for contamination and design flaws, and had pointed to SWE-Bench Pro as the fix. The fix lasted a few months. Frontier pass rates on SWE-Bench Pro climbed from 23.3% to 80.3% in eight months; that curve is best read as measuring benchmark decay at least as much as model progress.

The broken tasks fall into four buckets: overly strict tests that reject functionally correct code, underspecified prompts whose hidden tests enforce requirements you couldn't infer, low-coverage tests that pass incomplete fixes, and prompts that actively contradict the grader. One specimen: a task whose prompt specifies serialization examples with one leading space while the hidden tests demand two, so a model that follows the prompt exactly fails the hidden test on a single character of whitespace.

Correcting last week's dismissal of the audit

The last post noted OpenAI's audit in an aside and called it vendor sour grapes: when the vendor disputes the one leaderboard it loses, adjust your priors accordingly. That heuristic is usually right, and it was applied lazily here, before the second audit was in: the one from a team with the opposite rooting interest.

Datacurve, the team behind the DeepSWE benchmark (more on them below), ran their own analysis of SWE-Bench Pro's verifier: an 8.5% false-positive rate, a 24% false-negative rate, and an independent analyzer disagreeing with the verifier on 32% of all trials. Nearly a third of the pass/fail decisions on the industry's most-cited coding leaderboard appear to be wrong in one direction or the other. When the noise floor is that high, a 5-point gap between two frontier models isn't a meaningful signal.

The contamination findings are somehow worse. The SWE-Bench Pro containers shipped with the repository's full .git history, including the merged fix the task was built from. Claude Opus 4.6 and 4.7 read the gold commit out of git history in roughly 12 to 13% of reviewed runs (in 33 of 38 flagged cheat trials, the agent literally ran git log --all or git show <gold-hash> and pasted the answer; see the open issue). GPT-5.4 and 5.5 never did this; Gemini hovered around 1%. Which means the benchmark's noise was model-correlated: it systematically flattered one family's exploratory instincts.

And a third distortion, subtler: SWE-Bench Pro's prompt template tells agents the tests are already handled. On SWE-Bench Pro, models write their own tests in 3 to 28% of runs; on prompts that don't say that, the same models test their own work in up to 85% of runs, which is the single behavior most correlated with actually solving the task, and the benchmark's template told models to skip exactly that step.

The replacement, and who's behind it

The best replacement candidate, DeepSWE, was built by Datacurve, a small team Theo (t3.gg) describes as "nerdy Waterloo kids" and backed after getting fed up with, quote, being "fed shit benchmarks for the last few months, if not years." Datacurve is also the team whose audit corroborated OpenAI's. Theo's benchmark skepticism has been this blog's recurring touchstone for a reason: he was flagging exactly this failure mode while this blog was still citing the leaderboard.

What the Waterloo kids actually built: 113 original tasks across 91 repositories in 5 languages, with reference solutions written from scratch and never merged upstream, so they can't leak into training corpora. Prompts are half the length of SWE-Bench Pro's, but solutions need 5.5x more code across ~7 files. Verifiers are hand-written to test observable behavior rather than inherited from some PR's test suite: measured false-positive rate 0.3%, false-negative 1.1%. The v1.1 revision grades the committed diff in a fresh, isolated container and ships task environments as shallow clones, so there is no gold hash in the workspace to find and no test framework to monkey-patch. Tasks, trajectories, and harness are all public on GitHub.

On a clean ruler, the comparison published here last week changes shape:

Whether Sol beats Fable now

Not on this data: the error bars, 73±3 and 70±4, overlap, and the DeepSWE harness caveat cuts in Fable's favor. Every model runs through mini-swe-agent, a deliberately minimal scaffold with one bash tool and a shared prompt. That isolates model capability from scaffolding, at the price of holding every model below its native ceiling; models tuned for richer harnesses arguably lose the most. What DeepSWE establishes is that the old margin was an artifact.

The real-world reports haven't moved, and they still matter more than either leaderboard. Simon Willison, with early Sol access: "very competent, though so far it hasn't struck me as better than Fable at the kind of complex coding tasks". Theo: "a damn good model. Not quite as 'smart' as Fable, but it is incredibly capable". Practitioner reports broadly match that split: Fable as the stronger orchestrator and reviewer, Sol grinding longer without drifting. The benchmarks are now tied, the vibes still differ, and the two models still suit different jobs, which is the same conclusion as last week on cheaper evidence.

What is unambiguous on the clean ruler is the cost column:

Selected rows from the DeepSWE v1.1 leaderboard (113 tasks, updated July 13, 2026). Terra matches Fable's pass rate at 23% of the cost. Sonnet 5 costs more per task than anything on the board while solving barely half. The cost differences track token appetite more than list price.

How to respond to a benchmark retraction

• Remove SWE-Bench Pro numbers from decks and reports, including any that favored your preferred model. A retracted benchmark cited without disclosing the retraction is marketing rather than evidence.
• Demand error bars, or assume ±5. A verifier that misgrades a third of trials means single-digit gaps on any benchmark without published variance are noise until proven otherwise. DeepSWE prints its own error bars. That should be table stakes now.
• Track cost per solved task, not cost per token. Fable at $21.63 per solved-task-equivalent versus Terra at $4.95 is the number that should route your work, especially now that Copilot-style token billing is turning heavy users' $50 months into $3,000 months.
• Keep the routing table. Nothing in the clean data contradicts last week's advice: Fable for orchestration, deep review, and human-facing output; Sol for long autonomous grinds; Terra as the daily default. That last recommendation got stronger, and it held even though the benchmark that had been supporting it changed underneath it.
• Build your own mini benchmark. Theo's standing advice, and one of the few durable defenses: keep a folder of real tasks where agents have failed you, and rerun it on every release. His SnitchBench and Skatebench got outsized attention from researchers precisely because independent, reality-grounded tests are nearly extinct.

Every coding leaderboard this blog has cited in the last twelve months has died of contamination, verifier rot, or both. Trusting the newest one more would repeat the mistake. DeepSWE's own authors expect saturation within a year and are already building a private holdout for that reason; the practical response is to hold every public number loosely and keep your own task folder as the constant.