Kimi K3: benchmark claims versus measured results

What actually shipped

Moonshot unveiled Kimi K3 on July 16: "the world's first open 3T-class model." Live on their API today at $3/M in, $15/M out; weights promised on Hugging Face by July 27, technical report to follow. Until that upload happens, "open-weight" is a claim in a press release: no weights have shipped yet. Worth remembering for the next eleven days. One checkpoint, one reasoning tier ("max"; lower tiers listed as "planned"), 1M-token context, vision input.

And the parameter count is either 2.8T (Moonshot) or 2.7T (Fortune), depending on who you read; the active count is undisclosed entirely. The release is one day old and the most basic spec is already ambiguous..

The numbers, sorted by who measured them

House rule since the SWE-Bench Pro funeral: vendor numbers and independent numbers don't get to sit in the same column.

Independent (Artificial Analysis, Arena):

An open-weight model at #1 on a blind developer-preference arena, above Fable 5, is a first. It's one arena, frontend-only, but a real one.

Vendor (Moonshot's launch table, which, credit where due, footnotes its own harness-mixing): DeepSWE 67.5 vs Fable's 70.0 and Sol's 73.0 (the 67.3 on Datacurve's own leaderboard roughly confirms it); Terminal-Bench 88.3 on their harness vs the 85.0 AA measured independently, a 3.3-point gap between the two measurement methods on the same benchmark. Ofir Press flags that their Program Bench number uses average-implementation-percentage, which flatters partial work. This is the usual pattern for vendor benchmark tables at launch: the best-looking numbers up front, methodology caveats in the footnotes.

Not Mythos-class, and Moonshot says so

The launch post's limitations section states plainly: "K3 nonetheless exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT 5.6 Sol." Labs rarely volunteer this kind of admission in a launch post. Moonshot flags sensitivity to thinking history (quality degrades if the harness drops reasoning context, or the session switches models partway through) and excessive proactiveness (on ambiguous instructions it may act without waiting for confirmation; constrain it in AGENTS.md). Those are the long-horizon reliability properties that separate a Mythos-class orchestrator from a very strong executor, and Moonshot named them unprompted.

The independent picture agrees: 2.8 points below Fable on the composite, a knowledge-reliability score half of Grok's, and a hallucination rate that AA measured regressing to 51%, from K2.6's 39%. The people who tested it land the same way. Ethan Mollick: "Very good model, not Sol Max or Fable, but great open weights… closest to the frontier yet." Simon Willison found the vision input solid and the pelican acceptable, at 25 cents a pelican with 13,241 reasoning tokens included. MIT/DeepMind's Michiel Bakker, on the capability jump: "These results seem impossible to explain through distillation alone." And OpenAI's own Dean Ball, after hands-on agentic sessions, called it "very good," matching "the best public models from Q1 2026." That comparison point is six months old, not the current frontier.

How they got here: the technique

K3's launch post reads less like marketing than an architecture changelog, and the architecture is the focus of this release:

• Extreme sparsity. 896 experts, 16 active per token: roughly 1.8%. For scale: K2.6 ran 1T total / 32B active; DeepSeek V4 Pro 1.6T/49B; GLM-5.2 753B/40B. K3 nearly doubles the biggest open total ever while activating a sliver of it. (Active count undisclosed, so per-token compute comparisons are guesswork until the tech report.)
• Kimi Delta Attention (KDA): a hybrid linear-attention mechanism for information flow across the 1M context, plus Attention Residuals, which retrieve representations selectively across depth instead of accumulating uniformly.
• Quantile Balancing: expert routing balanced directly from router-score quantiles, deleting the usual pile of load-balancing hyperparameters.
• Per-Head Muon: the Muon optimizer extended to optimize each attention head independently, plus a SiTU activation and quantization-aware training from SFT onward (MXFP4 weights / MXFP8 activations).
• Net claim: ~2.5× scaling efficiency over K2. Vendor number, unverified, tech report pending. Emad Mostaque's estimate of a $15-25M training cost at ~1e25 FLOPs, if directionally right, would explain why "Scale isn't all you need!" was his conclusion.

The bill: this thing is slow

The architecture that produces those benchmark numbers has costs on the other side. AA measured 62 tokens/second (peer median ~72.7), ~2 seconds to first streamed chunk but 32 more seconds of reasoning before the first answer token, ~42 seconds end-to-end for a 500-token response. It also consumed 130M output tokens to complete the Intelligence Index against a 63M peer median, roughly double. Three compounding causes: only "max" reasoning effort exists at launch, so every request pays the full thinking tax; the model is verbose by temperament; and the 896-expert layout is why Moonshot recommends 64+ accelerator supernodes for serving. At $3/$15 per million, the full AA index cost $2,691 on K3 vs $2,824 on Sol and $5,631 on Fable. The result is cheap per token, token-hungry in aggregate, and slow in wall-clock time.

What to actually do

• Pilot it now, via API, on frontend and agentic coding: the Arena #1 and Terminal-Bench 85 are real signals, and per-task cost is Sol-tier or better.
• Don't orchestrate with it yet. Thinking-history sensitivity + excessive proactiveness + 51% hallucination rate is the wrong profile for the top of an agent tree. Keep a Mythos-class model in that seat; give K3 the execution lanes.
• Don't put it in anything latency-facing. 34 seconds to first answer token is a batch-job number.
• Wait for July 27 before repeating "open-weight," and for the tech report before repeating "2.5× efficiency."
• Recalibrate your priors: the gap between the best open model and the best model, full stop, is now 2.8 points on the only independent composite available. Twelve months ago it was a chasm. That gap is the subject of part 2.