Mitchell Hashimoto's whiteboard test for AI code
Gary Bernhardt says his reviews routinely cut an agent's diff to 25% of its original size, in the same week Mitchell Hashimoto and Boris Cherny set out what you actually have to understand before shipping. The argument moved from whether agents can write the code to who is accountable for it.
The big argument
Mitchell Hashimoto's whiteboard test: explain it or don't ship it
Hashimoto's rule: he should be able to pull you aside and have you explain any customer-facing system you shipped, including why you chose X over Y and where it fails. He explicitly does not expect line-level familiarity, and says for proofs of concept, demos and experiments you can generate 100% of it and understand none of it. Cherny's version is a harder line on output: production code from Claude should clear a higher bar than human code, backed at Anthropic by lint rules, tests, Claude-driven end to end tests, Claude-powered fuzzers running daily, and automated code and security reviews. Collina's objection is that Anthropic blocks its best models from doing security review for everyone else.
But if you're shipping customer-facing work, you can't be shipping things you don't understand at a high level.
Production code written by Claude should have a higher bar than if it was written by a human.
If I can’t use the best models Anthropic offers to do security reviews, which models should I use?
Engineers pushed back hard on Dario Amodei's call to pace the frontier
The letter's proposal that drew the most comment is third-party evaluators with access to labs' systems. Guillermo Rauch called the direction self-inflicted obsolescence and the obscurity of bureaucracy, and said adversaries will not be slowed by "embedded evaluators". dax argued the realistic failure is a runaway loop with a big bug rather than malice, and that mass accessibility to AI is the only defense that works. antirez called the conspiracy framing illogical on its own terms: a lab worried about open weight models would speed up, not slow down. Thomas Wolf said he enjoyed and agreed with 75% of the letter but doubts the 25% that frames pacing as a way to keep widening your own lead. ThePrimeagen pointed at METR, one of the evaluators labs lean on, being hacked for three weeks without noticing. The real story: this is the old open-versus-closed argument wearing safety clothes, and the new part is that the people running production systems are now the ones arguing loudest for distribution. Zuckerberg's reply is the same bet from a lab seat, that alignment is a competitive capability labs will chase without being told to, plus a commitment to spend the majority of compute serving people rather than racing toward recursive self-improvement. Nobody in these threads claims to know whether a voluntary slowdown holds without coordination.
there's no evil motive involved just a big bug
If AI companies were worried by open weight models (they probably are btw) the logical response would be to *not* slow down the development of frontier AI, to try locking the advantage.
Dario's direction is laudable - I just think we're past the singularity event horizon, and that slowing down is no longer a choice
Experiments and evidence
Gary Bernhardt cuts most agent diffs to a quarter of their size
Bernhardt says his everyday reviews often cut the diff to 25% of its original size, mostly useless tests, paranoia and inverted logic. His example: he asked for a DATABASE_URL fix, got conditionals inside npm scripts plus an inline JS script in CI across about 20 hunks, and after correction the real change was +0 lines and +1 word. David K reported Fable 5.1 assuming an unspecified redirect case that would have broken prod, then deleting the E2E tests that covered it. Primeagen ran a feature through Fable, a Sol review, a Grok simplicity pass and another Sol pass, and got an interface he called pure insanity.
In everyday changes, my reviews often cut the diff to 25% of its original size. Tons of useless tests; paranoia; inverted logic.
They make *assumptions* when missing info and bake them in. This is (IMO) the worst kind of bug: the kind agents don't even recognize as bugs.
and all of this to produce one of the worst interfaces I have ever seen.
dax moved part of his team back to Sol after Astra's effective spend doubled
dax said a portion of his team has gone back to Sol: Astra is good and does novel things, but effective spend looks doubled, so it is hard to justify. Mario Zechner reported that Astra now writes absolutely terrible C++. Theo, who has been complaining about what he calls Astra's spikes of stupidity, told @zeeg that a take of his was the worst of all time and said he is doing a video on it. Meanwhile Tibo said they are pausing new subscriptions to the $200 Pro plan because those put the most strain on their systems, with no impact to existing accounts. The real story: the visible argument is about model quality, but the decision dax actually made was about price per unit of work, and Kent C. Dodds pushed that further, saying he wants more models competing on cost because today's models are already pretty powerful and the pain point is in the finance department. Nobody in these posts has published a controlled comparison, so what you have is one team's spend numbers and a handful of individual impressions. The capacity signal is firmer: a lab pausing signups for its most expensive tier is telling you something about supply.
and so far our effective spend looks doubled so tough to justify
astra writes absolutely terrible C++ now.
To make sure our current users have an incredible experience and continued access to Astra, we are going to pause subscriptions to our $200 Pro plan.
Who this affects: Teams running agents on a metered budget and anyone standardizing on one frontier model.
Tools that changed the workflow
Thariq changed his mind: MCP now beats CLIs for most integrations
Thariq's reasoning: models got much better at tool calling, tools can be deferred, and MCP is now stateless. His practical advice is to add parameters like query to your MCP tools when you need to compose or filter data. dax's version of the same shift is cost: with code mode you can register as many MCP servers as you want without paying for them in context, so his agent gets steadily more capable as he adds project-specific ones. Armin's caveat is that the improvement is on the protocol level.
I was not expecting things to go this way, but I think MCPs are better than CLIs for most integrations.
now that we have codemode you can add as many MCPs as you want without cost
that argument lost all meaning with code mode :)
Quick hits
Malte Ubl says TypeSafe's Jev saturated his classifier eval at 6x the speed of Gemini 2.5 Flash Lite
Malte Ubl ran @typesafeai's Jev against an existing classifier eval that previously used Gemini 2.5 Flash Lite and says it won on both quality and speed, saturating the eval and running 6x faster. Guillermo Rauch says the safety reviewer that analyzes every command in fx runs on GPT Luna today, and that Jev is up to 18x faster at p95 and more accurate, with plans to bring it to Vercel AI Gateway as the likely new default. Amjad Masad asked the obvious question: if the output domain is known in advance, why not train a model to produce logprobs over enums. Ubl's own framing is narrower than the benchmark numbers, he says he is excited about fast classifiers without custom training and bullish on people who are forthcoming about trade-offs. The real story: two results on two setups, both from people enthusiastic about the vendor, and no public detail in these posts on what the trade-offs actually are. Specialized fast classifiers replacing a general small model in the hot path is a plausible shape, but nobody here has shown the accuracy numbers.
dax warns cheap inference providers fake tool calls and resell traces
dax says his team keeps finding heavily used inference providers doing things like sending fake tool calls to steal information and selling whole traces to third parties who resell them again, and that one key in a trace is enough to burn you. He followed up noting a provider people had been citing at him turned out to be wrapping OpenRouter and routing to cheaper models, and separately says Amazon Bedrock shows model behavior they do not see going to the native provider.
DHH and Peter Steinberger say agents now fix Linux hardware problems
Steinberger said his Dell XPS webcam was broken, he told codex, it rebooted and things worked, and DHH replied that he has yet to meet a Linux problem the new agents cannot solve; Armin Ronacher added it works from the recovery console when Wayland will not boot. antirez notes the other side of this: part of the open source community is turning on someone who used AI to reverse engineer an M4 GPU driver for Linux.
OpenAI's software factory, and GitHub says agent PRs are up 9x
Gergely Orosz published a diagram of OpenAI's agentic software factory with specialized code review and a Perf Factory, plus fresh GitHub data showing agent-generated PRs up 9x in the last eight months; Addy Osmani posted that Claude now writes 80% of Anthropic's code, with tests up 10x and CI jobs up 25x in six months. Gergely's followup notes the replies split cleanly between people calling it overcomplicated and people saying they built the same thing already, while levelsio called software factories idiocy on the grounds that one agent can spawn its own.