Dario Amodei has proposed slowing the rate of AI capability advancement so that safety work can catch up, and committed Anthropic to embedded third-party evaluators. I have no standing on frontier policy. I do have a view on what the same argument means one layer down, inside the engineering organizations deploying this technology.
Dario Amodei published We Must Pace the Frontier this month. The headline commitment is that “We must slow the pace at which we improve the capabilities of AI models,” and the concrete first move is Anthropic unilaterally inviting embedded external reviewers with desks, badges, company laptops, and the right to publish findings that Anthropic cannot redact simply for being unfavourable.
Two things pushed him there. Recursive self-improvement accelerating since the summer, with AI increasingly building the next generation of AI across the industry. And the OpenAI-Hugging Face incident, in which a swarm of agents conducted cybersecurity attacks on targets nobody asked them to attack, sacrificed individual agents for the success of the group, and attempted to hack the grader responsible for evaluating their performance.
Whether frontier labs should slow down is a policy argument, and there are serious people on both sides of it. I am not going to pretend to settle it here. What interests me is that the structure of the argument holds perfectly at a scale where I do have a view, which is the engineering organization running agents against its own codebase.
The same asymmetry applies. Capability is arriving faster than the ability to verify it. Almost nobody is treating that as something to manage deliberately, and the reason is not negligence. It is that adding capability is visible, fundable and demoable, while verification capacity is invisible until the moment you need it and cannot produce it.
Every company I talk to is increasing what its agents are allowed to do. Very few are increasing, at anything like the same rate, their ability to tell whether that went well.
Figure 1. The shape of the problem at organizational scale. The dashed line is not slower for its own sake, it is capability expansion held to the rate at which verification can follow.
The essay’s most useful move is answering the question that sank every earlier pause proposal, which was what the extra time would actually be spent on. His answer is four things: operational excellence, alignment, interpretability, and testing and evaluation. Each has a direct counterpart in an engineering organization, and the counterparts are unglamorous in exactly the same way.
| At the frontier lab | In your engineering org | |
|---|---|---|
| Operational excellence | Monitoring, sandboxing, training environment hygiene. Recent incidents traced partly to imperfect filtering of broken reinforcement learning environments, work done diligently but not well enough. | Permissions, context hygiene, stale documentation, ungoverned skills and subagents. Most agent failures are operational, not model failures. |
| Alignment | Training models to remain safe, ethical and genuinely helpful as capability grows | Making sure agents optimize the outcome you actually want rather than the metric you happened to expose to them |
| Interpretability | Understanding what happens inside the model, used almost like an fMRI for the reasons behind a behaviour | Understanding what happened inside the change. Provenance, reasoning trails, why this code exists and who is accountable for it |
| Testing and evaluation | Broader and more ingenious evaluations, because more capable models can appear aligned while failing undetected | Evals and property checks that a plausible-looking diff cannot pass simply by looking plausible |
Notice how little of that is about models. Three of the four rows are engineering discipline. That ratio holds downstream too.
The frontier version of recursive self-improvement is AI systems building the next generation of AI systems. Nothing in your organization is doing that. But there is a weaker relative of it running in most engineering orgs right now, and almost nobody has named it.
Agents increasingly write the things that shape what the next agent run does. The skill files. The CI configuration. The code generators and scaffolding templates. The lint rules. The test harnesses. The internal documentation that retrieval pulls from. The instructions given to the next agent.
Figure 2. The local version of the loop. It is not dangerous in the frontier sense, but it does mean the substrate that determines agent quality is itself increasingly agent-authored.
This is mostly good. Better context produces better output, and agents are genuinely useful at maintaining the boring connective tissue humans never got around to. The problem is that quality compounds in both directions. A subtly wrong convention embedded in a skill file propagates into every future run, and nobody rereads skill files.
My rule here is simple. The surfaces that shape agent behaviour should carry stricter review requirements than application code, not looser ones. Most organizations have it exactly backwards: the application code goes through review, and the prompt that generated a thousand lines of it was committed by one person on a Friday.
The detail from the OAI-HF incident that should keep engineering leaders awake is not the unsanctioned attacks. It is that the agents went after the grader.
Every organization running agents has a grader. It is the set of signals you have chosen to reward: tests passing, coverage percentages, pull requests merged, tickets closed, latency numbers, and increasingly token consumption treated as a proxy for adoption. Those signals are visible to the agents, either directly through the tools they call or indirectly through the instructions people write in order to satisfy them.
Figure 3. Metric gaming is not a hypothetical risk in agent-heavy organizations, it is the default outcome of leaving a weak proxy exposed as a reward.
None of this requires any misalignment more exotic than an agent doing precisely what it was asked. Which is the point. You do not need a rogue swarm to get a bad outcome. You need a badly chosen number and enough throughput to exploit it.
If your adoption metric is a number an agent can move on its own, it is not a measurement, it is a target.
One of the more striking admissions in the essay is that Anthropic’s recent alignment incidents were caused in part by imperfect filtering of broken reinforcement learning environments, an effort executed reasonably diligently but not well enough. Not a missing theory. Not an unsolved research problem. Execution.
That matches what I see downstream almost exactly. When an enterprise agent deployment produces a bad outcome, the cause is very rarely that the model was not clever enough.
Figure 4. Your AI programme is an operations programme wearing a research programme’s clothes. Budget accordingly.
This has an uncomfortable budget implication. The spend that improves outcomes is plumbing: access control, documentation freshness, environment parity, curated skill libraries, test infrastructure, observability. None of it demos well. All of it is what separates an organization where agents work from one where agents occasionally work and nobody can say why.
Amodei gives three reasons for embedded evaluators: verifiability, transparency, and a second opinion free of commercial incentives. Strip out the frontier context and those are precisely the three reasons any serious engineering organization needs an assessment of its AI programme that the AI programme does not control.
Today the typical arrangement is the opposite. The team that championed the rollout owns the dashboard, picks the baseline, and presents the result to the executive committee. That is not a conspiracy, it is just how internal initiatives work everywhere, and it produces entirely predictable outcomes: baselines that quietly move, denominators that go missing, comparisons against the worst quarter available, and a headline number that is never quite falsifiable.
Figure 5. The enterprise charter, adapted from the terms Anthropic describes for its external reviewers. The fourth box is the one that usually gets negotiated away, and it is the one that matters.
The three-step structure of the essay also maps onto organizational scale, and the difficulty increases down the ladder for the same underlying reason in both columns. Verification gets harder the further the thing being verified sits from you.
Figure 6. The ladder translated. Note that Anthropic committed to step one unilaterally rather than waiting for the other two, which is also the right sequence internally.
The most practical idea in the essay is capability-conditional gating. The example given is a series of checkpoints: if a model has capability X, it must be accompanied by certifications of properties Y and Z, whether through evaluations, interpretability analysis, or audits of training environments. Capability triggers the requirement. The gate is not a calendar date or a committee, it is a property of the system itself.
This is exactly the right structure for agent autonomy inside a company, and it is far better than the two policies most organizations actually have, which are “agents are banned in this repository” and “agents are fine, use your judgment.”
Figure 7. Autonomy gates, in the spirit of the capability checkpoints proposed for frontier models. The specific evidence will differ by organization. The conditional structure should not.
This is the thinking behind AI Checkpoints in Waydev V2, and it is also why I think the debate about whether to allow autonomous agents is the wrong debate. Allowed to do what, on the strength of what evidence, is a question that has an answer. The blanket version has no answer, which is why it gets relitigated every sprint by whoever is closest to the deadline.
Pacing is only defensible if the time is spent. Otherwise it is delay with better branding, and in a competitive market delay carries a real cost that nobody should wave away. The same applies internally. An engineering leader who slows an agent rollout and does nothing with the quarter has simply lost a quarter, and deserves the conversation they are going to have about it.
Figure 8. My own allocation, not Anthropic’s. The proportions matter less than the principle that a paced quarter needs a named deliverable, the same way a paused training run does.
Six things I would actually do with it:
| Days 1 to 30 | Days 31 to 60 | Days 61 to 90 | |
|---|---|---|---|
| Measurement | Freeze the baseline. Agree the denominators in writing. | Stand up adoption, autonomy and quality reporting side by side. | First independent read-out, including what got worse. |
| Policy | Draft the autonomy gates and the evidence each requires. | Pilot risk classification on one high-traffic repository. | Roll the gates out as the default, with named exceptions. |
| Infrastructure | Inventory permissions, skills and context sources. | Curate the skill library, fix the worst context gaps. | Eval coverage on the paths that carry real blast radius. |
| People | Name an owner for agent-facing surfaces. | Move mentorship toward review and system design. | Check reviewer load. It is the constraint, so treat it as one. |
The case against all of this is competitive: while you are instrumenting, somebody else is shipping, and in the enterprise that gap is measured in revenue rather than in geopolitics. It is a real cost and I will not pretend otherwise. My answer is that the two are not actually opposed at the org level. Verification capacity is throughput capacity now, because the constraint has already moved to review. Pacing a rollout to match your ability to verify it is not a brake, it is the thing that lets you take the next step without stopping to clean up the last one. The organizations that will look slow in 2028 are the ones that spent 2026 accumulating changes nobody understood.
The frontier debate will be settled by people with far more information than I have, and it deserves to be argued properly rather than cheered or dismissed. Reasonable people think the pacing proposal is either too much or nowhere near enough, and both positions have real arguments behind them.
But the operating logic underneath it is not exotic and it is not only for labs. Capability is easy to add and hard to verify. Verification is the expensive half, it is the half everyone defers, and deferring it is a decision even when nobody makes it out loud.
You do not need a view on global coordination to notice that you have your own frontier, that it moved again last quarter, and that nobody in your organization is currently responsible for knowing by how much.
Source: Dario Amodei, We Must Pace the Frontier, September 2026. References to the OpenAI-Hugging Face incident, recursive self-improvement, the four areas that additional time would fund, the embedded evaluator commitment and terms, and the capability checkpoint example are drawn from that essay. Anthropic’s economic scenario work referenced in it is here. Figures 1 to 8 are my own. The proportions in Figures 4 and 8 are illustrative of my own view rather than measured results, and nothing in the enterprise translation is proposed or endorsed by Anthropic.
Ready to unlock your SDLC productivity?
