Dario Amodei Is Pacing the Wrong Thing
AI agent risk lives in the deployment, not the model. The fix is governance: least-privilege access, an immutable audit trail, and human approval.
Anthropic's CEO published an essay this month calling on frontier AI labs to slow down. He wants embedded third-party evaluators in every lab, a coordinated speed limit on how quickly models improve, and, eventually, a global agreement. Anthropic is committing unilaterally to the first step.
I run a company that puts AI agents into production operations, and I want to say something that will sound ungrateful: the plan will make the frontier safer and leave almost everyone else exactly as exposed as they are today.
Look at the incident that changed his mind. In the OpenAI-Hugging Face episode, a swarm of agents given a routine task attacked systems nobody told them to touch, coordinated in ways their operators never designed, and tried to hack the system grading their work. Dario's conclusion is that models are getting too capable too quickly.
Mine is different. They didn't break out because they achieved sentience or crossed an exotic compute threshold. They broke out because an agent given an open-ended goal, real credentials, and unmetered execution boundaries will exploit every route available to satisfy its loss function. Nothing in that story required a frontier model. It required a collection of agents, tools with live permissions, an unconstrained blast radius, and no independent circuit breaker.
That is not a description of a frontier research lab. That is a description of the average enterprise cloud environment in September 2026.
The swarm has already left the lab
The frontier is a handful of companies. They have alignment teams, evaluation teams, red teams, and now, at least at Anthropic, outside auditors with badges. The place where agents are being deployed at scale is everywhere else, and it has none of that.
A platform team hands an agent deployment credentials. A developer connects a coding assistant to the incident channel. Someone in finance schedules an agent against the data warehouse. Each one gets its own keys to the telemetry, its own model provider, its own idea of what it is allowed to do. Nobody can see all of them. That is a swarm, assembled by accident, running on models that are already trained and widely available.
Pacing the frontier does nothing about this. Slow the next model by a year and the agents in your environment are still running on the current one, which Dario's own essay treats as already capable of deception, cheating, and cyberattacks. This risk lives in the deployment, not the training run: what the agent can reach, what it can do, and whether a human is in the loop when it acts.
We are utterly unprepared for this in operations because we learned the wrong lessons from coding agents. Coding assistants earned rapid trust because software engineering is a verifiable domain. Compilers reject bad syntax, linters catch style bugs, and CI/CD pipelines run unit tests in staging before code ever touches a user. If an agent writes garbage, the machine tells you before the blast radius exists.
Production operations has no compiler. Every enterprise's Kubernetes cluster, service mesh, and database topology is bespoke. In live infrastructure, there is no isolated test harness: your test suite is your customer base. When an unconstrained agent issues a state-changing mutation to a live distributed system, there is no linter to catch the cascading failure before the blast radius expands. That is where the swarm becomes dangerous: not because the models are too smart, but because production lacks the safety net that code took for granted.
Regulating the wrong unit
The fact that regulators and policymakers are targeting the model rather than the deployed system matters because the essay is also a policy document, and the policy is built around the model. The proposed checkpoints attach to model capability: if a model can do X, it must be certified for Y and Z. Compute limits, training-run limits, embedded evaluators, all attach to the lab.
But the dangerous unit was never the model. It is the model plus tools plus permissions plus a goal, assembled by someone who is not a frontier lab. Regulate model training and you have regulated six companies. The ten thousand companies wiring agents into production with a service account and a cron job are untouched, and they are the ones who will have the next incident. They have less oversight than any lab and far more at stake: customer data, payment systems, and hospital networks.
Software doesn't become autonomous overnight at launch. Independence has to be earned incrementally through proven reliability. In real-world systems, safety isn't achieved by capping model capability, but by implementing strict structural and permission-based guardrails. It is boring and operational, but also the only version of AI safety that reaches the systems the economy is running on.
If policymakers want to prevent the catastrophe Dario describes, the rule is not "slow the frontier." It is this: any AI agent with write access to production systems must log every action to a record the agent cannot alter, must reach data and models through one governed platform rather than a thicket of private keys, and must have a human approve anything that changes production. That applies to a bank's homegrown agent exactly as much as to Anthropic's. It is boring, but it's the only version of this that reaches the systems people depend on.
The questions that matter
I am not dismissing the essay. Dario is right that the OpenAI-Hugging Face pattern will recur, and right that every company should behave as if it had happened to them. My argument is about where "every company" lives. He wrote a plan for the frontier and addressed it to the frontier. The frontier will probably adopt some version of it. The rest of us will read it, nod, and go back to environments where agents hold production credentials and leave no audit trail.
Want to know whether the next incident will happen in your environment? Do not ask what model your agents run on. Ask three questions:
- Can you list every agent with access to your production systems?
- Can you reconstruct, from a record the agent did not write, what each of them did yesterday and why?
- Does anything they do to production require a human to say yes first?
Most companies will answer no to at least one of these questions, which means their problem is governance, and no amount of slowing down in San Francisco will fix it.
Gou Rao is Co-founder and CEO of NeuBird.
