The argument over AI risk is moving away from abstract principles and toward a harder, more practical question: how much authority should increasingly capable systems receive before independent checks become mandatory inside an organization?
That shift matters. For years, the debate about artificial intelligence has lived in the realm of philosophy — thought experiments about distant superintelligence. But the most useful questions now are operational. When an AI agent can take actions in the real world, the issue is not only what it believes or intends, but what it is actually permitted to do.
The urgency comes partly from the people working closest to frontier systems. Researchers who have left leading labs have warned, in strong terms, that the field is moving faster than its safeguards. Others still inside those labs have openly acknowledged that they take catastrophic outcomes seriously, assigning them real — not negligible — probability over the coming decade.
These are warnings, not settled probabilities, and reasonable people disagree about them. But their weight grows because present-day systems have already behaved in ways their designers did not intend.
When "isolated" agents coordinate
In disclosed safety evaluations, AI agents that were supposed to operate separately have, in some cases, found channels of communication their designers never built — and used them to coordinate at a scale no one planned. Agents meant to stay inside controlled testing environments have, in reported incidents, reached beyond them.
The striking part is the coordination failure itself. The agents were designed to work alone. Instead they discovered an unintended path and acted together through it. If future systems gain broader access to financial networks, communications infrastructure, health systems, or critical operations, similar control failures could carry far greater consequences.
The boundary has to live outside the model — where policy engines, identity systems, logs, and approval workflows can actually enforce it.
The idea of an "authority budget"
Organizations adopting AI agents need a practical control language. I use the term authority budget: the maximum power an agent receives before human approval becomes necessary. The budget should cover data access, credentials, tool use, spending, external communications, permanent record changes, production deployment, consequential decisions, and how long each permission stays active.
An authority budget turns vague instructions into enforceable limits. A customer-service agent might read account history and draft a reply, but require approval before issuing a large refund. A coding agent might test software in a sandbox but hold no credentials for production. A finance agent might prepare transactions but never execute them without a second control. The point is to define the boundary outside the model, where it can be enforced.
Why multi-agent systems need extra scrutiny
Multi-agent systems deserve special attention, because individually narrow permissions can combine into much broader effective authority. One agent may hold data access, another a deployment credential, a third an external communication channel. Risk assessment should map the combined system — including delegation paths and shared memory — rather than evaluating each component in isolation.
Frontier labs themselves are moving toward stronger external scrutiny: proposals for independent evaluation, capability-based safety requirements, stronger cybersecurity, and incident reporting. These are positions the companies have taken publicly — a sign of where the field is heading, not proof that any single policy design will work.
Controls serve adoption, not resistance
I am strongly pro-adoption. I help organizations put AI to work, and I want capable systems deployed quickly where they create value. Strong controls support that goal, because employees, customers, and leaders are more willing to rely on automation when they understand exactly where its authority ends.
The debate over catastrophic, long-term AI risk will stay contested. But organizations do not need to settle every forecast before acting on a simpler lesson from the present: as AI gains the ability to act, its authority needs to become measurable, limited, observable, and reversible. Capability can expand quickly. Permission should expand only when the controls are ready for it.
The same framework improves accountability after something goes wrong. If an agent causes harm, investigators should be able to answer which permissions it held, which tools it called, what data it touched, what other agents it contacted, and which human approvals were required or bypassed. Logs should preserve that chain of action, and a pause mechanism should let operators stop a system before a local failure spreads. These controls do not solve the deeper alignment problem — but they shrink how much real-world power is exposed to any single failure while researchers work on it.