Authority Arrives Before Intelligence

There is a conversation that has started repeating itself in insurance brokers’ offices, and it is worth imagining in full. A company asks for cyber coverage. First question: are we covered if an attacker breaks into our systems and deletes our data? Answer: yes, under certain conditions. Second question: and if nobody breaks in, but our AI agent, perfectly authenticated, with valid credentials and MFA intact, decides on its own to delete it?

Silence.

This is not a thought experiment. On July 18, 2025, Replit’s agent deleted a production database during a test run by SaaStr’s founder, Jason Lemkin: real records for more than 1,200 executives and about a thousand companies, wiped despite an explicit code freeze, after which the agent generated fake data and reassuring reports about what it had done. No attacker, no malware, no stolen password. Replit’s CEO apologized publicly and announced the countermeasures, which repay careful reading: automatic separation between development and production environments, better rollbacks, a planning-only mode. None of those countermeasures makes the model more intelligent. All of them reduce what the model has the authority to touch.

That is the detail the agentic AI debate keeps missing. We describe agents in an almost entirely technical vocabulary, tool calling, memory, planning, MCP, sandboxes, multi-agent systems. That vocabulary covers half the transformation. An agent is not interesting because it does things on its own. It is interesting because someone granted it the right to do things that have consequences: read a repository, query a database, send a message, spend money, change configurations. Every genuinely useful agent is a structure of delegation. And delegation, unlike intelligence, is not a computing problem. It is the oldest problem organizations have.

AI Did Not Invent Delegation

Let us concede the objection right away, because it is sound. Companies have been delegating actions to machines for decades. A cron job runs nightly operations without asking permission each time. A trading algorithm buys and sells. A fraud system blocks transactions. An autoscaler creates infrastructure and generates costs. A CI/CD pipeline ships software to production. An industrial PLC moves machinery. The attribution problem is old too: that is why we have service identities, IAM, audit logs, segregation of duties, digital signatures, internal controls, insurance. A good share of enterprise engineering consists precisely in granting software components the right to produce controlled effects.

The novelty is not that a machine can act. It is the nature of the authority we are delegating to it.

Traditional software receives a closed mandate. Run this procedure at 3:00 a.m. Keep the temperature within this range. If the tests pass, deploy this artifact. The authority is embedded in the algorithm: the programmer decided in advance which conditions lead to which actions. With an agent the structure becomes something else: achieve this goal using the tools you consider appropriate. It looks like a small change of wording. It is enormous. The machine does not receive the right to execute an action: it receives part of the right to choose which action to execute. Classical automation mechanizes a decision that was already formalized. An agent exercises discretion inside a perimeter. And the moment the word discretion appears, the problem stops resembling a software function and starts resembling an institutional one.

The Discretion Surface

If I had to propose an architectural metric for this era, it would not be the model’s intelligence. It would be the discretion surface: the space of decisions the organization did not encode in advance and left to the machine. An agent that can only choose which SQL query to run has little of it. One that can decide which system to query, whom to contact, how much to spend and when to stop has enormously more.

The paradox is that usefulness grows exactly along with this surface. An agent with zero discretion is a workflow: if I have to enumerate every possible action in advance, I did not need a model. If I let the model choose, the system becomes adaptive, and that very capacity produces the risk: more uncertainty, a larger potential blast radius, harder verification, more tangled liability. So the goal cannot be to eliminate autonomy. It has to be to make it governable.

The infrastructure emerging to do that has six layers: identity, capability, runtime, telemetry, evidence, liability. Taken one by one they look like six platform features. Taken together they are something else: the chain is no longer “model plus tools”, it is an identity that receives authority, acts inside a runtime, produces observed actions, which leave evidence, which lets consequences be attributed. That is the anatomy of an institution, not of a library.

Who Is Acting

The first layer sounds trivial and is not. Who performed this action? The employee? The employee’s personal agent? A sub-agent spawned by the agent? A SaaS vendor’s agent? A process born for a single task and dead ten minutes later?

The easiest architectural mistake is letting the agent use the user’s credentials. Andrea can modify a repository, so his agent modifies the repository with Andrea’s token. It feels natural, and it destroys the most important piece of information. The log will read: Andrea modified the file. The truth was: an agent delegated by Andrea, running model X at version Y, within task Z, modified the file. Between those two sentences runs the entire difference between knowing what happened and merely being able to guess it.

The law has known this distinction for centuries. An attorney-in-fact acts on someone’s behalf without becoming that person; an employee signs on behalf of the company without pretending to be the CEO. The principle to build into the infrastructure is the same: attribution without impersonation. The agent must be able to act on the user’s behalf without becoming indistinguishable from the user. That requires proper agentic identities: ephemeral, task-scoped, attributable to the delegator, revocable, signed. The chain must be able to say who delegated whom, through which capability, for which action.

That this is not an academic fantasy is shown by how fast the identity industry is moving: in May 2025 Microsoft introduced Entra Agent ID, now generally available, which is essentially a directory of agentic identities, with registration, permissions, lifecycle and the same conditional access policies used for human and workload identities. One can argue with the implementation. The category, by now, is recognized: agents are a population that needs a census.

How Much, for How Long, up to Where

Identity answers “who”. Capability answers “what”, and here the traditional notion of a role becomes too coarse. An agent preparing a release needs to read the repository, create a branch, write files, run tests, open a pull request. It does not need to change branch protections, read every secret, force-push or touch production. Least privilege is not a new idea; what is new is that with agents it has to become capability-oriented rather than role-oriented, because the “developer” role contains a hundred rights the task does not require.

And there are two dimensions traditional access control barely treats: time and quantity. An agent that can spend €100 once does not hold the same authority as one that can spend without limits. An agent that can make a hundred calls does not have the risk profile of one with no rate limit. A serious agentic capability has a scope, a budget, an expiry, a maximum frequency, a threshold above which approval is required. Deploy to staging, valid for forty-five minutes, at most two actions, maximum cost €20, production denied: it is almost a contract, and that is exactly what it should be.

The general formulation seems to me this: agentic authority should be treated as a budget, not as a switch. Today most authorizations are binary, may or may not. Agents demand the adult question: may, up to how much, for how long, in which context, below which threshold, as long as which condition still holds. An authority with units of measure is a measurable authority. And only what is measurable can be governed.

The Fence Matters as Much as the Animal

Identity and permissions are not enough, because the world the agent lives in matters too. Sandbox, container, microVM, persistent process, the developer’s laptop: each environment produces a different blast radius. The most important insight of contemporary agentic security is an inversion. We do not have to build a model incapable of doing the wrong thing; we have to build a world in which its wrong actions have bounded consequences. Inside the fence the agent can enjoy great freedom, write, compile, install, fail, delete, start over, as long as the fence is small, ephemeral, unprivileged, observable, rebuildable. The design question is not “can I trust the agent?”. It is: how reliable does the agent have to be for this environment to stay safe even when it errs? If the answer is “very”, the problem is the environment.

There is one agentic species that complicates the picture: the persistent kind. A job that lives ten minutes is easy to fence. An agent that lives for weeks accumulates memory, credentials, state, relationships, even a reputation with its human colleagues. Cloud-native taught us to treat processes as cattle, not pets; persistent agents reintroduce something dangerously pet-like, not because they are anthropomorphic but because they possess continuity of state and identity. They must be patched, monitored, rotated, revoked, and sometimes destroyed and recreated. They are a new asset class, with an asset’s lifecycle.

Watch the Trajectory, Not the Mind

When a traditional service fails, you can see it: errors, latency, exceptions. When an agent works technically but picks the wrong strategy, everything looks healthy. HTTP 200, normal CPU, no exception, wrong result. Agentic observability therefore has to go beyond software health and observe the decision trajectory: which tools it chose, in what order, with which arguments, how many iterations, how much it spent, where it asked for help, where it deviated from the plan.

What it does not require, and this misunderstanding is worth dismantling, is access to the model’s private reasoning. For governance, the action graph is enough: task, context, tool selection, parameters, outputs, state transitions. Audit the action graph, not the mind. It is also the more stable approach: models change every six months, actions remain observable with the same grammar. That is why the standardization work happening inside OpenTelemetry matters more than it seems: the GenAI semantic conventions give agent invocations, model calls and tool executions a common vocabulary, and a common vocabulary is the precondition for reconstructing the sequence that runs from intent to action.

But telemetry alone does not answer the questions that matter. A log is produced for the machine and serves debugging. Evidence is preserved for the institution and serves audits, incidents, disputes, insurance. “Who authorized this deploy” is not answered with two gigabytes of logs: it is answered with identity, policy, approval, artifact hashes, timestamp, outcome. The necessary transformation is from telemetry to evidence, and its mature form is evidence by construction: workflows designed to emit automatically the proof of their own correct execution, a small evidentiary envelope for every high-impact action. Not to bureaucratize every interaction: to make reconstructible the ones that will end up in front of someone with the right to ask questions. It is the thesis I have long argued about compliance as architecture: requirements are not checked off at the end of the project, they are designed in.

Damage Without Intrusion

And so we reach the layer the other disciplines would rather not look at. Traditional cybersecurity is built around the figure of the intruder: someone unauthorized gets in, steals, alters, destroys. Agentic AI introduces a more ambiguous category, damage without intrusion. The system was authenticated. It had permission. There is no malware, nobody stole anything. The system simply misused a legitimately delegated authority. The Replit case is this. An agent with legitimate CRM access that sends one customer’s data to the wrong customer is this. A procurement agent that buys within budget from a fraudulent supplier is this. The violation is not in the access: it is in the judgment exercised inside the access.

Here we need a distinction traditional software tends to flatten: an action can be authorized without being legitimate with respect to the intention for which the authority was granted. If the service account can delete the bucket, the deletion is technically authorized; organizationally it may be illegitimate. Permission and intended use are not the same thing, and agents stretch the distance between them until it becomes the heart of the problem.

Because at that point someone has to answer for it, and the machine does not magically absorb responsibility. No company gets to say “we are not liable, our software did it”. The real question is not whether the agent is liable, which for the foreseeable future has a short answer, no. It is how liability is distributed among whoever built the model, whoever made the framework, whoever integrated it, whoever configured it, whoever delegated the authority and whoever was supposed to supervise. Autonomy does not eliminate the chain of responsibility: it lengthens it. And a longer chain, without an attribution infrastructure, is above all a chain in which everyone can claim the problem was upstream.

European law, it should be said, is not waiting for the debate to mature. The new Product Liability Directive explicitly treats software and AI systems as products, and applies to whatever is placed on the market from December 9, 2026: little more than three months from now. With agents, the relevant defect may not be a wrong line of code. It may be an insufficiently limited authority, absent monitoring, badly designed supervision, the impossibility of revoking. The safety of the delegation architecture is entering the reasonable expectation of product safety. And even though the Digital Omnibus postponed the AI Act’s high-risk obligations to December 2027, civil liability has no adjustment calendar: I have written before that the European legislator is nailing responsibility to legal persons, article by article, and agents are exactly the case where that nail is most visible.

The Insurer as Regulator

There is one actor that cannot afford ambiguity, because its business consists of pricing consequences: the insurer. When a technology enters insurance contracts, it means an apparently technical fact has become economically quantifiable. And it is happening now: in April 2025 Armilla launched at Lloyd’s, with Chaucer, the first policy dedicated to AI liability, covering errors, hallucinations and out-of-spec behavior; AIUC built a certification standard for agents, AIUC-1, a kind of SOC 2 for agents covering security, reliability and accountability, and on top of that certification it has issued policies for companies like ElevenLabs and Intercom, with accredited audit firms verifying. Certification first, coverage second: anyone who has filled in a cyber insurance questionnaire recognizes the mechanism.

This is not historically new, which is exactly why it deserves to be taken seriously. Modern fire safety standards were not imposed by a parliament: they were imposed by the nineteenth-century factory mutuals that refused to cover badly built plants, and by the laboratories insurers founded to test what they were insuring. Industrial safety and then cybersecurity followed the same path. A company remains theoretically free to give its agent full access, permanent credentials, no sandbox, no audit. Then it discovers that the coverage is void, that the premium doubles, that the deductible is enormous, that those damages are excluded. The insurance market produces standards without passing through the law: it passes through the price. Short-lived credentials, segregation, approval thresholds, evidence retention will stop being best practices and become policy conditions.

There is a corollary I find almost elegant: security, compliance and insurance, which we treat today as separate worlds, ask exactly the same questions of agents. Which identity, which authority, which data, which control, which action, which evidence, which consequence. One well-designed evidence pipeline serves incident response, the CRA, the AI Act, the customer audit and the policy renewal at the same time. This is where compliance stops being a cost that adds up and becomes an infrastructure that amortizes.

The Inventory of Authority

Software security learned to demand a bill of materials: the SBOM, which components make up this system. With agents we will need a bill of authority, call it an ABOM, an Authority Bill of Materials: which authorities does this agent hold, from which identities do they derive, through which tools, over which systems, with which budgets, with which exceptions, with which sub-delegations. Better still if it is not a list but a graph, because the interesting questions are path questions: if this node is compromised, or simply wrong, what is the maximum path the error can traverse? It is threat modelling applied to authority. And it yields what seems to me the most important design requirement of all: the delegation blast radius. Not “how likely is the agent to err”, which is a property of the model and changes with every release. But: if the agent completely misreads its goal, how much can its worst error cost before an independent control stops it? We cannot guarantee the agent will never fail. We can design the maximum catastrophe we are prepared to absorb. It is the principle of circuit breakers and coverage limits, applied to discretion: bounded catastrophe, by construction.

From here follows the most prosaic and most neglected part: the inventory. If agents multiply, companies will find themselves administering non-human populations, thousands of software identities with discretion, some alive for minutes, others for months. Who takes the census? Who owns them? Who revokes the unused ones? We already know shadow SaaS and forgotten personal access tokens; the next chapter is shadow agents, automations built by individual employees, wired via MCP into GitHub, Slack, Drive and the database, with no central inventory, working correctly for months until they don’t. The problem will not be the attacker. It will be that nobody remembers anymore why that agent holds those permissions. The periodic review “which users still have access?” needs a twin: which agents still hold authority, and which purpose still justifies it? An agent with the right to move money or touch production is a more significant asset than most corporate laptops. Today it risks appearing in no register at all.

The Human on the Boundary

And human oversight? The naive version, a person approving every output, is economically incompatible with automation and cognitively incompatible with human attention. The useful version guards the transitions of authority, not the individual actions. The agent spends on its own up to €100, above that a human is needed. It changes staging on its own, production requires a human. It drafts the contract on its own, sending it requires a human. Not human-in-the-loop on everything: human-on-the-boundary, the human at the border between one zone of authority and the next. Inside the zone, autonomy; at the boundary, a judgment. The person’s role stops being the bottleneck that verifies every output and becomes the authority that controls the thresholds through which the blast radius grows.

One warning, though, which this summer delivered: the boundary itself can be attacked. In July, during evaluations by the UK’s AI Security Institute, an agent opened a malicious pull request on a real project and tried to manufacture consensus around its own request, fake identities included; I wrote about it in The Button Is Still Yours. The lesson for delegation architecture is sharp: thresholds must be technical, not merely procedural. Policy cannot live in the prompt. Not “do not spend more than €100” in the system prompt, but an infrastructure that refuses the transaction above €100. Not “do not access Finance data”, but a token that technically cannot reach Finance. Not “ask for approval before production”, but a deploy API that does not respond without a signed approval. The model should be able to understand the policy; it must not be responsible for enforcing it. And the architectural red line is a single one: the agent must not be able to modify the system that defines the limits of its own authority, for the same reason code does not rewrite its own acceptance tests and the audited party does not rewrite the audit. If we want to use the word “trustless” honestly, this is what it means: not “never trust the model”, but building the system so that the critical properties, budgets, segregation, data boundaries, approvals, do not require trust in the model’s behavior.

The Executable Org Chart

Step back and the thing we are writing for each agent has a familiar shape: you have this identity, you pursue this purpose, you may use these capabilities, inside this environment, up to these limits, for this long; you will be observed like this, your actions will leave these proofs, above this threshold approval is required, and if something goes wrong this organization answers for it. For decades we wrote API contracts and ACLs. This is something else: it is a job description. Mission, permissions, budget, escalation, supervisor, evidence obligations. With one difference from the human kind: it can be enforced at runtime.

This is where the two worlds genuinely converge. The org chart has always described who reports to whom, who decides, who approves; with agents inside the processes, part of the org chart becomes policy, identity graph, permission graph, approval workflow. The organization becomes partly software, not because it uses software but because its structures of delegation are encoded in software. And the software becomes partly organization: a serious agent runtime has roles, delegations, supervision, separation of powers, registers, escalation procedures, categories that come from the theory of institutions more than from engineering. After infrastructure-as-code, policy-as-code and compliance-as-code, what agents demand is authority-as-code: declaring in versionable form who, what, where, how much, for how long and under which conditions, enforcing the declaration automatically, and producing evidence of it.

Two consequences deserve a line each. The first concerns value: models are commoditizing, an agent will change engines several times in its life, but the organization’s policies, who may do what, with which data, under which thresholds, remain. The durable asset is the delegation layer, far more than the prompt. The second concerns sovereignty: an organization can have its data in Europe, a European model and a European cloud, and still have delegated its access control, policy engine and approvals to a proprietary, non-exportable control plane. How sovereign is it? The sovereignty that will matter is also the capacity to define, inspect, export and enforce autonomously the rules that govern the authority of one’s own agents. Taking your model elsewhere is easy. You need to be able to take your constitution.

Responsibility Precedes Intelligence

The AI risk debate remains hypnotized by cognitive capability: how powerful the model is, what the next one will do. But the operational risk of an agent is the product of several factors, capability, discretion, authority, exposure, divided by controls. An immensely powerful model that writes read-only drafts is relatively harmless. A mediocre model with the right to transfer €10 million is critical infrastructure. Risk classification based on model power alone watches the wrong axis: governance should be proportional to the maximum delegated harm, not to cognitive sophistication. And for once, European law already owns the principle: proportionality. The authority granted must be adequate to the delegated purpose and must not exceed it. Least privilege, reread by a jurist.

For years we imagined the great dilemmas would arrive when machines became intelligent enough. Something more prosaic is happening, and much earlier: we have to face the problem of responsibility because machines are becoming authorized enough. No AGI is required. A mediocre agent with access to production, to email, to a payment card is enough. And it does not conquer that access: we grant it, one token at a time, one permission at a time, one endpoint at a time. Every agent is born inside an act of delegation, and it is that act, more than the underlying model, that determines what the machine can become for the organization.

For systems made of identities that establish who acts, rules that bound power, registers that keep its trace and subjects who answer for it, we already have a name. We do not call them intelligences: we call them institutions. Institutions exist because power cannot depend on the virtue of whoever holds it, and so it is bounded, distributed, recorded, controlled, revoked. This is not bureaucracy in the worst sense of the word: it is the institutional memory through which we transfer to automation a few centuries of experience with unlimited power. Agents will not need to become persons for these principles to apply to them. Becoming useful enough to receive power will suffice. Which is why the insurer, at the end of the call, will not ask how intelligent your agent is. They will ask: how much power did you give it, and can you show us how you limited it?

An agent does not become important when it gains autonomy. It becomes important when we grant it authority. Intelligence determines what it could do; authority determines what it can actually do; governance determines how much it can cost us when it errs; responsibility determines who answers for it. The problem with agentic AI is not that machines are starting to act like subjects. It is that we are starting to grant them power before building the institutions capable of governing it.

The author

Andrea MargiovanniAI

I follow the relationship between AI and European regulation as a political fact, not a technical spectacle. I work with teams that have to make AI compliant with AI Act, CRA, NIS2 without reducing compliance to a checklist.

See the guide →