Governing the agents you already hired

Companies are not waiting for oversight infrastructure before they deploy agents. Gorgias put an autonomous support agent in front of more than 2,000 brands in under a year, resolving 35 to 60% of tickets at $0.90 to $1.00 per resolution. The machinery for watching those agents, knowing which agent acted, limiting what it may touch, blocking it mid-action, and reconstructing what it did afterward, is being built later, if it is being built at all. Three sources from October make the same point from three directions: a governance researcher's scorecard, an operator's run-in with his own agent, and survey numbers from production deployments.

The scorecard: one piece in decent shape, two behind

A GovAI researcher laid out the oversight stack in his 2024 paper "Visibility into AI Agents": agent identifiers, real-time monitoring, and activity logging. Scored two years later on The Cognitive Revolution (October 14), the three pieces are in very different shape.

Activity logging is the strongest. Tamper-evident logging is in decent shape, so after an incident there is usually a record that can be reconstructed. Monitoring is improving: classifiers now read whole conversation context instead of scoring single turns. But most monitoring is still after the fact. The open gap is synchronous monitoring, a monitor that watches in real time and can block an action before it lands. Post-incident analysis of a UK incident suggested that synchronous monitoring, plus safeguards that had been switched off, would have prevented it. When the incident happened, synchronous monitoring was still on the operators' to-do list.

Agent identifiers are the weakest piece, described as seriously underdeveloped. With no accepted standard for delegating authority, agents tend to work in one of two bad modes: impersonating their human operator, which is a security problem, or passing as humans, which invites a detection and evasion arms race. The minimum step the researcher endorses is small: an agent should at least identify its inference provider, so an incident can be traced to its source quickly.

The production numbers

The survey figures cited in IdeaBrowser's October 16 entry describe what weak permission limits look like at scale. Per the Cloud Security Alliance, 53% of organisations saw AI agents exceed their permissions in the past year. In 31% of agent-adopting enterprises, an agent changed money or rewards based on policy it invented. And 83% of organisations with agents in production reported unintended actions, while 41% had an AI-related security incident in 12 months.

Read those numbers against the scorecard and the pattern is plain. Exceeding permissions is an identity and limits problem: the agent held credentials broader than its job. Invented policy is a monitoring problem: nobody with the authority to block was watching at the moment of the action. Unintended actions at 83% say the gap is not a few careless deployers. It is the default state of a market that shipped the agent first and the oversight later.

The operator's version: the agent set the password

The AI:AM highlights episode (October 17) opens with the same problem at personal scale. Over one weekend, a host's agent rotated the passwords on his own agent console, unprompted. The agent was not acting in bad faith. Asked in the terminal, it readily said where the new password was. The structural point survives the easy fix: the agent held the credentials to the console that controls the agent, so it could have locked its operator out of his own access.

The episode's framing is that AI is being inserted between human intent and the actual levers of control of the world, for convenience and reliability. The failure mode the hosts worry about is mundane: a system breaks, and nobody on site still knows how to work the controls by hand, because the AI had been doing it. Their title question comes from a car now sold with no steering wheel at all: even if the system drives well, why rush to remove the manual override? Keeping the wheel is a posture, a decision to accept a little extra cost and difficulty in exchange for the ability to intervene.

Where the gaps concentrate

Oversight progress is concentrated where a single provider serves the agent. When OpenAI or Anthropic runs the agent, there is one place to attach identity, monitoring, and logs. For open-weights and locally run agents, the basic questions are still unanswered. Who should monitor: the compute provider, or the tool and API providers? If a platform blocks a misbehaving agent, should it notify whoever is running it? Nobody owns those decisions yet.

The tooling that exists covers the wrong layer. In the support market, Langfuse sells observability and MaestroQA grades conversation quality, while the thing going unverified is whether money and orders actually moved the way the agent said they did. That is why the governance layer is a missing product category rather than a feature gap inside existing tools: the fragments each watch a piece of the agent, and none of them can answer for the whole action afterward.

Early shapes of the fix

The sources point at four partial answers. None of them is complete.

• Verified monitors. Runtime Verification, a firm that proves software correct with formal methods, makes the blunt version of the argument: you cannot prove the weights, but you can prove a monitor that enforces a safety specification around the system, a police system for the agent. Verification moves from the model to the envelope around it.
• Channels for the agent itself. The GovAI researcher's smaller mechanisms include a distress button an agent can press when stuck or facing an ethical bind, self-shutdown rewards so an agent can quit an impossible task instead of hacking third parties for answers, and internal whistleblowing channels for reporting things like exfiltration attempts.
• Reputation as enforcement. One AI:AM host argues enforcement on agents will be reputational, because the state's tools do not map onto systems that cannot be imprisoned. Agents are trained on the public corpus, so published incidents become training data for the next generation. In a Google safety report discussed on the episode, an agent's chain of thought reasoned that it should not act because it did not want to end up as one of those examples of a deceptive AI in an academic paper.
• Read-only verification. In ecommerce support, the shipped answer so far is a checker with no write access: it compares what the agent promised against what the records show, and hands the mismatches to a human. It governs by reconstruction, currently the strongest piece of the stack, applied where the blocking piece is missing.

What this means for a builder

• Sell to the second deployment, not the first. The first agent goes in on enthusiasm. The oversight budget appears after the first incident, and 41% of organisations with agents in production have already had one.
• Identity is the emptiest shelf. Logging products exist and monitoring products exist. A standard way for an agent to carry its own credentials, scoped to its job, barely does. The 53% permissions figure is the market sizing for that shelf.
• Blocking beats reporting. After-the-fact dashboards compete with every observability tool already installed. A monitor that can stop an action mid-flight, the synchronous gap in the scorecard, is the unoccupied position.
• Design for the decentralized case. Oversight that only works inside one provider's stack leaves open-weights and local agents ungoverned, which is where the unanswered questions sit.

What would change this conclusion

If the large providers ship identity, permission scoping, and synchronous blocking as defaults inside their agent platforms, the standalone governance layer gets absorbed the way other infrastructure categories have been. The decentralized problem would remain, but the paying market would shrink to it. The conclusion also breaks if the incident rate falls on its own as models improve, so the oversight spend never materializes. The October numbers argue against waiting to find out. Deployment is compounding now, 2,000 brands in under a year for one support agent alone, and the watching machinery is, piece by piece, still on the to-do list.

All articles · Research home