In every meeting about putting an agent to work there is a moment when someone asks: “and what if it gets it wrong?” It is the right question. And the answer is not “it won’t,” because it will. The answer is what happens when it does.
We have already written about what an agent is and what it costs. This article is the one that goes between those two and the decision to buy: how you keep one under control, how you know when to trust it with more, and what you never trust it with.
The rule fits in one line
The agent prepares, the person signs.
Everything that leaves your company toward a customer, a supplier or an authority goes through someone who takes responsibility for it. The agent reads the email, searches the history, builds the quote and leaves it ready. A person looks at it and sends it. If it is wrong, they fix it before it goes out. If it is right, they lose thirty seconds.
That is not distrust of the technology. It is the same rule you already apply to a new hire: for the first few weeks someone checks their work before it reaches the customer, and you loosen the leash as they prove it is no longer needed.
The difference with the new hire is that the agent can be measured precisely.
What you measure: how often you correct it
For the first month, the agent works in draft mode. It prepares everything and sends nothing. And someone keeps count of one single thing: out of every ten drafts, how many did the person have to correct before sending.
That count is the only honest signal of whether the agent is ready. The vendor’s demo is not, because it always goes well. The opinion of whoever configured it is not, because they have an interest in it going well. What counts is how many times someone on your team, with your company’s real work, had to step in.
Three things worth writing down before you start counting:
What counts as a correction. Changing a part number is a correction. Changing “regards” to “kind regards” is not. If this is not agreed beforehand, everyone counts differently.
Who keeps the count. One person, by name. Almost always the same person who does the task by hand today, because they are the only one who knows when something is off.
What number is “good enough.” It depends on the cost of the error. For an internal draft, one correction in ten may be fine. For something a customer reads, you probably want to be at one in fifty before thinking about loosening the leash.
With that count week by week you get a curve. If it goes down, the agent is learning from the corrections or the process is being tuned, and at some point it crosses the number you agreed on. If it does not go down, the problem is not the model: it is the data you feed it or the process you asked it to follow, and no new version of the agent fixes that.
When you loosen the leash, and for what
Loosening the leash is not a switch you flip for everything. You loosen it by type of task, and there are tasks for which you never loosen it.
First, what can be undone. A reminder to a supplier who has not confirmed an order can go out without review early on: if it goes out unnecessarily, the supplier replies “already confirmed” and nothing happened. Sorting emails, chasing a missing certificate, preparing the shift handover note for the supervisor to correct: all of that lives fine without a signature.
Later, what gets seen. An internal report, a summary of the schedule that changed, an alert that a line will not make it. If it is wrong, someone notices before it costs money.
Never, what commits. A price to a customer, a delivery date, a movement in foreign-trade inventory, anything that goes to an authority. A person signs that, always, even if the agent has gone a year without a mistake. Not because the agent is bad, but because a price is a commercial commitment and a date is a promise, and promises are made by someone who answers for them.
This is not a rule we made up. It is the one regulators are writing. The European AI law requires high-risk systems to be overseeable by people while they run, with the ability to disregard their output or stop them. And the NIST risk management framework in the United States starts from the premise that a system’s risk is measured by what it can do, not by what it claims to know. If your customer is on the other side of the bridge, that is the language their compliance department will use with you sooner or later.
The trap of it working well
There is a risk that appears precisely when the agent is already good, and it is the hardest one to see.
When something gets it right ninety-eight times in a row, the person reviewing stops reviewing. They sign without reading. It is natural, and it has a name: automation bias. The European law names it explicitly among the things that whoever oversees a system must know can happen to them.
What works against it is not asking people to pay more attention. It is designing the review so it cannot be done without looking:
- The agent flags what it is unsure about. A part number it did not find in the history, a quantity that does not match the previous order, a new customer. The person reviews that first and in detail, and the rest more lightly.
- What changed relative to the previous draft is shown highlighted, not something you have to hunt for.
- Every so often, someone who is not the usual reviewer takes ten signed drafts and reviews them again. If they find errors that got through, the signature has become a formality and it is time to tighten up again.
The day it gets it wrong
It will happen. The question is whether you find out or your customer does.
In 2024, a tribunal in British Columbia ordered Air Canada to honour a discount its virtual assistant had promised a passenger and that did not exist. The airline argued the assistant was a separate entity, responsible for its own words. The tribunal did not accept it: the assistant was on the airline’s website and the airline answered for what it said. It was a chatbot that answered questions, not even an agent that did things. And still, what it said was a commitment.
Three things to have in place before switching anything on, with that day in mind:
A trail. Everything the agent prepared, what data it used and who signed it. When something goes wrong, the first thing you will want to know is whether the agent got it wrong or someone signed without reading. Without a trail there is no way to tell, and the discussion turns into opinions.
A way back. What can be undone and how. A sent email cannot be undone, but a created record can, and a sent quote can be corrected with a phone call if you find out the same day. The design should favour actions that can be reversed and put a signature on the ones that cannot.
An owner of exceptions. The agent will get stuck, and it will leave things unresolved. Someone has to receive them, the same day, and it cannot be “the team.” It is a person, and it is hours of their week that need to be counted.
What you give it and what you do not
An agent does things in your systems. That means it has permissions, and permissions are where you decide how much damage a mistake can do.
The rule is to give it only what its task needs, and nothing more. The agent that prepares quotes reads the history and writes drafts; it does not need to be able to delete quotes or change list prices. The security community that maintains the most widely used risk list for these applications gave the opposite a name: excessive agency, which is when an agent can do more than its job requires, and therefore when an error or a trick gives it access to more than it should.
And two caps we already mentioned when talking about costs, because they are also about control: a cap on attempts per task, so a failing step does not retry forever, and a cap on spend per day, so nothing can consume your month in an afternoon.
Six questions for the meeting with the vendor
If someone is offering you an agent, these six questions separate those who have put one to work from those who have given a demo.
- What leaves my company without a person approving it? If the answer is “everything, that is the point,” keep asking. If it is “nothing that commits a price, a date or an official record,” you are on the right track.
- How do I know how often it gets it wrong? There should be a count, somewhere you can see it yourself, not a vendor report.
- What permissions does it need in my systems, one by one? And if they ask for admin access “to keep it simple,” no.
- What is recorded about each thing it does? And who can see it.
- What happens when it cannot resolve something? Who it goes to, how, and within how long.
- How do I switch it off? It should be a one-minute action, not a support ticket.
Where to start
Pick a high-volume, low-risk task. Quote intake almost always wins, and we have already covered what it costs. Put it in draft mode, name who reviews and who receives the exceptions, and keep the count for four weeks.
At the end of four weeks you have a curve and a decision that does not depend on anyone trying to sell you something. And if the curve does not go down, you also know that before committing the budget.
Before that, two things come first: what data you can send to a model and what you cannot, because the agent will read your information and some things cannot leave your network. And if you are still not sure this is the right fight, Chira asks about your operation in a few minutes and tells you, in writing, whether an agent comes first or whether there is something to fix before.
References
- Moffatt v. Air Canada, 2024 BCCRT 149 — the decision of the Civil Resolution Tribunal of British Columbia, dated February 14, 2024, that made the airline answer for what its assistant said.
- Regulation (EU) 2024/1689, Article 14: human oversight — the official text of the European AI law. Article 14 requires high-risk systems to be overseeable by people, that those people can disregard or stop the system, and that they are aware of automation bias.
- AI Risk Management Framework 1.0, NIST — the voluntary framework of the United States government for managing the risk of AI systems, organized around govern, map, measure and manage.
- LLM06:2025 Excessive Agency, OWASP — the risk of giving an agent more permissions or more autonomy than its task needs, and how to limit it.
- Building effective agents, Anthropic — the design guide describing the pattern of an agent pausing at checkpoints to wait for a person.