The demo went well
The agent read the ticket, looked up the customer, drafted a reply and closed the case. Everyone in the room watched it happen in under a minute. Someone asked when it could go live.
That question usually gets answered with a date. It should get answered with a list, because a demo tests one thing: whether the agent can do the task when the inputs are clean and someone is watching. Production tests everything else.
When I've helped teams take agents from demo to production, the same five questions decide whether the launch holds up. None of them are about the model.
1. What is it allowed to touch?
An agent that can read and write to real systems needs permissions, and the easiest permissions to grant are the broad ones. The demo ran with an admin token because it was quicker to set up.
In production, the agent should have the narrowest access that lets it do the job. If it drafts replies, it doesn't need permission to delete accounts. If it reads order history, it doesn't need write access to billing.
Write the permission list down as a table: system, action, allowed or not. If the team can't fill that table in, the agent isn't ready to go live.
The same goes for what the agent reads. Agents that read emails, tickets or web pages can be steered by instructions hidden in that content. Treat anything the agent reads from outside as untrusted input, and make sure the permission list limits the damage if it gets misled.
2. What happens when it's wrong?
It will be wrong. The useful question is how wrong, how often and what it costs.
Sort the agent's actions into two groups:
- Reversible. A draft reply, a suggested tag, a note on a ticket. If it's wrong, someone fixes it.
- Hard to reverse. A refund issued, an email sent to a customer, a record deleted, a payment approved.
3. Who reviews it, and when?
"A human in the loop" means nothing until you name the human and their workload.
If the agent produces 300 drafts a day and one team lead is supposed to review them, the review will turn into clicking approve without reading within a week. That's worse than no review, because everyone believes there's a check that isn't happening.
Decide up front:
- Which outputs get reviewed: all of them, a random sample, or only the ones the agent marks as low confidence.
- Who does the review, and how many minutes per day that takes.
- What the reviewer does with a bad output: fix it, flag it, and where the flag goes.
4. What does it cost per task?
Demo costs are invisible. One run, a few cents, nobody notices.
At volume, an agent that calls a model several times per task, retries on errors and pulls long documents into every request can cost more than the person it was meant to help. I've seen teams find this out from the first monthly invoice.
Before launch, run 100 realistic tasks and measure the cost of each one. Set a spending limit per task and per day, and decide what the agent does when it hits the limit. Stopping and handing the task to a person is a fine answer.
5. How will you know it's working?
"It seems fine" is how most agent launches are judged in the first month. Then someone finds a pattern of bad outputs that had been going on for weeks.
Pick two or three numbers before launch and track them from day one:
- Task completion rate. How often the agent finishes without a person taking over.
- Correction rate. How often a reviewer changes or rejects what the agent did.
- Time saved. Measured against how long the same task took before, on the same kind of work.
A launch you can defend
None of these five questions are hard to answer. They take a week or two of work that doesn't look impressive in a demo: a permissions table, a review rota, a cost test, a dashboard with three numbers.
That week is the difference between an agent that's still running in six months and one that gets switched off after its first visible mistake. If you can't answer all five yet, launch the reversible half and keep a person on the rest. Expand from there as the numbers come in.