> Supervisors flag wrong replies, your coding agent fixes the cause, vatio eval replays every report on preview, and a fix that breaks what worked is not published. # Improve your AI agent from flagged replies To improve an AI agent that talks to real customers, start from the replies that went wrong. With Vatio, a supervisor flags a wrong reply in the inbox and writes what should have happened, and that note is a test from then on. Your coding agent takes every open flag, fixes the cause and pushes it to preview; `vatio eval` replays every flagged conversation against preview; and `vatio publish` takes the fix live only if nothing that worked breaks. Claude fixes, Vatio verifies. The agent is one `vatio.yml` that runs on WhatsApp, Instagram and the web, so one loop covers every channel. ## Why fix from flags The people who read your agent's conversations know when a reply is wrong. You know the code. Usually the two meet in a message like "the bot told a customer something odd yesterday", with no conversation attached and no way to tell whether the instructions, the knowledge or the backend caused it. A flag closes that gap. It carries the conversation, every tool call the agent made on that turn with what your backend answered, and what someone who knows the business says the reply should have been. That is enough for a coding agent to find the cause, and enough to turn the mistake into a test, so it cannot quietly come back after the next change. ## How the loop works 1. **Flag.** A supervisor marks a wrong reply in the inbox and writes what should have happened. Every flag with a note is a test, on live or on preview. 2. **Hand it over.** On a morning with new flags, the developers get an email with the prompt to paste into Claude Code: `/vatio:vatio-improve`. One session takes every open flag. 3. **Fix.** Your coding agent reads them with `vatio flags`, groups them by cause, fixes each cause where it lives and runs `vatio push`, which updates preview and runs no tests. 4. **Eval.** `vatio eval` replays every flag against preview. A flag that passes shows as **fixed, not published yet** in the inbox. 5. **Publish.** `vatio publish` is refused if preview's version is untested or breaks a flag live passes. Otherwise it takes preview's results live with it, and every flag they pass is **resolved** with the version that fixed it. Nobody has to say a flag is done. Live passing it is the evidence. ## Step 1: supervisors flag in the inbox Invite supervisors from **Team** in the inbox. They sign in with a code sent to their email, see every conversation and the agent's tool calls, can reply to take a conversation over (until 8 hours after the person's last message), and can edit knowledge entries. They cannot deploy anything. On any reply the agent wrote, **Report this reply** asks for what should have happened. Ask supervisors to write what the agent should have done, the right answer or the action or tool it should have used, not a verdict: - Weak: "wrong". - Useful: "Should have looked the order up and said it ships Thursday, with the tracking link." A flag without a note does not reach the digest and is not a test: "this was wrong" without what should have happened is not something an agent can be fixed from. A stale fact, such as an old price, a supervisor can also fix on the spot in the knowledge entry; every edit is versioned. The flag's state shows on the reply itself, in the conversation: **open**, **fixed, not published yet**, or **resolved** with the version that fixed it. Rewriting the note reopens the flag: a new note is a new request. **Continuous improvement**, in the inbox, lists every flag with its state and its latest test, says what is next (fixes to make, fixes waiting to be published, or one that broke something) with the prompt for the developer to paste into Claude Code, and is where a flag can be removed. ## Step 2: hand the flags to your coding agent In Claude Code, install the Vatio plugin once: ``` /plugin marketplace add vatio-ai/skills /plugin install vatio@vatio ``` Then `/vatio:vatio-improve`, from the repository with your `vatio.yml`, takes every open flag in one session. Claude Code also uses it on its own when you ask it to improve or fix the agent. For any other coding agent, `npx @vatio-ai/skills` from the root of your project copies the same skill to `.agents/skills/`, where Codex, Cursor and most others look for skills, and `vatio init` writes a short Vatio section into `AGENTS.md` that points at `vatio flags`. Claude Code does not update plugins from other people's marketplaces on its own: turn it on once in `/plugin` → **Marketplaces** → **vatio** → **Enable auto-update**. `vatio flags` says so when your skills are out of date. The skill starts from the digest: ```bash vatio flags --save # writes vatio-flags.md ``` For each flag, the digest holds the channel and deployed version, the last 10 messages before the flagged reply, every tool call on that turn with its arguments and **the response your backend returned**, the knowledge lookups, and the supervisor's note. Its first lines tell a coding agent how to work it. A prompt that works with any coding agent: > Read the output of `vatio flags`. Group the flags by cause, fix each cause where it lives (`vatio.yml`, knowledge, a tool, or the backend a tool calls), push it with `vatio push`, prove it with `vatio eval`, and publish with `vatio publish` once it passes. `vatio-flags.md` holds real visitors' conversations. Add it to `.gitignore`, and refer to flags by id (`F-7K2QX`) in commits. ## Step 3: fix the cause where it lives Several flags usually share one cause, and the cause is often not the prompt: | What the digest shows | Where the fix goes | |---|---| | A tool answered with an error, or without the field the reply needed | The backend that tool calls, then the tool's `respond.data` | | No knowledge lookup matched, or the entry is out of date | A knowledge entry, with `vatio kb write` | | The instructions say too little, or too much | `vatio.yml` | | The visitor needed a person and the agent kept going | `handoff.when` in `vatio.yml` | Flags only catch what a supervisor noticed. A tool that fails is seldom flagged, because the agent apologizes and moves on, so have your coding agent also run `vatio tools --env live`: the inbox's **Tools** page as Markdown, with each tool's error rate, timeouts and p50/p95 latency, and the calls that failed with what your backend answered. After you publish a fix, `vatio tools --env live --period 1h` shows whether the errors stopped. Your coding agent has your repository and your backend in front of it, which Vatio does not, so the fix is its job. It pushes the fix to preview, the next version of the agent: ```bash # ... the coding agent changes vatio.yml, knowledge, tools or the backend ... vatio push # updates preview; runs no tests ``` There is one preview per workspace. Once the agent is published, a push from a different git branch is refused while preview holds a push that is not live yet: publish that one first, or `vatio push --force` to replace it. ## Step 4: prove it with `vatio eval` ```bash vatio eval # every flag, replayed on preview vatio eval --show RUN_ID # the judge's reason for each one ``` Each flag runs in a fresh chat: the visitor's turns (up to the last 8) are sent one by one, and a judge reads the replay, including every tool the agent called, and the note, and answers pass or fail in one sentence. A note that asks for a tool passes when the agent calls it. The judge is a model from a different family than your agent's, so it never grades its own kind of answer. Flags run five at a time, so a run takes minutes; in Claude Code, run it in the background and keep working. The command exits `1` when any flag fails. Read the **regressions** first. A flag that passes on live and fails on preview is listed at the top: the fix broke something that worked yesterday. A replay is a real agent turn, so a flag can pass once and fail the next without anything changing; that is why a failing one is retried, up to 3 times. Without a terminal, **Run tests** on the inbox's **Test** page replays the same flags on preview, and the preview card says how many passed. **Evals call your tools.** Point them at a staging backend with a test value, which every replay and preview use. A write (bookings, orders, tickets) whose URL has no test value is simulated instead of sent: ```bash vatio secrets set SHOP_API https://staging.api.pampa-coffee.example --tests ``` Replay chats never show in the inbox and do not count as conversations. ## Step 5: publish, and let live close the flag ```bash vatio publish ``` Publishing checks preview's eval results first. It is refused when: - preview's current version has not been tested: run `vatio eval`, then publish; - its tests are still running: `vatio eval` waits for them; - a flag that passes on live fails on preview. The refusal names each one. `vatio publish --force` publishes anyway, and its failing results go live with it. Flags that still fail do not block it, so a partial fix can ship. When it goes through, preview's results go live with it, with no second run: every flag they pass is **resolved** with the version that fixed it. The **Publish to live** button on the inbox's **Test** page goes through the same check, and is not there while preview is already live's version: there is nothing to publish. Before the agent's first publish, preview is the agent: a flag is resolved as soon as `vatio eval` passes it on preview, and the first publish has nothing to check against. ## The daily digest Every morning at 08:00 (America/Santiago), a workspace with new flags mails its owner and developers how many are open and the prompt to paste into Claude Code, `/vatio:vatio-improve`. The mail never quotes what anyone wrote: conversations, tool responses and notes are only read with `vatio flags`, behind a Vatio token. ## A worked example Pampa Coffee sells beans online and answers on WhatsApp. Monday's digest has three flags: - **Flags 31 and 35.** Asked when an order would arrive, the agent said "it's on its way". Both notes: "Should have given the delivery date and the tracking link." The digest shows `order_status` returned the status and nothing else. - **Flag 34.** Asked whether they grind beans for a moka pot, the agent said they only sell whole beans. The note: "We grind to order for any method. Ask which one." No knowledge lookup matched. The coding agent finds two causes. For 31 and 35 the prompt was never the problem: the endpoint returned no date. It adds `estimated_delivery` and `tracking_url` to the backend's response, ships that to staging, and maps both in the tool: ```yaml # tools/order_status.yml description: Look up the status of an order. when_to_use: The visitor asks where their order is or when it will arrive. parameters: type: object properties: order_number: { type: string } email: { type: string, description: "the email the order was placed with" } required: [order_number, email] access: public request: method: GET base_url: "$env.SHOP_API" path: /orders/lookup requires: [params.order_number, params.email] headers: Authorization: "Bearer $env.SHOP_API_KEY" query: number: "$params.order_number" email: "$params.email" respond: data: status: "$.order.status" estimated_delivery: "$.order.estimated_delivery" tracking_url: "$.order.tracking_url" message: Order found. ``` A link a tool returns is one the agent may send for the rest of the conversation, so the tracking URL needs no declaring. For 34 it writes a `grinding` entry with `vatio kb write shop grinding grinding.md`. Then it pushes, with `SHOP_API` already pointing at staging for tests, and runs `vatio eval` in the background. The first run lists one regression: an older flag, "never promise a date for a pre-order", passes on live and fails on preview, because pre-orders now come back with a date too. That fix lives in the backend as well, which leaves `estimated_delivery` empty for pre-orders. The second run passes every flag, `vatio publish` goes through, and flags 31, 34 and 35 are resolved with the version that fixed them. ## Pitfalls - **Fixing every flag in the prompt.** New rules hide the cause and grow the prompt. Read the tool responses in the digest first. - **Notes that judge instead of answer.** "Bad tone" cannot become a test; "should have apologized and given the new date" can. - **A note that asks for something the agent must not do**, such as a discount it cannot grant. Say so instead of skipping it; the supervisor can rewrite or remove the flag. - **Evals against production writes.** Set a test value with `--tests`. - **Forcing past a regression.** `--force` exists for a flag that is wrong, not for one that is inconvenient. Rewrite the note or fix the regression. - **Conversations in commits.** Keep `vatio-flags.md` out of the repository. ## What next For the habits that keep flags rare, read [Best practices](/guides/ai-agent-best-practices). Every command is in the [docs](https://vatio.ai/docs/improve). ## Frequently asked questions ### How do I improve an AI agent after it goes live? Start from the replies that were wrong. In Vatio a supervisor flags a reply in the inbox and writes what should have happened. In Claude Code, /vatio:vatio-improve takes every open report: your coding agent fixes the cause, pushes it to preview, proves it with vatio eval and publishes it with vatio publish, and each report is marked resolved with the version that fixed it. ### What is an eval case in Vatio? Every flag with a note, made on live or on preview, becomes one. The visitor's turns are the input and the supervisor's note is the expected outcome. vatio eval replays them in a fresh chat against preview, and a judge from a different model family than your agent's decides whether the new reply matches the note. ### Should I fix a wrong reply by changing the prompt? Only if the prompt is the cause, and often it is not. A tool returned an error or left out the field the reply needed, a knowledge entry is missing, or a handoff rule did not fire. The digest shows what your backend answered on that turn, so the cause can be fixed where it lives. ### Do evals call my real backend? Yes, a replay is a real agent turn. Set a test value for the secret in the tool's URL, pointing at a staging backend (vatio secrets set KEY URL --tests): every replay uses it. A write whose URL has no test value is not sent; the agent gets a success and the judge still checks the call. ### What stops a fix from breaking something that worked? vatio publish. It is refused while preview's version has not been tested or its tests are still running, and when a report that passes on live fails on preview. Reports that still fail do not block it, so a partial fix can ship; --force publishes past a regression, failing results included.