#Improve your AI agent from flagged replies

To improve an AI agent that talks to real customers, start from the replies that went wrong. With Vatio, a supervisor flags a wrong reply in the inbox and writes what should have happened, and that note is a test from then on. Your coding agent takes every open flag, fixes the cause and pushes it to preview; vatio eval replays every flagged conversation against preview; and vatio publish takes the fix live only if nothing that worked breaks. Claude fixes, Vatio verifies. The agent is one vatio.yml that runs on WhatsApp, Instagram and the web, so one loop covers every channel.

#Why fix from flags

The people who read your agent’s conversations know when a reply is wrong. You know the code. Usually the two meet in a message like “the bot told a customer something odd yesterday”, with no conversation attached and no way to tell whether the instructions, the knowledge or the backend caused it.

A flag closes that gap. It carries the conversation, every tool call the agent made on that turn with what your backend answered, and what someone who knows the business says the reply should have been. That is enough for a coding agent to find the cause, and enough to turn the mistake into a test, so it cannot quietly come back after the next change.

#How the loop works

1. Flag. A supervisor marks a wrong reply in the inbox and writes what should have happened. Every flag with a note is a test, on live or on preview.
2. Hand it over. On a morning with new flags, the developers get an email with the prompt to paste into Claude Code: /vatio:vatio-improve. One session takes every open flag.
3. Fix. Your coding agent reads them with vatio flags, groups them by cause, fixes each cause where it lives and runs vatio push, which updates preview and runs no tests.
4. Eval. vatio eval replays every flag against preview. A flag that passes shows as fixed, not published yet in the inbox.
5. Publish. vatio publish is refused if preview’s version is untested or breaks a flag live passes. Otherwise it takes preview’s results live with it, and every flag they pass is resolved with the version that fixed it.

Nobody has to say a flag is done. Live passing it is the evidence.

#Step 1: supervisors flag in the inbox

Invite supervisors from Team in the inbox. They sign in with a code sent to their email, see every conversation and the agent’s tool calls, can reply to take a conversation over (until 8 hours after the person’s last message), and can edit knowledge entries. They cannot deploy anything.

On any reply the agent wrote, Report this reply asks for what should have happened. Ask supervisors to write what the agent should have done, the right answer or the action or tool it should have used, not a verdict:

• Weak: “wrong”.
• Useful: “Should have looked the order up and said it ships Thursday, with the tracking link.”

A flag without a note does not reach the digest and is not a test: “this was wrong” without what should have happened is not something an agent can be fixed from. A stale fact, such as an old price, a supervisor can also fix on the spot in the knowledge entry; every edit is versioned.

The flag’s state shows on the reply itself, in the conversation: open, fixed, not published yet, or resolved with the version that fixed it. Rewriting the note reopens the flag: a new note is a new request. Continuous improvement, in the inbox, lists every flag with its state and its latest test, says what is next (fixes to make, fixes waiting to be published, or one that broke something) with the prompt for the developer to paste into Claude Code, and is where a flag can be removed.

#Step 2: hand the flags to your coding agent

In Claude Code, install the Vatio plugin once:

/plugin marketplace add vatio-ai/skills
/plugin install vatio@vatio

Then /vatio:vatio-improve, from the repository with your vatio.yml, takes every open flag in one session. Claude Code also uses it on its own when you ask it to improve or fix the agent. For any other coding agent, npx @vatio-ai/skills from the root of your project copies the same skill to .agents/skills/, where Codex, Cursor and most others look for skills, and vatio init writes a short Vatio section into AGENTS.md that points at vatio flags. Claude Code does not update plugins from other people’s marketplaces on its own: turn it on once in /plugin → Marketplaces → vatio → Enable auto-update. vatio flags says so when your skills are out of date.

The skill starts from the digest:

vatio flags --save        # writes vatio-flags.md

For each flag, the digest holds the channel and deployed version, the last 10 messages before the flagged reply, every tool call on that turn with its arguments and the response your backend returned, the knowledge lookups, and the supervisor’s note. Its first lines tell a coding agent how to work it. A prompt that works with any coding agent:

Read the output of vatio flags. Group the flags by cause, fix each cause where it lives (vatio.yml, knowledge, a tool, or the backend a tool calls), push it with vatio push, prove it with vatio eval, and publish with vatio publish once it passes.

vatio-flags.md holds real visitors’ conversations. Add it to .gitignore, and refer to flags by id (F-7K2QX) in commits.

#Step 3: fix the cause where it lives

Several flags usually share one cause, and the cause is often not the prompt:

Flags only catch what a supervisor noticed. A tool that fails is seldom flagged, because the agent apologizes and moves on, so have your coding agent also run vatio tools --env live: the inbox’s Tools page as Markdown, with each tool’s error rate, timeouts and p50/p95 latency, and the calls that failed with what your backend answered. After you publish a fix, vatio tools --env live --period 1h shows whether the errors stopped.

Your coding agent has your repository and your backend in front of it, which Vatio does not, so the fix is its job. It pushes the fix to preview, the next version of the agent:

# ... the coding agent changes vatio.yml, knowledge, tools or the backend ...
vatio push                # updates preview; runs no tests

There is one preview per workspace. Once the agent is published, a push from a different git branch is refused while preview holds a push that is not live yet: publish that one first, or vatio push --force to replace it.

#Step 4: prove it with vatio eval

vatio eval                # every flag, replayed on preview
vatio eval --show RUN_ID  # the judge's reason for each one

Each flag runs in a fresh chat: the visitor’s turns (up to the last 8) are sent one by one, and a judge reads the replay, including every tool the agent called, and the note, and answers pass or fail in one sentence. A note that asks for a tool passes when the agent calls it. The judge is a model from a different family than your agent’s, so it never grades its own kind of answer. Flags run five at a time, so a run takes minutes; in Claude Code, run it in the background and keep working. The command exits 1 when any flag fails.

Read the regressions first. A flag that passes on live and fails on preview is listed at the top: the fix broke something that worked yesterday. A replay is a real agent turn, so a flag can pass once and fail the next without anything changing; that is why a failing one is retried, up to 3 times.

Without a terminal, Run tests on the inbox’s Test page replays the same flags on preview, and the preview card says how many passed.

Evals call your tools. Point them at a staging backend with a test value, which every replay and preview use. A write (bookings, orders, tickets) whose URL has no test value is simulated instead of sent:

vatio secrets set SHOP_API https://staging.api.pampa-coffee.example --tests

Replay chats never show in the inbox and do not count as conversations.

#Step 5: publish, and let live close the flag

vatio publish

Publishing checks preview’s eval results first. It is refused when:

• preview’s current version has not been tested: run vatio eval, then publish;
• its tests are still running: vatio eval waits for them;
• a flag that passes on live fails on preview. The refusal names each one. vatio publish --force publishes anyway, and its failing results go live with it.

Flags that still fail do not block it, so a partial fix can ship. When it goes through, preview’s results go live with it, with no second run: every flag they pass is resolved with the version that fixed it. The Publish to live button on the inbox’s Test page goes through the same check, and is not there while preview is already live’s version: there is nothing to publish.

Before the agent’s first publish, preview is the agent: a flag is resolved as soon as vatio eval passes it on preview, and the first publish has nothing to check against.

#The daily digest

Every morning at 08:00 (America/Santiago), a workspace with new flags mails its owner and developers how many are open and the prompt to paste into Claude Code, /vatio:vatio-improve. The mail never quotes what anyone wrote: conversations, tool responses and notes are only read with vatio flags, behind a Vatio token.

#A worked example

Pampa Coffee sells beans online and answers on WhatsApp. Monday’s digest has three flags:

• Flags 31 and 35. Asked when an order would arrive, the agent said “it’s on its way”. Both notes: “Should have given the delivery date and the tracking link.” The digest shows order_status returned the status and nothing else.
• Flag 34. Asked whether they grind beans for a moka pot, the agent said they only sell whole beans. The note: “We grind to order for any method. Ask which one.” No knowledge lookup matched.

The coding agent finds two causes. For 31 and 35 the prompt was never the problem: the endpoint returned no date. It adds estimated_delivery and tracking_url to the backend’s response, ships that to staging, and maps both in the tool:

# tools/order_status.yml
description: Look up the status of an order.
when_to_use: The visitor asks where their order is or when it will arrive.
parameters:
  type: object
  properties:
    order_number: { type: string }
    email: { type: string, description: "the email the order was placed with" }
  required: [order_number, email]
access: public
request:
  method: GET
  base_url: "$env.SHOP_API"
  path: /orders/lookup
  requires: [params.order_number, params.email]
  headers:
    Authorization: "Bearer $env.SHOP_API_KEY"
  query:
    number: "$params.order_number"
    email: "$params.email"
respond:
  data:
    status: "$.order.status"
    estimated_delivery: "$.order.estimated_delivery"
    tracking_url: "$.order.tracking_url"
  message: Order found.

A link a tool returns is one the agent may send for the rest of the conversation, so the tracking URL needs no declaring. For 34 it writes a grinding entry with vatio kb write shop grinding grinding.md. Then it pushes, with SHOP_API already pointing at staging for tests, and runs vatio eval in the background.

The first run lists one regression: an older flag, “never promise a date for a pre-order”, passes on live and fails on preview, because pre-orders now come back with a date too. That fix lives in the backend as well, which leaves estimated_delivery empty for pre-orders. The second run passes every flag, vatio publish goes through, and flags 31, 34 and 35 are resolved with the version that fixed them.

#Pitfalls

• Fixing every flag in the prompt. New rules hide the cause and grow the prompt. Read the tool responses in the digest first.
• Notes that judge instead of answer. “Bad tone” cannot become a test; “should have apologized and given the new date” can.
• A note that asks for something the agent must not do, such as a discount it cannot grant. Say so instead of skipping it; the supervisor can rewrite or remove the flag.
• Evals against production writes. Set a test value with --tests.
• Forcing past a regression. --force exists for a flag that is wrong, not for one that is inconvenient. Rewrite the note or fix the regression.
• Conversations in commits. Keep vatio-flags.md out of the repository.

#What next

For the habits that keep flags rare, read Best practices. Every command is in the docs.

Updated: October 11, 2026