#Your daily improvement routine

If you use Claude Code, the routine is three steps:

Report: in the inbox, mark a wrong reply and say what should have happened.
Improve: open your app's project and run /vatio:vatio-improve.
Review and publish: Claude prepares fixes; Vatio tests them against preview. Check the results and approve the update when you are ready.

New reports also arrive in your daily email with the same command. The rest of this page explains the checks and commands the skill runs for you.

The people reading your agent's conversations know when a reply is wrong. In the inbox, a supervisor flags that reply and writes what should have happened: a different answer, an action, a tool the agent should have used or does not have. From then on, that flag is a test. Your coding agent fixes it; Vatio checks the fix:

A digest. vatio flags prints every open flag as one markdown document written for a coding agent: the conversation up to the reply, every tool the agent called on that turn with what your backend answered, and the supervisor's note.
A test per flag. Every flag with a note is an eval case, wherever it was made, live or preview. The visitor's turns become the input and the note becomes the expected outcome. vatio eval replays every case against preview and a judge compares each new turn, tool calls included, with the note.
Resolution on evidence. A flag that passes on preview is fixed in preview; once that version is published it is resolved, with the version that fixed it. Nobody has to say it is done. Supervisors see each flag's state on the reply they flagged, and every flag with its latest tests in the inbox's Continuous improvement page.

Your coding agent has your repository and your backend in front of it, which Vatio does not. That is why the fix is its job: the cause is often a tool response missing a field, not the prompt.

Flags only catch what a supervisor noticed. A tool that fails seldom gets flagged, because the agent apologizes and moves on, so vatio tools --env live gives your coding agent the inbox's Tools page: each tool's error rate, timeouts and latency, and the calls that failed with what your backend answered. See Tool metrics.

#The loop

In Claude Code, from the repository with the vatio.yml:

/vatio:vatio-improve

One session takes every open flag. This is what it runs, and what you run with any other coding agent:

vatio flags --save     # vatio-flags.md: hand it to your coding agent
# ... your agent fixes vatio.yml, knowledge, tools or the backend they call ...
vatio push             # updates preview; runs no tests
vatio eval             # replays every flag on preview; exits 1 if one fails
vatio publish          # refused while preview's version is untested or breaks a flag live passes

vatio eval takes minutes: every case is a real agent turn, five run at a time, so 27 cases take three or four minutes. In Claude Code, run it in the background and keep working. Push and eval again until the flags you meant to fix pass, then publish: preview's results go live with it, so live is not replayed again and the flags they pass are resolved at once.

A prompt that works with any coding agent:

Read the output of vatio flags. Group the flags by cause, fix each cause where it lives — vatio.yml, knowledge, a tool, or the backend a tool calls — push it, prove it with vatio eval, and tell me which flags now pass.

The digest itself carries the same instructions at the top, so an agent handed only the file knows what to do with it.

#What publish checks

vatio publish and the Publish to live button on the inbox's Test page go through the same gate. It looks at preview's current version:

	`vatio publish`
It is already live's version	Nothing to publish: only preview's knowledge changes go live, and `vatio publish` prints a notice saying so. The Test page shows no Publish to live button then
The agent was never published	Publishes. If `vatio eval` already finished on that version, its results go live with it
No flags yet	Publishes
Not tested	Refused, `evals_missing`: run `vatio eval`, then publish
Its run is still going	Refused, `evals_running`: `vatio eval` waits for it
A flag that passes on live fails on it	Refused, `evals_regressed`, naming each flag. `vatio publish --force` publishes anyway, and its failing results go live with it
Otherwise	Publishes, and the flags its run passed are resolved with its short sha

Flags that still fail do not block a publish: a fix for three of five can go live while the other two wait. A version published with no results of its own, the first publish before any vatio eval for instance, has live replayed right after it.

#Let your coding agent find it on its own

So you do not have to remember the command:

The daily mail. Every morning with new flags, the workspace's owner and developers get how many are open and the prompt to paste into Claude Code, /vatio:vatio-improve (The daily digest).
AGENTS.md. vatio init writes a short Vatio section into the workspace's AGENTS.md, or appends one to an existing file that does not mention Vatio. Most coding agents read it without installing anything.
The vatio-improve skill, for Claude Code:
```
/plugin marketplace add vatio-ai/skills
/plugin install vatio@vatio
```
Claude Code then uses it when you ask it to improve or fix your agent, or as /vatio:vatio-improve. For any other coding agent, npx @vatio-ai/skills from the root of your project copies it to .agents/skills/, where Codex, Cursor and most others read skills. See Claude Code.

They only point at the CLI. The instructions come from the platform each time vatio flags runs, so none of them goes out of date.

#What a flag carries

In the digest	Where it comes from
Agent, environment, deployed version, channel	The chat the reply was in
The conversation up to the flagged reply	The last 10 messages before it
What the agent did on that turn	Each tool call: its arguments, the HTTP status, outcome and duration of the call to your backend, and what it returned; knowledge lookups; tokens
What should have happened	The supervisor's note
Eval case	The case Vatio created from the flag

A flag without a note is not in the digest and has no eval case: "this was wrong" without saying what should have happened is not something an agent can be fixed from. Removing the flag removes its case.

#A flag's state

State	Means
Open	It fails on the agent visitors talk to
Fixed in preview	`vatio eval` passed it on preview's current version, which is not published yet
Resolved	Live passes it. It shows the short sha of the version that fixed it, like `a1b2c3d`

Until the agent's first publish there is no live, so preview is the agent: a flag resolves as soon as vatio eval passes it there. Rewriting the note is a new request, so the flag goes back to open. A resolved flag that fails on preview is a regression, and preview's version cannot be published until it passes again.

#Eval runs

vatio eval replays every eval case of the workspace against one environment, matching cases to that environment's agents by slug. Each case runs in a fresh chat: the visitor's turns (up to the last 8) are sent one by one and the agent answers each. A judge then reads the replay, including every tool the agent called, and the expected outcome, and answers pass or fail with one sentence of reasoning. A note that asks for a tool passes when the agent calls it. Everything runs on Vatio's servers: your coding agent only starts it and reads the report.


Environment	`preview`; `--env live` to test what visitors get
Speed	Five cases at a time. A `vatio eval`, or Run tests on the inbox's Test page, while a run of the same version is still going joins it instead of starting another
Exit code	`1` when any case fails or errors, so it can gate CI
Report	The open flags, the ones a fix is for, are counted and listed apart from the regression cases: flags already resolved. Each row is named by its flag's id, like `F-7K2QX`: a flag and its case share one
Regressions	A case that passes on live and fails on preview is listed first. It blocks the publish
The judge's reasons	`vatio eval --show RUN_ID`
Judge	A model from a different family than your agent's, so it never grades its own kind of answer
Retries	A case the judge fails is replayed again, in a fresh chat, up to 3 attempts. Only one that fails all three counts as failing; one that passed on a later attempt says how many it took, like `F-7K2QX (2 attempts)`, and is unstable: worth fixing even though it passed
Judge errors	An answer the judge could not read is an error, not a fail, and does not count against the agent
Daily limit	2,000 executions (replay chats, each attempt counted) per workspace a day, a guard against loops. Past it, new runs start the next day, and `vatio eval` answers `429 daily_test_limit`
Disagreeing with the judge	Rewrite the flag's note in the conversation. A new note is a new request: the flag reopens and its case is judged against the new note from the next run
Replay chats	Not in the inbox and not counted as conversations, live included: they are Vatio testing the agent, not visitors

A replay is a real agent turn, so a case can pass once and fail the next without anything changing: that is why a failing case is retried.

Evals call your tools

A replay is a real agent turn, so the tools it calls reach your backend. Set a test value for the secrets in their URLs, like a staging backend (vatio secrets set SHOP_API https://staging.example.com --tests): every replay uses it, live's included, and so does preview. A replay's calls carry an X-Vatio-Test: 1 header. A write (any method but GET) whose URL has no test value is not sent at all: the agent gets a success, and the judge still checks the call it made and its arguments. See Secrets.

vatio push runs no tests. A run starts with vatio eval, with Run tests on the inbox's Test page, or with that page's Publish to live on a version nobody tested; a publish that has no results to take live replays live after it.

Run tests replays every case on preview, the same as vatio eval, for whoever has no terminal open. The preview card it sits on says how many cases the newest run of preview's version passed, or that it has not been tested yet.

#The daily digest

Every morning at 8:00 (America/Santiago), each workspace with new flags mails its owner and developers how many flags arrived and how many are open, with the prompt to paste into Claude Code, /vatio:vatio-improve, and a link to the inbox's Continuous improvement page.

The mail never quotes what anyone wrote. Conversations, tool responses and supervisors' notes are only ever readable with a Vatio token.

#API

Endpoint	Returns
`GET /api/developer/v1/:workspace/flags.md`	The digest, as `vatio flags` prints it
`GET /api/developer/v1/:workspace/flags`	The open flags as JSON
`POST /api/developer/v1/:workspace/evals/runs`	Starts a run; body `{ "environment": "preview", "agent": "main" }`, both optional
`GET /api/developer/v1/:workspace/evals/runs/:id`	One run's progress: `cases`, `done`, `passed`, `failed`, `finished`
`GET /api/developer/v1/:workspace/evals/runs/:id.md`	One run's report

All of them take the same token as the CLI. The proposals endpoints answer 410 proposals_removed: test with vatio eval and publish with vatio publish.

Latest CLI version: 0.79.0 · Changelog · The whole contract as Markdown: /docs/docs.md · or vatio docs