# Your daily improvement routine

If you use Claude Code, the routine is three steps:

1. **Report:** in the inbox, mark a wrong reply and say what should have happened.
2. **Improve:** open your app's project and run `/vatio:vatio-improve`.
3. **Review and publish:** Claude prepares fixes; Vatio tests them against preview.
   Check the results and approve the update when you are ready.

New reports also arrive in your daily email with the same command. The rest
of this page explains the checks and commands the skill runs for you.

The people reading your agent's conversations know when a reply is wrong.
In the [inbox](https://vatio.ai/docs/channels/inbox), a supervisor **flags** that reply and writes
what should have happened: a different answer, an action, a tool the agent
should have used or does not have. From then on, that flag is a test. Your
coding agent fixes it; Vatio checks the fix:

1. **A digest.** `vatio flags` prints every open flag as one markdown document
   written for a coding agent: the conversation up to the reply, every tool
   the agent called on that turn with **what your backend answered**, and the
   supervisor's note.
2. **A test per flag.** Every flag with a note is an eval case, wherever it
   was made, live or preview. The visitor's turns become the input and the
   note becomes the expected outcome. `vatio eval` replays every case against
   preview and a judge compares each new turn, tool calls included, with the
   note.
3. **Resolution on evidence.** A flag that passes on preview is **fixed in
   preview**; once that version is published it is **resolved**, with the
   version that fixed it. Nobody has to say it is done. Supervisors see each
   flag's state on the reply they flagged, and every flag with its latest
   tests in the inbox's
   [Continuous improvement](https://vatio.ai/docs/channels/inbox#continuous-improvement) page.

Your coding agent has your repository and your backend in front of it, which
Vatio does not. That is why the fix is its job: the cause is often a tool
response missing a field, not the prompt.

Flags only catch what a supervisor noticed. A tool that fails seldom gets
flagged, because the agent apologizes and moves on, so `vatio tools --env
live` gives your coding agent the inbox's **Tools** page: each tool's error
rate, timeouts and latency, and the calls that failed with what your backend
answered. See [Tool metrics](https://vatio.ai/docs/cli/improve#tool-metrics).

## The loop

In Claude Code, from the repository with the `vatio.yml`:

```
/vatio:vatio-improve
```

One session takes every open flag. This is what it runs, and what you run
with any other coding agent:

```bash
vatio flags --save     # vatio-flags.md: hand it to your coding agent
# ... your agent fixes vatio.yml, knowledge, tools or the backend they call ...
vatio push             # updates preview; runs no tests
vatio eval             # replays every flag on preview; exits 1 if one fails
vatio publish          # refused while preview's version is untested or breaks a flag live passes
```

`vatio eval` takes minutes: every case is a real agent turn, five run at a
time, so 27 cases take three or four minutes. In Claude Code, run it in the
background and keep working. Push and eval again until the flags you meant to
fix pass, then publish: preview's results go live with it, so live is not
replayed again and the flags they pass are resolved at once.

A prompt that works with any coding agent:

> Read the output of `vatio flags`. Group the flags by cause, fix each cause
> where it lives — `vatio.yml`, knowledge, a tool, or the backend a tool calls
> — push it, prove it with `vatio eval`, and tell me which flags now pass.

The digest itself carries the same instructions at the top, so an agent
handed only the file knows what to do with it.

### What publish checks

`vatio publish` and the **Publish to live** button on the inbox's **Test**
page go through the same gate. It looks at preview's current version:

| | `vatio publish` |
|---|---|
| It is already live's version | Nothing to publish: only preview's knowledge changes go live, and `vatio publish` prints a notice saying so. The **Test** page shows no **Publish to live** button then |
| The agent was never published | Publishes. If `vatio eval` already finished on that version, its results go live with it |
| No flags yet | Publishes |
| Not tested | Refused, `evals_missing`: run `vatio eval`, then publish |
| Its run is still going | Refused, `evals_running`: `vatio eval` waits for it |
| A flag that passes on live fails on it | Refused, `evals_regressed`, naming each flag. `vatio publish --force` publishes anyway, and its failing results go live with it |
| Otherwise | Publishes, and the flags its run passed are resolved with its short sha |

Flags that still fail do not block a publish: a fix for three of five can go
live while the other two wait. A version published with no results of its
own, the first publish before any `vatio eval` for instance, has live replayed
right after it.

### Let your coding agent find it on its own

So you do not have to remember the command:

- **The daily mail.** Every morning with new flags, the workspace's owner and
  developers get how many are open and the prompt to paste into Claude Code,
  `/vatio:vatio-improve` ([The daily digest](#the-daily-digest)).
- **`AGENTS.md`.** `vatio init` writes a short Vatio section into the
  workspace's `AGENTS.md`, or appends one to an existing file that does not
  mention Vatio. Most coding agents read it without installing anything.
- **The `vatio-improve` skill**, for Claude Code:

  ```
  /plugin marketplace add vatio-ai/skills
  /plugin install vatio@vatio
  ```

  Claude Code then uses it when you ask it to improve or fix your agent, or
  as `/vatio:vatio-improve`. For any other coding agent,
  `npx @vatio-ai/skills` from the root of your project copies it to
  `.agents/skills/`, where Codex, Cursor and most others read skills. See
  [Claude Code](https://vatio.ai/docs/integrations/claude-code).

They only point at the CLI. The instructions come from the platform each time
`vatio flags` runs, so none of them goes out of date.

## What a flag carries

| In the digest | Where it comes from |
|---|---|
| Agent, environment, deployed version, channel | The chat the reply was in |
| The conversation up to the flagged reply | The last 10 messages before it |
| What the agent did on that turn | Each tool call: its arguments, the HTTP status, outcome and duration of the call to your backend, and what it returned; knowledge lookups; tokens |
| What should have happened | The supervisor's note |
| Eval case | The case Vatio created from the flag |

A flag without a note is not in the digest and has no eval case: "this was
wrong" without saying what should have happened is not something an agent can
be fixed from. Removing the flag removes its case.

## A flag's state

| State | Means |
|---|---|
| **Open** | It fails on the agent visitors talk to |
| **Fixed in preview** | `vatio eval` passed it on preview's current version, which is not published yet |
| **Resolved** | Live passes it. It shows the short sha of the version that fixed it, like `a1b2c3d` |

Until the agent's first publish there is no live, so preview is the agent: a
flag resolves as soon as `vatio eval` passes it there. Rewriting the note is a
new request, so the flag goes back to open. A resolved flag that fails on
preview is a regression, and preview's version cannot be published until it
passes again.

## Eval runs

`vatio eval` replays every eval case of the workspace against one
environment, matching cases to that environment's agents by slug. Each case
runs in a fresh chat: the visitor's turns (up to the last 8) are sent one by
one and the agent answers each. A judge then reads the replay, including every
tool the agent called, and the expected outcome, and answers pass or fail with
one sentence of reasoning. A note that asks for a tool passes when the agent
calls it. Everything runs on Vatio's servers: your coding agent only starts
it and reads the report.

| | |
|---|---|
| Environment | `preview`; `--env live` to test what visitors get |
| Speed | Five cases at a time. A `vatio eval`, or **Run tests** on the inbox's **Test** page, while a run of the same version is still going joins it instead of starting another |
| Exit code | `1` when any case fails or errors, so it can gate CI |
| Report | The open flags, the ones a fix is for, are counted and listed apart from the regression cases: flags already resolved. Each row is named by its flag's id, like `F-7K2QX`: a flag and its case share one |
| Regressions | A case that passes on live and fails on preview is listed first. It blocks the publish |
| The judge's reasons | `vatio eval --show RUN_ID` |
| Judge | A [model](https://vatio.ai/docs/model) from a different family than your agent's, so it never grades its own kind of answer |
| Retries | A case the judge fails is replayed again, in a fresh chat, up to 3 attempts. Only one that fails all three counts as failing; one that passed on a later attempt says how many it took, like `F-7K2QX (2 attempts)`, and is unstable: worth fixing even though it passed |
| Judge errors | An answer the judge could not read is an error, not a fail, and does not count against the agent |
| Daily limit | 2,000 executions (replay chats, each attempt counted) per workspace a day, a guard against loops. Past it, new runs start the next day, and `vatio eval` answers `429 daily_test_limit` |
| Disagreeing with the judge | Rewrite the flag's note in the conversation. A new note is a new request: the flag reopens and its case is judged against the new note from the next run |
| Replay chats | Not in the inbox and not counted as conversations, live included: they are Vatio testing the agent, not visitors |

A replay is a real agent turn, so a case can pass once and fail the next
without anything changing: that is why a failing case is retried.

::: warning Evals call your tools
A replay is a real agent turn, so the tools it calls reach your backend. Set a
test value for the secrets in their URLs, like a staging backend
(`vatio secrets set SHOP_API https://staging.example.com --tests`): every
replay uses it, live's included, and so does preview. A replay's calls carry
an `X-Vatio-Test: 1` header. A write (any method but GET) whose URL has no
test value is not sent at all: the agent gets a success, and the judge still
checks the call it made and its arguments. See
[Secrets](https://vatio.ai/docs/deployments#secrets).
:::

`vatio push` runs no tests. A run starts with `vatio eval`, with **Run
tests** on the inbox's **Test** page, or with that page's **Publish to live**
on a version nobody tested; a publish that has no results to take live
replays live after it.

**Run tests** replays every case on preview, the same as `vatio eval`, for
whoever has no terminal open. The preview card it sits on says how many
cases the newest run of preview's version passed, or that it has not been
tested yet.

## The daily digest

Every morning at 8:00 (America/Santiago), each workspace with new flags mails
its owner and developers how many flags arrived and how many are open, with
the prompt to paste into Claude Code, `/vatio:vatio-improve`, and a link to
the inbox's [Continuous improvement](https://vatio.ai/docs/channels/inbox#continuous-improvement)
page.

The mail never quotes what anyone wrote. Conversations, tool responses and
supervisors' notes are only ever readable with a Vatio token.

## API

| Endpoint | Returns |
|---|---|
| `GET /api/developer/v1/:workspace/flags.md` | The digest, as `vatio flags` prints it |
| `GET /api/developer/v1/:workspace/flags` | The open flags as JSON |
| `POST /api/developer/v1/:workspace/evals/runs` | Starts a run; body `{ "environment": "preview", "agent": "main" }`, both optional |
| `GET /api/developer/v1/:workspace/evals/runs/:id` | One run's progress: `cases`, `done`, `passed`, `failed`, `finished` |
| `GET /api/developer/v1/:workspace/evals/runs/:id.md` | One run's report |

All of them take the same token as the CLI. The proposals endpoints answer
`410 proposals_removed`: test with `vatio eval` and publish with
`vatio publish`.