# Flags, evals and tools

| Command | Purpose |
|---|---|
| `vatio flags` | Every open report as markdown: the conversation, each tool call of that turn with your backend's status, duration and response, and what should have happened |
| `vatio flags --save [PATH]` | The same, written to `vatio-flags.md` or `PATH` to hand to a coding agent |
| `vatio flags --json` | The open reports as JSON rows |
| `vatio eval [--env NAME] [--agent SLUG] [--timeout N]` | Test every report against an environment, `preview` by default, wait for the judge, print the report; exits `1` if a case fails |
| `vatio eval --no-wait` | Start the runs and print their ids |
| `vatio eval --show RUN_ID` | One run's report |
| `vatio tools [--env NAME] [--period 1h\|1d\|7d\|30d]` | The inbox's **Tools** page as markdown: each tool's calls, error rate, timeouts and p50/p95 latency, then the calls that failed with what your backend answered |
| `vatio tools --json` | The same as JSON |

One session of your coding agent takes every open report:

```bash
vatio flags         # every open report, with how to work them at the top
# ... fix vatio.yml, knowledge, tools or the backend they call ...
vatio push          # updates preview; runs no tests
vatio eval          # tests every report on preview; exits 1 if one fails
vatio publish       # takes preview live, once eval has tested it
```

Every report with a note is a test, wherever it was made. `vatio eval` runs
them on Vatio's servers, five at a time, and waits up to 900 seconds
(`--timeout N`); the run keeps going if it stops waiting, and
`vatio eval --show RUN_ID` prints it later. It can take a few minutes, so in
Claude Code run it in the background and read the report when it finishes. A
report that passes on live and fails on preview is listed first, as a
regression: [`vatio publish`](https://vatio.ai/docs/cli/deploy#the-publish-gate) refuses it.
Replays call the environment's tools for real. The whole loop, and what
resolves a report, is in [Improve your agent from flags](https://vatio.ai/docs/improve).

When neither the Vatio plugin for Claude Code nor the skills that
`npx @vatio-ai/skills` copies into a project are installed, `vatio flags` and
`vatio eval` end with a line on stderr saying so: without them Claude Code has
no `/vatio:vatio-improve`. They also say when the plugin or the copy is older
than the published skills, and how to update it
([Claude Code](https://vatio.ai/docs/integrations/claude-code#keeping-them-up-to-date)). It shows at
most once a day per directory, never in CI, and `VATIO_NO_SKILLS_HINT=1` turns
it off.

## Tool metrics

A tool that fails seldom gets reported: the agent apologizes and moves on.
`vatio tools` is how your coding agent finds those, with the same numbers as
the inbox's [Tools page](https://vatio.ai/docs/tools#how-your-backend-is-answering):

```bash
vatio tools --env live                 # the last 7 days on live
vatio tools --env live --period 1h     # did the fix you just published stop the errors?
vatio tools --env live --json          # the same, for a script
```

It defaults to `live` and to the last 7 days. The periods are `1h`, `1d`,
`7d` and `30d`. `--env preview` only has the calls its own chats and eval
replays made; visitors' calls are on live.

The markdown starts with how to work it, like `vatio flags`, then:

- **By tool:** every tool the environment declares, never-called ones
  included, sorted by failed calls: calls, failed calls, error rate,
  timeouts, p50 and p95 latency, and the last failure.
- **Calls that failed:** the 50 most recent, each with the request
  (method and URL, without the query), HTTP status, outcome, duration, the
  `error_key` your backend returned, what your backend answered, and the
  `vatio chat show CHAT_ID` that prints the conversation.

A call failed when it ended any way but `ok`: `http_error` (a status other
than 2xx), `timeout` (8 seconds), `connection_error`, or `invalid_response`
(a 2xx that was not JSON). What your backend answered is kept for 30 days,
encrypted and cut to 4 KB, and can carry visitors' data: keep it out of
commits, pull requests and issues.

`--json` prints:

```json
{
  "environment": "live",
  "period": "7d",
  "since": "2026-10-02T14:00:00Z",
  "console_url": "https://inbox.vatio.ai/acme/tools?period=7d",
  "tools": [
    {
      "tool": "check_stock",
      "calls": 412,
      "failed": 9,
      "error_rate": 0.0218,
      "timeouts": 2,
      "p50_ms": 180,
      "p95_ms": 2400,
      "last_failure_at": "2026-10-09T13:12:44Z"
    }
  ],
  "failed_calls": [
    {
      "tool": "check_stock",
      "at": "2026-10-09T13:12:44Z",
      "outcome": "http_error",
      "http_status": 500,
      "error_key": null,
      "duration_ms": 312,
      "request_method": "GET",
      "request_url": "https://api.acme.com/stock",
      "response_body": "{\"error\":\"…\"}",
      "chat_id": 1234
    }
  ]
}
```

`error_rate`, `p50_ms`, `p95_ms` and `last_failure_at` are `null` for a tool
with no calls, or no failures, in the period.