#Flags, evals and tools

One session of your coding agent takes every open report:

vatio flags         # every open report, with how to work them at the top
# ... fix vatio.yml, knowledge, tools or the backend they call ...
vatio push          # updates preview; runs no tests
vatio eval          # tests every report on preview; exits 1 if one fails
vatio publish       # takes preview live, once eval has tested it

Every report with a note is a test, wherever it was made. vatio eval runs them on Vatio's servers, five at a time, and waits up to 900 seconds (--timeout N); the run keeps going if it stops waiting, and vatio eval --show RUN_ID prints it later. It can take a few minutes, so in Claude Code run it in the background and read the report when it finishes. A report that passes on live and fails on preview is listed first, as a regression: vatio publish refuses it. Replays call the environment's tools for real. The whole loop, and what resolves a report, is in Improve your agent from flags.

When neither the Vatio plugin for Claude Code nor the skills that npx @vatio-ai/skills copies into a project are installed, vatio flags and vatio eval end with a line on stderr saying so: without them Claude Code has no /vatio:vatio-improve. They also say when the plugin or the copy is older than the published skills, and how to update it (Claude Code). It shows at most once a day per directory, never in CI, and VATIO_NO_SKILLS_HINT=1 turns it off.

#Tool metrics

A tool that fails seldom gets reported: the agent apologizes and moves on. vatio tools is how your coding agent finds those, with the same numbers as the inbox's Tools page:

vatio tools --env live                 # the last 7 days on live
vatio tools --env live --period 1h     # did the fix you just published stop the errors?
vatio tools --env live --json          # the same, for a script

It defaults to live and to the last 7 days. The periods are 1h, 1d, 7d and 30d. --env preview only has the calls its own chats and eval replays made; visitors' calls are on live.

The markdown starts with how to work it, like vatio flags, then:

• By tool: every tool the environment declares, never-called ones included, sorted by failed calls: calls, failed calls, error rate, timeouts, p50 and p95 latency, and the last failure.
• Calls that failed: the 50 most recent, each with the request (method and URL, without the query), HTTP status, outcome, duration, the error_key your backend returned, what your backend answered, and the vatio chat show CHAT_ID that prints the conversation.

A call failed when it ended any way but ok: http_error (a status other than 2xx), timeout (8 seconds), connection_error, or invalid_response (a 2xx that was not JSON). What your backend answered is kept for 30 days, encrypted and cut to 4 KB, and can carry visitors' data: keep it out of commits, pull requests and issues.

--json prints:

{
  "environment": "live",
  "period": "7d",
  "since": "2026-10-02T14:00:00Z",
  "console_url": "https://inbox.vatio.ai/acme/tools?period=7d",
  "tools": [
    {
      "tool": "check_stock",
      "calls": 412,
      "failed": 9,
      "error_rate": 0.0218,
      "timeouts": 2,
      "p50_ms": 180,
      "p95_ms": 2400,
      "last_failure_at": "2026-10-09T13:12:44Z"
    }
  ],
  "failed_calls": [
    {
      "tool": "check_stock",
      "at": "2026-10-09T13:12:44Z",
      "outcome": "http_error",
      "http_status": 500,
      "error_key": null,
      "duration_ms": 312,
      "request_method": "GET",
      "request_url": "https://api.acme.com/stock",
      "response_body": "{\"error\":\"…\"}",
      "chat_id": 1234
    }
  ]
}

error_rate, p50_ms, p95_ms and last_failure_at are null for a tool with no calls, or no failures, in the period.