Agent Watchtower: a run log format, an eval checklist, and a cost and failure report for your agents

Run python3 watch_check.py eval evals.json runs/ --prices prices.json --min-score 90 and paste the RESULT line and every FAIL line.

From the pack.

# Agent Watchtower: log, score and test your agents before they do real work

A PaperChaseWebb pack, version 1.0, 2026-10-31. Works with any agent you can make write one line of JSON per step: your own Python or TypeScript agents, Claude Code or Codex hooks, n8n or Make flows, and Grok Bot routines. Plain Markdown plus two small Python files (3.8 or newer, nothing to install).

Most agents go live on a few good demos. Then a loop burns a day of tokens, a tool call fires without a yes, or a quiet failure rate sits at one run in five and nobody sees it. Watchtower is the small layer that catches this first: one log format, an eval checklist you write once, and a checker that turns the logs into a cost and failure report and a pass or fail gate before an agent gets real work.

## What is in the pack

| Section | What it is for |
|---|---|
| 01 What to watch | The five things that go wrong with agents in practice, and the signal for each |
| 02 The run log format | One JSON line per step: eight event types, required and optional fields, and a filled example |
| 03 Writing the log | `watch_log.py` for Python agents, plus the one-line pattern for any other tool |
| 04 The eval checklist | How to write test cases from real tasks, the fields a case can check, and a 12-point readiness list |
| 05 The checker | `watch_check.py log`, `report` and `eval`: every code and what to do about it |
| 06 The cost and failure report | Reading the report, setting limits, and a weekly review routine |
| 07 AGENTS.md | Rules that make an agent log itself honestly and stop at the right places |
| 08 How we tested it | Fixtures, unit tests, Python versions, and what was not tested |
| 09 The files | Both scripts, the unit tests, and sample logs, evals and prices |

## What you get

- **A run log format** small enough to add in an hour: run_start, model_call, tool_call, tool_result, retry, approval, error, run_end.
- **A checker** that finds malformed logs, keys pasted into logs, runs that never ended, loops (the same tool with the same arguments three times in a row), runs over a cost or step limit, and model calls with no price.
- **A cost and failure report** in plain text or Markdown: success rate, total cost, cost per successful run, the priciest run and the top failures.
- **An eval gate**: write cases from real tasks (must say, must not say, must call, must not call, needs a yes before, cost, steps and time limits) and get a score with a minimum you set.