# Agent Watchtower: a run log format, an eval checklist, and a cost and failure report for your agents

> One JSON line per agent step, a Python logger that hides keys and closes crashed runs, and watch_check.py: log checks, a cost and failure report that finds loops, runs over a cost or step limit and unpriced calls, and an eval gate with cases for must say, must call, must not call, needs a yes before, cost, steps and time.

- URL: https://paperchasewebb.ai/bots/marketplace/items/agent-watchtower-run-logs-evals-cost-and-failure-report/
- Type: Prompt pack
- Category: Vibe coding
- By: PaperChaseWebb
- Works with: Claude Code, Codex, Grok Bot, Cursor (named in the listing); Plain .md and Python 3.8 or newer scripts, standard library only; unit tested on Python 3.8 to 3.13
- Price: $39 USD
- Difficulty: Intermediate
- Setup time: 1 hr
- Time to result: Logging and a first report in an afternoon
- Format: Markdown (.md) file, agent-watchtower-run-logs-evals-cost-and-failure-report.md
- Version: 1.0, updated 2026-10-31
- License: Use for your own work and for clients. Do not resell or republish the pack itself.

## What you end up with

You see what each agent run did and cost, catch loops and unapproved actions, and only give an agent real work after it passes your cases.

## What you get

- 9 sections in one .md file
- The run log format: eight event types
- watch_log.py for Python agents
- The eval checklist and a 12-point readiness list
- watch_check.py: log, report and eval
- A weekly review routine
- AGENTS.md rules
- Sample logs, evals and prices
- 23 unit tests

## Who it is for

Builders who want to know an agent works, what it costs and where it fails before it gets real work, without a monitoring platform.

## Tools needed

- Any agent that can append one JSON line per step: Claude Code or Codex hooks, n8n or Make flows, Grok Bot routines, your own code
- Python 3.8 or newer: runs watch_check.py and watch_log.py; standard library only
- You need: An agent you can make write a log line per step
- You need: Python 3.8 or newer

Cost to run: The pack is the only purchase. The scripts use no hosted model tokens.

## Sample prompt: Gate an agent (Any agent that can append one JSON line per step)

```
Run python3 watch_check.py eval evals.json runs/ --prices prices.json --min-score 90 and paste the RESULT line and every FAIL line.
```

From the pack.

## Free preview of agent-watchtower-run-logs-evals-cost-and-failure-report.md

````markdown
# Agent Watchtower: log, score and test your agents before they do real work

A PaperChaseWebb pack, version 1.0, 2026-10-31. Works with any agent you can make write one line of JSON per step: your own Python or TypeScript agents, Claude Code or Codex hooks, n8n or Make flows, and Grok Bot routines. Plain Markdown plus two small Python files (3.8 or newer, nothing to install).

Most agents go live on a few good demos. Then a loop burns a day of tokens, a tool call fires without a yes, or a quiet failure rate sits at one run in five and nobody sees it. Watchtower is the small layer that catches this first: one log format, an eval checklist you write once, and a checker that turns the logs into a cost and failure report and a pass or fail gate before an agent gets real work.

## What is in the pack

| Section | What it is for |
|---|---|
| 01 What to watch | The five things that go wrong with agents in practice, and the signal for each |
| 02 The run log format | One JSON line per step: eight event types, required and optional fields, and a filled example |
| 03 Writing the log | `watch_log.py` for Python agents, plus the one-line pattern for any other tool |
| 04 The eval checklist | How to write test cases from real tasks, the fields a case can check, and a 12-point readiness list |
| 05 The checker | `watch_check.py log`, `report` and `eval`: every code and what to do about it |
| 06 The cost and failure report | Reading the report, setting limits, and a weekly review routine |
| 07 AGENTS.md | Rules that make an agent log itself honestly and stop at the right places |
| 08 How we tested it | Fixtures, unit tests, Python versions, and what was not tested |
| 09 The files | Both scripts, the unit tests, and sample logs, evals and prices |

## What you get

- **A run log format** small enough to add in an hour: run_start, model_call, tool_call, tool_result, retry, approval, error, run_end.
- **A checker** that finds malformed logs, keys pasted into logs, runs that never ended, loops (the same tool with the same arguments three times in a row), runs over a cost or step limit, and model calls with no price.
- **A cost and failure report** in plain text or Markdown: success rate, total cost, cost per successful run, the priciest run and the top failures.
- **An eval gate**: write cases from real tasks (must say, must not say, must call, must not call, needs a yes before, cost, steps and time limits) and get a score with a minimum you set.
````

## Test record

- Date: Oct 31, 2026
- What was run: 23 unit tests on Python 3.8, 3.9, 3.11, 3.12 and 3.13 with warnings as errors, from the source files and from the files extracted out of the pack, with the sample logs, evals and prices from the pack.
- Result: All 23 tests passed on every version from both copies.
- Not run: Live hooks inside Claude Code, Codex, n8n, Make or Zapier.

Copied from this item's dated test notes. What was not run stays in.

## Worth knowing

- Key detection uses patterns for common key formats and will not catch every secret.
- Prices in the sample file are examples; use your provider's current prices.
- Tested 2026-10-31: 23 unit tests on Python 3.8, 3.9, 3.11, 3.12 and 3.13 with warnings as errors, from the source files and from the files extracted out of the pack, with the sample logs, evals and prices from the pack. Result: All 23 tests passed on every version from both copies. Not run: Live hooks inside Claude Code, Codex, n8n, Make or Zapier.

## Questions

**Do I need a monitoring service?** No. Logs are plain files and the checker runs on your machine.

## How to buy

$39 one time at https://paperchasewebb.ai/bots/marketplace/items/agent-watchtower-run-logs-evals-cost-and-failure-report/. Stripe checkout, then an instant download page; the download link is also in your emailed receipt.

Seller: PaperChaseWebb Inc., mgmt@paperchasewebb.com