Release Watch: a per-change monitoring plan your agent writes from the diff, run by a fixed script
by PaperChaseWebb
After a change goes live, your agent reads the diff and writes a small watch plan for that change: which signals it could move, against which baseline, by how much, for how long. A fixed script runs the plan every five minutes, so no agent sits in a loop. On a trip, a second agent triages read-only and proposes one next step. Rollbacks, flag flips and alert changes stay with you.
- Price
- $24
- Difficulty
- Intermediate
- Setup
- 30 min
- Format
- .md file
release-watch-per-change-monitoring-plan-from-the-diff.md · 49 KB · instant download after checkout
Want this built for you instead? We can build it for you.
What you end up with
Each change gets its own watch tied to the paths it touched, a silent traffic drop trips as reliably as an error spike, and nothing is rolled back until you have read the triage and said yes.
What you get
- 18 sections in one .md file
- watch_run.py: increase, above and drop rules with persist, floors and minimum samples; CSV and Prometheus sources, GET only; exit 1 only on a trip, exit 2 on bad input
- A plan prompt that writes the watch from the diff and a read-only triage prompt that proposes and stops
- A skill file for Claude Code, Codex and Hermes, an AGENTS.md snippet, a Grok Bot rules block and timer setup
- Owner approvals in approvals.csv tied to the plan's sha256
- watch_check.py: fifteen read-only checks, with a fixture builder and the expected output
- A labeled eval set of six cases to count false alarms and misses before you trust it
- Tests: a runner selftest, a mock Prometheus test and one test_all.sh
- Ten prompts and a failure table
- A look at eight canary skills, rollback skills and runbooks, with sources and credit for the idea
Who it is for
Builders and small teams who merge many changes a day with help from agents, have metrics somewhere, and want each change watched without paying an agent to stare at dashboards.
Works with
Solid chips: the agent is named in this listing's own tools list. Dashed chip: A skill file for Claude Code, Codex and Hermes, an AGENTS.md snippet and a Grok Bot rules block; the runner and checker were tested on CSV metrics and a mock Prometheus, not a live production server, and the prompts were not run inside each harness.
Tools needed
Cost to run: The pack is the only purchase. The runner uses no model tokens; the agent is used only to write a plan and to triage a trip.
Time to result: A first plan and a tested runner in about half an hour
How it works
Run the tests
Copy the release-watch folder into your repo and run sh test_all.sh.
Expected: 14 ok and 3 ok with 0 failures, bad plan exits 2, 15 findings on seeded and 0 on fixed, eval caught 3, missed 0, false alarms 0.
List your sources
Have your agent draft WATCH-SOURCES.md from your dashboards with prompt 10: metric name, source, unit and what normal looks like.
Expected: A short list you have read and corrected.
Write the first plan
After your next merge, run the plan prompt. It runs watch_check.py until 0 findings and shows you the plan and its sha256.
Expected: A plan where every touched path is covered and every signal quotes the line it protects.
Approve and run
Add the approvals.csv row with the hash, then start the runner on a five-minute timer.
Expected: runlog.jsonl grows by one line per run and report.md shows WATCHING, then CLEAR or TRIPPED.
Sample prompt from the pack
Read the diff for this change and write the watch plan with PLAN-PROMPT.md. Run watch_check.py until it shows 0 findings. Show me the plan and its sha256, then stop. Do not start the watch.
Prompt 1 of 10. The owner adds the approval row.
The watch tripped. Run TRIAGE.md read-only and stop after writing triage.md. Propose one next step. Do not roll back, flip a flag or change an alert.
Prompt 5 of 10. Run it in a session with read-only tools.
Free preview of the file
# Release Watch: a per-change monitoring plan your agent writes from the diff, run by a fixed script Version 1.0, 2026-10-28. By PaperChaseWebb. Works with Grok Bot, Claude Code, Codex and Hermes. Plain Markdown prompts, a skill file, a rules block, an AGENTS.md snippet, a runner, a read-only checker and a labeled eval set, all Python 3 standard library. Tested as described in section 11: everything ran on Linux with Python 3.13.5 against CSV metrics and a mock Prometheus server on 127.0.0.1. It was not run against a live production Prometheus, and the prompts were not run inside each harness for this release. Both are marked where they matter. A change merges. Dashboards say nothing in particular. Two hours later someone notices checkout traffic is down a third, and nobody can say which of the eleven deploys that afternoon did it. The usual fixes do not help much. A wall of global alerts pages on everything or nothing. A post-deploy checklist gets skipped on a busy day. An agent told to "watch production" keeps a session open, reads the same graphs every minute, spends tokens doing it, and is free to type a rollback command when it gets nervous. Release Watch splits the job three ways. An agent reads the diff and writes a small plan for that one change: which signals the change could move, against which baseline, by how much, for how long. A fixed script runs the plan on a timer, so no agent sits in a loop. When a signal trips, a second agent triages read-only and writes
Worth knowing
- The runner was tested on CSV metrics and on a mock Prometheus query_range server on 127.0.0.1, not on a live production Prometheus. Try one plan on a quiet change before you rely on it.
- The plan and triage prompts and the skill file were not run inside Claude Code, Codex, Hermes or Grok Bot for this release. Read the first plan your agent writes before you approve it.
- This is not a replacement for global alerts on outages that have nothing to do with a change.
- On 2026-10-28, on Linux with Python 3.13.5, the runner selftest gave 14 ok, the mock Prometheus test 3 ok, both with 0 failures; a plan with no timezone exited 2; watch_check.py reported the fifteen expected findings on the seeded fixture and 0 on the fixed one, with files unchanged and exit 2 on a missing folder; the eval set gave caught 3, missed 0, false alarms 0, quiet and right 3.
Questions
Why not let the agent watch the dashboards itself?
An agent in a loop costs tokens every minute, can give different answers on the same data, and can be talked into a rollback. A fixed script gives the same answer every time, costs nothing to run, and only wakes an agent when something trips.
Will it roll back for me?
No, on purpose. The report names the rollback command as a proposal, triage writes up the evidence, and you decide. The checker flags any plan that sets a rollback or action to run automatically.
Do I need Prometheus?
No. The runner also reads a CSV with time, metric and value columns, so any tool that can export metrics works.
How do I know it is not too noisy?
Run the eval set, then add your own past incidents and quiet releases as labeled cases. It prints caught, missed and false alarms, so you tune with numbers.
Details
Version 1.0 · Updated Oct 28, 2026 · Markdown version of this page
License: Use in your own repos and on installs you do for clients and teams. Do not resell or republish the pack itself. Download terms
Related
A per-change watch plan template with five rules and a five-check script (free sample)
From Release Watch: five rules for watching a change after it goes live, a plan template, one prompt that has your agent write the plan from the diff, and a read-only script that checks the plan before anything watches it, with a worked example.
Turn production alerts into diagnosed fix PRs with a cloud agent
A small SaaS team wired its alerts to a Claude Code routine that finds the root cause, explains it in the Slack alert thread and opens a fix PR, and its first serious incident went from alert to merged fix in 13 minutes.
Two-Agent Build and Test Loop for paid apps: one agent builds, another tests, and nothing passes on its own say-so
A tested setup for apps that take money: one agent writes the code, a separate tester session writes the stories and tests and gives the verdicts, a small runner records every red, green and revert exit code itself, and the owner says yes to the plan, any payment or pricing work, each merge and each release. Test keys and sandboxes only. A read-only check with fifteen tests audits the finished run.
Planner Over Cheap Models: one strong planner, cheap workers, and a check that keeps both honest
A tested setup for splitting coding work between one strong model that plans and reviews and cheap or local models that write the code. Every task card carries its own check, a logger runs that check instead of trusting the worker, the owner says yes to the plan, any escalation and every merge, and a read-only check with fourteen tests audits the finished run. Includes a worker bake-off so you pick the cheap model on your own tasks.
Agent Readiness Self-Check: score one workflow before you hand it to an AI agent
A free self-check for agencies and small businesses. Score one workflow across eight areas (workflow mapped, data access, approval gates, owner per agent, handoffs, cost tracking, security and vetting, measurement) with an evidence rule and three gate rules, read your band and what to do next, plus an interview prompt for Grok Bot, Claude Code, Codex or Hermes, a score sheet and a worked example.
Shrink the Attack Surface: patch with proof
Your agent lists everything your app exposes, read-only: routes, open ports, public buckets, secrets in the repo, unused dependencies, debug endpoints, admin paths, CORS, API key scopes and its own tool permissions. You decide keep, remove or lock for each. Every fix gets a proof that fails before the change and passes after it, and the proof stays as a regression test. Red-team drills then check that the agent still asks before risky steps, stops retrying broken tools and never charges or emails twice.
Deal Alert Worker: a price watch your agent sets up and a fixed script runs
Tell your agent what you want and the most you will pay with delivery. It checks each site's robots.txt and terms, finds which ones answer a plain request and which need your own browser, saves a real page from each, and writes a small spec you approve by its hash. A fixed script then reads on a timer and alerts you, and only you, when an item clears your total on two reads in a row. Same listing, same price: one alert. Night finds wait for a morning batch. A broken source tells you it broke. Nothing ever buys.
Production Readiness Audit: decide from evidence whether an agent system can go live
Twelve areas, each pass, gap or na, and every pass points to a dated evidence file. Gaps get a named owner and a due date, five areas block go-live until fixed, drills are run rather than described, and the verdict is signed by someone who did not build the system. A fifteen-check script reads the folder, the report is sealed to a hash of the evidence, and the owner's go-live yes is tied to the report's sha256.