Replicate Before You React: a trial kit for claims about AI agents
by PaperChaseWebb
Test a claim about a coding agent on your own setup before you change anything: a security hole, a token-saving prompt, a hidden cost, a skill that makes the agent better. Restate the claim, run each setup 10 to 20 times in fresh containers with one capped key, use a harmless canary instead of a real payload, run a control arm for cost and quality, and get rates with honest intervals and a result card that says what you did not test.
- Price
- $19
- Difficulty
- Intermediate
- Setup
- 1 hr
- Format
- .md file
replicate-before-you-react-trial-kit-for-agent-claims.md · 52 KB · instant download after checkout
Want this built for you instead? We can build it for you.
What you end up with
When the next alarming or exciting post about your agent arrives, you can answer it with a result card from your own setups (versions, runs, an interval and a verdict) instead of a same-day settings change or a shrug.
What you get
- 17 sections and an appendix in one .md file, with a one-line unpack for all 14 files
- Plan templates for security, cost and quality claims
- Five safety rules, including a canary that only creates an empty marker file
- trials.py: ten read-only checks, Wilson intervals for rates, bootstrap intervals for cost changes, a verdict and a Markdown result card
- run_trial.sh: one fresh container per run, one capped key, version recorded per run
- A skill file and AGENTS.md snippet for Claude Code and Codex
- Two worked examples, 19 unit tests and a runner test with a fake docker
- Eight prompts and a failure table
Who it is for
Anyone who has to decide whether a claim about their agent changes settings, rules files or budget, and wants a reason they can show someone else.
Works with
Solid chips: the agent is named in this listing's own tools list. Dashed chip: Written for Claude Code and Codex: a skill file, an AGENTS.md snippet and runner commands for claude -p and codex exec. trials.py and the runner's bookkeeping were tested on Linux; the runner was not run against a live agent and the skill file was not loaded inside either harness.
Tools needed
Cost to run: The pack is the only purchase. Trials use the agent's normal API spend; set a spend cap on the test key, since run cost depends on your agent, model and task.
Time to result: A first trial card in an afternoon
How it works
Unpack and test
Save the pack as pack.md in an empty folder, run the unpack line from section 6, then sh test_all.sh.
Expected: 14 files written; 19 tests OK, runner tests OK, 0 findings on both examples.
Restate the claim
Copy the matching template to trials/<claim>/plan.json. Restate the claim in your words with its link and date, and write not stated for what the source left out. Use prompt 1 or 2.
Expected: trials.py check shows only C7 findings (no runs yet). A person approves the plan.
Run the setups
Build an image with the agent pinned to your version, then run run_trial.sh once per setup and arm with a capped key. Read the first two rows by hand.
Expected: results.csv with a version on every row and at least min_runs rows per setup and arm.
Read the card
Run trials.py check until it shows 0 findings, then trials.py card > RESULT.md, and use prompt 7 for the team note.
Expected: A verdict per setup with a 95% interval and a list of what was not tested.
Sample prompts, free to use
Here is a post about an agent. Restate its claim in one sentence in your own words, list what it did not say (versions, settings, prompts, number of runs), and tell me which template fits. Do not quote the post.
Prompt 1 of 8.
Draft plan.json for this claim from templates/plan-security.json. The canary must only create an empty file at a marker path. Propose cells for default and sandboxed modes. Stop and show me the plan.
Prompt 2 of 8.
Run trials.py summary and explain the verdict as if to someone who saw the original post. Say what the intervals mean. Do not call the claim false.
Prompt 6 of 8.
Free preview of the file
# Replicate Before You React: a trial kit for claims about AI agents Version 1.0, 2026-10-29. By PaperChaseWebb. A method and a small working kit for testing a claim about a coding agent on your own setup before you change anything because of it: a security hole someone found, a prompt that saves tokens, a hidden cost, a skill that makes the agent better. A plan file for each kind of claim, a runner that does each run in a fresh container, a script with ten read-only checks that turns the runs into rates with honest intervals and a one-page result card, two worked examples, eight prompts and the tests, all Python 3 standard library plus POSIX sh and Docker. Written for Claude Code and Codex, through a skill file and an AGENTS.md snippet. Tested as described in section 10: trials.py and its 19 tests ran on Linux with Python 3.13, and the runner was tested against a fake docker that checks its bookkeeping. The runner has not been run against a live agent for this release, and the example numbers are made up for illustration. Each is marked where it matters. A post says an agent gets hijacked nine times in ten by a file in your Downloads folder. Another says one line in your rules file cuts tokens by 40%. A third says a background feature is quietly eating a sixth of your bill. Some of these are true, some were true for one version, and some were one lucky run. The usual responses are to ignore them or to change your setup that afternoon. Both skip the cheap step in between: r
Test record
- Date
- Oct 29, 2026
- What was run
- The unit tests, the runner test against a fake docker, and the unpack line with test_all.sh. Linux, Python 3.13.
- Result
- 19 unit tests passed; the runner test passed; the unpack line wrote 14 files identical to the sources and test_all.sh passed.
- Not run
- Real Docker or a live agent. The skill file and AGENTS.md snippet inside either harness.
Copied from this item's dated test notes. What was not run stays in.
Worth knowing
- run_trial.sh was tested against a fake docker that checks its bookkeeping (rows, versions, marker detection, values, cleanup). It was not run against real Docker or a live agent for this release. Run one setup with RUNS=2 and read the rows before a full trial.
- The example numbers are made up with a fixed seed to show the output. They are not results from Claude Code, Codex or any agent, and the example claims are invented.
- A trial answers one claim on the setups you ran. A result that does not reproduce says nothing about other versions, settings or prompts, and a security claim that does reproduce goes to the vendor, not to a public post.
- Agent CLI flags change. The pack uses claude -p and codex exec for non-interactive runs; check --help on your version. The skill file and AGENTS.md snippet were not loaded inside either harness.
- On 2026-10-29, on Linux with Python 3.13: 19 unit tests passed (each of the ten checks fires alone on a planted fault, Wilson interval matches the textbook value, four verdicts, exit codes 0, 1 and 2); the runner test passed; the unpack line wrote 14 files byte-identical to the sources and test_all.sh passed in that folder.
Questions
Is it safe to test a security claim?
Only with the pack's rules: a canary that creates one empty file, a fresh container per run, one capped API key and no real accounts. trials.py refuses canary text that mentions network tools, secrets or deletes. Never reproduce a real payload.
How many runs do I need?
At least 20 per setup for security and 10 per arm for cost and quality. Three hits in 20 runs still gives an interval of about 5% to 36%, which is why the card shows intervals.
Does it work with Grok Bot or Hermes?
trials.py and the plan format work with any agent you can run from a command line in a container, but the pack's commands and skill file are written for Claude Code and Codex only.
How is this different from Check the Claim?
Check the Claim is a reading method for any claim you come across. This pack runs a claim about an agent on your own setups and measures it.
Details
Version 1.0 · Updated Oct 29, 2026 · Markdown version of this page
License: Use inside your own team or company, including on client work. Do not resell or republish the pack itself. Download terms
Related
Replicate Before You React: a claim card, five rules and a rate script (free sample)
From Replicate Before You React: a ten-line claim card to fill in before you test a claim about your agent, five safety rules (canary never payload, throwaway environment, one capped key, a control arm, say what you ran) and a script that shows how much a handful of runs can actually tell you.
Check the Claim (expanded edition): a method, prompts and a skill for checking AI claims
Make your agent check income, tool, popularity, quote, news and offer claims before they reach a brief, a pitch, a purchase or a post: label how much was read, run the creator's own arithmetic, trace sources back to first hand, and give each claim one of five verdicts on a card the owner approves. v1.1 adds an evidence level label (full, caption, summary or wall), four new arithmetic tests, a section on tracing a source back to first hand, screens for offers, quotes and slogans, four more red flags, a sixth worked example, a claim folder with a fourteen-check read-only checker and an owner approval helper, agent files for Grok Bot, Claude Code, Codex and Hermes, and twelve new prompts (26 in all). Price unchanged.
Evidence Standard: what counts as proof for each kind of agent work, who checks it, and what is refused
Agents report done from the plan as often as from the result. Five plain files write down, for each class of work, the proof that counts, who checks it (never the producer), what is refused and how long proof is kept. Agents log each claim with its evidence, seal it by sha256 and report in the words evidence.py say allows: done, partly done with the gap named, waiting on the owner, or reported, not verified. The agent's own summary is never proof. Fifteen read-only checks run on the standard and the claims ledger.
Vet Before You Install: twelve checks, a sandbox recipe and a tested vetting folder for third-party agent code
Decide whether a skill, plugin, mod, MCP server, connector, repo or model gets into your agent setup: sort the seat, read every file that runs, map reach and the data path, pin and vendor it, trial it in a sandbox with a planted fake key, then record a decision the owner approves at their own terminal. v1.1 adds a seat sort before any vetting, two new checks (data path and reach, approval surfaces), evidence tiers, twelve token classes including hidden text, encoded strings, startup file edits and approval hooks, a fresh-session second read, data path questions for proxies and routers, dependency and model weight pinning, a fifth decision (not vettable) with recheck dates and an approved shelf, more special cases, a vetting folder with a fifteen-check read-only checker and an owner approval helper, agent files for Grok Bot, Claude Code, Codex and Hermes, and twelve new prompts (27 in all). Price unchanged.
Agent Cost Levers: cut cost per solved task without cutting quality
A Markdown pack that measures what your coding agent spends per finished task, then works five levers one at a time: thinking depth, context curation, direct tool routing, cache and reuse, and a verified stop. Includes a baseline sheet, a rules block and 10 prompts.
Token saver: keep long command output out of your coding agent's context
Two shell scripts and one rule line that make Claude Code, Codex, Cursor and other agents read a one-page summary of builds, tests and searches instead of thousands of lines.
Shrink the Attack Surface: patch with proof, then attack your own agent's tools
Your agent lists everything your app exposes, read-only: routes, open ports, public buckets, secrets in the repo, unused dependencies, debug endpoints, admin paths, CORS, API key scopes and its own tool permissions. You decide keep, remove or lock for each. Every fix gets a proof that fails before the change and passes after it, and the proof stays as a regression test. Then attack drills run against your agent's own tool gate, offline: pressure, impersonation, injected tool output, stale and non-owner approvals and a failing tool. Each is judged by what the tools actually did, and every drill that gets through becomes a test that fails before the fix and passes after. A failing tool stops at a retry budget and asks you, and every approval is yours, typed at your own terminal and tied to a sha256.
Release Watch: a per-change monitoring plan your agent writes from the diff, run by a fixed script
After a change goes live, your agent reads the diff and writes a small watch plan for that change: which signals it could move, against which baseline, by how much, for how long. A fixed script runs the plan every five minutes, so no agent sits in a loop. On a trip, a second agent triages read-only and proposes one next step. Rollbacks, flag flips and alert changes stay with you.