# Replicate Before You React: a trial kit for claims about AI agents

> Test a claim about a coding agent on your own setup before you change anything: a security hole, a token-saving prompt, a hidden cost, a skill that makes the agent better. Restate the claim, run each setup 10 to 20 times in fresh containers with one capped key, use a harmless canary instead of a real payload, run a control arm for cost and quality, and get rates with honest intervals and a result card that says what you did not test.

- URL: https://paperchasewebb.ai/bots/marketplace/items/replicate-before-you-react-trial-kit-for-agent-claims/
- Type: Prompt pack
- Category: Vibe coding
- By: PaperChaseWebb
- Works with: Claude Code, Codex (named in the listing); Written for Claude Code and Codex: a skill file, an AGENTS.md snippet and runner commands for claude -p and codex exec. trials.py and the runner's bookkeeping were tested on Linux; the runner was not run against a live agent and the skill file was not loaded inside either harness
- Price: $19 USD
- Difficulty: Intermediate
- Setup time: 1 hr
- Time to result: A first trial card in an afternoon
- Format: Markdown (.md) file, replicate-before-you-react-trial-kit-for-agent-claims.md
- Version: 1.0, updated 2026-10-29
- License: Use inside your own team or company, including on client work. Do not resell or republish the pack itself.

## What you end up with

When the next alarming or exciting post about your agent arrives, you can answer it with a result card from your own setups (versions, runs, an interval and a verdict) instead of a same-day settings change or a shrug.

## What you get

- 17 sections and an appendix in one .md file, with a one-line unpack for all 14 files
- Plan templates for security, cost and quality claims
- Five safety rules, including a canary that only creates an empty marker file
- trials.py: ten read-only checks, Wilson intervals for rates, bootstrap intervals for cost changes, a verdict and a Markdown result card
- run_trial.sh: one fresh container per run, one capped key, version recorded per run
- A skill file and AGENTS.md snippet for Claude Code and Codex
- Two worked examples, 19 unit tests and a runner test with a fake docker
- Eight prompts and a failure table

## Who it is for

Anyone who has to decide whether a claim about their agent changes settings, rules files or budget, and wants a reason they can show someone else.

## Tools needed

- Claude Code or Codex: drafts plan.json from a template, reads results and writes the result card; the agent under test runs inside throwaway containers through claude -p or codex exec
- Python 3: runs trials.py (ten read-only checks, intervals, verdict and result card); standard library only, no network
- Docker: one fresh container per run through run_trial.sh, with one spend-capped API key and nothing else
- You need: Claude Code or Codex
- You need: Python 3.9 or newer
- You need: Docker, or throwaway VMs
- You need: A test API key with a spend cap

Cost to run: The pack is the only purchase. Trials use the agent's normal API spend; set a spend cap on the test key, since run cost depends on your agent, model and task.

## How it works

1. **Unpack and test.** Save the pack as pack.md in an empty folder, run the unpack line from section 6, then sh test_all.sh. Expected: 14 files written; 19 tests OK, runner tests OK, 0 findings on both examples.
2. **Restate the claim.** Copy the matching template to trials/<claim>/plan.json. Restate the claim in your words with its link and date, and write not stated for what the source left out. Use prompt 1 or 2. Expected: trials.py check shows only C7 findings (no runs yet). A person approves the plan.
3. **Run the setups.** Build an image with the agent pinned to your version, then run run_trial.sh once per setup and arm with a capped key. Read the first two rows by hand. Expected: results.csv with a version on every row and at least min_runs rows per setup and arm.
4. **Read the card.** Run trials.py check until it shows 0 findings, then trials.py card > RESULT.md, and use prompt 7 for the team note. Expected: A verdict per setup with a 95% interval and a list of what was not tested.

## Sample prompt: Restate a claim (Claude Code, Codex)

```
Here is a post about an agent. Restate its claim in one sentence in your own words, list what it did not say (versions, settings, prompts, number of runs), and tell me which template fits. Do not quote the post.
```

Prompt 1 of 8.

## Sample prompt: Plan a security trial (Claude Code, Codex)

```
Draft plan.json for this claim from templates/plan-security.json. The canary must only create an empty file at a marker path. Propose cells for default and sandboxed modes. Stop and show me the plan.
```

Prompt 2 of 8.

## Sample prompt: Explain the verdict (Claude Code, Codex)

```
Run trials.py summary and explain the verdict as if to someone who saw the original post. Say what the intervals mean. Do not call the claim false.
```

Prompt 6 of 8.

## Free preview of replicate-before-you-react-trial-kit-for-agent-claims.md

````markdown
# Replicate Before You React: a trial kit for claims about AI agents

Version 1.0, 2026-10-29. By PaperChaseWebb. A method and a small working kit for testing a claim about a coding agent on your own setup before you change anything because of it: a security hole someone found, a prompt that saves tokens, a hidden cost, a skill that makes the agent better. A plan file for each kind of claim, a runner that does each run in a fresh container, a script with ten read-only checks that turns the runs into rates with honest intervals and a one-page result card, two worked examples, eight prompts and the tests, all Python 3 standard library plus POSIX sh and Docker. Written for Claude Code and Codex, through a skill file and an AGENTS.md snippet. Tested as described in section 10: trials.py and its 19 tests ran on Linux with Python 3.13, and the runner was tested against a fake docker that checks its bookkeeping. The runner has not been run against a live agent for this release, and the example numbers are made up for illustration. Each is marked where it matters.

A post says an agent gets hijacked nine times in ten by a file in your Downloads folder. Another says one line in your rules file cuts tokens by 40%. A third says a background feature is quietly eating a sixth of your bill. Some of these are true, some were true for one version, and some were one lucky run. The usual responses are to ignore them or to change your setup that afternoon. Both skip the cheap step in between: r
````

## Test record

- Date: Oct 29, 2026
- What was run: The unit tests, the runner test against a fake docker, and the unpack line with test_all.sh. Linux, Python 3.13.
- Result: 19 unit tests passed; the runner test passed; the unpack line wrote 14 files identical to the sources and test_all.sh passed.
- Not run: Real Docker or a live agent. The skill file and AGENTS.md snippet inside either harness.

Copied from this item's dated test notes. What was not run stays in.

## Worth knowing

- run_trial.sh was tested against a fake docker that checks its bookkeeping (rows, versions, marker detection, values, cleanup). It was not run against real Docker or a live agent for this release. Run one setup with RUNS=2 and read the rows before a full trial.
- The example numbers are made up with a fixed seed to show the output. They are not results from Claude Code, Codex or any agent, and the example claims are invented.
- A trial answers one claim on the setups you ran. A result that does not reproduce says nothing about other versions, settings or prompts, and a security claim that does reproduce goes to the vendor, not to a public post.
- Agent CLI flags change. The pack uses claude -p and codex exec for non-interactive runs; check --help on your version. The skill file and AGENTS.md snippet were not loaded inside either harness.
- On 2026-10-29, on Linux with Python 3.13: 19 unit tests passed (each of the ten checks fires alone on a planted fault, Wilson interval matches the textbook value, four verdicts, exit codes 0, 1 and 2); the runner test passed; the unpack line wrote 14 files byte-identical to the sources and test_all.sh passed in that folder.

## Questions

**Is it safe to test a security claim?** Only with the pack's rules: a canary that creates one empty file, a fresh container per run, one capped API key and no real accounts. trials.py refuses canary text that mentions network tools, secrets or deletes. Never reproduce a real payload.

**How many runs do I need?** At least 20 per setup for security and 10 per arm for cost and quality. Three hits in 20 runs still gives an interval of about 5% to 36%, which is why the card shows intervals.

**Does it work with Grok Bot or Hermes?** trials.py and the plan format work with any agent you can run from a command line in a container, but the pack's commands and skill file are written for Claude Code and Codex only.

**How is this different from Check the Claim?** Check the Claim is a reading method for any claim you come across. This pack runs a claim about an agent on your own setups and measures it.

## How to buy

$19 one time at https://paperchasewebb.ai/bots/marketplace/items/replicate-before-you-react-trial-kit-for-agent-claims/. Stripe checkout, then an instant download page; the download link is also in your emailed receipt.

Seller: PaperChaseWebb Inc., mgmt@paperchasewebb.com