# Replicate Before You React: a claim card, five rules and a rate script (free sample)

Version 1.0, 2026-10-29. By PaperChaseWebb. For developers and small teams who keep getting forwarded posts about coding agents (a security hole, a token saver, a hidden cost) and want to test the claim before changing anything. One Markdown file and one Python 3 script, standard library only. The script was tested on three inputs and a bad input (exit 2). It was not run inside any agent harness or against a live agent.

Posts about agents arrive daily: one says a file in your Downloads folder hijacks the agent nine times in ten, another says one rule cuts tokens by 40%. Some hold up, some held for one version, and some were one lucky run. The step between ignoring them and acting on them is cheap: run the claim yourself, safely, enough times to know. This sample gives you the card to fill in, the rules that keep it safe, and a script that tells you how much a handful of runs can actually say.

## Install in one line

Save this page as FREE-SAMPLE.md, then:

```
python3 -c "import re;t=open('FREE-SAMPLE.md').read();open('rate_mini.py','w').write(re.search(r'~~~python\n(.*?)~~~',t,re.S).group(1))"
python3 rate_mini.py 3 20 0.9
```

## The claim card

Fill this in before any run. If you cannot fill a line, write "not stated".

| Line | Your answer |
|---|---|
| The claim, in your own words, one sentence | |
| Source link and date | |
| Agents and versions the source tested | |
| Settings or modes the source used | |
| The claimed figure (a rate, or a percent change) | |
| How many runs the source reported | |
| The setups you will test (agent, version, mode) | |
| Runs per setup (20 or more for security, 10 or more for cost and quality) | |
| How one run is scored (marker file exists, tokens per task, hidden tests passed) | |
| What you will not test | |

## Five rules

1. **Canary, never payload.** To test a security claim, the planted instruction only creates one empty file at a marker path. It never fetches, sends, deletes or reads secrets.
2. **Throwaway environment, every run.** A fresh container or VM per run, with a copy of the test files. Never your own machine, home folder or real accounts.
3. **One capped key.** The only credential inside is the agent's own API key, made for the test with a spend cap.
4. **A control for cost and quality.** Run the same tasks without the change. Judge quality with tests the agent cannot see.
5. **Say what you ran.** Report "reproduced" or "not reproduced on the setups we ran", with versions and the interval. Never "the post was fake".

## What a few runs can tell you

| Hits | Runs | 95% interval |
|---|---|---|
| 2 | 3 | 21% to 94% |
| 3 | 20 | 5% to 36% |
| 18 | 20 | 70% to 97% |
| 0 | 20 | 0% to 16% |

Zero hits in 20 runs still leaves room for a rate up to about one in six. That is why the full pack reports intervals and lists what was not tested.

## The script

~~~python
#!/usr/bin/env python3
"""rate_mini.py: how sure can you be about a hit rate? Free sample of Replicate Before You React (PaperChaseWebb).
usage: python3 rate_mini.py HITS RUNS [CLAIMED_RATE]     e.g. python3 rate_mini.py 3 20 0.9
Prints the rate, a 95% Wilson interval and how the claimed rate sits against it. Standard library only."""
import math, sys

def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d; h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return p, max(0.0, c - h), min(1.0, c + h)

def main(a):
    try:
        k, n = int(a[1]), int(a[2]); claim = float(a[3]) if len(a) > 3 else None
        if n <= 0 or not 0 <= k <= n or (claim is not None and not 0 <= claim <= 1): raise ValueError
    except (IndexError, ValueError):
        print(__doc__); return 2
    p, lo, hi = wilson(k, n)
    print(f"{k} of {n} = {p:.0%}, 95% interval {lo:.0%} to {hi:.0%}")
    if n < 20: print("fewer than 20 runs: the interval is wide, add runs before you decide")
    if claim is not None:
        v = "consistent with the claim" if lo <= claim <= hi else ("below the claim" if hi < claim else "above the claim")
        print(f"claimed {claim:.0%}: {v}")
    return 0

if __name__ == "__main__": sys.exit(main(sys.argv))
~~~

## The full pack

Replicate Before You React ($19) adds plan templates for security, cost and quality claims, a runner that does each run in a fresh container with one capped key, trials.py with ten read-only checks, intervals for rates and cost changes and a one-page result card, a skill file for Claude Code and Codex, two worked examples, eight prompts and the tests. The method draws on public write-ups by Chris Johnson (@ComputeGhost), Flamerix AI and @mano_sec, credited and linked in the full pack.