Two-Agent Build and Test Loop for paid apps: one agent builds, another tests, and nothing passes on its own say-so
by PaperChaseWebb
A tested setup for apps that take money: one agent writes the code, a separate tester session writes the stories and tests and gives the verdicts, a small runner records every red, green and revert exit code itself, and the owner says yes to the plan, any payment or pricing work, each merge and each release. Test keys and sandboxes only. A read-only check with fifteen tests audits the finished run.
- Price
- $29
- Difficulty
- Intermediate
- Setup
- 1 hr
- Format
- .md file
two-agent-build-and-test-loop-for-paid-apps.md · 67 KB · instant download after checkout
Want this built for you instead? We can build it for you.
What you end up with
The builder can no longer delete a failing test and call it done, every pass has a recorded red, green and revert behind it, and nothing near a checkout, price or store submission moves without your yes.
What you get
- 23 sections in one .md file
- The run folder: loop.json, cards.json, approvals.csv, ledger.csv, actions.csv, events.csv and report.md
- The split: what the builder may do, what the tester may do, and what neither ever does
- The story contract: behavior, test file, test command and a payment flag on every card
- Four owner approvals: plan, payment or pricing work, each merge, and each release or store submission
- Eleven paid-app stories worth writing first, with where to test each in Stripe, StoreKit, Google Play and RevenueCat sandboxes
- card_check.py: five preflight checks before the builder starts, including live keys in the project
- run_test.py: runs each story's test itself and records red, green and a revert proof with a hash of the test file
- loop_check.py: fifteen read-only checks on the finished run
- A seeded fixture with 15 planted faults and a tiny app whose tests really run, with the expected output
- Tester and builder rules blocks, twelve prompts and setup notes for Grok Bot, Claude Code, Codex and Hermes
- A one-afternoon install plan, go-live checklist, weekly review and failure table
- A look at seven test-first skills and agent packs, with sources
Who it is for
Founders and small teams with a paid app who use coding agents every day, have seen "all tests pass" on code that did not work, and want a builder and a separate tester with proof they can check.
Works with
Solid chips: the agent is named in this listing's own tools list. Dashed chip: Plain Markdown, so it also pastes into other agents such as Grok Bot, Hermes, OpenClaw or Muse. We list those as compatible by format only, not as tested.
Tools needed
Cost to run: The pack is the only purchase. Model usage is billed by those vendors. Test mode and sandboxes cost nothing to run.
Time to result: Set up and tested in about an hour; the first watched run takes about 90 minutes more
How it works
Test the checks
Save the three scripts from the file, build the fixture and run loop_check.py and card_check.py on both folders.
Expected: loop_check.py: 15 findings on seeded, 0 on fixed. card_check.py: 3, then 0.
Write loop.json
Make a runs/ folder outside your repo, copy loop.json from section 5, and put in your project path, test paths, two session names and your name as owner.
Expected: card_check.py runs on the folder and C1 does not appear.
Lock the two sessions
Map each role's tools to your harnesses and paste the tester and builder rules blocks. Try editing a source file from the tester and a test file from the builder.
Expected: Both edits are refused, and no session can push, merge, release or reach a live key.
Put test keys only in the project
Use test mode keys and sandbox accounts for every payment provider in the app. Keep live keys where neither agent can read them.
Expected: card_check.py shows no C5 finding.
Run two paid-app stories, watched
Pick two stories from section 8, run the twelve prompts in order, run run_test.py by hand for red, green and revert, and give each yes yourself.
Expected: loop_check.py prints 0 findings and the report matches the ledger.
Sample prompt from the pack
Turn this request and your notes from staging into cards.json for run {run}: one story per behavior a buyer would notice, each with one test file under {test_paths}, a command that runs only that test, and a payment flag.
Start with the paid-app stories in section 8 that apply. Then run card_check.py and fix every finding.Prompt 2 of 12. Runs in the tester session after the tester rules block in section 14.
Here is your card and the failing test output. Make the test pass by changing source only. Log your build and claim rows.
Report the files you changed. Do not touch anything under {test_paths} and do not give a verdict.Prompt 8 of 12. Runs in the builder session after the builder rules block.
Free preview of the file
# Two-Agent Build and Test Loop for paid apps: one agent builds, another tests, and nothing passes on its own say-so Version 1.0, 2026-10-28. By PaperChaseWebb. Works with Grok Bot, Claude Code, Codex and Hermes. Plain Markdown and three Python 3 scripts (standard library only), tested on a seeded fixture with fifteen planted faults. This pack sets up one way of working on an app that takes money: one agent writes the code, a second agent in a separate session writes the tests and gives the verdicts, a small script runs every test and records the exit code, and the owner says yes at four points: the plan, anything that touches payments or prices, each merge, and each release or store submission. The whole loop runs on test keys and sandboxes. No live key, no real charge. It gives you the run folder, the two role files, the story cards the tester writes, the four approval steps, a preflight, a test runner that records red, green and a revert proof, a fifteen-check read-only audit of the finished run, a seeded fixture, rules blocks for both agents, twelve prompts, setup for four harnesses, a one-afternoon install and a failure table. How to split one job between a strong planner and cheap workers is in Planner Over Cheap Models (pp-planner-over-cheap-models). Moving work between harnesses mid-week is in Cross-Harness Relay (pp-cross-harness-relay). This pack is the part those leave open: keeping the builder from grading its own homework when the homework is a checkout. ##
Worth knowing
- It needs a test command per story. Work you cannot check with a command (copy, visual taste, open research) does not fit this method.
- Test mode and sandboxes only. The pack never asks for live keys or real charges; adding live keys at release is the owner's step, outside the loop.
- The permissions in loop.json are a record. The real lock is your harness settings, so test both refusals before the first run.
- Store review rules and sandbox behavior change. Check Apple, Google, Stripe and RevenueCat docs before each release.
- On 2026-10-28, loop_check.py reported the fifteen expected findings on the seeded fixture, then 0 on the fixed one, with file hashes unchanged and exit 2 on a missing folder; card_check.py gave 3 then 0; run_test.py recorded real red, green and revert exit codes on the fixture app. Run the tests on your own machine first.
Questions
Why two agents instead of one agent that writes tests first?
One agent that writes the tests and the code can also change the tests to pass, and it shares its own blind spots. A separate tester that cannot edit source, plus a runner that records the exit codes, makes that move visible in the ledger.
Does it need two different AI products?
No. Two separate sessions are the minimum. A different harness or model for the tester is better, and the pack has setup notes for Grok Bot, Claude Code, Codex and Hermes in either role.
Will it ever touch real payments?
No. The loop runs on test keys and sandboxes only. card_check.py and loop_check.py both flag a live key in the project and any payment event outside test mode or a sandbox.
What is the revert proof?
After green, the same test runs on a copy of the project with the change taken out. It must fail again. If it passes, the test does not cover the change. run_test.py records a hash of the test file so a test softened in between shows up.
Details
Version 1.0 · Updated Oct 28, 2026 · Markdown version of this page
License: Use on your own projects and on installs you do for clients. Do not resell or republish the pack itself. Download terms
Related
The builder and tester contract and a red-then-green check (free sample)
From Two-Agent Build and Test Loop for paid apps: the six rules that keep a coding agent from grading its own work (two sessions, the tester never edits code, the builder never touches tests or says passed, red first, a script runs the tests, sandbox only with owner yeses on paid apps), a story card, and a read-only Python check on your run's ledger, with a worked example and its expected output.
Planner Over Cheap Models: one strong planner, cheap workers, and a check that keeps both honest
A tested setup for splitting coding work between one strong model that plans and reviews and cheap or local models that write the code. Every task card carries its own check, a logger runs that check instead of trusting the worker, the owner says yes to the plan, any escalation and every merge, and a read-only check with fourteen tests audits the finished run. Includes a worker bake-off so you pick the cheap model on your own tasks.
Cross-Harness Relay: Grok Bot, Claude Code, Codex and Hermes on one job
A bot template and teaching guide for running more than one agent harness on the same work. Six seat cards (front desk, planner, builder, reviewer, tester, night shift), a RELAY.md seat map, a relay packet for every handoff, four loop shapes, routine-or-judgment routing with a fallback when limits run tight, a preflight check, the front desk and night shift split with a proof footer, shared memory rules, eight relay drills, a rules block and 14 prompts.
Don't Get Sued: a pre-launch legal-risk gate for AI-built apps and agents
A read-only agent gate for the fourteen legal risks that hit small builders most: marketing texts and email, trackers before consent, accessibility, kids' data, biometrics, plain-text personal data, AI disclosure, AI claims you cannot prove, takedowns, policies that do not match the product, renewals and refunds, licenses, and agents that act for users. Evidence rule, gate register, seeded test app, 12 prompts, counsel packet. Not legal advice.
Run a pool of coding agents that only merge fixes proven red then green
A maintainer ran a small office of Claude Code workers against 13 Rust repos for five days and got about 170 pull requests merged, then open sourced the tool as ApolloOffice.
Have your coding agent run a release gate before App Store submission
An indie developer who builds everything with agents shares the pre-submit checks his agent runs, what they caught, and why Apple still rejected his Codex-built puzzle game.
Make your agent play the game before it says done
One prompt, two agents: Claude Code built a neon racing game far faster, while Grok Bot drove the race with real controls and caught reversed steering a script never touched.
Spec-First Build Docs: PRD, design system, architecture note, agents guide and readiness check
A Markdown pack of the four documents to write before an agent builds anything, sized to the stakes: a PRD an agent can build from without inventing scope, the smallest design system that holds, a one-page architecture note with reversal costs, an agents guide with a project tree for Claude Code, Codex and Cursor, a weekly staleness pass and a nine-area readiness check.
Ask-first rules for AI agents: hard nos, approval cards and a test set
Paste-in rules that make an agent ask before it sends, spends, posts or deletes, define what a real yes is, and stop retries after a no, plus 12 test scenarios to check it.
Vibe Coding with AI Agents: method, prompts, skills and checklists
A Markdown playbook for building real software with Claude Code, Codex or Claude: a seven-step method, project file templates, 18 prompts, five agent skills, release and hardware checklists, cost notes and 24 project ideas.
Game Jam Kit: a 48-hour Godot 4 plan and a two-day Unity PRD that a coding agent can follow
A tested jam method for working with a coding agent: seven milestones per engine, each with a first version, a first test and a written list of what to leave out, a JAM.md the agent cannot change without your new yes, three owner approvals (scope lock, art and audio licenses, upload), a scope-cut checklist and a fifteen-check read-only jam_check.py. Includes a playable Godot 4 starter that tests itself headless, and a tested proxy for games that call an LLM while people play.
Agent Status Hardware: one light and one e-ink display for every agent session, driven by hooks
Wire the hooks Claude Code, Codex and Hermes already have, plus a rules block for Grok Bot, to one small bridge on your machine. It turns agent events into one state for all sessions and drives a terminal, a USB busylight, a blink(1), an ESP32 LED, a 2.13 inch e-ink panel or a Home Assistant light. Waiting for you is a red blink nothing can cover, a session with no news turns amber, and the light never approves anything.
Release Watch: a per-change monitoring plan your agent writes from the diff, run by a fixed script
After a change goes live, your agent reads the diff and writes a small watch plan for that change: which signals it could move, against which baseline, by how much, for how long. A fixed script runs the plan every five minutes, so no agent sits in a loop. On a trip, a second agent triages read-only and proposes one next step. Rollbacks, flag flips and alert changes stay with you.
Shrink the Attack Surface: patch with proof
Your agent lists everything your app exposes, read-only: routes, open ports, public buckets, secrets in the repo, unused dependencies, debug endpoints, admin paths, CORS, API key scopes and its own tool permissions. You decide keep, remove or lock for each. Every fix gets a proof that fails before the change and passes after it, and the proof stays as a regression test. Red-team drills then check that the agent still asks before risky steps, stops retrying broken tools and never charges or emails twice.