# Pilot Scorecard **Agree what a passing pilot looks like before it starts. One page of terms the client owner signs, locked pass lines, and a scored report at the end that says pass or fail without an argument.** Version 1.0, 2026-11-03. By PaperChaseWebb Inc. Free. Works with Grok Bot, Claude Code, Codex and Hermes, or by hand. Python 3.8 or newer, no installs. ## The problem Most agent pilots end in a debate. The provider points at the hours saved, the client remembers the one bad reply, and nobody wrote down what "worked" meant. Worse, targets drift mid-pilot, so even a good result looks negotiated. The fix is to write the pass line first: a few numbers, where each comes from, and what happens after a pass or a fail. The client owner signs it. Then the numbers decide. ## What you get out of it - One terms file: one workflow, 30 days at most, up to 5 measures with today's number, a target and the source, plus data limits, an off switch, the fee and refund rule, and the next step either way. - A one-page terms sheet the client owner reads and signs at their own terminal. - Locked pass lines: if the terms change after sign-off, scoring is refused until the client owner signs again. - A separate, default-no case-study question, so consent is never bundled into the deal. - A report at the end: met or not met per measure, pass or fail, and the step both sides agreed to in advance. ## Who it is for Providers and small agencies who offer a trial before a monthly plan. Owners who want to try an agent workflow and know in advance what would make it worth keeping. Not a contract. It records what both sides agreed and scores it. Put the commercial terms in your own agreement. ## The terms | Field | What to write | | --- | --- | | `workflow` | one workflow only, in plain words | | `start`, `end` | dates, 30 days or fewer | | `measures` | 1 to 5, each with `name`, `unit`, `baseline`, `target`, `direction` (`up` or `down`) and `source` | | `pass_rule` | `"all"`, or how many measures must reach target | | `client_provides` | access, a contact, data: the list the pilot depends on | | `data_limits` | what is never read, stored or moved | | `off_switch` | the exact step that stops the agent, usable by the client | | `fee` | `amount_usd` (0 is fine) and a `refund_rule` if it is paid | | `after_pass`, `after_fail` | what happens next, written now | | `case_study` | `ask` and the `scope` of what would be shared; consent itself is only recorded at sign-off | ## What the checker catches | Code | Means | | --- | --- | | P01 | client, provider, client owner or workflow missing | | P02 | dates missing, reversed, or longer than 30 days | | P03 | no measures or more than 5; a measure with no unit or source; a target that is not better than the baseline | | P04 | a pass rule that asks for more measures than exist | | P05 | nothing listed for the client to provide; no data limits; no off switch | | P06 | a paid pilot with no refund rule written before it starts | | P07 | no after-pass or after-fail step | | P08 | consent written into the file instead of given at sign-off, or a case study asked with no scope | | S01 to S04 | at scoring: a measure with no result, a result that is not a number, a result for a measure not in the signed terms, or a date before the pilot started | Sign-off is refused inside an agent session, without a terminal, from anyone but the client owner, and if the terms changed after the sheet was made. Scoring is refused without a sign-off or if the terms changed after it. ## Set up (15 minutes) 1. Copy `extract_from_pack.py` from the appendix by hand, then run `python3 extract_from_pack.py pilot-scorecard-agree-the-pass-line-before-you-start.md pilot/` (the file you downloaded) to unpack the rest. 2. In `pilot/`, run the tests: `python3 -m unittest discover -s tests -p 'test_*.py'`. 3. Try the example, a made-up dental practice and its automation provider: `python3 pilot.py sheet example/pilot.json out/` and read out/pilot-terms.md. 4. At your own terminal: `python3 pilot.py approve example/pilot.json out/` (type Maya Lindqvist, then y, then answer the case-study question). Then `python3 pilot.py score example/pilot.json example/results.json out/`. You should see PASS: 2 of 3 measures met, 2 needed. 5. Change one target in example/pilot.json and score again. It is refused: the pass lines were locked at sign-off. 6. Copy templates/pilot.json and templates/results.json for your own pilot. Give your agent AGENTS.md. Claude Code and Codex read it from the folder (CLAUDE.md is the same text); for Grok Bot use agents/grok-standing-order.md, and for Hermes load agents/SKILL.md. ## Run it once before you rely on it Write terms for a pilot you already ran and score it with the real numbers. If the result surprises either side, the measures were wrong, and that is worth knowing before the next one. Stop and fix the setup if a measure has no source you can actually pull, if your agent signed off or edited the terms after sign-off, or if anything was shared without recorded consent. That is the kill rule. ## Limits - It scores what you give it. A weak source makes a weak result; pick sources both sides can see. - A short pilot is a small sample. A pass says the workflow met the agreed lines in that window, nothing more. - Baselines should be measured, not remembered. If you do not have one, spend the first week measuring it and start the pilot after. - Not a contract, and not legal advice. The refund rule here records what you agreed; your own agreement governs. ## Want this run for you A pilot with a written pass line is how we start new client work. If you would rather have the pilot set up, measured and reported for you, with every step drafted for your yes, that is Managed Ops: our agents work inside the tools you already use. Managed Ops starts at $5,000/mo with $2,500 onboarding and a 3-month minimum. Growth Fleet, starting at $2,500 setup + $750/mo, covers a smaller set of agents. Book a call on the paperchasewebb.ai homepage. Every price is a starting price. ## License Free for personal and commercial use inside your own business. Do not resell or republish the kit. ## Appendix: the files #### file: extract_from_pack.py ~~~~python #!/usr/bin/env python3 """Unpack every '#### file: PATH' block from pack.md (or FREE-SAMPLE.md) into a folder.""" import os, re, sys src, dst = sys.argv[1], sys.argv[2] t = open(src, encoding="utf-8").read() n = 0 for m in re.finditer(r"^#### file: (\S+)\n~~~~[a-z]*\n(.*?)\n~~~~$", t, re.S | re.M): p = os.path.join(dst, m.group(1)) os.makedirs(os.path.dirname(p) or ".", exist_ok=True) open(p, "w", encoding="utf-8").write(m.group(2) + "\n") n += 1 print(n, "files") ~~~~ #### file: pilot.py ~~~~python #!/usr/bin/env python3 """pilot.py: agree what a passing agent pilot looks like before it starts, then score it against those exact lines. python3 pilot.py check PILOT.json check the pilot terms python3 pilot.py sheet PILOT.json OUT/ write OUT/pilot-terms.md (one page for the client) and OUT/pending.json python3 pilot.py approve PILOT.json OUT/ client owner only, at their own terminal: sign off the pass lines python3 pilot.py score PILOT.json RESULTS.json OUT/ [--today D] write OUT/pilot-report.md, pass or fail per measure PILOT.json holds one workflow, the dates (30 days at most), up to 5 measures with a baseline, a target and where the number comes from, what the client provides, data limits, the off switch, the fee and refund rule, and what happens after a pass or a fail. Scoring is refused if the terms changed after sign-off. Case-study consent is asked separately at sign-off and defaults to no. Exit codes: 0 ok, 1 errors or a failed pilot, 2 usage, 3 refused (agent session or no terminal), 4 refused (missing or stale sign-off), 5 owner did not confirm. Python 3.8 or newer, standard library only. """ import hashlib, json, os, sys from datetime import date, datetime AGENT_ENV = ["CLAUDECODE", "CLAUDE_CODE_ENTRYPOINT", "CODEX_SANDBOX", "CODEX_SANDBOX_NETWORK_DISABLED", "CODEX_MANAGED_BY_NPM", "HERMES_AGENT", "HERMES_SESSION_ID", "GROK_BOT", "CURSOR_AGENT"] MAX_DAYS, MAX_MEASURES = 31, 5 def jload(p): with open(p, encoding="utf-8") as f: return json.load(f) def wtext(p, t): with open(p, "w", encoding="utf-8") as f: f.write(t) def to_date(s): try: return datetime.strptime(str(s), "%Y-%m-%d").date() except ValueError: return None def num(x): return isinstance(x, (int, float)) and not isinstance(x, bool) def fmt(x): return ("%d" % x) if float(x).is_integer() else ("%.2f" % x).rstrip("0").rstrip(".") def state_hash(p): with open(p, "rb") as f: return hashlib.sha256(f.read()).hexdigest() def check(p): e = [] for k in ("client", "provider", "client_owner", "workflow"): if not str(p.get(k) or "").strip(): e.append(("P01", "%s is missing" % k)) s, en = to_date(p.get("start")), to_date(p.get("end")) if not (s and en): e.append(("P02", "start and end must be YYYY-MM-DD")) elif en <= s: e.append(("P02", "end must be after start")) elif (en - s).days + 1 > MAX_DAYS: e.append(("P02", "pilot runs %d days; keep it to 30 or fewer" % ((en - s).days + 1))) ms = p.get("measures") if not isinstance(ms, list) or not 1 <= len(ms) <= MAX_MEASURES: e.append(("P03", "list 1 to %d measures" % MAX_MEASURES)) ms = ms if isinstance(ms, list) else [] seen = set() for i, m in enumerate(ms, 1): nm = str(m.get("name") or "").strip() tag = nm or "measure %d" % i if not nm or nm.lower() in seen: e.append(("P03", "%s: needs a unique name" % tag)) seen.add(nm.lower()) if not str(m.get("unit") or "").strip() or not str(m.get("source") or "").strip(): e.append(("P03", "%s: needs a unit and a source (where the number comes from)" % tag)) if m.get("direction") not in ("up", "down"): e.append(("P03", "%s: direction must be up or down" % tag)) elif not (num(m.get("baseline")) and num(m.get("target"))): e.append(("P03", "%s: baseline and target must be numbers" % tag)) elif (m["target"] - m["baseline"]) * (1 if m["direction"] == "up" else -1) <= 0: e.append(("P03", "%s: the target is not better than the baseline" % tag)) pr = p.get("pass_rule", "all") if not (pr == "all" or (isinstance(pr, int) and not isinstance(pr, bool) and 1 <= pr <= max(len(ms), 1))): e.append(("P04", "pass_rule must be \"all\" or how many measures must pass (1 to %d)" % max(len(ms), 1))) cp = p.get("client_provides") if not isinstance(cp, list) or not [x for x in cp if str(x).strip()]: e.append(("P05", "client_provides must list what the client gives (access, a contact, data)")) for k in ("data_limits", "off_switch"): if not str(p.get(k) or "").strip(): e.append(("P05", "%s is missing" % k)) fee = p.get("fee") or {} if not num(fee.get("amount_usd")) or fee["amount_usd"] < 0: e.append(("P06", "fee.amount_usd must be 0 or more")) elif fee["amount_usd"] > 0 and not str(fee.get("refund_rule") or "").strip(): e.append(("P06", "a paid pilot needs a refund_rule written before it starts")) for k in ("after_pass", "after_fail"): if not str(p.get(k) or "").strip(): e.append(("P07", "%s is missing" % k)) cs = p.get("case_study") or {} if "allowed" in cs or "consent" in cs: e.append(("P08", "case-study consent is recorded only by the client owner at sign-off; remove it from the file")) if cs.get("ask") and not str(cs.get("scope") or "").strip(): e.append(("P08", "case_study.scope must say what would be shared")) return e def pass_need(p): pr = p.get("pass_rule", "all") return len(p["measures"]) if pr == "all" else pr def make_sheet(path, out): p = jload(path) s, en = to_date(p["start"]), to_date(p["end"]) need = pass_need(p) L = ["# Pilot terms: %s" % p["workflow"], "", "%s and %s. %s to %s (%d days). Client owner: %s." % (p["client"], p["provider"], p["start"], p["end"], (en - s).days + 1, p["client_owner"]), "", "## What counts as a pass", "", "The pilot passes if %s of these %d measures reach their target by %s." % ("all" if need == len(p["measures"]) else need, len(p["measures"]), p["end"]), "", "| Measure | Today | Target | Where the number comes from |", "| --- | --- | --- | --- |"] for m in p["measures"]: L.append("| %s (%s, %s is better) | %s | %s %s | %s |" % (m["name"], m["unit"], "higher" if m["direction"] == "up" else "lower", fmt(m["baseline"]), "at least" if m["direction"] == "up" else "at most", fmt(m["target"]), m["source"])) fee = p["fee"] L += ["", "## Ground rules", "", "- The client provides: %s." % "; ".join(str(x) for x in p["client_provides"] if str(x).strip()), "- Data limits: %s" % p["data_limits"], "- Off switch: %s" % p["off_switch"], "- Fee: %s" % ("none" if fee["amount_usd"] == 0 else "$%s. %s" % (fmt(fee["amount_usd"]), fee["refund_rule"])), "- After a pass: %s" % p["after_pass"], "- After a fail: %s" % p["after_fail"]] cs = p.get("case_study") or {} if cs.get("ask"): L += ["- Case study: you will be asked separately. Saying no changes nothing else. What would be shared: %s." % cs["scope"].rstrip(".")] else: L += ["- Case study: not requested. Nothing about this pilot is published."] L += ["", "## Sign off", "", "Once you sign off, these lines are locked. If anything above changes, the pilot needs a new sign-off before it can be scored.", "", "At your own terminal: `python3 pilot.py approve pilot.json out/`", ""] os.makedirs(out, exist_ok=True) wtext(os.path.join(out, "pilot-terms.md"), "\n".join(L)) wtext(os.path.join(out, "pending.json"), json.dumps({"state": state_hash(path)}, indent=1) + "\n") def approve(path, out, ask=input, env=None, tty=None): env = os.environ if env is None else env tty = sys.stdin.isatty() if tty is None else tty for k in AGENT_ENV: if env.get(k): print("refused: agent session detected (%s); the client owner signs off" % k) return 3 if not tty: print("refused: no terminal; run this yourself") return 3 pp = os.path.join(out, "pending.json") if not os.path.exists(pp): print("refused: no terms sheet; run pilot.py sheet") return 4 if jload(pp)["state"] != state_hash(path): print("refused: the terms changed since the sheet was made; run pilot.py sheet again") return 4 p = jload(path) if check(p): print("refused: the terms do not pass check") return 1 if ask("Your name, as client_owner in the terms: ").strip() != p["client_owner"]: print("refused: that is not the client owner in the terms") return 5 if ask("Agree these pass lines and ground rules? y/n: ").strip().lower() != "y": print("not signed off") return 5 consent = False if (p.get("case_study") or {}).get("ask"): consent = ask("Separately: may %s publish a case study limited to: %s? y/N: " % (p["provider"], p["case_study"]["scope"])).strip().lower() == "y" rec = {"state": state_hash(path), "signed_by": p["client_owner"], "signed_at": datetime.now().strftime("%Y-%m-%dT%H:%M"), "case_study": consent} wtext(os.path.join(out, "signoff.json"), json.dumps(rec, indent=1) + "\n") print("signed off. Case study: %s. The pass lines are now locked." % ("yes, within the written scope" if consent else "no")) return 0 def check_results(p, r): e, names = [], {m["name"] for m in p["measures"]} got = r.get("results") if isinstance(r.get("results"), dict) else {} for n in names: if n not in got: e.append(("S01", "%s: no result" % n)) elif not num(got[n]): e.append(("S02", "%s: result must be a number" % n)) for n in got: if n not in names: e.append(("S03", "%s: not a measure in the signed terms" % n)) d = to_date(r.get("measured_on")) if not d or d < to_date(p["start"]): e.append(("S04", "measured_on must be a date on or after the pilot start")) return e def score(path, rpath, out, today=None): sp = os.path.join(out, "signoff.json") if not os.path.exists(sp): print("refused: no sign-off; the client owner runs pilot.py approve before the pilot starts") return 4 so = jload(sp) if so["state"] != state_hash(path): print("refused: the terms changed after sign-off; the old pass lines stand until the client owner signs new ones") return 4 p, r = jload(path), jload(rpath) errs = check_results(p, r) if errs: for c, m in errs: print("%s %s" % (c, m)) return 1 today = today or date.today() measured = to_date(r["measured_on"]) interim = measured < to_date(p["end"]) rows, passed = [], 0 for m in p["measures"]: v = r["results"][m["name"]] ok = v >= m["target"] if m["direction"] == "up" else v <= m["target"] passed += ok rows.append("| %s | %s | %s | %s | %s |" % (m["name"], fmt(m["baseline"]), fmt(m["target"]), fmt(v), "met" if ok else "not met")) need = pass_need(p) verdict = "PASS" if passed >= need else "FAIL" head = "INTERIM, on track" if interim and verdict == "PASS" else ("INTERIM, not yet passing" if interim else verdict) L = ["# Pilot result: %s" % p["workflow"], "", "%s and %s. Terms signed by %s on %s. Measured on %s. Report made %s." % ( p["client"], p["provider"], so["signed_by"], so["signed_at"][:10], r["measured_on"], today.isoformat()), "", "## %s: %d of %d measures met, %d needed" % (head, passed, len(p["measures"]), need), "", "| Measure | Before | Target | Result | |", "| --- | --- | --- | --- | --- |"] + rows + [""] if str(r.get("notes") or "").strip(): L += ["Notes: %s" % r["notes"].strip(), ""] L += ["## Next", ""] if interim: L.append("The pilot ends %s. Score it again with the final numbers then. Nothing below applies yet." % p["end"]) elif verdict == "PASS": L.append(p["after_pass"]) else: L.append(p["after_fail"]) if p["fee"]["amount_usd"] > 0: L.append("") L.append("Refund rule, as signed: %s" % p["fee"]["refund_rule"]) L += ["", "## Case study", "", ("The client owner agreed to a case study limited to: %s" % p["case_study"]["scope"]) if so.get("case_study") else "No consent on record. Nothing about this pilot is published, quoted or named.", ""] os.makedirs(out, exist_ok=True) wtext(os.path.join(out, "pilot-report.md"), "\n".join(L)) print("wrote %s/pilot-report.md: %s (%d of %d met, %d needed)" % (out, head, passed, len(p["measures"]), need)) return 0 if verdict == "PASS" else 1 def main(argv): today = None if "--today" in argv: i = argv.index("--today") today = to_date(argv[i + 1]) if i + 1 < len(argv) else None if not today: print("--today needs YYYY-MM-DD") return 2 argv = argv[:i] + argv[i + 2:] if not argv or argv[0] not in ("check", "sheet", "approve", "score") or len(argv) < 2: print(__doc__) return 2 cmd, a = argv[0], argv[1:] errs = check(jload(a[0])) if errs or cmd == "check": for c, m in errs: print("%s %s" % (c, m)) print("terms %s: %d error(s)" % ("OK" if not errs else "NOT OK", len(errs))) return 1 if errs else 0 if cmd == "sheet" and len(a) == 2: make_sheet(a[0], a[1]) print("wrote %s/pilot-terms.md; the client owner signs off with pilot.py approve" % a[1]) return 0 if cmd == "approve" and len(a) == 2: return approve(a[0], a[1]) if cmd == "score" and len(a) == 3: return score(a[0], a[1], a[2], today) print(__doc__) return 2 if __name__ == "__main__": sys.exit(main(sys.argv[1:])) ~~~~ #### file: AGENTS.md ~~~~markdown # AGENTS.md: pilot scorecard You help run a short agent pilot with the pass line agreed before it starts. You draft the terms, collect the numbers and score them. The client owner signs off. ## You may - Draft pilot.json with the provider: one workflow, 30 days at most, up to 5 measures with today's number, a target and where each number comes from. - Run `python3 pilot.py check pilot.json` and `sheet pilot.json out/`, then send the client owner out/pilot-terms.md. - During the pilot, collect each measure from the source named in the terms. Copy the number; never estimate one. - Fill results.json and run `python3 pilot.py score pilot.json results.json out/`. Send out/pilot-report.md to both sides. ## You never - Run `pilot.py approve` or write signoff.json. The client owner signs off at their own terminal. - Change pilot.json after sign-off. If the terms need to change, draft the change and the client owner signs again; until then the old lines stand. - Pick a source that flatters the result, drop a measure, or round a number toward the target. - Publish, quote or name the client unless signoff.json says case_study is true, and then only within the written scope. - Start the after-pass step (a paid plan, a new contract) without the client owner's yes in writing. ~~~~ #### file: CLAUDE.md ~~~~markdown # AGENTS.md: pilot scorecard You help run a short agent pilot with the pass line agreed before it starts. You draft the terms, collect the numbers and score them. The client owner signs off. ## You may - Draft pilot.json with the provider: one workflow, 30 days at most, up to 5 measures with today's number, a target and where each number comes from. - Run `python3 pilot.py check pilot.json` and `sheet pilot.json out/`, then send the client owner out/pilot-terms.md. - During the pilot, collect each measure from the source named in the terms. Copy the number; never estimate one. - Fill results.json and run `python3 pilot.py score pilot.json results.json out/`. Send out/pilot-report.md to both sides. ## You never - Run `pilot.py approve` or write signoff.json. The client owner signs off at their own terminal. - Change pilot.json after sign-off. If the terms need to change, draft the change and the client owner signs again; until then the old lines stand. - Pick a source that flatters the result, drop a measure, or round a number toward the target. - Publish, quote or name the client unless signoff.json says case_study is true, and then only within the written scope. - Start the after-pass step (a paid plan, a new contract) without the client owner's yes in writing. ~~~~ #### file: agents/grok-standing-order.md ~~~~markdown # Grok Bot standing order: pilot scorecard For each pilot in progress: 1. Before day one, confirm out/signoff.json exists. If not, the pilot has not started; remind the provider, and do nothing else. 2. Every Friday, collect each measure from the source named in pilot.json, fill results.json with today's date, and run `python3 pilot.py score pilot.json results.json out/`. Send the provider the interim line. 3. On the end date, collect the final numbers, score again and send out/pilot-report.md to the provider and the client owner. 4. Never change pilot.json after sign-off, never publish anything without case_study true in signoff.json, and never start the after-pass step without the client owner's written yes. ~~~~ #### file: agents/SKILL.md ~~~~markdown --- name: pilot-scorecard description: Set up a short agent pilot with written pass lines the client owner signs before it starts, then score the results against those exact lines and write the next step. Use when a client wants to try an agent workflow before committing to a plan. --- # Pilot scorecard 1. Terms: pilot.json holds one workflow, start and end (30 days at most), up to 5 measures (baseline, target, up or down, source), the pass rule, what the client provides, data limits, the off switch, fee and refund rule, and after-pass and after-fail steps. `python3 pilot.py check pilot.json`. 2. Sheet: `python3 pilot.py sheet pilot.json out/`. Send out/pilot-terms.md. 3. Sign-off: the client owner only, at their own terminal: `python3 pilot.py approve pilot.json out/`. Case-study consent is a separate question and defaults to no. 4. Run the pilot. Collect each number from the named source. 5. Score: fill results.json, then `python3 pilot.py score pilot.json results.json out/`. Refused if the terms changed after sign-off. Hard rules: never sign off for the client, never move a target after sign-off, never publish without recorded consent, never start the after-pass step without a written yes. ~~~~ #### file: templates/pilot.json ~~~~json { "client": "", "provider": "", "client_owner": "", "workflow": "", "start": "YYYY-MM-DD", "end": "YYYY-MM-DD", "measures": [ {"name": "", "unit": "", "baseline": 0, "target": 0, "direction": "down", "source": ""} ], "pass_rule": "all", "client_provides": [""], "data_limits": "", "off_switch": "", "fee": {"amount_usd": 0, "refund_rule": ""}, "after_pass": "", "after_fail": "", "case_study": {"ask": false, "scope": ""} } ~~~~ #### file: templates/results.json ~~~~json { "measured_on": "YYYY-MM-DD", "results": {}, "notes": "" } ~~~~ #### file: example/pilot.json ~~~~json { "client": "Harbor Street Dental", "provider": "Lanternfish Studio", "client_owner": "Maya Lindqvist", "workflow": "Replies to new patient inquiries from the website form and email", "start": "2026-11-02", "end": "2026-11-29", "measures": [ {"name": "First reply time", "unit": "minutes, weekday median", "baseline": 240, "target": 30, "direction": "down", "source": "inbox timestamps, weekdays 8am to 6pm"}, {"name": "Consults booked from inquiries", "unit": "per week", "baseline": 6, "target": 9, "direction": "up", "source": "booking calendar, tagged 'web inquiry'"}, {"name": "Front desk time on inquiries", "unit": "minutes per day", "baseline": 75, "target": 30, "direction": "down", "source": "front desk time log"} ], "pass_rule": 2, "client_provides": ["read and draft access to the inquiries inbox", "the booking calendar link", "one front desk contact for questions"], "data_limits": "No treatment or health details are read or stored. Drafts use name, contact details and the requested visit type only. Nothing leaves the practice's own email and calendar.", "off_switch": "Front desk turns off the 'inquiry drafts' rule in the inbox settings; replies go back to manual the same minute.", "fee": {"amount_usd": 750, "refund_rule": "If the pilot fails, half the fee is refunded within 14 days of the end date."}, "after_pass": "Move to a monthly plan at the quoted rate, starting the 1st of next month, only after the client owner says yes in writing.", "after_fail": "Switch the drafts off, hand back all access the same week, and send a one-page note on what did not work.", "case_study": {"ask": true, "scope": "the three measures above as percentages, with the practice unnamed"} } ~~~~ #### file: example/results.json ~~~~json { "measured_on": "2026-11-29", "results": {"First reply time": 22, "Consults booked from inquiries": 10, "Front desk time on inquiries": 34}, "notes": "Week one ran at half volume while the front desk tested the off switch." } ~~~~ #### file: tests/test_pilot.py ~~~~python """Tests for pilot.py. Run from the pack folder: python3 -m unittest discover -s tests -p 'test_*.py'""" import contextlib, io, json, os, shutil, sys, tempfile, unittest from datetime import date HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, HERE) import pilot as pl def rd(p): with open(p, encoding="utf-8") as f: return f.read() def quiet(fn, *a, **k): with contextlib.redirect_stdout(io.StringIO()) as buf: rc = fn(*a, **k) return rc, buf.getvalue() class Base(unittest.TestCase): def setUp(self): self.d = tempfile.mkdtemp() self.w = os.path.join(self.d, "ex") shutil.copytree(os.path.join(HERE, "example"), self.w) self.terms = os.path.join(self.w, "pilot.json") self.res = os.path.join(self.w, "results.json") self.out = os.path.join(self.d, "out") def tearDown(self): shutil.rmtree(self.d) def T(self): return pl.jload(self.terms) def save(self, obj, path=None): with open(path or self.terms, "w") as f: json.dump(obj, f) def codes(self, t): return [c for c, _ in pl.check(t)] def sign(self, answers=("Maya Lindqvist", "y", "n"), env=None, tty=True, sheet=True): if sheet: pl.make_sheet(self.terms, self.out) it = iter(answers) return quiet(pl.approve, self.terms, self.out, ask=lambda _: next(it), env=env or {}, tty=tty)[0] def score(self, today=date(2026, 11, 30)): return quiet(pl.score, self.terms, self.res, self.out, today) class Terms(Base): def test_example_is_valid(self): self.assertEqual(pl.check(self.T()), []) def test_template_is_not_valid_until_filled(self): self.assertIn("P01", self.codes(pl.jload(os.path.join(HERE, "templates", "pilot.json")))) def test_too_long(self): t = self.T(); t["end"] = "2026-12-31" self.assertIn("P02", self.codes(t)) def test_end_before_start(self): t = self.T(); t["end"] = "2026-11-01" self.assertIn("P02", self.codes(t)) def test_target_must_beat_baseline(self): t = self.T(); t["measures"][0]["target"] = 300 self.assertIn("P03", self.codes(t)) t = self.T(); t["measures"][1]["target"] = 5 self.assertIn("P03", self.codes(t)) def test_measure_needs_source(self): t = self.T(); t["measures"][0]["source"] = "" self.assertIn("P03", self.codes(t)) def test_too_many_measures(self): t = self.T(); t["measures"] = [dict(t["measures"][0], name="m%d" % i) for i in range(6)] self.assertIn("P03", self.codes(t)) def test_pass_rule_range(self): t = self.T(); t["pass_rule"] = 4 self.assertIn("P04", self.codes(t)) t["pass_rule"] = "all" self.assertNotIn("P04", self.codes(t)) def test_off_switch_and_data_limits_required(self): t = self.T(); t["off_switch"] = ""; t["data_limits"] = " " self.assertEqual(self.codes(t).count("P05"), 2) def test_paid_pilot_needs_refund_rule(self): t = self.T(); t["fee"]["refund_rule"] = "" self.assertIn("P06", self.codes(t)) t["fee"]["amount_usd"] = 0 self.assertNotIn("P06", self.codes(t)) def test_consent_cannot_be_written_into_file(self): t = self.T(); t["case_study"]["allowed"] = True self.assertIn("P08", self.codes(t)) def test_sheet_shows_pass_line_and_rules(self): pl.make_sheet(self.terms, self.out) s = rd(os.path.join(self.out, "pilot-terms.md")) self.assertIn("2 of these 3 measures", s) self.assertIn("at most 30", s) self.assertIn("Off switch:", s) self.assertIn("asked separately", s) class Signoff(Base): def test_refused_in_agent_session(self): for k in ("CLAUDECODE", "CODEX_SANDBOX", "HERMES_AGENT", "GROK_BOT"): self.assertEqual(self.sign(env={k: "1"}), 3) self.assertFalse(os.path.exists(os.path.join(self.out, "signoff.json"))) def test_refused_without_terminal(self): self.assertEqual(self.sign(tty=False), 3) def test_refused_without_sheet(self): self.assertEqual(self.sign(sheet=False), 4) def test_refused_wrong_name(self): self.assertEqual(self.sign(answers=("Someone Else", "y", "y")), 5) def test_refused_if_terms_edited_after_sheet(self): pl.make_sheet(self.terms, self.out) t = self.T(); t["measures"][0]["target"] = 60; self.save(t) self.assertEqual(self.sign(sheet=False), 4) def test_case_study_defaults_to_no(self): self.assertEqual(self.sign(answers=("Maya Lindqvist", "y", "")), 0) self.assertFalse(pl.jload(os.path.join(self.out, "signoff.json"))["case_study"]) def test_case_study_yes_is_recorded(self): self.assertEqual(self.sign(answers=("Maya Lindqvist", "y", "y")), 0) self.assertTrue(pl.jload(os.path.join(self.out, "signoff.json"))["case_study"]) def test_no_case_study_question_when_not_asked(self): t = self.T(); t["case_study"] = {"ask": False, "scope": ""}; self.save(t) self.assertEqual(self.sign(answers=("Maya Lindqvist", "y")), 0) class Score(Base): def test_refused_without_signoff(self): self.assertEqual(self.score()[0], 4) def test_refused_if_goalposts_move(self): self.sign() t = self.T(); t["measures"][1]["target"] = 10; self.save(t) rc, out = self.score() self.assertEqual(rc, 4) self.assertIn("old pass lines stand", out) def test_example_passes_two_of_three(self): self.sign() rc, out = self.score() self.assertEqual(rc, 0) r = rd(os.path.join(self.out, "pilot-report.md")) self.assertIn("PASS: 2 of 3 measures met, 2 needed", r) self.assertIn("Move to a monthly plan", r) self.assertIn("No consent on record", r) def test_fail_shows_refund_rule(self): self.sign() r = pl.jload(self.res); r["results"]["First reply time"] = 45; self.save(r, self.res) rc, _ = self.score() self.assertEqual(rc, 1) s = rd(os.path.join(self.out, "pilot-report.md")) self.assertIn("FAIL", s) self.assertIn("half the fee is refunded", s) self.assertIn("hand back all access", s) def test_interim_does_not_trigger_next_step(self): self.sign() r = pl.jload(self.res); r["measured_on"] = "2026-11-16"; self.save(r, self.res) self.score() s = rd(os.path.join(self.out, "pilot-report.md")) self.assertIn("INTERIM", s) self.assertNotIn("Move to a monthly plan", s) def test_case_study_line_follows_consent(self): self.sign(answers=("Maya Lindqvist", "y", "y")) self.score() self.assertIn("agreed to a case study limited to", rd(os.path.join(self.out, "pilot-report.md"))) def test_bad_results(self): self.sign() self.save({"measured_on": "2026-10-01", "results": {"First reply time": "fast", "Extra": 1}}, self.res) rc, out = self.score() self.assertEqual(rc, 1) for c in ("S01", "S02", "S03", "S04"): self.assertIn(c, out) def test_never_writes_terms_or_results(self): before = rd(self.terms) + rd(self.res) self.sign(); self.score() self.assertEqual(before, rd(self.terms) + rd(self.res)) if __name__ == "__main__": unittest.main() ~~~~