Rubric Library: name, version and own the scoring rubrics hiding in workflows
by PaperChaseWebb
Every score, rank, flag or triage step an agent runs has a rubric in it, usually unnamed and rewritten each time. Six plain files give each rubric an owner, a version, grades with definitions and actions, evidence rules, borderline examples and an adversarial set. Agents load a rubric by name and number with rubric.py and check every score; unnamed or default is unverdicted, and anything touching a person, a client or money goes to a person. Fifteen read-only checks run on the library itself.
- Price
- $19
- Difficulty
- Intermediate
- Setup
- 30 min
- Format
- .md file
rubric-library-name-version-own-scoring-rubrics.md · 86 KB · instant download after checkout
Want this built for you instead? We can build it for you.
What you end up with
The scoring rules inside your workflows written down, versioned and owned, loaded by your agents before they score, with low-confidence and high-stakes items routed to a person and every load turning unverdicted if anyone edits the library without your yes.
What you get
- 19 sections in one .md file
- The library folder: library.json, rubrics, grades, evidence rules, examples and adversarial sets, with blank templates
- Grades with definitions and actions, confidence thresholds, human gates and a refusal to profile people who did not opt in
- rubric.py: load by name and version, check-score (exit codes 0, 10, 11, 12), owner approval tied to a sha256, verify, and one use per yes
- rubric_check.py: fifteen read-only checks, with a seeded test library (15 findings, then 0)
- A skill file for Claude Code, Codex and Hermes, an AGENTS.md snippet and a Grok Bot rules block
- Ten prompts, a failure table and a worked example
- A look at rubric packs, eval kits and calibration guides, with sources
Who it is for
Anyone whose agents score things (saved research, owned calls, leads, bid notices, tickets) and who wants each score tied to a written, versioned rubric someone owns.
Works with
Solid chips: the agent is named in this listing's own tools list. Dashed chip: A skill file for Claude Code, Codex and Hermes, an AGENTS.md snippet and a Grok Bot rules block; the scripts were tested on Linux with Python 3.13, the agent files were not loaded inside each harness.
Tools needed
Cost to run: The pack is the only purchase. The scripts use no model tokens.
Time to result: A ratified library of two to five rubrics in an afternoon
How it works
Run the tests
Copy the rubric-library folder and run sh test_all.sh.
Expected: 15 findings on the seeded library and 0 on the fixed one, 15 of 15 single faults caught alone, checker tests 26 passed, rubric tests 38 passed, exit 2 on a missing folder.
Find and write your rubrics
Run rubric.py init, use prompts 1 to 5 to find the rubrics already in your prompts and write grades, evidence rules, borderline examples and adversarial items. Run rubric_check.py until only R15 remains.
Expected: Every active rubric has an owner, a number, defined grades with actions, and a human gate that fits what it decides.
Ratify and connect
The owner runs rubric.py approve DIR --library at their own terminal. Install the skill file, paste the AGENTS.md snippet, or paste the rules block into Grok Bot.
Expected: Each agent loads a named rubric version before scoring and follows the check-score exit code.
Live with it
Each scoring run ends in a report the owner approves by hash. Score the adversarial set after every version change, and review the library every quarter with prompt 10.
Expected: verify passes only for an unchanged report under the current library, within approval_days, once.
Sample prompts, free to use
Look through <the last month of prompts or workflow files>. List every place you scored, ranked, graded, flagged or triaged something. For each, write what was scored, the decision it fed and the grades you actually used. Mark every place the grades were never written down. Draft a rubric for <what is scored> feeding <decision>. Two to four grades, each with a definition someone new could apply and the action it triggers. Say which actions touch a person, a client or money. Version 1.0, status draft.
Prompts 1 and 2 of 10.
Load rubric <id> with rubric.py load and tell me the version. Score <the batch>: grade, one line of evidence and a confidence per item, and run check-score on each. Put anything below threshold or marked human in its own list without a verdict. Write the scoring report and give me its path and sha256.
Prompt 6 of 10.
Free preview of the file
# Rubric Library: name, version and own the scoring rubrics hiding in workflows Version 1.0, 2026-10-28. By PaperChaseWebb. A rubric library written as six plain files, a script your agents load rubrics from and check every score against, a read-only checker with fifteen checks, templates, a seeded test library, ten prompts and the tests, all Python 3 standard library. Works with Claude Code, Codex and Hermes through a skill file and an AGENTS.md snippet, and with Grok Bot through a rules block. Tested as described in section 11: everything ran on Linux with Python 3.13, including owner approvals typed at a pseudo-terminal. The skill file, snippet and rules block were not loaded inside the harnesses for this release, and the pack has not been used on a real client workflow. Each is marked where it matters. Most agent workflows score something. Saved posts get graded for hooks worth studying. Call transcripts get a churn flag. Bid notices get pursue, maybe or pass. Inbound requests get sorted by fit. Each of those runs on a rubric, and the rubric usually lives in the prompt: unnamed, unversioned, rewritten a little every time, and owned by nobody. The scores come back with two decimal places and nobody can say which definition of "strong" produced them. A line in one of the sources behind this pack puts it well: the rubric is the product. The model applies whatever it is given. What you can own, review and improve is the rubric. This pack makes the rubric something an agent
Test record
- Date
- Oct 28, 2026
- What was run
- rubric_check.py on a seeded library and a fixed copy, each planted fault alone, a missing input, and the unit tests. Linux, Python 3.13.
- Result
- 15 findings on the seeded library, 0 on the fixed one; 15 of 15 planted faults caught alone; exit 2 on missing input; nothing written. 26 checker and 38 rubric.py tests passed, 0 failed.
- Not run
- The agent files inside Claude Code, Codex, Hermes or Grok Bot. How well a model applies a rubric. A real client workflow. Windows.
Copied from this item's dated test notes. What was not run stays in.
Worth knowing
- rubric.py answers questions; it cannot stop an agent that never asks. The rules files tell each agent to load a rubric first, and the quarterly review catches drift, but this is not a sandbox.
- The skill file, AGENTS.md snippet and Grok Bot rules block were not loaded inside Claude Code, Codex, Hermes or Grok Bot for this release, so how reliably each harness loads a rubric first was not measured. How well a given model applies a given rubric was not measured either; the adversarial set is how you check that on your own material. The pack has not been used on a real client workflow. The test library is made up. Linux only; Windows was not tested.
- Not an eval harness for AI products, not a tool for hiring or credit decisions about people, and not a compliance certification (SOC 2, ISO 42001, EU AI Act).
- The approval step refuses piped input and the environment variables Claude Code, Codex, Hermes and Grok Bot set. That stops an agent approving by mistake; it is not a lock against someone with the same account.
- On 2026-10-28, on Linux with Python 3.13: rubric_check.py gave 15 findings on the seeded library (each check once) and 0 on the fixed one, caught 15 of 15 single faults with only its own check, wrote nothing, and exited 2 on a missing folder, any of the six library files missing, or broken library.json; checker tests 26 passed, 0 failed; rubric.py tests at a pseudo-terminal 38 passed, 0 failed.
Questions
How is this different from an eval rubric pack?
Eval packs grade model output for an AI product. This pack governs the business scoring steps inside agent workflows: who owns each rubric, which version is live, what each grade may trigger, and when a person decides instead.
What happens if the agent scores without naming a rubric?
There is no rubric to load, so the answer is unverdicted, exit 12. A score from an unnamed or default rubric is not a result.
Can a score trigger an email or a refund by itself?
No. Rubrics touching a person, a client or money are human_gate always, and check-score answers human (exit 11) at any confidence. The score prepares the decision; a person makes it.
What if I change a rubric?
Bump its version and keep the old one retired. Until you ratify the edited library, every load answers unverdicted and earlier scoring reports stop verifying.
Details
Version 1.0 · Updated Oct 28, 2026 · Markdown version of this page
License: Use on your own workflows and on libraries you set up for clients and teams. Do not resell or republish the pack itself. Download terms
Related
Rubric Library: five rules and a five-check script (free sample)
From the Rubric Library: five rules for the scoring inside agent workflows (a human owns the library, every rubric has a number, every grade says what it means and what it triggers, people, clients and money go to a person, no profiling people who did not opt in), and a read-only script that checks one rubric file for them.
Desk Authority Map: who in your agent fleet may decide what
Four plain files say, for every desk and action, decide, propose, ask-owner or never, with dollar limits, escalation and the evidence each desk owes. Agents ask authority.py before they act and get the answer as an exit code; anything not in the map is never. Owner yeses are typed at the owner's terminal, tied to one request's sha256, good for one use, and refused inside an agent. Fifteen read-only checks run on the map itself.
Governed Agent Teams: subagent personas, authority maps and handoffs
A Markdown pack for running a team of coding and research agents with clear limits: 7 subagent personas for Claude Code with Codex AGENTS.md and Hermes skill versions, a Desk Authority Map, three role risk classes, a handoff packet the receiver verifies, an evidence standard, a decision ledger and a reach inventory.
Research Contract for Agents: claims, sources and dates your agent must keep
A contract your agent follows on every research run, kept as plain files: each claim with its source, date and confidence, primary sources before secondary, quotes word for word with a location, contradictions listed, unknown allowed, and no citation to anything it did not open. A fifteen-check script reads the folder and names every break in a plain sentence, and nothing is published or acted on until the owner approves the exact report by its sha256.
Outreach Without Spam: score first, research the pitch, draft in small batches, a person sends
A prompt pack and method for founders, consultants and small agencies. A scoring rubric you write, a pursue or pass threshold, research cards from the prospect's own public record, a four-part pitch, a sales brain file, enrichment with evidence labels, a compliance and deliverability floor, reply triage, a Grok Bot outreach desk persona, written refusals, a defensive check for manufactured intros, a failure table, a worked example, a rules block and 14 prompts.
Production Readiness Audit: decide from evidence whether an agent system can go live
Twelve areas, each pass, gap or na, and every pass points to a dated evidence file. Gaps get a named owner and a due date, five areas block go-live until fixed, drills are run rather than described, and the verdict is signed by someone who did not build the system. A fifteen-check script reads the folder, the report is sealed to a hash of the evidence, and the owner's go-live yes is tied to the report's sha256.
Agent Readiness Self-Check: score one workflow before you hand it to an AI agent
A free self-check for agencies and small businesses. Name the decision, then score one workflow across eight areas (workflow mapped, data access, approval gates, owner per agent, handoffs, cost tracking, security and vetting, measurement) with an evidence rule, three gate rules and a skeptical pass, and read your band and the next steps it allows. v1.1 starts with the decision the score will change, scores unknowns as unknown, adds rules for conflicting facts and lookalike public records, ties every band to the next steps it allows with the Production Readiness Audit before go-live, makes owner approval concrete (prepared and approved status, a one-reply approval line, decisions the agent must leave open, a done test), adds agent files for Grok Bot, Claude Code, Codex and Hermes, a second worked example (missed call to booked job), a readiness folder with a fifteen-check read-only checker, a front door test with four questions to ask anyone selling you an agent install, and a plain note on how it was tested. Still free.
Evidence Standard: what counts as proof for each kind of agent work, who checks it, and what is refused
Agents report done from the plan as often as from the result. Five plain files write down, for each class of work, the proof that counts, who checks it (never the producer), what is refused and how long proof is kept. Agents log each claim with its evidence, seal it by sha256 and report in the words evidence.py say allows: done, partly done with the gap named, waiting on the owner, or reported, not verified. The agent's own summary is never proof. Fifteen read-only checks run on the standard and the claims ledger.