Evaluating the harness
Ships in 0.8.
The big picture: every gate script in Blueprint proves it can still fail with a
--self-test. Nothing proved the same of the agents. An edit to
.claude/agents/security-review.md that made it miss a whole class of bug passed every
check in the pipeline. The suite in evals/ closes that gap. It plants a known defect in
a throwaway repository, asks the agent to review it, and scores whether it found the
defect.
Why it matters: an agent’s prompt is code that nobody tests. A reworded paragraph, a
model change or a Claude Code upgrade can each quietly change what a gate catches. The
first sign is usually a defect that reached main.
How it works:
claude plugin evalruns each case three times with Blueprint’s agents and skills loaded and three times without them, and scores both.- A weekly CI job (
plugin-evals) runs the whole suite and keepsresults.jsonand an HTML report for 90 days. /kaizenstep 1i compares the two latest runs and reports any case that dropped, and any case where the agents made no difference.- A free lint (
evals-lint,make check-evals) runs on every MR. It checks each case’s structure, builds every fixture, and rejects graders that cannot fail or that a reply could pass by restating the briefing. It cannot show that a case tells a good agent from a bad one; only a paid run can.
The two arms and the delta
Section titled “The two arms and the delta”A score on its own says little. The model might find the defect without any help from Blueprint. So each case runs in two arms:
| Arm | What loads | What it tells you |
|---|---|---|
| with | Blueprint’s agents and skills, as a plugin named blueprint |
How often the harness gets it right |
| without | Nothing: the bare model | How often the model gets it right unaided |
Δ (delta) is the with-arm score minus the without-arm score: what the harness added. The delta is reported per case, but it never changes the job’s exit code. Only the with-arm score against the threshold (0.8) does.
Read the two numbers together:
| With | Δ | Reading |
|---|---|---|
| High | Large | The agent earns its slot on this class of defect |
| High | ≈ 0 | The model does this without help, or the fixture is too easy |
| Low | ≈ 0 | The fixture or a grader is broken. Fix the case before reading anything into it |
| Dropped since last run | any | An edit, a model or a CLI version made this defect harder to catch |
/kaizen flags any case whose |Δ| is 0.1 or less in two runs in a row. A missing delta,
from a one-arm run, is reported as “not comparable” and never as 0.
Why the format checks only count in the with-arm
Section titled “Why the format checks only count in the with-arm”Every review agent ends its report with Findings: 🔴 <n> · 🟡 <n> · 🟢 <n>, and every
case checks that the agent was dispatched at all. The bare model never produces either,
because it has never seen the agents’ output contract. Scoring those checks in both arms
would push the without-arm toward zero and inflate Δ for free. So they are marked
arm: with-only. In a two-arm run they are reported as pass/fail indicators and kept
out of both scores. The same goes for any tool_used: Skill grader.
A failing indicator is still a finding: /kaizen reports it as contract drift.
What the suite covers
Section titled “What the suite covers”Twenty-two cases in three groups. Each one plants one defect, or, for a negative control, none.
| Group | Case | Agent or skill | The planted defect |
|---|---|---|---|
completeness/ |
instance-fixed-class-missed |
completeness-check |
The API hides a field; the CSV export still writes it |
shared-rule-collateral |
completeness-check |
A narrowed permission helper also blocks token revocation | |
test-plan-dropped |
completeness-check |
A test the issue’s test plan asked for is missing | |
docs-stale-sibling |
completeness-check |
One docs page updated, two still name the old port | |
followup-without-issue |
completeness-check |
The MR description defers work to a follow-up with no issue | |
test-cannot-fail |
completeness-check |
The new test passes with the fix removed | |
release-path-only |
completeness-check |
A tag-only fallback reads a variable the branch removed | |
merged-tree-collision |
completeness-check |
Migration 0007 already exists on origin/main |
|
comment-overrides-criterion |
completeness-check |
The branch follows the issue body; a maintainer comment overruled it | |
completeness-clean-control |
completeness-check |
None: a complete branch (negative control) | |
review/ |
authz-detail-idor |
rbac-check |
Detail and delete actions fetch any object by id |
grant-path-parity |
rbac-check |
Project invites are capped; the emailed-invite path still grants admin | |
grant-path-shared-helper |
rbac-check |
Project invites go through the cap helper; the role-change path writes through a different helper and lets a member promote themselves to owner | |
grant-path-clean-control |
rbac-check |
None: every grant path calls the one cap helper (negative control) | |
rbac-clean-control |
rbac-check |
None: every action is scoped (negative control) | |
invariant-sibling-surface |
security-review |
A new event stream skips the “deactivated user loses access” rule | |
destructive-migration |
schema-check |
NOT NULL with no default on a populated table, plus a dropped column |
|
guard-fails-before-fix |
test-scaffold |
The regression test must be seen failing on the pre-fix code | |
skills/ |
adr-natural-phrasing |
/adr |
Fires on “write that decision down properly” and writes ADR 0004 |
plain-question-no-skill |
none | A plain question must not trigger any skill (negative control) | |
release-rollup-baseline |
/release Step 2a |
A stable release after a pre-release series must roll up from the last non-bug-fix stable baseline | |
release-rollup-control-prerelease |
/release Step 2a |
A pre-release target gets no stable-baseline roll-up (negative control) |
The negative controls are required. Without them, an agent that flags everything would score 100% on every positive case.
Running it
Section titled “Running it”ci/evals.yml is included by default, and both of its jobs skip themselves when evals/
does not exist.
evals-lintruns on every MR and on the default branch. It is free and blocking.plugin-evalsis paid. It never runs on a merge request, because an MR from a fork must not run an editedscaffold.shwith the API key in its environment. It runs from a pipeline schedule withEVALS_SCHEDULE=true, or from Run pipeline withEVALS_RUN=true, and only on a protected branch.
To turn it on:
- Add
ANTHROPIC_API_KEYunder Settings → CI/CD → Variables, masked and protected. - Set Settings → CI/CD → Variables → Minimum role to use pipeline variables to
Maintainer. A manual run on a protected branch carries the key, and pipeline
variables are how someone who cannot push there would steer it. Under CI,
run-evals.shalso ignores its test hooks. On every run, in CI or locally, it accepts only--case,--runsand--concurrencyinEVAL_ARGS. - Create a weekly pipeline schedule on the default branch with the variable
EVALS_SCHEDULE=true.
The artifacts are visible to project developers only (access: developer), and the
script replaces the key’s value with [MASKED] in every output file on every way out,
including a SIGTERM. It also stops the CLI at EVALS_WALL_SECONDS (9600 seconds), inside
the job’s 3-hour timeout, so the job is never killed with the key unscrubbed. A run
stopped that way exits 3.
The job pins Claude Code (CLAUDE_CODE_VERSION), the agent model (EVALS_MODEL,
claude-sonnet-5-5) and the judge model (EVALS_JUDGE_MODEL, claude-opus-5-5) to full
model IDs, never aliases. An alias moves on its own, and a moved model would read as a
regression in the harness. The judge is a larger model than the agent and never the same
one: the eval authoring guide built into Claude Code 2.1.304 asks for a sonnet-tier or
larger judge that is not the agent model, and 17 of the 22 cases have an llm
rubric.
The job soft-fails at first. Exit codes from scripts/run-evals.sh:
| Exit | Meaning | Pipeline |
|---|---|---|
| 0 | Every case at or above 0.8 | green |
| 1 | A case below the threshold | amber (allowed failure) |
| 2 | The EVALS_MAX_COST_USD ceiling stopped the run; results are partial |
amber (allowed failure) |
| 3 | The run measured nothing: no credential, a case or the plugin failed to load, a run ended on an API error or before it was graded (a failed scaffold), the job was killed | red |
| 4 | Every case that ran passed, but the core-bash cases did not run because the OS sandbox cannot start on this runner |
amber (allowed failure) |
Exit 4 also prints WARNING: pass core-bash SKIPPED (sandbox unavailable): did not run: <cases> in the job log and writes the same line to outcome.txt, so a run that skipped
cases never reads as a pass.
The artifact evals-out/ holds results.json (both passes merged), report.html,
outcome.txt, and each pass’s raw output.
Locally
Section titled “Locally”make eval EVALS_CONFIRM=1 EVAL_ARGS="--case adr-natural-phrasing --runs 1"make eval refuses to run without EVALS_CONFIRM=1, because every run costs money. It
uses your Claude Code login, or ANTHROPIC_API_KEY if set. EVAL_ARGS narrows the run,
with --case, --runs and --concurrency only; any other flag, --ablation none
included, exits 3. Without it, the whole suite runs.
To iterate on one case’s graders without paying for the baseline, run the CLI directly on an assembled plugin:
claude plugin eval "$(scripts/eval-plugin-dir.sh)" --case <name> --runs 1 \ --ablation none --scaffold --trust-pluginWhat it costs
Section titled “What it costs”An unmeasured estimate: 22 cases × 3 runs × 2 arms is 132 agent runs, plus three
judge calls per llm grader per run, about 306 judge calls, priced as Opus. That comes to
roughly $35–70 per full run. No full run has been measured; each run’s real figure is
costUsd in its results.json.
One data point is measured. A single with-arm run of invariant-sibling-surface
(security-review, on Claude Code 2.1.304) cost $0.47 and took 114 seconds. The
security-review agent ran on Opus, its frontmatter model, while the session ran on the
pinned Sonnet. EVALS_MAX_COST_USD (default 50) caps the whole run: the shell pass gets
whatever the first pass left. The ceiling is checked before each run starts, so the runs
already in flight can take spend past it. The default sits inside the estimate, so a full
run can hit it. A run that does is partial (exit 2), and eval-trend.py leaves it out of
the trend and names it.
Adding a case
Section titled “Adding a case”A case is a directory under evals/<group>/<case>/:
evals/review/authz-detail-idor/├── prompt.md # frontmatter: tags, limits, allowed_tools; body: what a person would type├── case.yaml # schema_version "1.1", name, context.scaffold_script├── scaffold.sh # builds the workspace└── graders/ ├── dispatched.md # step: the agent was dispatched (with-only) ├── contract.md # format: the Findings line came back (with-only) ├── located.md # result: the planted identifier is named (scored) └── judged.md # result: an llm rubric, only where judgment is the pointThe prompt
Section titled “The prompt”Phrase it the way a person in a Blueprint session would, and name the gate the way the
harness does: “Run the rbac-check gate on branch feat/x”. Never write
blueprint:rbac-check. Whether the description makes Claude choose the agent is part of
what the case measures. Do not set model: CI pins one model for the whole run. Tag every
case core, or core-bash if it needs a shell. Add needs-write if it writes files,
needs-bash with core-bash, and control for a negative control.
The scaffold
Section titled “The scaffold”Each run starts in an empty workspace. Nothing copies fixtures in, and no CLAUDE.md,
AGENTS.md or .claude/ from the workspace is loaded, even one the scaffold writes. So
each scaffold.sh builds everything itself:
- every file from a heredoc inside the script, so fixture code (deliberately vulnerable Python, for instance) never lands in the scanned source tree;
- one exception: a case for a user-invoked skill (
disable-model-invocation, which the workspace cannot load) extracts the text under test from the repo at scaffold time (release-rollup-*read Step 2a of/releasethis way), never from a static copy that could drift; the prompt then tells the agent to read that file. The text is in the workspace in both arms, so such a case scores the same with and without the plugin (Δ is 0 by construction) and/kaizenwill flag it every run; judge it on the with-plugin score alone; - a real git repository with a base commit on
mainand the change onfeat/x, withGIT_CONFIG_GLOBAL=/dev/nulland a fixed author and date; - the briefing files a reviewer without Bash needs:
diff.patch(git diff main...HEAD),changed-files.txt,issue.md(the issue body and its comments), andmain-tree.txtormr-description.mdwhere the case needs them. They are listed in.git/info/exclude, so the tree still reads clean; - no token-shaped strings, which the secret scanners would flag.
Scaffolds must pass shellcheck and the sigpipe check; both gates scan evals/.
The graders
Section titled “The graders”Give every case at least:
- one deterministic grader on the result (
regexorfile_exists, scored in both arms), usuallylocated.md: a regex for the identifier the fixture planted (export\.py,0007); - one grader on the steps (
tool_usedortool_order), usuallydispatched.md.
A located pattern that also appears in a briefing file (diff.patch, issue.md, …)
proves nothing alone: a reply that restates the diff matches it. 0007 is in
changed-files.txt whether or not the reviewer noticed the collision. Such a case needs a
second scored grader that demands the judgment, usually an llm rubric in judged.md
(“PASS only if it reports the collision as blocking”), and the lint rejects the case
without one.
Use an llm grader only where judgment is the point. Mentioning max_length is not the
same as saying the test cannot fail. Write its rubric as concrete PASS and FAIL lines, and
keep a deterministic grader beside it. Never assert tool_used ... max: 0 on a tool the
case was never granted: that grader cannot fail.
Every agent with a positive case needs a negative control, where the right answer is
“nothing to fix”. The lint warns for any agent that has positive cases and no negative
control. Today it warns for schema-check, security-review and test-scaffold.
Lint it, for free
Section titled “Lint it, for free”make check-evals # or: python3 scripts/check-evals.py --run-scaffoldsIt checks the rules above. It also runs every scaffold the way the CLI does: minimal
environment, 120-second limit. Every scaffold must create every file its prompt names, and
the located.md pattern must match something the scaffold actually planted. A grader that
hunts for something never planted can only fail. A case whose every scored grader is a
reply regex the briefing files already satisfy is rejected too: it can only pass.
Prove it discriminates, for money: weaken the agent and watch the case drop
Section titled “Prove it discriminates, for money: weaken the agent and watch the case drop”The lint cannot show that a case tells a good agent from a bad one. Before trusting a new case, run it once with the agent intact and once with the paragraph that makes it catch the defect deleted, from a copy of the plugin:
P="$(scripts/eval-plugin-dir.sh)"claude plugin eval "$P" --case <name> --scaffold --trust-plugin --ablation none# edit "$P/agents/<agent>.md": delete the rule this case exercisesclaude plugin eval "$P" --case <name> --scaffold --trust-plugin --ablation noneIf the score does not drop, the case is measuring something other than that rule. Tighten the fixture or the graders. The copy is throwaway, so the real agent is never touched. No case in this suite has been through this check: it needs a paid run per case.
Reading the trend
Section titled “Reading the trend”python3 scripts/eval-trend.pyThis is what /kaizen step 1i runs. It downloads results.json from the newest two
scheduled plugin-evals jobs that kept artifacts. Failed-but-allowed jobs count, because
those are the ones that matter. It reports:
- Errored: cases whose runs ended before grading, such as a failed scaffold. They measured nothing, so they appear in no other list.
- Missing: cases the previous run had and this one does not, such as a renamed or deleted case.
- Dropped: cases at or above 0.8 last run and below it now.
sustainedmeans two or more runs failed.single-runmeans one did, which is likely noise, so re-run before acting. - Delta ≈ 0: cases where the harness made no difference, in two runs in a row.
- Contract drift: with-only indicators failing.
- Confounders: a changed Claude Code version, agent model or judge model.
- STALE: the newest scheduled run is more than 14 days old.
Partial runs are listed and left out of the comparison.
How the suite is built
Section titled “How the suite is built”These are the decisions behind the layout, and the facts they rest on. They were checked against Claude Code 2.1.304 on 2026-10-20.
- The suite is not part of the plugin.
--eval-dironly accepts a directory below the plugin. When the agents and skills move into a distributed plugin (macrodream/blueprint#111), any case kept there would ship its fixtures to every marketplace user. So the suite lives at the repository root, andscripts/eval-plugin-dir.shassembles a throwaway plugin namedblueprintaround it. Its agents resolve asblueprint:<name>, before that move and after it. ADR 0002 §2 records this. - Only what the plugin ships can be tested. A run loads no
CLAUDE.md,AGENTS.mdor.claude/from the workspace. A rule that lives only inCLAUDE.mdor.claude/operating-rules.md, such as “watch the test fail before you trust it”, cannot be measured: its Δ is 0 by construction. Theguard-fails-before-fixcase targets the copy of that rule intest-scaffold§3a, which the plugin does ship. - Two passes.
--tagkeeps cases matching any tag and has no exclude, and an--allow-toolsgrant applies to every case in a run. So pass 1 runs thecorecases withWritegranted, and pass 2 runs thecore-bashcases with a narrow shell grant (./run-tests.sh,git show,git diff,cp). Bash only runs under Claude Code’s OS sandbox, sorun-evals.shprobes for it first (bubblewrapon Linux) and skips pass 2, loudly, when it cannot start. In an unprivileged Docker container, the default on most runners,bubblewrapcannot create its namespaces. A local run of the job’s image showed exactly that. On such a runnerguard-fails-before-fixdoes not run, the job exits 4 (amber), and the job log andoutcome.txtname the skipped case. It runs undermake evalon macOS, and on a runner that allows user namespaces. --modelpins the session, not the agents. In the probe run the session used the pinned Sonnet, andsecurity-reviewran on Opus, its frontmattermodel:. To compare models for an agent (the natural first A/B), editmodel:in the assembled copy.- Sub-agent tool calls are in the trace, but
tool_usedcounts only the session’s. In the probe, the dispatched agent’s ownReadandAgentcalls appeared in the trace with aparent_tool_use_id, but thetool_used: Agentgrader counted one call, the session’s. So step graders assert the dispatch. A check on what the agent itself ran (nogit stash) is aregexovertarget: trace. - The schedule, not every MR. About $35–70 a run (estimated) is too much per MR, and per-MR runs would need the API key in fork pipelines.
Go deeper: Test plugins with evals (the CLI’s own reference), Harness gates (the scripts that prove they can fail), and Testing AI-written code (the same “can the check fail” question, asked of application tests).