Skip to content

Evaluating the harness

Ships in 0.8.

The big picture: every gate script in Blueprint proves it can still fail with a --self-test. Nothing proved the same of the agents. An edit to .claude/agents/security-review.md that made it miss a whole class of bug passed every check in the pipeline. The suite in evals/ closes that gap. It plants a known defect in a throwaway repository, asks the agent to review it, and scores whether it found the defect.

Why it matters: an agent’s prompt is code that nobody tests. A reworded paragraph, a model change or a Claude Code upgrade can each quietly change what a gate catches. The first sign is usually a defect that reached main.

How it works:

  • claude plugin eval runs each case three times with Blueprint’s agents and skills loaded and three times without them, and scores both.
  • A weekly CI job (plugin-evals) runs the whole suite and keeps results.json and an HTML report for 90 days.
  • /kaizen step 1i compares the two latest runs and reports any case that dropped, and any case where the agents made no difference.
  • A free lint (evals-lint, make check-evals) runs on every MR. It checks each case’s structure, builds every fixture, and rejects graders that cannot fail or that a reply could pass by restating the briefing. It cannot show that a case tells a good agent from a bad one; only a paid run can.

A score on its own says little. The model might find the defect without any help from Blueprint. So each case runs in two arms:

Arm What loads What it tells you
with Blueprint’s agents and skills, as a plugin named blueprint How often the harness gets it right
without Nothing: the bare model How often the model gets it right unaided

Δ (delta) is the with-arm score minus the without-arm score: what the harness added. The delta is reported per case, but it never changes the job’s exit code. Only the with-arm score against the threshold (0.8) does.

Read the two numbers together:

With Δ Reading
High Large The agent earns its slot on this class of defect
High ≈ 0 The model does this without help, or the fixture is too easy
Low ≈ 0 The fixture or a grader is broken. Fix the case before reading anything into it
Dropped since last run any An edit, a model or a CLI version made this defect harder to catch

/kaizen flags any case whose |Δ| is 0.1 or less in two runs in a row. A missing delta, from a one-arm run, is reported as “not comparable” and never as 0.

Why the format checks only count in the with-arm

Section titled “Why the format checks only count in the with-arm”

Every review agent ends its report with Findings: 🔴 <n> · 🟡 <n> · 🟢 <n>, and every case checks that the agent was dispatched at all. The bare model never produces either, because it has never seen the agents’ output contract. Scoring those checks in both arms would push the without-arm toward zero and inflate Δ for free. So they are marked arm: with-only. In a two-arm run they are reported as pass/fail indicators and kept out of both scores. The same goes for any tool_used: Skill grader.

A failing indicator is still a finding: /kaizen reports it as contract drift.

Twenty-two cases in three groups. Each one plants one defect, or, for a negative control, none.

Group Case Agent or skill The planted defect
completeness/ instance-fixed-class-missed completeness-check The API hides a field; the CSV export still writes it
shared-rule-collateral completeness-check A narrowed permission helper also blocks token revocation
test-plan-dropped completeness-check A test the issue’s test plan asked for is missing
docs-stale-sibling completeness-check One docs page updated, two still name the old port
followup-without-issue completeness-check The MR description defers work to a follow-up with no issue
test-cannot-fail completeness-check The new test passes with the fix removed
release-path-only completeness-check A tag-only fallback reads a variable the branch removed
merged-tree-collision completeness-check Migration 0007 already exists on origin/main
comment-overrides-criterion completeness-check The branch follows the issue body; a maintainer comment overruled it
completeness-clean-control completeness-check None: a complete branch (negative control)
review/ authz-detail-idor rbac-check Detail and delete actions fetch any object by id
grant-path-parity rbac-check Project invites are capped; the emailed-invite path still grants admin
grant-path-shared-helper rbac-check Project invites go through the cap helper; the role-change path writes through a different helper and lets a member promote themselves to owner
grant-path-clean-control rbac-check None: every grant path calls the one cap helper (negative control)
rbac-clean-control rbac-check None: every action is scoped (negative control)
invariant-sibling-surface security-review A new event stream skips the “deactivated user loses access” rule
destructive-migration schema-check NOT NULL with no default on a populated table, plus a dropped column
guard-fails-before-fix test-scaffold The regression test must be seen failing on the pre-fix code
skills/ adr-natural-phrasing /adr Fires on “write that decision down properly” and writes ADR 0004
plain-question-no-skill none A plain question must not trigger any skill (negative control)
release-rollup-baseline /release Step 2a A stable release after a pre-release series must roll up from the last non-bug-fix stable baseline
release-rollup-control-prerelease /release Step 2a A pre-release target gets no stable-baseline roll-up (negative control)

The negative controls are required. Without them, an agent that flags everything would score 100% on every positive case.

ci/evals.yml is included by default, and both of its jobs skip themselves when evals/ does not exist.

  • evals-lint runs on every MR and on the default branch. It is free and blocking.
  • plugin-evals is paid. It never runs on a merge request, because an MR from a fork must not run an edited scaffold.sh with the API key in its environment. It runs from a pipeline schedule with EVALS_SCHEDULE=true, or from Run pipeline with EVALS_RUN=true, and only on a protected branch.

To turn it on:

  1. Add ANTHROPIC_API_KEY under Settings → CI/CD → Variables, masked and protected.
  2. Set Settings → CI/CD → Variables → Minimum role to use pipeline variables to Maintainer. A manual run on a protected branch carries the key, and pipeline variables are how someone who cannot push there would steer it. Under CI, run-evals.sh also ignores its test hooks. On every run, in CI or locally, it accepts only --case, --runs and --concurrency in EVAL_ARGS.
  3. Create a weekly pipeline schedule on the default branch with the variable EVALS_SCHEDULE=true.

The artifacts are visible to project developers only (access: developer), and the script replaces the key’s value with [MASKED] in every output file on every way out, including a SIGTERM. It also stops the CLI at EVALS_WALL_SECONDS (9600 seconds), inside the job’s 3-hour timeout, so the job is never killed with the key unscrubbed. A run stopped that way exits 3.

The job pins Claude Code (CLAUDE_CODE_VERSION), the agent model (EVALS_MODEL, claude-sonnet-5-5) and the judge model (EVALS_JUDGE_MODEL, claude-opus-5-5) to full model IDs, never aliases. An alias moves on its own, and a moved model would read as a regression in the harness. The judge is a larger model than the agent and never the same one: the eval authoring guide built into Claude Code 2.1.304 asks for a sonnet-tier or larger judge that is not the agent model, and 17 of the 22 cases have an llm rubric.

The job soft-fails at first. Exit codes from scripts/run-evals.sh:

Exit Meaning Pipeline
0 Every case at or above 0.8 green
1 A case below the threshold amber (allowed failure)
2 The EVALS_MAX_COST_USD ceiling stopped the run; results are partial amber (allowed failure)
3 The run measured nothing: no credential, a case or the plugin failed to load, a run ended on an API error or before it was graded (a failed scaffold), the job was killed red
4 Every case that ran passed, but the core-bash cases did not run because the OS sandbox cannot start on this runner amber (allowed failure)

Exit 4 also prints WARNING: pass core-bash SKIPPED (sandbox unavailable): did not run: <cases> in the job log and writes the same line to outcome.txt, so a run that skipped cases never reads as a pass.

The artifact evals-out/ holds results.json (both passes merged), report.html, outcome.txt, and each pass’s raw output.

Terminal window
make eval EVALS_CONFIRM=1 EVAL_ARGS="--case adr-natural-phrasing --runs 1"

make eval refuses to run without EVALS_CONFIRM=1, because every run costs money. It uses your Claude Code login, or ANTHROPIC_API_KEY if set. EVAL_ARGS narrows the run, with --case, --runs and --concurrency only; any other flag, --ablation none included, exits 3. Without it, the whole suite runs.

To iterate on one case’s graders without paying for the baseline, run the CLI directly on an assembled plugin:

Terminal window
claude plugin eval "$(scripts/eval-plugin-dir.sh)" --case <name> --runs 1 \
--ablation none --scaffold --trust-plugin

An unmeasured estimate: 22 cases × 3 runs × 2 arms is 132 agent runs, plus three judge calls per llm grader per run, about 306 judge calls, priced as Opus. That comes to roughly $35–70 per full run. No full run has been measured; each run’s real figure is costUsd in its results.json.

One data point is measured. A single with-arm run of invariant-sibling-surface (security-review, on Claude Code 2.1.304) cost $0.47 and took 114 seconds. The security-review agent ran on Opus, its frontmatter model, while the session ran on the pinned Sonnet. EVALS_MAX_COST_USD (default 50) caps the whole run: the shell pass gets whatever the first pass left. The ceiling is checked before each run starts, so the runs already in flight can take spend past it. The default sits inside the estimate, so a full run can hit it. A run that does is partial (exit 2), and eval-trend.py leaves it out of the trend and names it.

A case is a directory under evals/<group>/<case>/:

evals/review/authz-detail-idor/
├── prompt.md # frontmatter: tags, limits, allowed_tools; body: what a person would type
├── case.yaml # schema_version "1.1", name, context.scaffold_script
├── scaffold.sh # builds the workspace
└── graders/
├── dispatched.md # step: the agent was dispatched (with-only)
├── contract.md # format: the Findings line came back (with-only)
├── located.md # result: the planted identifier is named (scored)
└── judged.md # result: an llm rubric, only where judgment is the point

Phrase it the way a person in a Blueprint session would, and name the gate the way the harness does: “Run the rbac-check gate on branch feat/x”. Never write blueprint:rbac-check. Whether the description makes Claude choose the agent is part of what the case measures. Do not set model: CI pins one model for the whole run. Tag every case core, or core-bash if it needs a shell. Add needs-write if it writes files, needs-bash with core-bash, and control for a negative control.

Each run starts in an empty workspace. Nothing copies fixtures in, and no CLAUDE.md, AGENTS.md or .claude/ from the workspace is loaded, even one the scaffold writes. So each scaffold.sh builds everything itself:

  • every file from a heredoc inside the script, so fixture code (deliberately vulnerable Python, for instance) never lands in the scanned source tree;
  • one exception: a case for a user-invoked skill (disable-model-invocation, which the workspace cannot load) extracts the text under test from the repo at scaffold time (release-rollup-* read Step 2a of /release this way), never from a static copy that could drift; the prompt then tells the agent to read that file. The text is in the workspace in both arms, so such a case scores the same with and without the plugin (Δ is 0 by construction) and /kaizen will flag it every run; judge it on the with-plugin score alone;
  • a real git repository with a base commit on main and the change on feat/x, with GIT_CONFIG_GLOBAL=/dev/null and a fixed author and date;
  • the briefing files a reviewer without Bash needs: diff.patch (git diff main...HEAD), changed-files.txt, issue.md (the issue body and its comments), and main-tree.txt or mr-description.md where the case needs them. They are listed in .git/info/exclude, so the tree still reads clean;
  • no token-shaped strings, which the secret scanners would flag.

Scaffolds must pass shellcheck and the sigpipe check; both gates scan evals/.

Give every case at least:

  • one deterministic grader on the result (regex or file_exists, scored in both arms), usually located.md: a regex for the identifier the fixture planted (export\.py, 0007);
  • one grader on the steps (tool_used or tool_order), usually dispatched.md.

A located pattern that also appears in a briefing file (diff.patch, issue.md, …) proves nothing alone: a reply that restates the diff matches it. 0007 is in changed-files.txt whether or not the reviewer noticed the collision. Such a case needs a second scored grader that demands the judgment, usually an llm rubric in judged.md (“PASS only if it reports the collision as blocking”), and the lint rejects the case without one.

Use an llm grader only where judgment is the point. Mentioning max_length is not the same as saying the test cannot fail. Write its rubric as concrete PASS and FAIL lines, and keep a deterministic grader beside it. Never assert tool_used ... max: 0 on a tool the case was never granted: that grader cannot fail.

Every agent with a positive case needs a negative control, where the right answer is “nothing to fix”. The lint warns for any agent that has positive cases and no negative control. Today it warns for schema-check, security-review and test-scaffold.

Terminal window
make check-evals # or: python3 scripts/check-evals.py --run-scaffolds

It checks the rules above. It also runs every scaffold the way the CLI does: minimal environment, 120-second limit. Every scaffold must create every file its prompt names, and the located.md pattern must match something the scaffold actually planted. A grader that hunts for something never planted can only fail. A case whose every scored grader is a reply regex the briefing files already satisfy is rejected too: it can only pass.

Prove it discriminates, for money: weaken the agent and watch the case drop

Section titled “Prove it discriminates, for money: weaken the agent and watch the case drop”

The lint cannot show that a case tells a good agent from a bad one. Before trusting a new case, run it once with the agent intact and once with the paragraph that makes it catch the defect deleted, from a copy of the plugin:

Terminal window
P="$(scripts/eval-plugin-dir.sh)"
claude plugin eval "$P" --case <name> --scaffold --trust-plugin --ablation none
# edit "$P/agents/<agent>.md": delete the rule this case exercises
claude plugin eval "$P" --case <name> --scaffold --trust-plugin --ablation none

If the score does not drop, the case is measuring something other than that rule. Tighten the fixture or the graders. The copy is throwaway, so the real agent is never touched. No case in this suite has been through this check: it needs a paid run per case.

Terminal window
python3 scripts/eval-trend.py

This is what /kaizen step 1i runs. It downloads results.json from the newest two scheduled plugin-evals jobs that kept artifacts. Failed-but-allowed jobs count, because those are the ones that matter. It reports:

  • Errored: cases whose runs ended before grading, such as a failed scaffold. They measured nothing, so they appear in no other list.
  • Missing: cases the previous run had and this one does not, such as a renamed or deleted case.
  • Dropped: cases at or above 0.8 last run and below it now. sustained means two or more runs failed. single-run means one did, which is likely noise, so re-run before acting.
  • Delta ≈ 0: cases where the harness made no difference, in two runs in a row.
  • Contract drift: with-only indicators failing.
  • Confounders: a changed Claude Code version, agent model or judge model.
  • STALE: the newest scheduled run is more than 14 days old.

Partial runs are listed and left out of the comparison.

These are the decisions behind the layout, and the facts they rest on. They were checked against Claude Code 2.1.304 on 2026-10-20.

  • The suite is not part of the plugin. --eval-dir only accepts a directory below the plugin. When the agents and skills move into a distributed plugin (macrodream/blueprint#111), any case kept there would ship its fixtures to every marketplace user. So the suite lives at the repository root, and scripts/eval-plugin-dir.sh assembles a throwaway plugin named blueprint around it. Its agents resolve as blueprint:<name>, before that move and after it. ADR 0002 §2 records this.
  • Only what the plugin ships can be tested. A run loads no CLAUDE.md, AGENTS.md or .claude/ from the workspace. A rule that lives only in CLAUDE.md or .claude/operating-rules.md, such as “watch the test fail before you trust it”, cannot be measured: its Δ is 0 by construction. The guard-fails-before-fix case targets the copy of that rule in test-scaffold §3a, which the plugin does ship.
  • Two passes. --tag keeps cases matching any tag and has no exclude, and an --allow-tools grant applies to every case in a run. So pass 1 runs the core cases with Write granted, and pass 2 runs the core-bash cases with a narrow shell grant (./run-tests.sh, git show, git diff, cp). Bash only runs under Claude Code’s OS sandbox, so run-evals.sh probes for it first (bubblewrap on Linux) and skips pass 2, loudly, when it cannot start. In an unprivileged Docker container, the default on most runners, bubblewrap cannot create its namespaces. A local run of the job’s image showed exactly that. On such a runner guard-fails-before-fix does not run, the job exits 4 (amber), and the job log and outcome.txt name the skipped case. It runs under make eval on macOS, and on a runner that allows user namespaces.
  • --model pins the session, not the agents. In the probe run the session used the pinned Sonnet, and security-review ran on Opus, its frontmatter model:. To compare models for an agent (the natural first A/B), edit model: in the assembled copy.
  • Sub-agent tool calls are in the trace, but tool_used counts only the session’s. In the probe, the dispatched agent’s own Read and Agent calls appeared in the trace with a parent_tool_use_id, but the tool_used: Agent grader counted one call, the session’s. So step graders assert the dispatch. A check on what the agent itself ran (no git stash) is a regex over target: trace.
  • The schedule, not every MR. About $35–70 a run (estimated) is too much per MR, and per-MR runs would need the API key in fork pipelines.

Go deeper: Test plugins with evals (the CLI’s own reference), Harness gates (the scripts that prove they can fail), and Testing AI-written code (the same “can the check fail” question, asked of application tests).