Harness gates
The big picture: A gate is any check that can say no to a change — a CI job, a pre-push script, a review agent. Blueprint’s rule is simple: a green result has to be evidence, so every gate script can prove, on demand, that it still fails when it should.
Why it matters: A check that silently stopped working is worse than no check, because people keep trusting its green. Almost every rule on this page exists because a check somewhere reported “all clear” on a tree that broke the rule.
How it works:
- Every gate script has a
--self-testthat builds a broken example and confirms it’s rejected — run in the same CI job as the real check. - A gate that can’t reach what it needs (the network, the tracker) fails instead of passing quietly.
- Every CI gate can also run on your machine before you push (
make pre-push-checks). - Review agents leave a one-line result in each MR (the gate ledger), so you can tell which gates actually find problems.
The gate scripts
Section titled “The gate scripts”| Gate | Runs | Catches |
|---|---|---|
check-issue-collision.sh |
pre-push | A second MR for an issue that already has one |
check-security-deferred.sh |
pre-release Step 1.6 + release.sh pre-tag (GA tags only) |
An issue that ever carried security::deferred, in any milestone or state, that is unresolved: open with no valid accepted-risk note, or closed / unlabeled by someone who may not clear it; or a missing label — see Deferred security follow-ups |
check-spec-completeness.py |
pre-push (make check-specs) + CI (spec-check) |
A spec under docs/specs/ that is missing a required part (rules S-01..S-31: front matter, sections, users, acceptance criteria, resolvable and acyclic Depends on:, decisions that agree with the features they block, an Issue: #N on every feature of an imported spec, and --fix for document-export artifacts); prints one derivation line per finding and exits 0, 1, 2 or 3 by verdict — see Spec-driven development |
check-prepush-parity.sh |
pre-push + CI | A CI gate you can’t run before pushing |
check-gate-selftest-parity.sh |
pre-push + CI | A gate with no --self-test, or one that self-tests in a different job |
check-stale-references.sh |
CI (MRs, main, schedules) |
STUB/WIP markers; a reference to a renamed skill; an agent name used as a slash command; a slash command that matches no skill, built-in or agent; a link to an ADR file that does not exist; TODO(#N) and SUPPRESSED-UNTIL(#N) pointing at closed issues. Rule 7: rule 7 (/name matches no skill, built-in or agent) reads Markdown only. Known false negatives: a typo wrapped in emphasis (**/wvae**, _/wvae_) and a capitalized name (/Wvae) pass. A deliberate non-command takes slash-ok on its line; SLASH_EXTRA_OK replaces (does not extend) the default list of never-a-command names |
check-cc-claims.sh |
pre-push + CI | A Claude Code behavior claim with no code.claude.com link or checked: date |
check-mr-followups.sh |
CI (MRs) | A ## Follow-ups bullet or DEFERRED Requirements cell naming no open issue |
check-spec-trace.sh |
CI only (spec-trace: default branch + schedules) |
A status: imported spec whose Issue: #N is gone from the tracker, or whose issue’s Spec: <path>#F-nn line names a feature the spec lacks; fails closed when the tracker is unreachable (ALLOW_UNRESOLVED=1 downgrades to a warning). An open issue citing Spec: that no tracked spec accounts for (an orphan from an interrupted import) is reported by --orphans, which only /tracker-hygiene runs |
check-gate-ledger.py |
pre-push (self-test) + CI (MRs) | A ## Gates line whose outcome gate-yield.py cannot count (free text) |
check-rigor-floor.py |
pre-push (self-test) + CI (MRs, in mr-gate-ledger) |
An MR whose diff touches a path in .claude/rigor.json’s floor without the rigor: level its tier sets (strict, or strict-light for the reporting and ledger scripts) and that level’s gate lines (regression-check, completeness-check round 1 (on Opus at strict, any model at strict-light) and round 2 on Opus, the fix-diff re-check); see the rigor floor. Fails closed when the base or the config cannot be read |
check-bare-issue-refs.py |
pre-push + CI (bare-issue-refs) |
A bare #NNN Blueprint issue reference in a comment, docstring or doc of the shipped harness (scripts/, ci/, hooks, website pages); write macrodream/blueprint#NNN. Switches itself off once .blueprint-reset exists |
check-harness-reference.py |
pre-push + CI (harness-reference) |
A hook (.claude/hooks/*.sh, hooks/*), a scripts/wt subcommand or a make help target with no entry on its reference page (hooks, wt, make-targets), or a make-targets row for a target that no longer exists. A name in prose does not count: the entry must be a heading (hooks, wt) or a table row (make targets). A scan that finds nothing is an error, not a pass |
check-docs-stale.py |
pre-push + CI (docs-stale) |
A backticked make <target>, scripts/<path> or scripts/<path> --flag in docs/, website/, .claude/**/*.md, AGENTS.md, CLAUDE.md or changelog.d/ that names nothing in the tree, and, on a branch, a make target, script, flag or BLUEPRINT_* variable the diff removed that a doc still names on a line the diff did not add. Fails closed when the merge base cannot be computed. Intentional history takes an inline docs-stale: ok marker. Does not read for overstated claims, version tense or prose that contradicts behavior: those stay with completeness-check |
check-version-lockstep.py |
pre-push + CI | A version-bearing file the release script forgot to bump |
check-sigpipe-readers.sh |
pre-push + CI | An early-exit reader behind a pipe, which under pipefail reports a real match as missing |
check-shell-portability.sh |
pre-push + CI (shell-portability) |
A BSD-only or GNU-only shell idiom (stat -f tried before stat -c, sed -i without an attached suffix, date -d/-v with no fallback, readlink -f, grep -P, xargs -r, a GNU-only mktemp template or the BSD mktemp -t <name> form, echo -e; a backslash escape in a sed bracket, 0,/re/, gawk-only awk (also in multi-line single-quoted programs), tr [:print:], BusyBox-rejected grep --include/--exclude-dir/--line-buffered/-z); a reasoned # portability-ok: <reason> marker suppresses one line, and # portability-ok-file: <reason> a whole file, but only for the allowlist inside the script (today scripts/tests/pre-bash-deny.test.sh, whose flagged lines are quoted arguments) |
check-shellcheck.sh |
pre-push + CI | Shell scripts failing shellcheck |
check-nul-bytes.sh |
pre-push + CI | A stray NUL byte in a text file — Git and grep then treat the whole file as binary, so every tree-walking check silently skips it |
check-yaml-duplicate-keys.py, check-ci-include-all.py |
pre-push + CI (ci-yaml-lint) |
A repeated mapping key in .gitlab-ci.yml or ci/*.yml (YAML keeps the last and says nothing, which dropped variables from ci/python.yml); then a copy of .gitlab-ci.yml with every commented include enabled, checked for missing includes, duplicate keys and unknown keywords, plus GitLab CI Lint (dry run) when a GITLAB_LINT_TOKEN variable is set |
check-single-fastpath-table.sh |
pre-push + CI | A second copy of the fast-paths table, which would drift from the one in CLAUDE.md; a listed sentinel rule heading that opens a section in more than one of AGENTS.md, CLAUDE.md and their @-imports, or in none of them (a typo or stale entry in the list); an AGENTS.md that names .claude; a CLAUDE.md whose first line is not @AGENTS.md |
check-skill-names.sh |
pre-push + CI | A .claude/skills/* name equal to a Claude Code bundled skill, built-in command, or alias: the project skill would replace the built-in, or never run from the alias |
check-commands-doc.py |
pre-push + CI (commands-doc) |
A skill with no commands.md section, a section without its Before this: or Next: line, or a section for a skill that no longer exists |
check-agent-contract.sh |
pre-push + CI | A .claude/agents/*.md description with no “Do not use” clause; a review gate that does not end with the Findings: 🔴 <n> · 🟡 <n> · 🟢 <n> line or brings back a retired severity word; a gate with no Bash and no Brief contract section; an agent file in neither of the script’s two lists; a CLAUDE.md stop rule that does not name the scale, or a retired severity word in a .claude/hooks/ file |
check-claudemd-size.sh |
pre-push + CI | A CLAUDE.md over 200 lines (the root file loads in every session); nested exceptions go in scripts/claudemd-size-allowlist.txt, each with its own ceiling and a reason; also the byte budget in scripts/claudemd-byte-budget.txt for the always-loaded files |
check-load-bearing.sh |
pre-push + CI | Deleting a declared one-line guard |
check-edition-boundary.sh |
pre-push + CI (edition-boundary); a no-op until edition-boundary.conf names a package |
The open-source tree naming a configured enterprise package (an import, a require, or a string reference such as a settings entry) — see Open core and editions |
check-evals.py |
pre-push (make check-evals) + CI (evals-lint), only with an evals/ suite |
An eval case that cannot fail: no scored result grader or no step grader, an llm rubric with no deterministic grader beside it, tool_used max: 0 on a tool the case never has, an unknown frontmatter key, a scaffold that errors, or a located pattern the scaffold never planted — see Evaluating the harness (ships in 0.8) |
check-artifact-assertions.sh |
pre-push + CI | A tag-time publish job that doesn’t verify its artifact is non-empty |
check-tag-only-jobs.sh |
/release pre-flight + CI self-test |
A tag-only CI job changed since the last tag — it has never run in its current form |
check-release-pipeline.py |
CI (tags); --check-ci-file in pre-push + CI |
Publishing a tag from a commit where any job in its own branch pipeline failed (an allow_failure job included), or a tag job that does not needs: the gate, or a trigger job that can run on the release ref without strategy: depend / mirror (so its bridge would pass while its child pipeline failed), or a tag pipeline whose CI variables were overridden to aim it at another commit, project, ref or API host (job-by-job judgment, --check-ci-file and the variable checks ship in 0.8) |
check-docs-internal-links.py |
pre-push + CI (ci/docs.yml), only with a docs site |
A broken internal link, anchor, asset, or source link in the docs |
check-compose-project-names.sh |
pre-push + CI (ci/docker.yml) |
A compose file without its own pinned project name — two stacks can then share volumes (and data) or recreate each other’s containers |
check-migration-numbering.sh |
pre-push + CI (ci/python.yml) |
Two branches picking the same migration number — invisible until both merge |
check-serializer-ts-parity.py |
pre-push + CI (ci/python.yml) |
Backend schema and TypeScript types disagreeing on a field’s type |
check-ws-event-reachability.py |
pre-push + CI (ci/python.yml) |
A published real-time event no client can actually subscribe to |
check-added-files-covered.mjs |
pre-push + CI (ci/node.yml) |
A brand-new source file with no tests, which coverage tools silently omit |
check-chart-registry.sh |
CI (ci/helm.yml) |
A published Helm chart that doesn’t install the way the docs say |
check-trivyignore-expiry.sh |
CI (ci/trivyignore-expiry.yml, off by default) |
A .trivyignore entry with no exp:YYYY-MM-DD, a malformed or expired date, or one more than TRIVYIGNORE_MAX_DAYS (default 180) out; a missing file passes |
check-sonar-exclusions.sh |
CI (ci/sonar-exclusions.yml, off by default) |
A Sonar ignore-criterion ruleKey that does not exist or is inactive, a rejected token, or a multicriteria line it cannot parse; exits 2 (unresolved) when the Sonar API cannot be reached, unless ALLOW_UNRESOLVED=1 |
check-cognitive-complexity.py |
CI (ci/sonar-complexity.yml, off by default) |
A script entry point above its cognitive-complexity tier — a rough proxy, read its header before trusting it |
check_mutation_score.py |
CI (ci/mutation.yml, scheduled, off by default) |
A missing shard, a dead target, or an adjusted mutation score below the floor; fails closed |
nightly_load_test.py |
CI (python-nightly-load-test, scheduled) |
An endpoint whose p95 latency exceeds its committed budget |
osv-severity-gate.sh |
CI only (osv-scan) |
A dependency vulnerability at or above the threshold; fails closed if the scan didn’t finish |
gitleaks-precommit.sh |
pre-commit + CI self-test | A secret in staged changes, before it’s ever committed |
The app-aware gates (migration numbering, schema-to-types parity, event reachability,
added-file coverage) skip cleanly when your project has nothing for them to check. CI’s
app-fixture-gates job proves them, and the edition-boundary scan, against a generated
sample app, since Blueprint’s own tree has no app.
Go deeper: the rest of this page explains how the gates are built and why.
The four properties every gate shares
Section titled “The four properties every gate shares”Copy these into any gate you add.
1. It can fail on demand, in its own job. Every script takes --self-test: it
synthesizes a violating fixture, runs the real decision path, and asserts a rejection.
The half that rots is discovery, not judgment — judgment either answers or errors
loudly, while a scan that stops matching fails open and silently. One real instance: a
CI image shipped BusyBox grep, which rejects --exclude-dir; the || true on the
scan swallowed the usage error and the gate reported “no matches found” on every tree,
forever, while a real violation sat in two tracked files.
The in its own job half is not a detail. Both gates that had gone blind upstream
already carried passing test suites — the suites ran in a job with GNU grep and the
gates ran in a job with BusyBox grep, so they were evidence about that image and
nothing else. Same job is the only way to say “same image” in a CI config, so each gate
runs --self-test first, in the job that runs the real scan, and
check-gate-selftest-parity.sh asserts that every gate does. Its only opt-out category
is EXTERNAL — the gate’s input is not the repository, so no fixture can represent a
violation. There is deliberately no “verified by hand once” category: upstream parked
twelve gates in one, and when each was later neutered on purpose, every one still passed
on the real tree, because a compliant tree never executes the detection path at all.
make gate-self-tests remains a local convenience; it is not what proves the property.
2. It fails closed. A check whose oracle is outside the repo (issue state, an
advisory feed, a published artifact) treats an unreachable oracle as a failure, with
one named, reviewable opt-out (ALLOW_UNRESOLVED=1). A gate that silently skips when it
cannot answer is a gate that reports success for the wrong reason. osv-severity-gate.sh
is the concrete instance: a missing or unparseable results file means OSV-Scanner’s own
scan did not complete (verified against a blocked network path — no partial file, no
output at all), so the gate blocks rather than passing silently, distinct from a genuinely
empty scan (no lockfiles in the tree), which is a clean pass. Its oracle being outside the
repo is also why it is the one gate here that cannot run in pre-push at all — see its
entry in scripts/check-prepush-parity.sh’s OPT_OUT.
3. It scans the repository, not the directory. scripts/lib/git-ignored.sh provides
is_ignored and drop_ignored_lines. Without them a gate reads ignored artifacts — local
reports, scratch dirs, downloaded fixtures — that CI’s clean clone never sees, producing a
false RED locally only, naming a real file with a real violation. The identical gate
passing in CI is the only tell, and nothing in the output points at it.
4. The mirror list is derived, not hand-kept. make pre-push-checks is only worth its green
if it covers what CI covers. A hand-maintained list is correct the day it is written and
silently incomplete after — and two Make targets one character apart will hide the hole
from anyone reading either. check-prepush-parity.sh reads the CI configuration and
fails when a gate script has neither a Makefile mirror nor a recorded reason it cannot
have one. Add new gates to the Makefile, not to a list inside the parity check.
Declaring a load-bearing call site
Section titled “Declaring a load-bearing call site”Most of the gates above hunt for something that should not be in the tree. check-load-bearing.sh
is the inverse: it asserts something is there. It exists for one failure class — a guard
wired in by a single line, where deleting that line leaves the whole suite green and the
guard simply stops guarding.
The shape recurs: a validator installed at the top of a shared test-setup helper, a hook
entry in a config file, an ignore filter inside a tree walk. Upstream, one
installSchemaGuard(page) call was what made ~290 Playwright specs validate their mock
payloads against the OpenAPI schema; all 290 pass without it.
Declare a line in load-bearing.declarations when all three hold:
- It wires in a guard, hook, filter or validator — the line causes the work to happen rather than doing it.
- There is exactly one of it. Losing one of ten is visible; losing the only one is not.
- Deleting it leaves the suite green. This one is a test, not an argument — delete the line and run the suite before declaring it. If something already fails, that something is the gate, and a declaration adds only ceremony.
path: hooks/pre-pushcall: if ! make pre-push-checks 2>&1; thenwhy: The only local invocation of every pre-push gate. Delete it and the hook still installs, still runs, and checks nothing.call: is matched as a fixed string anywhere on a line, so indentation is not part of it
and regex metacharacters are literal. why: is required: a declaration with no stated
reason fails the gate rather than standing as a silent switch. When the gate fires you
restore the call, move the declaration to the guard’s new home, or delete the stanza and
say in the MR what covers this now — all three are reviewable one-line diffs. The gate
cannot stop a removal; it makes one visible.
Why a manifest and not a comment on the line. The obvious alternative — annotate the call site itself and grep for the marker — cannot work, structurally: deleting the call deletes the marker, the gate finds zero markers, and zero markers is exactly what a repo that declared nothing looks like. It reports OK. A marker can only prove its own presence, and the event this gate exists to catch is an absence, so the declaration has to outlive the line it describes. That is also the manifest’s cost — a declaration living away from the code can rot in the other direction — so both rot directions a text scan can see are errors: a declared file that no longer exists, and a declared call that no longer appears in it. A declared path that git ignores is an error too, because it would resolve on a developer’s machine and vanish in CI’s clean clone.
What it cannot see is reachability: a declared call inside a module nobody imports any
more still passes, and the verdict line says so on every run. And a repo that has declared
nothing passes trivially — a state indistinguishable from a broken gate, which is why the
--self-test rather than the green is what proves the machinery works.
The gate ledger
Section titled “The gate ledger”Agent gates (architect, security-review, rbac-check, …) leave their outcome in the
MR’s ## Gates section, one machine-readable line each. It is the only record of gate
yield: without it, a gate that runs on every MR and never finds anything is
indistinguishable from one that catches real defects — both look like compliance.
/kaizen parses those lines across recent MRs and reports find-rate per gate, which is
what earns a gate a fast-path exemption or costs it its slot.
0 findings is a real outcome and must never be omitted; n/a (out of scope) and
skipped (the user declined it) are different states; and “applied but not actually
run” has no token — so run it. A gate marked n/a with a confident one-line
justification is, in practice, usually an unrun gate whose justification was guessed.
The rigor floor
Section titled “The rigor floor”Ships in 0.8
Some paths are too costly to get wrong to leave the audit’s depth to judgment: the release
scripts and gates, the /release and /pre-release skills, the security gates, the Claude
Code and git hooks and settings, the helpers each of those loads, and the floor’s own
checker. They are listed as globs in .claude/rigor.json. A diff that touches one runs at
strict: its ledger records rigor: strict (floor: <a path it touches>) under
change-class:, and completeness-check runs on Opus with a round 2 every time. The
reporting and ledger scripts (gate-yield.py, check-gate-ledger.py) sit in a lighter
tier, strict-light: rigor: strict-light (floor: <path>), the same lines except that
round 1 may run on Sonnet, with round 2 and the fix-diff still on Opus. A diff touching both
tiers runs at strict. The exact ledger lines each level needs are listed once, in
docs/harness/rigor.md; any of them may instead be skipped (<reason>), the user’s
decision, which the person merging sees. check-rigor-floor.py fails the MR otherwise. The
floor is read from the MR’s base, so an MR that removes a glob, or moves it to the lighter
tier, is still held to it. A main → production promotion is not judged
again; a production-first hotfix is.
Check a branch before you open the MR with python3 scripts/check-rigor-floor.py --list,
and a draft description with --file <description.md>. Add your migrations and auth code
in place of the two placeholder globs; make customize lists them. There is no level below
standard and no per-issue override yet.
To read one MR’s evidence at a glance, python3 scripts/mr-verdict.py --file <description.md>
(or the description on stdin) renders a short verdict from the ## Gates,
## Requirements and ## Test plan sections and nothing else: each gate’s result, n/a
and skipped with their reasons as separate states, any gate the declared change-class:
requires that the ledger never mentions, the MET / DEFERRED / MISSING tally, and one
overall line: FAIL, INCOMPLETE, PASS WITH SKIPS (clean apart from a gate the user
declined) or PASS. It parses the ledger with gate-yield.py’s own parser and strips
HTML comments first, so a ledger or table left inside the template’s comments is not
evidence. Every cause, as the script states it:
- FAIL: a Requirements status cell starts with
MISSING,BLOCKERorBLOCKERSin any case, or has one of those words in uppercase anywhere in it, soMET: but BLOCKER remainsfails andMET: ... a missing filedoes not; the status cell is the column headed Status, or the last column when no header names one. - INCOMPLETE: no
change-class:line, or one the fast-path table does not know. - INCOMPLETE: a malformed or unparseable
gate:line. - INCOMPLETE: an
n/aorskippedwith no reason, or only template placeholder text such as<why ...>as its reason (a barecompleteness-check/fix-diff — n/ais accepted, ascompleteness-check.mddocuments that form; a bareskippedis not). - INCOMPLETE: a gate the declared class requires that the ledger does not list (a
/<mode>line such ascompleteness-check/fix-diffdoes not stand in for its base gate). - INCOMPLETE: every required gate
n/aorskipped, so no required gate actually ran. - INCOMPLETE: no gate lines, unless the declared class requires no gate.
- INCOMPLETE: no
## Requirementstable on an MR that closes an issue, unless the declared class requires no gate. - INCOMPLETE: a Requirements row that still holds template placeholder text such as
<criterion>. - INCOMPLETE: a DEFERRED row with no issue reference, or whose only references are issues the MR closes (
#N,group/proj#Nand issue URLs are compared by number). - INCOMPLETE: a Requirements status other than MET, DEFERRED, MISSING or BLOCKER.
The status word is what is judged: a leading MISSING or BLOCKER fails in any case, and
past it only the uppercase word does, so evidence prose such as “a missing file” after
MET does not stop /mr with a false FAIL. The one trade-off: a lowercase “still
missing” after MET is not caught. A BLOCKER the user chose to ship with is recorded in the
table as BLOCKER: <what is unresolved and why it ships>, so it reads FAIL rather than
hiding behind MET.
The verdict’s Tests and scans line (test-plan boxes checked out of total, and the
ledger’s scan-type gates: names containing security, dependency, scan, sast, secret,
trivy, osv, sonar or mutation) is information only. Unchecked test-plan boxes are listed
and never change the overall line, so an MR with 0 of 4 boxes checked can still read
PASS: the evidence it judges is the Gates ledger and the Requirements table.
A known inconsistency in the project’s own rules. CLAUDE.md’s prose says
regression-check runs on any source-touching branch, but the fast-path table’s schema,
backend, feature and trust-boundary rows do not list it. The verdict takes the table as the
only authority for required gates, so it does not mark such an MR incomplete; when a
source-touching class’s row lacks regression-check and the ledger records none, the
verdict prints a note saying so, without changing its state. Reconciling the prose and the
table is for the maintainers of those rules.
To render the verdict for an MR that already exists:
glab api projects/:id/merge_requests/<n> | jq -r .description | python3 scripts/mr-verdict.py. /mr runs it on the draft description before
creating the MR, stops on FAIL, and prints the full verdict to the author in its report.
No CI job posts it as an MR note, and /mr does not paste it into the description, so a
reviewer sees it only when the author shares it. It reads what was recorded and re-runs
nothing, so a fabricated 0 findings still reads as PASS.
How the tests test themselves
Section titled “How the tests test themselves”Every layer here answers a question the layer below it cannot. The point of the stack is not more coverage — it is that each layer’s green means something specific, and each one has a known way of lying that the next layer catches.
| Layer | Answers | Lies by |
|---|---|---|
| Unit + integration tests | Does this code do what I meant? | Passing on the unfixed build |
| E2E | Does the assembled product work? | Asserting on a stale bundle, or racing a refetch |
| Gate self-tests | Can the checks still detect anything? | — this is the layer that catches the others |
| Fuzzing | What did nobody think to write a test for? | Reporting clean for operations it never reached |
| Static analysis (Sonar / CodeQL) | What decays slowly across the whole tree? | Suppressions whose globs stopped matching |
1. Watch the test fail
Section titled “1. Watch the test fail”The base of the whole stack, and the cheapest. Revert the fix, run the new test,
require red, reapply. A test that has never been seen failing is an assertion about
the author’s intent. .claude/rules/tests.md catalogues the four vacuous shapes that
pass review and pass on the broken build — absence assertions that sample before the
action fires, key sequences that cancel themselves, name matchers that bind to a
neighboring node, parameterized guards whose parameter is ignored.
2. The gates test themselves
Section titled “2. The gates test themselves”Every scripts/check-*.sh ships a --self-test that synthesizes a violating fixture and
asserts a rejection, and CI runs it in the gate’s own job, before the real scan.
This exists because a gate’s silent failure mode is reporting OK forever — upstream, a
CI image whose BusyBox grep rejects --exclude-dir turned a || true scan into a
permanent clean report while a real violation sat in two tracked files. A self-test in a
different job would not have caught it: the suites existed and passed, on the other
image. scripts/check-gate-selftest-parity.sh is what keeps the rule from being prose.
3. Fuzzing — and a gate on the fuzzer
Section titled “3. Fuzzing — and a gate on the fuzzer”Contract fuzzing drives every documented API operation with generated input and asserts the responses match the schema. It finds the 500 nobody wrote a test for, because nobody imagined the input.
Two design decisions matter more than the fuzzer itself:
- It is scheduled and non-gating. A fuzz finding is a triage signal, not a merge
blocker — an unbounded generator on every MR trains people to ignore it. It runs
nightly with
allow_failure: true. - A coverage gate sits on top of the fuzzer’s own output. Nothing else distinguishes
“581 operations fuzzed, all clean” from “580 fuzzed, 1 never ran”. An operation
that errors gets zero checks and the run still ends green — so a separate script
reads the fuzzer’s JUnit report against the API schema and fails when an operation was
never actually exercised. That script has its own
--self-test, run in the same job.
4. Static analysis, and a gate on the suppressions
Section titled “4. Static analysis, and a gate on the suppressions”Sonar and CodeQL catch the slow decay a diff review cannot see. Both have the same structural weakness: the suppression list rots silently.
- A Sonar exclusion is a reviewed false positive — but nothing checks its glob still
points at anything. Upstream, a rename moved
useProjectChangelog.ts→.tsx, anactivity/*.tspattern stopped matching, four suppressed findings resurfaced, and the reliability rating fell A → D with no job going red. The fix is a pure path-matching gate (no network, no token, under a second) that runs on every MR — deliberately not tied to the scan itself, which is scheduled-only and non-gating. - Inline
// codeql[rule-id]comments suppress nothing. They are greppable human notes; the findings they annotate stay open. Under default setup there is no query-filter lever either — the only mechanism that clears a false positive is dismissing it in the UI. - A scheduled-only scan is invisible to every pre-merge gate, and its findings still read as open at pre-fix line numbers after the fix merges. Check when the last analysis actually ran before believing any number from it.
5. What none of it covers
Section titled “5. What none of it covers”Say the scope out loud, or a 0 findings gets read as safety. A dependency scanner reads
the lockfile — so a browser binary a test framework downloads at install time is outside
it entirely, and reports clean because it was never looked at. scripts/CLAUDE.md
makes this a rule: if a check bounds its own coverage, log what it dropped.