Harness gates — checks that can prove they still bite

A gate is a script that says yes or no about the tree, and its entire value is that a green result is evidence. Every rule in scripts/CLAUDE.md exists because a gate reported OK on a tree that violated it — which is worse than having no gate, because the green was believed.

Gate	Runs	What it catches
scripts/check-issue-collision.sh	pre-push	A second MR for an issue another session already claimed
scripts/check-prepush-parity.sh	pre-push + CI	A CI gate a developer cannot run before pushing
scripts/check-stale-references.sh	CI (MR, default branch, schedule)	STUB/WIP markers; TODO(#N) and SUPPRESSED-UNTIL(#N) pointing at closed issues
scripts/check-gate-selftest-parity.sh	pre-push + CI	A gate that ships without a --self-test, or whose self-test runs in a different job
scripts/check-version-lockstep.py	pre-push + CI (MR, default branch)	A version-bearing manifest the release script’s bump list forgot — before it ships stamped wrong
scripts/check-release-pipeline.sh	CI (tag)	A tag publishing from a commit whose own branch pipeline failed
scripts/check-sigpipe-readers.sh	pre-push + CI (MR, default branch)	An early-exit reader behind a pipe, which under pipefail reports a present match as missing

Four properties they all share — copy them into any gate you add:

1. It can fail on demand, in its own job. Every script takes --self-test: it synthesizes a violating fixture, runs the real decision path, and asserts a rejection. The half that rots is discovery, not judgment — judgment either answers or errors loudly, while a scan that stops matching fails open and silently. One real instance: a CI image shipped BusyBox grep, which rejects --exclude-dir; the || true on the scan swallowed the usage error and the gate reported “no matches found” on every tree, forever, while a real violation sat in two tracked files.

The in its own job half is not a detail. Both gates that had gone blind upstream already carried passing test suites — the suites ran in a job with GNU grep and the gates ran in a job with BusyBox grep, so they were evidence about that image and nothing else. Same job is the only way to say “same image” in a CI config, so each gate runs --self-test first, in the job that runs the real scan, and check-gate-selftest-parity.sh asserts that every gate does. Its only opt-out category is EXTERNAL — the gate’s input is not the repository, so no fixture can represent a violation. There is deliberately no “verified by hand once” category: upstream parked twelve gates in one, and when each was later neutered on purpose, every one still passed on the real tree, because a compliant tree never executes the detection path at all. make gate-self-tests remains a local convenience; it is not what proves the property.

2. It fails closed. A check whose oracle is outside the repo (issue state, an advisory feed, a published artifact) treats an unreachable oracle as a failure, with one named, reviewable opt-out (ALLOW_UNRESOLVED=1). A gate that silently skips when it cannot answer is a gate that reports success for the wrong reason.

3. It scans the repository, not the directory. scripts/lib/git-ignored.sh provides is_ignored and drop_ignored_lines. Without them a gate reads ignored artifacts — local reports, scratch dirs, downloaded fixtures — that CI’s clean clone never sees, producing a false RED locally only, naming a real file with a real violation. The identical gate passing in CI is the only tell, and nothing in the output points at it.

4. The mirror list is derived, not hand-kept. make pre-push is only worth its green if it covers what CI covers. A hand-maintained list is correct the day it is written and silently incomplete after — and two Make targets one character apart will hide the hole from anyone reading either. check-prepush-parity.sh reads the CI configuration and fails when a gate script has neither a Makefile mirror nor a recorded reason it cannot have one. Add new gates to the Makefile, not to a list inside the parity check.

The gate ledger

Agent gates (architect, security-review, rbac-check, …) leave their outcome in the MR’s ## Gates section, one machine-readable line each. It is the only record of gate yield: without it, a gate that runs on every MR and never finds anything is indistinguishable from one that catches real defects — both look like compliance. /kaizen parses those lines across recent MRs and reports find-rate per gate, which is what earns a gate a fast-path exemption or costs it its slot.

0 findings is a real outcome and must never be omitted; n/a (out of scope) and skipped (the user declined it) are different states; and “applied but not actually run” has no token — so run it. A gate marked n/a with a confident one-line justification is, in practice, usually an unrun gate whose justification was guessed.

How the tests test themselves

Every layer here answers a question the layer below it cannot. The point of the stack is not more coverage — it is that each layer’s green means something specific, and each one has a known way of lying that the next layer catches.

Note

What ships and what doesn’t. The unit/integration-test rules and a --self-test on every gate are in the template today. Blueprint ships no E2E suite, fuzzer, or static-analysis job, because those depend on your stack. The fuzzing and static-analysis sections below describe how to add them so their results can be trusted, and what went wrong in projects that added them without these safeguards.

Layer	Answers	Lies by
Unit + integration tests	Does this code do what I meant?	Passing on the unfixed build
E2E	Does the assembled product work?	Asserting on a stale bundle, or racing a refetch
Gate self-tests	Can the checks still detect anything?	— this is the layer that catches the others
Fuzzing	What did nobody think to write a test for?	Reporting clean for operations it never reached
Static analysis (Sonar / CodeQL)	What decays slowly across the whole tree?	Suppressions whose globs stopped matching

1. Watch the test fail

The base of the whole stack, and the cheapest. Revert the fix, run the new test, require red, reapply. A test that has never been seen failing is an assertion about the author’s intent. tests/CLAUDE.md.example catalogues the four vacuous shapes that pass review and pass on the broken build — absence assertions that sample before the action fires, key sequences that cancel themselves, name matchers that bind to a neighboring node, parameterized guards whose parameter is ignored.

2. The gates test themselves

Every scripts/check-*.sh ships a --self-test that synthesizes a violating fixture and asserts a rejection, and CI runs it in the gate’s own job, before the real scan. This exists because a gate’s silent failure mode is reporting OK forever — upstream, a CI image whose BusyBox grep rejects --exclude-dir turned a || true scan into a permanent clean report while a real violation sat in two tracked files. A self-test in a different job would not have caught it: the suites existed and passed, on the other image. scripts/check-gate-selftest-parity.sh is what keeps the rule from being prose.

3. Fuzzing — and a gate on the fuzzer

Contract fuzzing drives every documented API operation with generated input and asserts the responses match the schema. It finds the 500 nobody wrote a test for, because nobody imagined the input.

Two design decisions matter more than the fuzzer itself:

• It is scheduled and non-gating. A fuzz finding is a triage signal, not a merge blocker — an unbounded generator on every MR trains people to ignore it. It runs nightly with allow_failure: true.
• A coverage gate sits on top of the fuzzer’s own output. Nothing else distinguishes “581 operations fuzzed, all clean” from “580 fuzzed, 1 never ran”. An operation that errors gets zero checks and the run still ends green — so a separate script reads the fuzzer’s JUnit report against the API schema and fails when an operation was never actually exercised. That script has its own --self-test, run in the same job.

Caution

The honest caveat, and you must internalize it: allow_failure: true means the pipeline reads success while the job inside it failed. Upstream, the API fuzz job failed 14 of 21 consecutive nightlies, every pipeline green. Never conclude anything about fuzzing from a pipeline status — query the jobs, and read the artifact. An <error> result (never ran) and a <failure> result (found something) are completely different answers.

4. Static analysis, and a gate on the suppressions

Sonar and CodeQL catch the slow decay a diff review cannot see. Both have the same structural weakness: the suppression list rots silently.

• A Sonar exclusion is a reviewed false positive — but nothing checks its glob still points at anything. Upstream, a rename moved useProjectChangelog.ts → .tsx, an activity/*.ts pattern stopped matching, four suppressed findings resurfaced, and the reliability rating fell A → D with no job going red. The fix is a pure path-matching gate (no network, no token, under a second) that runs on every MR — deliberately not tied to the scan itself, which is scheduled-only and non-gating.
• Inline // codeql[rule-id] comments suppress nothing. They are greppable human notes; the findings they annotate stay open. Under default setup there is no query-filter lever either — the only mechanism that clears a false positive is dismissing it in the UI.
• A scheduled-only scan is invisible to every pre-merge gate, and its findings still read as open at pre-fix line numbers after the fix merges. Check when the last analysis actually ran before believing any number from it.

5. What none of it covers

Say the scope out loud, or a 0 findings gets read as safety. A dependency scanner reads the lockfile — so a browser binary a test framework downloads at install time is outside it entirely, and reports clean because it was never looked at. scripts/CLAUDE.md makes this a rule: if a check bounds its own coverage, log what it dropped.