External validation
The big picture: every gate the harness runs is configured by the same people, and the same agent, that wrote the code. External validators are tools you don’t control. They give a second opinion, and for a public project they give visitors a score they can check themselves.
Why it matters: an AI-assisted project can look clean to its own gates. A scanner with its own rules, its own history of your code, and a public dashboard is much harder to quietly argue with.
How it works:
- SonarQube Cloud scans the whole project for bugs, vulnerabilities, code smells, duplication, and coverage, not just the lines in an MR.
- Secret, dependency, and static-analysis scanners each catch something the others can’t.
- OpenSSF Scorecard grades supply-chain hygiene once the project is on GitHub.
SonarQube Cloud
Section titled “SonarQube Cloud”SonarQube Cloud (formerly SonarCloud) is SonarSource’s hosted service. Its plans are priced by lines of code (LOC) in private projects:
- Private projects, free tier: up to 50,000 LOC. Beyond that you pay (Team plan from $34/month for 100k LOC at the time of writing).
- Public open-source projects: SonarSource’s pricing page says only private-project LOC count toward the limit, and lists no cap for public ones. The whole codebase is analyzed, with the history of findings and a quality gate you can show as a badge.
Checked against sonarsource.com/plans-and-pricing on 2026-10-05. Plan terms change, so recheck before relying on them.
Set it up on GitLab:
- Sign in to SonarQube Cloud with your GitLab account and import the project. This creates the project key and organization.
- Add
sonar-project.propertiesat the repo root withsonar.projectKey,sonar.organization, and your source folders. - Store a token as the masked CI variable
SONAR_TOKEN. - Add a CI job that runs the
sonar-scanner-cliimage. GitLab projects need this job; the hosted “automatic analysis” shortcut is GitHub-only.
Sonar doesn’t run your tests. It imports coverage reports your test jobs already
produce. Point sonar.python.coverage.reportPaths (or the JavaScript equivalent) at them,
and make the scan job needs: the test jobs so the reports exist.
Run it nightly, not on every push. TruePPM scans on a schedule with
allow_failure, so a Sonar or network hiccup never turns the pipeline red and MR
latency doesn’t change. Sonar keeps one analysis per project, so the nightly result is
the dashboard. TruePPM’s lint:sonar-exclusions check also fails when an excluded
false-positive rule points at a path that no longer exists, so the exclusion list can’t
rot.
Check that your ignore criteria still do something. A
sonar.issue.ignore.multicriteria.<id>.ruleKey that is misspelled, was removed, or is
not active in your quality profile silently ignores nothing, and nothing goes red.
scripts/check-sonar-exclusions.sh asks the Sonar API about each key and fails when one
does not exist (HTTP 404) or is active in none of the project’s profiles. It is optional:
uncomment - local: ci/sonar-exclusions.yml in .gitlab-ci.yml and set a masked CI
variable SONAR_TOKEN (SONAR_HOST_URL for self-hosted SonarQube). Its design choices:
- It fails closed, like every check whose oracle is outside the repo. If it cannot
reach or understand the Sonar API (no token, network error, 3xx or 5xx, an unparseable
answer, a 404 that lacks Sonar’s own error body, usually a wrong
SONAR_HOST_URL), it exits 2 “unresolved”.ALLOW_UNRESOLVED=1is the one explicit, reviewable opt-out: a loud warning and exit 0. - It exits 1 on a key the API reports as missing or inactive, on HTTP 401/403 (a
revoked, expired, or underprivileged token is a configuration error), on a malformed
key, on a
multicriteriaproperty it cannot parse, and when criteria exist but thesonar.issue.ignore.multicriteria=e1,e2index line is missing (Sonar then ignores every criterion). - Protected-variable trap: if
SONAR_TOKENis a GitLab protected variable, merge request pipelines on unprotected branches never see it and the job exits 2. Make the variable unprotected (masked is fine), or setALLOW_UNRESOLVED=1on purpose. - The offline checks (parsing, key format, index line) run first and need no token.
Every Java-properties spelling is read (
=,:or whitespace separators, dashed or dotted ids, module-prefixed lines, continuation lines, CRLF). - It looks up each key with
api/rules/show, not the bulkapi/rules/search, which can omit a rule that is in fact active. - Wildcard keys (
python:*,?) are reported and skipped; the summary counts only keys it actually verified, and a run that verified none saysSKIPPED, not “ok”. sonar.organizationis optional, so self-hosted SonarQube works: setSONAR_HOST_URLand leave the organization out.- It warns, but does not fail, when an id is defined but not listed in the
sonar.issue.ignore.multicriteria=e1,e2index, or listed but not defined. - Limits: it reads only top-level
sonar.issue.ignore.*lines in one properties file. Criteria set in the Sonar UI, via-Dscanner arguments, or in other modules’ files are never seen.sonar.issue.enforce.multicriteria.*is parsed but not verified, and it does not check whether a rule still fires or aresourceKeyglob still matches a file. - The job is not
allow_failure; the opt-in is the include line. Its--self-testruns offline against a stubcurlin the same job before the real check.
Not the same thing: Blueprint’s ci/sonar-complexity.yml approximates one Sonar
rule (cognitive complexity) with a local script. It does not contact SonarQube Cloud.
Using a scanner to keep AI slop out
Section titled “Using a scanner to keep AI slop out”AI-written code fails in quiet ways: duplicated blocks, dead branches, swallowed errors, needless complexity, and copy-pasted insecure patterns. A scanner with a quality gate on new code turns these from review nitpicks into a failing check.
- Gate on new code (what this MR or this week added), not the whole backlog, so the bar rises without a cleanup project first.
- Watch duplication and cognitive complexity; they climb fastest when an agent adds code instead of reusing it.
- SonarSource also offers an MCP server and CLI so a coding agent can run the same analysis while it writes. Treat that as optional; the nightly scan is the independent check.
- Scanners are not the only option. Semgrep (custom rules for your own repeated mistakes) and CodeQL (free on public GitHub repositories) cover overlapping ground. Pick one or two, and read the findings, not the badge.
The rest of the scanner stack
Section titled “The rest of the scanner stack”| Layer | Catches | Blueprint ships | TruePPM adds |
|---|---|---|---|
| Secret scanning (pre-commit and CI) | Keys and tokens committed by mistake | GitLab secret-detection component, gitleaks-scan, .gitleaks.toml |
security:gitleaks |
| Static analysis (SAST) | Insecure code patterns | GitLab SAST component | security:semgrep with project rules |
| Dependency CVEs | Known-vulnerable packages | osv-scan |
security:osv, security:trivy:config |
| Licenses | A dependency whose license you can’t ship | python-license-check, node-license-check in ci/ |
license:check:* per package |
Blueprint ships the scanners in the table’s “Blueprint ships” column. SonarQube Cloud and Scorecard are described here but not wired in (only an opt-in check of Sonar ignore-criteria rule keys, ci/sonar-exclusions.yml, ships). Which scanners block an MR is set per job; check each job’s allow_failure and rules in your pipeline. TruePPM runs Sonar on a schedule and the others in the security stage.
Each layer misses what the others see, which is why they stack rather than replace each other. Secret scanning ignores bad code; SAST ignores a leaked key.
None of these scanners boots your image in a browser-like request path. For a single-page-app container, the served-assets smoke test is the check that fetches the JS and CSS the image claims to serve.
Making SAST and secret detection block (optional module)
Section titled “Making SAST and secret detection block (optional module)”The SAST and secret-detection components in .gitlab-ci.yml report; no finding turns
the pipeline red on its own. The optional module ci/scanner-gates.yml adds the blocking
half. It is off by default (uncomment - local: ci/scanner-gates.yml), so enabling it is
your decision and Blueprint’s own pipeline does not depend on it.
What ships:
scripts/sast-severity-gate.pyandscripts/secret-detection-gate.pyread the GitLab report (gl-sast-report.json,gl-secret-detection-report.json) and fail on any unsuppressed finding at or above a threshold (SAST_SEVERITY_THRESHOLD, defaultHIGH;SECRET_SEVERITY_THRESHOLD, defaultLOW, because a leaked credential is a finding whatever label it carries). Findings only below the threshold exit 2, shown as a warning rather than a green pass.- They fail closed. A missing, empty or non-GitLab report is a failure, not a clean scan,
and a finding with an
Unknownseverity blocks.ALLOW_UNRESOLVED=1is the one named opt-out for a missing report. - Suppressions live in
.sast-suppressions.json/.secret-suppressions.json. Each entry needs a reason and anexpiresdate at most 180 days out (SCANNER_SUPPRESSION_MAX_DAYS); an expired entry stops suppressing and the finding blocks again. Scope an entry withpath; one without it applies repo-wide, and the gate says so. A leaked secret still needs rotating: a suppression is for a documented fake, not a real key. - Each gate runs its
--self-testfirst in its own CI job, and both are inmake pre-push-checks. The raw secret is never printed. - A
gitleaks-historyjob scans the full commit history (the component only sees a diff), with gitleaks pinned by version and sha256.
What is advice, not shipped:
- Bumping the SAST and secret-detection components. The core pipeline pins them to exact
versions (
sast@3.5.0,secret-detection@2.4.0) andcheck-ci-image-pinsrejects@~latestor any other moving ref; bump them deliberately, and pin the same way in your project. - The job names the gates depend on (
semgrep-sast,secret_detection) are the components’ defaults. Edit theneedsif yours differ, and add one gate job per extra SAST analyzer (they all writegl-sast-report.json). - The thresholds. Start from the defaults, then tighten as the backlog reaches zero.
Not verified: the gates were exercised through their self-tests and synthetic reports, not inside a real pipeline with the components attached. Your first MR pipeline is the check.
Expiring Trivy suppressions
Section titled “Expiring Trivy suppressions”What ships:
scripts/check-trivyignore-expiry.shand the opt-in moduleci/trivyignore-expiry.yml. What is advice: everything about running Trivy itself. Blueprint ships no Trivy job, so no scan blocks your pipeline until you add one.
A .trivyignore line with no end date is a decision nobody revisits: it keeps hiding a
finding after a fix ships or the package is dropped. Write each suppression in Trivy’s own
syntax, with an expiry:
CVE-2024-12345 exp:2026-12-31 # waiting on upstream 2.4.1; owner: platformThe script fails on an entry that is undated, malformed (not a real calendar
date, or two exp: tokens), expired (the day of expiry itself still passes), or
too far out (more than TRIVYIGNORE_MAX_DAYS, default 180, from today). A missing
.trivyignore passes. To adopt it, uncomment - local: ci/trivyignore-expiry.yml in
.gitlab-ci.yml; the job runs only when a .trivyignore exists. Its --self-test runs
first in the same job, and make gate-self-tests runs it locally.
- It reads the repository, so it has no network oracle and no
ALLOW_UNRESOLVEDescape hatch. The pipeline goes red when the calendar passes an entry’s date, with no diff to blame. That is the check working: fix the finding, or renew the date with a reason in the same line. - It only reads the plain-text
.trivyignore. The YAML form (.trivyignore.yaml) and suppressions kept elsewhere (VEX documents, a scanner UI) are not seen. - Advice, not shipped: pin the scanner by digest (see Pinning CI images and tools), and make the Trivy job itself blocking on HIGH and CRITICAL findings, so an expired suppression surfaces the finding again rather than only a red expiry check.
Pinning CI images and tools
Section titled “Pinning CI images and tools”A tag such as alpine:3.23 moves: whoever pushes next decides what your pipeline runs, with
your CI variables in its environment. A digest (alpine:3.23@sha256:...) is a content hash,
so the pull fails if the bytes change. The tag stays for humans; the digest is what runs.
What ships:
- Every
image:andservices:entry in.gitlab-ci.ymlandci/*.ymlis pinned by digest, and the end-of-life bases were replaced with supported ones (Alpine 3.19 to 3.23, Node 20 to 22, Go 1.23 to 1.26, Docker 27 to 29). - Tools downloaded in a job are verified against a published sha256 before they run:
osv-scanner, gitleaks, kind, kubectl and helm.
govulncheckis installed at a fixed version instead of@latest. scripts/check-ci-image-pins.py(jobci-yaml-lint,make check-ci-yaml) fails on animage:orservices:entry without a 64-hex@sha256:digest, on a reference that holds a CI variable (except a whole$NAMEreference whose top-levelvariables:value is pinned by digest, names a trusted repository (stock alpine or this project’s ownci-baseimage; see the CI pipeline guide, Making the pipeline fast), and that no job overrides in its ownvariables:block), and ongo install ...@latest. Its--self-testruns first in the same job. Exceptions go inscripts/ci-image-pins.allow, one reference per line with a reason; a stale entry fails, so the list cannot become a blanket exemption.
Bumping a pin: resolve the digest from the registry yourself
(docker buildx imagetools inspect alpine:3.23) and change the tag and digest together. For a
downloaded tool, change the version and its sha256 together, taking the hash from the
publisher’s checksum file. If you override KIND_VERSION, KUBECTL_VERSION or HELM_VERSION
in ci/helm.yml, override the matching *_SHA256 too, or the job stops at the checksum.
What is advice, not shipped: a digest freezes the image, so it stops receiving security
fixes until you bump it. The repository’s renovate.json already carries a digest rule;
add :pinDigests to its extends if you want Renovate to raise the digest bumps as
reviewable MRs. Not covered: pip install and npx calls inside jobs (unpinned, or pinned by
version without hashes), the golangci-lint v1 image (v2 needs a config migration; v1.61 is
built with Go 1.23, so it cannot lint a module whose go directive is newer, while go-test
now runs Go 1.26, so move to v2 once your module needs it), the Kubernetes and Helm versions
the drill downloads (older than current), and the GitLab SAST and secret-detection components
pinned at sast@3.5.0 and secret-detection@2.4.0, which you bump deliberately and pin the same way in your project.
Signed images and SBOM attestations (optional)
Section titled “Signed images and SBOM attestations (optional)”A scanner says what is in an image. It does not tell a user that the image they pulled is the one your pipeline built, or that it is still the same a year later. Signing and SBOM attestation close that gap.
What ships (all optional, off by default):
ci/image-signing.yml, an opt-in module. Uncomment- local: ci/image-signing.ymlin.gitlab-ci.yml. It does nothing until you setSIGN_IMAGE_REFS(what to sign) orRELEASE_IMAGES(what to verify), so it cannot fail a clone that has no images.scripts/attest-image-sbom.sh, run from a tag pipeline. It resolves each image to a digest, signs the index and every platform manifest with cosign, and attaches a CycloneDX SBOM attestation to each architecture (generated per platform with syft, not copied across). It refuses to attest an empty or non-CycloneDX SBOM.scripts/check-release-attestations.sh, run by a pipeline schedule you create. It checks that the newest stable releases still carry their signatures and SBOM attestations. Its--self-testruns first in the same job, against stubcosignanddocker, and proves the check goes red on a missing signature, a missing SBOM on one architecture, a deleted tag, and an unreachable registry.scripts/lib/supply-chain-pins.sh, the single pin (version plus sha256) for cosign and syft. A Helm chart publish job can source the same file so charts and images are signed and verified by one cosign.
What is advice, not shipped: enabling it in your pipeline, choosing keyless or key-based signing, writing the identity regexp, setting the schedule, and making your registry keep the signature artifacts. Verifying signatures on the consumer side (an admission controller such as Kyverno or Sigstore policy-controller) is not covered.
Keys. The default is keyless: GitLab issues a short-lived OIDC token
(id_tokens: SIGSTORE_ID_TOKEN with aud: sigstore) and there is no key to leak.
Keyless signing writes the project path and ref to Sigstore’s public transparency log, so
use key-based signing (COSIGN_KEY as a masked, protected file variable) for a private
project that cannot allow that. Never commit a key.
Set the identity. In keyless mode the check requires SIGNING_CERT_IDENTITY_REGEXP
to name your project and tag refs. An empty or match-anything value is refused, because
it would accept anyone’s signature.
Read this before the first scheduled run:
- Older releases were never signed. Without
ATTESTATION_MIN_VERSIONset to the first signed release, the first run fails on every earlier release. Signing them after the fact would prove nothing about who built them. - It fails closed. The oracle is your registry plus Sigstore (Rekor). If either
cannot be reached, or no release tags are fetched (a shallow clone), the job exits 2
and goes red.
ALLOW_UNRESOLVED=1is the one explicit opt-out; it never hides a missing signature (exit 1). - Registry retention can delete proof. A registry that garbage-collects untagged
artifacts may remove the
.sigand.attentries. The first run is when you learn that. - Coverage is stated, not implied. The verdict line counts the tags, signatures and attestations it verified. It does not judge whether an SBOM is complete or the image is free of vulnerabilities.
- Run
--listand then the real check by hand once, and create the schedule only after one signed release exists.
OpenSSF Scorecard
Section titled “OpenSSF Scorecard”Scorecard grades a GitHub repository’s hygiene: branch protection, pinned dependencies, signed releases, a security policy, and more. It needs the project on GitHub, so it comes after you publish a GitHub mirror. The score is public, so what it checks is worth fixing before you go public.
Bottom line: let tools you don’t control grade the project. Start with SonarQube Cloud, since public projects get the whole-project scan for free.