Skip to content

External validation

The big picture: every gate the harness runs is configured by the same people, and the same agent, that wrote the code. External validators are tools you don’t control. They give a second opinion, and for a public project they give visitors a score they can check themselves.

Why it matters: an AI-assisted project can look clean to its own gates. A scanner with its own rules, its own history of your code, and a public dashboard is much harder to quietly argue with.

How it works:

  • SonarQube Cloud scans the whole project for bugs, vulnerabilities, code smells, duplication, and coverage, not just the lines in an MR.
  • Secret, dependency, and static-analysis scanners each catch something the others can’t.
  • OpenSSF Scorecard grades supply-chain hygiene once the project is on GitHub.

SonarQube Cloud (formerly SonarCloud) is SonarSource’s hosted service. Its plans are priced by lines of code (LOC) in private projects:

  • Private projects, free tier: up to 50,000 LOC. Beyond that you pay (Team plan from $34/month for 100k LOC at the time of writing).
  • Public open-source projects: SonarSource’s pricing page says only private-project LOC count toward the limit, and lists no cap for public ones. The whole codebase is analyzed, with the history of findings and a quality gate you can show as a badge.

Checked against sonarsource.com/plans-and-pricing on 2026-10-05. Plan terms change, so recheck before relying on them.

Set it up on GitLab:

  1. Sign in to SonarQube Cloud with your GitLab account and import the project. This creates the project key and organization.
  2. Add sonar-project.properties at the repo root with sonar.projectKey, sonar.organization, and your source folders.
  3. Store a token as the masked CI variable SONAR_TOKEN.
  4. Add a CI job that runs the sonar-scanner-cli image. GitLab projects need this job; the hosted “automatic analysis” shortcut is GitHub-only.

Sonar doesn’t run your tests. It imports coverage reports your test jobs already produce. Point sonar.python.coverage.reportPaths (or the JavaScript equivalent) at them, and make the scan job needs: the test jobs so the reports exist.

Run it nightly, not on every push. TruePPM scans on a schedule with allow_failure, so a Sonar or network hiccup never turns the pipeline red and MR latency doesn’t change. Sonar keeps one analysis per project, so the nightly result is the dashboard. TruePPM’s lint:sonar-exclusions check also fails when an excluded false-positive rule points at a path that no longer exists, so the exclusion list can’t rot.

Check that your ignore criteria still do something. A sonar.issue.ignore.multicriteria.<id>.ruleKey that is misspelled, was removed, or is not active in your quality profile silently ignores nothing, and nothing goes red. scripts/check-sonar-exclusions.sh asks the Sonar API about each key and fails when one does not exist (HTTP 404) or is active in none of the project’s profiles. It is optional: uncomment - local: ci/sonar-exclusions.yml in .gitlab-ci.yml and set a masked CI variable SONAR_TOKEN (SONAR_HOST_URL for self-hosted SonarQube). Its design choices:

  • It fails closed, like every check whose oracle is outside the repo. If it cannot reach or understand the Sonar API (no token, network error, 3xx or 5xx, an unparseable answer, a 404 that lacks Sonar’s own error body, usually a wrong SONAR_HOST_URL), it exits 2 “unresolved”. ALLOW_UNRESOLVED=1 is the one explicit, reviewable opt-out: a loud warning and exit 0.
  • It exits 1 on a key the API reports as missing or inactive, on HTTP 401/403 (a revoked, expired, or underprivileged token is a configuration error), on a malformed key, on a multicriteria property it cannot parse, and when criteria exist but the sonar.issue.ignore.multicriteria=e1,e2 index line is missing (Sonar then ignores every criterion).
  • Protected-variable trap: if SONAR_TOKEN is a GitLab protected variable, merge request pipelines on unprotected branches never see it and the job exits 2. Make the variable unprotected (masked is fine), or set ALLOW_UNRESOLVED=1 on purpose.
  • The offline checks (parsing, key format, index line) run first and need no token. Every Java-properties spelling is read (=, : or whitespace separators, dashed or dotted ids, module-prefixed lines, continuation lines, CRLF).
  • It looks up each key with api/rules/show, not the bulk api/rules/search, which can omit a rule that is in fact active.
  • Wildcard keys (python:*, ?) are reported and skipped; the summary counts only keys it actually verified, and a run that verified none says SKIPPED, not “ok”.
  • sonar.organization is optional, so self-hosted SonarQube works: set SONAR_HOST_URL and leave the organization out.
  • It warns, but does not fail, when an id is defined but not listed in the sonar.issue.ignore.multicriteria=e1,e2 index, or listed but not defined.
  • Limits: it reads only top-level sonar.issue.ignore.* lines in one properties file. Criteria set in the Sonar UI, via -D scanner arguments, or in other modules’ files are never seen. sonar.issue.enforce.multicriteria.* is parsed but not verified, and it does not check whether a rule still fires or a resourceKey glob still matches a file.
  • The job is not allow_failure; the opt-in is the include line. Its --self-test runs offline against a stub curl in the same job before the real check.

Not the same thing: Blueprint’s ci/sonar-complexity.yml approximates one Sonar rule (cognitive complexity) with a local script. It does not contact SonarQube Cloud.

AI-written code fails in quiet ways: duplicated blocks, dead branches, swallowed errors, needless complexity, and copy-pasted insecure patterns. A scanner with a quality gate on new code turns these from review nitpicks into a failing check.

  • Gate on new code (what this MR or this week added), not the whole backlog, so the bar rises without a cleanup project first.
  • Watch duplication and cognitive complexity; they climb fastest when an agent adds code instead of reusing it.
  • SonarSource also offers an MCP server and CLI so a coding agent can run the same analysis while it writes. Treat that as optional; the nightly scan is the independent check.
  • Scanners are not the only option. Semgrep (custom rules for your own repeated mistakes) and CodeQL (free on public GitHub repositories) cover overlapping ground. Pick one or two, and read the findings, not the badge.
Layer Catches Blueprint ships TruePPM adds
Secret scanning (pre-commit and CI) Keys and tokens committed by mistake GitLab secret-detection component, gitleaks-scan, .gitleaks.toml security:gitleaks
Static analysis (SAST) Insecure code patterns GitLab SAST component security:semgrep with project rules
Dependency CVEs Known-vulnerable packages osv-scan security:osv, security:trivy:config
Licenses A dependency whose license you can’t ship python-license-check, node-license-check in ci/ license:check:* per package

Blueprint ships the scanners in the table’s “Blueprint ships” column. SonarQube Cloud and Scorecard are described here but not wired in (only an opt-in check of Sonar ignore-criteria rule keys, ci/sonar-exclusions.yml, ships). Which scanners block an MR is set per job; check each job’s allow_failure and rules in your pipeline. TruePPM runs Sonar on a schedule and the others in the security stage.

Each layer misses what the others see, which is why they stack rather than replace each other. Secret scanning ignores bad code; SAST ignores a leaked key.

None of these scanners boots your image in a browser-like request path. For a single-page-app container, the served-assets smoke test is the check that fetches the JS and CSS the image claims to serve.

Making SAST and secret detection block (optional module)

Section titled “Making SAST and secret detection block (optional module)”

The SAST and secret-detection components in .gitlab-ci.yml report; no finding turns the pipeline red on its own. The optional module ci/scanner-gates.yml adds the blocking half. It is off by default (uncomment - local: ci/scanner-gates.yml), so enabling it is your decision and Blueprint’s own pipeline does not depend on it.

What ships:

  • scripts/sast-severity-gate.py and scripts/secret-detection-gate.py read the GitLab report (gl-sast-report.json, gl-secret-detection-report.json) and fail on any unsuppressed finding at or above a threshold (SAST_SEVERITY_THRESHOLD, default HIGH; SECRET_SEVERITY_THRESHOLD, default LOW, because a leaked credential is a finding whatever label it carries). Findings only below the threshold exit 2, shown as a warning rather than a green pass.
  • They fail closed. A missing, empty or non-GitLab report is a failure, not a clean scan, and a finding with an Unknown severity blocks. ALLOW_UNRESOLVED=1 is the one named opt-out for a missing report.
  • Suppressions live in .sast-suppressions.json / .secret-suppressions.json. Each entry needs a reason and an expires date at most 180 days out (SCANNER_SUPPRESSION_MAX_DAYS); an expired entry stops suppressing and the finding blocks again. Scope an entry with path; one without it applies repo-wide, and the gate says so. A leaked secret still needs rotating: a suppression is for a documented fake, not a real key.
  • Each gate runs its --self-test first in its own CI job, and both are in make pre-push-checks. The raw secret is never printed.
  • A gitleaks-history job scans the full commit history (the component only sees a diff), with gitleaks pinned by version and sha256.

What is advice, not shipped:

  • Bumping the SAST and secret-detection components. The core pipeline pins them to exact versions (sast@3.5.0, secret-detection@2.4.0) and check-ci-image-pins rejects @~latest or any other moving ref; bump them deliberately, and pin the same way in your project.
  • The job names the gates depend on (semgrep-sast, secret_detection) are the components’ defaults. Edit the needs if yours differ, and add one gate job per extra SAST analyzer (they all write gl-sast-report.json).
  • The thresholds. Start from the defaults, then tighten as the backlog reaches zero.

Not verified: the gates were exercised through their self-tests and synthetic reports, not inside a real pipeline with the components attached. Your first MR pipeline is the check.

What ships: scripts/check-trivyignore-expiry.sh and the opt-in module ci/trivyignore-expiry.yml. What is advice: everything about running Trivy itself. Blueprint ships no Trivy job, so no scan blocks your pipeline until you add one.

A .trivyignore line with no end date is a decision nobody revisits: it keeps hiding a finding after a fix ships or the package is dropped. Write each suppression in Trivy’s own syntax, with an expiry:

CVE-2024-12345 exp:2026-12-31 # waiting on upstream 2.4.1; owner: platform

The script fails on an entry that is undated, malformed (not a real calendar date, or two exp: tokens), expired (the day of expiry itself still passes), or too far out (more than TRIVYIGNORE_MAX_DAYS, default 180, from today). A missing .trivyignore passes. To adopt it, uncomment - local: ci/trivyignore-expiry.yml in .gitlab-ci.yml; the job runs only when a .trivyignore exists. Its --self-test runs first in the same job, and make gate-self-tests runs it locally.

  • It reads the repository, so it has no network oracle and no ALLOW_UNRESOLVED escape hatch. The pipeline goes red when the calendar passes an entry’s date, with no diff to blame. That is the check working: fix the finding, or renew the date with a reason in the same line.
  • It only reads the plain-text .trivyignore. The YAML form (.trivyignore.yaml) and suppressions kept elsewhere (VEX documents, a scanner UI) are not seen.
  • Advice, not shipped: pin the scanner by digest (see Pinning CI images and tools), and make the Trivy job itself blocking on HIGH and CRITICAL findings, so an expired suppression surfaces the finding again rather than only a red expiry check.

A tag such as alpine:3.23 moves: whoever pushes next decides what your pipeline runs, with your CI variables in its environment. A digest (alpine:3.23@sha256:...) is a content hash, so the pull fails if the bytes change. The tag stays for humans; the digest is what runs.

What ships:

  • Every image: and services: entry in .gitlab-ci.yml and ci/*.yml is pinned by digest, and the end-of-life bases were replaced with supported ones (Alpine 3.19 to 3.23, Node 20 to 22, Go 1.23 to 1.26, Docker 27 to 29).
  • Tools downloaded in a job are verified against a published sha256 before they run: osv-scanner, gitleaks, kind, kubectl and helm. govulncheck is installed at a fixed version instead of @latest.
  • scripts/check-ci-image-pins.py (job ci-yaml-lint, make check-ci-yaml) fails on an image: or services: entry without a 64-hex @sha256: digest, on a reference that holds a CI variable (except a whole $NAME reference whose top-level variables: value is pinned by digest, names a trusted repository (stock alpine or this project’s own ci-base image; see the CI pipeline guide, Making the pipeline fast), and that no job overrides in its own variables: block), and on go install ...@latest. Its --self-test runs first in the same job. Exceptions go in scripts/ci-image-pins.allow, one reference per line with a reason; a stale entry fails, so the list cannot become a blanket exemption.

Bumping a pin: resolve the digest from the registry yourself (docker buildx imagetools inspect alpine:3.23) and change the tag and digest together. For a downloaded tool, change the version and its sha256 together, taking the hash from the publisher’s checksum file. If you override KIND_VERSION, KUBECTL_VERSION or HELM_VERSION in ci/helm.yml, override the matching *_SHA256 too, or the job stops at the checksum.

What is advice, not shipped: a digest freezes the image, so it stops receiving security fixes until you bump it. The repository’s renovate.json already carries a digest rule; add :pinDigests to its extends if you want Renovate to raise the digest bumps as reviewable MRs. Not covered: pip install and npx calls inside jobs (unpinned, or pinned by version without hashes), the golangci-lint v1 image (v2 needs a config migration; v1.61 is built with Go 1.23, so it cannot lint a module whose go directive is newer, while go-test now runs Go 1.26, so move to v2 once your module needs it), the Kubernetes and Helm versions the drill downloads (older than current), and the GitLab SAST and secret-detection components pinned at sast@3.5.0 and secret-detection@2.4.0, which you bump deliberately and pin the same way in your project.

Signed images and SBOM attestations (optional)

Section titled “Signed images and SBOM attestations (optional)”

A scanner says what is in an image. It does not tell a user that the image they pulled is the one your pipeline built, or that it is still the same a year later. Signing and SBOM attestation close that gap.

What ships (all optional, off by default):

  • ci/image-signing.yml, an opt-in module. Uncomment - local: ci/image-signing.yml in .gitlab-ci.yml. It does nothing until you set SIGN_IMAGE_REFS (what to sign) or RELEASE_IMAGES (what to verify), so it cannot fail a clone that has no images.
  • scripts/attest-image-sbom.sh, run from a tag pipeline. It resolves each image to a digest, signs the index and every platform manifest with cosign, and attaches a CycloneDX SBOM attestation to each architecture (generated per platform with syft, not copied across). It refuses to attest an empty or non-CycloneDX SBOM.
  • scripts/check-release-attestations.sh, run by a pipeline schedule you create. It checks that the newest stable releases still carry their signatures and SBOM attestations. Its --self-test runs first in the same job, against stub cosign and docker, and proves the check goes red on a missing signature, a missing SBOM on one architecture, a deleted tag, and an unreachable registry.
  • scripts/lib/supply-chain-pins.sh, the single pin (version plus sha256) for cosign and syft. A Helm chart publish job can source the same file so charts and images are signed and verified by one cosign.

What is advice, not shipped: enabling it in your pipeline, choosing keyless or key-based signing, writing the identity regexp, setting the schedule, and making your registry keep the signature artifacts. Verifying signatures on the consumer side (an admission controller such as Kyverno or Sigstore policy-controller) is not covered.

Keys. The default is keyless: GitLab issues a short-lived OIDC token (id_tokens: SIGSTORE_ID_TOKEN with aud: sigstore) and there is no key to leak. Keyless signing writes the project path and ref to Sigstore’s public transparency log, so use key-based signing (COSIGN_KEY as a masked, protected file variable) for a private project that cannot allow that. Never commit a key.

Set the identity. In keyless mode the check requires SIGNING_CERT_IDENTITY_REGEXP to name your project and tag refs. An empty or match-anything value is refused, because it would accept anyone’s signature.

Read this before the first scheduled run:

  • Older releases were never signed. Without ATTESTATION_MIN_VERSION set to the first signed release, the first run fails on every earlier release. Signing them after the fact would prove nothing about who built them.
  • It fails closed. The oracle is your registry plus Sigstore (Rekor). If either cannot be reached, or no release tags are fetched (a shallow clone), the job exits 2 and goes red. ALLOW_UNRESOLVED=1 is the one explicit opt-out; it never hides a missing signature (exit 1).
  • Registry retention can delete proof. A registry that garbage-collects untagged artifacts may remove the .sig and .att entries. The first run is when you learn that.
  • Coverage is stated, not implied. The verdict line counts the tags, signatures and attestations it verified. It does not judge whether an SBOM is complete or the image is free of vulnerabilities.
  • Run --list and then the real check by hand once, and create the schedule only after one signed release exists.

Scorecard grades a GitHub repository’s hygiene: branch protection, pinned dependencies, signed releases, a security policy, and more. It needs the project on GitHub, so it comes after you publish a GitHub mirror. The score is public, so what it checks is worth fixing before you go public.

Bottom line: let tools you don’t control grade the project. Start with SonarQube Cloud, since public projects get the whole-project scan for free.