Testing AI-written code
The big picture: when an AI writes the code and the tests, a green suite with high coverage proves less than it seems. Two techniques check the tests themselves: fuzz (property-based) testing attacks the code with inputs nobody thought of, and mutation testing breaks the code on purpose to see whether the tests notice.
Why it matters: coverage says a line ran during a test, not that anything checked its result. An AI that misreads a requirement tends to write a test that agrees with its own misreading, so code and test are wrong together and stay green. You need a check that doesn’t share the author’s assumptions.
How it works:
- Fuzzing generates hundreds of inputs per run, including the odd ones, and checks that stated properties always hold (“the result is never negative”, “the API never returns a 500”).
- Mutation testing changes the code (flips a
<to<=, deletes a line), reruns the suite, and counts how many changes the tests caught. - Both run nightly or on a narrow scope, not on every push. They are slow.
Fuzz and property-based testing
Section titled “Fuzz and property-based testing”| Tool | Fuzzes | What it finds |
|---|---|---|
| Hypothesis (Python) | Functions and classes | Inputs that break a property you declared; shrinks to the smallest failing case |
| Schemathesis | A running API, from its OpenAPI schema | 500s, and responses that don’t match the schema |
An example test checks one input you chose. A property test checks a rule over many inputs you didn’t. The AI can’t have tuned the code to inputs it never saw.
Run it in two tiers:
- A deterministic profile on every MR. Fixed seed, small budget, so a red result is always a real, reproducible finding.
- A deep stochastic profile on a nightly schedule. Big budget, saved corpus of past failures, and it never blocks an MR. A red nightly is a triage signal: turn the failing case into a plain deterministic test, then fix.
Mutation testing
Section titled “Mutation testing”A surviving mutant is a code change the suite did not catch. It names a specific line whose behavior no test pins down.
- Score = caught mutants ÷ total. Read the survivor list, not just the number.
- Equivalent mutants change the code without changing its behavior, so no test can catch them. Remove them from the code or accept them; don’t chase 100%.
- Pin the tool version. A new mutation tool version can change what gets mutated and move the score with no change on your side.
- Report first, gate later. Run report-only until you have several nights of data, then set the floor from that data. Lower the floor only with a recorded reason. Raising it to quiet a red run deletes the finding.
- Scope it. Start with the few modules where a wrong answer is costly.
How TruePPM uses them
Section titled “How TruePPM uses them”TruePPM (the project management app built on this
harness) runs all of it in its own .gitlab-ci.yml:
| Job | Technique | Posture |
|---|---|---|
scheduler:fuzz-deep |
Hypothesis, deep profile, split across four shards |
Nightly only, never blocks an MR |
api:fuzz |
Schemathesis against the live API | Nightly, non-blocking (allow_failure) |
scheduler:mutation |
mutmut over the scheduler’s models, derive, and cli modules |
Nightly fails below a floor; MRs that change the measured code run it as informational; MRs that change the mutation setup itself gate |
api:mutation |
mutmut on a high-risk API module | Report-only until a baseline exists |
The same Hypothesis properties also run on every MR in the deterministic profile. TruePPM added the API mutation job after computations in that layer shipped wrong on a green suite twice.
What Blueprint ships today
Section titled “What Blueprint ships today”- Schema fuzzing:
python-schema-fuzzinci/python.yml. See CI pipeline. - A reference mutation gate:
ci/mutation.yml, an optional example module. See below. - Property tests: not shipped. Add Hypothesis to your own test suite.
The reference mutation gate
Section titled “The reference mutation gate”ci/mutation.yml is a nightly, sharded, floor-gated mutation job. It is a Python and
mutmut 2.x example, off by default: uncomment - local: ci/mutation.yml in
.gitlab-ci.yml and create a dedicated pipeline schedule on the default branch with
MUTATION_SCHEDULE=true and MUTATION_TARGETS set. The nightly jobs run only on a schedule
carrying that variable, so your other schedules never launch hours of mutation. Left
unconfigured (empty MUTATION_TARGETS) the jobs are skipped on purpose and the schedule
stays green, so confirm your first scheduled run actually lists the jobs.
The pieces are split so only the last one is tool-specific:
| Piece | Role |
|---|---|
scripts/check_mutation_score.py |
Stack-neutral. Merges per-shard JSON files, computes raw and adjusted scores, applies the floor, fails closed. |
scripts/mutmut_export.py |
The adapter. Turns a mutmut 2.x cache into one shard file, dropping the phantom rows mutmut 2.5.1 creates for duplicate lines. Swap it for another tool’s exporter. |
scripts/mutmut_config.py.example |
The shard hook: each shard skips the mutants other shards own (round-robin over one ordinal across all files). On any error it leaves a mutmut-hook-error marker that the exporter refuses on. |
ci/mutation.yml |
mutation-selftest on MRs; nightly mutation-shard (parallel) then mutation-report. |
You configure it with variables, not code: MUTATION_TARGETS (the modules),
MUTATION_SHARDS, MUTATION_MIN (the floor), MUTATION_EQUIVALENTS. Enforcing means setting MUTATION_MIN and setting
MUTATION_REPORT_ONLY to "false"; setting a floor while report-only is still on fails
the report instead of passing quietly. Shards split the target’s mutants round-robin, so
their sizes differ by at most one; keep MUTATION_SHARDS no larger than the mutant count
of your smallest target (a shard with an empty slice exits 2), and pick it so the slowest
shard fits the job timeout. The scripts have offline --self-tests
(fixture results, no mutmut needed), and each nightly job runs the self-test of the scripts
it runs, in its own image, before the real work (mutation-selftest adds early feedback on MRs).
From pilot to floor
Section titled “From pilot to floor”Blueprint ships no default floor. Visiban, the project this was distilled from, ratcheted to 90% after a report-only pilot; that is its number, not a recommendation. Set yours from your own baseline:
- Run report-only (
MUTATION_REPORT_ONLY: "true",MUTATION_MINunset) on one module for several nights. - Read the raw and adjusted scores per target in the report log.
- Set
MUTATION_MINjust below your baseline adjusted score. - Set
MUTATION_REPORT_ONLYto"false", then ratchetMUTATION_MINup as survivors are killed. Lower it only with a recorded reason.
Verification status: the hook, exit-code and cache-schema assumptions were checked
against the mutmut 2.5.1 source (run_mutation honors pre_mutation and context.skip),
and the shard and report scripts were run with the real mutmut 2.5.1 on Python 3.12: a
single file at 1 and 2 shards, a three-file package at 1, 2 and 3 shards, a duplicate-line
phantom row, a dict_synonyms setting, a run killed midway and a failing hook (both give
NOT MEASURED). The whole pipeline inside a real GitLab schedule has not been run, so your
first scheduled run is your validation. mutmut’s --use-coverage, --paths-to-exclude and
mutation-type options change what a run generates and are not mirrored; a project using them
gets a loud NOT MEASURED, not a wrong score.
The fail-closed rules
Section titled “The fail-closed rules”The gate exits non-zero (1 below the floor, 2 when it cannot measure) in every case where a green result would be a guess:
- A missing shard fails. You must say which targets and shard counts to expect
(
--expect); a shard that died or lost its artifact never reads as a smaller, complete run. A duplicate or foreign shard fails too. - A dead target fails. A module that produced no mutants has a run that died before mutating anything. It is not 100%, and it can’t drop out and let the other modules carry the floor.
- An unfinished run fails. Any mutant left
untestedmeans the job was cut short. - No floor is a config error, not a pass. The floor must be a fraction in (0, 1]
(
0.90, not90, not0). A bad floor is checked before anything is read and writes no merged stats, so it can’t pass by accident.--report-onlyis the one explicit way to run without a floor, and combining it with a floor is an error too. - Suspicious counts as not killed. A mutant mutmut marks suspicious (odd timing) is in the denominator and not the numerator.
- Equivalent mutants need a reason of at least 10 characters (and not one repeated character). Raw score counts every survivor; adjusted score excludes the mutants you recorded as equivalent. The floor gates the adjusted score, and both are printed. An equivalent with no reason, or one that no longer survives (the code moved), fails instead of quietly inflating the score.
# pragma: no mutatelines are outside both scores. mutmut generates no mutant on them, so they are neither raw nor adjusted. The report counts them (“N lines excluded by pragma”) so they stay visible, but their reasons are not checked; use the equivalents file for an exclusion that needs a reviewed reason.- Phantom cache rows are reconciled, not hidden. mutmut 2.5.1 can file a mutant under an earlier identical line and leave a row that never runs. The exporter drops such a row only if mutmut never generated it AND it has that signature (its line text is identical to another line’s, at an index the line does not have), and the report says how many it dropped. Any other never-run mutant, including one on a line with unique text, still fails the run.
- I/O failures are not measurements. An unreadable shard or an unwritable
--write-mergedpath exits 2 withNOT MEASURED, not a traceback. - Paths are contained. Every file the scripts read or write is resolved through
scripts/_paths.py, so a..or a symlink can’t point them outside the tree.
Assertions that kill mutants
Section titled “Assertions that kill mutants”A survivor usually names a missing assertion. Triage each one in order: can the change alter anything a caller could observe? If not, it is equivalent (record it with a reason). Is the changed line reached by any test? If not, add one. Otherwise add the smallest assertion that fails on the mutant:
- Assert the exact value, not “no exception” or “truthy”. A mutated return value survives any test that only checks the call finished.
- Sit exactly on every boundary. A
>=to>change survives unless a test uses the limit itself; testlimit - 1,limitandlimit + 1. - Assert the whole message. A string mutant that wraps text in extra characters survives
an
assertInon a fragment; assert the full string, or both its start and its end. - Cover the empty and single-item cases, and check ordering explicitly when order matters.
- Assert side effects, such as the row written, the event emitted, or the call made with its arguments, not only the return value.
- Make each branch of a condition decide an outcome. If flipping
andtoorchanges no test result, no test has a case where only one side is true. - Check the failure path: the exact error raised and the state left behind.
- Give a default a test that overrides it, and one that relies on it.
Two operating notes. Each shard runs your whole suite once per mutant, so keep the target list short and run the job on a runner that isn’t shared with latency-sensitive work. And a surviving mutant is usually a missing assertion: assert the exact value, boundary, or side effect, not just that the call didn’t raise.
Bottom line: treat coverage as a floor, not a verdict. When an AI writes both sides, let a fuzzer and a mutation run decide whether the tests are any good.