Skip to content

Testing AI-written code

The big picture: when an AI writes the code and the tests, a green suite with high coverage proves less than it seems. Two techniques check the tests themselves: fuzz (property-based) testing attacks the code with inputs nobody thought of, and mutation testing breaks the code on purpose to see whether the tests notice.

Why it matters: coverage says a line ran during a test, not that anything checked its result. An AI that misreads a requirement tends to write a test that agrees with its own misreading, so code and test are wrong together and stay green. You need a check that doesn’t share the author’s assumptions.

How it works:

  • Fuzzing generates hundreds of inputs per run, including the odd ones, and checks that stated properties always hold (“the result is never negative”, “the API never returns a 500”).
  • Mutation testing changes the code (flips a < to <=, deletes a line), reruns the suite, and counts how many changes the tests caught.
  • Both run nightly or on a narrow scope, not on every push. They are slow.
Tool Fuzzes What it finds
Hypothesis (Python) Functions and classes Inputs that break a property you declared; shrinks to the smallest failing case
Schemathesis A running API, from its OpenAPI schema 500s, and responses that don’t match the schema

An example test checks one input you chose. A property test checks a rule over many inputs you didn’t. The AI can’t have tuned the code to inputs it never saw.

Run it in two tiers:

  1. A deterministic profile on every MR. Fixed seed, small budget, so a red result is always a real, reproducible finding.
  2. A deep stochastic profile on a nightly schedule. Big budget, saved corpus of past failures, and it never blocks an MR. A red nightly is a triage signal: turn the failing case into a plain deterministic test, then fix.

A surviving mutant is a code change the suite did not catch. It names a specific line whose behavior no test pins down.

  • Score = caught mutants ÷ total. Read the survivor list, not just the number.
  • Equivalent mutants change the code without changing its behavior, so no test can catch them. Remove them from the code or accept them; don’t chase 100%.
  • Pin the tool version. A new mutation tool version can change what gets mutated and move the score with no change on your side.
  • Report first, gate later. Run report-only until you have several nights of data, then set the floor from that data. Lower the floor only with a recorded reason. Raising it to quiet a red run deletes the finding.
  • Scope it. Start with the few modules where a wrong answer is costly.

TruePPM (the project management app built on this harness) runs all of it in its own .gitlab-ci.yml:

Job Technique Posture
scheduler:fuzz-deep Hypothesis, deep profile, split across four shards Nightly only, never blocks an MR
api:fuzz Schemathesis against the live API Nightly, non-blocking (allow_failure)
scheduler:mutation mutmut over the scheduler’s models, derive, and cli modules Nightly fails below a floor; MRs that change the measured code run it as informational; MRs that change the mutation setup itself gate
api:mutation mutmut on a high-risk API module Report-only until a baseline exists

The same Hypothesis properties also run on every MR in the deterministic profile. TruePPM added the API mutation job after computations in that layer shipped wrong on a green suite twice.

  • Schema fuzzing: python-schema-fuzz in ci/python.yml. See CI pipeline.
  • A reference mutation gate: ci/mutation.yml, an optional example module. See below.
  • Property tests: not shipped. Add Hypothesis to your own test suite.

ci/mutation.yml is a nightly, sharded, floor-gated mutation job. It is a Python and mutmut 2.x example, off by default: uncomment - local: ci/mutation.yml in .gitlab-ci.yml and create a dedicated pipeline schedule on the default branch with MUTATION_SCHEDULE=true and MUTATION_TARGETS set. The nightly jobs run only on a schedule carrying that variable, so your other schedules never launch hours of mutation. Left unconfigured (empty MUTATION_TARGETS) the jobs are skipped on purpose and the schedule stays green, so confirm your first scheduled run actually lists the jobs.

The pieces are split so only the last one is tool-specific:

Piece Role
scripts/check_mutation_score.py Stack-neutral. Merges per-shard JSON files, computes raw and adjusted scores, applies the floor, fails closed.
scripts/mutmut_export.py The adapter. Turns a mutmut 2.x cache into one shard file, dropping the phantom rows mutmut 2.5.1 creates for duplicate lines. Swap it for another tool’s exporter.
scripts/mutmut_config.py.example The shard hook: each shard skips the mutants other shards own (round-robin over one ordinal across all files). On any error it leaves a mutmut-hook-error marker that the exporter refuses on.
ci/mutation.yml mutation-selftest on MRs; nightly mutation-shard (parallel) then mutation-report.

You configure it with variables, not code: MUTATION_TARGETS (the modules), MUTATION_SHARDS, MUTATION_MIN (the floor), MUTATION_EQUIVALENTS. Enforcing means setting MUTATION_MIN and setting MUTATION_REPORT_ONLY to "false"; setting a floor while report-only is still on fails the report instead of passing quietly. Shards split the target’s mutants round-robin, so their sizes differ by at most one; keep MUTATION_SHARDS no larger than the mutant count of your smallest target (a shard with an empty slice exits 2), and pick it so the slowest shard fits the job timeout. The scripts have offline --self-tests (fixture results, no mutmut needed), and each nightly job runs the self-test of the scripts it runs, in its own image, before the real work (mutation-selftest adds early feedback on MRs).

Blueprint ships no default floor. Visiban, the project this was distilled from, ratcheted to 90% after a report-only pilot; that is its number, not a recommendation. Set yours from your own baseline:

  1. Run report-only (MUTATION_REPORT_ONLY: "true", MUTATION_MIN unset) on one module for several nights.
  2. Read the raw and adjusted scores per target in the report log.
  3. Set MUTATION_MIN just below your baseline adjusted score.
  4. Set MUTATION_REPORT_ONLY to "false", then ratchet MUTATION_MIN up as survivors are killed. Lower it only with a recorded reason.

Verification status: the hook, exit-code and cache-schema assumptions were checked against the mutmut 2.5.1 source (run_mutation honors pre_mutation and context.skip), and the shard and report scripts were run with the real mutmut 2.5.1 on Python 3.12: a single file at 1 and 2 shards, a three-file package at 1, 2 and 3 shards, a duplicate-line phantom row, a dict_synonyms setting, a run killed midway and a failing hook (both give NOT MEASURED). The whole pipeline inside a real GitLab schedule has not been run, so your first scheduled run is your validation. mutmut’s --use-coverage, --paths-to-exclude and mutation-type options change what a run generates and are not mirrored; a project using them gets a loud NOT MEASURED, not a wrong score.

The gate exits non-zero (1 below the floor, 2 when it cannot measure) in every case where a green result would be a guess:

  • A missing shard fails. You must say which targets and shard counts to expect (--expect); a shard that died or lost its artifact never reads as a smaller, complete run. A duplicate or foreign shard fails too.
  • A dead target fails. A module that produced no mutants has a run that died before mutating anything. It is not 100%, and it can’t drop out and let the other modules carry the floor.
  • An unfinished run fails. Any mutant left untested means the job was cut short.
  • No floor is a config error, not a pass. The floor must be a fraction in (0, 1] (0.90, not 90, not 0). A bad floor is checked before anything is read and writes no merged stats, so it can’t pass by accident. --report-only is the one explicit way to run without a floor, and combining it with a floor is an error too.
  • Suspicious counts as not killed. A mutant mutmut marks suspicious (odd timing) is in the denominator and not the numerator.
  • Equivalent mutants need a reason of at least 10 characters (and not one repeated character). Raw score counts every survivor; adjusted score excludes the mutants you recorded as equivalent. The floor gates the adjusted score, and both are printed. An equivalent with no reason, or one that no longer survives (the code moved), fails instead of quietly inflating the score.
  • # pragma: no mutate lines are outside both scores. mutmut generates no mutant on them, so they are neither raw nor adjusted. The report counts them (“N lines excluded by pragma”) so they stay visible, but their reasons are not checked; use the equivalents file for an exclusion that needs a reviewed reason.
  • Phantom cache rows are reconciled, not hidden. mutmut 2.5.1 can file a mutant under an earlier identical line and leave a row that never runs. The exporter drops such a row only if mutmut never generated it AND it has that signature (its line text is identical to another line’s, at an index the line does not have), and the report says how many it dropped. Any other never-run mutant, including one on a line with unique text, still fails the run.
  • I/O failures are not measurements. An unreadable shard or an unwritable --write-merged path exits 2 with NOT MEASURED, not a traceback.
  • Paths are contained. Every file the scripts read or write is resolved through scripts/_paths.py, so a .. or a symlink can’t point them outside the tree.

A survivor usually names a missing assertion. Triage each one in order: can the change alter anything a caller could observe? If not, it is equivalent (record it with a reason). Is the changed line reached by any test? If not, add one. Otherwise add the smallest assertion that fails on the mutant:

  • Assert the exact value, not “no exception” or “truthy”. A mutated return value survives any test that only checks the call finished.
  • Sit exactly on every boundary. A >= to > change survives unless a test uses the limit itself; test limit - 1, limit and limit + 1.
  • Assert the whole message. A string mutant that wraps text in extra characters survives an assertIn on a fragment; assert the full string, or both its start and its end.
  • Cover the empty and single-item cases, and check ordering explicitly when order matters.
  • Assert side effects, such as the row written, the event emitted, or the call made with its arguments, not only the return value.
  • Make each branch of a condition decide an outcome. If flipping and to or changes no test result, no test has a case where only one side is true.
  • Check the failure path: the exact error raised and the state left behind.
  • Give a default a test that overrides it, and one that relies on it.

Two operating notes. Each shard runs your whole suite once per mutant, so keep the target list short and run the job on a runner that isn’t shared with latency-sensitive work. And a surviving mutant is usually a missing assertion: assert the exact value, boundary, or side effect, not just that the call didn’t raise.

Bottom line: treat coverage as a floor, not a verdict. When an AI writes both sides, let a fuzzer and a mutation run decide whether the tests are any good.