The mutation guard: 268 named breaks, each paired with the one test that must die

What you gain here is a way to read a green test suite without believing it. A passing test proves the code ran; it does not prove that anything would have failed had the code been wrong. Mutation testing closes that gap by asking the suite a question it cannot answer by being green: if this line were broken, which of you would notice? The BlazePhoenix guard asks it 268 times, by name, on every push, and prints the answer beside its denominator.

A mutant is a deliberate, single alteration of one line of source — a comparison flipped, a guard deleted, a constant doubled — compiled and run against the tests. If a test fails, the mutant is killed and that line is watched. If every test stays green, the mutant survived, and the suite has just been shown a hole it cannot see. The idea is from DeMillo, Lipton and Sayward in 1978; what is specific here is the shape the register takes.

Curated, not generated

Most mutation tools generate thousands of mutants mechanically and report a percentage. The guard in .github/scripts/mutants.py holds 268 hand-written mutants instead, and each pairs one exact line of source with the single test that must fail once that line is altered. The operator classes are the shapes a real regression takes: a guard deleted, a comparison flipped at the bound that decides a refusal, an authorisation condition widened, an error code swapped with its neighbour’s, a modifier removed.

Naming the killing test is the load-bearing choice. A generated score says “83 % of mutants died” and leaves you to guess which guards the surviving 17 % were protecting. A named pair says: this line is the pause check, this test is the one that must go red without it. When a future edit moves the line or weakens the test, the pair breaks by name, and the build says which guard just lost its watcher.

Three properties that make it a guard rather than a score

1. The paired test is run green on the unmutated tree first, so nothing is scored as a kill for an unrelated reason — a test that was already red would “kill” every mutant.
2. A mutation the optimiser removes — identical runtime bytecode before and after — is reported as inert, never as killed. A source edit the compiler folds away watched nothing, and a guard that counted it would be lying about its own denominator.
3. A static check verifies, in about a second, that every mutant still points at exactly one line of real code, so a refactor cannot silently orphan a watcher. Mutation-target integrity is the one metric here with a single correct value: anything below 100 % means a claim points at code that no longer exists.

Aimed at the invariants

Until 2026-09-05 no mutant named a stateful invariant as the test that must die to it. The 39 invariant functions were green, and nothing had asked whether any of them could go red: a campaign whose handler never reaches the state a property protects certifies that property over an empty universe, and looks identical from the outside to one that reaches it. So the same measurement was pointed at the campaigns: fifteen source mutations, one per property the campaigns claim, each run against all 39 invariants on one seed.

Eleven of fifteen were noticed on the first measurement and eighteen distinct invariant names went red at least once. The survivors were the finding, each read rather than counted: three had no campaign that could reach them — the protocol floor, the registry’s pair-proof, the bridge-residual sweep’s baseline — and were closed the same day by giving a campaign the action it lacked, each verified red without its guard. The fourth stays, as the table says, by design. Those eleven and the three closed rows are now guard entries paired with the invariant that dies to them.

What the guard admits, and what it refuses

The guard admits only mutants a named test kills. Two checks in the Router’s fee ledger cannot be made to fire by any test in isolation, because the predicate in front of them leaves no path that settles without paying; they are listed as belts, counted as covered nowhere, and kept out of the guard rather than folded into the killed count. A fourth guard on the stable curve — a cap on Newton’s iterate — was written, found unobservable once two fit checks hold, its mutant reported as decorative, and removed. A guard that cannot fire is not a weak guard; it is not a guard, and the register says so instead of carrying it.

Why 268 of 268 is a floor

The figure is always printed beside its denominator, because it is adequacy against this register and nothing else. The register is hand-curated: each mutant exists because a specific guard was worth watching. Killing all of them says the suite watches those guards. It does not say the set is representative of the faults that occur in the wild, and the empirical work on fault–mutant coupling reports that a substantial minority of real faults are not coupled to any mutant from the usual operators. A saturated score is a floor on what is watched, not a ceiling on what could go wrong.

That is also why the guard is paired with instruments that index something other than an injected fault — covering arrays over the regime a fixture fixes, metamorphic relations over how outputs must move, a second oracle written from the venue’s specification. Each sees a class the others cannot, and the honest number for the whole is the minimum, never the sum.

Reproduce

Both commands run from a clean checkout on the release profile; the first prints every mutant with its killing test and its outcome — killed, inert or surviving — and the second is the one-second integrity check.

python3 .github/scripts/mutants.py           # 268 mutants, each with its named killing test
python3 .github/scripts/check_targets.py     # every mutant still points at exactly one line