Verify your instrument: the research discipline that decides whether your findings are real

Security research on deployed code is an empirical discipline, and empirical disciplines have a failure mode that has nothing to do with the system under test: the instrument is wrong. A test harness, a simulation, a spreadsheet, a monitoring dashboard — each is a piece of software that can be defective in exactly the way the thing it measures can. When it is, the result is not noise. It is a confident, specific, wrong conclusion, and confident wrong conclusions travel further than uncertain right ones.

The rules below are stated as engineering discipline because each of them exists to prevent a specific class of confident error.

A negative result is a hypothesis, not a finding

The single most useful habit: when a probe reports that something is broken, the first hypothesis is that the probe is broken. Several results that read cleanly as protocol defects turn out, on investigation, to be faults in how they were measured — a harness that set up state the system can never reach, a comparison against a number computed by the wrong path, a simulation that skipped a step the real flow performs.

This is not scepticism for its own sake. It is a cost asymmetry: publishing a wrong finding costs credibility that takes years to rebuild, while spending an extra hour validating the instrument costs an hour. Verify your own instrument before you believe what it tells you about the system.

Severity is measured after reachability, never asserted before it

A severity rating is a claim about the WORLD — how bad can this get, how often, starting from states that actually occur. It is not a claim about the code path. Assign it from an algebraic argument and you will eventually assign it from an algebraic argument that is wrong.

The instructive failure looks like this: a bound is asserted ("this variable is capped, therefore that branch is unreachable, therefore the defect is minor"), several conclusions are built on the bound, and the bound turns out to be false because the reachable state space is wider than the argument assumed. Everything downstream inherits the error — including, in the worst case, severity ratings communicated publicly to the researchers who reported the issue.

The corollary is sharper still: a branch believed to be dead is not dead until reachability has been MEASURED. Deleting code on the strength of an unverified argument is how a live path gets removed from a deployed system. Measure first; delete second.

A correctness fix is a change, and changes regress

Fixes are written under pressure, in the region of the code that was just proven fragile, by someone who has spent hours convincing themselves the old behaviour was wrong. That is the highest-risk edit in the entire lifecycle, and it is routinely subjected to the least review.

The regression that matters most is a swap of failure MODES. Replacing tolerant arithmetic with strict arithmetic in a hot path is correct in the small — and if the quantity can genuinely reach the strict bound, it converts a wrong-number problem into a frozen-protocol problem, precisely when the system is already under stress. A correctness fix that introduces a liveness failure has not improved safety; it has moved it into a dimension nobody is watching.

Self-consistency is what a wrong number has with itself

Protocols publish figures for trustless verification, and users, dashboards, monitors and test suites all check the system THROUGH those figures. That makes the reporting surface the highest-leverage place a defect can hide: a bug that corrupts the instrument rather than the value is simultaneously invisible to every other dimension.

The wrong test is to check that the published views agree with each other. Two views computed from the same faulty helper agree perfectly and are both wrong; self-consistency is exactly the property a wrong number has with itself. The right test rebuilds every published figure from PRIMITIVE state, by an independent implementation, and compares. That is also why duplicate implementations of the same published quantity are a security problem rather than an aesthetic one — any figure computed in two places will eventually diverge, and the divergence is found only when a third, independent computation exists to expose it.

A checklist that generalises

For any system that distributes value, the following are checkable rather than aspirational: every accumulator is advanced before any write that changes its inputs; distribution and collection move in the same atomic step; no multiplier is keyed to a stored value that time can invalidate; every clock advance has a named, reachable destination for the value it represents; every published quantity has exactly one implementation; a participant absent for a window receives nothing from that window; the same operation costs the same regardless of which transaction drives it; every published figure is reproducible from primitive state by an independent party; every health metric has been deliberately driven into a known-bad state to prove it responds; and severity is assigned after reachability is measured.

The last item on any honest version of this list is the one that matters most: the dimensions absent from your map are where the next defect lives. A finding that appears in a dimension nobody had named is not bad luck — it is the predictable consequence of an incomplete map, and the correct response is to widen the map rather than to patch the instance.

Related engineering

• When invariants meet time: the vulnerability class conservation cannot detect ›
• Invariant-driven design: what a conservation law proves, and the dimension it cannot see ›
• The Nakamoto Test: five questions that measure any protocol against the original vision ›