Skip to content

fix(faults): a receipt line says when it is still in flight (#925) - #929

Merged
stephrobert merged 1 commit into
mainfrom
fix/925-the-receipt-says-in-flight
Nov 14, 2026
Merged

stephrobert merged 1 commit into
mainfrom
fix/925-the-receipt-says-in-flight

Conversation

@stephrobert

Copy link
Copy Markdown
Owner

Summary

A fault receipt line was written in two steps and only the second one was
visible. reserve appends the line when the rule fires; recordFired fills in
status, transport and abandoned when the handler finishes. Between the two the
line existed and said nothing, and nothing in the JSON said so.

That gap is not theoretical: it is a race the test suite was already losing.

The measurement that found it

On unmodified origin/main at 51612e6, in this repository:

run result
go test -race ./internal/core/emulator -run TestADroppedConnectionIsClaimedOnlyWhenItHappened -count=1, ten times 4 failures out of 10
the same at -count=8 3 failures out of 3

The CI of #922 was green on that same commit, which is how it shipped. The
author of #926, working on an unrelated area, hit it independently and
disclosed it in his pull request body as a pre-existing flake. It is the same
defect.

The reason: the test polled the receipt and read abandoned on a line that was
reserved and not yet completed. A zero value for "not yet" is indistinguishable
from a zero value for "no".

What changed

FiredFault.InFlight (internal/core/emulator/faults.go). reserve sets
it true. recordFired clears it last, after the epoch guard and after the
three values the effect produced, so a reader never sees a half-written line
claim to be finished. omitempty keeps a finished line byte-identical to what
version 3 produced.

What a version 3 reader was missing is worth stating precisely, because the
first draft of this said something false. A v3 reader was not blind: it
could infer "in flight" from the two-field invariant (status == 0 and
abandoned == false cannot both hold on a finished line, because
conformance.go hands recordFired !answered || status == 0 for
abandoned). But that invariant was undocumented, unenforced, and feint's own
CLI did not apply it. An invariant nobody can read is not an interface.

Note also that a line in flight does not hold all-zero fields: Transport
is named at the hijack by markTransport, so a hang carries its transport for
as long as the connection is held. That is the whole reason markTransport
exists, and the new CLI rendering depends on it.

One renderer instead of three (internal/cli/fault.go). describeFiredLine
now produces every receipt sentence, and run_faults.go calls it. A line still
in flight reads the connection is held open right now, no answer yet for a
hang, still in flight, no answer yet otherwise. The emulator import left
run_faults.go, which is the structural proof that only one renderer remains.

Schema versions. FaultsSchemaVersion 3 → 4. runReceiptSchemaVersion
2 → 3, because a run receipt is written to disk and cannot be re-polled.

The frozen fixture gained entry 4. The population now holds a raw
connection open across the capture loop, through waitForAHeldLine, because
without an in-flight line observed the field tree is unchanged and
frozen:update produces nothing. That is the trap this change walked into
first.

The whole testdata/frozen/faults.json file is reindented from 1 space to 2 by
the generator: the committed file had drifted from what the generator emits.
Entries 1 to 3 decode identically. conformance.json is still 1-space and is
untouched here.

The one that would have shipped broken

Changing the sentence broke tools/conformance/faults.sh:379, which greps
instance/v1/API.ListServers call 1: answered 403 against a real client.
mise run prepush does not play that suite, so nothing local said a word. It
was caught by the contract-guardian pass and is fixed, with a comment naming
why that grep exists. README.md:751 and README.fr.md:589 carry the same
sentence by hand, outside any generated block, so docs:check could not see
them either; both updated.

Type of change

  • A route added or changed
  • Bug fix
  • Refactor
  • Documentation
  • Chore / tooling
  • Security / supply chain

Runs

run verdict
mise run prepush green, 70 s
go test -race ./internal/core/emulator ./internal/cli, 6 repeats 0 failures
the previously flaky test, 10 separate processes 0 failures out of 10 (was 4 out of 10)
mise run falsify -- tools/falsify/specs/a-receipt-line-says-it-is-in-flight.json 4 mutations, all 4 bite
mise run conformance:leg -- fields green, and the broken assertion passes again

Mode: --vm off. Nothing here touches a machine runtime, so no leg under
FEINT_VM was played, and none is owed.

The fields leg is the row mise run testplan names for a response-shape
change, and it is the only leg that drives every client against one emulator.
It is also the only run that could have caught the faults.sh breakage.

Runs not played, and what that leaves unproven

  • The full mise run conformance pass. fields carries every client; what
    the full pass adds is the other legs' own gates, none of which judge the
    sentence this change edits.
  • The six other falsify specs mise run testplan names, because they mutate
    files this diff touches. mise run falsify:lint is green, so every mutation's
    find string still applies: that is the cheap half. That each of them still
    bites has not been replayed. This will be done before merge and the result
    posted here.

Falsification

tools/falsify/specs/a-receipt-line-says-it-is-in-flight.json, four mutations,
each biting TestAReceiptLineSaysWhenItIsStillInFlight:

  1. InFlight: true && false at reserve, so the line is never marked;
  2. InFlight = true in recordFired instead of false, so it is never cleared;
  3. clearing by operation name in a loop instead of by slot, which settles the
    wrong line when one rule has several firings;
  4. clearing before the epoch guard, so a stale firing settles the new
    campaign's line.

Mutations 3 and 4 are held by two further tests written for exactly those
shapes, TestSettlingOneFiringLeavesTheOtherInFlight and
TestAStaleFiringDoesNotSettleTheNewCampaignsLine.

Checklist

Always

  • mise run check passes
  • No new external Go dependency
  • internal/core still knows no provider
  • Nothing in the diff could be written identically for another provider
  • mise run testplan was run, and the runs not played are named above with
    what they leave unproven

When a model wrote a substantive part of this

  • An Assisted-by: trailer names the tool and the model version
  • mise run conformance:leg -- fields was run, not assumed. The full pass
    was not, and that is written above
  • No new field name: in_flight is this emulator's own debugging surface,
    not a provider shape. No provider response changed

When a route is added or changed

N/A. No route added or changed, no upstream operation touched.

When behaviour a client can observe changes

  • docs/limits.md names in_flight in what the receipt serves
  • No generated table moved. The two README sentences are hand-written and
    were updated by hand, which is the gap that let them go stale

When .github/workflows/ is touched

N/A.

When a machine runtime is involved

N/A. --vm off throughout.

Known gap, carried by its own issue

faultsDocument.SchemaVersion (internal/cli/fault.go:143) is decoded and read
nowhere, so feint fault list against an emulator at version 3 renders every
line as finished and an in-flight line reads answered 0. That is the lie this
pull request removes, reintroduced by a version skew.

#925 neither created nor worsened it, and it is not fixed here. It is #928,
with its closing criteria, rather than a sentence in a pull request body.

Related issues

Closes #925.

🤖 Generated with Claude Code

GET /_feint/faults reserves a line when a rule fires and completes it when
the handler ends. In between it published `status: 0, abandoned: false`
with nothing saying the firing was still running.

A reader was not blind to it: `abandoned` is `!answered || status == 0` at
the call site, so a finished line is always `status != 0, abandoned: false`
or `status == 0, abandoned: true`, and that pair belonged to a line in
flight alone. What was missing was a way to SAY it. The invariant was
undocumented, it holds only as long as `abandoned` keeps that formula, and
feint's own reader did not apply it: `feint fault list` printed
`answered 0` for a request nobody had answered.

The window is milliseconds for a delay and minutes for a `hang`, which is
exactly when a suite polls, since its client is still blocked.

How it was found, and what it says about the gate. The test that reads the
receipt right after its client gives up failed 4 times in 10 runs on main
at 51612e6, an hour after the lot that wrote it merged, and 3 times out of
3 under `-count=8`. The CI of that pull request was green on the same
commit: one draw of a coin that comes up tails four times in ten. A
verdict on an intermittent phenomenon needs repetition, not a single run.

What changed:

- `in_flight` on the receipt line, set at reserve and cleared in
  recordFired beside the three values the effect produced. It is omitempty,
  so a finished line is byte-identical for a version 3 reader.
  FaultsSchemaVersion 4.
- The frozen population holds a connection open across the capture. Without
  it the fixture would freeze version 4 without the only field that
  distinguishes it, and faults.json already carries the comment explaining
  why that matters: a gate whose population cannot contain the thing it
  judges is not a gate.
- `feint fault list` and a run's receipt read the field, through one shared
  describeFiredLine so the two renderings cannot drift. A run's receipt is
  written to disk and cannot be re-polled, so runReceiptSchemaVersion is 3.
- Four falsify mutations, two of them written by a falsifier against the
  first version of this lot: clearing the marker by operation name instead
  of by slot, and clearing it before the epoch guard. Both compiled, both
  lied, and the whole package stayed green under each, because every test
  here drove a single client.

The frozen fixture is reindented by the generator, which the committed file
had drifted from. Entries 1 to 3 decode identically; only version 4 is new.

Assisted-by: Claude Code (claude-opus-5)
@stephrobert
stephrobert merged commit ce59308 into main Nov 14, 2026
32 checks passed
@stephrobert

Copy link
Copy Markdown
Owner Author

Closing the run this pull request declared as not played.

The six falsify specs mise run testplan named, because they mutate files this diff touches, were replayed on the merged tree. All six exit 0, so every mutation still bites.

spec mutations verdict
a-failure-below-http 7 all bite
a-run-owns-its-lab 16 all bite
a-transport-fault-is-bounded 3 all bite
command-tables-two-audiences 6 all bite
frozen-cli-surface 7 all bite
injected-is-not-evidence 6 all bite

45 mutations, each compiled and each reddening its named test, baseline green before and after restoration in every run.

a-transport-fault-is-bounded is the one that mattered most here: its third mutation names TestADroppedConnectionIsClaimedOnlyWhenItHappened, the test this change stabilised. It still bites, so the fix did not make the test green by weakening it.

The full mise run conformance pass remains unplayed, as written in the description. fields carries every client, and it is the only leg whose gate judges the sentence this change edited.

Logs: .../scratchpad/replay/<spec>.log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A fault receipt line in flight is indistinguishable from a finished one, and the test loses that race 4 times in 10

1 participant