diff --git a/src/MeshWeaver.Documentation/Data/Architecture/TeardownVerdictsAreCausal.md b/src/MeshWeaver.Documentation/Data/Architecture/TeardownVerdictsAreCausal.md index db85056b19..e6ab23ed4a 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/TeardownVerdictsAreCausal.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/TeardownVerdictsAreCausal.md @@ -206,6 +206,31 @@ re-attempt is refused quickly during a teardown β€” i.e. the assertion was readi no right to observe. `ArmedRequestIds` exists for exactly this (its own remarks say so): name the correlation the verdict has to land on, and a second, unrelated arm cannot falsify it. +### A third: an ARMED watch is not an ACCEPTED patch + +The fence before the teardown waited for the caller's late watch to be armed and read that as "the +owner has the patch". It is not the same fact. The caller arms the watch **before** it posts +(#2882: register, then post, so no answer can slip past), so an armed watch means the patch was +SENT. The disposal NACK is registered by the owner's patch handler, which runs later. If +`Mesh.Dispose()` lands between the two, the patch reaches an owner that never registered a NACK. +That owner goes `Dead` owing nothing, and the assertion fails on a precondition the test never +established. + +That was the test's one red in the merge queue (run 36019287652, shard 4). The watch was armed at ++52 ms, `Mesh.Dispose()` ran 1 ms later, and the owner was `Dead` within the next 50 ms poll. The +watch was still armed and the window held no framework warning. It could not be reproduced by +looping: 0 failures in over 400 runs, alone, in bulk, on 2 CPUs, at the failing commit and on +`main`, because the handler normally runs 1–5 ms after the post. Holding the post back by 120 ms +after the watch arms reproduces it **40 of 40** with the identical assertion text. With the fence +below, the same delayed build passes 40 of 40. + +The fence now observes acceptance, and does not assume it. The handler stamps +`PATCH_MERGE_DISPATCHED` on the tree's request-fate ledger strictly after +`RegisterOwnerDisposingNack`, so the test waits for `PATCH_MERGE_DISPATCHED@` in +`Mesh.DescribeRequestFate(armedId)`. That is the same "accepted is observed, never assumed" wait +`DisposalRaceNackTest` makes on `ENQUEUED@`. Generalized: **a fence must observe the fact +that the assertion depends on, not an earlier step that usually comes just before it.** + ## The rule, generalized **If an answer is produced by a `RegisterForDisposal` registrant, a `ShuttingDown` handler, or diff --git a/test/MeshWeaver.Graph.Test/NackReachesTheWaiterDuringTeardownTest.cs b/test/MeshWeaver.Graph.Test/NackReachesTheWaiterDuringTeardownTest.cs index 36969975a0..a366c58c44 100644 --- a/test/MeshWeaver.Graph.Test/NackReachesTheWaiterDuringTeardownTest.cs +++ b/test/MeshWeaver.Graph.Test/NackReachesTheWaiterDuringTeardownTest.cs @@ -179,6 +179,25 @@ await Observable.Interval(TimeSpan.FromMilliseconds(50)).StartWith(0L) var armedId = registry.ArmedRequestIds.Single(); Output.WriteLine($"[fence] caller is armed on the late watch (request={armedId})"); + // 🚨 …and the OWNER has ACCEPTED it β€” OBSERVED, never assumed. The caller arms its late + // watch BEFORE it posts (#2882: register, THEN post, so no answer can slip past), so an + // armed watch says the patch has been SENT, not that the owner's handler has run. The + // disposal NACK this test is about is registered by that handler; a patch still in + // transit when the mesh goes down reaches an owner that never registered one, the owner + // goes Dead owing nothing, and the assertion below then fails on a precondition this test + // never established rather than on #2778. That is exactly the one red this test produced + // (queue run 36019287652: armed at +52 ms, Mesh.Dispose() 1 ms later, owner Dead inside + // the next 50 ms, watch still armed, not one framework warning). Holding the post back + // by 120 ms after the watch arms reproduces that failure 40 of 40. The handler stamps + // PATCH_MERGE_DISPATCHED strictly after RegisterOwnerDisposingNack, on the tree's shared + // request-fate ledger β€” the same "accepted is observed" wait DisposalRaceNackTest makes. + var acceptedStage = $"PATCH_MERGE_DISPATCHED@{owner.Address}"; + await Observable.Interval(TimeSpan.FromMilliseconds(20)).StartWith(0L) + .Select(_ => Mesh.DescribeRequestFate(armedId)) + .Where(trail => trail.Contains(acceptedStage, StringComparison.Ordinal)) + .FirstAsync().Timeout(TestTimeouts.Convergence).Await(ct); + Output.WriteLine($"[fence] the owner accepted the patch β€” {acceptedStage} is on its trail"); + // 🚨 The scenario. Dispose the MESH, not the node hub: the owner's disposal action then // runs with its parent already past DisposeHostedHubs β€” the exact window in which the // old guard skipped the post and the caller heard nothing.