Skip to content

RedundancyManager: a cycle that closes late counts against no replica (MRU-RM-023, v0.6.0) - #36

Merged
wbnns merged 3 commits into
mainfrom
fprime-late-cycles
Oct 22, 2026
Merged

wbnns merged 3 commits into
mainfrom
fprime-late-cycles

Conversation

@wbnns

@wbnns wbnns commented Oct 22, 2026

Copy link
Copy Markdown
Member

The flaw. On one processor, when the manager's thread falls behind (a slow state-file write, a starved processor), the rate group's next tick waits in the manager's queue in front of the replies. Each cycle then closes before its answers are dispatched, and after three cycles every replica is retired as LOST and the run halts. CI hit it once in the restart test (PR #35, all three replicas retired). Under CPU starvation in a container (half a CPU, six busy loops, 20 ms cycles) it halted at cycle 4 on every try.

The rule (MRU-RM-023). A cycle closes late when other messages wait in the queue as it closes, or when every one of two or more expected answers is missing. In such a cycle a missing answer counts against no replica (no miss, no known-answer test or admission failed for a missing answer); a cycle with an answer missing is withheld (nothing delivered, no strike), counted in the new Late telemetry with a throttled CycleWithheld event. A reply that arrives after its cycle closed clears the replica's misses. One replica left: silence counts as before.

Evidence.

  • 4 new unit tests (44 in all): answers queued behind the next tick, a known answer queued behind the tick, common silence, a replica that always answers late. With the rule disabled the manager suite fails.
  • Parity test unchanged: 36 of 60 identical, wrong results identical in all 60.
  • Load test (by hand): 15 starved runs, halted 9 times before and 0 after; replicas lost without cause 28 before, 14 after; 0 wrong in all. The losses left need one replica thread starved for three whole cycles (300 ms at the package's 100 ms cycle).
  • Deployment check and the full run 3 package in ESA's image (local ARM build): 147 s, the same counts and events as v0.5.1, 0 withheld.

Also. RunRecorder reads Late; the summary reports late=; scripts/fprime-deployment.sh allows one count of slack per withheld cycle. Requirements D3, MRU-RM-002, 009, 010, 014 updated, MRU-RM-023 added. Version 0.6.0.

wbnns added 3 commits October 22, 2026 19:12
… (MRU-RM-023, v0.6.0)

When the manager falls behind, the next cycle's tick waits in its queue in
front of the replies, so each cycle closed before its answers were
dispatched; three such cycles retired every replica as lost and halted.
Seen once in CI (the deployment's restart test) and reproduced under CPU
starvation. A cycle now closes late when other messages wait in the queue
as it closes, or when every one of two or more expected answers is
missing; then a missing answer counts against no replica, and a cycle with
an answer missing is withheld. A late reply clears the replica's misses.
New event CycleWithheld, telemetry Late, summary late=. Four unit tests;
requirements MRU-RM-002, 009, 010, 014, 023 and D3.
@wbnns
wbnns merged commit 215547b into main Oct 22, 2026
28 checks passed
@wbnns
wbnns deleted the fprime-late-cycles branch October 22, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant