[stephenctw]

When the settlement sender disappears, a finished Dave tournament can remain unstaged or an eligible staged result can remain unaccepted. Inputs can keep reaching L1 while outputs from the blocked epoch remain unavailable. This PR adds independent detection and operator-driven recovery for that specific F4 scenario.

Related fear: F4 — the app stops settling and nobody notices, from Mugen-Builders/qa-redteam-v0#9, specifically the won epoch that nobody stages or accepts. This addresses the sender-outage case, not every cause of F4.

Solution

• Detection: settlement-watch.py reads contract state independently of Sling, pins views to one block, and reports the current settlement phase. An unchanged unstaged winner or eligible acceptance becomes stalled after an operator-configured block allowance. Normal tournament/staging waits, failed roots, foreclosure, and observation errors remain distinct.
• Recovery: settlement-recover.py exports settlement material from a replay directory or verifies independently supplied evidence against chain state and simulates an unsigned staging/acceptance transaction. Exporting stored material does not itself establish correctness.
• Operator workflow: settlement-operator.py combines the tools. On a detected stall it asks whether to prepare recovery, verifies and simulates the action, then asks separately before submitting through a configured signer. It rechecks and simulates again after operator review. The default response and EOF decline; without a signer it prepares only.
• Runbook: documents reproduction, monitoring and signer configuration, evidence requirements, recovery verification, and the limits of each verdict. Neither the observer nor standalone preparation submits transactions; the wrapper submits only the explicitly confirmed action and never forecloses.

A winner is not automatically safe to settle: a correct terminal application can win but have no acceptable machine-validity proof. Recovery therefore requires checked replay material and a successful simulation rather than blindly progressing whichever result won.

Reproduction and evidence

Run just test-settlement-liveness in the configured development environment after just doctor passes.

Four deterministic Anvil scenarios exercise real Sling workers, Machine execution, and deployed contracts:

1. Outage before staging: settle a healthy epoch, execute an input in the next epoch, obtain its winner, then drop the sender's workers. Another input lands, but settlement does not advance and the completed epoch's outputs root stays invalid. A fresh runner replays L1 in its own state directory. The operator wrapper exports its material, stages and accepts through another account, and the test verifies receipts, the newly valid outputs root, and pending-input rollover.
2. Outage after staging: the detector respects the normal staging period, flags acceptance only after eligibility and the allowance, and another account restores progress.
3. Phase distinctions: an open tournament, a root without a winner, and foreclosure receive different verdicts.
4. Terminal winner: the true commitment wins, but staging its proof fails; the detection signal is not treated as permission to settle.

The worker stoppage is deterministic harness fault injection, not a process-signal test. The fresh runner shares Sling's implementation and emulator; this is not independent implementation assurance. The fixture establishes output-root eligibility, not an ERC-20 payout.

Validation and CI

Local checks passed:

• Nine detector tests and eight recovery/operator tests, including mismatches, unavailable actions, failed simulation, both confirmation boundaries, and changed results during operator review.
• Four chain-backed scenarios, including actual wrapper-driven staging/acceptance using exported replay material and a local RPC-managed signer.
• cargo check --locked.
• cargo clippy --locked --workspace --all-targets -- -D warnings.
• cargo fmt --all --check and git diff --check.

just test-settlement-tools is included in just check and the existing CI test job. Chain-backed scenarios run through the existing just test-node-harness CI step; they are not duplicated in the Python test step. Remote CI has not run yet at PR creation.

Compatibility and deployment limits

No contract, consensus, wire-format, or storage-schema changes. The tools target this checkout's contract ABI and stored proof layout. Block allowances are deployment settings; the four-block test value is not a mainnet recommendation.

This is operational hardening of an existing liveness dependency, not a demonstrated consensus-safety defect. Production keystore integration, paging/response latency, a running Sequencer during the outage, and actual ERC-20 withdrawal execution remain deployment checks. External supervision must detect observer absence and persistent observation failures. Transaction submission is reported separately from confirmation; operators must verify receipts and the expected epoch transition.

Red-team reproduction record

The primary incident and triage entry is now RT-16 / qa-redteam-v0#16. It contains pinned reproduction steps, a test-only single-sentry variant, and logs from canonical two-level and CI three-level fixtures. That incident PR is reproduction-first; the detector and recovery implementation here remains an optional solution resource for maintainers to prioritize.

The fixture-clock correction in this PR keeps synthetic block timestamps from moving ahead of real wall time when newly generated devnets fast-forward protocol clocks. Production timestamp freshness checks are unchanged. The extra single-sentry test is supplied in the red-team reproduction kit rather than added as new production logic here.