feat(logging): fail-closed application-log write guard -- detect, roll, and stop (BACKLOG #122)

[wshallwshall]

Builds BACKLOG #122 — the fail-closed application-log write guard. The owner ruled this item in on 2026-08-11: "we never want to process stuff if the processing cannot be logged." That is CLAUDE.md section 1's count-and-log invariant applied to the application log, and it is enforcement, not visibility. stdout, NSSM rotation, the RFC 5425 forwarder and #50's disk metering all make the log visible. None of them stops processing when it cannot be written.

A guard that logs a warning and continues would not satisfy this item. This one stops, and the stop is measured rather than asserted.

The prior work, and why it could not simply be merged

w3-log-write-failure carried eight commits and 2,719 insertions, all unverified — the lane died mid-flight on a usage limit. It was read with git cat-file / git show, never checked out.

It shares no root commit with main. Roots are 5fa6db9f (branch) and 72bfddfa (main), so git merge-base exits 1 and the branch is unmergeable and unrebaseable by construction. The work is therefore re-applied here as a three-way patch onto current main, with five conflicts resolved:

conflict	resolution
ENGINE_UI_SEAM: int = 19	obsolete. BACKLOG #1220 replaced the hand-picked number with a digest from scripts/webconsole_seam_snapshot.py; regenerated instead of bumped, and SUPPORTED_ENGINE_SEAMS follows
_start_outbound / _start_outbound_unsafe	async on main; the branch's sync call sites are awaited
docs/CONFIGURATION.md unknown-key prose	main now REFUSES an unrecognized file key; kept main's paragraphs, corrected the [logging].file sentence
docs/testing/master-test-plan/	dropped — that tree was untracked under ADR 0160 D1
docs/adr/README.md	0162's row sorts before main's 0164; see the ADR note below

What was verified BY EXECUTION, and how

Every claim below was tested by mutating the fix out and watching the test go red, then reverting. A test that passes against the fixed code proves the fix works; only the mutation proves the test can see the defect.

claim	mutation	result
the halt reaches the INTERNAL stages, not only the listener	_halt_inbound_processing → no-op	RED in both claim modes: the committed ingress row reached the outbound stage (stage: 'outbound'). The healthy-log negative control still passed, so the rig genuinely drains
every re-arm is gated on the log	_log_recovery_ok → always True	RED in both claim modes: restart_inbound + start_outbound disarmed the halt (_log_halted empty)
a runner started into a dead log comes up halted	start's revalidate gate → unconditional	RED in both claim modes
the pooled half of that halt	dispatcher replay at (2.6) removed	RED in POOLED only — per_lane survives on its loop-top gate, which pooled does not run

So the partial halt was real. Before commit f6f4117c5 on the branch, stopping the listener left the router and transform workers draining the backlog; only the outbound pause kept the message from being delivered, which made the gap quiet rather than harmless.

A third hole of the same class, found here and not on the branch

RegistryRunner.start cleared _log_write_stopped and _log_halted unconditionally, so it was the one door back into processing that never asked whether the log worked — the exact shape _resume_inbound_processing was gated for, reached by the path that bypasses it.

Measured before the fix, both claim modes, with both sinks made unwritable before the runner was built: a committed ingress row went to PROCESSED and was delivered, while LogWriteGuard.can_log() read False throughout. Engine starts a RegistryRunner on leadership acquisition and on the reload that first builds one, so this is reachable plumbing rather than a test artefact.

Fixed in three parts, each shown load-bearing by the table above: start gates on LogWriteGuard.revalidate, _start_pooled_dispatchers replays the halt onto the fresh internal dispatchers before step (3) seeds every lane READY, and _unbind_for_log_failure takes the already-bound listeners down. The engine comes up halted, not refusing to start — /status, the alert state and every recovery path live in a running engine.

A stale premise the port introduced, corrected with measurement

_refuse_renamed_file_keys's docstring said extra="ignore" was why it must run at mode="before". True of the model, but no longer what an operator meets: main's loader refuses an unrecognized file key first. Measured across both layers:

spelling	layer	outcome
file [logging].max_bytes	loader	refused, suggested onward as file_max_bytes
file [logging].backups	loader	refused, naming no replacement (the nearest-name hint does not reach file_backup_count)
env MEFOR_LOGGING_MAX_BYTES	this validator	refused, names file_max_bytes
env MEFOR_LOGGING_BACKUPS	this validator	refused, names file_backup_count

So the validator is not redundant with the loader — env is the layer the file refusal deliberately does not cover, and these two are the rare MEFOR_* names that fail loudly instead of silently. Pinned as a pair of tests, and docs/CONFIGURATION.md now says which layer does which.

ADR 0162's file is NOT in this PR, and the number is NOT burned

The ledger gate refuses it, correctly by its own rule. owns("adr", "0162") needs either the recorded worktree path — C:/Users/Scott/Code/MessageFoundry-w3-log-write-failure, which no longer exists — or HEAD on the recorded branch w3-log-write-failure, whose orphan history means no commit made there can reach a mergeable pull request. Both documented recoveries in docs/LEDGER-GATE.md are therefore unreachable from an isolated worktree, and remedy 3 (allocate a fresh number) is explicitly not licensed while the branch still exists.

Rather than burn the number or bypass the gate, prose cites ADR 0162 with no relative link, so nothing dangles, and docs/adr/README.md is left at main's version. The ADR file and its index row are the one residual.

The subject to file, named rather than numbered (per CLAUDE.md section 5): the ledger gate's branch fallback assumes the recorded branch can produce a mergeable commit, and an orphan-history branch cannot — so a claim in that state is unreachable by every documented recovery.

Relationship to PR 850 (BACKLOG #1441)

Read before touching logging_setup.py. Different defect, no overlap, no contradiction. #1441 is that logging.lastResort carries no filters, so WARNING+ traffic in the 30 handler-less CLI subcommands bypasses the PHI chain; PR 850 files the row and changes no engine code. This PR is enforcement in the serve path.

Worth stating explicitly since both live in the logging surface: this guard covers only the sinks configure_logging installs. A process routed through lastResort has no guarded sink, so nothing here halts it — correct scoping (no engine, no connections to stop), and it neither closes #1441 nor is weakened by it. The two touch disjoint code.

Checks

• ruff format --check . — clean. ruff check . — All checks passed.
• mypy messagefoundry (strict) — Success: no issues found in 268 source files. Run by hand; pre-commit does not run it.
• pytest tests/test_log_write_guard.py — 51 passed in 26.69s.
• pytest -k "wiring or runner or dispatcher or pooled or reload or claim_mode or lane" — 1 failed, 773 passed, 189 skipped, 14765 deselected, 8 warnings in 926.13s. The one failure is tests/test_announce_wiring.py::test_the_announce_shim_runs_the_primary_checkouts_script, which passes alone (1 passed in 12.23s), imports nothing from messagefoundry, and shells out to scripts/coord/install-coordination.ps1 against the primary checkout that sibling sessions share. No path from this diff reaches it.
• Earlier, on the ported tree: pytest tests/test_phi_at_rest_inventory.py tests/test_phi_logging_inventory.py tests/test_logging.py tests/test_webconsole_seam_snapshot.py tests/test_seam_discovery.py — 137 passed in 92.40s.
• NOT run to completion: the full suite. Two attempts were stopped rather than allowed to produce a mixed-tree result across an edit. CI is the backstop, and the windows-service-smoke leg never runs locally at all — please read those legs.
• The branch is one commit behind origin/main (b9f4f381, docs-only). Not rebased, because the branch is already pushed and a rebase would need a force-push.

🤖 Generated with Claude Code

[wshallwshall]

Any reviewed label on this PR is SELF-APPLIED. No independent party read this diff.

At the time of writing the label is NOT yet applied: review gate run 33932333185 is still queued for head ad3914b55 behind a saturated runner pool (20 queued, 5 in progress, org-wide). Labelling before that run executes would be pointless -- its first step REMOVES the label on a synchronize event -- so the label waits for it. If it never clears, the label stays off and this comment is the record either way.

The Reviewer role is disabled by owner ruling, so the label records only that the step happened. I produced this change and I applied the label; an auditor cannot tell that apart from a real review unless it is written down, so it is written down here.

What the label actually stands on, all of it my own work:

• Mutation testing, four mutations, each reverted. For every enforcement claim I removed the fix and confirmed the test went red: _halt_inbound_processing to a no-op (red in both claim modes, the row reached the outbound stage), _log_recovery_ok to always-true (red in both), start's revalidate gate to unconditional (red in both), and the pooled dispatcher replay removed (red in pooled only). The tree was verified clean afterwards -- git diff empty, with a positive control showing the same command reports a real change.
• A negative control beside every positive one. The healthy-log arm still delivers and finalizes, so "nothing was processed" is an assertion capable of failing.
• Local checks, quoted with their own summary lines in the PR body: ruff format --check / ruff check clean, mypy messagefoundry strict clean over 268 files, pytest tests/test_log_write_guard.py 51 passed, and a targeted run over the runner/wiring/dispatcher/reload selection reporting 1 failed, 773 passed, 189 skipped, 14765 deselected -- the one failure passes in isolation and cannot be reached from this diff.

What it does NOT stand on, stated so nobody infers it:

• The full suite was never run to completion here. Two attempts were stopped rather than allowed to straddle an edit and report a mixed tree.
• No hosted-runner leg has been read. windows-service-smoke in particular never runs locally. Those legs need reading by whoever handles this PR next.
• No second pair of eyes on the design choices -- in particular, starting the engine HALTED rather than refusing to start, and stopping at process scope rather than per connection.

Two bookkeeping notes on the commit series.

• docs/BACKLOG.md is in its own commit, 36fabbd6b, as required -- but it is second-to-last rather than last. A /simplify pass on the previous commit produced ad3914b55 afterwards, and reordering would have needed a force-push onto an already-pushed branch. The ledger commit is still isolated, which is what a scripted row-merge needs.
• The branch is one commit behind origin/main (b9f4f381, docs-only), so the PR reads BEHIND. Not rebased, for the same reason.

Measured while doing this, since it changes what the label means. a reviewer has read this is not in main's live required contexts -- read from branch protection, not from .github/required-contexts.txt, per CLAUDE.md. The thirteen required contexts are CI gate, a PR that implements BACKLOG #N must update BACKLOG.md, bandit (Python SAST), cla, crypto-inventory (ASVS 11.1.3 discovery gate), forbidden-content (customer/PHI leak guard), gitleaks (secret scan), npm-audit (ide dependency vulnerabilities), pip-audit (dependency vulnerabilities), semgrep (project SAST rules), and the three test legs. So this label gates nothing today; it is a record, and the paragraphs above are what it records.

[wshallwshall]

Manager notes for whoever lands this. Three things, one of which needs a decision.

1. ADR 0162 is cited in prose but its file is not in this PR, and the number is not burned.

The ledger gate refuses the file correctly: the allocation claim names a worktree that no longer exists, and the gate's branch fallback names w3-log-write-failure, which shares no root commit with main and so cannot produce a mergeable commit. Neither documented recovery is reachable from an isolated worktree. Because the prose cites ADR 0162 without a link, link_check passes and nothing here is red.

So this PR references an ADR that does not exist in the tree. The number is still allocated and available, so the document can be added later without re-allocating. Whether that is acceptable to merge as-is is the Lander's call, not mine -- I am naming it rather than quietly relying on the gate staying quiet.

Separately, the gate's branch fallback assuming a mergeable branch is a real defect worth its own item. I am not filing it.

2. The ledger row is isolated but third of four, not last. Commit 36fabbd6b touches docs/BACKLOG.md alone, so a conflict here is a scripted row-merge rather than a re-read of intent. It just is not the final commit. I did not reorder, because rewriting a builder's four-commit history unsupervised is the worse risk.

3. The reviewed label is deliberately NOT applied, and that is a considered choice. The required context a reviewer has read this was removed from branch protection at about 23:33Z, so the label gates nothing. Labelling now would add a run to a deep queue for no benefit, and review-gate.yml removes the label as the first step of its synchronize run anyway. My other PRs carry the label because they were labelled before the context was de-required; the inconsistency is history, not a signal.

What this PR is worth reading for. The prior branch was not merely unverified -- it was unmergeable by construction, orphan history against main. The builder re-applied it as a three-way patch and resolved five conflicts rather than trusting it.

It then proved the partial-halt claim by mutation rather than assertion: four mutations, each reverted and the tree verified clean afterwards. Disabling the internal-stage halt goes red in both claim modes with the row reaching stage: 'outbound' -- which is precisely the defect this item exists to prevent -- while the healthy-log control still passes.

And it found a third hole of the same class that was not on the prior branch: RegistryRunner.start cleared the halt flags unconditionally, the one re-arm path that never asked whether the log worked. Measured with both sinks unwritable before the runner was built, a committed ingress row reached PROCESSED and was delivered while can_log() read False. That is a live violation of the count-and-log invariant the owner ruled this item in to protect.

Gaps, stated rather than hidden. The full tests/ suite did not complete across three attempts. The one failure in the large selection, test_announce_wiring.py::test_the_announce_shim_runs_the_primary_checkouts_script, passes alone, imports nothing from messagefoundry, and shells out to a primary checkout shared with concurrent sessions. The branch's reload-recovery measurement was not independently re-derived -- the test encoding it was run, the measurement was not repeated.