fix(retention): let a dead row at ANY stage ride the dead-letter window (BACKLOG #1188)

[wshallwshall]

What this is

One of the builds BACKLOG #1188 names, landed. The item stays open and the ASVS cell is not re-scored — #1188 names several more builds, and this closes one of them plus a second over-retention that rode the same defect.

I did not touch the vault scorecard.

The defect, re-measured at this branch point

Both body purges scoped every queue statement to Stage.OUTBOUND, so a dead ingress or routed row was reached by neither. Those rows carry the full raw body:

Router raise dead-letters the ingress row	pipeline/wiring_runner.py:4996
Handler raise dead-letters the routed row	pipeline/wiring_runner.py:5030
purge_message_bodies	store/store.py:8919
purge_dead_letters	store/store.py:9252

Because dead is neither pending nor inflight, purge_message_bodies still found the message eligible and blanked messages.raw. So the message read as purged while a full raw PHI body would survive in queue.payload indefinitely, with no sweep able to reach it. On a first deployment that would accumulate raw PHI bodies with no engine-managed deletion. The test asserts that hazard directly before asserting the fix.

A second over-retention rode the same defect, and it is not in the item. _attachment_still_referenced_sql is already stage-agnostic — a dead row with a non-blank payload counts as a live holder whatever its stage. So an unreachable dead ingress row also pinned the message's detached streaming document (the very large PHI payload, e.g. a base64 PDF) forever. Blanking the row makes the predicate go false and the decref fire, so it closes with the same change. There is a test for it.

Why widening is honest here rather than convenient

replay() recovers "a dead-lettered ingress/routed row" by re-queueing it in place from its own payload. That is exactly the property that justified giving dead outbound rows a later window than the body, so the symmetry the research claimed holds in shipped code.

The stage predicate is dropped, not widened to a three-stage list, so a stage added later is covered by construction instead of silently re-opening this gap.

Two things deliberately stay scoped, both with the reason in the code:

• SQLite's store-once body_ref release. body_ref is written only by _insert_outbound_deliveries, so an ingress/routed/response row always has a NULL body_ref. A wider scope would match nothing extra.
• The dead-letter VIEW (list_dead, replay_dead). What an operator can enumerate is a separate question from what retention must delete — and an unenumerable body is the stronger reason to bound it, not a reason to leave it.

Assumption I had to make, stated per the one-turn rule

Per-connection overrides key on destination_name, which is NULL at ingress/routed/response, so CASE NULL matches no WHEN arm and those rows take the ELSE — always the global window. I chose that over inventing an inbound-keyed override, because dead_letter_days is declared on an OutboundConnection and no configuration surface declares an inbound-keyed one. Giving those stages their own override is a separate, unallocated follow-up. Verified as SQL semantics in all three dialects, and stated in every docstring.

Classification edits, in the same commit as the item asks

This closes half the tier, so a plain window claim would be false.

• docs/PHI.md §2 gains an eighth vocabulary form, dead-only, on the exact shape of the existing orphan-only — one state of a tier covered, the other not.
• The §8 residual row is rewritten, not deleted, to name the surviving gap.
• The §8 purge table and the serve-gate comment in __main__.py now state the real scope.
• Dead anchors inside PHI.md corrected (wiring_runner.py:4415/:4449 were 581 and 581 lines off).

The retention floor is untouched. MIN_PHI_RETENTION_WINDOWS is still 9, and no window was auto-bounded — the 2026-07-30 no-auto-bound ruling stands unchanged.

What is explicitly NOT done, so nobody re-scores on this

The permanently-pending ingress row is still unbounded, and it is now the tier's stated residual. Three sites re-pend with RetryPolicy(max_attempts=None) when the owning inbound is absent from the registry (wiring_runner.py:5057, :5296, :5480) — by design, so an ACKed-but-never-attempted message is never dropped for outliving a reload. There is no ingress-stage counterpart to the dead_letter_missing_destinations/_handlers startup sweeps, and adding one collides with that intent. Bounding it needs a rule telling "mid-reload" from "never coming back", which is a decision, not a patch.

I originally wrote that those two sweeps re-pend. They dead-letter. I caught it by reading them rather than trusting my own summary, and corrected the row before committing.

Also untouched and still open under #1188: the declared-set reference purge, the spill-directory sweep, the IDE saved-collection classification, a non-write-time eviction key for transform state, the preset ruling re-put on corrected facts, and the enumeration of PHI bodies written outside the store.

Anchor drift, re-measured

Real, and in one case 581 lines.

Cited	Actual
retention_classification.py:173	:181 (value still 9)
store/store.py:8384, :8659	:8919, :9252
wiring_runner.py:4415, :4449	:4996, :5030

Tests — each proven to fail first

Three behaviour tests, all failing with assert 0 == 1 on unmodified code before the fix:

• test_purge_dead_letters_reaches_a_dead_ingress_row — also asserts the hazard: messages.raw blanks while the ingress payload survives.
• test_purge_dead_letters_reaches_a_dead_routed_row
• test_dead_ingress_row_does_not_pin_a_streaming_attachment_forever

Plus test_no_backend_scopes_the_dead_letter_purge_to_one_stage, a source-level parity guard, because the Postgres and SQL Server purges run on hosted runners only — a re-narrowing there would otherwise reach main unseen. Mutation-tested: re-narrowing the Postgres statement reds it (assert 1 == 0).

Synthetic HL7 only. No real PHI in code, tests, logs or messages.

Checks run

Interpreter resolved into this worktree, as required:

$ /c/Users/Scott/Code/MessageFoundry/.venv/Scripts/python.exe -c "import messagefoundry; print(messagefoundry.__file__)"
C:\Users\Scott\Code\MessageFoundry\.claude\worktrees\agent-a9c318f08aca9fc31\messagefoundry\__init__.py

Check	Result
pytest tests/test_retention.py tests/test_retention_classification_drift.py	64 passed
pytest per-connection retention + attachment substrate + metadata bag	46 passed, 28 skipped
pytest accepts seam + reingress	47 passed
pytest PHI at-rest + logging inventory + claude citations	67 passed
pytest reference sets + redaction + refused-config-keys	349 passed
ruff check . / ruff format --check .	passed / 1241 formatted
mypy messagefoundry (strict)	no issues, 267 files
backlog_status_check / backlog_citation_check / dangling_citation_check	exit 0
pytest backlog citation + hygiene + status + ledger guards	101 passed

Postgres placeholder renumbering verified without a live database — both statements contiguous and complete at $1..$N, with and without overrides.

Not run: the full suite (BACKLOG #1014 port collision). Postgres and SQL Server store legs are hosted-runner only — this change touches store SQL on both, so CI is the authority there. Please read those legs plus windows-service-smoke. gh pr checks renders CANCELLED as "fail"; tally the check-runs API instead.

Commits

BACKLOG.md is in its own commit, last, as asked.

1. bbee82064 — the fix, tests, and classification edits
2. 6eed8177c — the BACKLOG.md note

🤖 Generated with Claude Code