docs(reliability): add offline ledger retention and recovery rehearsal

[LIHUA919]

Reliability diagnostics needs a reproducible operator path for whole-ledger export, deletion readback and recovery. This PR adds a bilingual manual offline rehearsal using the existing ledger and CLI, with a smoke that executes the literal documented shell steps through source and non-editable installed packages.

Addresses the bounded operations-preparation slice of #5211. It does not close the issue or qualify P0 exit.

• Require a frozen writer, exact record ownership, owner-selected finite retention period and scoped deletion/recovery authority. Preserve complete bytes, failure markers and fixed-time receipt/projection; delete must read back invalid/no_observations, and restore uses exclusive creation.
• Explicitly reject symlink ledgers and directories. Require the actual frozen provider directory to match the canonical <runtime-root>/reliability_diagnostics layout before CLI readback or export. Arbitrary provider directories remain supported by the provider but are outside this manual recipe; never infer their mapping from the parent directory.
• Replace the approximate Python-only rehearsal with literal-shell regressions for normal and invalid recovery, symlink, foreign/mixed/malformed/colliding ownership, source/copy tampering and occupied restore destinations. Exercise the existing DSH resolver and real file appender against canonical, custom and symlink directories through the installed CLI.
• Link the packaged guide from both capability READMEs and reconcile the bilingual RFC checkpoint.

Validation at ae2fb573e51306a4891e38acc9c351c97ce56d20, based on main at fd5f31bb3: 94 capability/provider tests; source and non-editable wheel retention smoke (11 literal-shell cases and 3 real producer-directory cases); Chat bundle and wheel builds; Chat HTTP recovery and upgrade smokes; Ruff, docs-governance, diff checks and public-boundary scan. Risk-based premerge passed all selected checks (3 diff, 1 compile, 9 catalog, 8 risk-profile and 1 boundary); its initial sandbox run had 3 runtime-startup failures that passed with local communication endpoints allowed. Current-head CI and independent re-review are required before merge; previous-head CI results do not qualify this revision.

Entry points: operator documentation and synthetic CLI validation. The existing CLI schema, default-off provider and frontend/Lark configuration contracts are unchanged, so no companion editor change is required. README additions follow the existing use instructions and leave the opening viewport unchanged.

Additional UI evidence: the full packaged personal-workspace smoke stopped at steward-model-settings with focus not returning after closing settings (19 prior scenarios passed; later scenarios were not run). The full retry reproduced it, while the isolated scenario passed. This PR leaves the frontend/runtime/scenario sources unchanged relative to its pinned base. Keep this failed local result visible; it is not a full packaged-UI pass or a verified base/latest-main attribution.

Remaining evidence: real observer shutdown, live C0/C1 and matched worker outcomes, CPU/RSS/bytes/latency, deployment-owner retention/deletion acceptance, automatic TTL, secure erasure and tenant isolation. No live case, benchmark or deployment was launched.

Future-facing pass: bring the actual shell/path boundary into the existing owner-local smoke; the current capability and provider remain sufficient without a new API or scheduler. Existing commits are preserved. Please leave merge to the maintainer.

[LIHUA919]

CI follow-up at head 422b30005b48aa723f3ca240cd16fb081000f5f0:

The two failed shards share a stale project-registry I/O census location. The same failures are present in the base main pipeline (run). #5239 moved check_contract's load_registry call from line 1014 to 1027; the manifest still recorded 1014.

The appended DCO-signed commit refreshes only that line. AST readback preserves all 250 sites, their classifications and the source policy. Both architecture test modules pass locally: 9 passed; the public-boundary scan and diff check are clean. Existing test assertions and enforcement rules are retained.

The new exact head has been pushed and CI restarted. Independent maintainer review/integration, real DSH C0/C1, measured observer overhead and deployment policy acceptance remain open; this correction does not qualify P0 exit.

[huangruiteng]

结论：REQUEST_CHANGES。新增 retention recipe 的符号链接检查没有 fail-fast，实际会继续 export/delete/restore；这是公共操作指引本身的 P1，不是未归因的红 CI。

English verdict: REQUEST_CHANGES - The documented POSIX guard does not stop a symlink ledger before export and deletion.

动机

#5211 与 reliability RFC S10/S11/S13 的这个有界切片，要给停止采集后的 operator 一条安装版 CLI 可复演的 export/delete/recovery 路径，并保留 degraded/invalid 的负向证据。它不需要顺便做自动 TTL、worker control 或完整 live prototype；当前 RFC 的 live C0/C1 与 CPU/RSS/bytes/latency hold 仍应保持。避免“恢复后看起来健康、实际丢了原证据”是有价值的长期目标，但新的用户操作路径还必须可靠拒绝不支持的文件形状。

改动思路

复用现有 ledger、status、receipt 和 synthetic fixture，先冻结并读取完整证据，再 raw-byte copy、hash/cmp、独立 readback；删除后必须呈现 invalid/no_observations，恢复只允许向不存在的目的文件 exclusive-create。它没有新增 restore API 或平行 retention owner。保留原字节与重新 ingest 并不等价：ingest 可能把不合法字段改写成 violation marker，因此不能用只回灌 accepted rows 代替历史恢复。

权限链仍是 operator 负责停采集、期限/hold、删除和重新保留的决定；安装、README 链接、CLI 可用不代表 observer 已开启。Goal/Todo/quota/gate/session 都不属于这个手动文件操作范围。

具体改动

七个文件、+396/-4：215 行双语 local-retention 指引，151 行 real-CLI durable smoke，双语 README/RFC 的短链接与 checkpoint，以及一处 census line anchor。没有新增生产 provider、持久化格式、scheduler 或 CLI retention 命令。README 入口位于已有内容之后，不改变 first viewport。

关键代码讲解

• main：创建 disposable synthetic runtime；--installed 检查 import 位于 checkout 外，分别演练 degraded 与包含持久 refusal marker 的 invalid 状态，而非跑 live observation。
• main.cli：用同一 sys.executable -m loopx.cli 在 checkout 外执行真实入口，设置隔离 runtime，校验进程退出和 JSON ok；不依赖嵌套 uv 或全局另一个 release。
• main.readback：固定 as_of 后比较完整 status/receipt；sequence gap、lost/drop、clock uncertainty 和拒绝记录必须仍存在，且 control siblings 不变。

这些 checks 有 durable regression 价值，但 smoke 用 Python 构造普通文件并 unlink，并没有执行下面的新 POSIX file guard；正常文件 green 不能覆盖实际 guide block。

对主干的风险

[P1] recipe 第 91 行 的 test ! -L "$diagnostic_ledger" && test -f "$diagnostic_ledger" 不会在 symlink case 终止 set -e 的 POSIX shell：失败的是 AND-list 第一个命令。后续 ownership parser 与 cp 会 follow link，rm 只移除 link，exclusive-create recovery 再把原路径变成普通文件。

我用同一 fixed-as_of synthetic fixture、实际文档 block 和 base/head 的 non-editable installed CLI 独立执行：symlink case 都 exit 0，export/delete/restore 全跑，link 被替换而 backing file 保留。原 byte/readback equality 仍能通过，恰好说明它不能证明文件边界。这个 recipe 是本 PR 新增的；在旧 runtime 上也复现是定位到新增指引，并不是可以忽略的 pre-existing unrelated failure。仅在 reviewer counterfactual 加显式 || exit 1 后，同一 case 才 exit 1，且没有 export/unlink、原 link/target 保留；该反例修复尚未进入 PR。

请拆成明确 fail-fast 的两条 test，或为组合 guard 加显式退出，并在现有薄 smoke 执行 literal shell recipe：普通文件恢复成功；symlink 在副作用前拒绝。保留 foreign/mixed ownership、malformed/collision、source/copy tamper 和 occupied destination 检查。无需新增大框架或降低手动 hold。

本地验证：source smoke 与两套 base/head non-editable installed smoke 通过；两边 wheel 和实际 Chat build 通过；head 既有 capability/provider/readback/census suite 143 passed，ruff/diff check 通过。base 是 142 passed 加一处旧 check_contract/load_registry census 行号失败，当前 PR 正好修复该 anchor，不把它算 blocker。独立 recipe 的其余 ownership/tamper/overwrite cases 正常，只有真实 symlink guard 不满足预期。全 canary 没有执行，也没有用安装版 catalog 冒充 exact-head coverage。

语义与 CI 对齐

当前义务是“仅在冻结、确切归属的 regular ledger 上操作，边界不成立就先停止”，新增 && guard 触发了违反；上述 literal shell effects 是证据。修复后重跑普通文件与 symlink 的实际 block，再跑 uv run --extra test python examples/reliability_diagnostics/ledger-retention-smoke.py 和 non-editable 的 --installed 变体。runtime read-only/authority-none 语义复用现有契约，未新增权限 vocabulary。远端 CI 未查询、未等待、未作为评审依据。

严格 change-quality 已按同一 exact diff 记录 fail，verify 保持非通过；未修复的实际问题没有被格式校验或 baseline 归因抹掉，也未进行自合并。

我的整体评价

这是比例合适、可回退的 offline recovery 切片，不应因它不是完整 live milestone 而否定；也不能因此宣布父目标完成。future-facing refine 应把实际 shell guard 收进既有薄验证，删除 Python-only 近似所留下的盲点，而不是再引入 retention service。默认关闭的 observer 与无控制权限路径在现有 readback/installed tests 中保持；但新 manual 用户旅程和后续删除 accounting 有具体回归，必须先修复并复演。未验证 live stop/resume、长期 overhead 或 secure erasure，且本 PR 并未声称具备它们。

本评审覆盖 f3ab983ef0b3c283f6043bc138c789f3afe812d8 相对 cf759d323b755e2040200f976eac1eeb89383b4d，按 loopx pr-review capability 的 review plan / five-block template 生成；精确正文写入 result.review_body 并在发布前 check-result。

[cocolord]

REQUEST_CHANGES — exact head f3ab983ef0b3c283f6043bc138c789f3afe812d8 adds a valuable offline recovery rehearsal, but the new operator recipe still fails two real supported boundaries.

动机

本 PR 为 #5211 补一个有界的 P0 operations-preparation 切片：shadow observer 停止写入后，operator 用同一安装版 LoopX 对单个 ledger 做完整导出、字节校验、删除后的 invalid/no_observations 回读和 exclusive-create 恢复，并保留 degraded/invalid 负向证据。它没有宣称完成 live C0/C1、长期 overhead、自动 TTL、secure erase 或部署 policy acceptance，范围选择合理。

默认目录下的 installed smoke 证明这个切片有真实价值；但公开操作路径还必须定位 provider 实际写入的同一文件，并在文件类型边界不成立时停止。当前 custom-directory 和 symlink 两个真实反例使该用户旅程尚未交付。

改动思路

实现复用现有 reliability ledger、CLI status/receipt、DSH fixture 和 read-only projection，没有新增 retention service、scheduler、restore API 或第二个状态 owner。正向流程冻结 writer，用固定 as_of 读取 receipt/projection，复制原字节并校验 hash 与独立 readback；删除后确认 no_observations，恢复用 xb 拒绝覆盖。权限仍由 operator 的冻结、hold、期限和删除决定承担。

问题在两个边界：DSH provider 接受任意 LOOPX_DSH_SHADOW_OBSERVER_LEDGER_DIR 并直接写入该目录，Python CLI 却固定读取 <runtime-root>/reliability_diagnostics/<goal>.ndjson；runbook 只要求把 custom directory 的 parent 当 runtime root，无法使路径相等。另一个边界中，POSIX set -e 对 AND-list 的非末尾失败不退出，因此组合 symlink guard 不会兑现 fail-fast。

具体改动

七个文件合计 +396/-4：新增 215 行 local-retention runbook 与 151 行 synthetic filesystem/CLI smoke；双语 capability README 增加入口；双语 RFC 更新 checkpoint 并保留后续 holds；semantic registry 只刷新一处生成行号。没有修改 provider、CLI schema、ledger 格式、默认启用状态或前台首屏。

关键代码讲解

examples/reliability_diagnostics/ledger-retention-smoke.py::main 在 disposable runtime 中经真实 CLI ingest fixture，覆盖 degraded 和含 durable refusal marker 的 invalid ledger；随后验证 raw-byte export、完整 JSON readback、删除、exclusive restore 与 sibling preservation。--installed 还会拒绝从 checkout import。

loopx/capabilities/reliability_diagnostics/docs/local-retention-v0.md 第 63–139 行是新的行为承诺：它从 CLI ledger_ref 拼出操作文件，再执行 ownership、copy/hash、delete 和 restore。packages/dsh-loopx-plugin/src/observer.ts::defaultLedgerDir/ledgerPath 允许任意配置目录，而 loopx/capabilities/reliability_diagnostics/ledger.py::ledger_ref 固定添加 reliability_diagnostics；默认布局相交不等于 custom layout 有契约。

对主干的风险

[P1] 当 provider 使用任意受支持的 custom ledger directory 时，按 runbook 将其 parent 传给 CLI 会读另一条固定路径。我用真实 provider resolver 在 custom-ledgers/goal-dsh-fixture.ndjson 写入完整 fixture；文件存在，但 installed CLI 返回 reliability_diagnostics/goal-dsh-fixture.ndjson、invalid/no_observations、persisted count 0。请明确只支持并验证 canonical layout，或让 CLI/recipe 安全映射实际目录，并增加真实 producer→installed CLI regression。

[P1] 第 91 行 test ! -L ... && test -f ... 在 POSIX set -e 下不会因第一个 test 失败而退出；literal symlink case 会继续执行。请改为明确 fail-fast，并让薄 smoke 执行实际 shell block，证明 regular file 成功、symlink 在任何 export/delete/restore 副作用前拒绝。

正向证据通过：源码 smoke；exact-head 6.2 MB wheel 在全新 Python 3.13 非 editable 环境的 --installed smoke；103 项 capability/provider/architecture tests；DSH fixture、docs-governance、Ruff check，以及 risk-based premerge 的 4 direct、9 catalog、8 risk-profile 和 1 boundary check。远端红灯已在 immutable base/head 复现为相同 dashboard workspace assertion，且 PR 未触碰该路径，属于独立 merge-readiness hold；当前 REQUEST_CHANGES 来自上述两个 PR-specific 反例。

语义与 CI 对齐

本 PR 复用现有 ledger、typed receipt reason 与 read-only/authority-none 词汇，没有新增状态分类或控制权限。未对齐的是 provider 的 custom-directory contract、CLI 固定 reference 与 runbook parent 映射，以及文档声明的 fail-fast 与 shell 实际行为。修复后应重跑 literal regular/symlink/custom-directory cases、source/installed smoke 和 risk-based premerge。

我的整体评价

这是方向正确、体量相称且可回滚的文档与 durable smoke 增量；原字节恢复、拒绝覆盖、default-off 和 authority-none 都值得保留，installed smoke 也证明默认 happy path 不是伪测试。

但新公开的核心操作路径仍会在受支持 custom directory 上读错文件，并可能越过 symlink guard 进入 destructive steps。这两项不能由默认路径 green 或 pre-existing 红 CI 抵消。请做最小路径契约与 fail-fast 修复并加入真实反例后再复审；live C0/C1、overhead 和 deployment policy 可继续作为 RFC 后续 hold。

English verdict: REQUEST_CHANGES — exact head f3ab983ef0b3c283f6043bc138c789f3afe812d8 adds a useful byte-preserving installed-package recovery rehearsal, but the recipe cannot map an arbitrary supported provider ledger directory to the CLI's fixed reliability_diagnostics/<goal>.ndjson path, and its POSIX symlink guard does not fail fast under set -e. Add real custom-directory and literal-shell symlink regressions, then rerun the source/installed smokes and risk-based premerge.

[mergify]

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @LIHUA919.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[LIHUA919]

两个 P1 已在 ae2fb573e51306a4891e38acc9c351c97ce56d20 修复，保留原有提交历史，请 @huangruiteng @cocolord 复审这个 head。

1. symlink guard 改为两条带显式退出的检查。实际文档 shell block 的 symlink case 现在非零退出，原 link 和 target 字节均保留，没有 ledger export、unlink 或 restore。
2. runbook 明确只支持 canonical layout，并要求传入冻结 provider 配置中的真实 ledger directory。目录与 <runtime-root>/reliability_diagnostics 不匹配或目录本身为 symlink 时，在 CLI 读回和创建 archive 前停止；不再把任意 custom directory 的 parent 当作映射，也没有新增 CLI/provider API。

现有薄 smoke 已替换 Python-only 近似操作，直接执行文档里的 shell block。11 个文件/证据场景和 3 个真实 DSH resolver/appender 目录场景均在 source 与最新 non-editable wheel 下通过，保留 foreign/mixed/malformed/collision、source/copy tamper 和 occupied destination 的反例。实际 appender 使用合成 session/event，不构成 native harness 或 live C0/C1 验收。

当前 head：94 项 capability/provider tests、docs/Ruff/public-boundary、Chat bundle/wheel build、Chat HTTP recovery/upgrade 和完整 risk-based premerge 通过。初次 sandbox premerge 的 3 项 runtime startup 失败，在允许本地通信端点后全部通过；没有修改测试规则。远端 CI 已针对这个 head 启动，仍须读取最终结果。

额外 packaged UI 全量验收：前 19 个场景通过，steward-model-settings 在关闭设置后恢复焦点处失败，后续未跑；完整重跑复现，单独场景通过。本 PR 相对 pinned base 未改 frontend/runtime/scenario source，保留该本地失败记录，不把它说成 full UI green，也未用未执行的 base/latest-main 结果替它归因。

RFC/issue 的 live C0/C1、overhead、部署 retention/deletion policy 和 tenant isolation holds 均保留；此 PR 只交付离线操作准备。README 首屏及 runtime/configuration contracts 未改。future-facing pass 收回了 literal shell 与真实目录映射的验证盲点，没有新增 scheduler 或平行 owner；merge 留给 maintainer。

English: Both P1 counterexamples are fixed on the exact head above. The literal POSIX recipe now fails before symlink effects, and validates the frozen provider directory against the explicitly supported canonical layout before readback/export. Source and non-editable installed regressions exercise the real shell and producer paths. Please re-review; current-head CI, independent acceptance and all live P0 qualification gates remain separate.

[huangruiteng]

Exact-head re-review: APPROVE

Reviewed head: ae2fb573e51306a4891e38acc9c351c97ce56d20; immutable base: fd5f31bb3ad57448c32df6c7cddde36d07446934. 使用 pull_request_review_execution_contract_v2 当前policy revision12，重审完整6文件 +522/-3，不仅检查旧review后的修补。当前没有阻塞发现。此前 f3ab983ef0b3c283f6043bc138c789f3afe812d8 的 custom-directory映射和POSIX symlink fail-fast 两项阻塞均已解决并独立验证。

动机

现有default-off诊断ledger需要在pilot前有可复现的导出、删除读回与恢复方法，且不能因恢复筛掉degraded、invalid或丢失标记来美化结果。Owning RFC和issue #5211需要这项有界离线准备，但其P0 live C0/C1、开销实测和部署owner政策验收仍然开放。这里不是自动retention或生产合格声明。

最小实现应复用现有schema、原始ledger和read-only CLI。新增scheduler/备份authority会扩大风险；只写指南而不执行原shell也不足。当前一份canonical recipe加真实路径验证能形成独立可用、可回滚的阶段结果。

改动思路

用户先记录冻结provider的实际目录、准确goal、有限deadline、数据owner与授权、固定带时区as-of及同一安装版本，由harness owner确认observer detach/dispose、最后flush且无writer。当前shell unset仅影响下一次launch，不冒充已停止活跃worker。

随后执行原recipe：canonical目录归属preflight → 全文件export/hash → 隔离copy的真实CLI receipt/projection比较 → 源hash再次核对 → 已授权单文件删除 → invalid/no_observations/persisted=0读回 → 重新保留数据授权 → absent目标exclusive raw restore → 字节与固定时间完整读回一致。整个流程不写Goal、Todo、quota或worker-session，不授予stop/resume/retry权限。

具体改动

• 新双语 local-retention-v0.md 给出人工deadline/hold、收集冻结、完整copy、删除/恢复和永久删除所有副本的边界；不宣称unlink安全擦除，也不把复制非法bytes称为public-safe。
• 旧custom目录假设已删除。实际DSH provider可写任意目录，但CLI固定加入 reliability_diagnostics，所以v0只支持canonical映射；provider resolved目录不匹配或为symlink时在readback/export前hold。不得移动、重ingest或删除不支持的ledger绕过它。
• 旧 test ! -L ... && test -f ... 改成各自显式失败退出，避免POSIX set -e 的AND-list例外。归属检查也拒绝foreign/mixed、不可解析记录与normalized文件名碰撞。
• smoke直接抽取并执行当前source或wheel中的原shell block，复用现有fixture、真实CLI和DSH TypeScript resolver/appender，而不是Python仿写。READMEs和RFC仅增加发现入口及准确partial checkpoint。

相关future-facing pass已应用在“单canonical guide由测试直接消费”的窄边界；无必要增加第二个retention owner或泛化service。两份README新增入口在第193/165行，未改变opening/hero/导航。没有settings/frontend/Lark companion：没有runtime/schema/provider/config editor改动，交互入口是人工CLI文档；Chat构建仅为wheel的现有装包前提，不声称新UI验收。

对主干的风险

最强反例不是happy-path hash相同，而是指南读取错目录、follow symlink、丢负面证据或覆盖已恢复writer。独立验证在source和隔离非editable wheel均运行11个实际文件系统场景（degraded、invalid、symlink、foreign、mixed、malformed、collision、copy/source/restore tamper、occupied）及3个真实provider布局（canonical、自定义、directory-symlink）。支持布局完整round-trip；不支持布局在创建archive前失败；完整loss/backpressure/clock和失败marker保留，旁路authority-sibling文件不变。--installed从wheel加载guide，明确拒绝checkout import；没有替换本机活动LoopX。

本次验证：uv run --extra test python examples/reliability_diagnostics/ledger-retention-smoke.py及隔离wheel的同脚本--installed均通过；103个既有capability/provider/registry边界tests、changed-smoke Ruff、docs governance通过；premerge9catalog+8risk+public-boundary通过，Chat/wheel构建通过。premerge没有自动包含这个新手工smoke，因此另行执行，并不以catalog结果替代它。初次pytest列错两个不存在文件、未进入产品断言；按真实文件纠正后103通过，原失败保留归因。现有Vite chunk-size warning不是此doc/test PR引入的改动或阻塞。

剩余边界明确：未运行模型/native worker或livepilot；冻结writer的事实需要harness owner按授权生命周期提供，不由synthetic smoke证明。未测CPU/RSS/latency、自动TTL/并发rotation/BYOC/租户隔离/secureerase；P0 exit保持开放。wait_for_ci=false，未读取或等待remote CI，不因无关红CI要求这个PR修其他代码。

我的整体评价

APPROVE。当前结果修复了旧review所指出的真实shell/provider边界，并交付可执行的bounded offline reference。保留完整raw负证据、精确目录hold和exclusive恢复比新增自动机制更符合当前需求；阶段不虚报父级生产资格。exact-head评审与merge authority/readiness不同，本轮未合并、未安装升级主机、未宣布Goal或P0完成。

English verdict: APPROVE - exact head ae2fb57. Both prior blockers are fixed and independently verified through the literal source and noneditable-wheel recipe, real CLI and real DSH producer layouts. Offline manual recovery is qualified; live/overhead and production retention acceptance remain with the existing issue owner.