Goal
Zero silent bugs. Not zero bugs — an agent fleet fails like an organization, not a program. The enforceable property is: any invariant violation becomes visible within one digest cycle (TTD ≤ 24h), and no trust-bearing label ("applied", DONE, synced, enabled) exists without a named check that earns it.
Principle: declared invariants, one reconciler. Components don't get bespoke watchers; they get a row in a manifest. Chokepoints guard the write path; a nightly reconciler audits the resting state; each catches the other's escapes. Evaluators are vibes — verification is structural.
Evidence base (why now)
One week of asking-by-hand found, among others: reports self-labeled "applied" while their code sat untracked in non-repos (fixed: claims verification, 09b409d); four disagreeing task-state vocabularies stalling ~80 tasks invisibly (fixed: DIP-0009 v1.1 + state_health); the overnight executor disabled for 11 days behind green dashboards; server checkouts on 10-day-stale work branches; split-brain remotes (7-megaphone). Every auditor added found real rot within hours.
Today's 30-minute manifest-style sweep found five recurring failed services nobody is watching (all failed again in the last 24h):
oauth-health-check (the guard is itself dead, exit 2) · nightshift-outbox (DIP-0017 processing, exit 15) · datacore-reindex (dies after 1h24m CPU, exit 15) · coinglass-scraper + strategy-sync (trading, failing to start) — plus an unreachable enterprise PLUR store with a queueing outbox, plur CLI version drift across machines (0.9.12-local hangs on recall / 0.10.1-server / 0.14 MCP), unpushed strays (4-forge, 6-meridian), and uncommitted churn (58 modified files in 1-datafund, 40 in 5-plur).
Phases
| Phase |
Scope |
Gate |
| R0 |
DIP-0033 "Reliability & Reconciliation" — invariant-manifest schema; label-must-name-check rule; incident→regression law (an incident closes only with a check that would have caught it); evaluator-dissent rule; canary requirement; TTD ≤ 24h as the metric. (0032 is claimed by the in-flight egress DIP.) |
none |
| R1 |
Invariant manifest + reconciler v1 (.datacore/reliability.yaml + nightshift lib/reconciler.py): repo topology (is-repo · branch=default · remote=canonical · unpushed-age · dirty-age), service SLAs (declared units active + last-success within max_silence — catches all five of today's corpses AND the 11-day-timer class), store health (plur doctor, outbox pending-age, remote-store reachability, CLI version parity), absorbs state_health. Output: digest section + NEED-YOU items; on-demand CLI. Absorbs root issues #31/#32 as declared invariants (unpushed-age catches both silent-loss modes). |
R0 |
| R2 |
Outcome acceptance hardening: code claims require committed-on-a-pushed-ref (extend claims_check with git verification, not just file existence); evaluator-dissent teeth (any single score ≤ 0.4 → forced actionable — the lone CTO 0.38 on the phantom builds had no mechanism); fix the evaluator-feedback rendering bug (all six quotes render empty in reports). Relates to #30 (policy guardrails). |
R1 |
| R3 |
Nightly canary: one synthetic :AI: task through the full pipeline (route → claim → execute → evaluate → triage → digest → verify every transition + artifact → clean up) in a sandbox space. Canary-didn't-run IS the alarm. Catches the unenumerable class. |
R1 |
| R4 |
Repair the found gaps (each small, each closes with a regression check per R0 law): the 5 failed services (diagnose → fix or formally retire; oauth-health-check first — it's a guard); enterprise-store outbox age alarm; plur CLI upgrades both machines (+ version-parity invariant); unpushed strays + churn cleanup; journal_parser same-day-cursor index gap (+ its relation to the dying datacore-reindex); routing-destination chokepoint (the deck-to-~/Data/content incident); corrupt April exec-state JSON. Split-brain 7-megaphone + org-workspace PyPI already tasked. |
R1 (alarm first, then fix) |
| R5 |
Fleet coverage: extend the manifest to hermes (Tris), plur-claw (Data), cos-server (Winston), blackpi git host. Pairs with the agent access-matrix audit (firm-20260719-agent-access-audit). Coordination: cos-server rows land only after the active Winston productization session merges — its scope (chief-of-staff repo, datacore-app, cos-server, dip-0032 branch) is hands-off until then. |
R1 proven + Winston session merged |
| R6 |
Governance ritual: reliability row in the weekly scorecard; monthly full-audit absorbing the memory-audit cadence; uptime-monitoring item from the Winston-gaps memo folds in here. |
R4 |
Non-goals
- Per-component bespoke monitors (manifest rows only)
- Blocking the fleet on perfect coverage — the reconciler ships useful on day one and grows by rows
- Touching the Winston session's scope before it merges
Conventions
Feature branch + PR per phase; findings route through the digest NEED-YOU loop (never silent auto-fix of someone else's state); every phase adds its own rows to the manifest it ships.
Goal
Zero silent bugs. Not zero bugs — an agent fleet fails like an organization, not a program. The enforceable property is: any invariant violation becomes visible within one digest cycle (TTD ≤ 24h), and no trust-bearing label ("applied", DONE, synced, enabled) exists without a named check that earns it.
Principle: declared invariants, one reconciler. Components don't get bespoke watchers; they get a row in a manifest. Chokepoints guard the write path; a nightly reconciler audits the resting state; each catches the other's escapes. Evaluators are vibes — verification is structural.
Evidence base (why now)
One week of asking-by-hand found, among others: reports self-labeled "applied" while their code sat untracked in non-repos (fixed: claims verification,
09b409d); four disagreeing task-state vocabularies stalling ~80 tasks invisibly (fixed: DIP-0009 v1.1 +state_health); the overnight executor disabled for 11 days behind green dashboards; server checkouts on 10-day-stale work branches; split-brain remotes (7-megaphone). Every auditor added found real rot within hours.Today's 30-minute manifest-style sweep found five recurring failed services nobody is watching (all failed again in the last 24h):
oauth-health-check(the guard is itself dead, exit 2) ·nightshift-outbox(DIP-0017 processing, exit 15) ·datacore-reindex(dies after 1h24m CPU, exit 15) ·coinglass-scraper+strategy-sync(trading, failing to start) — plus an unreachable enterprise PLUR store with a queueing outbox, plur CLI version drift across machines (0.9.12-local hangs on recall / 0.10.1-server / 0.14 MCP), unpushed strays (4-forge, 6-meridian), and uncommitted churn (58 modified files in 1-datafund, 40 in 5-plur).Phases
.datacore/reliability.yaml+ nightshiftlib/reconciler.py): repo topology (is-repo · branch=default · remote=canonical · unpushed-age · dirty-age), service SLAs (declared units active + last-success within max_silence — catches all five of today's corpses AND the 11-day-timer class), store health (plur doctor, outbox pending-age, remote-store reachability, CLI version parity), absorbsstate_health. Output: digest section + NEED-YOU items; on-demand CLI. Absorbs root issues #31/#32 as declared invariants (unpushed-age catches both silent-loss modes).claims_checkwith git verification, not just file existence); evaluator-dissent teeth (any single score ≤ 0.4 → forcedactionable— the lone CTO 0.38 on the phantom builds had no mechanism); fix the evaluator-feedback rendering bug (all six quotes render empty in reports). Relates to #30 (policy guardrails).:AI:task through the full pipeline (route → claim → execute → evaluate → triage → digest → verify every transition + artifact → clean up) in a sandbox space. Canary-didn't-run IS the alarm. Catches the unenumerable class.oauth-health-checkfirst — it's a guard); enterprise-store outbox age alarm; plur CLI upgrades both machines (+ version-parity invariant); unpushed strays + churn cleanup;journal_parsersame-day-cursor index gap (+ its relation to the dyingdatacore-reindex); routing-destination chokepoint (the deck-to-~/Data/contentincident); corrupt April exec-state JSON. Split-brain 7-megaphone + org-workspace PyPI already tasked.firm-20260719-agent-access-audit). Coordination: cos-server rows land only after the active Winston productization session merges — its scope (chief-of-staff repo, datacore-app, cos-server, dip-0032 branch) is hands-off until then.Non-goals
Conventions
Feature branch + PR per phase; findings route through the digest NEED-YOU loop (never silent auto-fix of someone else's state); every phase adds its own rows to the manifest it ships.