Repository navigation
Feat/incident rca app - #538
toasterman234 wants to merge 5 commits into
Conversation
| kind = "input" | ||
| from = ["Closed"] | ||
| to = "Reopened" | ||
| effect = ["recurrence_count += 1"] |
There was a problem hiding this comment.
Recurrences reuse old verification
Reopen keeps every flag from the previous closure. After a recurrence, an investigator can run BeginInvestigation → CompleteDocumentation → StateHypothesis → ProposeFix → BeginVerification → Close without attaching new evidence, reviewing prior incidents again, or recording new verification. Reset the investigation and verification flags when reopening, while keeping the previous record as history.
Knowledge Base Used: Execution semantics and action evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/incident.ioa.toml
Line: 127
Comment:
**Recurrences reuse old verification**
`Reopen` keeps every flag from the previous closure. After a recurrence, an investigator can run `BeginInvestigation → CompleteDocumentation → StateHypothesis → ProposeFix → BeginVerification → Close` without attaching new evidence, reviewing prior incidents again, or recording new verification. Reset the investigation and verification flags when reopening, while keeping the previous record as history.
**Knowledge Base Used:** [Execution semantics and action evaluation](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/execution-semantics.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| @@ -0,0 +1 @@ | |||
| [{"detected_at":"2026-10-07T00:00:00-05:00","evidence_refs":["memory/2026-10-21.md (incident pattern summary)"],"failure_mode":"ENROLLMENT_DOWN","root_cause_hypothesis":"VM-side tailscaled lost enrollment/connection state; `tailscale status` reported Not connected.","seed_id":"tunnel-2026-10-07","source_signal":"af-obsidian-icloud-sync failure + manual check","title":"Tailscale tunnel down — VM enrollment dropped","verification_evidence":"`tailscale up` restored the tunnel; missed sync completed green 2026-10-07 15:05 CDT and subsequent runs stayed green.","why_prior_fixes_failed":"First recorded incident of this mode; no prior fix."},{"detected_at":"2026-10-08T00:00:00-05:00","evidence_refs":["memory/2026-10-14.md (flap pattern summary)"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"Unknown. `tailscale status` claimed Connected while the egress proxy refused CONNECT; jobs held last-good state and backfilled on recovery. A separate same-day dropout was restored by `tailscale up`.","seed_id":"tunnel-2026-10-08","source_signal":"captures bundle/projector failures","title":"Tailscale tunnel outage (~12h), capture readers failed, self-recovered","verification_evidence":"Self-recovered; healthy runs backfilled with no captures lost (verified in the Oct 14 pattern summary).","why_prior_fixes_failed":"`tailscale up` treats enrollment loss; this mode presents as Connected, so the enrollment fix does not address it."},{"detected_at":"2026-10-10T15:29:00-05:00","evidence_refs":["memory/2026-10-10.md","memory/2026-10-21.md (incident pattern summary)"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"UNKNOWN — operator fix not recorded. VM-side state: `tailscale status` and `tailscale up` both claimed Connected; the port-3130 proxy accepted TCP but closed CONNECT without a response. Ben fixed it from his side at ~15:37 CDT ('I put some elbow grease on it'); Astro asked what the fix was and no answer is on record.","seed_id":"tunnel-2026-10-10","source_signal":"af-obsidian-icloud-sync failure (15:32 CDT run)","title":"Tailscale tunnel down — proxy closed CONNECTs; operator fixed from Mac side","verification_evidence":"Sync ran green immediately after the operator fix (mirrored 15:38 CDT); subsequent runs green.","why_prior_fixes_failed":"The VM-side fix (`tailscale up`) did not restore the data plane; the effective fix was Mac-side and is undocumented — which is itself the finding this corpus exists to prevent."},{"detected_at":"2026-10-11T07:32:00-05:00","evidence_refs":["memory/2026-10-11.md"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"Transient proxy CONNECT failure; the sync itself had completed and a retry read succeeded. Possibly the same underlying instability as 2026-10-08/10, unproven.","seed_id":"tunnel-2026-10-11","source_signal":"af-obsidian-icloud-sync run diagnostics (07:32 CDT)","title":"Transient tunnel blip during log inspection","verification_evidence":"Subsequent reads in the same run confirmed the 07:32:22 sync entry and no FAILED lines.","why_prior_fixes_failed":"No fix applied; recovered before intervention."},{"detected_at":"2026-10-14T12:32:00-05:00","evidence_refs":["memory/2026-10-14.md"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"Proxy closed connection during CONNECT on both the sync attempt and the log tail; VM-side `tailscale up` restored service this time. Same-day failures at 04:09 and 08:54 self-recovered.","seed_id":"tunnel-2026-10-14","source_signal":"af-obsidian-icloud-sync failure (12:32 CDT run)","title":"Tailscale tunnel down at 12:32 run; `tailscale up` restored","verification_evidence":"Missed sync ran green at 12:34 CDT (mirrored to iCloud); Ben notified.","why_prior_fixes_failed":"Recurrence of the 10-08/10-10 mode; no structural fix has ever been applied — recoveries only."},{"detected_at":"2026-10-20T16:17:00-05:00","evidence_refs":["memory/2026-10-20.md","memory/2026-10-21.md (incident pattern summary)"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"Proxy closed CONNECTs while `tailscale up` claimed Connected — the 10-10 presentation, but this time it self-recovered without the operator's Mac-side fix.","seed_id":"tunnel-2026-10-20-a","source_signal":"af-obsidian-icloud-sync failure (16:17 CDT run)","title":"Tailscale tunnel down at 16:17 run; self-recovered by 20:02","verification_evidence":"The 20:02 run connected and completed green (mirrored 20:04 CDT); Ben sent the correction.","why_prior_fixes_failed":"Recurrence; still no structural fix. The self-recovery weakens the 'Mac-side fault' theory from 10-10 and is recorded as such."},{"detected_at":"2026-10-20T20:02:00-05:00","evidence_refs":["memory/2026-10-20.md"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"Tail of the 16:17 outage: first attempt failed with proxy-closed CONNECT, immediate retry succeeded.","seed_id":"tunnel-2026-10-20-b","source_signal":"af-obsidian-icloud-sync (20:02 CDT run, first SSH attempt)","title":"Transient tunnel failure on first attempt of 20:02 run","verification_evidence":"Same run completed: mirrored to iCloud 20:04 CDT.","why_prior_fixes_failed":"No fix applied; recovered on retry."},{"detected_at":"2026-10-22T15:32:00-05:00","evidence_refs":["memory/2026-10-22.md"],"failure_mode":"TUNNEL_DOWN","root_cause_hypothesis":"`tailscale up` reported 'Tailscale stopped sending network configuration. Reconnecting.' while the proxy still closed CONNECTs; the Mac was reachable again by ~15:43 CDT with no intervention. Recurrence count for this mode is now high enough that 'transient' is no longer a satisfying classification.","seed_id":"tunnel-2026-10-22","source_signal":"af-obsidian-icloud-sync failure (15:32 CDT run)","title":"Tailscale tunnel down at 15:32 run; self-recovered within ~10 minutes","verification_evidence":"Mac reachable at ~15:43 CDT (SSH recon for the incident-rca build); sync runs resume on schedule.","why_prior_fixes_failed":"Every prior recovery was either enrollment repair, an undocumented operator fix, or self-recovery. Nothing has addressed the proxy/data-plane instability itself."}] No newline at end of file | |||
There was a problem hiding this comment.
Historical incidents never load
The installer never reads seeds/tunnel-incidents.json. It only loads seed-data/*.toml files containing [[instance]] blocks, so installing this app creates none of the eight historical incidents promised in APP.md. Convert the corpus to the supported seed format and include the actions needed to reach Closed.
Knowledge Base Used: Platform lifecycle and deployment
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/seeds/tunnel-incidents.json
Line: 1
Comment:
**Historical incidents never load**
The installer never reads `seeds/tunnel-incidents.json`. It only loads `seed-data/*.toml` files containing `[[instance]]` blocks, so installing this app creates none of the eight historical incidents promised in `APP.md`. Convert the corpus to the supported seed format and include the actions needed to reach `Closed`.
**Knowledge Base Used:** [Platform lifecycle and deployment](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/platform-lifecycle.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| guard = "prior_incidents_reviewed" | ||
| effect = ["has_root_cause_hypothesis = true"] | ||
| params = ["RootCauseHypothesis", "WhyPriorFixesFailed"] |
There was a problem hiding this comment.
Empty inputs pass required checks
The recording actions set their completion flags even when the caller supplies {} or empty strings. Declaring params and CSDL Nullable="false" does not require content here: the runtime accepts inputs without constraints. An incident can therefore close without evidence or a hypothesis, and a postmortem can enter review without root-cause text.
Add nonempty constraints to the evidence, prior-review, hypothesis, and verification inputs. Apply the same check to AddRootCause in the postmortem spec.
Knowledge Base Used:
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/incident.ioa.toml
Line: 83-85
Comment:
**Empty inputs pass required checks**
The recording actions set their completion flags even when the caller supplies `{}` or empty strings. Declaring `params` and CSDL `Nullable="false"` does not require content here: the runtime accepts inputs without constraints. An incident can therefore close without evidence or a hypothesis, and a postmortem can enter review without root-cause text.
Add nonempty constraints to the evidence, prior-review, hypothesis, and verification inputs. Apply the same check to `AddRootCause` in the postmortem spec.
**Knowledge Base Used:**
- [Specification authoring and compilation](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/specification-authoring.md)
- [Execution semantics and action evaluation](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/execution-semantics.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| name = "ProposeFix" | ||
| kind = "internal" | ||
| from = ["HypothesisStated"] | ||
| to = "FixProposed" | ||
| guard = "has_root_cause_hypothesis" | ||
| hint = "Propose the fix (creates a Remediation in Proposed state). Unreachable without prior review and a hypothesis — that is the point." |
There was a problem hiding this comment.
Proposing a fix creates nothing
ProposeFix only changes the incident's status. It has no creation effect or trigger, so it never creates the Remediation promised by its hint and APP.md. Callers reach FixProposed with nothing available for a human to approve. Add an explicit step that creates and links the proposed remediation.
Knowledge Base Used: Specification authoring and compilation
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/incident.ioa.toml
Line: 91-96
Comment:
**Proposing a fix creates nothing**
`ProposeFix` only changes the incident's status. It has no creation effect or trigger, so it never creates the `Remediation` promised by its hint and `APP.md`. Callers reach `FixProposed` with nothing available for a human to approve. Add an explicit step that creates and links the proposed remediation.
**Knowledge Base Used:** [Specification authoring and compilation](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/specification-authoring.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| name = "BeginVerification" | ||
| kind = "input" | ||
| from = ["FixProposed"] | ||
| to = "Verifying" | ||
| params = ["RemediationRef"] | ||
| hint = "Begin verification after the linked Remediation reaches Succeeded. Verification is against explicit criteria, not vibes." |
There was a problem hiding this comment.
Verification accepts unfinished fixes
BeginVerification accepts any RemediationRef without checking the referenced remediation. For example, {"RemediationRef":"not-real"} moves an incident from FixProposed to Verifying; a failed or rejected remediation works too. The investigator can then advance to closure without a successful fix. Require a remediation linked to this incident that has reached Succeeded before allowing this transition.
Knowledge Base Used:
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/incident.ioa.toml
Line: 99-104
Comment:
**Verification accepts unfinished fixes**
`BeginVerification` accepts any `RemediationRef` without checking the referenced remediation. For example, `{"RemediationRef":"not-real"}` moves an incident from `FixProposed` to `Verifying`; a failed or rejected remediation works too. The investigator can then advance to closure without a successful fix. Require a remediation linked to this incident that has reached `Succeeded` before allowing this transition.
**Knowledge Base Used:**
- [Specification authoring and compilation](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/specification-authoring.md)
- [Execution semantics and action evaluation](https://app.greptile.com/arni-labs/-/custom-context/knowledge-base/nerdsane/temper/-/docs/execution-semantics.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| name = "incident-rca" | ||
| description = "Governed incident + root-cause-analysis workflow: detected failures are investigated against the full prior-incident corpus, documented, fixed only through human-approved remediations, and verified closed. First deployment: the VM↔Mac mini Tailscale tunnel." | ||
| version = "0.1.0" |
There was a problem hiding this comment.
This adds an application-specific incident workflow to the kernel repository. The repository guide explicitly requires “Kernel code only” and says app and agent logic belongs in temperpaw. The incident state machine and Cedar policies are app behavior, not kernel support. Put the new app bundle in its owning repository before merging to satisfy that requirement.
Context Used: CLAUDE.md (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/app.toml
Line: 1-3
Comment:
**App violates kernel-only rule**
This adds an application-specific incident workflow to the kernel repository. The repository guide explicitly requires “Kernel code only” and says app and agent logic belongs in `temperpaw`. The incident state machine and Cedar policies are app behavior, not kernel support. Put the new app bundle in its owning repository before merging to satisfy that requirement.
**Context Used:** CLAUDE.md ([source](https://github.com/nerdsane/temper/blob/main/CLAUDE.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…der, sensor scripts, deployment notes (nerdsane#372)
| _, listing = call(tok, "GET", "/tdata/Incidents") | ||
| same_mode = [v for v in listing.get("value", []) | ||
| if v.get("fields", {}).get("FailureMode") == item["mode"]] |
There was a problem hiding this comment.
file_incident.py searches only the first page of incidents. The server returns 100 rows by default. When a matching incident is on a later page, the sensor creates another incident instead of reopening or reusing it. seed_loader.py has the same problem when checking existing titles.
Follow @odata.nextLink before deciding that no match exists.
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/sensor/file_incident.py
Line: 44-46
Comment:
**Later incidents are missed**
`file_incident.py` searches only the first page of incidents. The server returns 100 rows by default. When a matching incident is on a later page, the sensor creates another incident instead of reopening or reusing it. `seed_loader.py` has the same problem when checking existing titles.
Follow `@odata.nextLink` before deciding that no match exists.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| with open(args.queue, "w") as f: | ||
| for item in remaining: | ||
| f.write(json.dumps(item) + "\n") |
There was a problem hiding this comment.
Overlapping runs erase detections
If sensor runs overlap during an outage, they can erase undelivered detections. Each run reads the shared queue, keeps its own copy while SSH requests run, then replaces the whole file. If two runs read an empty queue, the second write drops the first run's detection.
Lock the queue across updates and save changes without exposing a partly written file.
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/sensor/tailnet-sensor.py
Line: 102-104
Comment:
**Overlapping runs erase detections**
If sensor runs overlap during an outage, they can erase undelivered detections. Each run reads the shared queue, keeps its own copy while SSH requests run, then replaces the whole file. If two runs read an empty queue, the second write drops the first run's detection.
Lock the queue across updates and save changes without exposing a partly written file.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| if seed["title"] in existing_titles: | ||
| print(f"skip (exists): {seed['seed_id']}") | ||
| continue |
There was a problem hiding this comment.
Seed retries skip unfinished records
Once creation succeeds, a later action failure leaves that incident unfinished. The next run skips it solely because its title exists, so retrying cannot finish the load. The loader also exits successfully after reporting partial loads.
Resume existing unfinished seeds from their current state and return a failure when any seed fails.
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/seeds/seed_loader.py
Line: 63-65
Comment:
**Seed retries skip unfinished records**
Once creation succeeds, a later action failure leaves that incident unfinished. The next run skips it solely because its title exists, so retrying cannot finish the load. The loader also exits successfully after reporting partial loads.
Resume existing unfinished seeds from their current state and return a failure when any seed fails.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| proc = subprocess.run( | ||
| [os.path.expanduser("~/workspace/bin/mac-ssh.sh"), "bencharney@100.71.118.10", | ||
| f"echo '{payload}' | base64 -d | python3 ~/temper-server/file_incident.py"], | ||
| capture_output=True, text=True, timeout=120) |
There was a problem hiding this comment.
--base is accepted and passed to file_incident, but that function ignores it and always sends detections to the hardcoded Mac over SSH. Someone selecting another server still files to the original deployment, or queues failures if that Mac is unavailable.
Honor the selected destination or remove the misleading option.
Prompt To Fix With AI
This is a comment left during a code review.
Path: os-apps/incident-rca/sensor/tailnet-sensor.py
Line: 65-68
Comment:
**Selected server is ignored**
`--base` is accepted and passed to `file_incident`, but that function ignores it and always sends detections to the hardcoded Mac over SSH. Someone selecting another server still files to the original deployment, or queues failures if that Mac is unavailable.
Honor the selected destination or remove the misleading option.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
This PR needs fixes before merging because sensor runs can lose detections and seed retries can leave historical records unfinished.
Fix with agent prompt
Summary
Adds an incident investigation workflow with human-approved fixes and postmortems.
Follow-up changes:
New findings cover incomplete list reads, lost queue updates, unfinished seed retries, and an ignored server option.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Tunnel classifier] --> B{Fault detected?} B -->|Yes| C[Detection plus queued items] B -->|No| D[Queued items] C --> E[SSH to Mac helper] D --> E E --> F{Delivery succeeds?} F -->|No| G[Keep detection in local queue] F -->|Yes| H[List incidents] H --> I{Matching incident?} I -->|Open| J[Reuse incident] I -->|Closed| K[Reopen incident] I -->|None| L[Create incident]Reviews (2) · Last reviewed commit: "As-built updates: operator identity in C..." · Reviewed by Greptile