History, incident records, fleet triage, autonomous watch and incident drills

README.md

@@ -179,9 +179,24 @@ Each case lists how to confirm it, its causes, fix steps tagged `[read]` /
 **`/incident [what you see]`** works the top case to resolution: confirm → fix the
 root cause → wait (`wait.until`, without spending model calls) → verify → report.
 When the agent finds a cause no case covered, it offers to record it with `kb.add`.
+
+cometcli remembers. Every triage is saved to the node's history, so it can tell you what
+changed since the last check and answer "since when?" (`node history --since 7d`).
+Every finished incident is saved as a postmortem with its root cause, actions,
+transaction hashes and a timeline from the audit log. Triage points out when the same
+problem comes back.
 Add your own cases in `~/.cometcli/kb/` — format in
 [docs/EXTENDING.md](docs/EXTENDING.md#knowledge-base-cases).
 
+Across nodes, `cometcli fleet triage` compares them all at once. It points out a node
+behind the others, a chain-wide halt, version mismatches, two nodes running one
+consensus key, and validators cut off from their sentries. `cometcli watch` does this on
+an interval for you. It works new incidents read-only and reports them to Slack,
+Discord or Telegram, or with `--mode fix` sends each approval to Telegram with
+Approve/Deny buttons. To check how well all of this works, `cometcli drill` breaks a
+throwaway local network on purpose and scores whether the agent finds and fixes each
+fault.
+
 Jail recovery is careful about fees: `val.unjail` refuses while the jail period runs,
 the node isn't synced, the signer's saved state is ahead of the chain, the node signs
 with a different consensus key than the one registered, self-delegation is below the
@@ -222,22 +237,25 @@ Tools are grouped by domain. Required args also bind positionally —
 
 | Domain | Highlights |
 |---|---|
-| `node` | `triage`, `status`, `peers`, `health`, `consensus`, `logs`, `config --action lint`, `service status\|restart`, `version-check` |
+| `node` | `triage`, `history`, `status`, `peers`, `health`, `consensus`, `logs`, `config --action lint`, `service status\|restart`, `version-check` |
 | `val` | `status`, `signing`, `jail-check`, `consensus`, `rewards`, `votes` · on-chain: `unjail`, `withdraw`, `vote`, `edit`, `create` |
 | `chain` | `validators` (`--status jailed`), `params`, `gov`, `upgrade-plan`, `pool`, `balance` |
 | `gov` | `propose` (text / upgrade / messages), `deposit` |
+| `incident` | `list`, `show`, `record` — postmortems of past incidents |
 | `kb` | `search`, `show`, `add` — the knowledge base of failure cases |
 | `wait` | `until` — synced, height, unjailable, signing, in-consensus, tx-committed, proposal-status, or any triage signal |
 | `evm` | `chainid`, `parity` (comet↔JSON-RPC drift), `gasprice`, `txpool` |
 | `tx` | `send`, `delegate`, `undelegate`, `redelegate`, `get` — every tx tool takes `--gas-price`, `--gas-limit`, `--fee-denom`, `--seq` |
 | `keys` | `add` (`--recover`/`--privkey-hex`), `list`, `show`, `rm`, `convert` (bech32↔0x) |
 | `sec` | `exposure` (listening ports), `perms` (key-file permissions), `doublesign` (priv_validator_state check) |
-| `mon` | `snapshot`, `watch` (live TUI), `alerts` (rules → stdout/Slack/Discord/Telegram) |
+| `mon` | `snapshot`, `watch` (live TUI), `record` (signal history), `alerts` (rules → stdout/Slack/Discord/Telegram) |
 | `upgrade` | `check`, `prepare` (download, verify, stage for cosmovisor), `watch` |
 | `snap` | `list`, `prune`, `statesync` |
 | `runbook` | `list`, `show`, `run` — builtins + your own YAML |
 | `net` | `add-peer`, `rm-peer` |
-| `fleet` | `status`, `exec` (a read-only tool on every profile), `shell` (one command on every host, one approval) |
+| `fleet` | `triage` (every node, compared), `status`, `exec` (a read-only tool on every profile), `shell` (one command on every host, one approval) |
+| `watch` | the autonomous watcher: triage on an interval, work new incidents, remote approvals |
+| `drill` | `testnet up\|down`, `list`, `run` — scored incident drills on a throwaway network |
 
 Global flags: `--profile`, `--json`, `-y/--yes` (auto-approve observe/diagnose
 prompts — never on-chain).
@@ -309,6 +327,8 @@ credentials       # API keys saved by `config set-key` (0600)
 settings.json     # permission rules, hooks, MCP servers (also per project: .cometcli/)
 COMET.md          # memory the agent always reads (also per project)
 kb/               # your own knowledge-base cases
+history/          # triage signal history per profile (30 days)
+incidents/        # incident records and postmortems per profile
 keys/             # ops keyring (file backend)
 sessions/         # saved conversations (cometcli -c / -r)
 runbooks/         # your custom YAML playbooks

docs/COMMANDS.md

@@ -99,7 +99,8 @@ down is reported, never fatal. Signals are flat names (`val.jailed`,
 `host.disk_used_pct`, `logs.apphash`, `node.key_mismatch`…).
 
 Each **case** says which signals point to it, how to confirm it, causes, fix steps tagged
-`[read]` / `[change]` / `[tx]`, how to verify, and what never to do. In node mode,
+`[read]` / `[change]` / `[tx]`, how to verify, and what never to do. In any session
+(name the node — `/incident val01 is down` — and cometcli switches to it),
 **`/incident [what you see]`** runs the method: triage → confirm the top case → fix the
 root cause first (approvals as usual) → wait with `wait.until` → verify → re-triage →
 report. When the agent finds a cause no case covered, it offers to record one with
@@ -116,6 +117,31 @@ format in [EXTENDING.md](EXTENDING.md#knowledge-base-cases).
 cometcli wait until --condition signal --value "host.disk_used_pct < 85 && proc.running == true"
 ```
 
+### History and incident records
+
+Every `node.triage` run is saved to the node's signal history
+(`~/.cometcli/history/<profile>/`, kept 30 days), and triage starts by saying what
+changed since the previous run (`node.peers 8→1, val.jailed false→true`). For trends:
+
+```bash
+cometcli node history                              # key health signals, last 24h
+cometcli node history --signals val.,node.peers --since 7d
+cometcli mon record --interval 5m                  # sample regularly, not only when someone triages
+```
+
+Each answer gives first → last, min/max, a trend line and when the value changed.
+
+When an incident is finished, the agent saves it with `incident.record`: title, root
+cause, matching case, evidence, actions and outcome. The timeline (prompts, tool calls,
+approvals) and transaction hashes are filled in from the audit log. Records live in
+`~/.cometcli/incidents/<profile>/` as JSON plus a Markdown postmortem. Triage
+mentions a node's past incidents and flags a case that keeps coming back.
+
+```bash
+cometcli incident list                             # last 30 days (--since 90d)
+cometcli incident show --id 20261007-0533-val-jailed-downtime
+```
+
 ### Incident tools
 
 Tools built for running an incident to resolution — usable directly or by the agent:
@@ -287,13 +313,97 @@ cometcli evm txpool                    # pending/queued EVM tx counts
 ## Fleet (all profiles at once)
 
 ```bash
+cometcli fleet triage                              # every node triaged at once, then compared
+cometcli fleet triage --chain mychain-1            # one chain only
 cometcli fleet status                              # health matrix: height, peers, signing, disk
 cometcli fleet exec --tool node.status             # run a read-only tool everywhere
 cometcli fleet exec --tool node.logs --args '{"lines":20}'
 cometcli fleet exec --tool val.signing --profiles val01,val02
 cometcli fleet shell "uptime"                      # shell on every host (one approval)
 ```
 
+`fleet triage` gives one row per node (height, peers, signing, jailed, top case), then
+what only a comparison shows:
+
+- a node behind the others on its chain
+- a chain-wide halt: every node stalled, so the problem is the chain, not one node
+- version mismatches between nodes
+- two nodes running the same consensus key (double-sign risk)
+- validators sharing a host
+- a validator whose sentries are down
+
+Tell cometcli which sentries a validator peers through so it can check them:
+`cometcli profile add val01 --sentries sentry1,sentry2`. In a session, questions about
+several nodes ("how is my network?") start with `fleet.triage`, in general mode too.
+
+## Watching on your behalf
+
+```bash
+cometcli watch                                     # every node, every 5m, diagnose mode
+cometcli watch --mode fix --interval 2m            # may act — approvals go to Telegram
+cometcli watch --once --mode notify                # one sweep, report only (cron-friendly)
+```
+
+Each sweep is a fleet triage. A problem that newly appears (a critical or high case on
+a node, or a chain halt, double-sign risk or cut-off validator) becomes an incident:
+
+| mode | what happens |
+|---|---|
+| `notify` | the finding is reported |
+| `diagnose` (default) | the agent works the incident read-only and reports its findings |
+| `fix` | the agent may act; **each approval is sent to Telegram with Approve / Deny buttons** |
+
+The watcher reports findings, agent reports and recoveries. It doesn't work the same
+problem again within the cooldown (default 1h), and remembers what it saw across
+restarts.
+
+```yaml
+# ~/.cometcli/config.yaml
+watch:
+  interval: 5m
+  mode: diagnose            # notify | diagnose | fix
+  profiles: [val01, val02]  # default: all
+  cooldown: 1h
+  approval_timeout: 15m     # an unanswered approval is a "no"
+  allow_tx: false           # transactions are refused in watch mode unless true
+  telegram_users: [123456]  # only these Telegram users' presses count
+  alerts:
+    slack_webhook: https://hooks.slack.com/…     # reports only
+    discord_webhook: https://discord.com/api/webhooks/…
+    telegram_chat_id: "-100123456"               # reports and approvals
+```
+
+Save the Telegram bot token with `cometcli config set-key TELEGRAM_BOT_TOKEN`, or put it
+in `watch.alerts.telegram_token`. Only presses from that chat, and from
+`telegram_users` if set, count. Slack and Discord get reports but can't answer
+approvals.
+
+## Incident drills
+
+Check how well cometcli handles real faults, on a throwaway network:
+
+```bash
+cometcli drill testnet up --image <evmd image>     # 4 local validators in docker, fast slashing
+cometcli drill list                                # the scenarios
+cometcli drill run                                 # all of them (--scenario oom,partition to pick)
+cometcli drill testnet down                        # remove containers, data and drill profiles
+```
+
+Each scenario breaks a drill node for real:
+
+- stopped until it's jailed for downtime
+- `config.toml` broken (crash loop)
+- `halt-height` set
+- container memory limit too low (OOM)
+- network detached
+
+For each one, the drill checks whether triage named the right root cause, then lets the
+agent work it with `/incident`, verifies the fix on the node itself, and puts the node
+back to health. The scoreboard shows triage x/y and fixed x/y, with steps, tokens and
+time per scenario. Results are saved in `~/.cometcli/drills/`, so model or prompt
+changes can be compared run to run. Approvals are granted automatically, **but only on
+drill profiles** (`metadata.drill: "true"`); anything else is refused.
+
 ## Monitoring
 
 ```bash
@@ -409,8 +519,8 @@ saved after each turn to `~/.cometcli/sessions/` (0600, already redacted).
 | `/memory`, `/remember [--user\|--node] <text>` | memory files loaded; add a line to COMET.md |
 | `/mcp`, `/agents` | connected MCP servers; subagents for the task tool |
 | `/<custom>` | commands from `.cometcli/commands/*.md` — see [EXTENDING.md](EXTENDING.md) |
-| `/incident [what you see]` | node mode: triage → known case → fix root cause → wait → verify → report |
-| `/recover-jail [notes]` | node mode: run the full jail-recovery procedure |
+| `/incident [what you see]` | triage → known case → fix root cause → wait → verify → record → report; name the node to switch to it |
+| `/recover-jail [notes]` | the full jail-recovery procedure (name the node, or the agent picks it) |
 | `/one [profile\|off]` | switch the session to node mode for a profile, or back to general (conversation kept) |
 
 ### General tools and permissions

internal/agent/builtin_cmds.go

@@ -10,13 +10,14 @@ var builtinCommands = map[string]*settings.Command{
 		Description: "Work an incident to resolution: triage, match a known case, fix the root cause, wait, verify, report",
 		Body: `Work this incident on the node to resolution. Keep the operator's checklist current with todo_write; don't stop between steps unless something needs their decision.
 
-1. node.triage (since=2h if the problem started earlier). Note which sources were unavailable.
+1. node.triage (since=2h if the problem started earlier). Note which sources were unavailable, what changed since the last check, and past incidents on this node — a repeat means the last fix missed something (incident.show it). For "since when?", use node.history.
 2. Take the top matched case: kb.show it, run its confirm steps. If confirmation fails, move to the next case or kb.search the symptoms/log lines. No case at all → investigate from the signals and logs (node.logs, bash read-only).
 3. If several cases match, fix the one that CAUSES the others first (disk full → crash → behind → jailed: disk first).
 4. Apply the fix steps by CALLING the tools. [change] and [tx] steps are approved at the tool's own prompt: state in one line what you're doing and why, then make the call in the same step — never stop to ask for approval in text. Never do anything a case lists under NEVER.
 5. Wait for the effect with wait.until (synced, signing, in-consensus, or condition=signal value="<expr>"). If progress stalls, re-triage instead of waiting longer.
 6. Verify with the case's verify steps, then node.triage again: the case must no longer match.
-7. Report: root cause with evidence, actions taken (tx hashes), final state. If you found a cause no case described, propose a kb.add case (with a test fixture) for the operator to approve.
+7. Record it with incident.record (title, root cause, case id, evidence, actions, outcome) — the timeline and tx hashes are filled in for you.
+8. Report: root cause with evidence, actions taken (tx hashes), final state. If you found a cause no case described, propose a kb.add case (with a test fixture) for the operator to approve.
 
 What the operator sees: $ARGUMENTS`,
 	},

internal/agent/commands.go

@@ -7,6 +7,7 @@ import (
 	"path/filepath"
 	"sort"
 	"strings"
+	"unicode"
 
 	"github.com/abhijitkrm/cometcli/internal/config"
 	"github.com/abhijitkrm/cometcli/internal/runbook"
@@ -232,7 +233,7 @@ func RunCommand(a *Agent, c *toolkit.Context, reg *toolkit.Registry, line string
 			return CmdResult{}, err
 		}
 		if len(f) == 1 {
-			txt := "general mode — shell and files on this machine; /one <profile> for node work"
+			txt := "general mode — shell and files on this machine; ask about a node by name and the agent switches to it (/one <profile> pins one)"
 			if a.Node() {
 				txt = "node mode: " + a.Ctx.Profile.Name + " — /one off for general mode"
 			}
@@ -468,14 +469,53 @@ func RunCommand(a *Agent, c *toolkit.Context, reg *toolkit.Registry, line string
 		if err := needAgent(); err != nil {
 			return CmdResult{}, err
 		}
+		prompt, text := cmd.Expand(args, nil, nil), "/"+name
 		if builtinNeedsNode[name] && !a.Node() {
-			return CmdResult{}, fmt.Errorf("/%s needs node mode — /one <profile> first", name)
+			p, err := a.nodeFor(args)
+			switch {
+			case err != nil:
+				return CmdResult{}, err
+			case p != nil:
+				// the operator named the node: work on it
+				a.SwitchScope(p)
+				text += " — working on " + p.Name
+			default:
+				// let the model pick the profile itself
+				prompt = "This is about one of the operator's nodes: call use_node with the right profile first.\n\n" + prompt
+			}
 		}
-		return CmdResult{Text: "/" + name, Prompt: cmd.Expand(args, nil, nil)}, nil
+		return CmdResult{Text: text, Prompt: prompt}, nil
 	}
 	return CmdResult{}, ErrUnknownCommand
 }
 
+// nodeFor finds the profile a node-only command is about: one named in
+// its arguments, or nil when none is (the model then picks with use_node).
+// It errors only when there are no profiles at all.
+func (a *Agent) nodeFor(args string) (*config.Profile, error) {
+	if a.Ctx == nil || a.Ctx.Cfg == nil || len(a.Ctx.Cfg.Profiles) == 0 {
+		return nil, fmt.Errorf("no node profiles yet — add one with `cometcli init` or `cometcli profile add`")
+	}
+	words := map[string]bool{}
+	for _, w := range strings.FieldsFunc(strings.ToLower(args), func(r rune) bool {
+		return r != '-' && r != '_' && r != '.' && !unicode.IsLetter(r) && !unicode.IsDigit(r)
+	}) {
+		words[strings.Trim(w, ".")] = true
+	}
+	var names []string
+	for n := range a.Ctx.Cfg.Profiles {
+		if words[strings.ToLower(n)] {
+			names = append(names, n)
+		}
+	}
+	if len(names) != 1 { // none, or ambiguous: the model decides
+		return nil, nil
+	}
+	p := a.Ctx.Cfg.Profiles[names[0]]
+	p.Name = names[0]
+	return p, nil
+}
+
 // memoryTarget picks the file /remember writes: --user, --node, or the
 // project's COMET.md (the user file without a project).
 func (a *Agent) memoryTarget(rest *string) (string, error) {

internal/agent/prompt.go

@@ -42,8 +42,11 @@ Incidents and "is something wrong?" questions — work the method:
    height, or signal="<triage expr>") instead of guessing; if progress
    stalls, re-triage.
 5. Verify with the case's verify steps, then re-run node.triage.
-6. Report cause + evidence, actions (tx hashes), final state. If no case
-   fit and you found the cause, offer to record it with kb.add.
+6. Save it with incident.record, then report cause + evidence, actions
+   (tx hashes), final state. If no case fit and you found the cause,
+   offer to record it with kb.add.
+Triage shows what changed since the last check and this node's past
+incidents; node.history answers "since when?" and "is it getting worse?".
 
 `)
 	b.WriteString(strings.Replace(generalToolsText, "General tools:", "General tools (they run on the node's host — local or over SSH):", 1))

internal/agent/scope.go

@@ -47,7 +47,7 @@ ENVIRONMENT: %s/%s, host %s, working directory %s, date %s
 		if ap, err := c.Cfg.ActiveProfile(""); err == nil && ap != nil {
 			active = " Active profile: " + ap.Name + "."
 		}
-		fmt.Fprintf(&b, "NODE PROFILES: %s.%s For any question about a node, validator or chain (jailing, sync, peers, upgrades, governance, txs), call use_node with the right profile yourself and continue — never ask the operator to switch. Any profile on a chain answers chain-wide questions; pick the active one when nothing points elsewhere.\n", strings.Join(names, "; "), active)
+		fmt.Fprintf(&b, "NODE PROFILES: %s.%s For any question about a node, validator or chain (jailing, sync, peers, upgrades, governance, txs), call use_node with the right profile yourself and continue — never ask the operator to switch. Any profile on a chain answers chain-wide questions; pick the active one when nothing points elsewhere. For questions about several nodes or the whole network (\"how is my fleet?\"), start with fleet.triage.\n", strings.Join(names, "; "), active)
 	}
 	return b.String()
 }

internal/agent/toolsearch.go

@@ -22,6 +22,7 @@ var coreTools = map[string]bool{
 	"val.status": true, "val.signing": true,
 	"val.jail-check": true, "val.consensus": true, "wait.until": true, "chain.validators": true,
 	"node.triage": true, "kb.show": true, "kb.search": true,
+	"node.history": true, "incident.record": true, "fleet.triage": true,
 }
 
 var toolSearchDef = ToolDef{
@@ -66,7 +67,7 @@ func (a *Agent) advertised(t toolkit.Tool) bool {
 		return true
 	}
 	if !a.Node() {
-		return false // node tools need a node: /one <profile>
+		return toolkit.IsFleetWide(t) // other node tools need a node (use_node)
 	}
 	if !a.deferred() || coreTools[t.Name()] {
 		return true