[CamiloSierraH]

Why

A recent sev-1 Keeper quorum loss turned on two facts the bundle did not carry:

1. One member had fallen behind the leader's retained log, so it could rejoin only through a full snapshot.
2. That snapshot was over 2 GiB and sent unchunked (snapshot_transfer_chunk_size = 0, the default). Past 2 GiB the transfer fails on every leader in turn (std::bad_alloc), and the ensemble ended up with no leader.

Both facts are one four-letter word away (lgif, conf), from any ClickHouse node, with no access to the Keeper hosts.

What changed

The collector sends seven read-only commands per member instead of three:

Command	Answers when not serving?	Gives
ruok	yes	liveness
isro	yes	ro/rw
lgif	yes	Raft log position: first/last index, committed vs leader-committed, last snapshot index
srvr, mntr	no	as before, including zk_latest_snapshot_size
conf	no	snapshot_transfer_chunk_size, write_snapshot_version, snapshot_distance, reserved_log_items … (in gov only numbers, booleans and plain words survive; paths, hosts, *_path, *_disk, dataDir, dataLogDir and server.N say REMOVED)
dirs	no	bytes in the snapshot and log directories

The ones that answer on a member that is not serving go first, because that is the state of a quorum loss. Keeper checks isServerActive() only for the others; I read this from its source at 26.8. Commands with side effects (csnp, rqld, ydld, rcvr, srst, crst, rclc, clrs) are never sent. Neither are those that list clients (cons, stat, dump, wchc, wchp).

• Parallel, fail-fast. Members are asked in parallel, and a member is asked nothing more after its first network failure. A member that accepts connections but never answers (seen in that incident) used to cost the 10 s read timeout once per command.
• "Not serving" is a status. A member answering This instance is not currently serving requests used to be filed as ok.
• Older Keepers stay ok. A command that an older Keeper lacks (lgif on 22.8) no longer makes the member partial; status is judged on ruok/srvr/mntr as before.

The pre-pass reads keeper/ for the first time:

• HC-3.18: availability from Keeper's own replies, per ensemble. It is critical only on the members' own word: two or more answer not currently serving requests and none is serving. A member not serving, or whose probe failed on the collector's side, is a warning that does not judge the quorum. The files hold the members this server knows, not proven voting membership.
• HC-3.19: within one ensemble, a member whose last_log_idx is below the observed leader's first_log_idx needs a full snapshot. Critical. Without a leader reply there is no verdict, because followers compact their logs independently. A truncated reply is never read.
• HC-3.20: a snapshot over 2 GiB with snapshot_transfer_chunk_size = 0 is critical (over 1 GiB, a warning). write_snapshot_version below 7, where sequential counters are still 32-bit, is a warning on 26.8+, where 8 is the default so a lower value was pinned. Before 26.8, where 6 is the default, it is info.
• With no conf in the file there is no finding: the chunk size is never assumed. Snapshot size and chunk setting are compared within one ensemble, and the format is judged per member against its own version.
• Ensembles are kept apart. system.zookeeper_connection lists the default connection and every auxiliary ZooKeeper, so each file now carries # ensemble: <name> (hashed in gov), and cross-member checks never mix ensembles. An auxiliary member's file is <ensemble>__<host>_<port>.txt, so a shared host:port never collides. Auxiliary blocks are never read as the default ensemble, even one named zookeeper.
• P-61 describes the pattern and the fix: keep the lagging member stopped, restart the up-to-date ones together, set snapshot_transfer_chunk_size and nuraft_snapshot_sync_ctx_timeout_ms on every member, and only then start the lagging one.

Measured (lab: 26.7, 3-node Keeper, 4 servers)

	before	after
Healthy ensemble, Keeper step	~1 ms per member	10 ms total, 7 commands × 3 members
One member frozen (docker pause: accepts, never answers)	30.02 s	10.01 s
Each command, median / max over 60 calls		0.25–0.33 ms / ≤ 1.9 ms, replies ≤ 2.7 KB

The commands do not grow with the data size: lgif reads counters, and dirs sums a few dozen files. That part comes from the source and was not measured on a 100M-znode Keeper.

Verification

• Collector tests: command order; stop after the first timeout, with the remaining commands written as skipped; members in parallel (three hung members fit inside one timeout); not serving; an older Keeper still ok; gov conf paths removed. Each one fails when its behaviour is reverted.
• Pre-pass tests: the quorum-loss shape (a 2.2 GiB unchunked snapshot, a member behind the retained log, format 6 pinned on 26.8, 2 of 3 serving is a warning, not critical); 1 of 3 serving is critical; a healthy ensemble with the fixes applied raises nothing; an older bundle without conf claims nothing; 26.7's default format is info. Each rule has a reverted-code mutation that fails its test.
• Lab, real binary:
  • The healthy run gives three members serving, one leader, and no false findings.
  • The run with a frozen member gives unreachable, as a 2-of-3 warning.
  • The 26.7 default format gets info.
• 22.8 Keeper: every new command exists except lgif, which returns an empty reply, now handled.
• Review round: each fix has a test that failed against the earlier code, and a mutation that reverts it fails that test. The fixes cover truncated replies, collector failures asserting an outage, mixed ensembles, a silent failed probe and the ensemble-wide version.
• From dashboard: a Keeper members panel — role, Raft log position and snapshot settings per member #58's review: Apache ZooKeeper's "This ZooKeeper instance is not currently serving requests" is recognised like Keeper's wording. A ZooKeeper member that was not serving used to come out ok. HC-3.19 requires exactly one serving leader, because mid-election two can report leader and the verdict depended on file order.
• Second review round: gov conf identifiers (Keeper and ZooKeeper reply formats), a shared endpoint across ensembles, a nested auxiliary zookeeper block, and no snapshot verdict without a leader. Each has a regression test, and each rule fails its test when reverted.
• gofmt, go vet (also GOOS=windows) and make test are green, with main (cluster: collect every host of a self-managed cluster from one command #52, archive: written to a temporary file and renamed into place only when complete; a failed archive exits 1 #53) merged in: 287 passed. 11 files: running-the-tool.md was updated with approval to exceed the usual 10.

Not covered: Keeper's own log files, which live on the Keeper hosts. The decisive bad_alloc lines are there, and the tool does not run on those hosts. That needs a separate Keeper-host collection.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Ensemble assumptions and reply handling can produce false critical diagnoses or hide probe failures.

5 open issues

• Reject truncated collector responses as incomplete · New
• Avoid asserting quorum failures without complete membership evidence · New
• Preserve ensemble identity when comparing member health · New
• Detect failed serving probes despite fail-fast collection · New
• Match format warnings to each member's version · New

What changed in this PR

Expands Keeper evidence collection and bundle analysis to diagnose quorum problems and snapshot-transfer failures.

Changes:

• Collects seven read-only commands per member, in parallel, stopping after network failures.
• Adds Keeper availability, Raft-log and snapshot-setting checks.
• Adds regression tests and documents P-61 recovery guidance.

File	Description
skills/​clickhouse-diagnostic/​scripts/​inspect_bundle.py	Parses Keeper replies and adds health checks.
skills/​clickhouse-diagnostic/​references/​known-patterns.md	Adds P-61 snapshot-transfer guidance.
skills/​clickhouse-diagnostic/​references/​health-checks.md	Documents HC-3.18–3.20.
skills/​clickhouse-diagnostic/​references/​file-guide.md	Explains new Keeper evidence and limitations.
skills/​clickhouse-diagnostic/​references/​bundle-layout.md	Documents reply sections and statuses.
README.md	Updates collection behavior and command documentation.
internal/​keeper/​fourletter.go	Expands commands, parallelizes collection and filters gov paths.
internal/​keeper/​fourletter_test.go	Tests collection behavior and filtering.
cmd/​prepass_keeper_test.go	Adds Keeper analysis fixtures and assertions.
cmd/​main.go	Updates CLI help and collection messages.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Gov redaction gaps, output collisions and unsupported ensemble or recovery conclusions remain unresolved.

4 open issues

• Prevent auxiliary zookeeper blocks from being labeled default · New
• Redact infrastructure identifiers and paths from gov bundles · New
• Prevent output collisions for shared endpoints across ensembles · New
• Avoid recovery verdicts without the observed leader log boundary · New

5 resolved since last review

• Preserve ensemble identity when comparing member health
• Avoid asserting quorum failures without complete membership evidence
• Reject truncated collector responses as incomplete
• Match format warnings to each member's version
• Detect failed serving probes despite fail-fast collection

_{Review effort: Balanced}