[CamiloSierraH]

Why

The alert summary says what is wrong, but not how to see it on the customer's own cluster, or which of several things it is. On a replication-lag case the engineer suspected thread-pool saturation but could not tell which pool, and there was nothing to hand the customer to check.

What changed

Under every alert the summary displays (fired, or its query failed), one collapsed line:

▸ queries to run on your cluster (N)

Opening it shows the following, each with a copy button, in the scrollable box the stack traces use:

1. The query that raised this alert. It is the rule's query exactly as it ran, with {sys.*} expanded for the bundle's mode, so it pastes into clickhouse-client on the same cluster and returns the alert's rows per host.
2. Follow-ups, from a new optional investigate: list in the rule YAML (title plus a read-only SELECT):
   • fetch_pool_saturated, schedule_pool_saturated, parts_propagation_lag: every background pool right now, per replica, busy against its size, and each pool's hourly peak over 24 h. This answers "which pool". On SharedMergeTree, propagation lag from a saturated pool is P-59.
   • replication_queue_errors: stuck entries grouped by table, type and cause (the message with numbers normalised, so one cause is one row), and replica health for the tables with a queue.

Further details:

• Not run by the tool. Follow-ups are only shown, so collection cost does not change. They go through the same ValidateQueryContent read-only check as the rule's query. A rule with an unsafe follow-up is reported as broken and shows no SQL.
• No SQL for clean rules. Rules that found nothing carry no SQL, so the page does not grow for them, and alerts_summary.json stays counts only (tested).
• Testable main query. The rule's main query now goes through the evaluator's existing query() seam, which runs the same ExecuteQueryWithFormat in production, so a fired result can be unit-tested.

Verification

• Lab, real evaluator and dashboard: I fired fetch_pool_saturated using a temporary copy with its thresholds set to zero (the shipped rule is unchanged). In headless Chrome the block renders collapsed. The three queries, copied from the rendered page as shown, all run on the cluster, and the alert's own query returns the row that fired.

• Every follow-up runs: on 26.7 in both the onprem and the cloud (clusterAllReplicas) form, and on the 22.8 floor. On a lab replica with a stuck queue, the replica-health follow-up shows the table and its delay.

• Tests:

  • a fired result carries its query, exactly as run and expanded for the mode;
  • a clean result carries none;
  • a failed query is shown;
  • a non-SELECT follow-up breaks the rule and runs nothing;
  • the shipped follow-ups have titles and pass the read-only check in every mode;
  • the summary JSON carries no SQL;
  • the template block is collapsed and escaped.

  Each of the 6 behaviours fails its test when reverted.

• Version overrides: the alerts/23.3.1.0/ overrides of the pool rules, which are what runs on every current server, carry the same follow-ups. A test requires every override's investigate: to equal its root's, and a RunAll at 26.7 checks the version-selected results.

• gofmt, go vet (also GOOS=windows) and make test are green: 273 passed. 11 files.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Versioned pool rules omit the follow-ups, and queue-error grouping combines distinct failure causes.

2 open issues

• Versioned alert overrides omit investigate follow-ups · New
• Group retry statistics by last_exception · New

What changed in this PR

Adds copyable diagnostic SQL beneath displayed dashboard alerts, helping engineers investigate findings on their own cluster.

Changes:

• Displays alert queries and optional follow-ups in collapsed, escaped blocks.
• Validates follow-ups without executing them and keeps summary JSON SQL-free.
• Adds pool and replication diagnostics, documentation, and tests.

File	Description
README.md	Documents displayed SQL and follow-up definitions.
internal/​dashboard/​generator.go	Renders collapsed queries with copy buttons.
internal/​dashboard/​alerts_panel_test.go	Tests query rendering and escaping.
internal/​alert/​alert.go	Validates and attaches diagnostic SQL.
internal/​alert/​alert_test.go	Tests SQL validation and result handling.
alerts/​schedule_pool_saturated.yaml	Adds pool diagnostic follow-ups.
alerts/​replication_queue_errors.yaml	Adds queue and replica-health queries.
alerts/​parts_propagation_lag.yaml	Adds pool diagnostics for propagation lag.
alerts/​fetch_pool_saturated.yaml	Adds fetch-pool diagnostic follow-ups.

_{Review effort: Balanced}