[karl-power]

Summary

Label lookups (/labels, /label/:name/values) for ClickHouse-backed PromQL sources now go through ClickHouse's prometheus_api_v1 handler, the same way /query and /query_range already do. SQL against timeSeriesTags() stays as the fallback.

• The HTTP API is used only when the source resolves to clickhouse-http, the server is 26.9+ (when the label endpoints shipped), and the existing handler probe passes. Otherwise lookups use SQL, so a source querying over HTTP on 26.6–26.8 keeps label autocomplete.
• Adds supportsPrometheusLabelsApi (26.9+) to common-utils. The probe cache now stores { query, labels } per connection, so no extra probe request is made.
• handleLabelLookup proxies with the source's database/table, start/end/limit/match[], and the ClickHouse auth and attribution headers. The chosen path is recorded on the span as hyperdx.promql.label_backend.
• On the HTTP path, limit is capped at PROMETHEUS_MAX_RESULT_ROWS (100k), including when absent or 0. The handler returns all values in one row, so max_result_rows can't bound it the way it bounds the SQL path.
• The SQL path's match[] error now names the HTTP handler on 26.9+ as the alternative to storing time ranges.
• Separate commit: the time ranges probe now matches ClickHouse errors by code (UNKNOWN_FUNCTION/UNKNOWN_TABLE) instead of message text. isClickHouseError/getClickHouseErrorType move from mcp/tools/query/helpers.ts to utils/clickhouseErrors.ts to avoid a circular import.

How to reproduce

Starting point: yarn dev (ClickHouse 26.10). In the ClickHouse play editor, run one statement at a time:

CREATE TABLE default.hdx5420_v8 ENGINE = TimeSeries

INSERT INTO default.hdx5420_v8 (metric_name, tags, samples) VALUES
  ('up', map('job', 'api'), [(now(), 1)]),
  ('up', map('job', 'worker'), [(now(), 1)]),
  ('http_requests_total', map('job', 'web'), [(now(), 1)])

In Team Settings → Sources, add a PromQL source on "Local ClickHouse" for default.hdx5420_v8 with the backend on Auto.

On the base branch:

1. Open /api/v1/prometheus/label/job/values?sourceId=<id>&match[]=up (logged in).
2. Run SYSTEM FLUSH LOGS, then:

   SELECT event_time, interface, substring(query, 1, 90) AS q
   FROM system.query_log
   WHERE event_time > now() - 300 AND type = 'QueryFinish' AND query_id LIKE 'hdx-api-%'
   ORDER BY event_time DESC

3. The lookup ran as SQL: interface = HTTP, query text contains timeSeriesTags(.

On this branch:

1. Same request. The response is {"status":"success","data":["api","worker"]}.
2. Same query-log check. The lookup ran inside the handler: interface = TCP, query text starts with SELECT arraySort(groupUniqArrayArray(, and there is no timeSeriesTags( row.
3. Set the source's backend to Table functions and repeat: same data, and the query log shows the timeSeriesTags( SQL again.

How to test on Vercel preview

N/A — non-UI change.

References

• Linear Issue: part of HDX-5420
• Related PRs: stacked on feat: Add a PromQL backend setting to ClickHouse PromQL sources #3601. Builds on feat: upgrade clickhouse versions and use promql remote endpoints #3142 (query proxy) and chore: upgrade ClickHouse to 26.10 #3624 (ClickHouse 26.10).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hyperdx-oss	Ready	Preview	Nov 12, 2026 1:59pm UTC
hyperdx-storybook	Ready	Preview	Nov 12, 2026 1:59pm UTC

[changeset-bot]

🦋 Changeset detected

Latest commit: d4468c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/common-utils	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Duplicate code analysis

No new duplicate code introduced by this PR.

---

jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

[github-actions]

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

• Diff size: 309 production lines changed (Tier 2 max: < 250)
• Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)
• Touches API routes or data models — hidden complexity risk

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats

• Production files changed: 6
• Production lines changed: 309 (+ 367 in test files, excluded from tier calculation)
• Branch: claude/hdx-5420-promql-labels-http-api
• Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

• 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
• 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
• Anything about surrounding code, or asking you to widen the change (hoist a helper,
  dedupe with another file, fix other call sites): never fix here, whatever the
  severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

1 minor

• 🔵 packages/api/src/routers/api/__tests__/prometheus.int.test.ts:835 — Live match[] test can't tell whether the handler served the narrowed request → mockFetch is only reset in the outer afterEach, so by the time of narrowed the call list already holds the unfiltered /label/job/values handler call. That makes toContain pass even if the match[] request fell back to SQL. The fixture table stores time bounds, so SQL would also return ['web'] and []. Fix: call mockFetch.mockClear() at the start of lookup (or assert on toEqual for that call only), and assert handlerPaths for the out-of-window lookup too.

---

_{Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.}

[github-actions]

Deep Review

✅ No critical issues found. The diff is a focused, defensively-written backend change: label lookups route to ClickHouse's prometheus_api_v1 handler on 26.9+ and fall back to SQL otherwise, reusing the shared probe cache and the existing hardened proxy path. Independent verification confirmed the renamed/moved symbols have no dangling references (connectionSupportsPrometheusHttpApi, proxyPath, and the relocated getClickHouseErrorType/isClickHouseError are all updated at every call site), the limit cap holds for the 0/absent/over-cap cases, and request-supplied database/table keys are stripped by the zod schema before the source's own values are applied.

🟡 P2 — recommended

• packages/api/src/routers/api/prometheus.ts:910 — the catch-all handler returns raw e.message to the client, so a ClickHouse SQL error on the fallback path can surface table/schema internals in the response body; this is pre-existing but now also governs the new backend split.
  • Fix: return a generic bad_data message to the client and log the underlying error, matching how the proxy path already redacts upstream detail.
  • _{security, correctness}

🔵 P3 nitpicks (1)

• packages/api/src/controllers/timeseriesEngine.ts:579 — isNoTimeRangesTableError now treats UNKNOWN_TABLE as expected, which also matches a genuinely missing TimeSeries table; the code comment argues the earlier Tags probe already surfaces that case, so the masking is intentional but relies on an invariant in a separate function.
  • Fix: add a focused test asserting a missing-table scenario is still observable upstream, so the documented invariant is guarded against regression.

---

Reviewers (13): correctness, security, adversarial, reliability, api-contract, performance, kieran-typescript, testing, maintainability, project-standards, previous-comments, agent-native, learnings-researcher.

Testing gaps:

• No route-level test asserts a table-function source reads labels via SQL on a 26.10 server (covered at the unit level in resolveClickhouseLabelBackend, and the HTTP cap/version-gate paths are well covered).
• No assertion that arbitrary extra query keys (e.g. a ClickHouse setting name) are stripped on the label path — safe today because the zod schema strips unknown keys by default, but untested against regression.

_{No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond}

[github-actions]

E2E Test Results

✅ All tests passed • 465 passed • 1 skipped • 1247s

Status	Count
✅ Passed	465
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

The merge-base changed after approval.