[knudtty]

Summary

Map key=value text index value lookups now run in parallel batches of 10 keys. Indexes with large dictionary blocks read each key separately and return its most frequent values with topKWeighted, so a high-cardinality key stops early and delays only its own batch. Indexes with small blocks keep main's query shape within each batch. Plain-column text indexes return the most frequent values with topKWeighted, in batches of 10 columns.

Stacked on #3635, which limits these reads to a configurable set of parts (20 median-sized by default).

The problem

getMapTextIndexKeyValues (Map key=value indexes, e.g. idx_log_attr_items) and getTextIndexKeyValues (text indexes on plain columns, e.g. TraceId):

1. Kept the alphabetically first values (groupUniqArray(limit) over a sorted dictionary), not the common ones.
2. Ran up to 100 keys in one query. A key with a value per row (request IDs, UUIDs) has a token per row, and every other key in that query waited on it.
3. Capped the whole query at max_rows_to_read (3M by default) with read_overflow_mode = 'break'. The dictionary is read in sorted token order, so once a high-cardinality key used up the budget, every key that sorts after it came back with no values and no error. On a real LogAttributes lookup, only 22 of 100 keys got values.

The fix

• topKWeighted(limit)(value, cardinality). cardinality is the token's row count in the part, so the values are the most frequent, most frequent first.
• Plain-column text indexes run in queries of at most 10 columns (TEXT_INDEX_KEYS_PER_QUERY). They stream in as each query finishes.
• Query shape depends on the index's dictionary block size, for Map key=value indexes:
  • 2048 or more (PER_KEY_TEXT_INDEX_MIN_DICTIONARY_BLOCK_SIZE): queries of at most 10 keys, each key in its own UNION ALL branch reading at most 20,000 index rows (TEXT_INDEX_TOKENS_PER_KEY), so a high-cardinality key's read stops early. Plain-column text indexes use the same cap. mergeTreeTextIndex() returns one row per part and value, so the cap counts a value once per part, and values that only appear in parts read after it are missed. That's enough to pick a filter's top values. "Load more" (mapKeyIndexMaxParts: null) wants every value, so it reads every part with no cap. No max_rows_to_read cap ('0', after the team's settings): the per-key LIMIT bounds each read.
  • Smaller, including ClickHouse's default of 512: main's query shape, in queries of at most 10 keys: one read with OR'd startsWith filters, groupUniqArray(limit), and the max_rows_to_read cap with read_overflow_mode = 'break'. It keeps main's known limits: the alphabetically first values, and keys that sort after a high-cardinality key can come back empty. Batches of 10 limit that to at most 9 other keys. The cap is marked with a shortcut: comment.
  • The block size comes from dictionary_block_size in the index definition, then the table's text_index_dictionary_block_size, then the server's MergeTree setting, then 512.
  • Plain-column text indexes keep one branch per column. Each branch reads a different index, so nothing is read twice. No max_rows_to_read cap either.

Why the block size decides

Every read of mergeTreeTextIndex() opens each part's index and loads its sparse index, which has one entry per dictionary block. Separate UNION ALL branches don't share that work.

On a local benchmark (1M rows, 3M tokens, 10 parts), a batch of 10 keys:

	Single read (OR'd startsWith)	One branch per key
TextIndexReadSparseIndexBlocks	10	100
Index files opened (cache misses)	20	~160

The sparse index was 17.7 KB per part with 512-token blocks and 1.3 KB with 8192. On ClickHouse Cloud, an earlier version with per-key branches on default-block indexes measured about 0.18 CPU-s and 10–30 MiB of cache reads per part per branch (26-part otel_logs: 804 ms for one key, 772 ms for a key that doesn't exist). With 8192-token blocks that fixed cost is small enough for per-key branches to pay off.

Cost and known limits

• Batches of 10 mean up to 10× more queries than main. Each opens every part's index once (small blocks) or once per key (large blocks).
• On small-block indexes, a high-cardinality key reads all its tokens in the parts read, so its batch is slower. Results stay complete.
• Block size detection is per table setting. Parts written before someone changed a table's text_index_dictionary_block_size keep their old block size. This only picks the query shape; results are the same either way.

Seed schemas: 8,192-token blocks as a table setting

The collector's migration now runs ALTER TABLE … MODIFY SETTING text_index_dictionary_block_size = 8192 on otel_logs, otel_traces and hyperdx_sessions after the seed, so the per-key path applies to them.

• Version gate: ClickHouse only has that setting from 26.6, checked against local 26.3, 26.5 and 26.6 images. The full-text seeds run on 26.2+, so the ALTER is limited to 26.6+, and 26.2–26.5 keep 512-token blocks.
• Existing installs get it too: new parts use 8,192-token blocks, and existing parts keep theirs until merged.
• Custom values are kept: a table that already sets the value is left alone.
• Failures don't block startup: they log a warning, like the TTL reconcile.
• Tested: a real migration against fresh 26.5 and 26.6 containers. On 26.6, a rerun left a manually set 1024 in place.
• topKWeighted uses about 1.6× the CPU per token of groupUniqArray. Memory is the same.
• getEngineSetting moved from DBRowTable.tsx to common-utils/core/utils.ts so both use one parser for engine_full settings.

How to reproduce

Starting point: yarn dev against ClickHouse 26.3+, with the OTel otel_logs schema (key=value text indexes idx_log_attr_items, idx_res_attr_items), and a Map key with one rare value that sorts first and one common value, e.g. map('freq.key', if(number = 0, 'a-rare', 'z-common')) from numbers(4).

On main:

1. On Search, turn on showing all values in the filter panel. freq.key lists a-rare first, and system.query_log shows one values query for all keys.

On this branch:

1. Repeat step 1 on ClickHouse 26.6+ with a fresh stack, so the migration sets text_index_dictionary_block_size = 8192. freq.key lists z-common first.
2. In system.query_log, there is one values query per 10 keys, with one UNION ALL branch per key. Run ALTER TABLE otel_logs MODIFY SETTING text_index_dictionary_block_size = 512 and reload: the query shape follows the table setting, so each values query now has main's shape for 10 keys.

References

• Related PRs: perf: share one Map key read between concurrent callers #3611, perf: read a configurable set of parts for text index key and value lookups #3635. Absorbs the row cap change from fix: drop the row cap from text index value lookups and read a limited set of parts #3637 (closed).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hyperdx-oss	Ready	Preview	Nov 13, 2026 8:27pm UTC
hyperdx-storybook	Ready	Preview	Nov 13, 2026 8:27pm UTC

[changeset-bot]

🦋 Changeset detected

Latest commit: 46be94f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/otel-collector	Patch
@hyperdx/common-utils	Patch
@hyperdx/api	Patch
@hyperdx/app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Duplicate code analysis

5 new clones introduced by this PR.

• packages/api/src/models/source.ts:179-205 ↔ packages/api/src/models/source.ts:235-261 — 27 lines
• packages/common-utils/src/core/metadata.ts:1291-1315 ↔ packages/common-utils/src/core/metadata.ts:1339-1360 — 25 lines
• packages/common-utils/src/core/metadata.ts:1813-1822 ↔ packages/common-utils/src/core/metadata.ts:1942-1951 — 10 lines
• packages/common-utils/src/core/metadata.ts:1822-1835 ↔ packages/common-utils/src/core/metadata.ts:1951-1965 — 14 lines
• packages/common-utils/src/types.ts:2769-2783 ↔ packages/common-utils/src/types.ts:2816-2830 — 15 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.

---

jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.17%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

[github-actions]

E2E Test Results

✅ All tests passed • 465 passed • 1 skipped • 1274s

Status	Count
✅ Passed	465
❌ Failed	0
⚠️ Flaky	3
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →