Skip to content

MCP docs: teach quoted Lucene exact-match form and the body-text token path #3560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wanjinhao1 wants to merge 2 commits into from
Closed

MCP docs: teach quoted Lucene exact-match form and the body-text token path #3560

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dddda9a
Fix #3538: document quoted Lucene exact-match form and token path for…
wanjinhao1 Nov 4, 2026
7b8dde0
Add changeset for MCP Lucene exact-match docs
wanjinhao1 Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/mcp-lucene-exact-match-docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@hyperdx/api': patch
---

docs: Teach the quoted Lucene exact-match form (field:"value") and the bare-term/hasToken body-text path in MCP tool descriptions
14 changes: 8 additions & 6 deletions packages/api/src/mcp/prompts/dashboards/content.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -881,11 +881,13 @@ Lucene field:value does NOT mean exact equality. It translates to ilike(field, '
ServiceName:api* does NOT mean "starts with api". It means "contains a substring starting with api".
In practice this often works for prefix matching, but it is NOT equivalent to SQL LIKE 'api%'.

For exact matching or true prefix/suffix/contains patterns, use SQL instead:
Exact: whereLanguage: "sql", where: "SpanKind = 'Server'"
Prefix: whereLanguage: "sql", where: "ServiceName LIKE 'api%'"
Contains: whereLanguage: "sql", where: "Body LIKE '%timeout%'"
Regex: whereLanguage: "sql", where: "match(ServiceName, '^api-v[0-9]+')"
For exact matching or true prefix/suffix/contains patterns:
Exact (stay in Lucene, QUOTE the value): SpanKind:"Server" (compiles to SpanKind = 'Server', uses the primary key)

@github-actions github-actions Bot Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — The recommended exact-match example SpanKind:"Server" matches no rows on real span data

The repo stores SpanKind as SPAN_KIND_SERVER (see packages/app/tests/e2e/seed-clickhouse.ts:214, packages/api/src/mcp/tests/trace.int.test.ts:260, and the hdx-eval generators). Today the unquoted SpanKind:Server still matches by substring. The new quoted form compiles to SpanKind = 'Server' (queryParser.ts:2076 → eq) and returns zero rows, so an agent that follows this advice gets an empty tile. Change the example to an enum column whose stored value equals the literal, e.g. ServiceName:"checkout", or use SpanKind:"SPAN_KIND_SERVER". Make the same change in schemas.ts:23.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

@github-actions github-actions Bot Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — 'uses the primary key' is false for SpanKind in the default schema

otel_traces is ORDER BY (ServiceName, SpanName, toDateTime(Timestamp)) (docker/otel-collector/schema/seed/00005_otel_traces.sql:41), so SpanKind = ... cannot use the primary key. Remove the claim. In schemas.ts:63, change 'can use the primary key' to say it applies only when the column is part of the table's ORDER BY.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Exact (SQL): whereLanguage: "sql", where: "SpanKind = 'Server'"
Prefix: whereLanguage: "sql", where: "ServiceName LIKE 'api%'"
Body text: prefer a bare Lucene term (timeout) or SQL hasToken(lower(Body), 'timeout') over LIKE '%timeout%' --
the token path uses the text index; Body:term still compiles to substring ILIKE and scans the whole column.
Regex: whereLanguage: "sql", where: "match(ServiceName, '^api-v[0-9]+')"

When Lucene substring matching is FINE: free-text columns like Body, StatusMessage where substring is the intent.

@github-actions github-actions Bot Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The new body-text guidance contradicts the next line, which still calls Body:term substring 'FINE'

Line 889 now says Body:term compiles to ILIKE and scans the whole column. Line 892 still recommends Lucene substring matching on Body. Reword line 892 so free-text search points to a bare term or token search instead of Body:value.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

When Lucene substring matching is WRONG: enum-like columns (SpanKind, SeverityText, StatusCode, ServiceName) where you want exact or true prefix match.
Expand All @@ -901,7 +903,7 @@ Lucene queries that reference map-attribute keys via dotted paths (http.status_c

Reliable in Lucene (top-level columns only):
Duration:>1000000000 (top-level numeric range)
ServiceName:checkout (top-level string -- but remember this is a substring match, not exact!)
ServiceName:"checkout" (top-level string, quoted for exact match; unquoted ServiceName:checkout is substring!)

Workaround: switch to SQL with bracket access. Map values are stored as strings, so quote the comparison value.

Expand Down
7 changes: 5 additions & 2 deletions packages/api/src/mcp/tools/query/schemas.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ const WHERE_DESCRIPTION =
'WRONG: level = "error" (SQL syntax with whereLanguage:"lucene")\n\n' +
'SUBSTRING TRAP: Lucene field:value matches ANY row containing "value" as a substring, not exact equality.\n' +
' SpanKind:Server matches "Server", "ServerStreaming", "InternalServer", etc.\n' +
" For exact match, use SQL: SpanKind = 'Server'";
' For an exact match stay in Lucene and QUOTE the value: SpanKind:"Server" (= SpanKind = \'Server\').\n' +
" For exact match in SQL: SpanKind = 'Server'";

const WHERE_LANGUAGE_DESCRIPTION =
'Query language for the "where" filter. Default: lucene.\n' +
Expand All @@ -31,7 +32,9 @@ const WHERE_LANGUAGE_DESCRIPTION =
'and map attributes via dot notation. Use "sql" for IN(...) lists, complex expressions, or function calls.\n' +
'IMPORTANT: Lucene field:value is a SUBSTRING match (ilike), not exact equality. ' +
'field:val* is prefix-within-substring, not a true prefix match. ' +
"For exact matching or reliable wildcards, use SQL: WHERE field = 'value' or WHERE field LIKE 'val%'.\n\n" +
'For an exact match in Lucene, quote the value: field:"value" compiles to field = \'value\' and can use the primary key. ' +
"For reliable wildcards, use SQL: WHERE field = 'value' or WHERE field LIKE 'val%'.\n" +
"For free-text body/message columns, prefer a bare Lucene term (e.g. timeout) or SQL hasToken(lower(Body), 'timeout') over LIKE: they use the text index instead of scanning the whole column.\n\n" +
'Common mistake: writing Column:value (Lucene) but setting whereLanguage to "sql". ' +
'If your filter uses colon syntax, leave whereLanguage as "lucene" (the default).';

Expand Down
Loading