SEARCH.md: a code query engine on treebank, and its prior art - #140
Open
zmaril wants to merge 1 commit into
Open
SEARCH.md: a code query engine on treebank, and its prior art#140zmaril wants to merge 1 commit into
zmaril wants to merge 1 commit into
Conversation
A sketch of an indexed structural search engine: supertype bits
materialised at index time (§2 fact 4 makes them underivable afterwards),
interval labels so containment is an integer comparison, a trigram
prefilter, and a re-parse verification pass so the index is allowed to be
approximate.
The first draft claimed the XML twig-query approach had never been applied
to code. That is false, and the correction is worth more than the design.
srcML has marked code up as XML and queried it with XPath since 2002.
Babelfish parsed many languages into a Universal AST annotated with
language-agnostic ROLES -- the same word this repo uses -- and queried it
with XPath, which is the thesis of DESIGN.md §3 shipped eight years
earlier. Kythe, Glean and CodeQL all index structure and query it.
What is left is narrow and stated as such: where the vocabulary is enforced
(in the parse, not a post-hoc annotation layer that can drift), and
published per-language confidence. And a warning -- source{d} shut down and
Babelfish is unmaintained. The hard part was never the index, it was
keeping N frontends mapped onto one vocabulary while N languages moved.
That is exactly this repo's cost structure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZQR1QTvqArX8vMxYKK3rv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A design sketch, not a commitment. Independent of the measurement stack —
based on
main, no code.The design. An indexed structural engine: supertype bits materialised at
index time (DESIGN.md §2 fact 4 says supertype membership is
derivation-based, so it is not recoverable from
(kind, parent)after thefact — it has to be written while the tree is in hand);
(pre, post)interval labels so containment is an integer comparison rather than a tree
walk; a trigram prefilter; and a re-parse verification pass so the index is
allowed to be approximate. Crates named throughout —
roaring,bitpacking,fst,tantivy,rkyv,memmap2,redb.One nice accident: the per-
(parent, field, child)edge counts added in#137 for coverage measurement are exactly the cardinality estimator a query
planner needs.
The prior art, which is the part worth reading. My first draft claimed
this approach had "never been applied to code." That is false:
queries it with XPath/XSLT.
a Universal AST annotated with language-agnostic roles — the same word
this repo uses — and queried it with XPath so one query worked across
languages. That is the thesis of DESIGN.md §3, shipped eight years
earlier.
What survives is narrow, and the doc says so: where the vocabulary is
enforced (in the parse, versus Babelfish's post-hoc annotation layer — which
is precisely the "query layer that can drift" the project exists to avoid),
and published per-language confidence, which none of them have.
And a warning. source{d} shut down; Babelfish is unmaintained. The hard
part was never the index — it was keeping N language frontends mapped onto
one vocabulary while N languages kept moving. That is exactly this repo's
cost structure, and it belongs in the record.