Repository navigation
feat(repo): read a SCIP code index (#41) - #339
Conversation
Add dokimi.repos.code_index: load a SCIP index.scip written by another tool and answer definitions, references, enclosing scopes, base classes and per-file occurrences. dokimi runs no indexer. The protobuf wire format is decoded with the standard library only, so no dependency is added. Unknown fields are skipped, so an index from a newer tool still reads. A malformed, empty or truncated file raises CodeIndexError. root_prefix maps document paths to repo paths, and a local N id is scoped to its document: the queries that take a symbol refuse it. Tests: a committed fixture built with scip-python 0.6.8 from a three-file package, cross-checked against scip print, plus hand-built bytes for unknown fields, truncation at every cut, random input and bad input. The docs gain a "Code index input" subsection (1.2). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…es (intel#41) Self-review of the code index reader on a scip-clang index: - bases() reduces scip-clang's ancestor lists to direct bases; - an occurrence repeated identically in one document is kept once; - definitions() looks a query up as a full symbol first, then as a name; - Occurrence gains end_line for 4-value ranges; - docs say a cut at a document boundary reads with fewer documents. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
scip-python lists direct bases, so a Python class C(B, A) with B(A) lost its direct base A. The reduction now runs only when most classes with a grandparent list every ancestor (scip-clang: 1,716 of 1,733 on oneDAL; scip-python: 0 of 38 on dokimi). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ntel#41) A small scip-python index whose only deep class is C(B, A) with B(A) won the vote and lost A. scip-clang and scip-python are now decided by name; the majority vote covers other tools, with a test per branch. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 13 selected item(s). Review details
|
| best: Occurrence | None = None | ||
| best_span = -1 | ||
| for first, last, occ in self._scopes.get(path, ()): | ||
| if first <= line <= last and (best is None or last - first < best_span): |
There was a problem hiding this comment.
Must-fix: SCIP range endpoints are exclusive, but the decoder keeps the ending line without its column and this comparison includes it unconditionally. For example, a definition with enclosing_range=[1, 0, 9, 0] makes enclosing("a.py", 9) return that definition even though its scope ends before line 9. Normalize the last included line during decoding (subtract one when a multiline range ends at column zero), and test the first excluded line. The current decoder boundary test explicitly expects the incorrect result.
There was a problem hiding this comment.
Fixed in de746b6. Enclosing ranges are half-open now: a multiline end at column 0 leaves the line before it as the last covered line (_enclosing). enclosing("a.py", 9) is None for [1, 0, 9, 0]. Tests: test_enclosing_is_half_open_so_an_end_at_column_zero_excludes_that_line, test_enclosing_ending_after_column_zero_includes_its_end_line.
| return tuple(base for base in listed | ||
| if not any(base in above[other] and other not in above[base] for other in listed)) |
There was a problem hiding this comment.
Must-fix: transitive reduction cannot always recover declared direct bases from an ancestor listing. Valid C++ such as struct A {}; struct B : A {}; struct C : B, A {}; produces C:[B,A] and B:[A], but this filtering returns only B despite A also being a declared direct base. These relationships cannot distinguish that case from C inheriting only B. Either retain/report the ambiguity or document the result as a reduced ancestor graph rather than guaranteed direct bases; update the direct-base promise in the repository contract as well and add a redundant-inheritance case.
There was a problem hiding this comment.
Not changed in this push: this needs a contract decision, not only code. Options are to keep the direct-base promise and report the ambiguous cases, or document the result as a reduced ancestor graph and update the repository contract. Will follow up once decided.
There was a problem hiding this comment.
Decided: document it as a truncated ancestor graph (option b). bases() returns a minimal set that still reaches every ancestor, not declared direct bases. Docstring and repo-evaluation-contract.md updated in 1b12c9d. The C:[B,A], B:[A] case is test_bases_drop_an_ancestor_of_another_listed_base.
| if len(rng) not in (3, 4): | ||
| raise CodeIndexError(f"an occurrence range has {len(rng)} values, expected 3 or 4") | ||
| if enc and len(enc) not in (3, 4): | ||
| raise CodeIndexError(f"an enclosing range has {len(enc)} values, expected 3 or 4") |
There was a problem hiding this comment.
Must-fix: range validation checks only the number of coordinates, so malformed ranges are returned as valid occurrences. For example, _index(index(document("a.py", (occurrence([1, 5, 2], SYM, roles=1),)))) succeeds with start_col=5 and end_col=2; a four-value range can likewise end before its starting line. This violates the malformed-input error contract and exposes impossible source spans to callers. Validate nonnegative int32 coordinates and lexicographic start/end ordering for both rng and enc, raising CodeIndexError on invalid spans, and add regression tests.
There was a problem hiding this comment.
Fixed in de746b6. Occurrence and enclosing spans are refused with CodeIndexError when a value is outside int32 or the end comes before the start (_check_span). Tests: test_an_occurrence_span_must_be_ordered_and_fit_int32, test_an_enclosing_span_must_be_ordered_and_fit_int32.
| raise CodeIndexError(f"an occurrence range has {len(rng)} values, expected 3 or 4") | ||
| if enc and len(enc) not in (3, 4): | ||
| raise CodeIndexError(f"an enclosing range has {len(enc)} values, expected 3 or 4") | ||
| enclosing = (enc[0], enc[2] if len(enc) == 4 else enc[0]) if enc else None |
There was a problem hiding this comment.
Must-fix: SCIP ranges are half-open, so an enclosing range [1, 0, 9, 0] excludes line 9 entirely. Converting it to (1, 9) makes CodeIndex.enclosing("a.py", 9) incorrectly return this definition, potentially attributing the following declaration or blank line to the preceding scope. For a four-value range whose end column is zero, store end_line - 1 as the last included line. Update test_enclosing_holds_the_first_and_last_line_of_the_range, which currently asserts the incorrect boundary behavior, and cover a nonzero end column as well.
Suggestion:
| enclosing = (enc[0], enc[2] if len(enc) == 4 else enc[0]) if enc else None | |
| enclosing = (enc[0], enc[2] - (enc[3] == 0) if len(enc) == 4 else enc[0]) if enc else None |
There was a problem hiding this comment.
Fixed in de746b6, same change as the other half-open thread: [1, 0, 9, 0] no longer covers line 9, and a nonzero end column ([1, 0, 9, 4]) keeps line 9. Updated test_enclosing_is_half_open_so_an_end_at_column_zero_excludes_that_line and added the nonzero case.
| pos += 1 | ||
| elif name and mark == "(": | ||
| # a method: `name(disambiguator).` | ||
| close = desc.find(")", pos) |
There was a problem hiding this comment.
Must-fix: find(")") treats a parenthesis inside a backtick-escaped method disambiguator as the closing delimiter. For example, descriptor_name("scip-test python pkg 1.0 Foo#bar(x)y).") returns None although this valid descriptor defines bar. Consequently, CodeIndex.definitions("bar") omits the definition. Parse the disambiguator using the escaped-name rules (while retaining support for empty ()) before checking the closing )., and add a regression test for an escaped disambiguator containing ).
There was a problem hiding this comment.
Fixed in de746b6. The method disambiguator is read with the escaped-name rules (_method_end), so a ) inside backticks no longer closes it; an empty () still works. Test: test_an_escaped_disambiguator_may_hold_a_closing_parenthesis.
| value |= (byte & 0x7F) << shift | ||
| if byte < 0x80: | ||
| return value, pos |
There was a problem hiding this comment.
Must-fix: the tenth varint byte must have a payload of at most 1; accepting any terminating byte permits values larger than protobuf's uint64 limit. For example, decode_index(memoryview(b"\x0a\x00\xc0\x02" + b"\x80" * 9 + b"\x02")) silently accepts an overflowing varint in unknown field 40 instead of raising CodeIndexError, violating the malformed-input contract. Reject a tenth byte greater than 1 and test both the valid uint64 maximum and an overflowing unknown-field value following metadata.
There was a problem hiding this comment.
Fixed in de746b6. A tenth varint byte above 1 is refused, and a continuation bit on the tenth byte is refused too (read_varint). Tests: test_a_varint_of_ten_bytes_must_fit_in_64_bits (max uint64 accepted; overflow refused).
There was a problem hiding this comment.
🟡 Changes recommended
Scope lookup remains incorrect because half-open range endpoints are mishandled, local-symbol scopes are dropped, and typed SCIP range encodings are not decoded.
3 open issues
What changed in this PR
3 new findings.
This PR adds an internal SCIP index reader for repository analysis under #41. It reads externally generated indexes without running an indexer or changing existing evaluation flows.
Changes:
- Adds standard-library decoding and queries for definitions, references, enclosing scopes, base classes, and file occurrences.
- Adds fixture-based and synthetic tests covering decoding, queries, and malformed input.
- Documents the library API and introduces
CodeIndexError.
| File | Description |
|---|---|
dokimi/tests/test_repo_code_index.py |
Tests queries against a real SCIP fixture. |
dokimi/tests/test_repo_code_index_decode.py |
Tests wire decoding, malformed input, and query edge cases. |
dokimi/tests/fixtures/code_index/sample_pkg/util.py |
Provides cross-file calls for the fixture. |
dokimi/tests/fixtures/code_index/sample_pkg/core.py |
Provides inheritance, definitions, and local variables. |
dokimi/tests/fixtures/code_index/sample_pkg/base.py |
Provides the fixture’s base class. |
dokimi/dokimi/repos/errors.py |
Adds CodeIndexError. |
dokimi/dokimi/repos/code_index/wire.py |
Implements bounded protobuf wire reads. |
dokimi/dokimi/repos/code_index/symbols.py |
Parses symbol names and identifies local symbols. |
dokimi/dokimi/repos/code_index/scip.py |
Decodes occurrences, metadata, and relationships. |
dokimi/dokimi/repos/code_index/index.py |
Implements loading and index queries. |
dokimi/dokimi/repos/code_index/__init__.py |
Exports the reader’s public entry points. |
dokimi/docs/repo-evaluation-contract.md |
Documents index input and reader limitations. |
dokimi/changelog.d/20261031_120000_napetrov_code-index-reader.md |
Records the new internal reader. |
Review effort: Balanced
| if not occ.symbol or is_local(occ.symbol): | ||
| continue | ||
| self._by_symbol.setdefault(occ.symbol, []).append(occ) | ||
| if not occ.is_definition: | ||
| continue | ||
| if occ.symbol not in names: | ||
| names[occ.symbol] = descriptor_name(occ.symbol) | ||
| name = names[occ.symbol] | ||
| if name is not None: | ||
| self._by_name.setdefault(name, []).append(occ) | ||
| if occ.enclosing is not None: | ||
| first, last = occ.enclosing | ||
| self._scopes.setdefault(occ.path, []).append((first, last, occ)) |
There was a problem hiding this comment.
Fixed in de746b6. Scopes are collected before the local-symbol filter, so a local N definition keeps its enclosing scope and has_enclosing_ranges sees it. Test: test_a_local_definition_keeps_its_enclosing_scope.
| if field == 1: | ||
| pos = read_ints(body, pos, wire, rng) |
There was a problem hiding this comment.
Fixed in de746b6. Typed ranges (fields 8-11, verified against upstream scip.proto) are decoded and take precedence over range and enclosing_range. Tests: test_typed_ranges_alone_decode, test_typed_ranges_take_precedence_over_the_deprecated_ones.
| raise CodeIndexError(f"an occurrence range has {len(rng)} values, expected 3 or 4") | ||
| if enc and len(enc) not in (3, 4): | ||
| raise CodeIndexError(f"an enclosing range has {len(enc)} values, expected 3 or 4") | ||
| enclosing = (enc[0], enc[2] if len(enc) == 4 else enc[0]) if enc else None |
There was a problem hiding this comment.
Fixed in de746b6, same half-open change as above. The expectations at test_repo_code_index_decode.py:110,118,242 now encode the exclusive end ((1, 8), (290, 309), and enclosing(9) is None).
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #339 +/- ##
==========================================
+ Coverage 94.05% 94.11% +0.06%
==========================================
Files 297 302 +5
Lines 38018 38442 +424
Branches 6424 6514 +90
==========================================
+ Hits 35756 36179 +423
- Misses 1452 1453 +1
Partials 810 810
🚀 New features to boost your workflow:
|
…opes - enclosing ranges are half-open: a multiline end at column 0 leaves the line before it as the last covered line - occurrence and enclosing spans are refused when a value is outside int32 or the end comes before the start - typed range fields 8-11 are decoded, and take precedence over range and enclosing_range - a varint of ten bytes must fit in 64 bits - a local definition keeps its enclosing scope - a method disambiguator may hold an escaped name with a closing parenthesis Thread: PRRT_kwDOVBg7g86yWxm0 Thread: PRRT_kwDOVBg7g86yWxm4 Thread: PRRT_kwDOVBg7g86yWxm6 Thread: PRRT_kwDOVBg7g86yWxnA Thread: PRRT_kwDOVBg7g86yWxnC Thread: PRRT_kwDOVBg7g86yWxsJ Thread: PRRT_kwDOVBg7g86yWxsR Thread: PRRT_kwDOVBg7g86yWxsa Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| best: Occurrence | None = None | ||
| best_span = -1 | ||
| for first, last, occ in self._scopes.get(path, ()): | ||
| if first <= line <= last and (best is None or last - first < best_span): |
There was a problem hiding this comment.
Must-fix: an empty enclosing range still matches a line. For example, the accepted input in test_a_zero_width_span_is_valid has enclosing_range=[4,3,4,3], but enclosing("a.py", 4) returns that definition even though a half-open empty range contains no position. Preserve whether the decoded enclosing range is empty and exclude it from scope lookup; add an assertion that this input returns None. This is separate from choosing between equal-span nested scopes.
There was a problem hiding this comment.
Fixed in 38c4779. An empty enclosing range (equal start and end) now holds no line, so enclosing() returns None for it. Test: test_an_empty_enclosing_range_holds_no_line.
| if first <= line <= last and (best is None or last - first < best_span): | ||
| best, best_span = occ, last - first |
There was a problem hiding this comment.
Must-fix: equal line spans leave the first definition selected, even when it is the outer scope. For example, put an outer definition with enclosing_range=[1,0,9,20] before an inner definition with enclosing_range=[1,15,9,10]. Both decode to (1,9), so enclosing(path,5) returns the outer definition although the inner range contains that line and is strictly nested. Preserve enclosing columns internally and use range containment/full-range extent to resolve nesting, rather than occurrence order; add an outer-first equal-line-span regression test.
There was a problem hiding this comment.
Fixed in 38c4779. When two enclosing spans have the same lines, the one nested inside the other by columns wins; otherwise the shorter line span. Test: test_the_inner_of_two_equal_line_spans_is_the_one_nested_by_columns, both orders.
| pos = read_ints(body, pos, wire, enc) | ||
| elif field in (8, 9, 10, 11) and wire == LEN: | ||
| raw, pos = read_delimited(body, pos) | ||
| typed[field] = _typed_range(raw, multi=field in (9, 11)) |
There was a problem hiding this comment.
Must-fix: protobuf merges repeated instances of a singular embedded message rather than replacing the whole message. For example, typed(8, 4, 1) + ld(8, vint(3, 7)) is a valid single-line range [4, 1, 7], but this assignment replaces the first message with [0, 0, 7], silently returning the wrong location. Merge explicitly supplied fields before applying defaults. The same replacement problem occurs with repeated tool_info messages in _metadata_tool and repeated metadata messages in decode_index; add split-message regression cases for those too.
There was a problem hiding this comment.
Fixed in 38c4779. Typed ranges (8–11) and Metadata.tool_info now merge field by field as protobuf does: a later field replaces the earlier one, an absent one keeps it. Tests: test_a_split_typed_range_merges_as_protobuf_does, test_a_split_tool_info_merges_name_and_version_across_metadata_messages.
| if field == 0: | ||
| raise CodeIndexError(f"field number 0 at byte {pos}: not a protobuf message") |
There was a problem hiding this comment.
Must-fix: validate the upper bound on protobuf field numbers as well as zero. Field numbers cannot exceed 2^29−1, but decode_index(memoryview(b"\x0a\x00\x80\x80\x80\x80\x10\x00")) accepts an invalid field numbered 2^29 and returns an empty index. This contradicts the malformed-input rejection contract. Reject field numbers above the maximum and test both the maximum legal number and the first illegal number.
There was a problem hiding this comment.
Fixed in 38c4779. read_key rejects field numbers above 2^29-1 with CodeIndexError. Tests: test_the_largest_field_number_is_legal_and_the_next_one_is_not and a malformed-input row for 2^29 (the bytes from the thread).
…uf messages, bound field numbers - An empty enclosing range holds no line. Of two definitions whose enclosing line spans are equal, the one nested inside the other by columns is the inner scope. - A repeated singular typed range (single/multi line, enclosing) and a repeated Metadata.tool_info merge field by field as protobuf does: a later field replaces the earlier one, an absent field keeps it. - A field number above 2^29-1 is refused with CodeIndexError. Tests: red on the previous head for each of the three, green here. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 3b30d89. The decoder and the query layer hold up, but three test gaps let real regressions through, and the direct-base promise is still open. Three inline comments.
What I ran, on a Linux review host at 3b30d89:
make ci-local: 30 passed, 0 failed, 7 skipped (other-Python legs, semgrep, conda not run locally); changed-line coverage gate passed.- A mutation run: 75 single-line mutants of the four new modules against the two new test files, 20 of which pass the full suite. The three that matter are inline.
Open bot points I confirm:
- I agree with @exai-bot that
bases()cannot recover a declared direct base such asAinstruct C : B, AwhenB : A. New evidence: a three-class index from an unknown tool with onlyC(B, A)andB(A)getslists_ancestorsTrue from the one-sample vote andbases(C)returns['B#'], losingA, while the docs promise direct bases. The author left this for a contract decision, so it is still open.
Checked, no issues
- Wire format: a 10-byte varint above 64 bits, field number 0 and 2^29, truncation at every cut and negative int32 ranges all raise
CodeIndexError. - Symbols:
descriptor_namereturns the right name for scip-python, scip-clang style and escaped symbols. - Hygiene: the new tests are offline and write only to
tmp_path; every commit carries aSigned-off-by.
Bot points I checked that no longer apply at this head (12)
- The half-open range, span-check, local-scope, typed-range, escaped-disambiguator, varint, field-number, empty-range, nesting and split-message points from @exai-bot and Copilot do not apply at this head: each is fixed, and the new tests pass.
…iew cases bases() reduces scip-clang's listed ancestors to a minimal set that still reaches each ancestor, so it is not the declared direct bases. The docstring and the repo-evaluation contract now say so, and an inherited base can be absent. Tests for the review of intel#339, each checked by a mutation: - a definition with extra role bits (roles 9) is still a definition; - a typed field-11 multi-line enclosing range keeps its end column; - the same symbol record in two documents gives each base once. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The last commit adds the three tests I asked for and documents bases() as a truncated ancestor graph, which settles the direct-base question. One nit inline.
What I ran, on a Linux review host at 1b12c9d:
make ci-local: 28 passed, 2 failed, 7 skipped. The two failures are the gitleaks install (HTTP 504 on download) and the commit scan that needs it; neither reads this PR's tree. Tests, mypy, coverage of changed lines, docs build, fragment and pip-audit pass.- Mutants of the three new tests:
self.roles == DEFINITION,field in (9,)and plain concatenation of the bases each fail their new test. - The
C : B, AwithB : Aexample from the contract gives(B,), as documented.
Checked, no issues
- Threads: the role-bit, typed multi-line end column and de-duplicated bases tests each kill the mutant I named.
- Hygiene: all 8 commits carry a
Signed-off-by; the changelog fragment is present; the new tests write nothing outsidetmp_path.
Bot points I checked that no longer apply at this head (13)
- exai-bot's point on the exclusive end of an enclosing range does not apply:
[1, 0, 9, 0]no longer covers line 9. - exai-bot's point on direct bases does not apply as a defect: the contract and the docstring now say
bases()is a truncated ancestor graph, and theC : B, Acase is tested. - exai-bot's point on span validation does not apply: unordered and out-of-int32 spans raise
CodeIndexError. - exai-bot's second point on the exclusive end of an enclosing range does not apply: it is fixed with the half-open change.
- exai-bot's point on
)inside an escaped method disambiguator does not apply: it is parsed with the escaped-name rules. - exai-bot's point on the tenth varint byte does not apply: a tenth byte above 1 is refused.
- Copilot's point on scopes of
local Nsymbols does not apply: scopes are collected before the local filter. - Copilot's point on typed ranges (fields 8-11) does not apply: they are decoded and take precedence over the legacy fields.
- Copilot's point on the exclusive end line does not apply: the expectations in the decode tests encode the exclusive end.
- exai-bot's point on empty enclosing ranges does not apply: an empty range holds no line.
- exai-bot's point on equal line spans does not apply: the span nested by columns wins, in both orders.
- exai-bot's point on split embedded messages does not apply: typed ranges and
tool_infomerge field by field. - exai-bot's point on field numbers above 2^29-1 does not apply:
read_keyrejects them.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to dcca093, which fix the nit from my review:
dcca093:dokimi/docs/repo-evaluation-contract.md:178and thebases()docstring inindex.py: the lead now saysbases()returns the bases the index lists, and a truncated ancestor graph only on an index that lists every ancestor (scip-clang).
make ci-local at dcca093: 30 passed, 0 failed, 7 skipped.
Bring in main at ad734f5 (intel#339, a SCIP code index reader; intel#340, eval coverage against a frozen need graph). Both sides register new artifact kinds at the end of the same lists, which are in insertion order, not sorted. Each conflict keeps both sides, main's entries first: - dokimi/dokimi/artifacts.py: the module docstring names need_graph and eval_coverage and then repo_hits_judged; SCHEMA_VERSIONS lists need_graph.v1 and eval_coverage.v1, then repo_hits_judged.v1. - dokimi/dokimi/runs/kinds.py: FIELDS keeps the need_graph and eval_coverage rows, then the repo_hits_judged row. - dokimi/tests/test_runs_scan.py: EXPECTED keeps the need_graph.v1 and eval_coverage.v1 rows, then the repo_hits_judged.v1 row. The auto-merged docs/reference/cli-interface.json is what scripts/cli_surface.py dump writes for the merged tree, so it needs no change. cli_routes.py, tests/data/cli_flags_allowlist.txt, tests/test_runs_show.py and the contract keep the entries of both sides. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Bring in main at ad734f5 (intel#339, a SCIP code index reader; intel#340, eval coverage against a frozen need graph). No conflicts. CI's schema_compat compares the branch with main, which gained eval_coverage.v1 and need_graph.v1 after this branch's base. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
#349) * feat(repo): read a SCIP code index (#41) Add dokimi.repos.code_index: load a SCIP index.scip written by another tool and answer definitions, references, enclosing scopes, base classes and per-file occurrences. dokimi runs no indexer. The protobuf wire format is decoded with the standard library only, so no dependency is added. Unknown fields are skipped, so an index from a newer tool still reads. A malformed, empty or truncated file raises CodeIndexError. root_prefix maps document paths to repo paths, and a local N id is scoped to its document: the queries that take a symbol refuse it. Tests: a committed fixture built with scip-python 0.6.8 from a three-file package, cross-checked against scip print, plus hand-built bytes for unknown fields, truncation at every cut, random input and bad input. The docs gain a "Code index input" subsection (1.2). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): direct bases, one copy of repeated occurrences, spaced names (#41) Self-review of the code index reader on a scip-clang index: - bases() reduces scip-clang's ancestor lists to direct bases; - an occurrence repeated identically in one document is kept once; - definitions() looks a query up as a full symbol first, then as a name; - Occurrence gains end_line for 4-value ranges; - docs say a cut at a document boundary reads with fewer documents. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): reduce bases only on an index that lists ancestors (#41) scip-python lists direct bases, so a Python class C(B, A) with B(A) lost its direct base A. The reduction now runs only when most classes with a grandparent list every ancestor (scip-clang: 1,716 of 1,733 on oneDAL; scip-python: 0 of 38 on dokimi). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): decide lists_ancestors by tool name first, pin the vote (#41) A small scip-python index whose only deep class is C(B, A) with B(A) won the vote and lost A. scip-clang and scip-python are now decided by name; the majority vote covers other tools, with a test per branch. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Address review: typed ranges, half-open scopes, span checks, local scopes - enclosing ranges are half-open: a multiline end at column 0 leaves the line before it as the last covered line - occurrence and enclosing spans are refused when a value is outside int32 or the end comes before the start - typed range fields 8-11 are decoded, and take precedence over range and enclosing_range - a varint of ten bytes must fit in 64 bits - a local definition keeps its enclosing scope - a method disambiguator may hold an escaped name with a closing parenthesis Thread: PRRT_kwDOVBg7g86yWxm0 Thread: PRRT_kwDOVBg7g86yWxm4 Thread: PRRT_kwDOVBg7g86yWxm6 Thread: PRRT_kwDOVBg7g86yWxnA Thread: PRRT_kwDOVBg7g86yWxnC Thread: PRRT_kwDOVBg7g86yWxsJ Thread: PRRT_kwDOVBg7g86yWxsR Thread: PRRT_kwDOVBg7g86yWxsa Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(code-index): nest enclosing scopes by columns, merge split protobuf messages, bound field numbers - An empty enclosing range holds no line. Of two definitions whose enclosing line spans are equal, the one nested inside the other by columns is the inner scope. - A repeated singular typed range (single/multi line, enclosing) and a repeated Metadata.tool_info merge field by field as protobuf does: a later field replaces the earlier one, an absent field keeps it. - A field number above 2^29-1 is refused with CodeIndexError. Tests: red on the previous head for each of the three, green here. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(code-index): type the enclosing-scope best span for mypy Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(code-index): bases() is a truncated ancestor graph; test the review cases bases() reduces scip-clang's listed ancestors to a minimal set that still reaches each ancestor, so it is not the declared direct bases. The docstring and the repo-evaluation contract now say so, and an inherited base can be absent. Tests for the review of #339, each checked by a mutation: - a definition with extra role bits (roles 9) is still a definition; - a typed field-11 multi-line enclosing range keeps its end column; - the same symbol record in two documents gives each base once. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(repo): inspect --index adds a structure section from a SCIP index Modules (directories, split past max(30, files/20)), reference fan-in, cross-tree bridges, name ambiguity and guidance mentions per module, read with the code-index reader. Info findings for the ten kept modules with most fan-in. Part of #41. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): write the module split limit as integer division Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(repo): inspect --index review fixes: dedupe across documents, exact ambiguity, guidance tokens - SCIP reader: one path in several Document messages keeps each identical occurrence once. - structure: a name is ambiguous only when a different function of that name is defined elsewhere; the ratio is exact, so 500 of 1001 is no finding (rounding is display only). - in_guidance: the module path as a token (hyphen or extension continues it); the bare name counts only between backticks. - Tests for each case, and the module order by fan-in. - Changelog: repo inspect --index reads the index. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(repo): pin the reference fan-in of the ordered modules Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): fix the mentioned() docstring on the backtick rule Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(repo): a name defined in another module is ambiguous here too A name is ambiguous for a module when two or more symbols share it and one is defined in another module. The old check missed a name whose symbols were all defined in the module, when one of them was also defined elsewhere. The guidance-mention tests now pin the characters before a path token (word, hyphen, dot). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): dunder names are not counted, no names is no ambiguity share, root is in guidance - structure: dunder names (__init__, __repr__, ...) are neither counted nor ambiguous; most classes define them and nobody looks them up by name. - structure: a module with no function names has ambiguity null, not 0.0; the schema allows null, findings and the CLI line skip it. - structure: the root "." is in guidance when a root instruction file exists; §8.1 says the root is a tree of its own. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): fan-in counts references to functions, not type uses or definitions The structure section and the evaluation contract said a type use or a header declaration counts toward reference_fanin. The code keeps only function and method symbols, and it skips definition occurrences, so a type use is not counted and a declaration counts only when the indexer records it as a reference. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): count tree files a code index lacks A SCIP file cut at a document boundary still reads, and the structure section then reported "1/1 index documents in the tree" with nothing missing. index.tree_files_not_in_index counts the non-test tree files under the prefix, of an indexed extension, that have no document. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): a leading ./ names the module in guidance The lookbehind rejected "/" before the path, so "./src/core", "`./src/core/`" and "(./src/core/README.md)" never counted and the module printed as not in guidance. An optional "./" is now consumed after the unchanged lookbehind; "tests/src/core" stays rejected. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): C++ operator overloads are not counted as names Dunders were skipped because most classes define them; operator==, operator= and the like are the same case in a scip-clang index, and they pushed every module of such classes past the 0.5 ambiguity bar. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(repo): pin the root module guidance through inspect_repo Only _row was tested: passing root_guidance=False, or bool(instructions), from inspect_repo left every test green. The root module is now checked with a root AGENTS.md, with only src/AGENTS.md, and with CLAUDE.local.md. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): a module with its own instruction file is in guidance The root rule put "." in guidance since no path names it; a module with a path-specific src/core/AGENTS.md is the same case and printed as not in guidance. analyze now takes the directories that have an instruction file of their own; a local-private file does not count. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>
Summary
Adds
dokimi.repos.code_index, an internal reader for SCIPindex.scipfiles that another tool wrote. It answers definitions, references, enclosing scopes, base classes and per-file occurrences. dokimi runs no indexer, and nothing calls the reader yet.CodeIndexError(new, inrepos/errors.py). A file cut exactly at a document boundary still reads, with fewer documents; the format cannot tell.bases()gives direct bases. scip-clang lists the whole ancestor chain; on such an index the reader drops a listed base that is a base of another listed one.lists_ancestorsis decided by the tool name for scip-clang and scip-python; for another tool, by a majority of the classes that have a grandparent (the data agrees with the names: 1,716 of 1,733 such classes on oneDAL list the full chain, 0 of 38 on dokimi's scip-python index). The vote is a fallback for indexers not tested here (scip-java, scip-typescript); on a small index with few deep classes it can guess wrong.definitions(query)looks a query up as a full symbol first, then as a name, so names with spaces (operator new, template specialisations) are reachable.root_prefixmaps document paths (relative to the indexer's directory) to repo paths.local Nid is scoped to its document, sodefinitions,referencesandbasesrefuse it. Read it withoccurrences(path).docs/repo-evaluation-contract.md.Part of #41.
Evidence
Decoder cross-checked against
scip print --json, occurrence by occurrence:dokimi/)Decode cost on a real large index (oneDAL, scip-clang 0.4.0, 67.9 MB; no enclosing ranges, as that tool writes none):
Decode takes about 5.6 s on an idle host; dropping the repeats adds about 10%. On the oneDAL index,
bases(PredictionContainerIface)gives the one base inprediction.h:42(AlgorithmContainerImpl), where the raw relationships list four ancestors.Tests (92 new, all passing): the committed fixture, hand-built bytes covering unknown fields at every level, packed vs unpacked and 3- vs 4-value ranges, repeated occurrences, a three-level ancestor chain, length prefixes one byte past the end, every truncation of a valid index, 500 seeded random inputs, bad-input messages,
local Nscoping, and symbol-name parsing.make ci-localpasses locally (mypy clean, comment blocks, docs, fragment, test-quality audit).Open question for reviewers:
definitions(name)also returns parameters and module metas (self,__init__:). Kept for now; the descriptor kind can be exposed when the first caller (repo inspectstructure) needs to filter.Merge Danger
repos.code_indexyet. The only shared edit is one new class inrepos/errors.py.sample.scipis a binary file (5 KB). It was built with scip-python 0.6.8 from the three.pyfiles beside it. Rebuild it only with the same tool version, or the decode tests will need new expected values.🤖 Generated with Claude Code