Skip to content

Parse MODULE.bazel with Python's ast in the msan guard - #433

Merged
zzcgumn merged 2 commits into
developfrom
refactor/msan_guard_ast
Oct 16, 2026
Merged

zzcgumn merged 2 commits into
developfrom
refactor/msan_guard_ast

Conversation

@zzcgumn

@zzcgumn zzcgumn commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #423, which was merged with a hand-written Starlark lexer in python/tests/ci_msan_test.py. This applies the same rewrite as #424.

Why

The lexer kept drawing review findings. The latest: it joined separate statements into a fictitious call. With

dep_function = bazel_dep
("toolchains_llvm", "9.9.9")
bazel_dep(name = "toolchains_llvm", version = "1.11.2")

_toolchains_llvm_bazel_dep_version() returned 9.9.9. A one-line version bump had also grown the file by ~1400 lines.

What

Starlark's syntax is a subset of Python's, so the guard now parses MODULE.bazel with ast.parse. It walks the ast.Call nodes whose callee is a bare bazel_dep / *_override name. Python's tokenizer handles strings, escapes, raw strings, comments, continuations and statement boundaries exactly:

  • attribute calls such as ext.bazel_dep(...) are excluded;
  • grouping parens around a callable are transparent;
  • any argument that isn't a single string literal still fails closed.

The regression test for the example above was added first and confirmed failing. Tests that only exercised tokenizer details (escapes, raw-string backslash parity, comment placement, continuations) were removed. The fail-closed cases are now subtests of one test. The file goes from 1683 to 614 lines; the non-parser tests are unchanged. Full //python/... suite green (24/24).

🤖 Generated with Claude Code

Follow-up to #423, mirroring #424. Starlark's syntax is a subset of
Python's, so ast.parse handles strings, comments, continuations and
statement boundaries exactly. This fixes the scanner joining
"dep_function = bazel_dep" and a following "(...)" tuple into a
fictitious call, and replaces ~1000 lines of lexer and its
tokenizer-level tests with ~70 lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzcgumn zzcgumn self-assigned this Oct 16, 2026
@zzcgumn
zzcgumn requested a review from tameware October 16, 2026 16:50
Mirrors #424. ast records keyword unpacking as a keyword with arg None, so
local_path_override(**{"module_name": "toolchains_llvm", ...}) read as
having no module_name and slipped past the guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzcgumn

zzcgumn commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Mirrored #424's keyword-unpacking fix (6085e98): local_path_override(**{"module_name": "toolchains_llvm", ...}) slipped past the override guard because ast records ** as a keyword with arg=None. _argument() now returns _UNRESOLVED in that case, so the guard fails closed. Regression tests were added red first. Full //python/... suite green (24/24).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused rewrite addresses the parsing defect, with regression coverage and no identified blocking issues.

0 open findings

What changed in this PR

Simplifies the MemorySanitizer CI guard by replacing its hand-written Starlark lexer with Python’s ast parser.

Changes:

  • Fixes incorrect joining of separate statements into calls.
  • Preserves dependency-version and override checks.
  • Consolidates parser tests and adds statement-boundary and argument-unpacking coverage.
File Description
python/​tests/​ci_msan_test.py Replaces the lexer with AST helpers and updates regression tests.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@zzcgumn
zzcgumn merged commit 43e5d85 into develop Oct 16, 2026
16 of 17 checks passed
@zzcgumn
zzcgumn deleted the refactor/msan_guard_ast branch October 16, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants