[exKAZUu]

Why

Valid code such as if True:\n print(1 +\n1) produces errors and detaches following statements from its suite. This addresses tree-sitter/tree-sitter-python#256. Python permits arbitrary indentation on implicit continuation lines inside brackets.

The scanner guesses whether it is inside brackets by checking whether a closing token is currently valid. After an operator, the parser needs an operand first, so no closing token is valid and the scanner incorrectly forces a dedent. Removing forced dedents also accepts invalid unbracketed continuations; tracking the actual bracket context preserves that recovery behavior.

Requirements

• required: Improve our forked tree-sitter and grammar repositories as much as possible, ignoring backward compatibility; review our issues and major upstream issues, PRs and recent commits and create dedicated nonduplicate PRs from latest main for eligible items.

• required: Fix the reproduced valid dedented bracket-continuation case while keeping meaningful public corpus, query and edit validation; run verification and the requested review-booster, complete-pr and release workflow.

• required: Run review-booster and complete-pr concurrently, then release the changes.

• chosen: Track actual opening and closing punctuation in serialized scanner state instead of guessing from possible closing tokens. Preserve forced dedent recovery outside brackets because removing it newly accepts incomplete assignments and unbracketed expressions across a dedent.

• chosen: Preserve existing accepted node trees, fields, ranges and following-statement ownership; this lexical correction needs no new named nodes or runtime ABI change.

• chosen: Preserve the existing 510-level indentation capacity at every bracket depth using a fixed 32-bit bracket counter and variable-length increasing indentation gaps. A 510-suite/127-bracket reproduction showed that adding extended-depth bytes before fixed-width indents truncates state; the gap encoding fits every stack representable by the prior format within 902 of 1024 bytes.

• chosen: Scan format-specifier content as a hidden external token before layout, since its whitespace is meaningful payload. An explicit context token preserves the existing format field text/ranges without relying on a heuristic over closing-token validity.

Customer Summary

Expressions inside parentheses, square brackets and braces can continue on a less-indented line without ending their surrounding suite. Incomplete unbracketed expressions still report errors, and following statements retain their scope.

Technical Summary

grammar.js exposes opening and closing punctuation to the scanner using the existing anonymous token symbols. src/scanner.c tracks bracket depth, includes it in serialization, and clears abandoned bracket context when recovery emits a dedent. F-string brace paths update the same state. Punctuation is not emitted across skipped explicit line continuations, allowing the internal lexer to preserve their named syntax nodes. A hidden format-content token preserves spaces, tabs and carriage returns before nested or closing format braces; it replaces the internal regular expression without introducing visible nodes. Canonical generated sources carry the external-token changes.

The public incremental regression removes and restores continuation indentation for calls, groups, subscripts, lists, sets and dictionaries, 129 nested brackets and indentation widths at the gap-encoding boundaries. It checks suite ownership, standard query captures, exact operand ranges and fresh/incremental agreement. Negative cases ensure unbracketed incomplete statements remain errors and preserve the following top-level statement. Additional LF/CRLF query-and-edit coverage preserves explicit line-continuation nodes and ranges before opening and closing punctuation. The deep-indentation regression combines 510 suites with 127 nested brackets. Public format-specifier queries retain whitespace text/ranges and agree after edits for nested braces, tabs and LF/CRLF.

Testing

• The new public incremental test and corpus case fail against the prior parser; the original published 6.0.0 parser also rejects the issue's source.
• Canonical bun run build/ci; all 210 native and Wasm corpus cases pass.
• Twelve relevant test files (20 tests) pass, covering corpus edit fuzzing, f-string formats, type/field queries, deep indentation, error-recovery performance and artifact freshness.
• cargo test --locked and bun run verify pass.
• Differential checks against published 6.0.0 cover 2,020 CPython-checked sources across seven suite contexts, indentation styles, LF/CRLF, comments and Unicode identifiers: 1,290 previously rejected valid cases are fixed; all 728 already-accepted trees, fields and ranges are unchanged; no new invalid acceptance or lost following call occurs. All 12 tracked real-world fixture trees, fields and ranges are unchanged. An additional 170 format-content cases preserve accepted trees and ranges across quoting/prefix styles, Unicode whitespace, and literal NUL versus EOF.
• The exact pinned CPython test_compile.py compiles and now parses without errors; the published parser rejects its dedented attribute continuations. Its obsolete expected failure is removed. PR CI parses all 5,901 external files with three expected failures, matching the updated list. All PR checks pass, including external examples, browser/Workers, fuzzing, sanitizers, Rust, Linux ARM and both macOS architectures.

Notes

The existing scanner dispatch exceeds code-gauge limits (complexity 184, 211 NCSS; previously 175/202). Reducing it below the limits would require a separate string/layout scanner redesign or mechanically move the same complexity into helpers. The six other existing errors are imported fixtures: Python2 GrammarTests.testFuncdef in both newline variants (159 NCSS), Python3 equivalents (187 NCSS), and the Python3.8 example file (1,200 NCSS) and its test_funcdef (232 NCSS). Forty-six existing warnings remain outside changed code. Intentional LF/CRLF fixture duplicates would be suitable for a code-gauge exclusion; no exclusion configuration is added.