[exKAZUu]

Why

The grammar silently accepts physical newlines that CPython rejects, including def function(value)\n:\n return value and value = 1 +\n 2. Global whitespace consumed those newlines before they could terminate or invalidate a logical line. This addresses tree-sitter/tree-sitter-python#178.

Requirements

• required: Improve our forked tree-sitter repositories as much as possible, ignoring backward compatibility.
• required: Check existing PRs and integrate latest main; do not create new PRs. Complete and merge only PRs already created in this session.
• required: Reject unjoined physical newlines while preserving valid implicit and explicit continuation, multiline literals, comments, indentation, and following statements.
• required: Validate actual CPython acceptance and rejection, native and public Wasm parsing, and incremental edit and undo.
• required: Preserve the existing exact source ranges of following class and function declarations; keep their assertions unchanged.
• required: Preserve existing line_continuation nodes and ranges for valid inputs; correct only the proven invalid EOF corpus expectation to ERROR, preserving all input bytes and valid trees.
• required: Run review-booster and complete-pr concurrently, then release the changes.
• chosen: Treat LF, CRLF, and bare CR as physical line endings: CPython accepts them, and the new CR boundary path must preserve comments and following statements rather than changing their ownership or swallowing them.
• chosen: Track explicit joins separately from logical starts. Reject incomplete expressions followed by unjoined blank/comment lines, including its pre-existing gap. Look through joins to the actual suite body and remember joins before the first unjoined ending: stopping at column-zero joins detaches bodies or closes suites before joined statements. Emit visible joins separately before comment lookahead delegates to the internal lexer; otherwise mixed CR/LF loses context. Reject an explicit join if EOF follows its whole LF/CRLF/CR ending: CPython rejects them (LF/CRLF acceptance pre-exists, CR is introduced). Blank/comment/horizontal following lines remain valid, with join ranges unchanged. Keep the token external-only: internal fallback bypasses EOF rejection.
• chosen: Consume contiguous ignored endings together and query columns only when line-start context is needed, because repeated suffix scans amplified a 4,008-character input to 8,003,216 characters requested on the reviewed head.
• chosen: Emit a logical newline during recovery only after a real physical ending, and retire queued indentation levels without emitting a token the grammar cannot accept. Native traces showed a malformed header repeatedly resuming the condition-expression state, then an unexpected EOF dedent wrapping independent statements in an error; preserving that behavior would violate the following-statement contract.
• chosen: Keep syntax boundaries zero-width and let a separate hidden scanner extra consume physical line endings, because grammar-wide whitespace cannot distinguish physical and logical lines and the declarations must exclude their trailing newline. A consuming logical-boundary token was rejected because it broke six existing exact-range CI assertions.
• chosen: Retain get_column and require published runtime 14.0.0 with matching ABI10 generation: core cache repairs tracked through WillBooster/tree-sitter#255 make the unchanged seven-source 16x public bound pass for comments, brackets, statements, Unicode, BOM, and joins. Released 12.0.2 still requests 421,018 characters against a 384,128 bound for CR statements; actual14 scales 1k/2k/4k/8k statements to 11,790/23,838/47,932/96,120 characters. Absolute-position serialization was rejected because it increased prefix-edit work and depended on a separately defective getter. Keep the publication block if expanded regressions fail.
• chosen: Skip line-start classification inside brackets and after known comment-only endings: LF bracket comments otherwise request 1,276,799 characters versus base 75,261; the narrow guard reduces this to 63,016. Keep zero-width boundaries and separate ending consumption to preserve declaration endpoints. Alternating same-process actual14/matching-generator controls measure 654–667 ms versus main 633–644 ms for 400 stdlib files, 196–198 versus 173–177 ms for body comments, and 79–84 versus 66–67 ms for bracket comments. These residual costs are measured, not throughput parity; the actual released dependency and unchanged bound now pass.
• chosen: Reject unescaped physical endings in non-triple string content and format text, including empty content, while retaining escaped joins. Bare-CR acceptance is introduced by this branch; LF/CRLF single-line format acceptance predates it but also violates the requested newline rejection. CPython syntax checks require odd-backslash joins to remain valid and even-backslash endings to fail. Consuming an escaped slash or quote must not also consume the following physical ending: 12 LF/CRLF raw parity cases already fail at the base, while this branch introduces six additional accepted CR cases; make those escape alternatives exclusive without changing valid raw triple content or quote escapes. Keep the existing triple-format scanning and capture ranges, because those valid multiline captures are an asserted caller contract. Empty-content error recovery keeps the existing LF shape, exposing the following call outside string content even when recovery attaches it to the malformed assignment.
• chosen: Retain the explicitly declared FEFF/U+2060/U+200B extras as tolerant horizontal trivia rather than remove an unrelated grammar extension. CPython rejects these mid-file characters; validate the normalized program separately instead of claiming CPython acceptance. Actual native UTF8 and public UTF16 lexer probes confirm one get_column unit per character, despite three UTF8 bytes; count that input width with zero indentation width. Recognize them consistently in indentation, ignored-ending, and join lookahead, while rejecting real unjoined endings and preserving literal/comment contents. The base accepts these lines but can already detach following suite calls; normalized syntax and suite ownership govern this repair.

Customer Summary

Unjoined newlines before a header's colon or inside an unbracketed expression produce syntax errors. Valid continuations, multiline strings, comments, indentation, and following statements retain their structure and source ranges. An explicit join requires a following physical line. Consumers must use the WillBooster runtime 14.x.

Technical Summary

The grammar restricts horizontal whitespace and gives the scanner ownership of physical endings. Logical boundaries and queued dedents remain zero-width; a separate hidden extra consumes the physical ending so declarations exclude their trailing newline. Serialized join context preserves visible continuation tokens, suite ownership, and terminal declaration endpoints. Recovery emits logical boundaries only after real physical endings and retires indentation levels that the grammar cannot consume. Comment handling stops at either ending character and avoids unnecessary line-start classification inside brackets or after a known comment-only line.

The continuation token is scanner-owned only: an internal token previously bypassed scanner rejection. A complete LF/CRLF/CR ending followed immediately by EOF is rejected, while a following blank/comment/horizontal line remains valid.

Non-triple strings and format text reject unescaped physical endings, including empty content. Raw quote/slash escapes and escaped endings are exclusive; valid odd-slash joins and triple-format captures retain their source ranges. Explicit FEFF/U+2060/U+200B extras remain tolerant horizontal trivia with one input-column unit and zero indentation width.

Runtime dependencies, Rust documentation, and README usage require the actual published 14.0.0 release. Canonical ABI10 parser tables use its matching published CLI, the 210 unchanged corpus input bodies, and 12 tracked profile inputs. This includes the core column-cache repairs without storing absolute source positions in the Python scanner.

Testing

• bun run build/ci: matching published 14.0.0 CLI, optimization z, canonical corpus/source profile; passes.
• CARGO_BUILD_JOBS=1 cargo test --locked --lib: actual published crate loads the generated grammar; passes.
• bun run verify: passes; remaining scanner code-gauge findings are explained below.
• Twenty-nine focused tests across thirteen files pass, including all 210 corpus cases (one corrected invalid EOF expectation), the six unchanged exact declaration-range assertions, visible continuation ranges, string/format captures, deep indentation, recovery, and incremental edit/undo. The unchanged seven-source 16x public input-read bound now passes for LF blanks, CR comments, Unicode, BOM, joins, brackets, and statements. The same CR-statement test remains preserved as RED against 12.0.2 (421,018 characters versus a 384,128 bound); on 14.0.0, 1k/2k/4k/8k CR statements request 11,790/23,838/47,932/96,120 characters with no errors.
• Fifty-four actual CPython/native/public EOF controls agree: nine terminal joins are invalid and 45 joins followed by blank/comment/horizontal lines remain valid. Eighteen insertion/undo steps agree with fresh parsing; valid visible continuation text and indices remain unchanged.
• Actual CPython, native CLI, and public Wasm agree on 540 string-boundary cases (252 valid), with every following call preserved outside string content. Raw slash parity, escaped quotes, ordinary/bytes/formatted/raw/triple literals, and LF/CRLF/CR are covered. Existing exact triple-format assertions remain unchanged. String insertion/undo compares fresh syntax, errors, text, indices, and points.
• Native/public layout controls cover 120 cases and 36 native XML suite-ownership checks. Their normalized programs match CPython (96 valid, 24 invalid); the original mid-file Unicode extras are deliberately beyond CPython syntax. Existing 72 layout insertion/undo steps preserve suite/token text, indices, points, and endpoints. Native UTF8/public UTF16 probes establish each extra's input width.
• Fourteen prefix insertion/undo controls pass with the actual 14.0.0 runtime; three included-range outputs exactly match the prior candidate's nodes and coordinates. Existing physical-newline and continuation edit/undo tests retain function/class/try suite ownership and following-call coordinates across all three endings.
• On runtime-integration snapshot 92f207e, same-process alternating controls use its source and latest-main 686abce grammar generated/profiled with the same actual14 CLI and runtime. LF function/body-comment requests are 145,744/227,257 versus main 145,748/227,261; bracket-comment requests are 63,016 versus 75,261. CR bracket requests scale linearly (500/1k/2k: 5,032/10,052/20,092 versus main 4,773/9,793/19,833). Main's CR-statement parses have errors, so their linear counts are not valid-source parity evidence. Four timing rounds over 400 stdlib files report no errors: candidate 654–667 ms versus main 633–644 ms; 20k functions 171–173 versus 152–155 ms; body comments 196–198 versus 173–177 ms; bracket comments 79–84 versus 66–67 ms.
• Earlier published grammar 9.0.0 controls establish that the new physical-newline acceptance/edit regressions fail before this repair, and preserve meaningful LF continuation source-range comparisons. The old ABI07 grammar is not loaded into the ABI10 runtime.
• The local focused command excludes only the existing broad reparses the corpus consistently fuzz test; its full-suite CI execution remains enabled. No corpus expectation or existing source-range assertion was weakened.

Notes

The scanner still reports cognitive complexity 246, function NCSS 288, and file NCSS 579. Reaching those thresholds would require restructuring unrelated string scanning and splitting the scanner translation unit; moving branches between helpers would only relocate complexity. No gauge exclusion was added.

Zero-width boundaries and separate ending consumption preserve the existing declaration endpoint contract at the measured residual parsing cost above; no exact throughput parity is claimed. The initial cache integration was blocked until the expanded public regression passed against an actual published dependency. The retained 12.0.2 snapshot documents its residual keyword-lookahead replay failure; 14.0.0 is verified with the unchanged stronger bound and edit/range controls.

Blank/comment-only lines after explicit joins, bare-CR continuations, LF/CRLF single-line formats, and twelve raw slash-parity cases already failed the intended contract at the original base. Their independent repairs complete the recorded newline requirement and are not counted as branch-introduced review defects. Bare-CR string acceptance was introduced by the branch. Empty-content unterminated strings retain the base LF recovery shape: a following call survives outside string content but can belong to the malformed assignment during recovery.

FEFF/U+2060/U+200B are explicitly declared original grammar extras. Retaining them consistently is an intentional tolerant syntax policy, not a claim of CPython acceptance. Since the base can already detach suite calls after extra-prefixed indentation, normalized syntax and explicit suite ownership govern those controls.

The Function definitions corpus input ends in a terminal explicit join that CPython rejects. Its final expected line_continuation is corrected to ERROR/UNEXPECTED backslash, preserving every input byte and all valid trees. The whole legacy fixture also contains a Python 2 tuple parameter that CPython 3 rejects earlier; an isolated terminal function and the independent EOF matrix establish the EOF error separately. LF/CRLF EOF acceptance pre-exists, while bare-CR EOF acceptance is introduced by this branch; the unified repair follows the recorded newline contract.