Stop the lexer reading past the end of a byte_range - #3040
Merged
Conversation
4 tasks
ksss
force-pushed
the
ksss/lexer-end-pos
branch
2 times, most recently
from
July 28, 2026 11:06
7becc4e to
8df3c79
Compare
ksss
force-pushed
the
ksss/lexer-end-pos
branch
2 times, most recently
from
August 13, 2026 09:52
fc60d48 to
798d5b3
Compare
`rbs_next_char` ends the input when `byte_pos == end_pos`. `rbs_skip` advances by a whole character, so a multibyte character starting before `end_pos` and ending after it steps over the boundary and equality never holds again — the lexer then reads to the end of the string. The realistic way to land inside a character is to pass a character offset where a byte offset is expected, the mistake ruby#2945 fixed in `parse_inline_*_annotation`. `"日本語"` is 5 characters but 11 bytes, so offset 5 falls inside `本`: Parser.parse_type('"日本語" | Integer', byte_range: 0...5) #=> Types::Union spanning the whole input, rather than an error `require_eof: true` does not catch it, because the lexer really is at EOF by then. It needs a character to straddle the boundary, so ASCII-only input never hits it. Compare with `>=` so stepping over the boundary still ends the input.
ksss
force-pushed
the
ksss/lexer-end-pos
branch
from
August 13, 2026 09:55
798d5b3 to
7c1006a
Compare
soutaro
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rbs_next_charends the input whenbyte_pos == end_pos.rbs_skipadvances by a whole character, so a multibyte character starting beforeend_posand ending after it steps over the boundary — equality never holds again, and the lexer reads to the end of the string.The realistic way to hit this is passing a character offset where a byte offset is expected — the mistake #2945 fixed in
parse_inline_*_annotation."日本語"is 5 characters but 11 bytes, so offset 5 falls inside本:require_eof: truedoes not catch it, because by then the lexer really is at EOF. It takes a character straddling the boundary, so ASCII-only input never hits it. Now that #3082 allows non-ASCII identifiers, more inputs can reach this.Fix
Compare with
>=so stepping over the boundary still ends the input.Test plan
rake testtest_parse__byte_range_ending_mid_character, confirmed to fail with the==comparison restored.