Repository navigation
Refactor Location tracking to be an offset-based Locatable model - #1845
TwitchBronBron wants to merge 32 commits into
Conversation
Tokens and AST nodes now implement `Locatable` directly (`pos`, `end`, `source`) instead of carrying a `location: Location` object tree. - `Locatable` + `SourceInfo` types in Token.ts - Lexer emits offsets and builds `lineStarts` in one pass (drops line/column tracking) - AstNode declares abstract `pos`/`end`/`source`; removes `location` and `range` - node constructors use `util.setBounds()` instead of `createBoundingLocation()` - `util.getLocation()` builds a `Location` on demand - creators take a `Locatable` to copy from instead of a `Location` Consumers are updated in the next commit, so this one does not compile on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Everything that read `.location` off a token or AST node (diagnostics, LSP processors, validators, BrsFile lookups, etc.) now calls `util.getLocation()` to build the `Location` at that boundary. Comparisons still happen on the resulting line/character ranges, so behavior is unchanged. - `util.createBoundingLocation()` accepts `Locatable` items too - `TypeChainEntry` keeps a `locatable` and builds its `location` lazily Transpile/sourcemap consumers are updated in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `TranspileState` resolves line/character for sourcemaps with `util.getLocation()` (SG tokens still carry a `location`, so both shapes are supported) - `getSource()` reads the uri from `source.uri` - `lineage` and `transpileEndBlockToken` take locatables directly - ternary transpile copies the `?` token's location into synthetic tokens Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- read token/node locations through `util.getLocation()` - hand-built parser test tokens use a new `testLocatable()` helper (fixed-width test source) - clone tests skip `pos`/`end`/`source` the same way they used to skip `range` (`source` is intentionally shared between an original and its clone) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Can you
|
|
I ran the scenario benchmarks -- finding an overall ~25% heap memory reduction with this! |
Scenario benchmarks: this branch vs
|
| project | load | validate | heap | peak RSS | re-validate (shared/leaf, body/api) |
|---|---|---|---|---|---|
| roku-googlephotos | -7% | 0% | -27% (38.8 → 28.3MB) | -14% | +1 / 0 / -1 / -3% |
| jellyfin-roku | -11% | -3% | -25% (258 → 195MB) | -22% | -3 / -8 / -8 / -5% |
| jellyrock | -15% | -2% | -27% (337 → 245MB) | -19% | +1 / +2 / -7 / -6% |
| jellyfin-fork | -14% | +6% | -15% (285 → 241MB) | -10% | -10 / -5 / +7 / -7% |
Heap is consistently down and load is faster. Validate and re-validate times are within run-to-run noise.
Breaks rooibos
The two rooibos projects report different diagnostics, so I'm leaving their timings out:
- brighterscript-game-engine: 467 → 1318 diagnostics
- Unamed OTT App: 276 → 493 diagnostics. It also ends up with fewer files and scopes on this branch (1410 → 1367 files, 363 → 320 scopes), which I haven't traced yet.
The new diagnostics are the same on both:
RBS2212 General error : Cannot read properties of undefined (reading 'range')Duplicate identifier '_'(1175 of them on game-engine)
The throw is in rooibos-roku 6.0.0-alpha.55, dist/lib/rooibos/TestSuiteBuilder.js:158 (createTestCases):
const lineNumber = statement.func.location.range.start.line;Stack: createTestCases ← processClassMethod ← processClass ← addSuiteIfValid ← processFile ← RooibosPlugin.afterProvideFile.
The Duplicate identifier '_' errors come from the same throw. Rooibos tests are written as function _(), and createTestCases renames each one to a unique rooiboos_test_case_<hash>_<n>. That rename comes after line 158, so every test method keeps the name _.
Other places in rooibos that read .location.range and will hit the same problem:
Annotation.js:205TestGroup.js:80,235TestSuite.js:112utils/Diagnostics.js:13-21(every rooibos diagnostic position)CodeCoverageProcessor.js(10 places, only reached with code coverage on)
So node.location / token.location going away is a breaking plugin API change. The options are to keep a lazy location getter on nodes and tokens (built via util.getLocation()) for compatibility, or to update rooibos (and probably bslint) alongside this PR.
|
Looking really good so far! I haven't even reviewed this yet, so will try and do that early next week. But I'm thinking we might be able to get some similar gains with similar techniques acrouss the codebase. flattening certain things that don't need to be so nested. I'm totally happy with this being a huge breaking change, it's well worth it. |
`util.getLocation()` returns a new object each call, so comparing ranges with `===` never matched. Compare the tokens instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `util.getLocation`: LF/CRLF newline tokens, trailing newline + Eof, surrogate pairs, multi-line template strings, split `elseif`/`exitwhile` tokens, multi-line nodes - `util.setBounds`: min/max, mixed sources, no-source items - `findChildAtPosition`: inclusive end, column 0 after a newline, surrogate pairs, LF and CRLF Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An end offset at a line start was always reported on the previous line, which is only right for tokens that end with a newline. Derived ranges like an empty block (end of `)` to start of `end sub`) now end at `(nextLine, 0)` as they did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests for hover, definition, references, document/workspace symbols, semantic tokens, selection ranges, completions, signature help, inlay hints, code actions, diagnostics and sourcemaps, covering CRLF, surrogate pairs, multi-line template strings, column 0, end of identifiers, last line without a trailing newline, and empty blocks. They only use the public Position/Location APIs, and all of them also pass against v1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, faster lookups - Node bounds no longer end at the start of the next line when their last piece is a newline (fixes a bare `print` pulling the next line's comment onto it) - Compute every node's bounds once at construction (Body, Block, annotations, literals included); empty blocks get their bounds from the parent's constructor via `Block.setBoundsBetween()` - Offset-based token/expression lookups in BrsFile, start-only position lookups for sourcemaps, and fewer redundant getLocation() calls - Add `util.setLocation()`, `getContentEnd()`, `getStartPosition()` and `getOffset()`; document the location model in docs/plugins.md - Smaller fixes: ImportStatement clone double-shrink, mixed-source block bounds, duplicate-function check compares callables Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Construction-time bounds, getLocation() call hoisting, lazy shadowing ranges, and the ImportStatement clone fix move to their own PRs to keep this one focused on the location change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
# Conflicts: # src/bscPlugin/codeActions/CodeActionsProcessor.ts # src/parser/Statement.ts # src/parser/TranspileState.ts # src/parser/tests/statement/PrintStatement.spec.ts
|
Consider adding getters and setters for AstNode /**
* @deprecated use `util.getLocation(node)`. Computed on every read
*/
public get location(): Location | undefined {
return util.getLocation(this);
}
public set location(value: Location | undefined) {
util.setLocation(this, value);
}
/**
* @deprecated use `util.getLocation(node)?.range`
*/
public get range(): Range | undefined {
return util.getLocation(this)?.range;
}Token.ts const tokenPrototype = {
get location() { return util.getLocation(this); },
set location(value) { util.setLocation(this, value); }
};
export function createTokenObject(): Token {
return Object.create(tokenPrototype);
}--- I'll just make a PR on |
5d45475 to
af08c66
Compare
Reword the Locatable, AstNode and plugin docs to match how LSP describes a Position (a gap between characters, like an editor cursor), instead of character indexes with an "exclusive" end. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…1913) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…1918) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/bscPlugin/validation/BrsFileValidator.ts # src/parser/AstNode.spec.ts # src/parser/Parser.ts # src/parser/Statement.ts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Bronley Plumb <bronley@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Bronley Plumb <bronley@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Bronley Plumb <bronley@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Bronley Plumb <bronley@gmail.com>
The literal and goto-label limit checks from v1 used token.location, which no longer exists; they now keep Locatables like the other validators. Block, Body, call args, and array/AA literals spread their children into util.setBounds, which overflowed the call stack for huge lists (the new 131k-statement literal-limit test). setBounds now also takes arrays and walks them without spreading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/parser/Parser.ts
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…gram It was missing from the list of upgraded events, so plugins using it were never called and got no warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hey there! I just built a new temporary npm package based on 3d97ee7. You can download it here or install it by running the following command: npm install https://github.com/rokucommunity/brighterscript/releases/download/v0.0.0-packages/brighterscript-1.0.0-alpha.57-flatten-locations.20261016193811.tgz |
Implements #1843.
Tokens and AST nodes now carry their position as
startPosition/endPosition/sourceInfo(aSourceInfoshared by everything from the same parse) instead of alocation: Locationobject tree.util.getLocation(item)builds a line/characterLocationon demand.Compatibility:
locationandrangestay on tokens and nodes as deprecated accessors, computed on every read. With #1928 they can also be assigned,Tokendeclareslocationagain, and thecreate*functions accept aLocationor aLocatable. bslint v1 compiles and runs unchanged against it. Diagnostics accept aLocatableand resolve it to aLocationonly when diagnostics are read.Review commit by commit
Locatable/SourceInfo. The lexer emits offsets and buildslinePositionsas it scans. Node constructors useutil.setBounds()instead ofcreateBoundingLocation(). Creators copy position from an existingLocatable.x.location→util.getLocation(x).TranspileStatehandles both shapes, since XML tokens still uselocation.testLocatable()helper for hand-built parser tokens.Phases 1–3 don't compile on their own. Later commits are review fixes and the stacked follow-ups (#1865–#1881).
Notes
SGToken,SGElement) is unchanged and still usesLocation.(L, c+1)end.util.setBounds()takes arrays for large lists (spreading 100k+ statements overflowed the stack).finalizeCloneno longer copies the original's location; clones rebuild it from their cloned tokens.util.createBoundingLocationFromTokens().TypeChainEntry'slocationoption is nowlocatable.getTokenAtto offset comparisons. These can be follow-ups.🤖 Generated with Claude Code