Skip to content

Refactor Location tracking to be an offset-based Locatable model - #1845

Open
TwitchBronBron wants to merge 32 commits into
v1from
flatten-locations
Open

TwitchBronBron wants to merge 32 commits into
v1from
flatten-locations

Conversation

@TwitchBronBron

@TwitchBronBron TwitchBronBron commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Implements #1843.

Tokens and AST nodes now carry their position as startPosition / endPosition / sourceInfo (a SourceInfo shared by everything from the same parse) instead of a location: Location object tree. util.getLocation(item) builds a line/character Location on demand.

Compatibility: location and range stay on tokens and nodes as deprecated accessors, computed on every read. With #1928 they can also be assigned, Token declares location again, and the create* functions accept a Location or a Locatable. bslint v1 compiles and runs unchanged against it. Diagnostics accept a Locatable and resolve it to a Location only when diagnostics are read.

Review commit by commit

  1. Types and producers. Adds Locatable / SourceInfo. The lexer emits offsets and builds linePositions as it scans. Node constructors use util.setBounds() instead of createBoundingLocation(). Creators copy position from an existing Locatable.
  2. Readers. Mostly x.location → util.getLocation(x).
  3. Transpile and sourcemaps. TranspileState handles both shapes, since XML tokens still use location.
  4. Tests. Adds a testLocatable() helper for hand-built parser tokens.

Phases 1–3 don't compile on their own. Later commits are review fixes and the stacked follow-ups (#1865–#1881).

Notes

  • The XML/SG side (SGToken, SGElement) is unchanged and still uses Location.
  • An end position maps to the line of the last character, so a token ending in a newline keeps its old (L, c+1) end.
  • Node bounds are computed once in the constructor. util.setBounds() takes arrays for large lists (spreading 100k+ statements overflowed the stack).
  • finalizeClone no longer copies the original's location; clones rebuild it from their cloned tokens.
  • Removed: util.createBoundingLocationFromTokens(). TypeChainEntry's location option is now locatable.
  • Not included: the location-override WeakMap, the sourcemap last-hit cache, and moving position lookups like getTokenAt to offset comparisons. These can be follow-ups.
  • No heap or perf benchmarks have been run yet.

🤖 Generated with Claude Code

TwitchBronBron and others added 4 commits September 26, 2026 14:30
Tokens and AST nodes now implement `Locatable` directly (`pos`, `end`,
`source`) instead of carrying a `location: Location` object tree.

- `Locatable` + `SourceInfo` types in Token.ts
- Lexer emits offsets and builds `lineStarts` in one pass (drops line/column tracking)
- AstNode declares abstract `pos`/`end`/`source`; removes `location` and `range`
- node constructors use `util.setBounds()` instead of `createBoundingLocation()`
- `util.getLocation()` builds a `Location` on demand
- creators take a `Locatable` to copy from instead of a `Location`

Consumers are updated in the next commit, so this one does not compile on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Everything that read `.location` off a token or AST node (diagnostics, LSP
processors, validators, BrsFile lookups, etc.) now calls `util.getLocation()`
to build the `Location` at that boundary. Comparisons still happen on the
resulting line/character ranges, so behavior is unchanged.

- `util.createBoundingLocation()` accepts `Locatable` items too
- `TypeChainEntry` keeps a `locatable` and builds its `location` lazily

Transpile/sourcemap consumers are updated in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `TranspileState` resolves line/character for sourcemaps with `util.getLocation()`
  (SG tokens still carry a `location`, so both shapes are supported)
- `getSource()` reads the uri from `source.uri`
- `lineage` and `transpileEndBlockToken` take locatables directly
- ternary transpile copies the `?` token's location into synthetic tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- read token/node locations through `util.getLocation()`
- hand-built parser test tokens use a new `testLocatable()` helper (fixed-width test source)
- clone tests skip `pos`/`end`/`source` the same way they used to skip `range`
  (`source` is intentionally shared between an original and its clone)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@markwpearce markwpearce added this to the v1.0.0 milestone Sep 26, 2026
@markwpearce

Copy link
Copy Markdown
Collaborator

Can younpm run benchmark:scenarios do get some data on those projects ? feel free to add any local to you projects too in benchmarks/scenarios/projects.local.json (it's gitignored,
see projects.local.example.json):

  • name: "big-app"
  • path:
  • bsconfig: the bsconfig I normally build with
  • set "install": true if plugins like bslint/rooibos need their npm deps
  • editFiles: pick a widely-used utility file for "shared" and a single
    component's script for "leaf"

@TwitchBronBron
TwitchBronBron marked this pull request as draft September 26, 2026 20:03
@markwpearce

Copy link
Copy Markdown
Collaborator

I ran the scenario benchmarks -- finding an overall ~25% heap memory reduction with this!

@markwpearce

Copy link
Copy Markdown
Collaborator

Scenario benchmarks: this branch vs v1 (bc06b96)

npm run benchmark:scenarios -- run --bsc <v1 build> local --runs 3, best of 3.

Diagnostics are identical to v1 on these four projects, and edits restore cleanly:

project load validate heap peak RSS re-validate (shared/leaf, body/api)
roku-googlephotos -7% 0% -27% (38.8 → 28.3MB) -14% +1 / 0 / -1 / -3%
jellyfin-roku -11% -3% -25% (258 → 195MB) -22% -3 / -8 / -8 / -5%
jellyrock -15% -2% -27% (337 → 245MB) -19% +1 / +2 / -7 / -6%
jellyfin-fork -14% +6% -15% (285 → 241MB) -10% -10 / -5 / +7 / -7%

Heap is consistently down and load is faster. Validate and re-validate times are within run-to-run noise.

Breaks rooibos

The two rooibos projects report different diagnostics, so I'm leaving their timings out:

  • brighterscript-game-engine: 467 → 1318 diagnostics
  • Unamed OTT App: 276 → 493 diagnostics. It also ends up with fewer files and scopes on this branch (1410 → 1367 files, 363 → 320 scopes), which I haven't traced yet.

The new diagnostics are the same on both:

  • RBS2212 General error : Cannot read properties of undefined (reading 'range')
  • Duplicate identifier '_' (1175 of them on game-engine)

The throw is in rooibos-roku 6.0.0-alpha.55, dist/lib/rooibos/TestSuiteBuilder.js:158 (createTestCases):

const lineNumber = statement.func.location.range.start.line;

Stack: createTestCases ← processClassMethod ← processClass ← addSuiteIfValid ← processFile ← RooibosPlugin.afterProvideFile.

The Duplicate identifier '_' errors come from the same throw. Rooibos tests are written as function _(), and createTestCases renames each one to a unique rooiboos_test_case_<hash>_<n>. That rename comes after line 158, so every test method keeps the name _.

Other places in rooibos that read .location.range and will hit the same problem:

  • Annotation.js:205
  • TestGroup.js:80,235
  • TestSuite.js:112
  • utils/Diagnostics.js:13-21 (every rooibos diagnostic position)
  • CodeCoverageProcessor.js (10 places, only reached with code coverage on)

So node.location / token.location going away is a breaking plugin API change. The options are to keep a lazy location getter on nodes and tokens (built via util.getLocation()) for compatibility, or to update rooibos (and probably bslint) alongside this PR.

@TwitchBronBron

Copy link
Copy Markdown
Member Author

Looking really good so far! I haven't even reviewed this yet, so will try and do that early next week. But I'm thinking we might be able to get some similar gains with similar techniques acrouss the codebase. flattening certain things that don't need to be so nested.

I'm totally happy with this being a huge breaking change, it's well worth it.

TwitchBronBron and others added 6 commits September 26, 2026 15:27
`util.getLocation()` returns a new object each call, so comparing ranges with
`===` never matched. Compare the tokens instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `util.getLocation`: LF/CRLF newline tokens, trailing newline + Eof, surrogate pairs,
  multi-line template strings, split `elseif`/`exitwhile` tokens, multi-line nodes
- `util.setBounds`: min/max, mixed sources, no-source items
- `findChildAtPosition`: inclusive end, column 0 after a newline, surrogate pairs, LF and CRLF

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An end offset at a line start was always reported on the previous line, which
is only right for tokens that end with a newline. Derived ranges like an empty
block (end of `)` to start of `end sub`) now end at `(nextLine, 0)` as they
did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests for hover, definition, references, document/workspace symbols, semantic tokens,
selection ranges, completions, signature help, inlay hints, code actions, diagnostics
and sourcemaps, covering CRLF, surrogate pairs, multi-line template strings, column 0,
end of identifiers, last line without a trailing newline, and empty blocks.

They only use the public Position/Location APIs, and all of them also pass against v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, faster lookups

- Node bounds no longer end at the start of the next line when their last piece is a newline (fixes a bare `print` pulling the next line's comment onto it)
- Compute every node's bounds once at construction (Body, Block, annotations, literals included); empty blocks get their bounds from the parent's constructor via `Block.setBoundsBetween()`
- Offset-based token/expression lookups in BrsFile, start-only position lookups for sourcemaps, and fewer redundant getLocation() calls
- Add `util.setLocation()`, `getContentEnd()`, `getStartPosition()` and `getOffset()`; document the location model in docs/plugins.md
- Smaller fixes: ImportStatement clone double-shrink, mixed-source block bounds, duplicate-function check compares callables

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Construction-time bounds, getLocation() call hoisting, lazy shadowing ranges, and the ImportStatement clone fix move to their own PRs to keep this one focused on the location change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
# Conflicts:
#	src/bscPlugin/codeActions/CodeActionsProcessor.ts
#	src/parser/Statement.ts
#	src/parser/TranspileState.ts
#	src/parser/tests/statement/PrintStatement.spec.ts
@markwpearce

markwpearce commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Consider adding getters and setters for location & range - allows this not to be so huge a breaking change.

AstNode

/**
 * @deprecated use `util.getLocation(node)`. Computed on every read
 */
public get location(): Location | undefined {
    return util.getLocation(this);
}
public set location(value: Location | undefined) {
    util.setLocation(this, value);
}

/**
 * @deprecated use `util.getLocation(node)?.range`
 */
public get range(): Range | undefined {
    return util.getLocation(this)?.range;
}

Token.ts

const tokenPrototype = {
    get location() { return util.getLocation(this); },
    set location(value) { util.setLocation(this, value); }
};
export function createTokenObject(): Token {
    return Object.create(tokenPrototype);
}

--- I'll just make a PR on flatten-locations

TwitchBronBron and others added 3 commits October 6, 2026 10:09
Reword the Locatable, AstNode and plugin docs to match how LSP describes a Position (a gap between characters, like an editor cursor), instead of character indexes with an "exclusive" end.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…1913)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…1918)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@TwitchBronBron TwitchBronBron changed the title Flatten token/node locations into pos/end/source Refactor token and AST location architecture to an offset-based Locatable model Oct 15, 2026
TwitchBronBron and others added 10 commits October 15, 2026 15:48
# Conflicts:
#	src/bscPlugin/validation/BrsFileValidator.ts
#	src/parser/AstNode.spec.ts
#	src/parser/Parser.ts
#	src/parser/Statement.ts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bronley Plumb <bronley@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bronley Plumb <bronley@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bronley Plumb <bronley@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@TwitchBronBron TwitchBronBron changed the title Refactor token and AST location architecture to an offset-based Locatable model Refactor Location tracking to be an offset-based Locatable model Oct 16, 2026
markwpearce and others added 4 commits October 16, 2026 12:23
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bronley Plumb <bronley@gmail.com>
The literal and goto-label limit checks from v1 used token.location, which no longer exists; they now keep Locatables like the other validators.

Block, Body, call args, and array/AA literals spread their children into util.setBounds, which overflowed the call stack for huge lists (the new 131k-statement literal-limit test). setBounds now also takes arrays and walks them without spreading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TwitchBronBron and others added 2 commits October 16, 2026 14:47
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…gram

It was missing from the list of upgraded events, so plugins using it were never called and got no warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@markwpearce
markwpearce marked this pull request as ready for review October 16, 2026 19:37
@markwpearce markwpearce added the create-package create a temporary npm package on every commit label Oct 16, 2026
@rokucommunity-bot

Copy link
Copy Markdown
Contributor

Hey there! I just built a new temporary npm package based on 3d97ee7. You can download it here or install it by running the following command:

npm install https://github.com/rokucommunity/brighterscript/releases/download/v0.0.0-packages/brighterscript-1.0.0-alpha.57-flatten-locations.20261016193811.tgz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

create-package create a temporary npm package on every commit

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants