fix(tokenizer): add the missing 'whitespace' char class and EndStatemend to the 'lower' group

[geircodes]

fix(tokenizer): add the missing 'whitespace' char class and EndStatement to the 'lower' group

Branch fix/lexer-whitespace-charclass, targets version-1.0.4.

What happens

Tokenizing a large source file is dominated by the lexer rather than the structure pass: on a 770 KB / 14,512-line module, lexing took 587.8 ms of a 633 ms full tokenize. Every retokenize while typing pays that, and every feature that needs fresh tokens waits for it.

Separately, a lowercase end at column 0 tokenizes as Variable "nd" — the leading e is dropped — while END at column 0 and an indented lowercase end both tokenize correctly as EndStatement.

Root cause

PatternMatcher.getCharClass classifies ' ' and '\t' as 'whitespace', but charClassGroups had no 'whitespace' entry. The per-position lookup in ClarionTokenizer.tokenizeLines:

const relevantTypes = PatternMatcher.getPatternsByCharClass().get(charClass) || types;

fell through undefined to types — all 33 token patterns — at every whitespace position. Whitespace is roughly a third of the characters in a Clarion source file, and the tokenizer never skips it (it advances one character when nothing matches), so instrumented on the file above, 92.5% of the 5,666,201 regex exec calls in one tokenize started at a whitespace position, and only 1.29% of all exec calls matched at index 0.

That fallback was masking a second gap. TokenType.EndStatement is listed in the 'upper' char-class group but was missing from 'lower':

'lower': [
    TokenType.Label,
    TokenType.Directive, TokenType.ClarionDocument,
    TokenType.ExecutionMarker, TokenType.ConditionalContinuation, TokenType.Structure, TokenType.Keyword,
    ...

A lowercase end only tokenized correctly when indented, because the all-patterns fallback at the preceding space matched EndStatement's /^\s*(END)\b/ alternative there. At column 0 there is no preceding space, no pattern in 'lower' accepted the word, and the single-character fallback advance consumed the e. Adding the 'whitespace' group on its own therefore regressed 9 of 241 files (every indented lowercase end), which is why both changes ship together.

The fix

server/src/tokenizer/PatternMatcher.ts:

'lower': [
    ...
    TokenType.ExecutionMarker, TokenType.ConditionalContinuation, TokenType.Structure, TokenType.Keyword,
    TokenType.EndStatement,
    ...
],
...
'whitespace': [],

An empty whitespace group is sound: nothing needs to match at a whitespace position. LineContinuation re-matches at the | itself, and the MODULE (/^\s*MODULE\b/) and TOOLBAR (/^[ \t]*TOOLBAR\b.../) structure patterns re-match at the keyword through their zero-width leading \s*.

Measured

• Lex-only on the 770 KB / 14,512-line module: 587.8 ms → ~80 ms (7x). Full tokenize 633 ms → 170 ms.
• Token streams compared as type|value|line|start for every token across 241 source files (14.3 MB, including this repository's own test fixtures): 0 differences.

Testing

New server/src/test/ClarionTokenizer.LowercaseEndCharClass.test.ts (5 tests): column-0 lowercase end is an EndStatement; column-0 END and indented lowercase end still are; EndStatement is present in both the 'upper' and 'lower' groups; a 'whitespace' group is defined so the lookup can never fall back to every pattern again.

Without the fix, 3 of the 5 fail — the column-0 lowercase end case with exactly Variable "nd", plus the two group assertions. With the fix the full suite is 3073 passing, 0 failing, 4 pending.

Scope

Only the two char-class table entries. The patterns themselves are unchanged. A separate, smaller opportunity remains and is deliberately not touched here: nearly every token pattern is unanchored and executed against line.slice(position) with the result discarded unless match.index === 0, so the engine scans to end of line before the result is thrown away; replaying the recorded call sequence with sticky (/y) patterns was 81% faster. After this change the remaining lex cost is small enough that it is not worth the semantic risk in the same PR — going sticky against the full line would activate the currently inert (?<![:\w.]) lookbehinds, which the tokenizer re-implements by hand today.

[msarson]

Merged, thanks Geir — a big win: across our 7,237-file corpus tokenizing dropped from 67 s to 19 s, and the longest event-loop block while loading the DirectSystems solution from 1.4 s to 0.9 s.

One thing the full-corpus comparison turned up: letting lower case reach the END pattern also turned access_token:end (libcurldropbox.clw) into an END statement, closing the enclosing IF early. The upper-case form already had that bug, so we fixed it first as #584 (the END pattern now skips a match after ':' or '.') and merged your PR on top; with that in place the only token changes are your column-0 lowercase end fixes in cpxml.inc and cpxmlif.inc.