fix(completion): rank word completion candidates by scope tier (#555)

[geircodes]

fix(completion): rank word completion candidates by scope tier (#555)

Closes #555.

Branch (pushed): geircodes/Clarion-Extension:fix/word-completion-sorttext-tiers → version-1.0.4

Motivation

WordCompletionProvider set no sortText anywhere, so the client ordered every candidate alphabetically against every other one. Once #554 added project-wide EQUATEs from the declaration index, a cross-file constant could outrank a local variable on a shared prefix by pure alphabetical luck — the constant is valid to the compiler so it belongs in the list, but it shouldn't compete for the top of it.

The tiers

One tier per source, alphabetical within each:

tier	source	collectors
1	locals and parameters	collectVariables, collectParameters, the document's own EQUATEs
2	module and PROGRAM data	collectProgramGlobalDataSymbols
3	MAP procedures	collectProcedures
4	project-wide index entries	collectProjectEquates
5	static catalogs	keywords, built-ins, data types, controls, attributes, directives

This matches what tsserver does with its own SortText enum — locals first, globals and keywords later, auto-import suggestions (the closest analogue to a cross-file EQUATE reached through an un-inlined INCLUDE) last.

Two details worth review attention

Every candidate gets a tier, including the catalogs the issue doesn't name. Leaving any unstamped would not preserve their position: the client compares an unstamped item's label against a stamped item's sortText, and since digits precede letters, every unstamped candidate would sink below every stamped one. The catalogs reach seen without going through add(), so a post-pass stamps whatever is left:

for (const item of seen.values()) {
    if (!item.sortText) {
        item.sortText = tierSortText(SORT_TIER.CATALOG, String(item.label));
    }
}

The tier number is zero-padded and the label half lowercased. Without padding, a tenth tier would sort between 1 and 2. Without lowercasing — Clarion being case-insensitive — MyConst and myconst would swap places depending on capitalisation. Two digits is deliberate: the tiers are spaced by ten, so there is room for nine more between any two existing ones and the ceiling is 99.

function tierSortText(tier: SortTier, label: string): string {
    return `${tier.toString().padStart(2, '0')}_${label.toLowerCase()}`;
}

Implementation

add() stamps sortText only on first insertion, never on the merge path, so the first (higher-priority) tier to claim a label keeps its rank — the same first-writer-wins rule the closure already followed for label/kind. A small addIn(tier) wrapper binds a tier to the callback, so none of the 13 collector signatures had to change:

const addIn = (tier: SortTier) =>
    (label: string, kind: CompletionItemKind, detail?: string, documentation?: string, typeText?: string) =>
        add(label, kind, detail, documentation, typeText, tier);

collectEquates takes a second callback for its project-wide delegation, so the document's own EQUATEs rank as locals while indexed ones rank at tier 4 — the actual complaint in #555.

Testing

WordCompletionSortTextTiers555.test.ts — 3 tests:

• the issue's stated ordering: a local, a PROGRAM global and a project-wide EQUATE sharing a prefix come back in that order by sortText
• each tier forms one contiguous block, alphabetical within itself under case-insensitive comparison (this is what would catch a missing toLowerCase())
• every candidate carries a sortText, so none is ranked by its bare label

Proved real: reverted the provider, recompiled, reran — all 3 fail. Restored → all 3 pass.

The fixture is a real PROGRAM + MEMBER pair on disk (the #565 pattern, since PROGRAM globals are read from the file) plus a prototype-stubbed SDI and SolutionManager (the #312/#554 pattern), so all three tiers have a candidate sharing the prefix without needing a solution load.

Full suite: 3053 passing, 4 pending, 0 failing (pre-existing pendings, unrelated) — no regressions.

Performance

No measurable cost. Measured against a large PROGRAM file (770 KB, 14,512 lines) with a ~20,000-entry declaration index, averaged over repeated requests:

	avg	median	p95
without the tiers	0.39 ms	0.35 ms	0.65 ms
with the tiers	0.43 ms	0.40 ms	0.76 ms

The per-request stage breakdown puts the tier post-pass at 0–1 ms against a total dominated entirely by tokenization. An A/B on the same file — same typing sequence, sortText stamped versus omitted entirely — produced statistically identical request times (slow requests 649–716 ms with tiers, 668–698 ms without; fast requests 18–22 ms versus 21–29 ms), with ~97% of every slow request attributable to re-lexing rather than to anything in this change.

Scope

• Ordering only. No candidate is added, removed or relabelled; only sortText is set.
• Client-side recency is deliberately untouched. VS Code already handles "what did I pick last time" through editor.suggestSelection (recentlyUsed / recentlyUsedByPrefix), and that changes which item is preselected, not the order — so it layers on top of these tiers rather than competing with them. LSP also gives the server no signal about which completion was accepted, so recency can't be tracked server-side anyway. Tiers stay pure scope distance.
• PROJECT_EQUATE_LIMIT truncation is a separate, pre-existing issue, not addressed here. collectProjectEquates walks index.byName — a plain Map in file-scan order — and stops at 300 matches, so a broad prefix silently drops candidates that a narrower one reveals (e.g. typing AB_ can omit entries that AB_C then shows). That drops valid candidates rather than merely misordering them, so it deserves its own issue and fix.