[ajitpratap0]

Summary

Adds SQL syntax highlighting as a first-class GoSQLX capability.

Core — new pkg/highlight

• highlight.Spans(sql, opts...) classifies SQL into sorted, non-overlapping byte-offset spans with 13 stable categories: keyword, function, type, identifier, quoted-identifier, string, number, constant, operator, punctuation, comment, placeholder, error.
• Dialect-aware via WithDialect.
• Context rules:
  • functions are recognised in call position;
  • types in type position: column definitions, x::int, CAST(x AS DATE), Nullable(String);
  • prefixed literals stay whole: N'..', E'..', 0x1F, :name, @@var.
• Never fails: invalid SQL is highlighted up to the first lexical error, and the rest becomes one error span. Works for incomplete editor input.
• Works around tokenizer quirks (unchanged in this PR):
  • token start positions can include a preceding comment, so starts are derived from end positions;
  • tab-as-4 column counting is handled;
  • some unterminated-literal errors report column 0 or end-of-input, so error positions are validated, with a fallback scan for the unclosed opener.
• Renderers:
  • ANSI: 256-color, true color, or attributes only; resets at each newline.
  • HTML: escaped; class-based or inline styles.
  • CSS stylesheet generation.
  • Themes: dark, light, mono. Theme colors and class prefixes are sanitized.
• Spans marshal to JSON.

Surfaces

• API: gosqlx.Highlight, gosqlx.HighlightANSI, gosqlx.HighlightHTML.
• CLI: gosqlx highlight (alias hl) with -f ansi|html|json|css, --theme, --dialect, --color auto|always|never (honors TTY, NO_COLOR and COLORTERM) and --standalone.
• LSP: semanticTokens/full now uses pkg/highlight.
  • Standard token types; positions in UTF-16.
  • Multi-line tokens are split one per line.
  • The dialect comes from initializationOptions.dialect.
  • The VS Code extension restarts the server when gosqlx.dialect changes.
• WASM: gosqlxHighlight(sql[, dialect]) returns spans with UTF-16 offsets.
• Playground: CodeMirror decorations driven by GoSQLX spans in the selected dialect.
  • Lezer highlighting is the fallback until WASM loads, on failure, and for documents over 200k characters.
  • Colors use the site design tokens.
• Docs: docs/HIGHLIGHTING.md, plus updates to the CLI/API/LSP/usage guides, README, llms.txt, CLAUDE.md and the CHANGELOG [Unreleased] section.

Test Plan

• task check: fmt, vet, golangci-lint (0 issues), race tests; plus task test:cbinding.
• pkg/highlight tests:
  • classification table;
  • error-recovery cases, including tokenizer location quirks;
  • tabs and UTF-8;
  • all 11 dialects;
  • corpus test over the repository's ~130 testdata SQL files (no unclassified text);
  • renderer escaping and injection tests;
  • concurrency; examples.
  • Coverage about 91%.
• Fuzzing FuzzSpans for about 2.6M executions with strict invariants:
  • spans are ordered and in bounds;
  • spans don't start or end with whitespace;
  • no significant text is left unclassified;
  • ANSI output round-trips.
  • It found a tab/error-path panic, which is fixed and kept as a seed.
• LSP tests for the legend, UTF-16 positions (é, 😀), CRLF, multi-line splitting, invalid SQL, and the dialect option.
• CLI command tests for each format, colour mode, NO_COLOR, bad flags, stdin and file input.
• WASM bridge UTF-16 conversion tests and contract fixture; GOOS=js GOARCH=wasm build.
• Website: typecheck, lint (no new warnings) and unit tests 32/32.
• Manual check of the playground in a local dev server and real Chrome, in dark and light themes, with UTF-16 offsets verified via gosqlxHighlight in the console.
• Reviews: per-surface specialist reviews (Go, TypeScript) and an independent broad review. All findings were addressed, including a high-severity tab-offset bug.
• Benchmark: BenchmarkSpans takes about 45µs for 2 KB, roughly twice the cost of tokenizing alone. The tokenizer and parser hot paths are untouched.

Breaking Changes

None to the Go API. The LSP semantic token legend changed:

• identifier is replaced by the standard variable type.
• Quoted identifiers are now variable; they were string.
• Comments, functions, types and parameters are now emitted.
• Positions are now UTF-16.

Clients read the legend at initialize, so nothing breaks at the protocol level. Users with custom styling keyed on the old identifier type (for example Neovim @lsp.type.identifier) need to update it. This is recorded under Behavior changes in the CHANGELOG.

Follow-ups, not in this PR:

• Tokenizer error locations for unterminated quoted identifiers (column 0) and dollar quotes (end of input) could be fixed at the source; that change needs golden-file updates.
• Other LSP features still ignore the dialect.
• The CLI dialect help text and the VS Code gosqlx.dialect enum list fewer dialects than the code accepts.

Type of Change

• New feature (non-breaking change which adds functionality)
• Documentation update
• Breaking change: no Go API break; the LSP legend change is listed under Breaking Changes below

Performance Impact

• No performance impact on the existing tokenizer and parser hot paths. Highlighting is a separate, opt-in package.

BenchmarkSpans-10   26042   45935 ns/op   44.63 MB/s   112895 B/op   282 allocs/op   (2 KB of SQL)
BenchmarkHTML-10    80782   15603 ns/op                 48809 B/op    19 allocs/op

Test Coverage

• pkg/highlight: about 91% (new package)

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my own code, plus specialist and independent reviews
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings (golangci-lint 0 issues; website lint has no new warnings)
• I have added tests that prove my feature works
• New and existing unit tests pass locally with my changes (race-enabled)
• No new dependencies

🤖 Generated with Claude Code

https://claude.ai/code/session_01KTj1PEbxzVCETkeYrZKcrE

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
gosqlx-website	Ready	Preview	Oct 21, 2026 9:08am UTC

[github-actions]

Claude finished @ajitpratap0's task in 26s —— View job

---

Review of PR #825: dialect-aware syntax highlighting

I read the core of pkg/highlight (highlight.go, render.go). I did not read the CLI, LSP, WASM, website or docs changes in detail, and I did not run the tests.

Overall this is well-structured and carefully defended. The renderers escape output, sanitize prefixes, and walk skips bad spans. Spans never fails. The list below is mostly minor.

Potential issues

1. Recovery can silently drop most of the input (highlight.go:97-120). After maxRecoveries (4) failed attempts, or when off <= 0, the whole document becomes a single Error span. A long document with an early, unusual lexical error will lose all highlighting, and so will an input where the offset estimate is wrong. This is probably acceptable, but a test that pins this fallback behaviour would be useful.
2. keywordsFor holds a global mutex and keywords.New resets package state (highlight.go:722-740). Construction is serialized, but if any other code calls keywords.New concurrently, the comment's "resets package state" suggests a race. Please confirm that the other callers of keywords.New (parser, tokenizer) can't race with this one. task test:race passing is a good sign, but the concurrency test should cover that mix.
3. upper() comment is inaccurate (highlight.go:696-700). It says ASCII input is returned without allocating, but the function is a bare strings.ToUpper. ToUpper does return the input unchanged when it is already uppercase, but it allocates for mixed or lowercase input. Since category calls it per lexeme, that is a per-token allocation. Either fix the comment or add a non-allocating comparison, for example a switch on a pre-uppercased cached value in lexeme. The 45µs/2KB benchmark suggests this isn't a big cost, but it is cheap to improve.
4. observe uses ls[i-1].value as the paren opener (highlight.go:405-410). For a quoted or prefixed token before (, the opener is the raw token value. That is fine for the heuristics, but upper("") or punctuation values can match nothing silently. Low risk.
5. Heuristic false positives. functionKeywords includes YEAR/MONTH/DAY/LEFT/RIGHT, and dataTypeNames includes YEAR, KEY-like words and NUMBER. typeContext returns true whenever st.ddl is set and the previous lexeme is an identifier. For example, CREATE TABLE t (name text, id int) is fine, but ALTER TABLE t RENAME a TO b-style statements may classify b oddly. These are cosmetic. A few more negative test cases for DDL would help.
6. ddl state is only reset on ; (highlight.go:415-418). A script without semicolons that starts with CREATE ... and then continues with SELECT x::... stays in DDL mode. Words like date or timestamp after an identifier would be coloured as types in later statements. This is cosmetic, but you could also reset on statement-starting keywords such as SELECT, INSERT, UPDATE and DELETE.

Security

• HTML escaping is complete (&<>"'). Class prefixes and theme colors are sanitized (parseHex is used for colors), and cssDecl only emits parsed values. I see no injection path.
• ANSI: source text is written raw, so a source string containing ESC sequences passes through to the terminal. This matters for gosqlx highlight on untrusted input (terminal escape injection). Consider stripping or escaping control characters (other than \n, \t, \r) in the ANSI renderer, or at least documenting it. The fuzz "round-trip" invariant would need to allow for this.

Performance

• Spans converts sql to []byte (an extra copy) and re-tokenizes up to 5 times on errors. That is fine for editor-sized input.
• WriteHTML builds htmlEscaper.WriteString(w, text) against w rather than sw, so the writer type is wrapped inconsistently. It is harmless.
• The playground cut-off at 200k characters is a sensible guard.

Tests

The coverage described is strong: corpus test, fuzzing, UTF-16 conversion tests and dialect tests. Suggestions:

• Add a test for ESC and control characters in the ANSI renderer (see Security above).
• Add a negative DDL test for state leaking past a missing semicolon (item 6).
• Pin the "error in the first few bytes means one Error span" behaviour (item 1).

Notes

• The LSP legend change is a user-visible behaviour change. It is documented in the CHANGELOG, which is good.
• The follow-ups you listed (tokenizer error locations, dialect list in the CLI and VS Code enum) are reasonable to defer. The dialect list mismatch is a small fix that you could do in this PR.

No blocking issues found.
· branch feat/syntax-highlighting

[github-actions]

Claude finished @ajitpratap0's task in 35s —— View job

---

Review of PR #825: dialect-aware syntax highlighting

I read pkg/highlight/highlight.go and render.go, and grepped for the specific points below. I did not run the tests. I did not read the CLI, LSP, WASM, website or docs changes in detail. This update replaces the earlier review comment, which was a first pass.

No blocking issues. The earlier review's items still stand; this pass checked them against the code.

Findings

1. ANSI escape injection (most worth fixing). render.go writes source text to the terminal unchanged. It defines only its own SGR sequences (render.go:72, render.go:123-145) and has no control-character handling. A SQL comment or string containing \x1b]... or cursor-movement sequences would reach the terminal when someone runs gosqlx highlight on untrusted SQL. Consider replacing C0 control characters (except \t, \n, \r) with the visible form ␛ or ^[ in the ANSI renderer, or at least documenting the behaviour. The fuzz round-trip invariant would need to allow for the substitution. Add a test with a payload such as SELECT '\x1b[2J'.

2. upper() comment is wrong (highlight.go:696-700). It says ASCII-uppercase input is returned without allocating, but the function is a bare strings.ToUpper. ToUpper does skip allocation when the input is already uppercase, but lowercase or mixed-case input still allocates. This runs per lexeme. Fix the comment, or cache the uppercase form once per lexeme.

3. ddl state resets only on ; (highlight.go:391, highlight.go:415-422). In a script without semicolons, such as editor scratch input, the state stays set after CREATE .... dataTypeNames[w] && st.ddl (highlight.go:499) then colours words like date or timestamp as types in later SELECT statements. This is cosmetic. Consider also resetting on statement-leading keywords (SELECT, INSERT, UPDATE, DELETE, WITH). A negative test would pin the behaviour.

4. Recovery fallback. After maxRecoveries (4) attempts, or when the offset estimate is <= 0 (highlight.go:114), the rest of the document becomes a single Error span. That is reasonable, but I couldn't find a test that pins it. Add one for an early error followed by a long valid tail.

5. keywords.New and shared state (highlight.go:727). The mutex only serializes this package's own calls. If the parser or tokenizer call keywords.New concurrently, the comment says it "resets package state", which would be a race. task test:race passing is encouraging. Please confirm that keywords.New is safe to call from other packages concurrently, or extend the concurrency test to mix highlight with parse calls.

6. Dialect list mismatch. You listed this as a follow-up. It's a small fix (CLI help text and the VS Code gosqlx.dialect enum), so it could go into this PR.

Other notes

• HTML escaping, class-prefix sanitization and theme colour parsing look sound. I found no injection path in the HTML or CSS output.
• Performance is fine for editor-sized input. The 200k-character playground cutoff is a sensible guard.
• The LSP legend change is documented as a behaviour change in the CHANGELOG.
• Test coverage is strong (corpus, fuzz, UTF-16, dialects). The additions suggested above are items 1, 3 and 4.
  · branch feat/syntax-highlighting

[github-actions]

Claude finished @ajitpratap0's task in 22s —— View job

---

Review of PR #825: dialect-aware syntax highlighting

I re-checked the points raised in the earlier reviews against pkg/highlight/highlight.go and render.go. I did not run the tests, and I did not read the CLI, LSP, WASM, website or docs changes line by line. There are no blocking issues. Everything below is a follow-up.

Worth fixing

1. ANSI terminal escape injection (render.go:74-145)

   • Source text is written to the terminal as-is.
   • gosqlx highlight on untrusted SQL can therefore emit OSC or cursor-control sequences, for example from SELECT '\x1b]0;title\x07' or from a comment.
   • Replace C0 controls and DEL, except \t, \n and \r, with a visible form such as ␛ or ^[ in the ANSI renderer. Alternatively, document the behavior.
   • highlight_test.go:425-426 already skips inputs containing \x1b in the round-trip fuzz invariant, so that skip would need to move to the substituted form.
   • Add a test with an ESC payload.

2. Inaccurate upper() comment (highlight.go:687-691)

   • It claims ASCII-uppercase input is returned without allocating. In fact it is a bare strings.ToUpper.
   • ToUpper only avoids allocating when the input has no lowercase letters. Lowercase or mixed-case input allocates.
   • It is called per lexeme (:399, :411 and others). Either fix the comment or compute the uppercase value once per lexeme and reuse it.

3. ddl state leaks without ; (highlight.go:406-414)

   • st.ddl is only cleared on a semicolon.
   • In a semicolon-less script, such as editor scratch input, CREATE ... followed by SELECT a date, ... can still colour type-like words as types.
   • Consider also resetting the state on statement-leading keywords (SELECT, INSERT, UPDATE, DELETE, WITH).
   • Add a negative test for this.

Minor

4. Recovery fallback is untested

   • After maxRecoveries (4) attempts, or when the offset estimate is <= 0 (:114), the rest of the document becomes one Error span.
   • Add a test pinning an early error followed by a long valid tail.

5. keywords.New shared state (highlight.go:709-723)

   • The mutex only serializes this package's own calls.
   • If the tokenizer or parser call keywords.New concurrently, the "resets package state" comment implies a possible race.
   • Please confirm those callers are safe, or extend the concurrency test to mix highlight and parse calls under -race.

6. Dialect lists differ

   • The CLI help text and the VS Code gosqlx.dialect enum list fewer dialects than the code accepts.
   • It is a small fix and could go in this PR.

Looks good

• HTML escaping, class-prefix sanitization and theme-color parsing: I found no injection path.
• Test strategy: the corpus test, fuzzing, UTF-16 and CRLF cases, and per-dialect coverage are strong.
• The 200k-character playground cutoff is a sensible guard.
• The LSP legend change is recorded in the CHANGELOG.
  · branch feat/syntax-highlighting

[github-actions]

Claude finished @ajitpratap0's task in 20s —— View job

---

Review of PR #825: dialect-aware syntax highlighting

I re-checked the earlier review points against pkg/highlight with grep, not by reading every file. I did not run the tests. I did not read the CLI, LSP, WASM, website or docs changes line by line. This pass found nothing blocking. Three points below are worth fixing before merge.

Worth fixing

1. Terminal escape injection in the ANSI renderer (pkg/highlight/render.go)

   • Grep finds no control-character handling in render.go. The only \x1b occurrences are its own SGR sequences (render.go:72, render.go:145).
   • Source text appears to be written to the terminal as-is. A comment or string literal containing \x1b]0;...\x07 or a cursor-movement sequence would reach the terminal when someone runs gosqlx highlight on untrusted SQL.
   • Suggested fix: in the ANSI renderer only, replace C0 controls and DEL, except \t, \n and \r, with a visible form such as ␛.
   • Update the fuzz round-trip invariant to match. A seed \x1b[m is already in the fuzz corpus (testdata/fuzz/FuzzSpans/96cc2c3d5e52083c), and the test currently skips inputs containing ESC.
   • Add a test with an ESC payload such as SELECT '\x1b[2J'.

2. Inaccurate upper() comment (highlight.go:689)

   • The function is a bare strings.ToUpper, and it runs per lexeme. ToUpper allocates for lowercase or mixed-case input.
   • Either fix the comment or compute the uppercase value once per lexeme and reuse it. The benchmark shows 282 allocs per 2 KB, so this is a cheap win.

3. ddl state leaks without a semicolon (highlight.go:382, :407-413, :490, :543)

   • st.ddl is cleared only on ;.
   • In a script without semicolons, such as editor scratch input, CREATE ... followed by SELECT a date ... still colours type-like words as types.
   • Consider also resetting on statement-leading keywords: SELECT, INSERT, UPDATE, DELETE, WITH.
   • Add a negative test for this.

Minor

4. Recovery fallback is untested (highlight.go:114)

   • After maxRecoveries (4) attempts, or when off <= 0, the rest of the document becomes one error span.
   • Add a test that pins an early error followed by a long valid tail.

5. keywords.New shared state (highlight.go:710-720)

   • The comment says keywords.New "resets package state", but the mutex only serializes this package's own calls.
   • Please confirm that the tokenizer and parser either don't call it concurrently or are safe to.
   • Otherwise, extend the concurrency test to mix highlight and parse calls under -race.

6. Dialect lists differ

   • The CLI help text and the VS Code gosqlx.dialect enum list fewer dialects than the code accepts. It is a small fix that could go into this PR.

Looks good

• HTML escaping, class-prefix sanitization (sanitizePrefix) and theme-colour parsing: I saw no injection path.
• Test strategy: the corpus test, fuzzing, UTF-16 and CRLF cases, and per-dialect coverage are strong.
• The 200k-character playground cutoff is a sensible guard.
• The LSP legend change is recorded in the CHANGELOG.
  · branch feat/syntax-highlighting

[github-actions]

Claude finished @ajitpratap0's task in 20s —— View job

---

Review of PR #825: dialect-aware syntax highlighting

I re-checked the points from the earlier reviews against pkg/highlight using grep. I did not run the tests, and I did not read the CLI, LSP, WASM, website or docs changes line by line. I found nothing blocking. The three "Worth fixing" items below are still open.

Worth fixing

1. Terminal escape injection in the ANSI renderer (pkg/highlight/render.go)

   • render.go has no control-character handling. The only \x1b bytes it contains are its own SGR sequences (render.go:72, render.go:145).
   • Source text is written to the terminal unchanged. gosqlx highlight on untrusted SQL can therefore emit OSC or cursor-control sequences from a string literal or comment.
   • Suggested fix, in the ANSI renderer only: replace C0 controls and DEL, except \t, \n and \r, with a visible form such as ␛.
   • highlight_test.go:426 skips inputs containing \x1b in the round-trip fuzz invariant. That skip would need to change to account for the substitution.
   • Add a test with a payload such as SELECT '\x1b[2J'.

2. upper() comment is inaccurate (highlight.go:689)

   • The function is a bare strings.ToUpper, called per lexeme. It allocates for lowercase or mixed-case input, which contradicts the comment.
   • Either fix the comment or compute the uppercase value once per lexeme and reuse it.

3. ddl state is only cleared on ; (highlight.go:382, :407, :490, :543)

   • In a script without semicolons, such as editor scratch input, CREATE ... followed by SELECT a date ... can still colour type-like words as types.
   • Consider also resetting on statement-leading keywords: SELECT, INSERT, UPDATE, DELETE, WITH.
   • Add a negative test for this.

Minor

4. Recovery fallback is untested (highlight.go:114)

   • After maxRecoveries (4) attempts, or when off <= 0, the rest of the document becomes one error span.
   • Add a test that pins an early error followed by a long valid tail.

5. keywords.New shared state (highlight.go:710-720)

   • The mutex only serializes this package's own calls. If the parser or tokenizer call keywords.New concurrently, the "resets package state" comment points to a possible race.
   • Please confirm those callers are safe, or extend the concurrency test to mix highlight and parse calls under -race.

6. Dialect lists differ

   • The CLI help text and the VS Code gosqlx.dialect enum list fewer dialects than the code accepts. This is a small fix that could go into this PR.

Looks good

• HTML escaping, class-prefix sanitization and theme-colour parsing: I saw no injection path.
• The test strategy is strong: corpus test, fuzzing, UTF-16 and CRLF cases, and per-dialect coverage.
• The 200k-character playground cutoff is a sensible guard.
• The LSP legend change is recorded in the CHANGELOG.
  · branch feat/syntax-highlighting