diff --git a/README.md b/README.md index d2815d78..8666a9f0 100644 --- a/README.md +++ b/README.md @@ -166,7 +166,7 @@ Formatting turns one-line SQL into a readable query: - **[Analysis](https://gosqlx.dev/docs/reference/linting-rules/)** - SQL injection scanner, 30 lint rules (L001-L030), 20 optimizer rules and metadata extraction. - **[Tooling](https://gosqlx.dev/docs/cli-guide/)** - AST-based formatter, query transforms API, VS Code extension and GitHub Action. - **[Syntax highlighting](docs/HIGHLIGHTING.md)** - dialect-aware highlighting for terminals (ANSI), HTML/CSS, JSON spans, LSP semantic tokens and WASM; `gosqlx highlight query.sql` or `gosqlx.HighlightANSI(sql)`. -- **[Multi-dialect](https://gosqlx.dev/docs/reference/sql-compatibility/)** - PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, SQLite, Snowflake, ClickHouse. +- **[Multi-dialect](https://gosqlx.dev/docs/reference/sql-compatibility/)** - 8 dialects with dedicated grammar support: PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, SQLite, Snowflake, ClickHouse. The `--dialect` flag and `WithDialect` accept 11 names: those 8 (`postgresql`, `mysql`, `mariadb`, `sqlserver`, `oracle`, `sqlite`, `snowflake`, `clickhouse`) plus `generic`, and `bigquery` and `redshift`, which have keyword recognition and basic parsing only. - **[AI-ready](https://gosqlx.dev/docs/integrations/mcp/)** - MCP server with 7 tools, public remote endpoint, Streamable HTTP. - **[Tested](https://gosqlx.dev/docs/reference/sql-compatibility/)** - race-detector clean in CI, ~80-85% SQL-99 (estimated). diff --git a/cmd/gosqlx/cmd/doc.go b/cmd/gosqlx/cmd/doc.go index d3e18257..38f5d910 100644 --- a/cmd/gosqlx/cmd/doc.go +++ b/cmd/gosqlx/cmd/doc.go @@ -22,10 +22,17 @@ // - parse: AST generation and inspection with multiple output formats // - analyze: Security and complexity analysis with grading // - lint: Style and quality checking with L001-L010 rules +// - highlight (alias hl): Syntax highlighting as ANSI, HTML, CSS or JSON spans // - lsp: Language Server Protocol server for IDE integration // - config: Configuration file management // - completion: Shell autocompletion setup // +// validate, parse, format, analyze, lint and highlight accept --dialect with +// one of the 11 names from keywords.DialectNames(): generic, postgresql, +// mysql, mariadb, sqlserver, oracle, sqlite, snowflake, bigquery, redshift +// and clickhouse; an unknown name is rejected with an error. Each +// command's --help names its default (highlight uses the generic keyword set). +// // # Architecture // // The package follows a modular design with separation of concerns: @@ -44,6 +51,7 @@ // ├── sql_analyzer.go - AST-based analysis engine // ├── analysis_types.go - Analysis data structures // ├── lint.go - Lint command definition +// ├── highlight.go - Highlight command (pkg/highlight renderers) // ├── lsp.go - LSP server command // ├── config.go - Config management commands // ├── config_manager.go - Config management logic @@ -314,13 +322,24 @@ // Key types: Analyzer, SQLAnalyzer, AnalysisReport // Key functions: Analyze(), DisplayReport(), scoreQuery() // +// ## highlight Command +// +// Syntax-highlights SQL through pkg/highlight. --format selects ansi (the +// default), html, css or json spans; --theme selects dark, light or mono; +// --color auto|always|never controls ANSI coloring; --standalone emits a full +// HTML document. Highlighting is lexical and never fails: invalid SQL is +// highlighted up to the first lexical error and the rest is marked as an +// error. +// +// Implementation: highlight.go +// // ## lint Command // // Checks SQL for style violations. // // Implementation: lint.go // Key functions: lintRun(), createLinter() -// Rules: L001-L010 (see pkg/linter) +// Rules: L001-L010 (see pkg/linter); --dialect is passed to Linter.SetDialect // // ## lsp Command // diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md index 917a8306..9ab459da 100644 --- a/docs/API_REFERENCE.md +++ b/docs/API_REFERENCE.md @@ -19,7 +19,7 @@ New to GoSQLX? Start with [Getting Started](GETTING_STARTED.md), then pick a tas | [errors](api/errors.md) | `pkg/errors` | Structured errors with codes, positions and hints | | [metrics](api/metrics.md) | `pkg/metrics` | Production counters and timings | | [security](api/security.md) | `pkg/sql/security` | SQL injection pattern detection | -| [highlight](#highlight-package) | `pkg/highlight` | Dialect-aware SQL syntax highlighting as spans, ANSI, HTML and CSS | +| [highlight](api/highlight.md) | `pkg/highlight` | Dialect-aware SQL syntax highlighting as spans, ANSI, HTML and CSS | | [linter](api/linter.md) | `pkg/linter` | The lint engine and its rules | | [lsp](api/lsp.md) | `pkg/lsp` | The Language Server Protocol server | | [mcp](api/mcp.md) | `pkg/mcp` | The Model Context Protocol server | @@ -65,124 +65,7 @@ Moved to the [security package reference](api/security.md). ## Highlight Package -`github.com/ajitpratap0/GoSQLX/pkg/highlight` classifies SQL into byte-offset spans for terminals, HTML and editors. The guide with examples is [Syntax Highlighting](HIGHLIGHTING.md). - -### Spans - -```go -func Spans(sql string, opts ...Option) []Span -func WithDialect(d keywords.SQLDialect) Option // type Option func(*config) - -type Span struct { - Start int `json:"start"` // byte offset, inclusive - End int `json:"end"` // byte offset, exclusive - Category Category `json:"category"` -} -func (s Span) Text(src string) string -``` - -`Spans` returns spans sorted by `Start` that never overlap and never carry `Plain`; uncovered bytes are plain text. It never returns an error: after the first lexical error the rest of the input is one `Error` span. Input the tokenizer refuses outright (beyond its size or token limits) yields no spans. `Spans` is safe for concurrent use. `WithDialect` accepts the dialect identifiers listed below; an empty or unknown dialect uses the generic keyword set. - -### Categories - -```go -type Category uint8 - -const ( - Plain Category = iota // never on a Span; implicit category of gaps - Keyword - Function - DataType - Identifier - QuotedIdentifier - String - Number - Constant - Operator - Punctuation - Comment - Placeholder - Error -) - -func Categories() []Category // every category except Plain, in declaration order -func ParseCategory(name string) (Category, bool) -func (c Category) String() string // stable name, e.g. "quoted-identifier" -``` - -Stable names, used as JSON values, HTML class suffixes and theme keys: `keyword`, `function`, `type` (`DataType`), `identifier`, `quoted-identifier`, `string`, `number`, `constant`, `operator`, `punctuation`, `comment`, `placeholder`, `error`. `Category` marshals to and from these names in JSON. - -### Terminal Output - -```go -func ANSI(src string, spans []Span, opts ANSIOptions) string -func WriteANSI(w io.Writer, src string, spans []Span, opts ANSIOptions) error - -type ANSIOptions struct { - Theme Theme // zero Theme means DarkTheme - Mode ColorMode // zero value is Color256 -} - -type ColorMode int // Color256 (default), TrueColor, NoColor -``` - -`NoColor` emits only bold, italic and underline. Styles are reset before each newline and re-applied after it, so output stays correct through `head`, `grep` or a pager. - -### HTML and CSS - -```go -func HTML(src string, spans []Span, opts HTMLOptions) string -func WriteHTML(w io.Writer, src string, spans []Span, opts HTMLOptions) error -func CSS(theme Theme, classPrefix string) string - -type HTMLOptions struct { - ClassPrefix string // empty means DefaultClassPrefix ("gsx-") - InlineStyles bool // emit style="..." from Theme instead of classes - Theme Theme // used with InlineStyles; zero means DarkTheme - Wrap bool // wrap in
-}
-
-const DefaultClassPrefix = "gsx-"
-```
-
-All source text is HTML-escaped. Spans become ``. `CSS` returns one rule per category for the same class names; it styles the token classes only.
-
-### Themes
-
-```go
-type Theme struct {
- Name string
- Styles map[Category]Style
-}
-
-type Style struct {
- Color string // "#RRGGBB" or "#RGB"; other values are ignored
- Bold bool
- Italic bool
- Underline bool
-}
-
-func DarkTheme() Theme
-func LightTheme() Theme
-func MonoTheme() Theme
-func ThemeByName(name string) (Theme, bool) // case-insensitive
-func ThemeNames() []string // "dark", "light", "mono"
-func (t Theme) Style(c Category) Style
-```
-
-Each constructor returns a fresh copy that the caller may modify.
-
-### High-Level Helpers (gosqlx)
-
-```go
-func gosqlx.Highlight(sql string, opts ...highlight.Option) []highlight.Span
-func gosqlx.HighlightANSI(sql string, opts ...highlight.Option) string // dark theme, 256 colors
-func gosqlx.HighlightHTML(sql string, opts ...highlight.Option) string // wrapped in
-```
-
-### Dialects
-
-`highlight.WithDialect` and the CLI `--dialect` flag accept `generic`, `postgresql`, `mysql`, `mariadb`, `sqlserver`, `oracle`, `sqlite`, `snowflake`, `bigquery`, `redshift` and `clickhouse` (the values of `keywords.AllDialects()`). With no dialect, the generic keyword set is used.
+Moved to the [highlight package reference](api/highlight.md). The guide with examples is [Syntax Highlighting](HIGHLIGHTING.md).
## Linter Package
diff --git a/docs/MCP_GUIDE.md b/docs/MCP_GUIDE.md
index 3cd07b02..43fe15a4 100644
--- a/docs/MCP_GUIDE.md
+++ b/docs/MCP_GUIDE.md
@@ -74,12 +74,12 @@ All 7 GoSQLX SQL tools are available instantly:
- `parse_sql` - Parse SQL into AST statement types
- `extract_metadata` - Extract tables, columns, and functions
- `security_scan` - Detect SQL injection patterns
-- `lint_sql` - Lint against the style rules (L001–L010)
+- `lint_sql` - Lint against the ten style rules L001–L010 (not the L011–L030 safety, performance and naming rules)
- `analyze_sql` - Run all 6 tools in one concurrent request
### Rate Limits
-The public server has smart rate limiting to ensure fair usage:
+The public server has smart rate limiting to ensure fair usage. These limits come from the rate-limit middleware that the `gosqlx-mcp` binary installs (`NewRateLimitMiddleware`); a server embedded with `pkg/mcp` through `Server.Start` or `Server.Handler` has none of them unless you add the middleware yourself (see [Embedding as a Go Library](#embedding-as-a-go-library)):
- **Burst**: 10 requests/second per IP
- **Sustained**: 120 weighted-requests/minute per IP
- Heavier tools (`analyze_sql`) consume more budget than simple ones (`validate_sql`)
@@ -521,7 +521,7 @@ curl -s -X POST http://127.0.0.1:8080/mcp \
### lint\_sql
-**Description**: Lint SQL against the GoSQLX style rules (L001–L010).
+**Description**: Lint SQL against the ten GoSQLX style rules L001–L010, the same set `gosqlx lint` runs, with L005's maximum line length fixed at 100 characters. The safety, performance and naming rules L011–L030, which `gosqlx.Lint` also runs, are not applied.
#### Parameters
@@ -745,6 +745,26 @@ func main() {
}
```
+`Start` and `Handler` apply bearer auth only. They do **not** apply the rate limiting described under [Rate Limits](#rate-limits): there is no per-client rate limit, no `GET` stream cap, no request body size cap and no body read deadline. That is fine on `127.0.0.1`, but before exposing an embedded server to untrusted clients, serve `Handler` wrapped in `NewRateLimitMiddleware`, as `gosqlx-mcp` does:
+
+```go
+cfg, err := mcp.LoadConfig()
+if err != nil {
+ log.Fatal(err)
+}
+srv := mcp.New(cfg)
+defer srv.Shutdown(context.Background())
+
+httpSrv := &http.Server{
+ Addr: cfg.Addr(),
+ Handler: mcp.NewRateLimitMiddleware(cfg, srv.Handler()),
+ ReadHeaderTimeout: 10 * time.Second,
+}
+log.Fatal(httpSrv.ListenAndServe())
+```
+
+`NewRateLimitMiddleware` honours the trusted-proxy settings in `cfg` (see [Client IP and reverse proxies](#client-ip-and-reverse-proxies)). `Shutdown` stops the idle-session sweeper that `Handler` starts; it does not close an `http.Server` you built yourself.
+
### Public API
| Symbol | Signature | Description |
diff --git a/docs/MIGRATION.md b/docs/MIGRATION.md
index b5f4d6ba..646956ca 100644
--- a/docs/MIGRATION.md
+++ b/docs/MIGRATION.md
@@ -1,5 +1,54 @@
# Migration Guide
+## v1.14.0 → v1.15.0 (2026-07-30)
+
+**Source-compatible upgrade**: exported signatures are unchanged apart from additions, so code that compiled against v1.14.0 compiles against v1.15.0. Some runtime behavior changed; read the checklist below. The [CHANGELOG](../CHANGELOG.md) has the full list.
+
+### Why upgrade
+
+- **High-level `Tree` API**: `gosqlx.ParseTree` with typed walkers, `Rewrite` and `Clone`, and functional options instead of the `ParseWith*` combinations.
+- **ClickHouse at 83/83 of the QA corpus**: `ARRAY JOIN`, scalar CTEs, `CREATE MATERIALIZED VIEW ... ENGINE/TO`, `SAMPLE ... OFFSET` and `LIMIT offset, count`.
+- **Oracle gaps closed**: `ROWNUM`/`ROWID`/`LEVEL`/`SYSDATE` pseudo-columns, `PIVOT`/`UNPIVOT` `IN` lists with aliases, and literal backslashes in strings outside MySQL/PostgreSQL.
+- **Correctness sweep**: structured parser errors, pool leak fixes, and `Children()` coverage for 14 node types that `Walk`/`Inspect` used to skip.
+
+### New APIs to adopt
+
+- `gosqlx.ParseTree(ctx, sql, opts...)` returning `*Tree`, with `WalkSelects`, `WalkInserts`, `WalkUpdates`, `WalkDeletes`, `WalkCreateTables`, `WalkJoins`, `WalkCTEs`, `WalkFunctionCalls`, `WalkIdentifiers`, `WalkBinaryExpressions`, generic `WalkBy[T]`, `Tree.Rewrite` and `Tree.Clone`.
+- Options `WithDialect`, `WithStrict`, `WithTimeout`, `WithRecovery` and `WithMaxBytes`; `ParseReader` / `ParseReaderMultiple`, with `errors.Is(err, gosqlx.ErrTooLarge)` when the `WithMaxBytes` limit is exceeded.
+- `FormatTree` / `FormatAST` to format without re-parsing.
+- `gosqlx.SplitStatements(sql, dialect)`: dialect-aware script splitting on top-level `;`.
+- `gosqlx.Lint(sql)` (all 30 rules), `gosqlx.Analyze(sql)`, and `ErrorCode`, `ErrorLocation` and `ErrorHint` to unwrap structured error details.
+- `pkg/sql/dialect`: typed `Dialect` and `Capabilities`, exposed through `Parser.DialectTyped()`, `Parser.Capabilities()` and `Parser.Is()` helpers.
+- `pkg/linter/config`: `.gosqlx.yml` linter configuration with rule enable/disable, severity overrides and ignore globs.
+- `keywords.Conflicts()`, `ast.UnsupportedStatement` with `AST.HasUnsupportedStatements()`, and `ast.PoolLeakCount()`.
+
+### Behavior changes to act on
+
+- [ ] **Lint findings in nested queries**: fourteen rules now walk the whole AST, so violations inside subqueries and CTEs are reported. Expect more findings.
+- [ ] **Structured parser errors**: parser errors are `*errors.Error` values with a code and position, and the legacy `Parse`-family functions wrap the `ErrSyntax`, `ErrTokenize`, `ErrTimeout` and `ErrUnsupportedDialect` sentinels. Match with `errors.Is` / `errors.As` instead of comparing message strings.
+- [ ] **Cancellation**: a cancelled context returns `ctx.Err()` directly, so `errors.Is(err, context.Canceled)` works.
+- [ ] **`*errors.Error.WithX` methods copy** instead of mutating the receiver. Use the returned value.
+- [ ] **`parser.ValidateBytes("")` returns an error**, matching `gosqlx.Validate("")`.
+- [ ] **Snowflake stub statements** (`USE`, `COPY INTO`, `PUT`, `GET`, `LIST`, `REMOVE`, `CREATE STAGE/STREAM/TASK/PIPE/...`) are `*ast.UnsupportedStatement`, not `*ast.DescribeStatement`, and the formatter emits `-- UNSUPPORTED: ` for them. Update type switches.
+- [ ] **Metrics error statistics** are counted per error code instead of per `err.Error()` string.
+
+### Deprecations
+
+- `pkg/sql/monitor` is deprecated in favour of `pkg/metrics`.
+- `parser.Parse([]token.Token)`, `ParseFromModelTokensWithPositions` and `ConversionResult.PositionMapping` are scheduled for removal in v2.0.
+
+### Companion release versions
+
+| Component | v1.14.0 | v1.15.0 |
+|-----------|---------|---------|
+| Library (`github.com/ajitpratap0/GoSQLX`) | 1.14.0 | **1.15.0** (the `gosqlx.Version` constant still reads 1.14.0) |
+| CLI (`cmd/gosqlx`, `gosqlx --version`) | 1.14.0 | 1.14.0 (stale constant) |
+| VS Code extension (`ajitpratap0.gosqlx`) | 1.14.0 | **1.15.0** (version taken from the release tag) |
+| OpenTelemetry integration (`integrations/opentelemetry`) | requires v1.14.0 | requires v1.14.0 |
+| GORM integration (`integrations/gorm`) | requires v1.14.0 | requires v1.14.0 |
+
+---
+
## v1.13.0 → v1.14.0 (2026-04-12)
**Drop-in upgrade** — no breaking API changes. Anything that compiled against v1.13.0 will still compile and run identically against v1.14.0.
diff --git a/docs/README.md b/docs/README.md
index 7d0d5257..c37e3a39 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -8,7 +8,7 @@ GoSQLX is a production-ready, high-performance SQL parsing SDK for Go with compr
### Core Capabilities
- **High-Performance Parsing** - 1.38M+ operations/second sustained, 1.5M peak with zero-copy tokenization
-- **Multi-Dialect Support** - PostgreSQL, MySQL, SQL Server, Oracle, SQLite with ~80-85% SQL-99 compliance
+- **Multi-Dialect Support** - PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, SQLite, Snowflake and ClickHouse (plus `generic`, `bigquery` and `redshift` as accepted dialect names) with ~80-85% SQL-99 compliance
- **Thread-Safe Operations** - Race-free concurrent processing validated with 20,000+ concurrent operations
- **Memory Efficient** - Object pooling architecture with 60-80% memory reduction
- **Production Ready** - Comprehensive error handling, position tracking, and recovery
@@ -46,6 +46,8 @@ GoSQLX is a production-ready, high-performance SQL parsing SDK for Go with compr
| [**GETTING_STARTED.md**](GETTING_STARTED.md) | 5-minute quickstart guide for new users | Beginners |
| [**CLI_GUIDE.md**](CLI_GUIDE.md) | Command-line tool usage and examples | CLI Users |
| [**LSP_GUIDE.md**](LSP_GUIDE.md) | Language Server Protocol integration for IDEs | IDE Users/Developers |
+| [**MCP_GUIDE.md**](MCP_GUIDE.md) | MCP server for AI assistants: tools, configuration, rate limits and embedding | AI Tooling/DevOps |
+| [**CHATGPT_APP.md**](CHATGPT_APP.md) | How `gosqlx-mcp` works as a ChatGPT app and in MCP Apps hosts | AI Tooling |
### Core Documentation
@@ -57,6 +59,7 @@ GoSQLX is a production-ready, high-performance SQL parsing SDK for Go with compr
| [**TROUBLESHOOTING.md**](TROUBLESHOOTING.md) | Common issues, error messages, debugging techniques, and FAQ | Support/Debug |
| [**LINTING_RULES.md**](LINTING_RULES.md) | Complete linting rules reference (L001-L030) with examples | Developers/QA |
| [**CONFIGURATION.md**](CONFIGURATION.md) | Configuration file (.gosqlx.yml) guide with all options | DevOps/Teams |
+| [**HIGHLIGHTING.md**](HIGHLIGHTING.md) | SQL syntax highlighting: categories, themes, CLI, LSP semantic tokens and WASM ([API](api/highlight.md)) | Developers/Tool Builders |
### Reference Documentation
@@ -86,6 +89,7 @@ GoSQLX is a production-ready, high-performance SQL parsing SDK for Go with compr
| Document | Description |
|----------|-------------|
+| [**MIGRATION.md**](MIGRATION.md) | Release-by-release upgrade notes: new APIs, behavior changes and companion versions |
| [**UPGRADE_GUIDE.md**](UPGRADE_GUIDE.md) | Version upgrade guide with breaking changes |
| [**migration/FROM_JSQLPARSER.md**](migration/FROM_JSQLPARSER.md) | Migrating from JSqlParser |
| [**migration/FROM_PG_QUERY.md**](migration/FROM_PG_QUERY.md) | Migrating from pg_query |
@@ -387,7 +391,7 @@ The command-line tool now includes:
### Comprehensive SQL Support
- **80-85% SQL-99 Compliance**: Window functions, CTEs, set operations
-- **Multi-Dialect**: PostgreSQL, MySQL, SQL Server, Oracle, SQLite
+- **Multi-Dialect**: PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, SQLite, Snowflake, ClickHouse
- **Advanced Features**: MERGE, GROUPING SETS, materialized views
- **Modern SQL**: Full window function and CTE support
diff --git a/docs/api/highlight.md b/docs/api/highlight.md
new file mode 100644
index 00000000..67957dcd
--- /dev/null
+++ b/docs/api/highlight.md
@@ -0,0 +1,229 @@
+# highlight package
+
+Dialect-aware SQL syntax highlighting: classify SQL into byte-offset spans, then render them for terminals, HTML pages or editors.
+
+Import path: `github.com/ajitpratap0/GoSQLX/pkg/highlight` ([pkg.go.dev](https://pkg.go.dev/github.com/ajitpratap0/GoSQLX/pkg/highlight))
+
+Part of the [GoSQLX API reference](../API_REFERENCE.md). The guide with examples, CLI usage and the WASM build is [Syntax Highlighting](../HIGHLIGHTING.md).
+
+The same classifier drives `gosqlx.Highlight*`, the `gosqlx highlight` command and the LSP server's semantic tokens.
+
+## Quick Example
+
+```go
+import (
+ "fmt"
+
+ "github.com/ajitpratap0/GoSQLX/pkg/highlight"
+ "github.com/ajitpratap0/GoSQLX/pkg/sql/keywords"
+)
+
+sql := "SELECT id, name FROM users WHERE active = TRUE"
+spans := highlight.Spans(sql, highlight.WithDialect(keywords.DialectPostgreSQL))
+
+for _, s := range spans {
+ fmt.Printf("%-12s %q\n", s.Category, s.Text(sql))
+}
+
+fmt.Print(highlight.ANSI(sql, spans, highlight.ANSIOptions{}))
+
+html := highlight.HTML(sql, spans, highlight.HTMLOptions{Wrap: true})
+css := highlight.CSS(highlight.DarkTheme(), "")
+```
+
+## Classifying SQL
+
+### `Spans(sql string, opts ...Option) []Span`
+
+Classifies `sql` for syntax highlighting.
+
+- Spans are sorted by `Start`, never overlap and never carry `Plain`. Any byte not covered by a span (typically whitespace) is plain text.
+- `Spans` never returns an error. When the tokenizer rejects the input, everything before the offending position is still classified and the rest of the input is one `Error` span, so it is safe to call on incomplete editor input.
+- Input the tokenizer refuses outright (beyond its size or token limits) yields no spans. An empty string yields `nil`.
+- `Spans` is safe for concurrent use.
+
+### `WithDialect(d keywords.SQLDialect) Option`
+
+Selects the dialect used for tokenization and keyword recognition. It accepts the 11 values of `keywords.AllDialects()`: `generic`, `postgresql`, `mysql`, `mariadb`, `sqlserver`, `oracle`, `sqlite`, `snowflake`, `bigquery`, `redshift` and `clickhouse`. An empty or unknown dialect uses the generic keyword set.
+
+```go
+type Option func(*config)
+```
+
+### `Span`
+
+```go
+type Span struct {
+ Start int `json:"start"` // 0-based byte offset, inclusive
+ End int `json:"end"` // 0-based byte offset, exclusive
+ Category Category `json:"category"`
+}
+```
+
+`src[s.Start:s.End]` is the span's text. Offsets always fall on UTF-8 boundaries. Spans marshal to JSON as `{"start":0,"end":6,"category":"keyword"}`.
+
+### `(s Span) Text(src string) string`
+
+Returns the source text covered by the span, or `""` when the span does not fit within `src`.
+
+## Categories
+
+### `Category`
+
+```go
+type Category uint8
+
+const (
+ Plain Category = iota // never on a Span; implicit category of gaps
+ Keyword // SELECT, FROM, RETURNING, LATERAL
+ Function // a function name in call position: COUNT, now
+ DataType // a type name in type position: VARCHAR, x::int
+ Identifier // unquoted table, column, alias or schema name
+ QuotedIdentifier // "name", `name` or [name]
+ String // string literals, including N'', E'', X'' and dollar-quoted
+ Number // numeric literals
+ Constant // TRUE, FALSE, NULL
+ Operator // =, <>, ||, ->>, ::
+ Punctuation // comma, period, semicolon, parentheses, brackets, braces
+ Comment // -- line and /* block */ comments
+ Placeholder // $1, ?, :name, @var, @@var
+ Error // input the tokenizer rejected; always runs to the end of the input
+)
+```
+
+Each category has a stable lowercase name, used as the JSON value, the HTML class suffix and the theme key. The names will not change; new categories may be appended in later releases.
+
+| Category | Name |
+|---|---|
+| `Plain` | `plain` |
+| `Keyword` | `keyword` |
+| `Function` | `function` |
+| `DataType` | `type` |
+| `Identifier` | `identifier` |
+| `QuotedIdentifier` | `quoted-identifier` |
+| `String` | `string` |
+| `Number` | `number` |
+| `Constant` | `constant` |
+| `Operator` | `operator` |
+| `Punctuation` | `punctuation` |
+| `Comment` | `comment` |
+| `Placeholder` | `placeholder` |
+| `Error` | `error` |
+
+### `(c Category) String() string`
+
+Returns the stable name, for example `"keyword"` or `"quoted-identifier"`.
+
+### `(c Category) MarshalText() ([]byte, error)` and `(c *Category) UnmarshalText(text []byte) error`
+
+Encode and decode the stable name, so `Category` round-trips through JSON. Unknown names and out-of-range values return an error.
+
+### `ParseCategory(name string) (Category, bool)`
+
+Returns the category with the given stable name.
+
+### `Categories() []Category`
+
+Returns every category that can appear on a `Span`, in declaration order, excluding `Plain`. The result is a fresh slice.
+
+## Terminal Output
+
+### `ANSI(src string, spans []Span, opts ANSIOptions) string`
+
+Renders `src` with ANSI escape sequences.
+
+### `WriteANSI(w io.Writer, src string, spans []Span, opts ANSIOptions) error`
+
+Like `ANSI`, but writes to `w`.
+
+```go
+type ANSIOptions struct {
+ Theme Theme // the zero Theme means DarkTheme
+ Mode ColorMode // the zero value is Color256
+}
+
+type ColorMode int
+
+const (
+ Color256 ColorMode = iota // xterm 256-color palette (default)
+ TrueColor // 24-bit color; use when COLORTERM is "truecolor" or "24bit"
+ NoColor // bold, italic and underline only
+)
+```
+
+Styles are reset before every newline and re-applied after it, so output stays correct when piped through `head`, `grep` or a pager.
+
+Source text cannot inject terminal escape sequences. C0 control characters other than tab, newline and carriage return are shown as Unicode control pictures (ESC becomes U+241B), DEL as U+2421, and C1 controls and invalid UTF-8 bytes as U+FFFD. Only the renderer's own SGR sequences reach the terminal.
+
+## HTML and CSS
+
+### `HTML(src string, spans []Span, opts HTMLOptions) string`
+
+Renders `src` as HTML. All source text is escaped; only `` elements and the optional `` wrapper are emitted.
+
+### `WriteHTML(w io.Writer, src string, spans []Span, opts HTMLOptions) error`
+
+Like `HTML`, but writes to `w`.
+
+```go
+type HTMLOptions struct {
+ ClassPrefix string // empty means DefaultClassPrefix
+ InlineStyles bool // emit style="..." from Theme instead of class names
+ Theme Theme // used with InlineStyles; the zero Theme means DarkTheme
+ Wrap bool // wrap the output in
+}
+
+const DefaultClassPrefix = "gsx-"
+```
+
+Spans render as `` and so on. A `ClassPrefix` containing characters other than ASCII letters, digits, `-` and `_` is replaced by `DefaultClassPrefix`. `InlineStyles` suits contexts without a stylesheet, such as e-mail.
+
+### `CSS(theme Theme, classPrefix string) string`
+
+Returns a stylesheet with one rule per category for the class names `HTML` emits, for example `.gsx-keyword{color:#569CD6;font-weight:bold}`. The zero `Theme` means `DarkTheme`. It styles the token classes only, not the `` wrapper.
+
+## Themes
+
+```go
+type Theme struct {
+ Name string
+ Styles map[Category]Style
+}
+
+type Style struct {
+ Color string // "#RRGGBB" or "#RGB"; empty or malformed means the default foreground
+ Bold bool
+ Italic bool
+ Underline bool
+}
+```
+
+Categories without an entry in `Styles` render with the default style. Colors are validated before use, so a theme cannot inject CSS or escape sequences.
+
+### `(t Theme) Style(c Category) Style`
+
+Returns the style for `c`, or the zero `Style` when the theme has none.
+
+### `DarkTheme() Theme`, `LightTheme() Theme`, `MonoTheme() Theme`
+
+The built-in themes: `DarkTheme` is the default and suits dark backgrounds, `LightTheme` suits light backgrounds, and `MonoTheme` uses only bold, italic and underline. Each call returns a fresh copy that the caller may modify.
+
+### `ThemeByName(name string) (Theme, bool)`
+
+Returns a fresh copy of the built-in theme with the given case-insensitive name. An empty name returns `DarkTheme`.
+
+### `ThemeNames() []string`
+
+Lists the built-in theme names: `dark`, `light` and `mono`.
+
+## High-Level Helpers
+
+The `gosqlx` package wraps the common cases:
+
+```go
+func Highlight(sql string, opts ...highlight.Option) []highlight.Span // same as highlight.Spans
+func HighlightANSI(sql string, opts ...highlight.Option) string // DarkTheme, Color256
+func HighlightHTML(sql string, opts ...highlight.Option) string // class-based, wrapped in
+```
+
+For another theme, true color, inline styles or a custom class prefix, call `highlight.ANSI` or `highlight.HTML` with the spans directly.
diff --git a/docs/api/linter.md b/docs/api/linter.md
index f7de8253..320275cb 100644
--- a/docs/api/linter.md
+++ b/docs/api/linter.md
@@ -13,7 +13,7 @@ The linter package provides SQL linting with 30 built-in rules (L001-L030) and a
### `Linter`
```go
type Linter struct {
- rules []Rule
+ // unexported fields: rules, ignore matcher, dialect
}
```
@@ -86,6 +86,21 @@ linter := linter.New(
)
```
+### `(l *Linter) SetDialect(dialect keywords.SQLDialect)`
+Select the SQL dialect used when tokenizing and parsing SQL for the rules that inspect tokens or the AST. An empty string or `keywords.DialectUnknown` restores the default dialect (PostgreSQL). Call it before linting; it is not synchronised with concurrent `Lint*` calls on the same `Linter`.
+
+This affects dialect-specific syntax such as SQL Server `TOP` and `[bracketed]` identifiers, the MySQL/SQLite `?` placeholder or MySQL `LIMIT offset, count`.
+
+```go
+import sqlkeywords "github.com/ajitpratap0/GoSQLX/pkg/sql/keywords"
+
+l := linter.New(rules...)
+l.SetDialect(sqlkeywords.DialectSQLServer)
+result := l.LintString("SELECT TOP 10 [id] FROM orders", "query.sql")
+```
+
+The CLI sets it from `gosqlx lint --dialect`.
+
### `(l *Linter) LintFile(filename string) FileResult`
Lint a single SQL file.
@@ -130,6 +145,21 @@ Format violation for display.
### `FormatResult(result Result) string`
Format linting results for display.
+## Linting With a Dialect From gosqlx
+
+### `gosqlx.LintWithDialect(sql string, dialect keywords.SQLDialect) (*gosqlx.LintResult, error)`
+Like `gosqlx.Lint`, which runs all 30 built-in rules, but tokenizes and parses the SQL with the given dialect, so the AST-based rules (L011-L030) also run on dialect-specific syntax such as SQL Server's `TOP` or `[bracketed]` identifiers. An empty dialect behaves exactly like `gosqlx.Lint`. An unknown dialect returns an error wrapping `gosqlx.ErrUnsupportedDialect`, as `gosqlx.ParseWithDialect` does. Safe for concurrent use.
+
+```go
+result, err := gosqlx.LintWithDialect("SELECT TOP 10 [id] FROM orders", sqlkeywords.DialectSQLServer)
+if err != nil {
+ log.Fatal(err)
+}
+for _, v := range result.Violations {
+ fmt.Println(linter.FormatViolation(v))
+}
+```
+
## Built-in Rules
All 30 rules are listed below. The [Linting Rules reference](../LINTING_RULES.md) documents each one with examples.
diff --git a/docs/api/lsp.md b/docs/api/lsp.md
index a7045d73..f84cdf89 100644
--- a/docs/api/lsp.md
+++ b/docs/api/lsp.md
@@ -232,6 +232,58 @@ Parameters:
// Quick fix: Convert 'select' to 'SELECT'
```
+### **9. Semantic Tokens**
+- `textDocument/semanticTokens/full` - Syntax highlighting computed by the server
+
+The server classifies the document with [pkg/highlight](highlight.md) under the configured dialect and returns the standard LSP relative encoding. Only full-document requests are supported (`full: true`, `range: false`). Invalid SQL is still highlighted up to the first lexical error; an unknown document returns an empty token stream.
+
+The legend uses only standard LSP token types and modifiers, so editor themes color them without extra configuration:
+
+| `pkg/highlight` category | Token type | Modifiers |
+|---|---|---|
+| `Keyword` | `keyword` | |
+| `Function` | `function` | |
+| `DataType` | `type` | |
+| `Identifier`, `QuotedIdentifier` | `variable` | |
+| `String` | `string` | |
+| `Number` | `number` | |
+| `Constant` (`TRUE`, `FALSE`, `NULL`) | `keyword` | `readonly` |
+| `Operator` | `operator` | |
+| `Comment` | `comment` | |
+| `Placeholder` | `parameter` | |
+
+The advertised legend is `tokenTypes: ["keyword", "function", "type", "variable", "string", "number", "operator", "comment", "parameter"]` and `tokenModifiers: ["readonly"]`. Punctuation is not reported (there is no standard token type for it), and lexical errors are reported as diagnostics instead. Positions are 0-based lines and UTF-16 characters, and a token that spans several lines (a block comment or multi-line string) is split into one token per line.
+
+## SQL Dialect
+
+One dialect drives diagnostics, formatting, document symbols, completion, hover and semantic tokens. The default is `generic`. Accepted values are the 11 names from `keywords.AllDialects()`: `generic`, `postgresql`, `mysql`, `mariadb`, `sqlserver`, `oracle`, `sqlite`, `snowflake`, `bigquery`, `redshift` and `clickhouse`. Values are trimmed and lower-cased; a missing, non-string, empty or unrecognised value selects `generic`.
+
+### At startup: `initializationOptions`
+
+```json
+{
+ "method": "initialize",
+ "params": {
+ "initializationOptions": { "dialect": "mysql" }
+ }
+}
+```
+
+### At runtime: `workspace/didChangeConfiguration`
+
+```json
+{
+ "method": "workspace/didChangeConfiguration",
+ "params": {
+ "settings": { "gosqlx": { "dialect": "sqlserver" } }
+ }
+}
+```
+
+The server reads only `settings.gosqlx.dialect`. Settings without that entry (for example the `null` settings some clients send) keep the current dialect; a present but invalid value selects `generic`, exactly as at startup. When the dialect actually changes, the server re-publishes diagnostics for every open document and, if the client advertised `workspace.semanticTokens.refreshSupport` in `initialize`, sends `workspace/semanticTokens/refresh` so the client re-requests highlighting. Hover, completion, formatting and document symbols read the dialect on every request.
+
+The VS Code extension sends its `gosqlx.dialect` setting as `initializationOptions.dialect`.
+
## LSP Usage
### **Starting the LSP Server**
diff --git a/pkg/models/location.go b/pkg/models/location.go
index e60d6dcf..9e329278 100644
--- a/pkg/models/location.go
+++ b/pkg/models/location.go
@@ -21,12 +21,14 @@ package models
//
// Fields:
// - Line: Line number in source code (starts at 1)
-// - Column: Column number within the line (starts at 1)
+// - Column: Byte column within the line (starts at 1)
//
// Example:
//
// loc := models.Location{Line: 5, Column: 20}
-// // Represents position: line 5, column 20 (5th line, 20th character)
+// // Represents line 5, byte column 20: the position 19 bytes after the
+// // start of the 5th line. A multi-byte UTF-8 character before it counts
+// // all of its bytes, so this is not necessarily the 20th character.
//
// Usage in error reporting:
//
@@ -38,10 +40,15 @@ package models
//
// Integration with LSP (Language Server Protocol):
//
-// // Convert to LSP Position (0-based)
+// LSP positions are 0-based and, by default, count UTF-16 code units, so the
+// line converts directly but the column must be re-measured against the line
+// text. Subtracting 1 from Column is only correct for ASCII-only lines:
+//
+// lineText := lines[location.Line-1]
+// prefix := lineText[:location.Column-1] // bytes before the position
// lspPos := lsp.Position{
-// Line: location.Line - 1, // Convert to 0-based
-// Character: location.Column - 1, // Convert to 0-based
+// Line: location.Line - 1,
+// Character: len(utf16.Encode([]rune(prefix))),
// }
//
// Columns produced by the tokenizer count bytes: every byte of the line before
diff --git a/vscode-extension/CHANGELOG.md b/vscode-extension/CHANGELOG.md
index 3e347f97..f742f8d6 100644
--- a/vscode-extension/CHANGELOG.md
+++ b/vscode-extension/CHANGELOG.md
@@ -2,6 +2,31 @@
All notable changes to the "GoSQLX" extension will be documented in this file.
+## [1.15.0] - 2026-07-30
+
+Bundles the GoSQLX v1.15.0 language server. The extension's own settings and
+commands are unchanged. See the
+[main CHANGELOG](https://github.com/ajitpratap0/GoSQLX/blob/main/CHANGELOG.md)
+for the full list.
+
+### Added
+- **ClickHouse at 83/83 of the QA corpus**: `ARRAY JOIN`, scalar CTEs,
+ `CREATE MATERIALIZED VIEW` with `ENGINE`/`TO`, numeric `SAMPLE` with
+ `OFFSET`, and `LIMIT offset, count` no longer produce diagnostics.
+- **Oracle**: `ROWNUM`/`ROWID`/`LEVEL`/`SYSDATE` pseudo-columns and
+ `PIVOT`/`UNPIVOT` `IN` lists with `AS` aliases parse.
+- Named `WINDOW` clauses and `EXTRACT(field FROM expr)` parse.
+
+### Fixed
+- **Document symbols** report each statement's real range instead of line 0
+ to the end of the file.
+- 37 more parser errors carry an error code and a source position, so their
+ diagnostics show a code and point at the offending token.
+
+### Security
+- `brace-expansion` pinned to `^5.0.8` in the extension's lockfile (build
+ tooling).
+
## [1.14.0] - 2026-04-12
### Added
diff --git a/website/scripts/check-docs-content.mjs b/website/scripts/check-docs-content.mjs
index f5f56e9f..d4f0741f 100644
--- a/website/scripts/check-docs-content.mjs
+++ b/website/scripts/check-docs-content.mjs
@@ -50,6 +50,8 @@ const LINE_RULES = [
name: 'per-release heading label',
re: /^#{1,6}\s+(?:v\d+\.\d+(?:\.\d+)?\b|.*(\(\s*[^)]*\bv\d+\.\d+(\.\d+)?\+?[^)]*\)|\bNEW v\d+\.\d+|\bNew in v\d|\bAdded in v\d|\bin v\d+\.\d+(\.\d+)?\s*$))/i,
hint: 'evergreen docs should not label headings with a release',
+ // The upgrade guide is organised by release on purpose ("## v1.13.0 → v1.14.0").
+ skip: (file) => file.endsWith(`${join('docs', 'MIGRATION.md')}`),
},
{
name: 'bold per-release callout',
diff --git a/website/src/lib/constants.ts b/website/src/lib/constants.ts
index e4cd6572..f8cee569 100644
--- a/website/src/lib/constants.ts
+++ b/website/src/lib/constants.ts
@@ -60,6 +60,7 @@ export const DOCS_SIDEBAR: DocsSidebarCategory[] = [
{ slug: 'guides/production', file: 'PRODUCTION_GUIDE.md', label: 'Run in Production', description: 'Run GoSQLX in production: sizing object pools, concurrency and race safety, metrics and monitoring, error handling, timeouts and deployment advice.' },
{ slug: 'guides/performance-tuning', file: 'PERFORMANCE_TUNING.md', label: 'Performance Tuning', description: 'Tune GoSQLX for throughput: reuse pooled tokenizers and ASTs, choose the right API, profile hot paths and read benchmark results for your workload.' },
{ slug: 'guides/troubleshooting', file: 'TROUBLESHOOTING.md', label: 'Troubleshooting', description: 'Fix common GoSQLX problems: parse errors, LSP server and linter issues, security scanner surprises, performance and memory problems, with debugging tips.' },
+ { slug: 'guides/syntax-highlighting', file: 'HIGHLIGHTING.md', label: 'Syntax Highlighting', description: 'Highlight SQL with GoSQLX for terminals, HTML pages and editors: the categories, themes, the highlight CLI command, LSP semantic tokens and the WASM build.' },
]},
{ category: 'Integrations', summary: 'Editors, language server and AI assistants.', items: [
{ slug: 'integrations/vscode', file: 'editors/vscode.md', label: 'VS Code', description: 'Install the GoSQLX VS Code extension for SQL diagnostics, formatting, completion and hover, and configure the language server for your project.' },
@@ -67,6 +68,7 @@ export const DOCS_SIDEBAR: DocsSidebarCategory[] = [
{ slug: 'integrations/jetbrains', file: 'editors/jetbrains.md', label: 'JetBrains', description: 'Connect JetBrains IDEs such as GoLand, IntelliJ and DataGrip to the GoSQLX language server for SQL diagnostics, formatting and completion, step by step.' },
{ slug: 'integrations/lsp', file: 'LSP_GUIDE.md', label: 'Language Server (LSP)', description: 'Set up the GoSQLX Language Server for diagnostics, completion, hover, formatting and symbols in VS Code, Neovim, JetBrains and other LSP editors.' },
{ slug: 'integrations/mcp', file: 'MCP_GUIDE.md', label: 'MCP Server', description: 'Expose GoSQLX to AI assistants through the Model Context Protocol: available tools for parsing, validating, formatting and analyzing SQL, plus setup steps.' },
+ { slug: 'integrations/chatgpt', file: 'CHATGPT_APP.md', label: 'ChatGPT App', description: 'How gosqlx-mcp works as a ChatGPT app and in MCP Apps hosts: tool and resource metadata, the analyze_sql report widget, structured content and testing.' },
]},
{ category: 'Reference', summary: 'API, CLI configuration, errors, rules and dialects.', items: [
{ slug: 'reference/api', file: 'API_REFERENCE.md', label: 'API Reference', description: 'GoSQLX API reference index: every package from the high-level gosqlx API to the tokenizer, parser and AST, with pooling advice and a complete example.' },
@@ -79,6 +81,7 @@ export const DOCS_SIDEBAR: DocsSidebarCategory[] = [
{ slug: 'reference/api/errors', file: 'api/errors.md', label: 'errors package', depth: 1, description: 'Reference for the GoSQLX errors package: structured errors with codes, source positions, hints and help links, and helpers to build and inspect them.' },
{ slug: 'reference/api/metrics', file: 'api/metrics.md', label: 'metrics package', depth: 1, description: 'Reference for the GoSQLX metrics package: enable, record and read counters and timings for tokenizer and parser operations in production services.' },
{ slug: 'reference/api/security', file: 'api/security.md', label: 'security package', depth: 1, description: 'Reference for the GoSQLX security package: the SQL injection scanner, its finding and result types, severity levels and every detection pattern.' },
+ { slug: 'reference/api/highlight', file: 'api/highlight.md', label: 'highlight package', depth: 1, description: 'Reference for the GoSQLX highlight package: classify SQL into byte-offset spans by category, then render them as ANSI, HTML or CSS with built-in themes.' },
{ slug: 'reference/api/linter', file: 'api/linter.md', label: 'linter package', depth: 1, description: 'Reference for the GoSQLX linter package: the lint engine, the Rule interface, violations, auto-fix support and how to build a linter with your rules.' },
{ slug: 'reference/api/lsp', file: 'api/lsp.md', label: 'lsp package', depth: 1, description: 'Reference for the GoSQLX lsp package: the Language Server Protocol server, its handlers and capabilities, and how to embed it in your own tool.' },
{ slug: 'reference/api/mcp', file: 'api/mcp.md', label: 'mcp package', depth: 1, description: 'Reference for the GoSQLX mcp package: the Model Context Protocol server, its configuration and the SQL tools it registers for AI assistants.' },
@@ -92,7 +95,8 @@ export const DOCS_SIDEBAR: DocsSidebarCategory[] = [
{ slug: 'concepts/architecture', file: 'ARCHITECTURE.md', label: 'Architecture', description: 'How GoSQLX is built: the zero-copy tokenizer, recursive descent parser, AST and visitor pattern, object pools and package dependencies, explained.' },
{ slug: 'concepts/security', file: 'SECURITY.md', label: 'Security Model', description: 'The GoSQLX security model: test results, architecture, CLI input sanitization, best practices for users, hardening, and the SQL injection scanner.' },
]},
- { category: 'Migration', summary: 'Move to GoSQLX from other SQL parsers and linters.', items: [
+ { category: 'Migration', summary: 'Upgrade between GoSQLX releases, or move to GoSQLX from other SQL parsers and linters.', items: [
+ { slug: 'migration/upgrading', file: 'MIGRATION.md', label: 'Upgrading GoSQLX', description: 'Upgrade GoSQLX between releases: new APIs worth adopting, behavior changes to check, deprecations and companion component versions, release by release.' },
{ slug: 'migration/from-jsqlparser', file: 'migration/FROM_JSQLPARSER.md', label: 'From JSQLParser', description: 'Migrate from JSqlParser to GoSQLX: how Java parsing concepts map to Go APIs, equivalent AST nodes, visitor usage and a step-by-step porting checklist.' },
{ slug: 'migration/from-pg-query', file: 'migration/FROM_PG_QUERY.md', label: 'From pg_query', description: 'Migrate from pg_query_go to GoSQLX: swap the cgo dependency for a pure Go parser, map AST nodes and functions, and handle multiple SQL dialects.' },
{ slug: 'migration/from-sqlfluff', file: 'migration/FROM_SQLFLUFF.md', label: 'From SQLFluff', description: 'Migrate from SQLFluff to GoSQLX: map lint rules and configuration to GoSQLX equivalents, compare speed and dialects, and move CI checks over.' },