perf: repo-wide optimization audit (2026-10-11) — tracking issue #566

Activity

1. ajitpratap0

   added

   performance

   optimization

   P1High priorityHigh priority

   on Oct 10, 2026
2. ajitpratap0 commented on Oct 11, 2026

   

   ajitpratap0

   on Oct 11, 2026

   OwnerAuthor

   More actions

   Fix status (2026-10-11)

   Fixes were implemented by a multi-agent workflow (Fable plans and validates; Opus/Sonnet implement and review). Every PR below passed the full race gate (go test -race on all packages except cbinding), go vet, golangci-lint, an independent code review, and a benchstat comparison; branches deliberately do not touch CHANGELOG.md or performance_baselines.json so they merge without conflicts. One consolidated CHANGELOG/baselines PR should follow once these land.

   PR	Issues	Headline (allocs/op or time, measured)
   #577	#544 #545 #547 #549	tokenizer: 1 MB input 1.75 s -> 22 ms; -25% allocs; identifier scan -18% ns/op; -45% B/op on 10 KB+ inputs
   #578	#551 #553 #554	parser: ComplexSelect 46 -> 22 allocs; qualified refs -1 alloc each; paginated join 37 -> 26 allocs
   #581	#555 #556	ast: double-Put and stale-reset defects fixed; PutExpression after a 20k-element list 2995 ns -> 19 ns; ReleaseAST B/op -90%
   #575	#558	ast SQL(): 15-line query 122 -> 1 allocs, ~3.5x faster
   #569	#559	SplitStatements 855 -> 1 allocs
   #571	#560	release hygiene: balanced pool accounting in batch/transform/Transpile
   #572	#561	ExtractMetadata 114 -> 24 allocs, ~2.5x faster
   #568	#562	gosqlx.Lint -30 allocs/call; L015 96 KB copy removed
   #574	#563	security scanner: clean-query scan 28.6 us -> 7.0 us; extra Opus security review, zero finding differences over 400k inputs
   #573	#564	LSP: 10k-line diagnostics 789 MiB / 512 ms -> 1.2 MiB / 7 ms
   #570	#565	schema loaders: 3N+1 -> 4 queries per load; MaxOpenConns=1 deadlock and in-memory SQLite pitfall fixed

   Deferred by maintainer decision (not started): #546, #548, #550 (the tokenizer-2 chain, stacked on #577), #552, #557. Branch perf/tokenizer-zero-copy-and-dialect exists at the #577 head with no own commits.

   Note: external PR #576 (fold identifier case without per-word allocation) overlaps #545, which is included in #577; one of the two should be preferred.
3. ajitpratap0

   added a commit that references this issue on Oct 13, 2026

   docs(changelog): restore CHANGELOG.md emptied by the merge in #526 (#593

   ...

   0a2b8fb
4. ajitpratap0 commented on Oct 13, 2026

   

   ajitpratap0

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Status update 2026-10-13 (batch 2)

   Batch 1 (#544 #545 #547 #549 #551 #553 #554 #555 #556 #558 #559 #560 #561 #562 #563 #564 #565) is fully merged (PRs #568–#575, #577, #578, #581).

   Batch 2 PRs, each with an independent Opus review and validation (full race gate, vet, lint, benchstat A/B before/after, golden/differential tests where output equivalence matters):

   • fix(schema): correct foreign-key join keys in the Postgres and MySQL loaders (#579) #582 — schema: foreign-key join keys in the Postgres and MySQL loaders (follow-up to #570) #579 schema FK join keys: OPEN
   • fix(cli): render ClickHouse parametric aggregate parameters in gosqlx format (#531) #583 — [BUG] ClickHouse parametric aggregates drop params group on render (quantile(0.5)(x) → quantile(x)) #531 ClickHouse parametric render: OPEN
   • fix(parser): accept WITH ... SELECT as the source of INSERT and keep leading CTEs in gosqlx format (#537) #584 — [BUG] insert with CTE #537 INSERT ... WITH CTE: OPEN
   • fix(parser): support three-part identifiers and parenthesized join groups (#521) #585 — [BUG] Valid MySQL query fails on three-part identifiers and parenthesized joins #521 three-part identifiers / parenthesised joins: OPEN
   • perf(parser): reuse a parser-owned preprocess buffer and the pooled Statements slice (#552) #586 — perf(parser): reuse a parser-owned preprocess buffer and the pooled AST Statements slice #552 parser preprocess buffer: OPEN
   • perf(ast): stop heap-copying nodes in Children() so Walk/Inspect allocate less (#557) #588 — perf(ast): stop heap-copying nodes in Children() so Walk/Inspect allocate less #557 AST Children() no-copy (rebased onto main after perf(ast): fix release-path defects and stop ReleaseAST pooling node types the parser never draws (#555, #556) #581): OPEN
   • perf(tokenizer): Oracle bind variables, substring token values, zero-copy quoted literals, dialect pooling (#520, #546, #548, #550) #595 — [BUG] Does not support Oracle placeholders like :1, :2, :name. #520 perf(tokenizer): eliminate per-token Word and string-copy allocations (nil Word for bare words, substring token values) #546 perf(tokenizer): fast-path escape-free quoted identifiers/strings and remove the unused pooled buffer from Tokenize #548 perf(tokenizer): stop rebuilding keyword maps on every dialect-aware call; pool dialect tokenizers and parsers #550 tokenizer chain: OPEN
   • docs(changelog): restore CHANGELOG.md emptied by the merge in #526 #593 — CHANGELOG.md restore (emptied by feat(cli): add --dialect flag to parse, format, analyze, lint #526 merge): MERGED
   • chore: add .gitattributes enforcing LF line endings #594 — .gitattributes LF policy (Windows CI golden fixtures): OPEN

   #595 is the largest: #520 (Oracle :1/:name binds), #546 (substring token values, slab Words, TokenizeString), #548 (zero-copy escape-free quoted literals, 400+ → 2 allocs on quote-heavy input), #550 (drop the never-read keyword table; dialect-aware entry points now cost the same as the default path, e.g. ParseBytesWithDialect 70 → 28 allocs). It also drops the token slice in Parser.Reset so a pooled parser no longer retains the previous caller's SQL string.

   Side findings while doing this work:

   • New issues filed: [BUG] gosqlx format panics (nil pointer) on NOT EXISTS; CLI formatter lacks ExistsExpression case #587 (gosqlx format nil-pointer panic on NOT EXISTS), [BUG] SQL() and formatter drop numeric window-frame bounds (ROWS BETWEEN 1 PRECEDING ... renders as ROWS BETWEEN PRECEDING ...) #589 (SQL()/formatter drop numeric window-frame bounds), [BUG] Pooled parser (GetParser) reports different Capabilities() than NewParser() for the default dialect #590 (GetParser() capabilities are zero while NewParser() is permissive).
   • docs(changelog): restore CHANGELOG.md emptied by the merge in #526 #593: PR feat(cli): add --dialect flag to parse, format, analyze, lint #526's merge had truncated CHANGELOG.md to zero bytes; restored (merged).
   • chore: add .gitattributes enforcing LF line endings #594: Windows CI checks golden fixtures out with CRLF; .gitattributes now enforces LF (tests in perf(ast): stop heap-copying nodes in Children() so Walk/Inspect allocate less (#557) #588/perf(tokenizer): Oracle bind variables, substring token values, zero-copy quoted literals, dialect pooling (#520, #546, #548, #550) #595 also normalize as belt and braces).
   • External PR perf(tokenizer): fold identifier case without per-word allocation #576 overlaps perf(tokenizer): replace per-identifier strings.ToUpper with allocation-free ASCII upper-casing #545 (already merged in perf(tokenizer): linear position tracking, ASCII fast paths, counted token-slice capacity (#544, #545, #547, #549) #577); needs a maintainer decision.

   Still owed: one consolidated PR updating CHANGELOG.md and performance_baselines.json for all merged perf PRs (branches deliberately did not touch either).
5. ajitpratap0 commented on Oct 13, 2026

   

   ajitpratap0

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Status update (2026-10-13): merge train complete

   All open PRs from this effort are merged. Each was rebased onto the current main, passed the full race test suite locally, went green on every CI check (including Vercel), and had its review comments answered. Each was then squash-merged.

   Merged in this train

   PR	Change
   #594	.gitattributes with LF line endings, fixing Windows CRLF golden failures
   #588	Visitors get pointers into the AST (#557)
   #582	Schema foreign-key join keys (#579)
   #586	Parser preprocess buffer reuse (#552)
   #595	Tokenizer chain: zero-copy tokens, Oracle bind variables, keyword table (#520, #546, #548, #550)
   #585	MySQL three-part identifiers and parenthesized join groups (#521)
   #591	CLI NOT EXISTS nil dereference (contributor PR)
   #592	Numeric window-frame offsets (#589, contributor PR)
   #584	INSERT ... WITH (#537); gosqlx format keeps leading CTEs on INSERT, UPDATE and DELETE
   #583	ClickHouse parametric aggregate params in gosqlx format (#531)
   #597	Hardening for adversarial input (details below)

   #597 contains:

   • iterative comment skipping
   • comments counted toward MaxTokens
   • the E2007 depth limit extended to unary operators, NOT and derived tables
   • linear-time set-operation rendering
   • $name placeholders keeping their text

   Consolidated docs: #598 adds CHANGELOG [Unreleased] entries for everything since v1.14.0, starting with a Behavior changes section, and refreshes performance_baselines.json.

   Follow-up issues found during review were filed as #599–#620. Each was reproduced on main 61ca6c8, and none duplicates an existing issue.

   • CLI and formatters
     • cli: gosqlx format silently corrupts many expressions; switch the CLI to pkg/formatter #599: umbrella issue; move gosqlx format onto pkg/formatter
     • ast/formatter: SQL() and pkg/formatter drop IGNORE NULLS, named arguments, ON DUPLICATE KEY UPDATE and INSERT OUTPUT; parser discards SELECT ... SETTINGS #600, ast/formatter: NOT EXISTS (subquery) renders as NOT (EXISTS (...) NOT ), which does not re-parse #601, formatter: readable style emits ; inside CTE/IN/EXISTS subqueries and set operations, and ;; after INSERT ... SELECT / CREATE VIEW #602, parser/ast: JOIN ... USING (cols) is stored as a plain condition and renders as ON cols #603, ast/formatter: SQL Server table hints WITH (NOLOCK) are never rendered; SQL() also drops PIVOT #604, ast/formatter: SQL() and FormatAST drop table-function call arguments in FROM (generate_series(1,10) -> generate_series) #607, ast/formatter: named windows render as OVER (w) ... WINDOW w AS (w ...) and do not round-trip; golden file pins the broken output #608, ast/formatter: quoted identifiers that are keywords or case-sensitive lose their quotes on render (SELECT "select", CTE/view column lists, table names) #611
   • Parser gaps: parser/clickhouse: parametric aggregates move DISTINCT to the wrong group and reject chained calls or subscripts #605, parser/clickhouse: table functions in FROM (numbers(10), remote(...)) fail to parse #606, parser: window frames reject GROUPS, EXCLUDE and INTERVAL '1' DAY bounds; ast doc example uses Type:"2 PRECEDING" #609, parser: INSERT gaps: RETURNING after INSERT ... SELECT, DEFAULT VALUES, MySQL INSERT ... SET, INSERT INTO t (SELECT ...) #610, parser: parenthesized set operations not accepted as derived tables or statements; ((SELECT ...)) t gives a misleading join-group error #612
   • Performance and errors: parser: parseQualifiedName concatenates dotted table names quadratically #613, tokenizer: pooled Tokenizer keeps lineStarts capacity of the largest input it has seen #614, errors: error string grows with the error column on long single-line input (full source line plus caret padding) #615, parser: several E2002 errors pass hint text as the SQL source, printing a bogus context snippet #616
   • AST API and nondeterminism: ast: CreateTable, FunctionDesc, TriggerExecBody and TriggerReferencing return child nodes by value from Children() #617, linter: L006 and text-based L009 results depend on map iteration order #618, gosqlx: ExtractMetadata and Extract* return tables/columns/functions in random map order #619
   • Schema/postgres loader: schema/postgres: loader misreports index columns and key flags (column order, INCLUDE, expression and partitioned indexes, read-only roles) #620

   Also open from earlier: #596 (arithmetic on the right-hand side of a comparison).
6. ajitpratap0

   added 2 commits that reference this issue on Oct 14, 2026

   docs: consolidate CHANGELOG for the perf audit and batch-2 fixes; ref…

   ...

   86e1f72

   docs: consolidate CHANGELOG for the perf audit and batch-2/batch-3 fi…

   ...

   d7338ce
7. ajitpratap0 commented on Oct 14, 2026

   

   ajitpratap0

   on Oct 14, 2026

   OwnerAuthor

   More actions

   Batch 3 (#599–#620) is complete

   All 22 issues are fixed in 12 PRs, merged through one merge train. Every PR got an Opus adversarial and security review before merge, and the fix commits were independently validated by Opus.

   PR	Issues	Merge commit
   #621	#617	e1d54df
   #625	#618, #619	c35349f
   #626	#614, #615	c1f69d5
   #622	#620	41a5035
   #629	#608, #609	82982e5
   #623	#610	3baad9d
   #628	#605, #606, #607	1cd8f78
   #630	#603, #612	458b3c3
   #631	#602, #600	1be0808
   #627	#613, #616	95978a4
   #632	#601, #611, #604	728c544
   #624	#599	baf528e

   The CHANGELOG and refreshed baselines are in #598 (merged as d7338ce).

   Security: the review of #632 found render-time SQL injection, and main had it too. A quoted alias such as "a; DROP TABLE x; --" came back from gosqlx.Format and SQL() as two statements. #632 closes the whole class:

   • every quoted-name position is now quoted when needed;
   • the tokenizer treats smart quotes inside ASCII-quoted tokens as ordinary characters;
   • keyword fields are checked against exact allow-lists.

   It was validated with hostile-name fuzzing (549k cases) and AST-mutation fuzzing: 0 injected statements and 0 injectable paths.

   Follow-ups found during batch 3, not yet filed

   Per the earlier instruction not to add new issues for now, these are recorded here rather than filed. Each one is already on main.

   Rendering and round-trip, all fail-safe: the output does not re-parse, or the CLI refuses it

   • SQL Server CROSS APPLY / OUTER APPLY render as CROSS APPLY JOIN …, and TOP is lost in the derived table.
   • MySQL MATCH (…) AGAINST (…) renders as AGAINST AGAINST(…).
   • Oracle START WITH / CONNECT BY and QUALIFY are not rendered.
   • ALTER TABLE … ADD CONSTRAINT renders as an empty string from SQL() and as ALTER from the formatter. ALTER ROLE / POLICY / CONNECTOR are not rendered.
   • ALTER TABLE … ADD COLUMN IF NOT EXISTS / DROP COLUMN IF EXISTS are not parsed. The AST fields exist.
   • ClickHouse: SQL() drops ORDER BY after ENGINE = MergeTree(), the formatter drops ARRAY JOIN, and INTO OUTFILE after SETTINGS is not rendered.
   • CREATE TABLE t AS SELECT … renders as CREATE TABLE t (), and Snowflake CREATE SEQUENCE renders as an empty string. Snowflake CREATE STAGE RawSQL is lost.
   • fn()(x) rendering, JOIN-side table functions, and INSERT … SET t.a / = DEFAULT.
   • Dialect-specific INTERVAL rendering, and the MySQL-only SET form when transpiling.

   Quoting and dialects

   • Quoting is dialect-agnostic: ANSI "…" everywhere. MySQL output needs backticks unless ANSI_QUOTES is set. The CLI now refuses rather than emit wrong MySQL.
   • A name quoted only for its case ("MixedCase") renders bare, because the AST does not record quoting for aliases or column lists. Recording Quoted everywhere would let case-sensitive names round-trip.
   • In the default dialect, "#x" renders bare and does not re-parse.
   • Backslash escaping is not per dialect: 'a\b' becomes 'a\\b' in SQL Server, Oracle, SQLite, Snowflake and ClickHouse. The MySQL tokenizer does not handle \" inside double-quoted strings.
   • Lowercase any(...) turns into ALL.
   • Window frame-type casing and dialect gating.

   Security scanner

   • It does not descend into derived tables, CTEs or comma joins.
   • The NULL-column UNION check never fires.

   API, schema and linter

   • cfg.Rules warning order in the linter config.
   • Partitioned and invalid index metadata, and an expression-key marker in the Postgres schema loader.
   • CreateTableBuilder.Build() shares Columns/Constraints storage between built statements. Since ast: CreateTable, FunctionDesc, TriggerExecBody and TriggerReferencing return child nodes by value from Children() #617, Inspect edits propagate across them.
   • TableReference padding: +16 B/op from parser: parenthesized set operations not accepted as derived tables or statements; ((SELECT ...)) t gives a misleading join-group error #612. Identifier grew from 48 to 64 B.
   • The ClickHouse ORDER BY error caret position.

   CI

   • TestFormatAST_SetOperationChainLinearTime uses a 2 s wall-clock limit. It failed once on a starved windows-latest runner (4.6 s) alongside unrelated lsp/security timeouts. A rerun passed. It should use a relative or operation-count bound.
   • Dependabot [Parser] Add DISTINCT keyword support #142 (website sprintf-js) is untouched.
8. ajitpratap0 commented on Oct 18, 2026

   

   ajitpratap0

   on Oct 18, 2026

   OwnerAuthor

   More actions

   Status update for the issues tracked under this one:

   • parser: arithmetic on the right-hand side of a comparison fails (WHERE a = 1 + 2) #596: closed via PR fix(parser): parse arithmetic on the right-hand side of comparisons (#596) #734, squash-merged 2026-10-18T10:53:13Z, branch deleted. All required checks passed. The Vercel check reported "Canceled by Ignored Build Step", which confirms the new ignoreCommand in website/vercel.json works. No admin merge, force-push, or CHANGELOG edit.
   • [BUG] Pooled parser (GetParser) reports different Capabilities() than NewParser() for the default dialect #590: closed via PR fix(parser): make pooled parser capabilities match NewParser (#590) #735, squash-merged 2026-10-18T11:03:38Z, branch deleted. The PR was behind main and was updated with gh pr update-branch (new head 791e502), after which all checks passed. The Vercel check showed the ignored-build success on both the original and updated heads. No admin merge, force-push, or CHANGELOG edit.

   Issue #566 stays open.
9. ajitpratap0 commented on Oct 18, 2026

   

   ajitpratap0

   on Oct 18, 2026

   OwnerAuthor

   More actions

   Follow-up outcomes for #781-#784 (found while fixing #596/#590):

   • parser(mysql): '||' parsed as string concat, but MySQL's default meaning is logical OR #781: merged via PR fix(parser): parse || as logical OR in MySQL and MariaDB (#781) #790 (squash, branch deleted). Needed gh pr update-branch twice to catch up with main. Merged while Vercel was still deploying on the final head; not re-checked after merge.
   • ast(snowflake): VariantPath.SQL() renders a bare ':' so round-trip fails #782: merged via PR fix(ast): render Snowflake VariantPath in SQL() and the formatter #786 (squash, branch deleted) after gh pr update-branch. All checks passed, Vercel passed.
   • parser(oracle): CONNECT BY condition rejects arithmetic (PRIOR id = parent_id + 1) #783: merged via PR fix(parser): accept general expressions in CONNECT BY (PRIOR as unary operator) #791 (squash, branch deleted). Resolved a parser.go Parser-struct conflict by keeping both sides. Limited one test's concat expectation to Oracle, since main now makes || logical OR in MariaDB by default. Merge commit 6c0cbe5 pushed to the PR branch; all checks passed.
   • parser: zero-value &parser.Parser{} still reports all-false Capabilities() (doc examples use it) #784: merged via PR fix(parser): make zero-value Parser{} capabilities match NewParser (#784) #789 (squash, branch deleted). Resolved conflicts with the ast(snowflake): VariantPath.SQL() renders a bare ':' so round-trip fails #782 pipes-as-OR work by keeping both sides. Merge commit f398f3a pushed; all checks passed.

   All four issues are closed. No admin merges, force-pushes, or CHANGELOG edits were made, and no scratch worktrees remain from this work.