perf(tokenizer): zero-copy @param and @@var token values [W2-03] by ajitpratap0 · Pull Request #925 · ajitpratap0/GoSQLX

ajitpratap0 · 2026-11-02T10:10:52Z

Summary

W2-03 (TOK-7): @param and SQL Server @@var token values are now t.src[atStart:pos] slices instead of "@"+identToken.Value concatenations, and the name is scanned with scanIdentifierTail directly (no throwaway Word, no compound lookahead). Names that readIdentifier treats specially (U&" escapes, single letter before a quote, compound-keyword starts like @group by, non-ASCII names) fall back to the previous path with the position restored, so output is identical. No API change; no golden diff.

Test Plan

TestAtParam_Golden (new, testdata/at_param_golden.txt): 41 inputs x 9 dialects, types/values/quote/spans and error code+position. Now also pins the non-ASCII guard (@rıght join x, @ınner join x, @groupıng sets, @@rıght join x: the old path joins these into one Placeholder) and keyword-named params (@select, @From, @Where). The golden was regenerated on origin/main's tokenizer.go and passes on this branch, so any diff is a behaviour change.
TestAtParam_ValueIsSourceSlice now checks that values alias the TokenizeString input (unsafe.StringData range check).
TestAtParam_NoPerTokenAlloc: fails before (71 allocs), passes after.
task check, task test:cbinding, go test -count=1 ./pkg/compatibility/..., staticcheck on ./pkg/sql/tokenizer: all pass.
Local task bench:ab BASE=origin/main BENCH='./pkg/sql/tokenizer=^BenchmarkTokenizeRW$' (shared M4, indicative): sec/op geomean -0.28%, all cells ~ (no significant time change); allocs/op: sqlserver small 4 -> 0, medium 8 -> 0, large 162 -> 9 (-94%); mariadb small 2 -> 0, large -64%; mysql large -52%; snowflake large -67%; oracle large -31%; sqlite large -15%. B/op: sqlserver small 33 -> 1, medium 71 -> 7 (-90%). No regressions.
CI perf-gate (shadow), latest run 41431883548 (head cbbee20d, merge 907178d, base f03332f): PASS, no gated metric moved past its threshold; A/A control 0 of 32 moved >3%. geomean -0.18% (p=0.029). ParseRW/sqlserver/medium -0.69% (p<0.001), allocs/op 936 -> 928 (-allocs), B/op -0.07%. Normalize/simple_select is now -0.62%, confirming the earlier +3.34% flag on run 41429649507 was code layout noise. Other target cells are flat on time: mysql -0.19%, mariadb +0.42%, snowflake +0.20% (none significant). All gated allocs/op unchanged or lower, B/op unchanged.
- Plan threshold note: the PLAN time target (<= -2% on sqlserver/mysql/mariadb/snowflake) is NOT met on amd64; the allocation target (TokenizeRW sqlserver allocs <= -80%) is met locally (-90% to -100%) but TokenizeRW is not gated. C0.md section 3 re-scoped W2-03 to "RW allocs only", so this PR asks for allocation-only acceptance.

Breaking Changes

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

vercel · 2026-11-02T10:10:56Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
gosqlx-website	Ready	Preview	Nov 2, 2026 1:57pm UTC

github-actions · 2026-11-02T10:11:15Z

Claude finished @ajitpratap0's task in 29s —— View job

Review of PR #925: zero-copy `@param` / `@@var` token values

Verdict: looks good, with a few non-blocking notes. I read the diff but didn't run the tests. The sandbox denied go test, so I haven't checked the golden or the alloc test myself. Please rely on the CI run for those.

Read the diff and tests
Checked scanAtName against the readIdentifier special cases
Ran the tests (not possible here: command approval was denied)

Correctness

Fallback conditions. scanAtName returns false with the position restored for the cases readIdentifier treats specially:
- a non-ASCII start or non-ASCII name;
- U&' and U&" escapes;
- a single letter before ';
- a compound-keyword start.
In each case the old path runs unchanged, so the zero-copy path only handles plain ASCII names. I found no behaviour difference.
Position restore. saved := t.pos is a value copy, so t.pos = saved restores the position fully, assuming Position holds only plain value fields. That includes line and column state.
Slice source. t.src[atStart:t.pos.Index] is only valid if t.src and t.input are the same bytes with matching offsets. The invariant comment at tokenizer.go:586 suggests they are. The golden also covers this, since it would catch BOM or offset drift.
Single-letter check. The len(ident)==1 && next=='\'' guard matches @N'x'-style inputs. The golden covers @N, @n, @X, @B, @E and @q.
Compound keywords. Dialect-specific keywords that are not compound starts, such as @select, get the @name value from the source text. This is the same as the old "@"+Value unless readIdentifier normalised the case of keyword values. The golden's @@ROWCOUNT and @group cases suggest it doesn't. I'd still add a lowercase non-compound keyword case, such as @select, @from or @Where, to the golden to pin this. It is cheap to add.

Performance

The allocation reductions are plausible: one string concat per parameter is removed, and the throwaway Word and compound lookahead are skipped.
kbuf is a fixed stack array and asciiUpperHash writes into it, so the new path shouldn't allocate.
Retention: token values now alias the whole input string. This is already true for other zero-copy tokens, so it is not a new class of issue.

Tests

The golden has 36 inputs across 9 dialects. It was generated before the change, which is the right way to do it. Covering edge cases like @@ x, a bare @, @>, @@ (the Postgres text-search operator), BOM-free CRLF input and long names is good.
TestAtParam_NoPerTokenAlloc has a threshold of <= 20 against a roughly 71-alloc baseline. That is a reasonable margin, but it depends on slice-growth behaviour. If it flakes across Go versions, assert against a baseline measured in the same test instead.
Optional additions:
- a BOM-prefixed input, to prove the offsets;
- a pooled-tokenizer reuse case, to prove that slices from an earlier call stay valid after PutTokenizer.

Minor

TestAtParam_ValueIsSourceSlice only checks string equality, so its name overclaims. It can't prove the value is a slice. Rename it, or drop it in favour of the alloc test.
The CHANGELOG entry is accurate. It is long, though, compared with its neighbours.

No security concerns: there is no new unsafe code, and the changes are bounds-checked.

github-actions · 2026-11-02T10:23:29Z

Perf gate: PASS (shadow mode: report only, this check never fails)

Base 98f7489cb0f4, head 238e0d91b3d9. 10 interleaved rounds, benchtime 200ms, -cpu 1, pinned to CPU 3 (SMT siblings 2-3) with taskset.

no gated metric moved past its threshold

Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark	time	p	A/A	B/op	allocs/op	status
`ParseE2E/simple_select`	-0.66%	0.241	+0.35%	+0.00%	5 → 5	ok
`ParseE2E/select_where`	+0.10%	0.570	-0.07%	+0.00%	18 → 18	ok
`ParseE2E/join`	-0.58%	0.247	+0.55%	+0.00%	17 → 17	ok
`ParseE2E/aggregate`	+0.49%	0.052	-0.37%	+0.00%	18 → 18	ok
`ParseE2E/subquery`	-0.23%	0.472	+0.28%	+0.00%	14 → 14	ok
`ParseE2E/cte`	+0.34%	0.096	-0.19%	+0.00%	21 → 21	ok
`ParseE2E/window`	+0.10%	0.820	+1.05%	+0.00%	11 → 11	ok
`ParseE2E/complex`	+1.92%	0.014	+0.69%	+0.00%	50 → 50	ok
`ParseE2E/insert`	+1.72%	0.003	+0.23%	+0.00%	17 → 17	ok
`ParseE2E/update`	+0.88%	0.004	+0.34%	+0.00%	13 → 13	ok
`ParseE2E/delete`	+1.47%	0.023	+0.86%	+0.00%	5 → 5	ok
`ParseRW/postgresql/medium`	+0.76%	0.218	+0.21%	+0.00%	646 → 646	ok
`ParseRW/mysql/medium`	+0.55%	0.015	+0.16%	+0.00%	365 → 365	ok
`ParseRW/mariadb/medium`	+0.64%	0.315	+0.10%	+0.00%	523 → 523	ok
`ParseRW/sqlserver/medium`	+0.45%	0.218	+0.05%	-0.07%	936 → 928	-allocs
`ParseRW/oracle/medium`	+1.60%	0.002	-0.28%	+0.00%	1093 → 1093	ok
`ParseRW/sqlite/medium`	+0.74%	0.015	-0.18%	+0.00%	346 → 346	ok
`ParseRW/snowflake/medium`	+0.25%	0.579	-0.59%	+0.00%	777 → 777	ok
`ParseRW/clickhouse/medium`	-0.25%	0.853	-0.13%	+0.00%	384 → 384	ok
`ParseFresh/simple_select`	+0.53%	0.007	-0.04%	+0.00%	6 → 6	ok
`ParseFresh/join`	+0.39%	0.199	-0.25%	+0.00%	18 → 18	ok
`ParseFresh/complex`	+0.72%	0.070	+0.59%	+0.00%	51 → 51	ok
`PoolGetPut/serial`	+1.38%	0.162	+0.24%	+0.00%	0 → 0	ok
`Tokenize/scratch/join`	-0.62%	0.009	+0.10%	+0.00%	0 → 0	ok
`Tokenize/scratch/complex`	+0.45%	0.089	+0.30%	+0.00%	0 → 0	ok
`Tokenize/scratch/insert`	-0.69%	0.021	-0.15%	+0.00%	0 → 0	ok
`Normalize/simple_select`	+0.82%	0.008	+0.25%	+0.00%	7 → 7	ok
`Normalize/join`	+1.09%	0.006	+0.49%	+0.00%	23 → 23	ok
`Normalize/complex`	+0.12%	0.481	-0.28%	+0.00%	69 → 69	ok
`TransformParse/simple_select`	-0.21%	0.820	+0.10%	+0.00%	5 → 5	ok
`TransformParse/join`	+0.24%	0.450	+0.09%	+0.00%	17 → 17	ok
`TransformParse/complex`	-0.63%	0.105	+0.35%	+0.00%	50 → 50	ok
geomean	+0.43%	0.052	+0.15%

A/A control: 0 of 32 benchmarks moved more than 3%.

Instructions per operation (cmd/perfdriver)

case	instructions/op	status
`e2e/aggregate`	+0.00%	ok
`e2e/complex`	+0.01%	ok
`e2e/cte`	+0.02%	ok
`e2e/delete`	+0.10%	ok
`e2e/insert`	+0.01%	ok
`e2e/join`	+0.01%	ok
`e2e/select_where`	-0.03%	ok
`e2e/simple_select`	+0.13%	ok
`e2e/subquery`	+0.01%	ok
`e2e/update`	-0.01%	ok
`e2e/window`	+0.02%	ok
`rw/postgresql/medium`	-0.01%	ok

Secondary metric: default GC, GOMAXPROCS=1 (informational)

benchmark	time	p
`ParseE2E/simple_select`	+0.07%	0.796
`ParseE2E/select_where`	-0.22%	0.850
`ParseE2E/join`	+1.09%	0.112
`ParseE2E/aggregate`	+0.76%	0.623
`ParseE2E/subquery`	+1.52%	0.315
`ParseE2E/cte`	+0.90%	0.545
`ParseE2E/window`	+0.03%	0.940
`ParseE2E/complex`	+1.69%	0.190
`ParseE2E/insert`	+0.40%	0.579
`ParseE2E/update`	+0.93%	0.496
`ParseE2E/delete`	+1.94%	0.028
`ParseRW/postgresql/medium`	+1.11%	0.165
`ParseRW/mysql/medium`	+1.03%	0.280
`ParseRW/mariadb/medium`	-0.59%	0.684
`ParseRW/sqlserver/medium`	+1.11%	0.796
`ParseRW/oracle/medium`	+1.35%	0.075
`ParseRW/sqlite/medium`	+0.62%	0.190
`ParseRW/snowflake/medium`	-0.30%	0.796
`ParseRW/clickhouse/medium`	+0.25%	0.436
`ParseFresh/simple_select`	+0.82%	0.075
`ParseFresh/join`	-0.12%	0.570
`ParseFresh/complex`	-0.28%	0.796
`PoolGetPut/serial`	-1.39%	0.364
`Tokenize/scratch/join`	-0.49%	0.012
`Tokenize/scratch/complex`	+1.27%	0.034
`Tokenize/scratch/insert`	-0.69%	0.023
`Normalize/simple_select`	+1.17%	0.186
`Normalize/join`	-0.29%	0.579
`Normalize/complex`	+0.46%	0.623
`TransformParse/simple_select`	-0.31%	0.384
`TransformParse/join`	-1.68%	0.190
`TransformParse/complex`	-0.40%	0.579
geomean	+0.36%

Environment

date:        2026-11-02T13:58:50Z
base:        98f7489cb0f4a09dcc22a6c902c355665c4b74e6
head:        working tree of 238e0d91b3d9770aea535f1fd016ffb03e5ef03f
go:          go version go1.26.1 linux/amd64
kernel:      Linux 7.0.0-1017-azure x86_64
pinning:     pinned to CPU 3 (SMT siblings 2-3) with taskset
clocksource: tsc
load:        1.27, 0.43, 0.16
instr count: cachegrind
perf probe:  Error: No supported events found. The instructions:u event is not supported. 

Architecture:                            x86_64
CPU(s):                                  4
Model name:                              AMD EPYC 9V74 80-Core Processor
Thread(s) per core:                      2
Core(s) per socket:                      2
Socket(s):                               1
Hypervisor vendor:                       Microsoft
L1d cache:                               64 KiB (2 instances)
L1i cache:                               64 KiB (2 instances)
L2 cache:                                2 MiB (2 instances)
L3 cache:                                32 MiB (1 instance)

benchstat (primary metric)

goos: linux
goarch: amd64
pkg: github.com/ajitpratap0/GoSQLX/pkg/fingerprint
cpu: AMD EPYC 9V74 80-Core Processor                
                        │    base     │                head                │
                        │   sec/op    │   sec/op     vs base               │
Normalize/simple_select   2.011µ ± 1%   2.028µ ± 1%  +0.82% (p=0.006 n=10)
Normalize/join            6.519µ ± 1%   6.590µ ± 2%  +1.09% (p=0.004 n=10)
Normalize/complex         16.89µ ± 1%   16.91µ ± 1%       ~ (p=0.481 n=10)
geomean                   6.050µ        6.091µ       +0.68%

                        │     base     │                 head                  │
                        │     B/op     │     B/op      vs base                 │
Normalize/simple_select     896.0 ± 0%     896.0 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            2.680Ki ± 0%   2.680Ki ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         7.492Ki ± 0%   7.492Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                   2.600Ki        2.600Ki       +0.00%
¹ all samples are equal

                        │    base    │                head                 │
                        │ allocs/op  │ allocs/op   vs base                 │
Normalize/simple_select   7.000 ± 0%   7.000 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            23.00 ± 0%   23.00 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         69.00 ± 0%   69.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                   22.31        22.31       +0.00%
¹ all samples are equal

pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
                          │    base     │                head                │
                          │   sec/op    │   sec/op     vs base               │
ParseE2E/simple_select      1.443µ ± 0%   1.433µ ± 3%       ~ (p=0.239 n=10)
ParseE2E/select_where       2.934µ ± 0%   2.937µ ± 1%       ~ (p=0.578 n=10)
ParseE2E/join               4.431µ ± 1%   4.406µ ± 2%       ~ (p=0.247 n=10)
ParseE2E/aggregate          3.234µ ± 1%   3.250µ ± 4%       ~ (p=0.052 n=10)
ParseE2E/subquery           2.858µ ± 1%   2.852µ ± 2%       ~ (p=0.470 n=10)
ParseE2E/cte                3.881µ ± 0%   3.894µ ± 0%       ~ (p=0.093 n=10)
ParseE2E/window             2.513µ ± 1%   2.515µ ± 1%       ~ (p=0.810 n=10)
ParseE2E/complex            11.68µ ± 2%   11.90µ ± 3%       ~ (p=0.011 n=10)
ParseE2E/insert             1.953µ ± 1%   1.987µ ± 5%  +1.72% (p=0.001 n=10)
ParseE2E/update             1.758µ ± 0%   1.773µ ± 5%  +0.88% (p=0.002 n=10)
ParseE2E/delete             1.051µ ± 2%   1.067µ ± 1%       ~ (p=0.020 n=10)
ParseRW/postgresql/medium   142.4µ ± 1%   143.5µ ± 1%       ~ (p=0.218 n=10)
ParseRW/mysql/medium        76.44µ ± 0%   76.86µ ± 3%       ~ (p=0.015 n=10)
ParseRW/mariadb/medium      109.9µ ± 1%   110.6µ ± 1%       ~ (p=0.315 n=10)
ParseRW/sqlserver/medium    210.6µ ± 1%   211.5µ ± 2%       ~ (p=0.218 n=10)
ParseRW/oracle/medium       256.0µ ± 1%   260.1µ ± 1%  +1.60% (p=0.002 n=10)
ParseRW/sqlite/medium       66.22µ ± 0%   66.71µ ± 3%       ~ (p=0.015 n=10)
ParseRW/snowflake/medium    156.3µ ± 1%   156.7µ ± 0%       ~ (p=0.579 n=10)
ParseRW/clickhouse/medium   73.93µ ± 1%   73.75µ ± 1%       ~ (p=0.853 n=10)
ParseFresh/simple_select    5.644µ ± 0%   5.674µ ± 1%  +0.53% (p=0.005 n=10)
ParseFresh/join             9.462µ ± 1%   9.498µ ± 1%       ~ (p=0.197 n=10)
ParseFresh/complex          16.85µ ± 1%   16.97µ ± 1%       ~ (p=0.066 n=10)
PoolGetPut/serial           56.53n ± 4%   57.32n ± 2%       ~ (p=0.159 n=10)
geomean                     10.26µ        10.31µ       +0.58%

                          │     base     │                head                 │
                          │     B/s      │     B/s       vs base               │
ParseE2E/simple_select      17.19Mi ± 0%   17.31Mi ± 3%       ~ (p=0.224 n=10)
ParseE2E/select_where       33.48Mi ± 0%   33.45Mi ± 1%       ~ (p=0.643 n=10)
ParseE2E/join               27.12Mi ± 1%   27.28Mi ± 2%       ~ (p=0.239 n=10)
ParseE2E/aggregate          24.77Mi ± 1%   24.65Mi ± 4%       ~ (p=0.050 n=10)
ParseE2E/subquery           27.36Mi ± 1%   27.43Mi ± 2%       ~ (p=0.470 n=10)
ParseE2E/cte                30.47Mi ± 0%   30.37Mi ± 0%       ~ (p=0.100 n=10)
ParseE2E/window             35.68Mi ± 1%   35.63Mi ± 1%       ~ (p=0.839 n=10)
ParseE2E/complex            30.88Mi ± 2%   30.29Mi ± 3%  -1.88% (p=0.010 n=10)
ParseE2E/insert             37.11Mi ± 1%   36.49Mi ± 5%  -1.67% (p=0.001 n=10)
ParseE2E/update             29.85Mi ± 0%   29.58Mi ± 4%  -0.89% (p=0.001 n=10)
ParseE2E/delete             47.18Mi ± 2%   46.51Mi ± 1%       ~ (p=0.019 n=10)
ParseRW/postgresql/medium   25.19Mi ± 1%   25.00Mi ± 1%       ~ (p=0.224 n=10)
ParseRW/mysql/medium        35.41Mi ± 0%   35.21Mi ± 3%       ~ (p=0.014 n=10)
ParseRW/mariadb/medium      32.26Mi ± 1%   32.06Mi ± 1%       ~ (p=0.289 n=10)
ParseRW/sqlserver/medium    31.60Mi ± 1%   31.46Mi ± 2%       ~ (p=0.197 n=10)
ParseRW/oracle/medium       32.75Mi ± 1%   32.23Mi ± 1%  -1.57% (p=0.002 n=10)
ParseRW/sqlite/medium       31.19Mi ± 0%   30.97Mi ± 2%       ~ (p=0.014 n=10)
ParseRW/snowflake/medium    32.52Mi ± 1%   32.43Mi ± 0%       ~ (p=0.541 n=10)
ParseRW/clickhouse/medium   30.48Mi ± 1%   30.56Mi ± 1%       ~ (p=0.839 n=10)
geomean                     30.61Mi        30.45Mi       -0.53%

                          │      base      │                 head                  │
                          │      B/op      │     B/op      vs base                 │
ParseE2E/simple_select        816.0 ± 0%       816.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where       1.523Ki ± 0%     1.523Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join               2.305Ki ± 0%     2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate          1.812Ki ± 0%     1.812Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery           1.828Ki ± 0%     1.828Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                2.336Ki ± 0%     2.336Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window             1.281Ki ± 0%     1.281Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex            6.399Ki ± 0%     6.399Ki ± 0%       ~ (p=0.211 n=10)
ParseE2E/insert               824.0 ± 0%       824.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update               736.0 ± 0%       736.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete               400.0 ± 0%       400.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium   70.51Ki ± 0%     70.51Ki ± 0%       ~ (p=0.115 n=10)
ParseRW/mysql/medium        43.72Ki ± 0%     43.72Ki ± 0%       ~ (p=1.000 n=10)
ParseRW/mariadb/medium      64.83Ki ± 0%     64.83Ki ± 0%       ~ (p=0.487 n=10)
ParseRW/sqlserver/medium    90.61Ki ± 0%     90.55Ki ± 0%  -0.07% (p=0.000 n=10)
ParseRW/oracle/medium       113.8Ki ± 0%     113.8Ki ± 0%       ~ (p=0.474 n=10)
ParseRW/sqlite/medium       39.89Ki ± 0%     39.89Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium    83.38Ki ± 0%     83.38Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium   37.13Ki ± 0%     37.13Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select      896.0 ± 0%       896.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join             2.383Ki ± 0%     2.383Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex          6.477Ki ± 0%     6.477Ki ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                  ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │     base      │                 head                 │
                          │   allocs/op   │  allocs/op   vs base                 │
ParseE2E/simple_select       5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where        18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join                17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate           18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery            14.00 ± 0%      14.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                 21.00 ± 0%      21.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window              11.00 ± 0%      11.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex             50.00 ± 0%      50.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert              17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update              13.00 ± 0%      13.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete              5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium    646.0 ± 0%      646.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mysql/medium         365.0 ± 0%      365.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mariadb/medium       523.0 ± 0%      523.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlserver/medium     936.0 ± 0%      928.0 ± 0%  -0.85% (p=0.000 n=10)
ParseRW/oracle/medium       1.093k ± 0%     1.093k ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlite/medium        346.0 ± 0%      346.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium     777.0 ± 0%      777.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium    384.0 ± 0%      384.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select     6.000 ± 0%      6.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join              18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex           51.00 ± 0%      51.00 ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial            0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                 ²                -0.04%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │    base     │                head                │
                          │  sec/stmt   │  sec/stmt    vs base               │
ParseRW/postgresql/medium   11.87µ ± 1%   11.96µ ± 1%       ~ (p=0.225 n=10)
ParseRW/mysql/medium        9.555µ ± 0%   9.608µ ± 3%       ~ (p=0.016 n=10)
ParseRW/mariadb/medium      12.21µ ± 1%   12.29µ ± 1%       ~ (p=0.315 n=10)
ParseRW/sqlserver/medium    10.03µ ± 1%   10.07µ ± 2%       ~ (p=0.197 n=10)
ParseRW/oracle/medium       10.67µ ± 1%   10.84µ ± 1%  +1.60% (p=0.002 n=10)
ParseRW/sqlite/medium       11.04µ ± 0%   11.12µ ± 3%       ~ (p=0.015 n=10)
ParseRW/snowflake/medium    12.03µ ± 1%   12.06µ ± 0%       ~ (p=0.565 n=10)
ParseRW/clickhouse/medium   10.56µ ± 1%   10.54µ ± 1%       ~ (p=0.853 n=10)
geomean                     10.96µ        11.02µ       +0.59%

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer
                         │    base     │                head                │
                         │   sec/op    │   sec/op     vs base               │
Tokenize/scratch/join      1.938µ ± 1%   1.926µ ± 1%  -0.62% (p=0.006 n=10)
Tokenize/scratch/complex   5.269µ ± 1%   5.293µ ± 0%       ~ (p=0.085 n=10)
Tokenize/scratch/insert    930.3n ± 3%   923.9n ± 2%       ~ (p=0.017 n=10)
geomean                    2.118µ        2.112µ       -0.29%

                         │     base     │                head                 │
                         │     B/s      │     B/s       vs base               │
Tokenize/scratch/join      62.02Mi ± 1%   62.41Mi ± 1%  +0.64% (p=0.006 n=10)
Tokenize/scratch/complex   68.42Mi ± 1%   68.11Mi ± 0%       ~ (p=0.079 n=10)
Tokenize/scratch/insert    77.92Mi ± 3%   78.45Mi ± 2%       ~ (p=0.017 n=10)
geomean                    69.15Mi        69.35Mi       +0.29%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/ajitpratap0/GoSQLX/pkg/transform
                             │    base     │                head                │
                             │   sec/op    │   sec/op     vs base               │
TransformParse/simple_select   1.437µ ± 1%   1.434µ ± 1%       ~ (p=0.811 n=10)
TransformParse/join            4.787µ ± 1%   4.798µ ± 2%       ~ (p=0.448 n=10)
TransformParse/complex         13.11µ ± 1%   13.02µ ± 2%       ~ (p=0.105 n=10)
geomean                        4.483µ        4.474µ       -0.20%

                             │     base     │                 head                  │
                             │     B/op     │     B/op      vs base                 │
TransformParse/simple_select     816.0 ± 0%     816.0 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            2.305Ki ± 0%   2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         6.398Ki ± 0%   6.398Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                        2.273Ki        2.273Ki       +0.00%
¹ all samples are equal

                             │    base    │                head                 │
                             │ allocs/op  │ allocs/op   vs base                 │
TransformParse/simple_select   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         50.00 ± 0%   50.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                        16.20        16.20       +0.00%
¹ all samples are equal

Run, raw data and benchstat

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

@r

… errors [W2-03] Add @rıght join, @ınner join, @groupıng sets, @@rıght join and keyword-named params to the golden (regenerated on origin/main's tokenizer, HEAD passes). Golden errors now record code and position only. ValueIsSourceSlice checks that values alias the input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

github-actions · 2026-11-02T10:32:55Z

Claude finished @ajitpratap0's task in 22s —— View job

Review of PR #925: zero-copy `@param` / `@@var` token values

Verdict: looks good. I read the tokenizer.go diff only. I didn't run the tests or read the test file or golden, so rely on CI for those.

Correctness

Fallback conditions. scanAtName (tokenizer.go:~1243) returns false with t.pos restored for the cases readIdentifier treats specially:
- a non-ASCII start or name;
- U&' and U&" escapes;
- a single letter before ';
- a compound-keyword start.
The old path runs unchanged in those cases.
Early returns. The non-ASCII start and U& checks return before t.pos is touched. Only the later checks need saved. The restore is correct.
Slice bounds. t.src[atStart:t.pos.Index] assumes t.src and t.input are the same bytes with the same offsets. The diff does not show that invariant, so please confirm it holds for every entry point. These include a BOM, any input normalisation, and pooled-tokenizer reuse.
Keyword-length guard. For names longer than keywordKeyBufLen, the compound lookup is skipped. That is safe only if no compound-start keyword is longer than that buffer, so please confirm.
Keyword values. @select, @from and @Where now take their value from the source text. This matches the old "@"+Value only if readIdentifier never normalises keyword case. A lowercase non-compound keyword case in the golden would pin that. It is cheap to add.

Performance

The allocation drops are plausible. The string concat, the throwaway Word and the compound lookahead are gone. kbuf is a fixed stack array.
Tokens now alias the whole input string. Other zero-copy tokens already do this, so it is not a new class of issue.
The perf-gate result is clean. The +1.01% on ParseRW/sqlserver/medium is under the 3% threshold. Allocations on that cell went down, so layout noise is a reasonable reading.
The time target in the PLAN isn't met, as the PR description already states.

Tests

Generating the golden before the change is the right approach.
TestAtParam_NoPerTokenAlloc uses a fixed threshold. If it flakes across Go versions, compare against a baseline measured in the same test.
Optional extra cases:
- a BOM-prefixed input;
- slices from an earlier call staying valid after PutTokenizer.
TestAtParam_ValueIsSourceSlice, if it only checks string equality, overclaims in its name. Rename it or drop it in favour of the alloc test.

Security

No new unsafe code, and all slicing is bounds-checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

github-actions · 2026-11-02T10:49:45Z

Claude finished @ajitpratap0's task in 29s —— View job

Review of PR #925: zero-copy `@param` / `@@var` token values

Verdict: looks good, no blocking issues. I read the tokenizer.go diff and compared scanAtName with readIdentifier. I didn't run the tests, so rely on CI for those.

Correctness

Fallbacks match readIdentifier. scanAtName returns false for these cases:
- a non-ASCII start or name;
- U&' and U&";
- a single letter before ';
- a compound-keyword start.
In the last three cases t.pos is restored, and the non-ASCII-start and U& checks return before t.pos is touched. The old path then runs unchanged.
Keyword length. keywordKeyBufLen is 32, and readIdentifier uses the same bound. For ASCII names longer than 32 it also skips the compound lookup, so skipping it in scanAtName is equivalent.
Keyword values. readIdentifier returns Value: ident, the raw source text, with no case normalisation. A slice of the source therefore equals the old "@"+Value for names like @select. The description says the golden pins @select, @From, @Where.
Source offsets. t.src and t.input are assigned together at tokenizer.go:590-591, so the offsets match.
saved := t.pos. It is a value copy, consistent with how readIdentifier restores its own position.

Performance

Removing the concat, the throwaway Word and the compound lookahead accounts for the alloc drops. kbuf is a stack array.
The perf gate flagged Normalize/simple_select at +3.34%. That benchmark doesn't touch @ tokens, the allocs and B/op are unchanged, and the A/A control is flat. Instructions/op is flat. I read this as layout noise, as the PR description does. A re-run would settle it.
Token values now alias the whole input string, as other zero-copy tokens already do.

Tests

The earlier review rounds suggested pinning lowercase keyword names and renaming TestAtParam_ValueIsSourceSlice. The PR description says both are done, with keyword-named params in the golden and an aliasing check on the value.
TestAtParam_NoPerTokenAlloc uses a fixed threshold. If it flakes across Go versions, compare against a baseline measured in the same test.
Optional extra cases: a BOM-prefixed input, and slices from an earlier call staying valid after PutTokenizer. The src assignment makes both safe, since the slices are of an immutable string.

Minor

The scanIdentifierTail comment change is accurate. It should cover the first-character case that scanAtName relies on.
The CHANGELOG entry is long, but it is accurate and in line with its neighbours.

Security

No unsafe code in the tokenizer change, and all slicing is bounds-checked.

github-actions · 2026-11-02T13:57:46Z

Claude finished @ajitpratap0's task in 23s —— View job

Review of PR #925: zero-copy `@param` / `@@var` token values

Verdict: looks good, no blocking issues. I read the tokenizer.go and CHANGELOG diffs and checked the src/input assignment sites. I did not run the tests or read the golden file, so rely on CI for those.

Correctness

Fallbacks. scanAtName returns false for these cases:
- a non-ASCII start or name;
- U&' and U&" escapes;
- a single letter before ';
- a compound-keyword start.
The first two return before t.pos changes. The last two restore saved, which is a value copy. The old readIdentifier path then runs unchanged.
Source offsets. t.input and t.src are assigned together at tokenizer.go:590-591. pool.go:188-189 clears both together on PutTokenizer. Because src is an immutable string, token values stay valid after the tokenizer is returned to the pool.
Keyword-length guard. keywordKeyBufLen is 32. Skipping the compound lookup for longer names is only safe if no compound-start keyword is longer than 32 bytes. readIdentifier uses the same bound, so the behaviour matches.
Non-ASCII check. isASCII(ident) runs after the tail scan. That catches names that start ASCII but contain non-ASCII characters, and they take the fallback path.
First-character comment. The scanIdentifierTail comment update is accurate. scanAtName relies on the first character having already passed isIdentifierStart.

Performance

The saving comes from dropping the string concatenation, the throwaway Word and the compound lookahead. kbuf is a stack array.
The perf gate passes. The time target in the plan isn't met on amd64, but the PR description says so and asks for allocation-only acceptance. That seems reasonable given the re-scoping in C0.md.
Token values now alias the whole input string, as other zero-copy tokens already do.

Tests

Generating the golden on the old tokenizer before the change is the right approach, and the 41 inputs x 9 dialects cover the edge cases well.
TestAtParam_NoPerTokenAlloc uses a fixed threshold. If it flakes across Go versions, assert against a baseline measured in the same test.
Optional extra cases:
- a BOM-prefixed input;
- a pooled-tokenizer reuse case, where values from the first call are re-checked after PutTokenizer and a second tokenization.

Security

There is no new unsafe code in the tokenizer change, and all slicing is bounds-checked. The unsafe.StringData use is only in the test.

Minor

The CHANGELOG entry is long but accurate.

perf(tokenizer): zero-copy @PARAM and @@var token values [W2-03]

ac5dc97

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

vercel Bot deployed to Preview November 2, 2026 10:11

View deployment

github-actions Bot added documentation Improvements or additions to documentation tokenizer Tokenizer / lexer changes labels Nov 2, 2026

ajitpratap0 and others added 2 commits November 2, 2026 15:57

merge: origin/main into perf/w2-03-zero-copy-at-params

ccd6246

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

vercel Bot deployed to Preview November 2, 2026 10:33

View deployment

docs(tokenizer): correct scanIdentifierTail first-character note [W2-03]

cbbee20

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

Merge branch 'main' into perf/w2-03-zero-copy-at-params

af586d9

vercel Bot deployed to Preview November 2, 2026 13:57

View deployment

ajitpratap0 merged commit afec16a into main Nov 2, 2026
48 checks passed

ajitpratap0 deleted the perf/w2-03-zero-copy-at-params branch November 2, 2026 14:13

Navigation Menu

Repository navigation

perf(tokenizer): zero-copy @param and @@var token values [W2-03] - #925

perf(tokenizer): zero-copy @param and @@var token values [W2-03]#925
ajitpratap0 merged 5 commits into
mainfrom
perf/w2-03-zero-copy-at-params

ajitpratap0 commented Nov 2, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ajitpratap0 commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Breaking Changes

Uh oh!

vercel Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #925: zero-copy @param / @@var token values

Correctness

Performance

Tests

Minor

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Perf gate: PASS (shadow mode: report only, this check never fails)

Gated suite (GOGC=off, GOMAXPROCS=1)

Instructions per operation (cmd/perfdriver)

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #925: zero-copy @param / @@var token values

Correctness

Performance

Tests

Security

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #925: zero-copy @param / @@var token values

Correctness

Performance

Tests

Minor

Security

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #925: zero-copy @param / @@var token values

Correctness

Performance

Tests

Security

Minor

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ajitpratap0 commented Nov 2, 2026 •

edited

Loading

vercel Bot commented Nov 2, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Review of PR #925: zero-copy `@param` / `@@var` token values

github-actions Bot commented Nov 2, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Review of PR #925: zero-copy `@param` / `@@var` token values

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Review of PR #925: zero-copy `@param` / `@@var` token values

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Review of PR #925: zero-copy `@param` / `@@var` token values