Repository navigation
fix(security,cbinding): scan raw-text statements, recover from panics in C exports - #871
Conversation
… in C exports - Scanner.Scan(tree) scans standalone VALUES and statements the parser keeps as text (utility, SET, transaction, USE, unsupported, T-SQL blocks, PREPARE/EXECUTE, CREATE POLICY/TRIGGER) with the ScanSQL rules, so payloads such as VALUES (SLEEP(5)) and IF 1=1 WAITFOR DELAY are reported; no finding is reported twice. - Every C-binding export recovers from a panic and returns its normal error result with a fixed message instead of crashing the host. - DialectExtras()/DialectModifiers() read accessors return values, so the shared zero-value stand-ins can no longer be mutated through them; EnsureExtras() stays the mutable path. No new allocations. - tokscratch panics with a clear message when the tokenizer is not linked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Claude finished @ajitpratap0's task in 20s —— View job Review of PR #871I read the diff for the scanner, C binding recovery, Scanner (
C binding (
AST accessors (
Tests
Verdict: Approve-worthy once you decide on the "shallow copy" doc wording. The other points are optional. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 19s —— View job Review of PR #871I read the diffs for Scanner (
C binding (
AST accessors (
Tests
Verdict: No blockers. I'd fix the "shallow copy" wording and the fixture regeneration note before merging. The other points are optional. |
…879) Scan(tree) only inspected top-level SELECT WHERE/HAVING/columns/FROM, INSERT VALUES rows, UPDATE/DELETE WHERE and set-operation leaves, so time-based, out-of-band and tautology patterns inside CTEs, subqueries, derived tables, CASE/CAST/IN/BETWEEN, ORDER/GROUP BY, JOIN ON, window specs, FILTER, LIMIT, LATERAL/APPLY, PREWHERE, QUALIFY, CONNECT BY, RETURNING, ON CONFLICT, INSERT ... SELECT, MERGE, CREATE TABLE AS, CREATE [MATERIALIZED] VIEW and EXPLAIN went undetected while ScanSQL caught them. Scan now uses ast.Inspect: every FunctionCall and table function is checked once, tautologies are checked in every condition position, UNION checks run on each outermost set operation (including those nested in CTEs and subqueries), and text-bodied nodes keep the text path. JOIN Left references are skipped to avoid double counting, and qualified identifiers such as a.id = b.id are no longer reported as tautologies. T-SQL IF ... WAITFOR and standalone VALUES were already covered on main (#871) and gain regression tests. Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 Co-authored-by: Ajit Pratap Singh <ajitpratapsingh@Ajits-Mac-mini-2655.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…st lower-case mode (#886) * fix(parser): end VACUUM, ANALYZE and LISTEN at their grammar PostgreSQL, Redshift and SQLite VACUUM / ANALYZE and PostgreSQL LISTEN, UNLISTEN and NOTIFY were kept verbatim, so trailing same-line tokens (VACUUM t junk) were swallowed into the statement text. They now follow the GRANT / COPY pattern: a small per-dialect grammar finds the end and utilityTailError rejects anything after it. Generic mode accepts the union of the dialect forms; MySQL / MariaDB / Oracle ANALYZE TABLE keeps its verbatim path. Adds regression tests for the already-fixed CREATE TRIGGER / PREPARE / GRANT / COPY boundaries and for the splitter's MySQL backslash escapes and PostgreSQL CREATE RULE ... DO (a; b) command lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(security): walk the full AST in Scanner.Scan for ScanSQL parity Scan(tree) only inspected top-level SELECT WHERE/HAVING/columns/FROM, INSERT VALUES rows, UPDATE/DELETE WHERE and set-operation leaves, so time-based, out-of-band and tautology patterns inside CTEs, subqueries, derived tables, CASE/CAST/IN/BETWEEN, ORDER/GROUP BY, JOIN ON, window specs, FILTER, LIMIT, LATERAL/APPLY, PREWHERE, QUALIFY, CONNECT BY, RETURNING, ON CONFLICT, INSERT ... SELECT, MERGE, CREATE TABLE AS, CREATE [MATERIALIZED] VIEW and EXPLAIN went undetected while ScanSQL caught them. Scan now uses ast.Inspect: every FunctionCall and table function is checked once, tautologies are checked in every condition position, UNION checks run on each outermost set operation (including those nested in CTEs and subqueries), and text-bodied nodes keep the text path. JOIN Left references are skipped to avoid double counting, and qualified identifiers such as a.id = b.id are no longer reported as tautologies. T-SQL IF ... WAITFOR and standalone VALUES were already covered on main (#871) and gain regression tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(parser): reserve TOP in SQL Server names; parse full MariaDB DELETE HISTORY SQL Server: a delimited [top] / "top" is now an ordinary name (the select list no longer reads it as the TOP clause) and keeps its delimiters in SQL() and the formatter. Bare TOP is rejected where a table name, DML/MERGE target, table alias or select-item alias is expected, matching T-SQL, instead of being accepted in FROM/INSERT/DELETE but rejected in UPDATE. MariaDB: DELETE HISTORY FROM t [PARTITION (p, ...)] [BEFORE SYSTEM_TIME [TIMESTAMP | TRANSACTION] expr] parses any point expression and round-trips the partition list and unit keyword. The new fields live in DMLTargetFields.History (DeleteHistoryOptions) so DeleteStatement keeps its size class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(parser,tokenizer): stop GRANT cursor panic; MySQL/MariaDB # line comments - utility grammar cursor no longer stores the -1 groupEnd/bracketEnd sentinel before failing (GRANT ... TO [ / [AppUser / CURRENT_USER(, MariaDB IDENTIFIED VIA ... USING PASSWORD(, COPY (); Parser.seek clamps negative indexes to EOF. These inputs now return E2002 instead of panicking. - With the MySQL or MariaDB dialect, '#' starts a line comment in the tokenizer (recorded in Comments, preserved by the formatter), the statement splitter and its procedural lookahead, highlight error recovery and the dropped-word checker. Other dialects keep their '#' meaning. - Regression tests over all 8 dialects, fuzz seeds, realworld corpus cases; literal golden regenerated (only mysql/mariadb digests change for inputs containing '#'). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(lsp): advertise textDocumentSync and survive malformed client input - initialize now sends the LSP 3.17 key textDocumentSync instead of the non-standard textDocumentSyncOptions, so clients send open/change/close. - readMessage rejects a non-positive Content-Length instead of panicking in makeslice, and drains oversized bodies so message framing survives. - didChange ranges are clamped to the document, read as UTF-16 code units and respect CRLF line ends; negative positions are rejected and reversed ranges are swapped, so slicing cannot go out of bounds. - A panic in a message handler is logged and answered with InternalError, and background diagnostics recover too, so one bad input cannot end the server process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(transpiler): keep set-operation LIMIT whole for SQL Server and survive long operator chains - A LIMIT that belongs to a UNION/INTERSECT/EXCEPT was pushed into the last branch as TOP when targeting SQL Server, silently changing results. It is now emitted on the set operation as ORDER BY ... OFFSET n ROWS FETCH NEXT m ROWS ONLY; without ORDER BY, Transpile returns an error. A TOP branch moved to a LIMIT dialect keeps the set-level ORDER BY. - operatorRetargeter.walk and forEachLiteral now use explicit stacks, so a 490,000-term OR/AND/+ chain transpiles instead of crashing the process with an unrecoverable stack overflow. Rewrites that would nest output more than 512 levels deep return an error, and || flattening into CONCAT stays linear. - Near-ceiling chain tests skip under -race to stay within the package timeout; a 50k-term case runs everywhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(gosqlx): keep statement terminators with comments; visit each node once in Extract - formatSQL's comment-preserving path cleared the layout terminator but only the plain path added ';' to statements rendered as raw text, so with AddSemicolon and any comment, PostgreSQL DO $$...$$, CREATE FUNCTION bodies and Oracle BEGIN ... END blocks lost their ';' and the output failed to parse or merged two statements. Both paths now use formatter.EnsureStatementTerminator. - metadataCollector visited WITH clauses, CTE bodies, set-operation operands, INSERT queries and UPDATE assignments explicitly and again through Children(), doubling the work at each level: nested CTEs at depth 10 took about 59 s and long UNION ALL chains never finished. Children() already covers them, so the explicit visits are gone and every node is visited once. Results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(security): close ScanSQL bypasses via quoted dollar tags, truncation and inline comments - Dollar-quote stripping ran inside quoted strings, so SELECT '$x$'; DROP TABLE users; SELECT '$x$' scanned clean. A $tag$ is now stripped only when every quoting interpretation (backslash or doubled quote escapes, PostgreSQL E-strings, "..." strings, [bracket] identifiers, nested comments, # comments) agrees it is outside strings and comments. - Tautology and comment checks only looked at the first 10,000 bytes. They now cover the whole input; inputs above 10 MiB get a CRITICAL INPUT_TOO_LARGE finding regardless of the severity filter. - Comments between keywords (UNION/**/SELECT, SLEEP/**/(5), ;--\nDROP, # comments, /*/, MySQL 1--1) evaded matching. Patterns now also run over comment-free views of the text for each quoting convention; views only add findings. /*! */ and /*M! */ bodies are scanned. - The AST UNION NULL-padding check now matches the parser's null literal. - The tautology check compares operands without allocating per candidate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(formatter): keep Oracle / lines, U& spellings and GO comments; fast lower-case mode - Oracle SQL*Plus / execute lines survive every formatting entry point. Statement counts are unchanged: AST.ExecuteSlash marks statements that a / follows, and no ';' is added between a statement and its / beyond a PL/SQL block's own END;. - PostgreSQL U&"..." identifiers keep their source spelling via AST.UnicodeIdentifiers when the target is PostgreSQL or unset; other targets and all AST name fields keep the decoded name. - Comments next to SQL Server GO lines and trailing comments of statements without ';' stay in place, using recorded statement end positions, and GO n keeps its count and comments. - Lower-case keyword mode renders common expressions straight into one builder; it now costs 1.2-1.5x upper-case instead of 3.5-10x, with 5x to 500x fewer allocations. The slot path remains for uncommon syntax. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * docs(changelog): drop entries duplicated by the main merge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 --------- Co-authored-by: Ajit Pratap Singh <ajitpratapsingh@Ajits-Mac-mini-2655.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(parser): end VACUUM, ANALYZE and LISTEN at their grammar PostgreSQL, Redshift and SQLite VACUUM / ANALYZE and PostgreSQL LISTEN, UNLISTEN and NOTIFY were kept verbatim, so trailing same-line tokens (VACUUM t junk) were swallowed into the statement text. They now follow the GRANT / COPY pattern: a small per-dialect grammar finds the end and utilityTailError rejects anything after it. Generic mode accepts the union of the dialect forms; MySQL / MariaDB / Oracle ANALYZE TABLE keeps its verbatim path. Adds regression tests for the already-fixed CREATE TRIGGER / PREPARE / GRANT / COPY boundaries and for the splitter's MySQL backslash escapes and PostgreSQL CREATE RULE ... DO (a; b) command lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(security): walk the full AST in Scanner.Scan for ScanSQL parity Scan(tree) only inspected top-level SELECT WHERE/HAVING/columns/FROM, INSERT VALUES rows, UPDATE/DELETE WHERE and set-operation leaves, so time-based, out-of-band and tautology patterns inside CTEs, subqueries, derived tables, CASE/CAST/IN/BETWEEN, ORDER/GROUP BY, JOIN ON, window specs, FILTER, LIMIT, LATERAL/APPLY, PREWHERE, QUALIFY, CONNECT BY, RETURNING, ON CONFLICT, INSERT ... SELECT, MERGE, CREATE TABLE AS, CREATE [MATERIALIZED] VIEW and EXPLAIN went undetected while ScanSQL caught them. Scan now uses ast.Inspect: every FunctionCall and table function is checked once, tautologies are checked in every condition position, UNION checks run on each outermost set operation (including those nested in CTEs and subqueries), and text-bodied nodes keep the text path. JOIN Left references are skipped to avoid double counting, and qualified identifiers such as a.id = b.id are no longer reported as tautologies. T-SQL IF ... WAITFOR and standalone VALUES were already covered on main (#871) and gain regression tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(parser): reserve TOP in SQL Server names; parse full MariaDB DELETE HISTORY SQL Server: a delimited [top] / "top" is now an ordinary name (the select list no longer reads it as the TOP clause) and keeps its delimiters in SQL() and the formatter. Bare TOP is rejected where a table name, DML/MERGE target, table alias or select-item alias is expected, matching T-SQL, instead of being accepted in FROM/INSERT/DELETE but rejected in UPDATE. MariaDB: DELETE HISTORY FROM t [PARTITION (p, ...)] [BEFORE SYSTEM_TIME [TIMESTAMP | TRANSACTION] expr] parses any point expression and round-trips the partition list and unit keyword. The new fields live in DMLTargetFields.History (DeleteHistoryOptions) so DeleteStatement keeps its size class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(parser,tokenizer): stop GRANT cursor panic; MySQL/MariaDB # line comments - utility grammar cursor no longer stores the -1 groupEnd/bracketEnd sentinel before failing (GRANT ... TO [ / [AppUser / CURRENT_USER(, MariaDB IDENTIFIED VIA ... USING PASSWORD(, COPY (); Parser.seek clamps negative indexes to EOF. These inputs now return E2002 instead of panicking. - With the MySQL or MariaDB dialect, '#' starts a line comment in the tokenizer (recorded in Comments, preserved by the formatter), the statement splitter and its procedural lookahead, highlight error recovery and the dropped-word checker. Other dialects keep their '#' meaning. - Regression tests over all 8 dialects, fuzz seeds, realworld corpus cases; literal golden regenerated (only mysql/mariadb digests change for inputs containing '#'). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(lsp): advertise textDocumentSync and survive malformed client input - initialize now sends the LSP 3.17 key textDocumentSync instead of the non-standard textDocumentSyncOptions, so clients send open/change/close. - readMessage rejects a non-positive Content-Length instead of panicking in makeslice, and drains oversized bodies so message framing survives. - didChange ranges are clamped to the document, read as UTF-16 code units and respect CRLF line ends; negative positions are rejected and reversed ranges are swapped, so slicing cannot go out of bounds. - A panic in a message handler is logged and answered with InternalError, and background diagnostics recover too, so one bad input cannot end the server process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(transpiler): keep set-operation LIMIT whole for SQL Server and survive long operator chains - A LIMIT that belongs to a UNION/INTERSECT/EXCEPT was pushed into the last branch as TOP when targeting SQL Server, silently changing results. It is now emitted on the set operation as ORDER BY ... OFFSET n ROWS FETCH NEXT m ROWS ONLY; without ORDER BY, Transpile returns an error. A TOP branch moved to a LIMIT dialect keeps the set-level ORDER BY. - operatorRetargeter.walk and forEachLiteral now use explicit stacks, so a 490,000-term OR/AND/+ chain transpiles instead of crashing the process with an unrecoverable stack overflow. Rewrites that would nest output more than 512 levels deep return an error, and || flattening into CONCAT stays linear. - Near-ceiling chain tests skip under -race to stay within the package timeout; a 50k-term case runs everywhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(gosqlx): keep statement terminators with comments; visit each node once in Extract - formatSQL's comment-preserving path cleared the layout terminator but only the plain path added ';' to statements rendered as raw text, so with AddSemicolon and any comment, PostgreSQL DO $$...$$, CREATE FUNCTION bodies and Oracle BEGIN ... END blocks lost their ';' and the output failed to parse or merged two statements. Both paths now use formatter.EnsureStatementTerminator. - metadataCollector visited WITH clauses, CTE bodies, set-operation operands, INSERT queries and UPDATE assignments explicitly and again through Children(), doubling the work at each level: nested CTEs at depth 10 took about 59 s and long UNION ALL chains never finished. Children() already covers them, so the explicit visits are gone and every node is visited once. Results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(security): close ScanSQL bypasses via quoted dollar tags, truncation and inline comments - Dollar-quote stripping ran inside quoted strings, so SELECT '$x$'; DROP TABLE users; SELECT '$x$' scanned clean. A $tag$ is now stripped only when every quoting interpretation (backslash or doubled quote escapes, PostgreSQL E-strings, "..." strings, [bracket] identifiers, nested comments, # comments) agrees it is outside strings and comments. - Tautology and comment checks only looked at the first 10,000 bytes. They now cover the whole input; inputs above 10 MiB get a CRITICAL INPUT_TOO_LARGE finding regardless of the severity filter. - Comments between keywords (UNION/**/SELECT, SLEEP/**/(5), ;--\nDROP, # comments, /*/, MySQL 1--1) evaded matching. Patterns now also run over comment-free views of the text for each quoting convention; views only add findings. /*! */ and /*M! */ bodies are scanned. - The AST UNION NULL-padding check now matches the parser's null literal. - The tautology check compares operands without allocating per candidate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * fix(cli,transpiler,lsp): close re-verification follow-ups - analyze --security and lint's security pass merge ScanSQL findings with the AST scan, so stacked DROP statements are no longer reported clean; inline SQL containing '/' is no longer treated as a missing file. - SQL Server: LIMIT/OFFSET and bare OFFSET without ORDER BY get ORDER BY (SELECT NULL); FETCH FIRST gets OFFSET 0 ROWS. - Row limits in derived tables, JOIN operands and scalar subqueries are rendered for the target dialect; a cheap text pre-check keeps the common select list on the old path with no extra allocations. - QueryTailFrom keeps WITH TIES; targets that cannot express it error. - LSP unknown methods return MethodNotFound (-32601). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 * docs(changelog): drop entries duplicated by the main merge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2 --------- Co-authored-by: Ajit Pratap Singh <ajitpratapsingh@Ajits-Mac-mini-2655.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Follow-ups from the pre-merge security and AST reviews of #866–#868:
Scanner.Scan(tree)now sees newly parseable statements. StandaloneVALUESand every statement the parser keeps as raw text go through theScanSQLrules: utility,SET, transaction,USE, unsupported, T-SQLIF/WHILE/BEGINbodies,PREPARE/EXECUTE,CREATE POLICY/TRIGGER.VALUES (SLEEP(5))andIF 1=1 WAITFOR DELAY '0:0:5'are now reported as TIME_BASED. Each statement takes exactly one path, so nothing is reported twice.//exportrecovers from a panic and returns its normal error result with a fixed message. A panic can no longer crash the host process, and panic values (which may contain SQL or paths) are not exposed.DialectExtras()/DialectModifiers()return values instead of pointers to shared zero values, so a caller can no longer mutate every node through them.EnsureExtras()remains the mutable path. These accessors are unreleased, so the signature change is safe.internal/tokscratchpanics with a clear message if the tokenizer isn't linked.Test Plan
gofmt,go vet,golangci-lint run(0 issues),task test:race,task test:cbindingScanSQLproduced is preserved, andScan(tree)now matchesScanSQLon the newly parseable statementsBenchmarkRealWorldParseFormatvs main, 4 interleaved runs: median 9.68 vs 9.63 ms/op, same B/op and allocs/opBreaking Changes
DialectExtras()/DialectModifiers()now return values (SelectExtras,FunctionExtras,BinaryExtras,InsertDialectFields,DMLTargetFields,TableModifiers) instead of pointers. These were added after v1.16.0 and are unreleased.🤖 Generated with Claude Code
https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2