Story 21.1 — quoted identifiers: one lexing surface (backtick + double quote)

[fupelaqu]

Part 1 of 2 for #252 — closed by the 21.2 PR.

Story 21.1 (_bmad-output/implementation-artifacts/21-1-quoted-identifier-lexing-surface.md).
A column name or alias may now be written quoted, in either the ANSI SQL-92 double-quote or the
MySQL backtick spelling, everywhere an identifier is accepted. Quoting is folded into identifier
itself, so the ~35 sites ending in | identifier inherit it with a zero-line diff — no
SelectParser / GroupByParser / OrderByParser / WhereParser / function/** edits, which is
the point of AD-2.

---

🔴 The one deviation that matters: AD-13 reverses PD-3

The spec was internally inconsistent, and only the round-trip suite could show it. PD-3 kept a
bare double-quoted lexeme a STRING in operand position; AC-5 requires the render to be a fixed
point. Because Identifier.sql normalises to ONE canonical delimiter, those cannot both hold:

SELECT MAX(`amount`)   ->  .sql = SELECT MAX("amount")  ->  re-parses as MAX('amount')

Six AC-5 matrix rows failed this way (MAX, CAST, TRY_CAST, CONVERT ×2, DATE_TRUNC,
EXTRACT, HAVING COUNT(id)), and SELECT (amount + 1) re-parsed into a hard type mismatch
rejection. MaterializedViewExtension persists that render and runs client.run(alter.sql) — so
this was a live silent corruption, not cosmetics.

Switching the canonical delimiter to the backtick also fixes it, in one line, and was refuted:
DuckDB's Keywords and Identifiers documentation accepts " and only ", and
softclient4es-arrow's JoinPlanner:297 feeds field.identifier.sql straight into a DuckDB SELECT
list. Emitting backticks would break the sidecar's JOIN path for every quoted column.

Fix — one token: quotedIdentifier inserted immediately before identifierWithValue in
identifierWithIntervalFunction. No alternation ORDER moves (AD-3 intact). It subsumes AD-10 —
the ten Tableau SQL-92 corpus shapes are a strict subset — so qualifiedQuotedIdentifier was never
shipped.

Measured blast radius. MAX("salary"), CAST("col" AS BIGINT), DATE_TRUNC("event_ts", MONTH),
UPPER("abc"), ABS("amount") and DATE_PARSE's first operand now read the double-quoted lexeme as
a COLUMN. Every genuine VALUE position is untouched (WHERE a = "x", >, BETWEEN, IN,
LIKE). Zero existing tests depended on the old reading — 604/605 green immediately after the
flip, the single failure being an unrelated deliberate widening. One-line revert if the lead
disagrees, at the cost of AC-5.

---

Task 0 — baseline re-measured on 66cfbcff (11 of 12 rows exactly as specced)

Statement	Spec expectation	Measured
SELECT UPPER("abc") AS u FROM t	parses; UPPER('abc'); name == ""	✅
SELECT MAX("salary") AS m FROM t	parses; MAX('salary')	✅
SELECT CAST("amount" AS BIGINT) AS a FROM t	parses; CAST('amount' AS BIGINT)	✅
SELECT "category" FROM t	parses; name == "category"	✅
SELECT e."category" FROM bi_events e	name == "e.", fieldAlias == Some("category")	✅
SELECT t.from FROM x t	name == "from", tableAlias == Some("t")	✅
SELECT a.[0] FROM t	name == "a.[0]"	✅
SELECT "" AS e FROM t	name == "", functions EMPTY	✅
SELECT "a\b" FROM t	name == "a\b" (one backslash)	✅
SELECT "amount" + 1 AS a FROM t	REJECTED (type mismatch)	⚠️ REJECTED, different reason: end of input expected
SELECT a . b FROM t	REJECTED	✅
SELECT id FROM bi_events "e"	REJECTED	✅

Row 10's verdict matches; its stated cause does not, and the real one is load-bearing — see the
|-commits limitation below.

Task 5.2 — ordering audit re-run at dev HEAD: ONE spec row was wrong

The audit recorded date_parse / datetime_parse as keeping the string reading "because they list
literal before identifier". True, and not decisive: literal sits at slot 4, BEHIND
identifierWithIntervalFunction at slot 2. They flip with the rest of the operand family
(DATE_PARSE("2024-01-01", …) is now the column named 2024-01-01; the documented single-quoted
spelling is unaffected). Corrected in-source and pinned. Every other row was measured and is
unchanged. No alternation ORDER moved anywhere.

AD-9 parse cost — the patched tree is FASTER, ranges non-overlapping

sql/testOnly *ParserSpec, 11 runs per tree in one sbt session, first discarded as warm-up:

tree	median	range (n=10)
baseline 66cfbcff	3.748 s	3.526 – 3.877
patched feature/21.1	3.018 s	2.938 – 3.218

−19.5 %, and patched-max < baseline-min. Plausibly the shorter first-part regex plus packrat
memoisation on smaller parts. Well inside the ~20 % escalation budget and in the good direction; not
over-claimed — ParserSpec is a suite, not a parse microbenchmark.

AD-11 — cross-story merge contract with 21.5 / #274

One shared file, sql/.../sql/package.scala. 21.1 landed FIRST and makes the ONLY executable
scanner edits (the backtick joins the quote set in all three scanners, with the backslash-escape
branch suppressed for it). '' doubling needs no scanner change, so 21.5 has no competing logic
hunk — it owns the merge and must re-run core/testOnly *SplitStatementsSpec,
sql/testOnly *ScriptFunctionParenSpec and sql/testOnly *QuotedIdentifierSpec.
unquoteName stays identifier-only; whoever lands second folds its q == '"' interior onto 21.5's
unescapeStringLiteral(content, delimiter) and leaves the backtick arm local (doubling only).
🔴 AD-13 strengthens 21.5's coherence argument: "x" is now an identifier in many more places,
so the single quote is the only unambiguous string delimiter in operand position.

AD-12 — interaction with 21.3

The bucketNames digit-in-name crash is pinned, not fixed, for BOTH quoted spellings, in
rejectsInternally shape (Left carrying Parser.InternalParseFailure + IndexOutOfBounds, via
21.4's boundary catch). 21.3 RETARGETS those two rows to its own rejection message — never
deletes them. And GROUP BY `1` is the column named 1 (name == "1", no functions), which is
what 21.3's ordinal predicate separates by construction.

21.4 hand-off re-check (dev-findings item 11)

identifierWithArithmeticExpression's default arm is a hard err and alternative #2 of 9 in
WhereParser.any_identifier. Verdict: still unreachable. Its arms cover
ArithmeticExpression | Identifier | FunctionWithIdentifier | Function, and every valueExpr
alternative including AD-13's yields an Identifier (GenericIdentifier is the sealed trait's only
implementor). The err was NOT softened to failure.

Deliberate behaviour changes, all pinned

• N1/N2 SELECT a..b, SELECT a. → rejected.
• N3 SELECT e."category" FROM bi_events e was a live silent corruption (rendered
  SELECT e AS category FROM bi_events AS e); now the column e.category.
• N4 SELECT "" is the empty string literal, not a nameless column.
• N5 "a\b" unescapes to ab, consistent with what the regex always accepted.
• Widenings: SELECT a . b, "a"::BIGINT / `a`::BIGINT, and a quoted FROM table alias.
• NOT a narrowing: SELECT a.[0] still parses.

Two residual limits, pinned and documented (not fixed)

1. Arithmetic cannot START with a quoted operand, unparenthesised. SELECT `amount` + 1 and
   SELECT "amount" + 1 are both REJECTED (on main too); SELECT amount + 1,
   SELECT (`amount` + 1) and SELECT MAX(`amount` + 1) all work. Cause: | COMMITS, and
   quotedIdentifier is first in SelectParser.field — the ordering that makes SELECT "category"
   a column. Needs a lookahead, not a reorder.
2. A name ending in a dot swallows the next word (rep("." ~> part) skips whitespace):
   ORDER BY b. DESC → ORDER BY "b.DESC" ASC, direction silently lost. Both inputs are malformed
   SQL whose old parse was equally nonsense. Hand-off to 21.2 / 21.7, which touch these
   productions.

PD-4 — release notes owed by 0.23.0

• 🔴 "x" inside a function, aggregate, conversion or arithmetic operand now means the COLUMN
  x, not the string x. Use single quotes for a literal there.
• .sql renderings change for any statement using a quoted identifier or alias — they gain canonical
  ANSI double quotes. Anything downstream pinning generated SQL text must look.
• GenericIdentifier and Alias gain a field ⇒ binary-incompatible (case-class arity) ⇒
  downstream rebuild on the next core bump. Source-compatible (defaulted, appended last; no
  case Alias(x) => pattern exists anywhere).
• CREATE OR REPLACE MATERIALIZED VIEW's "definition unchanged" short-circuit compares a stored
  render, so a view whose definition quotes a name rebuilds once at version + 1.

Verification

sql 654 · core 868 (both green on 2.13.16 and 2.12.20) · softclient4es-sql-bridge
129 · es6bridge 129 · softclient4es7-sql-bridge 129 · softclient4es8-sql-bridge
129 · macrosTests 19. Named gates, each run alone: ParserSpec 305 · ParserTotalitySpec
35 · AlterTableRoundTripSpec 4 · ScriptFunctionParenSpec 12 · SQLKeywordsSpec 9 ·
DialectCensusSpec 16 (source-derived; did not move) · SplitStatementsSpec 18.
scalafmtCheckAll and headerCheck green. + sql/compile / + core/compile and
++ 2.12.20 sql|core/Test/compile clean.

⚠️ softclient4es9-sql-bridge/test was not executed: it dies with UnsupportedClassVersionError
on the default JDK (elastic4s for es9 is Java-17 bytecode — 21.4's dev-findings item 20) and this run
is constrained to plain sbt. It compiles, and the identical copyBridge template ran 129 green
on the template, es6, es7 and es8 legs. CI runs temurin 17.

Also reproduced twice: es7 and es8 each printed [success] with no Tests: line on their first
invocation (stale generated tree) and 129 on the second.

Notes

• The softclient4es-web MDX twin of dql_statements.md is deferred to the 0.23.0 release doc
  sweep — the site documents released behaviour and this ships on 0.23.0-SNAPSHOT.
• No statement count is claimed here; 21.6 measures the real corpus movement by replay (PD-1).
• build.sbt:23 verified at 0.23.0-SNAPSHOT and left alone — feature/21.4 carried the bump.

Post-merge record — lead sign-off and the two issues this PR did not fix (2026-09-06)

Added after the merge, because both items were still open when it landed.

AD-13 — CONFIRMED by the lead (2026-09-06)

AD-13 reverses PD-3, a product decision that had already been reviewed: a double-quoted lexeme in a
function / CAST / aggregate / arithmetic operand position now means the COLUMN, where it used to
mean the string. The lead confirmed AD-13 and declined the one-line revert.

The grounds, restated so the reversal is a decision and not a discovery: PD-3 and AC-5 are mutually
incompatible once the render normalises to one canonical delimiter. MAX(amount) renders
MAX("amount") and, under PD-3, re-parses as MAX('amount') — an aggregate silently becoming an
aggregate over a constant. MaterializedViewExtension persists that render and runs
client.run(alter.sql), so it was a live corruption, not a cosmetic round-trip failure.

Measured costs, accepted: MAX("salary"), CAST("col" AS BIGINT), DATE_TRUNC("event_ts", MONTH),
UPPER("abc") and DATE_PARSE("2024-01-01", 'fmt') all flip from string to column. Value positions
are untouched — WHERE a = "x", >, BETWEEN, IN and LIKE each list literal ahead of the
identifier in their own alternation.

⚠️ Process note, recorded honestly: this PR was opened non-draft (the workflow requires
--draft) and merged before the lead had ruled on AD-13. The approval above is retrospective.
Nothing needs undoing, but a reversal of a lead-reviewed decision should block at the design gate in
future stories rather than ship and flag.

Filed as separate issues — NOT fixed here, deliberately

Both are pinned by tests and documented in documentation/sql/known_limitations.md, so they are
recorded decisions. Neither is fixable inside 21.1 without moving an alternation order AD-3 forbids.

• Arithmetic cannot start with a quoted operand unless parenthesised #284 — arithmetic cannot START with a quoted operand unless parenthesised. SELECT `amount` + 1
  is rejected while SELECT (`amount` + 1) and SELECT MAX(`amount` + 1) work. | commits to
  the first succeeding alternative and quotedIdentifier is first in SelectParser.field /
  WhereParser.any_identifier — the very ordering that makes SELECT "category" a column. Needs a
  lookahead, not a reorder. Pre-existing for the double-quoted spelling; 21.1 makes the backtick
  spelling inherit it.
• A name ending in a dot swallows the next word across whitespace, silently losing ORDER BY direction #285 — a name ending in a dot swallows the next word across whitespace. ORDER BY b. DESC
  parses as ORDER BY "b.DESC" ASC, silently losing the sort direction. Both this and
  SELECT a. AS x are malformed inputs whose pre-21.1 parse was equally nonsense, so it trades one
  broken reading for another. Closing it means a whitespace-free dot+part regex, which also reverts
  the deliberate SELECT a . b widening this PR pins. Hand-off to 21.2 / 21.7, which both touch
  these productions.

#252 stays OPEN — verified after the merge. Closes #252 rides 21.2's PR, as specced.

🤖 Generated with Claude Code