Skip to content

feat(sql): quoted and qualified table names in FROM, JOIN and DELETE (story 21.2) - #294

Merged
fupelaqu merged 2 commits into
mainfrom
feature/21.2
Sep 6, 2026
Merged

feat(sql): quoted and qualified table names in FROM, JOIN and DELETE (story 21.2)#294
fupelaqu merged 2 commits into
mainfrom
feature/21.2

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Closes #252
Closes #85
Closes #292
Closes #293

Story 21.2 — the FROM/JOIN table-name surface. FROM (and JOIN, and DELETE FROM) becomes a
call-site swap onto story 21.1's exported lexing surface plus a leading-qualifier rep, and the
qualifier a statement writes is preserved in the AST instead of being discarded.

The rule

Quoting is the discriminator, and it is #57's shipped decision generalised — not replaced. A
qualifier is the maximal leading run of parts that are each quoted AND each followed by a dot;
the index name is everything after it. A bare dot never separates, which is what has always
protected logs-2025.03.

written index read qualifier
FROM elastic.bi_events elastic.bi_events
FROM "elastic".bi_events bi_events elastic
FROM `elastic`.`bi_events` bi_events elastic
FROM "logs-2025.03" logs-2025.03
FROM "elasticsearch"."prod-cluster"."bi_events" bi_events 2 levels

The parser preserves, it does not interpret (#85). Table.name stays byte-for-byte its previous
value; Table.parts / StandardJoin.parts record the ordered NameParts. Nothing in sql, core,
bridge or any client reads parts — role assignment (schema? catalog? server alias?) belongs to
the resolver, which knows the venue.

Task 0 baseline — re-measured on the CURRENT origin/main (99d1f438), not the spec's ac54a079

The spec's baseline was four merges stale, and two of its "rejected today" rows had become live
defects
, both created by 21.1:

statement on 99d1f438 after
JOIN "prod_us".customers c parses, JOIN leg reads index prod_us.customers while the FROM leg reads orders both read the bare index
FROM elastic."bi_events" parses, reads index elastic. (the quoted lexeme was stolen as the ALIAS) elastic.bi_events
FROM bi_events `bi_events` / FROM bi_events "bi_events" parses (21.1's alias flip) unchanged
JOIN `customers` c / JOIN "customers" c parses (21.1's source flip) unchanged

The six review rows reproduced exactly: the CROSS JOIN message verbatim,
FROM "prod_us".orders o, "prod_eu".orders pListMap(orders -> p),
CREATE TABLE dest (COL INTEGER) parsing, CREATE LOCAL TEMPORARY TABLE rejecting with
regex '(?i)OR\b' expected but 'L' found, FROM elastic."bi_events" and the ON-less/ON-ful CROSS
JOIN pair. Corpus counts re-derived and confirmed (25 backtick + 23 double-quote qualified
statements, zero overlap; 24 DDL probes).

The four Issue-#57 tests are green, byte-unmodified

ParserSpec.scala's // ── Schema-qualified table names (Issue #57) ── block is untouched; this PR
appends two tests to it. If one of the four had gone red, the implementation would have joined
the parts and AD-1 would have been violated.

Design decisions taken at dev time

  • AD-9tableParts uses 21.1's joined qualifiedName; qualifiedName itself was not
    touched, so 21.1's "one lexing surface" holds with a zero-line diff. Splitting the tail would
    render elastic."bi_events" as "elastic"."bi_events", which re-parses to a different index.
  • AD-8qualifierPart keeps a separate dot terminal rather than one adjacency-strict
    regex. Measured: the strict form sends FROM "elastic" .bi_events (accepted today, reads
    bi_events) down qualifiedName, whose tail matches .bi_events adjacently, and it would
    silently start reading elastic.bi_events.
  • AD-6′ — the alias-map key is the qualified reference only where the bare name is ambiguous
    in its own FROM. The unconditional form was implemented first and measured to break
    Identifier.table and two softclient4es-extensions call sites that look tables up by the bare
    index name, for statements that work today.
  • AD-6″ (review follow-up) — Identifier.table is an alias-map key, so anything in sql
    comparing it against an index name must speak the same key language. New From.joinSourceKeys;
    TemporalLiterals' cross-index join-leg guard consumes it.
  • AD-7′ (review follow-up) — Join.validate's CrossJoin exemption was unsatisfiable for every
    input before and after AD-7, so it is deleted rather than left as a second, unreachable
    statement of the same rule.

Consumer sweep for tableAliases / aliasesToTable / joinAliases (AC-12)

site how it reads the map status
sql/…/package.scala:1266 Identifier.update reverse lookup → Identifier.table key language now shared via joinSourceKeys
sql/…/query/TemporalLiterals.scala:304,349 compares Identifier.table against join sources fixed in this PR
sql/…/query/OrderBy.scala:58 values only unaffected
sql/…/query/package.scala:119 re-exposes the map inherits
aliasesToTable no consumer beyond its definition, in this repo or any sibling
extensions graph/JoinDependencyGraph.scala:517-528, graph/FieldAnalyzer.scala:94,97,152,250,296 look tables up in a schemas map keyed by the bare index why AD-6′ is conditional; unaffected for every unambiguous statement
arrow planner/PredicatePushdown.scala:52-65 a Set of aliases unrelated

#85 — the consumer note

softclient4es-arrow's JoinPlanner.extractCatalogList / stripAllCatalogPrefixes can retire for
the FROM/JOIN half on the next core bump and read Table.parts / StandardJoin.parts instead,
which also removes nextCatalogFor's positional mis-association. A retiring resolver must treat
an UNREGISTERED qualifier as "no qualifier", not as an error
— after story 20.3 a BI tool's
qualifier is routinely the cluster name.

⚠️ The spec's "arrow is unaffected by anything this story does" is refuted:
CatalogTablePattern needs a backtick qualifier + a bare table name, so a fully quoted
FROM `prod_us`.`orders` is never stripped, now parses, and both legs silently run against the
default cluster where they used to fail loudly. Cross-repo, so not fixable here — warned in
joins.md and known_limitations.md, and recorded in the local arrow follow-up record.

Release notes

  1. Table.sql / From.sql renderings now carry the qualifier the statement had, canonicalised to
    the ANSI double quote. Anything pinning generated SQL text must look.
  2. Table and StandardJoin gained a parts field ⇒ binary-incompatible; downstream rebuild
    required. Shared with 21.1's GenericIdentifier/Alias note — one line for the train.
  3. CREATE OR REPLACE MATERIALIZED VIEW on an unchanged definition REBUILDS once after the upgrade
    (the extension string-compares the stored render against a fresh one). Not data loss; an
    expensive surprise on first re-deploy. Shared with 21.1 — one train-level line.
  4. CROSS JOIN without ON now validates; the alias maps of a table whose bare name is ambiguous
    in its own FROM
    are keyed by its qualified name (visible only to callers introspecting
    From.tableAliases / aliasesToTable / Identifier.table).
  5. A JOIN leg with a quoted qualifier reads a different index. JOIN "prod_us".customers c
    read prod_us.customers before and reads customers now, matching what the FROM leg of the
    same statement has always done. Write the dotted name unquoted to keep the old reading.
  6. A JOIN source can no longer be an expression. JOIN customers::BIGINT c and
    JOIN DISTINCT customers c parsed before — the cast and the DISTINCT reached nothing, while the
    render kept them — and are loud rejections now, matching the FROM side.
  7. CREATE WATCHER … FROM a JOIN b ON … parses the JOIN and then silently discards it #191's multi-index watcher guard now fires on a cross-qualifier FROM.
    CREATE OR REPLACE WATCHER … FROM "a".orders o, "b".orders p WHERE o.x = 1 … was accepted and is
    now rejected. A wholly unqualified self-join still defeats it, as before.
  8. Federation only: a fully quoted cross-cluster reference now parses and silently runs against
    the default cluster where it used to fail loudly. Quote the prefix, leave the table name bare
    (FROM `prod_us`.orders), until the arrow-side retirement lands.

Story 21.7 owns the rest

Parser.ident — quoted names for INSERT / UPDATE / CREATE / DROP / ALTER, table and
column — is not in this PR and stays rejected, pinned as such. known_limitations.md names the
residual as DML/DDL-only. The #-prefixed temp-table probe outcome and the temp-table product
decision ride 21.7.

Verification

sql 772 · core 887 · softclient4es-sql-bridge 165 · macrosTests 19 ·
+ sql/test and + core/test green on 2.12.20 and 2.13.16 · + sql/compile / + core/compile
· ++ 2.12.20 sql/Test/compile · scalafmtCheckAll. No emitted-JSON change — the index list is
byte-identical by construction. build.sbt's 0.23.0-SNAPSHOT verified and untouched. No core,
bridge or es{N} source was modified. Every SQL example added to the docs was parse-probed
against the built parser.

🤖 Generated with Claude Code

fupelaqu and others added 2 commits September 6, 2026 20:09
…(story 21.2)

The FROM/JOIN table-name surface becomes a call-site swap onto story 21.1's
exported lexing surface plus a leading-qualifier `rep`, and the qualifier a
statement writes is PRESERVED in the AST instead of being discarded.

Lexing (#252 part 2)
- new `Parser.qualifierPart` / `Parser.tableParts`; `FromParser.table`,
  `source` and `join` rewritten onto them, `quotedSchemaPrefix` deleted
- `identifierRegexStr` / `identifierRegex` deleted: zero callers left in this
  repo and zero in jdbc / arrow / extensions
- `source` now returns a partially-built `StandardJoin`, which types
  `(unnest | source)` and removes an unchecked erasure pattern in `join`

Semantics (#85) - the parser preserves, it does not interpret
- `Table.parts` / `StandardJoin.parts` record the ordered `NamePart`s the
  statement wrote; `Table.name` stays byte-for-byte its previous value and
  `SingleSearch.sources` is unchanged for every statement that parsed before
- `Table.render` emits each part as ONE lexeme (never split on its dots) with
  qualifier parts always quoted; `StandardJoin` overrides `sql` to use it, so a
  dotted index name on a JOIN leg cannot be re-quoted into a different index
- the render stops deleting the qualifier - in SELECT, DELETE, CTAS and
  MATERIALIZED VIEW bodies alike

Two approved fixes in the same surface
- CROSS JOIN without ON now parses and validates: `Join.validate`'s CrossJoin
  exemption was unreachable dead code behind `StandardJoin.validate`'s
  unconditional ON requirement. A bare JOIN without ON stays rejected
- two tables differing only by qualifier keep both aliases: `From.tableAliases`
  keys an entry by its qualified reference when - and only when - this FROM
  uses that bare name for more than one distinct qualified reference. The
  unconditional form the spec prescribed was measured to break `Identifier.table`
  and two softclient4es-extensions call sites that look tables up by the bare
  index name

Also fixed in-branch, found while implementing
- `Delete.sql` rendered the bare index, so a DELETE's qualifier vanished from
  the rendering and the AST fixed point broke once `Table` carried `parts`
- `FromlessSelect.toSingleSearch` built its `Table` without `parts`, so the
  rewrite it documents as parser-equivalent was no longer AST-equal to one

Behaviour changes worth a release note
- renderings now carry the qualifier, canonicalised to the ANSI double quote
- `Table` and `StandardJoin` gained a field => binary-incompatible
- a JOIN source can no longer be an expression (`JOIN customers::BIGINT c` and
  `JOIN DISTINCT customers c` parsed before, and silently discarded the cast)

Tests: `QuotedTableNameSpec` (39) and `QuotedTableRoundTripSpec` (40) new, two
appended to ParserSpec's Issue-#57 block (its four originals untouched and
green), one retargeted in QuotedIdentifierSpec. sql 768, core 887,
softclient4es-sql-bridge 165, macrosTests 19; `+ sql/test` and `+ core/test`
green on 2.12.20 and 2.13.16; `++ 2.12.20 sql/Test/compile` green.

Docs: a Qualified and quoted table names section in dql_statements.md and a
rewritten quoting section in known_limitations.md; every SQL example in both
was parse-probed against the built parser.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…w follow-ups (story 21.2)

Independent-review findings on feature/21.2. The lead ruled: KEEP AD-6' (the alias-map key is
qualified only where the bare name is ambiguous) and fix the desync it left behind.

F1 - the desync AD-6's scaladoc claimed did not exist
`Identifier.table` IS a `From.tableAliases` key, so in the ambiguous branch it is the QUALIFIED
reference - while `TemporalLiterals` built its cross-index join-leg guard out of the BARE
`joinAliases` source names. Measured on
`SELECT o.a, p.b FROM orders o JOIN "prod_eu".orders p ON o.cid = p.id`: main compares
`prod_eu.orders` against `{prod_eu.orders}` and the guard fires; the branch compared it against
`{orders}` and it did not, so a temporal literal was resolved against the wrong index's schema.
New `From.joinSourceKeys` applies the same `aliasKey` to every JOIN leg and `TemporalLiterals`
consumes it. The AD-6' scaladoc no longer claims to be self-sufficient and states the obligation
any new `Identifier.table` consumer inherits.

F4 - `Join.validate`'s `CrossJoin` exemption was STILL dead after AD-7
`Unnest` fixes `joinType = None` and `StandardJoin` returns on its own `on` match before
delegating, so the predicate is unsatisfiable for every input, before and after. AD-7 duplicated
the exemption rather than making the original reachable. Deleted, under the same rule that deleted
`identifierRegex` in this story.

F3d - an UNDECLARED behaviour change, now pinned
`CREATE OR REPLACE WATCHER ... FROM "a".orders o, "b".orders p WHERE o.x = 1 ...` is ACCEPTED
before this story and REJECTED after: #191's multi-index guard tests whether any predicate
identifier resolves to a table, and AD-6' keeps both aliases where the map used to drop one. A
wholly unqualified self-join still defeats it. Source comment corrected, both halves pinned.

F5 - the `joinReferences` comment stated the OPPOSITE of measured behaviour (two same-name JOIN
legs under different qualifiers do NOT collapse; keying them bare is what would collapse them).

F8 - pins for behaviour that already measured correct: the mixed qualified/bare ambiguous FROM,
the three-table mix where one name is ambiguous and another is not, and the F1 interaction.

Documentation
- F2: the spec's "arrow is unaffected by anything this story does" is REFUTED. Federation's
  `CatalogTablePattern` matches a backtick qualifier + a BARE table name, so a fully quoted
  ``FROM `prod_us`.`orders` `` is never stripped, now parses, and both legs silently run against
  the default cluster where they used to fail loudly in the parser. Cross-repo, so not fixable
  here: warned at the site that publishes the syntax (joins.md) and in known_limitations.md, and
  `docs/issues/local-arrow-positional-catalog-loss.md` records the loud-to-silent transition.
- F3a: a JOIN leg with a quoted qualifier reads a DIFFERENT index than on current main
  (`prod_us.customers` -> `customers`) - documented as a behaviour change, not a widening.
- F3b: AC-2's "the index never changes" wording was too strong; both shapes that move go from a
  wrong index to the intended one, and both are pinned. Test title softened to match.
- F6: "Four things" listed five. F7: the qualified key shares a namespace with real dotted index
  names - recorded, deliberately not engineered around.

sql 772, core 887, softclient4es-sql-bridge 165, macrosTests 19; `+ sql/compile`,
`+ core/compile`, `++ 2.12.20 sql/Test/compile` and `scalafmtCheckAll` green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fupelaqu
fupelaqu marked this pull request as ready for review September 6, 2026 21:03
@fupelaqu
fupelaqu merged commit 8c42625 into main Sep 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment