fix(sql,core): resolve temporal literals against the mapped date format before rendering the query (#276)

[fupelaqu]

Closes #276

Story BIDC-4 (BI defect-closure epic). Branch feature/BIDC-4 off origin/main e55bd937. Version line 0.23.0-SNAPSHOT verified, untouched.

What

WHERE event_ts >= '2026-06-04 00:00:00.000000' — the SQL-standard spelling Superset / SQLAlchemy render — was forwarded verbatim into the range/term query and rejected by Elasticsearch's default date format (strict_date_optional_time||epoch_millis); only the T-separated form matched. Superset's relative-date filter could not run, on JDBC and Flight SQL alike.

The string literals a WHERE clause compares against a date-mapped column are now resolved against that column's mapping format ONCE per statement, on the AST, before the Elasticsearch query is rendered:

• new sql object query.TemporalLiterals — walks where.criteria (Predicate, NESTED/CHILD/PARENT wrappers, GenericExpression with = <> != >= > <= <, literal BETWEEN, string IN), resolves a function-free identifier through schema.find, and rewrites the StringValue in place so the bridge receives the normalised value with zero bridge change and no second parse. LIKE/RLIKE, function-wrapped columns, TIME columns, keyword/text columns, HAVING and CASE are never touched.
• core seam SearchApi.resolveTemporalLiterals invoked from every SingleSearch -> ElasticQuery conversion: search / searchAsync (single statement and each UNION ALL request), the inner-hits search, ScrollApi.scroll — so GatewayApi.run (REPL, JDBC driver, Flight SQL sidecar and its cross-index JOIN legs, which re-enter gateway.run) is covered transitively. Schema access = the existing 5-minute IndicesApi.loadSchema cache via the this match { case indices: IndicesApi => … } runtime pattern (SELECT without LIMIT returns only 10 rows on the non-scroll search path #209 precedent) — no signature change; the new SearchApi trait field (the negative cache) means the usual downstream rebuild on 0.23.0.
• the same resolution is applied to the WHERE of DELETE / UPDATE … WHERE at the five by-query render sites in IndicesApi (review R4-4), so a literal the SELECT accepts is accepted by the DML too.

Format rule (AD-S4-1.1) — a strict-grammar APPROXIMATION, measured against all four ES parsers

format = the column's mapping format else the default; split on ||. Per literal, in order: a number (leniently, +1/1e3 included) or now-anchored date math is verbatim; a || date-math literal has its HEAD normalised by the same rule and its math kept (2026-06-04 10:30:15||+1d → 2026-06-04T10:30:15||+1d), never rejected; a literal a CUSTOM alternative already parses is verbatim (a format: "yyyy-MM-dd HH:mm:ss" column keeps working — the fix does not invert the defect); if an ISO optional-time built-in is present, the RECOGNISED calendar shape (yyyy-MM-dd, optionally T or ONE space, HH[:mm[:ss[.f]]], a zone) is validated with java.time — the space form is rewritten to T (fraction digits and zone preserved, a lower-case z upper-cased), the T form is verbatim; a literal that merely STARTS like an ISO date (a signed or 5-digit year, 2026-6-4, T1:02:03, an ordinal/week date, a zone id, an offset with seconds, a bare T, any unmodelled tail) is verbatim — Elasticsearch decides. A literal is rejected (400 naming literal, field and format) only when it cannot be an ISO date at all or carries an invalid recognised calendar/time component, and only under formats made solely of the STRICT ISO names and the epoch names; under the non-strict date_optional_time, a custom pattern or an unrecognised built-in nothing is rejected. The accept set is pinned as a table in TemporalLiteralsSpec against the real DateFormatter parsers of ES 6.8.23, 7.17.29, 8.18.3 and 9.0.3 (56 literals + the four zone shapes below; every REJECT row fails on all four, every VERBATIM is a no-op, and every value the rewrite can PRODUCE — a T form with an optional fraction and an optional Z / +01 / +0100 / +01:00 / +01:00:00 — is accepted on all four: zone ids (UTC, GMT+1) and other offset shapes (+010000, +0100:00, +01:0000), which some or all majors reject, are never produced and stay verbatim — second-look R4-13).

Schema-absent path = verbatim (AD-S4-1.2), stated

The lookup is skipped when the WHERE carries no candidate literal (zero cost for the common statement), when the FROM does not name exactly one concrete index (several sources, * wildcard, , list), when the client is not an IndicesApi, and when the schema cannot be loaded (an index alias, an unknown index without a template, a cluster error, a thrown lookup). An identifier qualified with a cross-index JOIN alias is never resolved against the FROM table's schema. A cache miss costs one GET <index> per index per 5 minutes, never per row — the lookup is synchronous on the caller thread (also on searchAsync/scroll, like the DDL executors' own loadSchema calls) and not coalesced across concurrent first statements on a cold index (accepted trade-off, R4-10). A 404 miss is remembered for the same TTL in a per-client negative cache (bounded like #238's shardCountCache: expired entries purged above 256, cleared above 1024), because loadSchema caches successes only and an alias source on the es8/es9 clients would otherwise pay two round trips + two WARN lines per statement; a transient non-404 failure is never remembered. An un-LIMITed searchAsync row query runs the pass twice (once in searchAsync, once in the scroll it routes to) — idempotent, one cache hit + one AST walk (R4-11).

Spike results (T2 — measured, not argued)

• T2a — the spec's "verified current state" premise was FALSE for every EXECUTING path. SingleSearch.update(Some(schema)) has exactly one production caller, Table.mergeWithSearch (CTAS/INSERT…SELECT target derivation). SearchApi.search/searchAsync/ScrollApi.scroll and GatewayApi's DQL executor never attach a schema (ecosystem-wide grep: sql/core/bridge/es*/macros/extensions/jdbc/arrow). AD-S4-1 as specced would have shipped with zero covered entry points; this PR adds the hop. A throwaway bridge probe (17 shapes, real parser + real bridge) showed: an operand CAN see the resolved column after update(Some(schema)) (GenericIdentifier.col = Some(event_ts DATE); col is not exposed on the sealed Identifier trait), a mapping format survives into Column.options as a StringValue, and attaching the schema globally is emission-neutral on 13/13 parsing shapes. AD-S4-2 (dev decision, for the lead to veto): the hop is a statement-level Either-returning pass invoked from core rather than a global update(Some(schema)) inside GenericExpression.update — AC 6 needs a rejection seam and update() cannot return one without a throw (forbidden after 21.4), and the pass does not re-run the whole update() nor flip col/out for every identifier.
• T2b — no JOIN-venue divergence. Call chain: JoinPlanner.plan parses only to plan (JoinPlanner.scala:109); each leg is re-issued as a SQL string — SubQueryExecutor.scala:307 → LocalConnection (gateway: GatewayApi) → ElasticSqlExecutor.scala:112/127/144 gateway.run(sql) → core GatewayApi.run → SearchExecutor → api.searchAsync(single) / api.scroll(single). Legs re-enter the SAME core seam as direct queries; whatever the SELECT path does covers both venues.
• date_nanos resolves to SQLTypes.Any from a mapping (SQLTypes.apply default arm) and is therefore excluded by construction — pinned by a unit test over a real date_nanos mapping document (AC 4: excluded, not covered).
• Four temporal-vs-string shapes do not even parse on main (DATE_TRUNC(ts, DAY) >= '…', CAST(ts AS DATE) = '…' are parse-time type-mismatch rejections; DATE(ts) / CAST('…' AS DATE) in WHERE are end of input expected). Out of scope (function-wrapped operands never reach rangeQuery), recorded for 21.3/21.5.

Covered entry points (AC 5)

Entry point	Coverage
SearchApi.search (single, UNION ALL)	TemporalLiteralSearchSpec (core, Docker-free) + TemporalLiteralSpec on real ES
SearchApi.searchAsync	TemporalLiteralSearchSpec
ScrollApi.scroll	TemporalLiteralSearchSpec (rejection route) + un-LIMITed row queries in TemporalLiteralSpec (routed through scroll by #209)
GatewayApi.run (REPL / JDBC / sidecar / JOIN legs)	TemporalLiteralSpec gatewayIds on real ES
inner-hits search	seam wired, same helper (no dedicated test — deprecated API)
schema-absent: no candidate / multi-source / wildcard / unknown index	TemporalLiteralSearchSpec (verbatim + lookup count 0/1)

Tests

• sql: TemporalLiteralsSpec (25) — format classification, every literal rule, statement walk incl. alias, AND/OR, NOT/!=/NOT IN, UNNEST nested wrapper, keyword/TIME/LIKE/function/unknown/custom-format negative controls, rejection messages (not startWith "Internal parser error"), real-mapping resolution incl. date_nanos exclusion.
• bridge template + hand-maintained es6/bridge: TemporalLiteralQuerySpec (6) — the GENERATED JSON: range/term/terms carry the T form; the keyword negative control is byte-identical with and without the schema (AC 3); epoch millis and the custom-format column verbatim.
• core: TemporalLiteralSearchSpec (14) — the seam on a recording NopeClientApi: rewrite on search/searchAsync/UNION ALL, verbatim when the schema is absent / several sources / wildcard, zero lookups when no candidate, 400 rejection naming literal + field on search, searchAsync and scroll (stream fails with the same error), the client never reached.
• testkit template TemporalLiteralSpec + five client subclasses (es6 rest, es6 jest, es7, es8, es9): 8-row default-format date index — the three spellings select the SAME ids e4..e8 (AC 1), equality/BETWEEN/IN/< in the space form, epoch millis (AC 4), keyword negative control on a date-shaped string (AC 3), custom format: "yyyy-MM-dd HH:mm:ss" index parity (AC 2), gateway venue, AC 6 rejection.
• Falsification: with SearchApi.scala/ScrollApi.scala reverted to origin/main, 7/10 core seam tests go red with the original symptom (space form forwarded verbatim, no 400, scroll stream does not fail) and the 3 negative controls stay green; on real ES 8.18 the same revert turns 4/7 integration tests red with the issue's exact error (search_phase_execution_exception … all shards failed) while the epoch / keyword / custom-format controls stay green.
• Code review (bmad-code-review, inline layers): 7 patch findings applied before commit — lower-case z zone in the rewrite, T-form/date-only literals with invalid calendar values now rejected under fully-understood formats (AC 6 consistency), rejection literal bounded (120 chars) + control characters collapsed (20.4 log/SQLException discipline), now date-math grammar tightened (nowhere is not date math), cross-index JOIN-alias identifiers never resolved against the FROM schema, the negative-miss cache above, es6 bridge import. 2 pre-existing items deferred (below).

Evidence

Suite (final tree = 6eeffebe, after the review follow-up)	Result
sbt "+ sql/test" (2.12.20 + 2.13.16)	679 / 679 on both legs (TemporalLiteralsSpec 24 incl. the 56-literal oracle table)
sbt "core/test" (2.13.16) + ++ 2.12.20 core/Test/compile	880 / 880 (TemporalLiteralSearchSpec 12) · 2.12 compiles
softclient4es-sql-bridge/test / es6bridge/test / softclient4es7-sql-bridge/test / softclient4es8-sql-bridge/test	135 / 135 each
sbt17 "softclient4es9-sql-bridge/test"	135 / 135
++ 2.12.20 sql/Test/compile	green
ES 6.8 sbt17 "es6rest/testOnly *TemporalLiteralSpec"	9 / 9 (incl. UPDATE + DELETE)
ES 6.8 sbt17 "es6jest/testOnly *TemporalLiteralSpec"	9 / 9
ES 7.17 sbt17 "es7rest/testOnly *TemporalLiteralSpec"	9 / 9
ES 8.18 sbt17 "es8java/testOnly *TemporalLiteralSpec"	9 / 9
ES 9.0 sbt17 "es9java/testOnly *TemporalLiteralSpec"	9 / 9
Commit 5 (delta-review fixes): sql/testOnly *TemporalLiteralsSpec 25/25, core/testOnly *TemporalLiteralSearchSpec 14/14, ++ 2.12.20 Test/compile (sql + core), scalafmtAll alone, five ES legs of TemporalLiteralSpec	11/11 on ES 6.8 rest, 6.8 jest, 7.17, 8.18, 9.0
Commit 4 (aliases): sql/testOnly *TemporalLiteralsSpec, core/testOnly *TemporalLiteralSearchSpec, bridges, ++ 2.12.20 test compile, es8java/compile, five ES legs of TemporalLiteralSpec (now 11 cases)	sql/test 680/680, core/test 881/881, ++ 2.12.20 Test/compile (sql + core) green, TemporalLiteralSpec 11/11 on ES 6.8 rest, 6.8 jest, 7.17, 8.18 and 9.0
ES 8.18 falsification on commit 1 (core seams reverted to origin/main)	4 / 7 RED with search_phase_execution_exception … all shards failed; epoch / keyword / custom-format controls green
Parser oracle (DateFormatter.forPattern(spec).toDateMathParser()), 56 literals × {default, date_optional_time, nanos}	run against elasticsearch-6.8.23 (JDK 11), 7.17.29 (JDK 11), 8.18.3+lucene-core-9.12.1 (JDK 21) by the reviewer and 9.0.3+lucene-core-10.1.0 (JDK 21) for this follow-up — 9.0.3 identical to 8.18.3

Release note (0.23.0) — AC 7

• Behaviour change, customer-visible (good): a space-separated timestamp literal compared to a date column ('2026-06-04 00:00:00', '2026-06-04 00:00:00.000000', with or without a zone offset) now matches the same rows as the ISO T form on every venue (client API, REPL, JDBC, Flight SQL, JOIN legs). Previously it failed with search_phase_execution_exception … failed to parse date field.
• Error message change: a literal that cannot be a date under the default mapping format ('not-a-date', '2026-13-45 99:99:99', '') now fails BEFORE reaching Elasticsearch with HTTP 400 Cannot parse '<literal>' as a date/time value for date field '<field>' (mapping format '…'). Anything matching the raw Elasticsearch shard error text will not see it any more.
• Keyword negative control: keyword/text columns compared to a date-shaped string are byte-identical to before — the rewrite keys on the MAPPED type, never on the literal's shape. Custom-format date columns whose format already parses the literal are untouched. date_nanos columns are not normalised (they resolve to ANY on the AST) — the space form still fails there as before.
• DELETE / UPDATE … WHERE get the same resolution as SELECT (the space form now deletes/updates the rows the SELECT matches; a literal the SELECT rejects is rejected with the same 400, operation = deleteByQuery|updateByQuery).
• Zone ids in the space form are no longer rewritten (deliberate, since a8b8453e): the rewrite emits only values every shipped major accepts — a T form with an optional fraction and an optional Z / +01 / +0100 / +01:00 / +01:00:00. '2026-06-04 10:30:15UTC' and other unmodelled zone shapes are forwarded verbatim and judged by Elasticsearch (7+ accepts a zone id, 6.8 does not), exactly as when the user types the T form.
• Index aliases (lead ruling, commit 4): an alias over exactly ONE index now resolves to that index's mapping on every client — the temporal-literal resolution applies through it, and so do SHOW TABLE / DESCRIBE / INSERT INTO … SELECT / COPY INTO through such an alias (they used to fail "not found" on es8/es9 and see an empty schema on es6/es7). An alias over SEVERAL indices is ambiguous and is now reported as not found on every client (es6/es7 used to answer an empty schema): the literal is forwarded verbatim there, as before.
• New network call class on the SELECT path: when a WHERE compares a string literal to a column and the FROM names one concrete index, the index mapping is read through the schema cache (one GET <index> per index every 5 minutes). Statements without such a predicate, multi-index/wildcard reads and unloadable schemas skip it and behave exactly as before.

Aliases — lead ruling on R4-5: extend this PR (commit 4)

GET /<alias> answers with the concrete index document(s) keyed by THEIR names. The rule lives in ONE place, Index.apply(name, root): when root has no entry named name, an alias over exactly one index resolves to that index's document (the schema keeps the alias as its name); an alias over several is ambiguous and IndicesApi.loadIndexAsSchema reports it not found (uniform on every client — es6/es7 REST and jest already returned the concrete-keyed body; es8/es9 executeGetIndex now hand the whole concrete-keyed map through instead of looking the alias up as a key). Audited getIndex consumers (IndicesApi.scala:1219 insertByQuery, :1458 copyInto): both already map None to a named "not found", so a single-index alias target now resolves its metadata (primary key, mappings) instead of failing, and a MULTI-index alias target now fails with that honest "not found" instead of silently proceeding on one member's mapping (es6/es7) or a 404-after-template (es8/es9). SHOW TABLE / DESCRIBE through a single-index alias resolve for the same reason. A resolved alias is a schema-cache SUCCESS, so the negative cache never remembers it as a miss; the cache is keyed by the alias name (a remap is seen after the 5-minute TTL). No optional "all members agree" merge for multi-index aliases — kept out of scope for safety. Tests: Index.apply rule (sql, GET-index JSON with one / two documents), the core seam through an overridden executeGetIndex (single → rewritten, multi → getIndex = None and verbatim), and TemporalLiteralSpec alias cases (single alias: three spellings via search + run; two-index alias: no named 400) on the five clients.

Delta review of the alias work (commit 5)

APPROVE, no HIGH/MEDIUM. Two of its LOW findings were silent-failure shaped and were fixed rather than recorded: R4-18 — the ambiguity test counted mapping-BEARING documents, so a two-index alias whose other member has no mapping resolved silently to the first; it now counts top-level index entries, with the mappings/settings/aliases check kept only as the shape test (one index document vs a map of them), pinned by a unit case. R4-21 — invalidateSchema(<concrete>) left a cached ALIAS schema behind, so after an ALTER TABLE a query through the alias kept a stale mapping (hence a stale format) for up to the 5-minute TTL; Index now carries the concrete name it resolved from, IndicesApi keeps an alias → target map beside the schema cache, and invalidating or updating an index drops every alias entry pointing at it. Also R4-19 (a bounded ambiguous alias is now asserted: second statement, zero further lookups), R4-20 (the testkit multi-alias case asserts the literal is verbatim IN THE EMITTED QUERY — the alias spans two custom-format indices so the query reaches ES and its body can be read), R4-17 (a resolved alias is no longer listed among its own ALIASES, so SHOW CREATE TABLE <alias> has no self-referential entry) and R4-22 (these test counts). R4-14 stays recorded, not fixed: es6/es7 flatten any ≥ 400 on GET <index> into a synthetic 404 that is then remembered as a miss — pre-existing #184 family.

Independent review follow-up (commits 2 and 3 on this branch)

Second look on 6eeffebe: APPROVE. Commit 3 lands the three LOW nits — R4-13 zone grammar tightened to the four offset shapes + Z (oracle rows added), R4-15 local deferred-work record updated, R4-16 test ADT nested in the spec. R4-14 (es6/es7 REST flatten any >= 400 on GET <index> to a synthetic 404, so such a status is remembered as a miss for 5 min on those clients) is pre-existing #184 territory — recorded in the findings log, not changed here.

R4-1 false 400 under date_optional_time (non-strict name dropped from the rejection-eligible set) · R4-2 conservative rejection (only "cannot be an ISO date" or invalid recognised component; unmodelled tails verbatim; oracle table on 4 majors incl. ES 9.0.3 run for this follow-up) · R4-3 negative cache bounded (404-only, purge > 256, clear > 1024) · R4-4 DML WHERE wired at the five by-query render sites + 2 integration cases · R4-6 stale "date + zone" claim removed · R4-7 || head normalised · R4-9 docs JOIN sentence · R4-12 licence headers on the sql/bridge test files · PR-body wording (R4-8/R4-10/R4-11).

Coordination notes (T6)

• ↔ 21.1 (merged): the quoted flag never enters the name comparison — GenericIdentifier.update resolves col with the unquoted name and this pass keys on identifier.name the same way; the rewritten where.sql re-renders quoted names through Identifier.sql unchanged.
• ↔ 21.5 / SQL-standard single-quote escaping ('') is unsupported anywhere — any literal containing an apostrophe is a parse error #274 (not started): this pass consumes StringValue.value AFTER lexing; '' doubling changes only the lexer. Either order works; no shared file.

Docs

documentation/sql/dql_statements.md — new ### Temporal literals against date columns under ## WHERE. Web MDX twin (softclient4es-web/src/content/docs/sql/dql.mdx) to be synced by the orchestrator (feedback_dual_docs_sync).

🤖 Generated with Claude Code