fix(bridge,client,core): STDDEV/VARIANCE over a transformed expression — per-major emission (#222)

[fupelaqu]

Closes #222

Story BIDC-3 (BI defect-closure epic). PR base = main (with origin/main merged in, so D-1 could be fixed alongside the refusal it must be symmetric with — lead ruling 2026-09-06). (BIDC-2's PR #290 merged into feature/BIDC-2 rather than main, so this child was never carried onward and #222 stayed correctly open; feature/BIDC-3 sits directly on 10f3f8fc, which IS on main.) Superseded note: it was originally cut as a stacked branch — PR base = feature/BIDC-2 (STACKED, lead decision 2026-09-06) — BIDC-2's PR is held, not merged, and this story builds on its corrected buckets_path / naming (ElasticAggregation.scala neighbourhood). Cut from origin/feature/BIDC-2 4368d9dc, rebased onto BIDC-2's head 10f3f8fc (its review follow-ups a5f376f2 · b09aaf27 · b5a86cf9 · a363f579 · 10f3f8fc — every rebase clean; the last two touch BOTH AggregationNamingSpec copies, so the merged tree was re-compiled and re-run, not just merged). git log origin/main..feature/BIDC-3 therefore lists BIDC-2's two commits as well — expected; GitHub retargets to main when BIDC-2 merges. Version line 0.23.0-SNAPSHOT verified, untouched. No publishLocal, no GitHub issue operation. Upstream reference: Philippus/elastic4s#4100 (no closing keyword — a WATCH item, see below).

Per-major behaviour (lead ruling R-1b)

Major	Before	After this PR	Mechanism
ES 8 / ES 9	statistic of the RAW field ("extended_stats":{"field":"createdAt"}), or extended_stats: {} rejected by ES	computed over the transform — the emitted extended_stats carries its script	client-module serializer on elastic4s's 2-arg SearchBodyBuilderFn with a handler for the bridge's marker (AD-S3-1′)
ES 7	same silent wrong statistic	computed over the transform	elastic4s 7.17.26 (nl.gn0s1s, backport #4105) emits the script natively; the module UNWRAPS the marker and lets the stock 1-arg builder render it (AD-S3-4)
ES 6 (rest + jest)	same	refused, 400, naming Elasticsearch 6 — permanent	refusing serializers in es6/rest and es6/jest; the 6.x elastic4s line is unmaintained, so no fix can reach it

Scope extension (lead, 2026-09-06): ES 7 was specced — and first shipped here — as a loud reject
"lifted when the 7.17.26 backport ships". It shipped, so the lift is part of this PR.
RestHighLevelClientExtendedStatsRejectionSpec is replaced by
RestHighLevelClientExtendedStatsEmissionSpec, and the testkit's #222 cases branch on
computesTransformedStats (ES ≥ 7) so the ES 7 leg asserts the same value oracle as ES 8 / ES 9.

The ES 7 fork migration — and what it surfaced

com.sksamuel.elastic4s 7.17.4 → nl.gn0s1s 7.17.26 (7.17.26 exists only under the fork; the
sksamuel 7.x line stops at 7.17.4). Package names are unchanged, so the entire es7 tree recompiled
with zero source changes on both Scala legs.

🔴 It needed a Jackson exclusion, and resolution alone did not reveal that. 7.17.26 is a recent
release of an old line: it declares jackson-databind/core/annotations 2.22.2, while this build
curates a coherent Jackson per ES major. Resolved as-is, that left databind 2.22.2 beside the curated
jackson-module-scala 2.19.0, which validates its databind at class-init — so every emitted query
died with ExceptionInInitializerError ("Scala module 2.19.0 requires Jackson Databind >= 2.19.0 and
< 2.20.0 - Found jackson-databind version 2.22.2") on the first XContentBuilder.string. Fixed with
excludeAll(jacksonExclusions) on both es7 elastic4s artifacts — exactly what elasticDependencies
already does for org.elasticsearch:elasticsearch. Verified twice: the classpath re-resolves to one
coherent 2.19.0 set, and the suites that exercise that path run green.

es8 / es9 are unchanged and stay pinned at 8.18.2 / 9.0.0 (lead decision): the latest 8.x (8.19.1)
and 9.x (9.3.0) releases predate #4106 / #4100, so their builder still drops the script — the marker
and handler remain.

Raw-field STDDEV(salary) is byte-identical on every major (pinned). Every other scripted metric (MAX(YEAR(x)), …) is untouched (pinned: the es8/es9 serializer is byte-identical to the default for any request without a marker, over 7 shapes incl. HAVING/bucket_script/nested).

Why the handler cannot key on ExtendedStatsAggregation — AD-S3-1′ (LEAD-RATIFIED 2026-09-06)

The spec's AD-S3-1 named a handler keyed on ExtendedStatsAggregation: that mechanism is refuted by measurement, not merely superseded — it is unreachable dead code, independently re-derived from the elastic4s 8.18.2 / 9.0.0 sources by the fresh-context reviewer and ratified by the lead. The seam's SHAPE (client-module-injected serializer; template contributes only a discriminator; es6/es7 reject in the client modules) is unchanged; only the handler's key moves, to a marker type.

elastic4s's AggregationBuilderFn (8.18.2, byte-identical in 9.0.0) matches the typed arm case agg: ExtendedStatsAggregation => ExtendedStatsAggregationBuilder(agg, handler) BEFORE case other => customAggregationHandler(other) — the customAggregation seam is consulted only for a type the library does not know. So the bridge binds a transform-bearing extended_stats (both binds: plain :199 and windowed :225, es6 :200/:226) to a template-owned marker ScriptedExtendedStatsAggregation(inner: ExtendedStatsAggregation) — an elastic4s Aggregation whose abstract members are byte-identical on 6.7.8 / 7.17.4 / 8.18.2 / 9.0.0 (javap) — and the es8/es9 handler keys on the marker. The template stays major-agnostic; the seam is a SearchBodySerializer trait injected as an implicit-with-default on requestToElasticSearchRequest AND sqlQueryToAggregations (both doors), carried on ElasticSearchRequest (defaulted last field). SearchBodySerializer.Default (= today's 1-arg builder) REFUSES a marker loudly and by name (it can never emit the script; without the guard elastic4s throws NotImplementedError("…Please add a PR!") on it — loud on every line, but misleading).

Surfacing: singleSearchToJsonQuery returns a String, and translation runs synchronously inside searchAsync / scroll before any Future exists — a refusal used to escape GatewayApi.run as a raw exception (the same route the bridge's require(...) failures take). ONE boundary in GatewayApi.run(statement) — catch for a synchronous throw (every in-tree route) and .recover on dispatch for a refusal an out-of-tree extension defers into its own Future, so the contract holds for the SPI's callers too (pinned by a deferring-extension test) — the point every route converges on, extension (CoreDqlExtension's quota-capped scroll calls client.scroll directly and never enters an executor) or executor — turns a thrown status-bearing ElasticError into ElasticFailure (operation = "dql"). The windowed row query's scroll translated its window-aggregation request INSIDE Future(...) and built its source lazily, so the refusal only failed the stream at materialisation: its translation is now hoisted onto the calling thread (additive SearchApi.windowAggregationQuery; the protected one-argument executeWindowAggregations keeps its signature). Contract: a refusal known at translation time is answered at run — on every route this PR owns, i.e. for the extended-stats refusal, executor and extension alike (GatewayRefusalBoundarySpec, 6 routes incl. an extension deferring into its own Future). The claim is universal, and now true for BOTH pre-execution rejection kinds: D-1 is fixed in this PR (see below), so a temporal-literal rejection (#276) takes the same path as the extended-stats refusal. Pinned by GatewayRefusalBoundarySpec (9 cases) over both kinds × the executor, extension-scroll, one-shot, windowed and deferring-extension routes.

T2 — reproduction captured on the base commit BEFORE the fix (both bridge copies, both doors, both binds)

-- door 1 = ElasticSearchRequest.query (what the client sends); door 2 = sqlQueryToAggregations
P1  SELECT id, STDDEV(YEAR(createdAt)) AS s FROM t GROUP BY id
    "aggs":{"s":{"extended_stats":{"field":"createdAt"}}}            -- NO script: stddev of the raw timestamps
P2  SELECT id, STDDEV(ABS(salary)) AS s FROM t GROUP BY id
    "aggs":{"s":{"extended_stats":{}}}                               -- neither field nor script: ES rejects
P3  SELECT id, VARIANCE(ABS(salary)) AS v FROM t GROUP BY id
    "aggs":{"v":{"extended_stats":{}}}
P5  SELECT id, STDDEV(DATE_TRUNC(createdAt, MINUTE)) AS s …          "extended_stats":{"field":"createdAt"}
P6  STDDEV_POP / STDDEV_SAMP / VAR_POP / VAR_SAMP (YEAR(createdAt))  all "extended_stats":{"field":"createdAt"}
P7  SELECT id, MAX(YEAR(createdAt)) AS m … (control)
    "m":{"max":{"field":"createdAt","script":{"lang":"painless","source":"def param1 = (doc['createdAt'].size() == 0 ? null : doc['createdAt'].value.toInstant().atZone(ZoneId.of('Z')).get(ChronoField.YEAR)); param1"}}}
W1  SELECT id, name, STDDEV(YEAR(createdAt)) OVER (PARTITION BY id) AS s FROM t      (:225 window bind)
    "aggs":{"id":{"terms":{…},"aggs":{"s":{"extended_stats":{"field":"createdAt"}}}}}
W2  STDDEV(ABS(salary)) OVER (PARTITION BY id)      "extended_stats":{}
W3  VARIANCE(ABS(salary)) OVER (PARTITION BY id)    "extended_stats":{}
-- door 2 (per-aggregation body) identical for every shape:  e.g. P1 -> {"query":{"match_all":{}},"size":0,"aggs":{"s":{"extended_stats":{"field":"createdAt"}}}}

After, ES 8/9 (door 1; door 2 identical modulo the wrapper):

P1  "aggs":{"s":{"extended_stats":{"field":"createdAt","script":{"lang":"painless","source":"def param1 = (doc['createdAt'].size() == 0 ? null : doc['createdAt'].value.toInstant().atZone(ZoneId.of('Z')).get(ChronoField.YEAR)); param1"}}}
P3  "aggs":{"v":{"extended_stats":{"script":{"lang":"painless","source":"def param1 = (doc['salary'].size() == 0 ? null : doc['salary'].value); (param1 == null) ? null : Double.valueOf(Math.abs(param1))"}}}
W1  "aggs":{"id":{"terms":{…},"aggs":{"s":{"extended_stats":{"field":"createdAt","script":{…YEAR…}}}}}

After, ES 6/7: ElasticFailure(400, "STDDEV/VARIANCE over a transformed expression is not supported on Elasticsearch 7: the underlying elastic4s builder drops the aggregation script (elastic4s#4100), so the statistic would silently be computed over the raw field. Aggregate over a raw field, or use Elasticsearch 8+.") — before any JSON exists.

Null-safety (lead directive 2026-09-06)

The scripts reaching ES 8/9 through extended_stats for the first time are the metric scripts the other aggregates already emit (def param1 = (doc[..].size() == 0 ? null : <chain>); param1, (param1 == null) ? null : Double.valueOf(Math.abs(param1))). AggregationNamingSpec's structural guard (both bridge copies) now walks 9 transform-bearing extended_stats shapes (plain + windowed) off the marker tree; the es8/es9 spec re-checks the rendered JSON.

Fifth door found, exempt with a comment

ElasticMultiSearchRequest.query (MultiSearchBuilderFn, default handler) is referenced nowhere in the repository — core builds _msearch bodies from each request's own singleSearchToJsonQuery (ElasticQueries.multiQuery), and requestToMultiSearchRequest builds elastic4s's MultiSearchRequest, not this case class. Exempt with a comment; a marker inside it would still fail loudly on every line (elastic4s's NotImplementedError). queryToJson exempt (query-only) as specified.

Downstream fixture sweep (AC 8, read-only)

extended_stats / STDDEV / VARIANCE / VAR_POP / VAR_SAMP / std_deviation / ElasticSearchRequest / SearchBodyBuilderFn / singleSearchToJsonQuery / elastic.sql.bridge across softclient4es-jdbc, softclient4es-arrow, softclient4es-extensions: 0 hits — nothing downstream pins the moved emission or consumes the bridge directly. Binary-incompatible arity on ElasticSearchRequest (new defaulted last field) — downstream rebuilds on 0.23.0 anyway (same class as BIDC-2 / 21.1).

Upstream watch (AC 9, AD-S3-3)

Local record docs/issues/local-BIDC-3-elastic4s-4100-upstream-watch.md (never staged). All three upstream PRs are MERGED: #4100 (series/9.x), #4105 (series/7.x), #4106 (series/8.x). Trigger 2 is DONE and consumed by this PR — #4105 shipped in elastic4s 7.17.26, which is why ES 7 now computes. Trigger 1 remains open: the latest 8.x (8.19.1) and 9.x (9.3.0) releases predate #4106 / #4100, so the es8 / es9 marker + handler stay until a release carrying them exists. es6 never.

Release-note items (0.23.0)

1. ES 6: STDDEV, STDDEV_SAMP, STDDEV_POP, VARIANCE, VAR_SAMP, VAR_POP over a transformed expression (STDDEV(YEAR(x)), VARIANCE(ABS(x)), plain or OVER (PARTITION BY …)) now REFUSE with a 400 naming the release. They used to return the statistic of the raw field (or fail at Elasticsearch with an empty extended_stats). Aggregate over a raw field there, or use ES 7+. (The Single-table MATERIALIZED VIEW rejected with internal error 'Number of stages must be positive' #185 precedent: honest rejection replacing silent acceptance.) The refusal is permanent — the 6.x client library line is unmaintained.
2. ES 7, ES 8 and ES 9: the same statements now return the statistic over the transform. Anyone who had a number from them before had the wrong number.
3. The ES 7 client library moved from com.sksamuel.elastic4s 7.17.4 to nl.gn0s1s 7.17.26 (a groupId change, 22 patch versions on) — anyone pinning elastic4s themselves alongside this driver must follow, and the es7 closure now excludes elastic4s's own Jackson in favour of the curated set.
4. ElasticSearchRequest gains a defaulted last field (serializer), and requestToElasticSearchRequest / sqlQueryToAggregations each gain a defaulted implicit parameter — all three source-compatible, all three binary-incompatible (arity). Downstream rebuilds on 0.23.0.
5. The windowed-scroll FALLBACK arm (window aggregation failed, failOnWindowError = false) now builds its base ElasticQuery with sql = Some(baseQuery.sql) like the success arm — one shared, eagerly-built value; the core SQL log line for that arm gains the statement text.
6. A client-layer refusal raised during SQL translation is now answered by GatewayApi.run as an ElasticFailure (400) instead of escaping as an exception — REPL, JDBC and Arrow see the message.
7. A temporal-literal rejection (Space-separated timestamp literals are forwarded verbatim to ES and rejected — only T-separated ISO-8601 works (breaks Superset date filters) #276) on the un-LIMITed row route now surfaces as an ElasticFailure(400) from run, instead of an ElasticSuccess carrying a stream that fails at materialisation (D-1, folded in by lead ruling). Direct SearchApi.scroll callers now receive the thrown ElasticError rather than a failed Source — the same shape the extended-stats refusal always had. ⚠️ This changes behaviour shipped in fix(sql,core): resolve temporal literals against the mapped date format before rendering the query (#276) #289: the rejection was already loud, but only once somebody ran the stream, and run reported success until then.

Docs

documentation/sql/functions_aggregate.md (STDDEV / VARIANCE family — "Transformed operands are per-major" + table), documentation/sql/dql_statements.md (one paragraph), documentation/sql/known_limitations.md (new section). MDX twins to sync by the web maintainer: softclient4es-web/src/content/docs/sql/functions-aggregate.mdx (or wherever the STDDEV family lives), sql/dql.mdx, sql/known-limitations.mdx — not edited here.

Test evidence

Evidence (SUPERSEDED — the tree as it stood before the es7 extension and the D-1 fold-in; the authoritative numbers are in "D-1 fold-in — final evidence (head fc56fb7e)" below):

Suite	Result
+ softclient4es-sql-bridge/test (template, ES 8 elastic4s)	174/174 on 2.12.20 AND 2.13.16 (157 after BIDC-2's follow-up + 17 new: ExtendedStatsEmissionSpec 16 + the marker-tree null-safety guard in AggregationNamingSpec)
+ es6bridge/test (hand-maintained)	174/174 on both legs — same specs, same pins
softclient4es7-sql-bridge/test, softclient4es8-sql-bridge/test (generated)	174/174 each on the SECOND invocation (first = 158, the documented stale-copy race)
softclient4es9-sql-bridge/test (generated, sbt17)	174/174
+ core/test	874/874 on both legs (868 + GatewayRefusalBoundarySpec 6: SearchExecutor aggregation shape, SearchExecutor LIMIT row, CoreDqlExtension quota-capped scroll, windowed un-LIMITed scroll — all answered at run —, non-refusing control, plus an extension deferring its refusal into its own Future)
es8java / es9java JavaClientExtendedStatsEmissionSpec (sbt17, Docker-free)	9/9 each — exact JSON pins (plain, ABS pure-script, windowed), all six family members × both binds, door 2, verbatim marker script, byte-identical-to-Default over 7 non-marker shapes, elastic4s's own NotImplementedError preserved for an aggregation nobody can build
es7rest / es6rest / es6jest *ExtendedStatsRejectionSpec (sbt17, Docker-free)	5/5 each — every family member × both binds refused with the named 400 before emission, door 2 refused, raw field untouched, MAX(YEAR(x)) keeps its script
cross-compile	+ softclient4es-sql-bridge/compile + es6bridge/compile + core/compile + ++ 2.12.20 <bridge, es6bridge, core, es8java, es7rest, es6rest, es6jest>/Test/compile + ++ 2.12.20 softclient4es-core-testkit/compile — 11/11 green
gates	scalafmtCheckAll green; Compile headerCheck green on all 9 modules; es7rest/Test/headerCheck red on 12 PRE-EXISTING headerless specs only (none new; every new client-module spec carries the header)

Docker (sbt17, one major per invocation, WindowFunctionSpec = 38 existing + 2 new #222 cases)
— snapshot at 398dd5e2, BEFORE the ES 7 scope extension; the ES 7 row is superseded by the
post-extension re-run recorded further down:

Major	Client	Result	#222 branch taken
8.18.3	es8java	40/40	VALUES — per department, plain: Engineering sd 2.1602 / vs 4.6667 / vp 4.0000 / sdp 2.0000 (years 2019,2018,2020,2017,2021,2016,2015 → mean 2018, SS 28); Sales 1.4142 / 2.0 / 1.6667 / 1.2910; Marketing 1.2910 / 1.6667 / 1.25 / 1.1180; HR 1.5275 / 2.3333 / 1.5556 / 1.2472 — every one an exact fixture oracle; windowed OVER (PARTITION BY department): the same sd per department on every row. The raw-field statistic (STDDEV(salary) 19334.15, and hire_date in millis) sits beside them, unchanged
9.0.3	es9java	40/40	VALUES — identical to ES 8, all four departments, plain + windowed
7.17.29	es7rest	40/40	REFUSAL (SUPERSEDED — ES 7 now COMPUTES, see below) — client.run → ElasticFailure(400, "…not supported on Elasticsearch 7 … (elastic4s#4100)…") for the plain AND the windowed statement; the direct API throws the same ElasticError
6.8.23	es6rest	39/40 + 1 canceled	REFUSAL naming Elasticsearch 6 (both cases); the cancel = the pre-existing 14.4 assume (sample keys need ES 7.7+)
6.8.23	es6jest	39/40 + 1 canceled	REFUSAL naming Elasticsearch 6 (both cases); same pre-existing cancel

Re-runs at that intermediate base (superseded, kept for the audit trail): bridge template 174/174 ×2 legs, es6 bridge 174/174 ×2 legs, core 873/873 ×2 legs (+ …/test); generated es7 + es8 174/174 (both invocations); generated es9 (sbt17) 172/174 on the FIRST invocation — the two IN over an aggregate pins BIDC-2's b09aaf27 rewrote, compared against the stale previous test copy (the documented race) — then 174/174 on the second; es8 JavaClientExtendedStatsEmissionSpec 8/8; ES 8.18.3 WindowFunctionSpec 40/40.

Falsification: es8 handler's script line removed ⇒ JavaClientExtendedStatsEmissionSpec 6/8 RED (the two survivors are the parity + injection checks); restored, green.

Review: bmad-code-review's three layers run inline by the implementer (no reviewer sub-agent may be spawned per the run brief) — 4 findings applied during the run (boundary moved from SearchExecutor to GatewayApi.run; lazy windowed translation hoisted; a test-scope implicit silently falling back to the Default serializer; an over-strict null-safety proxy replaced by verbatim marker-script equality); no open HIGH/MEDIUM. No independent fresh-context reviewer ran on this diff.

Post-review re-runs (on 10f3f8fc, after the R3 follow-up commit): scalafmtCheckAll green; bridge template 176/176 ×2 legs; es6 bridge 176/176 ×2 legs; core 874/874 ×2 legs; generated es7 + es8 176/176 (both invocations each); generated es9 (sbt17) 176/176 (both); es8 + es9 JavaClientExtendedStatsEmissionSpec 9/9 each (the R3-1 diagnostic test included); GatewayRefusalBoundarySpec 6/6 (the deferring-extension route included); ES 8.18.3 WindowFunctionSpec 40/40 on real Elasticsearch.

Post-extension re-run (ES 7 now COMPUTES, tree 4a6d3bed)

ES 7.17.29 es7rest RestHighLevelClientWindowFunctionSpec 40/40 — VALUES, the same fixture
oracle ES 8 / ES 9 assert (Engineering sd 2.1602 / vs 4.6667 / vp 4.0000 / sdp 2.0000, plain AND
windowed) — the row the snapshot table above records as REFUSAL. Whole es7 tree cross-compiled on
both Scala legs after the nl.gn0s1s migration; RestHighLevelClientExtendedStatsEmissionSpec 10/10
replaces the deleted rejection spec.

D-1 fold-in — final evidence (merged tree, head fc56fb7e)

origin/main merged in at d1269d29 (clean; five files overlap BIDC-4 — ScrollApi, SearchApi,
es8/es9 JavaClientApi, dql_statements.md), D-1 fixed at fc56fb7e.

Suite	Result
scalafmtCheckAll	green
sql/test	687/687
+ softclient4es-sql-bridge/test (template)	182/182
+ es6bridge/test (hand-maintained)	182/182
+ core/test	896/896 (GatewayRefusalBoundarySpec now 9: 4 extended-stats routes + deferring extension + non-refusing control + 3 new temporal-literal cases)
++ 2.12.20 <bridge, es6bridge, core, es8java, es7rest>/Test/compile	5/5 green

Falsification (D-1): Source.failed(error) restored at ScrollApi ⇒ exactly the three new
temporal-literal cases go red, nothing else; restored, green. TemporalLiteralSearchSpec's scroll pin
was RETARGETED (not deleted): it used to assert the failure arrives on the stream, it now asserts
run answers the 400 before any stream exists (client.lastQuery shouldBe None).

Symmetry audit (as instructed): every SearchApi call site of resolveTemporalLiterals already
returned ElasticResult.failure; ScrollApi was the only one that could not, because it must return
a Source. No other Source.failed(<status-bearing ElasticError>) remains on a pre-execution path —
ScrollApi's other three carry IllegalArgumentException / UnsupportedOperationException /
RuntimeException, and file/package.scala's are IO.

Docker, D-1 tree, both specs per invocation (TemporalLiteralSpec 11 + WindowFunctionSpec 40 = 51):

Major	Client	Result
8.18.3	es8java	51/51
7.17.29	es7rest	51/51
9.0.3	es9java	51/51
6.8.23	es6rest	50/51 + 1 canceled (the pre-existing 14.4 assume: sample keys need ES 7.7+)
6.8.23	es6jest	50/51 + 1 canceled (same pre-existing cancel)

All five majors run, one per sbt17 invocation. origin/main (99d1f438) is an ancestor of the
branch head, so the PR merges into main as a fast-forward.

🤖 Generated with Claude Code