fix(bridge,client,core): STDDEV/VARIANCE over a transformed expression — per-major emission (#222)

[fupelaqu]

Closes #222

Story BIDC-3 (BI defect-closure epic). PR base = feature/BIDC-2 (STACKED, lead decision 2026-09-06) — BIDC-2's PR is held, not merged, and this story builds on its corrected buckets_path / naming (ElasticAggregation.scala neighbourhood). Cut from origin/feature/BIDC-2 4368d9dc, rebased onto BIDC-2's head 10f3f8fc (its review follow-ups a5f376f2 · b09aaf27 · b5a86cf9 · a363f579 · 10f3f8fc — every rebase clean; the last two touch BOTH AggregationNamingSpec copies, so the merged tree was re-compiled and re-run, not just merged). git log origin/main..feature/BIDC-3 therefore lists BIDC-2's two commits as well — expected; GitHub retargets to main when BIDC-2 merges. Version line 0.23.0-SNAPSHOT verified, untouched. No publishLocal, no GitHub issue operation. Upstream reference: Philippus/elastic4s#4100 (no closing keyword — a WATCH item, see below).

Per-major behaviour (lead ruling R-1b)

Major	Before	After this PR	Mechanism
ES 8 / ES 9	statistic of the RAW field ("extended_stats":{"field":"createdAt"}), or extended_stats: {} rejected by ES	computed over the transform — the emitted extended_stats carries its script	client-module serializer on elastic4s's 2-arg SearchBodyBuilderFn with a handler for the bridge's marker
ES 7	same silent wrong statistic	refused, 400, "STDDEV/VARIANCE over a transformed expression is not supported on Elasticsearch 7: … (elastic4s#4100) …"	refusing serializer in es7/rest; lifted when the 7.17 backport ships and elastic74s bumps
ES 6 (rest + jest)	same	refused, 400, naming Elasticsearch 6 — permanent	refusing serializers in es6/rest and es6/jest

Raw-field STDDEV(salary) is byte-identical on every major (pinned). Every other scripted metric (MAX(YEAR(x)), …) is untouched (pinned: the es8/es9 serializer is byte-identical to the default for any request without a marker, over 7 shapes incl. HAVING/bucket_script/nested).

Why the handler cannot key on ExtendedStatsAggregation — AD-S3-1′ (LEAD-RATIFIED 2026-09-06)

The spec's AD-S3-1 named a handler keyed on ExtendedStatsAggregation: that mechanism is refuted by measurement, not merely superseded — it is unreachable dead code, independently re-derived from the elastic4s 8.18.2 / 9.0.0 sources by the fresh-context reviewer and ratified by the lead. The seam's SHAPE (client-module-injected serializer; template contributes only a discriminator; es6/es7 reject in the client modules) is unchanged; only the handler's key moves, to a marker type.

elastic4s's AggregationBuilderFn (8.18.2, byte-identical in 9.0.0) matches the typed arm case agg: ExtendedStatsAggregation => ExtendedStatsAggregationBuilder(agg, handler) BEFORE case other => customAggregationHandler(other) — the customAggregation seam is consulted only for a type the library does not know. So the bridge binds a transform-bearing extended_stats (both binds: plain :199 and windowed :225, es6 :200/:226) to a template-owned marker ScriptedExtendedStatsAggregation(inner: ExtendedStatsAggregation) — an elastic4s Aggregation whose abstract members are byte-identical on 6.7.8 / 7.17.4 / 8.18.2 / 9.0.0 (javap) — and the es8/es9 handler keys on the marker. The template stays major-agnostic; the seam is a SearchBodySerializer trait injected as an implicit-with-default on requestToElasticSearchRequest AND sqlQueryToAggregations (both doors), carried on ElasticSearchRequest (defaulted last field). SearchBodySerializer.Default (= today's 1-arg builder) REFUSES a marker loudly and by name (it can never emit the script; without the guard elastic4s throws NotImplementedError("…Please add a PR!") on it — loud on every line, but misleading).

Surfacing: singleSearchToJsonQuery returns a String, and translation runs synchronously inside searchAsync / scroll before any Future exists — a refusal used to escape GatewayApi.run as a raw exception (the same route the bridge's require(...) failures take). ONE boundary in GatewayApi.run(statement) — catch for a synchronous throw (every in-tree route) and .recover on dispatch for a refusal an out-of-tree extension defers into its own Future, so the contract holds for the SPI's callers too (pinned by a deferring-extension test) — the point every route converges on, extension (CoreDqlExtension's quota-capped scroll calls client.scroll directly and never enters an executor) or executor — turns a thrown status-bearing ElasticError into ElasticFailure (operation = "dql"). The windowed row query's scroll translated its window-aggregation request INSIDE Future(...) and built its source lazily, so the refusal only failed the stream at materialisation: its translation is now hoisted onto the calling thread (additive SearchApi.windowAggregationQuery; the protected one-argument executeWindowAggregations keeps its signature). Contract: a refusal known at translation time is answered at run on every route (GatewayRefusalBoundarySpec, 5 routes).

T2 — reproduction captured on the base commit BEFORE the fix (both bridge copies, both doors, both binds)

-- door 1 = ElasticSearchRequest.query (what the client sends); door 2 = sqlQueryToAggregations
P1  SELECT id, STDDEV(YEAR(createdAt)) AS s FROM t GROUP BY id
    "aggs":{"s":{"extended_stats":{"field":"createdAt"}}}            -- NO script: stddev of the raw timestamps
P2  SELECT id, STDDEV(ABS(salary)) AS s FROM t GROUP BY id
    "aggs":{"s":{"extended_stats":{}}}                               -- neither field nor script: ES rejects
P3  SELECT id, VARIANCE(ABS(salary)) AS v FROM t GROUP BY id
    "aggs":{"v":{"extended_stats":{}}}
P5  SELECT id, STDDEV(DATE_TRUNC(createdAt, MINUTE)) AS s …          "extended_stats":{"field":"createdAt"}
P6  STDDEV_POP / STDDEV_SAMP / VAR_POP / VAR_SAMP (YEAR(createdAt))  all "extended_stats":{"field":"createdAt"}
P7  SELECT id, MAX(YEAR(createdAt)) AS m … (control)
    "m":{"max":{"field":"createdAt","script":{"lang":"painless","source":"def param1 = (doc['createdAt'].size() == 0 ? null : doc['createdAt'].value.toInstant().atZone(ZoneId.of('Z')).get(ChronoField.YEAR)); param1"}}}
W1  SELECT id, name, STDDEV(YEAR(createdAt)) OVER (PARTITION BY id) AS s FROM t      (:225 window bind)
    "aggs":{"id":{"terms":{…},"aggs":{"s":{"extended_stats":{"field":"createdAt"}}}}}
W2  STDDEV(ABS(salary)) OVER (PARTITION BY id)      "extended_stats":{}
W3  VARIANCE(ABS(salary)) OVER (PARTITION BY id)    "extended_stats":{}
-- door 2 (per-aggregation body) identical for every shape:  e.g. P1 -> {"query":{"match_all":{}},"size":0,"aggs":{"s":{"extended_stats":{"field":"createdAt"}}}}

After, ES 8/9 (door 1; door 2 identical modulo the wrapper):

P1  "aggs":{"s":{"extended_stats":{"field":"createdAt","script":{"lang":"painless","source":"def param1 = (doc['createdAt'].size() == 0 ? null : doc['createdAt'].value.toInstant().atZone(ZoneId.of('Z')).get(ChronoField.YEAR)); param1"}}}
P3  "aggs":{"v":{"extended_stats":{"script":{"lang":"painless","source":"def param1 = (doc['salary'].size() == 0 ? null : doc['salary'].value); (param1 == null) ? null : Double.valueOf(Math.abs(param1))"}}}
W1  "aggs":{"id":{"terms":{…},"aggs":{"s":{"extended_stats":{"field":"createdAt","script":{…YEAR…}}}}}

After, ES 6/7: ElasticFailure(400, "STDDEV/VARIANCE over a transformed expression is not supported on Elasticsearch 7: the underlying elastic4s builder drops the aggregation script (elastic4s#4100), so the statistic would silently be computed over the raw field. Aggregate over a raw field, or use Elasticsearch 8+.") — before any JSON exists.

Null-safety (lead directive 2026-09-06)

The scripts reaching ES 8/9 through extended_stats for the first time are the metric scripts the other aggregates already emit (def param1 = (doc[..].size() == 0 ? null : <chain>); param1, (param1 == null) ? null : Double.valueOf(Math.abs(param1))). AggregationNamingSpec's structural guard (both bridge copies) now walks 9 transform-bearing extended_stats shapes (plain + windowed) off the marker tree; the es8/es9 spec re-checks the rendered JSON.

Fifth door found, exempt with a comment

ElasticMultiSearchRequest.query (MultiSearchBuilderFn, default handler) is referenced nowhere in the repository — core builds _msearch bodies from each request's own singleSearchToJsonQuery (ElasticQueries.multiQuery), and requestToMultiSearchRequest builds elastic4s's MultiSearchRequest, not this case class. Exempt with a comment; a marker inside it would still fail loudly on every line (elastic4s's NotImplementedError). queryToJson exempt (query-only) as specified.

Downstream fixture sweep (AC 8, read-only)

extended_stats / STDDEV / VARIANCE / VAR_POP / VAR_SAMP / std_deviation / ElasticSearchRequest / SearchBodyBuilderFn / singleSearchToJsonQuery / elastic.sql.bridge across softclient4es-jdbc, softclient4es-arrow, softclient4es-extensions: 0 hits — nothing downstream pins the moved emission or consumes the bridge directly. Binary-incompatible arity on ElasticSearchRequest (new defaulted last field) — downstream rebuilds on 0.23.0 anyway (same class as BIDC-2 / 21.1).

Upstream watch (AC 9, AD-S3-3)

Local record docs/issues/local-BIDC-3-elastic4s-4100-upstream-watch.md (never staged). Three upstream PRs, all OPEN — awaiting merge + release (checked 2026-09-06): #4100 (series/9.x) and its cherry-picks #4106 (series/8.x) and #4105 (series/7.x), each carrying the golden test red-without/green-with. Triggers: #4106 / #4100 released ⇒ bump elastic84s / elastic94s, retire the es8/es9 handler; #4105 released ⇒ bump elastic74s (note the coordinate change to nl.gn0s1s from 7.17.5) and lift the es7 refusal; es6 never.

Release-note items (0.23.0)

1. ES 6 / ES 7: STDDEV, STDDEV_SAMP, STDDEV_POP, VARIANCE, VAR_SAMP, VAR_POP over a transformed expression (STDDEV(YEAR(x)), VARIANCE(ABS(x)), plain or OVER (PARTITION BY …)) now REFUSE with a 400 naming the release. They used to return the statistic of the raw field (or fail at Elasticsearch with an empty extended_stats). Aggregate over a raw field there, or use ES 8+. (The Single-table MATERIALIZED VIEW rejected with internal error 'Number of stages must be positive' #185 precedent: honest rejection replacing silent acceptance.)
2. ES 8 / ES 9: the same statements now return the statistic over the transform. Anyone who had a number from them before had the wrong number.
3. ElasticSearchRequest gains a defaulted last field (serializer), and requestToElasticSearchRequest / sqlQueryToAggregations each gain a defaulted implicit parameter — all three source-compatible, all three binary-incompatible (arity). Downstream rebuilds on 0.23.0.
4. The windowed-scroll FALLBACK arm (window aggregation failed, failOnWindowError = false) now builds its base ElasticQuery with sql = Some(baseQuery.sql) like the success arm — one shared, eagerly-built value; the core SQL log line for that arm gains the statement text.
5. A client-layer refusal raised during SQL translation is now answered by GatewayApi.run as an ElasticFailure (400) instead of escaping as an exception — REPL, JDBC and Arrow see the message.

Docs

documentation/sql/functions_aggregate.md (STDDEV / VARIANCE family — "Transformed operands are per-major" + table), documentation/sql/dql_statements.md (one paragraph), documentation/sql/known_limitations.md (new section). MDX twins to sync by the web maintainer: softclient4es-web/src/content/docs/sql/functions-aggregate.mdx (or wherever the STDDEV family lives), sql/dql.mdx, sql/known-limitations.mdx — not edited here.

Test evidence

Evidence (final tree, rebased onto BIDC-2's final head b5a86cf9):

Suite	Result
+ softclient4es-sql-bridge/test (template, ES 8 elastic4s)	174/174 on 2.12.20 AND 2.13.16 (157 after BIDC-2's follow-up + 17 new: ExtendedStatsEmissionSpec 16 + the marker-tree null-safety guard in AggregationNamingSpec)
+ es6bridge/test (hand-maintained)	174/174 on both legs — same specs, same pins
softclient4es7-sql-bridge/test, softclient4es8-sql-bridge/test (generated)	174/174 each on the SECOND invocation (first = 158, the documented stale-copy race)
softclient4es9-sql-bridge/test (generated, sbt17)	174/174
+ core/test	874/874 on both legs (868 + GatewayRefusalBoundarySpec 6: SearchExecutor aggregation shape, SearchExecutor LIMIT row, CoreDqlExtension quota-capped scroll, windowed un-LIMITed scroll — all answered at run —, non-refusing control, plus an extension deferring its refusal into its own Future)
es8java / es9java JavaClientExtendedStatsEmissionSpec (sbt17, Docker-free)	9/9 each — exact JSON pins (plain, ABS pure-script, windowed), all six family members × both binds, door 2, verbatim marker script, byte-identical-to-Default over 7 non-marker shapes, elastic4s's own NotImplementedError preserved for an aggregation nobody can build
es7rest / es6rest / es6jest *ExtendedStatsRejectionSpec (sbt17, Docker-free)	5/5 each — every family member × both binds refused with the named 400 before emission, door 2 refused, raw field untouched, MAX(YEAR(x)) keeps its script
cross-compile	+ softclient4es-sql-bridge/compile + es6bridge/compile + core/compile + ++ 2.12.20 <bridge, es6bridge, core, es8java, es7rest, es6rest, es6jest>/Test/compile + ++ 2.12.20 softclient4es-core-testkit/compile — 11/11 green
gates	scalafmtCheckAll green; Compile headerCheck green on all 9 modules; es7rest/Test/headerCheck red on 12 PRE-EXISTING headerless specs only (none new; every new client-module spec carries the header)

Docker (sbt17, one major per invocation, WindowFunctionSpec = 38 existing + 2 new #222 cases):

Major	Client	Result	#222 branch taken
8.18.3	es8java	40/40	VALUES — per department, plain: Engineering sd 2.1602 / vs 4.6667 / vp 4.0000 / sdp 2.0000 (years 2019,2018,2020,2017,2021,2016,2015 → mean 2018, SS 28); Sales 1.4142 / 2.0 / 1.6667 / 1.2910; Marketing 1.2910 / 1.6667 / 1.25 / 1.1180; HR 1.5275 / 2.3333 / 1.5556 / 1.2472 — every one an exact fixture oracle; windowed OVER (PARTITION BY department): the same sd per department on every row. The raw-field statistic (STDDEV(salary) 19334.15, and hire_date in millis) sits beside them, unchanged
9.0.3	es9java	40/40	VALUES — identical to ES 8, all four departments, plain + windowed
7.17.29	es7rest	40/40	REFUSAL — client.run → ElasticFailure(400, "…not supported on Elasticsearch 7 … (elastic4s#4100)…") for the plain AND the windowed statement; the direct API throws the same ElasticError
6.8.23	es6rest	39/40 + 1 canceled	REFUSAL naming Elasticsearch 6 (both cases); the cancel = the pre-existing 14.4 assume (sample keys need ES 7.7+)
6.8.23	es6jest	39/40 + 1 canceled	REFUSAL naming Elasticsearch 6 (both cases); same pre-existing cancel

Final-base re-runs (after the last rebase onto b5a86cf9): bridge template 174/174 ×2 legs, es6 bridge 174/174 ×2 legs, core 873/873 ×2 legs (+ …/test); generated es7 + es8 174/174 (both invocations); generated es9 (sbt17) 172/174 on the FIRST invocation — the two IN over an aggregate pins BIDC-2's b09aaf27 rewrote, compared against the stale previous test copy (the documented race) — then 174/174 on the second; es8 JavaClientExtendedStatsEmissionSpec 8/8; ES 8.18.3 WindowFunctionSpec 40/40.

Falsification: es8 handler's script line removed ⇒ JavaClientExtendedStatsEmissionSpec 6/8 RED (the two survivors are the parity + injection checks); restored, green.

Review: bmad-code-review's three layers run inline by the implementer (no reviewer sub-agent may be spawned per the run brief) — 4 findings applied during the run (boundary moved from SearchExecutor to GatewayApi.run; lazy windowed translation hoisted; a test-scope implicit silently falling back to the Default serializer; an over-strict null-safety proxy replaced by verbatim marker-script equality); no open HIGH/MEDIUM. No independent fresh-context reviewer ran on this diff.

Post-review re-runs (on 10f3f8fc, after the R3 follow-up commit): scalafmtCheckAll green; bridge template 176/176 ×2 legs; es6 bridge 176/176 ×2 legs; core 874/874 ×2 legs; generated es7 + es8 176/176 (both invocations each); generated es9 (sbt17) 176/176 (both); es8 + es9 JavaClientExtendedStatsEmissionSpec 9/9 each (the R3-1 diagnostic test included); GatewayRefusalBoundarySpec 6/6 (the deferring-extension route included); ES 8.18.3 WindowFunctionSpec 40/40 on real Elasticsearch.

🤖 Generated with Claude Code