Skip to content

fix(sql,bridge): name un-aliased aggregates after the whole expression; bucket pipelines read params.<metric> null-guarded (#54, #223) - #288

Merged
fupelaqu merged 6 commits into
mainfrom
feature/BIDC-2
Sep 6, 2026
Merged

fix(sql,bridge): name un-aliased aggregates after the whole expression; bucket pipelines read params.<metric> null-guarded (#54, #223)#288
fupelaqu merged 6 commits into
mainfrom
feature/BIDC-2

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Closes #54
Closes #223

Story BIDC-2 (BI defect-closure epic). Branch feature/BIDC-2 off origin/main e55bd937. Version line 0.23.0-SNAPSHOT verified, untouched. No publishLocal, no GitHub issue operation.

What

One naming rule, one rendering rule, one resolver.

  1. Identifier.metricName (sql) — an un-aliased aggregate is now named after the WHOLE expression (AliasUtils.normalize over the DISTINCT-aware chain): count_x, max_x, count_distinct_x, max_year_createdat, max_abs_salary, count_emails_address. COUNT(*) keeps count_all / count_distinct_all; an alias always wins. The auxiliary aggregation created for a HAVING / ORDER BY-only aggregate is named with it, so buckets_path resolves by construction.
  2. Bucket-pipeline rendering (sql)Identifier.painless(None) on an aggregate is params.<metricName> (no transform — it lives once, in the metric aggregation's own script); Expression.painless(None) on an aggregate predicate is ONE parenthesised, null-guarded expression: (params.k == null ? false : (params.k > 2020)), both operands guarded when the right-hand side is an aggregate. The temporal-literal → epoch-millis conversion moved into it (it used to be appended to the whole predicate by MetricSelectorScript).
  3. resolveBucketMetric (bridge template + hand-maintained es6/bridge)metricSelectorForBucket and extractMetricsPathForBucket share one resolver; buckets_path values are the aggregation's carried local name (agg.name), never a string-split of the .-joined aggName. No case-class arity change.
  4. Arithmetic over aggregates (MAX(x) - MIN(x) AS d) — documented for years, never worked (no operand aggregations, self-referencing buckets_path {"d":"d"}, params.x - params.x). auxiliaryAggs now creates the operands (the Aggregations referenced only in HAVING or WHERE are not created #53 mechanism), BucketScriptAggregation.update registers only aggregate operands.
  5. auxiliaryAggs dedup is order-preserving (was groupBy → hash order → unpinnable JSON).
  6. HAVING cnt > 1 where cnt aliases a SELECT aggregate now filters (Having.resolveAggregateAliases, sql). Found by the live run: the bare alias carries no aggregate function, so the selector rendered 1 == 1 and the emitted query had no having_filter — every group came back. This is the "workaround" issue HAVING COUNT(field) fails when COUNT(field) has no alias #54 itself prescribes; it never worked. Scoped to HAVING (ORDER BY already resolves aliases by name; WHERE keeps reading a bare name as a field).
  7. core: sqlAggregationToClientAggregation gains the BucketScriptAggregation arm (AggregationType.BucketScript). Found by the live run: the bridge emitted a correct bucket_script, and core rejected it before the request left the JVM (Unsupported aggregation type: bucket_script).

Why — what T2 captured at e55bd937 (bridge template, ES 8 elastic4s)

#54 as filed does not reproduce — every un-aliased COUNT(x) already emitted a consistent agg / buckets_path / params name (__c2 in SELECT, x HAVING-only) because #53 aliases the auxiliary Field with metricName. But that name was the bare field, which is the real defect behind the title:

-- K1  SELECT id FROM t GROUP BY id HAVING COUNT(x) > 5 AND MAX(x) > 3        (SILENT WRONG ANSWER)
"aggs":{"x":{"value_count":{"field":"x"}},
 "having_filter":{"bucket_selector":{"buckets_path":{"x":"x"},"script":{"source":"params.x > 5 && params.x > 3"}}}}
-- MAX(x) was dropped (auxiliaryAggs dedups by name); both terms compared the count.

-- K2  ... HAVING MAX(x) > 3 ORDER BY MIN(x) DESC   -> terms carried NO "order" key at all.

-- N4  nested level, HAVING-only COUNT(e.address)   -> agg "emails.address", NO having_filter emitted
--     (the selector's params\.(\w+) scanner saw `emails`, found no aggregation, dropped the condition).
-- T2  HAVING MAX(profile.age) > 30                 -> "script":"params.profile.age > 30" (nested-property read).

#223 reproduces exactly as filed:

-- S3  SELECT id, COUNT(x) AS c FROM t GROUP BY id HAVING MAX(YEAR(createdAt)) > 2020
"createdAt":{"max":{"field":"createdAt","script":{"source":"def param1 = (doc['createdAt'].size() == 0 ? null : doc['createdAt'].value.toInstant().atZone(ZoneId.of('Z')).get(ChronoField.YEAR)); param1"}}},
"having_filter":{"bucket_selector":{"buckets_path":{"createdAt":"createdAt"},
  "script":{"source":"def left = (doc['createdAt'].size() == 0 ? null : doc['createdAt'].value.toInstant().atZone(ZoneId.of('Z')).get(ChronoField.YEAR)).get(ChronoField.YEAR); left == null ? false : (left > 2020)"}}}
--   doc[] in a bucket_selector, the transform twice, `.get(...)` invoked on a possibly-null group.

-- S3d ... HAVING MAX(YEAR(createdAt)) > 2020 OR COUNT(x) > 5
"source":"def left = ...; left == null ? false : (left > 2020) || params.c > 5"
--   `? :` binds looser than `||`: a null left swallowed the OR's right-hand side.

-- S4  SELECT id, COUNT(x) AS c FROM t GROUP BY id ORDER BY MAX(ABS(salary)) DESC
"terms":{...,"order":{"":"desc"}},"aggs":{"c":{...},"":{"max":{"script":{...Math.abs...}}}}

-- B1  SELECT id, MAX(x) - MIN(x) AS d FROM t GROUP BY id
"aggs":{"d":{"bucket_script":{"buckets_path":{"d":"d"},"script":"params.x - params.x"}}}

After (same shapes): K1 → count_x + max_x, (params.count_x == null ? false : (params.count_x > 5)) && (params.max_x == null ? false : (params.max_x > 3)); K2 → "order":{"min_x":"desc"}; N4 → count_e_address + its having_filter; S3 → metric max_year_createdat (script unchanged, transform once) + (params.max_year_createdat == null ? false : (params.max_year_createdat > 2020)); S3d → two guarded terms joined by ||; S4 → "order":{"max_abs_salary":"desc"}; B1 → max_x, min_x + {"max_x":"max_x","min_x":"min_x"}, params.max_x - params.min_x. Nested direct-child path unchanged: "nb":"e>nb" (N1, now pinned — it was not).

Deviation from the spec's AC-1 wording

AC-1 expected {"x": "count_x"} (key x, agg count_x). The key and the aggregation name are ONE name by construction (the auxiliary Field's alias); splitting them would mean un-aliasing auxiliary Fields — a larger change with no observable gain. Shipped: {"count_x": "count_x"} / params.count_x. The invariant AC-1 states — the path names the aggregation Elasticsearch actually emits — holds, and is now enforced structurally (AggregationNamingSpec: every params.<k> a pipeline script reads is a declared buckets_path key).

Lead directive (2026-09-06) — AC 4b, null-safety

  • bucket_selector scripts read only declared params.<k>, never doc[, and null-guard every compared metric (uniformly, COUNT included). Enforced by AggregationNamingSpec "a bucket pipeline script should read only declared, null-guarded params and never a document" over 23 shapes, in BOTH bridge copies.
  • No emitted Painless dereferences a (… ? null : …) group or an unguarded nullable def. Enforced by "no emitted Painless should dereference or compare a possibly-null value unguarded" (structural walk over every script the 23 shapes emit, both copies). The metric scripts already satisfied it (def param1 = (doc[..] ? null : <chain>); param1, (param1 == null) ? null : Double.valueOf(Math.abs(param1))).
  • gap_policy decision: bucket_selector keeps Elasticsearch's default (skip); nothing explicit is emitted. Contract: a bucket whose compared metric is missing never passes a HAVING comparison — proven live in both directions (HAVING MAX(age) > 0 and HAVING MAX(age) < 1000 both drop the Nice bucket whose only user has no age, no request error). Having.script (the extensions MV transform path) shares the producer, so it is guarded identically.

Downstream fixture sweep (AC 7)

buckets_path / bucket_selector / having_filter / ElasticAggregation( / SQLAggregation( across softclient4es-jdbc, softclient4es-arrow, softclient4es-extensions:

  • jdbc: 0 hits.
  • arrow: 2 hits, both prose comments (SchemaProbeSpec.scala:80, ElasticFlightProducer.scala:2097) — no pinned JSON.
  • extensions: community/.../graph/Stage.scala builds the MV transform's own having_filter from Having.script (:291) into a TransformBucketSelectorConfig (:295) with its own extractAggregatePaths (:310, whose empty-name fallback :319 already uses AliasUtils.normalize(identifier.identifierName) — the same spelling adopted here). No pinned JSON. Behaviour change reaching it: Having.script now emits the guarded (params.k == null ? false : (params.k > n)) form and params.<metric> for transform-bearing aggregates (was doc[]). ⚠️ Quieter failure, not fixable here (R2-8): buckets_path there is built from SELECT aggregates only, so a HAVING mixing a SELECT-aliased aggregate with a HAVING-only one ships a script reading an undeclared params.<k> — before, a runtime failure at the transform checkpoint; now the guard yields false and the view is silently empty. Extensions follow-up: buildBucketSelector should assert every params.<k> in having.script is a bucketsPath key (mirror core's structural test) and fail loudly.

Nothing downstream pins the moved emissions; no "size":65536-style sweep is needed for this PR.

Review follow-up (second commit — independent review, approve-with-fixes: 0 HIGH / 3 MEDIUM / 8 LOW)

  • R2-1 HAVING <alias> over a bucket_script (MAX(x) - MIN(x) AS d … HAVING d > 3) was silently ignored, and this PR made the statement executable. RESOLVED rather than rejected: the alias map covers isBucketScript SELECT items, the substituted identifier routes to the bucket rendering (params.d), Identifier.allMetricsPath publishes d -> d, and the bridge resolver finds the sibling bucket_script by name (Elasticsearch lets a bucket_selector read a sibling pipeline aggregation). Inline arithmetic over aggregates in HAVING (HAVING MAX(x) - MIN(x) > 3, no alias) is now a validation error naming the remedy.
  • R2-2 MetricSelectorScript negated the LEFT operand of A AND NOT B (pre-existing). The NOT is now pushed INTO the right-hand expression (> 3<= 3 inside its guard), so a bucket whose metric is missing still fails the negated test as SQL's three-valued NOT would; compound right sides fall back to !( … ).
  • R2-3 BETWEEN / IN over an aggregate bypassed the guarded rendering (BETWEEN even emitted the chained 1 <= p <= 5 Painless rejects). Both now render through an overridable bucketPipelineCheck(p == null ? false : [!](p >= a && p <= b)), (p == null ? false : [!](p == v1 || p == v2)) (an equality chain, NOT [v1,v2].contains(p) — see the second look, S2-1) — and their NOT forms are pinned. Note: COUNT(x) IN (…) is rejected by the pre-existing type validator (BIGINT vs ARRAY<BIGINT>); MAX(x) IN (…) is accepted and guarded.
  • R2-4 the AC 4b guard asserts a bucket_selector is PRESENT for every HAVING shape (no vacuous pass). R2-5 IS [NOT] NULL over an aggregate renders the null test alone (no contradictory outer guard). R2-6 extractMetricNames drops __now__ in both bridge copies — a nested-level HAVING with now - interval … keeps its condition (pinned). R2-9 auxiliaryAggs dedups against SELECT by expression; an alias naming a SELECT aggregate and a DIFFERENT HAVING/ORDER BY aggregate is a validation error. R2-11 docs updated (alias reference works; what is rejected).
  • Lead-confirmed 2026-09-06 (product): an aggregate in WHERE is now REJECTED (Aggregate functions are not allowed in WHERE (found …); use HAVING) instead of silently dropped — the house-style default, ratified by the lead.
  • Not fixable here: R2-8 (extensions, above). Records: R2-7 below.

Second look (third and fourth commits — independent second-look review of a5f376f2)

  • S2-1 (introduced by R2-3, caught by the ES 8 leg BEFORE the report arrived): HAVING MAX(age) IN (40, 50) rendered [40,50].contains(params.max_age), and Painless List.contains compares the boxed Double a buckets_path value arrives as (40.0) with the Integer literals through equals — it NEVER matched (the live run returned the zero-bucket NULL row). Rendered as a disjunction of == (numeric promotion; equals for strings): (p == null ? false : (p == 40 || p == 50)); NOT IN = !(…). Pinned in AggregationNamingSpec and live on all five majors (IN (40, 50) → Lyon, Marseille). Commit b09aaf27.
  • S2-2 (pre-existing, Dangling AND/OR in WHERE/HAVING silently drops the whole clause — DELETE … WHERE id = 1 AND empties the index #280 data-loss family): the WHERE-aggregate reject lived only in SingleSearch.validate(), so DELETE FROM t WHERE COUNT(x) > 5 still parsed as DELETE FROM t (index wiped) and the same UPDATE hit every document. The reject moved into Where.validate(), which fires for SELECT, DELETE and UPDATE (and, since the third look, CREATE ENRICH POLICY's source WHERE); Update/Delete gained a validate() over their WHERE (Parser.apply validates every parsed statement, so the parse path covers DML). Pinned (DELETE, UPDATE, id = 1 AND MAX(x) > 5; plain DML stays Right; every rejection asserts not startWith Parser.InternalParseFailure) and live (DELETE + UPDATE rejected, count and ages unchanged). Commit b5a86cf9.
  • S2-3 / S2-4: shape count corrected to 35; release-note item 5 made truthful (IN is guarded AND correct); negated()'s compound fallback commented as grammar-unreachable; scalafmtCheckAll and the 2.12 test compiles re-run on the final tree (evidence table).

Third look (fifth commit a363f579 — independent third-look review of b5a86cf9: APPROVE, no HIGH/MEDIUM residual)

  • T3-1 CreateEnrichPolicy.validate() never validated its source WHERE — an aggregate there was dropped and the policy enriched from every source document (match_all). It now uses the same where.map(_.validate()).getOrElse(Right(())) idiom as Update/Delete, so the S2-2 reject reaches the enrich path too. Pinned (rejection + the plain WHERE status = 'active' form still accepted, not startWith Parser.InternalParseFailure).
  • T3-2 wording: the S2-2 reject covers SELECT, DELETE and UPDATE (and CREATE ENRICH POLICY's source WHERE) — not "every statement kind".
  • T3-3 COUNT(x) IN (1, 2) was rejected by the shared Expression.validate comparing the element with the list's ARRAY type (BIGINT vs ARRAY<BIGINT>) while the untyped MAX(x) IN (…) passed through Any. InExpr now overrides validate() and compares against the list's element type (contained: one override, no other validator behaviour moved). Pins: COUNT / COUNT NOT IN / MAX / string IN / a typed document-level YEAR(...) IN (…), plus the genuine mismatch COUNT(x) IN ('a','b') still rejected and naming 'BIGINT'/'VARCHAR'.
  • T3-4 (house rule — what the fix newly REACHES): T3-3 makes COUNT(x) IN (…) newly ACCEPTED, so a previously rejected shape now reaches the bucket-pipeline emission, and every existing IN pin was a MAX(x) one (COUNT(x) IN had a parse-level pin only, which cannot see a wrong script). Added: emitted-JSON pins for HAVING COUNT(x) IN (1, 2) and its NOT IN twin in BOTH bridge copies (guarded == chain over params.count_x, buckets_path key count_x -> count_x), both shapes in the structural guards' list (35 → 37), and one live case beside MAX(age) IN (40, 50)HAVING COUNT(name) IN (2, 3) expects exactly Paris, so an empty result (a value_count long vs Integer literals — the S2-1 boxing class) and an all-pass guard both fail loudly. Commit 10f3f8fc.
  • Lead ruling 2026-09-06: the WHERE-aggregate loud reject is CONFIRMED as shipped; every "lead to confirm" marker is flipped to "lead-confirmed 2026-09-06" and release-note item 9 carries the behaviour change.

Release-note items (0.23.0)

  1. HAVING / ORDER BY over a transformed aggregate (MAX(YEAR(x)), MAX(ABS(x)), MAX(DATE_TRUNC(x, …))) now execute correctly; they used to fail at Elasticsearch (doc[] in a bucket_selector) or emit an aggregation named "".
  2. Two different aggregates over the same column in HAVING / ORDER BY (HAVING COUNT(x) > 5 AND MAX(x) > 3) now filter on both; they used to be collapsed onto one — a silent wrong answer.
  3. A HAVING inside a nested level (JOIN UNNEST … GROUP BY e.domain HAVING COUNT(e.address) > 1) is now applied; it used to be silently dropped.
  4. Arithmetic over aggregates (MAX(x) - MIN(x) AS d) now works as documented (bucket_script with real operands); it used to fail at Elasticsearch.
  5. Every generated bucket_selector comparison — <, <=, =, !=, >=, >, BETWEEN, IN, and their NOT forms — is null-guarded: a group whose compared metric is missing never passes a HAVING test, negated or not, and the request does not fail.
  6. Internal aggregation names changed for HAVING / ORDER BY-only aggregates (xcount_x, salarymax_salary); they never appear in result rows. Anyone pinning generated JSON downstream must update having_filter scripts to the guarded form.
  7. HAVING <alias> where the alias names a SELECT aggregate (COUNT(name) AS cnt … HAVING cnt > 1) or an arithmetic expression over aggregates (… AS age_range … HAVING age_range > 3) now filters; both used to return every group.
  8. HAVING A AND NOT B now negates B (it negated A).
  9. Three shapes that were silently dropped are now rejected with an explicit message: an aggregate function in WHERE (use HAVING), arithmetic over aggregates written inline in HAVING (alias it in SELECT), and an alias naming one aggregate in SELECT and a different one in HAVING/ORDER BY. Behaviour change (lead-confirmed 2026-09-06): a statement with an aggregate in WHERE used to parse and silently mis-execute — the predicate was dropped, so a SELECT returned every group and a DELETE/UPDATE matched every document (data loss) — and now fails with a named 400 suggesting HAVING. It covers SELECT, DELETE, UPDATE and CREATE ENRICH POLICY's source WHERE (whose aggregate widened the enrich source query to match_all).
  10. (R2-7) Parser.apply returns the updated AST, so a statement written HAVING cnt > 1 re-renders as HAVING COUNT(name) > 1. It re-parses to the same AST; the visible effects are the REPL's AST→SQL echo and a one-time "definition changed → replace" on the first CREATE OR REPLACE MATERIALIZED VIEW of an existing view written with HAVING <alias> (extensions compares the rendered SQL to the stored definition).
  11. (R2-8, extensions) In the materialized-view transform, a HAVING that mixes a SELECT-aliased aggregate with a HAVING-only one used to fail at the transform checkpoint (undeclared params.<k>); with the guarded script it now yields an empty view silently — an extensions follow-up should assert every params.<k> is declared in buckets_path.
  12. <aggregate> IN (…) type-checks the aggregate against the list's element type: HAVING COUNT(x) IN (1, 2) used to be rejected as BIGINT vs ARRAY<BIGINT> while the untyped MAX(x) IN (…) passed. A genuine mismatch (COUNT(x) IN ('a','b')) stays rejected and now names the element types.
  13. An aggregate in the source WHERE of CREATE ENRICH POLICY is rejected: CreateEnrichPolicy.validate() never validated its WHERE, so the aggregate was dropped and the policy enriched from EVERY source document (match_all).

21.3 interlock — assertions 21.3's AC-10 must re-run after this lands

metricSelectorForBucket was rewritten (shared resolver). 21.3's AC-10 pin (an aggregate-free HAVING degenerates to terms include/exclude and emits no bucket_selector) still holds by construction — MetricSelectorScript.metricSelector returns 1 == 1 for non-aggregate expressions and metricSelectorForBucket strips it to "" (unchanged lines) — but 21.3 must re-run its emitted-JSON assertion on top of this branch, and re-run AggregationNamingSpec (both copies) after its own change. The exact assertions to re-run are listed in .epic-BIDC-dev-findings.md § BIDC-2.

Docs

documentation/sql/dql_statements.md (GROUP BY and HAVING): three bullets — HAVING/ORDER BY-only aggregates need no alias and may wrap a transform; arithmetic over aggregates; the null-metric contract. MDX twin to sync by the web maintainer: softclient4es-web/src/content/docs/sql/dql.mdx (not edited here).

Test evidence

Unit (plain sbt, default JDK):

Suite Result
sql/test first commit 655/655; follow-up 659/659 (2.13; HavingAggregateResolutionSpec added)
core/test 868/868 (2.13) — re-run green after the follow-up
softclient4es-sql-bridge/test (template, ES 8 elastic4s) first commit 148/148; follow-up 157/157AggregationNamingSpec 30 tests (27 JSON pins + 3 structural guards over 37 shapes) and the 10 moved having_filter fixtures
es6bridge/test (hand-maintained) first commit 148/148; follow-up 157/157 — same spec, same fixtures, identical emission
softclient4es7-sql-bridge/test (generated) first commit 148/148 (second invocation — the first compiles the previous template copy, see findings); b5a86cf9 157/157
softclient4es9-sql-bridge/test (generated, sbt17) first commit 148/148 (idem); b5a86cf9 157/157
softclient4es-sql-bridge + es6bridge *AggregationNamingSpec on the final head 10f3f8fc 30/30 each (the two COUNT(x) IN emission pins added by T3-4); sql/testOnly *HavingAggregateResolutionSpec 7/7; softclient4es-core-testkit/compile and scalafmtAll clean
+ sql/compile, + softclient4es-sql-bridge/compile, ++ 2.12.20 {sql,bridge,es6bridge}/Test/compile, softclient4es-core-testkit/compile green — the three ++ 2.12.20 …/Test/compile and scalafmtCheckAll re-run on the final head b5a86cf9, all exit 0 (S2-4)
+ sql/test + core/test + softclient4es-sql-bridge/test + es6bridge/test (both legs, AC-6) 655 / 868 / 148 / 148 on 2.12.20 AND on 2.13.16 — all green, exit 0

Integration (real Elasticsearch via Testcontainers, sbt17, one major per invocation, GatewayApiIntegrationSpec = 50 existing + 11 new BIDC-2 tests with exact row oracles, 61 per major; final head 10f3f8fc):

Major Client First commit 4368d9dc (57 tests) b5a86cf9 (61 tests) Final head 10f3f8fc (61 tests)
8.18.3 es8java JavaClient 57/57 61/61 (also 60/60 on b09aaf27, where the ES 8 leg found S2-1) 61/61
7.17.29 es7rest REST high-level 57/57 61/61 61/61
9.0.3 es9java JavaClient 57/57 61/61 61/61
6.8.23 es6rest REST high-level 56/57 60/61 — 1 canceled = pre-existing supportsEnrichPolicies assume (:1875, enrich needs ES 7.5+) 60/61 — same pre-existing cancel
6.8.23 es6jest Jest 56/57 60/61 — same pre-existing cancel 60/61 — same pre-existing cancel

The final head re-runs all five majors because T3-3 changed what the emission RECEIVES (COUNT(x) IN is newly accepted) — and the S2-1 defect of the same family was found by a live leg, not by a unit test. ES 8 capture of the newly reached shape: "buckets_path":{"count_name":"count_name"}, "script":{"source":"(params.count_name == null ? false : (params.count_name == 2 || params.count_name == 3))"} ⇒ exactly Paris, as the oracle demands.

The 11 new live tests (7 from the first commit, 4 from the follow-ups): aliased / un-aliased / HAVING-only COUNT(name) (same rows); COUNT(age) >= 1 AND MAX(age) > 45 (Marseille only — the collision case); MAX(YEAR(birthdate)) > 1990 and MAX(ABS(age)) > 45 OR COUNT(*) > 1 (exact bucket sets, the no-age bucket excluded); ORDER BY MAX(ABS(age)) DESC (exact order, missing metric last); the AC 4b contract (MAX(age) > 0 AND MAX(age) < 1000 both drop the no-age bucket, no error); MAX(age) - MIN(age) AS age_range alone and beside MAX(age) AS oldest (exact doubles); then HAVING age_range > 3 through the bucket_script alias (Paris only, R2-1); BETWEEN 40 AND 49 / NOT BETWEEN / IN (40, 50) / COUNT(*) >= 1 AND NOT MAX(age) > 45 (exact bucket sets, R2-2/R2-3/S2-1); an aggregate in WHERE rejected with the use HAVING message; DELETE/UPDATE … WHERE MAX(age) > 5 rejected with the snapshot n=5 / oldest=50 / youngest=25 unchanged afterwards (S2-2). The first ES 8 run failed two of the original seven and found the alias-in-HAVING drop and the core bucket_script rejection described above; the ES 8 run of the review follow-up found S2-1 — each fixed, then all five majors re-run green on the final head.

Falsification: with the six sql sources reverted to origin/main in place (bridge resolver kept), AggregationNamingSpec goes red with the original symptoms — agg x, params.x > 5 && params.x > 3, the missing order key, no having_filter for the alias form. Restored; green again.

Review: first commit — bmad-code-review's three layers run inline by the implementer (2 patches applied, 1 defer). Then an independent fresh-context review of 4368d9dc (.review-BIDC-2.md: approve-with-fixes, 0 HIGH / 3 MEDIUM / 8 LOW → commit a5f376f2) and an independent second look of a5f376f2 (S2-1..S2-4 → commits b09aaf27, b5a86cf9); every disposition is in the findings log under BIDC-2 — review follow-up.

🤖 Generated with Claude Code

fupelaqu and others added 6 commits September 6, 2026 03:44
…n; bucket pipelines read params.<metric> null-guarded

An un-aliased aggregate's metricName was the bare field name, so two aggregates over one field in HAVING/ORDER BY collapsed onto one aggregation (HAVING COUNT(x) > 5 AND MAX(x) > 3 compared the count twice, ORDER BY MIN(x) beside HAVING MAX(x) lost its order key), a dotted field leaked into buckets_path keys (a nested-level HAVING silently emitted no having_filter), and an aggregate over a self-contained function (MAX(ABS(salary))) was named "". Identifier.metricName now derives the name from the whole expression (count_x, max_abs_salary, count_distinct_x; COUNT(*) keeps count_all).

The bucket_selector script read doc[...] and re-applied the transform the metric aggregation had already applied (HAVING MAX(YEAR(x)) > 2020). A context-free rendering of an aggregate is now the bucket-pipeline form: params.<metric> only, the transform once in the metric script, and every compared metric null-guarded as one parenthesised expression (lead directive, AC 4b) so it composes under AND/OR; the temporal literal is converted to epoch millis inside the guard. Both bridge copies resolve buckets_path through one resolver keyed on the aggregation's carried local name.

Found by the live run and fixed here too: HAVING <alias> of a SELECT aggregate returned every group (Having.resolveAggregateAliases); arithmetic over aggregates (MAX(x) - MIN(x) AS d) never created its operands and core rejected the bucket_script before Elasticsearch (auxiliaryAggs operands, AggregationType.BucketScript); auxiliaryAggs dedup is now order-preserving. Ten pinned having_filter fixtures per bridge copy move to the guarded form; AggregationNamingSpec pins the emitted JSON and adds two structural guards (no empty aggregation/order key; no unguarded possibly-null dereference or comparison in any emitted Painless). Seven live tests with exact row oracles green on ES 6.8 (rest + jest), 7.17, 8.18, 9.0.

Story BIDC-2

Closes #54

Closes #223

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…NOT on the right operand, guarded BETWEEN/IN, loud rejects (R2-1..R2-9)

R2-1: HAVING <alias> of a bucket_script (MAX(x) - MIN(x) AS d ... HAVING d > 3) was silently ignored and the first commit made the statement executable; the alias now resolves to the SELECT item, the selector reads params.d from the sibling pipeline aggregation (buckets_path d -> d), and inline arithmetic over aggregates in HAVING is a validation error naming the remedy. R2-2: MetricSelectorScript negated the LEFT operand of A AND NOT B; the NOT is pushed into the right-hand expression so a missing metric still fails the negated test. R2-3: BETWEEN and IN over an aggregate bypassed the guarded bucket rendering (BETWEEN even emitted a chained comparison Painless rejects); both render through bucketPipelineCheck with their own negation. R2-4: the AC 4b guard asserts a bucket_selector is present for every HAVING shape. R2-5: IS [NOT] NULL renders the null test alone. R2-6: extractMetricNames drops __now__ (both bridge copies) so a nested-level HAVING with now - interval keeps its condition. R2-9: auxiliary aggregates dedup against SELECT by expression; an alias naming a SELECT aggregate and a different HAVING/ORDER BY aggregate is a validation error.

Lead to confirm: an aggregate function in WHERE is now rejected (it used to be silently dropped while its aggregation was still created) — the house-style default, reversible. Docs: HAVING alias references and the rejected shapes.

Story BIDC-2

Closes #54

Closes #223

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…Painless == (R2-3, live)

The ES 8.18 leg found that the bucket form [v1,v2].contains(params.<metric>) never matched: a buckets_path value arrives as a boxed Double while the literals are Integers, and List.contains uses Java equals. The membership is now rendered as a disjunction of Painless == comparisons, which promote numerics and use equals for strings; the two IN pins in both bridge copies follow.

Story BIDC-2

Closes #54

Closes #223

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ELETE and UPDATE included (S2-2)

The reject lived only in SingleSearch.validate(), so DELETE FROM t WHERE COUNT(x) > 5 still became match_all and wiped the index, and the same UPDATE touched every document (#280 data-loss family). The arm now lives in Where.validate(), and Update / Delete gain a validate() that validates their WHERE; Parser.apply already validates every parsed statement, so the parse path covers DML. Pinned in HavingAggregateResolutionSpec (DELETE, UPDATE, AND-combined; plain forms stay accepted) and live (both rejected, count and ages unchanged). Also: negated()'s compound fallback documented as grammar-unreachable.

Story BIDC-2

Closes #54

Closes #223

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…checks the element, not the array (third look)

Third-look tidy-up on top of the BIDC-2 review follow-ups (verdict: approve, no
HIGH/MEDIUM residual).

- T3-1 `CreateEnrichPolicy.validate()` never validated its source WHERE, so an
  aggregate there was dropped and the policy enriched from EVERY source document
  (`match_all`). It now uses the same `where.map(_.validate())` idiom as
  `Update`/`Delete`, which brings it under the S2-2 aggregate reject.
- T3-3 `HAVING COUNT(x) IN (1, 2)` was rejected by the shared
  `Expression.validate` comparing the element with the list's ARRAY type
  (`BIGINT` vs `ARRAY<BIGINT>`), while the untyped `MAX(x) IN (...)` passed
  through `Any`. `InExpr` overrides `validate()` and compares against the list's
  ELEMENT type; a genuine mismatch (`COUNT(x) IN ('a','b')`) stays loud and now
  names the element types.
- Lead ruling 2026-09-06: the loud reject of an aggregate in WHERE (SELECT,
  DELETE, UPDATE, and now CREATE ENRICH POLICY's source WHERE) is CONFIRMED as
  shipped; the "lead to confirm" markers become "lead-confirmed 2026-09-06" and
  the behaviour change is a 0.23.0 release note.

Tests: sql `*ParserTotalitySpec *HavingAggregateResolutionSpec` 42/42 (two new
pins, each asserting the rejection is the grammar's and not the boundary catch);
`softclient4es-sql-bridge` and `es6bridge` `*AggregationNamingSpec` 28/28 each;
`++ 2.12.20 sql/Test/compile` and `scalafmtAll` clean.

Story BIDC-2

Closes #54
Closes #223

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…newly accepted by T3-3

T3-3 made `COUNT(x) IN (…)` pass validation, so a shape that used to be rejected
now reaches the bucket-pipeline emission -- and every existing IN pin was a
`MAX(x)` one, with `COUNT(x) IN` covered only at parse level, which cannot see a
wrong script.

- `AggregationNamingSpec` (template and the hand-maintained es6 copy, kept
  byte-identical): two emitted-JSON pins for `HAVING COUNT(x) IN (1, 2)` and its
  NOT twin -- the guarded `==` chain over `params.count_x` and the declared
  `buckets_path` key `count_x -> count_x`, the same assertions the MAX pins make.
  Both shapes added to the structural guards' list (35 -> 37 shapes).
- `GatewayApiIntegrationSpec`: one live case beside `MAX(age) IN (40, 50)` --
  `HAVING COUNT(name) IN (2, 3)` over the `having_naming` table (Paris 2, Lyon 1,
  Marseille 1, Nice 1) expects exactly Paris, so an empty result (a `value_count`
  long compared with Integer literals -- the S2-1 boxing class of defect) and an
  all-pass guard both fail loudly.

Tests: sql `*HavingAggregateResolutionSpec` 7/7, `softclient4es-sql-bridge` and
`es6bridge` `*AggregationNamingSpec` 30/30 each, `softclient4es-core-testkit`
compile clean, `scalafmtAll` clean.

Story BIDC-2

Closes #54
Closes #223

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fupelaqu
fupelaqu marked this pull request as ready for review September 6, 2026 11:14
@fupelaqu
fupelaqu merged commit d51e50c into main Sep 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant