Repository navigation
feat(databricks): Unity Catalog metric views from exposes[].semantics - #865
Merged
Merged
Conversation
The MCP query compiler kept its own agg-to-SQL table and percentile check, and the Databricks metric view needs the same rendering. Both now go through forge_datamodel/semantics_builder.py (AGGREGATE_FUNCTIONS, DEFAULT_PERCENTILE, percentile_params, aggregate_call, parens_balanced), so one contract aggregates the same way on every SQL path. The compiler's SQL and its QueryValidationError messages are unchanged; the dbt MetricFlow bridge keeps its own default, pinned equal by its tests.
github-actions
Bot
added
ci
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
…ntics block fluid apply now creates <catalog>.<schema>.<semantics.name> (<table>_semantic_model without a name) beside the table, in the product's own schema, as metric view YAML 1.1, so Genie, AI/BI dashboards and SQL read the contract's semantic model natively (SELECT ..., MEASURE(m) ... GROUP BY ALL). Mapping: entities and dimensions are dimensions (a time dimension at its timeGranularity: a DATE at day as it is, coarser TRUNC, else DATE_TRUNC); a measure is its aggregate (the shared semantics_builder.aggregate_call); a simple metric is that aggregate with its filter as FILTER (WHERE ...); a ratio is MEASURE(n) / NULLIF(MEASURE(d), 0); nonAdditiveDimension is a window measure (semiadditive last for max, first for min). Every column reference is written source.<column>: measured live, a bare name reads a field or a window measure first. Comments, display names (businessName) and synonyms (labels.synonyms) carry over; the model's ai-synonyms and ai-instructions go into the view comment. What the YAML cannot hold is refused at validate: derived metrics, a ratio with a filter, a non-additive dimension that is no time dimension, hour/minute on a DATE, colliding names, unsafe expressions or calls that would lend the owner's rights to readers. The provider has no metric view resource (terraform-provider-databricks#4827), so the view is created on the SQL warehouse after tofu apply, like the column masks: CREATE VIEW when missing (never OR REPLACE: it drops grants and changes the owner), ALTER VIEW ... AS when changed (keeps grants), nothing when its digest and stored structure match, DROP VIEW once no longer declared. SELECT on it is exactly the table's readers. A name held by an object this module did not create is refused before the apply, and when a restricted table's masks cannot be bound its views are closed too. Dry-run prints every statement. Borrowed: the mapping follows the Apache Ossie Databricks converter (apache/ossie-converters, databricks/, Apache-2.0); YAML per the Databricks metric view reference; deploying it as SQL on the warehouse as the databricks-agent-skills metric-views skill does. Live (Databricks Free Edition, SQL 2026.45): the semantics block of industry-model dp019_site_360 verbatim applied +77, the view and 3 grants created, MEASURE() returned each market at its latest snapshot, a second apply changed nothing; information_schema lists it as METRIC_VIEW with its column comments and grants.
…n by a quoted or qualified name
The denylist of functions that would lend the view owner's rights to its readers
(secret(), http_request(), ai_*(), read_*(), reflection, the querying identity) matched
only a bare name before "(". `secret`(...), system.builtin.secret(...) and a UDF of
another schema (cat.sch.fn(...)) passed, and IDENTIFIER('secret')(...) named the function
from a string. Every call in an expression is now read, quoted or not, qualified or not:
a quoted or qualified call is refused, IDENTIFIER() is refused, and the denylist is
checked against the name itself. Parentheses inside a string literal no longer count
toward the balance check (a ')' in '...' closes nothing).
Found by the security review of this branch (LOW: the contract author can already run
SQL as the deploy identity through a dbt job).
…ss when checking a metric view expression
The checks blanked string literals with a pattern that did not know backtick-quoted
names, which the allowlist accepts. A ' inside one (`a'`) opened a literal that ran to
the next ', so a call between them (secret(), http_request(), a qualified UDF), an
aggregate, or a ')' that closes FILTER (WHERE ...) early went unseen, while Databricks
reads `a'` as a name and runs the call. One left-to-right pass now finds backtick names
and string literals as Databricks lexes them: the call, aggregate and MEASURE checks see
the quoted names, the parentheses check ignores both, and an expression whose quote or
backtick does not close is refused.
The derived-metric and name-clash messages read plainer ("a managed table, not a metric
view").
Found by the Opus security judge of this branch (LOW, as the earlier gap).
The OpenTofu engine handed the plugins' pre_apply / post_apply hooks cprint, which reads its text as Rich markup. The Databricks hooks print SQL statements and server errors that quote the contract (a metric view's YAML, a mask predicate): a lowercase "[word]" in a description vanished from the dry-run, and a "[/word]" raised MarkupError, which the engine does not map to an access-enforcement error, so the run ended as an unexpected error and the remaining views were not attempted. The hooks now get cprint_literal, which keeps redaction and spells control characters.
…narrowing, stale views From the diff-mode inspection and defect scan of this branch: - The view is <table>_semantic_model, never semantics.name: forge's flat emitter and the dbt importer name the model after its table, so the view took the table's own name and every such contract was refused; the schema calls the name human-readable. - An entity is its key column under the column's name, and one whose key is no column is left out with a validate warning (a metric view holds no join): forge's template entity "customer" over customer_id no longer collides with a dimension "customer", and a Data Vault satellite's entity no longer refuses the contract. - A bare name that is no column of the table is refused at validate, not at CREATE after tofu apply; <table>.<column> is source.<column>; a unit or type keyword (EXTRACT(YEAR FROM d), INTERVAL 1 DAY, x::date, DATEADD(MONTH, ...)) is no column reference; two fields differing only in a literal's case collide instead of merging. - Readers narrow first: pre_apply revokes reading each existing view from whoever the contract no longer names, before tofu apply, and post_apply runs every revoke on its own, then the definition, then each grant, so a failed GRANT or ALTER VIEW no longer leaves a removed reader able to read. ALL PRIVILEGES held by a non-reader is revoked. - Stale views are found catalog-wide (table-summaries, *_semantic_model, this product's fluid.product_id, owned by the deploy identity) and dropped before the apply, so a schema the apply removes (a table moved, an expose removed) is empty for OpenTofu's destroy. A stale view whose name SQL cannot hold is left and said, not a ValueError. - The plan action's op names no OpenTofu kind, so generate iac --shadow no longer counts the view as a missing module resource; a "..."( or `x`name( call is read as a call.
…hemas The catalog-wide stale-view drop of the previous commit matched only this product's fluid.product_id and the deploy identity, so two environments of one product in one catalog (an overlay that changes only the schema: prod main.sales, staging main.sales_stg) dropped each other's view on every apply. The drop is back to the schemas the contract's tables are in (table-summaries filtered by schema and *_semantic_model), still before the apply. The cost is documented, as for the column masks' functions: a view left in a schema the contract stops using is not found, and OpenTofu's destroy of that schema fails until it is dropped by hand. The first argument of date_add / datediff / dateadd / timestampadd / timestampdiff is a unit only when it is a unit word, so the 2-argument forms (date_add(d, 1), datediff(end, start)) qualify their date column again. The name-clash refusal no longer points at semantics.name. Found by the security re-check (MEDIUM) and the fix re-check of this branch.
…view steps in their order After the rebase on #863 the hook checks the Genie space's client first, then settles the metric views (refuse a name another object holds, revoke readers the contract no longer names, drop this product's views it no longer declares), then binds the masks and row filters.
… same apply made With extensions.databricksGenie and a semantics block, one fluid apply creates the metric view in post_apply and then converges the Genie space (#863), which reads the metric views over the product's tables from Unity Catalog: the space holds the table and the view, and says "1 table(s), 1 metric view(s)". Pins the order of the two hooks and that the view's YAML source is one Genie matches.
…ot_date time dimension Since the metric views, a Databricks contract whose semantics the metric view YAML cannot hold fails validate. The Genie tests' Site 360 fixture ordered its non-additive site_count by snapshot_date without declaring that dimension, which MetricFlow refuses too (a non-additive dimension is a time dimension of its own model), so 44 Genie tests stopped on the refusal. The fixture now declares snapshot_date as a day-grain time dimension, as dp019 does; Genie reads no semantics.dimensions, so its space is unchanged. The no-SDK test's second half (without the block nothing needs the client before tofu) now uses a contract with no semantics either: with them the apply makes a metric view, which needs databricks-sdk, as the CHANGELOG says.
fas89
force-pushed
the
feat/databricks-metric-views
branch
from
November 12, 2026 22:34
d1f3448 to
5cea568
Compare
fas89
enabled auto-merge (squash)
November 12, 2026 22:35
github-actions
Bot
removed
the
needs-docs
…o feat/databricks-metric-views # Conflicts: # .secrets.baseline
…o feat/databricks-metric-views # Conflicts: # HONESTLY_TESTED.md # fluid_build/iac/providers/databricks.py
…o feat/databricks-metric-views # Conflicts: # HONESTLY_TESTED.md
…o feat/databricks-metric-views # Conflicts: # CHANGELOG.md
fas89
added a commit
that referenced
this pull request
Nov 12, 2026
iac-tests.yml: the Databricks fake lane keeps both new tiers, the Genie estate's and the metric views'.
fas89
added a commit
that referenced
this pull request
Nov 12, 2026
…ut keys After main's metric views (#865) the stand-in keeps a metric view among its tables with no constraints list, and _with_uc_joins read one for every data source of a new space: a KeyError on create (test_the_products_genie_space_holds_the_metric_view_the_apply_made). It reads constraints only where a table has them. The secrets baseline's line numbers move with the merge.
fas89
added a commit
that referenced
this pull request
Nov 12, 2026
1 of 3 tasks
fas89
added a commit
that referenced
this pull request
Nov 13, 2026
…a products (#870) * fix(databricks): a table leaving a Genie space is said as a table, not a metric view An update named every data source the space no longer holds as a metric view, since anything other than the space's own tables counted as one: taking an expose out of a product printed "updated: metric views". Each data source leaving the space is now read in Unity Catalog. One that is still a table is said as a table; a metric view, or one that is gone, as a metric view, as before. The read only names the change, so a failed read never fails the update. * refactor(databricks): Genie converge speaks through a Voice and takes a marker of its own So a space kept by another command can reuse the converge. The messages that named `fluid apply`, the contract's block or "one space per product" now read them from a Voice, set for the converge's duration (a ContextVar, reset after); the product's wording is unchanged byte for byte. Space.token overrides the marker, Compiled.voice carries the voice, ACCESS_LINE and require_warehouse are shared, and a found space whose marker is not a product's is named by its own marker in the left-alone note. The summary counts joins, and an update names a change of instructions.join_specs as "joins". * feat(databricks): fluid genie apply, one Genie space over several data products `fluid genie apply <contracts|globs|folders...> --estate <id>` creates or updates one AI/BI Genie space over the Databricks tables of every contract given (founder decisions 2026-11-12: this flag shape, and CAN_RUN only for the principals that read every table of the estate, since the space describes them all). Each table carries what a product space says of it. The space adds a join_spec for each foreign key the contracts declare between its tables, resolved by databricks_keys.foreign_keys among the estate's own tables, so each join is the Unity Catalog FOREIGN KEY fluid apply declares: MANY_TO_ONE from the referencing table, ONE_TO_ONE when the key is that table's primary key, one join for a composite key, a second alias for a key to its own table, and every column qualified by its alias. A reference outside the estate, or to no single primary key, is said. One text instruction with a section per product; no measure snippets. The space reuses the product space's identity rule and converge (databricks_genie.py): it is found by [fluid-genie-estate:<id>] in the deploying identity's own folder, updated with its etag only when it differs, and converges to exactly the contracts given. Each run checks that every table exists before it writes, and names the product to apply first. `fluid genie remove --estate <id>` trashes it. Options: --title, --env (overlays), --dry-run. Refusals and API failures are typed CLI errors (genie_estate_refused, genie_estate_failed) with the fix. Measured on Free Edition 2026-11-12 and modelled in the stand-in: join_specs are kept as sent; a self-join under two aliases and ONE_TO_ONE are accepted; an unknown relationship tag is refused ("Failed to parse export proto"), and so is a 51st table ("A space can have at most 50 tables": databricks-agent-skills' 30 is stale); 60 sample questions and an instruction of 60,000 characters are accepted. Borrowed: the join_spec shape and its rules from databricks/databricks-agent-skills (skills/databricks-genie-agents/references/create-genie-agent.md); the many-to-one orientation of apache/ossie (OSI core-spec relationships[], whose `from` is the many side); contract-file globbing as `fluid publish` does it. * docs: fluid genie apply, one Genie space over several products, in AGENTS.md and CHANGELOG The command, what the space holds beyond a product's own (the joins from the declared foreign keys), who gets CAN RUN, how it is found again, and removal; the product space's update now names a leaving table as a table. * refactor(databricks): Genie differences from a table of parts; shared test helpers Prompted by ripwire --quality-delta on the estate branch. differences names a change from two small tables of serialized_space parts instead of a ladder of comparisons (complexity 18 -> under the bar), and parses the live space once (_held). The stand-in's whole-space limits (one text instruction, a known relationship tag, 50 tables) move out of _genie_check into _genie_limit_problem. Estate.joins replaces the CLI's own count. The Genie tests read their writes through _fake_databricks.writes_since, and the estate tests write contracts through _databricks_cli.write_contract, which now takes a path below the root. * fix(cli): fluid genie in `fluid --help`; one header line instead of one overlay line per contract Found walking the command's journeys. `fluid --help` did not list `genie`; it now sits under Integrations next to publish and market, and the help-style lint covers it. A pack read with --env printed the loader's "overlay_applied" once per contract (19 lines for the tower pack): the header now names the overlay once, a missing overlay still warns, and --log-level DEBUG shows every line. The header no longer prints "0 table(s), 0 join(s)" before a refusal: the counts are shown only when the space compiled. * fix(databricks): an estate says what its tables' keys lose; the share note names its command Found by the diff review. The estate dropped the warnings databricks_keys.compile_keys gives (a primary key on a column without required: true, a labels.references that names no product and column), so a parent whose key column is nullable was reported as tagging no primary key, and a garbled reference made no join without a word. The estate now says both, as fluid apply says them of its Unity Catalog keys, and the no-key note names both conditions. The note on direct CAN_EDIT/CAN_MANAGE holders named `fluid apply` in an estate run; it reads the Voice. * fix(cli,databricks): genie apply gates each contract; globs one level; remove names itself Found by the diff defect scan. - `fluid genie apply` skipped the gate `fluid apply` runs, so a schema-invalid contract died as "Unexpected error" (exit 2) and a pre-0.7 one was not refused. Each contract now passes plan.py's _gate_contract_for_plan_or_apply (genie_contract_invalid, contract_version_unsupported), and a file the loader cannot read is contract_load_failed; both name the file. - A glob followed symlinked folders: with two symlink loops in a pack, `**` never ended. Globs now expand as `fluid publish` expands them (one level per `**`); a folder argument is the recursive search, past no symlink. A glob whose match is an ignored folder itself (`*` on `target/`) is skipped as the docstring says. - A failed `fluid genie remove` said to re-run `fluid genie apply`; it names itself. - The product's "an apply in the next minute" note had become "another apply": the default Voice now keeps it word for word, and a test pins it. * fix(databricks): a Genie space keeps only the joins it is sent, after Genie adds its own Measured on Free Edition 2026-11-13 over the 19 tower-management tables, whose Unity Catalog foreign keys fluid apply declares. Whenever a table enters a space (a create, or an update that adds one), Genie adds a join for each foreign key that touches it, with its own id and the parent aliased <table>_2, beside the joins it was sent: 24 sent, 48 held. An update with the same tables keeps exactly what it is sent. So the first run made the space, and the second "updated: joins" it. Each write is now read back, and a space the service added to is set back once and said ("Genie added N join(s) of its own ..."), so the next run finds no change. The stand-in adds the same joins on a create and on an update that adds a table. * docs: the Genie estate's live run, contract gate and Genie's own joins in the docs HONESTLY_TESTED gains the estate's unit (40), stand-in (1) and live Free Edition rows: the 19 tower-management products applied into fluid_iactest_estate, a cross-product question answered through the Genie Conversation API by the join the contracts declare, a second run with no change once Genie's own joins are taken out, removal, and the sweep (0 left). CHANGELOG and AGENTS.md say each contract passes fluid apply's gate and that a run takes out the joins Genie adds from Unity Catalog foreign keys. * chore: refresh the secrets baseline's line numbers; drop an unused import in the estate tests * refactor(databricks): smaller estate helpers for joins, refusals and contract paths ripwire --quality-delta flagged the estate's own new functions over the complexity bar: join_specs (27), contract_paths (18) and _refusals (16). Each now delegates one concern: _with_a_parent and _say_unjoined (which references can join, and what is said of the rest), _member_refusals (what one contract makes the estate refuse), and _matches and _folder_contracts (what one argument names). No behaviour change; the 40 estate tests pin it. * test(databricks): no side effect inside an assert in the estate tests Flagged by the code-quality bot on #870: remove() and Run(...)() ran inside assert statements, which python -O skips. Each call now runs on its own line and the assert checks its result (5 places; the bot named 3). * test(databricks): the stand-in's Genie joins skip a data source without keys After main's metric views (#865) the stand-in keeps a metric view among its tables with no constraints list, and _with_uc_joins read one for every data source of a new space: a KeyError on create (test_the_products_genie_space_holds_the_metric_view_the_apply_made). It reads constraints only where a table has them. The secrets baseline's line numbers move with the merge.
fas89
added a commit
that referenced
this pull request
Nov 13, 2026
…#871) * feat(dq): read a contract's DQ rules once, as checks or refusals fluid_build/util/dq_checks.py turns exposes[].contract.dq.rules into engine-neutral checks (not_null, unique, accepted_values, regex, bounds, reference, freshness, schema) and a refusal, with its fix, for each rule no check can run as written. The allowed values of a valid_values rule come from the selected column: its validationRules (enum 'A | B' or 'A,B', regex, range) or labels.references <product>.<column> through consumes[]. Thresholds below 1.0 are shares computed as fluid test computes them; critical/error map to error, warn/info to warn. Shared with the Databricks DQ sessions (monitoring, results table, Lakeflow expectations) as the one reading of the rules. * feat(databricks): a dbt job runs its contract's DQ rules as dbt tests Each check of the rules on the exposes a Databricks dbt build writes is a singular dbt test, <test path>/fluid_dq/<model>/<check>.sql, uploaded beside the project; the user's files upload byte for byte (dbt describes a model in one properties entry only). dbt build runs each test after its model, so the run page lists every rule by id and an error-severity failure fails the run. not_null, unique, accepted_values and relationships run dbt's built-in test SQL through adapter.dispatch; regex, bounds, freshness, schema and shares are forge-cli's own SQL after dbt_utils (recency, not_null_proportion) and dbt_expectations (match_regex, table_columns_to_match_set), both Apache-2.0, with no package to install on serverless. Every value reaches the file as an escaped Jinja string and every SQL literal through databricks_access.string_literal. A rule no test can run, a Git-sourced build with rules, a model or a referenced source the project lacks are refused at validate and plan. The plan lists each job's tests (dq_tests); report() reads dbt's result lines from a run's output. * feat(dq): a reference to a product not consumed keeps no expose A source-aligned product consumes nothing, so a labels.references to another product's column no longer refuses in the shared reader: Reference.expose_id is None and the engine decides whether it can find that table (the Databricks dbt job cannot, and refuses). Two exposes of one consumed product stay refused as ambiguous. Each refusal carries the rule's type, so an engine that runs a type another way (Databricks data quality monitoring for anomaly_detection) can tell its own. * feat(databricks): the dbt job refuses an unconsumed reference; a run's report covers rules dbt stopped before The dbt test of a reference reads the other product from the dbt source the project declares for the expose it consumes, so a reference to a product consumes[] does not list is refused there. report() says once when the task's output holds no result of any rule's test (dbt stopped before them). The job's dbt task key is a constant, DBT_TASK_KEY, which a run's report finds the dbt output under. * fix(databricks): the schema test refs its model while dbt parses dbt records a test's parents while it parses, with execute false, and dbt build --select +model+ selects tests by them. The schema test called ref() only inside its execute guard, so it had no parent and was never selected: live on Databricks Free Edition dbt found 11 tests and ran 10. The relation is now set before the guard. Every kind is pinned to ref its model at parse time. * docs(databricks): changelog entry and the live row for DQ rules as dbt tests The Unreleased entry says what runs for each rule type, what is refused, and what fluid apply prints. HONESTLY_TESTED records the unit tiers and the manual live run on Databricks Free Edition (2026-11-12): a seeded bad row failed the run on its two error rules and warned on two, a clean run passed all 11 tests, the next plan was empty, and everything was swept; it also lists what was not run live. * refactor(dq): smaller units in the rule reader and the test compiler; two hardening fixes From the branch's security review and ripwire --quality-delta: - The schema test escapes the expected type as it escapes the type the warehouse reports. It is unreachable today (every Databricks column type passes _column_type first), but no longer depends on it (LOW). - A reference refusal quotes the labels.references parts it names (LOW). - compile_tests runs in stages (a _Project with its paths and lazily read sources, _test_folder, _add_test); _declared_sources reads through _properties_docs and _source_tables; _Rule._allowed reads each validationRule kind in its own method; read_rules and has_rules share _parts; report's wording is _outcome. - The job's tests ride on _Source, so _emit_job keeps its signature and its task dict stays one line. * feat(dq): each check keeps its rule's severity, threshold and operator as written DqCheck.rule_severity, rule_threshold and rule_operator carry the rule as the contract wrote it, beside the two-level severity a dbt test runs at and the share a lower threshold becomes: Lakeflow expectations fail an update on critical and drop the row on error, and a results table reports one row per rule. * fix(databricks): refuse a test name the project uses and a selection that skips the model From the diff's delivery review: - A rule's test is named after its id, and dbt names a singular test once per project: a project test of the same name made the whole job fail to parse ("dbt found two data_tests with the name"). It is refused at plan. - A build whose dbt selection provably leaves the model out would never run its tests: refused, with the model's fqn and path evaluated by databricks_retention.selects_model. A tag, config or state selection cannot be told here; the run's report names any test that did not run. - A real dbt (dbt-core with dbt-duckdb, as CI installs) parses the upload and dbt ls --select +orders+ lists every rule's test; with the schema test's ref put back behind its execute guard, it fails. - Wording: a share is fluid test's own only for completeness and uniqueness; the anomaly and drift refusal says what fluid test does with them; dq_checks is the reading for the engines that run the rules, this job its first. * feat(databricks): fluid apply reports what dbt found for each contract rule after a run Under --mode amend-and-build, once the job's run ends (success or failure), the run report reads the dbt task's output and prints one line per contract rule's test: pass, warn or fail with its failing-row count, error, skip, or not in the output (with the note when Databricks kept only its end), then the summary. The rules come from the contract alone (databricks_dbt_tests.expected_tests); the task is found by DBT_TASK_KEY, so the fluid_keys task beside it is never read as dbt's. runs/get-output is read once per task and feeds the dbt summary, the failed task's tail and the rules alike (_task_output). A rule report is printed for the dbt task when a later task failed too: its tests ran. Never fails a build: an unreadable output or contract only drops the report. * fix(dq): refuse a threshold or bound no float holds, and a range no value meets From the diff's defect scan: an integer threshold of hundreds of digits raised OverflowError out of fluid validate and plan, and so did a window too long to count; an infinite or NaN threshold, or a range bound past a float's range, reached the SQL as a bare 'inf'. Each is now a refusal (_finite, and parse_range drops a non-finite bound). A range no value can meet (between 100 and 0, > 5 and < 1, > 3 and <= 3) is refused instead of failing every row of its test. * fix(databricks): defect-scan fixes to the DQ tests and the keys task - Only the exposes with rules are tested, so only their models are looked for: a sibling output with no rules (no model of its name, or left out of the selection) was refused. - properties.model names the model of the one table a build writes; a build that writes several (no outputs writes every expose) tests each through the model named after its table. - fluid validate, which reads the contract before {{ env.* }} is substituted, no longer refuses a model named by a placeholder; the apply writes those tests. - The schema test spells a decimal as the table emitter does (_canonical_type): decimal(12) is decimal(12,0), as Databricks reports. - _jinja_str keeps characters outside the BMP whole: ensure_ascii wrote a surrogate pair, which dbt cannot encode (one emoji in a rule id failed the whole project's parse). - The test-name clash check also covers the project's own files under fluid_dq/, and a project YAML nested past the recursion limit is passed over like any file that does not parse. - The fluid_keys task (#860) runs with run_if ALL_DONE: a failing rule fails dbt after its model replaced the table, which dropped its keys. * docs(databricks): AGENTS.md and the live row for the DQ tests and their report AGENTS.md says the job runs the contract's DQ rules as dbt tests and where the reader and the renderer live; the changelog says the keys task now runs whatever dbt's outcome; HONESTLY_TESTED counts the tests again (48 + 54 + 4) and lists what the review waves added. * feat(databricks): an anomaly_detection rule is left to data quality monitoring #864 turns Databricks data quality monitoring on for an expose with an anomaly_detection rule, and the monitor learns the rule's bounds, which a dbt test cannot. The job's tests therefore leave such a rule to it (databricks_quality.monitoring_request), as agreed with that change. An expose that turns monitoring off (contract.dq.monitoring.enabled: false) still has its anomaly rule refused, and the fix names the opt-out. drift_detection stays refused. * fix(databricks): refuse an unpaired surrogate; a placeholder selection waits for the apply From the fix wave's re-check: - Since the Jinja strings keep raw Unicode, an unpaired surrogate (YAML's "\ud83d" escape loads one) in a rule id, an enum value or a regex raised UnicodeEncodeError out of fluid plan. The reader refuses one with the control characters, in ids too, and a value from the project's own YAML (a source name) that UTF-8 cannot hold is a refusal of its own. - fluid validate, before {{ env.* }} is substituted, refused a build whose properties.select still held a placeholder as leaving the model out; the selection is judged only once it can be read, at the apply. * docs(databricks): the anomaly hand-off to monitoring, and the counts, in the changelog and the live row * fix(databricks): refuse a test path no file can hold; monitoring takes over only the history refusal From the round-2 re-check: - An unpaired surrogate in dbt_project.yml's first test path (YAML's "\ud83d" escape loads one) reached the generated tests' paths and raised UnicodeEncodeError out of fluid plan. It is a test-path refusal now, and the message names the path escaped. - On an expose Databricks data quality monitoring watches, every refusal of an anomaly_detection rule was dropped, so a bad severity or an id holding a control character passed silently. DqRefusal.needs_history marks the one refusal monitoring takes over (the rule needs the table's history); the others are refused as before. * fix(databricks): a DQ refusal names a lone surrogate from the project escaped The round-3 re-check found model-paths reaching the model refusal's text raw, and a lone surrogate there (a YAML escape loads one) made fluid plan fail on printing the refusal. Every DQ refusal is built in _refusal, which now spells such a character as its backslash escape; a source or table name from the project's YAML in a reference refusal is covered the same way. * docs(databricks): the live row counts the round-3 tests and names the escaped refusal text * fix(dq): a valid_values rule allows what fluid test allows #867 landed fluid test's reading of a valid_values rule, providers/quality_engine.py::value_domain: the values the rule itself lists, else every declaration its column makes (the first enum, regex and range validationRule, and labels.references), all of which must hold. dq_checks had its own, which let the column's declarations win over the rule's list, split an enum on | only when it held one, and read every validationRule of a kind; the dbt tests could then check other values than fluid test does. dq_checks now takes the values, pattern and bounds from value_domain and keeps its refusals on top: a range fluid test cannot read (it passes one over) or no value can meet, a value or pattern holding a control character, and a labels.references that consumes[] cannot resolve. Its own parse_enum and parse_range are gone; the range grammar is quality_engine's. The Tower Management pack still reads as 330 checks with none refused, check for check as before, and the demo's dbt products as 230 tests. * docs(dq): valid_values reads as fluid test's value_domain, in AGENTS.md, the changelog and the live row * chore(secrets): baseline line numbers after the rebase on #865 and #869 * test(databricks): the schema test's expected rows join their parts with an explicit +
This was referenced Nov 13, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
fluid applyon a Databricks contract now creates a Unity Catalog metric view from each expose'ssemanticsblock. Genie, AI/BI dashboards and SQL then read the contract's semantic model natively:<catalog>.<schema>.<table>_semantic_model, beside the table in the product's own schema. It is named after the table, notsemantics.name. Forge's own emitters and the dbt importer name the model after its table, and the schema calls the name human-readable.timeGranularity: aDATEatdaystays as it is, a coarser grain on aDATEbecomesTRUNC(col, 'MONTH'), and anything elseDATE_TRUNC.FILTER (WHERE ...).MEASURE(num) / NULLIF(MEASURE(den), 0).nonAdditiveDimensionbecomes a window measure:order: <dim>, range: current, withsemiadditive: lastformaxandfirstformin.source.<col>, and<table>.<col>is rewritten the same way. Measured live, a bare name in a metric view reads a field or window measure before a column. Unit and type keywords stay keywords:EXTRACT(YEAR ...),INTERVAL 1 DAY,::date, the unit argument ofDATEADD.businessNamebecomes the dimension'sdisplay_name, and itslabels.synonymsitssynonyms.ai-synonymsandai-instructionslabels go into the view's comment.SELECTis granted to them.SELECTandALL PRIVILEGESare revoked from anyone else.fluid validate, not approximated:hourorminutegrain on aDATE;_sql_safetyallowlist, unclosed quotes, unbalanced parentheses;secret(),http_request(),ai_*(),read_*(), reflection, identity functions,IDENTIFIER(), and quoted or qualified function names.aggTimeDimensionanddefaultAggTimeDimension(MetricFlow'smetric_time),createMetric, a metric'sowner, the model'stagsandai-example-questions.How
The databricks/databricks provider has no metric view resource (terraform-provider-databricks#4827). So the view uses the column masks' seam: SQL on the warehouse around
tofu apply(databricks_enforce.py), influid_build/iac/providers/databricks_metrics.py.fluid.product_id, and a view another identity owns.CREATE VIEW ... WITH METRICS LANGUAGE YAML TBLPROPERTIES (...)when the view is missing. NeverOR REPLACE: measured live, a replace drops the view's grants and makes the replacer its owner.ALTER VIEW ... AS $$...$$when the definition changed; it keeps the grants.--mode dry-runprints every statement. The reportfluid diff --outwrites lists the view as adatabricks_metric_viewaction, whichgenerate iac --shadowleaves out of the OpenTofu parity count.fluid plandoes not list it (0.7.x contracts plan through_plan_with_provider_actions).amountt * 2) passesfluid validate. Databricks then refuses theCREATE VIEWaftertofu apply, and the run fails withdatabricks-metric-view-failed. A single-nameexprthat is no column is refused at validate. Checking every name needs a SQL parser; that is a follow-up.post_apply, before the space converges, so the first apply already lists it beside its table. A fake e2e test pins that.semantics_builder.aggregate_call). The compiler's SQL and errors are unchanged.cprint_literal. Before, a[word]in a description vanished from the dry-run and a[/word]raisedMarkupError.Behaviour changes (both in the CHANGELOG):
semanticsnow needs the[databricks]extra at apply time, as column masks already do.fluid validatefor a Databricks binding.Known limit: a view left in a schema the contract stops using (its table moved, or its last expose removed) is not found. OpenTofu's destroy of that schema then fails until the view is dropped by hand. The mask functions already have the same limit. A state-aware cleanup for both is a follow-up.
Borrow before build
databricks_enforce.py: the SQL seam, client and fail-safe, reused.output_ports/mcp/query_compiler.py: the strict aggregate rendering, now shared.engines/dbt/semantic_models.py: the MetricFlow reading of the same block.semantics_builder.pyandtime_grains.py.copilot/schemas/osi.py,emit/osi_sidecar.py) runs on the forge pipeline's in-memory model, and nothing converts a contract back to OSI.databricks/(Apache-2.0): its OSI-to-metric-view mapping and refusals. Adapted, not depended on, because it is Java.databricks-metric-views: deploying as SQL on the warehouse.metric_viewmaterialization (1.12): not used, since this is the IaC path.Validation
broken pipein the metadata fake tier) also reproduces on origin/main under load.fluidCLI,tofuand provider 1.140.0, added to theiac-testsDatabricks lane. The fifth appliessemanticswithextensions.databricksGenie, and the space holds the table and the view.tofu, offline provider mirror): 61 passed. The startup budget: 9 passed..secrets.baseline, HONESTLY_TESTED.md and the plugin, all independent additions, and both sides were kept. The plugin docstring's stale<semantics.name>naming was fixed in the merge.site_countbysnapshot_datewithout declaring that dimension. MetricFlow refuses that too. The fixture now declaressnapshot_dateas a day-grain time dimension, as dp019 does. Genie reads nosemantics.dimensions, so its space is unchanged.fluid_iactest_metrics, droppedCASCADEafterwards:+77. The view was created and SELECT granted to the 3 reader principals.MEASURE()returned each market at its latest snapshot, and each day by date.+0 ~0 -0, and no statement ran.unchanged, and a stale probe view carrying the product's id was dropped before the apply.information_schema.tableslists the view asMETRIC_VIEW, and.columnsand.table_privilegesshow its comments and grants.CREATE OR REPLACEdrops grants;ALTER VIEW ... ASkeeps them.comment,display_nameandsynonyms.semiadditive: lastis per output group.Reviews
Documentation
Docs PR linked
No docs needed
Docs TODO: forge_docs
docs/providers/databricks.mdneeds a "Metric views" section. It should cover:semanticsmakes (<table>_semantic_model), and querying it withMEASURE();fluid validaterefuses;--mode dry-run;[databricks]extra, the known limit, and that the Genie space lists the view.A verified draft is written but not committed yet. The Trello card https://trello.com/c/nb8MjAzn (P2) and the
needs-docslabel track it. Squash auto-merge is armed here, so the docs PR may land after this one. Until then, the module docstring, CHANGELOG and HONESTLY_TESTED.md describe the behaviour.