Repository navigation
feat(databricks): Unity Catalog primary and foreign keys from the contract - #860
Merged
Merged
Conversation
A capped name kept its first characters and ended in an 8-digit digest of the whole in gcp_governance._bounded; the Databricks key constraints need the same with an underscore. naming.bounded_ident is that helper, and _bounded delegates to it (same output).
…tract A column tagged primary-key becomes the table's PRIMARY KEY (composite in schema order; each column must be required), and one tagged foreign-key with labels.references: <product id>.<column> a FOREIGN KEY to that product's table, so Catalog Explorer draws the ERD and Genie can join the products. FLUID 0.7.6/0.7.7 has no key field ($defs.column is closed), so these are the tag conventions the ODCS exporter and the changelog rules read. - databricks_keys.py: compile, resolve, plan, reconcile. A parent in the same contract is one of its exposes; any other is found live by the fluid_product_id tag every forge table carries (system.information_schema.table_tags), its live PRIMARY KEY must match, the child's catalog wins a tie, an ambiguous one is skipped and said. - fluid apply declares them in post_apply on the SQL warehouse (terraform-provider-databricks has no constraint argument: #4157, PR #4205 unmerged), re-declares what is missing, replaces or drops only keys it named (fluid_pk_<table>, fluid_fk_<table>_<cols>), and never fails the run on a key: each skip says why and what to do. --mode dry-run prints the statements. - Measured on a warehouse (dbsql 2026.45): CREATE OR REPLACE drops a table's keys and every foreign key referencing it; INSERT OVERWRITE, tags and masks survive. So a Databricks dbt job that writes a keyed table gets a second task, fluid_keys, a SQL file task whose CONTINUE handler passes over a failing statement (re-adding keys dbt dropped), and a dbt build that amend-and-build runs locally is followed by the same re-declaration (post_build hook). - The pattern is dbt-databricks' own: ALTER TABLE ... ADD CONSTRAINT after every create or replace (macros/relations/constraints.sql). - _fake_databricks serves the keys with the live rules and error texts, and the table_tags lookup.
Any table owner can tag a table fluid_product_id, so a lookalike table in the child's own catalog outranked the parent this pipeline deployed in another catalog. A candidate the deploying identity owns (read-back owner vs SCIM Me, as the mask functions are checked) now wins a tie before the catalog does. Nothing is refused: a parent another identity deployed is still used when it is the only match, so a cross-team foreign key keeps working.
…es, coverage
From the diff's security, delivery and defect reviews:
- A dry run of --mode amend-and-build no longer declares keys: the post-build hook runs only
when the apply is real, and runs after a build that failed part way too (it may already
have rebuilt a table).
- The post-build hook gets the contract as the apply did, {{ env.* }} templates resolved;
before, a templated binding made it skip the keys without a word.
- fluid validate (and the verify inventory) of a templated location no longer raises: a
key statement for a name that is no plain identifier yet is left out of the job script.
- emit looks up other products' tables only after the refusal and sovereignty checks, waits
at most 120 s, and a lookup that fails is a plan notice instead of silence; one unreadable
or oddly named tagged table no longer discards the others.
- A composite key and a single-column key to two tables of one product are both declared; a
column that is half of a composite key is said as such.
- A foreign key this module declared is kept only while it still says what the contract does
(or its product cannot be looked up); one a new key covers, or whose reference moved, is
dropped.
- A job's script also re-declares the contract's own foreign keys into the tables it rebuilds.
- Key names can no longer collide in a schema: fluid_fk_<table>__<col>[__<col>], with a
digest when a part could read two ways.
- Own exposes' declared primary keys count even when a parent's table could not be read.
- The hooks' output (server error classes, the install hint) prints literally.
- Docs: the job task passes over a key it cannot add (a nullable key column) and needs a
warehouse that runs SQL scripting.
The review round made one unreadable tagged table no longer discard the others, but it then left its product "resolved" without that table, so the next fluid apply dropped a correct foreign key to it and said the contract no longer declared it. A product with a table that could not be read is now unknown, like one whose lookup failed, and its keys stay. A key that is dropped says what changed: a new key replaces it, its columns reference another product now, or the contract no longer declares it.
…oduct When one table of the referenced product could not be read and a sibling with the same primary key could, the apply moved a correct foreign key to the sibling, where the next apply then left it. While not every table of that product can be read, a key that is in place stays there, and the apply says why it was not moved.
The partial-view guard kept a key only to stop a move to another table; a key whose parent is the same and whose parent columns changed is replaced, and the drop says that instead of "its parent changed" (fix re-check, LOW).
github-actions
Bot
added
cli
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
The expected statements in a key-plan test are joined with an explicit +, not adjacent literals; the unused Optional import goes; REBUILD_TIMEOUT_SECONDS is in __all__, since databricks_jobs imports it (the per-file analysis saw no use).
fas89
enabled auto-merge (squash)
November 12, 2026 21:31
fas89
added a commit
that referenced
this pull request
Nov 12, 2026
…-space CHANGELOG [Unreleased] Added: kept both the Genie space and the Unity Catalog keys entries.
This was referenced Nov 12, 2026
Merged
Merged
fas89
added a commit
that referenced
this pull request
Nov 12, 2026
- Only the exposes with rules are tested, so only their models are looked
for: a sibling output with no rules (no model of its name, or left out
of the selection) was refused.
- properties.model names the model of the one table a build writes; a
build that writes several (no outputs writes every expose) tests each
through the model named after its table.
- fluid validate, which reads the contract before {{ env.* }} is
substituted, no longer refuses a model named by a placeholder; the
apply writes those tests.
- The schema test spells a decimal as the table emitter does
(_canonical_type): decimal(12) is decimal(12,0), as Databricks reports.
- _jinja_str keeps characters outside the BMP whole: ensure_ascii wrote
a surrogate pair, which dbt cannot encode (one emoji in a rule id
failed the whole project's parse).
- The test-name clash check also covers the project's own files under
fluid_dq/, and a project YAML nested past the recursion limit is passed
over like any file that does not parse.
- The fluid_keys task (#860) runs with run_if ALL_DONE: a failing rule
fails dbt after its model replaced the table, which dropped its keys.
1 of 3 tasks
fas89
added a commit
that referenced
this pull request
Nov 13, 2026
…#871) * feat(dq): read a contract's DQ rules once, as checks or refusals fluid_build/util/dq_checks.py turns exposes[].contract.dq.rules into engine-neutral checks (not_null, unique, accepted_values, regex, bounds, reference, freshness, schema) and a refusal, with its fix, for each rule no check can run as written. The allowed values of a valid_values rule come from the selected column: its validationRules (enum 'A | B' or 'A,B', regex, range) or labels.references <product>.<column> through consumes[]. Thresholds below 1.0 are shares computed as fluid test computes them; critical/error map to error, warn/info to warn. Shared with the Databricks DQ sessions (monitoring, results table, Lakeflow expectations) as the one reading of the rules. * feat(databricks): a dbt job runs its contract's DQ rules as dbt tests Each check of the rules on the exposes a Databricks dbt build writes is a singular dbt test, <test path>/fluid_dq/<model>/<check>.sql, uploaded beside the project; the user's files upload byte for byte (dbt describes a model in one properties entry only). dbt build runs each test after its model, so the run page lists every rule by id and an error-severity failure fails the run. not_null, unique, accepted_values and relationships run dbt's built-in test SQL through adapter.dispatch; regex, bounds, freshness, schema and shares are forge-cli's own SQL after dbt_utils (recency, not_null_proportion) and dbt_expectations (match_regex, table_columns_to_match_set), both Apache-2.0, with no package to install on serverless. Every value reaches the file as an escaped Jinja string and every SQL literal through databricks_access.string_literal. A rule no test can run, a Git-sourced build with rules, a model or a referenced source the project lacks are refused at validate and plan. The plan lists each job's tests (dq_tests); report() reads dbt's result lines from a run's output. * feat(dq): a reference to a product not consumed keeps no expose A source-aligned product consumes nothing, so a labels.references to another product's column no longer refuses in the shared reader: Reference.expose_id is None and the engine decides whether it can find that table (the Databricks dbt job cannot, and refuses). Two exposes of one consumed product stay refused as ambiguous. Each refusal carries the rule's type, so an engine that runs a type another way (Databricks data quality monitoring for anomaly_detection) can tell its own. * feat(databricks): the dbt job refuses an unconsumed reference; a run's report covers rules dbt stopped before The dbt test of a reference reads the other product from the dbt source the project declares for the expose it consumes, so a reference to a product consumes[] does not list is refused there. report() says once when the task's output holds no result of any rule's test (dbt stopped before them). The job's dbt task key is a constant, DBT_TASK_KEY, which a run's report finds the dbt output under. * fix(databricks): the schema test refs its model while dbt parses dbt records a test's parents while it parses, with execute false, and dbt build --select +model+ selects tests by them. The schema test called ref() only inside its execute guard, so it had no parent and was never selected: live on Databricks Free Edition dbt found 11 tests and ran 10. The relation is now set before the guard. Every kind is pinned to ref its model at parse time. * docs(databricks): changelog entry and the live row for DQ rules as dbt tests The Unreleased entry says what runs for each rule type, what is refused, and what fluid apply prints. HONESTLY_TESTED records the unit tiers and the manual live run on Databricks Free Edition (2026-11-12): a seeded bad row failed the run on its two error rules and warned on two, a clean run passed all 11 tests, the next plan was empty, and everything was swept; it also lists what was not run live. * refactor(dq): smaller units in the rule reader and the test compiler; two hardening fixes From the branch's security review and ripwire --quality-delta: - The schema test escapes the expected type as it escapes the type the warehouse reports. It is unreachable today (every Databricks column type passes _column_type first), but no longer depends on it (LOW). - A reference refusal quotes the labels.references parts it names (LOW). - compile_tests runs in stages (a _Project with its paths and lazily read sources, _test_folder, _add_test); _declared_sources reads through _properties_docs and _source_tables; _Rule._allowed reads each validationRule kind in its own method; read_rules and has_rules share _parts; report's wording is _outcome. - The job's tests ride on _Source, so _emit_job keeps its signature and its task dict stays one line. * feat(dq): each check keeps its rule's severity, threshold and operator as written DqCheck.rule_severity, rule_threshold and rule_operator carry the rule as the contract wrote it, beside the two-level severity a dbt test runs at and the share a lower threshold becomes: Lakeflow expectations fail an update on critical and drop the row on error, and a results table reports one row per rule. * fix(databricks): refuse a test name the project uses and a selection that skips the model From the diff's delivery review: - A rule's test is named after its id, and dbt names a singular test once per project: a project test of the same name made the whole job fail to parse ("dbt found two data_tests with the name"). It is refused at plan. - A build whose dbt selection provably leaves the model out would never run its tests: refused, with the model's fqn and path evaluated by databricks_retention.selects_model. A tag, config or state selection cannot be told here; the run's report names any test that did not run. - A real dbt (dbt-core with dbt-duckdb, as CI installs) parses the upload and dbt ls --select +orders+ lists every rule's test; with the schema test's ref put back behind its execute guard, it fails. - Wording: a share is fluid test's own only for completeness and uniqueness; the anomaly and drift refusal says what fluid test does with them; dq_checks is the reading for the engines that run the rules, this job its first. * feat(databricks): fluid apply reports what dbt found for each contract rule after a run Under --mode amend-and-build, once the job's run ends (success or failure), the run report reads the dbt task's output and prints one line per contract rule's test: pass, warn or fail with its failing-row count, error, skip, or not in the output (with the note when Databricks kept only its end), then the summary. The rules come from the contract alone (databricks_dbt_tests.expected_tests); the task is found by DBT_TASK_KEY, so the fluid_keys task beside it is never read as dbt's. runs/get-output is read once per task and feeds the dbt summary, the failed task's tail and the rules alike (_task_output). A rule report is printed for the dbt task when a later task failed too: its tests ran. Never fails a build: an unreadable output or contract only drops the report. * fix(dq): refuse a threshold or bound no float holds, and a range no value meets From the diff's defect scan: an integer threshold of hundreds of digits raised OverflowError out of fluid validate and plan, and so did a window too long to count; an infinite or NaN threshold, or a range bound past a float's range, reached the SQL as a bare 'inf'. Each is now a refusal (_finite, and parse_range drops a non-finite bound). A range no value can meet (between 100 and 0, > 5 and < 1, > 3 and <= 3) is refused instead of failing every row of its test. * fix(databricks): defect-scan fixes to the DQ tests and the keys task - Only the exposes with rules are tested, so only their models are looked for: a sibling output with no rules (no model of its name, or left out of the selection) was refused. - properties.model names the model of the one table a build writes; a build that writes several (no outputs writes every expose) tests each through the model named after its table. - fluid validate, which reads the contract before {{ env.* }} is substituted, no longer refuses a model named by a placeholder; the apply writes those tests. - The schema test spells a decimal as the table emitter does (_canonical_type): decimal(12) is decimal(12,0), as Databricks reports. - _jinja_str keeps characters outside the BMP whole: ensure_ascii wrote a surrogate pair, which dbt cannot encode (one emoji in a rule id failed the whole project's parse). - The test-name clash check also covers the project's own files under fluid_dq/, and a project YAML nested past the recursion limit is passed over like any file that does not parse. - The fluid_keys task (#860) runs with run_if ALL_DONE: a failing rule fails dbt after its model replaced the table, which dropped its keys. * docs(databricks): AGENTS.md and the live row for the DQ tests and their report AGENTS.md says the job runs the contract's DQ rules as dbt tests and where the reader and the renderer live; the changelog says the keys task now runs whatever dbt's outcome; HONESTLY_TESTED counts the tests again (48 + 54 + 4) and lists what the review waves added. * feat(databricks): an anomaly_detection rule is left to data quality monitoring #864 turns Databricks data quality monitoring on for an expose with an anomaly_detection rule, and the monitor learns the rule's bounds, which a dbt test cannot. The job's tests therefore leave such a rule to it (databricks_quality.monitoring_request), as agreed with that change. An expose that turns monitoring off (contract.dq.monitoring.enabled: false) still has its anomaly rule refused, and the fix names the opt-out. drift_detection stays refused. * fix(databricks): refuse an unpaired surrogate; a placeholder selection waits for the apply From the fix wave's re-check: - Since the Jinja strings keep raw Unicode, an unpaired surrogate (YAML's "\ud83d" escape loads one) in a rule id, an enum value or a regex raised UnicodeEncodeError out of fluid plan. The reader refuses one with the control characters, in ids too, and a value from the project's own YAML (a source name) that UTF-8 cannot hold is a refusal of its own. - fluid validate, before {{ env.* }} is substituted, refused a build whose properties.select still held a placeholder as leaving the model out; the selection is judged only once it can be read, at the apply. * docs(databricks): the anomaly hand-off to monitoring, and the counts, in the changelog and the live row * fix(databricks): refuse a test path no file can hold; monitoring takes over only the history refusal From the round-2 re-check: - An unpaired surrogate in dbt_project.yml's first test path (YAML's "\ud83d" escape loads one) reached the generated tests' paths and raised UnicodeEncodeError out of fluid plan. It is a test-path refusal now, and the message names the path escaped. - On an expose Databricks data quality monitoring watches, every refusal of an anomaly_detection rule was dropped, so a bad severity or an id holding a control character passed silently. DqRefusal.needs_history marks the one refusal monitoring takes over (the rule needs the table's history); the others are refused as before. * fix(databricks): a DQ refusal names a lone surrogate from the project escaped The round-3 re-check found model-paths reaching the model refusal's text raw, and a lone surrogate there (a YAML escape loads one) made fluid plan fail on printing the refusal. Every DQ refusal is built in _refusal, which now spells such a character as its backslash escape; a source or table name from the project's YAML in a reference refusal is covered the same way. * docs(databricks): the live row counts the round-3 tests and names the escaped refusal text * fix(dq): a valid_values rule allows what fluid test allows #867 landed fluid test's reading of a valid_values rule, providers/quality_engine.py::value_domain: the values the rule itself lists, else every declaration its column makes (the first enum, regex and range validationRule, and labels.references), all of which must hold. dq_checks had its own, which let the column's declarations win over the rule's list, split an enum on | only when it held one, and read every validationRule of a kind; the dbt tests could then check other values than fluid test does. dq_checks now takes the values, pattern and bounds from value_domain and keeps its refusals on top: a range fluid test cannot read (it passes one over) or no value can meet, a value or pattern holding a control character, and a labels.references that consumes[] cannot resolve. Its own parse_enum and parse_range are gone; the range grammar is quality_engine's. The Tower Management pack still reads as 330 checks with none refused, check for check as before, and the demo's dbt products as 230 tests. * docs(dq): valid_values reads as fluid test's value_domain, in AGENTS.md, the changelog and the live row * chore(secrets): baseline line numbers after the rebase on #865 and #869 * test(databricks): the schema test's expected rows join their parts with an explicit +
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Databricks target now declares Unity Catalog primary and foreign keys from the contract, so Catalog Explorer draws the entity-relationship diagram and Genie can join the products.
primary-key(orprimaryKey) becomes the table's PRIMARY KEY. A composite key takes every tagged column, in schema order. Each key column must berequired: true, because Unity Catalog declares a key only on NOT NULL columns. A key column that is not required makes afluid validatewarning, and the table gets no key.foreign-keywithlabels.references: <product id>.<column>becomes a FOREIGN KEY to that product's table. References to one product that exactly cover a composite primary key form one key. A single-column key to another table of the same product works alongside it.$defs.columnis closed, andsemantics.entitiesnames no target table. These are the tag conventions the ODCS exporter and the changelog rules already read.Design calls
fluid_product_idtag every forge table carries, viasystem.information_schema.table_tags, wherever an overlay put it and whichever contract or repo applied it.fluid_pk_<table>andfluid_fk_<table>__<col>[__<col>]. They are unique in the schema; a part that could read two ways adds a digest, and a name over 255 characters is cut with a digest.fluid_prefix.CREATE OR REPLACE. So each Databricks dbt job that writes a keyed table gets a second task,fluid_keys: asql_taskon the same warehouse that runs an uploaded script after dbt succeeds.DECLARE CONTINUE HANDLER FOR SQLEXCEPTIONpasses over a key it cannot add, such as a key column the dbt model leaves nullable. The nextfluid applyreports that.--mode amend-and-buildruns locally is followed by the same re-declaration through an advisorypost_buildplugin hook. The hook is skipped on dry runs and gets the template-resolved contract.There is no public contract, schema or CLI change.
Measured on Databricks SQL (Free Edition, serverless warehouse, dbsql 2026.45)
CREATE OR REPLACE TABLE, as SELECT or with a column list, drops the table's keys and every FOREIGN KEY on another table that references it, in any schema.INSERT OVERWRITEkeeps keys. Tags and column masks survive a replace.DROP COLUMNtakes the key on that column with it.DROP TABLEof a referenced parent succeeds and takes the child's FK.ADD CONSTRAINThas noIF NOT EXISTS. Dropping a referenced PRIMARY KEY fails withoutCASCADE. A key needs NOT NULL columns, and an FK needs the parent's PK columns. A cross-schema FK works.Borrow receipts
databricks_enforce.SqlRunnerandread_table(the table GET already returnstable_constraints[]);databricks_access.quote,quote_tableandstring_literalfor all SQL text;_PRODUCT_TAGS(fluid_product_id) as the parent index;databricks_jobs._emit_filesand_under_homefor the uploaded script;build_runners.base.is_dbt_build;_console.cprint_literal.naming.bounded_identreplaces a copy, and GCP's_boundednow delegates to it.macros/relations/constraints.sql) runsALTER TABLE ... ADD CONSTRAINTafter every create or replace. Pattern adapted.sql_exporter.py: PK only, no FK.primaryKeyandrelationships.information_schemakey views.Validation
ruff, andblack24.10.0--check: clean..[dev,local]venv, the CI way (-n 6 --dist loadscope --ignore=tests/perf, tofu hidden): 34501 passed. The 3 failures are local baselines that fail identically on pristine main: twotests/providers/test_duckdb_sandbox_workspace_root.pytests, andtest_build_runners.py::...test_uses_configured_dbt_command_prefix_verbatim. Perf budgets: 41 passed.test_iac_databricks_fake_e2e.py,test_iac_databricks_jobs_fake_e2e.pyandtest_iac_databricks_keys.py: 116 passed.fluid apply.fluid_keystask, whose second plan is+0 ~0 -0.tests/iac/test_iac_databricks_keys.py(67 tests) covers:ripwire --quality-delta, with the branch diff on a fresh base worktree:is_dbt_build) and theapply.py::runcomplexity bump.DatabricksIacPlugin+44 lines forpost_buildand the emit lookup ordering,_emit_job+13) and the stand-in's new statement branches.Live proof (scratch catalog
fluid_iactest_keys, now dropped together with its job and uploads)Two contracts:
iactest.keys.sites_v1(PKsite_id), andiactest.keys.visits_v1(PKvisit_id, FKsite_idtoiactest.keys.sites_v1.site_id, built by a dbt job).fluid apply sitesgavetofu +9, thendone: primary key fluid_pk_sites (site_id).fluid apply visitsgavetofu +16, then the PK anddone: foreign key ... (site_id) references fluid_iactest_keys.keys_sites.sites (site_id). The parent was found across contracts by its tag.information_schema.table_constraintslists the two PRIMARY KEYs and the FOREIGN KEY.referential_constraintslinks the FK tokeys_sites.fluid_pk_sites, andkey_column_usagehas the columns.sites(PKsite_id) ←visits(PKvisit_id, FKsite_id). The screenshot is not attached because it shows a personal account.dbtSUCCESS, thenfluid_keysSUCCESS. The table history shows v1CREATE OR REPLACE TABLE AS SELECTby dbt, and the PK and FK are back after the run.fluid apply sitesgave+0 ~1 -0and re-added the PK;fluid apply visitsgave+0 ~0 -0and re-added the FK.+0 ~1 -0(only the script changed), and a second dbt run'sfluid_keysre-declaredfluid_pk_visitsandfluid_fk_visits__site_id.Reviews
security-review: PASS. The first pass found one candidate: a lookalike table tagged
fluid_product_id. The false-positive filter rated it 3/10, and the Opus judge ruled it LOW (an informational key only, no data access). It is hardened anyway: a parent the deploying identity owns wins a tie. Re-runs on the fix diffs found nothing.world-class-inspection (diff mode, delivery review): CONCERNS, then fixed. It raised two P2s, both fixed and re-verified:
{{ env.* }}templates.defect-scan (diff mode, 3 hunters, 2 skeptic batches, referee). Its REAL_BUGs, all fixed and re-verified:
--dry-runamend-and-build declared keys.fluid validatecrashed on a templated location with a job.Also fixed: LOW findings on stale-key rules, the script missing same-contract children, name collisions, own-candidate notes, per-table lookup resilience, hooks after a partly failed build, and literal hook output (
test_hints_print_literally).Re-check rounds: 3. Round 2 fixed a Medium the round-1 fix had introduced: a parent unreadable now dropped its FK. Round 3 fixed a pre-existing Medium: a key moved to a sibling table on a partial view. A cosmetic LOW was fixed in the same pass.
Follow-ups
fluid_keystask, including the nullable-column note and the SQL-scripting requirement.fluid verifycould check the keys live.fluid apply.