Repository navigation
feat(databricks): a dbt job runs its contract's DQ rules as dbt tests - #871
Merged
Merged
Conversation
fluid_build/util/dq_checks.py turns exposes[].contract.dq.rules into engine-neutral checks (not_null, unique, accepted_values, regex, bounds, reference, freshness, schema) and a refusal, with its fix, for each rule no check can run as written. The allowed values of a valid_values rule come from the selected column: its validationRules (enum 'A | B' or 'A,B', regex, range) or labels.references <product>.<column> through consumes[]. Thresholds below 1.0 are shares computed as fluid test computes them; critical/error map to error, warn/info to warn. Shared with the Databricks DQ sessions (monitoring, results table, Lakeflow expectations) as the one reading of the rules.
Each check of the rules on the exposes a Databricks dbt build writes is a singular dbt test, <test path>/fluid_dq/<model>/<check>.sql, uploaded beside the project; the user's files upload byte for byte (dbt describes a model in one properties entry only). dbt build runs each test after its model, so the run page lists every rule by id and an error-severity failure fails the run. not_null, unique, accepted_values and relationships run dbt's built-in test SQL through adapter.dispatch; regex, bounds, freshness, schema and shares are forge-cli's own SQL after dbt_utils (recency, not_null_proportion) and dbt_expectations (match_regex, table_columns_to_match_set), both Apache-2.0, with no package to install on serverless. Every value reaches the file as an escaped Jinja string and every SQL literal through databricks_access.string_literal. A rule no test can run, a Git-sourced build with rules, a model or a referenced source the project lacks are refused at validate and plan. The plan lists each job's tests (dq_tests); report() reads dbt's result lines from a run's output.
A source-aligned product consumes nothing, so a labels.references to another product's column no longer refuses in the shared reader: Reference.expose_id is None and the engine decides whether it can find that table (the Databricks dbt job cannot, and refuses). Two exposes of one consumed product stay refused as ambiguous. Each refusal carries the rule's type, so an engine that runs a type another way (Databricks data quality monitoring for anomaly_detection) can tell its own.
…s report covers rules dbt stopped before The dbt test of a reference reads the other product from the dbt source the project declares for the expose it consumes, so a reference to a product consumes[] does not list is refused there. report() says once when the task's output holds no result of any rule's test (dbt stopped before them). The job's dbt task key is a constant, DBT_TASK_KEY, which a run's report finds the dbt output under.
dbt records a test's parents while it parses, with execute false, and dbt build --select +model+ selects tests by them. The schema test called ref() only inside its execute guard, so it had no parent and was never selected: live on Databricks Free Edition dbt found 11 tests and ran 10. The relation is now set before the guard. Every kind is pinned to ref its model at parse time.
…t tests The Unreleased entry says what runs for each rule type, what is refused, and what fluid apply prints. HONESTLY_TESTED records the unit tiers and the manual live run on Databricks Free Edition (2026-11-12): a seeded bad row failed the run on its two error rules and warned on two, a clean run passed all 11 tests, the next plan was empty, and everything was swept; it also lists what was not run live.
… two hardening fixes From the branch's security review and ripwire --quality-delta: - The schema test escapes the expected type as it escapes the type the warehouse reports. It is unreachable today (every Databricks column type passes _column_type first), but no longer depends on it (LOW). - A reference refusal quotes the labels.references parts it names (LOW). - compile_tests runs in stages (a _Project with its paths and lazily read sources, _test_folder, _add_test); _declared_sources reads through _properties_docs and _source_tables; _Rule._allowed reads each validationRule kind in its own method; read_rules and has_rules share _parts; report's wording is _outcome. - The job's tests ride on _Source, so _emit_job keeps its signature and its task dict stays one line.
…r as written DqCheck.rule_severity, rule_threshold and rule_operator carry the rule as the contract wrote it, beside the two-level severity a dbt test runs at and the share a lower threshold becomes: Lakeflow expectations fail an update on critical and drop the row on error, and a results table reports one row per rule.
…that skips the model
From the diff's delivery review:
- A rule's test is named after its id, and dbt names a singular test once
per project: a project test of the same name made the whole job fail to
parse ("dbt found two data_tests with the name"). It is refused at plan.
- A build whose dbt selection provably leaves the model out would never
run its tests: refused, with the model's fqn and path evaluated by
databricks_retention.selects_model. A tag, config or state selection
cannot be told here; the run's report names any test that did not run.
- A real dbt (dbt-core with dbt-duckdb, as CI installs) parses the upload
and dbt ls --select +orders+ lists every rule's test; with the schema
test's ref put back behind its execute guard, it fails.
- Wording: a share is fluid test's own only for completeness and
uniqueness; the anomaly and drift refusal says what fluid test does
with them; dq_checks is the reading for the engines that run the rules,
this job its first.
…t rule after a run Under --mode amend-and-build, once the job's run ends (success or failure), the run report reads the dbt task's output and prints one line per contract rule's test: pass, warn or fail with its failing-row count, error, skip, or not in the output (with the note when Databricks kept only its end), then the summary. The rules come from the contract alone (databricks_dbt_tests.expected_tests); the task is found by DBT_TASK_KEY, so the fluid_keys task beside it is never read as dbt's. runs/get-output is read once per task and feeds the dbt summary, the failed task's tail and the rules alike (_task_output). A rule report is printed for the dbt task when a later task failed too: its tests ran. Never fails a build: an unreadable output or contract only drops the report.
…alue meets From the diff's defect scan: an integer threshold of hundreds of digits raised OverflowError out of fluid validate and plan, and so did a window too long to count; an infinite or NaN threshold, or a range bound past a float's range, reached the SQL as a bare 'inf'. Each is now a refusal (_finite, and parse_range drops a non-finite bound). A range no value can meet (between 100 and 0, > 5 and < 1, > 3 and <= 3) is refused instead of failing every row of its test.
- Only the exposes with rules are tested, so only their models are looked
for: a sibling output with no rules (no model of its name, or left out
of the selection) was refused.
- properties.model names the model of the one table a build writes; a
build that writes several (no outputs writes every expose) tests each
through the model named after its table.
- fluid validate, which reads the contract before {{ env.* }} is
substituted, no longer refuses a model named by a placeholder; the
apply writes those tests.
- The schema test spells a decimal as the table emitter does
(_canonical_type): decimal(12) is decimal(12,0), as Databricks reports.
- _jinja_str keeps characters outside the BMP whole: ensure_ascii wrote
a surrogate pair, which dbt cannot encode (one emoji in a rule id
failed the whole project's parse).
- The test-name clash check also covers the project's own files under
fluid_dq/, and a project YAML nested past the recursion limit is passed
over like any file that does not parse.
- The fluid_keys task (#860) runs with run_if ALL_DONE: a failing rule
fails dbt after its model replaced the table, which dropped its keys.
…ir report AGENTS.md says the job runs the contract's DQ rules as dbt tests and where the reader and the renderer live; the changelog says the keys task now runs whatever dbt's outcome; HONESTLY_TESTED counts the tests again (48 + 54 + 4) and lists what the review waves added.
…onitoring #864 turns Databricks data quality monitoring on for an expose with an anomaly_detection rule, and the monitor learns the rule's bounds, which a dbt test cannot. The job's tests therefore leave such a rule to it (databricks_quality.monitoring_request), as agreed with that change. An expose that turns monitoring off (contract.dq.monitoring.enabled: false) still has its anomaly rule refused, and the fix names the opt-out. drift_detection stays refused.
…n waits for the apply
From the fix wave's re-check:
- Since the Jinja strings keep raw Unicode, an unpaired surrogate (YAML's
"\ud83d" escape loads one) in a rule id, an enum value or a regex
raised UnicodeEncodeError out of fluid plan. The reader refuses one with
the control characters, in ids too, and a value from the project's own
YAML (a source name) that UTF-8 cannot hold is a refusal of its own.
- fluid validate, before {{ env.* }} is substituted, refused a build whose
properties.select still held a placeholder as leaving the model out; the
selection is judged only once it can be read, at the apply.
… in the changelog and the live row
…s over only the history refusal From the round-2 re-check: - An unpaired surrogate in dbt_project.yml's first test path (YAML's "\ud83d" escape loads one) reached the generated tests' paths and raised UnicodeEncodeError out of fluid plan. It is a test-path refusal now, and the message names the path escaped. - On an expose Databricks data quality monitoring watches, every refusal of an anomaly_detection rule was dropped, so a bad severity or an id holding a control character passed silently. DqRefusal.needs_history marks the one refusal monitoring takes over (the rule needs the table's history); the others are refused as before.
… escaped The round-3 re-check found model-paths reaching the model refusal's text raw, and a lone surrogate there (a YAML escape loads one) made fluid plan fail on printing the refusal. Every DQ refusal is built in _refusal, which now spells such a character as its backslash escape; a source or table name from the project's YAML in a reference refusal is covered the same way.
… escaped refusal text
#867 landed fluid test's reading of a valid_values rule, providers/quality_engine.py::value_domain: the values the rule itself lists, else every declaration its column makes (the first enum, regex and range validationRule, and labels.references), all of which must hold. dq_checks had its own, which let the column's declarations win over the rule's list, split an enum on | only when it held one, and read every validationRule of a kind; the dbt tests could then check other values than fluid test does. dq_checks now takes the values, pattern and bounds from value_domain and keeps its refusals on top: a range fluid test cannot read (it passes one over) or no value can meet, a value or pattern holding a control character, and a labels.references that consumes[] cannot resolve. Its own parse_enum and parse_range are gone; the range grammar is quality_engine's. The Tower Management pack still reads as 330 checks with none refused, check for check as before, and the demo's dbt products as 230 tests.
…md, the changelog and the live row
github-actions
Bot
added
security
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
fas89
enabled auto-merge (squash)
November 13, 2026 00:02
fas89
disabled auto-merge
November 13, 2026 00:06
github-actions
Bot
removed
the
needs-docs
fas89
enabled auto-merge (squash)
November 13, 2026 00:09
fas89
disabled auto-merge
November 13, 2026 00:16
fas89
enabled auto-merge (squash)
November 13, 2026 00:22
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On the Databricks target, a dbt build's contract data-quality rules (
exposes[].contract.dq.rules) now run as dbt tests inside the product's Databricks job. The job run page lists every rule by its id: passed, warned or failed. A rule of severitycriticalorerrorthat fails also fails the build. Under--mode amend-and-build,fluid applyprints what dbt found for each rule once the run ends.fluid_build/util/dq_checks.py(new) reads the rules once, into engine-neutral checks or refusals, each refusal with its fix. Avalid_valuesrule allows whatfluid testallows. It reads through feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard #867'squality_engine.value_domain: the values the rule's description lists come first. Otherwise every declaration of the selected column must hold: itsvalidationRules(enumasA | BorA,B,regex,range) and itslabels.references: <product id>.<column>, resolved throughconsumes[](the same convention the Unity Catalog keys of feat(databricks): Unity Catalog primary and foreign keys from the contract #860 read). The other Databricks DQ work (monitoring, results table, Lakeflow expectations) shares this module as the one reading of the rules.fluid_build/iac/providers/databricks_dbt_tests.py(new) turns each check into a singular dbt test,<first test path>/fluid_dq/<model>/<rule id>.sql. The test goes into the copy of the project that the job uploads. The user's own files upload byte for byte: dbt describes a model in one properties entry only, and that entry stays the user's.databricks_jobs.pyadds the test files to the WORKSPACE upload. It refuses a Git-sourced build with rules, since no file can be added to a checkout. The plan's job action lists itsdq_tests. Thefluid_keystask (feat(databricks): Unity Catalog primary and foreign keys from the contract #860) now runs withrun_if: ALL_DONE. A failing rule fails dbt after its model has replaced the table, and the task still declares the keys again.build_runners/_databricks_job_run.py(feat(databricks): amend-and-build runs the dbt job now and waits for it #861): once the run ends, the run report reads the dbt task's output once. It prints one line per rule's test: pass, warn or fail with the failing-row count, error, skip, or not in the output. The rules come from the contract alone, and the task is found byDBT_TASK_KEY. An unreadable output never fails the build.unique(one column; a list is refused, asfluid testreads one)not_nullon each selected column (a,b,cis one test per column)accepted_values. Otherwise each declaration of the column: anenumrunsaccepted_values; aregexruns forge-cli's regex test; arangeruns a bounds test;labels.referencesruns dbt'srelationshipsagainst the dbt source table the project declares for that consumed expose<operator> <threshold>(the boundfluid testchecks)windowSeverity:
criticalanderrorbecome dbterror;warnandinfobecome dbtwarn. Each file sets its severity inconfig(), which wins overdbt_project.yml. A threshold of 1.0, or no threshold, means every row must pass. A threshold below 1.0 is the share of rows that must pass, computed asfluid testcomputes it (providers/quality_engine.py). The built-in tests run dbt's own SQL throughadapter.dispatch. Regex, bounds, freshness, schema and share tests are forge-cli's own SQL, so no dbt package is installed on serverless.Founder decisions (asked once): tests as files in the uploaded copy; forge-cli's own tests, no dbt packages; shares below 1.0.
fluid validateandfluid planrefuse these cases, each with its fix:valid_valueswith no allowed values, an unknown column, an unparseable window, a non-temporal freshness column, a threshold no float holds, a range no value meets, or an unpaired surrogateBorrowed
Searched: dbt's duplicate-patch rule;
data_testsnames andarguments;severityanderror_if; rate tests; dbt_expectations on Spark; Databricks Free Edition egress; dbt task artifacts;run_results.json.Prior art: dbt-core's built-in generic tests (
default__test_unique,not_null,accepted_values,relationships); dbt_utilsrecencyandnot_null_proportion; dbt_expectationsexpect_column_values_to_match_regexandexpect_table_columns_to_match_set. Both packages are Apache-2.0. Their patterns are adapted, not depended on, because Free Edition limits serverless egress to trusted domains. dbt_expectations also calls its Spark support experimental and pulls in dbt_date.Reused in-repo:
quality_engine's ratios andparse_duration_seconds;databricks_access.string_literalandquote;engines/dbt/_types.sql_type.Validation
tests/util/test_dq_checks.py(51)tests/iac/test_iac_databricks_dbt_tests.py(65): every kind rendered in a dbt-shaped Jinja context at parse and at execute, and hostile values stay inside their literal. A real dbt-core with dbt-duckdb parses the upload, anddbt ls --select +orders+lists every rule's test.tests/build_runners/test_databricks_job_run.py(+4, the report hook)dbt parseanddbt ls --select +active_tenancy+(dbt-core 1.12.5) select all 20 of DP005's tests.fluid_iactest_dq(swept after, 0 objects left): one dbt product, 9 rules as 11 tests, run withfluid apply --mode amend-and-build.RUN_EXECUTION_ERROR. The two error rules showedFAIL 1, two warn rulesWARN 1, six testsPASS, all printed byfluid apply. The run also showed that the schema test was never selected: itsrefsat behindif execute. Fixed infix(databricks): the schema test refs its model while dbt parses.SUCCESS, all 11 testsPASS. The next plan was+0 ~0 -0.b633329is Cache copilot agent system prompt (rebuilt per iteration despite module-level constant) #6's merge, which makes the generator write chore: pre-release fixes — build-manifest version + test deselections #4's fixes. After the rebase on feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard #867 and thevalue_domainalignment, the pack reads check for check as before, and the demo compiles to the same 230 tests.tests/iac -k databricks(1321 passed after the rebase on feat(databricks): Unity Catalog metric views from exposes[].semantics #865 and feat(databricks): one Genie space across deploying identities (parentPath); plan, diff and generate iac show it #869).value_domainalignment before the last rebase: 35502 passed, 9 failed, 2 errors. Cleanmain(d1749326) has the same 9 failures and 2 errors in this machine's shared.venv: dlt credential allow-lists ×4, the DuckDB sandbox root ×2, lazy core-command imports ×2, a Snowflake-login test that reads the machine's own AWS credentials under load ×1, and the bashlex import ×2.Reviews
/ship ran three review rounds, each followed by a re-check of its fixes:
dbt ls)fluid_dq/run_ifThe last re-check found nothing new at Medium or above.
After rebasing on #867, which landed
fluid test's own reading of avalid_valuesrule (quality_engine.value_domain),dq_checksreads through it. Before, the column's declarations won over the rule's list, an enum was split on|only when it held one, and every validationRule of a kind was read. The brief asked for one rule translation across the DQ work. An independent verifier compared the two readings over 764 rule and column cases (the test fixture and all 19 Tower Management contracts) and found no disagreement. Its 14 edge cases behaved as described, and 427 tests pass.Pre-existing, not changed here: a lone surrogate in the contract
id, a buildidorrepositoryraises iniac/naming.py, with or without rules. Adbt_project.ymlnested past the recursion limit raises in_project_files.Follow-ups:
validValues(validValues: 5, which the schema forbids) raises inquality_engine.extract_valid_values, influid testand here alike; the fix belongs there.engines/dbt/_test_mapping.py) still has its own reading ofvalid_values.Documentation
Docs PR linked:
No docs needed
Docs TODO: forge_docs
docs/providers/databricks.mdneeds a "Contract rules as dbt tests" part in its dbt job section. It should cover:<test path>/fluid_dq/<model>/<rule id>.sql), and that the project's own files upload unchanged;criticalanderrorfail the run,warnandinfodo not), and a threshold below 1.0 as a share;valid_valuesrule allows (asfluid testreads it), andanomaly_detectiongoing to data quality monitoring;fluid validateandfluid planrefuse, each with its fix;fluid apply --mode amend-and-buildprints for each rule after the run.No draft yet, so the docs PR may land after this one. Until then, the module docstrings, CHANGELOG and HONESTLY_TESTED.md describe the behaviour.