Repository navigation
feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard - #867
Merged
Merged
Conversation
…n Unity Catalog
fluid test answered "Live validation against Unity Catalog is not
available" for platform: databricks. It now reads the table's columns from
the Unity Catalog tables API (no compute) and runs the row count and every
dq rule on DATABRICKS_WAREHOUSE_ID (providers/databricks_validation.py),
through the existing databricks_enforce.SqlRunner, which gains :name
parameters. A table that does not exist runs no statement.
The native engine runs what a contract already declares: a valid_values
rule's values from its column's enum, regex or range validationRule or its
labels.references (resolved on Databricks by the fluid_product_id tag,
the tested table's catalog first); comma-listed columns for completeness,
accuracy and valid_values; schema rules from the live column comparison.
Every rule's result, passing ones included, carries what was measured.
Fixed in passing: columns were quoted "col" on BigQuery and Databricks,
where that is a string literal, and string literals doubled their quotes,
which Databricks reads as adjacent literals ('it''s' becomes its). New
_sql_safety.quote_string_literal_for(dialect, value).
--results-table CATALOG.SCHEMA.TABLE appends one row per check of the run, the same cases --output junit reports (cli/_test_cases.py), to a Unity Catalog table created if missing in a schema the operator already has; the catalog and schema are never created, the product's own schema is refused, and a table without the result columns is never written. Every value is a bound parameter. --results-dashboard WORKSPACE_PATH keeps a Lakeview dashboard over the table (pass rate of the latest runs, pass rate per product by layer, failing checks, daily trend), found by its workspace path so no state file is needed, updated only when its definition differs, and published without embedded credentials. The row shape follows Databricks Labs DQX's summary-metrics table and Elementary's test results; the path lookup follows databricks-solutions/ai-dev-kit; widget shapes follow DQX's .lvdash.json. No code is taken from them (DQX is under the Databricks License).
From the branch's security, inspection and defect-scan reviews: - A schema rule on BigQuery no longer passes when the table was never read: the BigQuery check now says whether it compared, as the generic one does. - Two rules sharing an id: each outcome pairs with its own result, so one rule's pass cannot mask the other's failure. - A quality error no rule case carries (an expose whose rules could not run) keeps its JUnit and results-table case. - valid_values checks every declaration its column makes (enum, regex and range in one statement, and a reference besides them), not only the first. - An accuracy rule with no threshold is reported as not run; it passed whenever the column held a value. - Databricks schemas and row counts are always read live, never from the hour-long validation cache. - A run is appended in one INSERT, its rows bound as one JSON parameter, so a failure never leaves half a run that the dashboard would take as the latest. - The dashboard's latest run is the latest one that tested rules, and a draft a failed publish left behind is published on the next run. - --results-table refuses --no-data, and refuses the product's own schema before any check runs; a missing databricks-sdk names the extra to install. - A warehouse that stops answering stops the remaining statements instead of each waiting out its timeout; a failed table lookup is reported once. - The ambiguous-reference message no longer names other tables: it is written to the shared results table. - --engine soda renders the same column declarations as the native engine (valid values, valid regex, valid min / max, one check per listed column); a reference or a strict bound stays unmapped.
- The early product-schema check reads the contract through fluid_build.loader.load_contract, as the no-raw-readers rule asks, and a malformed `exposes` is left to the validator instead of raising. - The dashboard's publish-time comparison reads timestamps with any fraction length (Python 3.10's fromisoformat takes only 3 or 6 digits). - A range bound too large for a float is no bound (it reached SQL as inf). - The SodaCL reason for a bound with no threshold no longer says the native engine passes it. - HONESTLY_TESTED: the reference count is the 23 that ran on deployed tables.
github-actions
Bot
added
provider
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
github-actions
Bot
removed
the
needs-docs
1 of 3 tasks
fas89
added a commit
that referenced
this pull request
Nov 12, 2026
#867 landed fluid test's reading of a valid_values rule, providers/quality_engine.py::value_domain: the values the rule itself lists, else every declaration its column makes (the first enum, regex and range validationRule, and labels.references), all of which must hold. dq_checks had its own, which let the column's declarations win over the rule's list, split an enum on | only when it held one, and read every validationRule of a kind; the dbt tests could then check other values than fluid test does. dq_checks now takes the values, pattern and bounds from value_domain and keeps its refusals on top: a range fluid test cannot read (it passes one over) or no value can meet, a value or pattern holding a control character, and a labels.references that consumes[] cannot resolve. Its own parse_enum and parse_range are gone; the range grammar is quality_engine's. The Tower Management pack still reads as 330 checks with none refused, check for check as before, and the demo's dbt products as 230 tests.
This was referenced Nov 12, 2026
fas89
added a commit
that referenced
this pull request
Nov 13, 2026
…#871) * feat(dq): read a contract's DQ rules once, as checks or refusals fluid_build/util/dq_checks.py turns exposes[].contract.dq.rules into engine-neutral checks (not_null, unique, accepted_values, regex, bounds, reference, freshness, schema) and a refusal, with its fix, for each rule no check can run as written. The allowed values of a valid_values rule come from the selected column: its validationRules (enum 'A | B' or 'A,B', regex, range) or labels.references <product>.<column> through consumes[]. Thresholds below 1.0 are shares computed as fluid test computes them; critical/error map to error, warn/info to warn. Shared with the Databricks DQ sessions (monitoring, results table, Lakeflow expectations) as the one reading of the rules. * feat(databricks): a dbt job runs its contract's DQ rules as dbt tests Each check of the rules on the exposes a Databricks dbt build writes is a singular dbt test, <test path>/fluid_dq/<model>/<check>.sql, uploaded beside the project; the user's files upload byte for byte (dbt describes a model in one properties entry only). dbt build runs each test after its model, so the run page lists every rule by id and an error-severity failure fails the run. not_null, unique, accepted_values and relationships run dbt's built-in test SQL through adapter.dispatch; regex, bounds, freshness, schema and shares are forge-cli's own SQL after dbt_utils (recency, not_null_proportion) and dbt_expectations (match_regex, table_columns_to_match_set), both Apache-2.0, with no package to install on serverless. Every value reaches the file as an escaped Jinja string and every SQL literal through databricks_access.string_literal. A rule no test can run, a Git-sourced build with rules, a model or a referenced source the project lacks are refused at validate and plan. The plan lists each job's tests (dq_tests); report() reads dbt's result lines from a run's output. * feat(dq): a reference to a product not consumed keeps no expose A source-aligned product consumes nothing, so a labels.references to another product's column no longer refuses in the shared reader: Reference.expose_id is None and the engine decides whether it can find that table (the Databricks dbt job cannot, and refuses). Two exposes of one consumed product stay refused as ambiguous. Each refusal carries the rule's type, so an engine that runs a type another way (Databricks data quality monitoring for anomaly_detection) can tell its own. * feat(databricks): the dbt job refuses an unconsumed reference; a run's report covers rules dbt stopped before The dbt test of a reference reads the other product from the dbt source the project declares for the expose it consumes, so a reference to a product consumes[] does not list is refused there. report() says once when the task's output holds no result of any rule's test (dbt stopped before them). The job's dbt task key is a constant, DBT_TASK_KEY, which a run's report finds the dbt output under. * fix(databricks): the schema test refs its model while dbt parses dbt records a test's parents while it parses, with execute false, and dbt build --select +model+ selects tests by them. The schema test called ref() only inside its execute guard, so it had no parent and was never selected: live on Databricks Free Edition dbt found 11 tests and ran 10. The relation is now set before the guard. Every kind is pinned to ref its model at parse time. * docs(databricks): changelog entry and the live row for DQ rules as dbt tests The Unreleased entry says what runs for each rule type, what is refused, and what fluid apply prints. HONESTLY_TESTED records the unit tiers and the manual live run on Databricks Free Edition (2026-11-12): a seeded bad row failed the run on its two error rules and warned on two, a clean run passed all 11 tests, the next plan was empty, and everything was swept; it also lists what was not run live. * refactor(dq): smaller units in the rule reader and the test compiler; two hardening fixes From the branch's security review and ripwire --quality-delta: - The schema test escapes the expected type as it escapes the type the warehouse reports. It is unreachable today (every Databricks column type passes _column_type first), but no longer depends on it (LOW). - A reference refusal quotes the labels.references parts it names (LOW). - compile_tests runs in stages (a _Project with its paths and lazily read sources, _test_folder, _add_test); _declared_sources reads through _properties_docs and _source_tables; _Rule._allowed reads each validationRule kind in its own method; read_rules and has_rules share _parts; report's wording is _outcome. - The job's tests ride on _Source, so _emit_job keeps its signature and its task dict stays one line. * feat(dq): each check keeps its rule's severity, threshold and operator as written DqCheck.rule_severity, rule_threshold and rule_operator carry the rule as the contract wrote it, beside the two-level severity a dbt test runs at and the share a lower threshold becomes: Lakeflow expectations fail an update on critical and drop the row on error, and a results table reports one row per rule. * fix(databricks): refuse a test name the project uses and a selection that skips the model From the diff's delivery review: - A rule's test is named after its id, and dbt names a singular test once per project: a project test of the same name made the whole job fail to parse ("dbt found two data_tests with the name"). It is refused at plan. - A build whose dbt selection provably leaves the model out would never run its tests: refused, with the model's fqn and path evaluated by databricks_retention.selects_model. A tag, config or state selection cannot be told here; the run's report names any test that did not run. - A real dbt (dbt-core with dbt-duckdb, as CI installs) parses the upload and dbt ls --select +orders+ lists every rule's test; with the schema test's ref put back behind its execute guard, it fails. - Wording: a share is fluid test's own only for completeness and uniqueness; the anomaly and drift refusal says what fluid test does with them; dq_checks is the reading for the engines that run the rules, this job its first. * feat(databricks): fluid apply reports what dbt found for each contract rule after a run Under --mode amend-and-build, once the job's run ends (success or failure), the run report reads the dbt task's output and prints one line per contract rule's test: pass, warn or fail with its failing-row count, error, skip, or not in the output (with the note when Databricks kept only its end), then the summary. The rules come from the contract alone (databricks_dbt_tests.expected_tests); the task is found by DBT_TASK_KEY, so the fluid_keys task beside it is never read as dbt's. runs/get-output is read once per task and feeds the dbt summary, the failed task's tail and the rules alike (_task_output). A rule report is printed for the dbt task when a later task failed too: its tests ran. Never fails a build: an unreadable output or contract only drops the report. * fix(dq): refuse a threshold or bound no float holds, and a range no value meets From the diff's defect scan: an integer threshold of hundreds of digits raised OverflowError out of fluid validate and plan, and so did a window too long to count; an infinite or NaN threshold, or a range bound past a float's range, reached the SQL as a bare 'inf'. Each is now a refusal (_finite, and parse_range drops a non-finite bound). A range no value can meet (between 100 and 0, > 5 and < 1, > 3 and <= 3) is refused instead of failing every row of its test. * fix(databricks): defect-scan fixes to the DQ tests and the keys task - Only the exposes with rules are tested, so only their models are looked for: a sibling output with no rules (no model of its name, or left out of the selection) was refused. - properties.model names the model of the one table a build writes; a build that writes several (no outputs writes every expose) tests each through the model named after its table. - fluid validate, which reads the contract before {{ env.* }} is substituted, no longer refuses a model named by a placeholder; the apply writes those tests. - The schema test spells a decimal as the table emitter does (_canonical_type): decimal(12) is decimal(12,0), as Databricks reports. - _jinja_str keeps characters outside the BMP whole: ensure_ascii wrote a surrogate pair, which dbt cannot encode (one emoji in a rule id failed the whole project's parse). - The test-name clash check also covers the project's own files under fluid_dq/, and a project YAML nested past the recursion limit is passed over like any file that does not parse. - The fluid_keys task (#860) runs with run_if ALL_DONE: a failing rule fails dbt after its model replaced the table, which dropped its keys. * docs(databricks): AGENTS.md and the live row for the DQ tests and their report AGENTS.md says the job runs the contract's DQ rules as dbt tests and where the reader and the renderer live; the changelog says the keys task now runs whatever dbt's outcome; HONESTLY_TESTED counts the tests again (48 + 54 + 4) and lists what the review waves added. * feat(databricks): an anomaly_detection rule is left to data quality monitoring #864 turns Databricks data quality monitoring on for an expose with an anomaly_detection rule, and the monitor learns the rule's bounds, which a dbt test cannot. The job's tests therefore leave such a rule to it (databricks_quality.monitoring_request), as agreed with that change. An expose that turns monitoring off (contract.dq.monitoring.enabled: false) still has its anomaly rule refused, and the fix names the opt-out. drift_detection stays refused. * fix(databricks): refuse an unpaired surrogate; a placeholder selection waits for the apply From the fix wave's re-check: - Since the Jinja strings keep raw Unicode, an unpaired surrogate (YAML's "\ud83d" escape loads one) in a rule id, an enum value or a regex raised UnicodeEncodeError out of fluid plan. The reader refuses one with the control characters, in ids too, and a value from the project's own YAML (a source name) that UTF-8 cannot hold is a refusal of its own. - fluid validate, before {{ env.* }} is substituted, refused a build whose properties.select still held a placeholder as leaving the model out; the selection is judged only once it can be read, at the apply. * docs(databricks): the anomaly hand-off to monitoring, and the counts, in the changelog and the live row * fix(databricks): refuse a test path no file can hold; monitoring takes over only the history refusal From the round-2 re-check: - An unpaired surrogate in dbt_project.yml's first test path (YAML's "\ud83d" escape loads one) reached the generated tests' paths and raised UnicodeEncodeError out of fluid plan. It is a test-path refusal now, and the message names the path escaped. - On an expose Databricks data quality monitoring watches, every refusal of an anomaly_detection rule was dropped, so a bad severity or an id holding a control character passed silently. DqRefusal.needs_history marks the one refusal monitoring takes over (the rule needs the table's history); the others are refused as before. * fix(databricks): a DQ refusal names a lone surrogate from the project escaped The round-3 re-check found model-paths reaching the model refusal's text raw, and a lone surrogate there (a YAML escape loads one) made fluid plan fail on printing the refusal. Every DQ refusal is built in _refusal, which now spells such a character as its backslash escape; a source or table name from the project's YAML in a reference refusal is covered the same way. * docs(databricks): the live row counts the round-3 tests and names the escaped refusal text * fix(dq): a valid_values rule allows what fluid test allows #867 landed fluid test's reading of a valid_values rule, providers/quality_engine.py::value_domain: the values the rule itself lists, else every declaration its column makes (the first enum, regex and range validationRule, and labels.references), all of which must hold. dq_checks had its own, which let the column's declarations win over the rule's list, split an enum on | only when it held one, and read every validationRule of a kind; the dbt tests could then check other values than fluid test does. dq_checks now takes the values, pattern and bounds from value_domain and keeps its refusals on top: a range fluid test cannot read (it passes one over) or no value can meet, a value or pattern holding a control character, and a labels.references that consumes[] cannot resolve. Its own parse_enum and parse_range are gone; the range grammar is quality_engine's. The Tower Management pack still reads as 330 checks with none refused, check for check as before, and the demo's dbt products as 230 tests. * docs(dq): valid_values reads as fluid test's value_domain, in AGENTS.md, the changelog and the live row * chore(secrets): baseline line numbers after the rebase on #865 and #869 * test(databricks): the schema test's expected rows join their parts with an explicit +
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
fluid teston a Databricks product now runs its checks live, and can keep the results in Unity Catalog with an AI/BI dashboard over them.fluid testused to answer "Live validation against Unity Catalog is not available" forplatform: databricks. It now reads the table's columns from the Unity Catalog tables API (no compute) and runs the row count and everydq.rules[]entry onDATABRICKS_WAREHOUSE_ID, signing in asfluid applyandfluid verifydo. Statements go through the existingdatabricks_enforce.SqlRunner(now with:nameparameters).--results-table CATALOG.SCHEMA.TABLEappends one row per check of the run: each rule, then each structural check, the same cases--output junitreports (which keeps working beside it, so Jenkins still shows the test results). Columns: run_id, run_time, product_id, product_version, layer, expose_id, table_name, rule_id, rule_type, severity, passed, observed_value, threshold, message, fluid_version. Created if missing in a schema the operator already has; forge never creates the catalog or schema, refuses the product's own schema, and never writes into an existing table without these columns. Every value is a bound parameter.--results-dashboard WORKSPACE_PATHkeeps a Lakeview dashboard over that table: pass rate of the latest runs, pass rate per product coloured by layer (Bronze sits beside the dbt-built layers), failing checks, daily trend. Found by its workspace path (no state file), created on first use, PATCHed only when its definition differs, published without embedded credentials.valid_valuesrule's values from its column'senum,regexorrangevalidationRule, or itslabels.references(resolved on Databricks by thefluid_product_idtable tag, same catalog first);completeness,accuracyandvalid_values;schemarules from the live column comparison.On the tower-management pack that is every rule of every deployed product, against 38 of 168 before.
"col", which both read as a string literal, and string literals doubled quotes, which Databricks reads as adjacent literals ('it''s'becomesits, checked on a warehouse). New_sql_safety.quote_string_literal_for(dialect, value).Decisions (founder, 2026-11-12)
Two flags on
fluid test; the table created if missing with the columns above; the dashboard found by workspace path; forge reads what the spec allows. Pack rules: the 24*_refrules run in forge, not removed. Thenonnegative_energyrule was made executable in the pack by industry-models#4 (arangerule per column), which forge now reads; my industry-models#5 was closed in its favour.Prior art (borrow-before-build)
resource_idof a.lvdash.jsonpath), checked against databricks-sdk-py v0.166.0service/dashboards.py.aibi_dashboards/dashboards.py): find an existing dashboard by its workspace path, update or create, then publish. Adopted: no state file, so ephemeral CI agents never duplicate the dashboard.DQX_Dashboard.lvdash.jsongave the widget shapes (counter v2, bar/line v3, table v1 with its full column keys). DQX records the dashboard id in install state instead. DQX and lsql are under the Databricks License, so the patterns are adapted and no code is taken.elementary_test_results: one row per test per invocation, with status and failures.databricks_enforce.SqlRunner(now with:nameparameters),databricks_auth.sdk_config,default_clientandscrub; the emitter's type table (engines/dbt/_types.sql_type) andnormalize_column_type;_sql_safety; thefluid_product_idtag lookup that feat(databricks): Unity Catalog primary and foreign keys from the contract #860 uses for foreign keys.Validation
lint-imports(6 kept), product guardrail and detect-secrets 1.5.0 on changed files are all clean. The touched and dependent suites pass in random order: 1,354 passed.-n 6 --dist loadscope --ignore=tests/perf, tofu hidden,--randomly-seed=20261112): 35,362 passed, 9 failed, 2 errors at b6e8eb9 (12 min); theTestSnowflakeLoginfailure is the known xdist-only flake and passes alone (5 passed). Every failure is in the shared venv's known baseline (dlt dotenv x4, DuckDB sandbox root x2, lazy-import probes x2, 2bashlexcollection errors).fluid test <contract> --env databricks --output junit --results-table ... --results-dashboard ...on the 19 tower-management products innorthwind_towers(read only), with contracts derived from industry-models main. The results table and dashboard lived in a scratch catalog, now dropped.dp019_site_360exits 1: its table is not deployed, so its rules are reported as not run and no statement is sent.warn(the data is hours old against a 15-minute window). Among them are 23 references (3 resolved only by the tag lookup) and chore: pre-release fixes — build-manifest version + test deselections #4's range rule over five columns.Reviews
Size L (
/ship,caveman:lean-build). One fix round plus a pass over the LOW notes.Security review of the branch diff against its merge base: PASS. No finding at confidence 7 or above. An Opus judge confirmed it, including a 200,000-case fuzz of the new literal quoter on Databricks and BigQuery. One LOW was fixed: the ambiguous-reference message named other tables, and that text lands in the shared results table. Security re-check of the fix diff: PASS.
World-class inspection, Diff mode: the delivery review first returned FAIL, with three findings:
schemarule passed on BigQuery when the table was never read;All three are fixed. In the UX walk of the changed CLI surface, every refusal exits 2 with its fix, JUnit is written before the table, and the status lines go to stderr. A missing SDK now names the extra to install.
Defect scan, Diff mode: three hunters (logic, runtime, cross-file contracts). 15 findings were confirmed, 2 High and the rest Medium. The main ones:
--engine sodadisagreed with the native engine.A re-check auditor found all 15 FIXED and nothing new at Medium or above. A pre-existing defect was found in passing:
databricks_enforce.SqlRunnerdoes not cancel a statement after a transport error while polling. It is listed below.Documentation
Docs PR linked
No docs needed
Docs TODO: this changes user-facing behaviour, so forge_docs needs:
docs/cli/test.md:--results-tableand--results-dashboard(the 15 result columns, what each flag refuses, the dashboard found by its workspace path), with a Jenkins example that keeps--output junit; and the live checks on Databricks, which replace the "not available" warning.enum,regexandrangevalidationRules,labels.references, comma-listed columns, andschemarules.The forge_docs PR is a follow-up. If this PR merges first, the docs use the house marker On forge-cli
main, not yet in a release (forge-cli feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard #867).Follow-ups
fluid test --results-table/--results-dashboardand the Databricks live checks (docs live in forge_docs).:nameparameters too (defense in depth: the quoting holds, fuzzed).fluid_product_idtag to equal the contract id, so a contract cannot pointfluid testat an unrelated table whose MIN/MAX would land in the shared results table. Other providers' sinks have the same pattern today.databricks_enforce.SqlRunner: cancel the statement when a poll fails with a transport error (pre-existing).environmentcolumn, so dev and prod runs of one product are told apart on the dashboard. That changes the agreed table schema, so it needs a decision.--results-tablewith--engine soda(refused today, with a clear message).nonnegative_energyrange form).util/dq_checks.pylands.