[fas89]

A Databricks product can now be built as a Lakeflow Declarative Pipeline (formerly Delta Live Tables), with the contract's data-quality rules as the pipeline's expectations. The pipeline graph's Data quality tab then shows each rule's passed and failed records, and fluid apply --mode amend-and-build prints the same counts.

What changed

• Contract surface (founder, 2026-11-12): engine: lakeflow joins the 0.7.7 preview schema (stable 0.7.6 is unchanged; a 0.7.6 contract that uses it is told to move to 0.7.7). The build is pattern: embedded-logic with properties.sql, the SELECT of the dataset; upstreams are named by their Unity Catalog names.
• The pipeline (iac/providers/databricks_lakeflow.py): one serverless databricks_pipeline per build, publishing into the expose's catalog and schema, with its source as a workspace file under the deploying identity's home (beside a dbt job's upload). forge-cli writes the source: CREATE OR REFRESH MATERIALIZED VIEW <table> with the expose's columns (types, NOT NULL, comments) and one CONSTRAINT <rule id> EXPECT (...) per row-level rule, then the contract's query, last in the file. A materialized view, not a streaming table: the query reads tables that a dbt CREATE OR REPLACE rewrites.
• Rules → expectations, read once by the shared util/dq_checks.py::read_rules (feat(databricks): a dbt job runs its contract's DQ rules as dbt tests #871), as the dbt tests read them:
  • completeness → col IS NOT NULL; valid_values → the column's validationRules (IN, RLIKE, range comparisons), NULL passing;
  • uniqueness, freshness, and a valid_values rule whose column carries labels.references: <product id>.<column> need the whole table: one PRIVATE materialized view per check in the same pipeline (never published), whose one expectation is named after the rule. A reference resolves the referenced product's table through the workspace (fluid.workspace.yaml, FLUID_UPSTREAM_CONTRACTS), as an embedded-SQL build resolves consumes[], whether or not consumes[] lists that product;
  • schema → the declared column list itself (the update fails when the query does not give those columns and types);
  • refused: accuracy/anomaly/drift with no expression, a rule with no allowed values, a threshold an action cannot carry, two rules with one id (Lakeflow refuses two expectations of one name at the first update).
• Severity (my call, as asked): info and warn → EXPECT (row kept, counted); error → ON VIOLATION DROP ROW (row never published, counted as dropped); critical → ON VIOLATION FAIL UPDATE (update fails, the table's write rolls back). No quarantine table: the rejected rows would sit outside the contract's access policy. A whole-table check has no row of the product to drop, so error only counts there; a critical one fails the update after the table was written (Lakeflow runs a check view after the table it reads). Uniqueness is checked by a private view: SELECT <key>, count(*) ... GROUP BY <key> with EXPECT (_fluid_rows = 1), NULL keys left out as fluid test and dbt's unique leave them out.
• Coexistence (founder): the pipeline owns its table: no databricks_sql_table, its comments are the definition's, its readers get SELECT on the product's schema (exists before the first update; a precondition fails the plan if the schema holds any other table or view), and its table and column tags are set after each update forge-cli runs. A tag the contract no longer declares is removed only when the deploying identity set it (the tag's updated_by), so a steward's tag and an inherited one stay. Refused for now: masks, row filters, write grants, partitions, lifecycle.expire, another writer, a schema shared with another table. Not set up on a pipeline's table yet, each with a fluid validate warning: Unity Catalog keys (a materialized view refuses ALTER TABLE ADD CONSTRAINT), data quality monitoring (feat(databricks): data quality monitoring from a contract's freshness and completeness SLOs #864), its metric view (feat(databricks): Unity Catalog metric views from exposes[].semantics #865) and its place in the Genie space (feat(databricks): a Genie space for a data product (extensions.databricksGenie) #863, feat(databricks): one Genie space across deploying identities (parentPath); plan, diff and generate iac show it #869). The last three are set up at or right after tofu apply, and the pipeline creates its table at its first update, after that; a product whose tables are all a pipeline's gets no Genie space and no refusal.
• Running (founder): fluid apply --mode amend-and-build starts an update of the pipeline this apply's OpenTofu state holds (by its fluid_product / fluid_build tags, as feat(databricks): amend-and-build runs the dbt job now and waits for it #861 finds a dbt job), waits, prints each rule's counts from the update's event log, and fails on a failed update with Lakeflow's error, less the row a failed expectation quotes (replace-and-build: a full refresh). amend and create-only only provision; a continuous pipeline is not started, but its tags are set once its table exists. A triggered pipeline sets pipelines.numUpdateRetryAttempts to 0 (found live: Lakeflow retried a failed update by itself and kept spending compute after the apply had failed).
• fluid verify now compiles a Databricks expose from its contract file, as apply does (found live: verify expected 1 tag on a pipeline table that carries 10).
• One pipeline per product build (founder): measured first on Free Edition, two pipelines exist side by side and a second update starts while the first runs.
• Shared with feat(databricks): amend-and-build runs the dbt job now and waits for it #861's job runner: the state scan, the build selection, the Ctrl-C/SIGTERM guard and the two "nothing was started" error paths are one implementation in build_runners/_databricks_job_run.py, used by both runners (ripwire flagged the twins; the job runner prints what it printed before and its tests pass unchanged).

Industry pack

Under the industry-repo rule: Agenticstiger/industry-models#4 (merged, 8d82512b) made the pack's rules readable as written: the 57 enum constraints were A | B | C (forge-cli reads comma lists, as FLUID's own example writes them), and dp009's smart_meters_nonnegative_energy named five columns with no allowed values (now range >= 0 validationRules). The 24 *_ref rules stay as written: their columns carry labels.references. (A sibling session fixed the pack's generator to write the same form in industry-models#6.)

Borrow-before-build

• Checked locally: engines/dbt/_test_mapping.py and providers/quality_engine.py (rule readers), databricks_jobs.py (workspace upload and run seam), build_runners/_embedded_sql_io.py (consumes resolution), build_runners/dlt/ (dlthub ingestion, not Lakeflow: nothing to reuse). The shared reader util/dq_checks.py comes from the dbt-tests session; this branch uses it.
• Searched: dlt-meta/sdp-meta (metadata-driven DLT with DQE JSON and quarantine; license not Apache), databrickslabs/dqx v0.17.2 (ODCS → DQX checks: the mapping table followed here), datacontract-cli v1.2.9 (DQX exporter; no DLT exporter), Lakeflow's own expectation patterns (validation views for keys and references), the provider's databricks_pipeline docs, the event-log docs. No project emits Lakeflow expectations from a data contract.
• Reuse: adapt the pattern (DQX mapping, Lakeflow validation views, the dbt job runner's polling and interrupt handling, now shared); no new dependency.

Validation

• Gates, with CI's own commands on the branch head: ruff check fluid_build/ tests/, the dead-code gate, black==24.10.0 --check, the product guardrail (self-test, then the run), license headers, lint-imports (6 + 1 contracts kept), detect-secrets-hook 1.5.0 on the changed files (clean after a line-number refresh of the baseline), and scripts/mypy_strict.py.
• Tests: the Databricks, runner, verify and tofu validate suites: 1651 passed, 24 skipped on the head (one stand-in-workspace test lost its connection to the stand-in server under load and passes alone). The fake-workspace e2e files pass 59/59 at -n 2; two of them flaked under -n 6 load and pass alone.
• Full suite, CI's way (-n 6 --dist loadscope --ignore=tests/perf, tofu off PATH, pytest-randomly seed 20261112), then tests/perf serially: 35753 passed on the head (35b9490). The 9 failures and 2 collection errors are the shared venv's known baselines (lazy-core ×2, dlt dotenv ×4, the two tests/test_ci_* ImportErrors), two that fail the same way on pristine main (a test that needs dbt on PATH, and the v0.24.0 compat gate, which CI skips), and one load flake (a Snowflake MCP test whose local stand-in refused a connection; it passes alone 3/3). tests/perf: 41 passed.
• ripwire --quality-delta (the branch diff applied to a fresh origin/main worktree). What stays:
  • run_builds_from_args complexity 110 → 114: the lakeflow branch shares the dbt job branch.
  • verify_databricks_expose 16 → 19: the pipeline's "created by its first update" wording.
  • Test-idiom clones with unrelated tests, and short-horizon churn from the parallel Databricks PRs.
  • The shape of run_job_build / run_pipeline_build, and rows inside feat(databricks): a dbt job runs its contract's DQ rules as dbt tests #871's dq_checks.py.
• Live, Free Edition (manual; scratch catalog fluid_iactest_ldp, dropped after each run; nothing left). The 2026-11-12 row in HONESTLY_TESTED has the first run: counts per rule as promised, tags, verify, a critical failure, and the no-retry setting. 2026-11-13, rebased on feat(databricks): data quality monitoring from a contract's freshness and completeness SLOs #864 and feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard #867:
  • The plan held no data quality monitor for a product with a completeness rule, and the views-and-tables precondition passed.
  • The update completed in 54 s with the same counts; verify passed all 10 tags.
  • After site_id's sensitivity was removed from the contract, the second apply (plan +0 ~0 -0) removed the column's tag, which the deploying identity had set, and kept the 8 table tags.
  • On a scratch schema, through the real entity-tag-assignments API, updated_by was the SCIM userName the build phase compares it with.

Reviews

One gate round; each review's findings were fixed and re-checked.

• /security-review (on the rebased branch diff; an Opus reviewer, then an Opus false-positive filter per candidate): no HIGH or MEDIUM finding. One candidate: a failed critical expectation's error quotes the row that broke it ("Input data: '{...}'. Output record: '{...}'"), and fluid apply printed it to the terminal and CI log. Filtered to LOW, 5/10 (a contract author controls the query and its grants anyway, and masked tables are refused), below the report's cutoff; fixed anyway (ca082e7f). A follow-up pass over the commits after it found nothing. An earlier pass, before the rebase, found two issues below the cutoff, both fixed: a query ending in an unclosed /* swallowed forge-written text up to a */ in a rule description, so the rest ran as a second statement (the query now comes last in the file, and an open comment, string or backtick name is refused, lexed as Spark lexes it); readers' schema-level SELECT covered other teams' tables in an adopted schema (a precondition now fails the plan unless the schema holds only the pipeline's table).
• /world-class-inspection (Diff mode, delivery review): no P0 or P1. Four P2, all fixed: a dbt build naming the expose beside a lakeflow build crashed the compile with a KeyError (now refused as another writer); the tag sync read a column the table no longer had and failed a completed update; fluid verify told a pipeline's table to re-run plain fluid apply for its tags; the data-loss gate's count of a pipeline's removal had no test.
• /defect-scan (Diff mode): two Medium, fixed: two rules with one id gave the pipeline two expectations of one name, which Lakeflow refuses at the first update; the tag keys forge-cli managed were recorded in TBLPROPERTIES, and a refresh of a changed definition dropped that record, so a label the contract stopped declaring stayed on the table forever (the record is gone; removal goes by the tag's updated_by, live-checked). Seven Low, fixed (a continuous pipeline's tags never set; Ctrl-C during the start request left an update running; raw strings lexed with escapes; malformed event payloads raised; a rule's own severity ignored; failing warn rules marked as errors; the precondition missing views; verify's wording for a missing pipeline table; the USE CATALOG notice missing a pipeline schema's grant). The re-check (one auditor on the findings and the fix diff) found both Medium fixed and no new defect, plus one Low, fixed: the tag sync read only the first page of a tag list.
• At the rebase onto feat(databricks): data quality monitoring from a contract's freshness and completeness SLOs #864: tofu validate failed for a lakeflow build with a completeness rule (the data quality monitor named a databricks_sql_table that a pipeline's table does not have); fixed with a test that fails without it.

Notes for the merge

• Rebased onto main after feat(databricks): amend-and-build runs the dbt job now and waits for it #861 and feat(databricks): data quality monitoring from a contract's freshness and completeness SLOs #864–feat(databricks): fluid genie apply, one Genie space over several data products #870; feat(databricks): a dbt job runs its contract's DQ rules as dbt tests #871 is merged in (a merge commit: the branch was already pushed, and its tree is the rebased tree the suites ran on). Two of them met this branch: feat(databricks): data quality monitoring from a contract's freshness and completeness SLOs #864's data quality monitor depended on a databricks_sql_table that a pipeline's table does not have (tofu validate: "Reference to undeclared resource"), and feat(databricks): Unity Catalog metric views from exposes[].semantics #865's metric views and the Genie space are set up before a pipeline creates its table. Both fixed here, with tests that fail without the fix.
• fluid_build/util/dq_checks.py came with feat(databricks): a dbt job runs its contract's DQ rules as dbt tests #871: this PR no longer changes it (its earlier commits here were snapshots of that branch).
• providers/quality_engine.py (feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard #867, fluid test) and util/dq_checks.py now read a valid_values rule the same way (feat(databricks): a dbt job runs its contract's DQ rules as dbt tests #871 reads it through quality_engine.value_domain).

Follow-ups (not in this PR)

• Data quality monitoring, the metric view and the Genie space on a pipeline's table: set them up after the pipeline's first update (and check live whether the schema-level monitoring scan covers a materialized view).
• Unity Catalog keys in the pipeline's definition (Lakeflow takes PRIMARY KEY / FOREIGN KEY constraints there), and masks and row filters in the definition.
• A stand-in-workspace (_fake_databricks.py) tier for pipelines; a cron trigger; quarantine as an opt-in.
• Spark errors other than expectation violations (a cast error, say) can still name one offending value when printed; the dbt job runner prints them the same way, so it is one change for both.
• fluid genie apply (feat(databricks): fluid genie apply, one Genie space over several data products #870) puts a pipeline's table in an estate space (it runs after the products are built), but shares the space (CAN_RUN) only with principals holding a table grant, and a pipeline's readers hold SELECT on its schema: a reader of only a pipeline's table is not shared the estate space yet.
• A Genie space left as it is (a product of pipeline tables only) keeps its last CAN_RUN shares until a later apply converges it: metadata only, since Genie runs questions on the asker's own grants. A converted table's old metric view is not dropped.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.