Skip to content

docs(databricks): contract DQ rules run as dbt tests in the dbt job (forge-cli#871) - #210

Merged
fas89 merged 1 commit into
mainfrom
docs/databricks-dbt-dq-tests-871
Nov 13, 2026
Merged

fas89 merged 1 commit into
mainfrom
docs/databricks-dbt-dq-tests-871

Conversation

@fas89

@fas89 fas89 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What

Documents forge-cli #871: a Databricks dbt job now runs its contract's data-quality rules as dbt tests.

docs/providers/databricks.md, in the dbt jobs section, gets a "Contract rules as dbt tests" part:

  • where the test files go (<test path>/fluid_dq/<model>/<name>.sql), how they are named, and that the project's own files upload unchanged;
  • the dbt test each rule type runs, and why no dbt package is installed;
  • severity (critical and error fail the run, warn and info do not) and a threshold below 1.0 read as a share;
  • what a valid_values rule allows, read as fluid test reads it, and anomaly_detection being left to data quality monitoring;
  • what fluid validate, fluid generate iac and fluid apply refuse, each with its fix (two tables);
  • what fluid apply --mode amend-and-build prints for each rule, with the two live captures from the implementing session (workspace host and ids replaced).

Short pointers go into the overview, "Where the dbt project comes from", "What is refused", "What validate refuses", Limitations and "How it is tested". docs/upgrading.md gets an entry under "Not in a release yet".

Status

#871 merged on 2026-11-13 as 1e20c739a132829446c2872848700ffa889bb3c3, so every added sentence about it uses the main marker: *On forge-cli `main`, not yet in a release (forge-cli #871)*. The pin bump that follows the next forge-cli release turns these into release statements.

Checked against forge-cli d50084b17bab84f83ab50c8ddbbdc4e7d5598985, the last head of the #871 branch. The squash commit has the same tree (git diff d50084b1 1e20c739 is empty), so what was checked is what is on main.

What the CLI does that differs from #871's description

fluid plan does not refuse these rules and does not list the tests, although #871's description and its CHANGELOG entry say it does. On a 0.7.x contract it writes the abstract actions and never reaches the Databricks planner. fluid validate, fluid generate iac and fluid apply refuse. The page says what the CLI does. If fluid plan is changed to refuse, one sentence in "What the rules refuse" changes with it.

Verification

  • A verifier agent (sonnet, high effort) checked about 78 claims in the diff against the CLI on that commit. It ran validate, generate iac, plan and apply --dry-run on about 90 scratch contracts, and a real dbt-core with dbt-duckdb on the generated upload. It found no false statement about the CLI and 11 imprecise ones, all fixed in this PR: the <name> and <model> rules, the range, threshold and operator rows of the refusal table, "one file per rule", the byte limits, two scoping fixes in the upgrading entry, share thresholds, and "when the output can be read".
  • Not re-run: the live runs of 2026-11-12 (the two captures come from the implementing session's transcript, and their format lines were checked against this commit's code), and the Databricks run page in a browser.
  • scripts/check_cli_docs.py (full, --version-only, --markers-only), scripts/check_providers.py and scripts/gen_contract_reference.py --check, with the pinned CLI 0.29.0 on Python 3.11: all pass.
  • npm ci && npm run docs:build && node scripts/check-dist-links.mjs: pass. Every #anchor the diff adds resolves to a heading id in the built page. The source link check (lychee, offline, with CI's route mirror) reports 0 errors.

Left alone

  • The fluid_keys task now runs with run_if: ALL_DONE (#871). Nothing on this site describes that task, so nothing here does.
  • docs/cli/test.md still says a valid_values rule with no list checks nothing. That has been stale since #867; its own docs TODO is open. The new text describes the current reading.
  • The branch markers for #861 elsewhere on these pages still read "feat/databricks-build-run-now branch", and #861 merged on 2026-11-12.

…forge-cli#871)

A "Contract rules as dbt tests" part in the dbt jobs section: where the test
files go and that the project's own files upload unchanged, the dbt test each
rule type runs, severity and thresholds, what a valid_values rule allows,
anomaly_detection left to data quality monitoring, what fluid validate,
generate iac and apply refuse (each with its fix), and what fluid apply prints
for each rule after a run, with two live captures. Short pointers in the
overview, the project upload, what is refused, what validate refuses,
limitations and how it is tested, and an upgrading.md entry.

Checked against forge-cli d50084b1, the last head of #871; the squash commit
1e20c739 has the same tree.
@fas89
fas89 merged commit 99d13bc into main Nov 13, 2026
9 checks passed

This branch was successfully deployed

1 active deployment
github-pages — bf96abcf Deployed Nov 13, 2026 by fas89 via deploy #303
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant