Repository navigation
docs(databricks): Lakeflow pipelines, a build whose expectations are the contract's DQ rules (forge-cli#872) - #213
Merged
Merged
Conversation
…the contract's DQ rules (forge-cli#872) A build with engine: lakeflow (schema 0.7.7) runs as a Lakeflow Declarative Pipeline that owns its table, and the contract's dq rules become its expectations. The new section on the Databricks page covers opting in, how each rule type and severity maps, what fluid apply --mode amend-and-build prints, the tag sync, what is refused or not set up next to a pipeline's table, fluid verify, and a two-product example taken from the live runs. Plus one marked line each on apply, builds/exposes/bindings, providers, operating in CI and environment variables.
…able gets none yet Main gained the Metric views section (#211) while this draft waited for review. Its intro said every expose with semantics gets a view. A table a Lakeflow pipeline publishes gets none yet, and fluid validate warns ("its semantics get no metric view yet", checked on 0.30.1). The Lakeflow list now links that section instead of forge-cli #865.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Documents forge-cli #872: a build with
engine: lakeflow(the 0.7.7 preview schema) runs as a Lakeflow Declarative Pipeline that owns its table, and the contract'sdq.rulesbecome its expectations. It shipped in forge-cli 0.30.0.docs/providers/databricks.mdgets a "Lakeflow pipelines" section, before "Landing a DuckDB acquisition":fluid applyprovisions, with the pipeline source forge-cli generates for the worked example;fluid apply: each mode, a liveamend-and-buildcapture, the per-rule marks, a failed update, how long it waits, dry-run, and the tag sync on the pipeline's table;fluid validatewarning;fluid verifyon a pipeline's table;Plus one line each in that page's overview, prerequisites, provisions table, validate list, verify pointer, Limitations and "How it is tested", and one in the Metric views intro (a pipeline's table gets no view yet). The dbt run-now section's quote of
apply_databricks_sdk_missingnow has the text #872 gave it ("a dbt or lakeflow build ...").Elsewhere, one marked sentence each in
cli/apply.md(the build phase, the SDK error row),concepts/builds-exposes-bindings.md,providers/README.md,advanced/operating-in-ci.mdandadvanced/environment-variables.md(FLUID_DATABRICKS_RUN_TIMEOUT_SECONDSalso caps a pipeline update).Status
0.30.0 contains #872 (
d2ddd7ce), and 0.30.1 changes nothing this documents, so every marker is the release form,*(since 0.30.0, [forge-cli #872](<url>))*.The draft was written against forge-cli
84da9dde(the commitv0.30.0was cut from), then rebased onto main after #211 and #212. The one conflict was the end of "How it is tested", where #211 added two paragraphs. Both stay, followed by this PR's not-proven line. The second commit links the new Metric views section from the Lakeflow list, and says in that section that a pipeline's table gets no view.Verification
fluid generate iacoutput._report_qualityand_errorscode, and says so.fluid validateandfluid generate iacoutput (the 0.7.6 refusal, the SLO and semantics warnings, the cron refusal) was re-run on 0.30.1 (9980ac6d) and is unchanged. #876's notes channel only moved the dbt-comments message.scripts/check_cli_docs.py(full,--version-only, and--markers-onlyagainst forge-cli main's CHANGELOG, newest release 0.30.1),scripts/check_providers.pyandscripts/gen_contract_reference.py --check, with the pinned CLI 0.29.0 on Python 3.11: all pass.npm ci && npm run docs:buildandnode scripts/check-dist-links.mjs: pass. All 22#anchorlinks the change adds resolve to heading ids in the built HTML. lychee offline over the source, with CI's route mirror: 0 errors, canaries 0 and 2.84da9dde.replace-and-build, a continuous pipeline, a table that exists outside its pipeline, a grant to a real group, a steward's tag, a tag list longer than one page, a failingcriticalrule with today's output, and the Data quality tab itself (the counts are the event log's).Left alone
docs/upgrading.md: the engine is opt-in (a 0.7.7 contract withengine: lakeflow), so nothing changes for an existing contract, and #872's CHANGELOG lines are all under Added.fluid generate iacemits a module for a 0.7.6 contract withengine: lakeflow, because it does not schema-validate;fluid validateandfluid applyrefuse it. That is a forge-cli follow-up, not a docs one.