Repository navigation
docs(databricks): data quality monitoring from freshness and completeness SLOs (forge-cli#864) - #208
Merged
Merged
Conversation
…ness SLOs (forge-cli#864) A Databricks expose that states a freshness or completeness SLO now turns on Databricks data quality monitoring for the schema the product owns. Document what turns it on and the opt-out, the monitor the apply creates, why the SLO values are not thresholds, where the results show, the cost and the Free Edition quota, adoption, the ungated removal and a refused monitor, plus an upgrading note. The markers use the main form: #864 merged as 215b51d7.
Merged
1 of 3 tasks
1 of 3 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents Agenticstiger/forge-cli#864: a Databricks expose that states a freshness or completeness SLO turns on Databricks data quality monitoring for the schema the product owns.
What changed
docs/providers/databricks.md: a new "Data quality monitoring" section. It covers what turns monitoring on and the opt-out, the monitorfluid applycreates, why the SLO values are not thresholds (and thedq.monitoring.windowwarning), where the results show, the cost and the Free Edition quota, adoption, the ungated removal and a refused monitor. Also an Overview bullet, a row in "Whatfluid applyprovisions", a row in "Changes the apply gates", a sentence in "What validate refuses", a Limitations bullet and a sentence in the live row of "How it is tested".docs/upgrading.md: a bullet under "Not in a release yet". A contract that already states such an SLO gets a monitor at its next apply, that apply can now fail if Databricks refuses the monitor, and the bullet says how to opt out.Markers
forge-cli #864 merged at 22:49Z as
215b51d7, so each new passage carries the main form: "On forge-climain, not yet in a release (forge-cli #864)." That is 8 markers on 2 pages. The pin bump for the next release turns them into "since" markers.forge-cli #861 merged at 22:38Z, so its branch markers from #207 on these two pages are stale now. This PR does not touch them.
Checked against
forge-cli
215b51d7, the squash of #864. The checks ran at the PR head (39e692ce, thenbefe92ea). Against the squash, the PR's files differ only by #861's CHANGELOG entry and itsapplied_stateplumbing in the apply engine, neither of which this page documents.fluid validateandfluid generate iacrun at the PR head, the plan note fromplan_noticescalled offline (as the Certification section does), and the removal and refusal output from the PR's six stand-in e2e tests (realtofuand provider 1.140.0 against the in-process stand-in), all six passing.HONESTLY_TESTED.md. Databricks facts (the indicator's statuses, who can readtable_results, the billing product,MANAGE) come from Databricks' anomaly detection page.tofu providers schema. It found nothing wrong and 7 imprecise claims, all fixed here.Where the page differs from the PR's own wording:
HONESTLY_TESTED.mdsaysnotification_settingsis not in provider 1.140.0. It is, but underdata_profiling_config(on_failureonly).anomaly_detection_configholds onlyexcluded_table_full_names, so the page says the anomaly detection settings have no alert setting.Gates, run locally (Python 3.11, data-product-forge 0.29.0)
scripts/check_cli_docs.py(full,--version-only,--markers-only): OKscripts/check_providers.py: OKscripts/gen_contract_reference.py --check: OKnpm ci && npm run docs:build && node scripts/check-dist-links.mjs: OK, and the six new anchors are in the built pagelychee --offlineon both pages over CI's route mirror: 0 errors