From e99be24b46bb14af3789163b0a612007b57f5d62 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 12 Nov 2026 22:37:48 +0100 Subject: [PATCH] docs(databricks): amend-and-build runs the dbt job and waits for it (forge-cli#861) Document forge-cli #861: on Databricks, --mode amend-and-build and replace-and-build start a run of a dbt build's provisioned job, wait for it and fail the apply when it fails. Covers the modes, how the job is found, what the run leaves alone, the output, the wait cap FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS and cancellation, the missing-SDK refusal, --dry-run and building an estate in lineage order. Every new sentence carries the branch marker: #861 is not merged yet. The Prerequisites row no longer says only fluid verify needs the [databricks] extra; apply needs it for masks and DuckDB landing too. --- docs/advanced/environment-variables.md | 1 + docs/advanced/error-codes.md | 2 + docs/advanced/operating-in-ci.md | 1 + docs/cli/apply.md | 3 + docs/providers/README.md | 2 +- docs/providers/databricks.md | 98 ++++++++++++++++++++++++-- docs/upgrading.md | 6 ++ 7 files changed, 106 insertions(+), 7 deletions(-) diff --git a/docs/advanced/environment-variables.md b/docs/advanced/environment-variables.md index f2c5bf1..4f6135e 100644 --- a/docs/advanced/environment-variables.md +++ b/docs/advanced/environment-variables.md @@ -65,6 +65,7 @@ Since `0.20.0`, a project's `.env`, `.env.` and `.env.local` may ho |---|---| | `FLUID_STATE_BACKEND` | Default of `--state-backend` for `fluid apply`, `fluid diff` and `fluid verify --state-drift`: `s3://[/]` or `gcs://[/]` (see [Remote state](../cli/apply.md#remote-state)). A spec that names only a bucket gives every contract its own key (see below); the flag does not. An empty `--state-backend ""` forces local state for one run. | | `FLUID_TOFU_TIMEOUT_SECONDS` | Wall-clock cap per `tofu` invocation. Default `1800`. A timed-out call reports exit code 124. *(since 0.27.0, [forge-cli #787](https://github.com/Agenticstiger/forge-cli/pull/787))* `tofu` is interrupted first, so it releases its state lock, and killed only if it is still running 30 seconds later. | +| `FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS` | *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* How long `fluid apply --mode amend-and-build` or `replace-and-build` waits for a Databricks job run, in whole seconds, before it cancels the run and fails the build. Unset, an hour, or the job's own timeouts and retries plus 15 minutes when that is longer. A value that is not a whole number of seconds above 0, in at most nine digits, is not used, and the apply says so. `fluid doctor --env` lists it, and a project `.env` may set it. See [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply). | | `FLUID_TOFU_S3_LOCKFILE` | *(since 0.27.0, [forge-cli #787](https://github.com/Agenticstiger/forge-cli/pull/787))* `0`, `false`, `no` or `off` stops `fluid apply` from asking `tofu init` to lock S3 state (`-backend-config=use_lockfile=true`, OpenTofu 1.10 or newer). Set it for a role that cannot write `.tflock`, in the environment of the job: a project `.env` cannot set it (see [project `.env` trust](./project-dotenv-trust.md)). Unset, or any other value, keeps the lock on. See [Locking](../concepts/state.md#locking). | | `FLUID_OPENTOFU_VERSION` | Pins the OpenTofu version that [`fluid apply --ensure-opentofu`](../cli/apply.md) installs when `tofu` is missing. | | `FLUID_REDSHIFT_WORKGROUP`, `FLUID_REDSHIFT_DATABASE`, `FLUID_REDSHIFT_SQL` | Set by the generated AWS module for its Redshift Serverless statements. They are outputs of the generator, not settings. | diff --git a/docs/advanced/error-codes.md b/docs/advanced/error-codes.md index 277ca67..4765484 100644 --- a/docs/advanced/error-codes.md +++ b/docs/advanced/error-codes.md @@ -228,6 +228,8 @@ The state refusals (`state_shared_with_another_provider`, `state_migration_ambig *(since 0.20.0, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714) and [forge-cli #715](https://github.com/Agenticstiger/forge-cli/pull/715))* `apply_contract_invalid`, `apply_plan_missing_contract` and `unsupported_binding` (kind `mwaa-environment-not-emitted`) are not catalogued, so they print no curated suggestions. [Production troubleshooting](./production-troubleshooting.md#environment-mode-and-plan-file-errors) explains the first two, and [Scheduling on AWS](../providers/aws.md#scheduling-on-aws) the third. `contract_version_unsupported`, catalogued [above](#contract-version-unsupported), is now raised by `fluid apply` on both engines and in every mode. +*On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* `apply_databricks_sdk_missing` is not catalogued either: its own `hint` line gives the fix. [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply) explains it. + ### planner_failed `ERR_PLANNER_FAILED`. The documentation link lands on [Production troubleshooting](./production-troubleshooting.md). diff --git a/docs/advanced/operating-in-ci.md b/docs/advanced/operating-in-ci.md index c5e6963..c2ccd07 100644 --- a/docs/advanced/operating-in-ci.md +++ b/docs/advanced/operating-in-ci.md @@ -209,6 +209,7 @@ Cloud applies go through the OpenTofu engine, which runs the `tofu` binary: - **State.** CI wipes the workspace after every run, and a wiped local state re-plans every resource as new. Set `FLUID_STATE_BACKEND=s3://` or `gcs://` once for the pipeline: each contract and provider then gets its own key. See [Environment variables](./environment-variables.md#apply-state-and-opentofu). - **S3 state lock.** *(since 0.27.0, [forge-cli #787](https://github.com/Agenticstiger/forge-cli/pull/787))* With OpenTofu 1.10 or newer, S3 state is locked through a `.tflock` object, and `tofu plan` takes the lock too. Every role in the pipeline that plans or applies, a plan-only job and a drift check (`fluid diff`, `fluid verify --state-drift`) included, needs `s3:GetObject`, `s3:PutObject` and `s3:DeleteObject` on that object, or `FLUID_TOFU_S3_LOCKFILE=0` in the job's environment (a project `.env` cannot set it). A job that is killed hard can leave the lock behind; `tofu force-unlock ` in the contract's workdir clears it. See [Locking](../concepts/state.md#locking). - **Timeouts.** Each `tofu` call is capped at 1800 s by default; raise it for a large first apply with `FLUID_TOFU_TIMEOUT_SECONDS`. *(since 0.27.0, [forge-cli #787](https://github.com/Agenticstiger/forge-cli/pull/787))* A call that runs past the cap is interrupted first, so `tofu` releases its state lock, and killed only if it is still running 30 seconds later. +- **Databricks job runs.** *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* Under `--mode amend-and-build`, a dbt build that runs as a Databricks job holds the CI job until its run ends, up to `FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS`. A cancelled CI job that stops the apply with SIGINT, SIGTERM or SIGHUP cancels the run too. See [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply). - **Review before apply.** `fluid generate iac ` emits a deterministic, credential-free `main.tf.json` you can archive or review in a pull request; see [`fluid generate iac`](../cli/generate-iac.md). ## Contract SQL in CI diff --git a/docs/cli/apply.md b/docs/cli/apply.md index 06a3340..4731daf 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -154,6 +154,8 @@ amend-and-build (or replace-and-build) to run that build, or drop --build-id. Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. +*On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* On Databricks, a dbt build with `execution.runtime.platform: databricks` runs as the job the apply provisions. The build phase starts a run of the job and waits for it to end, and a failed run fails the apply. See [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply). + ## Contract validation *([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), since 0.20.0)* `fluid apply` checks the contract against its schema before it plans anything, on both engines and in every mode, `--dry-run` and the build modes included. A GCP contract that still carries the deprecated `metadata.policies` mapping stops before `tofu` runs: @@ -624,6 +626,7 @@ Each failure prints the typed event name and an `[ERR_]` slug, and exits | `apply_build_id_requires_build_mode` | `--build-id` with a mode that runs no build. | | `apply_build_identifier_invalid` | *(since 0.27.0, [forge-cli #803](https://github.com/Agenticstiger/forge-cli/pull/803))* A build mode, and an `id` or `builds[].id` the build phase cannot use. Nothing is applied. See [Build ids](#build-ids). | | `apply_build_id_not_found` | *(since 0.27.0, [forge-cli #808](https://github.com/Agenticstiger/forge-cli/pull/808))* A build mode, and a `--build-id` that matches no `builds[].id`. The error lists the ids the contract has. Nothing is applied. See [Build ids](#build-ids). | +| `apply_databricks_sdk_missing` | *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* A build mode without `--dry-run`, a selected dbt build that runs as a Databricks job and is not continuous, and no `databricks-sdk`. Nothing is applied. Install `data-product-forge[databricks]`, or apply with `--mode amend`. See [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply). | | `apply_mode_data_loss_blocked` | `replace` or `replace-and-build` without `--allow-data-loss`. | | `opentofu_data_loss_gate` | An OpenTofu plan destroys a resource and `--allow-data-loss` is not set. *(since 0.27.0, [forge-cli #793](https://github.com/Agenticstiger/forge-cli/pull/793))* Also a plan that changes the column types of a Glue table in place. | | `opentofu_plan_failed` | `tofu plan` failed. *(since 0.27.0, [forge-cli #786](https://github.com/Agenticstiger/forge-cli/pull/786))* A plan that failed on a held state lock suggests `tofu force-unlock `. | diff --git a/docs/providers/README.md b/docs/providers/README.md index fa3e65f..40b41fd 100644 --- a/docs/providers/README.md +++ b/docs/providers/README.md @@ -21,7 +21,7 @@ Fluid Forge uses one contract format across local and provider-backed execution | [GCP](./gcp.md) | Yes (OpenTofu) | Prefer `fluid generate schedule` | Production | | [AWS](./aws.md) | Yes (OpenTofu) | Prefer `fluid generate schedule` | Production | | [Snowflake](./snowflake.md) | Yes (OpenTofu) | Prefer `fluid generate schedule` | Production | -| [Databricks](./databricks.md) | Yes (OpenTofu) | dbt builds run as Databricks jobs on their own trigger | Added in 0.26.0 | +| [Databricks](./databricks.md) | Yes (OpenTofu) | dbt builds run as Databricks jobs on their own trigger. *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* `--mode amend-and-build` also starts a run and waits for it | Added in 0.26.0 | | [Local](./local.md) | Yes (native DuckDB) | Local-first onboarding | Production | "Production" covers provisioning and builds. Governance differs by cloud: on GCP, `fluid apply` emits dataset grants, column policy tags and, on `0.7.6`, retention and Cloud KMS keys; on Snowflake, as of 0.18.1, `accessPolicy` grants, column restrictions and masking are not applied. Each provider page lists what its module emits. diff --git a/docs/providers/databricks.md b/docs/providers/databricks.md index 1f8c79b..51628ce 100644 --- a/docs/providers/databricks.md +++ b/docs/providers/databricks.md @@ -14,7 +14,7 @@ Deploy data products to Databricks Unity Catalog (a schema, a managed Delta tabl - ✅ **Plan & Apply**: a schema, a managed Delta table and one grant per principal, through OpenTofu - ✅ **Existing catalog**: `location.database` names a catalog that must already exist. forge-cli reads it and never creates, grants on or destroys it -- ✅ **dbt builds as jobs**: a dbt build whose `execution.runtime.platform` is `databricks` becomes a serverless job that Databricks runs on its own trigger +- ✅ **dbt builds as jobs**: a dbt build whose `execution.runtime.platform` is `databricks` becomes a serverless job that Databricks runs on its own trigger. *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* `fluid apply --mode amend-and-build` also starts a run now and waits for it (see [Running the job from `fluid apply`](#running-the-job-from-fluid-apply)) - ✅ **Residency**: `binding.location.region` is checked against the contract's `sovereignty` policy and against the workspace's metastore (see [Residency](#residency)) - ✅ **Verify**: `fluid verify` checks the live table's columns, `USE CATALOG` for every reader, the residency, the governance tags, the bound masks and row filter, and that each dbt job exists exactly once - ✅ **One pinned credential**: the workspace comes from `DATABRICKS_HOST` only, and a run never falls back to a stray profile @@ -34,7 +34,7 @@ Deploy data products to Databricks Unity Catalog (a schema, a managed Delta tabl | A Unity Catalog workspace with an existing catalog | The catalog is read, never created. The identity that applies needs `USE CATALOG` and `CREATE SCHEMA` on it. | | A SQL warehouse, its id in `DATABRICKS_WAREHOUSE_ID` | Every table statement runs on it, and so do the dbt jobs. Find the id under SQL Warehouses, Connection details. The id is not a secret. | | A Unity Catalog volume of its own, `/Volumes///` in `DATABRICKS_STAGING_VOLUME` | Only for a DuckDB acquisition that lands in Databricks (see [Landing a DuckDB acquisition](#landing-a-duckdb-acquisition)). | -| `pip install 'data-product-forge[databricks]'` | The extra brings `databricks-sdk`. Only `fluid verify` needs it. | +| `pip install 'data-product-forge[databricks]'` | The extra brings `databricks-sdk`. `fluid verify` needs it, and so does `fluid apply` to bind [column masks and row filters](#column-masks-and-row-filters) and to [land a DuckDB acquisition](#landing-a-duckdb-acquisition). *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* An apply that [runs a dbt job](#running-the-job-from-fluid-apply) needs it too. | ## Credentials @@ -497,6 +497,8 @@ job 'fluid sales.orders_dbx transform' is created paused (pause_status PAUSED): The next `fluid apply`, once the bindings match, unpauses it in place, with no drift from unpausing it by hand. A manual job has no trigger to pause. `fluid validate` and the pure compile never read the workspace. +*On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* Under `--mode amend-and-build`, the run the apply starts (see [Running the job from `fluid apply`](#running-the-job-from-fluid-apply)) comes after the bindings, so it fills a protected table. It leaves the job paused: the next `fluid apply` unpauses the schedule, as above. + #### What masks and filters refuse Refused rather than dropped, since a table applied without them would be readable without what the contract promises: @@ -596,7 +598,7 @@ Never uploaded: `profiles.yml`, `target/`, a hidden file or directory, a symboli | --- | --- | | `trigger.type: schedule` (or no type) with `cron` | A Quartz `schedule`, days of the week by name. `trigger.timezone` is an IANA name; the default is `UTC`. | | `trigger.type: continuous` | Databricks' `continuous` trigger. | -| `trigger.type: manual`, or no trigger | No trigger. Start a run from the job's page or with `databricks jobs run-now`. | +| `trigger.type: manual`, or no trigger | No trigger. Start a run from the job's page or with `databricks jobs run-now`. *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* Or apply with `--mode amend-and-build`, which starts one and waits for it (see [Running the job from `fluid apply`](#running-the-job-from-fluid-apply)). | | `retries.maxAttempts` | The first run counts, so `3` is two retries. | | `retries.initialDelay` | The fixed interval between attempts. | | `notifications` of `type: email` | The job's email notifications, `condition` of `success`, `failure` or `always` (default `failure`). | @@ -620,10 +622,94 @@ A job runs as the identity that applies it. A contract can name a service princi ### Applying and removing -`fluid apply --mode amend-and-build` leaves such a build to its job. It does not run dbt on the applying machine; it prints the job's name and how to start a run now. A dbt build without `runtime.platform: databricks` still runs locally with the dbt-databricks profile. forge-cli's own schedules (`fluid generate schedule`, the Airflow DAG `fluid apply` writes) give a Databricks job build no task: Databricks starts its runs. +`fluid apply` never runs such a build's dbt on the applying machine: the build runs as its job. A dbt build without `runtime.platform: databricks` still runs locally with the dbt-databricks profile. forge-cli's own schedules (`fluid generate schedule`, the Airflow DAG `fluid apply` writes) give a Databricks job build no task: Databricks starts its runs. In 0.29.0, `--mode amend-and-build` provisions the job, then prints its name and how to start a run now. + +*On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* `--mode amend-and-build` and `--mode replace-and-build` start a run of the job and wait for it, as the next section says. Removing a job, an uploaded file or a folder from the contract needs no `--allow-data-loss`, since no table data is lost. The job's run history goes with the job, and the apply says so. +### Running the job from `fluid apply` + +*On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* + +Under `--mode amend-and-build` and `--mode replace-and-build`, `fluid apply` starts a run of each such build's job and waits for it to end. The run starts once `tofu apply` has succeeded and the [masks and row filters](#column-masks-and-row-filters) are bound, so it writes tables that are already protected. A run that fails fails the apply. + +| `--mode` | The job | +| --- | --- | +| `amend-and-build` | Provisioned, then one run starts now, and the apply waits for it. | +| `replace-and-build` | The same, and that run adds `--full-refresh` to the job's `dbt build`. | +| `amend`, `create-only`, `replace` | Provisioned only. It runs on its trigger. | + +`--build-id ` starts only that build's job. A continuous job (`trigger.type: continuous`) is never started: Databricks keeps it running, so a run of it never ends. The apply says so and goes on. + +- **The job is the one this apply provisioned.** The apply finds it in its own OpenTofu state by the job's `fluid_product` and `fluid_build` tags, never by its name: two environments of one contract in one workspace each own a job of that name. When the state holds no such job, or more than one, no run starts and the build fails. +- **The run changes nothing on the job.** The apply starts the run and, at the time limit or on an interrupt (below), cancels it. It writes nothing else. The job's settings, schedule and pause state stay as they are, so a scheduled job keeps its schedule and a job [created paused](#a-job-that-writes-a-table-not-yet-protected) stays paused. The `--full-refresh` of `replace-and-build` goes with that run only, and the job's own commands keep none. +- **A busy job queues the run.** The run waits in the job's queue, instead of being skipped, when the job is busy (it runs one run at a time, and a scheduled run may be going) or the workspace is at its limit. The start request carries an idempotency token and is retried with the same token, so a retry never starts a second run. + +What a run prints, captured live on Databricks Free Edition on 2026-11-12 for a one-model dbt product, with the workspace host and id replaced: + +```text +🚀 Build 'summarize' - started run 864120962919474 of Databricks job 'fluid iactest.build.sites_summary summarize' (711974842827325) + run page: https:///?o=#job/711974842827325/run/864120962919474 + 0:00 RUNNING + 0:57 TERMINATED SUCCESS + task 'dbt': Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 +✅ Build 'summarize' - Databricks run 864120962919474 succeeded in 57s +``` + +- The run's page comes first. Then each state the run enters gets a line, with the time since the run started, and a `still ` line follows each minute while nothing changes. +- A queued run says why. Databricks' own reason is used when it gives one. It usually gives none (seen live), so the line names one of two causes, or none when the job's runs cannot be listed: + + ```text + 0:00 QUEUED - behind run of this job, which runs one at a time + 0:00 QUEUED - waiting for the workspace's limit on runs or tasks at once (Databricks Free Edition runs at most 5 job tasks at a time) + ``` + +- On success, dbt's own `Done. PASS=...` line is printed for each task that has one. +- Every credential value in the environment is masked in what is printed, and terminal escape sequences are removed. + +**When the run fails.** A run that ends in anything but `SUCCESS` fails its build, and the apply exits `1`. The apply prints the run's end state and message. For each task that did not succeed, it prints the task's error and up to the last 30 lines of its output, then the run page again. A live run of a model that names a column its source does not have (abridged, with the wrapped error line joined): + +```text + 0:00 RUNNING + 1:08 still RUNNING + 1:18 TERMINATED RUN_EXECUTION_ERROR +❌ Build 'summarize' - Databricks run 973093456770242 ended RUN_EXECUTION_ERROR after 1m18s: Task dbt failed with message: Workload failed, see run output for details. + task 'dbt': Command 'dbt build --select +sites_summary+' … returned non-zero exit status 1. + dbt output: + + dbt deps + … + 19:59:14 Database Error in model sites_summary (models/sites_summary.sql) + [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `height_meters` cannot be resolved. Did you mean one of the following? [`height_m`, `site_id`, `kind`]. SQLSTATE: 42703; line 19 pos 47 + … + 19:59:14 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 + run page: https:///?o=#job/711974842827325/run/973093456770242 +``` + +**How long it waits.** `FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS` caps the wait for each run, in whole seconds. Unset, the cap is an hour, or the job's own limit plus 15 minutes when that is longer, so a contract's `runtime.timeout` is never cut short. The job's own limit is its timeout when it has one (a job forge-cli emits has none), else each task's timeout times its attempts, plus the intervals between them. The build in the example under [dbt builds as jobs](#dbt-builds-as-jobs) (`timeout: 1h`, `maxAttempts: 3`, `initialDelay: 5m`) is waited for up to 3 h 25 min: three attempts of an hour, two 5-minute intervals and the 15 minutes. A task with no timeout leaves the cap at an hour. A value that is not a whole number of seconds above 0, in at most nine digits, is not used: the apply says so, and names the cap it waits for instead. + +**Stopping it.** At the cap, the run is cancelled and the build fails, and the message names the variable to raise. Ctrl-C while the run is waited on cancels it too, and so do SIGTERM and SIGHUP (a cancelled CI job, a closed terminal); the apply then stops as Ctrl-C stops it. A signal that is ignored stays ignored, so under `nohup` the apply and the run outlive the terminal. A cancel that fails is said: `could not cancel run : `. + +**A run it cannot read.** A failed read of the run is tried again, and the first failure is said. After five failed reads in a row, the apply stops waiting and the build fails. The run is left as it is, since it may still be going, and its page is printed when an earlier read returned it. Failed reads alone never cancel a run. + +**It needs `databricks-sdk`.** The run is started through it. Without the `[databricks]` extra, an apply that would start a run is refused before anything is applied: + +```text +❌ apply_databricks_sdk_missing [ERR_APPLY_DATABRICKS_SDK_MISSING] + error: a dbt build runs on Databricks, and the build phase starts its job through databricks-sdk, which is not installed; nothing was applied + hint: pip install 'data-product-forge[databricks]', or apply with --mode amend to provision the job without starting it +``` + +`--dry-run` is not refused. It runs `tofu plan`, applies and starts nothing, and names the run it would start: + +```text +🔍 [DRY RUN] Build 'summarize' - would start a run of Databricks job 'fluid iactest.build.sites_summary summarize' and wait for it. +``` + +Under `replace-and-build` the line ends `with dbt build --full-refresh and wait for it.` + +**Several products, in lineage order.** Each apply starts its own product's jobs only. To build an estate whose products read each other, provision every product first, in parallel, with `--mode amend`. Then run `--mode amend-and-build` one lineage level at a time, upstream first. For a product that has not changed since, the second apply plans no change (live: `tofu plan: +0 ~0 -0`, about 30 seconds for some 75 resources), except to unpause a job [created paused](#a-job-that-writes-a-table-not-yet-protected). Then it starts the runs and waits for them. + ## Tables a dbt build writes dbt's `table` materialization rebuilds a table with `CREATE OR REPLACE`. So after the apply creates a table that a dbt build writes (locally or as a job), the module leaves its definition to dbt. It ignores the table's columns, partitions, clustering, properties and comment, and sets no column mapping. Each apply reports the column drift it no longer reconciles, and `fluid verify` is the gate. *(since 0.28.0, [forge-cli #831](https://github.com/Agenticstiger/forge-cli/pull/831))* The comment is left to dbt too, so the contract's descriptions are not written there; `fluid validate` warns and says so (see [Comments on a dbt-written table](#comments-on-a-dbt-written-table)). The tags are still set. @@ -794,7 +880,7 @@ The full matrix is in `HONESTLY_TESTED.md` in forge-cli. In short: | Unit | 471 tests across the emitter, jobs, credentials, engine hooks, cron conversion, the dbt runner, verify, doctor and the welcome scan, against a stubbed SDK client. | | `tofu validate` | Both job modules (Git and workspace upload) and the dbt-written table's `ignore_changes` list validate against provider 1.140.0's schema. | | Stand-in workspace | The real `fluid` CLI, `tofu` and provider 1.140.0 against an in-process stand-in for the Unity Catalog, SQL statement, permissions and jobs APIs: plan, apply, an empty second plan, the data-loss gate, adoption, job and file removal. The stand-in does not check SQL dialect or privilege names. | -| Live, Databricks Free Edition | One manual run on 2026-11-01 with the real `fluid` CLI: validate, plan, dry-run and apply, an empty second plan, one job run that succeeded and built the table, `fluid verify` with the real SDK, a model change, and removal of the job and its files. Everything the run created was swept afterwards. | +| Live, Databricks Free Edition | One manual run on 2026-11-01 with the real `fluid` CLI: validate, plan, dry-run and apply, an empty second plan, one job run that succeeded and built the table, `fluid verify` with the real SDK, a model change, and removal of the job and its files. Everything the run created was swept afterwards. *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* A second manual run on 2026-11-12, in a scratch catalog dropped afterwards, ran the job from `fluid apply`: `amend-and-build` started a run, waited for it and exited 0; a broken model exited 1 with the task's error and dbt's output; `replace-and-build` on a scheduled job ran `--full-refresh`, and the job kept its schedule and its commands; a run queued behind another run of the job; a missing `databricks-sdk` was refused before `tofu`; `--dry-run` and `--mode amend` started no run. | | Emulator (nightly, not gating) | First run on 2026-10-30 passed 1 of 4: creates and the replace gate work, but a second plan is not empty against the emulator. | -Not proven live: a Git source, profile or OAuth auth, a run-as service principal, table adoption, a larger project, and re-adopting a job after lost state. +Not proven live: a Git source, profile or OAuth auth, a run-as service principal, table adoption, a larger project, and re-adopting a job after lost state. *On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* Nor are the reason line of a queued run and a run queued for the workspace's limit. diff --git a/docs/upgrading.md b/docs/upgrading.md index 2008db0..d01a680 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -849,6 +849,12 @@ These changes to AI authoring can change what a script, a CI job or an IDE agent - *(since 0.29.0, [forge-cli #850](https://github.com/Agenticstiger/forge-cli/pull/850))* **`fluid ai test` probes the real call.** A rejected key exits `3`, a model the provider does not have exits `4`, and other failed calls exit `5`. See [`ai test`](./cli/ai.md#ai-test). - *(since 0.29.0, [forge-cli #852](https://github.com/Agenticstiger/forge-cli/pull/852))* **`LlmProvider.list_available_models` takes an `endpoint` keyword**, the address the call will go to. Forge passes it only when an endpoint is stated, so code that subclasses `LlmProvider` and overrides the method with two arguments keeps working until then. With an endpoint stated and the model check on (`--require-llm`, `--tiered` or `FLUID_LLM_MODEL_PREFLIGHT`), the check fails with `copilot_llm_model_preflight_failed`; add the keyword to avoid it. +## Not in a release yet + +*On forge-cli's feat/databricks-build-run-now branch ([forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861)), not yet on `main` or in a release.* + +- **`fluid apply --mode amend-and-build` on Databricks runs each dbt job build now, and waits for it.** In 0.29.0 it only provisioned the job, so a manual job never ran. A pipeline that applies with `amend-and-build` or `replace-and-build` now waits for each run, up to `FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS` (by default an hour, or the job's own limits plus 15 minutes), and fails when a run fails. It needs `databricks-sdk`: without `pip install 'data-product-forge[databricks]'`, the apply is refused before anything is applied. To provision a job without running it, apply with `--mode amend`. See [Running the job from `fluid apply`](./providers/databricks.md#running-the-job-from-fluid-apply). + ## DuckDB 1.x to 2.x `0.22.0` raised the local engine's DuckDB range to `duckdb>=1.5.0` with no upper bound, so