Repository navigation
feat(databricks): amend-and-build runs the dbt job now and waits for it - #861
Merged
Merged
Conversation
On Databricks, `fluid apply --mode amend-and-build` provisioned a dbt build's serverless job and then only printed "start a run now from the job's page". A manual-trigger job never ran, so an estate applied in lineage order built nothing on Databricks, unlike every other target. Now `amend-and-build` and `replace-and-build` start a run of the job and wait for it: - The job is the one this apply's OpenTofu state holds for the build. The engine hands back the state after `post_apply` binds the masks, and the job is matched by the emitter's `fluid_product` / `fluid_build` tags, never by its name. - run-now carries an idempotency token and `queue.enabled`, so a busy job queues instead of skipping. replace-and-build adds `--full-refresh` to the job's own `dbt build` for that run only (`dbt_commands`). Nothing else is written to the job: the schedule, the pause state and the commands stay as the apply set them. - Progress shows the run page, each state with its elapsed time, and a line a minute. A queued run says why: Databricks' reason, or "behind run N of this job" / "the workspace's limit" from runs/list. - Success shows dbt's `Done. PASS=...` line. Any other end fails the build with the task's error and the last 30 lines of its output, with credentials masked and escape codes stripped. - The wait cap is FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS: an hour, or the job's own timeouts and retries plus 15 minutes. At the cap or on Ctrl-C the run is cancelled. - A continuous job is not started. `amend` and `create-only` keep provisioning only. A missing databricks-sdk is refused before tofu runs. Polling follows databricks-sdk-py's run waiter (wait_get_run_job_terminated_or_skipped, Apache-2.0). The failed-task report follows Apache Airflow's Databricks provider (extract_failed_task_errors, Apache-2.0) and `databricks bundle run`.
…o end The in-process Databricks stand-in serves run-now, runs/get, runs/get-output and runs/cancel. Each run plays `run_states` and ends as `run_result` says. The jobs e2e then drives the real `fluid` CLI, `tofu` and provider: - amend-and-build starts the job this apply's state holds (one run, queued, three polls) and writes nothing else to the job; - a failed run fails the apply with dbt's output; - amend provisions the job and starts no run.
AGENTS.md's apply-mode notes say what amend-and-build and replace-and-build do on Databricks and how to build an estate in lineage order. CLAUDE.md records the seam and the API facts checked live on Free Edition. The secrets baseline follows AGENTS.md's shifted line.
…only The queued-run reason names the run ahead of ours from GET runs/list. That id is remote data, so it is now coerced to an integer like every other id printed, and anything that is not a positive number is not counted as a run.
…imeout From the delivery review of this branch: - A failed runs/get, after the SDK's own retries, ended the build and left the run going. It is now cancelled like a run past the wait cap. A cancel that fails too is said, with the run page. - FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS that is not a whole number of seconds above 0 (`1h`, `0`, `-5`) was ignored without a word. It is now named, with the wait used instead. - Durations of an hour or more read `1h00m00s`, not `60m00s`. CHANGELOG [Unreleased] records the new amend-and-build behaviour.
…and keys From the defect scan of this branch: - A failed runs/get is tried again: 5 reads in a row, with a growing pause. The SDK does not repeat a 500, 502 or 504. One such reply cancelled a healthy run (since e422ed4b). A run that cannot be seen is never cancelled; the build fails and says it may still be going. - run-now is tried 3 times with the same idempotency token. The SDK does not repeat a POST that timed out, and Databricks starts one run per token, so a retry never starts a second run. If it still fails, the job's page is named. - A dbt build with no `id` (the schema does not require one) gets its job tagged by its position, as before. The runner now looks it up by that position through one shared key (`job_build_id`). - Ctrl-C, SIGTERM and SIGHUP cancel the run only while it is waited on. An interrupt while a finished run's output is read cancels nothing. - The wait budget is decided before run-now. An env value int() cannot read (`²`, 5000 digits) is named rather than raised after the start. - A queued run is "behind" only a run of the job that is not queued. - An empty state read back after the apply is said, and the build's message names both causes. - The SDK gate runs only for the OpenTofu engine; the native engine starts no job.
…he SDK From the re-check of the defect-scan fixes and the full suite in a CI-like venv: - During the wait, SIGTERM and SIGHUP cancel the run. A signal the operator chose to ignore now stays ignored, so `nohup fluid apply` keeps running when its terminal closes. - The apply-level hand-off tests report databricks-sdk as installed. CI's `.[dev,local]` install has no SDK, so the gate refused first. - The apply hands over the jobs as a keyword, not through `build_kwargs.update`. tests/test_engine.py's `args\.` scan read that as an `args.update` field. - The docstring and AGENTS.md say that failed reads alone never cancel a run, and name SIGHUP.
github-actions
Bot
added
security
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
fas89
enabled auto-merge (squash)
November 12, 2026 21:09
github-actions
Bot
removed
the
needs-docs
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Code-quality review on #861: the empty except in _cancelled_if_interrupted's restore loop now says why. A handler Python cannot set back, such as one installed from C, which reads as None, is left as it is: the wait is over, and failing there would hide the build's own result. Behaviour is unchanged.
The Stage 1 Databricks lane installs .[dev,test-emulators], without the [databricks] extra, so the two amend-and-build e2e tests stopped at the apply's SDK gate (apply_databricks_sdk_missing). When the SDK is missing, they now start the job through the stand-in's own HTTP client and lift only that gate. The rest of the path still runs for real: CLI, tofu, provider, state hand-off, run-now, polling and the report. With the extra installed they use the real SDK as before. Passes both ways locally: 9/9 without the SDK, 3/3 with it.
…o feat/databricks-build-run-now # Conflicts: # CHANGELOG.md # fluid_build/cli/apply.py # tests/iac/test_iac_databricks_jobs_fake_e2e.py
…o feat/databricks-build-run-now # Conflicts: # .secrets.baseline # tests/iac/_fake_databricks.py
…o feat/databricks-build-run-now # Conflicts: # CHANGELOG.md
This was referenced Nov 12, 2026
fas89
added a commit
that referenced
this pull request
Nov 13, 2026
…written, plan refuses a blocked residency (#868) * fix(redact): mask a JWT by its JOSE header, never a fully qualified table name fluid apply printed "databricks.load table=***REDACTED*** rows=340" for northwind_towers.bronze_smart_meters.smart_meters and the real name for a table whose last part had 11 characters. The prefix-agnostic _JWT_RE masked any three dotted runs of 12 or more characters, and a catalog.schema.table name (or project.dataset.table, DB.SCHEMA.TABLE) has that shape. Such a run is now masked only when its first part is a JOSE header: base64url of a JSON object, the check detect-secrets' JwtTokenDetector.is_formally_valid makes (Yelp/detect-secrets plugins/jwt.py). The eyJ-anchored rule, which gitleaks' jwt rule and trufflehog's JWT detector also anchor on, now covers a JWE (a direct-encryption one has an empty encrypted key) and short or empty payloads and signatures. One redact_jose_tokens() serves both redaction layers: the Snowflake twin calls it, as it imports _URL_USERINFO_RE, so they cannot drift. The rules start where no base64url character precedes them instead of at \b, which matched after every '-' and made the scan quadratic on a-a-a-... input (200k characters: about 30 s before, 0.07 s now). keep_dotted_names goes: logs and printed errors apply the same rule now. * fix(databricks): write Unity Catalog tag values as the contract states them The governance labels became Unity Catalog tags with GCP's label slugs: lowercased, spaces and punctuation as '-', cut at 63 characters (business_value = 'a-single-unrenewed-ground-lease-can-strand-a-multi-tenant- tower', fluid_jurisdiction = 'us'). A UC tag value takes up to 256 characters with case, spaces and punctuation, and may not start or end with a space (docs.databricks.com, "Apply tags to Unity Catalog securable objects"). governance_labels() gains verbatim_values: the Databricks module asks for the contract's text, stripped, with the regulatory frameworks joined by ", ", and refuses a value over 256 as databricks-tag-value-too-long (one _too_long refusal now serves the product tags too). Keys keep GCP's label alphabet, each '-' an '_'; GCP labels and AWS tags are unchanged. An existing deployment updates in place: databricks_entity_tag_assignment's tag_value carries no RequiresReplace at provider v1.140.0 and Update sends UpdateMask "tag_value" (entity_tag_assignment/resource_entity_tag_assignment.go), and the address and import id depend on the key alone. Pinned by a fake- workspace e2e (slug values, then the real ones: +0 ~2 -0, nothing replaced) and observed on Free Edition: +0 ~17 -0, the table's rows untouched, verify green on all 27 tags. * feat(plan)!: refuse a contract its own sovereignty policy blocks, by default fluid validate and fluid plan --check-sovereignty refused a contract whose binding region its strict sovereignty denies ("Region 'us-west-2' is explicitly denied by sovereignty policy"), but plain fluid plan wrote a plan for it, and --check-sovereignty itself wrote the plan before it exited 1. plan now runs the built-in policy engine, the checker validate runs, before it plans, and refuses (exit 1, sovereignty_violation, no plan.json) whenever validate would: strict findings, or a region in deniedRegions, which blocks in every mode. It builds no provider, so a plain plan stays off the cloud SDKs (Command Center's plan endpoint relies on that) and prints nothing when nothing blocks. --check-sovereignty keeps asking the provider's hook and printing the verdict, now before the write. --no-check-sovereignty (argparse.BooleanOptionalAction) skips the check and logs a WARNING with the event sovereignty_check_skipped, the shape of apply's --no-verify-plan-binding; it changes nothing in apply, which still refuses the placement on Databricks. The design follows HCP Terraform's policy enforcement levels: a mandatory policy stops the run, and an override is explicit and recorded. BREAKING CHANGE: fluid plan exits 1 and writes no plan for a contract its sovereignty policy blocks; pass --no-check-sovereignty to plan it anyway. * fix(redact,plan): a linear two-segment JWT rule, and --check-sovereignty never weaker From the branch's own review. - _JWT_TWO_SEGMENT_RE still started at \b, so eyJ-eyJ-... stayed quadratic (200k characters: 9.75 s through redact_secret_text, 0.30 s now), and the linear test fed eyJ_ (no \b after '_'), so it could not see it. The rule now starts where _JWT_ANCHORED_RE does; _JWT_RE takes two runs as well as three, so a two-segment token glued after '-' is still masked by its header. The test feeds eyJ- too. - _JWT_ANCHORED_RE took up to two more segments greedily, so "got <jwt>.Next" lost "Next". It now takes a JWS (three segments) or a JWE (five). - --check-sovereignty asked the provider's hook first, so a hook that reported nothing passed a contract the new default refuses. It now runs the default gate first. --no-check-sovereignty also says so on the console. - The changelog no longer says a failing plan stage leaves no plan file: one an earlier run left at --out stays. * fix(plan,redact,databricks): the defect scan's findings - fluid plan's sovereignty refusal printed its findings through Rich markup, which took each finding's "[expose_id]" for a tag and dropped it; findings print literally now (here and in --check-sovereignty's report). The refusal also carries the findings as the CLIError's context errors and message, the shape local_plan_validation_failed has: Command Center's plan endpoint surfaces an engine refusal that has them, and fell back to an ungated parse-only plan for one that had not. - _JWT_RE takes up to five segments, empty ones included, so a JWE with direct encryption whose header is written with a space or a newline is masked whole again (origin/main masked its last three segments; the branch left it). Its header check still decides, so names stay readable. - governance_labels' verbatim values spell a YAML true/false as written and make no tag for null: fluid generate iac emits without the schema gate, which allows only strings, and wrote Python's "True". * chore(secrets): refresh the baseline's line numbers after the redactor's edits * fix(redact): find a glued JOSE header in any segment of a dotted run From the security judge's re-check. A token glued after a dotted name (northwind_towers.x-<header>.<payload>) and a header written with spaces or newlines glued to a word (token_eyAi..., x-ewog...) were masked on origin/main, whose rule masked any three 12+ runs, but not on the branch, which looked for a header only in the run's first segment and only at "eyJ". _mask_if_jose now checks every segment of the run and masks from the first that holds a header, keeping the name before it. A glued header starts at an 'e' after '-' or '_' or at an "eyJ" (base64url of '{' always starts with 'e'), six tries at most per segment. _JWT_RE takes the whole dotted run, and _is_jose_header requires the "alg" member RFC 7515 and RFC 7516 (4.1.1) make mandatory, so a short JSON part of a name such as e30 ({}) is no header. 200,000-character adversarial runs stay under 0.5 s. * fix(databricks): the job-run cleaner drops keep_dotted_names, now the default #861's clean() asked redact_secret_text for keep_dotted_names=True, the keyword this branch removes: every log and printed error now keeps a dotted name and masks a JWT by its header, so the call passes nothing extra. Without this, every Databricks job-run message raised TypeError (39 tests in tests/build_runners/test_databricks_job_run.py after rebasing onto #861). * style: one import style per module in the new tests; explicit concatenation in the catalog From the code-quality review of #868. The two sovereignty_violation suggestions split over adjacent literals now use the file's own parenthesised "+" form, and the tests import fluid_build.observability.secret_redactor, the Snowflake logging module and fluid_build.cli.plan one way each. No behaviour change; the baseline moves line numbers only. * test(redact): isolate the JOSE tests from secrets other tests registered Test (Python 3.14) failed test_the_words_after_a_token_are_kept with "got ***REDACTED******REDACTED***.Next step". The exact-match registry is process-wide, and a masking test in the same xdist worker had registered the base64url of bytes(range(32)), which is this test's signature, so layer 1 masked it first and the JOSE rule then masked "header.payload." on its own. The token was masked whole either way; the exact-string expectation was not. Both JOSE test classes now set the registry aside for each test and restore it after (forget_known_secrets / register_secrets). Reproduced with that key registered first: the old file fails 1, the new one passes 55.
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
On Databricks,
fluid apply --mode amend-and-buildprovisioned a dbt build's serverless job and then printed "start a run now from the job's page". A job withexecution.trigger.type: manualtherefore never ran, andamend-and-buildbuilt nothing on Databricks, unlike every other target. An estate applied in lineage order had to call the Jobs API itself.Now
amend-and-buildandreplace-and-buildstart a run of that job, wait for it, and fail the apply when it fails:post_applyhas bound masks and row filters (applied_state). The job is then found by the emitter's tags (fluid_product,fluid_build), never by name: two environments of one contract in one workspace each own a job of that name.run-nowand, at the cap or on an interrupt,runs/cancel. A scheduled job keeps its schedule, and a job created paused stays paused.run-nowcarries an idempotency token and is retried with that same token, so a retry cannot start a second run.queue.enabledis set, so a busy job queues instead of being skipped.replace-and-buildadds--full-refreshto the job's owndbt buildfor that run only (dbt_commands), as the local runner does.Done. PASS=…line is printed. Any other end fails the build with the task'serrorand the last 30 lines of dbt's output. Credentials are masked and escape sequences stripped. The run page is repeated after the failure.amendandcreate-onlyonly provision. A continuous job is not started, because it never ends.Design calls (no new CLI flag)
FLUID_DATABRICKS_RUN_TIMEOUT_SECONDS, the same form asFLUID_TOFU_TIMEOUT_SECONDS.runtime.timeoutis never cut short.--no-wait.-and-buildmeans the build has run when the apply returns. Provision-only is--mode amend.--mode amend, then run--mode amend-and-buildlevel by level in lineage order. The second apply plans no change: live,tofu plan: +0 ~0 -0; the measured cost is about 30 s for ~75 resources. A build-only flag that skips that re-plan would be a new public flag, so it is left as a follow-up.databricks-sdk: refused beforetofuruns (apply_databricks_sdk_missing), only on the OpenTofu engine, and not for a dry run.Borrowed
JobsAPI.wait_get_run_job_terminated_or_skipped: one more second per poll, at most 10 s), Apache-2.0. Rawapi_client.dokeeps the repo's client seam (databricks_enforce.default_client) and the in-process fake.extract_failed_task_errors: the latest attempt of each failed task, itsget-outputerror), Apache-2.0, anddatabricks bundle run's failure output.jobs run-now --no-wait/--timeout 20m(Databricks License). The 20 min default is too short for dbt.bundle run24 h wait. Without a cap, a queued run could hold CI for 48 h.Validation
fluid_iactest_build(droppedCASCADEafterwards, scratch jobs deleted). One upstream table (4 rows) and one dbt product reading it:amend-and-build: created the job (+13 resources), started the run, waited (0:00 RUNNING→0:57 TERMINATED SUCCESS), exit 0. The table has 3 rows.[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column … height_meters cannot be resolvedand dbt's tail, plus the run page.replace-and-buildon a cron-scheduled job: the run executeddbt build --select +sites_summary+ --full-refresh. Afterwards the job still had0 0 3 1 1 ?/UNPAUSEDand its stored commands without--full-refresh.0:00 QUEUED→0:47 RUNNING→ success. A raw probe showed that a queued run carries onlystatus.state: QUEUED, which is why the reason is inferred.tofu.--dry-run: prints a would-start line.amend: no run started.tests/build_runners/test_databricks_job_run.py,tests/cli/test_apply_databricks_job_runs.py, additions totest_databricks_job_builds.py). Real-tofue2e against the in-process fake, which now serves runs:tests/iac/test_iac_databricks_jobs_fake_e2e.py+test_iac_databricks_fake_e2e.py, 54 passed.tests/fragments/test_compat_gate.py: 45 passed..[dev,local]venv,tofuhidden): 34,575 passed and 6 failed.fix(databricks): keep an ignored SIGHUP ignored:.[dev,local]venv has none;tests/test_engine.py'sargs\.scan readbuild_kwargs.updateas a field.test_duckdb_sandbox_workspace_root.pytests (macOS/var, they fail on main as well);test_uses_configured_dbt_command_prefix_verbatim, because hidingtofualso hiddocker. It passes with docker on PATH.Reviews
/security-review, two rounds (Opus judge): no vulnerabilities.
/world-class-inspection (diff): CONCERNS, with 3 P2: a run left going after a failed read, no CHANGELOG entry, a bad timeout value silently ignored. All fixed and re-verified. UX 9/10.
/defect-scan (diff): 9 confirmed (2 Medium, 7 Low), 3 refuted. Fixed and re-verified:
The round-2 re-check found one new Medium, fixed: the SIGTERM/SIGHUP hook overrode a SIGHUP ignored by
nohup.Follow-ups
amend-and-buildon Databricks andFLUID_DATABRICKS_RUN_TIMEOUT_SECONDS. Docs live there, not in this repo.fluid_keystask afterdbt, and itsbuild_after_applyforwardsdatabricks_jobsunchanged. Whichever PR lands second rebases.Documentation
mainform)fluid apply --mode amend-and-build/replace-and-buildon Databricks now run the dbt job and wait for it, and the new env varFLUID_DATABRICKS_RUN_TIMEOUT_SECONDS. The forge_docs PR is a follow-up. Squash auto-merge is armed here, so it may land after this PR. In that case it uses the house marker On forge-climain, not yet in a release (forge-cli feat(databricks): amend-and-build runs the dbt job now and waits for it #861).