Repository navigation
chore(streaming): sync #784 with main as a true merge, restoring main's ancestry - #815
Merged
fas89 merged 30 commits intoNov 4, 2026
Merged
Conversation
Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.14.2 to 1.14.3. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@dc37677...7245906) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.6.0 to 4.7.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@dbcb813...81c6f93) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…Catalog (#785) * feat(databricks): column restrictions and row filters compile to Unity Catalog SQL functions policy.authz.columnRestrictions and rowFilters were refused. They now compile to SQL functions named by a hash of their definition, with the semantics GCP and AWS give the same fields: deny is a mask that returns NULL, mask is last_four, first_four or NULL, one mask per column with a deny beating a mask, and one row filter per table that is the union of its principals' predicates for their members and every row for everyone else. A predicate is parsed as one boolean expression, checked node by node against an allowlist and rendered from the tree. access: allow, a principal SQL cannot test and a text mask on a non-string column are refused. Nothing runs them yet. * feat(databricks): fluid apply binds the masks and row filter around tofu apply, and revokes SELECT when it cannot The provider has no function resource and no mask or filter attribute, so the statements run on the SQL warehouse through the SQL Statement Execution API. pre_apply binds on the tables that exist before tofu apply runs, so a grant the apply adds lands on a protected table; post_apply binds on the tables the apply created, drops a mask or filter this module bound that the contract no longer asks for, and, for a table it cannot bind, revokes SELECT from the principals the contract restricts and exits non-zero. Live bindings are read from the Tables API and only what differs runs, so a second apply executes nothing. A dry run prints the statements and runs none. The stand-in executes the statements and serves the bindings. * feat(verify): fluid verify checks a Databricks table's row filter and column masks The contract's restrictions are Unity Catalog functions bound on the table, named by a hash of their definition. Verify compares the table's live row_filter and each column's mask, from the GetTable it already reads, with the names the contract expects: a binding that is missing, or is another function (a changed predicate is a new name), is CRITICAL and the fix is fluid apply. No restriction in the contract skips the dimension. * test(databricks): a real tofu apply binds the masks and row filter against the stand-in, and fails safe Apply binds the filter and masks and a second apply executes no statement; a table that exists is bound before the apply can grant on it; a changed predicate binds a new function and keeps the old; a removed restriction is dropped and said to widen access; a binding that fails revokes SELECT from the restricted principals only, fails the run, and the next apply binds first and grants again; a failure before the apply stops it with nothing applied. * docs(databricks): the changelog and tested-honestly rows say what is enforced, how, and what is not proven live * fix(databricks): a contract with no restriction needs no databricks-sdk at apply and says nothing without it * fix(databricks): a mask or filter tests every identity as a group and as a user, so a remapped principal is still restricted The test was chosen from the logical principal's prefix. binding.principals may map a group: principal to a service principal's application ID or a user's email, and the group test then matched no one while the grant to that identity still applied: the mask and filter never took effect. Every identity is tested as is_account_group_member(x) OR current_user() = x; a name that is both a group and a user is restricted either way, which errs closed. The prefix only refuses domain: and unprefixed principals now. * fix(databricks): a mask rule belongs to the principal that names it, and the strictest matching one applies One rules[column] kept the last restriction's rule for every masked member, so a nullify for one principal and a last_four for another gave the first the last four characters, a weaker mask than declared. Members are grouped by rule and the function has one WHEN branch per rule, strictest first: deny, nullify, first_four, last_four, so a member of several groups gets the strictest that matches. Each text rule is type-checked against the column. restrictions_for takes rule_per_principal for a platform whose mask tests each principal itself; GCP and AWS keep refusing two rules for one column. * fix(databricks): a tofu apply that fails part way still binds the masks and row filter of the tables that exist A failed apply raised before post_apply ran, so a table it created, its grants and the dbt job that fills it could exist with no mask or filter, and a scheduled run would expose unmasked rows. Before the apply error is raised, post_apply now runs best effort in a bind-only mode (after_failed_apply): it binds what exists, revokes SELECT where it cannot bind, skips a table that does not exist, and drops nothing, since dropping widens access and waits for an apply that succeeded. A failure there is printed; the run still raises opentofu_apply_failed. * docs(databricks): the changelog and tested-honestly rows describe per-identity tests, per-principal mask rules and the bind-only pass after a failed apply * fix(databricks): a function under a hashed name is read back and must be the one this module creates, before it is bound The name fluid_mask_<table>_<column>_<hash8> can be computed from the contract, so anyone with CREATE FUNCTION on the schema could create it first with a body that returns the value unmasked: CREATE FUNCTION IF NOT EXISTS kept theirs, SET MASK bound it and fluid verify, which compared names only, passed. After each create and before any bind the function is read (GET /api/2.1/unity-catalog/functions/{name}) and its owner, body, parameters and return type must be the expected ones, whitespace-normalised; the owner must be the identity the run authenticates as (userName of scim/v2/Me, who CREATE FUNCTION makes the owner). A mismatch is a failed bind, printed with what differs. fluid verify reads every bound function back and checks the same, accepting the table's owner as well since verify may run as someone else. * fix(databricks): a row filter or mask set by hand is never replaced silently SET ROW FILTER and SET MASK overwrite whatever is bound, so a filter or mask an admin set (a function without the fluid_ prefix) was lost without a word when the contract asked for its own. Every table is read and planned before any statement runs, and a live binding this module did not create, where the contract asks for one, is refused naming the table, the column and the live function, with the remedy: remove it, or move its rule into the contract. Nothing is changed on any table. After an apply it is a failed bind (the fail-safe runs). * fix(databricks): a Tables API read that fails for one table is a failed bind for it, and the other tables are still tried _read raised outside the try that triggers the fail-safe, so a transient 429 or 503 on one table aborted post_apply with that table granted and unbound and the rest never tried. Read failures are caught per table, after an apply and after a failed one: a table with restrictions is a failed bind (the fail-safe runs, the error names it) and the loop goes on; one without is a note. A read that fails before the apply still stops the run. * fix(databricks): a failed bind binds a deny-all row filter first, and only then locks the table by revoking The fail-safe revoked SELECT from the restricted principals alone, which misses a user in another granted group and SELECT inherited from the schema or catalog, and the message claimed none of them could read unmasked rows. A failed bind now binds a fixed-purpose deny-all row filter for the restricted identities (a zero-column function, RETURN NOT <any restricted identity>, content-hashed and checked for owner and body like every function): it blocks them whatever the grant path. If that cannot be bound, or would replace a row filter this module did not create, SELECT (and ALL PRIVILEGES) is revoked on the table from every grantee but its owner, listed through the permissions API. The message says exactly what was done and names each grant path still open, a schema or catalog grant a table-level REVOKE cannot close, with the statement to run. * fix(databricks): a row filter that needs a column the apply adds is held by a deny-all filter until then pre_apply skipped a filter whose columns the table does not have yet, and the table exists with rows, so the restricted principals read every row until post_apply. When the real filter cannot bind and the table exists, pre_apply now binds the interim deny-all row filter (the fail-safe's own: a zero-column function, SET ROW FILTER ... ON (), owner and body checked) for the restricted identities, and post_apply replaces it with the real one. A dry run shows it; a hostile function under its name, or a row filter this module did not create, stops the run before the apply. SET replaces a bound filter directly: a DROP first would leave a window with no filter. * fix(databricks): generate iac refuses a module a plain tofu apply would leave unmasked The masks and row filters are bound on the SQL warehouse around tofu apply, not in the module, so fluid generate iac wrote a module that creates the table and grants SELECT with none of them and told the reader to review and apply it with OpenTofu. generate iac now asks the plugin (standalone_module_refusal) before it writes one, and refuses a Databricks contract that compiles any mask or row filter as databricks-enforcement-outside-module, saying that a plain tofu apply would grant unmasked access and to use fluid apply. emit stays pure; fluid apply, plan and validate are unaffected. * fix(databricks): a dbt job that writes a table not yet protected is created paused A continuous or frequent cron job created in the same tofu apply as its restricted table and the grant on it could run and fill the table before post_apply binds the masks and row filter. DatabricksIacPlugin.emit (the apply path; the pure compile validate uses is unchanged) now reads the live bindings of each restricted table a job writes and sets pause_status to PAUSED on the job's schedule or continuous block unless the table exists, its row filter and masks are the expected functions on the expected columns, and each function is the one this module creates. An unreadable workspace counts as not protected. The next fluid apply, once they are bound, unpauses it in place, with no drift from unpausing by hand. A manual job has nothing to pause. plan_notices says why a job is created paused and that run-now works meanwhile. The engine comment that claimed binding happens before any build can fill a table is corrected. * docs(databricks): the changelog and tested-honestly rows describe the verified functions, the deny-all fail-safe, the paused jobs and what is still unproven live * chore(tests): the unused names the code-quality review found are gone An unused import of copy and Optional, two unused helper imports and an unused unpacking target in the Databricks tests; the stand-in's deploy identity is now the one constant the module itself uses.
Bumps [reviewdog/action-actionlint](https://github.com/reviewdog/action-actionlint) from 1.80.6 to 1.81.1. - [Release notes](https://github.com/reviewdog/action-actionlint/releases) - [Commits](reviewdog/action-actionlint@48e36f5...619a81e) --- updated-dependencies: - dependency-name: reviewdog/action-actionlint dependency-version: 1.81.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…her (#745) Dependabot split the 4.39.1 bump into #745 (init) and #747 (analyze). CodeQL fails any run whose steps are on different versions, so each PR alone was red on all three Analyze jobs. This one now bumps both to the same pinned SHA (9b3254085494b55a0f7c4c2b3d2d52cc712ce370, checked against the upstream v4.39.1 tag), and dependabot.yml groups github/codeql-action* so init, analyze and upload-sarif arrive in one PR from now on.
…nTofu apply failure (#786) * feat(doctor): --strict exits 1 when doctor reports "Action needed" fluid doctor printed "Action needed" for a copilot that is not ready and still exited 0, so a CI gate on doctor could not fail. With --strict it exits 1 whenever the summary says "Action needed", in the text report and with --json, whose payload now carries action_needed and copilot_ready. The default exit code is unchanged; --strict has no effect with --features-only, which prints no summary. Found by an external hands-on assessment of 0.26.0. * fix(roadmap): stop claiming a 1.0 release; name every provider in the description fluid roadmap printed "forge-cli v1.0 shipped 2026-04-23" while the package is 0.x and classified Beta. The roadmap now says so and calls its "v1" headings an internal milestone, and a test fails if it announces a 1.x release while the package is below 1.0. The package description named GCP, Snowflake and Local; it now also names AWS and Databricks. Found by an external hands-on assessment of 0.26.0. * fix(apply): a failed OpenTofu apply names its diagnostic and the way on, redacted The runner uses -json, so the error a failed apply printed was the last 800 characters of diagnostic JSON lines, with no hint. The error now reads the first error diagnostic (summary, resource address, provider detail), falls back to the first "Error:" or the exit code, and says that OpenTofu records state as it goes, so fixing the cause and re-running fluid apply normally continues; it names the workdir for `tofu state list`, and only a state-lock error points at `tofu force-unlock`. The raw tail goes to --debug. Every value a CLIError prints is now redacted and has its control characters spelled out, as the other error handlers already did: the new detail field carries provider text, which can hold a credential. Found by an external hands-on assessment of 0.26.0. * fix(cli): error values go through the full secret redactor; a locked plan says how to clear it The top-level CLIError handler ran context values through _console's narrow pattern only (bare password/token/secret/api key followed by : or =), which leaves an AKIA access key id, a Bearer token, a URL password, JSON-quoted and *_secret_key shapes and env-literal secrets in clear. Context values and suggestions now go through redact_secret_text first, the redactor the logs already use. _apply_failure redacts the diagnostic before cutting it at 800 characters, since a credential split at the limit (a URL that lost its @) matches no pattern. A stale state lock first surfaces at tofu plan, which printed only the raw tail; _plan_failure now redacts it and, for a state-lock error only, says to wait for the other run or clear it with tofu force-unlock in the named workdir. Found by the security review of this branch. * test: allowlist the deliberate fake credentials the secret scan flags detect-secrets failed Lint & Format on the AWS-example key and the admin:s3cretpw DSN these redaction tests feed in on purpose. Each value now carries `# pragma: allowlist secret` on every line that holds it, not just the first: detect-secrets reports one finding per value per file, so allowlisting only lines 690/696 and 412/413 would have surfaced the copies at 695/701 and 424 on the next run. Two lines were pre-split so the pragma stays on the value. With the comment appended, black would otherwise rewrap them and leave the pragma on a closing bracket, where detect-secrets does not look.
…792) run_welcome_scan drops any probe result that misses its 0.4s/1.0s budget, and pick_mode takes its default from those results. On a loaded runner the workspace probe overran and test_pick_mode_default_picks_from_product_when_ workspace_has_products saw "ai" instead of "from_product" (PR #782, Python 3.14 coverage leg). Add a deterministic_welcome_scan fixture that lifts the budgets and points the dbt probe at "no dbt" (it sizes its `dbt --version` subprocess from the same budget), and apply it to the four modules whose tests assert on probe results. Exposure audit: with every probe result dropped, 13 tests in those four modules fail; the other test files that reach the scan do not. A new test pins the original failure with a deliberately slow workspace probe. The product budget is unchanged.
…788) * security: validate_ident let a name with a trailing newline through Python's `$` also matches just before a trailing newline, so `_SAFE_IDENT.match("orders\n")` succeeded and validate_ident returned "orders\n" unchanged. Every DDL emitter routes identifiers through it, and a name written into a line-based file (a Java properties file, as in #784) gains a second line. The four whole-string allowlists in _sql_safety.py now end in `\Z`. Only _SAFE_IDENT changes behaviour: the type-name, type-parameter and RLS-expression helpers already strip their input, so `\Z` there keeps a later edit that drops the strip from reopening the gap. tests/providers/test_sql_safety_type_param.py pins validate_ident against "\n", "\r\n" and an embedded "\n", and each pattern against a trailing "\n". * docs(changelog): validate_ident refuses a name with a trailing newline * chore(secrets): move the two _sql_safety baseline hits to their new lines
* ci: freeze dependency weight and startup cost before the diet Guards at today's numbers, so nothing grows while later PRs move litellm, mcp and openlineage out of the base install. - scripts/ci/dep_budget.py and .github/workflows/dep-budget.yml resolve the base install with `uv pip compile` for linux/3.12, linux/3.10 and windows/3.12 and fail above a committed ceiling (88/86/89), naming the packages added and removed against a baseline. It is its own workflow (PRs touching pyproject, main, weekly), because an upstream release can move the count with no change here. - scripts/ci/bare_install_smoke.sh, called from ci.yml build-check and release.yml, caps the bare wheel's build_parser() import cost (869) and runs validate, plan, bundle, generate iac and doctor on an AWS example. - .importlinter-external is an import-linter forbidden contract with external packages: litellm, mcp, openlineage and boto3 stay off the core modules. The 8 lazy routes that exist today are listed one by one. - ruff TID253 bans module-level imports of the heavy SDKs, with 4 exemptions. It does not see an import inside a module-level try:, which the config says. - deptry 0.25.1 joins [dev]. CI runs it as advisory until 0.29.0. - tests/perf/test_startup_budget.py lowers MAX_MODULES from 1100 to 895 (852 on 3.12, plus 5%) and adds a discover_providers() probe capped at 916. Both counts print in the CI log. Borrowed, not built: uv, import-linter (include_external_packages, ignore_imports), ruff flake8-tidy-imports, deptry. * ci: keep the bare-install smoke out of the release build job release.yml's build job holds id-token: write and attestations: write. Running the wheel and its ~88 transitive dependencies (resolved at release time, not hash-locked) through validate, plan, bundle, generate iac and doctor there widened an exposure that `fluid init` already had. A failing doctor check could also block a tag. A tag's merge commit has already run the same smoke on the same wheel build in ci.yml build-check. So release.yml goes back to its --version and init check, and the smoke runs only in CI. * ci: the dependency budget fails loudly and ratchets Fixes from the defect hunt on the budget guard: - A uv failure (a resolver conflict or an index outage) now prints uv's own stderr. CalledProcessError's message left it out. - `name @ url` requirements are counted. Any line the parser does not understand raises, so no package slips past the budget uncounted. - --update compares against the committed baseline before rewriting it, so an over-ceiling run still names the packages added and removed. - A count under its ceiling prints a notice to lower the ceiling. - Pushes to main run the workflow only when pyproject.toml, the script, the baselines or the workflow change, as PRs already did. Upstream drift still shows in the weekly run without turning unrelated pushes red. The bare-install smoke now resolves the venv path to an absolute one, because its probe runs from /tmp, and removes its temp dir on exit.
* fix: install hints name data-product-forge, not fluid-build Doctor, the copilot, the MCP, ODCS and DuckDB errors, the generated Jenkinsfile and the agent docs told users to `pip install fluid-build[...]`. The distribution is `data-product-forge`; every hint now names it with the right extra. - Add an empty `mcp` extra so `data-product-forge[mcp]` resolves today; it is filled when mcp leaves the base install. - tests/test_no_wrong_package_name.py fails on any `fluid-build` not followed by `.` (domains are a separate fence). The allowlist matches a line, not a file (only OTel's service.namespace), and a coverage guard keeps the scan honest. Pattern borrowed from tests/test_no_reserved_tld_defaults.py. * test: catch every shape of a wrong install hint, and missing extras Follow-up from the security review: - An extras-exist check: every `data-product-forge[...]` in the scanned files must name extras that pyproject declares. It failed first on two existing hints. tracing.py now names the OTel packages it imports (opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http), and the CONTRIBUTING.md example names `atlasclient`. Neither extra existed, so pip installed the base package and the feature still failed. - A sentence-final "fluid-build." no longer passes as a domain. - Install commands (pip, pip3, pipx, uv pip) fail on any fluid[-_.]build spelling, since PyPI normalises them to one name. The import package `fluid_build` stays legal everywhere else. - The scan now covers the non-.py files shipped in the wheel and CONTRIBUTING.md, and asserts that it read them. - odcs.py quotes its extra, which zsh rejected unquoted. * fix: the vowl hint keeps its extra, and the secrets baseline follows From the verifier: - `fluid odcs validate` printed the vowl hint through Rich markup, which read `[odcs-strict]` as a tag and dropped it. Users were told to install the base package, which has no vowl. It now prints with markup=False, as generate_ci already does. A test drives the real cprint. - tests/test_litellm_backend.py moved a baselined fake key from line 566 to 607. CI's detect-secrets-hook on changed files would exit 3, so .secrets.baseline is refreshed. The diff is that line number and generated_at only; no secret entry was added.
…#789) fluid diff --baseline read an expose's columns from exposes[].schema, but a 0.7.x contract keeps them in exposes[].contract.schema, so a removed column, a type change or a new NOT NULL column reported "No changes detected.", --fail-on-breaking exited 0, and the fluid forge --refine preview said "No changes vs the existing contract." Columns now come from contract.schema when schema is absent, and a column is NOT NULL when required is true or nullable is false. * fix(diff): fluid diff --baseline sees the columns of a 0.7.x contract * fix(diff): a column is NOT NULL when required or nullable: false says so * test(diff): reuse the shared 0.7.6 fixture and pin which spelling names a nullability finding
* feat!: retire redshift and datamesh-manager as providers `redshift` only forwarded plan and apply to `aws`. Its real job, the Redshift rollback DDL, is dispatched by snapshot name and never went through the registry. Data Mesh Manager is a catalog, already reached through `fluid publish --target datamesh-manager` and `fluid datamesh-manager`. `fluid providers` now lists aws, databricks, gcp, local and snowflake, plus third-party plugins. - datamesh_manager sets __fluid_no_autoregister__, as the exporters did in 0.10.0. RedshiftProvider stays as the rollback-only helper. - The error catalog and the user-facing provider lists name the real built-ins: no azure, and databricks added. AGENTS.md describes Data Mesh Manager as a catalog backend. BREAKING CHANGE: `--provider redshift` and `--provider datamesh-manager` (also through FLUID_PROVIDER) are no longer accepted. They exit 2 with a one-line fix: use `--provider aws`, or `fluid publish --target datamesh-manager`. The fix comes from one map, RETIRED_PROVIDERS, which the global flag check, build_provider and the plan gate share. A plugin that registers either name still wins. * fix: reserve the retired provider names, and serve a redshift platform with aws Fixes from the security review and the defect hunt: - Plugins cannot claim `redshift` or `datamesh-manager`. Entry points load before the built-ins, so a plugin could take a retired name and replace its hint. Discovery now skips such a registration before `ep.load()`, so the plugin's code never runs, and logs a WARNING (`provider_retired_name_reserved`) naming the distribution. An explicit `register_provider(...)` call in code still works. Pattern from llm/provider_plugins._reserved_builtin_names. - A contract that names `redshift` as a platform resolves to `aws` through RETIRED_SUCCESSORS (`builds[].execution.runtime.platform` still allows it). This covers resolve_provider_from_contract, apply._declared_platforms, policy-apply and export. So plan, apply and generate iac agree again, as when RedshiftProvider forwarded to AwsProvider. A `--provider redshift` the user types still gets the hint. - The hint no longer needs provider discovery, so it still shows when the registry fails to load or an allowlist hides every provider. - CHANGELOG: Redshift objects apply through `aws`. The rollback claim is dropped. * fix: retired-name helpers pass a non-string name through From the security re-review: resolve_retired_platform and _claims_retired_name raised AttributeError on a non-string name, which an LLM-drafted contract can carry. The previous commit passed such a value through unchanged. Both now return it untouched. * fix: the reserved-name warning names the plugin an operator must remove The provider_retired_name_reserved WARNING carried the plugin only in a log extra, which the CLI formatters drop. An operator saw the event but not which package to uninstall, as two reviewers found. The message now names the plugin and the retired name, repr'd so control characters are escaped. The evt and plugin extras stay for log parsers. * test: one import style per module in test_retired_providers Code-quality review: fluid_build.providers.datamesh_manager and fluid_build.cli were each imported with both `import` and `from ... import` in this file. The test still checks that DataMeshManagerProvider is exported, now through the module.
…787) * fix(iac): lock OpenTofu state on S3 with use_lockfile The S3 backend was emitted as bucket and key only, so a scheduled apply and a manual one (or an apply and a teardown) wrote the same state with no lock. tofu init now gets -backend-config=use_lockfile=true for an S3 backend when the installed OpenTofu is 1.10 or newer; the emitted module is unchanged, so golden files stay valid. An older or unreadable OpenTofu skips the flag and logs s3_state_unlocked with the fix, and FLUID_TOFU_S3_LOCKFILE=0 opts out (listed in fluid doctor). A workdir kept from before this change recorded the backend without the lockfile, and tofu refuses the changed config ("Backend configuration changed"). init adds -reconfigure only when the recorded backend is this same S3 bucket and key (and region, when the module names one) without use_lockfile; any other difference is left to tofu's own error rather than silently repointed. Pinned by unit tests and an old-then-new apply on one kept workdir against moto. Upgrade note: tofu plan takes the lock too, so every role that runs plan, apply --dry-run, fluid diff or verify --state-drift against S3 state needs s3:GetObject, s3:PutObject and s3:DeleteObject on <key>.tflock, or sets FLUID_TOFU_S3_LOCKFILE=0. Found by an external hands-on assessment of 0.26.0; Trello #738 (the locking half). * fix(iac): the S3 lock opt-out works on a kept workdir; a timed-out tofu is interrupted first tofu reuses the partial backend config a workdir saved whenever init gets no -backend-config override, so once a workdir had initialised with use_lockfile=true, FLUID_TOFU_S3_LOCKFILE=0 (or a tofu too old or unreadable for the flag) left locking on: the operator who needed the escape hatch stayed locked out. init now adds -reconfigure in that case, only when the saved backend is this same bucket and key, so no state moves. A run that hit FLUID_TOFU_TIMEOUT_SECONDS was SIGKILLed, so tofu never released its lock and every later run failed until someone ran tofu force-unlock. _run now sends SIGINT, which makes tofu stop and release the lock, and kills only after a 30 s grace period; the timeout result is unchanged. The lockfile-driven -reconfigure is never added with -force-copy, where tofu would let it win and silently skip the state copy. Found by the security review of this branch. * test(iac): fake Popen, not subprocess.run, where tests stand in for tofu runner._run now starts tofu with subprocess.Popen so a timeout can interrupt it before killing it. Two tests in test_iac_state_drift.py patched subprocess.run, so their fake was bypassed and a real Popen of /bin/tofu failed on CI. They now patch Popen with a stand-in child. * fix(credentials): a project .env cannot turn S3 state locking off FLUID_TOFU_S3_LOCKFILE=0 disables a safety control, so it is classified as protected: the operator's environment may set it, a checked-in .env may not. test_every_fluid_variable_in_the_package_is_classified failed on CI because the new variable was unclassified.
* security: refuse a trailing newline in whole-string validators Python's `$` also matches just before a final newline, so `re.match(r"^[a-z]+$", "orders\n")` succeeds and the caller goes on to use "orders\n". #788 closed this in providers/_sql_safety.py. The same `^...$` plus `.match` shape guarded 36 more validator patterns across fluid_build/, fed by contract YAML (a `|` block scalar ends in a newline), .env values, CLI flags, LLM tool calls and percent-encoded URL queries. 31 of those patterns now end in `\Z`. Two call sites use `fullmatch` instead: the catalog quoting chokepoint, which also covers the BigQuery patterns callers pass in, and the federation workspace id, whose error message prints the pattern. Seen through the CLI before the fix, `fluid generate iac` put `region = "eu-west-1\n"` on the AWS provider, and `branch = "main\n"` on a Databricks job from `?branch=main%0A`. 81 other `$`-ending patterns stay as they are. Their input is stripped first, or they are deny lists and detectors where `\Z` would loosen the check, or they are parsers whose captured groups are what is used. tests/security/test_trailing_newline_anchors.py pins each fixed site with a value and the same value plus "\n". All 39 cases fail on the parent commit. The fix follows Django's username validators (ticket #30257, `$` to `\Z`) and the MCP Python SDK's tool-name check (#3076, `fullmatch`). * docs(changelog): 36 more validators refuse a value with a trailing newline * fix(iac): refuse an AWS binding region with whitespace around it With `\Z`, provider_block_for dropped a binding region of "eu-west-1\n" as no region code. That left the provider on the environment's region and skipped _guard_region_move. Pinning the stripped value is no better. Both sovereignty checkers (policy/sovereignty.py, providers/aws/util/sovereignty.py) compare the region as written, so "us-east-1\n" passes `deniedRegions: [us-east-1]` while the provider would be pinned to us-east-1. provider_block_for now raises UnsupportedBindingError (aws-region-whitespace) for a region code with whitespace around it. generate iac and apply render it as a typed CLI error before anything is applied, and fluid diff reports the inspector error. On main the value went onto the provider with its newline. * security: match the BigQuery catalog and imported-schema patterns whole copilot/catalog/bigquery.py::_validate_ident checked the same `$` patterns with `.match` that the quoting chokepoint now checks with `fullmatch`. So `scope.tables` values and FQN parts such as "orders\n" reached `table_name IN ('...')`. The ODCS and ODPS importers compile the bundled schema's identifier and semver patterns, which end in `$`, and also used `.match`: an imported id "orders\n" was kept verbatim, and "1.2.3\n" counted as semver. All three now call `fullmatch`. The two patterns that keep `$` because their call sites use `fullmatch` (`_DEFAULT_IDENTIFIER`, `_WORKSPACE_ID_RE`) now say so in a comment beside them. * fix: show the refused newline in AWS name and mission path messages These messages quoted the value as '{value}', so a name refused for a trailing newline printed as a valid-looking name split over two lines. They now use repr, which prints any other value exactly as before. * chore: keep the domain slug patterns in step with the profile-name guard The comment on forge_domain_enrichment._SAFE_DOMAIN_RE says it matches forge_copilot_prompts._PROFILE_NAME_RE, and the comment on _ai_setup_storage._DOMAIN_SLUG_RE says it mirrors _SAFE_DOMAIN_RE. The profile-name guard now ends in `\Z`, so both do too. Neither is reachable with a trailing newline: one runs after a disk lookup, the other on stripped input. * docs: drop the validator count and reword validate_id's summary The count in the changelog entry depended on how patterns were grouped. The entry now lists what is covered instead, including the AWS region refusal and the imports. validate_id's summary said it refused a trailing newline "as well", which the shared grammar now does itself. * test: skip the BigQuery validation pin without the GCP SDK bigquery_validation imports google.cloud.bigquery at module load. The iac Stage 1 unit lane installs no GCP extra, so the new pin failed collection there with ModuleNotFoundError. It now skips via pytest.importorskip, the guard tests/providers/test_validation_sql_injection.py already applies to the same module.
…e refuses them (#793) * fix(iac): column types reach Glue and BigQuery faithfully or are refused at validate fluid validate --strict exited 0 while the emitters wrote wrong or invalid column types. Glue wrote 47 of the 79 schema types as `string`, 29 of them scalars with a real Hive type (smallint, int4, timestamptz, real, varbinary, money ...). BigQuery got `ARRAY` / `STRUCT` with no element type or fields through the ODCS fallback, which no BigQuery table accepts, so the apply failed late. `array(string)` lost its element type on both clouds. fluid_build/iac/column_types.py classifies every type once per cloud, and emit and validate both call it, so they cannot disagree: - exact: the scalar maps to its faithful Glue / BigQuery type; - lossy: a fallback the target needs (json/jsonb/variant/super/time/ interval/bignumeric -> string on Glue; geometry -> GEOGRAPHY on BigQuery) is kept with a validate warning, so --strict fails; - refused (kind column-type): array, struct, map, object, record, row and any parameterised structured type on both clouds (the schema cannot declare element types or fields yet), bit(n) with n > 1, unmapped types, and on Glue geography/geometry/geom/point/hll and a decimal Hive rejects. The BigQuery fallback to the ODCS table is gone. _hive_type and _bq_type stay total for fluid verify, fluid diff and the Dataform engine. On a Glue table applied by an earlier version, the next apply changes the column types in place (string -> smallint and so on). That update is not a delete, so --allow-data-loss never sees it. A new optional plugin hook, plan_update_notices(plan), lets the AWS plugin announce it on the dry run and before the apply: one note per table, `column: old -> new`. The CHANGELOG upgrade notes cover this, the verify/diff drift a user sees until that apply, and every new refusal and warning. Found by an external hands-on assessment of 0.26.0. Measured against moto with real tofu: base-applied table, dry-run notice, apply without --allow-data-loss, live types after. * fix(iac): an in-place Glue column-type change needs --allow-data-loss; print plan data literally A Glue column whose type changes in place (string -> smallint on a table an earlier version applied) can stop Athena reading files written under the old type until they are rewritten; reverting the type restores reads. The AWS plugin now implements plan_data_changes: each aws_glue_catalog_ table update whose column types differ is listed, `column: old -> new`, and the data-loss gate refuses the apply (and the dry run) until --allow-data-loss. A whitespace-only difference (decimal(10, 2) -> decimal(10,2)) never trips it. The advisory plan_update_notices hook from the previous commit is gone; the gate carries the list. Like Databricks, an AWS apply with in-place updates whose saved plan cannot be read back now needs the flag too (fail-closed). Plan data quotes live catalog values, so the gate list and plan notices print through cprint_literal: a live type "string[/]" no longer crashes every apply with a Rich MarkupError, and "[link=...]" no longer renders a hyperlink. Also from review: type parameters must be ASCII digits (decimal(١٠,٢) was classified exact and emitted as written); a BigQuery decimal beyond NUMERIC's range (scale > 9 or more than 29 integer digits) warns and suggests bignumeric; a malformed binding is skipped by the type check instead of raising and silently skipping the governance checks after it. Measured against moto with real tofu: refused on dry run and apply without the flag (live table untouched), applied with it; whitespace-only applied without it.
…and its Lambda code takes contract values as literals (#794) * fix(aws): the planner keeps its action order, so the emitted module is deterministic DependencyGraph.nodes was a set, so the order of independent actions, and with it the emitted module, followed Python's hash seed. The IaC emitter interpolates a Lambda's attributes only when the function was emitted first, so a Lambda permission, schedule target, event-source mapping or S3 notification named its function by literal ARN in one run and by ${aws_lambda_function.<name>.arn} in the next, and tofu sometimes had no ordering edge to the function. nodes is now an insertion-ordered dict. Golden check on 21 AWS contracts (examples, test fixtures, synthetic scheduled ones), base under three hash seeds against this branch under two: modules without a Lambda are byte-identical; scheduled ones flipped on base and are stable here, always emitting the reference. * fix(aws): a schedule is never dropped silently, and the placeholder it emits says so The workflow Lambda was named fluid-workflow-<id> raw, so a dotted id (the schema's example and 27 of 28 examples use dots) or one over ~49 characters failed the AWS plan validator, and native_actions swallowed that refusal at DEBUG: generate iac and apply logged "Planned EventBridge schedule" and wrote no Lambda, permission or scheduler, exit 0. - Every AWS name the schedule planner derives from a contract id (Lambda functions, permission statement ids, the schedule, event rules, the state machine, the MWAA environment) goes through bounded_aws_name: disallowed characters become '-', and past the service limit the name is cut and ends in a hash of the whole id, so long ids stay apart. Each rule cites the AWS API reference; every name starts with a fixed letter prefix. - A plan its own validator rejects raises PlanValidationError, which native_actions re-raises by type through the cause chain, like a sovereignty refusal: generate iac exits 1 and apply raises. A planner that could not run (no SDK) stays best-effort. - An AWS account that cannot be resolved is refused by name (AwsAccountUnresolvedError) before any name is derived from it, instead of surfacing as an invalid bucket "None-fluid-staging". Offline generate iac keeps its placeholder account. - The scheduled Lambda only logs its tasks and both roles it names are never created, so every plan that emits it warns schedule_placeholder_lambda, naming the roles. A real AWS scheduling path stays on Trello #875. Found by an external hands-on assessment of 0.26.0; Trello #875. * fix(aws): generated Lambda code takes contract values as literals; names never collide From the security review of this branch: - The workflow and trigger Lambdas' Python source was built by splicing contract values into string literals (TASKS = json.loads('''...'''), 'contract_id': '<id>', FunctionName='<name>'). A schema-valid orchestration task param such as '''+__import__('os').system('id')+''' closed the literal and ran on every schedule tick under the workflow role; an id with a trailing newline (which the id pattern admits) made a Lambda that could not import. Pre-existing on main. Every contract value now reaches the source only as a repr() literal; tests parse each generated source, walk its calls and round-trip hostile ids and params. - bounded_aws_name appends a hash of the raw name whenever sanitising changed it, so a.b and a-b no longer share one Lambda in an account. A name the service already accepts is unchanged, so nothing deployed is renamed. - The orchestration and schedule-planner failure paths raised a plain ProviderError that native_actions swallowed, still dropping the schedule with exit 0; they raise PlanValidationError now. - The S3 bucket rule allows dots as S3 does (no '..', '.-', '-.'), so a legal dotted bucket no longer fails generate iac now that validator refusals are fatal. - An unresolved account with boto3 missing says to install data-product-forge[aws] instead of blaming credentials. * test(iac): license header on the action-order test The test file split out of test_iac_aws_schedule_names.py had no Apache header, which the License Headers check requires. Added with scripts/add_license_headers.py. * fix(aws): an unresolved account refuses only a contract that needs the native plan The account check ran first in AwsProvider.plan() and refused every contract, so fluid apply of a schedule-less AWS contract failed when boto3 could not resolve the account (no [aws] extra, no STS access) although tofu resolves it itself and main applied such contracts fine; CI's test_state_backend_env_default.py caught it. Locally the tests passed because this machine's credentials resolve STS. The sovereignty check now runs first (the only place AWS enforces jurisdiction, and it needs no account). Then, with no account, a contract that declares orchestration (a schedule, triggers or tasks) is refused by name, since its Lambda, permission and schedule come from no other plan; any other contract gets a plain ProviderError, which native_actions treats as a planner that could not run, as before. * perf(aws): keep the new plan-error imports off the fluid --help path provider.py is loaded by fluid --help, and its new module-scope imports pulled _plan_validation, plan.schedule and util.config onto that budgeted path. They are now imported inside the methods that use them, so build_parser() loads exactly the fluid_build modules it does on main.
…gh a pattern (#799) * security: schema validation let a value with a trailing newline through a pattern JSON Schema writes `pattern` in ECMA-262, where `$` matches only at the end of the value. jsonschema runs it with Python's re.search, whose `$` also matches before a final newline, so `fluid validate` accepted `id: "orders\n"` (what an `id: |` block scalar produces) under every bundled schema, 0.7.1 to 0.7.7. An editor validating the same schema in JavaScript refuses it. schema_manager.schema_validator(schema) extends the declared dialect's validator with a `pattern` keyword that rewrites each `$` anchor to `\Z`. Contract and fragment validation (_validate_with_jsonschema) and bundle validation (forge.core.validators.validate_json_schema) use it. The schemas are unchanged, so cached and fetched ones are covered too, and forge_docs, the fluid spec and Command Center's models need no change. When the line break is the only problem, the error says so. patternProperties keys keep jsonschema's re.search, as its own additionalProperties and _locate_error read them. No FLUID schema uses the keyword, and a test fails if one starts to. The catalogAuth token check, `not: {pattern: ...(?:_|$)}`, now reads `$` as ECMA-262 does too; the name pattern beside it already refuses any value that ends in a newline, pinned in test_catalog_auth_schema.py. Borrowed: the keyword override from check-jsonschema (jsonschema.validators.extend), and the `$` to `\Z` rewrite from fastjsonschema 2.21.2 and js-regex 1.0.1. Their (?<!\\)\$ finder turns `[$]` into `[\Z]` (bad escape) and misses `\\$`, so this scan skips escapes and sets. Conformance case from JSON-Schema-Test-Suite draft2020-12/optional/ecmascript-regex.json. * security: bundle validation stays an error when jsonschema cannot be imported Review follow-ups for the pattern end-anchor change: - validate_json_schema checked importlib.util.find_spec, which finds a jsonschema that then fails to import. schema_validator then raised NameError, reported as a SCHEMA999 warning when not strict, where the base returned the SCHEMA-LIB-MISSING error. It now reads schema_manager.JSONSCHEMA_AVAILABLE, the flag schema_validator needs. - The schema_validator docstring says FLUID-schema validation goes through it, and names the second place jsonschema's own re.search remains: a subschema that declares its own $schema, or a $ref to the root. - CHANGELOG: a sink.catalogAuth field gets no line-break hint, since its value is never shown. - Tests: the library-missing error, and a not-pattern reading $ as ECMA-262 does. * test: one import style per module in the end-anchor tests The code-quality check flagged re and fluid_build.schema_manager as imported with both import and from-import. re now comes in as `from re import sub`, and the jsonschema-missing test patches the flag by its dotted name.
…ff the clock (#800) run_welcome_scan documented a 50ms scan that stays within _TOTAL_TIMEOUT_S, but the budget only decided which results were kept: the ThreadPoolExecutor context joined every probe thread. Run from a directory that holds many repos, the specialization probe's rglob took 177s and the whole scan with it. Each probe's wait window also started when the collector reached it, so a cold first import counted against it. On a loaded machine the AI probe (a lazy pydantic import inside the AI-config load) missed its window and the scan reported "AI not configured" although it is. - Probes run on daemon threads, one Future each. The scan returns at _TOTAL_TIMEOUT_S and abandons stragglers. An executor is deliberately not used: its non-daemon workers are joined at interpreter exit, so an abandoned probe would keep the CLI alive. - _warm_probe_imports imports the modules the probes load lazily while the probes run, then the budget starts. A test fails if a probe imports something that is not listed in _WARM_MODULES. - _PROBE_TIMEOUT_S is removed: once the probes start together it can never bind before the total. _TOTAL_TIMEOUT_S (1.0s) is the single budget. The dbt probe gets its own _DBT_VERSION_TIMEOUT_S (0.32s, was 0.4 * 0.8), which must stay below the budget so the child is killed by its own timeout rather than orphaned. - scan_duration_ms uses a monotonic clock and covers the whole call. - The "<50ms" claims (module docstring, forge_modes comment, CLAUDE.md) are corrected, and the deterministic_welcome_scan fixture drops the removed constant. - tests/test_welcome_scan_budget.py pins the cap, the exit behaviour of an abandoned probe, the cold-import accounting, the warm list and the dbt cap. - Review follow-up: _spawn_probe records only Exception; SystemExit and KeyboardInterrupt end the probe thread and are dropped at the deadline.
* fix(apply): refuse an unusable build id before tofu apply `fluid apply --mode amend-and-build` (and replace-and-build) checked contract.id and builds[].id only in the build runner. On the OpenTofu engine the runner starts after `tofu apply`, so an id the schema admits but the build phase refuses (1orders, more than 128 characters, a trailing newline) was refused after the product was provisioned, and rendered as "Unexpected error". The runner's rule is now one function, build_runners/_ids.py validate_build_ids, and apply runs it right after the schema gate in a build mode, ahead of the apply hooks, tofu and the runner, on both engines. The refusal is a typed CLIError, apply_build_identifier_invalid, that names the field and the value and says nothing was applied. The runner keeps its own check as the backstop. * docs(changelog): build ids are checked before tofu apply * test(apply): import fluid_build.cli one way in the build-id test
…ement follow-ups (#801) * fix(databricks): the fail-safe lock never revokes from the deploy identity and says when the owner could not be skipped * fix(databricks): the lock names a restricted table owner as a path it cannot close * fix(databricks): the fail-safe says when the permission listing cannot show inherited grants * fix(databricks): the pause decision is made once per run, and verify flags a job paused for an unprotected table * fix(databricks): post_apply drops this module's superseded functions that no table in the schema uses * fix(databricks): a residency failure prints one line with the declared region, the live region and the fix * docs(databricks): HONESTLY_TESTED records the 2026-11-03 manual live run and what it left open * docs(databricks): CHANGELOG entries for the masks and row filter follow-ups * fix(databricks): the orphan drop leaves a function the deploy identity does not own * feat(databricks): the acquisition runner lands bronze into the Unity Catalog table A DuckDB acquisition build writing a `platform: databricks` expose no longer falls back to a local file. The runner masks at landing as before, writes Parquet, uploads it to DATABRICKS_STAGING_VOLUME through the Files API, loads it with one INSERT OVERWRITE ... SELECT <contract columns> FROM read_files(...) on DATABRICKS_WAREHOUSE_ID (the table's definition, grants, tags, row filter and masks stay), checks the row count, and deletes the staged files whether or not the load worked. A missing table fails naming `fluid apply`; the runner never creates or alters a table. Because the runner applies the mask before the data leaves the machine, the emitter now accepts policy.privacy.masking on an expose that only DuckDB acquisition builds write, and still refuses it for dbt and every other writer. * feat(databricks): lifecycle.expire is accepted on a full-refresh table whose Delta history fits the retention A Databricks table rebuilt in full every run holds no row older than a run except in Delta history, so expire is accepted when every build writing the expose is a full-refresh writer and lifecycle.retention covers the effective history (the larger of delta.logRetentionDuration, 30 days by default, and delta.deletedFileRetentionDuration, 7 days by default, or a dbt model's tblproperties). Full-refresh writers: a dbt build that selects a model resolving to materialized=table (SQL config(), then the model's YAML config, then dbt_project.yml, default view; anything not a plain literal is refused with the fix), and a DuckDB acquisition build, whose Databricks landing overwrites the whole table. ISO 8601 durations count years and months at their shortest, so a too-short retention never passes. New refusals: databricks-retention-incremental, -not-table, -dbt-unresolved, -too-short, -period; -unmapped stays for every other writer. fluid verify gains a retention dimension: it reads the table's properties and fails when the live history exceeds the contract, and warns when predictive optimization is off (VACUUM would then need scheduling). * fix(databricks): the landing reads only its own file and deletes it on an interrupt read_files now names the one Parquet file this run uploaded, not the run directory, so a file someone with WRITE VOLUME drops there is not loaded. Ctrl-C or SystemExit during the upload or the INSERT cancels the statement on the warehouse (SqlRunner) and still deletes the staged files before the interrupt propagates. A run killed outright cannot clean up, so each load first deletes what earlier runs of the same build left under <volume>/<product>/<build>/ and records it as facets.databricks_load.earlier_runs_staging_deleted. * fix(databricks): close the review's low findings on the landing, retention and orphan drop - the landing's error names the files it could not delete first, so a shortened message keeps them, and logs them in full - the dbt resolver does not walk a model path that reaches outside the project through a linked folder on the way - the orphan drop lists tables with include_browse, so a table the run may only browse makes the schema unreadable (nothing dropped) instead of a binding going unseen - verify's retention dimension warns when GetTable serves no properties, instead of passing on Delta's assumed defaults * docs(databricks): say why the landing uploads with one Files API PUT * refactor(databricks): the landing reuses the runner's path-part helper, and the orphan drop's per-function step is its own function * fix(databricks): close the inspection's findings on the landing, retention and docs - the INSERT OVERWRITE lists its target columns, so a live table that keeps its columns in another order (one fluid apply appended) gets each value in the column of its name, not by position - the stale-staging sweep deletes an earlier run's directory only when every file in it is over an hour old, so a concurrent run keeps its upload, and it follows next_page_token - retention accepts a DuckDB acquisition only with a source mode the landing runs (LOAD_MODES, full_refresh): another mode is databricks-retention- incremental, none databricks-retention-unmapped; the too-short message shows the declared Delta properties, not the defaults - the dbt config scan reads Jinja tags, comments and raw blocks in linear time (the lazy regexes were quadratic on unclosed openers), matching them on 200k random inputs - the runner and landing use public scrub/is_not_found names, and retention reuses util.contract.is_duckdb_acquisition_build - a DROP FUNCTION that fails is named and the apply still succeeds (tested) - CHANGELOG and HONESTLY_TESTED say what the code now does: masking at landing, orphan drop, lock identity, the second live run, counts * fix(databricks): the masking refusal starts its reason as a sentence * feat(databricks): an accessPolicy grant scoped by resources lands on the tables it names The emitter refused every grant with resources (databricks-grant-scope-unmapped), which blocked products whose grants carry $.exposes[?(@.exposeId=='<id>')]. iac/access.py now parses resources once (AccessGrant.scope, parse_grant_scope): one expose by id (either quote, spaces allowed) or every expose ($.exposes[*], $.exposes). The Databricks emitter grants only on the matching tables. Any other JSONPath, alone or beside a good one, and an exposeId the contract does not have stay refused with a message that names the resource; covers() is false for an unevaluable scope, so a refused grant is never placed on more tables. AWS and GCP emit what they did: GCP still applies every grant to every expose. * fix(databricks): close the defect scan's and the second security pass's findings - fluid validate leaves a table name, retention or source mode that is still an {{ env.X }} placeholder to the emitter (it refused contracts fluid apply accepts), and fluid verify compiles the contract with its placeholders substituted, as apply does; a placeholder retention promises nothing - the emitter refuses what the DuckDB acquisition runner would refuse at run time after apply created the table: the source mode and sink format (databricks-landing-unsupported) and a masking rule it cannot apply (databricks-masking-invalid) - the stale-staging sweep deletes only directories named like a run id that hold nothing but Parquet files, all over an hour old; an empty directory, a subdirectory or another file means the directory is left - a blank accessPolicy resource is unevaluable, never "every expose" - _contract_key returns None for a contract JSON cannot key (mixed-type or date keys, a cycle) instead of crashing emit, and no pause decision is reused then - dbt fqn selectors match as dbt does: a prefix of the fqn with or without the project's name (shop, marts, marts.orders) - the too-short fix rounds the history up to whole days; a COUNT that fails after a successful INSERT says the load ran; the lock names an owner that is also the run's identity once * docs(databricks): CHANGELOG for the placeholder, landing, sweep and grant-scope fixes * fix(databricks): match dbt selectors as dbt-core does, and refuse an unresolved landing mode at apply - _fqn_selects ports dbt-core 1.11's is_selected_node / node_is_match (Apache-2.0): exact leaf name, else an exact fqn prefix up to the first wildcard part and one fnmatch of the rest, with and without the project's name. The per-part glob it replaces accepted ord*, ma*ts and *.core, which dbt does not select, so lifecycle.expire could be accepted for a model the build never rebuilds. Checked against dbt-core itself on 34,944 selector/fqn pairs. - _refuse_unlandable skips a placeholder mode or sink format only while validating; at apply one still unresolved (its variable unset) is refused before the table is created. * chore: refresh the secrets baseline's line numbers * test(databricks): one blank line between the license header and the docstring, as the writer makes it * refactor(databricks): scrub and is_not_found are the enforce module's public names; drop a pass-through lambda and repeated json imports
…'s ancestry The previous sync (#798) was squash-merged, so b6b901f has one parent and main never became an ancestor of feat/managed-flink-iceberg-streaming. Once main moved on, #784 went CONFLICTING and GitHub stopped running its pull_request workflows (CI, Flink Iceberg preview, actionlint). This is a real merge commit of origin/main (61ccba5) into b6b901f. It brings in #787, #793, #794, #796, #797, #799, #800, #801 and #803. Merge base. #798's head (4d37c2c) has the same tree as b6b901f and descends from a true merge with e59e325 (#789). So the feature's delta on top of main is e59e325..b6b901f. git's own base, b2d05c0, predates #798's merge. Against it, 11 files conflict, because the feature side still carries their #789-era text. Against e59e325 the merge is clean. Outside those 11 files, both bases give the same tree. The recorded tree is the clean merge against e59e325 (8e4c1af). Resolutions, all 11 files git reported as conflicted: - Took main's version, unchanged. The feature did not touch these since e59e325, so each conflict came from the squash alone: CHANGELOG.md, HONESTLY_TESTED.md, fluid_build/cli/_verify_databricks.py, fluid_build/iac/providers/databricks.py, fluid_build/iac/providers/databricks_enforce.py (add/add), tests/conftest.py, tests/iac/_fake_databricks.py, tests/iac/test_iac_databricks_enforce.py (add/add), tests/iac/test_iac_databricks_fake_e2e.py and tests/test_forge_mode_picker.py. - fluid_build/cli/verify.py: kept both sides. Main's #801 adds the retention dimension and the warn status to _render_findings_dimensions. #784 adds --streaming-timeout and the managed-streaming branch in run(). The hunks do not overlap. Not conflicted: - fluid_build/api/changelog_rules.py stays byte-identical to #798's judge-verified resolution. Main has not touched it since #789. - Both sides changed these files since e59e325, and git merged them hunk by hunk with no overlap: README.md, build_runners/base.py, cli/apply.py, cli/plan.py, iac/providers/aws.py and schema_manager.py. _report_externally_run_build keeps #784's contract argument, and its only caller passes it. Main's #803 build-id gate sits after #784's early streaming return in apply.run(), as each side placed it.
…rry a credential to another origin (#804) * refactor: one HTTP client in the base install, httpx The base install carried three HTTP stacks: httpx, requests and aiohttp. Every site now uses httpx, which is already a base dependency. - fluid_build/util/http_client.py: new_client and new_async_client keep requests' behaviour (up to 30 redirects, environment proxies, REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE, TLS always verified). RetryTransport gives urllib3 Retry semantics. httpx's own transport retries cover only connection failures, and httpx-retries (MIT) would add a base dependency, so its shape is adapted here. - Ported: - util/network - observability/reporter: refusing a foreign redirect is now a response hook that runs on every hop - cli/_command_center: HEAD does not follow redirects and GET does, as before - cli/marketplace - the Command Center catalog connector: aiohttp becomes httpx.AsyncClient with aiohttp's limits - the Data Mesh Manager session: GET/PUT/DELETE retried on 429/5xx with backoff, honouring Retry-After - local mocks - util/network.safe_request reported status_code None for every 4xx and 5xx, because the error response is falsy. It now reports the status. - requests leaves [project] dependencies. It still installs through openlineage-python and tiktoken, so the package count drops when those leave the base install in 0.28.0. - Tests move from requests mocks to respx and httpx. The new tests/test_http_client.py covers redirects, retries, proxies and TLS against a loopback server. * fix: redirects are judged on the URL httpx follows, and the Data Mesh Manager key stays on its origin Security review of the httpx port: - refuse_foreign_redirect resolves the Location as httpx does. A Location with a scheme but no host (https:/x) used to pass as same-origin while httpx sent the key and body to the default port. - Retry-After is capped at the 120 s backoff ceiling; an unparseable or overflowing value is ignored rather than crashing the CLI. - The Data Mesh Manager session refuses a cross-origin redirect instead of forwarding x-api-key (older than the port); the error names the new origin and says to update DMM_API_URL. - A missing REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE disables the reporter with a warning naming the variable; TLS is never relaxed. - The catalog connector keeps the system trust store aiohttp used. Drift from requests found by the defect hunt: - a non-JSON 200 from a registry falls back again instead of crashing; - params merge with the URL's own query string (merge_query); - safe_request raises only on 4xx/5xx and translates allow_redirects; - the two SSRF wiring tests fail again when the gate is removed; - the catalog connector's search reads the real SearchFilters fields (broken before the port, hidden by a test shim). * test: the redirect hook agrees with where httpx follows, for 427 Location spellings * chore: refresh .secrets.baseline line numbers * chore: no silent except in the ported HTTP sites
…805) * fix(diff): validate both contracts before fluid diff --baseline compares them The diff rules read only the shapes a schema allows, so a contract no schema accepts could hide a change from --fail-on-breaking: a column retyped from decimal(18,2) to the number 18 passed with exit 0. compare_contracts now validates both sides first, as fluid plan does: the JSON Schema of each contract's own fluidVersion, offline, from the bundled schemas. It raises the new ContractValidationError, which lists every error of every invalid side. A pre-0.7 contract is refused with the upgrade fluid validate asks for. validate=False keeps the old behaviour, and the fluid forge --refine preview uses it, because the contract being refined may be the invalid one. fluid diff --baseline maps the error to diff_contract_invalid, exit 2 (diff(1)'s "trouble", never 0 or 1). It logs each schema error, names each invalid file with its fluid validate command, and prints or writes nothing. Prior art: datacontract-cli's breaking command parses both contracts into typed models and refuses an invalid one (HTTP 422). * fix(diff): carry a 0.7.x contract's quality, sensitivity and primary key A 0.7.x contract keeps them under exposes[].contract, where the diff did not look, so the gate missed a quality rule made failing, any sensitivity change, and a primary-key column made nullable (reported non-breaking). - Quality: contract.dq.rules (by id), contract.quality (by rule) and, from 0.7.6, each column's quality (by id) are read beside the pre-0.7 quality.tests. An escalation is judged by fluid_build.severity.is_error, so critical fails as error does: warn -> critical is now an escalation and critical -> error no longer is. - Sensitivity: a column whose sensitivity becomes or stops being pii or phi (the set docs_build and the MCP output port use) is column_pii_added / column_pii_removed, reported at .sensitivity. Any other change is the new column_sensitivity_changed (info). - Primary key: a column tagged primary-key (or primaryKey) is in the key, the tag the ODCS exporter turns into primaryKey: true, so one that stops being required is primary_key_nullable (breaking). The PK predicate and the sensitivity block move out of _primary_key_set and diff_columns, which keeps both under their pre-change complexity. * fix(diff): read the columns of a schema wrapped as {fields: [...]} _verify_reconcile, forge_copilot_drift_guard and tools/contract_tests accept an expose schema written as {"fields": [...]}; diff_columns read the mapping's keys instead and saw no columns, so a removed column went unreported. The wrapper is now unwrapped at schema and contract.schema alike, and its columns are reported at ...schema.fields.<column>. No bundled schema accepts the wrapper, so fluid diff --baseline refuses such a contract as invalid; the unwrap serves the unvalidated callers, the fluid forge --refine preview among them. * fix(diff): print contract names as data in text, Markdown and JSON A column or expose name reached the terminal and Markdown raw. In text it could drive the terminal (an escape sequence, an OSC 8 link to another URL, a carriage return or line feed that forges a line, a bidi control that reorders it), and Rich read [bold] as a style, swapped 😄 for an emoji and raised on [/], which crashed the command. In Markdown a name could add a link, an image or an HTML comment that hides the rest of a PR comment, or mention someone. - render_text spells every non-printable or invisible character as repr spells it (\x1b, ) and doubles a backslash, and fluid diff prints it literally (cprint_literal) and soft-wrapped, as the drift report is, so each entry stays one line. - render_markdown puts the path and the description each in one code span. A backslash escape is not enough: GitHub's renderer still links \@user, \#1, \:smile\: and foo\@example\.com (checked against its /markdown API); a code span is literal on every host. The fence and padding follow mdformat's code_inline (MIT). - JSON and Markdown are printed verbatim (cprint_json): Rich wrapped a long line at 80 columns in CI, which put a line break inside a JSON string, so --format json could print JSON that does not parse. - The fluid forge --refine preview prints the same text without markup. * fix(diff): redact each value, then write JSON and Markdown verbatim From the security review of this branch: cprint_json ran the console's redaction regex over the finished document, and its value pattern ([^\s,;]+) took the closing fence of a Markdown code span with it. A column named "[a](https://e.x) token=z" (or ending in token=, token:) closed its path span early, the next fence opened the description, and the description rendered as live Markdown: a link, an image, a mention. The same pass took the closing quote of a JSON string, so --format json could print JSON that does not parse. fluid diff now redacts every value of the report first, with the redactor logging uses (observability.secret_redactor.redact_value, a wider set of secret shapes than the console's), and writes the JSON and Markdown documents verbatim. No regex runs over a finished document, so a dangling "token=" cannot reach a fence either. Text keeps cprint_literal, whose pass cannot break a line's structure. Tests render each name through the CLI and parse the Markdown with markdown-it-py, and check a credential in a name or description is redacted while the JSON still parses. * fix(diff): refuse a contract whose fluidVersion is not a version From the architecture review: detect_version raises on a fluidVersion that is not a version (7, "garbage"), and the error escaped compare_contracts' gate, so fluid diff --baseline exited 1 (diff_failed), which a pipeline reads as "breaking". Any failure to validate a side is now that side's error, raised as ContractValidationError, so the CLI exits 2 (diff_contract_invalid) as for any other invalid contract. * docs(diff): --baseline help names the validation; correct two docstrings From the documentation review: fluid diff --help did not say that both contracts must be schema-valid or that an invalid one exits 2 before any output, which only the CHANGELOG said. The --baseline help says so now. The _code docstring states its padding rule as the code applies it, and the module docstring no longer places the renderers in cli/diff.py. * fix(diff): a duplicate rule id cannot hide a severity escalation From the codebase review: nothing makes a quality rule's id unique, and _rules_by_name kept the last rule of each name, so a new contract with dq.rules [d1 error, d1 warn] against a baseline [d1 warn] reported no escalation, and --fail-on-breaking passed. A baseline [d1 error, d1 warn] against [d1 error, d1 error] was reported as one. Each name now keeps all of its rules, and each version is judged by its most severe rule of that name, in every location the diff reads. * test(diff): pin the code-span padding and the defaults the review found untested From the test-suite review (mutation probes): the space-at-both-ends and backtick-at-both-ends cases embedded the name in a longer path, so they never reached _code's padding branch, and a mutant that dropped it survived. _code is now tested on the raw strings, parsed with markdown-it-py. Also pinned: a quality test with no severity counts as warn; a column rule named by name (no id) is compared; an invalid baseline alone is named in diff_contract_invalid; the pre-0.7 per-column primaryKey flag is still read. * fix(diff): the defect scan's findings in the diff engine From the scoped defect scan of this branch (hunters, skeptic, referee): - A read-only or missing HOME made fluid diff --baseline exit 1, as for a breaking change: the schema manager creates ~/.fluid/schema_cache when it is built. It now falls back to a temporary cache; the diff validates offline against the bundled schemas, so nothing a bundled version needs is lost. - A column the dbt engine builds as a key (engines/dbt/_test_mapping.py column_is_key: semanticType identifier or primary_key, a labels constraint of primary_key or unique, the key flags) is in the primary key, as a primary-key tag already was, so one that stops being required is primary_key_nullable. The predicate is mirrored, because importing the engine loads every engine; a test pins the two together. - A PII change reported at .sensitivity carries the sensitivity values, not the derived booleans, and a pre-0.7 pii: true rewritten as sensitivity: pii is not a change. - The blank Braille pattern (U+2800), a printable symbol that draws as a space, is spelled out like the other characters that draw nothing. * fix(diff): one redacted, JSON-safe envelope for stdout and --out From the scoped defect scan of this branch: - The per-value redaction used the logging redactor alone, which misses shapes the console pattern caught on main: "api key: x" with a space, a separator padded with more than eight spaces, a very long value. A value now goes through both, as the CLI's error renderer does (_redact_str(redact_secret_text(...))). - --out was written from the raw report while stdout was redacted. Both now get the same redacted envelope (_safe_report). - A value JSON cannot hold (a date, bytes or a set that YAML put in a column's quality rule) made json.dump raise after the text report had printed, left --out cut short and exited 1, as for a breaking change; a NaN was written as a bare token. Such values are written as text. * refactor(diff): judge a key with the dbt engine's own predicate, not a copy The diff mirrored engines/dbt/_test_mapping.py::column_is_key (and its is_truthy) because importing the engine loads every engine; ripwire's quality delta flagged the 141-token copy. column_is_key, is_truthy and _PK_KEYS move unchanged to fluid_build/util/contract.py (stdlib-only), _test_mapping re-exports them so every _tm.* caller is unchanged, and _in_primary_key is the ODCS tag check plus that one predicate. The parity test becomes an identity check. * fix(diff): redact a credential held in a mapping key, too From the round-2 review: _safe_report redacted every string value, but a mapping's keys went out as written, and a removed column's free-form labels reach before/after. A key such as "password: x" was printed in --format json and written to --out; on main the whole-text pass of the console had masked it on stdout. Keys now go through both redactors.
… are reserved behind an operator grant (#807) * feat(iac): IaC plugins load on first use, and built-in platform names are reserved A fluid_build.iac_providers entry point is no longer imported with fluid_build.iac. Its name is read with importlib.metadata, and the plugin is loaded, by name, when a contract's platform or --provider needs it (the stevedore DriverManager shape). The allow/block lists and requires_cli are checked before the load. An entry point that claims a built-in platform or alias is skipped before import, with a WARNING naming the distribution. The operator can grant an override with a distribution-qualified FLUID_PLUGINS_ALLOWLIST entry (<dist>:<name>, plugin_manager.override_granted). Retired names are never granted. * fix(iac): grants match the real distribution name, policy matches every spelling, and the state guard sees unloaded plugins Security review and defect hunt on the lazy IaC plugin registry: - override_granted reads ep.dist.name (PEP 508 fullmatch) instead of the first word of "<name> <version>", so a self-declared "acme-fluid-aws spoof" no longer satisfies the grant acme-fluid-aws:aws. - Allow/block entries match case-insensitively with - == _, so an entry point named Kubernetes or on-prem cannot slip past a blocklist written kubernetes or on_prem; policy, reserved names and lookup share one key. - The state-sharing guard loads every admitted plugin before attributing state, so an --allow-data-loss apply on a shared key still sees a plugin cloud's resources that nothing loaded earlier in the process. - Distribution and entry-point names are repr'd in the reserved-name hint and the unavailable reason, so a name cannot inject escapes. - A grant cancelled by a bare allowlist says so; two distributions shipping one platform name are refused rather than last-one-wins. - get_iac_plugin re-checks under the lock (one instantiation per platform) and is stable for any spelling; reset restores a built-in that a grant replaced. - Docs: requires_cli is checked right after the import, and the global --provider flag accepts built-ins only (auto-detect reaches plugins). * chore(iac): assign the entry-point index cache on its own line
… extras ship alongside the base install (#806) * feat: optional dependencies name their extra, and silent fallbacks warn Step 5 of the dependency diet. Every optional-dependency failure now says what to install, in one line, and exits 1 with no traceback. Nothing changes when everything is installed. - fluid_build/_optional.py adds require(module, extra, feature) and MissingOptionalDependency, a subclass of ImportError. It is adapted from pandas' import_optional_dependency (BSD-3). - It probes with find_spec before importing, top-level package first, since find_spec on a dotted name raises when the parent is missing (bpo-30436). - A package that is installed but fails to import keeps its own error. - The CLI prints the message with its stable code and exits 1. - These now go through it: ai setup's key validation, the litellm backend, LLM fallback routing, the MCP server and output-port server, the dbt, hosted and discovery MCP delegates, and fetch_sample_rows (duckdb). - fluid doctor never imports litellm, mcp or openlineage; it reads their versions from package metadata. That takes it from 3,287 to 1,035 modules and drops litellm's cost-map fetch. New MCP SDK and OpenLineage rows name their extras. - AWS: plan and diff used to fail with "Bucket name 'None-fluid-staging' is invalid". They now name the [aws] extra when boto3 is missing, and say "Could not resolve the AWS account id" when no credentials resolve. Glue import discovery warns instead of silently dropping its blocks. - Lineage emission without openlineage-python warns once, naming [lineage], and apply still succeeds. The ODCS and credential fallbacks warn instead of logging at DEBUG. - _mcp_compat probes mcp before mcp.server.mcpserver. - Corrected comments that called litellm a hard dependency, or said sqlglot powers _sql_safety. - An empty `lineage` extra lets the [lineage] hint resolve now; the extras PR fills it. - .secrets.baseline: one entry's line number moved. * fix: the lineage warning keeps its [lineage] extra on the console From the security review of the optional-dependency commit: - _warn_once printed through _console.warning, which wraps the text in Rich markup. Rich read `[lineage]` as a tag and dropped it, so the console said `pip install 'data-product-forge'`. It now prints with literal_text. The test asserts the console copy as well as the log record; it failed before this change. - The row_access comment claimed the separator and comment check is skipped without sqlglot. That check runs first and still holds. Only the structural checks (subquery, UNION/GROUP/ORDER/LIMIT) and the undeclared-column check go unchecked. * feat: litellm, mcp and lineage extras alongside the base install 0.27.0 adds the extras that 0.28.0 relies on when litellm, mcp and openlineage leave the base install. The base install is unchanged here; a package listed in both is harmless to pip and uv. - [litellm] carries both base lines verbatim, keeping the Python 3.10 cap from PR #555, and [ai] aliases it. - [mcp] is mcp plus pyjwt, which output_ports/mcp/auth.py imports directly. - [lineage] is openlineage-python. - [standard] is litellm + mcp, as in fastapi[standard]. - all and dev include the new extras. - CI jobs that run LLM or MCP code now name their extras (llm-smoke, ai-provider-matrix, four integration jobs), so 0.28.0 changes nothing for them. - Every Docker profile installs [standard], so the images keep AI and `fluid mcp serve`. - `fluid doctor` prints one line: AI/MCP move to extras in 0.28.0. - mcp_output_port_certify.py names [mcp] when the SDK is missing. - tests/ci_guards/extras_alongside_base.py pins the marker lines and the contents of [standard], [ai], all and dev. * fix: pyjwt floor 2.15.0 in [mcp], and Docker images keep lineage From the security review of the extras commit: - [mcp] declared pyjwt>=2.10.1, mcp's own floor. Every release up to 2.14.0 carries advisories on the PyJWKClient and decode path that output_ports/mcp/auth.py uses (e.g. GHSA-ffc3-869f-jxw9, critical). 2.15.0 has none. The floor is now pyjwt[crypto]>=2.15.0, which resolves on 3.10 and 3.12 alongside every mcp in range. - openlineage-python leaves the base install in 0.28.0, but [standard] is only litellm + mcp. Every Docker profile now names [lineage] too, so the images keep OpenLineage emission. Today's image is unchanged, since all of it is still in base. * chore: keep the real [lineage] extra, drop the empty alias * fix: install hints survive every output path, and a broken litellm degrades like a missing one From the defect hunt: - fluid forge without litellm printed the hint as Rich markup, which swallowed [litellm], and dumped tracebacks; the hint is now literal and a missing optional dependency logs no traceback. - An installed litellm that fails to import (>= 1.98 on Python 3.10) escaped list_available_models as a raw ImportError and broke the model preflight; it is a CopilotGenerationError again. - doctor probed the openlineage namespace package, so any openlineage-* distribution read as installed; it probes openlineage.client. - The dbt and hosted MCP delegates logged only the exception type, so the [mcp] hint never reached the user; a MissingOptionalDependency message (no secrets in it) now goes through. - The lineage warning keeps the pip line on its own line. - The AWS account-id error no longer claims there are no credentials when STS failed for another reason. - MissingOptionalDependency pickles and copies. - A test left a stub output-port module in sys.modules. * chore: one blank line after the license header, as the header writer emits * docs(changelog): the extras and the optional-dependency gate * fix: a failed mcp serve leaves no policy behind, and AWS account errors are main's - fluid mcp serve installed its process-wide policy before building the app, so a missing SDK left a policy with another directory's readable paths installed; in-process MCP tool calls after it were refused. The app is built first now. - The serve test clears the cached app, which an earlier in-process test could have built, so it exercises the SDK import every time. - The AWS account-id guard added on this branch is gone: main now reports an unresolved account itself (#794), after the sovereignty check and with the [aws] hint, and the guard pre-empted it. * test: one import style per module in the optional-deps tests * test: run the no-SDK output-port check in a subprocess, and reset the mcp version cache around blocked imports * chore: refresh .secrets.baseline line numbers
Main moved while the first sync merge (6d34df5) was being checked: #804, #805, #806 and #807 landed. Without them, #805 conflicts with #784 in fluid_build/cli/diff.py and #784 stays CONFLICTING. This is a normal merge; its base is 61ccba5, which is now a real ancestor. Resolution, one conflict: - fluid_build/cli/diff.py, imports. Kept main's `from ._logging import error, info, warn` (#805's schema-error reporting uses `error`) and #784's `from .console import cprint` (the managed Flink drift summary uses it). Both sides also changed these files since 61ccba5; git merged them hunk by hunk with no conflict: - fluid_build/api/changelog_rules.py: - #805 replaces the PII drift with `_sensitivity_changes`, reads a `schema: {fields: [...]}` wrapper in `_expose_schema` and adds the sovereignty and quality-severity rules. - #784's container-child recursion and nested nullability stay. - Every function except `diff_columns` equals the version from the side that changed it (AST comparison against 61ccba5). - `diff_columns` runs #784's `_diff_container_children` call, then #805's `_sensitivity_changes`. - fluid_build/iac/providers/aws.py: #806's account-id warnings in `_resolve_catalog_id`. #784's nested `_columns` branch is unchanged. The tree differs from origin/main (ef28db6) in exactly the feature's 39 files, and from 6d34df5 in exactly main's 88 files.
`fluid apply --mode amend-and-build --build-id <id>` (and replace-and-build) checked the id only in the build runner, which keeps the builds whose id equals it and, with none left, logs "Build not found" and returns 1. On the OpenTofu engine the runner starts after `tofu apply`, so a mistyped id provisioned the product and then failed. apply now runs the runner's match in a build mode, right after the build-id grammar gate, ahead of the apply hooks, tofu and the runner, on both engines. The refusal is a typed CLIError, apply_build_id_not_found, that names the id, lists the contract's build ids and says nothing was applied. Ids match exactly, as in the runner. An empty --build-id and a contract without builds are not checked, as before. The runner keeps its own check as the backstop. The Command Center run now reports apply_build_id_not_found, where it reported build_not_found:<id> after the infrastructure was applied. * docs(changelog): an unknown --build-id is refused before tofu apply
The feature branch moved while this sync was in review: #809 (the security gate) was squash-merged onto b6b901f as f4768da. Merging it here keeps this sync mergeable into the feature branch. The base is b6b901f. Resolution, one conflict: - fluid_build/cli/diff.py, imports. Kept main's `_redact_str` and `redact_secret_text` (#805's redacted schema-error output) and #809's `FluidUserError`, which `run()` now re-raises as a typed error. pyproject.toml merged without conflict. #809 adds its three modules to the mypy strict allowlist, and main's #806 extras stay. The tree differs from f4768da in exactly main's files, and from 95c5c1c in exactly #809's 12 files. In the two files both sides changed, each side's diff is unchanged.
Brings in #808, which landed during review. The merge is clean, with base ef28db6. #808 and #784 both change two files: - fluid_build/cli/apply.py. #808's `_gate_selected_build_for_apply` runs in the general gate block after `_gate_build_ids_for_apply`. Like #803's gate, that is after #784's managed-streaming early return, which refuses any --build-id on its own (`apply_streaming_build_selection`). - fluid_build/build_runners/base.py. #808 changes only a comment in the build filter; #784's `_report_externally_run_build` is unchanged.
…e=False #805 made compare_contracts validate both contracts by default. This test's contract is schema-invalid on purpose (a `name` inside a map value), so it now raised ContractValidationError before the rule under test ran. The test checks the diff rule, not validity, so it opts out with validate=False, the path #805 keeps for a contract that is still being repaired.
github-actions
Bot
added
provider
This was referenced Nov 4, 2026
fas89
pushed a commit
that referenced
this pull request
Nov 4, 2026
Managed Flink to Iceberg streaming preview, in FLUID 0.7.7 (author: Dustlab). - Recursive record types. - Kafka record shapes declared in FLUID (properties.stream and properties.mapping). - Generated FlinkDeployment resources. - Plan, apply, verify and diff integration. - Checkpoint and catalog verification. - fluid generate flink. The operator security gate covers several things: - Allowlists for the kube context, namespace, image, service account and Secrets. - Listed hostnames for every destination the job contacts: the catalog, the token endpoint, S3, the Kafka brokers and the Schema Registry. - NAME@host pairing for catalogAuth. - A pinned-IP request guard. - Refusals: no env: refs in catalog text, an https-only OAuth2 endpoint, no SQL hints, and a contract-SQL allowlist. Hardened through stacked PRs #795, #809, #817, #819, #825 and #826, and syncs #815, #821 and #824. The guide is in forge_docs (advanced/managed-flink-iceberg). The remaining live proof is tracked on Trello.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Sync PR #798 was squash-merged. b6b901f therefore has a single parent, and
mainnever became an ancestor offeat/managed-flink-iceberg-streaming(merge-base b2d05c0). Oncemainmoved on, #784 wentCONFLICTING. GitHub then stopped running itspull_requestworkflows: CI, Flink Iceberg preview and actionlint. The PR looked green because only CodeQL, label, docs-reminder and dependabot reported.Merge this PR with "Create a merge commit", never squash. A squash drops main's ancestry again, which is the bug this PR fixes.
What
Four real merge commits and one test fix. Nothing else changes.
The final tree differs from
main(635ea30) in exactly the feature's own files, 44 of them: #784's delta plus #809's. It differs from the feature branch (f4768da) in exactly main's changes since #789.How the conflicts were resolved
6d34df5. #798's head (4d37c2c) has the same tree as b6b901f and descends from a true merge with e59e325 (#789). So the feature's delta on top of main is
e59e325c..b6b901f6.CHANGELOG.md,HONESTLY_TESTED.md,cli/_verify_databricks.py,iac/providers/databricks.py,iac/providers/databricks_enforce.py,tests/conftest.py,tests/iac/_fake_databricks.py,tests/iac/test_iac_databricks_enforce.py,tests/iac/test_iac_databricks_fake_e2e.pyandtests/test_forge_mode_picker.py.cli/verify.pykeeps both sides: main's feat(databricks): bronze landing, retention, scoped grants and enforcement follow-ups #801 retention dimension and warn rendering, and feat: add managed Flink to Iceberg streaming preview #784's--streaming-timeoutand managed-streaming branch.api/changelog_rules.pystays byte-identical to chore(streaming): sync #784 with main after #789 and resolve the changelog-rules conflict #798's judge-verified resolution.95c5c1c.
cli/diff.pyhas one import conflict. The resolution keeps main'sfrom ._logging import error, info, warn(fix(diff): close the --baseline gate gaps found while shipping #789 #805) and feat: add managed Flink to Iceberg streaming preview #784'sfrom .console import cprint.changelog_rules.pymerged without conflict. An AST comparison against 61ccba5 shows that every top-level function exceptdiff_columnsmatches the side that changed it.diff_columnsruns feat: add managed Flink to Iceberg streaming preview #784's_diff_container_children, then fix(diff): close the --baseline gate gaps found while shipping #789 #805's_sensitivity_changes.ed6aae7.
cli/diff.pyhas one import conflict. The resolution keeps main's_redact_strandredact_secret_text(fix(diff): close the --baseline gate gaps found while shipping #789 #805) and security(streaming): operator allowlists and egress rules for the managed Flink lane (#784 run 2) #809'sFluidUserError.pyproject.tomlmerged without conflict. security(streaming): operator allowlists and egress rules for the managed Flink lane (#784 run 2) #809 adds three modules to the mypy strict allowlist.f6ca76b. A clean merge. #808's
_gate_selected_build_for_applysits after #784's streaming early return, as #803's gate does. That branch refuses any--build-iditself.2e0952f. #805 made
compare_contractsvalidate both contracts by default. #784'stest_a_container_child_carrying_its_own_name_is_still_diffed_at_its_slotdiffs a contract that is schema-invalid on purpose, anameinside a map value, so it now raisedContractValidationErrorbefore the rule under test ran. It now passesvalidate=False, the path #805 keeps for a contract that is still being repaired.Review
An Opus judge, in its own worktree, reviewed each merge's resolution. Verdict: PASS on all four, with no merge-induced finding at MEDIUM or above. Carried to follow-ups, all failing closed or pre-existing:
iac/providers/aws.py::_glue_column_type_findings) refuses astruct/array/mapcolumn before feat: add managed Flink to Iceberg streaming preview #784's nested_columnsbranch can render it. A typed nested column on a Glue table therefore failsfluid validateandfluid generate iac, with a message that is wrong on 0.7.7: "the contract schema cannot declare them yet". It fails closed. The flink-iceberg example uses a REST catalog, so it is unaffected.streaming/compiler.pyvalidatesparameters.flinkwith a plainDraft202012Validator, so its$-anchored patterns admit a trailing newline. security(streaming): operator allowlists and egress rules for the managed Flink lane (#784 run 2) #809's exact-match allowlists refuse it everywhere exceptdeployment.nameandimage, and neither of those lets anything past an allowlist.Gates (final tree, local)
Run on the final tree. The pytest runs used the main checkout's
.venvwithPYTHONPATHpointing at this worktree (fluid_build.__file__checked), and CI-faithful lean.[dev,local]venvs for the perf, compat and Flink-job runs.ruff check fluid_build/ tests/(0.15.12 and latest 0.17.3), the dead-code gate,black --check24.10.0 (the ci.yml pin): clean.black26.10.1 (the pyproject pin): the same 33 files as on main, with nothing extra from this branch.lint-imports(6 contracts kept) andlint-imports --config .importlinter-external(1 kept): clean.scripts/mypy_strict.pyin an isolated mypy env: clean on 11 files, including security(streaming): operator allowlists and egress rules for the managed Flink lane (#784 run 2) #809's 3.detect-secrets-hookin apython:3.12-slimcontainer: rc 0 with the baseline unchanged, both on this PR's 185 changed files and on feat: add managed Flink to Iceberg streaming preview #784's 44 files against main.mypy --strict, pytest with the coverage gate): 450 passed, coverage 92.57%.tests/streamingcollects 204.tests/fragments/test_compat_gate.py, full history in a lean venv: 45 passed. CI's shallow clone skips this test.tests/perf/test_startup_budget.py, lean venvs: branch 857/786 modules, main 856/786 (budgets 895/916); 7 passed on both.tests/streaming. Result: 9803 passed. The 2 failures (test_duckdb_sandbox_workspace_root, macOS/var) fail the same way on pristine main 635ea30. The third was thetest_standardscase fixed in 2e0952f, whose suites now give 384 passed.The CI
pull_requesttrigger only covers PRs to main, soci.ymlandflink-iceberg.ymlare dispatched on this branch.