diff --git a/docs/advanced/error-codes.md b/docs/advanced/error-codes.md index 4765484..e002f32 100644 --- a/docs/advanced/error-codes.md +++ b/docs/advanced/error-codes.md @@ -347,7 +347,7 @@ The state refusals (`state_shared_with_another_provider`, `state_migration_ambig | Kind | Triggered by | Fix | |---|---|---| | `databricks-tag-collision` | Two labels that become one tag key once `-` is read as `_` (`cost-center` and `cost_center`), or a label that becomes one of the product's own tag keys *(since 0.28.0, [forge-cli #831](https://github.com/Agenticstiger/forge-cli/pull/831))*. Only a key the contract states counts: with no `metadata.productType`, a label `fluid_product_type` is no collision | Rename the label so its key differs | -| `databricks-tag-value-too-long` | A tag value over 256 characters: the contract's `id`, `name`, `domain`, `metadata.layer`, `metadata.productType`, `metadata.owner.team`, `fluidVersion` or the expose's `version`, or a column's `sensitivity` | Shorten it to 256 characters or fewer. forge-cli refuses rather than cuts, since a cut id names another product | +| `databricks-tag-value-too-long` | A tag value over 256 characters: the contract's `id`, `name`, `domain`, `metadata.layer`, `metadata.productType`, `metadata.owner.team`, `fluidVersion` or the expose's `version`, or a column's `sensitivity`. *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* Also a governance label's value, which is no longer cut to 63 characters | Shorten it to 256 characters or fewer. forge-cli refuses rather than cuts, since a cut id names another product | | `databricks-tags-too-many` | More than 50 tags on a table or schema, counting the product's own tags with the governance labels, or more than 1,000 column tags on a table *(since 0.28.0, [forge-cli #831](https://github.com/Agenticstiger/forge-cli/pull/831))* | Remove labels until 50 or fewer remain, or classify 1,000 columns or fewer | | `databricks-comment-unsafe` | A table or column description that holds a backslash (the provider escapes only quotes in `COMMENT '...'`, and Databricks reads a backslash in a literal as an escape, so a trailing one, or one before a quote, would break the literal) or `${` (Spark may substitute a variable there) | Remove the backslash, or write `$ {` for `${` | | `databricks-certification-invalid` | The label `certification` has any value but `certified` (case ignored), on a product that is not `deprecated`. Unity Catalog's `system.certification_status` takes `certified` or `deprecated` | Set `labels.certification: certified` or remove it. Mark a deprecated product with `lifecycle.state: deprecated` | @@ -423,7 +423,10 @@ The `fluid generate iac` and `fluid apply` runs print the refusal and exit befor - Move the offending binding into an allowed region, or widen 'sovereignty.allowedRegions' in the contract - Run 'fluid validate <contract>' to see the same findings with full detail -- Set 'sovereignty.enforcementMode: advisory' to report without blocking +- Set 'sovereignty.enforcementMode: advisory' to report a region outside 'allowedRegions' without blocking; a region in 'deniedRegions' blocks in every mode +- Pass --no-check-sovereignty to write the plan anyway (logged at WARNING; fluid apply keeps its own checks) + +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* `fluid plan` raises it by default, before it writes a plan, when the contract's own sovereignty policy blocks; see [Sovereignty gate](../cli/plan.md#sovereignty-gate-since-0-15-0). The third suggestion is reworded and the fourth is new. Up to 0.29.0 the third read "Set 'sovereignty.enforcementMode: advisory' to report without blocking", although a region in `deniedRegions` blocks in every mode. ## Product authoring diff --git a/docs/advanced/governance.md b/docs/advanced/governance.md index 3bb5027..7341a41 100644 --- a/docs/advanced/governance.md +++ b/docs/advanced/governance.md @@ -220,7 +220,7 @@ A contract's `sovereignty` block declares where its data may live. `enforcementM | `enforcementMode` | Severity | Effect | |---|---|---| -| `strict` *(the schema default)* | ❌ error | [`fluid validate`](../cli/validate.md) exits 1; [`fluid plan --check-sovereignty`](../cli/plan.md#sovereignty-gate-since-0-15-0) blocks. | +| `strict` *(the schema default)* | ❌ error | [`fluid validate`](../cli/validate.md) exits 1; [`fluid plan --check-sovereignty`](../cli/plan.md#sovereignty-gate-since-0-15-0) blocks, and *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* so does plain `fluid plan`. | | `advisory` | ⚠️ warning | Reported, does not block, though `fluid validate --strict` promotes warnings to errors. | | `audit` | ℹ️ info | Logged only. | @@ -292,7 +292,7 @@ A Cloud KMS key ring and a Data Catalog taxonomy are placed at their dataset's m | Stage | What it does | |---|---| | [`fluid validate`](../cli/validate.md) | Runs the policy engine on every contract; severity follows `enforcementMode`. | -| [`fluid plan --check-sovereignty`](../cli/plan.md#sovereignty-gate-since-0-15-0) | Opt-in when you run `plan` yourself. Since 0.17.0 every pipeline that `fluid generate ci` emits runs stage 6 as `fluid plan ... --out runtime/plan.json --check-sovereignty`, so a strict violation fails before any stage reaches a cloud. A contract with no `sovereignty` block prints `NOT CHECKED` and passes. | +| [`fluid plan`](../cli/plan.md#sovereignty-gate-since-0-15-0) | *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* Runs the policy engine by default and refuses what `fluid validate` refuses, before it writes a plan; `--no-check-sovereignty` skips it with a warning. `--check-sovereignty` also asks the provider's hook and prints the verdict; up to 0.29.0 only that flag checked. Since 0.17.0 every pipeline that `fluid generate ci` emits runs stage 6 as `fluid plan ... --out runtime/plan.json --check-sovereignty`, so a strict violation fails before any stage reaches a cloud. With that flag, a contract with no `sovereignty` block prints `NOT CHECKED` and passes. | | [`fluid generate iac`](../cli/generate-iac.md) / [`fluid apply`](../cli/apply.md) | The AWS provider, and since 0.17.0 the GCP provider, refuse a placement outside the policy and exit 1, and **no module is written**. GCP checks every location the plan emits, including one a resource inherits by default. | | Embedded-SQL builds on BigQuery | The read and the landing are checked against the policy too (`EmbeddedSqlSovereigntyError`). | | [`fluid mcp output-port serve`](mcp.md#caller-jurisdiction-enforcement-since-0-15-0) | A pinned `jurisdiction` becomes a query-time gate on the **verified** caller jurisdiction, enforced by default, with the escape hatches in the contract and not in a flag. | diff --git a/docs/cli/plan.md b/docs/cli/plan.md index 9e01c73..55f4811 100644 --- a/docs/cli/plan.md +++ b/docs/cli/plan.md @@ -25,7 +25,8 @@ fluid apply runtime/plan.json --mode amend-and-build --yes ```bash fluid plan CONTRACT [--env ENV] [--mode MODE] [--out PATH] [--provider NAME] [--project PROJECT] [--region REGION] - [--validate-actions] [--estimate-cost] [--check-sovereignty] + [--validate-actions] [--estimate-cost] + [--check-sovereignty | --no-check-sovereignty] [--html [PATH]] [--verbose] ``` @@ -41,7 +42,8 @@ fluid plan CONTRACT [--env ENV] [--mode MODE] [--out PATH] | `--verbose`, `-v` | Show detailed action information | | `--validate-actions` | Validate generated provider actions against the ProviderAction SDK schema | | `--estimate-cost` | Ask the provider to estimate cost | -| `--check-sovereignty` | *(since 0.15.0)* Resolve a real sovereignty verdict and name which source produced it — the provider's `validate_sovereignty` hook when it has one (the `gcp` provider has one since 0.17.0), otherwise the built-in policy engine (the same checker [`fluid validate`](./validate.md) runs), otherwise `NOT CHECKED`. **Exits 1 when the check fails.** Opt-in, off by default. See [Sovereignty gate](#sovereignty-gate-since-0-15-0). | +| `--check-sovereignty` | *(since 0.15.0)* Resolve a real sovereignty verdict and name which source produced it — the provider's `validate_sovereignty` hook when it has one (the `gcp` provider has one since 0.17.0), otherwise the built-in policy engine (the same checker [`fluid validate`](./validate.md) runs), otherwise `NOT CHECKED`. **Exits 1 when the check fails.** *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* `plan` checks the policy without this flag too, and refuses before it writes a plan; the flag adds the provider's hook and the printed verdict. See [Sovereignty gate](#sovereignty-gate-since-0-15-0). | +| `--no-check-sovereignty` | *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* Skip the sovereignty check `plan` runs by default, and write the plan whatever the policy says. Prints a warning and logs a WARNING (`sovereignty_check_skipped`). `fluid apply` keeps its own checks. | | `--provider` | Override the provider from the contract | | `--project` | Override the project / account from the contract | | `--region` | Override the region / location from the contract | @@ -95,7 +97,44 @@ A bundle is never re-overlaid: `fluid plan bundle.tgz --env prod` on a bundle bu ## Sovereignty gate (since 0.15.0) -`--check-sovereignty` stays **opt-in and off by default**, but on `0.15.0` it is a gate rather than a label. The verdict resolves in a fixed order, and the output always names which source answered: +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* `fluid plan` checks the contract's sovereignty policy by default, before it plans. It asks the built-in policy engine, the checker [`fluid validate`](./validate.md) runs, and refuses what `validate` refuses: + +- Under `enforcementMode: strict`, the schema default, the findings that block include a binding outside `allowedRegions`, one in a region of another jurisdiction, and one on AWS, GCP, Azure or Databricks that declares no region (the platform would pick it). +- In every mode, a binding in a region listed in `deniedRegions` blocks. + +A refusal exits 1 with `ERR_SOVEREIGNTY_VIOLATION` and writes no plan. A plan file an earlier run left at `--out` stays as it was. The check builds no provider, so it needs no cloud SDK and no credentials. When nothing blocks it prints nothing, and the plan is written as before: for a contract with no `sovereignty` block, and for one whose findings are only warnings or info lines (under `advisory` or `audit`, every finding but a denied region). + +This is the refusal on 0.30.0 for a contract with `enforcementMode: strict`, `allowedRegions: [eu-central-1, eu-west-1]`, `deniedRegions: [us-west-2]` and one binding in `us-west-2`: + +```text +Sovereignty check: 3 finding(s) — source: built-in policy engine, enforcementMode=strict + - ❌ [customer_events] Region 'us-west-2' is explicitly denied by sovereignty policy + 💡 Use one of the allowed regions: eu-central-1, eu-west-1 + - ❌ [customer_events] Region 'us-west-2' not in allowed regions list + 💡 Allowed regions: eu-central-1, eu-west-1 + - ❌ [customer_events] Region 'us-west-2' (jurisdiction: US) does not match required jurisdiction: EU + 💡 Consider using regions in EU jurisdiction + +❌ Sovereignty check FAILED — enforcementMode is 'strict'. +❌ sovereignty_violation [ERR_SOVEREIGNTY_VIOLATION] + command: plan + contract: .../contract.fluid.yaml + errors: ["[customer_events] Region 'us-west-2' is explicitly denied by sovereignty policy", "[customer_events] Region 'us-west-2' not in allowed regions list", "[customer_events] Region 'us-west-2' (jurisdiction: US) does not match required jurisdiction: EU"] + message: The contract's sovereignty policy (enforcementMode strict) blocks where it puts data, so fluid plan wrote no plan. +... +``` + +Each finding names its expose in brackets, and the error carries the findings as `errors`. The suggestions printed after it are listed under [`sovereignty_violation`](../advanced/error-codes.md#sovereignty-violation). + +Up to 0.29.0 the check was opt-in: plain `fluid plan` wrote a plan for this contract and exited 0. To write the plan anyway now, pass [`--no-check-sovereignty`](#no-check-sovereignty). + +::: warning Behavior change in 0.30.0 +A script or pipeline that runs `fluid plan` on a contract `fluid validate` refuses for sovereignty now stops at plan, with exit 1 and no plan file. See [Changes in 0.30.0](../upgrading.md#changes-in-0-30-0). +::: + +### `--check-sovereignty` + +`--check-sovereignty` runs the same check first. If nothing blocks, plan then resolves a verdict and prints it, naming which source answered, in a fixed order: 1. The target provider's `validate_sovereignty` hook, when it returns a verdict. 2. Otherwise the **built-in policy engine** — the same sovereignty checker `fluid validate` runs, so the two stages cannot reach opposite verdicts on the same contract. @@ -124,9 +163,11 @@ Sovereignty check: 10 violation(s) — source: gcp provider hook ❌ sovereignty_violation [ERR_SOVEREIGNTY_VIOLATION] ``` +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* The second output predates 0.30.0. Now the default check runs first and refuses that contract under `strict` before the hook is asked: 0.30.0 prints the built-in engine's two findings for the `us-central1` binding (`not in allowed regions list` and the jurisdiction mismatch) and writes no plan. The hook's verdict is printed when the engine finds no error. + The pipeline `fluid generate ci --system github` writes runs `fluid plan ... --check-sovereignty` in its plan job, and the Jenkins pipeline's stage 6 does the same. -A failing check prints a `Sovereignty check: N finding(s)` header naming the source, then its findings as a bulleted list, then `❌ Sovereignty check FAILED`, and **exits 1**. The plan file is still written; the non-zero exit is what makes the flag usable as a CI gate. +A failing check prints a `Sovereignty check: N finding(s)` header naming the source, then its findings as a bulleted list, then `❌ Sovereignty check FAILED`, and **exits 1**. *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* It writes no plan, as the default check does. Up to 0.29.0 the plan file was still written, and only the non-zero exit stopped a pipeline. On `0.14.1` and earlier none of that happened. No shipped provider implemented `validate_sovereignty`, the hook helper returned an empty violation list for a hook that was absent (and for one that raised, because the invoker swallows the exception and hands back its first argument), and an empty list rendered as `Sovereignty check: PASS` with exit 0 — printed on contracts `fluid validate` rejects with two residency errors. The flag was also skipped outright when the provider failed to build. `PASS` is now printed only when a check actually ran and found nothing. @@ -134,6 +175,20 @@ On `0.14.1` and earlier none of that happened. No shipped provider implemented ` A pipeline that already passes `--check-sovereignty` moves from a step that could only ever be green to one that can fail. Two rules decide whether it blocks: a region named in `deniedRegions` is an **error in every mode**, and everything else follows the contract's own `sovereignty.enforcementMode`, whose default is `strict`. Full table: [Governance → Sovereignty enforcement modes](../advanced/governance.md#sovereignty-enforcement-modes-since-0-15-0). ::: +### `--no-check-sovereignty` + +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* `--no-check-sovereignty` skips the check and writes the plan whatever the policy says: + +```bash +fluid plan contract.fluid.yaml --no-check-sovereignty --out runtime/plan.json +``` + +```text +⚠️ Sovereignty check skipped (--no-check-sovereignty): the plan is written whatever the contract's sovereignty policy says. +``` + +It also logs a WARNING with the event `sovereignty_check_skipped`, so the skip is on record. It changes nothing in `fluid apply`, which keeps its own checks; [Where sovereignty is enforced](../advanced/governance.md#where-sovereignty-is-enforced) lists them. + ## Examples ### Basic plan diff --git a/docs/concepts/sovereignty.md b/docs/concepts/sovereignty.md index a3c4079..8403169 100644 --- a/docs/concepts/sovereignty.md +++ b/docs/concepts/sovereignty.md @@ -138,7 +138,7 @@ This is the part most worth getting straight, because the two gates share a bloc |---|---|---| | **Question** | May the data *sit* here? | May this *caller* read it? | | **Compares** | each expose's `binding.location.region` → jurisdiction, against `sovereignty.jurisdiction` | the caller's **verified** jurisdiction claim, against `sovereignty.jurisdiction` | -| **Runs on** | `fluid validate` (always); `fluid plan --check-sovereignty` (run by stage 6 of a generated pipeline); `fluid generate iac` and `fluid apply` on AWS and GCP | each `tools/call` that reaches the policy check at `fluid mcp output-port serve` | +| **Runs on** | `fluid validate` (always); `fluid plan` (by default *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)*; up to 0.29.0 only with `--check-sovereignty`, which stage 6 of a generated pipeline passes); `fluid generate iac` and `fluid apply` on AWS and GCP | each `tools/call` that reaches the policy check at `fluid mcp output-port serve` | | **Since** | `0.7.1`, blocking by default since `0.15.0` | `0.15.0` | | **Relaxed by** | `enforcementMode: advisory` / `audit` | `crossBorderTransfer: true` | @@ -237,7 +237,7 @@ fluid apply plan.json --yes # after editing plan.json by hand The two digests it prints are yours, not ours. A `planDigest` is a function of your plan, so the property to rely on is *stable across re-runs, different after an edit*, never a particular hex string. -So the division of labour is: `validate` (and `plan --check-sovereignty` in CI) decides whether the contract's residency claim is consistent; on AWS and GCP the provider refuses to create a resource outside the policy, including where it fills in a default; and the digest guarantees the artifact reaching your cloud is the one that decision was made about. +So the division of labour is: `validate` (and `plan`, which checks by default since 0.30.0) decides whether the contract's residency claim is consistent; on AWS and GCP the provider refuses to create a resource outside the policy, including where it fills in a default; and the digest guarantees the artifact reaching your cloud is the one that decision was made about. ## When `policy-compile` or `policy-apply` fails diff --git a/docs/providers/databricks.md b/docs/providers/databricks.md index 3c5db24..ab6d120 100644 --- a/docs/providers/databricks.md +++ b/docs/providers/databricks.md @@ -313,7 +313,7 @@ Whether a real metastore reports its region in the spelling a contract declares A contract's governance labels reach AWS as tags and Glue parameters and GCP as labels. On Databricks they are Unity Catalog tags, one `databricks_entity_tag_assignment` per tag, built from the same source as the other two clouds (`iac/governance_labels.py`). Before this change the Databricks module never read `labels`, and nothing said so. -The tags a contract carries are its and the expose's `labels`, plus `fluid_classification`, `fluid_jurisdiction`, `fluid_regulation` and `fluid_residency`, from the classification (the expose's `policy.classification`, else `metadata.classification`) and the contract's `sovereignty` block. Keys and values first pass through the label alphabet the three clouds share (lower-case letters, digits, `_` and `-`, at most 63 characters), so a tag carries the same text on each. +The tags a contract carries are its and the expose's `labels`, plus `fluid_classification`, `fluid_jurisdiction`, `fluid_regulation` and `fluid_residency`, from the classification (the expose's `policy.classification`, else `metadata.classification`) and the contract's `sovereignty` block. Keys first pass through the label alphabet the three clouds share (lower-case letters, digits, `_` and `-`, at most 63 characters), so a tag has the same key on each. Values are written as the contract states them; see **Values** below. | On | Carries | | --- | --- | @@ -322,9 +322,31 @@ The tags a contract carries are its and the expose's `labels`, plus `fluid_class Two rules shape the keys: -- **A `-` in a key becomes `_`.** A Unity Catalog tag key may not contain `. , - = / :`, and a governance label key is lower-case letters, digits, `_` and `-`. `data-classification` is the tag `data_classification`; `fluid_jurisdiction` stays as it is. The value is not changed further. +- **A `-` in a key becomes `_`.** A Unity Catalog tag key may not contain `. , - = / :`, and a governance label key is lower-case letters, digits, `_` and `-`. `data-classification` is the tag `data_classification`; `fluid_jurisdiction` stays as it is. - **Nothing is dropped, nothing is overwritten.** Two labels that become one key (`cost-center` and `cost_center`) are refused as `databricks-tag-collision`. More than the 50 tags a Unity Catalog securable holds is refused as `databricks-tags-too-many`. `fluid validate` reports both. *(since 0.28.0, [forge-cli #831](https://github.com/Agenticstiger/forge-cli/pull/831))* On a table, the product's own tags (see [Comments and product tags](#comments-and-product-tags)) count toward the 50 with the governance labels, and a label that becomes one of their keys is also `databricks-tag-collision`. +**Values.** *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* A tag value is the contract's text, less the surrounding whitespace Unity Catalog refuses. It keeps case, spaces and punctuation, and `fluid_regulation` joins the frameworks with `, `. A value over 256 characters is refused as `databricks-tag-value-too-long` rather than cut, and `fluid validate` reports it too. Up to 0.29.0 a value went through the label alphabet as well: lower-cased, with each character other than a letter, a digit, `_` or `-` turned into `-`, and cut at 63 characters. GCP labels and AWS tags still do. + +These labels and `sovereignty` fields, compiled offline with `fluid generate iac` on each release, give the same keys and these values: + +```yaml +labels: + business-value: "A single unrenewed ground lease can strand a multi-tenant tower" + cost-center: CC-42 +sovereignty: + jurisdiction: US + regulatoryFramework: [GDPR, CCPA] +``` + +| Tag | 0.29.0 | 0.30.0 | +| --- | --- | --- | +| `business_value` | `a-single-unrenewed-ground-lease-can-strand-a-multi-tenant-tower` | `A single unrenewed ground lease can strand a multi-tenant tower` | +| `cost_center` | `cc-42` | `CC-42` | +| `fluid_jurisdiction` | `us` | `US` | +| `fluid_regulation` | `gdpr_ccpa` | `GDPR, CCPA` | + +**A deployment made before 0.30.0** gets the new values at its next `fluid apply`, which updates each changed tag in place (`~` in the plan). A tag assignment's address and import id depend on its key alone, and the provider (1.140.0) updates `tag_value` without replacing the assignment. So no table, schema or data is replaced or destroyed, and no `--allow-data-loss` is needed. Until that apply, `fluid verify` reports each old value as a changed tag, `CRITICAL`. + A contract with no labels gets the module it always did. *(since 0.28.0, [forge-cli #831](https://github.com/Agenticstiger/forge-cli/pull/831))* Since 0.28.0 that is no longer so for a table: a contract with no labels still gets the product's own tags on it. An assignment that already exists is imported before the first apply, best-effort as grants are, so a run after lost state adopts it instead of failing. Removing a label destroys only its assignment. That holds no data, so it does not need `--allow-data-loss`: the apply prints it, as it does a revoked grant. @@ -335,7 +357,7 @@ An assignment that already exists is imported before the first apply, best-effor Databricks does not document whether a table keeps its tags across the `CREATE OR REPLACE` a dbt `table` materialization runs, and forge-cli has not proven it. If tags go missing after a build, `fluid apply` sets them again. `fluid verify` is the gate. ::: -Not run against a live workspace. +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* forge-cli ran the change on Databricks Free Edition: a product carrying the 0.29.0 values planned `+0 ~17 -0` and applied, `information_schema.table_tags` then showed the values as the contract states them, and `fluid verify` passed on all 27 tags. ### Comments and product tags diff --git a/docs/upgrading.md b/docs/upgrading.md index 941f6cf..6420495 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1,6 +1,6 @@ --- title: Upgrading -description: How to upgrade and pin data-product-forge, the checklist for each step from 0.15.x to 0.28.0, and what moving the local engine from DuckDB 1.x to 2.x changes — what breaks, what to do, and how to check it. +description: How to upgrade and pin data-product-forge, the checklist for each step from 0.15.x to 0.30.0, and what moving the local engine from DuckDB 1.x to 2.x changes — what breaks, what to do, and how to check it. --- # Upgrading @@ -851,12 +851,29 @@ These changes to AI authoring can change what a script, a CI job or an IDE agent ## Changes in 0.30.0 -These changes shipped in 0.30.0. +These changes shipped in 0.30.0. The first can stop a plan that used to pass: read it before you upgrade a pipeline. + +### `fluid plan` checks sovereignty by default + +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* `fluid plan` now refuses a contract that `fluid validate` refuses for its sovereignty policy. It exits 1 with `ERR_SOVEREIGNTY_VIOLATION` and writes no plan. Up to 0.29.0 plain `fluid plan` wrote a plan for such a contract and exited 0; only `--check-sovereignty` checked. Under `enforcementMode: strict`, the schema default, the refused contracts include one with a binding outside `allowedRegions`, in another jurisdiction, or on AWS, GCP, Azure or Databricks with no region declared. In every mode they include one with a binding in a region listed in `deniedRegions`. + +- **A plan step can now fail.** Before you upgrade, run `fluid validate` on each contract: the sovereignty errors it reports are the ones `plan` now refuses. To write the plan anyway, pass `--no-check-sovereignty`. It prints a warning and logs a WARNING (`sovereignty_check_skipped`), and `fluid apply` keeps its own checks. +- **A failing plan stage writes no plan file.** The pipelines `fluid generate ci` writes already pass `--check-sovereignty`. On 0.29.0 a failing stage still wrote `plan.json`; now it writes none, and a file an earlier run left at `--out` stays as it was. A provider hook that reports nothing no longer passes a contract `fluid validate` refuses, because the same check runs first. + +See [Sovereignty gate](./cli/plan.md#sovereignty-gate-since-0-15-0). ### Databricks - *([forge-cli #865](https://github.com/Agenticstiger/forge-cli/pull/865), since 0.30.0)* **A Databricks expose with `semantics` now gets a metric view, and the apply needs `databricks-sdk`.** `fluid apply` creates `_semantic_model` beside the table. Without `pip install 'data-product-forge[databricks]'`, the apply is refused before anything is applied. A `semantics` block the view cannot hold (a derived metric, say) now fails `fluid validate` for a Databricks binding: express it as a ratio, or remove it. See [Metric views](./providers/databricks.md#metric-views). - *([forge-cli #873](https://github.com/Agenticstiger/forge-cli/pull/873), since 0.30.0)* **A schema the contract stops using is now destroyed, not left failing the apply.** When a table moves to another schema or a schema's last expose is removed, `fluid apply` drops the metric views and the mask and row filter functions this product made there, when the deploy identity owns them, so OpenTofu can destroy the schema. Anything else the schema holds is named and left. `--mode dry-run` lists the drops and runs none. See [A schema the contract stops using](./providers/databricks.md#a-schema-the-contract-stops-using). +- *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* **Unity Catalog tag values are written as the contract states them.** A tag value is now the contract's text less surrounding whitespace. Up to 0.29.0 it was the GCP label slug: lower-cased, with each character other than a letter, a digit, `_` or `-` turned into `-`, and cut at 63 characters. So `fluid_jurisdiction` is `US` where it was `us`, and `fluid_regulation` is `GDPR, CCPA` where it was `gdpr_ccpa`. Tag keys are unchanged, and so are GCP labels and AWS tags. See [Governance tags](./providers/databricks.md#governance-tags). + - **The next `fluid apply` updates the tags in place.** The plan shows each changed tag as `~`. Nothing is replaced or destroyed, and no `--allow-data-loss` is needed. Until that apply, `fluid verify` reports each old value as a changed tag (`CRITICAL`). On Databricks Free Edition, a product carrying the old values planned `+0 ~17 -0`. + - **A value over 256 characters is refused** as `databricks-tag-value-too-long` by `fluid validate`, `fluid generate iac` and `fluid apply`. Up to 0.29.0 it was cut to 63 characters. Shorten the value before you upgrade. + - **Anything that reads a tag value sees the new text**, for example a query on `information_schema.table_tags` that matches `'us'`. + +### Other changes + +- *([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* Logs and printed errors no longer mask a fully qualified name such as `catalog.schema.table`: `fluid apply` printed `databricks.load table=***REDACTED***`, and now prints the table's name. A JWT or JWE is still masked, recognised by its JOSE header (a first part that decodes to a JSON object). `redact_secret_text()` no longer takes the `keep_dotted_names` keyword, so code that passes it gets a `TypeError`. ## Not in a release yet diff --git a/docs/walkthrough/11-stage-pipeline.md b/docs/walkthrough/11-stage-pipeline.md index 64f3c1a..03bd791 100644 --- a/docs/walkthrough/11-stage-pipeline.md +++ b/docs/walkthrough/11-stage-pipeline.md @@ -321,20 +321,23 @@ sovereignty: enforcementMode: strict ``` -The plan stage prints: +On 0.30.0 the plan stage prints: ```text -Sovereignty check: 1 finding(s) — source: built-in policy engine, enforcementMode=strict; the aws provider has no sovereignty hook - - ❌ Region 'us-east-1' (jurisdiction: US) does not match required jurisdiction: EU +Sovereignty check: 1 finding(s) — source: built-in policy engine, enforcementMode=strict + - ❌ [customer_events] Region 'us-east-1' (jurisdiction: US) does not match required jurisdiction: EU 💡 Consider using regions in EU jurisdiction ❌ Sovereignty check FAILED — enforcementMode is 'strict'. -CLI command error ❌ sovereignty_violation [ERR_SOVEREIGNTY_VIOLATION] + command: plan contract: .../sov.fluid.yaml + errors: ["[customer_events] Region 'us-east-1' (jurisdiction: US) does not match required jurisdiction: EU"] + message: The contract's sovereignty policy (enforcementMode strict) blocks where it puts data, so fluid plan wrote no plan. +... ``` -The plan file is still written; the exit code is what stops the pipeline. `enforcementMode: advisory` reports without blocking. See [Sovereignty](../concepts/sovereignty.md), and [Governance](../advanced/governance.md) for how the enforcement modes apply to the plan and to apply. +*([forge-cli #868](https://github.com/Agenticstiger/forge-cli/pull/868), since 0.30.0)* The stage writes no `plan.json` and no `plan.html`; a plan file an earlier run left stays as it was. Up to 0.29.0 the plan file was still written, and only the exit code stopped the pipeline. The same contract is now refused without `--check-sovereignty` too. `enforcementMode: advisory` reports without blocking, except for a region listed in `deniedRegions`, which blocks in every mode. See [Sovereignty](../concepts/sovereignty.md), and [Governance](../advanced/governance.md) for how the enforcement modes apply to the plan and to apply. ### Stage 7: apply diff --git a/scripts/cli-docs-allowlist.yml b/scripts/cli-docs-allowlist.yml index 1383706..8669a38 100644 --- a/scripts/cli-docs-allowlist.yml +++ b/scripts/cli-docs-allowlist.yml @@ -71,6 +71,11 @@ flag_oracle_ok: # prints the command's options and # `fluid generate-custom-scaffold --help` prints the # top-level usage. The pages document the name that runs. + - docs/cli/plan.md::--no-check-sovereignty + # Newer than the pin: forge-cli 0.30.0 (#868) made + # --check-sovereignty a BooleanOptionalAction, and the + # pinned 0.29.0 parser registers only --check-sovereignty. + # Remove this entry when cli-version.json moves to 0.30.0. # Files the version sweep must not scan, as paths relative to the repo root # (glob syntax allowed). RELEASE_NOTES_*.md is excluded unconditionally and