diff --git a/README.md b/README.md index b818be8..aacf0f1 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ The site is organised the way a reader uses it (get started, tutorials, how-to g ## Current Versioning -- Current CLI release documented here: `0.27.0` +- Current CLI release documented here: `0.28.0` - Contract schema: `fluidVersion: 0.7.6` is the stable default; `0.7.7` is an opt-in preview Those are different on purpose. `fluid version` reports the installed CLI release, while `fluidVersion` inside a contract selects the contract schema version. diff --git a/docs/.vuepress/cli-version.json b/docs/.vuepress/cli-version.json index cac7a06..55f9109 100644 --- a/docs/.vuepress/cli-version.json +++ b/docs/.vuepress/cli-version.json @@ -1,6 +1,6 @@ { - "supportedCliVersion": "0.27.0", - "supportedCliInstallSpec": "data-product-forge==0.27.0", + "supportedCliVersion": "0.28.0", + "supportedCliInstallSpec": "data-product-forge==0.28.0", "supportedFluidContractVersion": "0.7.6", "quickstartScaffoldVersion": "0.7.5", "supportedSdkVersion": "0.10.0", diff --git a/docs/README.md b/docs/README.md index 5ba2e8c..4467a89 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,7 @@ fluid forge fluid forge --domain retail ``` -`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. An API-key provider needs `pip install "data-product-forge[litellm]"`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without an API key, `fluid forge --llm-provider claude-code` uses your Claude Code login, and `fluid forge --template starter` uses no AI. To forge a data model from a business intent file and generate dbt from it: +`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. An API-key provider needs `pip install "data-product-forge[litellm]"`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without an API key, `fluid forge --llm-provider claude-code` uses your Claude Code login, and `fluid forge --template starter` uses no AI. To forge a data model from a business intent file and generate dbt from it: ```bash fluid forge data-model from-intent --example retail > intent.yaml diff --git a/docs/advanced/environment-variables.md b/docs/advanced/environment-variables.md index d64794b..9894052 100644 --- a/docs/advanced/environment-variables.md +++ b/docs/advanced/environment-variables.md @@ -151,7 +151,7 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke | `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. | | `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. | | `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. | -| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. *(from 0.28.0)* Emitting needs the `lineage` extra. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* | +| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. Emitting needs the `lineage` extra. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* | | `FLUID_OPENLINEAGE_ENDPOINT` | Path appended to the URL. | | `FLUID_OPENLINEAGE_API_KEY` (or `OPENLINEAGE_API_KEY`) | Bearer key for the receiver. | | `FLUID_OPENLINEAGE_TIMEOUT_SECONDS` | Request timeout. Default `5`. | diff --git a/docs/advanced/forge-copilot-discovery.md b/docs/advanced/forge-copilot-discovery.md index 2b9bd88..1dd46e7 100644 --- a/docs/advanced/forge-copilot-discovery.md +++ b/docs/advanced/forge-copilot-discovery.md @@ -37,7 +37,7 @@ Forge may inspect local: ## Optional helpers -These helpers are not the AI backend. Generating a contract with an API-key provider needs the `litellm` extra (`pip install "data-product-forge[litellm]"`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* +These helpers are not the AI backend. Generating a contract with an API-key provider needs the `litellm` extra (`pip install "data-product-forge[litellm]"`). *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* If you want richer Parquet and Avro schema inspection, install the optional discovery helpers: diff --git a/docs/advanced/litellm-backend.md b/docs/advanced/litellm-backend.md index ddd82b9..626bfc9 100644 --- a/docs/advanced/litellm-backend.md +++ b/docs/advanced/litellm-backend.md @@ -1,9 +1,9 @@ # LiteLLM Backend -[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is a core dependency of `data-product-forge` up to 0.27.0; from 0.28.0 it is installed with the `litellm` extra (alias `ai`) or `standard`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md). +[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is not in the base install: install it with the `litellm` extra (alias `ai`) or `standard`. It was a core dependency up to 0.27.0. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md). ::: tip Where this fits -LiteLLM is a hard dependency of core `data-product-forge` (`litellm>=1.83.14,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`. In 0.27.0 the base install still contains litellm, so installing the extra changes nothing yet. *(from 0.28.0)* litellm leaves the base install. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `fluid forge` and `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `--agent-loop` and `fluid ai setup` then need `pip install "data-product-forge[litellm]"` (or `[ai]`, or `[standard]` to add `mcp`); an explicit API-key `--llm-provider` and `fluid ai setup` print that line and exit 1 before doing any work when it is missing. `fluid init`'s AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). +LiteLLM was a hard dependency of core `data-product-forge` (`litellm>=1.83.14,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`; in 0.27.0 the base install still contains litellm, so installing the extra changes nothing there. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* litellm is not in the base install. `fluid forge` and `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `--agent-loop` and the `fluid ai setup` provider wizard (not `--clear` or `--source`) need `pip install "data-product-forge[litellm]"` (or `[ai]`, or `[standard]` to add `mcp`); an explicit API-key `--llm-provider` and the `fluid ai setup` wizard print that line and exit 1 before doing any work when it is missing. `fluid init`'s AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#changes-in-0-28-0). ::: ## Which providers you can select diff --git a/docs/advanced/llm-providers.md b/docs/advanced/llm-providers.md index f5aa090..7869086 100644 --- a/docs/advanced/llm-providers.md +++ b/docs/advanced/llm-providers.md @@ -16,7 +16,7 @@ fluid forge data-model from-intent intent.yaml -o customer_orders.fluid.yaml ## Supported providers -The `openai`, `anthropic`, `gemini` and `ollama` providers run through LiteLLM and need `pip install 'data-product-forge[litellm]'` (alias `[ai]`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. The four local agent CLIs need neither extra; only Claude Code needs no setup of its own, since the others reuse their own CLI login or key. +The `openai`, `anthropic`, `gemini` and `ollama` providers run through LiteLLM and need `pip install 'data-product-forge[litellm]'` (alias `[ai]`). *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. The four local agent CLIs need neither extra; only Claude Code needs no setup of its own, since the others reuse their own CLI login or key. | Provider | `--llm-provider` | Default model | Notes | | --- | --- | --- | --- | diff --git a/docs/advanced/mcp.md b/docs/advanced/mcp.md index c571e13..6a077d6 100644 --- a/docs/advanced/mcp.md +++ b/docs/advanced/mcp.md @@ -15,7 +15,7 @@ The first half of this page covers the authoring server. The second half is a de ### Start the server -Install the MCP SDK first: `pip install 'data-product-forge[mcp]'`. The extra exists since 0.27.0. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A base install no longer contains `mcp`, and `fluid mcp serve`, `fluid mcp output-port serve` and `fluid scaffold-ide` print that install line and exit 1 before they do any work. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). +Install the MCP SDK first: `pip install 'data-product-forge[mcp]'`. The extra exists since 0.27.0. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* A base install does not contain `mcp`, and `fluid mcp serve`, `fluid mcp output-port serve` and `fluid scaffold-ide` print that install line and exit 1 before they do any work. See [Upgrading](../upgrading.md#changes-in-0-28-0). ```bash fluid mcp serve --read-only # inspection only diff --git a/docs/advanced/network-safety.md b/docs/advanced/network-safety.md index faefe9e..4adae15 100644 --- a/docs/advanced/network-safety.md +++ b/docs/advanced/network-safety.md @@ -60,7 +60,7 @@ The webhook, federation and Command Center variables take comma-separated host s An IPv4-mapped address (`::ffff:...`) or a NAT64 address (`64:ff9b::/96`) that carries a link-local or one of these metadata addresses is refused too. The surrounding ranges stay allowed: a Tailscale tailnet (CGNAT, `100.100.100.100` included) and an IPv6 cluster on ULA addresses need no change. -*On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* A fourth check, the `catalogAuth` host check of the Kafka Connect and embedded Debezium Iceberg sink, refuses the same addresses, even when the host is listed. See [Operator allowlists](source-aligned-acquisition.md#operator-allowlists). +*(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828))* A fourth check, the `catalogAuth` host check of the Kafka Connect and embedded Debezium Iceberg sink, refuses the same addresses, even when the host is listed. See [Operator allowlists](source-aligned-acquisition.md#operator-allowlists). ## Cloud metadata and the credential resolver diff --git a/docs/advanced/production-troubleshooting.md b/docs/advanced/production-troubleshooting.md index 4050615..eb32c5c 100644 --- a/docs/advanced/production-troubleshooting.md +++ b/docs/advanced/production-troubleshooting.md @@ -272,11 +272,11 @@ Update the key at the source `fluid ai status` reports, the provider environment | `fluid mcp serve`, `fluid mcp output-port serve`, `fluid scaffold-ide`, `mcp-sampling` (it runs inside `fluid mcp serve`) | `mcp` | `mcp`, or `standard` | | Lineage events when `OPENLINEAGE_URL` is set | `openlineage-python` | `lineage` | -*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `litellm`, `mcp` and `lineage` rows above describe a base install from 0.28.0. +The `litellm`, `mcp` and `lineage` rows above describe a base install. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* -Through 0.27.0 a base install contains all three, so you meet this only if you removed one. From 0.28.0 a base install omits them. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step. +A base install omits all three; through 0.27.0 it contained them, so on those releases you meet this only if you removed one. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step. -*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* From 0.28.0 an explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) stop before they do any work. The forge message names `--llm-provider claude-code` and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. `fluid init` does not exit 1: it prints the line and the options, then continues in template mode. Lineage is the exception: without `lineage`, `fluid apply` through the OpenTofu engine and acquisition builds print one warning on the console (and log it) and carry on; a native local apply never emits lineage. AWS commands that need boto3 point at `[aws]`; boto3 no longer arrives through litellm. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). +An explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) stop before they do any work. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The forge message names `--llm-provider claude-code` and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. `fluid init` does not exit 1: it prints the line and the options, then continues in template mode. Lineage is the exception: without `lineage`, `fluid apply` through the OpenTofu engine and acquisition builds print one warning on the console (and log it) and carry on; a native local apply never emits lineage. AWS commands that need boto3 point at `[aws]`; boto3 no longer arrives through litellm. See [Upgrading](../upgrading.md#changes-in-0-28-0). ## Where logs live diff --git a/docs/advanced/project-dotenv-trust.md b/docs/advanced/project-dotenv-trust.md index de1d4ab..ed32c2f 100644 --- a/docs/advanced/project-dotenv-trust.md +++ b/docs/advanced/project-dotenv-trust.md @@ -75,7 +75,7 @@ What it may not set is any variable that project file has nothing to supply there. The full list is `fluid_build/credentials/protected_env.py` in forge-cli. -*On forge-cli `main`, not yet in a release ([forge-cli #833](https://github.com/Agenticstiger/forge-cli/pull/833)).* The embedded Debezium Server inherits the environment `fluid apply` loads, and its SmallRye Config reads a variable over `application.properties`. Its JVM reads `-D` system properties, which outrank both, from `JDK_JAVA_OPTIONS` or the launcher's `JAVA_OPTS`. So a project file could send a `sink.catalogAuth` credential to a catalog the operator never paired with it, change the source database, load another config file, or start another `java`. A project dotenv file can no longer set `DEBEZIUM_*`, `QUARKUS_*`, `SMALLRYE_*` or `MP_CONFIG_*` variables in the forms SmallRye Config matches, profile forms included (`debezium.sink.iceberg.uri`, `_PROD_DEBEZIUM_...`, `%prod....`), nor `JDK_JAVA_OPTIONS`, `JAVA_OPTS`, `JAVA_HOME`, or the launcher's `EXTRA_CONNECTOR`, `JMX_HOST`, `JMX_PORT`, `JMX_EXPORTER_PORT`, `JMX_EXPORTER_CONFIG` and `OTEL_JAVAAGENT_*`. Each is ignored with the usual warning. Set it in the shell or CI instead. +*(since 0.28.0, [forge-cli #833](https://github.com/Agenticstiger/forge-cli/pull/833))* The embedded Debezium Server inherits the environment `fluid apply` loads, and its SmallRye Config reads a variable over `application.properties`. Its JVM reads `-D` system properties, which outrank both, from `JDK_JAVA_OPTIONS` or the launcher's `JAVA_OPTS`. So a project file could send a `sink.catalogAuth` credential to a catalog the operator never paired with it, change the source database, load another config file, or start another `java`. A project dotenv file can no longer set `DEBEZIUM_*`, `QUARKUS_*`, `SMALLRYE_*` or `MP_CONFIG_*` variables in the forms SmallRye Config matches, profile forms included (`debezium.sink.iceberg.uri`, `_PROD_DEBEZIUM_...`, `%prod....`), nor `JDK_JAVA_OPTIONS`, `JAVA_OPTS`, `JAVA_HOME`, or the launcher's `EXTRA_CONNECTOR`, `JMX_HOST`, `JMX_PORT`, `JMX_EXPORTER_PORT`, `JMX_EXPORTER_CONFIG` and `OTEL_JAVAAGENT_*`. Each is ignored with the usual warning. Set it in the shell or CI instead. ::: tip Two more since `0.23.0` `FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, which allows an acquisition source's `env://` diff --git a/docs/advanced/source-aligned-acquisition.md b/docs/advanced/source-aligned-acquisition.md index 3e04120..0913cf5 100644 --- a/docs/advanced/source-aligned-acquisition.md +++ b/docs/advanced/source-aligned-acquisition.md @@ -524,7 +524,7 @@ The same contract declaring `fluidVersion: "0.7.5"` fails, because the 0.7.5 sch - **`oauth2ServerUri`** starts with a lowercase `http://` or `https://` and has no `user:password@` part. *(since 0.27.0, [forge-cli #818](https://github.com/Agenticstiger/forge-cli/pull/818))* `fluid validate` and the run then refuse `http://`; see [Operator allowlists](#operator-allowlists). Iceberg treats any other spelling as a path on the catalog host and sends the credentials there. The query string is not checked, so keep secrets out of it too. - **`oauth2ServerUri`, `scope`, `audience` and `resource`** are written as they are. None may hold a `${...}` expression, which Debezium Server would expand in `application.properties` while Kafka Connect would pass it to Iceberg as text, a control character such as a line break or a tab, or a value starting with `env:`, which Iceberg would resolve from the environment. Only the secret comes from the environment. -- **Override text.** *On forge-cli `main`, not yet in a release ([forge-cli #830](https://github.com/Agenticstiger/forge-cli/pull/830)).* With `catalogAuth`, a catalog URL that holds `${`, starts with `env:` or holds a control character is refused wherever it is set, in `binding.location.uri` or in an override map, at `fluid validate` and in the run's preflight. The check reads the text before the sink expands it: Debezium Server expands a `${...}` expression when it starts, Kafka Connect resolves a `${provider:path:key}` reference with the worker's config providers, and Iceberg replaces a value that starts with `env:` by that variable. The same text is refused in every other override value Iceberg reads: Kafka Connect `iceberg_catalog_overrides`, the `iceberg.*` keys of `sink_connector_config` except `iceberg.kafka.*`, and every key of an embedded Debezium `server.sink.config`. A storage credential comes from the worker's or the server's own credential chain, not from the contract. Kafka-side keys such as `consumer.override.*` and `iceberg.kafka.*` keep their config providers. A build without `catalogAuth` is unchanged. +- **Override text.** *(since 0.28.0, [forge-cli #830](https://github.com/Agenticstiger/forge-cli/pull/830))* With `catalogAuth`, a catalog URL that holds `${`, starts with `env:` or holds a control character is refused wherever it is set, in `binding.location.uri` or in an override map, at `fluid validate` and in the run's preflight. The check reads the text before the sink expands it: Debezium Server expands a `${...}` expression when it starts, Kafka Connect resolves a `${provider:path:key}` reference with the worker's config providers, and Iceberg replaces a value that starts with `env:` by that variable. The same text is refused in every other override value Iceberg reads: Kafka Connect `iceberg_catalog_overrides`, the `iceberg.*` keys of `sink_connector_config` except `iceberg.kafka.*`, and every key of an embedded Debezium `server.sink.config`. A storage credential comes from the worker's or the server's own credential chain, not from the contract. Kafka-side keys such as `consumer.override.*` and `iceberg.kafka.*` keep their config providers. A build without `catalogAuth` is unchanged. A message about a `catalogAuth` field that fails a check never shows the field's value, because it may be a secret pasted in place of a name. A variable name appears in a message only after it has passed the name checks. @@ -581,7 +581,7 @@ For each host the credential reaches, the run needs: - **The pairing**, `NAME@host` in `FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, unless the host is loopback. - **A host entry for a hostname**, in `FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST`, because the worker (or Debezium Server) resolves the name after forge-cli's check, so no lookup made by forge-cli shows where it will connect. `localhost` needs none. -An IP address keeps the deployment host list's rules instead of needing an entry as a name does: a link-local or cloud metadata address is never allowed, a private one needs no host entry (only the pairing), and a public one only when listed. A loopback host needs neither entry. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* +An IP address keeps the deployment host list's rules instead of needing an entry as a name does: a link-local or cloud metadata address is never allowed, a private one needs no host entry (only the pairing), and a public one only when listed. A loopback host needs neither entry. *(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828))* An OAuth2 credential reaches two hosts, so it needs a pairing for the token endpoint and one for the catalog (one entry covers both when they are the same host, as with the Polaris recipe below). The example contract needs: @@ -599,19 +599,19 @@ Two cautions on how entries match: So prefer `NAME@host`, and list the exact host. -The run refuses before it makes any Connect REST call, writes `application.properties` or reads the variable. `fluid validate` may run where the operator's lists are not set, so for a host that only lacks an entry it warns, with the `export` line the run needs (the warning above), and does not fail. It refuses outright, with no `export` line, an `http://` URL and a link-local, unspecified or cloud metadata address, which no entry allows. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* With `oauth2ServerUri: http://keycloak:8080/...`: +The run refuses before it makes any Connect REST call, writes `application.properties` or reads the variable. `fluid validate` may run where the operator's lists are not set, so for a host that only lacks an entry it warns, with the `export` line the run needs (the warning above), and does not fail. It refuses outright, with no `export` line, an `http://` URL and a link-local, unspecified or cloud metadata address, which no entry allows. *(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828))* With `oauth2ServerUri: http://keycloak:8080/...`: ```text 1. iceberg sink (build 'stream_orders'): sink.catalogAuth.oauth2ServerUri must be an https:// URL: the OAuth2 token endpoint receives the credential in LAKEKEEPER_CREDENTIAL (the client id and secret) on every request, and http would carry it in clear text, so a loopback host needs https too ``` -With `binding.location.uri: https://169.254.169.254`, a cloud metadata address, the message reads as follows. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* +With `binding.location.uri: https://169.254.169.254`, a cloud metadata address, the message reads as follows. *(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828))* ```text 1. iceberg sink (build 'stream_orders'): the Iceberg catalog host '169.254.169.254' (binding.location.uri) is link-local, unspecified or a known cloud metadata endpoint; no Iceberg catalog is ever there, and this is refused even with an allowlist entry ``` -Both lists are operator-only: a value set in a project `.env`, `.env.` or `FLUID_SECRETS_FILE` is ignored (`credentials/protected_env.py` lists both). Set them in the shell or the CI job that runs `fluid`. [Network safety](network-safety.md) lists the further metadata addresses that the orchestrator guard, the OpenLineage emitter and the managed Flink host check refuse; this sink check refuses those too. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* It refuses the same addresses as the orchestrator guard, the OpenLineage emitter and the managed Flink host check, even when listed. Before that change it refused link-local and unspecified addresses only. +Both lists are operator-only: a value set in a project `.env`, `.env.` or `FLUID_SECRETS_FILE` is ignored (`credentials/protected_env.py` lists both). Set them in the shell or the CI job that runs `fluid`. [Network safety](network-safety.md) lists the further metadata addresses that the orchestrator guard, the OpenLineage emitter and the managed Flink host check refuse; this sink check refuses those too. *(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828))* It refuses the same addresses as the orchestrator guard, the OpenLineage emitter and the managed Flink host check, even when listed. Before that change it refused link-local and unspecified addresses only. Related text checks, which apply whether or not the sink has a `catalogAuth`: `binding.location` text that reaches the derived sink config may not start with `env:`, hold `${`, or hold a control character, and `binding.location.region` must be one host label (letters, digits and inner hyphens, at most 63). Only `sink.catalogAuth` may name a variable. A value is never quoted in the message. With `warehouse: "env:AWS_SECRET_ACCESS_KEY"`: @@ -1370,7 +1370,7 @@ The emitter is chosen by environment, not by a key in the contract. With no endp export OPENLINEAGE_URL=https://marquez.example.com ``` -Emitting needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* +Emitting needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* `FLUID_OPENLINEAGE_URL` takes precedence over `OPENLINEAGE_URL`. The endpoint path defaults to `api/v1/lineage` (change it with `FLUID_OPENLINEAGE_ENDPOINT`), and `OPENLINEAGE_API_KEY` or `FLUID_OPENLINEAGE_API_KEY` sets the bearer token. A private or loopback endpoint is allowed unless `FLUID_OPENLINEAGE_ALLOW_PRIVATE=false`; link-local addresses, such as cloud metadata endpoints, are blocked either way. diff --git a/docs/cli/README.md b/docs/cli/README.md index 61f3e5b..8186073 100644 --- a/docs/cli/README.md +++ b/docs/cli/README.md @@ -1,6 +1,6 @@ # CLI Reference -This section tracks the command surface of `forge-cli` `0.27.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands. +This section tracks the command surface of `forge-cli` `0.28.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands. ::: tip Looking up *how to do something* (not *what a command does*)? The page you actually want is [**CLI by task →**](./tasks/) — narrative walkthroughs organized around things you might be trying to accomplish, like "deploy to a new cloud", "add quality rules", "debug a failed run", or "add agent governance". Each task page links to the relevant commands. **This** page is the command index, grouped by purpose — better for lookup, worse for onboarding. diff --git a/docs/cli/ai.md b/docs/cli/ai.md index 474f078..bd38c66 100644 --- a/docs/cli/ai.md +++ b/docs/cli/ai.md @@ -2,7 +2,7 @@ Configure the AI / LLM provider used by Forge Copilot, test that it works, and inspect the current AI configuration. -`fluid ai setup` needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, it prints that line, names the keyless ways to author (`fluid forge --llm-provider claude-code|codex|cursor|kiro`, `--llm-provider mcp-sampling`, `fluid forge --template starter`) and exits 1 before it reads the OS keyring or asks for a key. +The `fluid ai setup` provider wizard needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`; `--clear` and `--source` do not. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without it, the wizard prints that line, names the keyless ways to author (`fluid forge --llm-provider claude-code|codex|cursor|kiro`, `--llm-provider mcp-sampling`, `fluid forge --template starter`) and exits 1 before it reads the OS keyring or asks for a key. ## Syntax diff --git a/docs/cli/contract-tests.md b/docs/cli/contract-tests.md index 870da3d..7fa4640 100644 --- a/docs/cli/contract-tests.md +++ b/docs/cli/contract-tests.md @@ -87,7 +87,7 @@ The contract is loaded the way `fluid plan` loads it, with `$ref` fragments and ```yaml - name: Schema gate run: | - pip install data-product-forge==0.27.0 + pip install data-product-forge==0.28.0 fluid contract-tests contract.fluid.yaml --baseline baseline.schema.json ``` diff --git a/docs/cli/doctor.md b/docs/cli/doctor.md index 18806e5..9d5faae 100644 --- a/docs/cli/doctor.md +++ b/docs/cli/doctor.md @@ -203,7 +203,7 @@ The exit code is `1`. Leave `--extended` out unless your project provides its ow AI/MCP move to extras in 0.28.0: pip install 'data-product-forge[standard]' ``` -*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* After the change it reads `AI and MCP are extras since 0.28.0: pip install 'data-product-forge[standard]'`, and a configured provider is no longer reported as ready when litellm is missing: the Next steps panel says ` needs litellm, which is not installed` and prints `pip install 'data-product-forge[litellm]'`. +*(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* On 0.28.0 the notice reads `AI and MCP are extras since 0.28.0: pip install 'data-product-forge[standard]'` and prints only when litellm or mcp is not installed (0.27.0 printed it every run), and a configured provider is no longer reported as ready when litellm is missing: the Next steps panel says ` needs litellm, which is not installed` and prints `pip install 'data-product-forge[litellm]'`. It also reports three optional packages in the AI checks: the LiteLLM backend, the MCP SDK and the OpenLineage client. Each shows its version when installed, or `Not installed (optional)` with the `pip install 'data-product-forge[]'` line. None of them is imported: doctor reads the version from the package metadata, so it makes no network call for litellm's cost map. @@ -219,7 +219,7 @@ The `fix:` line for a missing binary, and the `doc` field in `--json` output, po | `helm`, `kubectl` | Managed-mode acquisition engines that provision infrastructure with Helm on Kubernetes. | [Helm install guide](https://helm.sh/docs/intro/install/) and [Kubernetes tools](https://kubernetes.io/docs/tasks/tools/). | | `duckdb` | The `local` provider and DuckDB acquisition builds. | `pip install "data-product-forge[local]"`, which installs `duckdb` at the version the CLI requires for its contract-SQL sandbox. See [DuckDB sandbox](../advanced/duckdb-sandbox.md). | | `dlt` | `dlt` acquisition builds. | `pip install dlt`. As of 0.18.1 `data-product-forge` has no `dlt` extra. | -| `litellm`, `mcp`, `openlineage` | *(from 0.27.0)* AI forge, `fluid mcp serve` and lineage emission. | Part of a base install through 0.27.0. From 0.28.0: `pip install "data-product-forge[standard]"`, or `[ai]`, `[mcp]` or `[lineage]` for one. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* | +| `litellm`, `mcp`, `openlineage` | *(from 0.27.0)* AI forge, `fluid mcp serve` and lineage emission. | Part of a base install through 0.27.0; since 0.28.0 install them with `pip install "data-product-forge[standard]"`, or `[ai]`, `[mcp]` or `[lineage]` for one. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* | | `httpx` | A core dependency of the CLI. | Installed with `data-product-forge`; a failure here points to a broken Python environment. | | `snowflake.connector` | The `snowflake` provider. | `pip install "data-product-forge[snowflake]"`. The `fix:` line for this check reads `pip install snowflake`, which is not the package that provides the connector; use the extra. | diff --git a/docs/cli/forge.md b/docs/cli/forge.md index 29509e1..3352a7a 100644 --- a/docs/cli/forge.md +++ b/docs/cli/forge.md @@ -2,7 +2,7 @@ Author a data product contract with an AI copilot, an offline guided interview, or an empty skeleton. By default forge writes `contract.fluid.yaml` and `.fluid/forge-receipt.json`; a scaffold (`--scaffold`), a CI pipeline (`--ci`) and engine artifacts add files. -AI forge with an API-key provider needs the `litellm` extra (alias `ai`): `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, an explicit `--llm-provider` that names an API-key provider (`openai`, `anthropic`, `gemini`, `ollama`), on `fluid forge`, on `fluid forge data-model` or with `--agent-loop`, prints that line and exits 1 with `[ERR_COPILOT_LITELLM_UNAVAILABLE]`, before the interview reads any context or memory. `fluid forge data-model` with no `--llm-provider`, `--llm-model` or `--llm-endpoint` runs offline and needs no extra (`--llm-model` alone resolves to `ollama`). The forge message names two ways to author without it: `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`, because the agent CLIs lack tool use. `codex`, `cursor` and `kiro` also work without litellm, but they reuse their own CLI login or key. `--blank` and the other templates need no AI. `--llm-provider mcp-sampling` works only inside `fluid mcp serve`, which needs the `mcp` extra. Bare interactive `fluid forge` with no provider prints the hint and offers guided mode. +AI forge with an API-key provider needs the `litellm` extra (alias `ai`): `pip install 'data-product-forge[litellm]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without it, an explicit `--llm-provider` that names an API-key provider (`openai`, `anthropic`, `gemini`, `ollama`), on `fluid forge`, on `fluid forge data-model` or with `--agent-loop`, prints that line and exits 1 with `[ERR_COPILOT_LITELLM_UNAVAILABLE]`, before the interview reads any context or memory. `fluid forge data-model` with no `--llm-provider`, `--llm-model` or `--llm-endpoint` runs offline and needs no extra (`--llm-model` alone resolves to the provider saved with `fluid ai setup`, and to `ollama` when none is saved). The forge message names two ways to author without it: `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`, because the agent CLIs lack tool use. `codex`, `cursor` and `kiro` also work without litellm, but they reuse their own CLI login or key. `--blank` and the other templates need no AI. `--llm-provider mcp-sampling` works only inside `fluid mcp serve`, which needs the `mcp` extra. Bare interactive `fluid forge` with no provider uses Claude Code when it is installed, and otherwise prints the hint and offers guided mode. ## Syntax diff --git a/docs/cli/init.md b/docs/cli/init.md index 0f0e2a2..b575e97 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -8,7 +8,7 @@ Create a new project: a working example, an empty skeleton, a bundled blueprint, fluid init [NAME] [OPTIONS] ``` -With no mode flag, `fluid init NAME` starts the AI-assisted path (the same copilot as [`fluid forge`](./forge.md)), which needs `pip install "data-product-forge[litellm]"` for an API-key provider. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Every example below picks a mode explicitly, so none of them needs an AI key. +With no mode flag, `fluid init NAME` starts the AI-assisted path (the same copilot as [`fluid forge`](./forge.md)), which needs `pip install "data-product-forge[litellm]"` for an API-key provider. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Every example below picks a mode explicitly, so none of them needs an AI key. ## Examples diff --git a/docs/cli/mcp.md b/docs/cli/mcp.md index c816fdf..682c1a8 100644 --- a/docs/cli/mcp.md +++ b/docs/cli/mcp.md @@ -25,7 +25,7 @@ fluid mcp output-port serve ... # consumer-side data-access server Runs the authoring MCP stdio server so an MCP client can inspect, validate, and edit forge artifacts. It is a foreground process — no daemon, no HTTP port, no background service. -Needs the `mcp` extra: `pip install 'data-product-forge[mcp]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it the command prints that line and exits 1, with `[ERR_MCP_SDK_UNAVAILABLE]` and no traceback. +Needs the `mcp` extra: `pip install 'data-product-forge[mcp]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without it the command prints that line and exits 1, with `[ERR_MCP_SDK_UNAVAILABLE]` and no traceback. ### Syntax @@ -83,8 +83,8 @@ Both servers run on the official `mcp` Python SDK. | | | | --- | --- | -| **Dependency pin** | `mcp>=1.22,<3.0` *(since 0.20.0, [forge-cli #710](https://github.com/Agenticstiger/forge-cli/pull/710); 0.19.0 and earlier pin `mcp>=1.20,<3.0`)*. The `mcp` extra then needs `mcp>=1.30,!=2.0.*,!=2.1.*,<3.0`, because earlier releases have published advisories. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* | -| **What a fresh install resolves** | The newest `2.x`. Up to 0.27.0 `mcp` is a core dependency, so `pip install data-product-forge` picks it up. *(from 0.27.0)* The `mcp` extra (and `standard`) names it; *(from 0.28.0)* a base install no longer includes it, and `fluid mcp serve`, MCP output ports, `fluid scaffold-ide` and the keyless `mcp-sampling` provider need `pip install "data-product-forge[mcp]"`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A fresh 0.18.1 install resolved `mcp 2.3.0` on 5 Oct 2026. | +| **Dependency pin** | `mcp>=1.22,<3.0` *(since 0.20.0, [forge-cli #710](https://github.com/Agenticstiger/forge-cli/pull/710); 0.19.0 and earlier pin `mcp>=1.20,<3.0`)*. The `mcp` extra needs `mcp>=1.30,!=2.0.*,!=2.1.*,<3.0`, because earlier releases have published advisories. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* | +| **What a fresh install resolves** | The newest `2.x`. Up to 0.27.0 `mcp` was a core dependency, so `pip install data-product-forge` picked it up. *(from 0.27.0)* The `mcp` extra (and `standard`) names it. A base install does not include it, and `fluid mcp serve`, MCP output ports, `fluid scaffold-ide` and the keyless `mcp-sampling` provider need `pip install "data-product-forge[mcp]"`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* A fresh 0.18.1 install resolved `mcp 2.3.0` on 5 Oct 2026. | | **Existing environments** | Do not move until you upgrade them. Pinning `mcp>=1.22,<2.0` yourself remains supported. mcp 1.20 and 1.21 fail to import with pydantic 2.14 or later, which removed a private helper they import; keep pydantic below 2.14 if you must stay on them. | | **3.x** | Outside the pin. | @@ -100,7 +100,7 @@ On 1.x a client could self-attest identity (`model`, `useCase`, tenant attribute Binds **one expose** from a FLUID contract and serves it to MCP clients as a governed, read-only data port. This is distinct from `fluid mcp serve`: the output port queries production data, so its threat model and policy surface differ entirely. -Needs the `mcp` extra as well (`pip install 'data-product-forge[mcp]'`); add `local` for the DuckDB engine, as in `pip install 'data-product-forge[local,mcp]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* +Needs the `mcp` extra as well (`pip install 'data-product-forge[mcp]'`); add `local` for the DuckDB engine, as in `pip install 'data-product-forge[local,mcp]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The server is **read-only by default** and exposes a small surface: an agent can `describe` the data product, `sample` a few rows, and run a predeclared **semantic** `query`. Free-form SQL (`query_sql`) is OFF unless you opt in with `--allow-sql`. diff --git a/docs/cli/mission.md b/docs/cli/mission.md index 791e857..d075fe2 100644 --- a/docs/cli/mission.md +++ b/docs/cli/mission.md @@ -249,7 +249,7 @@ cosmetics ([`0.13.0` security fixes](../RELEASE_NOTES_0.13.0.md#security)). ```yaml - name: Mission gate run: | - pip install data-product-forge==0.27.0 + pip install data-product-forge==0.28.0 fluid mission check quality-coverage contract.fluid.yaml ``` diff --git a/docs/cli/providers.md b/docs/cli/providers.md index 482cbe7..faa0bb4 100644 --- a/docs/cli/providers.md +++ b/docs/cli/providers.md @@ -136,7 +136,7 @@ Raised when an action in an execution plan names a provider that is missing from pip install 'data-product-forge[gcp]' ``` -The package defines `local`, `gcp`, `snowflake` and `aws` extras, and others for catalogs and tooling; [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0) lists them all. +The package defines `local`, `gcp`, `snowflake` and `aws` extras, and others for catalogs and tooling; [Upgrading](../upgrading.md#changes-in-0-28-0) lists them all. diff --git a/docs/cli/scaffold-ci.md b/docs/cli/scaffold-ci.md index 4208ee3..ee5a04b 100644 --- a/docs/cli/scaffold-ci.md +++ b/docs/cli/scaffold-ci.md @@ -41,7 +41,7 @@ The generated file installs `data-product-forge` from PyPI (`pip install --quiet | publish | On `main` with `DMM_API_URL` set: `fluid publish $CONTRACT --catalog $CATALOG`. | ::: warning Pin the version in a pipeline that holds credentials -An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.27.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file. +An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.28.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file. ::: The settings are `CONTRACT` (the path you passed), `PROVIDER` (default `default`), `BUILD_ID` (default empty; set it to a `builds[].id` to run that build), `CATALOG` (default `datamesh-manager`), `AIRFLOW_DAGS_DEST` and `DMM_API_URL`. GitLab reads them as CI/CD variables, GitHub Actions as `env`, repository `vars` and `secrets`, Jenkins from the `environment` block. diff --git a/docs/cli/scaffold-ide.md b/docs/cli/scaffold-ide.md index fd84b2d..297562d 100644 --- a/docs/cli/scaffold-ide.md +++ b/docs/cli/scaffold-ide.md @@ -2,7 +2,7 @@ Generate agentic-IDE configuration so an AI-assisted editor can drive Fluid Forge — steering rules, hooks, and an MCP server entry, written from one pack. -Needs the `mcp` extra: `pip install 'data-product-forge[mcp]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it the command writes nothing, prints that line and exits 1 with `[ERR_MCP_SDK_UNAVAILABLE]`, because every entry it writes launches `fluid mcp serve`. +Needs the `mcp` extra: `pip install 'data-product-forge[mcp]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without it the command writes nothing, prints that line and exits 1 with `[ERR_MCP_SDK_UNAVAILABLE]`, because every entry it writes launches `fluid mcp serve`. ## Syntax diff --git a/docs/cli/version.md b/docs/cli/version.md index d89180d..bb66151 100644 --- a/docs/cli/version.md +++ b/docs/cli/version.md @@ -69,7 +69,7 @@ Python: 3.11.15 (darwin) System: Darwin arm64 ``` -`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.27.0`. +`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.28.0`. ## Options diff --git a/docs/concepts/quality-sla-lineage.md b/docs/concepts/quality-sla-lineage.md index 38a8c08..c94886d 100644 --- a/docs/concepts/quality-sla-lineage.md +++ b/docs/concepts/quality-sla-lineage.md @@ -264,7 +264,7 @@ dist/artifacts/policy/bindings.json | **ODPS-Bitol** (Bitol Open Data Product Standard) | `odps-bitol/` | Bitol-aware catalogs | | **OPDS** (Open Data Product Specification v4.1, LF/ODPI) | `opds/` | Generic catalog ingest | -OpenLineage is not written to a file. Runs send OpenLineage run events over HTTP when `OPENLINEAGE_URL` (or `FLUID_OPENLINEAGE_URL`) is set: acquisition builds, and `fluid apply` through OpenTofu. Marquez and DataHub accept them. With no URL set, nothing is sent. Sending needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, those runs print one warning naming the extra on the console (and log it) and carry on without sending events. A native local apply never emits lineage. +OpenLineage is not written to a file. Runs send OpenLineage run events over HTTP when `OPENLINEAGE_URL` (or `FLUID_OPENLINEAGE_URL`) is set: acquisition builds, and `fluid apply` through OpenTofu. Marquez and DataHub accept them. With no URL set, nothing is sent. Sending needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without it, those runs print one warning naming the extra on the console (and log it) and carry on without sending events. A native local apply never emits lineage. ## Where to look next diff --git a/docs/faq/README.md b/docs/faq/README.md index d227d12..5c6ab4e 100644 --- a/docs/faq/README.md +++ b/docs/faq/README.md @@ -39,7 +39,7 @@ The same cause as above: the `[local]` extra is missing. Install `data-product-f ## A command says `needs , which is not installed`. -*(from 0.27.0)* An optional package is missing. The message names the extra: run the `pip install 'data-product-forge[]'` line it prints. From 0.28.0 the AI, MCP and lineage packages are no longer in the base install, so `pip install 'data-product-forge[standard]'` restores AI forge and MCP, or name one of `[litellm]`, `[mcp]` and `[lineage]`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* See [Production troubleshooting](/forge_docs/advanced/production-troubleshooting.html#an-optional-package-is-not-installed). +*(from 0.27.0)* An optional package is missing. The message names the extra: run the `pip install 'data-product-forge[]'` line it prints. The AI, MCP and lineage packages are not in the base install (they were through 0.27.0), so `pip install 'data-product-forge[standard]'` restores AI forge and MCP, or name one of `[litellm]`, `[mcp]` and `[lineage]`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* See [Production troubleshooting](/forge_docs/advanced/production-troubleshooting.html#an-optional-package-is-not-installed). ## A local output file contains `id,value` and `1,materialized`. @@ -58,7 +58,7 @@ pip install --upgrade "data-product-forge[local]" fluid version ``` -Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.27.0`), not a range. +Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.28.0`), not a range. Your contracts do not need a new `fluidVersion`: `0.27.0` reads `0.7.1` to `0.7.6`, and `0.7.7` as a preview. diff --git a/docs/forge-data-model.md b/docs/forge-data-model.md index ab5fe60..c5d2c37 100644 --- a/docs/forge-data-model.md +++ b/docs/forge-data-model.md @@ -181,7 +181,7 @@ For dbt output, the generator fails clearly if it produces zero `models/**/*.sql ## AI provider runs -A run that calls a provider with an API key needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it the command prints that line and exits 1 before it does any work. `--llm-provider claude-code`, `codex`, `cursor` and `kiro` use that agent's own login instead. +A run that calls a provider with an API key needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without it the command prints that line and exits 1 before it does any work. `--llm-provider claude-code`, `codex`, `cursor` and `kiro` use that agent's own login instead. Provider keys are never part of the intent file or contract. Export one provider key for the shell session or run `fluid ai setup`: diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md index ff5575d..a62239c 100644 --- a/docs/getting-started/README.md +++ b/docs/getting-started/README.md @@ -51,7 +51,7 @@ The `local` extra installs DuckDB (`duckdb>=1.5.0`), pandas and numpy, which the | `lineage` | *(from 0.27.0)* Emitting OpenLineage events when `OPENLINEAGE_URL` is set | `openlineage-python` | | `standard` | *(from 0.27.0)* `litellm` and `mcp` together: the AI and MCP packages a 0.27.0 base install has | `litellm`, `mcp` | -The table lists the extras a first project needs. From 0.28.0 a base install has none of `litellm`, `mcp` and `openlineage-python`, so name the extra for any AI, MCP or lineage feature you use, for example `pip install "data-product-forge[local,litellm]"`; it works on 0.27.0 too. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0) lists every extra, and what changes for a base install in 0.28.0. +The table lists the extras a first project needs. A base install has none of `litellm`, `mcp` and `openlineage-python` (it had them through 0.27.0), so name the extra for any AI, MCP or lineage feature you use, for example `pip install "data-product-forge[local,litellm]"`; it works on 0.27.0 too. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* [Upgrading](../upgrading.md#changes-in-0-28-0) lists every extra, and what changes for a base install in 0.28.0. ```bash pip install "data-product-forge[local,gcp]" # several extras at once diff --git a/docs/providers/README.md b/docs/providers/README.md index 52aff1e..04508f1 100644 --- a/docs/providers/README.md +++ b/docs/providers/README.md @@ -8,7 +8,7 @@ Fluid Forge uses one contract format across local and provider-backed execution ## Docs baseline -- CLI release covered by the primary docs: `0.27.0` +- CLI release covered by the primary docs: `0.28.0` - Contract schema: `0.7.6` is the current stable version, and the default since `0.26.0`; `0.7.7` is a preview you opt into with `fluidVersion: "0.7.7"`. The GCP retention, encryption and `binding.principals` fields need `0.7.6`. - Which `fluidVersion` each scaffolding command writes is listed in [`fluid init`](../cli/init.md#which-fluidversion-each-path-writes). diff --git a/docs/providers/aws.md b/docs/providers/aws.md index f27de7f..9309fa2 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -437,7 +437,7 @@ fluid odcs export contract.fluid.yaml --output standards/product.odcs.yaml `fluid diff` reads the Glue table in the region `fluid apply` uses and compares its columns and types with the contract. Since `0.18.1` its live check resolves `{{ env.* }}` placeholders in the binding before it reads the target. -`fluid diff` and `fluid verify` read AWS through boto3; install it with `pip install 'data-product-forge[aws]'`. Builds that land data with DuckDB need the `local` extra (`duckdb>=1.5.0`). boto3 used to arrive through litellm, so a base install of 0.28.0 no longer has it: `fluid plan`, `fluid validate` and `fluid generate iac` run without it; install `[aws]` (or set `AWS_ACCOUNT_ID`, where only the account id is needed) before any command that calls AWS or needs the account id: `diff`, `verify`, live reads and `apply`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* +AWS reads go through boto3; install it with `pip install 'data-product-forge[aws]'`. Builds that land data with DuckDB need the `local` extra (`duckdb>=1.5.0`). boto3 used to arrive through litellm, so a base install no longer has it: `fluid plan`, `fluid validate` and `fluid generate iac` run without it; install `[aws]` before a command that calls AWS. A native AWS plan that cannot resolve the account id is refused, but only for a contract that declares orchestration (`orchestration.schedule`, `triggers` or `tasks`); other contracts skip the native plan. `fluid apply` without boto3 warns that the Glue catalog import blocks are omitted. Setting `AWS_ACCOUNT_ID` avoids the account lookup. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* ## Where a build lands data diff --git a/docs/recipes/per-environment-overlays.md b/docs/recipes/per-environment-overlays.md index 2504e83..b1a9c1a 100644 --- a/docs/recipes/per-environment-overlays.md +++ b/docs/recipes/per-environment-overlays.md @@ -250,7 +250,7 @@ jobs: matrix: env: [dev, staging, prod] steps: - - run: pip install "data-product-forge==0.27.0" + - run: pip install "data-product-forge==0.28.0" - run: fluid validate contract.fluid.yaml --env ${{ matrix.env }} - run: fluid plan contract.fluid.yaml --env ${{ matrix.env }} ``` diff --git a/docs/reference/contract-0.7.6.md b/docs/reference/contract-0.7.6.md index 0daa9d3..a938e9e 100644 --- a/docs/reference/contract-0.7.6.md +++ b/docs/reference/contract-0.7.6.md @@ -9,7 +9,7 @@ description: "The fields of the FLUID 0.7.6 contract schema (stable): path, type Status: **stable**. Schema 0.7.6 is the newest stable schema bundled with the CLI. The newer 0.7.7 schema is a preview; see the [0.7.7 preview delta](./contract-0.7.7-preview.md). -Generated from `fluid_build/schemas/fluid-schema-0.7.6.json` in the `data-product-forge` 0.27.0 package (sha256 `61ca91a63087b89b`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. +Generated from `fluid_build/schemas/fluid-schema-0.7.6.json` in the `data-product-forge` 0.28.0 package (sha256 `61ca91a63087b89b`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. Each table row is one field, addressed by its path from the contract root: `.` descends into an object, `[]` marks the items of a list, and `` stands for any key of a map. **Required** means required inside its parent object; the parent can itself be optional. **Allowed values** lists the enum, pattern and range the schema enforces. An object typed plainly `object` rejects keys the schema does not list; `object (open)` accepts extra keys and `object (free-form)` has no listed keys. A row that begins "Only when ..." exists only under that condition. See [how to read this reference](./README.md#how-to-read-a-row). diff --git a/docs/reference/contract-0.7.7-preview.md b/docs/reference/contract-0.7.7-preview.md index b5e7593..e21911f 100644 --- a/docs/reference/contract-0.7.7-preview.md +++ b/docs/reference/contract-0.7.7-preview.md @@ -9,7 +9,7 @@ description: "What the FLUID 0.7.7 preview schema adds to, and changes in, the s Status: **preview**. A contract opts in with `fluidVersion: "0.7.7"`; a contract that declares 0.7.6 is unaffected. This page lists the difference between the 0.7.6 schema and the 0.7.7 schema as bundled with the CLI: added fields, fields whose definition changed, and fields removed. A field not listed here has the same type, required flag, allowed values, default and description in both; for those, use the [0.7.6 reference](./contract-0.7.6.md). A definition used at several paths is listed once, at its first path. -Generated from `fluid_build/schemas/fluid-schema-0.7.7.json` in the `data-product-forge` 0.27.0 package (sha256 `0f11fb44bc9dafbc`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. +Generated from `fluid_build/schemas/fluid-schema-0.7.7.json` in the `data-product-forge` 0.28.0 package (sha256 `0f11fb44bc9dafbc`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. Each table row is one field, addressed by its path from the contract root: `.` descends into an object, `[]` marks the items of a list, and `` stands for any key of a map. **Required** means required inside its parent object; the parent can itself be optional. **Allowed values** lists the enum, pattern and range the schema enforces. An object typed plainly `object` rejects keys the schema does not list; `object (open)` accepts extra keys and `object (free-form)` has no listed keys. A row that begins "Only when ..." exists only under that condition. See [how to read this reference](./README.md#how-to-read-a-row). diff --git a/docs/sdk-and-plugins/journeys/custom-validator.md b/docs/sdk-and-plugins/journeys/custom-validator.md index 6e5c01c..bbb7349 100644 --- a/docs/sdk-and-plugins/journeys/custom-validator.md +++ b/docs/sdk-and-plugins/journeys/custom-validator.md @@ -459,7 +459,7 @@ Three places this typically gets installed: ``` 3. **CI** — the bundle from [your-own-ci](./your-own-ci.md) already has a `validate` stage. Add `my-org-validators` to the `pip install` line: ```yaml - - run: pip install --index-url "" "data-product-forge==0.27.0" "my-org-validators==" + - run: pip install --index-url "" "data-product-forge==0.28.0" "my-org-validators==" - run: fluid validate contract.fluid.yaml --strict env: FLUID_PLUGINS_ALLOWLIST: "steward-required,cost-center-required,classification-from-vocab,local" diff --git a/docs/sdk-and-plugins/journeys/your-own-scaffolding.md b/docs/sdk-and-plugins/journeys/your-own-scaffolding.md index e3ee72d..d7ac6d1 100644 --- a/docs/sdk-and-plugins/journeys/your-own-scaffolding.md +++ b/docs/sdk-and-plugins/journeys/your-own-scaffolding.md @@ -176,7 +176,7 @@ keywords = [ dependencies = [ "pydantic>=2.0", - "data-product-forge==0.27.0", + "data-product-forge==0.28.0", ] [project.optional-dependencies] diff --git a/docs/sdk-and-plugins/reference/companion-packages.md b/docs/sdk-and-plugins/reference/companion-packages.md index b15e29f..bbad991 100644 --- a/docs/sdk-and-plugins/reference/companion-packages.md +++ b/docs/sdk-and-plugins/reference/companion-packages.md @@ -48,7 +48,7 @@ The PyPI name reflects the product brand (`data-product-forge`). The import path ```toml dependencies = [ - "data-product-forge==0.27.0", # pin exact for reproducibility + "data-product-forge==0.28.0", # pin exact for reproducibility ] ``` diff --git a/docs/sdk-and-plugins/reference/trust-model.md b/docs/sdk-and-plugins/reference/trust-model.md index 64bdc59..8613562 100644 --- a/docs/sdk-and-plugins/reference/trust-model.md +++ b/docs/sdk-and-plugins/reference/trust-model.md @@ -177,7 +177,7 @@ Plugins are tested, scanned, and approved by your platform team before showing u ```toml # In your deploy environment's pyproject.toml or requirements: -data-product-forge==0.27.0 +data-product-forge==0.28.0 data-product-forge-sdk==0.10.0 data-product-forge-custom-scaffold==0.4.1 my-org-validators==1.2.0 diff --git a/docs/upgrading.md b/docs/upgrading.md index 676ebb5..225babc 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1,6 +1,6 @@ --- title: Upgrading -description: How to upgrade and pin data-product-forge, the checklist for each step from 0.15.x to 0.26.0, and what moving the local engine from DuckDB 1.x to 2.x changes — what breaks, what to do, and how to check it. +description: How to upgrade and pin data-product-forge, the checklist for each step from 0.15.x to 0.28.0, and what moving the local engine from DuckDB 1.x to 2.x changes — what breaks, what to do, and how to check it. --- # Upgrading @@ -61,7 +61,7 @@ Upgrading the CLI on your machine does not upgrade that pipeline. Regenerate it, ## Contracts do not need a new `fluidVersion` -Upgrading the CLI does not change the schema your contracts declare. `0.27.0` reads +Upgrading the CLI does not change the schema your contracts declare. `0.28.0` reads `fluidVersion` `0.7.1` to `0.7.6`, and `0.7.7` as a preview (the list `fluid version` prints). `0.7.6` is stable since `0.26.0`: it is the default, and the contracts forge-cli writes declare it. A contract that declares `0.7.5` still validates as `0.7.5`. @@ -732,26 +732,32 @@ These changes shipped in 0.27.0. Most fail closed where a contract used to apply in `host 'connect.example.com', connecting to '93.184.216.34', is public`. A URL it cannot parse is refused with "it has no host". -## Extras in 0.27.0 and the install change in 0.28.0 +## Extras in 0.27.0 -*(from 0.27.0)* Four extras are new. In 0.27.0 the base install still contains all of their packages, so installing them changes nothing yet. +*(since 0.27.0)* Four extras were added next to the base install, so you could switch before 0.28.0 took their packages out of it. [Changes in 0.28.0](#changes-in-0-28-0) says what that change does. | Extra | Installs | For | | --- | --- | --- | | `litellm`, alias `ai` | `litellm` | The AI backend of `fluid forge`, `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup` and the copilot | | `mcp` | `mcp` | The MCP server SDK: `fluid mcp serve`, MCP output ports, `fluid scaffold-ide` and the keyless `mcp-sampling` provider | | `lineage` | `openlineage-python` | Lineage events when `OPENLINEAGE_URL` is set | -| `standard` | `litellm` and `mcp` | Today's full AI and MCP behaviour | +| `standard` | `litellm` and `mcp` | The AI and MCP behaviour a base install had through 0.27.0 | -The extras that already exist are `all`, `aws`, `catalogs`, `copilot`, `databricks`, `datahub`, `dev`, `docs`, `gcp`, `local`, `odcs-strict`, `odps`, `opds`, `postgres`, `sdk`, `snowflake`, `test-emulators`, `test-emulators-heavy`, `tui`, `vector` and `viz`. +The extras that already existed are `all`, `aws`, `catalogs`, `copilot`, `databricks`, `datahub`, `dev`, `docs`, `gcp`, `local`, `odcs-strict`, `odps`, `opds`, `postgres`, `sdk`, `snowflake`, `test-emulators`, `test-emulators-heavy`, `tui`, `vector` and `viz`. Since 0.28.0 `viz`, `tui` and `odps` are empty aliases: nothing imports `graphviz`, `textual` or `ruamel.yaml`, and `pip install 'data-product-forge[viz]'` still works. -### The base install in 0.28.0 +*(since 0.27.0)* When an optional package is missing, the CLI prints a message naming the feature, the package and the extra, then exits 1 with no traceback; see [Production troubleshooting](./advanced/production-troubleshooting.md#an-optional-package-is-not-installed). `fluid doctor` reads the litellm version from package metadata instead of importing it. -*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* litellm, mcp and openlineage-python leave the base install. This is a breaking change to the install, not to any contract or command. Counted with `uv pip compile` for a base install with no extras, the number of packages goes from 88 to 41 (Linux, Python 3.12), from 86 to 45 (Linux, Python 3.10) and from 89 to 40 (Windows, Python 3.12). litellm alone was 30 of them. +## Changes in 0.28.0 + +These changes shipped in 0.28.0. The first is a breaking change to the install, not to any contract or command: read it before you move a pipeline, a Docker image or a CI install line. After it come the pins that changed with it, the dependency list, the Debezium `.env` names, the Iceberg `catalogAuth` checks, Databricks and the smaller changes. + +### The base install + +*(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* litellm, mcp and openlineage-python are no longer in the base install; with litellm went boto3, aiohttp, openai, tokenizers and about fifty more packages. Counted with `uv pip compile` for a base install with no extras, the number of packages goes from 88 to 41 (Linux, Python 3.12), from 86 to 45 (Linux, Python 3.10) and from 89 to 40 (Windows, Python 3.12); litellm alone was 30 of them. `packaging` is now a declared base dependency, since it used to arrive through litellm. The base install keeps the core pipeline: `validate`, `plan`, `apply`, `verify`, `publish`, `generate iac`, `bundle`, `diff`, `doctor`, and template or blank `forge`. If that is all you run, you need nothing. -If you use AI forge, MCP or lineage, name the extra. These install lines also work on 0.27.0, where they change nothing: +If you use AI forge, MCP or lineage, name the extra. These install lines also work on 0.27.0, where they change nothing. ```bash pip install "data-product-forge[litellm]" # AI forge; [ai] is the same extra @@ -762,15 +768,15 @@ pip install "data-product-forge[standard]" # litellm and mcp: the pre-0.28.0 AI | You run | You need | | --- | --- | -| `fluid forge` with an API-key provider (`--llm-provider openai`, `anthropic`, `gemini`, `ollama`), `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup`, the `fluid init` hand-off to AI, `--agent-loop` | `litellm` (alias `ai`) | +| `fluid forge` with an API-key provider (`--llm-provider openai`, `anthropic`, `gemini`, `ollama`), `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), the `fluid ai setup` provider wizard (not `--clear` or `--source`), the `fluid init` hand-off to AI, `--agent-loop` | `litellm` (alias `ai`) | | `fluid mcp serve`, `fluid mcp output-port serve`, `fluid scaffold-ide` | `mcp` | | `OPENLINEAGE_URL` set on `fluid apply` through the OpenTofu engine, or on an acquisition build | `lineage` | -| An AWS command that needs the account id or calls AWS (`diff`, `verify`, live reads, `apply`) | `aws` | +| A native AWS plan that must resolve the account id (a contract that declares orchestration), live AWS reads, the Glue catalog import blocks of `apply` | `aws` | | `litellm` and `mcp` together, the AI and MCP packages a 0.27.0 base install had | `standard` | `[dev]` and `[all]` include all three. The Docker image profiles already install `[standard]` and `[lineage]`. -**The CLI stops before it does any work.** An explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) print the exact `pip install` line and exit 1, with no traceback. For AI forge the check runs before the interview reads any context or memory, and before `fluid ai setup` reads the OS keyring or asks for a key. The forge message names `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. Only Claude Code needs no setup of its own: `codex`, `cursor` and `kiro` reuse their own CLI login or key, and `mcp-sampling` runs inside `fluid mcp serve`, which needs `[mcp]`. `fluid forge data-model` with no provider flags runs offline. `fluid init` does not exit 1: its AI hand-off prints the line and the options, then continues in template mode (it picks Claude Code if `claude` is on `PATH`). Bare interactive `fluid forge` with no provider prints the hint and offers guided mode. Template and blank forge are unchanged. `fluid scaffold-ide` writes nothing without `mcp`, because every entry it writes launches `fluid mcp serve`. +**The CLI stops before it does any work.** An explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), the `fluid ai setup` wizard and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) print the exact `pip install` line and exit 1, with no traceback. `--llm-model` alone resolves to the provider saved with `fluid ai setup`, and to `ollama` when none is saved. For AI forge the check runs before the interview reads any context or memory, and before the `fluid ai setup` wizard reads the OS keyring or asks for a key. The forge message names `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. Only Claude Code needs no setup of its own: `codex`, `cursor` and `kiro` reuse their own CLI login or key, and `mcp-sampling` runs inside `fluid mcp serve`, which needs `[mcp]`. `fluid forge data-model` with no provider flags runs offline. `fluid init` does not exit 1: its AI hand-off prints the line and the options, then continues in template mode (it picks Claude Code if `claude` is on `PATH`). Bare interactive `fluid forge` with no provider uses Claude Code when it is installed, and otherwise prints the hint and offers guided mode. Template and blank forge are unchanged. `fluid scaffold-ide` writes nothing without `mcp`, because every entry it writes launches `fluid mcp serve`. ```text ❌ The MCP server needs mcp, which is not installed. @@ -781,7 +787,7 @@ pip install "data-product-forge[standard]" # litellm and mcp: the pre-0.28.0 AI **Lineage is the one that does not stop.** Lineage is emitted only by `fluid apply` through the OpenTofu engine and by acquisition builds; a native local apply never emits it. Without `lineage`, those print one warning naming the extra on the console (and log it) and carry on without sending events. -**AWS needs `aws`.** boto3 used to arrive through litellm. Without it, an AWS command that needs the account id fails with `The AWS account id could not be resolved: boto3 is not installed, so sts:GetCallerIdentity cannot be called`, followed by the install line, instead of the misleading bucket-name error it used to give. Setting `AWS_ACCOUNT_ID` also avoids the lookup. +**AWS needs `aws`.** boto3 used to arrive through litellm. Without it, a native AWS plan that cannot resolve the account id fails with `The AWS account id could not be resolved: boto3 is not installed, so sts:GetCallerIdentity cannot be called`, followed by the install line, instead of the misleading bucket-name error it used to give. Only a contract that declares orchestration (`orchestration.schedule`, `triggers` or `tasks`) reaches that plan; other contracts skip it. `fluid apply` without boto3 instead warns that the Glue catalog import blocks are omitted. Setting `AWS_ACCOUNT_ID` avoids the lookup. `fluid doctor` no longer reports a configured provider as ready when litellm is missing; its Next steps panel names the extra. @@ -789,15 +795,32 @@ pip install "data-product-forge[standard]" # litellm and mcp: the pre-0.28.0 AI ### Pins that changed with it -*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `mcp` extra now needs `mcp>=1.30,!=2.0.*,!=2.1.*,<3.0` (it was `mcp>=1.22`), because earlier releases have published advisories, among them an SSE session that was not bound to the principal that opened it; `fluid mcp output-port serve` uses that transport. The `litellm` extra excludes litellm 1.107.0, a malicious release since removed from PyPI (MAL-2026-2144). +*(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The `mcp` extra now needs `mcp>=1.30,!=2.0.*,!=2.1.*,<3.0` (it was `mcp>=1.22`), because earlier releases have published advisories, among them an SSE session that was not bound to the principal that opened it; `fluid mcp output-port serve` uses that transport. The `litellm` extra excludes litellm 1.107.0, a malicious release since removed from PyPI (MAL-2026-2144). + +### The dependency list now matches the imports + +*(since 0.28.0, [forge-cli #829](https://github.com/Agenticstiger/forge-cli/pull/829))* `typing-extensions` and the Windows-only `colorama` leave the base install, because nothing imports them (`pydantic` and `click` still bring them where needed). `anyio` is named in it. The `[gcp]` extra drops `google-auth-oauthlib` and `google-cloud-iam`. The GCP project-access probe, which is the one user of `google-api-python-client` (still not in `[gcp]`), reports "not checked" when that package is absent, where it used to report "no access". `fluid odcs validate --roundtrip` needs `deepdiff`, which now belongs to the `[odcs-strict]` extra; a missing one says so. + +### A project `.env` can no longer configure the embedded Debezium Server + +*(since 0.28.0, [forge-cli #833](https://github.com/Agenticstiger/forge-cli/pull/833))* If you set any of these in a project `.env`, `.env.` or `.env.local`, move it to the shell or the CI job: it is ignored now, with the usual warning. The names are `DEBEZIUM_*`, `QUARKUS_*`, `SMALLRYE_*` and `MP_CONFIG_*` in the forms SmallRye Config matches, `JDK_JAVA_OPTIONS`, `JAVA_OPTS`, `JAVA_HOME`, and the launcher's `EXTRA_CONNECTOR`, `JMX_HOST`, `JMX_PORT`, `JMX_EXPORTER_PORT`, `JMX_EXPORTER_CONFIG` and `OTEL_JAVAAGENT_*`. See [Project dotenv trust](./advanced/project-dotenv-trust.md). + +### Iceberg sink `catalogAuth` checks are stricter + +*(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828) and [#830](https://github.com/Agenticstiger/forge-cli/pull/830))* On a Kafka Connect or embedded Debezium build with `sink.catalogAuth`, `fluid validate` and the run's preflight now refuse a catalog or token endpoint at a cloud metadata address (AWS, GCP, Oracle and Alibaba, IPv6 and NAT64 forms included) even when you list it, and a catalog URL that holds `${`, starts with `env:` or holds a control character, wherever it is set (`binding.location.uri`, `iceberg_catalog_overrides`, the `iceberg.*` keys of `sink_connector_config` except `iceberg.kafka.*`, `server.sink.config`). A build without `catalogAuth` is unchanged. See [Operator allowlists](./advanced/source-aligned-acquisition.md#operator-allowlists) and [Network safety](./advanced/network-safety.md). + +### Databricks tables get tags and comments -### Faster `--help`, and nine modules removed +*(since 0.28.0, [forge-cli #831](https://github.com/Agenticstiger/forge-cli/pull/831))* `fluid apply` writes the table comment and column comments from the contract (on a table a dbt build writes, the comments stay dbt's), and every Databricks table gets the product's identity tags (`fluid_product_id`, `fluid_layer`, `fluid_owner_team` and others), which count toward Unity Catalog's 50 tags per securable together with your labels. A contract that used to apply may now be refused: a label that becomes one of those keys (`databricks-tag-collision`), a tag value over 256 characters (`databricks-tag-value-too-long`), more than 1,000 column tags on a table (`databricks-tags-too-many`), and a description holding a backslash or `${` (`databricks-comment-unsafe`). A table is certified through Unity Catalog's `system.certification_status`: `certified` when the label `certification: certified` is set, `deprecated` when `lifecycle.state` is `deprecated`; any other value of the label is refused (`databricks-certification-invalid`). Setting it needs the ASSIGN permission on that governed tag for the deploy identity (from Databricks' documentation; not verified on a live workspace), and the plan names the tables. After a load, *(since 0.28.0, [forge-cli #835](https://github.com/Agenticstiger/forge-cli/pull/835))* the bronze landing deletes the local masked Parquet copy under `.fluid/staging//` and fails the run if it cannot. See [Databricks](./providers/databricks.md). -*On forge-cli `main`, not yet in a release ([forge-cli #836](https://github.com/Agenticstiger/forge-cli/pull/836)).* `fluid --help` and each command import only the module of the command being run: about 300 modules on a bare install instead of about 857. The command list and every help line are unchanged. +### Other changes -*On forge-cli `main`, not yet in a release ([forge-cli #832](https://github.com/Agenticstiger/forge-cli/pull/832)).* Nine internal modules that nothing imported are removed: `fluid_build.logging_utils`, `fluid_build.cli.init_samples`, `fluid_build.copilot.schemas.table_metadata`, `fluid_build.providers.local.ducksql`, `.mocks` and `.samples`, and `fluid_build.runtimes.dataform_gcp`, `.dbt_bigquery` and `.dbt_duckdb`. No command or contract behaves differently. It matters only to code that imported one of them directly. +- *(since 0.28.0, [forge-cli #840](https://github.com/Agenticstiger/forge-cli/pull/840))* For a local-provider deployment, `fluid publish` reports a relative `location.path` anchored at the contract's directory, so the Command Center gets the file the engine writes. The contract itself is unchanged. +- *(since 0.28.0, [forge-cli #835](https://github.com/Agenticstiger/forge-cli/pull/835))* `fluid generate iac` prints the reason, fix and doc of a sovereignty refusal, as `fluid apply` does. +- *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The CLI prints install hints as written, and masks GitHub, Anthropic and OpenAI keys, JWTs and `password=...` pairs in the error text it prints. +- *(since 0.28.0, [forge-cli #836](https://github.com/Agenticstiger/forge-cli/pull/836))* `fluid --help` and each command import only the module of the command being run: about 300 modules on a bare install instead of about 857. The command list and every help line are unchanged. +- *(since 0.28.0, [forge-cli #832](https://github.com/Agenticstiger/forge-cli/pull/832))* Nine internal modules that nothing imported are removed: `fluid_build.logging_utils`, `fluid_build.cli.init_samples`, `fluid_build.copilot.schemas.table_metadata`, `fluid_build.providers.local.ducksql`, `.mocks` and `.samples`, and `fluid_build.runtimes.dataform_gcp`, `.dbt_bigquery` and `.dbt_duckdb`. No command or contract behaves differently. It matters only to code that imported one of them directly. -*(from 0.27.0)* When an optional package is missing, the CLI prints a message naming the feature, the package and the extra, then exits 1 with no traceback; see [Production troubleshooting](./advanced/production-troubleshooting.md#an-optional-package-is-not-installed). `fluid doctor` reads the litellm version from package metadata instead of importing it. ## DuckDB 1.x to 2.x diff --git a/docs/vision.md b/docs/vision.md index e7010db..c4c09b7 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -147,7 +147,7 @@ fluid market --search "customer analytics" ## Versioning in the docs -- Current CLI release baseline: `0.27.0` +- Current CLI release baseline: `0.28.0` - Current scaffolded contract examples: `fluidVersion: 0.7.5` That split is intentional. The CLI release and the contract schema version move on related but different timelines. diff --git a/docs/walkthrough/ai-forge-data-model.md b/docs/walkthrough/ai-forge-data-model.md index 74c0499..88f2254 100644 --- a/docs/walkthrough/ai-forge-data-model.md +++ b/docs/walkthrough/ai-forge-data-model.md @@ -25,7 +25,7 @@ export OLLAMA_HOST=http://localhost:11434 ## Choose The Right Entry Point -Every AI row below that uses an API-key provider needs `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* With an explicit API-key `--llm-provider`, a base install prints that line and exits 1 before doing any work; `--llm-provider claude-code`, `codex`, `cursor` or `kiro` avoids the extra (the last three reuse their own CLI login or key), and `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. +Every AI row below that uses an API-key provider needs `pip install 'data-product-forge[litellm]'`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* With an explicit API-key `--llm-provider`, a base install prints that line and exits 1 before doing any work; `--llm-provider claude-code`, `codex`, `cursor` or `kiro` avoids the extra (the last three reuse their own CLI login or key), and `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. | Goal | Command | AI needed | | --- | --- | --- | diff --git a/docs/walkthrough/mcp-output-port.md b/docs/walkthrough/mcp-output-port.md index 5076b5f..143ea73 100644 --- a/docs/walkthrough/mcp-output-port.md +++ b/docs/walkthrough/mcp-output-port.md @@ -42,7 +42,7 @@ No cloud account, no credentials, no cost. Everything runs on DuckDB reading a l pip install 'data-product-forge[local,mcp]' ``` -The `[local]` extra pulls in DuckDB (1.5.0 or later), which is the reference engine for the output port. Without it, `fluid mcp output-port doctor` fails its `engine_health` check with `duckdb is not installed`. The `[mcp]` extra pulls in the MCP SDK that `fluid mcp output-port serve` runs on. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A base install no longer contains it, and `serve` prints `pip install 'data-product-forge[mcp]'` and exits 1. On 0.27.0 the base install has the SDK already and the extra changes nothing. +The `[local]` extra pulls in DuckDB (1.5.0 or later), which is the reference engine for the output port. Without it, `fluid mcp output-port doctor` fails its `engine_health` check with `duckdb is not installed`. The `[mcp]` extra pulls in the MCP SDK that `fluid mcp output-port serve` runs on. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* A base install does not contain it, and `serve` prints `pip install 'data-product-forge[mcp]'` and exits 1. On 0.27.0 the base install has the SDK already and the extra changes nothing. ### Verify the command is wired diff --git a/scripts/forge-cli-releases.json b/scripts/forge-cli-releases.json index ae6e436..73fea1f 100644 --- a/scripts/forge-cli-releases.json +++ b/scripts/forge-cli-releases.json @@ -21,7 +21,7 @@ ], "repo": "Agenticstiger/forge-cli", "changelogUrl": "https://raw.githubusercontent.com/Agenticstiger/forge-cli/main/CHANGELOG.md", - "reviewedThroughRelease": "0.27.0", + "reviewedThroughRelease": "0.28.0", "releaseOfPr": { "707": "0.20.0", "709": "0.20.0",