Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The site is organised the way a reader uses it (get started, tutorials, how-to g

## Current Versioning

- Current CLI release documented here: `0.27.0`
- Current CLI release documented here: `0.28.0`
- Contract schema: `fluidVersion: 0.7.6` is the stable default; `0.7.7` is an opt-in preview

Those are different on purpose. `fluid version` reports the installed CLI release, while `fluidVersion` inside a contract selects the contract schema version.
Expand Down
4 changes: 2 additions & 2 deletions docs/.vuepress/cli-version.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"supportedCliVersion": "0.27.0",
"supportedCliInstallSpec": "data-product-forge==0.27.0",
"supportedCliVersion": "0.28.0",
"supportedCliInstallSpec": "data-product-forge==0.28.0",
"supportedFluidContractVersion": "0.7.6",
"quickstartScaffoldVersion": "0.7.5",
"supportedSdkVersion": "0.10.0",
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ fluid forge
fluid forge --domain retail
```

`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. An API-key provider needs `pip install "data-product-forge[litellm]"`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without an API key, `fluid forge --llm-provider claude-code` uses your Claude Code login, and `fluid forge --template starter` uses no AI. To forge a data model from a business intent file and generate dbt from it:
`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. An API-key provider needs `pip install "data-product-forge[litellm]"`. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* Without an API key, `fluid forge --llm-provider claude-code` uses your Claude Code login, and `fluid forge --template starter` uses no AI. To forge a data model from a business intent file and generate dbt from it:

```bash
fluid forge data-model from-intent --example retail > intent.yaml
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke
| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<adapter><2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. |
| `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. |
| `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. |
| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. *(from 0.28.0)* Emitting needs the `lineage` extra. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* |
| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. Emitting needs the `lineage` extra. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* |
| `FLUID_OPENLINEAGE_ENDPOINT` | Path appended to the URL. |
| `FLUID_OPENLINEAGE_API_KEY` (or `OPENLINEAGE_API_KEY`) | Bearer key for the receiver. |
| `FLUID_OPENLINEAGE_TIMEOUT_SECONDS` | Request timeout. Default `5`. |
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/forge-copilot-discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Forge may inspect local:

## Optional helpers

These helpers are not the AI backend. Generating a contract with an API-key provider needs the `litellm` extra (`pip install "data-product-forge[litellm]"`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).*
These helpers are not the AI backend. Generating a contract with an API-key provider needs the `litellm` extra (`pip install "data-product-forge[litellm]"`). *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))*

If you want richer Parquet and Avro schema inspection, install the optional discovery helpers:

Expand Down
4 changes: 2 additions & 2 deletions docs/advanced/litellm-backend.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# LiteLLM Backend

[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is a core dependency of `data-product-forge` up to 0.27.0; from 0.28.0 it is installed with the `litellm` extra (alias `ai`) or `standard`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md).
[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is not in the base install: install it with the `litellm` extra (alias `ai`) or `standard`. It was a core dependency up to 0.27.0. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md).

::: tip Where this fits
LiteLLM is a hard dependency of core `data-product-forge` (`litellm>=1.83.14,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`. In 0.27.0 the base install still contains litellm, so installing the extra changes nothing yet. *(from 0.28.0)* litellm leaves the base install. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `fluid forge` and `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `--agent-loop` and `fluid ai setup` then need `pip install "data-product-forge[litellm]"` (or `[ai]`, or `[standard]` to add `mcp`); an explicit API-key `--llm-provider` and `fluid ai setup` print that line and exit 1 before doing any work when it is missing. `fluid init`'s AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).
LiteLLM was a hard dependency of core `data-product-forge` (`litellm>=1.83.14,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`; in 0.27.0 the base install still contains litellm, so installing the extra changes nothing there. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* litellm is not in the base install. `fluid forge` and `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `--agent-loop` and the `fluid ai setup` provider wizard (not `--clear` or `--source`) need `pip install "data-product-forge[litellm]"` (or `[ai]`, or `[standard]` to add `mcp`); an explicit API-key `--llm-provider` and the `fluid ai setup` wizard print that line and exit 1 before doing any work when it is missing. `fluid init`'s AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#changes-in-0-28-0).
:::

## Which providers you can select
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/llm-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ fluid forge data-model from-intent intent.yaml -o customer_orders.fluid.yaml

## Supported providers

The `openai`, `anthropic`, `gemini` and `ollama` providers run through LiteLLM and need `pip install 'data-product-forge[litellm]'` (alias `[ai]`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. The four local agent CLIs need neither extra; only Claude Code needs no setup of its own, since the others reuse their own CLI login or key.
The `openai`, `anthropic`, `gemini` and `ollama` providers run through LiteLLM and need `pip install 'data-product-forge[litellm]'` (alias `[ai]`). *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. The four local agent CLIs need neither extra; only Claude Code needs no setup of its own, since the others reuse their own CLI login or key.

| Provider | `--llm-provider` | Default model | Notes |
| --- | --- | --- | --- |
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The first half of this page covers the authoring server. The second half is a de

### Start the server

Install the MCP SDK first: `pip install 'data-product-forge[mcp]'`. The extra exists since 0.27.0. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A base install no longer contains `mcp`, and `fluid mcp serve`, `fluid mcp output-port serve` and `fluid scaffold-ide` print that install line and exit 1 before they do any work. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).
Install the MCP SDK first: `pip install 'data-product-forge[mcp]'`. The extra exists since 0.27.0. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* A base install does not contain `mcp`, and `fluid mcp serve`, `fluid mcp output-port serve` and `fluid scaffold-ide` print that install line and exit 1 before they do any work. See [Upgrading](../upgrading.md#changes-in-0-28-0).

```bash
fluid mcp serve --read-only # inspection only
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/network-safety.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ The webhook, federation and Command Center variables take comma-separated host s

An IPv4-mapped address (`::ffff:...`) or a NAT64 address (`64:ff9b::/96`) that carries a link-local or one of these metadata addresses is refused too. The surrounding ranges stay allowed: a Tailscale tailnet (CGNAT, `100.100.100.100` included) and an IPv6 cluster on ULA addresses need no change.

*On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* A fourth check, the `catalogAuth` host check of the Kafka Connect and embedded Debezium Iceberg sink, refuses the same addresses, even when the host is listed. See [Operator allowlists](source-aligned-acquisition.md#operator-allowlists).
*(since 0.28.0, [forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828))* A fourth check, the `catalogAuth` host check of the Kafka Connect and embedded Debezium Iceberg sink, refuses the same addresses, even when the host is listed. See [Operator allowlists](source-aligned-acquisition.md#operator-allowlists).

## Cloud metadata and the credential resolver

Expand Down
6 changes: 3 additions & 3 deletions docs/advanced/production-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,11 +272,11 @@ Update the key at the source `fluid ai status` reports, the provider environment
| `fluid mcp serve`, `fluid mcp output-port serve`, `fluid scaffold-ide`, `mcp-sampling` (it runs inside `fluid mcp serve`) | `mcp` | `mcp`, or `standard` |
| Lineage events when `OPENLINEAGE_URL` is set | `openlineage-python` | `lineage` |

*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `litellm`, `mcp` and `lineage` rows above describe a base install from 0.28.0.
The `litellm`, `mcp` and `lineage` rows above describe a base install. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))*

Through 0.27.0 a base install contains all three, so you meet this only if you removed one. From 0.28.0 a base install omits them. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step.
A base install omits all three; through 0.27.0 it contained them, so on those releases you meet this only if you removed one. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step.

*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* From 0.28.0 an explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) stop before they do any work. The forge message names `--llm-provider claude-code` and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. `fluid init` does not exit 1: it prints the line and the options, then continues in template mode. Lineage is the exception: without `lineage`, `fluid apply` through the OpenTofu engine and acquisition builds print one warning on the console (and log it) and carry on; a native local apply never emits lineage. AWS commands that need boto3 point at `[aws]`; boto3 no longer arrives through litellm. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).
An explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) stop before they do any work. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* The forge message names `--llm-provider claude-code` and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. `fluid init` does not exit 1: it prints the line and the options, then continues in template mode. Lineage is the exception: without `lineage`, `fluid apply` through the OpenTofu engine and acquisition builds print one warning on the console (and log it) and carry on; a native local apply never emits lineage. AWS commands that need boto3 point at `[aws]`; boto3 no longer arrives through litellm. See [Upgrading](../upgrading.md#changes-in-0-28-0).

## Where logs live

Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/project-dotenv-trust.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ What it may not set is any variable that
project file has nothing to supply there. The full list is
`fluid_build/credentials/protected_env.py` in forge-cli.

*On forge-cli `main`, not yet in a release ([forge-cli #833](https://github.com/Agenticstiger/forge-cli/pull/833)).* The embedded Debezium Server inherits the environment `fluid apply` loads, and its SmallRye Config reads a variable over `application.properties`. Its JVM reads `-D` system properties, which outrank both, from `JDK_JAVA_OPTIONS` or the launcher's `JAVA_OPTS`. So a project file could send a `sink.catalogAuth` credential to a catalog the operator never paired with it, change the source database, load another config file, or start another `java`. A project dotenv file can no longer set `DEBEZIUM_*`, `QUARKUS_*`, `SMALLRYE_*` or `MP_CONFIG_*` variables in the forms SmallRye Config matches, profile forms included (`debezium.sink.iceberg.uri`, `_PROD_DEBEZIUM_...`, `%prod....`), nor `JDK_JAVA_OPTIONS`, `JAVA_OPTS`, `JAVA_HOME`, or the launcher's `EXTRA_CONNECTOR`, `JMX_HOST`, `JMX_PORT`, `JMX_EXPORTER_PORT`, `JMX_EXPORTER_CONFIG` and `OTEL_JAVAAGENT_*`. Each is ignored with the usual warning. Set it in the shell or CI instead.
*(since 0.28.0, [forge-cli #833](https://github.com/Agenticstiger/forge-cli/pull/833))* The embedded Debezium Server inherits the environment `fluid apply` loads, and its SmallRye Config reads a variable over `application.properties`. Its JVM reads `-D` system properties, which outrank both, from `JDK_JAVA_OPTIONS` or the launcher's `JAVA_OPTS`. So a project file could send a `sink.catalogAuth` credential to a catalog the operator never paired with it, change the source database, load another config file, or start another `java`. A project dotenv file can no longer set `DEBEZIUM_*`, `QUARKUS_*`, `SMALLRYE_*` or `MP_CONFIG_*` variables in the forms SmallRye Config matches, profile forms included (`debezium.sink.iceberg.uri`, `_PROD_DEBEZIUM_...`, `%prod....`), nor `JDK_JAVA_OPTIONS`, `JAVA_OPTS`, `JAVA_HOME`, or the launcher's `EXTRA_CONNECTOR`, `JMX_HOST`, `JMX_PORT`, `JMX_EXPORTER_PORT`, `JMX_EXPORTER_CONFIG` and `OTEL_JAVAAGENT_*`. Each is ignored with the usual warning. Set it in the shell or CI instead.

::: tip Two more since `0.23.0`
`FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, which allows an acquisition source's `env://`
Expand Down
Loading
Loading