Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ fluid forge
fluid forge --domain retail
```

`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. To forge a data model from a business intent file and generate dbt from it:
`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. An API-key provider needs `pip install "data-product-forge[litellm]"`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without an API key, `fluid forge --llm-provider claude-code` uses your Claude Code login, and `fluid forge --template starter` uses no AI. To forge a data model from a business intent file and generate dbt from it:

```bash
fluid forge data-model from-intent --example retail > intent.yaml
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke
| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<adapter><2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. |
| `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. |
| `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. |
| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. *(from 0.28.0)* Emitting needs the `lineage` extra. |
| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. *(from 0.28.0)* Emitting needs the `lineage` extra. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* |
| `FLUID_OPENLINEAGE_ENDPOINT` | Path appended to the URL. |
| `FLUID_OPENLINEAGE_API_KEY` (or `OPENLINEAGE_API_KEY`) | Bearer key for the receiver. |
| `FLUID_OPENLINEAGE_TIMEOUT_SECONDS` | Request timeout. Default `5`. |
Expand Down
2 changes: 2 additions & 0 deletions docs/advanced/forge-copilot-discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ Forge may inspect local:

## Optional helpers

These helpers are not the AI backend. Generating a contract with an API-key provider needs the `litellm` extra (`pip install "data-product-forge[litellm]"`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).*

If you want richer Parquet and Avro schema inspection, install the optional discovery helpers:

```bash
Expand Down
4 changes: 2 additions & 2 deletions docs/advanced/litellm-backend.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# LiteLLM Backend

[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is a core dependency of `data-product-forge` up to 0.27.0; from 0.28.0 it is installed with the `litellm` extra (alias `ai`) or `standard`. The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md).
[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is a core dependency of `data-product-forge` up to 0.27.0; from 0.28.0 it is installed with the `litellm` extra (alias `ai`) or `standard`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md).

::: tip Where this fits
LiteLLM is a hard dependency of core `data-product-forge` (`litellm>=1.83.7,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`. In 0.27.0 the base install still contains litellm, so installing the extra changes nothing yet. *(from 0.28.0)* litellm leaves the base install: `fluid forge`, `fluid forge data-model`, `fluid ai setup` and the copilot then need `pip install "data-product-forge[standard]"` (or `[ai]`). The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).
LiteLLM is a hard dependency of core `data-product-forge` (`litellm>=1.83.14,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`. In 0.27.0 the base install still contains litellm, so installing the extra changes nothing yet. *(from 0.28.0)* litellm leaves the base install. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `fluid forge` and `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `--agent-loop` and `fluid ai setup` then need `pip install "data-product-forge[litellm]"` (or `[ai]`, or `[standard]` to add `mcp`); an explicit API-key `--llm-provider` and `fluid ai setup` print that line and exit 1 before doing any work when it is missing. `fluid init`'s AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).
:::

## Which providers you can select
Expand Down
2 changes: 2 additions & 0 deletions docs/advanced/llm-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ fluid forge data-model from-intent intent.yaml -o customer_orders.fluid.yaml

## Supported providers

The `openai`, `anthropic`, `gemini` and `ollama` providers run through LiteLLM and need `pip install 'data-product-forge[litellm]'` (alias `[ai]`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. The four local agent CLIs need neither extra; only Claude Code needs no setup of its own, since the others reuse their own CLI login or key.

| Provider | `--llm-provider` | Default model | Notes |
| --- | --- | --- | --- |
| Anthropic | `anthropic` (alias `claude`) | `claude-sonnet-4-6` | Tool-forced structured output and provider-native prompt caching. Streamed runs report token usage in cost summaries. |
Expand Down
2 changes: 2 additions & 0 deletions docs/advanced/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ The first half of this page covers the authoring server. The second half is a de

### Start the server

Install the MCP SDK first: `pip install 'data-product-forge[mcp]'`. The extra exists since 0.27.0. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A base install no longer contains `mcp`, and `fluid mcp serve`, `fluid mcp output-port serve` and `fluid scaffold-ide` print that install line and exit 1 before they do any work. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).

```bash
fluid mcp serve --read-only # inspection only
fluid mcp serve # full surface, scoped to the working directory
Expand Down
10 changes: 7 additions & 3 deletions docs/advanced/production-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,11 +268,15 @@ Update the key at the source `fluid ai status` reports, the provider environment

| Feature | Package | Extra |
|---|---|---|
| `fluid forge`, `fluid forge data-model`, `fluid ai setup`, the copilot | `litellm` | `ai` (alias of `litellm`), or `standard` |
| `fluid mcp serve`, MCP output ports, the keyless `mcp-sampling` provider | `mcp` | `mcp`, or `standard` |
| `fluid forge` with an API-key provider, `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup`, the `fluid init` hand-off to AI, `--agent-loop` | `litellm` | `litellm`, its alias `ai`, or `standard` |
| `fluid mcp serve`, `fluid mcp output-port serve`, `fluid scaffold-ide`, `mcp-sampling` (it runs inside `fluid mcp serve`) | `mcp` | `mcp`, or `standard` |
| Lineage events when `OPENLINEAGE_URL` is set | `openlineage-python` | `lineage` |

Through 0.27.0 a base install contains all three, so you meet this only if you removed one. From 0.28.0 a base install omits them. A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step.
*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `litellm`, `mcp` and `lineage` rows above describe a base install from 0.28.0.

Through 0.27.0 a base install contains all three, so you meet this only if you removed one. From 0.28.0 a base install omits them. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step.

*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* From 0.28.0 an explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) stop before they do any work. The forge message names `--llm-provider claude-code` and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. `fluid init` does not exit 1: it prints the line and the options, then continues in template mode. Lineage is the exception: without `lineage`, `fluid apply` through the OpenTofu engine and acquisition builds print one warning on the console (and log it) and carry on; a native local apply never emits lineage. AWS commands that need boto3 point at `[aws]`; boto3 no longer arrives through litellm. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0).

## Where logs live

Expand Down
2 changes: 2 additions & 0 deletions docs/advanced/source-aligned-acquisition.md
Original file line number Diff line number Diff line change
Expand Up @@ -1368,6 +1368,8 @@ The emitter is chosen by environment, not by a key in the contract. With no endp
export OPENLINEAGE_URL=https://marquez.example.com
```

Emitting needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).*

`FLUID_OPENLINEAGE_URL` takes precedence over `OPENLINEAGE_URL`. The endpoint path defaults to `api/v1/lineage` (change it with `FLUID_OPENLINEAGE_ENDPOINT`), and `OPENLINEAGE_API_KEY` or `FLUID_OPENLINEAGE_API_KEY` sets the bearer token. A private or loopback endpoint is allowed unless `FLUID_OPENLINEAGE_ALLOW_PRIVATE=false`; link-local addresses, such as cloud metadata endpoints, are blocked either way.

The schema's `lineage` block has one key, `emit`, and no runner reads it. A build with `lineage: { emit: false }` still emits when `OPENLINEAGE_URL` is set. Out-of-tree emitters subclass `LineageEmitter` from `fluid_build.api.lineage`.
Expand Down
2 changes: 2 additions & 0 deletions docs/cli/ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Configure the AI / LLM provider used by Forge Copilot, test that it works, and inspect the current AI configuration.

`fluid ai setup` needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, it prints that line, names the keyless ways to author (`fluid forge --llm-provider claude-code|codex|cursor|kiro`, `--llm-provider mcp-sampling`, `fluid forge --template starter`) and exits 1 before it reads the OS keyring or asks for a key.

## Syntax

```bash
Expand Down
6 changes: 4 additions & 2 deletions docs/cli/doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,12 +197,14 @@ The exit code is `1`. Leave `--extended` out unless your project provides its ow

## The 0.28.0 install change

*(from 0.27.0)* `fluid doctor` prints one line once per run:
*(from 0.27.0)* `fluid doctor` prints one line once per run. On 0.27.0 it reads:

```text
AI/MCP move to extras in 0.28.0: pip install 'data-product-forge[standard]'
```

*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* After the change it reads `AI and MCP are extras since 0.28.0: pip install 'data-product-forge[standard]'`, and a configured provider is no longer reported as ready when litellm is missing: the Next steps panel says `<provider> needs litellm, which is not installed` and prints `pip install 'data-product-forge[litellm]'`.

It also reports three optional packages in the AI checks: the LiteLLM backend, the MCP SDK and the OpenLineage client. Each shows its version when installed, or `Not installed (optional)` with the `pip install 'data-product-forge[<extra>]'` line. None of them is imported: doctor reads the version from the package metadata, so it makes no network call for litellm's cost map.

## Installing what doctor checks
Expand All @@ -217,7 +219,7 @@ The `fix:` line for a missing binary, and the `doc` field in `--json` output, po
| `helm`, `kubectl` | Managed-mode acquisition engines that provision infrastructure with Helm on Kubernetes. | [Helm install guide](https://helm.sh/docs/intro/install/) and [Kubernetes tools](https://kubernetes.io/docs/tasks/tools/). |
| `duckdb` | The `local` provider and DuckDB acquisition builds. | `pip install "data-product-forge[local]"`, which installs `duckdb` at the version the CLI requires for its contract-SQL sandbox. See [DuckDB sandbox](../advanced/duckdb-sandbox.md). |
| `dlt` | `dlt` acquisition builds. | `pip install dlt`. As of 0.18.1 `data-product-forge` has no `dlt` extra. |
| `litellm`, `mcp`, `openlineage` | *(from 0.27.0)* AI forge, `fluid mcp serve` and lineage emission. | Part of a base install through 0.27.0. From 0.28.0: `pip install "data-product-forge[standard]"`, or `[ai]`, `[mcp]` or `[lineage]` for one. |
| `litellm`, `mcp`, `openlineage` | *(from 0.27.0)* AI forge, `fluid mcp serve` and lineage emission. | Part of a base install through 0.27.0. From 0.28.0: `pip install "data-product-forge[standard]"`, or `[ai]`, `[mcp]` or `[lineage]` for one. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* |
| `httpx` | A core dependency of the CLI. | Installed with `data-product-forge`; a failure here points to a broken Python environment. |
| `snowflake.connector` | The `snowflake` provider. | `pip install "data-product-forge[snowflake]"`. The `fix:` line for this check reads `pip install snowflake`, which is not the package that provides the connector; use the extra. |

Expand Down
2 changes: 2 additions & 0 deletions docs/cli/forge.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Author a data product contract with an AI copilot, an offline guided interview, or an empty skeleton. By default forge writes `contract.fluid.yaml` and `.fluid/forge-receipt.json`; a scaffold (`--scaffold`), a CI pipeline (`--ci`) and engine artifacts add files.

AI forge with an API-key provider needs the `litellm` extra (alias `ai`): `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, an explicit `--llm-provider` that names an API-key provider (`openai`, `anthropic`, `gemini`, `ollama`), on `fluid forge`, on `fluid forge data-model` or with `--agent-loop`, prints that line and exits 1 with `[ERR_COPILOT_LITELLM_UNAVAILABLE]`, before the interview reads any context or memory. `fluid forge data-model` with no `--llm-provider`, `--llm-model` or `--llm-endpoint` runs offline and needs no extra (`--llm-model` alone resolves to `ollama`). The forge message names two ways to author without it: `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`, because the agent CLIs lack tool use. `codex`, `cursor` and `kiro` also work without litellm, but they reuse their own CLI login or key. `--blank` and the other templates need no AI. `--llm-provider mcp-sampling` works only inside `fluid mcp serve`, which needs the `mcp` extra. Bare interactive `fluid forge` with no provider prints the hint and offers guided mode.

## Syntax

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/init.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Create a new project: a working example, an empty skeleton, a bundled blueprint,
fluid init [NAME] [OPTIONS]
```

With no mode flag, `fluid init NAME` starts the AI-assisted path (the same copilot as [`fluid forge`](./forge.md)). Every example below picks a mode explicitly, so none of them needs an AI key.
With no mode flag, `fluid init NAME` starts the AI-assisted path (the same copilot as [`fluid forge`](./forge.md)), which needs `pip install "data-product-forge[litellm]"` for an API-key provider. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Every example below picks a mode explicitly, so none of them needs an AI key.

## Examples

Expand Down
Loading
Loading