diff --git a/docs/README.md b/docs/README.md index acb2223..5ba2e8c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,7 @@ fluid forge fluid forge --domain retail ``` -`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. To forge a data model from a business intent file and generate dbt from it: +`fluid init` scaffolds from a fixed template or blueprint. `fluid forge` drafts a contract from the files in your directory with the LLM provider you configure with `fluid ai setup`. An API-key provider needs `pip install "data-product-forge[litellm]"`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without an API key, `fluid forge --llm-provider claude-code` uses your Claude Code login, and `fluid forge --template starter` uses no AI. To forge a data model from a business intent file and generate dbt from it: ```bash fluid forge data-model from-intent --example retail > intent.yaml diff --git a/docs/advanced/environment-variables.md b/docs/advanced/environment-variables.md index de3acdc..d64794b 100644 --- a/docs/advanced/environment-variables.md +++ b/docs/advanced/environment-variables.md @@ -151,7 +151,7 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke | `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. | | `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. | | `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. | -| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. *(from 0.28.0)* Emitting needs the `lineage` extra. | +| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. *(from 0.28.0)* Emitting needs the `lineage` extra. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* | | `FLUID_OPENLINEAGE_ENDPOINT` | Path appended to the URL. | | `FLUID_OPENLINEAGE_API_KEY` (or `OPENLINEAGE_API_KEY`) | Bearer key for the receiver. | | `FLUID_OPENLINEAGE_TIMEOUT_SECONDS` | Request timeout. Default `5`. | diff --git a/docs/advanced/forge-copilot-discovery.md b/docs/advanced/forge-copilot-discovery.md index 4df78f1..2b9bd88 100644 --- a/docs/advanced/forge-copilot-discovery.md +++ b/docs/advanced/forge-copilot-discovery.md @@ -37,6 +37,8 @@ Forge may inspect local: ## Optional helpers +These helpers are not the AI backend. Generating a contract with an API-key provider needs the `litellm` extra (`pip install "data-product-forge[litellm]"`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* + If you want richer Parquet and Avro schema inspection, install the optional discovery helpers: ```bash diff --git a/docs/advanced/litellm-backend.md b/docs/advanced/litellm-backend.md index 17b6eea..ddd82b9 100644 --- a/docs/advanced/litellm-backend.md +++ b/docs/advanced/litellm-backend.md @@ -1,9 +1,9 @@ # LiteLLM Backend -[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is a core dependency of `data-product-forge` up to 0.27.0; from 0.28.0 it is installed with the `litellm` extra (alias `ai`) or `standard`. The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md). +[LiteLLM](https://github.com/BerriAI/litellm) is the LLM backend for the `openai`, `anthropic`, `gemini` and `ollama` providers. It is a core dependency of `data-product-forge` up to 0.27.0; from 0.28.0 it is installed with the `litellm` extra (alias `ai`) or `standard`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `mcp-sampling` provider and the local agent CLIs (`claude-code`, `codex`, `cursor`, `kiro`) do not go through it; see [LLM Providers](./llm-providers.md). ::: tip Where this fits -LiteLLM is a hard dependency of core `data-product-forge` (`litellm>=1.83.7,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`. In 0.27.0 the base install still contains litellm, so installing the extra changes nothing yet. *(from 0.28.0)* litellm leaves the base install: `fluid forge`, `fluid forge data-model`, `fluid ai setup` and the copilot then need `pip install "data-product-forge[standard]"` (or `[ai]`). The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). +LiteLLM is a hard dependency of core `data-product-forge` (`litellm>=1.83.14,<2` on Python 3.11 and later) up to 0.27.0. *(from 0.27.0)* The `litellm` extra, with the alias `ai`, names it, and `standard` adds `mcp`. In 0.27.0 the base install still contains litellm, so installing the extra changes nothing yet. *(from 0.28.0)* litellm leaves the base install. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `fluid forge` and `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `--agent-loop` and `fluid ai setup` then need `pip install "data-product-forge[litellm]"` (or `[ai]`, or `[standard]` to add `mcp`); an explicit API-key `--llm-provider` and `fluid ai setup` print that line and exit 1 before doing any work when it is missing. `fluid init`'s AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion [LLM Providers](./llm-providers.md) page covers which providers `--llm-provider` accepts and which environment variables to set; this page covers the routing layer. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). ::: ## Which providers you can select diff --git a/docs/advanced/llm-providers.md b/docs/advanced/llm-providers.md index 8810107..f5aa090 100644 --- a/docs/advanced/llm-providers.md +++ b/docs/advanced/llm-providers.md @@ -16,6 +16,8 @@ fluid forge data-model from-intent intent.yaml -o customer_orders.fluid.yaml ## Supported providers +The `openai`, `anthropic`, `gemini` and `ollama` providers run through LiteLLM and need `pip install 'data-product-forge[litellm]'` (alias `[ai]`). *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. The four local agent CLIs need neither extra; only Claude Code needs no setup of its own, since the others reuse their own CLI login or key. + | Provider | `--llm-provider` | Default model | Notes | | --- | --- | --- | --- | | Anthropic | `anthropic` (alias `claude`) | `claude-sonnet-4-6` | Tool-forced structured output and provider-native prompt caching. Streamed runs report token usage in cost summaries. | diff --git a/docs/advanced/mcp.md b/docs/advanced/mcp.md index 8328504..c571e13 100644 --- a/docs/advanced/mcp.md +++ b/docs/advanced/mcp.md @@ -15,6 +15,8 @@ The first half of this page covers the authoring server. The second half is a de ### Start the server +Install the MCP SDK first: `pip install 'data-product-forge[mcp]'`. The extra exists since 0.27.0. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A base install no longer contains `mcp`, and `fluid mcp serve`, `fluid mcp output-port serve` and `fluid scaffold-ide` print that install line and exit 1 before they do any work. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). + ```bash fluid mcp serve --read-only # inspection only fluid mcp serve # full surface, scoped to the working directory diff --git a/docs/advanced/production-troubleshooting.md b/docs/advanced/production-troubleshooting.md index c546fe0..4050615 100644 --- a/docs/advanced/production-troubleshooting.md +++ b/docs/advanced/production-troubleshooting.md @@ -268,11 +268,15 @@ Update the key at the source `fluid ai status` reports, the provider environment | Feature | Package | Extra | |---|---|---| -| `fluid forge`, `fluid forge data-model`, `fluid ai setup`, the copilot | `litellm` | `ai` (alias of `litellm`), or `standard` | -| `fluid mcp serve`, MCP output ports, the keyless `mcp-sampling` provider | `mcp` | `mcp`, or `standard` | +| `fluid forge` with an API-key provider, `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup`, the `fluid init` hand-off to AI, `--agent-loop` | `litellm` | `litellm`, its alias `ai`, or `standard` | +| `fluid mcp serve`, `fluid mcp output-port serve`, `fluid scaffold-ide`, `mcp-sampling` (it runs inside `fluid mcp serve`) | `mcp` | `mcp`, or `standard` | | Lineage events when `OPENLINEAGE_URL` is set | `openlineage-python` | `lineage` | -Through 0.27.0 a base install contains all three, so you meet this only if you removed one. From 0.28.0 a base install omits them. A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step. +*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `litellm`, `mcp` and `lineage` rows above describe a base install from 0.28.0. + +Through 0.27.0 a base install contains all three, so you meet this only if you removed one. From 0.28.0 a base install omits them. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A pipeline of only `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor` needs none of them. In CI, put the extra in `FLUID_PACKAGE_SPEC` or in your install step. + +*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* From 0.28.0 an explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) stop before they do any work. The forge message names `--llm-provider claude-code` and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. `fluid init` does not exit 1: it prints the line and the options, then continues in template mode. Lineage is the exception: without `lineage`, `fluid apply` through the OpenTofu engine and acquisition builds print one warning on the console (and log it) and carry on; a native local apply never emits lineage. AWS commands that need boto3 point at `[aws]`; boto3 no longer arrives through litellm. See [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0). ## Where logs live diff --git a/docs/advanced/source-aligned-acquisition.md b/docs/advanced/source-aligned-acquisition.md index 8ae0e9c..b640ff0 100644 --- a/docs/advanced/source-aligned-acquisition.md +++ b/docs/advanced/source-aligned-acquisition.md @@ -1368,6 +1368,8 @@ The emitter is chosen by environment, not by a key in the contract. With no endp export OPENLINEAGE_URL=https://marquez.example.com ``` +Emitting needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* + `FLUID_OPENLINEAGE_URL` takes precedence over `OPENLINEAGE_URL`. The endpoint path defaults to `api/v1/lineage` (change it with `FLUID_OPENLINEAGE_ENDPOINT`), and `OPENLINEAGE_API_KEY` or `FLUID_OPENLINEAGE_API_KEY` sets the bearer token. A private or loopback endpoint is allowed unless `FLUID_OPENLINEAGE_ALLOW_PRIVATE=false`; link-local addresses, such as cloud metadata endpoints, are blocked either way. The schema's `lineage` block has one key, `emit`, and no runner reads it. A build with `lineage: { emit: false }` still emits when `OPENLINEAGE_URL` is set. Out-of-tree emitters subclass `LineageEmitter` from `fluid_build.api.lineage`. diff --git a/docs/cli/ai.md b/docs/cli/ai.md index bdd7d8a..474f078 100644 --- a/docs/cli/ai.md +++ b/docs/cli/ai.md @@ -2,6 +2,8 @@ Configure the AI / LLM provider used by Forge Copilot, test that it works, and inspect the current AI configuration. +`fluid ai setup` needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, it prints that line, names the keyless ways to author (`fluid forge --llm-provider claude-code|codex|cursor|kiro`, `--llm-provider mcp-sampling`, `fluid forge --template starter`) and exits 1 before it reads the OS keyring or asks for a key. + ## Syntax ```bash diff --git a/docs/cli/doctor.md b/docs/cli/doctor.md index 46c44bc..18806e5 100644 --- a/docs/cli/doctor.md +++ b/docs/cli/doctor.md @@ -197,12 +197,14 @@ The exit code is `1`. Leave `--extended` out unless your project provides its ow ## The 0.28.0 install change -*(from 0.27.0)* `fluid doctor` prints one line once per run: +*(from 0.27.0)* `fluid doctor` prints one line once per run. On 0.27.0 it reads: ```text AI/MCP move to extras in 0.28.0: pip install 'data-product-forge[standard]' ``` +*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* After the change it reads `AI and MCP are extras since 0.28.0: pip install 'data-product-forge[standard]'`, and a configured provider is no longer reported as ready when litellm is missing: the Next steps panel says ` needs litellm, which is not installed` and prints `pip install 'data-product-forge[litellm]'`. + It also reports three optional packages in the AI checks: the LiteLLM backend, the MCP SDK and the OpenLineage client. Each shows its version when installed, or `Not installed (optional)` with the `pip install 'data-product-forge[]'` line. None of them is imported: doctor reads the version from the package metadata, so it makes no network call for litellm's cost map. ## Installing what doctor checks @@ -217,7 +219,7 @@ The `fix:` line for a missing binary, and the `doc` field in `--json` output, po | `helm`, `kubectl` | Managed-mode acquisition engines that provision infrastructure with Helm on Kubernetes. | [Helm install guide](https://helm.sh/docs/intro/install/) and [Kubernetes tools](https://kubernetes.io/docs/tasks/tools/). | | `duckdb` | The `local` provider and DuckDB acquisition builds. | `pip install "data-product-forge[local]"`, which installs `duckdb` at the version the CLI requires for its contract-SQL sandbox. See [DuckDB sandbox](../advanced/duckdb-sandbox.md). | | `dlt` | `dlt` acquisition builds. | `pip install dlt`. As of 0.18.1 `data-product-forge` has no `dlt` extra. | -| `litellm`, `mcp`, `openlineage` | *(from 0.27.0)* AI forge, `fluid mcp serve` and lineage emission. | Part of a base install through 0.27.0. From 0.28.0: `pip install "data-product-forge[standard]"`, or `[ai]`, `[mcp]` or `[lineage]` for one. | +| `litellm`, `mcp`, `openlineage` | *(from 0.27.0)* AI forge, `fluid mcp serve` and lineage emission. | Part of a base install through 0.27.0. From 0.28.0: `pip install "data-product-forge[standard]"`, or `[ai]`, `[mcp]` or `[lineage]` for one. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* | | `httpx` | A core dependency of the CLI. | Installed with `data-product-forge`; a failure here points to a broken Python environment. | | `snowflake.connector` | The `snowflake` provider. | `pip install "data-product-forge[snowflake]"`. The `fix:` line for this check reads `pip install snowflake`, which is not the package that provides the connector; use the extra. | diff --git a/docs/cli/forge.md b/docs/cli/forge.md index bd0bc68..29509e1 100644 --- a/docs/cli/forge.md +++ b/docs/cli/forge.md @@ -2,6 +2,8 @@ Author a data product contract with an AI copilot, an offline guided interview, or an empty skeleton. By default forge writes `contract.fluid.yaml` and `.fluid/forge-receipt.json`; a scaffold (`--scaffold`), a CI pipeline (`--ci`) and engine artifacts add files. +AI forge with an API-key provider needs the `litellm` extra (alias `ai`): `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, an explicit `--llm-provider` that names an API-key provider (`openai`, `anthropic`, `gemini`, `ollama`), on `fluid forge`, on `fluid forge data-model` or with `--agent-loop`, prints that line and exits 1 with `[ERR_COPILOT_LITELLM_UNAVAILABLE]`, before the interview reads any context or memory. `fluid forge data-model` with no `--llm-provider`, `--llm-model` or `--llm-endpoint` runs offline and needs no extra (`--llm-model` alone resolves to `ollama`). The forge message names two ways to author without it: `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`, because the agent CLIs lack tool use. `codex`, `cursor` and `kiro` also work without litellm, but they reuse their own CLI login or key. `--blank` and the other templates need no AI. `--llm-provider mcp-sampling` works only inside `fluid mcp serve`, which needs the `mcp` extra. Bare interactive `fluid forge` with no provider prints the hint and offers guided mode. + ## Syntax ```bash diff --git a/docs/cli/init.md b/docs/cli/init.md index 6f478a0..0f0e2a2 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -8,7 +8,7 @@ Create a new project: a working example, an empty skeleton, a bundled blueprint, fluid init [NAME] [OPTIONS] ``` -With no mode flag, `fluid init NAME` starts the AI-assisted path (the same copilot as [`fluid forge`](./forge.md)). Every example below picks a mode explicitly, so none of them needs an AI key. +With no mode flag, `fluid init NAME` starts the AI-assisted path (the same copilot as [`fluid forge`](./forge.md)), which needs `pip install "data-product-forge[litellm]"` for an API-key provider. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Every example below picks a mode explicitly, so none of them needs an AI key. ## Examples diff --git a/docs/cli/mcp.md b/docs/cli/mcp.md index b392a0b..c816fdf 100644 --- a/docs/cli/mcp.md +++ b/docs/cli/mcp.md @@ -25,6 +25,8 @@ fluid mcp output-port serve ... # consumer-side data-access server Runs the authoring MCP stdio server so an MCP client can inspect, validate, and edit forge artifacts. It is a foreground process — no daemon, no HTTP port, no background service. +Needs the `mcp` extra: `pip install 'data-product-forge[mcp]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it the command prints that line and exits 1, with `[ERR_MCP_SDK_UNAVAILABLE]` and no traceback. + ### Syntax ```bash @@ -81,8 +83,8 @@ Both servers run on the official `mcp` Python SDK. | | | | --- | --- | -| **Dependency pin** | `mcp>=1.22,<3.0` *(since 0.20.0, [forge-cli #710](https://github.com/Agenticstiger/forge-cli/pull/710); 0.19.0 and earlier pin `mcp>=1.20,<3.0`)* | -| **What a fresh install resolves** | The newest `2.x`. Up to 0.27.0 `mcp` is a core dependency, so `pip install data-product-forge` picks it up. *(from 0.27.0)* The `mcp` extra (and `standard`) names it; *(from 0.28.0)* a base install no longer includes it, and `fluid mcp serve`, MCP output ports and the keyless `mcp-sampling` provider need `pip install "data-product-forge[mcp]"`. A fresh 0.18.1 install resolved `mcp 2.3.0` on 5 Oct 2026. | +| **Dependency pin** | `mcp>=1.22,<3.0` *(since 0.20.0, [forge-cli #710](https://github.com/Agenticstiger/forge-cli/pull/710); 0.19.0 and earlier pin `mcp>=1.20,<3.0`)*. The `mcp` extra then needs `mcp>=1.30,!=2.0.*,!=2.1.*,<3.0`, because earlier releases have published advisories. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* | +| **What a fresh install resolves** | The newest `2.x`. Up to 0.27.0 `mcp` is a core dependency, so `pip install data-product-forge` picks it up. *(from 0.27.0)* The `mcp` extra (and `standard`) names it; *(from 0.28.0)* a base install no longer includes it, and `fluid mcp serve`, MCP output ports, `fluid scaffold-ide` and the keyless `mcp-sampling` provider need `pip install "data-product-forge[mcp]"`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A fresh 0.18.1 install resolved `mcp 2.3.0` on 5 Oct 2026. | | **Existing environments** | Do not move until you upgrade them. Pinning `mcp>=1.22,<2.0` yourself remains supported. mcp 1.20 and 1.21 fail to import with pydantic 2.14 or later, which removed a private helper they import; keep pydantic below 2.14 if you must stay on them. | | **3.x** | Outside the pin. | @@ -98,6 +100,8 @@ On 1.x a client could self-attest identity (`model`, `useCase`, tenant attribute Binds **one expose** from a FLUID contract and serves it to MCP clients as a governed, read-only data port. This is distinct from `fluid mcp serve`: the output port queries production data, so its threat model and policy surface differ entirely. +Needs the `mcp` extra as well (`pip install 'data-product-forge[mcp]'`); add `local` for the DuckDB engine, as in `pip install 'data-product-forge[local,mcp]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* + The server is **read-only by default** and exposes a small surface: an agent can `describe` the data product, `sample` a few rows, and run a predeclared **semantic** `query`. Free-form SQL (`query_sql`) is OFF unless you opt in with `--allow-sql`. ### The output-port subcommands diff --git a/docs/cli/scaffold-ide.md b/docs/cli/scaffold-ide.md index b491173..fd84b2d 100644 --- a/docs/cli/scaffold-ide.md +++ b/docs/cli/scaffold-ide.md @@ -2,6 +2,8 @@ Generate agentic-IDE configuration so an AI-assisted editor can drive Fluid Forge — steering rules, hooks, and an MCP server entry, written from one pack. +Needs the `mcp` extra: `pip install 'data-product-forge[mcp]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it the command writes nothing, prints that line and exits 1 with `[ERR_MCP_SDK_UNAVAILABLE]`, because every entry it writes launches `fluid mcp serve`. + ## Syntax ```bash diff --git a/docs/concepts/quality-sla-lineage.md b/docs/concepts/quality-sla-lineage.md index 02b7e52..38a8c08 100644 --- a/docs/concepts/quality-sla-lineage.md +++ b/docs/concepts/quality-sla-lineage.md @@ -264,7 +264,7 @@ dist/artifacts/policy/bindings.json | **ODPS-Bitol** (Bitol Open Data Product Standard) | `odps-bitol/` | Bitol-aware catalogs | | **OPDS** (Open Data Product Specification v4.1, LF/ODPI) | `opds/` | Generic catalog ingest | -OpenLineage is not written to a file. Runs send OpenLineage run events over HTTP when `OPENLINEAGE_URL` (or `FLUID_OPENLINEAGE_URL`) is set: acquisition builds, and `fluid apply` through OpenTofu. Marquez and DataHub accept them. With no URL set, nothing is sent. +OpenLineage is not written to a file. Runs send OpenLineage run events over HTTP when `OPENLINEAGE_URL` (or `FLUID_OPENLINEAGE_URL`) is set: acquisition builds, and `fluid apply` through OpenTofu. Marquez and DataHub accept them. With no URL set, nothing is sent. Sending needs the `lineage` extra: `pip install 'data-product-forge[lineage]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it, those runs print one warning naming the extra on the console (and log it) and carry on without sending events. A native local apply never emits lineage. ## Where to look next diff --git a/docs/faq/README.md b/docs/faq/README.md index 0854f69..d227d12 100644 --- a/docs/faq/README.md +++ b/docs/faq/README.md @@ -39,7 +39,7 @@ The same cause as above: the `[local]` extra is missing. Install `data-product-f ## A command says `needs , which is not installed`. -*(from 0.27.0)* An optional package is missing. The message names the extra: run the `pip install 'data-product-forge[]'` line it prints. From 0.28.0 the AI, MCP and lineage packages are no longer in the base install, so `pip install 'data-product-forge[standard]'` restores AI forge and MCP. See [Production troubleshooting](/forge_docs/advanced/production-troubleshooting.html#an-optional-package-is-not-installed). +*(from 0.27.0)* An optional package is missing. The message names the extra: run the `pip install 'data-product-forge[]'` line it prints. From 0.28.0 the AI, MCP and lineage packages are no longer in the base install, so `pip install 'data-product-forge[standard]'` restores AI forge and MCP, or name one of `[litellm]`, `[mcp]` and `[lineage]`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* See [Production troubleshooting](/forge_docs/advanced/production-troubleshooting.html#an-optional-package-is-not-installed). ## A local output file contains `id,value` and `1,materialized`. diff --git a/docs/forge-data-model.md b/docs/forge-data-model.md index 05e3ddd..ab5fe60 100644 --- a/docs/forge-data-model.md +++ b/docs/forge-data-model.md @@ -181,6 +181,8 @@ For dbt output, the generator fails clearly if it produces zero `models/**/*.sql ## AI provider runs +A run that calls a provider with an API key needs the `litellm` extra: `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* Without it the command prints that line and exits 1 before it does any work. `--llm-provider claude-code`, `codex`, `cursor` and `kiro` use that agent's own login instead. + Provider keys are never part of the intent file or contract. Export one provider key for the shell session or run `fluid ai setup`: ```bash diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md index c84c610..ff5575d 100644 --- a/docs/getting-started/README.md +++ b/docs/getting-started/README.md @@ -46,12 +46,12 @@ The `local` extra installs DuckDB (`duckdb>=1.5.0`), pandas and numpy, which the | `gcp` | BigQuery | Google Cloud client libraries, `dbt-bigquery` | | `aws` | S3, Glue, Athena | `boto3` | | `snowflake` | Snowflake | `snowflake-connector-python`, `dbt-snowflake` | -| `litellm` (alias `ai`) | *(from 0.27.0)* The AI backend for `fluid forge`, `fluid forge data-model`, `fluid ai setup` and the copilot | `litellm` | -| `mcp` | *(from 0.27.0)* The MCP server SDK for `fluid mcp serve`, MCP output ports and the keyless `mcp-sampling` provider | `mcp` | +| `litellm` (alias `ai`) | *(from 0.27.0)* The AI backend for `fluid forge` with an API-key provider, `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup` and the copilot | `litellm` | +| `mcp` | *(from 0.27.0)* The MCP server SDK for `fluid mcp serve`, MCP output ports, `fluid scaffold-ide` and `mcp-sampling` (which runs inside `fluid mcp serve`) | `mcp` | | `lineage` | *(from 0.27.0)* Emitting OpenLineage events when `OPENLINEAGE_URL` is set | `openlineage-python` | -| `standard` | *(from 0.27.0)* `litellm` and `mcp` together: the AI and MCP behaviour a base install has today | `litellm`, `mcp` | +| `standard` | *(from 0.27.0)* `litellm` and `mcp` together: the AI and MCP packages a 0.27.0 base install has | `litellm`, `mcp` | -The table lists the extras a first project needs. [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0) lists every extra, and what changes for a base install in 0.28.0. +The table lists the extras a first project needs. From 0.28.0 a base install has none of `litellm`, `mcp` and `openlineage-python`, so name the extra for any AI, MCP or lineage feature you use, for example `pip install "data-product-forge[local,litellm]"`; it works on 0.27.0 too. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* [Upgrading](../upgrading.md#extras-in-0-27-0-and-the-install-change-in-0-28-0) lists every extra, and what changes for a base install in 0.28.0. ```bash pip install "data-product-forge[local,gcp]" # several extras at once diff --git a/docs/providers/aws.md b/docs/providers/aws.md index 34502af..f27de7f 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -437,7 +437,7 @@ fluid odcs export contract.fluid.yaml --output standards/product.odcs.yaml `fluid diff` reads the Glue table in the region `fluid apply` uses and compares its columns and types with the contract. Since `0.18.1` its live check resolves `{{ env.* }}` placeholders in the binding before it reads the target. -`fluid diff` and `fluid verify` read AWS through boto3; install it with `pip install 'data-product-forge[aws]'`. Builds that land data with DuckDB need the `local` extra (`duckdb>=1.5.0`). +`fluid diff` and `fluid verify` read AWS through boto3; install it with `pip install 'data-product-forge[aws]'`. Builds that land data with DuckDB need the `local` extra (`duckdb>=1.5.0`). boto3 used to arrive through litellm, so a base install of 0.28.0 no longer has it: `fluid plan`, `fluid validate` and `fluid generate iac` run without it; install `[aws]` (or set `AWS_ACCOUNT_ID`, where only the account id is needed) before any command that calls AWS or needs the account id: `diff`, `verify`, live reads and `apply`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* ## Where a build lands data diff --git a/docs/upgrading.md b/docs/upgrading.md index 7f7f635..676ebb5 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -738,22 +738,66 @@ These changes shipped in 0.27.0. Most fail closed where a contract used to apply | Extra | Installs | For | | --- | --- | --- | -| `litellm`, alias `ai` | `litellm` | The AI backend of `fluid forge`, `fluid forge data-model`, `fluid ai setup` and the copilot | -| `mcp` | `mcp` | The MCP server SDK: `fluid mcp serve`, MCP output ports and the keyless `mcp-sampling` provider | +| `litellm`, alias `ai` | `litellm` | The AI backend of `fluid forge`, `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup` and the copilot | +| `mcp` | `mcp` | The MCP server SDK: `fluid mcp serve`, MCP output ports, `fluid scaffold-ide` and the keyless `mcp-sampling` provider | | `lineage` | `openlineage-python` | Lineage events when `OPENLINEAGE_URL` is set | | `standard` | `litellm` and `mcp` | Today's full AI and MCP behaviour | The extras that already exist are `all`, `aws`, `catalogs`, `copilot`, `databricks`, `datahub`, `dev`, `docs`, `gcp`, `local`, `odcs-strict`, `odps`, `opds`, `postgres`, `sdk`, `snowflake`, `test-emulators`, `test-emulators-heavy`, `tui`, `vector` and `viz`. -**Coming in 0.28.0 (announced, not released):** litellm, mcp and openlineage leave the base install, which shrinks from 88 packages to 41 (Linux, Python 3.12, resolved with uv). If you use AI forge or MCP, install: +### The base install in 0.28.0 + +*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* litellm, mcp and openlineage-python leave the base install. This is a breaking change to the install, not to any contract or command. Counted with `uv pip compile` for a base install with no extras, the number of packages goes from 88 to 41 (Linux, Python 3.12), from 86 to 45 (Linux, Python 3.10) and from 89 to 40 (Windows, Python 3.12). litellm alone was 30 of them. + +The base install keeps the core pipeline: `validate`, `plan`, `apply`, `verify`, `publish`, `generate iac`, `bundle`, `diff`, `doctor`, and template or blank `forge`. If that is all you run, you need nothing. + +If you use AI forge, MCP or lineage, name the extra. These install lines also work on 0.27.0, where they change nothing: ```bash -pip install "data-product-forge[standard]" +pip install "data-product-forge[litellm]" # AI forge; [ai] is the same extra +pip install "data-product-forge[mcp]" +pip install "data-product-forge[lineage]" +pip install "data-product-forge[standard]" # litellm and mcp: the pre-0.28.0 AI and MCP behaviour +``` + +| You run | You need | +| --- | --- | +| `fluid forge` with an API-key provider (`--llm-provider openai`, `anthropic`, `gemini`, `ollama`), `fluid forge data-model` with an API-key provider (`--llm-provider openai|anthropic|gemini|ollama`), `fluid ai setup`, the `fluid init` hand-off to AI, `--agent-loop` | `litellm` (alias `ai`) | +| `fluid mcp serve`, `fluid mcp output-port serve`, `fluid scaffold-ide` | `mcp` | +| `OPENLINEAGE_URL` set on `fluid apply` through the OpenTofu engine, or on an acquisition build | `lineage` | +| An AWS command that needs the account id or calls AWS (`diff`, `verify`, live reads, `apply`) | `aws` | +| `litellm` and `mcp` together, the AI and MCP packages a 0.27.0 base install had | `standard` | + +`[dev]` and `[all]` include all three. The Docker image profiles already install `[standard]` and `[lineage]`. + +**The CLI stops before it does any work.** An explicit `--llm-provider` naming an API-key provider (`fluid forge`, `fluid forge data-model`, `--agent-loop`), `fluid ai setup` and the MCP commands (`mcp serve`, `mcp output-port serve`, `scaffold-ide`) print the exact `pip install` line and exit 1, with no traceback. For AI forge the check runs before the interview reads any context or memory, and before `fluid ai setup` reads the OS keyring or asks for a key. The forge message names `--llm-provider claude-code` (your Claude Code login) and `--template starter`; the `--agent-loop` message names only `--template starter`; `ai setup` and `init` name `claude-code`, `codex`, `cursor`, `kiro`, `mcp-sampling` and `--template starter`. Only Claude Code needs no setup of its own: `codex`, `cursor` and `kiro` reuse their own CLI login or key, and `mcp-sampling` runs inside `fluid mcp serve`, which needs `[mcp]`. `fluid forge data-model` with no provider flags runs offline. `fluid init` does not exit 1: its AI hand-off prints the line and the options, then continues in template mode (it picks Claude Code if `claude` is on `PATH`). Bare interactive `fluid forge` with no provider prints the hint and offers guided mode. Template and blank forge are unchanged. `fluid scaffold-ide` writes nothing without `mcp`, because every entry it writes launches `fluid mcp serve`. + +```text +❌ The MCP server needs mcp, which is not installed. + + pip install 'data-product-forge[mcp]' +[ERR_MCP_SDK_UNAVAILABLE] ``` -or `[ai]`, `[mcp]` or `[lineage]` for one of them. If you only run `validate`, `plan`, `apply`, `bundle`, `generate iac` and `doctor`, you need nothing. Put the extra in `FLUID_PACKAGE_SPEC` in CI before you move. +**Lineage is the one that does not stop.** Lineage is emitted only by `fluid apply` through the OpenTofu engine and by acquisition builds; a native local apply never emits it. Without `lineage`, those print one warning naming the extra on the console (and log it) and carry on without sending events. + +**AWS needs `aws`.** boto3 used to arrive through litellm. Without it, an AWS command that needs the account id fails with `The AWS account id could not be resolved: boto3 is not installed, so sts:GetCallerIdentity cannot be called`, followed by the install line, instead of the misleading bucket-name error it used to give. Setting `AWS_ACCOUNT_ID` also avoids the lookup. + +`fluid doctor` no longer reports a configured provider as ready when litellm is missing; its Next steps panel names the extra. + +**Before you move,** put the extra in `FLUID_PACKAGE_SPEC` in CI, or in your own install step. + +### Pins that changed with it + +*On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* The `mcp` extra now needs `mcp>=1.30,!=2.0.*,!=2.1.*,<3.0` (it was `mcp>=1.22`), because earlier releases have published advisories, among them an SSE session that was not bound to the principal that opened it; `fluid mcp output-port serve` uses that transport. The `litellm` extra excludes litellm 1.107.0, a malicious release since removed from PyPI (MAL-2026-2144). + +### Faster `--help`, and nine modules removed + +*On forge-cli `main`, not yet in a release ([forge-cli #836](https://github.com/Agenticstiger/forge-cli/pull/836)).* `fluid --help` and each command import only the module of the command being run: about 300 modules on a bare install instead of about 857. The command list and every help line are unchanged. + +*On forge-cli `main`, not yet in a release ([forge-cli #832](https://github.com/Agenticstiger/forge-cli/pull/832)).* Nine internal modules that nothing imported are removed: `fluid_build.logging_utils`, `fluid_build.cli.init_samples`, `fluid_build.copilot.schemas.table_metadata`, `fluid_build.providers.local.ducksql`, `.mocks` and `.samples`, and `fluid_build.runtimes.dataform_gcp`, `.dbt_bigquery` and `.dbt_duckdb`. No command or contract behaves differently. It matters only to code that imported one of them directly. -*(from 0.27.0)* When an optional package is missing, the CLI prints a message naming the feature, the package and the extra, then exits 1 with no traceback; see [Production troubleshooting](./advanced/production-troubleshooting.md#an-optional-package-is-not-installed). `fluid doctor` prints `AI/MCP move to extras in 0.28.0: pip install 'data-product-forge[standard]'` once per run, and reads the litellm version from package metadata instead of importing it. +*(from 0.27.0)* When an optional package is missing, the CLI prints a message naming the feature, the package and the extra, then exits 1 with no traceback; see [Production troubleshooting](./advanced/production-troubleshooting.md#an-optional-package-is-not-installed). `fluid doctor` reads the litellm version from package metadata instead of importing it. ## DuckDB 1.x to 2.x diff --git a/docs/walkthrough/ai-forge-data-model.md b/docs/walkthrough/ai-forge-data-model.md index 570f6ba..74c0499 100644 --- a/docs/walkthrough/ai-forge-data-model.md +++ b/docs/walkthrough/ai-forge-data-model.md @@ -25,6 +25,8 @@ export OLLAMA_HOST=http://localhost:11434 ## Choose The Right Entry Point +Every AI row below that uses an API-key provider needs `pip install 'data-product-forge[litellm]'`. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* With an explicit API-key `--llm-provider`, a base install prints that line and exits 1 before doing any work; `--llm-provider claude-code`, `codex`, `cursor` or `kiro` avoids the extra (the last three reuse their own CLI login or key), and `mcp-sampling` works only inside `fluid mcp serve`, which needs `[mcp]`. + | Goal | Command | AI needed | | --- | --- | --- | | Create a blank contract scaffold | `fluid forge --blank` | No | diff --git a/docs/walkthrough/mcp-output-port.md b/docs/walkthrough/mcp-output-port.md index b93878e..5076b5f 100644 --- a/docs/walkthrough/mcp-output-port.md +++ b/docs/walkthrough/mcp-output-port.md @@ -1,6 +1,6 @@ # Walkthrough: MCP Output Port -**Time:** 10 minutes | **Difficulty:** Beginner | **Prerequisites:** Python 3.10+, pip, Node.js (for the MCP Inspector CLI) +**Time:** 10 minutes | **Difficulty:** Beginner | **Prerequisites:** Python 3.10+, pip, Node.js (for the MCP Inspector CLI), and the `local` and `mcp` extras (Step 1) > **Why it matters** > Give an AI agent safe, read-only access to a published data product — governed by the same contract that governs people. @@ -36,13 +36,13 @@ No cloud account, no credentials, no cost. Everything runs on DuckDB reading a l ## Step 1: Setup -### Install Fluid Forge with the local extra +### Install Fluid Forge with the local and mcp extras ```bash -pip install 'data-product-forge[local]' +pip install 'data-product-forge[local,mcp]' ``` -The `[local]` extra pulls in DuckDB (1.5.0 or later), which is the reference engine for the output port. Without it, `fluid mcp output-port doctor` fails its `engine_health` check with `duckdb is not installed`. +The `[local]` extra pulls in DuckDB (1.5.0 or later), which is the reference engine for the output port. Without it, `fluid mcp output-port doctor` fails its `engine_health` check with `duckdb is not installed`. The `[mcp]` extra pulls in the MCP SDK that `fluid mcp output-port serve` runs on. *On forge-cli `main`, not yet in a release ([forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837)).* A base install no longer contains it, and `serve` prints `pip install 'data-product-forge[mcp]'` and exits 1. On 0.27.0 the base install has the SDK already and the extra changes nothing. ### Verify the command is wired