fix(ai): saved AI choice in non-interactive runs, JSON-only --agent stdout, mission agent loop by fas89 · Pull Request #854 · Agenticstiger/forge-cli

fas89 · 2026-11-12T12:27:52Z

Description

This fixes two defects that hit fluid forge without a TTY (CI, scripts, IDE agents), plus one they exposed. All three are follow-ups to #850 and #852. There is no issue.

1. The saved fluid ai setup choice is used in runs without a TTY (fix(ai)).

Before, only the interactive inline setup applied the saved provider, model and key. check_llm_readiness (fluid doctor) honoured the saved choice, but the real selection never read it. So doctor and forge disagreed.
Selection now walks one ladder, and readiness calls the same function:
1. a named provider (--llm-provider, FLUID_LLM_PROVIDER, FLUID_FORGE_AGENT);
2. exactly one key variable;
3. the saved choice;
4. exactly one keyring key;
5. a local Ollama, then (non-interactive, nothing configured) the first coding-agent CLI on PATH.
The saved model and Ollama host fill in what the run does not state, for the saved provider only. The environment's OLLAMA_HOST wins over the saved host.
A run that names its provider takes nothing from the saved choice. Without --llm-model, a named provider gets its catalog model, as before. So Command Center's fluid forge data-model from-intent --llm-provider <p> drafting call keeps its provider and model even where a saved config exists. A test pins this for openai, anthropic, claude and gemini, with no saved config, a saved config for another provider, and one for the same provider.
The saved endpoint field is not applied. That is the founder's decision: a run states its endpoint with --llm-endpoint or FLUID_LLM_ENDPOINT.
A saved coding agent (or MCP sampling) is passed over wherever the auto-pick is: a stated endpoint or model (flag or FLUID_LLM_* variable) or the agent loop. The no-provider message names it.
Saved names go through the provider registry, so a hand-written claude or claude_code counts as anthropic or claude-code.
Keys still resolve through _resolve_api_key. The saved choice is never written as a named provider.
tests/conftest.py gains an autouse fixture that hides the developer's saved choice from every test, as _hermetic_keyring does for the keyring.

2. fluid forge --agent stdout carries only the JSON-Lines events (fix(forge)).

AGENT_IDE.md promises one JSON object per line on stdout, but in blank mode 28 of 32 stdout lines were Rich panels.
forge.run now keeps the real stdout for the events and runs the rest under contextlib.redirect_stdout(sys.stderr). That is the same stdlib pattern cli/mcp/dispatch.py uses to capture these events in-process.
A missing extra still propagates from forge.run, so the MCP forge_run still gets the exception. The CLI entry point prints its remedy on stderr under --agent.

3. fluid mission run resolves its LLM as an agent-loop run (fix(mission)).

A mission always runs the tool-use loop, but it auto-picked (and, with fix 1, would have selected a saved) coding agent. The loop then failed mid-run with NotImplementedError.

Commits: one per unit, each green on its own (checked in scratch worktrees).

Overlap: the FLUID_LLM_API_KEY scope PR also touches check_llm_readiness and provider resolution. Whichever lands second merges main in and resolves the overlap. This PR never writes the saved choice into --llm-provider or FLUID_LLM_PROVIDER, so that PR's named and picked distinction holds.

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would break existing functionality)
Documentation update
Refactor / code cleanup
New provider or provider enhancement

Documentation

Docs PR linked:
- Docs PR:
No docs needed — this change does not affect user-facing behaviour
- Justification:
Docs TODO — I know docs are needed and will create a PR in forge_docs before this is merged

Pages to update in forge_docs:

llm-providers: the selection order now includes the saved fluid ai setup choice for runs without a TTY. A named provider takes nothing from it. The saved endpoint is not applied.
advanced/production-troubleshooting: the "No LLM provider is selected" section from forge_docs#205 quotes "this command does not read the saved choice", which is no longer said. Add the message for a saved coding agent that a run passes over.
error-codes: the copilot_no_llm_provider message variants.
cli/forge: under --agent, stdout carries only the JSON-Lines events and human-facing output is on stderr. Non-interactive runs use the saved choice.
cli/doctor: readiness walks the same ladder as forge, including FLUID_FORGE_AGENT.
environment-variables: FLUID_LLM_ENDPOINT and FLUID_LLM_MODEL (and their routing twins) also keep a saved or auto-picked coding agent from being used.
upgrading: a short "on main" note for the three behaviour changes.
The fluid mission page, if there is one: a mission never selects a coding agent.

Checklist

I have read the Contributing Guide
My code follows the project's coding standards
I have added tests that prove my fix/feature works
All new and existing tests pass (pytest). The local full suite fails only on the known shared-venv baseline; see Testing.
I have run ruff check and black with no errors
New maintained Python files include license headers (no new files)

Testing

Each fix was written test-first. The new tests failed on main and pass here.

Readiness and selection agreement: a matrix over the ladder in tests/test_llm_keyless_autopick.py, using the world fixture with no real ~/.fluid and no real keyring.
Command Center's call through the real parser.
Every stdout line of an --agent run parses as JSON: blank mode as a subprocess, AI mode with the generation seam mocked, and a missing extra through the CLI entry point.
The mission test.

Gates on the rebased head:

ruff, the dead-code sweep, black 24.10.0, product_guardrail, license headers, lint-imports and strict mypy all pass.
detect-secrets-hook 1.5.0 on the changed files passes. .secrets.baseline needed only a line-number refresh.
Targeted tests: 693 passed, including the Agent Skills tests from feat(agent-skills): an Agent Skills pack, its Claude Code plugin, scaffold-ide install and an eval harness #851.

Full suite (-n 6, shared venv): 34616 passed. The 11 failures and 2 collection errors are the known baseline:

test_lazy_core_commands (2) and test_duckdb_sandbox_workspace_root (2);
the dotenv dlt tests (4);
the bashlex collection errors (2);
AWS credential-chain tests that pass alone (TestSnowflakeLogin, test_an_operator_exported_aws_endpoint_is_used);
a databricks fake-e2e test that passes alone.

Reviews:

security-review: no findings, before and after the fix round.
Delivery review (world-class-inspection, Diff mode): four P2 findings, all fixed.
defect-scan (Diff mode): one candidate refuted. Mission (MEDIUM), test hermeticity (MEDIUM-LOW) and saved-name aliases (LOW) were fixed. An independent re-check confirmed every fix.

Screenshots / Output

Live runs used a scratch HOME and a null keyring.

fluid forge --non-interactive --context intent.txt with a saved anthropic and no key:

before: ❌ No LLM provider is selected for this run: 'fluid ai setup' saved anthropic, but this command does not read the saved choice. Pass --llm-provider anthropic.
after:  ❌ No API key was configured for the anthropic copilot adapter.

fluid forge --agent --context intent.txt -d ws with a saved Ollama qwen3-coder:30b, a real local generation:

rc = 0
stdout: 3 lines, 0 not JSON
  {"event": "forge.start", ..., "mode": "ai", ...}
  {"event": "forge.contract_written", "path": "ws/contract.fluid.yaml", "action": "created", "size": 2409}
  {"event": "forge.done", ..., "exit_code": 0}
stderr: the Performance and "Next steps" panels

The run's receipt records llm_provider: ollama, llm_model: qwen3-coder:30b, and fluid doctor reports the same provider, model and endpoint.

fluid forge --agent --blank: 4 of 4 stdout lines are JSON. Before, 28 of 32 were not.

`fluid mission run` always drives the tool-use agent loop, but it resolved its LLM without saying so. `_agents_allowed` therefore stayed true, so a run without a TTY auto-picked a coding agent from PATH. Coding agents have no tool use, so the loop then failed mid-run with NotImplementedError after the agent had been invoked. Resolve with a copy of the args carrying `agent_loop=True`. No coding agent is chosen, the caller's args are untouched, and a run with nothing else configured stops with `copilot_no_llm_provider` before any agent runs.

Only the interactive inline setup applied the provider, model and key that `fluid ai setup` saves. `check_llm_readiness` (`fluid doctor`) read the saved choice too, but the real selection (`_provider_name_from`) never did. So non-interactive runs (CI, scripts, `fluid forge --agent` with an AI flag) ignored it, and doctor and forge disagreed about which provider would run. Selection now walks one ladder, and readiness calls it (`_provider_name_from` with no args) instead of keeping its own copy: 1. A named provider: `--llm-provider`, `FLUID_LLM_PROVIDER` or `FLUID_FORGE_AGENT`. 2. Exactly one key environment variable. 3. The saved `fluid ai setup` choice. 4. Exactly one keyring key. 5. A local Ollama, then (non-interactive, nothing configured) the first coding-agent CLI on PATH. - The saved model and Ollama host fill in what the run does not state, for the saved provider only. The environment's `OLLAMA_HOST` wins over the saved host. - A run that names its provider takes nothing from the saved choice. Without `--llm-model`, a named provider gets its catalog model, as before. Command Center's `fluid forge data-model from-intent --llm-provider <p>` call relies on that, so its provider and model are unchanged even where a saved config exists. - The saved `endpoint` field is not applied. It has had no writer since the multi-provider config (#363). Applied next to an environment key, it would send that key to a host it was not saved with. - A saved agent (coding-agent CLI or MCP sampling) is passed over wherever the auto-pick is. That covers a stated endpoint or model, given as a flag or as a `FLUID_LLM_*` variable, and the agent loop. A stated destination is never swapped for an agent. When nothing else selects a provider, `no_provider_message` names the saved agent. - Saved names are read through the provider registry, so a hand-written `claude` or `claude_code` means `anthropic` or `claude-code`. - Keys still resolve through `_resolve_api_key`. The saved choice is never written into `--llm-provider` or `FLUID_LLM_PROVIDER`. - The lean (no litellm) preflight follows the same order. A saved coding agent runs without litellm. - Three re-exports that only the old readiness ladder used are removed. tests/conftest.py gains an autouse fixture that hides the developer's saved choice (`ai_config.json` and the unified `config.yaml`) from every test, as `_hermetic_keyring` does for the OS keyring. Without it, the Command Center pin test failed on a machine with a saved gemini config. The precedence follows layered-config practice (flags, then environment, then the config file, then defaults), as in ConfigArgParse and pydantic-settings.

AGENT_IDE.md promises one JSON object per line on stdout. But Rich panels ("Forge Complete", "Next steps"), prints and error text landed between the `{"event": ...}` lines. In blank mode, 28 of 32 stdout lines were not JSON. Under `--agent`, `forge.run` keeps the stdout the run started with for the events (`_AGENT_EVENTS`) and runs the rest under `contextlib.redirect_stdout(sys.stderr)`. Every human-facing writer moves to stderr without being touched: - Rich consoles built without `file=` (Rich looks `sys.stdout` up at each write, rich/console.py); - `print`; - a provider SDK's banner. A missing extra still propagates from `forge.run`, so an in-process caller (the MCP `forge_run`) still gets the exception. The CLI entry point prints the remedy on stderr under `--agent` (`_render_missing_optional`). Outside `--agent`, nothing changes. The redirect is the stdlib pattern `cli/mcp/dispatch.py` already uses to capture the same events in-process.

#854 walks _provider_name_from and the saved fluid ai setup choice in check_llm_readiness. There FLUID_LLM_API_KEY counts only for a named provider (_provider_is_named(None, env, ...)), never for the saved choice. resolve_llm_config keeps #854's 'not saved' auto-pick condition and this branch's unnamed-provider note.

Runs without a TTY use the provider fluid ai setup saved (#854). That provider is picked, not named, so it gets its own keyring key and the generic key is unused; without a key of its own it is not ready and resolve_llm_config raises copilot_missing_llm_api_key. Both tests fail on main 3b7e8c1.

… stay off project hosts (#853) * security(ai): FLUID_LLM_API_KEY goes only to a provider you name FLUID_LLM_API_KEY does not say which vendor it is for, yet forge sent it to whatever provider it picked, including one it only inferred: a running Ollama or the one OLLAMA_HOST names (as a bearer token, in cleartext over http), the provider of the one keyring key, or the provider of the one key variable set. So an OpenAI key could reach a local or remote Ollama, or Anthropic. The key is now used only when --llm-provider or FLUID_LLM_PROVIDER names the provider the run uses (founder decision, 2026-11-12). A provider forge picks itself gets its own key variable, then its keyring key, and a warning says the generic key was not used. Interactive fluid forge and fluid init record what the operator named before their setup replaces args.llm_provider, so setup's pick counts as picked. fluid ai test reads the key only with --llm-provider, fluid doctor only with FLUID_LLM_PROVIDER. A project .env can no longer pair an OpenAI, Anthropic, Gemini or GitHub endpoint with FLUID_LLM_API_KEY: forge can pick those providers without a name, and then the operator's keyring key, not the project's key, would go out. * security(ai): interactive setup keeps your saved key off a project's host A project .env may point a provider at its own host (OPENAI_BASE_URL, ANTHROPIC_BASE_URL, GEMINI_API_BASE, GITHUB_API_BASE) next to a key of its own; credentials/protected_env pairs it because that key is what the readers send. Interactive fluid forge and fluid init then loaded the provider that fluid ai setup saved and copied its keyring key into the provider's key variable with set_session_env, over the project's key (for Gemini into GOOGLE_API_KEY, which forge reads before GEMINI_API_KEY). The operator's key went to the project's host. set_session_env now leaves the variable alone while a project endpoint of that provider is in effect (dotenv_store.project_endpoints_in_effect, as the Snowflake readers already do), so the project's own key goes there, and logs that the operator's key was not used. fluid ai setup checked a typed or imported key with one litellm call, which reads the endpoint from the environment, so the check sent that key to the project's host too. While such an endpoint is in effect, setup now asks for no key, skips a key it would import, and _validate_api_key sends nothing. For a provider without a mapped group, any LLM endpoint a project file set counts. * fix(ai): build the unused-key warnings from names, never from a key CodeQL (py/clear-text-logging-sensitive-data) flagged the two warnings this branch adds: - resolve_llm_config logged a note unpacked from the same tuple as the API key, returned by _api_key_for_run, a call whose name reads as a password source. The note now comes from _unnamed_provider_note, from the provider names and the named flag alone; the key comes from _resolve_api_key on its own. - set_session_env's warning logged the provider string, which setup can get by classifying a typed key (_classify_key_shape -> detect_provider_from_ api_key). The notice now names the endpoint group from _PROVIDER_ENDPOINT_SERVICES, picked by comparison so it is the table's own constant, and the endpoint variables; the provider string is never logged. * chore(ai): one literal per suggestion, one import style in the key-scope tests Addresses the two code-quality review threads on #853: - the copilot_missing_llm_api_key suggestion this branch added was two adjacent string literals in a list (implicit concatenation, read as a possibly missing comma); it is one literal now; - tests/test_llm_api_key_scope.py imported fluid_build.llm.providers both as a module and with from-imports; it uses the module import only, and the new interactive-setup matrix test does the same. * test(ai): the saved fluid ai setup choice never gets FLUID_LLM_API_KEY Runs without a TTY use the provider fluid ai setup saved (#854). That provider is picked, not named, so it gets its own keyring key and the generic key is unused; without a key of its own it is not ready and resolve_llm_config raises copilot_missing_llm_api_key. Both tests fail on main 3b7e8c1. * chore(tests): one import style for fluid_build.llm.providers in the pairing matrix The file imported the module as llm_providers in two places and with 'from ... import LlmConfig' in _run_codex (github-code-quality on #853). _run_codex now uses llm_providers.LlmConfig as well.

Moves [Unreleased] under [0.29.0] — 2026-11-12, leaving an empty [Unreleased] above it. release.yml extracts this section as the GitHub Release body: 22 entries under Added, Changed, Fixed and Security. Two corrections ride along so the release notes are accurate: - #854 merged without a CHANGELOG change. Its three entries are added: runs without a terminal use the saved fluid ai setup choice, --agent stdout carries only the JSON-Lines events, and fluid mission run no longer picks a coding agent it cannot use. - #852's no-provider entry quoted a "does not read the saved choice" variant that #854 replaced; it now names the saved coding agent the run passes over.

fas89 added 3 commits November 12, 2026 13:24

github-actions Bot added security Security-related changes or reports cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes labels Nov 12, 2026

fas89 merged commit 3b7e8c1 into main Nov 12, 2026
33 checks passed

This was referenced Nov 12, 2026

chore(release): cut v0.29.0 changelog entry #855

Merged

docs: pin forge-cli 0.29.0, and document FLUID_LLM_API_KEY scope, the saved choice and JSON-only --agent stdout (forge-cli#853, #854) Agenticstiger/forge_docs#206

Merged

This was referenced Nov 12, 2026

fix(ai): stated provider flags win, keys for two providers stop, github never inferred #857

Merged

fix(forge,ai): clean machine-readable output — forge.error event, no litellm banner, one contract event #858

Merged

Navigation Menu

Repository navigation

fix(ai): saved AI choice in non-interactive runs, JSON-only --agent stdout, mission agent loop - #854

fix(ai): saved AI choice in non-interactive runs, JSON-only --agent stdout, mission agent loop#854
fas89 merged 3 commits into
mainfrom
fix/forge-noninteractive-saved-ai-config

fas89 commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fas89 commented Nov 12, 2026

Description

Type of Change

Documentation

Checklist

Testing

Screenshots / Output

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant