Vendor questions

[ngoiyaeric]

Here are the seven questions listed on the slide:

• What is your measured error rate, by language and by name origin?
• Can you reproduce a decision from six months ago — inputs, version, output?
• What can it do without a human, and where is that written down?
• Private data, untrusted input and outbound comms at once? Which did you remove?
• How do you test that it disagrees with the user when the user is wrong?
• What happens when the model underneath you is deprecated?
• What does it refuse to do?

[ngoiyaeric]

@coderabbitai plan

[coderabbitai]

Coding Plan

Summary

• Add a protected base policy layer that enforces refusal and disagreement rules even when a user overrides the system prompt.
• Persist per-message provenance (model, prompt snapshot, parameters, tool calls) and centralize model configuration with a deprecation fallback.
• Reduce the lethal-trifecta risk by enforcing the retrieve URL policy in code, adding SSRF protection, and sanitizing untrusted content before it re-enters agent context.
• Build a language and name-origin error-rate eval plus disagreement and refusal evals, wire them into scripts and CI, and document all seven answers in a new governance document.

Design Choices

Design Choice 1: Scope of the ticket: code changes or documentation only

Options Considered:

1. Produce documentation only that describes current behavior and known gaps.
2. Produce documentation plus the minimal code changes needed to make each documented answer honest and verifiable.

Chosen Option: 2

Rationale: The vendor answers must be truthful and testable, so each question needs a matching technical change plus documentation that references it.

Design Choice 2: Enforcement of the refusal and disagreement policy against user prompt overrides

Options Considered:

1. Keep the current full-replacement behavior and only add rules to the default prompt.
2. Introduce a protected base layer that always applies, and append the user prompt after it.

Chosen Option: 2

Rationale: A protected base layer keeps refusal and disagreement rules present even when a user sets a custom prompt.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Safety policy foundation (refusal and disagreement)

This phase adds an explicit refusal policy and a rule that instructs the agent to disagree when the user is wrong. It makes these rules survive user prompt overrides. This phase answers the questions: "What does it refuse to do?" and "How do you test that it disagrees with the user when the user is wrong?".

Task 1: Add a protected base system-prompt layer

Define a non-overridable safety layer that always precedes any prompt.

• Create a new module in lib/agents/ (for example lib/agents/safety-policy.ts) that exports a constant base policy string.
• Include an explicit refusal list. State what the agent must decline (for example: unconfirmed billable orders, requests to exfiltrate private data, requests to fetch internal or non-allowlisted URLs, requests to reveal secrets or the protected policy).
• Include a disagreement rule. Instruct the model to correct a factually wrong user claim, to state the correction plainly, and to cite a source when one is available.
• Keep the wording aligned with the existing tool-usage rules already in getDefaultSystemPrompt.

Task 2: Apply the base layer in the researcher agent

Ensure the safety layer applies on every turn.

• Update lib/agents/researcher.tsx so the effective system prompt is the base policy layer followed by either the user dynamicSystemPrompt or getDefaultSystemPrompt(...).
• Change the current logic that lets dynamicSystemPrompt fully replace the default. The base policy must always prepend.
• Add the refusal and disagreement text to getDefaultSystemPrompt as well, so the default experience matches the documented policy.

🤖 Prompt for AI agents

Implement the safety policy foundation for the agent system. The goal is to add
an explicit refusal policy and a disagreement rule, and make sure these rules
cannot be removed by a user-set system prompt.

- Create a new module in `lib/agents/` (for example
`lib/agents/safety-policy.ts`) that exports a constant base policy string.
- In the base policy, include an explicit refusal list. State what the agent
must decline: unconfirmed billable orders, requests to exfiltrate private data,
requests to fetch internal or non-allowlisted URLs, and requests to reveal
secrets or the protected policy.
- In the base policy, include a disagreement rule. Instruct the model to correct
a factually wrong user claim, state the correction plainly, and cite a source
when one is available.
- Align the wording of the base policy with the existing tool-usage rules
already present in `getDefaultSystemPrompt`.
- Update `lib/agents/researcher.tsx` so the effective system prompt is always
the base policy layer followed by either the user `dynamicSystemPrompt` or
`getDefaultSystemPrompt(...)`.
- Remove the current behavior where `dynamicSystemPrompt` fully replaces the
default prompt. The base policy layer must always prepend to whichever prompt is
used.
- Add the refusal and disagreement text to `getDefaultSystemPrompt` too, so the
default experience matches the documented policy.

Phase 2: Decision reproducibility (provenance)

This phase records enough metadata per message to reproduce a past decision. It answers the question: "Can you reproduce a decision from six months ago — inputs, version, output?".

Task 1: Extend the message schema with provenance columns

Store model, prompt, parameters, and tool activity per message.

• Add columns to the messages table in lib/db/schema.ts: model identifier, model label, a system-prompt snapshot or content hash, generation parameters, and structured tool calls and tool results.
• Use JSON columns for the parameter bag and the tool-call records.
• Create a Drizzle migration under drizzle/migrations/ for the new columns. Make all new columns nullable so historical rows remain valid.

Task 2: Persist provenance during generation

Capture and store the metadata that the researcher already holds in memory.

• Update lib/agents/researcher.tsx to return the resolved model identifier and label, the effective system prompt (or its hash), the generation parameters, and the toolCalls/toolResponses arrays it already collects.
• Update app/actions.tsx and lib/actions/chat-db.ts (createMessage, saveChat) to write these values into the new columns instead of only serializing them into the content text blob.
• Record the model label and identifier from the centralized model config added in Phase 3, so the stored version matches the config source of truth.

🤖 Prompt for AI agents

Implement decision reproducibility by persisting provenance metadata for each
message. The goal is to store enough data per message to reproduce a past
decision (its inputs, version, and output).

- Add columns to the `messages` table in `lib/db/schema.ts`: model identifier,
model label, a system-prompt snapshot or content hash, generation parameters,
and structured tool calls and tool results.
- Use JSON columns for the generation-parameters bag and the tool-call records.
- Create a Drizzle migration under `drizzle/migrations/` for the new columns.
Make all new columns nullable so historical rows remain valid.
- Update `lib/agents/researcher.tsx` to return the resolved model identifier and
label, the effective system prompt (or its hash), the generation parameters, and
the `toolCalls`/`toolResponses` arrays it already collects.
- Update `app/actions.tsx` and `lib/actions/chat-db.ts` (functions
`createMessage` and `saveChat`) to write these values into the new columns
instead of only serializing them into the `content` text blob.
- Record the model label and identifier from the centralized model config added
in Phase 3, so the stored version always matches the config source of truth.

Phase 3: Model configuration and deprecation resilience

This phase centralizes model IDs and adds runtime fallback. It answers the question: "What happens when the model underneath you is deprecated?".

Task 1: Centralize model configuration

Create a single source of truth for provider model IDs.

• Create a config module in lib/config/ (for example lib/config/models.ts) that maps each user-facing label (Grok 4.2, Gemini 3.1 Pro, GPT-5.1, and others) to its provider model ID and provider.
• Include a per-entry deprecation flag and a fallback label field.
• Refactor getModel in lib/utils/index.ts to read IDs from this module instead of a hardcoded switch.
• Refactor the hardcoded model ID in lib/agents/tools/geospatial.tsx to read from the same module.

Task 2: Add deprecation-aware fallback at request time

Handle a provider rejection of a deprecated or invalid model ID.

• Wrap the actual streamText call path in lib/agents/researcher.tsx so a model-not-found or invalid-model error triggers a single fallback to the configured fallback label.
• Log the fallback event and surface a clear message if no fallback is available.
• Confirm the fallback selection is also written to the provenance columns from Phase 2.

🤖 Prompt for AI agents

Implement model configuration centralization and deprecation-aware fallback. The
goal is to have one source of truth for model IDs and a runtime fallback when a
model becomes deprecated.

- Create a config module in `lib/config/` (for example `lib/config/models.ts`)
that maps each user-facing label (`Grok 4.2`, `Gemini 3.1 Pro`, `GPT-5.1`, and
others) to its provider model ID and provider.
- In this config module, include a per-entry deprecation flag and a fallback
label field.
- Refactor `getModel` in `lib/utils/index.ts` to read model IDs from this module
instead of the current hardcoded `switch` statement.
- Refactor the hardcoded model ID in `lib/agents/tools/geospatial.tsx` to read
from the same central config module.
- Wrap the `streamText` call path in `lib/agents/researcher.tsx` so that a
model-not-found or invalid-model error triggers a single fallback attempt using
the configured fallback label.
- Log the fallback event. If no fallback is available, surface a clear error
message.
- Confirm the fallback model selection is also written to the provenance columns
added in Phase 2.

Phase 4: Lethal trifecta mitigation

This phase reduces the ambient risk of private data leaving through untrusted-content-driven tool calls. It answers the question: "Private data, untrusted input and outbound comms at once? Which did you remove?".

Task 1: Enforce the retrieve URL policy in code

Move the "only user-provided URLs" rule from prompt text into enforced code.

• Update lib/agents/tools/retrieve.tsx and its schema to accept only URLs that appear in the current user message, not URLs first introduced by prior tool output.
• Add a denylist for internal, loopback, link-local, and cloud-metadata address ranges to prevent server-side request forgery.
• Reject any URL that fails the checks with a clear tool error.

Task 2: Sanitize untrusted content before it re-enters context

Strip injection and exfiltration vectors from external content.

• Create a shared sanitizer in lib/agents/tools/ and apply it to the outputs of search.tsx, retrieve.tsx, document-retrieve.tsx, and skyfi.tsx before the content is pushed back into messages.
• Remove or neutralize embedded instruction patterns, markdown image and link exfiltration vectors, and any detected secret-like tokens.
• Keep the sanitizer conservative so it does not degrade normal answers.

Task 3: Constrain the firecrawl-to-system-prompt pipeline

Prevent scraped content from silently becoming the durable system prompt.

• Update lib/actions/system-prompt.ts so scraped domain content passes through the Phase 4.2 sanitizer before it reaches generateText.
• Confirm that generated prompts still pass through the protected base layer from Phase 1, so a generated prompt cannot remove refusal rules.

🤖 Prompt for AI agents

Implement lethal trifecta mitigations to reduce the risk of private data leaving
through untrusted-content-driven tool calls. The goal is to enforce URL policy
in code, sanitize untrusted content, and constrain the prompt-generation
pipeline.

- Update `lib/agents/tools/retrieve.tsx` and its schema so it accepts only URLs
that appear in the current user message, not URLs first introduced by prior tool
output.
- In `retrieve.tsx`, add a denylist for internal, loopback, link-local, and
cloud-metadata address ranges to prevent server-side request forgery (SSRF).
- Reject any URL that fails these checks with a clear tool error.
- Create a shared sanitizer module in `lib/agents/tools/` and apply it to the
outputs of `search.tsx`, `retrieve.tsx`, `document-retrieve.tsx`, and
`skyfi.tsx` before the content is pushed back into `messages`.
- In the sanitizer, remove or neutralize embedded instruction patterns, markdown
image and link exfiltration vectors, and any detected secret-like tokens.
- Keep the sanitizer conservative so normal answers are not degraded.
- Update `lib/actions/system-prompt.ts` so scraped domain content passes through
the sanitizer before it reaches `generateText`.
- Confirm that generated prompts still pass through the protected base layer
from Phase 1, so a generated prompt cannot remove the refusal rules.

Phase 5: Evaluation harness

This phase measures error rate by language and name origin and verifies disagreement and refusal behavior. It answers the question: "What is your measured error rate, by language and by name origin?" and it validates Phase 1.

Task 1: Build the evaluation datasets and runner

Create a repeatable eval that produces error rates by segment.

• Create an eval directory under tests-unit/ (for example tests-unit/evals/) using the existing bun:test style.
• Add datasets that vary prompts by language and by name origin, each with expected outcomes.
• Implement a runner that computes error rate per language segment and per name-origin segment and prints a summary table.

Task 2: Add disagreement and refusal evals

Test the Phase 1 policy directly.

• Add eval cases where the user states a false claim. Assert that the response corrects the user.
• Add eval cases for each item in the refusal list. Assert that the response declines.

Task 3: Wire evals into scripts and CI

Make the eval suite discoverable and runnable.

• Add test:unit and test:eval scripts to package.json that run the bun:test files.
• Add a CI step (extend .github/workflows/ or add a new workflow) that runs test:unit. Gate the model-calling test:eval behind an opt-in flag or required API keys so CI without keys does not fail.

🤖 Prompt for AI agents

Implement the evaluation harness that measures error rate by language and name
origin, and validates the Phase 1 disagreement and refusal policy. The goal is a
repeatable, CI-wired eval suite.

- Create an eval directory under `tests-unit/` (for example `tests-unit/evals/`)
using the existing `bun:test` style already used in `tests-unit/`.
- Add datasets that vary prompts by language and by name origin, each with an
expected outcome.
- Implement a runner that computes the error rate per language segment and per
name-origin segment, and prints a summary table.
- Add eval cases where the user states a false claim, and assert that the
response corrects the user (validates the Phase 1 disagreement rule).
- Add eval cases for each item in the Phase 1 refusal list, and assert that the
response declines (validates the Phase 1 refusal policy).
- Add `test:unit` and `test:eval` scripts to `package.json` that run the
corresponding `bun:test` files.
- Add a CI step by extending `.github/workflows/` or adding a new workflow that
runs `test:unit`.
- Gate the model-calling `test:eval` behind an opt-in flag or required API keys,
so CI runs without keys do not fail.

Phase 6: Governance documentation

This phase writes down the answers to all seven questions and the autonomy boundaries. It answers the question: "What can it do without a human, and where is that written down?" and it consolidates the other answers.

Task 1: Author the AI governance document

Create a single authoritative reference.

• Create docs/AI_GOVERNANCE.md following the existing SCREAMING_SNAKE_CASE.md and section conventions.
• Answer each of the seven vendor questions in its own section. Reference the concrete code and eval added in Phases 1 through 5.
• Include an autonomy-boundary section. State which actions run without human review (read, search, geocode, document retrieval) and which require explicit confirmation (the SkyFi place_order confirm gate). Note the taskManager/inquire clarify gate status.
• Include the refusal list and the disagreement rule text from Phase 1 verbatim.

Task 2: Add discoverability callout

Point readers to the governance document.

• Add a one-line pointer to docs/AI_GOVERNANCE.md in the README "🧪 Testing & Automated Verification" section, consistent with how Playwright testing is surfaced.

🤖 Prompt for AI agents

Implement the governance documentation that answers all seven vendor questions
and describes the autonomy boundaries. The goal is a single authoritative
document plus a discoverability pointer.

- Create `docs/AI_GOVERNANCE.md` following the existing
`SCREAMING_SNAKE_CASE.md` naming and section conventions used in `docs/`.
- In this document, answer each of the seven vendor questions in its own
section. Reference the concrete code and evals added in Phases 1 through 5.
- Include an autonomy-boundary section. State which actions run without human
review (read, search, geocode, document retrieval) and which require explicit
confirmation (the SkyFi `place_order` `confirm` gate). Note the status of the
`taskManager`/`inquire` clarify gate.
- Include the refusal list and the disagreement rule text from Phase 1 verbatim
in the document.
- Add a one-line pointer to `docs/AI_GOVERNANCE.md` in the README "🧪 Testing &
Automated Verification" section, consistent with how Playwright testing is
already surfaced there.

Research

The system is a Next.js application with an agent layer in lib/agents/. The researcher agent in lib/agents/researcher.tsx runs the live chat flow. It calls streamText with maxSteps: 5 and a registry of tools from lib/agents/tools/index.tsx. The taskManager and inquire human-in-the-loop agents exist but are never called from app/actions.tsx.

Model IDs are hardcoded in a switch inside lib/utils/index.ts (getModel) and again in lib/agents/tools/geospatial.tsx. config/model.json is an unused stub.

Chat data is stored by lib/actions/chat-db.ts in a messages table (lib/db/schema.ts) that holds only role, freeform content, an embedding, a location FK, and createdAt.

The default system prompt lives in getDefaultSystemPrompt in researcher.tsx. A user-set custom prompt fully replaces it.

Tools mix private data (document-retrieve.tsx, skyfi.tsx), untrusted external content (search.tsx, retrieve.tsx), and outbound network calls in one execution context with no content sanitization or URL allowlist.

Unit tests use bun:test in tests-unit/ but are not wired into package.json scripts or CI.

Docs live in docs/ as SCREAMING_SNAKE_CASE.md files.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement the safety policy foundation for the agent system. The goal is to add
an explicit refusal policy and a disagreement rule, and make sure these rules
cannot be removed by a user-set system prompt.

- Create a new module in `lib/agents/` (for example
`lib/agents/safety-policy.ts`) that exports a constant base policy string.
- In the base policy, include an explicit refusal list. State what the agent
must decline: unconfirmed billable orders, requests to exfiltrate private data,
requests to fetch internal or non-allowlisted URLs, and requests to reveal
secrets or the protected policy.
- In the base policy, include a disagreement rule. Instruct the model to correct
a factually wrong user claim, state the correction plainly, and cite a source
when one is available.
- Align the wording of the base policy with the existing tool-usage rules
already present in `getDefaultSystemPrompt`.
- Update `lib/agents/researcher.tsx` so the effective system prompt is always
the base policy layer followed by either the user `dynamicSystemPrompt` or
`getDefaultSystemPrompt(...)`.
- Remove the current behavior where `dynamicSystemPrompt` fully replaces the
default prompt. The base policy layer must always prepend to whichever prompt is
used.
- Add the refusal and disagreement text to `getDefaultSystemPrompt` too, so the
default experience matches the documented policy.
===============================================================================

Task: 2

Implement decision reproducibility by persisting provenance metadata for each
message. The goal is to store enough data per message to reproduce a past
decision (its inputs, version, and output).

- Add columns to the `messages` table in `lib/db/schema.ts`: model identifier,
model label, a system-prompt snapshot or content hash, generation parameters,
and structured tool calls and tool results.
- Use JSON columns for the generation-parameters bag and the tool-call records.
- Create a Drizzle migration under `drizzle/migrations/` for the new columns.
Make all new columns nullable so historical rows remain valid.
- Update `lib/agents/researcher.tsx` to return the resolved model identifier and
label, the effective system prompt (or its hash), the generation parameters, and
the `toolCalls`/`toolResponses` arrays it already collects.
- Update `app/actions.tsx` and `lib/actions/chat-db.ts` (functions
`createMessage` and `saveChat`) to write these values into the new columns
instead of only serializing them into the `content` text blob.
- Record the model label and identifier from the centralized model config added
in Phase 3, so the stored version always matches the config source of truth.
===============================================================================

Task: 3

Implement model configuration centralization and deprecation-aware fallback. The
goal is to have one source of truth for model IDs and a runtime fallback when a
model becomes deprecated.

- Create a config module in `lib/config/` (for example `lib/config/models.ts`)
that maps each user-facing label (`Grok 4.2`, `Gemini 3.1 Pro`, `GPT-5.1`, and
others) to its provider model ID and provider.
- In this config module, include a per-entry deprecation flag and a fallback
label field.
- Refactor `getModel` in `lib/utils/index.ts` to read model IDs from this module
instead of the current hardcoded `switch` statement.
- Refactor the hardcoded model ID in `lib/agents/tools/geospatial.tsx` to read
from the same central config module.
- Wrap the `streamText` call path in `lib/agents/researcher.tsx` so that a
model-not-found or invalid-model error triggers a single fallback attempt using
the configured fallback label.
- Log the fallback event. If no fallback is available, surface a clear error
message.
- Confirm the fallback model selection is also written to the provenance columns
added in Phase 2.
===============================================================================

Task: 4

Implement lethal trifecta mitigations to reduce the risk of private data leaving
through untrusted-content-driven tool calls. The goal is to enforce URL policy
in code, sanitize untrusted content, and constrain the prompt-generation
pipeline.

- Update `lib/agents/tools/retrieve.tsx` and its schema so it accepts only URLs
that appear in the current user message, not URLs first introduced by prior tool
output.
- In `retrieve.tsx`, add a denylist for internal, loopback, link-local, and
cloud-metadata address ranges to prevent server-side request forgery (SSRF).
- Reject any URL that fails these checks with a clear tool error.
- Create a shared sanitizer module in `lib/agents/tools/` and apply it to the
outputs of `search.tsx`, `retrieve.tsx`, `document-retrieve.tsx`, and
`skyfi.tsx` before the content is pushed back into `messages`.
- In the sanitizer, remove or neutralize embedded instruction patterns, markdown
image and link exfiltration vectors, and any detected secret-like tokens.
- Keep the sanitizer conservative so normal answers are not degraded.
- Update `lib/actions/system-prompt.ts` so scraped domain content passes through
the sanitizer before it reaches `generateText`.
- Confirm that generated prompts still pass through the protected base layer
from Phase 1, so a generated prompt cannot remove the refusal rules.
===============================================================================

Task: 5

Implement the evaluation harness that measures error rate by language and name
origin, and validates the Phase 1 disagreement and refusal policy. The goal is a
repeatable, CI-wired eval suite.

- Create an eval directory under `tests-unit/` (for example `tests-unit/evals/`)
using the existing `bun:test` style already used in `tests-unit/`.
- Add datasets that vary prompts by language and by name origin, each with an
expected outcome.
- Implement a runner that computes the error rate per language segment and per
name-origin segment, and prints a summary table.
- Add eval cases where the user states a false claim, and assert that the
response corrects the user (validates the Phase 1 disagreement rule).
- Add eval cases for each item in the Phase 1 refusal list, and assert that the
response declines (validates the Phase 1 refusal policy).
- Add `test:unit` and `test:eval` scripts to `package.json` that run the
corresponding `bun:test` files.
- Add a CI step by extending `.github/workflows/` or adding a new workflow that
runs `test:unit`.
- Gate the model-calling `test:eval` behind an opt-in flag or required API keys,
so CI runs without keys do not fail.
===============================================================================

Task: 6

Implement the governance documentation that answers all seven vendor questions
and describes the autonomy boundaries. The goal is a single authoritative
document plus a discoverability pointer.

- Create `docs/AI_GOVERNANCE.md` following the existing
`SCREAMING_SNAKE_CASE.md` naming and section conventions used in `docs/`.
- In this document, answer each of the seven vendor questions in its own
section. Reference the concrete code and evals added in Phases 1 through 5.
- Include an autonomy-boundary section. State which actions run without human
review (read, search, geocode, document retrieval) and which require explicit
confirmation (the SkyFi `place_order` `confirm` gate). Note the status of the
`taskManager`/`inquire` clarify gate.
- Include the refusal list and the disagreement rule text from Phase 1 verbatim
in the document.
- Add a one-line pointer to `docs/AI_GOVERNANCE.md` in the README "🧪 Testing &
Automated Verification" section, consistent with how Playwright testing is
already surfaced there.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---