Skip to content

test(mcp): add HTTP Code Mode agent evaluations - #1675

Draft
betegon wants to merge 1 commit into
bt/code-mode-httpfrom
bt/code-mode-http-evals
Draft

betegon wants to merge 1 commit into
bt/code-mode-httpfrom
bt/code-mode-http-evals

Conversation

@betegon

@betegon betegon commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Prerequisite #1676 is merged. Stack: #1670 (HTTP Code Mode) → this PR. Merge #1670 before rebasing/retargeting this PR to main.

Tests whether agents discover HTTP Code Mode themselves and complete API tasks missing from the normal MCP catalog. Real Claude/Codex clients use local Dynamic Workers and stateful API fixtures; independent verifiers check the resulting state, including collateral writes, rather than trusting the agent's answer.

The harness covers dashboard copying, releases/deploys, repository pagination, alerts, monitors, partial-write recovery and permission failures. A catalog-only control measures the existing capability gap. Reports retain model usage, tool/API calls and provenance. Runtime, permissions and their safety tests stay in #1670; SDK comparisons remain future work.

Validated after separation: 56 fixture/adapter tests + 4 discovery integration tests, both Code Mode typechecks, lint, documentation links and frozen install. After restacking onto the lifecycle fix, the four discovery integration tests and both Code Mode typechecks pass again. No new model runs were performed for this split or restack; earlier results and their limits are preserved in the evaluation guide. Fixtures do not establish production authorization or hosted CPU enforcement.

@vercel

vercel Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sentry-local Ready Ready Preview Oct 27, 2026 5:14pm UTC

Request Review

@betegon
betegon force-pushed the bt/code-mode-http-evals branch from 8d7e92d to 954c0cd Compare October 27, 2026 13:28
betegon added a commit that referenced this pull request Oct 27, 2026
The Cloudflare test plugin still used an older Miniflare/workerd than
Wrangler, so updating the direct dependency alone left different
runtimes in local and integration tests.

Switch to `@cloudflare/vitest-plugin` 1.4.6 and align Miniflare/workerd
with Wrangler. Miniflare 5 removed `fetchMock`; an Undici outbound
service preserves the existing fixtures and `disableNetConnect()`
policy.

This is the prerequisite for #1670 (HTTP Code Mode), followed by #1675
(agent evals). Only seven test/dependency files move here; the broker,
permissions and production integration remain in #1670.

Validated independently on Node 24: **443 Cloudflare tests across 34
files**, root typechecks, lint and frozen install. The earlier full
workspace run had unrelated CLI fixture/platform failures; CLI fixes are
excluded. No deployment performed.
@betegon
betegon force-pushed the bt/code-mode-http-evals branch from 954c0cd to d518be2 Compare October 27, 2026 13:44

This branch was successfully deployed

2 active deployments
Preview — 63999543 Deployed Oct 27, 2026 by vercel[bot]
Actions — 63999543 Deployed Oct 27, 2026 by betegon via eval #1382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant