Skip to content

docs(architecture): use Promptfoo-native coding-agent evals - #485

Open
christso wants to merge 44 commits into
mainfrom
docs/execution-gateway-adr
Open

christso wants to merge 44 commits into
mainfrom
docs/execution-gateway-adr

Conversation

@christso

@christso christso commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • replace the proposed one-shot AllAgents Gateway with Promptfoo's built-in Claude Agent SDK and Codex SDK providers
  • run each expanded evaluation row serially against ./.eval/workspace, recreated from a protected immutable seed by lifecycle extensions
  • keep source staging, workspace reset, deterministic filesystem checks, receipts, cleanup, and disposable-job isolation as small job-owned glue
  • use Promptfoo-native output, provider metadata, OpenTelemetry spans, and trajectory:* assertions instead of a custom provider and ATIF conversion
  • document provider-specific sandbox, session, cache, deferred-grading, transcript, redaction, and cleanup limits
  • retain the prior gateway, credential-broker, Harbor, and incumbent analysis as clearly labeled historical/future-service research
  • document how the PR 679 Promptfoo experiment can move repository, workdir, cache, and skill setup out of top-level metadata into a source/workspace catalog plus fixed working_dir

Selected boundary

V1 is a trusted, single-tenant local/CI evaluation running inside a disposable rootless container or VM. Promptfoo owns matrices, repetitions, assertions, scoring, traces, and reports. The harness owns exact source staging and one private workspace copy per row.

Live-workspace assertions are deterministic-only. If semantic grading is also required, the deterministic phase persists unique row evidence and a separate evaluation grades that immutable artifact. Promptfoo currently logs afterEach failures rather than failing the command, so the extension writes receipts/failure sentinels and the job wrapper validates them before accepting a run.

A separate execution service is reconsidered only for concrete requirements such as hostile tenants, remote callers, strict per-row process isolation, credential/network phase separation, hidden verifiers, durable cancellation/recovery, mandatory ATIF, OMP support, or cross-machine retention and scheduling.

Verification

  • bun run docs:build
  • git diff --check
  • all relative Markdown links and anchors resolve across the seven affected decision/plan/research documents
  • final architecture review found no remaining high- or medium-priority findings

Documentation-only change; no CLI runtime behavior changed, so no CLI E2E was applicable.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Deploying allagents with  Cloudflare Pages  Cloudflare Pages

Latest commit: c00ebc7
Status: ✅  Deploy successful!
Preview URL: https://58bbe935.allagents.pages.dev
Branch Preview URL: https://docs-execution-gateway-adr.allagents.pages.dev

View logs

@christso
christso force-pushed the docs/execution-gateway-adr branch from be79c77 to d3df7ba Compare September 18, 2026 07:57
@christso christso changed the title docs(architecture): define coding-agent execution gateway docs(architecture): simplify coding execution gateway Sep 19, 2026
@christso
christso force-pushed the docs/execution-gateway-adr branch from 7635b4c to f6e09c2 Compare September 22, 2026 02:30
@christso christso changed the title docs(architecture): simplify coding execution gateway docs(architecture): adopt UHP through HarnessRouter Sep 22, 2026
@christso
christso force-pushed the docs/execution-gateway-adr branch from eb94b1f to 50c67a9 Compare September 23, 2026 23:41
Focus the ADR on user and operator impact while keeping implementation plumbing in the linked plan.
Document E2B's sandbox and self-hosting boundaries, explain why it is not a HarnessRouter replacement, and keep E2B-derived runtime work out of version one.
@christso
christso force-pushed the docs/execution-gateway-adr branch from fbb7184 to a40a408 Compare September 27, 2026 08:12
@christso christso changed the title docs(architecture): adopt UHP through HarnessRouter docs(architecture): define AllAgents Gateway v1 Sep 27, 2026
@christso christso changed the title docs(architecture): define AllAgents Gateway v1 docs(architecture): define HarnessRouter workspace execution Sep 27, 2026
@christso christso changed the title docs(architecture): define HarnessRouter workspace execution docs(architecture): define Git and OCI workspace execution Sep 27, 2026
@christso christso changed the title docs(architecture): define Git and OCI workspace execution docs(architecture): define AllAgents Gateway workspace execution Sep 27, 2026
@christso christso changed the title docs(architecture): define AllAgents Gateway workspace execution docs(architecture): split workspace preparation from execution Sep 28, 2026
@christso christso changed the title docs(architecture): split workspace preparation from execution docs(architecture): define one-shot AllAgents Gateway Sep 28, 2026
@christso christso changed the title docs(architecture): define one-shot AllAgents Gateway docs(architecture): use Promptfoo-native coding-agent evals Sep 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant