[Synaeve]

Unblocks adcontextprotocol/adcp#8929.

Problem

A grader that tests with its own credential passes it in a run-level test_kit. resolveDeclaredTestKit loads a storyboard's declared kit (prerequisites.test_kit) only when options.test_kit is unset. So that run-level kit replaces every declared kit, and no storyboard gets its own fixtures:

• assets for $build_assets_from_format;
• commercial_relationship for phase skip_if;
• creative_build_probes;
• creative_formats.

AAO's hosted comply() does exactly this (adcp#8929). adcp#8930 works around it by copying one kit's assets into the run kit.

routeStoryboard cannot fix this from the host side:

• Its result has no kit field.
• comply() does not call it for the discovery-diagnostic branch, for capability-gated storyboards, or in runWithDegradedProfile.
• A replacement route.storyboard cannot change the kit fields the runner reads directly.

Changes

src/lib/testing/storyboard/test-kit.ts

• resolveDeclaredTestKit takes test_kit_auth, a credential laid over the declared kit:
  • A kit that declares sandbox: true keeps every field except auth, which the overlay replaces whole. None of the kit's own api_key, basic or probe_task sits next to the caller's credential.
  • A kit with no credential of its own (no auth.api_key or auth.basic, as in runner-contract kits) gets the overlay as its auth.
  • A kit that declares sandbox: false, or carries a credential without declaring sandbox: true, keeps its own auth (or none). It may be the live-mode principal the storyboard tests (comply_controller_mode_gate grades false FAIL against conformant sellers: from_test_kit never resolves the storyboard-declared kit (silent no-auth locally; seller-bearer substitution in the hosted engine) adcp#6735). An absent sandbox is never read as sandbox, the same polarity as getSandboxEntities().
  • A storyboard that declares no kit gets { auth: test_kit_auth }, which is what a run-level kit gives it today.
  • A declared kit that cannot be loaded (no compliance root, or no file) gets no overlay. Nothing says it is not a live-mode principal, so its from_test_kit steps fail as a configuration error, as they do without test_kit_auth.
  • The kit is validated after the overlay. The overlay is always consumed: the returned options never carry test_kit_auth.
• New validateTestKitAuthOverlay, also exported from the storyboard barrel next to validateTestKit:
  • It refuses test_kit together with test_kit_auth, with TestKitValidationError on field test_kit_auth.
  • It checks the overlay with the test_kit.auth rules: probe_task is required and must be allowlisted, and api_key must be printable ASCII.
  • Those rules move into one helper. The test_kit.auth messages are unchanged.
• Loading the declared file moves into loadDeclaredTestKit, which keeps the same containment, symlink and error behaviour.

src/lib/testing/types.ts, agent-tester.ts, index.ts

• New exported TestKitAuth type, the shape of test_kit.auth.
• TestOptions.test_kit_auth?: TestKitAuth, documented.

src/lib/testing/storyboard/runner.ts

• createAuthOverrideClient (used for auth: none steps and invalid-key probes) drops test_kit_auth as it already drops test_kit, so neither can be promoted back into the probe's credential.
• runCredentialValues treats test_kit_auth as a credential the run holds, for header exclusion and evidence scrubbing.
• Both are defense in depth: the overlay is consumed before any step runs.

src/lib/testing/compliance/comply.ts

• comply() validates test_kit_auth up front, before contacting the agent.
• Every storyboard comply() runs, on any branch, goes through runStoryboard, which resolves the kit per storyboard.

src/lib/testing/client.ts

• Without run-level auth, the transport falls back to test_kit_auth when there is no test_kit. comply()'s own discovery then authenticates the way it does with test_kit: { auth }.
• The check stays null-safe for a JavaScript caller that passes test_kit: null.

src/lib/testing/local-agent-runner.ts

• The resolvePerStoryboard JSDoc says a per-storyboard test_kit override on a run with test_kit_auth must also set test_kit_auth: undefined.

Docs

• docs/guides/VALIDATE-YOUR-AGENT.md: how to grade with your own credential and keep declared kits.
• docs/guides/CREATIVE-BUILD-PROBE.md: test_kit_auth keeps the declared kit's recipes.

Tests

New file: test/lib/storyboard-test-kit-auth-overlay.test.js, 15 tests. All of them fail on main. The two added for the sandbox polarity fix also fail on the first commit of this PR.

Unit tests on resolveDeclaredTestKit:

• a sandbox kit keeps its fixtures, and its whole auth block, basic included, becomes the overlay;
• resolving twice is a no-op;
• a live-mode kit keeps its credential;
• a kit with no credential (no auth, or probe_task only) gets the overlay;
• a kit that carries a credential and omits sandbox keeps it;
• over every kit in the bundled compliance caches: live kits declare sandbox: false and keep their principal, sandbox: true kits and runner kits take the overlay, and a credential-bearing kit without sandbox keeps its credential;
• a storyboard that declares no kit gets { auth };
• a declared kit that cannot be loaded gets no overlay, and the overlay is still consumed;
• a kit's stale probe_task is replaced before validation;
• one set of run options resolves different kits for different storyboards without being changed;
• the "both" refusal and the overlay validation work.

comply() tests against a loopback MCP agent that records each probe's Authorization and x-adcp-auth headers and its arguments. Each probe sits in a phase gated on its own kit's fixture_marker, and the kit-less storyboards have a phase that runs only if a marker leaked. They check that:

• every storyboard gets its own fixtures;
• sandbox and kit-less storyboards send the overlay credential;
• the live kit sends its own key;
• an unloadable declared kit sends nothing;
• auth: none steps send no Authorization and no x-adcp-auth, with and without run-level auth;
• no sandbox demo key reaches the agent, and no marker leaks;
• without run-level auth, discovery authenticates with the overlay;
• test_kit + test_kit_auth, and an overlay without probe_task, are refused with no request to the agent.

Existing suites:

• test/lib/storyboard-declared-test-kit.test.js: 20/20.
• The test-kit and comply()/testing-client suites pass. The only local failures come from incomplete schema caches in my checkout and fail identically on main.
• tsc --noEmit, eslint (no new warnings) and prettier are clean.
• npm run review:codex -- --all --base main was not run: the codex CLI is not available in this environment.

Changeset

@adcp/sdk minor (.changeset/test-kit-auth-overlay.md): a new option and type, no breaking change.

Notes for graders: kit principals under the overlay

Inventory of the kits in the 4 bundled SDK caches, adcp source, and the 68 published adcp bundles:

• the only live-mode kit (acme-outdoor-live.yaml) declares sandbox: false;

• every brand kit declares sandbox: true;

• runner-contract kits omit sandbox, and billing-gate-runner.yaml is the only one with a credential.

• billing-gate-runner.yaml.

  • The kit omits sandbox and carries demo-billing-passthrough-v1, the spoofable test principal mapped to commercial_relationship: passthrough_only. So it keeps that credential.
  • On hosted runs, the optional per_agent_gate_reject phase then runs for sellers that advertise agent billing, under that demo principal, exactly as a local run does.
  • A seller must map that principal, or its own convention, to passthrough-only, as the kit already requires.
  • adcp maintainers should confirm this is the hosted behaviour they want.

• canonical_formats. It describes the kit key as "a different buyer principal from the one controller seeding ran under". With test_kit_auth set to the transport credential, the two principals are the same, as on hosted runs today.

Out of scope / follow-ups

• adcontextprotocol/adcp, once this is released:
  • hosted comply() passes the owner credential as test_kit_auth instead of a run-level test_kit;
  • delete the adcp#8930 assets copy and its drift guard;
  • delete liveModeTestKitRoute (adcp#7772) if it is no longer needed;
  • confirm the billing-gate-runner note and decide the canonical_formats note above.
• Once AAO switches, the docs/guides/CREATIVE-BUILD-PROBE.md sentence "Hosted auth defaults … do not merge missing recipes from a separately declared brand kit" goes stale and needs updating.
• No CLI flag is added. A local run can already pass a whole kit with --test-kit.

🤖 Generated with Claude Code

[aao-secretariat]

Ladon verdict: Approve

Approve — additive test_kit_auth credential overlay with correct minor changeset.

What I checked:

• Changeset correctness: minor is correct for the additive overlay feature.
• Witness-not-translator: overlay is consumed from returned options; no fabrication of upstream fields.
• Credential safety: runCredentialValues redaction and createAuthOverrideClient (auth: none) both strip/cover test_kit_auth.
• Validation: validateTestKitAuthOverlay refuses test_kit+test_kit_auth together before the agent is contacted (tests assert agent.requests.length === 0).
• Refactor of resolveDeclaredTestKit → loadDeclaredTestKit preserves no-overlay behavior and throw ordering; unloadable-kit path fails closed.
• No new external I/O, timeouts, or async surface.

Medium findings (1):

• src/lib/testing/storyboard/test-kit.ts:318 — live-mode guard keys on kit.sandbox === false, so a declared live kit omitting sandbox is overlaid and could grade a conformant seller FAIL (opposite polarity to the !kit.sandbox convention at sandbox-entities.ts:115). Worth a follow-up fix.

A single medium finding with no blocking issues, no escalation trigger, no gated paths, and no team gate falls through to row 9 → approve. gated_paths is false (row 2 does not apply despite review_decision: REVIEW_REQUIRED). high_risk is false.

Medium findings

• src/lib/testing/storyboard/test-kit.ts:318 — test_kit_auth live-mode protection fails open for kits that omit sandbox

[aao-secretariat]

Ladon verdict: Approve

Approve — prior medium finding resolved, no new findings in the delta.

This is a subsequent review (prior outcome: approve). The earlier medium finding — test_kit_auth live-mode protection failing open for kits that omit sandbox (src/lib/testing/storyboard/test-kit.ts:318) — has been addressed. The reworked withAuthOverlay (test-kit.ts:308) now keeps a declared kit's own api_key/basic credential instead of taking the caller's overlay when sandbox is absent. Polarity matches getSandboxEntities, carriesCredential covers the two inline credential kinds (OAuth resolves via options.auth, never inline), and a new bundled-cache test exercises the credential-bearing-without-sandbox path. Changeset present and docs updated.

Decision table: no critical/high findings (row 1 n/a); gated_paths false (row 2 n/a); high_risk false (rows 3/5 n/a); no medium findings at all (rows 4/8 n/a); prior decision was approve, not escalate (row 6 n/a); no no-auto-approve team match (row 7 n/a). Falls through to row 9 → approve.