Skip to content

fix(testing): validate capabilities against the served release; key compliance_testing in the cross-check - #3280

Merged
bokelley merged 3 commits into
mainfrom
bokelley/capabilities-served-release
Oct 15, 2026
Merged

bokelley merged 3 commits into
mainfrom
bokelley/capabilities-served-release

Conversation

@bokelley

@bokelley bokelley commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes two capability-discovery bugs that make conformant agents fail testCapabilityDiscovery. Both were found while fixing the AdCP repo's external smoke check (adcontextprotocol/adcp#8208), which carries workarounds for them.

#3278: validate get_adcp_capabilities against the served release, not the pin

A 3.0 pin can't carry adcp_version, so buildVersionEnvelope sends only adcp_major_version: 3. Per the versioning spec, the seller then serves its highest 3.x release and echoes it in adcp_version. Buyers SHOULD validate against that release, not their pin ("Server echoes", docs/reference/versioning.mdx). discoverAgentProfile and TaskExecutor.validateResponseSchema both validated strictly against the pin. A 3.2 response from the reference agent therefore produced false capabilities_schema_issues, and getAdcpCapabilities() failed in strict mode. The two false issues were media_buy.features.bidding_policy must be boolean and the closed 3.0 compliance_testing.scenarios enum.

  • New internal helper: resolveServedValidationVersion(response, pin) in src/lib/validation/schema-loader.ts. It lives there rather than in a new module, which keeps the client/core eager graph and the packed file count inside their budgets.
    • It returns the echoed adcp_version only when the echo is release-precision, in the pin's major, and covered by a schema bundle the SDK ships (hasSchemaBundle).
    • Otherwise it returns the pin: echo missing, malformed, from another major, or not bundled. Validation stays strict, so an echo the SDK can't validate never silences the check.
    • If the echo is the pinned release, it returns the exact pin, so a patch-precise pin still applies.
    • An external schemaRoot bound to the pin wins over the echo. That matches the storyboard runner's response_schema precedence.
    • An echo older than the pin is accepted. That's the spec's downshift rule.
    • supported_versions is not consulted. With no echo, the spec says to validate against the pin.
  • Used in discoverAgentProfile (testCapabilityDiscovery, comply(), storyboard runner) and in TaskExecutor.validateResponseSchema, for get_adcp_capabilities only. responseAdcpVersionForValidation() / effectiveResponseAdcpVersion() are untouched, so the unwrapper is unaffected.
  • This also covers the capabilities case of Pick the validation line for a major-only get_adcp_capabilities response from the response itself, not a hard-coded 3.0 (follow-up to #3124, item 2) #3125 (major-only envelope, previously hard-coded to '3.0'). Pick the validation line for a major-only get_adcp_capabilities response from the response itself, not a hard-coded 3.0 (follow-up to #3124, item 2) #3125's supported_versions fallback and its all-tasks scope are not included.

#3279: cross-check comply_test_controller under compliance_testing

crossValidateProtocolsAndTools keyed COMPLIANCE_TOOLS as compliance, but parseCapabilitiesResponse reports the block as compliance_testing. Every agent that declared the block failed with Tools suggest unreported protocols: compliance (has: [comply_test_controller]).

  • The key is now compliance_testing, and the table carries satisfies Partial<Record<AdcpProtocol, …>>, so a key the parser never emits fails to compile.
  • I checked the other keys against AdcpProtocol / TOOL_PROTOCOL_MAP and none are mismatched. brand is absent from the table, which is an omission, not a false failure; left as is.
  • Behavior change: a declared compliance_testing block with no comply_test_controller tool now fails "reported but no matching tools found". The schema says the block declares the controller.

Both changesets are patch. No exports, types or wire shapes change.

Test plan

  • New test/lib/capabilities-served-release-validation.test.js (17 tests). I checked the first 14 against unpatched main: 11 fail, and the 3 that pass are the fallback guards. The 3 added after review cover prerelease and legacy pins, external-root precedence, and the major-only envelope.
    • It runs a local MCP seller that receives a major-only probe (asserted) from a 3.0 pin and echoes 3.2 with 3.2 shapes.
    • Covered: no issues and all steps pass; still strict against 3.2 for an invalid 3.2 payload; pin fallback for no echo, a 4.0 echo, and an unbundled 3.99 echo; TaskExecutor strict success and failure; cross-check pass, tool without block, and block without tool; helper unit cases.
  • npm test locally. Every non-Postgres suite passes, including the eslint-plugin and reference-renderers workspaces. The only failures are Postgres-backed tests (ECONNREFUSED, no local database). CI's Postgres-backed unit tests are green.
  • npm run typecheck, npm run lint (0 errors, no new warnings), prettier --check, ci:doc-links, ci:docs-check, build:lib

The backport to 15.9.x for latest is a separate PR.

🤖 Generated with Claude Code

…ompliance_testing in the cross-check

Capability discovery strictly validated get_adcp_capabilities against the
caller's pin. A 3.0 pin sends only adcp_major_version: 3, so the seller
serves and echoes its highest 3.x release, and the SDK reported that
release's shapes as false capabilities_schema_issues (and failed
getAdcpCapabilities() in strict mode). discoverAgentProfile and
TaskExecutor now validate against the echoed adcp_version when it is a
same-major release the SDK bundles, falling back to the pin otherwise,
per the versioning spec ("Server echoes"). Validation stays strict.

The testCapabilityDiscovery protocol/tool cross-check keyed
comply_test_controller as "compliance" while the capabilities parser
reports "compliance_testing", so every agent declaring the block failed
with "Tools suggest unreported protocols: compliance". The key is now
compliance_testing and the table's keys are type-checked against
AdcpProtocol.

Fixes #3278
Fixes #3279

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Oct 15, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean validation bug fix, no blocking or medium findings.

Checked:

  • New resolveServedValidationVersion validates get_adcp_capabilities against the seller's echoed release, wired into both TaskExecutor.ts:6131 and testing/client.ts:845.
  • Protocol/tool cross-check now keys comply_test_controller under compliance_testing (the key actually reported by parseCapabilitiesResponse) instead of the never-matching compliance, satisfies-pinned to AdcpProtocol.
  • Fail-closed design verified: served-version.ts:166 restricts the echo to a same-major, bundled, release-precision string; RELEASE_PRECISION rejects patch-precision and traversal payloads (../3.2) before reaching hasSchemaBundle/resolveSchemaRoot. Non-conforming echoes fall back to the pin with strict validation preserved — no witness/translator violation; only the validation version is affected.
  • Test coverage confirms strict validation survives (3.2-invalid response still fails, no regression to the 3.0 "must be boolean" message) and that the request is major-only.
  • Two patch changesets, correct for a validation fix that removes no exports and changes no wire shape.

No findings. gated_paths is false; high_risk is false; no no-auto-approve team match. review_decision is REVIEW_REQUIRED but row 2 does not apply (gated_paths false). Falls through to row 9 → approve.

…he echo; cover prerelease and major-only pins

Matches the storyboard runner's response_schema precedence. Documents the
deliberate downshift acceptance and tightens the cross-check changeset.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Oct 15, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

Delta since prior approve (4 code/changeset files, ~127 lines): adds an external-schema-root precedence guard to resolveServedValidationVersion (src/lib/validation/served-version.ts:51-55) with a fail-closed try/catch that falls back to the pin on throw, plus changeset wording and new test coverage. The guard is only reached for numeric pins (legacy aliases return early), external root wins over the echo, strict validation preserved — consistent with the storyboard runner's response_schema precedence. Witness-not-translator posture holds; no fabrication or re-shaping. Changesets are present and consistent with the wire impact.

No critical, high, or medium findings in this delta.

Gate checks: no no-auto-approve team match; gated_paths is false (review_decision REVIEW_REQUIRED does not force escalate because the path gate is not armed); high_risk is false. None of rows 1–8 fired → row 9 approve.

…oader

A separate module grew the @adcp/sdk/client/core eager graph past its
210-module budget and the packed file count past 6630.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — pure refactor, no findings.

resolveServedValidationVersion was moved from the deleted served-version.ts into schema-loader.ts with import-site updates in TaskExecutor.ts, testing/client.ts, and the test file. The function body is byte-identical to the prior-approved version (constant rename + merged JSDoc only). Reviewer verified no dangling served-version references, all dependencies resolve in-file with no new import cycle, the symbol remains @internal with no public-surface break, and feature changesets cover the PR.

No blocking findings; no medium findings. high_risk is false, gated_paths is false, no author team gate. Falls through to row 9 → approve. Prior decision was approve and remains valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant