Repository navigation
fix(testing): validate capabilities against the served release; key compliance_testing in the cross-check - #3280
Conversation
…ompliance_testing in the cross-check
Capability discovery strictly validated get_adcp_capabilities against the
caller's pin. A 3.0 pin sends only adcp_major_version: 3, so the seller
serves and echoes its highest 3.x release, and the SDK reported that
release's shapes as false capabilities_schema_issues (and failed
getAdcpCapabilities() in strict mode). discoverAgentProfile and
TaskExecutor now validate against the echoed adcp_version when it is a
same-major release the SDK bundles, falling back to the pin otherwise,
per the versioning spec ("Server echoes"). Validation stays strict.
The testCapabilityDiscovery protocol/tool cross-check keyed
comply_test_controller as "compliance" while the capabilities parser
reports "compliance_testing", so every agent declaring the block failed
with "Tools suggest unreported protocols: compliance". The key is now
compliance_testing and the table's keys are type-checked against
AdcpProtocol.
Fixes #3278
Fixes #3279
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — clean validation bug fix, no blocking or medium findings.
Checked:
- New
resolveServedValidationVersionvalidatesget_adcp_capabilitiesagainst the seller's echoed release, wired into bothTaskExecutor.ts:6131andtesting/client.ts:845. - Protocol/tool cross-check now keys
comply_test_controllerundercompliance_testing(the key actually reported byparseCapabilitiesResponse) instead of the never-matchingcompliance,satisfies-pinned toAdcpProtocol. - Fail-closed design verified:
served-version.ts:166restricts the echo to a same-major, bundled, release-precision string;RELEASE_PRECISIONrejects patch-precision and traversal payloads (../3.2) before reachinghasSchemaBundle/resolveSchemaRoot. Non-conforming echoes fall back to the pin with strict validation preserved — no witness/translator violation; only the validation version is affected. - Test coverage confirms strict validation survives (3.2-invalid response still fails, no regression to the 3.0 "must be boolean" message) and that the request is major-only.
- Two
patchchangesets, correct for a validation fix that removes no exports and changes no wire shape.
No findings. gated_paths is false; high_risk is false; no no-auto-approve team match. review_decision is REVIEW_REQUIRED but row 2 does not apply (gated_paths false). Falls through to row 9 → approve.
…he echo; cover prerelease and major-only pins Matches the storyboard runner's response_schema precedence. Documents the deliberate downshift acceptance and tightens the cross-check changeset. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Approve
Approve.
Delta since prior approve (4 code/changeset files, ~127 lines): adds an external-schema-root precedence guard to resolveServedValidationVersion (src/lib/validation/served-version.ts:51-55) with a fail-closed try/catch that falls back to the pin on throw, plus changeset wording and new test coverage. The guard is only reached for numeric pins (legacy aliases return early), external root wins over the echo, strict validation preserved — consistent with the storyboard runner's response_schema precedence. Witness-not-translator posture holds; no fabrication or re-shaping. Changesets are present and consistent with the wire impact.
No critical, high, or medium findings in this delta.
Gate checks: no no-auto-approve team match; gated_paths is false (review_decision REVIEW_REQUIRED does not force escalate because the path gate is not armed); high_risk is false. None of rows 1–8 fired → row 9 approve.
…oader A separate module grew the @adcp/sdk/client/core eager graph past its 210-module budget and the packed file count past 6630. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — pure refactor, no findings.
resolveServedValidationVersion was moved from the deleted served-version.ts into schema-loader.ts with import-site updates in TaskExecutor.ts, testing/client.ts, and the test file. The function body is byte-identical to the prior-approved version (constant rename + merged JSDoc only). Reviewer verified no dangling served-version references, all dependencies resolve in-file with no new import cycle, the symbol remains @internal with no public-surface break, and feature changesets cover the PR.
No blocking findings; no medium findings. high_risk is false, gated_paths is false, no author team gate. Falls through to row 9 → approve. Prior decision was approve and remains valid.
Summary
Fixes two capability-discovery bugs that make conformant agents fail
testCapabilityDiscovery. Both were found while fixing the AdCP repo's external smoke check (adcontextprotocol/adcp#8208), which carries workarounds for them.#3278: validate
get_adcp_capabilitiesagainst the served release, not the pinA 3.0 pin can't carry
adcp_version, sobuildVersionEnvelopesends onlyadcp_major_version: 3. Per the versioning spec, the seller then serves its highest 3.x release and echoes it inadcp_version. Buyers SHOULD validate against that release, not their pin ("Server echoes",docs/reference/versioning.mdx).discoverAgentProfileandTaskExecutor.validateResponseSchemaboth validated strictly against the pin. A 3.2 response from the reference agent therefore produced falsecapabilities_schema_issues, andgetAdcpCapabilities()failed in strict mode. The two false issues weremedia_buy.features.bidding_policy must be booleanand the closed 3.0compliance_testing.scenariosenum.resolveServedValidationVersion(response, pin)insrc/lib/validation/schema-loader.ts. It lives there rather than in a new module, which keeps theclient/coreeager graph and the packed file count inside their budgets.adcp_versiononly when the echo is release-precision, in the pin's major, and covered by a schema bundle the SDK ships (hasSchemaBundle).schemaRootbound to the pin wins over the echo. That matches the storyboard runner'sresponse_schemaprecedence.supported_versionsis not consulted. With no echo, the spec says to validate against the pin.discoverAgentProfile(testCapabilityDiscovery,comply(), storyboard runner) and inTaskExecutor.validateResponseSchema, forget_adcp_capabilitiesonly.responseAdcpVersionForValidation()/effectiveResponseAdcpVersion()are untouched, so the unwrapper is unaffected.'3.0'). Pick the validation line for a major-only get_adcp_capabilities response from the response itself, not a hard-coded 3.0 (follow-up to #3124, item 2) #3125'ssupported_versionsfallback and its all-tasks scope are not included.#3279: cross-check
comply_test_controllerundercompliance_testingcrossValidateProtocolsAndToolskeyedCOMPLIANCE_TOOLSascompliance, butparseCapabilitiesResponsereports the block ascompliance_testing. Every agent that declared the block failed withTools suggest unreported protocols: compliance (has: [comply_test_controller]).compliance_testing, and the table carriessatisfies Partial<Record<AdcpProtocol, …>>, so a key the parser never emits fails to compile.AdcpProtocol/TOOL_PROTOCOL_MAPand none are mismatched.brandis absent from the table, which is an omission, not a false failure; left as is.compliance_testingblock with nocomply_test_controllertool now fails "reported but no matching tools found". The schema says the block declares the controller.Both changesets are
patch. No exports, types or wire shapes change.Test plan
test/lib/capabilities-served-release-validation.test.js(17 tests). I checked the first 14 against unpatchedmain: 11 fail, and the 3 that pass are the fallback guards. The 3 added after review cover prerelease and legacy pins, external-root precedence, and the major-only envelope.TaskExecutorstrict success and failure; cross-check pass, tool without block, and block without tool; helper unit cases.npm testlocally. Every non-Postgres suite passes, including the eslint-plugin and reference-renderers workspaces. The only failures are Postgres-backed tests (ECONNREFUSED, no local database). CI's Postgres-backed unit tests are green.npm run typecheck,npm run lint(0 errors, no new warnings),prettier --check,ci:doc-links,ci:docs-check,build:libThe backport to 15.9.x for
latestis a separate PR.🤖 Generated with Claude Code