Skip to content

test(adcp/v3): validate wire fixtures against their schemas - #617

Merged
torbenbrodt merged 4 commits into
adcontextprotocol:mainfrom
affinity-com:ci/v3-fixture-validation
Nov 11, 2026
Merged

torbenbrodt merged 4 commits into
adcontextprotocol:mainfrom
affinity-com:ci/v3-fixture-validation

Conversation

@torbenbrodt

Copy link
Copy Markdown
Collaborator

Check 1 of #564: wire fixtures in adcp/v3's tests are validated against their schemas in the pinned bundle, and the helper fails closed. Test code and AGENTS.md only; no exported Go changes.

What changes

Commit 1: the helper (schema_fixture_test.go).

  • The regex list is explicit, and fails closed. test(adcp/v3): validate wire fixtures against their JSON schema #587 already translates the bundle's 8 patterns that Go's RE2 can't run into exact RE2 equivalents (re2Translations, 23 locations in 11 files, listed by schema pointer), so nothing is skipped. TestTranslateSchemaFailsClosed now proves the load fails for an unlisted pattern, an entry for another location or file, a changed pattern text and a bad patternProperties key.
  • A missing bundle fails under CI. With CI set, assertSchemaValid and requireSchemaBundle fail instead of logging; locally they still log or skip. Only ci.yml's test job runs these tests, after its bundle download step.
  • Known-invalid fixtures. knownInvalidFixtures (empty in this commit; commit 2 adds its entries) records, per fixture, the owning issue, the exact error it must still produce, and a Patch that repairs that one defect. A listed fixture must fail with exactly that error and validate fully once patched, so an entry never hides a second defect, and it fails once the fixture becomes valid. The list records its bundle version (3.2.2) and fails when schemas/VERSION moves.

Commit 2: the call sites. 116 call sites in 20 files (108 assertSchemaValid, 8 through the new assertEncodingSchemaValid; the helper's own tests not counted), up from 57 in 14. Many fixtures turned out to be schema-invalid and are corrected; each expected string changed with its value. The main ones:

Old → new File
days mon, tue → monday, tuesday; reach unit user → individuals; disclosure overlay_top_left → overlay, persistent → continuous; metric roas → conversion_rate; feedback source buyer → buyer_attribution generated_core_refs_test.go
idempotency keys under 16 characters lengthened; missing required fields added (buying_mode, proposal_id, total_budget, status, cache_scope, check_id, expires_at, revision, …) generated_core_refs_test.go, inputs_test.go, responses_test.go
rating system tvpg → tv_parental; landing page required → any; special category sports → championship; readiness ready → good; forecast historical → modeled; selection signal_ids → by_id; publisher_domain, SignalID.Source and image asset fields added generated_product_refs_test.go
catalog type products → product; preview manifest format_kind; Deployment gains agent_url inputs_test.go, generated_capabilities_test.go
CreativeInput three-state localization fixtures extended to a full item creative_input_test.go

Ten fixtures stay invalid because the SDK produces them, each owned by an issue:

  • #611 (3): GetProductsResponse.Products is omitempty, so an empty products list is not sent.
  • #612 (7): response builders emit no or an empty envelope status (create_media_buy success, get_media_buys, five comply_test_controller results).

Not in this PR

  • format is not asserted by jsonschema-go, so a valid fixture can still carry a malformed date-time.
  • Fixtures left unvalidated: deliberately invalid inputs (canceled: false against const: true) and a nil optional pointer shown to be omitted say why in a comment; zero-value and error-path tests and stub payloads, for example in idempotent_test.go, optional_test.go, example_optional_test.go, addtool_test.go and validation_test.go (not touched here), are not validated and carry no comment.
  • A known-invalid entry whose test is deleted is not detected; it is clutter, not a hidden failure.

Checks run locally

go vet and go test -race ./... in adcp/v3 on Go 1.26 and 1.27 with CI=true; golangci-lint; regeneration reproduces types_gen.go; lint.py --strict; --coverage-max-unreviewed-any 0; the Python generator tests. Each new guard was broken in a copy to show its test fails, including a corrected value put back inside a known-invalid fixture.

Related: #564.

🤖 Generated with Claude Code

torbenbrodt and others added 2 commits November 11, 2026 10:29
…id list

Prove the fail-closed behaviour of the fixture helper and give the tests
a place to record fixtures that are invalid because of a Go type.

translateSchema takes the RE2 translation list as a parameter, and
TestTranslateSchemaFailsClosed shows that an unlisted pattern, an entry
for another location or file, a changed pattern and a bad
patternProperties key each fail the load, while pattern-like text in
example data is ignored.

Without the bundle the helpers now fail when CI is set instead of
leaving every fixture unchecked, and skip or log otherwise.
TestBundleMissing covers both through a recording test double.

knownInvalidFixtures records a fixture by test name and schema, with the
link to the issue that tracks the defect, the exact validation error of
the raw fixture, and a Patch that repairs that one defect. The raw
fixture must fail with exactly that error, and the patched fixture must
validate in full, so an entry never stops the rest of the fixture from
being checked. An entry fails once its fixture validates, the list is
recorded against one bundle version, and
TestKnownInvalidFixturesWellFormed fails when schemas/VERSION moves or
an entry lacks a link, an error or a patch. The list is empty here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The fixture helper checked 57 call sites in 14 test files; the other
wire fixtures went unchecked, among them the CreativeInput localization
fixtures and the capabilities, plan, response-builder and test
controller encodings. They now go through assertSchemaValid or the new
assertEncodingSchemaValid: 116 call sites in 20 files, several of them
table driven. jsonschema-go does not assert the format keyword, so a
date-time or URI value is checked for its type only.

Many of those fixtures were invalid and are corrected without weakening
an assertion: values outside the schema enum (day names, reach unit,
catalog type, content rating system, disclosure position and
persistence, landing page requirement, forecast method, selection type,
feedback source, metric type), idempotency keys under the 16 character
minimum, and missing required fields (status, cache_scope, check_id,
revision, confirmed_at, publisher_domain, agent_url, source, and the
asset_type, width and height of an image). An expected string that
named a corrected value or its field order changes with it.

Fixtures that are invalid because of a Go type stay as they are and are
listed in knownInvalidFixtures, each with its issue link, exact error
and a patch for that one defect: an empty products list that
GetProductsResponse drops, and responses that SDK builders send with no
or an empty envelope status (create_media_buy success, get_media_buys
from a list, comply_test_controller). Fixtures that are invalid on
purpose, such as canceled false, and a nil optional pointer shown to be
omitted, are not validated and say why.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@torbenbrodt
torbenbrodt marked this pull request as ready for review November 11, 2026 10:07

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — test-only PR with no blocking or medium findings.

This PR touches 11 files (all *_test.go plus AGENTS.md); no production, schema, or generated-type changes. It hardens the adcp/v3 schema-fixture helper and extends validation from 57 to 116 call sites.

Checked and clean:

  • schema_fixture_test.go known-invalid control flow fails closed; one entry can't mask a second defect; assertKnownInvalid/translateSchema fail-closed on unlisted RE2 patterns.
  • Global-map mutation in TestAssertSchemaValidReports is safe — no t.Parallel() in the package, subtests awaited serially, no race.
  • Fixture value corrections are hand-built test inputs with want strings updated in lockstep; no assertion weakened.
  • Ten remaining-invalid fixtures are pre-existing SDK wire bugs tracked to issues (#611, #612), not introduced here.
  • No touch to types_gen.go, tmproto signing, identity-agent TEE, protocol-managed skills, or any wire-shape/exported-API — changeset/minor-pathway rules don't apply.

Gate check: gated_paths is false, no high-risk flag, no no-auto-approve team match, no findings. Rows 1–8 do not fire; falls through to row 9 (approve). Note review_decision is REVIEW_REQUIRED, but row 2 only applies when gated_paths is true, which it is not.

@torbenbrodt
torbenbrodt merged commit 2c1dbef into adcontextprotocol:main Nov 11, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant