Skip to content

feat(conformance): Socrate fixtures for mocks and a nightly live conformance suite - #119

Merged
ovander merged 4 commits into
mainfrom
feat/conformance-suite
Oct 11, 2026
Merged

ovander merged 4 commits into
mainfrom
feat/conformance-suite

Conversation

@ovander

@ovander ovander commented Oct 10, 2026

Copy link
Copy Markdown
Owner

What and why

Closes #101. Each consumer has built its own Socrate mock and found the differences in production (GPWA, then Lakebridge with test/socrate-mock). This PR adds two things:

1. A conformance package. It imports no other backendkit package, so there is no new import edge. It embeds 25 fixtures of what Socrate really issues:

  • access tokens from the authorization code flow, with and without MFA; refresh; dual audience; custom claims; client_credentials, plain, dual-audience, with custom claims and DPoP-bound;
  • ID tokens, the refresh token, token responses, userinfo, introspection (active, client_credentials, inactive);
  • discovery (with and without DPoP), the JWKS and the JOSE header.

Volatile values are placeholders documented in the package. The API:

func Names() []string
func Raw(name string) ([]byte, error)
func Load(name string, p Params) (map[string]any, error)    // a fixture with p filled in
func Sign(name string, p Params, key Key) (string, error)   // a Socrate-shaped RS256 token for a mock
func JWKS(keys ...Key) (map[string]any, error)
func Discovery(issuer string) (map[string]any, error)
func Match(name string, got any, p Params) error            // fails on a missing or extra key; never prints a token
func NewKey() (Key, error)

A consumer's mock loads or signs these instead of hand-writing claims. That's the "Done when" of #101: Lakebridge can drop test/socrate-mock.

2. A live suite behind the conformance build tag. Plain go test ./... and the required CI jobs are unaffected. Each package tests itself against a real Socrate, importing only what it already may:

  • conformance: every fixture against live tokens and documents. This covers the hosted sign-in, consent and code flow, refresh, userinfo, introspection, a TOTP enrolment followed by an MFA sign-in, client_credentials, a DPoP-bound token, and the dual-audience and claim-mapped clients.
  • jwtauth: real tokens against the live JWKS. It rejects a wrong audience, a wrong issuer and a tampered signature, and checks dual audience for user and service tokens and the namespaced tenant claim.
  • socrate: ServiceToken, IntrospectToken, GetCurrentUserProfile, GetProfile, GetUserAsService, RefreshToken, RevokeToken and Logout.
  • bff: a minimal BFF runs login, hosted sign-in and callback (a replay is refused), then the proxy (401 without a session, 403 without CSRF), refresh rotation and logout. A reused refresh token is a fatal refresh error.
  • pep: real Decide answers (deny, no_applicable_rule, pep_mode_accepted), and the Enforcer in off mode and with MinimumMode enforce.

.github/workflows/conformance.yml:

  • Triggers: nightly (17 3 * * *), workflow_dispatch with a socrate_ref input, and pull requests that change the suite. It is informational and not a required check.
  • What it runs: it builds Socrate from ovander/go-oauth2 (main by default) with that repo's Go version, on a Postgres 16 service, then runs scripts/conformance-local.sh. The script boots Socrate on loopback, seeds three clients and one user per package with secrets generated per run, and runs go test -tags conformance -run '^TestConformance' ./....
  • On failure: the server log goes to the job summary and an artifact. A separate report job, the only one with issues: write, opens or comments on a conformance issue. It never reports for pull requests.

How it was tested

  • Live suite run end to end locally against Socrate built from go-oauth2 main (dcf8f01): 16 tests and 6 subtests pass, 0 fail.

  • A deliberately broken fixture made the live check fail, as it should.

  • All fixtures are checked live. The one exception is the DPoP-off discovery, which a unit test pins to the live-checked discovery_dpop minus its DPoP key.

  • This PR changes the suite, so it also triggers the workflow, which is the first run on GitHub.

  • go mod tidy && git diff --exit-code go.sum leaves go.sum unchanged

  • go build ./... passes

  • go vet ./... passes, also with -tags conformance

  • go test -race -count=1 -timeout=120s ./... passes

  • golangci-lint run ./... (v2.14.0) reports no issue, with and without --build-tags conformance

  • govulncheck ./...: not run locally (vuln.go.dev is blocked here); CI runs it. No dependency change.

  • actionlint passes on the workflow

  • A line is added under ## [Unreleased] in CHANGELOG.md

Compatibility

  • Exported-API change: yes, additive only, in a new conformance package; no other package changes.
  • Behaviour change for existing callers: none.
  • Breaking change: none.
  • Risk: the seed program uses go-oauth2's internal model, so an upstream model change fails the nightly run. That's on purpose, and it shows up as a conformance issue rather than in production.
  • Noticed, not changed: socrate.IntrospectResponse.Role is a field Socrate never sends. That would be a separate change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GKRxaeYxyDhmt42cehLsGA


Generated by Claude Code

…suite

Each consumer wrote its own Socrate mock and found the differences in
production (#101). The new conformance package embeds what Socrate really
issues: the claim sets of its access, ID and refresh tokens (authorization
code with and without MFA, refresh, client_credentials, AUDIENCE_MODE=dual,
namespaced custom claims, DPoP-bound), its token, userinfo and
introspection responses, and its discovery and JWKS documents, with
placeholders for volatile values. Names, Raw, Load, Sign, JWKS, Discovery,
NewKey and Match let a mock serve and sign Socrate-shaped documents and
check its own output. The package imports nothing else from backendkit.

Live tests behind the conformance build tag match every fixture against a
running Socrate (hosted sign-in and consent, TOTP enrolment, a DPoP proof)
and test jwtauth, socrate, bff (authorize, callback, refresh, logout) and
pep against it, each from its own package's tests. go test ./... is
unchanged. scripts/conformance-local.sh builds Socrate from a checkout,
starts it on a scratch database, seeds it with
conformance/testdata/socrate-seed (throwaway secrets) and runs them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRxaeYxyDhmt42cehLsGA
Builds Socrate from ovander/go-oauth2 (main, or a ref given on dispatch)
with the Go its go.mod names, starts it on a PostgreSQL 16 service, seeds
it and runs the conformance suite through scripts/conformance-local.sh.
Runs nightly, on demand, and on pull requests that change the suite; it is
informational and not a required check. A failed nightly or manual run
opens or updates an issue labelled conformance from a separate job, the
only one with issues: write; the server log goes to the job summary and an
artifact.

Refs #101

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRxaeYxyDhmt42cehLsGA
Resolve the CHANGELOG conflict with #117: keep both Unreleased entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRxaeYxyDhmt42cehLsGA
Resolve the CHANGELOG conflict with #118: the conformance entry stays
under Added, the pending-login fix under Fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRxaeYxyDhmt42cehLsGA
@ovander
ovander merged commit d6d1f7d into main Oct 11, 2026
10 checks passed
ovander pushed a commit that referenced this pull request Oct 11, 2026
Moves [Unreleased] under [1.26.0] - 2026-10-11 (#117 scopes, #118 pending-login UPDATE check, #119 conformance suite), updates the compare links and the README install snippet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRxaeYxyDhmt42cehLsGA
@ovander ovander mentioned this pull request Oct 11, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

conformance: a suite run nightly against a Socrate built from go-oauth2, exporting claim fixtures

2 participants