feat(conformance): Socrate fixtures for mocks and a nightly live conformance suite

.github/workflows/conformance.yml

@@ -0,0 +1,182 @@
+# Conformance: backendkit against a real Socrate (#101).
+#
+# Builds Socrate from ovander/go-oauth2 (main by default), starts it on a
+# PostgreSQL 16 service container, seeds three clients and one user per suite
+# package (conformance/testdata/socrate-seed, throwaway secrets generated in
+# the job), and runs the live tests behind the `conformance` build tag: the
+# fixtures in conformance/ against real tokens and documents, and jwtauth,
+# socrate, bff and pep against the live server. scripts/conformance-local.sh
+# does all of it, here and on a workstation alike.
+#
+# Informational: it is not a required check and never gates a backendkit pull
+# request. It runs nightly, on demand, and on pull requests that change the
+# suite itself. A failed nightly or manual run opens an issue labelled
+# `conformance` (or comments on the open one).
+name: Conformance
+
+on:
+  schedule:
+    - cron: "17 3 * * *"
+  workflow_dispatch:
+    inputs:
+      socrate_ref:
+        description: ovander/go-oauth2 ref to build (branch, tag or commit)
+        required: false
+        default: main
+  pull_request:
+    paths:
+      - .github/workflows/conformance.yml
+      - conformance/**
+      - scripts/conformance-local.sh
+      - "**/*conformance*_test.go"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: conformance-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  conformance:
+    name: Socrate conformance
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_USER: socrate
+          POSTGRES_PASSWORD: ci-only
+          POSTGRES_DB: socrate_conformance
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd "pg_isready -U socrate" --health-interval 5s
+          --health-timeout 5s --health-retries 10
+    env:
+      CONFORMANCE_DATABASE_URL: postgres://socrate:ci-only@localhost:5432/socrate_conformance?sslmode=disable
+      CONFORMANCE_WORKDIR: ${{ github.workspace }}/conformance-work
+      SOCRATE_REF: ${{ inputs.socrate_ref || 'main' }}
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          path: backendkit
+
+      - uses: actions/checkout@v4
+        with:
+          repository: ovander/go-oauth2
+          ref: ${{ env.SOCRATE_REF }}
+          path: socrate
+
+      # Each module is built with the Go its go.mod names: its toolchain line,
+      # or its go line when there is none (the rule both repositories' CI use).
+      - name: Read the Go versions from go.mod
+        id: go
+        run: |
+          gover() { awk '/^toolchain /{t=$2} /^go /{g="go"$2} END{print (t != "" ? t : g)}' "$1" | sed 's/^go//'; }
+          echo "socrate=$(gover socrate/go.mod)" >> "$GITHUB_OUTPUT"
+          echo "backendkit=$(gover backendkit/go.mod)" >> "$GITHUB_OUTPUT"
+          echo "Socrate $(git -C socrate rev-parse --short HEAD) ($SOCRATE_REF)" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Set up Go ${{ steps.go.outputs.socrate }} (Socrate)
+        uses: actions/setup-go@v5
+        with:
+          go-version: ${{ steps.go.outputs.socrate }}
+          cache: true
+          cache-dependency-path: |
+            socrate/go.sum
+            backendkit/go.sum
+
+      - name: Build Socrate
+        working-directory: socrate
+        env:
+          WANT_GO: ${{ steps.go.outputs.socrate }}
+        run: |
+          test "$(go env GOVERSION)" = "go$WANT_GO"
+          go build -o "$RUNNER_TEMP/socrate" ./cmd/server
+
+      - name: Set up Go ${{ steps.go.outputs.backendkit }} (backendkit)
+        uses: actions/setup-go@v5
+        with:
+          go-version: ${{ steps.go.outputs.backendkit }}
+          cache: false
+
+      # Boots the binary on loopback (migrating the empty database), seeds it,
+      # and runs `go test -tags conformance -run '^TestConformance' ./...`.
+      - name: Run the conformance suite
+        working-directory: backendkit
+        env:
+          SOCRATE_SRC: ${{ github.workspace }}/socrate
+          SOCRATE_BIN: ${{ runner.temp }}/socrate
+          WANT_GO: ${{ steps.go.outputs.backendkit }}
+        run: |
+          test "$(go env GOVERSION)" = "go$WANT_GO"
+          scripts/conformance-local.sh
+
+      - name: Socrate log to the job summary
+        if: failure()
+        run: |
+          {
+            echo "### Socrate server log (last 200 lines)"
+            echo '```'
+            tail -200 "$CONFORMANCE_WORKDIR/server.log" 2>/dev/null || echo "(no log: the server did not start)"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload the Socrate log
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: socrate-server-log
+          path: ${{ env.CONFORMANCE_WORKDIR }}/server.log
+          if-no-files-found: ignore
+          retention-days: 14
+
+  # Separate job so only this one holds issues: write.
+  report:
+    name: Report a failed run
+    needs: conformance
+    if: failure() && github.event_name != 'pull_request'
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+    steps:
+      - uses: actions/github-script@v7
+        env:
+          SOCRATE_REF: ${{ inputs.socrate_ref || 'main' }}
+        with:
+          script: |
+            const label = 'conformance';
+            const repo = context.repo;
+            const run = `${context.serverUrl}/${repo.owner}/${repo.repo}/actions/runs/${context.runId}`;
+            const body = [
+              `The Socrate conformance run failed (${context.eventName}, ovander/go-oauth2 \`${process.env.SOCRATE_REF}\`).`,
+              '',
+              `Run: ${run}`,
+              '',
+              'Either Socrate changed a shape backendkit or its fixtures rely on (update conformance/fixtures, ' +
+              'and the affected package, in a pull request), or the suite itself broke. ' +
+              'Reproduce with `scripts/conformance-local.sh`.',
+            ].join('\n');
+            try {
+              await github.rest.issues.getLabel({ ...repo, name: label });
+            } catch (e) {
+              if (e.status !== 404) throw e;
+              await github.rest.issues.createLabel({
+                ...repo, name: label, color: 'b60205',
+                description: 'Nightly Socrate conformance run (#101)',
+              });
+            }
+            const { data: open } = await github.rest.issues.listForRepo({
+              ...repo, state: 'open', labels: label, per_page: 1,
+            });
+            if (open.length > 0) {
+              await github.rest.issues.createComment({ ...repo, issue_number: open[0].number, body });
+              core.info(`Commented on #${open[0].number}`);
+            } else {
+              const { data: issue } = await github.rest.issues.create({
+                ...repo, title: 'Socrate conformance run failed', labels: [label], body,
+              });
+              core.info(`Opened #${issue.number}`);
+            }

CHANGELOG.md

@@ -15,6 +15,24 @@ All notable changes to backendkit are documented here. Format:
   writer, before any identity is injected; with no non-empty scope it rejects every token. A
   malformed `scope`/`scp` claim only matters with `RequireScopes` (it is then a 403): callers that do
   not require scopes see no change.
+- **`conformance`: what Socrate really issues, as fixtures for your Socrate mock, checked nightly
+  against a live Socrate** (#101). Each consumer built its own Socrate mock and found the
+  differences in production. The new package (no other backendkit import) embeds the claim sets
+  of Socrate's access, ID and refresh tokens (authorization code with and without MFA, refresh,
+  `client_credentials`, `AUDIENCE_MODE=dual`, namespaced custom claims, DPoP-bound), its token,
+  userinfo and introspection responses, and its discovery and JWKS documents, with placeholders
+  for volatile values.
+  - `Names`, `Raw`, `Load(name, Params)`, `Sign(name, Params, Key)`, `JWKS`, `Discovery`,
+    `NewKey`, and `Match(name, doc, Params)`, which checks a document against a fixture (same
+    keys, values by rule).
+  - Live tests behind the `conformance` build tag check every fixture against a running Socrate,
+    and test `jwtauth`, `socrate`, `bff` (authorize → callback → refresh → logout) and `pep`
+    against it. `go test ./...` and CI are unchanged.
+  - `scripts/conformance-local.sh` builds Socrate from a checkout, starts and seeds it on a
+    scratch database (`conformance/testdata/socrate-seed`), and runs them. The new **Conformance**
+    workflow does the same nightly, on demand, and on pull requests that change the suite. It is
+    informational, not a required check. A failed nightly or manual run opens or updates an issue
+    labelled `conformance`.
 
 ### Fixed
 - **`WithPendingLoginManagedSchema` now also checks the `UPDATE` privilege.** `Put` is an

CONTRIBUTING.md

@@ -49,6 +49,14 @@ govulncheck ./...
 ```
 
 - Tests sit next to the code (`*_test.go`), table-driven.
+- Live tests against a real Socrate sit in `*_conformance_test.go` files behind the `conformance`
+  build tag, named `TestConformance…`, and import only what their package already may. They are
+  not part of the required checks: the **Conformance** workflow runs them nightly (and on a pull
+  request that touches them) against Socrate built from `ovander/go-oauth2` `main`, and opens an
+  issue labelled `conformance` when they fail. Run them locally with
+  `scripts/conformance-local.sh` (it needs a scratch PostgreSQL database and a go-oauth2
+  checkout), and check they still build with `go vet -tags conformance ./...`. When Socrate
+  changes a shape on purpose, update `conformance/fixtures` in the same pull request.
 - Do not weaken a check to get green: no skipped or deleted tests, and no `//nolint` or `t.Skip`
   without a one-line reason.
 

README.md

@@ -31,7 +31,8 @@ Go developers writing an API or a BFF in the Socrate suite.
   [apierror](#apierror) · [ctxutil](#ctxutil) · [httpware](#httpware) ·
   [gormlogger](#gormlogger) · [socrate](#socrate) · [jwtauth](#jwtauth) · [bff](#bff) ·
   [pep](#pep) · [tiering](#tiering) · [aigateway](#aigateway) · [ailang](#ailang) ·
-  [ainarration](#ainarration) · [pagination](#pagination) · [buildinfo](#buildinfo)
+  [ainarration](#ainarration) · [pagination](#pagination) · [buildinfo](#buildinfo) ·
+  [conformance](#conformance)
 - [Environment variables](#environment-variables)
 - [Testing](#testing)
 - [Troubleshooting](#troubleshooting)
@@ -87,6 +88,7 @@ backendkit puts that foundation in one versioned dependency:
 | [`ainarration`](#ainarration) | Generic LRU+TTL narration cache and `CacheKey` helper |
 | [`pagination`](#pagination) | Query-param parsing and `PagedResponse` |
 | [`buildinfo`](#buildinfo) | Build-time version metadata (`-ldflags`) and a `/version` HTTP handler |
+| [`conformance`](#conformance) | Fixtures of what Socrate really issues (token claim sets, token, userinfo and introspection responses, discovery, JWKS) and helpers to sign Socrate-shaped tokens and check documents against them, for your Socrate mock; verified nightly against a live Socrate |
 
 `go get` pulls the whole module, but importing one package brings in only what it builds on: the
 shared `ctxutil` and `apierror` primitives, plus `socrate` for the two packages that exist to talk
@@ -114,6 +116,7 @@ to Socrate (`bff`, `pep`) and `ailang` for `aigateway`.
 | Parse `?page`/`?per_page` and return paged lists | [`pagination`](#pagination) |
 | Log GORM queries through logrus / flag slow queries | [`gormlogger`](#gormlogger) |
 | Expose build/version info on a `/version` endpoint | [`buildinfo`](#buildinfo) |
+| Test against Socrate-shaped tokens and documents instead of a hand-written mock | [`conformance`](#conformance) |
 
 ---
 
@@ -1047,6 +1050,49 @@ info := buildinfo.Get()
 
 Full API: [pkg.go.dev/…/buildinfo](https://pkg.go.dev/github.com/ovander/backendkit/buildinfo).
 
+### conformance
+
+What Socrate really issues, as fixtures a consumer's Socrate mock loads instead of hand-writing
+claims: the claim sets of its access, ID and refresh tokens (authorization code with and without
+MFA, refresh, `client_credentials`, `AUDIENCE_MODE=dual`, namespaced custom claims, DPoP-bound),
+its token, userinfo and introspection responses, and its discovery and JWKS documents. Volatile
+and deployment-specific values are placeholders (`<sub>`, `<client_id>`, `<exp>`, `<ns>tenant_id`,
+…) that `Params` fills. The package imports nothing else from backendkit.
+
+```go
+key, _ := conformance.NewKey()                     // RSA-2048, UUID kid, like Socrate's key ring
+jwks, _ := conformance.JWKS(key)                   // serve at /.well-known/jwks.json
+disc, _ := conformance.Discovery(mock.URL)         // serve at /.well-known/openid-configuration
+
+tok, _ := conformance.Sign("access/authorization_code", conformance.Params{
+    Issuer: mock.URL, ClientID: "my-app", Subject: "42", Role: "editor",
+}, key)                                            // the token Socrate would issue
+
+claims, _ := conformance.Load("access/client_credentials", conformance.Params{AppID: 12})
+// claims["sub"] == "app:12", claims["aud"] == ["my-app"], claims["scope"] == "api"
+
+err := conformance.Match("access/authorization_code", myMockClaims, conformance.Params{})
+// nil only if the document has exactly the fixture's keys and every value fits
+```
+
+| Function | Purpose |
+|---|---|
+| `Names()` | Every fixture name, e.g. `access/client_credentials_custom_claims`, `discovery` |
+| `Raw(name)` | The fixture JSON, placeholders unresolved |
+| `Load(name, Params)` | The fixture with every placeholder filled (defaults: the `Default…` constants) |
+| `Sign(name, Params, Key)` | A token fixture as an RS256 JWS under Socrate's JOSE header |
+| `JWKS(keys…)` / `Discovery(issuer)` / `NewKey()` | The documents a mock serves, and a signing key |
+| `Match(name, doc, Params)` | Check a document against a fixture: same keys, literal values equal, placeholders by rule (exactly, when the `Params` field is set) |
+
+The fixtures stay honest because a nightly workflow (`.github/workflows/conformance.yml`) builds
+Socrate from `ovander/go-oauth2` `main`, seeds it, and checks every fixture against the real
+tokens and documents; the same run tests `jwtauth`, `socrate`, `bff` and `pep` against the live
+server. Those tests sit behind the `conformance` build tag, so `go test ./...` never needs a
+server; `scripts/conformance-local.sh` runs them locally (see [Testing](#testing)). A failed
+nightly run opens an issue labelled `conformance`.
+
+Full API: [pkg.go.dev/…/conformance](https://pkg.go.dev/github.com/ovander/backendkit/conformance).
+
 ---
 
 ## Environment variables
@@ -1109,6 +1155,21 @@ takes any `bff.TokenRefresher`; `bff.Gateway.Now` fixes the clock.
 runnable `Example*` functions (visible on
 [pkg.go.dev](https://pkg.go.dev/github.com/ovander/backendkit)) that double as usage docs.
 
+When a test needs Socrate itself — a JWKS, a discovery document, a token with the claims a real
+server issues — build the mock from [`conformance`](#conformance) rather than writing claims by
+hand: the fixtures are checked against a live Socrate every night.
+
+backendkit's own live suite runs the same way locally. It wipes the given scratch database,
+builds Socrate from a checkout, starts and seeds it on loopback, and runs the tests behind the
+`conformance` build tag (they read `SOCRATE_*` variables, listed in the `conformance` package
+documentation, and fail without them):
+
+```bash
+SOCRATE_SRC=../go-oauth2 \
+CONFORMANCE_DATABASE_URL='postgres://user:pass@127.0.0.1:5432/socrate_conf?sslmode=disable' \
+scripts/conformance-local.sh
+```
+
 ---
 
 ## Troubleshooting