Skip to content

Sign-in stays broken until restart if Ministry Platform blips during the one OIDC discovery fetch #101

Closed

Description

@jonnydcakes

Hi Chris,

Thanks for turning #95–#97 around so quickly. This is the follow-up #87 called "worth its own ticket", which the downstream playbook lists under Known-open items. You deferred dropping discoveryUrl because it would lose JWKS verification of the id_token, so both fixes below keep JWKS.

Our experience

We run four production apps built on MPNext at The Moody Church, and Ministry Platform does blip for us:

  • When several of our apps restart together, their startup cache warming hits MP at the same moment and MP returns connect timeouts. We stagger deploys because of it.
  • Earlier this year, bursts of about 50 simultaneous requests to MP caused connect timeouts in one of our apps (the mapWithConcurrency cap came out of that).

On better-auth 1.6 a blip like that healed itself, because 1.6 fetched discovery again on each sign-in. While planning our move to 1.7 we found that the same blip, if it lands on 1.7's one discovery fetch, takes sign-in down until the app is restarted, even after MP is back. Nothing obvious flags it: /signin still returns 200, so an uptime check stays green, and the only trace is a single log line at the moment it failed. You hit a transient discovery failure yourself during #87.

We sidestepped it in our apps by not using discovery (see the end), so we have not had this outage in production. But the blips that trigger it are ones we see.

What users see

  • /signin loads and shows "Redirecting to sign in..." with a spinner that never finishes. There is no error message.
  • The page's POST /api/auth/sign-in/social gets 404 {"message":"Provider not found","code":"PROVIDER_NOT_FOUND"}.
  • The server logs once: ERROR [Better Auth]: Provider "ministry-platform": discovery left no usable authorization endpoint or token exchange. Provider skipped.
  • It stays that way after MP recovers. Only restarting the app fixes it.
  • If MP accepts the connection but never answers, the first request waits about 300 s (undici's default headers timeout), and every page that checks the session hangs during that window. Then it ends in the same permanent 404.

We checked the failure and the non-recovery on 6c0c46b with a production build (next build + next start), with curl and in Chromium, against the fake MP below. We measured the ~300 s hang (301 s) on ea2e0ad; nothing in #96–#98 touches that path. No real Ministry Platform was involved.

How to reproduce (about 5 minutes, no MP needed)

On main (6c0c46b), in a checkout with npm ci done:

1. Save this as fake-mp.mjs somewhere outside the repo. It serves the discovery document and can be switched to failing or hanging:

// fake-mp.mjs: stands in for Ministry Platform's OIDC discovery endpoint.
// Switch behaviour at runtime: curl "http://127.0.0.1:4000/__mode?m=ok|fail|hang"
import http from "node:http";

const BASE = "http://127.0.0.1:4000";
let mode = "ok";
let discoveryRequests = 0;

http.createServer((req, res) => {
  const { pathname, searchParams } = new URL(req.url, BASE);
  if (pathname === "/__mode") { mode = searchParams.get("m"); return res.end(`mode=${mode}\n`); }
  if (pathname === "/__stats") return res.end(JSON.stringify({ mode, discoveryRequests }) + "\n");
  if (pathname === "/oauth/.well-known/openid-configuration") {
    discoveryRequests++;
    if (mode === "hang") return; // accept the connection, never answer
    if (mode === "fail") { res.statusCode = 500; return res.end("error"); }
    res.setHeader("content-type", "application/json");
    return res.end(JSON.stringify({
      issuer: `${BASE}/oauth`,
      authorization_endpoint: `${BASE}/oauth/connect/authorize`,
      token_endpoint: `${BASE}/oauth/connect/token`,
      userinfo_endpoint: `${BASE}/oauth/connect/userinfo`,
      end_session_endpoint: `${BASE}/oauth/connect/endsession`,
      jwks_uri: `${BASE}/oauth/.well-known/openid-configuration/jwks`,
      id_token_signing_alg_values_supported: ["RS256"],
    }));
  }
  res.statusCode = 404;
  res.end();
}).listen(4000, "127.0.0.1", () => console.log(`fake MP on ${BASE}`));

2. Create .env.local. Generate BETTER_AUTH_SECRET with openssl rand -base64 32; everything else is a dummy value.

MINISTRY_PLATFORM_BASE_URL=http://127.0.0.1:4000
MINISTRY_PLATFORM_CLIENT_ID=fake
MINISTRY_PLATFORM_CLIENT_SECRET=fake
OIDC_CLIENT_ID=fake
OIDC_CLIENT_SECRET=fake
BETTER_AUTH_URL=http://localhost:3000
BETTER_AUTH_SECRET=<output of openssl rand -base64 32>
MP_SECURITY_ROLES=Test
NEXT_PUBLIC_MINISTRY_PLATFORM_FILE_URL=http://127.0.0.1:4000/files
NEXT_PUBLIC_APP_NAME=repro

3. Start the fake MP (healthy) and build. The build fetches discovery too, so let it succeed:

node fake-mp.mjs      # terminal 1, leave running
npm run build         # terminal 2

4. Make MP fail, then start the app:

curl "http://127.0.0.1:4000/__mode?m=fail"
npm run start
curl http://127.0.0.1:4000/__stats   # note discoveryRequests (the build's fetches); starting the app does not change it

5. Open http://localhost:3000/signin. It stays on "Redirecting to sign in...". The request the page makes, with curl:

curl -i -X POST http://localhost:3000/api/auth/sign-in/social \
  -H 'Content-Type: application/json' -H 'Origin: http://localhost:3000' \
  -d '{"provider":"ministry-platform","callbackURL":"/"}'
# HTTP/1.1 404 NOT_FOUND
# {"message":"Provider not found","code":"PROVIDER_NOT_FOUND"}

discoveryRequests has gone up by exactly one, and the server log shows Provider skipped.

6. Bring MP back and try again:

curl "http://127.0.0.1:4000/__mode?m=ok"
# repeat the curl from step 5, or reload /signin: still 404, still spinning
curl http://127.0.0.1:4000/__stats   # unchanged: discovery is never requested again

7. Restart npm run start. The same curl now returns 200 with an authorize URL.

To see the hang instead, use ?m=hang in step 4: the first sign-in, and any page render that checks the session, waits about 300 s and then fails the same way. /sign-in/social is rate-limited, so leave a few seconds between attempts.

Why it happens

This is better-auth 1.7.4. The genericOAuth plugin (dist/plugins/generic-oauth/index.mjs) is byte-identical in 1.7.5 and 1.7.6, and both behave the same.

  • genericOAuth fetches the discovery document inside the plugin's init, once, with no timeout, retry or re-fetch.
  • Every auth request, and every in-process auth.api.* call, waits for that init.
  • If the fetch fails, the provider is dropped for the life of the auth instance.
  • The fetch starts when betterAuth() is constructed, i.e. when src/lib/auth.ts is first evaluated. Under next start that is the first request that loads it, so nothing is fetched at boot. Since fix(auth): share one auth instance per process (logout id_token_hint + sign-out revocation) #97 every bundle layer shares that one instance, so a single failed fetch takes out sign-in and every page's session check for the whole process.

The library side is tracked in better-auth/better-auth#11404 and #10999.

Ways to address it

Both keep JWKS verification.

  1. Explicit endpoints, and verify the id_token in getUserInfo.
    • Replace discoveryUrl with authorizationUrl, tokenUrl and userInfoUrl (the discovery document's own values), plus accountSubject: ({ profile }) => profile.sub ?? "".
    • In your getUserInfo, verify tokens.idToken with jose before the sub cross-check: createRemoteJWKSet on MP's JWKS URL, then jwtVerify with issuer, audience and RS256.
    • Also set endSessionEndpoint to ${mpBaseUrl}/oauth/connect/endsession. Without discovery, better-auth otherwise has no provider-logout URL, and the id_token_hint that Security review 2026-09-28: session lifetime, fail-closed roles, sign-in DoS cap, logout, rate-limit IP #96's handleSignOut reads from it disappears. With it, your auth.shared-instance.test.ts passes 5/5; without it, its id_token_hint test fails.
    • We tried this in a scratch copy: it started with MP down, signed in without a restart once MP came back, and refused a bad signature.
    • It also leaves the provider with no id-token config, so better-auth's own /sign-in/social handler refuses the idToken branch (ID_TOKEN_NOT_SUPPORTED). Your route filter and hooks.before guard still answer first; better-auth's refusal is a backstop if the hook is ever removed.
    • Cost: jose becomes a direct dependency. Its key set loads lazily, with a 5 s timeout.
  2. Keep discoveryUrl, but add a timeout and retries.
    • Wrap the genericOAuth plugin object's init with a per-attempt timeout and a retry with a short delay.
    • We tested it on 1.7.4 in a standalone script rather than in MPNext itself: a bare betterAuth + genericOAuth with your discovery settings, against a local fake MP, with a 1 s per-attempt timeout and a 200 ms × attempt delay. Attempt 1 hung, attempt 2 got a 500, and attempt 3 registered the provider, JWKS config included, after about 1.6 s. 1.7.5 and 1.7.6 behave the same.
    • It depends on the plugin's init shape, and it still gives up if MP stays down past the retry budget. The timed-out fetch isn't aborted, so when it finally fails it logs a stray Provider skipped, though the registered provider is unaffected.

For reference, our four apps use explicit endpoints (no discoveryUrl) and don't verify the id_token against JWKS. MPNext on 1.6 didn't verify it either: it set discoveryUrl, but 1.6's genericOAuth never checked the id_token signature. In the code flow the id_token comes straight from MP's token endpoint over TLS, and OIDC Core §3.1.3.7 allows TLS server validation in place of the signature check in that case. You've already weighed that trade-off, so we're not pushing it.

Two small doc fixes

  • .claude/references/auth.md tells readers to look for discovery returned no valid data, but no release since 1.7.3 contains that string. It was the error 1.7.0–1.7.2 threw from init when no accountIssuer was set; 1.7.3 and later log the Provider skipped line above. The same passage says sign-in is broken "until discovery returns", but it stays broken until the process restarts.
  • The "inverted failure mode" Known-open note can't happen with a discoveryUrl-only config on 1.7.3 or later: a discovery fetch that fails (refused, non-2xx, or a non-JSON body) always ends in PROVIDER_NOT_FOUND, never in an unverified sign-in.

Happy to build it

If you'd like us to, I can build whichever option you prefer and submit a PR with tests, including a regression test that fails discovery at startup and checks that sign-in recovers without a restart.

Jonathon Huff, The Moody Church

Activity

  1. chriskehayias commented on Sep 29, 2026

    @chriskehayias
    Contributor

    So this was partially addressed - I have claude chewing on it now for full alignment.

  2. chriskehayias commented on Sep 29, 2026

    @chriskehayias
    Contributor

    Thanks, Jonathon. This is an excellent report: the repro, the root-cause analysis and both options were spot on, and it saved a lot of time.

    Where things stood. Your repro is against 6c0c46b. A partial fix (5e1d2cc, merged in #99 shortly after that commit) wrapped the auth instance in a selfHealingAuth facade. It rebuilt the instance on the next sign-in after a 30 s cooldown, which fixed "broken until restart" for a failed fetch. It did not fix the hang you described. The facade waited on the first instance's $context with no timeout, so a hung MP still stalled the first sign-in and every session check for about 300 s. It also still depended on better-auth internals. So your ticket stood.

    What we did: your option 1, with one correction. PR #107 (fixes this issue):

    • No discoveryUrl. The provider is configured with authorizationUrl, tokenUrl and endSessionEndpoint (you were right that the id_token_hint depends on it), and accountSubject reads profile.sub. Building the auth instance, next build, cold starts and session checks now make no MP call at all, so the provider always registers.

    • The id_token is still verified against MP's JWKS, now in our getUserInfo, before the sub binding. It uses the same jose jwtVerify better-auth used (signature, iss, aud, exp), and RS256 is pinned.

    • The correction: the JWKS URL isn't pinned. Your fake MP serves it at /oauth/.well-known/openid-configuration/jwks, the IdentityServer default. A live MP discovery document we checked lists /oauth/.well-known/jwks. Hardcoding either path could break sign-in on some tenant. So issuer and jwks_uri still come from discovery, but it's fetched lazily, at the first callback that needs it:

      • 5 s timeout;
      • one fetch shared by concurrent callbacks;
      • cached on success;
      • never cached on failure.

      A blip now refuses only the sign-ins that land during it (/auth-error, "please try again"), and the next attempt works on the same process. jose likewise doesn't cache a failed JWKS fetch, and there's a test for that.

    • F12 side effect, as you predicted. With no id_token config, better-auth refuses the idToken branch itself (ID_TOKEN_NOT_SUPPORTED). We kept the route filter and hooks.before guard as the primary controls.

    • selfHealingAuth has been removed.

    • jose is now a direct dependency.

    We checked MP's discovery document for authorization_response_iss_parameter_supported. It isn't advertised, so losing better-auth's RFC 9207 callback iss check costs nothing.

    Verified with your repro, adapted to https (env.ts now requires it for the MP URL in production), against next build + next start:

    • Fake MP in fail mode before start: POST /api/auth/sign-in/social returns 200 with an authorize URL, and discoveryRequests stays at 0.
    • Fake MP in hang mode on a fresh process: /api/auth/get-session, /sign-in/social, /signin and / all answer in 10–40 ms, where you measured about 301 s.

    There's a new regression suite (src/auth.oidc-discovery.test.ts) covering:

    • MP rejecting, returning 500 or hanging at boot, with no MP call made;
    • discovery or JWKS failing at the callback, then recovering on the same instance;
    • the timeout and redirect refusal;
    • single-flight fetching and caching;
    • fail-closed handling of invalid discovery documents.

    Re-adding discoveryUrl fails 11 of its 23 tests.

    Your two doc fixes. discovery returned no valid data and "until discovery returns" were already gone by #99's docs sweep. The references and downstream playbooks now describe the new behaviour. The "Dropping discoveryUrl … deferred" and F12-residue Known-open notes are marked done, with a porting section for forks. That includes what to delete if a fork already took selfHealingAuth.

    Thanks again for the offer to build it. We went ahead since it touched the security suites anyway, but a review of the PR from you would be very welcome, especially against your four apps' setup.

  3. added a commit that references this issue on Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions