Repository navigation
Sign-in stays broken until restart if Ministry Platform blips during the one OIDC discovery fetch #101
Description
Hi Chris,
Thanks for turning #95–#97 around so quickly. This is the follow-up #87 called "worth its own ticket", which the downstream playbook lists under Known-open items. You deferred dropping discoveryUrl because it would lose JWKS verification of the id_token, so both fixes below keep JWKS.
Our experience
We run four production apps built on MPNext at The Moody Church, and Ministry Platform does blip for us:
- When several of our apps restart together, their startup cache warming hits MP at the same moment and MP returns connect timeouts. We stagger deploys because of it.
- Earlier this year, bursts of about 50 simultaneous requests to MP caused connect timeouts in one of our apps (the
mapWithConcurrencycap came out of that).
On better-auth 1.6 a blip like that healed itself, because 1.6 fetched discovery again on each sign-in. While planning our move to 1.7 we found that the same blip, if it lands on 1.7's one discovery fetch, takes sign-in down until the app is restarted, even after MP is back. Nothing obvious flags it: /signin still returns 200, so an uptime check stays green, and the only trace is a single log line at the moment it failed. You hit a transient discovery failure yourself during #87.
We sidestepped it in our apps by not using discovery (see the end), so we have not had this outage in production. But the blips that trigger it are ones we see.
What users see
/signinloads and shows "Redirecting to sign in..." with a spinner that never finishes. There is no error message.- The page's
POST /api/auth/sign-in/socialgets404 {"message":"Provider not found","code":"PROVIDER_NOT_FOUND"}. - The server logs once:
ERROR [Better Auth]: Provider "ministry-platform": discovery left no usable authorization endpoint or token exchange. Provider skipped. - It stays that way after MP recovers. Only restarting the app fixes it.
- If MP accepts the connection but never answers, the first request waits about 300 s (undici's default headers timeout), and every page that checks the session hangs during that window. Then it ends in the same permanent 404.
We checked the failure and the non-recovery on 6c0c46b with a production build (next build + next start), with curl and in Chromium, against the fake MP below. We measured the ~300 s hang (301 s) on ea2e0ad; nothing in #96–#98 touches that path. No real Ministry Platform was involved.
How to reproduce (about 5 minutes, no MP needed)
On main (6c0c46b), in a checkout with npm ci done:
1. Save this as fake-mp.mjs somewhere outside the repo. It serves the discovery document and can be switched to failing or hanging:
// fake-mp.mjs: stands in for Ministry Platform's OIDC discovery endpoint.
// Switch behaviour at runtime: curl "http://127.0.0.1:4000/__mode?m=ok|fail|hang"
import http from "node:http";
const BASE = "http://127.0.0.1:4000";
let mode = "ok";
let discoveryRequests = 0;
http.createServer((req, res) => {
const { pathname, searchParams } = new URL(req.url, BASE);
if (pathname === "/__mode") { mode = searchParams.get("m"); return res.end(`mode=${mode}\n`); }
if (pathname === "/__stats") return res.end(JSON.stringify({ mode, discoveryRequests }) + "\n");
if (pathname === "/oauth/.well-known/openid-configuration") {
discoveryRequests++;
if (mode === "hang") return; // accept the connection, never answer
if (mode === "fail") { res.statusCode = 500; return res.end("error"); }
res.setHeader("content-type", "application/json");
return res.end(JSON.stringify({
issuer: `${BASE}/oauth`,
authorization_endpoint: `${BASE}/oauth/connect/authorize`,
token_endpoint: `${BASE}/oauth/connect/token`,
userinfo_endpoint: `${BASE}/oauth/connect/userinfo`,
end_session_endpoint: `${BASE}/oauth/connect/endsession`,
jwks_uri: `${BASE}/oauth/.well-known/openid-configuration/jwks`,
id_token_signing_alg_values_supported: ["RS256"],
}));
}
res.statusCode = 404;
res.end();
}).listen(4000, "127.0.0.1", () => console.log(`fake MP on ${BASE}`));2. Create .env.local. Generate BETTER_AUTH_SECRET with openssl rand -base64 32; everything else is a dummy value.
MINISTRY_PLATFORM_BASE_URL=http://127.0.0.1:4000
MINISTRY_PLATFORM_CLIENT_ID=fake
MINISTRY_PLATFORM_CLIENT_SECRET=fake
OIDC_CLIENT_ID=fake
OIDC_CLIENT_SECRET=fake
BETTER_AUTH_URL=http://localhost:3000
BETTER_AUTH_SECRET=<output of openssl rand -base64 32>
MP_SECURITY_ROLES=Test
NEXT_PUBLIC_MINISTRY_PLATFORM_FILE_URL=http://127.0.0.1:4000/files
NEXT_PUBLIC_APP_NAME=repro
3. Start the fake MP (healthy) and build. The build fetches discovery too, so let it succeed:
node fake-mp.mjs # terminal 1, leave running
npm run build # terminal 24. Make MP fail, then start the app:
curl "http://127.0.0.1:4000/__mode?m=fail"
npm run start
curl http://127.0.0.1:4000/__stats # note discoveryRequests (the build's fetches); starting the app does not change it5. Open http://localhost:3000/signin. It stays on "Redirecting to sign in...". The request the page makes, with curl:
curl -i -X POST http://localhost:3000/api/auth/sign-in/social \
-H 'Content-Type: application/json' -H 'Origin: http://localhost:3000' \
-d '{"provider":"ministry-platform","callbackURL":"/"}'
# HTTP/1.1 404 NOT_FOUND
# {"message":"Provider not found","code":"PROVIDER_NOT_FOUND"}discoveryRequests has gone up by exactly one, and the server log shows Provider skipped.
6. Bring MP back and try again:
curl "http://127.0.0.1:4000/__mode?m=ok"
# repeat the curl from step 5, or reload /signin: still 404, still spinning
curl http://127.0.0.1:4000/__stats # unchanged: discovery is never requested again7. Restart npm run start. The same curl now returns 200 with an authorize URL.
To see the hang instead, use ?m=hang in step 4: the first sign-in, and any page render that checks the session, waits about 300 s and then fails the same way. /sign-in/social is rate-limited, so leave a few seconds between attempts.
Why it happens
This is better-auth 1.7.4. The genericOAuth plugin (dist/plugins/generic-oauth/index.mjs) is byte-identical in 1.7.5 and 1.7.6, and both behave the same.
- genericOAuth fetches the discovery document inside the plugin's
init, once, with no timeout, retry or re-fetch. - Every auth request, and every in-process
auth.api.*call, waits for thatinit. - If the fetch fails, the provider is dropped for the life of the auth instance.
- The fetch starts when
betterAuth()is constructed, i.e. whensrc/lib/auth.tsis first evaluated. Undernext startthat is the first request that loads it, so nothing is fetched at boot. Since fix(auth): share one auth instance per process (logout id_token_hint + sign-out revocation) #97 every bundle layer shares that one instance, so a single failed fetch takes out sign-in and every page's session check for the whole process.
The library side is tracked in better-auth/better-auth#11404 and #10999.
Ways to address it
Both keep JWKS verification.
- Explicit endpoints, and verify the id_token in
getUserInfo.- Replace
discoveryUrlwithauthorizationUrl,tokenUrlanduserInfoUrl(the discovery document's own values), plusaccountSubject: ({ profile }) => profile.sub ?? "". - In your
getUserInfo, verifytokens.idTokenwithjosebefore thesubcross-check:createRemoteJWKSeton MP's JWKS URL, thenjwtVerifywith issuer, audience and RS256. - Also set
endSessionEndpointto${mpBaseUrl}/oauth/connect/endsession. Without discovery, better-auth otherwise has no provider-logout URL, and theid_token_hintthat Security review 2026-09-28: session lifetime, fail-closed roles, sign-in DoS cap, logout, rate-limit IP #96'shandleSignOutreads from it disappears. With it, yourauth.shared-instance.test.tspasses 5/5; without it, itsid_token_hinttest fails. - We tried this in a scratch copy: it started with MP down, signed in without a restart once MP came back, and refused a bad signature.
- It also leaves the provider with no id-token config, so better-auth's own
/sign-in/socialhandler refuses the idToken branch (ID_TOKEN_NOT_SUPPORTED). Your route filter andhooks.beforeguard still answer first; better-auth's refusal is a backstop if the hook is ever removed. - Cost:
josebecomes a direct dependency. Its key set loads lazily, with a 5 s timeout.
- Replace
- Keep
discoveryUrl, but add a timeout and retries.- Wrap the genericOAuth plugin object's
initwith a per-attempt timeout and a retry with a short delay. - We tested it on 1.7.4 in a standalone script rather than in MPNext itself: a bare
betterAuth+genericOAuthwith your discovery settings, against a local fake MP, with a 1 s per-attempt timeout and a 200 ms × attempt delay. Attempt 1 hung, attempt 2 got a 500, and attempt 3 registered the provider, JWKS config included, after about 1.6 s. 1.7.5 and 1.7.6 behave the same. - It depends on the plugin's
initshape, and it still gives up if MP stays down past the retry budget. The timed-out fetch isn't aborted, so when it finally fails it logs a strayProvider skipped, though the registered provider is unaffected.
- Wrap the genericOAuth plugin object's
For reference, our four apps use explicit endpoints (no discoveryUrl) and don't verify the id_token against JWKS. MPNext on 1.6 didn't verify it either: it set discoveryUrl, but 1.6's genericOAuth never checked the id_token signature. In the code flow the id_token comes straight from MP's token endpoint over TLS, and OIDC Core §3.1.3.7 allows TLS server validation in place of the signature check in that case. You've already weighed that trade-off, so we're not pushing it.
Two small doc fixes
.claude/references/auth.mdtells readers to look fordiscovery returned no valid data, but no release since 1.7.3 contains that string. It was the error 1.7.0–1.7.2 threw frominitwhen noaccountIssuerwas set; 1.7.3 and later log theProvider skippedline above. The same passage says sign-in is broken "until discovery returns", but it stays broken until the process restarts.- The "inverted failure mode" Known-open note can't happen with a
discoveryUrl-only config on 1.7.3 or later: a discovery fetch that fails (refused, non-2xx, or a non-JSON body) always ends inPROVIDER_NOT_FOUND, never in an unverified sign-in.
Happy to build it
If you'd like us to, I can build whichever option you prefer and submit a PR with tests, including a regression test that fails discovery at startup and checks that sign-in recovers without a restart.
Jonathon Huff, The Moody Church
Activity
chriskehayias commentedon Sep 29, 2026
So this was partially addressed - I have claude chewing on it now for full alignment.
chriskehayias commentedon Sep 29, 2026
Thanks, Jonathon. This is an excellent report: the repro, the root-cause analysis and both options were spot on, and it saved a lot of time.
Where things stood. Your repro is against
6c0c46b. A partial fix (5e1d2cc, merged in #99 shortly after that commit) wrapped the auth instance in aselfHealingAuthfacade. It rebuilt the instance on the next sign-in after a 30 s cooldown, which fixed "broken until restart" for a failed fetch. It did not fix the hang you described. The facade waited on the first instance's$contextwith no timeout, so a hung MP still stalled the first sign-in and every session check for about 300 s. It also still depended on better-auth internals. So your ticket stood.What we did: your option 1, with one correction. PR #107 (fixes this issue):
No
discoveryUrl. The provider is configured withauthorizationUrl,tokenUrlandendSessionEndpoint(you were right that theid_token_hintdepends on it), andaccountSubjectreadsprofile.sub. Building the auth instance,next build, cold starts and session checks now make no MP call at all, so the provider always registers.The id_token is still verified against MP's JWKS, now in our
getUserInfo, before thesubbinding. It uses the same josejwtVerifybetter-auth used (signature,iss,aud,exp), and RS256 is pinned.The correction: the JWKS URL isn't pinned. Your fake MP serves it at
/oauth/.well-known/openid-configuration/jwks, the IdentityServer default. A live MP discovery document we checked lists/oauth/.well-known/jwks. Hardcoding either path could break sign-in on some tenant. Soissuerandjwks_uristill come from discovery, but it's fetched lazily, at the first callback that needs it:A blip now refuses only the sign-ins that land during it (
/auth-error, "please try again"), and the next attempt works on the same process. jose likewise doesn't cache a failed JWKS fetch, and there's a test for that.F12 side effect, as you predicted. With no id_token config, better-auth refuses the
idTokenbranch itself (ID_TOKEN_NOT_SUPPORTED). We kept the route filter andhooks.beforeguard as the primary controls.selfHealingAuthhas been removed.joseis now a direct dependency.We checked MP's discovery document for
authorization_response_iss_parameter_supported. It isn't advertised, so losing better-auth's RFC 9207 callbackisscheck costs nothing.Verified with your repro, adapted to https (
env.tsnow requires it for the MP URL in production), againstnext build+next start:failmode before start:POST /api/auth/sign-in/socialreturns200with an authorize URL, anddiscoveryRequestsstays at 0.hangmode on a fresh process:/api/auth/get-session,/sign-in/social,/signinand/all answer in 10–40 ms, where you measured about 301 s.There's a new regression suite (
src/auth.oidc-discovery.test.ts) covering:Re-adding
discoveryUrlfails 11 of its 23 tests.Your two doc fixes.
discovery returned no valid dataand "until discovery returns" were already gone by #99's docs sweep. The references and downstream playbooks now describe the new behaviour. The "DroppingdiscoveryUrl… deferred" and F12-residue Known-open notes are marked done, with a porting section for forks. That includes what to delete if a fork already tookselfHealingAuth.Thanks again for the offer to build it. We went ahead since it touched the security suites anyway, but a review of the PR from you would be very welcome, especially against your four apps' setup.
Merge pull request #107 from MinistryPlatform-Community/fix/101-oidc-…
docs(playbooks): add a downstream port playbook for issue #101
fix(auth): verify the MP id_token lazily instead of fetching discover…
docs: describe the MinistryPlatform-Community#101 OIDC change across …