Repository navigation
Pin openai below 3: its truststore dependency breaks SSL under gevent - #191
Merged
Merged
Conversation
Every production image built since 2026-10-14 pulled openai 3.x, whose new HTTP stack depends on truststore. truststore wraps the SSL socket with a do_handshake() that does not match what gevent's patched SSLSocket calls, so every LLM request under the gevent Celery pool failed with "SSLSocket.do_handshake() takes 1 positional argument but 2 were given" and students saw "AI generation failed". Reproduced in isolated virtualenvs against api.openai.com with a dummy key, under monkey.patch_all(): gevent 26.9.0 + openai 3.31.0 fails; gevent 26.9.0 + openai 2.16.0 returns the expected 401; gevent 25.9.1 + openai 3.31.0 still fails, so the SDK is the variable, not gevent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
CoffeePoweredComputers
added a commit
that referenced
this pull request
Oct 16, 2026
…ments.txt lock requirements.txt was lower bounds only, so every image build installed whatever was newest on PyPI that day. That is how openai 3.x reached production unreviewed and broke AI grading for a day (#191). - requirements.in now holds the direct dependencies with their safety floors (the old file, header rewritten, one duplicate removed). - requirements.txt is generated from it by `make lock` (uv pip compile, resolved for Python 3.11 with universal markers so the 3.12 CI matrix and local 3.14 install the same pins). Docker, CI, e2e and the README keep installing requirements.txt unchanged. - CI's lint job recompiles without --upgrade and fails if the lock is stale, so editing requirements.in without running `make lock` cannot land. The locked set was installed into a real Python 3.11 and the LLM client was exercised under gevent monkey-patching: expected 401, no handshake error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CoffeePoweredComputers
added a commit
that referenced
this pull request
Oct 16, 2026
…ments.txt lock (#192) requirements.txt was lower bounds only, so every image build installed whatever was newest on PyPI that day. That is how openai 3.x reached production unreviewed and broke AI grading for a day (#191). - requirements.in now holds the direct dependencies with their safety floors (the old file, header rewritten, one duplicate removed). - requirements.txt is generated from it by `make lock` (uv pip compile, resolved for Python 3.11 with universal markers so the 3.12 CI matrix and local 3.14 install the same pins). Docker, CI, e2e and the README keep installing requirements.txt unchanged. - CI's lint job recompiles without --upgrade and fails if the lock is stale, so editing requirements.in without running `make lock` cannot land. The locked set was installed into a real Python 3.11 and the LLM client was exercised under gevent monkey-patching: expected 401, no handshake error. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production outage
AI grading in production has been failing with:
Cause
requirements.txtuses lower bounds only, so every image build pulls the newest of everything. OpenAI SDK 3.30 was published on 2026-10-14 and every image built since (all of today's deploys, starting with #182) picked up 3.x. OpenAI 3.x depends on a new HTTP stack plustruststore, which wraps the SSL socket with ado_handshake()whose signature does not match what gevent's patchedSSLSocketcalls. The Celery worker runs with--pool=geventand monkey-patchesssl, so every LLM request died in the TLS handshake.Bisect
Reproduced in isolated virtualenvs under
monkey.patch_all()againstapi.openai.comwith a dummy key. A healthy stack returns the expected 401; a broken one raises in the handshake.The SDK is the variable, not gevent.
Fix
Pin
openai>=1.40.6,<3. The code uses onlyopenai.OpenAI(...)andchat.completions, which 2.x provides. No code change.Follow-up worth doing
Lower-bound-only requirements mean any upstream release can break a deploy with no code change. A lock file (
pip-compileor equivalent) would make image builds reproducible.🤖 Generated with Claude Code