Fix staging cold-start timeout - #3
Merged
Merged
Conversation
… timeouts Staging returned 503/500 on every request: the worker's cold-start import (torch + now pydantic/pydantic-core from the lf_toolkit v1.1.1 bump) takes ~34s on the 1024MB Lambda, longer than shimmy's 30s worker-send-timeout, so shimmy killed the half-booted worker before it became ready and never recovered. - evaluation.py / models/__init__.py: dispatch models via importlib against an AVAILABLE_MODELS allowlist instead of eagerly importing every model module. torch is no longer pulled in at worker startup, only when a model that needs it actually runs. Cold import drops ~34s -> ~8s (measured CPU/mem-constrained to match the Lambda). - models/utils.py: build the torch NeuralLM class lazily so importing utils for csv_to_lists / shard_for (shannon_letters_ngram, shannon_words_ngram) does not pull in torch either. - Dockerfile: raise FUNCTION_WORKER_START_TIMEOUT / FUNCTION_WORKER_SEND_TIMEOUT to 150s (< the 175s Lambda timeout) so shimmy waits out a slow first request instead of discarding the worker; make FUNCTION_RPC_TRANSPORT=stdio explicit. Verified in AWS RIE Lambda mode, container constrained to 1024MB / 0.6 vCPU: cold torch-free model 13.9s -> 200; cold basic_nn (first torch load) 35s -> 200 on direct invoke; warm calls ~0.2s. Worker now survives the slow first call. pytest + flake8 pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UsgxJUsq1vG176rbznHRqz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staging returned 503/500 on every request: the worker's cold-start import (torch + now pydantic/pydantic-core from the lf_toolkit v1.1.1 bump) takes ~34s on the 1024MB Lambda, longer than shimmy's 30s worker-send-timeout, so shimmy killed the half-booted worker before it became ready and never recovered.
Verified in AWS RIE Lambda mode, container constrained to 1024MB / 0.6 vCPU: cold torch-free model 13.9s -> 200; cold basic_nn (first torch load) 35s -> 200 on direct invoke; warm calls ~0.2s. Worker now survives the slow first call. pytest + flake8 pass.
Claude-Session: https://claude.ai/code/session_01UsgxJUsq1vG176rbznHRqz