Skip to content

Fix staging cold-start timeout - #3

Merged
m-messer merged 1 commit into
mainfrom
fix-lambda-cold-start
Sep 1, 2026
Merged

Fix staging cold-start timeout#3
m-messer merged 1 commit into
mainfrom
fix-lambda-cold-start

Conversation

@m-messer

@m-messer m-messer commented Sep 1, 2026

Copy link
Copy Markdown
Member

Staging returned 503/500 on every request: the worker's cold-start import (torch + now pydantic/pydantic-core from the lf_toolkit v1.1.1 bump) takes ~34s on the 1024MB Lambda, longer than shimmy's 30s worker-send-timeout, so shimmy killed the half-booted worker before it became ready and never recovered.

  • evaluation.py / models/init.py: dispatch models via importlib against an AVAILABLE_MODELS allowlist instead of eagerly importing every model module. torch is no longer pulled in at worker startup, only when a model that needs it actually runs. Cold import drops ~34s -> ~8s (measured CPU/mem-constrained to match the Lambda).
  • models/utils.py: build the torch NeuralLM class lazily so importing utils for csv_to_lists / shard_for (shannon_letters_ngram, shannon_words_ngram) does not pull in torch either.
  • Dockerfile: raise FUNCTION_WORKER_START_TIMEOUT / FUNCTION_WORKER_SEND_TIMEOUT to 150s (< the 175s Lambda timeout) so shimmy waits out a slow first request instead of discarding the worker; make FUNCTION_RPC_TRANSPORT=stdio explicit.

Verified in AWS RIE Lambda mode, container constrained to 1024MB / 0.6 vCPU: cold torch-free model 13.9s -> 200; cold basic_nn (first torch load) 35s -> 200 on direct invoke; warm calls ~0.2s. Worker now survives the slow first call. pytest + flake8 pass.

Claude-Session: https://claude.ai/code/session_01UsgxJUsq1vG176rbznHRqz

… timeouts

Staging returned 503/500 on every request: the worker's cold-start import
(torch + now pydantic/pydantic-core from the lf_toolkit v1.1.1 bump) takes
~34s on the 1024MB Lambda, longer than shimmy's 30s worker-send-timeout, so
shimmy killed the half-booted worker before it became ready and never
recovered.

- evaluation.py / models/__init__.py: dispatch models via importlib against an
  AVAILABLE_MODELS allowlist instead of eagerly importing every model module.
  torch is no longer pulled in at worker startup, only when a model that needs
  it actually runs. Cold import drops ~34s -> ~8s (measured CPU/mem-constrained
  to match the Lambda).
- models/utils.py: build the torch NeuralLM class lazily so importing utils for
  csv_to_lists / shard_for (shannon_letters_ngram, shannon_words_ngram) does
  not pull in torch either.
- Dockerfile: raise FUNCTION_WORKER_START_TIMEOUT / FUNCTION_WORKER_SEND_TIMEOUT
  to 150s (< the 175s Lambda timeout) so shimmy waits out a slow first request
  instead of discarding the worker; make FUNCTION_RPC_TRANSPORT=stdio explicit.

Verified in AWS RIE Lambda mode, container constrained to 1024MB / 0.6 vCPU:
cold torch-free model 13.9s -> 200; cold basic_nn (first torch load) 35s -> 200
on direct invoke; warm calls ~0.2s. Worker now survives the slow first call.
pytest + flake8 pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UsgxJUsq1vG176rbznHRqz
@m-messer
m-messer merged commit 4979fce into main Sep 1, 2026
1 check passed
@m-messer
m-messer deleted the fix-lambda-cold-start branch September 1, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant