Repository navigation
Avoid compaction stalls during checkpoint restore - #3
Conversation
jackf-beamcloud
left a comment
There was a problem hiding this comment.
Looks right: the ExpectHugepages guard in adviseHugepageRanges covers both callers (AdviseHugepageRanges and PrefaultRanges), the reader default change only affects restore reads (writer and NewPagesFileFDReader callers untouched), and the checkpoint_version input is passed via env and regex-checked before reaching the shell, so no injection. Non-blocking: guarding on ExpectHugepages rather than AdviseHugepage means a host that expects THP but has advice disabled will still madvise here; fine if that's intended, maybe worth a comment.
|
Using |
VoxCPM2 checkpoints intermittently spent tens of seconds inside runsc restore even when the image and checkpoint were already local. Pin the exact node-tested runsc hotfix from [beam-cloud/gvisor#3](beam-cloud/gvisor#3): honor the existing host huge-page policy to avoid synchronous compaction, and reduce shared-memory contention between checkpoint readers. The artifact preserves the deployed checkpoint compatibility version (`ebbaed3f9dd5`). Serialized state is unchanged; strict version checks and the worker compatibility key remain stable. Its new source revision, release tag and SHA512 identify the hotfix separately. Validation used this exact binary and the real beta9 runtime on reserved RTX 4090 GPUs. An unchanged VoxCPM2 checkpoint captured on .22 restored on .19 in 13.56–14.34 s and .22 in 11.64–13.24 s, retaining its nonce and identical audio. The deployed runtime reproduced a 77.36 s managed restore; a paired private .22 control took 19.31 s. Registered host memory and GPU state also passed, with restore improving from 12.05 to 7.23 s. The published asset was downloaded through the Dockerfile URL and verified byte-identical to the tested binary. Some warm deployed restores were faster; these tests establish removal of a compaction path, not a guaranteed seven-second restore or production p99. No scheduler-free 5090 GPU was available for this artifact's GPU test. The runtime dependency is approved and merged; the artifact remains a prerelease and production has not been changed. [Canonical build](https://github.com/beam-cloud/gvisor/actions/runs/37680251173) · [Pinned artifact](https://github.com/beam-cloud/gvisor/releases/tag/release-20260714.0-beam.10)
Restoring CUDA checkpoints advised huge pages even when the destination host policy had disabled them. On the affected host this caused direct compaction while loading and pinning restored memory. Honor the existing immutable
ExpectHugepagespolicy at both advice sites, retaining eager page loading, writable prefaulting and CUDA pinning.Use eight 4 MiB page reads instead of 128 256 KiB reads, keeping 32 MiB in flight. Writer defaults, read limits, short-read/error handling and serialized checkpoint fields remain unchanged.
The optional artifact-build input preserves
ebbaed3f9dd5for this state-compatible hotfix. Both strict checkpoint-version checks remain enabled, and the worker's runtime-version fingerprint input is unchanged. Identify the artifact by its new source revision, immutable release tag and SHA512; default builds retain their existing git-described version behavior.Validation: the canonical Linux artifact restored the original, unmodified owned VoxCPM2 checkpoint with retained nonce and identical audio. Final runtime durations were 13.56–14.34s on the affected host and 11.64–13.24s on the control host; one corresponding deployed control took 19.31s with 12.28s in prefault versus 4.03–4.65s for the candidate. Registered host memory, GPU allocations, module and stream state also passed. Profiling demonstrated direct compaction in deployed restores and none in the policy-guard cases.
These private runtime measurements exclude scheduling and checkpoint transfer. Some warmed deployed restores were faster, and the reader change's independent full-model benefit is not established; this addresses compaction stalls rather than guaranteeing every restore is faster. No production runtime was replaced during validation.