Repository navigation
policy: let soft drain proceed when pending demand fits without the worker - #50
Merged
Merged
Conversation
…orker plan_soft_drain treated any pending create as a veto: it never selected a surplus worker while pending_count > 0, and cleared an in-progress drain whenever the worker could serve a pending request. Hours of pending demand that fit on one worker pinned every worker that held parks. surplus() already places pending demand (resources and exact placement shapes) on the survivors, so with the new ScalePolicy flag drain_on_park_with_pending_enabled (default off) pending demand only blocks a drain through that check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
plan_soft_draintreats any pending create as a veto. Whilepending_count > 0it never selects a surplus worker, and it clears an in-progress drain whenever the worker could serve a pending request.Queued creates keep a
queued_createpending row from enqueue untilcomplete(), including while they are deferred onimage_building. So a steady trickle of creates waiting on first-use builds keepspending_countabove zero for hours.Production hit this: six workers each held a handful of parked sandboxes with no compute in use, and 24–104 vCPU of pending creates would have fit on one or two workers. Soft drain logged "can serve pending demand" and cancelled every time, so the fleet never shrank.
Change
New
ScalePolicy.drain_on_park_with_pending_enabled, default off:surplus()check, which already places pending demand (resources and exact placement shapes) on the remaining workers.The PR also:
docs/architecture.md.Tests
Decision tests in
tests/test_soft_drain.py, with plain inputs and an explicitnow:Ran the unit tier (737 passed) and the unit plus contract tiers against PostgreSQL (2,371 passed, 22 skipped).
scripts/check_test_rules.pypasses. I did not run the fullscripts/check.sh(Rust, Go, SDK).Rollout
After release, set
"drain_on_park_with_pending_enabled": truein the deployment policy. Only gateway-side code changes.🤖 Generated with Claude Code