Skip to content

policy: let soft drain proceed when pending demand fits without the worker - #50

Merged
rlrs merged 1 commit into
mainfrom
fix/soft-drain-pending-veto
Oct 24, 2026
Merged

rlrs merged 1 commit into
mainfrom
fix/soft-drain-pending-veto

Conversation

@rlrs

@rlrs rlrs commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Problem

plan_soft_drain treats any pending create as a veto. While pending_count > 0 it never selects a surplus worker, and it clears an in-progress drain whenever the worker could serve a pending request.

Queued creates keep a queued_create pending row from enqueue until complete(), including while they are deferred on image_building. So a steady trickle of creates waiting on first-use builds keeps pending_count above zero for hours.

Production hit this: six workers each held a handful of parked sandboxes with no compute in use, and 24–104 vCPU of pending creates would have fit on one or two workers. Soft drain logged "can serve pending demand" and cancelled every time, so the fleet never shrank.

Change

New ScalePolicy.drain_on_park_with_pending_enabled, default off:

  • Flag off: behaviour is unchanged.
  • Flag on: pending demand blocks a drain only through the existing surplus() check, which already places pending demand (resources and exact placement shapes) on the remaining workers.

The PR also:

  • parses the flag in config (older configs without the key still load);
  • shows it in the dashboard policy dump;
  • adds a short paragraph in the soft-drain section of docs/architecture.md.

Tests

Decision tests in tests/test_soft_drain.py, with plain inputs and an explicit now:

  • Several workers with few parks plus pending demand that fits on one worker. Flag off, nothing is selected; flag on, the emptiest worker is selected.
  • An in-progress drain is cleared with the flag off and kept with it on.
  • Pending demand that doesn't fit without the worker still blocks selection and clears a drain with the flag on.
  • The config round-trip covers the new key and its default.

Ran the unit tier (737 passed) and the unit plus contract tiers against PostgreSQL (2,371 passed, 22 skipped). scripts/check_test_rules.py passes. I did not run the full scripts/check.sh (Rust, Go, SDK).

Rollout

After release, set "drain_on_park_with_pending_enabled": true in the deployment policy. Only gateway-side code changes.

🤖 Generated with Claude Code

…orker

plan_soft_drain treated any pending create as a veto: it never selected a
surplus worker while pending_count > 0, and cleared an in-progress drain
whenever the worker could serve a pending request. Hours of pending demand
that fit on one worker pinned every worker that held parks.

surplus() already places pending demand (resources and exact placement
shapes) on the survivors, so with the new ScalePolicy flag
drain_on_park_with_pending_enabled (default off) pending demand only blocks a
drain through that check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rlrs
rlrs merged commit ec60905 into main Oct 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant