fix: wake deferred AI step on lease release, cap backoff at 120s, expose lease utilization (#3499)

[chubes4]

Summary

AI concurrency backpressure starved deferred jobs: each waiter slept on a per-job exponential backoff (capped at 600s) and nothing woke a waiter when a lease slot was released. A slot freed and the next waiter stayed asleep for up to ten minutes. This PR adds a release-side wake-up, lowers the cap to a filterable 120s, and surfaces lease utilization in wp datamachine worker status.

Fixes #3499

Observed starvation shape (from the #3499 diagnosis, live 2026-09-15)

• datamachine_resume_ai_step pending: 791 (791 distinct jobs), default pipeline_ai_concurrency_limit=3
• ai_resume_generation histogram of waiters: gen 8:93, gen 9:47, gen 10:162, gen 11:432 — 55% parked at the 600s cap
• Lease slots held (limit 3, sampled 1/sec for 10s): 2 2 2 2 2 2 1 1 2 2 — slots idle while 791 jobs wait
• After raising the limit to 10: 6 7 6 7 7 7 7 8 8 8 — still not saturated, same cause
• Jobs kept completing (100–500 per 10 min), so the system was not stuck, just starved

(Re-checked blog 7 while writing this PR: the backlog has since drained to 0 pending resume actions — consistent with the diagnosis; the fix removes the need to manually raise the limit.)

Behavior change

1. Release-side wake-up (AIConcurrencyBackpressure::wakeEarliestDeferred())

Called from AIStep::executeStep()'s finally immediately after $ai_concurrency_lease->release() — the precise signal that a held, used slot just freed. Selection rule: one pending datamachine_resume_ai_step action, earliest-scheduled first, restricted to actions whose scheduled time is still in the future (a waiter that backed off). Due-now pending actions are never selected — the queue is already self-waking there. The store query is bounded (hook + status + date > now, orderby date ASC, per_page 1) through as_get_scheduled_actions, so no full scan of actionscheduler_actions.

Consistency with the generation/claim protocol. Action Scheduler 3.9 has no scheduled-date update API (verified against the bundled store: ActionScheduler_wpPostStore exposes save_action/cancel_action but no update), so the wake goes cancel+recreate, in this order:

1. Schedule the replacement first at now, with the waiter's exact args and continuation group (continuationGroup()), unique=false — the still-pending predecessor would block a unique schedule.
2. Cancel the predecessor via ActionScheduler_Store::instance()->cancel_action().
3. Repoint ownership via new AIConcurrencyBackpressure::repointScheduledAction(), a CAS-style engine-data mutation fenced by flow_step_id + generation + status='scheduled' + action_id == previous. No token is needed because the exact previous action ID is the fence.

If step 1 fails, the wake aborts with the predecessor untouched. If step 2 or 3 fails, the waiter temporarily has two live actions — harmless, because datamachine_resume_ai_step_action execution-fences through beginGeneration() (only the exact scheduled → running transition wins; the duplicate no-ops). The whole wake is wrapped in try/catch(\Throwable) and returns a report array — a failed wake-up can never break the releasing job.

2. Lowered, filterable backoff cap

• AIStep::AI_CONCURRENCY_MAX_DEFER_DELAY: 600 → 120 seconds.
• New filter datamachine_ai_concurrency_max_defer_delay (args: value, provider, job_id). The cap now only bounds recovery when a wake-up is missed. AI_CONCURRENCY_MAX_DEFER_AGE (stranded threshold) is unchanged.

3. wp datamachine worker status lease telemetry

New columns appended at the end of the snapshot (table and JSON; existing keys/order untouched):

Column	Meaning
ai_lease_site_slots_held / ai_lease_site_limit	occupied slot rows / site concurrency limit
ai_lease_provider_scopes	per-provider held/limit as openai:1/2|anthropic:0/3 (empty when unconfigured)
ai_resume_pending_actions	exact pending count (one indexed count query)
ai_resume_sampled_actions / sample stats	median + max ai_resume_generation over a bounded sample (default 500, earliest-scheduled; sample_capped flags truncation)

Implementation: new PipelineAIConcurrencyLimiter::utilization() (read-only slot-row scan, site limit resolution extracted into a shared siteLimit() helper) and AIConcurrencyBackpressure::deferredSnapshot(). Everything is bounded and exception-contained.

Judgment calls

• Wake hook placement: AIStep finally, not PipelineAIConcurrencyLease::release(). The lease/limiter stay generic slot primitives — no resume-hook knowledge, no injected-callback plumbing. AIConcurrencyBackpressure already owns RESUME_HOOK and the generation protocol, and AIStep is the AI-specific orchestrator. The limiter's partial-release path (multi-scope acquire failure) deliberately does not wake: nothing was actually used there, and the deferring job reschedules itself anyway.
• No provider-scope filter on the wake target. Resume action args don't carry provider/scope, the site scope is the primary contention point, and a mismatched wake degrades gracefully to a normal re-defer (one extra limiter check).
• Schedule-before-cancel (opposite of the naive order) so the waiter never loses its continuation if the process dies mid-wake.
• camelCase method names kept to match the codebase's existing convention (scheduleContinuation, delaySeconds, …).

Files touched

• inc/Engine/AI/AIConcurrencyBackpressure.php — wakeEarliestDeferred(), repointScheduledAction(), deferredSnapshot(), scheduler-readiness guard
• inc/Core/Steps/AI/AIStep.php — cap 600 → 120 + filter, wake on lease release
• inc/Engine/AI/PipelineAIConcurrencyLimiter.php — utilization(), heldSlots(), siteLimit() extraction
• inc/Cli/Commands/WorkerCommand.php — status snapshot additions
• tests/ai-step-backpressure-smoke.php — cap/filter/wake assertions (+ missing is_wp_error stub that was breaking this smoke file on main pre-existing)
• tests/ai-concurrency-release-wakeup-smoke.php — new, 43 assertions

Verification

• php -l on every touched file — clean.
• php tests/ai-concurrency-release-wakeup-smoke.php — 43 assertions, 0 failures (wake selection picks earliest future waiter only; no-op with no/due-only waiters; ownership repoint keeps token/generation and stays fenced against duplicate execution; reschedule/cancel failure containment; snapshot count/median/max/capping; limiter utilization incl. provider scopes and release reset).
• php tests/ai-step-backpressure-smoke.php — 64 assertions, 0 failures (note: this smoke file was already fatally broken on main — missing is_wp_error() stub at Case 5; fixed here as part of extending it).
• phpcs --standard=WordPress before/after diff on touched files: all new findings on added lines fixed except the MethodNameInvalid camelCase sniffs, which match the repo-wide existing naming convention. (No repo phpcs.xml.dist exists, so composer lint runs a bare default standard that the tab-indented codebase fails wholesale pre-existing; phpcs comparison was done against HEAD baselines instead.)
• phpunit suite (e.g. JobLifecycleTransitionTest, which covers the resume protocol) runs in the WP Codebox CI environment (WordPress + MySQL) and will gate this PR; no unit-test coupling to the old 600s cap exists (grepped).

Authored by Extra Chill Bot (AI agent) via kimaki minion; not yet human-reviewed.