fix(scheduling): lock flow schedule reconcile before boot(), not after - #3494
Merged
Merged
Conversation
#3492: a deploy set the flow-schedule reconciliation marker, and datamachine_reconcile_marked_flow_schedules() ran FlowRoutines::reconcile(true) inline on init of every web request while it was set. FlowRoutines::reconcile() called self::boot() — a full ~700-routine registration pass through the Agents API Action Scheduler bridge — BEFORE the registry's own reconcile lock was taken. With 60 concurrent php-fpm workers, 59 of them paid the full boot() cost before discovering the lock was already held, and lock contention was misclassified as failure, so the marker was retained and every subsequent request repeated the same cost. Under that load a RedisException from update_option() inside the AS bridge escaped boot() before HashGatedRoutineBackend::persist() ran, losing that request's fingerprints and making the next boot() reschedule everything again — self-sustaining until the DB pinned out and the network took a 60-site 504 outage. Restore FlowScheduleReconciliationLock (a compare-and-set option lock deleted by #3475's routines convergence, alongside the smoke test that already exercised it) and take it in FlowRoutines::reconcile() BEFORE boot() runs. A losing caller's cost is now one option read: it returns immediately with success => true, skipped => true, reason => 'locked', having done zero registry or Action Scheduler work. The lock is a ->query() compare-and-set with an explicit wp_cache_delete(), not add_option()/delete_option(), so a stale object-cache-only entry can never block it forever (a failure mode observed on the underlying agents-api registry lock during recovery). Move the deferred reconcile off the request path: init only enqueues a single Action Scheduler async action (deduplicated with as_has_scheduled_action()); the actual reconcile runs inside that action's callback. A locked/skipped result there is a silent no-op — marker untouched, nothing logged — while a genuine failure retains the marker and arms a 5-minute backoff transient so a persistently failing reconcile doesn't re-enqueue on every request. The WP-CLI path (wp datamachine flows reconcile-schedules --apply) still calls FlowRoutines::reconcile() synchronously and now prints a clear message when it loses the lock race. Wrap boot()'s registration loops in try/finally so HashGatedRoutineBackend::persist() always runs, and give each routine's registration its own try/catch so one throwing registration logs and continues instead of losing every fingerprint recorded so far this request. Add two pure-PHP smoke tests: FlowRoutines::reconcile()'s lock-before-boot ordering and boot()'s persist-survives-a-throw guarantee, and the setup/flow-schedules.php enqueue/backoff/no-op-on-skip lifecycle. Run: php tests/flow-routines-reconcile-lock-before-boot-smoke.php, php tests/flow-schedule-reconciliation-worker-smoke.php, php tests/flow-schedule-reconciliation-lock-smoke.php (all pass). composer lint (WordPress-Extra) verified clean on every touched production file relative to the pre-existing baseline. Closes #3492
FlowScheduleReconciliationLock still relied on add_option() for the fast-acquire path and get_option() for the staleness check. Both go through the persistent object cache before touching the database, which reproduces the exact failure mode the lock brief and #3492 warned against: if a killed request leaves the lock cached (e.g. Redis) with no backing row in wp_options, add_option() always fails (cache hit), get_option() keeps serving the cached payload as if it were live for up to STALE_AFTER seconds, and once it's finally treated as stale the compare-and-set UPDATE matches zero rows (there is no row to update) — so acquire() is locked forever. This is what actually happened to the underlying agents-api registry lock (agents_routine_reconcile_lock) during the incident's recovery. Every locking decision now reads straight from wp_options via a direct ->get_row(), never the cache: - acquire(): no DB row -> wp_cache_delete() then INSERT IGNORE (unique option_name key resolves concurrent first-acquire races; a lost race re-reads and evaluates the winner's row instead of assuming failure). DB row present -> stale check and compare-and-set against the exact DB-read raw value, never a re-serialized-from-cache copy. - release() / refresh(): read and compare against the DB-read row; both now call wp_cache_delete() unconditionally after any mutation attempt, including the 0-row case, so a stale cache entry can never survive. No call to add_option()/get_option() remains anywhere in the class. Extended tests/flow-routines-reconcile-lock-before-boot-smoke.php with a cache/DB-split fake (separate cache and DB globals, matching a real persistent-object-cache-in-front-of-MySQL topology) and two new cases: a ghost cache entry with no backing DB row must not block acquire(), and two acquires racing against an empty DB must yield exactly one winner. Updated the fake in tests/flow-schedule-reconciliation-lock-smoke.php to back reads with get_row()/INSERT IGNORE against a DB-only store (it was fatally broken by the SELECT this class now issues) while keeping its original acquire/release/refresh/stale-takeover coverage. Run: php tests/flow-routines-reconcile-lock-before-boot-smoke.php (32/32), php tests/flow-schedule-reconciliation-lock-smoke.php (13/13), php tests/flow-schedule-reconciliation-worker-smoke.php (18/18, unaffected). composer lint (WordPress-Extra) on the touched file shows only the pre-existing filename-convention findings already present before this change — no new violations.
Both new smokes hand-rolled a DataMachine\Core\ActionScheduler\GroupRegistrar stub whose sole purpose was the GROUP constant, duplicating the 'data-machine' slug literal already owned by the real class. Require the real inc/Core/ActionScheduler/GroupRegistrar.php file instead — it is a plain constant holder plus DB-touching methods neither smoke ever calls, so requiring it has no side effects, and it matches the existing convention used by tests/ai-step-backpressure-smoke.php, tests/action-scheduler-group-registration-smoke.php, tests/batch-scheduler-retry-budget-smoke.php, and tests/fanout-adoption-recovery-smoke.php. Run: php tests/flow-routines-reconcile-lock-before-boot-smoke.php (32/32), php tests/flow-schedule-reconciliation-worker-smoke.php (18/18).
This was referenced Sep 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3492
Summary
The v0.176.8 deploy took the whole Extra Chill network down for ~45 minutes because
FlowRoutines::reconcile()calledself::boot()(a full ~700-routine registration pass through the Agents API Action Scheduler bridge) before any lock was taken. Only the registry's own reconcile step was guarded, so every one of 60 concurrent php-fpm workers paid the fullboot()cost before finding out the registry lock was already held — and lock contention was misclassified as failure, so the deploy marker was retained and every subsequent request repeated the same cost. Under that load aRedisExceptionfromupdate_option()inside the Action Scheduler bridge escapedboot()beforeHashGatedRoutineBackend::persist()ran, losing that request's fingerprints and making the nextboot()reschedule everything again — self-sustaining until the DB pinned out.Changes
Lock before boot. Restored
FlowScheduleReconciliationLock— a compare-and-set$wpdb->query()option lock with an explicitwp_cache_delete()(notadd_option()/delete_option(), which can leave a stale object-cache-only entry that blocks forever — this was observed on the underlying agents-api registry lock during recovery). It was deleted by refactor(scheduling): converge flow and system-task scheduling on Agents API Routines #3475's routines convergence, along with its production class but not its smoke test (tests/flow-schedule-reconciliation-lock-smoke.php), which has been silently broken (require_oncea file that no longer existed) since that PR landed.FlowRoutines::reconcile()now acquires this lock before callingboot(). A losing caller returns immediately withsuccess => true, skipped => true, reason => 'locked', having done zero registry or Action Scheduler work.Lock contention is not failure.
datamachine_reconcile_marked_flow_schedules()(soondatamachine_run_deferred_flow_schedule_reconciliation(), see WP_Abilities_Registry::register called incorrectly - missing 'category' string in ability properties #3) treats askippedresult as a silent no-op: marker untouched, nothing logged. Only a genuine failure retains the marker, and it now arms a 5-minute backoff transient so a persistently failing reconcile doesn't re-enqueue on every request.Moved off the request path.
initno longer runsreconcile(true)inline — it only enqueues a single Action Scheduler async action (deduplicated withas_has_scheduled_action()), and the actual reconcile runs inside that action's callback. The WP-CLI path (wp datamachine flows reconcile-schedules --apply) still callsFlowRoutines::reconcile()synchronously, and now prints a clear message when it loses the lock race instead of reporting a misleading empty "success".persist()survives a throw.boot()'s registration loops now run inside atry/finallysoHashGatedRoutineBackend::persist()always runs, and each routine's registration has its owntry/catchso one throwing registration (e.g. the Redis exception from the incident) logs and continues instead of silently losing every fingerprint recorded so far that request.Testing
Two new pure-PHP smoke tests (existing style — no PHPUnit/WP bootstrap):
tests/flow-routines-reconcile-lock-before-boot-smoke.php— proves (a) a held lock makesreconcile()skip without ever callingboot()orpersist(), and that normal operation resumes once released; (c)boot()persists once even when a registration throws (loop continues to the next routine), and persists via its top-levelfinallyeven when an unexpected throw outside the per-routine catch propagates out ofboot().tests/flow-schedule-reconciliation-worker-smoke.php— proves (b) theinithandler never callsFlowRoutines::reconcile()itself (only enqueues, deduplicated against an already-pending action), a locked/skipped worker result is a silent no-op, (d) a genuine failure retains the marker and arms the backoff transient (and theinithandler respects it), and (e) a genuine success clears the marker and logs completion.Ran:
composer lint(WordPress-Extra standard — this repo has nophpcs.xml, so I diffed against the pre-existing per-file baseline rather than a barephpcsrun against PSR2 defaults, which flags unrelated pre-existing style across the whole codebase): every touched production file is clean relative to its pre-existing baseline. The one alignment issue my own change introduced (ReconcileFlowSchedulesAbility.php's newskipped/reasonoutput-schema keys) was fixed withphpcbf.Not in scope / follow-ups
agents_routine_reconcile_lockstill usesadd_option()/delete_option()and has the same stale-object-cache failure mode this PR's lock is designed to avoid. That's an agents-api issue, filed separately per the network's layer-purity convention — not fixed here since it's owned by a different repo/layer.agents_routine_action_generation_*option rows observed during recovery are also an agents-api-side leak, out of scope for this DM-side fix.AI disclosure
This PR was authored by an AI coding agent (Extra Chill Bot / Claude) per repository convention.