Skip to content

fix(scheduling): lock flow schedule reconcile before boot(), not after - #3494

Merged
chubes4 merged 3 commits into
mainfrom
fix/reconcile-lock-before-boot
Sep 12, 2026
Merged

chubes4 merged 3 commits into
mainfrom
fix/reconcile-lock-before-boot

Conversation

@chubes4

@chubes4 chubes4 commented Sep 12, 2026

Copy link
Copy Markdown
Member

Closes #3492

Summary

The v0.176.8 deploy took the whole Extra Chill network down for ~45 minutes because FlowRoutines::reconcile() called self::boot() (a full ~700-routine registration pass through the Agents API Action Scheduler bridge) before any lock was taken. Only the registry's own reconcile step was guarded, so every one of 60 concurrent php-fpm workers paid the full boot() cost before finding out the registry lock was already held — and lock contention was misclassified as failure, so the deploy marker was retained and every subsequent request repeated the same cost. Under that load a RedisException from update_option() inside the Action Scheduler bridge escaped boot() before HashGatedRoutineBackend::persist() ran, losing that request's fingerprints and making the next boot() reschedule everything again — self-sustaining until the DB pinned out.

Changes

  1. Lock before boot. Restored FlowScheduleReconciliationLock — a compare-and-set $wpdb->query() option lock with an explicit wp_cache_delete() (not add_option()/delete_option(), which can leave a stale object-cache-only entry that blocks forever — this was observed on the underlying agents-api registry lock during recovery). It was deleted by refactor(scheduling): converge flow and system-task scheduling on Agents API Routines #3475's routines convergence, along with its production class but not its smoke test (tests/flow-schedule-reconciliation-lock-smoke.php), which has been silently broken (require_once a file that no longer existed) since that PR landed. FlowRoutines::reconcile() now acquires this lock before calling boot(). A losing caller returns immediately with success => true, skipped => true, reason => 'locked', having done zero registry or Action Scheduler work.

  2. Lock contention is not failure. datamachine_reconcile_marked_flow_schedules() (soon datamachine_run_deferred_flow_schedule_reconciliation(), see WP_Abilities_Registry::register called incorrectly - missing 'category' string in ability properties #3) treats a skipped result as a silent no-op: marker untouched, nothing logged. Only a genuine failure retains the marker, and it now arms a 5-minute backoff transient so a persistently failing reconcile doesn't re-enqueue on every request.

  3. Moved off the request path. init no longer runs reconcile(true) inline — it only enqueues a single Action Scheduler async action (deduplicated with as_has_scheduled_action()), and the actual reconcile runs inside that action's callback. The WP-CLI path (wp datamachine flows reconcile-schedules --apply) still calls FlowRoutines::reconcile() synchronously, and now prints a clear message when it loses the lock race instead of reporting a misleading empty "success".

  4. persist() survives a throw. boot()'s registration loops now run inside a try/finally so HashGatedRoutineBackend::persist() always runs, and each routine's registration has its own try/catch so one throwing registration (e.g. the Redis exception from the incident) logs and continues instead of silently losing every fingerprint recorded so far that request.

Testing

Two new pure-PHP smoke tests (existing style — no PHPUnit/WP bootstrap):

  • tests/flow-routines-reconcile-lock-before-boot-smoke.php — proves (a) a held lock makes reconcile() skip without ever calling boot() or persist(), and that normal operation resumes once released; (c) boot() persists once even when a registration throws (loop continues to the next routine), and persists via its top-level finally even when an unexpected throw outside the per-routine catch propagates out of boot().
  • tests/flow-schedule-reconciliation-worker-smoke.php — proves (b) the init handler never calls FlowRoutines::reconcile() itself (only enqueues, deduplicated against an already-pending action), a locked/skipped worker result is a silent no-op, (d) a genuine failure retains the marker and arms the backoff transient (and the init handler respects it), and (e) a genuine success clears the marker and logs completion.

Ran:

php tests/flow-routines-reconcile-lock-before-boot-smoke.php   # 19/19 pass
php tests/flow-schedule-reconciliation-worker-smoke.php        # 18/18 pass
php tests/flow-schedule-reconciliation-lock-smoke.php          # 13/13 pass (previously broken by #3475's dangling require)
php tests/setup-schema-runtime-smoke.php                       # pass, unaffected

composer lint (WordPress-Extra standard — this repo has no phpcs.xml, so I diffed against the pre-existing per-file baseline rather than a bare phpcs run against PSR2 defaults, which flags unrelated pre-existing style across the whole codebase): every touched production file is clean relative to its pre-existing baseline. The one alignment issue my own change introduced (ReconcileFlowSchedulesAbility.php's new skipped/reason output-schema keys) was fixed with phpcbf.

Not in scope / follow-ups

  • The agents-api registry's own agents_routine_reconcile_lock still uses add_option()/delete_option() and has the same stale-object-cache failure mode this PR's lock is designed to avoid. That's an agents-api issue, filed separately per the network's layer-purity convention — not fixed here since it's owned by a different repo/layer.
  • The ~300k leaked agents_routine_action_generation_* option rows observed during recovery are also an agents-api-side leak, out of scope for this DM-side fix.

AI disclosure

This PR was authored by an AI coding agent (Extra Chill Bot / Claude) per repository convention.

#3492: a deploy set the flow-schedule
reconciliation marker, and datamachine_reconcile_marked_flow_schedules()
ran FlowRoutines::reconcile(true) inline on init of every web request
while it was set. FlowRoutines::reconcile() called self::boot() — a full
~700-routine registration pass through the Agents API Action Scheduler
bridge — BEFORE the registry's own reconcile lock was taken. With 60
concurrent php-fpm workers, 59 of them paid the full boot() cost before
discovering the lock was already held, and lock contention was
misclassified as failure, so the marker was retained and every
subsequent request repeated the same cost. Under that load a
RedisException from update_option() inside the AS bridge escaped
boot() before HashGatedRoutineBackend::persist() ran, losing that
request's fingerprints and making the next boot() reschedule everything
again — self-sustaining until the DB pinned out and the network took a
60-site 504 outage.

Restore FlowScheduleReconciliationLock (a compare-and-set option lock
deleted by #3475's routines convergence, alongside the smoke test that
already exercised it) and take it in FlowRoutines::reconcile() BEFORE
boot() runs. A losing caller's cost is now one option read: it returns
immediately with success => true, skipped => true, reason => 'locked',
having done zero registry or Action Scheduler work. The lock is a
->query() compare-and-set with an explicit wp_cache_delete(), not
add_option()/delete_option(), so a stale object-cache-only entry can
never block it forever (a failure mode observed on the underlying
agents-api registry lock during recovery).

Move the deferred reconcile off the request path: init only enqueues a
single Action Scheduler async action (deduplicated with
as_has_scheduled_action()); the actual reconcile runs inside that
action's callback. A locked/skipped result there is a silent no-op —
marker untouched, nothing logged — while a genuine failure retains the
marker and arms a 5-minute backoff transient so a persistently failing
reconcile doesn't re-enqueue on every request. The WP-CLI path
(wp datamachine flows reconcile-schedules --apply) still calls
FlowRoutines::reconcile() synchronously and now prints a clear message
when it loses the lock race.

Wrap boot()'s registration loops in try/finally so
HashGatedRoutineBackend::persist() always runs, and give each routine's
registration its own try/catch so one throwing registration logs and
continues instead of losing every fingerprint recorded so far this
request.

Add two pure-PHP smoke tests: FlowRoutines::reconcile()'s lock-before-boot
ordering and boot()'s persist-survives-a-throw guarantee, and the
setup/flow-schedules.php enqueue/backoff/no-op-on-skip lifecycle.

Run: php tests/flow-routines-reconcile-lock-before-boot-smoke.php,
php tests/flow-schedule-reconciliation-worker-smoke.php,
php tests/flow-schedule-reconciliation-lock-smoke.php (all pass).
composer lint (WordPress-Extra) verified clean on every touched
production file relative to the pre-existing baseline.

Closes #3492
FlowScheduleReconciliationLock still relied on add_option() for the
fast-acquire path and get_option() for the staleness check. Both go
through the persistent object cache before touching the database, which
reproduces the exact failure mode the lock brief and #3492 warned
against: if a killed request leaves the lock cached (e.g. Redis) with no
backing row in wp_options, add_option() always fails (cache hit),
get_option() keeps serving the cached payload as if it were live for up
to STALE_AFTER seconds, and once it's finally treated as stale the
compare-and-set UPDATE matches zero rows (there is no row to update) —
so acquire() is locked forever. This is what actually happened to the
underlying agents-api registry lock (agents_routine_reconcile_lock)
during the incident's recovery.

Every locking decision now reads straight from wp_options via a direct
->get_row(), never the cache:

- acquire(): no DB row -> wp_cache_delete() then INSERT IGNORE (unique
  option_name key resolves concurrent first-acquire races; a lost race
  re-reads and evaluates the winner's row instead of assuming failure).
  DB row present -> stale check and compare-and-set against the exact
  DB-read raw value, never a re-serialized-from-cache copy.
- release() / refresh(): read and compare against the DB-read row; both
  now call wp_cache_delete() unconditionally after any mutation attempt,
  including the 0-row case, so a stale cache entry can never survive.

No call to add_option()/get_option() remains anywhere in the class.

Extended tests/flow-routines-reconcile-lock-before-boot-smoke.php with a
cache/DB-split fake  (separate cache and DB globals, matching a real
persistent-object-cache-in-front-of-MySQL topology) and two new cases:
a ghost cache entry with no backing DB row must not block acquire(), and
two acquires racing against an empty DB must yield exactly one winner.
Updated the fake  in tests/flow-schedule-reconciliation-lock-smoke.php
to back reads with get_row()/INSERT IGNORE against a DB-only store (it
was fatally broken by the SELECT this class now issues) while keeping
its original acquire/release/refresh/stale-takeover coverage.

Run: php tests/flow-routines-reconcile-lock-before-boot-smoke.php (32/32),
php tests/flow-schedule-reconciliation-lock-smoke.php (13/13),
php tests/flow-schedule-reconciliation-worker-smoke.php (18/18, unaffected).
composer lint (WordPress-Extra) on the touched file shows only the
pre-existing filename-convention findings already present before this
change — no new violations.
Both new smokes hand-rolled a DataMachine\Core\ActionScheduler\GroupRegistrar
stub whose sole purpose was the GROUP constant, duplicating the
'data-machine' slug literal already owned by the real class. Require the
real inc/Core/ActionScheduler/GroupRegistrar.php file instead — it is a
plain constant holder plus DB-touching methods neither smoke ever calls,
so requiring it has no side effects, and it matches the existing
convention used by tests/ai-step-backpressure-smoke.php,
tests/action-scheduler-group-registration-smoke.php,
tests/batch-scheduler-retry-budget-smoke.php, and
tests/fanout-adoption-recovery-smoke.php.

Run: php tests/flow-routines-reconcile-lock-before-boot-smoke.php (32/32),
php tests/flow-schedule-reconciliation-worker-smoke.php (18/18).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deploy-time flow schedule reconciliation on init runs on every request under lock contention and took the network down (v0.176.8)

1 participant