Summary
The 0.176.8 deploy (2026-09-12 17:16 UTC) took the whole Extra Chill network down twice within an hour. datamachine_maybe_ensure_current_schema() set datamachine_flow_schedule_reconciliation_pending on every site; datamachine_reconcile_marked_flow_schedules() (init:10) then ran FlowRoutines::reconcile( true ) on every request on the Events site (717 persisted flow routines). Reactivating DM at 18:00 (which calls activate_for_site() → re-arms the same marker) reproduced the storm immediately.
Chain of causes (all in inc/setup/flow-schedules.php + inc/Engine/Scheduling/FlowRoutines.php)
FlowRoutines::reconcile() runs boot() before the reconcile lock is taken. boot() re-declares all 717 routines; only then does WP_Agent_Routine_Registry::reconcile() try acquire_reconcile_lock(). Every concurrent request pays the full boot cost before discovering it lost the lock.- Lock contention retains the marker.
reconcile() returns success:false with _lock => 'Another routine reconcile is already running.', so datamachine_reconcile_marked_flow_schedules() logs "marker retained" and every subsequent request retries. With 60 php-fpm workers that is 60 concurrent full boots, continuously.
- The hash gate did not hold. Each boot went through
WP_Agent_Routine_Action_Scheduler_Bridge::register() for every routine (as_unschedule_all_actions + as_schedule_recurring_action). Observed: every one of the 717 flow-N routines accrued 44 canceled actions in 15 minutes — i.e. ~44 complete re-registration passes. datamachine_routine_schedule_hashes exists with 717 entries, so either schedule_hash() inputs changed in 0.176.8 or persist() never won the race under contention. Open question — needs a test that a version bump does not invalidate all fingerprints.
- O(n²) via Action Scheduler canceled rows.
as_unschedule_all_actions(hook, args, group) is planned on the group_id index and scans every agents-api row including canceled ones. Canceled rows went 0 → 44k in the first hour and 2 → 45,504 in ~9 minutes after reactivation, so each unschedule got slower as the storm progressed. 60 MariaDB threads all in Sending data on SELECT a.action_id FROM c8c_7_actionscheduler_actions a LEFT JOIN c8c_7_actionscheduler_groups g ..., load 64, every public URL timing out.
- Deactivating DM does not stop in-flight boots.
max_execution_time=120 excludes DB wait time on Linux and request_terminate_timeout is unset, so 60 workers kept grinding for >10 minutes after DM was inactive in DB + Redis. The only thing that drained them was repeatedly purging the canceled rows (which makes each scan cheap) — see mitigation.
Evidence
c8c_7_actionscheduler_logs rows/min: 17:48 → 47,655; 18:01 → 25,058 (reactivation); sustained ~2,500/min until the purge.wp_agent_routine_run_scheduled status on blog 7 at 18:12: canceled 45,504 / pending 960 / failed 85. After purge + drain: canceled ~0 / pending 596.- DM log wall-to-wall:
Deferred flow schedule reconciliation failed; marker retained / Another routine reconcile is already running.
- Marker also survived in Redis (
2:options:…, 11:options:…, 13:options:…) after delete_option on other blogs — the object cache needed an explicit flush of that key.
- Secondary blast radius:
extrachill-ai-adventure and extrachill-studio fatal on every request once DM is inactive (unguarded hard dependency on DM-vendored AgentsAPI\… classes) — filed separately.
Mitigation applied (twice)
Kill runaway queries; delete_option + Redis delete of the marker on every blog; DELETE … WHERE hook='wp_agent_routine_run_scheduled' AND status='canceled' (plus logs) on blog 7 in a loop until php-fpm drained; deactivate data-machine network-wide. DM is currently inactive on production and must stay that way until this is fixed — activate_for_site() re-arms the marker.
Proposed fix
- Take the reconcile lock (or a DM-side transient lock) before
boot(), and make lock contention a clean no-op that does not retain the marker on that request (the lock holder clears it on success).
- Gate the deferred reconcile behind a single scheduled action (
as_enqueue_async_action) rather than running it inline on init of every request — activation/deploy should enqueue one reconcile, not arm every request.
- Add a regression test: bumping
DATAMACHINE_VERSION with unchanged flow scheduling must not invalidate datamachine_routine_schedule_hashes.
- Upstream (Automattic/agents-api, filed separately): the AS bridge's unschedule path must not degrade with canceled rows in the group, and cancels should not be retained indefinitely.
Related: #3458 (legacy hook retirement), agents-api bridge issue (link below once filed).
Summary
The 0.176.8 deploy (2026-09-12 17:16 UTC) took the whole Extra Chill network down twice within an hour.
datamachine_maybe_ensure_current_schema()setdatamachine_flow_schedule_reconciliation_pendingon every site;datamachine_reconcile_marked_flow_schedules()(init:10) then ranFlowRoutines::reconcile( true )on every request on the Events site (717 persisted flow routines). Reactivating DM at 18:00 (which callsactivate_for_site()→ re-arms the same marker) reproduced the storm immediately.Chain of causes (all in
inc/setup/flow-schedules.php+inc/Engine/Scheduling/FlowRoutines.php)FlowRoutines::reconcile()runsboot()before the reconcile lock is taken.boot()re-declares all 717 routines; only then doesWP_Agent_Routine_Registry::reconcile()tryacquire_reconcile_lock(). Every concurrent request pays the full boot cost before discovering it lost the lock.reconcile()returnssuccess:falsewith_lock => 'Another routine reconcile is already running.', sodatamachine_reconcile_marked_flow_schedules()logs "marker retained" and every subsequent request retries. With 60 php-fpm workers that is 60 concurrent full boots, continuously.WP_Agent_Routine_Action_Scheduler_Bridge::register()for every routine (as_unschedule_all_actions+as_schedule_recurring_action). Observed: every one of the 717flow-Nroutines accrued 44 canceled actions in 15 minutes — i.e. ~44 complete re-registration passes.datamachine_routine_schedule_hashesexists with 717 entries, so eitherschedule_hash()inputs changed in 0.176.8 orpersist()never won the race under contention. Open question — needs a test that a version bump does not invalidate all fingerprints.as_unschedule_all_actions(hook, args, group)is planned on thegroup_idindex and scans everyagents-apirow including canceled ones. Canceled rows went 0 → 44k in the first hour and 2 → 45,504 in ~9 minutes after reactivation, so each unschedule got slower as the storm progressed. 60 MariaDB threads all inSending dataonSELECT a.action_id FROM c8c_7_actionscheduler_actions a LEFT JOIN c8c_7_actionscheduler_groups g ..., load 64, every public URL timing out.max_execution_time=120excludes DB wait time on Linux andrequest_terminate_timeoutis unset, so 60 workers kept grinding for >10 minutes after DM was inactive in DB + Redis. The only thing that drained them was repeatedly purging the canceled rows (which makes each scan cheap) — see mitigation.Evidence
c8c_7_actionscheduler_logsrows/min: 17:48 → 47,655; 18:01 → 25,058 (reactivation); sustained ~2,500/min until the purge.wp_agent_routine_run_scheduledstatus on blog 7 at 18:12: canceled 45,504 / pending 960 / failed 85. After purge + drain: canceled ~0 / pending 596.Deferred flow schedule reconciliation failed; marker retained/Another routine reconcile is already running.2:options:…,11:options:…,13:options:…) afterdelete_optionon other blogs — the object cache needed an explicit flush of that key.extrachill-ai-adventureandextrachill-studiofatal on every request once DM is inactive (unguarded hard dependency on DM-vendoredAgentsAPI\…classes) — filed separately.Mitigation applied (twice)
Kill runaway queries;
delete_option+ Redis delete of the marker on every blog;DELETE … WHERE hook='wp_agent_routine_run_scheduled' AND status='canceled'(plus logs) on blog 7 in a loop until php-fpm drained; deactivatedata-machinenetwork-wide. DM is currently inactive on production and must stay that way until this is fixed —activate_for_site()re-arms the marker.Proposed fix
boot(), and make lock contention a clean no-op that does not retain the marker on that request (the lock holder clears it on success).as_enqueue_async_action) rather than running it inline oninitof every request — activation/deploy should enqueue one reconcile, not arm every request.DATAMACHINE_VERSIONwith unchanged flow scheduling must not invalidatedatamachine_routine_schedule_hashes.Related: #3458 (legacy hook retirement), agents-api bridge issue (link below once filed).