Skip to content

Ensure world termination completes under IO saturation - #475

Merged
adamruzicka merged 3 commits into
Dynflow:masterfrom
jakduch:fix/world-termination-io-starvation
Sep 25, 2026
Merged

adamruzicka merged 3 commits into
Dynflow:masterfrom
jakduch:fix/world-termination-io-starvation

Conversation

@jakduch

@jakduch jakduch commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem

World termination schedules its timeout watchdog as a Concurrent::Promises.future and its completion callback on the same global IO executor used by persistence work. When PostgreSQL is unavailable and that executor is saturated by retries, the shutdown can reach local component cleanup but never complete.

This is exposed by the concurrent-ruby 1.3.x update in #473. The same update also exposes timing-sensitive WithPollingSubPlans assertions and a Ruby 3.0 incompatibility between multi_json 1.19.1 and JSON 3.

Changes

  • expose a resolvable future as the termination result
  • run the bounded watchdog on a dedicated short-lived thread
  • resolve the world termination event in one place
  • wait for polling timers to be scheduled before advancing the managed clock in WithPollingSubPlans tests
  • keep JSON below 3 on Ruby below 3.2, whose compatible multi_json release cannot use the JSON 3 keyword-argument API

Verification

  • concurrent-ruby 1.3.8 / concurrent-ruby-edge 0.7.2: real PostgreSQL outage terminated after 80 seconds, within the 180-second Bats timeout
  • Ruby 3.3/PostgreSQL suite with 1.3.x: 402 runs, 1488 assertions, no failures or errors
  • Ruby 3.2/PostgreSQL suite with 1.3.x: 402 runs, 1483 assertions, no failures or errors
  • Ruby 3.0 and 3.2 targeted action/future-execution tests: 41 runs, 131 assertions, no failures or errors on each version
  • RuboCop: 3 changed files inspected, no offenses

Tracking issue: https://projects.theforeman.org/issues/39799

Related: #473 and https://projects.theforeman.org/issues/39353

Use JSON 2 on Ruby versions that cannot install the multi_json release with JSON 3 keyword-argument support. Wait for polling timers to be scheduled before advancing the managed clock.
@adamruzicka

Copy link
Copy Markdown
Contributor

It looks reasonable, could we get some sort of test for the saturated IO executor situation?

@jakduch

jakduch commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

Good point. I added a deterministic regression test in d885a7a. It replaces the global IO executor with a single-thread pool, blocks its only worker, and verifies that world termination still resolves through the watchdog. The test passes with this change and fails against master on the termination wait timeout.

@adamruzicka
adamruzicka merged commit 012510a into Dynflow:master Sep 25, 2026
12 checks passed
@adamruzicka

Copy link
Copy Markdown
Contributor

Thank you @jakduch !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants