Skip to content

Fix queue=<int> deadlock when three or more state runs start together - #70614

Open
pujitha24 wants to merge 1 commit into
saltstack:masterfrom
pujitha24:auto/issue-70580
Open

pujitha24 wants to merge 1 commit into
saltstack:masterfrom
pujitha24:auto/issue-70580

Conversation

@pujitha24

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes a deadlock in check_prior_running_states() where three or more simultaneous state runs with queue= wait on each other forever. A running job that has a queue flag file is treated as waiting and blocks only higher JIDs.

What issues does this PR fix or reference?

Fixes #70580

Previous Behavior

With three simultaneous queue= runs, after the first finishes the other two block on each other indefinitely.

New Behavior

Waiting jobs only block younger jobs, so runs execute in JID order.

Merge requirements satisfied?

  • Docs
  • Changelog
  • Tests written/updated (fails before, passes after; live-minion repro not run)

Commits signed with GPG?

No (DCO sign-off only)

This PR was written in part with the assistance of generative AI.

Fixes #70580

Motivation: check_prior_running_states() treats every running state.* job
with a PID as blocking regardless of JID order. A job waiting in the
queue=<int> path (state._wait) is itself a running job with a PID, so once
the first run finishes, the remaining two waiting jobs block on each other
forever. Two concurrent runs are unaffected; three or more deadlock.

Approach: a running job that has a queue flag file (written by
state._set_queue_flag and removed before the state run starts) is treated
like a queued placeholder: it blocks only jobs with a higher JID. Jobs
without a flag keep blocking unconditionally, preserving the behaviour
added for the concurrent-run race.

Validation: added a unit test with two waiting jobs and flag files. It
fails without the change and passes with it. Ran
`pytest tests/pytests/unit/modules/state/test_state.py
tests/pytests/unit/utils/test_state.py` (68 passed). The end-to-end
multi-job reproduction on a live minion was not run here.

Report: saltstack#70580
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>

@twangboy twangboy left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the pre-commit and lint failures. Also, open this against the 3008.x branch, please.

@twangboy twangboy added the test:full Run the full test suite label Oct 30, 2026
@twangboy twangboy added this to the Argon v3008.7 milestone Oct 30, 2026

This branch was previously deployed

1 inactive deployment
ci — 0cd82a31 Deployed Oct 30, 2026 by twangboy via Build Onedir Packages / RPM (x86_64) #28183
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: state.apply queue=<int> deadlocks when three or more state runs start at once on a minion (regression from #69826)

2 participants