Skip to content

Refuse service redrive of reused steps its SDKs can't replay yet - #1006

Merged
rmcdaniel merged 2 commits into
mainfrom
fix/864-service-redrive-fence
Nov 3, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
fix/864-service-redrive-fence

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Implements the product fence General chose (#864) for Security's 2.19.0 service-mode hold (#864 comment 6614022929). It's a prerequisite for the 2.19.0 release PR #1005.

Problem

A run started by a service worker can be redriven on the same definition when the worker presents a matching fingerprint. Since #1001 and #1002, its successor can reuse MemoUpserted, SearchAttributesUpserted and signal-wait events (SignalWaitOpened, SignalReceived, SignalApplied) with reused_recorded_at. No service-mode SDK replays those reused keys yet (sdk-python#235, sdk-php#244, sdk-rust#172). An SDK worker could replay that successor at the redrive time, and documentation can't make an already-writable history safe.

Change

FailedRunRedriveEligibility::evaluate() refuses with service_redrive_history_not_qualified when:

  • the source's WorkflowStarted.workflow_definition_fingerprint_source is worker, and
  • its plan reuses any memo upsert, search-attribute upsert or signal-wait step.

The refusal runs before the definition decision and before any write. Run detail and describe report the same reason.

Unchanged:

  • embedded runs,
  • worker-sourced reuse of the already-qualified set: activities, side effects, version markers, fired timers and start time.

The fence lifts only through a replay-capability/version check once those SDK issues ship installed-qualified. The CHANGELOG entry under Unreleased says so.

Tests

Local runs, PHP 8.4, SQLite + Redis:

  • DefaultWorkflowControlPlaneRedriveRejectionTest::testAServiceRunRefusesReusedStepsItsSdksAreNotQualifiedToReplay has cases for a memo upsert, a search-attribute upsert and a signal wait. A remote.redrive service run gets that step at sequence 1 and a failed activity at 2. Each case asserts:

    • the typed reason,
    • a byte-identical snapshot of every engine table (instance, run including message_cursor_position, commands, history, tasks, links, memos, search attributes, activities, failures),
    • an identical snapshot of the projections (WorkflowRunSummary, WorkflowTimelineEntry),
    • the describe redrive_blocked_reason.

    All three are accepted on main and refused here.

  • testAServiceRunStillRedrivesAQualifiedReusedSideEffect: a service run reusing a side effect is still accepted, with the reuse keys.

  • Suites: rejection 47/47, V2FailedRunRedriveTest 28/28 (embedded unchanged), V2EmbeddedReplayRegressionCorpusTest 19/19, FailedRunRedrivePlanTest 11/11.

  • Replay corpus: new redrive-service-unqualified-history fixture and scenario. validate-regression-corpus.py --base-ref c5b63c2c passes with replay counterfactual_fixture_paths: 1.

  • ecs and phpstan are clean.

🤖 Generated with Claude Code

…#864)

Redrive of a run a service worker started (WorkflowStarted.
workflow_definition_fingerprint_source = worker) could reuse memo upserts,
search-attribute upserts and signal waits, whose reused_recorded_at
service-mode SDKs do not replay yet. Such a run now refuses with
service_redrive_history_not_qualified before any successor, history,
cursor, row or projection write.

Embedded runs and the already-qualified worker-sourced steps (activities,
side effects, version markers, fired timers, start time) redrive as
before. The fence lifts only through a replay-capability check once
sdk-python#235, sdk-php#244 and sdk-rust#172 ship.

Counterfactuals: a service run reusing each of a memo upsert, a
search-attribute upsert and a signal wait is accepted on main and refused
here with every engine table and projection unchanged; a service run
reusing a side effect is still accepted. Replay corpus evidence:
redrive-service-unqualified-history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

@rmcdaniel

rmcdaniel commented Nov 3, 2026 •

Copy link
Copy Markdown
Member Author

Security final clearance for exact head a154b23.

I independently verified:

  • the product delta is the reviewed worker-provenance fence: a service-worker-started run that would reuse a signal wait, memo upsert, or search-attribute upsert refuses with stable service_redrive_history_not_qualified before the definition decision or any successor/history/cursor/row/projection write;
  • embedded runs and worker-sourced histories using the already-qualified side-effect path remain accepted;
  • the three no-mutation counterfactuals cover signal, memo, and search-attribute histories, and the immutable replay fixture proves the new refusal against main;
  • the only delta after source review was the two-line ECS indentation correction in the corpus test;
  • hosted CI is terminal at this exact SHA: 16 success, 7 intended skips, 0 failure, 0 pending; GitHub reports CLEAN.

Cleared to mark ready and merge only this exact head. Any source/head change returns to review.

This clears the service fence only. Workflow #1005 and the 2.19.0 tag remain held until the release branch is refreshed onto this merge, the fence entry and public notes accurately say worker-sourced runs are refused, and that exact release head is separately reviewed.

@rmcdaniel
rmcdaniel merged commit a2daf6a into main Nov 3, 2026
23 checks passed
@rmcdaniel
rmcdaniel deleted the fix/864-service-redrive-fence branch November 3, 2026 20:21
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security merge-boundary readback for Workflow #1006:

  • origin main is a2daf6a;
  • its parents are prior main c5b63c2 and accepted head a154b23;
  • merge tree 0d083ddf7eefabbb8314d58bdd5307ac49e52b52 exactly equals the accepted head tree.

The reviewed service-mode fence is durably installed without merge-time source drift. Workflow #1005 may now be refreshed onto this merge, but its release head and public notes remain a separate Security gate; no 2.19.0 tag is cleared by this readback.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security forward-gate clarification after exact SDK reviews: service_redrive_history_not_qualified must be lifted by a per-runtime, per-reused-event capability, not by one Boolean derived from closing sdk-python#235 / sdk-php#244 / sdk-rust#172.

  • Rust already authors and compares both search-attribute values and attribute_types, and Workflow stuck. No more job in queue. #173 is converging on valid successor evidence for memo/search/signal reuse.
  • Python currently has no signal-wait primitive. Its draft Phantom Workflows #236 can prove signal delivery/value handling and clock non-leakage, but not replay of an authored signal wait; its new search-attribute comparison also ignores recorded attribute_types today.
  • PHP authors condition waits rather than SignalWaitOpened; its mixed Core signal-wait cancellation path fails sequence matching. Draft Potential Deadlock Issue with Workflow Status Field in Kubernetes Environment #246 is value-only search-attribute hardening and likewise has no declared-type identity.

Therefore keep the 2.19 fence for Python/PHP signal-wait reuse and for any search-attribute history whose complete value+type identity the runtime cannot author and verify. Memo reuse can be qualified separately. Unknown runtime/capability combinations must continue to fail closed. The SDK PR comments carry the exact fixture and source holds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant