Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,20 @@
can be redriven again. A row written by workflow code with no matching
reused upsert step is still refused with `unsupported_visibility_metadata`
(#864).
- Redrive a failed run whose completed steps include a signal wait
(`await('name')`) that received and applied its signal. The successor
records the wait, receipt and application with `reused_from_run_id` and
`reused_recorded_at`, and replay uses the original time, so the workflow
gets the same value without a new signal. Redrive also carries the source's
message cursor, so a consumed message is not delivered again. A wait with a
timeout, cancellation scope or parallel group is refused with
`unsupported_signal_wait`, a signal the source received but never applied
with `unsupported_pending_signal`, and a wait without its receipt and
application with `incomplete_signal_step`. A receipt that names another
run, instance, command or wait is refused with `mismatched_signal_receipt`,
and an externally stored signal payload with
`external_signal_payload_not_supported`. Updates in the completed prefix
are still refused (#864).

## 2.18.0 - 2026-11-03

Expand Down
6 changes: 3 additions & 3 deletions docs/api-stability.md
Original file line number Diff line number Diff line change
Expand Up @@ -749,10 +749,10 @@ class, source file path, or stack trace to replay or handle a failure.
| `TimerScheduled` | `timer_id`, `sequence`, `delay_seconds`, `fire_at`, `timer_kind`, `condition_wait_id`, `condition_key`, `condition_definition_fingerprint`, `signal_wait_id`, `signal_name` | `WorkflowStepHistory`, `QueryStateReplayer`, `RunTimerView`, `ConditionWaits`, `SignalWaits` |
| `TimerFired` | `timer_id`, `sequence`, `delay_seconds`, `fire_at`, `fired_at`, `timer_kind`, `condition_wait_id`, `condition_key`, `condition_definition_fingerprint`, `signal_wait_id`, `signal_name`, `cancellation_scope_id` (candidate 1.20), `reused_from_run_id`, `reused_recorded_at` | `WorkflowStepHistory`, `QueryStateReplayer`, `WorkflowFiberRunner`, `RunTimerView`, `ConditionWaits`, `SignalWaits`, `RedriveReplayClock` |
| `TimerCancelled` | `timer_id`, `sequence`, `delay_seconds`, `fire_at`, `timer_kind`, `condition_wait_id`, `condition_key`, `condition_definition_fingerprint`, `signal_wait_id`, `signal_name`, `cancelled_at` | `WorkflowStepHistory`, `RunTimerView`, `ConditionWaits`, `SignalWaits`, `HistoryTimeline` |
| `SignalReceived` | `workflow_command_id`, `signal_id`, `workflow_instance_id`, `workflow_run_id`, `signal_name`, `signal_wait_id`, `arguments`, `payload_codec` | `SignalWaits`, `RunSignalView`, public and worker history payload consumers |
| `SignalApplied` | `workflow_command_id`, `signal_id`, `signal_name`, `signal_wait_id`, `sequence`, `value` | `WorkflowStepHistory`, `SignalWaits`, `RunSignalView`, `QueryStateReplayer` |
| `SignalReceived` | `workflow_command_id`, `signal_id`, `workflow_instance_id`, `workflow_run_id`, `signal_name`, `signal_wait_id`, `arguments`, `payload_codec`, `reused_from_run_id`, `reused_recorded_at` | `SignalWaits`, `RunSignalView`, public and worker history payload consumers, `RedriveReplayClock` |
| `SignalApplied` | `workflow_command_id`, `signal_id`, `signal_name`, `signal_wait_id`, `sequence`, `value`, `reused_from_run_id`, `reused_recorded_at` | `WorkflowStepHistory`, `SignalWaits`, `RunSignalView`, `QueryStateReplayer`, `WorkflowExecutor`, `WorkflowFiberRunner`, `RedriveReplayClock` |
| `SignalRejected` | `workflow_command_id`, `signal_id`, `signal_name`, `signal_wait_id`, `command_sequence`, `rejection_reason`, `terminal_reason` | `PendingSignalCloser`, `RunSignalView`, `HistoryTimeline` |
| `SignalWaitOpened` | `signal_name`, `signal_wait_id`, `sequence`, `timeout_seconds`, `cancellation_scope_id` (candidate 1.20) | `WorkflowStepHistory`, `SignalWaits`, `RunSignalView`, `HistoryTimeline` |
| `SignalWaitOpened` | `signal_name`, `signal_wait_id`, `sequence`, `timeout_seconds`, `cancellation_scope_id` (candidate 1.20), `reused_from_run_id`, `reused_recorded_at` | `WorkflowStepHistory`, `SignalWaits`, `RunSignalView`, `HistoryTimeline` |
| `SignalWaitCancelled` (candidate 1.20) | `signal_name`, `signal_wait_id`, `sequence`, `timeout_seconds`, `timer_id`, `cancelled_at`, `cancellation_scope` | `ScopedWaitCancellation`, `SignalWaits`, `HistoryTimeline`, scope delivery and worker history consumers |
| `UpdateAccepted` | `workflow_command_id`, `update_id`, `workflow_instance_id`, `workflow_run_id`, `update_name`, `arguments`, `ordering_state`, `queued_behind_command_id`, `queued_behind_command_sequence`, `queued_behind_command_type` | `RunUpdateView`, worker history payload consumers |
| `UpdateRejected` | `workflow_command_id`, `update_id`, `workflow_instance_id`, `workflow_run_id`, `update_name`, `arguments`, `validation_errors` | `RunUpdateView`, `HistoryTimeline`, command-contract projections |
Expand Down
23 changes: 23 additions & 0 deletions src/V2/Support/DefaultWorkflowControlPlane.php
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,9 @@ public function redrive(string $instanceId, string $failedRunId, array $options
],
), null, $command);

// Messages the source consumed stay consumed for the successor.
MessageStreamCursor::transferCursor($source, $continuedRun);

app(MemoUpsertService::class)->copyForRedrive($source, $continuedRun);
app(SearchAttributeUpsertService::class)->copyForRedrive($source, $continuedRun);

Expand All @@ -250,6 +253,26 @@ public function redrive(string $instanceId, string $failedRunId, array $options
? $reused['reused_recorded_at']
: $step['recorded_at'];

if (isset($step['signal_events'])) {
// A completed signal wait: the successor records the
// wait, the receipt and the application without a signal
// row or command to deliver again.
foreach ($step['signal_events'] as $signalEvent) {
$signalPayload = $signalEvent['payload'];
WorkflowHistoryEvent::record($continuedRun, $signalEvent['event_type'], array_merge(
$signalPayload,
[
'reused_from_run_id' => $source->id,
'reused_recorded_at' => is_string($signalPayload['reused_recorded_at'] ?? null)
? $signalPayload['reused_recorded_at']
: $signalEvent['recorded_at'],
],
));
}

continue;
}

if ($step['event_type'] === HistoryEventType::TimerFired) {
// The timer already fired, so the successor records it
// fired with no timer row or task to wait on.
Expand Down
2 changes: 1 addition & 1 deletion src/V2/Support/FailedRunRedriveEligibility.php
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ final class FailedRunRedriveEligibility
* reason: string|null,
* resume_step_sequence: int|null,
* completed: list<array<string, mixed>>,
* reused_steps: list<array{event_type: HistoryEventType, payload: array<string, mixed>, recorded_at: string|null, scheduled_payload?: array<string, mixed>}>
* reused_steps: list<array{event_type: HistoryEventType, payload: array<string, mixed>, recorded_at: string|null, scheduled_payload?: array<string, mixed>, signal_events?: list<array{event_type: HistoryEventType, payload: array<string, mixed>, recorded_at: string|null}>}>
* }|null,
* started_payload: array<string, mixed>
* }
Expand Down
210 changes: 207 additions & 3 deletions src/V2/Support/FailedRunRedrivePlan.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@
* event_type: HistoryEventType,
* payload: array<string, mixed>,
* recorded_at: string|null,
* scheduled_payload?: array<string, mixed>
* scheduled_payload?: array<string, mixed>,
* signal_events?: list<array{event_type: HistoryEventType, payload: array<string, mixed>, recorded_at: string|null}>
* }>
* }
*/
Expand All @@ -45,6 +46,7 @@
$activityEvents = [];
$markerEvents = [];
$timerEvents = [];
$signalEvents = [];
$lastActivityEvent = null;

foreach ($events as $event) {
Expand Down Expand Up @@ -113,6 +115,26 @@
continue;
}

if ($event->event_type === HistoryEventType::MessageCursorAdvanced) {
// Message stream bookkeeping, not a step. Redrive carries the
// source's cursor position to the successor instead.
continue;
}

if (in_array($event->event_type, [
HistoryEventType::SignalWaitOpened,
HistoryEventType::SignalReceived,
HistoryEventType::SignalApplied,
], true)) {
if ($started === null) {
return self::reject('invalid_step_sequence');
}

$signalEvents[] = $event;

continue;
}

if (in_array($event->event_type, [HistoryEventType::TimerScheduled, HistoryEventType::TimerFired], true)) {
$payload = $event->payload;
$sequence = is_array($payload) ? ($payload['sequence'] ?? null) : null;
Expand Down Expand Up @@ -173,7 +195,17 @@
return self::reject('unrecorded_failure_boundary');
}

foreach ([...array_keys($markerEvents), ...array_keys($timerEvents)] as $sequence) {
$signalSteps = self::signalSteps($signalEvents, $run);

if (is_string($signalSteps)) {
return self::reject($signalSteps);
}

foreach ([
...array_keys($markerEvents),
...array_keys($timerEvents),
...array_keys($signalSteps),
] as $sequence) {
if ($sequence >= $boundary) {
return self::reject('invalid_failed_step');
}
Expand All @@ -191,7 +223,13 @@
}
}

if (count($activityEvents) + count($markerEvents) + count($timerEvents) !== $boundary) {
foreach (array_keys($signalSteps) as $sequence) {
if (isset($activityEvents[$sequence]) || isset($markerEvents[$sequence]) || isset($timerEvents[$sequence])) {
return self::reject('duplicate_step_sequence');
}
}

if (count($activityEvents) + count($markerEvents) + count($timerEvents) + count($signalSteps) !== $boundary) {
return self::reject('activity_sequence_gap');
}

Expand All @@ -215,6 +253,14 @@
continue;
}

$signalStep = $signalSteps[$sequence] ?? null;

if (is_array($signalStep)) {
$reusedSteps[] = $signalStep;

continue;
}

$timerStep = $timerEvents[$sequence] ?? null;

if (is_array($timerStep)) {
Expand Down Expand Up @@ -400,6 +446,164 @@
];
}

/**
* Group signal events into completed signal-wait steps keyed by step
* sequence. Only a plain wait (no timeout, cancellation scope or parallel
* group) whose received signal was applied is reusable. A received signal
* the source never applied is refused, because the successor would lose it.
*
* The receipt must belong to the source run and instance, carry the same
* command as the application, and name the same wait. A receipt recorded
* before its wait opened has no wait id yet, which is the only allowed
* difference. External signal payloads are refused, because the successor
* gets no signal row or command that owns the stored reference.
*
* @param list<WorkflowHistoryEvent> $events
*
* @return array<int, array{event_type: HistoryEventType, payload: array<string, mixed>, recorded_at: string|null, signal_events: list<array{event_type: HistoryEventType, payload: array<string, mixed>, recorded_at: string|null}>}>|string
*/
private static function signalSteps(array $events, WorkflowRun $run): array|string

Check warning on line 465 in src/V2/Support/FailedRunRedrivePlan.php

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

This method has 13 returns, which is more than the 3 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_workflow&issues=AaGitYfT-Lu4Ca1PiWgN&open=AaGitYfT-Lu4Ca1PiWgN&pullRequest=1002

Check failure on line 465 in src/V2/Support/FailedRunRedrivePlan.php

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 50 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_workflow&issues=AaGitYfT-Lu4Ca1PiWgO&open=AaGitYfT-Lu4Ca1PiWgO&pullRequest=1002
{
$opened = [];
$applied = [];
$received = [];

foreach ($events as $event) {
$payload = is_array($event->payload) ? $event->payload : [];
$waitId = $payload['signal_wait_id'] ?? null;
$name = $payload['signal_name'] ?? null;

if ($event->event_type === HistoryEventType::SignalReceived) {
$signalId = $payload['signal_id'] ?? null;

if (! is_string($signalId) || $signalId === '' || isset($received[$signalId])) {
return 'unsupported_signal_wait';
}

$received[$signalId] = $event;

continue;
}

$sequence = $payload['sequence'] ?? null;

if (! is_int($sequence) || $sequence < 1 || ! is_string($waitId) || $waitId === '' || ! is_string($name)) {
return 'invalid_step_sequence';
}

if (
($payload['timeout_seconds'] ?? null) !== null
|| ($payload['cancellation_scope_id'] ?? null) !== null
|| ($payload['parallel_group_id'] ?? null) !== null
) {
return 'unsupported_signal_wait';
}

if ($event->event_type === HistoryEventType::SignalWaitOpened) {
if (isset($opened[$sequence])) {
return 'duplicate_step_sequence';
}

$opened[$sequence] = $event;

continue;
}

if (isset($applied[$sequence])) {
return 'duplicate_step_sequence';
}

$applied[$sequence] = $event;
}

$steps = [];
$appliedSignalIds = [];

foreach ($opened as $sequence => $wait) {
$application = $applied[$sequence] ?? null;
$applicationPayload = $application?->payload;
$signalId = is_array($applicationPayload) ? ($applicationPayload['signal_id'] ?? null) : null;
$receipt = is_string($signalId) ? ($received[$signalId] ?? null) : null;

if (
! $application instanceof WorkflowHistoryEvent
|| ! $receipt instanceof WorkflowHistoryEvent
|| $applicationPayload['signal_wait_id'] !== $wait->payload['signal_wait_id']
|| $applicationPayload['signal_name'] !== $wait->payload['signal_name']
) {
return 'incomplete_signal_step';
}

$receiptPayload = is_array($receipt->payload) ? $receipt->payload : [];
$receiptWaitId = $receiptPayload['signal_wait_id'] ?? null;
$commandId = $applicationPayload['workflow_command_id'] ?? null;

if (
($receiptPayload['signal_name'] ?? null) !== $wait->payload['signal_name']
|| ($receiptWaitId !== null && $receiptWaitId !== $wait->payload['signal_wait_id'])
|| ! is_string($commandId)
|| $commandId === ''
|| ($receiptPayload['workflow_command_id'] ?? null) !== $commandId
|| ($receiptPayload['workflow_run_id'] ?? null) !== $run->id
|| ($receiptPayload['workflow_instance_id'] ?? null) !== $run->workflow_instance_id
) {
return 'mismatched_signal_receipt';
}

// One admitted signal is one workflow input: a receipt may satisfy
// only one wait.
if (isset($appliedSignalIds[$signalId])) {
return 'mismatched_signal_receipt';
}

// Every replay path resumes the wait with the application's value,
// so it must be recorded inline.
if (! array_key_exists('value', $applicationPayload)) {
return 'incomplete_signal_step';
}

if (
self::isExternalResult($receiptPayload['arguments'] ?? null)
|| self::isExternalResult($applicationPayload['value'])
) {
return 'external_signal_payload_not_supported';
}

$appliedSignalIds[$signalId] = true;
$signalEvents = [];

foreach ([$wait, $receipt, $application] as $event) {
$eventPayload = $event->payload;
// The task and command are the source run's recording context.
unset($eventPayload['task'], $eventPayload['command']);
$signalEvents[] = [
'event_type' => $event->event_type,
'payload' => $eventPayload,
'recorded_at' => $event->recorded_at?->toIso8601String(),
];
}

$steps[$sequence] = [
'event_type' => HistoryEventType::SignalApplied,
'payload' => $signalEvents[2]['payload'],
'recorded_at' => $signalEvents[2]['recorded_at'],
'signal_events' => $signalEvents,
];
}

if (count($applied) !== count($opened)) {
return 'incomplete_signal_step';
}

foreach (array_keys($received) as $signalId) {
if (! isset($appliedSignalIds[$signalId])) {
return 'unsupported_pending_signal';
}
}

return $steps;
}

private static function isExternalResult(mixed $result): bool
{
return (is_array($result) && isset($result['external_storage']))
Expand Down
6 changes: 6 additions & 0 deletions src/V2/Support/HistoryEventPayloadContract.php
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,8 @@ final class HistoryEventPayloadContract
'signal_wait_id',
'arguments',
'payload_codec',
'reused_from_run_id',
'reused_recorded_at',
],
'SignalRejected' => [
'workflow_command_id',
Expand All @@ -494,6 +496,8 @@ final class HistoryEventPayloadContract
'parallel_group_size',
'parallel_group_index',
'parallel_group_path',
'reused_from_run_id',
'reused_recorded_at',
],
'SignalWaitOpened' => [
'signal_name',
Expand All @@ -507,6 +511,8 @@ final class HistoryEventPayloadContract
'parallel_group_size',
'parallel_group_index',
'parallel_group_path',
'reused_from_run_id',
'reused_recorded_at',
],
'UpdateAccepted' => [
'workflow_command_id',
Expand Down
2 changes: 1 addition & 1 deletion src/V2/Support/QueryStateReplayer.php
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ public function replayState(WorkflowRun $run): ReplayState
$this->syncWorkflowCursor($workflow, $sequence + 1);
$current = $workflowExecution->send(
$this->signalValue($signalEvent, $run),
$signalEvent->recorded_at,
RedriveReplayClock::recordedAt($signalEvent),
);

++$sequence;
Expand Down
5 changes: 4 additions & 1 deletion src/V2/Support/WorkflowExecutor.php
Original file line number Diff line number Diff line change
Expand Up @@ -1437,7 +1437,10 @@ private function resumeSignal(
$sequence = &$frame->sequence;
try {
$this->syncWorkflowCursor($workflow, $sequence + 1);
$current = $workflowExecution->send($this->signalValue($signalEvent, $run), $signalEvent->recorded_at);
$current = $workflowExecution->send(
$this->signalValue($signalEvent, $run),
RedriveReplayClock::recordedAt($signalEvent),
);
} catch (Throwable $throwable) {
$this->failRun($run, $task, $throwable, 'workflow_run', $run->id);
return null;
Expand Down
Loading
Loading