Rust Server train: definition routing, worker admission, poll waits, schedule probe by rmcdaniel · Pull Request #675 · durable-workflow/server

rmcdaniel · 2026-10-22T18:13:30Z

One CI run for four reviewed Rust Server #325 changes on current main (17dc12cd, after #672). Each constituent keeps its own description:

Route pinned runs by their recorded definition like PHP; run every fixture check before failing #667: workflow definition routing like Server 2.10.0 (definition_routing read surfaces, routing of pinned runs to compatible workers);
Native: admit worker registrations like Server 2.10.0 #670: worker registration admission and refusals like PHP 2.10.0;
Native: wait DW_WORKER_POLL_TIMEOUT on polls and check operation names like PHP #669: the default worker poll wait from DW_WORKER_POLL_TIMEOUT, plus operation-name validation;
Stop the schedule fixture probe from applying a published PHP write twice #674: the schedule fixture probe no longer applies a published PHP write twice, and its contracts print the responses behind a difference.

The branches merged without conflicts. The train adds one commit: it drops a test module's use super::* that #667 had left unused, so the build has no warnings.

Expected step-10 differences against the frozen 2.5.14 corpus, until its re-baseline:

signals-before-worker: native follows Server 2.10.0's SignalRejected after termination;
worker-listing: process_metrics._worker_protocol_version. Server 2.10.0 also stores and lists this key (WorkflowReplayEligibility::PROTOCOL_METRIC).

After this merges I'll close #667, #669, #670 and #674 with a link here.

Update (Oct 22): the train now also carries #679, which moves the parity reference to published Server 2.11.1 and keeps the catalog comparisons on the frozen 2.5.14 layout. Without it, the Shared Server fixtures jobs fail on signals-before-worker and worker-listing, because native already follows 2.10+ there.

Update: the train also merges #676 (the operator-metrics repair-window contract), so one run qualifies the fixtures together. #676 closes when the train lands on main.

Update: the train also merges #680 (activity lease renewal like Server 2.11) and #681 (service catalog reads like Server 2.11). Both were checked locally against PHP 2.11.1, and their fixtures pass on PHP, embedded and native.

Update: the train also merges #678 (namespace delete). It was re-checked against PHP 2.11.1 and closes when the train lands.

Server 2.10.0's WorkflowDefinitionRouting decides at claim which worker may take a workflow task of a run whose start recorded its worker cohort's definition fingerprint. Native offered such tasks to any worker. Now a worker advertising the recorded fingerprint takes the task. A worker with another definition waits while an eligible peer has the recorded one, then takes it only if it advertises workflow_entry_arguments_validation. Workers without a fingerprint, or runs without a recorded one, get nothing. Eligible peers follow WorkflowReplayEligibility: fresh, on the run's queue and build, supporting the type, and able to replay its cooperative cancellation and metadata history. Registrations and heartbeats now record their protocol version under _worker_protocol_version in the process metrics, as PHP 2.10.0 does; worker listings show it.

Server 2.10.0 reports WorkflowDefinitionRouting's decision for a run's next due ready workflow task as definition_routing in describe, repair and debug (inspect), and samples a queue's ready runs in task queue describe (forQueue). Native now does the same: the recorded context, PHP's status, eligible worker counts by definition, up to ten example workers, blocked, runnable_task and PHP's resolution text. A blocked route is a debug warning (workflow_definition_routing_blocked, diagnostic_status needs_attention) and marks the queue's repair as needing attention.

…outing

The SQLite, PostgreSQL and MySQL fixture steps stopped as soon as native's record of the frozen corpus failed, which skipped every later check in the step: the SDK direction, patch, marker, range and restart profiles. One expected difference (native's Server 2.10.0 SignalRejected in signals-before-worker) therefore hid everything after it. The native record and the comparison now keep their status, the rest of the step runs, and the step fails at its end if either failed. Nothing about the expectations or the frozen artifacts changes.

A workflow or activity poll without timeout_seconds now waits max(0, DW_WORKER_POLL_TIMEOUT) seconds (30 by default), as Server 2.10.0's LongPoller does, instead of 5. A query-task poll starts from the same setting, capped by DW_QUERY_TASK_POLL_TIMEOUT. An explicit null wait takes the default, and a requested wait follows Laravel's integer rule from 0 to 60. Signal, update and query names get PHP's validateOperationName errors: empty, over 256 bytes or with control characters. Native keeps its 255-byte storage guard for signal and update names, which PHP's own columns cannot exceed. A query poll from a worker registered without capabilities answered 503; it now polls empty, as PHP does.

The worker-fencing contract compares whole run reads across a live registration replacement. Native and Server 2.10.0 describe a run with definition_routing, which reports how the next ready workflow task would route now; releasing the lease makes that task runnable without changing the run or its history. The preservation checks now compare the reads without that block, and a unit case covers both a routing-only change and a real run change.

Registration now runs PHP's Laravel request rules, with the capability manifest required from protocol 1.18, and then PHP's capability guards in order: prepared local activities, Nexus and cooperative cancellation, the manifest's flag and floor agreement, workflow metadata capabilities and message streams. Refusals carry PHP's fields plus protocol_version and server_capabilities. An unnamed worker gets a ULID, missing type lists are stored empty, and the manifest is stored normalized. The rule engine gains numeric and required_array_keys, and a nested wildcard rule covers only array items, as Laravel's expansion does. The execution tests' router adds the manifest an SDK sends; the new admission tests use the server's own router.

The SDK query contract compares the run read before and after a query. definition_routing reports how the next ready task would route now, so a worker leaving changes its counts without touching the run or its history.

…outing

Main's Shared Server fixtures fail schedule-policies on native SQLite intermittently, with a different step each time: a 503 on DELETE, a first skip trigger reported as skipped, and a 404 on the second quota trigger. The contract printed only positions and outcomes, so the cause was not in the log. A status difference now names each differing step with its response, and an outcome difference shows the trigger responses with their reasons.

The schedule-policies and schedule-catalog differences on main come from the published PHP reference, not native: no PHP pass line precedes them. PHP on SQLite can report "database is locked" for a write that already committed (the started run's task is queued after the trigger's transaction), and the probe sent the write again. A retried trigger then read as skipped (its first run was running), and a retried quota trigger used the second action, so the next trigger found the schedule deleted. A write is now sent again only when its schedule reads exactly as it did before the write. Otherwise the failure is recorded, and the recorded lock replays the whole sequence in a new namespace, now up to five times. Reads are still retried. Native never reports this lock. In a local run of the CI step on these sources, published PHP passed both schedule fixtures (five safe retries and one replay for schedule-policies), and native matched them on the first attempt.

…rust-325-train-oct22c

…5-train-oct22c

… rust-325-train-oct22c

…o rust-325-train-oct22c

The operator metrics probe reads metrics, then health, in separate requests. Past the redispatch window, published PHP's TaskWatchdog repairs the overdue task and projects the run before TaskDispatcher redispatches it and projects the run again. A health read in between sees the run needing repair (durable_resume_paths); one after it sees the task redispatched. Both are later points of the same PHP repair, not differences from the metrics read before them. For PHP at that stage only, health may now be at the metrics' state or a later point of the repair. Native runs no repair loop and stays held to exactly the state its metrics show. Main's postgres-17 fixtures failed this way on the published PHP reference (run 39815830012).

rmcdaniel · 2026-10-22T19:48:48Z

Merged main 842d58a5 (#666) into the train at d917bef2d; CI restarts there. On the previous head c020a6a19, every failed job failed only in its fixture comparison step. All other steps, including the Rust suites, passed. The differences:

signals-before-worker and worker-listing on mysql-8.0, existing-php-matrix and execution-and-codec. Main shows the same differences.
operator-metrics on postgres-17. That came from the published PHP reference: its repair window falls between the probe's metrics and health reads. Accept published PHP health read during or after its own task repair #676 fixes it.

I'll merge manually once the Rust jobs on d917bef2d are verified.

Record what merged since October 20, the open PRs, the two native fixture differences that follow published Server 2.10.0 against the frozen 2.5.14 reference, the PHP-side operator-metrics window, and the route surface: 97 of published PHP's 175 API routes, with the missing routes by family.

DELETE /api/namespaces/{namespace} (dw namespace:delete, the SDK's deleteNamespace) follows PHP's NamespaceController::destroy over NamespaceLifecycleCleanup. One transaction deletes the namespace's rows in PHP's order, matching each table by the run, instance or namespace columns the verified schema has, and reports each table's non-zero count under PHP's keys. Native also clears its own poll receipts and worker incarnations for the namespace. A missing namespace answers PHP's 404 namespace_not_found; deleting needs an admin, like updating. Native does not delete external payload objects yet, so a namespace that still references one refuses with PHP's 503 external_payload_storage_driver_unavailable before changing anything. The new reviewed namespace-delete fixture runs through the unchanged published PHP SDK: one search attribute definition and one started run, the exact counts published PHP 2.5.14 reports, the 404s afterwards, the SDK's typed refusal of a second delete, an untouched run elsewhere and the freed workflow ID. PHP's asynchronously filled timeline, timer and lineage projections may be counted. Embedded has no HTTP namespaces.

This PR's own postgres-17 run caught published PHP's metrics inside the repair window: the task repaired once and still overdue, and the run projected as needing repair (runs.repair_needed and backlog.repair_needed_runs 1). The contract now models PHP's three states past the redispatch window in order, overdue, repairing and redispatched, each with its counts and warnings. With a repair count, PHP's metrics may show repairing or redispatched; health may show the metrics' state or a later one. Native still has only the state its metrics show, and a run needing repair without a repaired task fails.

Published PHP and its consumers have moved to Server 2.10 and later. Against 2.11.1 three reviewed expectations change: - signals a closed run never applied are recorded as SignalRejected after its terminal event; - every registration stores its protocol version in its process metrics, including one that reports none; - SQLite lock pressure answers 503 backend_lock_pressure, which the schedule probe now retries under its existing rule. Native already behaved as 2.11.1 does. The consumer tuple is unchanged; the frozen 2.5.14 baseline stays for the cohorts that pin it.

… into rust-325-train-oct22c

…ust-325-train-oct22c # Conflicts: # .github/workflows/server-parity.yml # scripts/conformance/server-parity/contract.mjs # scripts/conformance/server-parity/probe.php

…rust-325-train-oct22c # Conflicts: # .github/workflows/server-parity.yml # scripts/conformance/server-parity/contract.mjs # scripts/conformance/server-parity/probe.php

The raw-environment auth probe launches native with a non-text value for each authentication setting and expects an authentication refusal before storage is opened. DW_RUNTIME_CREDENTIALS_ENABLED was read only after the database opened, and DW_AUTH_BACKWARD_COMPATIBLE only through the lossy contract audit, so both reached 'DB_DATABASE is required'. Both are now read before storage selection. Readable values behave as before.

…istration Published Server 2.11.1 on SQLite settles a departing worker's leases with NOWAIT locks while its one-second maintenance loop runs, and answers a contended DELETE /api/worker/registrations with 503 backend_lock_pressure (retryable, nothing recorded). The published Rust SDK worker deregisters once at shutdown without retrying, so the patch-deployment replacement phase failed on the reference in two consecutive runs. The patch probe's HTTP observer now opts in to retrying exactly that documented refusal for non-poll requests, below the SDK, as lockPressureOperationRetryDelay does for PHP clients. Only the final reply is recorded and returned; poll pressure stays visible, and every other user of the observer is unchanged.

The SQLite reference's one-second repair loop can refuse a workflow-task completion with a retryable backend_lock_pressure 503 while the restart probe prepares its runs. The probe retried only polls, so one contended completion aborted the step. It now uses lockPressureOperationRetryDelay: polls keep their no-task check, and any other request is retried only when the server marked its refusal retryable, so nothing was committed.

…probe

rmcdaniel · 2026-10-23T05:24:44Z

Requalifying on 33c212cf6. Main was merged in (d35836c51, required by branch protection), and the namespace-delete fixture probe now retries the reference's retryable lock refusal.

On d35836c51, execution-and-codec failed only on namespace-delete: 503 !== 200. The SQLite PHP reference refused the namespace delete with a retryable backend lock refusal while its repair pass held the database lock; the maintenance container log shows the database is locked trace. The probe's raw HTTP client was the one probe client without the shared lock-pressure retry. Every other check on d35836c51 that had finished had passed.

Since the fixture step runs every later profile before failing, the MySQL jobs take 75 to 85 minutes and were cancelled at their 75-minute limit in each of the last four runs. The PostgreSQL and codec jobs, which share the limit, take about 60.

rmcdaniel · 2026-10-23T08:17:49Z

Requalifying on 59a816fd2. The last run on 06a6d1552 (after merging main's #689) had three failures, none from native behavior:

execution-and-codec: the published-SDK patch profile's Python worker got the published PHP reference's 503 backend_lock_pressure while PHP's maintenance repair pass held the SQLite database (database is locked). This is the same class as the namespace-delete probe fix on this branch.
mysql-execution (mysql-8.0 and existing-php-mysql-matrix): cancelled at their 75-minute limit during the query-authority profile. Since the fixture step runs every later profile before failing, the MySQL jobs need 75 to 85 minutes; they were cancelled the same way in each of the last four runs. PostgreSQL runs the same profiles in about 60 minutes.

59a816fd2 raises the three parity database jobs' limit to 105 minutes. Scenario budgets are unchanged.

rmcdaniel · 2026-10-23T10:09:51Z

Every check passed on 59a816fd2 (26 pass, 1 skipped), including execution-and-codec, all three mysql-execution jobs and both postgres-storage jobs. Main then gained #691, a PHP-only update routing fix, so I merged main into this branch (db7254b3f) to meet the up-to-date rule. No Rust changes. I'll merge manually once the Rust suites pass again on this head.

rmcdaniel · 2026-10-23T11:49:16Z

Checks on db7254b3f: 24 pass, 1 skipped, 2 failures, both reruns of jobs that passed on 59a816fd2 with the same Rust code:

codec-deadline-health (mysql-8.4): a canonical event landed 1.008 s after its deadline, against a 1 s allowance.
mysql-execution (existing-php-mysql-matrix): the python-clock-genuine-one corrected-marker fixture differed once (false !== true) on the Rust target. Every other fixture in that job passed on all targets.

Main's only change in this merge is #691 (PHP). I've rerun both failed jobs and will merge when they pass. If the clock fixture differs again, I'll treat it as a real difference and investigate it before merging.

rmcdaniel · 2026-10-23T13:06:06Z

The rerun on db7254b3f passed both failed jobs. codec-deadline-health (mysql-8.4) passed, and python-clock-genuine-one and the rest of the corrected-marker step passed in mysql-execution (existing-php-mysql-matrix), so the earlier difference was a one-off.

Main has since gained #693, which changes only PHP files (HistoryController::export, ControlPlaneProtocol, a feature test), none under the parity paths. I merged it in (6d0ae5bf0). The native suites pass locally on SQLite: 233 lib and 408 execution tests. The Rust tree is unchanged from the head whose parity run passed. Following the merge-under-runner-scarcity default, I've cancelled the duplicate Shared Server fixtures run on this head. I'll merge once the required checks (PHPUnit feature suite, Sonar) pass.

rmcdaniel · 2026-10-23T13:17:45Z

Main gained #695 (Workflow 2.11.4 pin in composer.json/composer.lock), which is again outside the parity paths. I merged it as fb018211e; the Rust tree is unchanged. I cancelled the duplicate Shared Server fixtures run again, and will merge on the required checks.

sonarqubecloud · 2026-10-23T13:20:57Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.5% Duplication on New Code

See analysis details on SonarQube Cloud

rmcdaniel added 18 commits October 22, 2026 13:58

Merge remote-tracking branch 'origin/main' into rust-325-definition-r…

785d48b

…outing

Register unit-test workers with the new protocol argument

c6f5ddf

Leave the live routing view out of the query preservation check

1950f38

The SDK query contract compares the run read before and after a query. definition_routing reports how the next ready task would route now, so a worker leaving changes its counts without touching the run or its history.

Merge remote-tracking branch 'origin/main' into rust-325-definition-r…

b2999ca

…outing

Merge remote-tracking branch 'origin/main' into rust-325-poll-wait

d08b2d1

Merge remote-tracking branch 'origin/rust-325-worker-admission' into …

3dda863

…rust-325-train-oct22c

Merge remote-tracking branch 'origin/rust-325-poll-wait' into rust-32…

f740110

…5-train-oct22c

Merge remote-tracking branch 'origin/rust-325-sched-diagnostics' into…

544c66a

… rust-325-train-oct22c

Merge remote-tracking branch 'origin/rust-325-definition-routing' int…

437f60d

…o rust-325-train-oct22c

Drop the incarnation test module's unused super import

c020a6a

rmcdaniel mentioned this pull request Oct 22, 2026

Accept published PHP health read during or after its own task repair #676

Merged

Merge remote-tracking branch 'origin/main' into rust-325-train-oct22c

d917bef

rmcdaniel added 4 commits October 22, 2026 19:51

rmcdaniel mentioned this pull request Oct 22, 2026

Rust #325: service catalog reads like Server 2.11 #681

Merged

rmcdaniel added 3 commits October 22, 2026 21:57

Merge remote-tracking branch 'origin/rust-325-activity-lease-renewal'…

1d94aa4

… into rust-325-train-oct22c

Merge remote-tracking branch 'origin/rust-325-service-catalog' into r…

dfb03f6

…ust-325-train-oct22c # Conflicts: # .github/workflows/server-parity.yml # scripts/conformance/server-parity/contract.mjs # scripts/conformance/server-parity/probe.php

Merge remote-tracking branch 'origin/rust-325-namespace-delete' into …

1448737

…rust-325-train-oct22c # Conflicts: # .github/workflows/server-parity.yml # scripts/conformance/server-parity/contract.mjs # scripts/conformance/server-parity/probe.php

rmcdaniel added 2 commits October 23, 2026 00:49

rmcdaniel mentioned this pull request Oct 23, 2026

Start every Server pip install from the patched, bundle-only pip (GHSA-6q48-6c7w-9gw2) #673

Merged

rmcdaniel added 3 commits October 23, 2026 02:18

Merge remote-tracking branch 'origin/main' into rust-325-train-oct22c

d35836c

Retry the reference's retryable lock refusal in the namespace-delete …

33c212c

…probe

rmcdaniel added 2 commits October 23, 2026 06:31

Merge remote-tracking branch 'origin/main' into rust-325-train-oct22c

06a6d15

Merge remote-tracking branch 'origin/main' into rust-325-train-oct22c

db7254b

Merge remote-tracking branch 'origin/main' into rust-325-train-oct22c

6d0ae5b

Merge remote-tracking branch 'origin/main' into rust-325-train-oct22c

fb01821

rmcdaniel merged commit 44fd778 into main Oct 23, 2026
14 of 19 checks passed

rmcdaniel deleted the rust-325-train-oct22c branch October 23, 2026 13:31

This was referenced Oct 23, 2026

Rust Server train: command batches, named namespaces, completion checks, external inputs, operator surface, leases, deployments (#325) #696

Merged

Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325

Open

Navigation Menu

Repository navigation

Rust Server train: definition routing, worker admission, poll waits, schedule probe - #675

Rust Server train: definition routing, worker admission, poll waits, schedule probe#675
rmcdaniel merged 42 commits into
mainfrom
rust-325-train-oct22c

rmcdaniel commented Oct 22, 2026 •

edited

Loading

Uh oh!

rmcdaniel commented Oct 22, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

sonarqubecloud Bot commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rmcdaniel commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rmcdaniel commented Oct 22, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

rmcdaniel commented Oct 23, 2026

Uh oh!

sonarqubecloud Bot commented Oct 23, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rmcdaniel commented Oct 22, 2026 •

edited

Loading