Skip to content

Add bounded activity lease recovery and runtime renewal - #879

Merged
rmcdaniel merged 26 commits into
mainfrom
fix/878-activity-lease
Oct 21, 2026
Merged

rmcdaniel merged 26 commits into
mainfrom
fix/878-activity-lease

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Scope

Refs #878. The Core source component is merged; this does not close the issue or claim that published workers automatically renew their activity leases.

Qualified candidate 3b3018a56f0fad5c953a6b02aceb5b626c624cb8 merged as 39413974d68ea32ce246dc1230171dc3468ce249 at 07:37:26 UTC, exact reviewed/tested tree c8175dc77e93f70352781ff89c3bf773462a1b65. Every applicable candidate check passes, including build 39556756587, quality, MySQL feature/unit-contract/corpus/tracing and independent Sonar/boundary. Sonar issue API is independently zero OPEN/CONFIRMED; the two documented method-count acceptances remain acceptances. Fresh-main full qualification is separate and pending. No version/tag/package/image was published and no native/frozen adoption follows. Earlier failed originals below remain evidence, not rescued runs.

Original workflow deadline correction

Two new unit counterexamples prove that a live activity lease previously overrode an elapsed original run/execution deadline. Two actual Server controls also reproduce a deadline expiring during external result resolution: preflight alone returned success after the bound. The shared authority check now receives the original locked run in application heartbeats, runtime renewal and every publication check, including after payload preparation. It preserves UTC6 boundaries and returns run_timeout / execution_timeout without moving the original clock.

The exact source component passes 187 Core bridge/portable-cancellation/claim/authority cases / 1,311 assertions on file-backed SQLite with shared Redis. Its explicit Server source cohort passes 170 cases / 4,594 assertions. Focused PHPStan, ECS across every changed PHP file, Server Pint and diff checks pass. Selections overlap; these are not published or all-database acceptance.

The preceding d863 hosted ECS failure is a real formatting refusal, now corrected. Its MySQL portable-abandon test recorded a one-attempt limit but expected a second after lease loss. The same failure is reproduced locally. Recovery now explicitly records two attempts, with a complementary one-attempt exhaustion test preserving failed history and parent state; both pass (2 cases / 27 assertions) within the broader selection above. The first local combined invocation inherited an in-memory Testbench database across resets; another inherited the repository's unused PostgreSQL defaults. The normal file-backed/shared-Redis harness passes with actual environment propagation. No production retry limit or assertion is relaxed. Original failed CI remains retained; new whole-head checks qualify independently.

Latest source correction and transport

The original claim budget incorrectly froze the application heartbeat deadline even after confirmed, same-attempt progress. Test-first 33096bf2 retains two failures. Corrected 80f9ec32 passes 42 claim cases / 124 assertions and seven actual SQLite bridge cases / 54 assertions, plus focused ECS/PHPStan. It preserves immutable close, schedule-to-close, run and execution deadlines while observing only newer, exactly recorded heartbeat progress within the original live owner/attempt. Runtime renewal does not manufacture progress. The five affected source/interface/test files are identical after the current main integration; no complete-tree pass is transferred.

Server #564 prepares the optional authenticated renewal route, effective lease mapping, original workflow deadline projection and preflight refusal. Its earlier explicit source cohort and locked compatibility results remain component evidence. Python #183 now implements capability-gated automatic maintenance with 65 focused and an overlapping 557 passing / two skipped combined selection. Neither draft adopts a published new Core or proves released long-activity acceptance.

The previous #879 head's repository-contract job failed at its durable replay/codec evidence step. That failed original is retained. With dependencies installed, the current preparation reproduced the precise refusal: replay evidence had not grown (68 before and after). Two new immutable histories now cover a lease-loss retry followed by success and a terminal lease-loss failure caught by workflow code before recovery. Results use the official Avro serializer.

Both new histories pass cold replay (2 tests / 47 assertions) and fail the preceding history contract on the three new diagnostic fields. The complete unit replay consumer passes 67 tests / 2,496 assertions. The full official regression-policy command also passes against 9776f4a4, with counterfactual enforcement enabled: replay inventory grows 68 to 70, and both new fixture paths independently fail the base consumer and pass the current consumer. No policy or expectation was relaxed. The first structural invocation lacked dependencies; the first full invocation used a borrowed PHP image without Redis support. Those setup failures are separate from the passing PHP 8.3.35 / Redis / file-backed SQLite consumer run. Fresh hosted whole-head checks remain independent and pending.

Recorded recovery

An expired running activity lease consumes the recorded retry policy instead of silently putting the same task back on the ready queue. Record the original attempt and lease loss, schedule the recorded backoff only while attempts remain, and record ActivityFailed when exhausted. Late-result fencing remains. Lease loss is distinct from declared activity timeouts and does not establish that the old callback stopped.

The current history contract explicitly documents optional lease-loss diagnostics (failure_reason, lease_expires_at, lease_owner) on retry/failure records. Existing required keys retain their meanings; no frozen conformance/native schema changes. Wrong-owner refusal, repeat settlement, visible storage errors and atomic rollback are covered.

Repair can already retain a run/task lock. It now refuses contested attempt/execution locks rather than waiting against a worker that needs that run. Ordinary worker/timeout enforcement keeps its canonical lock order and waiting behavior.

Runtime renewal role

Add the optional ActivityLeaseRenewal contract rather than changing required methods on custom ActivityTaskBridge implementations. The default bridge validates the live original owner/attempt, acknowledges the current lease, and refuses expired, replaced, cancelled or timed-out authority. Renewal writes no history or application heartbeat and changes none of the authored execution/progress deadlines. Read-only status remains read-only.

The configured embedded duration reaches claims, heartbeats and discovery through one resolver. DW_V2_ACTIVITY_TASK_LEASE_SECONDS accepts integers 1–86400; invalid values retain the existing 300-second default. A shorter configured duration cannot shorten an already held lease. The live operator view shows the current lease only for the same still-running original attempt; terminal history remains authoritative.

Earlier component evidence

  • Tests-first 325358d9 fails both original retry-policy assertions. bcc31941 proves the configured 19-second claim still receives 300 seconds and records four missing-renewal-API errors before implementation.
  • ebbee1dd reproduces an actual PostgreSQL two-connection repair wait of 5.015 seconds while retaining the run lock. The corrected bounded-refusal control passes.
  • Broad bridge/lease/repair/operator-view selection: 116 cases / 1541 assertions PASS using the normal shared Redis feature-harness configuration.
  • Recovery/history-contract/timeout replay/timeout feature selection: 75 cases / 543 assertions PASS after the lock correction; one explicit row-locking-only case skipped on SQLite. These selections overlap and are not a unique-case total.
  • PostgreSQL 17.11 and MySQL 8.0.46: 17 focused lease/configuration/recovery controls / 79 assertions per backend PASS. Five additional PostgreSQL heartbeat-timeout and worker/watchdog concurrency cases / 107 assertions PASS.
  • Focused PHPStan, ECS and git diff --check PASS. Normal hosted checks for this exact whole head remain independent.

Earlier observations remain failed/interrupted: missing SQLite/Redis preparation, initially undocumented diagnostic keys, a mutable test-clock comparison, timeout-sweep diagnostic ownership, and a Carbon/string presentation comparison. The corrected presentation comparison then proves the actual stale operator-lease value. An older recovery fixture assumed unbudgeted redispatch; it now explicitly records two attempts and zero backoff and claims the recorded replacement task, retaining all stale/duplicate-result assertions. One broader SQLite heartbeat run failed with a database lock while the feature harness used process-local cache; the standard shared-cache run is separate and does not establish the failed run's cause. A local invocation was stopped to isolate parallel fixtures' Redis databases; it is not a pass.

Retained source verification and remaining delivery

The preceding 0014655d hosted build and quality checks pass, but its independent PR issue API review reported 11 OPEN Sonar findings. Current 4d25f6a26ee9931f9b78f5cd4609381cb0cb1fe9 / tree b82cc583ea0bcac60597042d6974b625e67616a4 addresses the authority, claim, row-lock, observation, timeout and repair reports with smaller source methods. Original transaction/lock order, deadline priority, owner/generation fences, retry counts, history and application-progress separation are retained. No rule/profile/threshold changed; fresh analysis must establish dispositions.

Actual refactored-source verification on PHP 8.3.35 / Laravel 13.38.0 / PHPUnit 12.5.40: 257 tests / 1,825 assertions PASS, with one explicit network-only skip on SQLite. PostgreSQL 17.11 and MySQL 8.0.46 each pass 13 tests / 123 assertions, including actual two-connection repair and heartbeat/timeout locking, original renewal identity, retry exhaustion and history rollback. ECS, focused PHPStan and diff checks pass. Both task databases stopped with exit 0 / no OOM and were removed. Fresh normal hosted gates qualify this whole tree independently; no earlier pass is carried across heads.

The preceding bc03c96d also extracts the remaining nested lock-mode expression. Its Sonar workflow and app gate pass; independent issue review has zero OPEN/CONFIRMED. Two method-count findings are individually ACCEPTED: AaFc9-3mhJ6JuRzzzgyK (enforcer 24 methods) and AaFc9-9ThJ6JuRzzzgyL (repair 22 methods). These private transition helpers belong to their atomic owners; splitting them solely for the rule's 20-method count would not create an independent responsibility. The technical reasons are recorded on the findings. These are accepted tradeoffs, not fixed findings, and no rule/profile/threshold changed. Fresh analysis must independently establish the new head's result.

Worker-departure follow-through

Separate preparation 3b3018a56f0fad5c953a6b02aceb5b626c624cb8 / tree c8175dc77e93f70352781ff89c3bf773462a1b65 adds original-worker revocation, recorded retry settlement and optional lease_revoked_at history diagnostics, then integrates current test-only main 22b2651a. Production bytes match the tested departure component 52650566 / docs successor 16e5f2ab. Actual broad Core verification again passes 257 / 1,825; the explicit Server component passes SQLite and real PostgreSQL/MySQL deregistration, stale receipt, exhaustion, deadline-priority, rollback and two-connection contention controls. Exact counts, failed originals and limits are recorded in #878's component receipt.

This source is now merged at the identity above after its own build 39556756587, quality 39556745148, Sonar 39556756560 and boundary 39556756515 pass. Fresh main must qualify separately. The separate Server preparation contains qualified tracing main and its published Core 2.9.0 lock; it has not adopted the new lease API from a published package. Source overlay tests remain distinct. All substantive Core/Server component resources and dependencies are verified absent at 06:10:14 UTC.

  • Finish the prepared Server renewal/discovery/configuration transport through its dependency, documentation and exact-artifact qualification.
  • PHP/Python/Rust worker and embedded runtime maintenance, independent of user heartbeat calls.
  • Publication budgets that advance liveness only after actual same-attempt acknowledgement while preserving original business deadlines and generation fences.
  • Full source/database/race qualification, including delayed renewal, completion, worker loss and rollback.
  • Published activity beyond the original five-minute lease plus killed-worker recovery: no healthy overlapping redelivery, bounded recorded attempts/backoff, retained history and normal installed consumers.
  • Full exact-artifact release acceptance. No current image, native capability or completed Long activities without heartbeats are re-run every 5 minutes and never complete #878 acceptance is claimed.

@rmcdaniel rmcdaniel changed the title Respect recorded activity retry policy after lease loss Add bounded activity lease recovery and runtime renewal Oct 20, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

The MySQL rollback case needs a fixture correction before full database qualification. On the unchanged #879 source, testExpiredActivityLeaseSettlementRollsBackWhenItsHistoryCannotBeRecorded fails because its injected history-write error matches only double-quoted SQL; MySQL uses backticks, so the storage failure is never injected.

Prepared 1c4a0bad on prep/878-mysql-rollback-portability matches the INSERT/table independently of identifier quoting. It retains the original storage-error, state-rollback, task-count and no-history assertions. Actual MySQL 8.0.46 and SQLite each pass this case with four assertions; focused ECS and diff checks pass. No runtime code changes, no complete #878 delivery claim, and no replacement full matrix is submitted. Earlier 17-case network selections did not include this rollback case.

The isolated temporary database/tool resources are fully removed; both database and cache processes exit 0 without OOM. The admitted #879 source/checks remain evidence for their exact original head until this prepared correction is integrated with the next source component.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Prepared renewal-budget component: 422310a4, on prep/878-confirmed-lease-budget. The existing PR #879 head remains unchanged. This component is not an automatic-renewal implementation or a release claim.

ActivityTaskClaim now captures the original task/run/execution/attempt/owner and accepts only an actual successful renewal acknowledgement for that identity. The request's monotonic send clock anchors its remaining lease budget, so transport latency consumes the grant. Strict UTC timestamps, original run/execution deadlines, stale or mismatched replies, conflicting duplicates, expired claims and replies older than the known original grant are fenced. Renewal cannot extend business deadlines or mutate history. The interface documents the required acknowledgement fields and request-clock capture.

The test-first component retains 21 missing-method errors and two real missing-deadline failures on the preceding implementation. A separate adversarial control then reproduced accepting an acknowledgement older than the original lease grant. The corrected runtime 441c7ba passes 33 focused tests / 101 assertions, focused ECS and PHPStan. The final interface comment is documentation only. No full suite, network database, automatic Worker renewal or published-artifact acceptance is claimed for this preparation.

Remaining work includes independent embedded and SDK lease maintenance, Server renewal/auth/claim fencing and configuration wiring, application-progress heartbeat handling, real worker-loss and publication races, and the published six-minute activity/kill-recovery acceptance. Runtime lease maintenance must not fabricate user-progress heartbeats or revive an expired claim. The original #878 acceptance criteria remain open.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 21, 2026 07:37
@rmcdaniel
rmcdaniel merged commit 3941397 into main Oct 21, 2026
20 checks passed
@rmcdaniel
rmcdaniel deleted the fix/878-activity-lease branch October 21, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants