Repository navigation
Add bounded activity lease recovery and runtime renewal - #879
Conversation
|
The MySQL rollback case needs a fixture correction before full database qualification. On the unchanged #879 source, Prepared The isolated temporary database/tool resources are fully removed; both database and cache processes exit 0 without OOM. The admitted #879 source/checks remain evidence for their exact original head until this prepared correction is integrated with the next source component. |
|
Prepared renewal-budget component: 422310a4, on prep/878-confirmed-lease-budget. The existing PR #879 head remains unchanged. This component is not an automatic-renewal implementation or a release claim. ActivityTaskClaim now captures the original task/run/execution/attempt/owner and accepts only an actual successful renewal acknowledgement for that identity. The request's monotonic send clock anchors its remaining lease budget, so transport latency consumes the grant. Strict UTC timestamps, original run/execution deadlines, stale or mismatched replies, conflicting duplicates, expired claims and replies older than the known original grant are fenced. Renewal cannot extend business deadlines or mutate history. The interface documents the required acknowledgement fields and request-clock capture. The test-first component retains 21 missing-method errors and two real missing-deadline failures on the preceding implementation. A separate adversarial control then reproduced accepting an acknowledgement older than the original lease grant. The corrected runtime 441c7ba passes 33 focused tests / 101 assertions, focused ECS and PHPStan. The final interface comment is documentation only. No full suite, network database, automatic Worker renewal or published-artifact acceptance is claimed for this preparation. Remaining work includes independent embedded and SDK lease maintenance, Server renewal/auth/claim fencing and configuration wiring, application-progress heartbeat handling, real worker-loss and publication races, and the published six-minute activity/kill-recovery acceptance. Runtime lease maintenance must not fabricate user-progress heartbeats or revive an expired claim. The original #878 acceptance criteria remain open. |
|



Scope
Refs #878. The Core source component is merged; this does not close the issue or claim that published workers automatically renew their activity leases.
Qualified candidate
3b3018a56f0fad5c953a6b02aceb5b626c624cb8merged as39413974d68ea32ce246dc1230171dc3468ce249at 07:37:26 UTC, exact reviewed/tested treec8175dc77e93f70352781ff89c3bf773462a1b65. Every applicable candidate check passes, including build39556756587, quality, MySQL feature/unit-contract/corpus/tracing and independent Sonar/boundary. Sonar issue API is independently zero OPEN/CONFIRMED; the two documented method-count acceptances remain acceptances. Fresh-main full qualification is separate and pending. No version/tag/package/image was published and no native/frozen adoption follows. Earlier failed originals below remain evidence, not rescued runs.Original workflow deadline correction
Two new unit counterexamples prove that a live activity lease previously overrode an elapsed original run/execution deadline. Two actual Server controls also reproduce a deadline expiring during external result resolution: preflight alone returned success after the bound. The shared authority check now receives the original locked run in application heartbeats, runtime renewal and every publication check, including after payload preparation. It preserves UTC6 boundaries and returns
run_timeout/execution_timeoutwithout moving the original clock.The exact source component passes 187 Core bridge/portable-cancellation/claim/authority cases / 1,311 assertions on file-backed SQLite with shared Redis. Its explicit Server source cohort passes 170 cases / 4,594 assertions. Focused PHPStan, ECS across every changed PHP file, Server Pint and diff checks pass. Selections overlap; these are not published or all-database acceptance.
The preceding
d863hosted ECS failure is a real formatting refusal, now corrected. Its MySQL portable-abandon test recorded a one-attempt limit but expected a second after lease loss. The same failure is reproduced locally. Recovery now explicitly records two attempts, with a complementary one-attempt exhaustion test preserving failed history and parent state; both pass (2 cases / 27 assertions) within the broader selection above. The first local combined invocation inherited an in-memory Testbench database across resets; another inherited the repository's unused PostgreSQL defaults. The normal file-backed/shared-Redis harness passes with actual environment propagation. No production retry limit or assertion is relaxed. Original failed CI remains retained; new whole-head checks qualify independently.Latest source correction and transport
The original claim budget incorrectly froze the application heartbeat deadline even after confirmed, same-attempt progress. Test-first
33096bf2retains two failures. Corrected80f9ec32passes 42 claim cases / 124 assertions and seven actual SQLite bridge cases / 54 assertions, plus focused ECS/PHPStan. It preserves immutable close, schedule-to-close, run and execution deadlines while observing only newer, exactly recorded heartbeat progress within the original live owner/attempt. Runtime renewal does not manufacture progress. The five affected source/interface/test files are identical after the current main integration; no complete-tree pass is transferred.Server #564 prepares the optional authenticated renewal route, effective lease mapping, original workflow deadline projection and preflight refusal. Its earlier explicit source cohort and locked compatibility results remain component evidence. Python #183 now implements capability-gated automatic maintenance with 65 focused and an overlapping 557 passing / two skipped combined selection. Neither draft adopts a published new Core or proves released long-activity acceptance.
The previous #879 head's repository-contract job failed at its durable replay/codec evidence step. That failed original is retained. With dependencies installed, the current preparation reproduced the precise refusal: replay evidence had not grown (68 before and after). Two new immutable histories now cover a lease-loss retry followed by success and a terminal lease-loss failure caught by workflow code before recovery. Results use the official Avro serializer.
Both new histories pass cold replay (2 tests / 47 assertions) and fail the preceding history contract on the three new diagnostic fields. The complete unit replay consumer passes 67 tests / 2,496 assertions. The full official regression-policy command also passes against
9776f4a4, with counterfactual enforcement enabled: replay inventory grows 68 to 70, and both new fixture paths independently fail the base consumer and pass the current consumer. No policy or expectation was relaxed. The first structural invocation lacked dependencies; the first full invocation used a borrowed PHP image without Redis support. Those setup failures are separate from the passing PHP 8.3.35 / Redis / file-backed SQLite consumer run. Fresh hosted whole-head checks remain independent and pending.Recorded recovery
An expired running activity lease consumes the recorded retry policy instead of silently putting the same task back on the ready queue. Record the original attempt and lease loss, schedule the recorded backoff only while attempts remain, and record
ActivityFailedwhen exhausted. Late-result fencing remains. Lease loss is distinct from declared activity timeouts and does not establish that the old callback stopped.The current history contract explicitly documents optional lease-loss diagnostics (
failure_reason,lease_expires_at,lease_owner) on retry/failure records. Existing required keys retain their meanings; no frozen conformance/native schema changes. Wrong-owner refusal, repeat settlement, visible storage errors and atomic rollback are covered.Repair can already retain a run/task lock. It now refuses contested attempt/execution locks rather than waiting against a worker that needs that run. Ordinary worker/timeout enforcement keeps its canonical lock order and waiting behavior.
Runtime renewal role
Add the optional
ActivityLeaseRenewalcontract rather than changing required methods on customActivityTaskBridgeimplementations. The default bridge validates the live original owner/attempt, acknowledges the current lease, and refuses expired, replaced, cancelled or timed-out authority. Renewal writes no history or application heartbeat and changes none of the authored execution/progress deadlines. Read-only status remains read-only.The configured embedded duration reaches claims, heartbeats and discovery through one resolver.
DW_V2_ACTIVITY_TASK_LEASE_SECONDSaccepts integers 1–86400; invalid values retain the existing 300-second default. A shorter configured duration cannot shorten an already held lease. The live operator view shows the current lease only for the same still-running original attempt; terminal history remains authoritative.Earlier component evidence
325358d9fails both original retry-policy assertions.bcc31941proves the configured 19-second claim still receives 300 seconds and records four missing-renewal-API errors before implementation.ebbee1ddreproduces an actual PostgreSQL two-connection repair wait of 5.015 seconds while retaining the run lock. The corrected bounded-refusal control passes.git diff --checkPASS. Normal hosted checks for this exact whole head remain independent.Earlier observations remain failed/interrupted: missing SQLite/Redis preparation, initially undocumented diagnostic keys, a mutable test-clock comparison, timeout-sweep diagnostic ownership, and a Carbon/string presentation comparison. The corrected presentation comparison then proves the actual stale operator-lease value. An older recovery fixture assumed unbudgeted redispatch; it now explicitly records two attempts and zero backoff and claims the recorded replacement task, retaining all stale/duplicate-result assertions. One broader SQLite heartbeat run failed with a database lock while the feature harness used process-local cache; the standard shared-cache run is separate and does not establish the failed run's cause. A local invocation was stopped to isolate parallel fixtures' Redis databases; it is not a pass.
Retained source verification and remaining delivery
The preceding
0014655dhosted build and quality checks pass, but its independent PR issue API review reported 11 OPEN Sonar findings. Current4d25f6a26ee9931f9b78f5cd4609381cb0cb1fe9/ treeb82cc583ea0bcac60597042d6974b625e67616a4addresses the authority, claim, row-lock, observation, timeout and repair reports with smaller source methods. Original transaction/lock order, deadline priority, owner/generation fences, retry counts, history and application-progress separation are retained. No rule/profile/threshold changed; fresh analysis must establish dispositions.Actual refactored-source verification on PHP 8.3.35 / Laravel 13.38.0 / PHPUnit 12.5.40: 257 tests / 1,825 assertions PASS, with one explicit network-only skip on SQLite. PostgreSQL 17.11 and MySQL 8.0.46 each pass 13 tests / 123 assertions, including actual two-connection repair and heartbeat/timeout locking, original renewal identity, retry exhaustion and history rollback. ECS, focused PHPStan and diff checks pass. Both task databases stopped with exit 0 / no OOM and were removed. Fresh normal hosted gates qualify this whole tree independently; no earlier pass is carried across heads.
The preceding
bc03c96dalso extracts the remaining nested lock-mode expression. Its Sonar workflow and app gate pass; independent issue review has zero OPEN/CONFIRMED. Two method-count findings are individually ACCEPTED:AaFc9-3mhJ6JuRzzzgyK(enforcer 24 methods) andAaFc9-9ThJ6JuRzzzgyL(repair 22 methods). These private transition helpers belong to their atomic owners; splitting them solely for the rule's 20-method count would not create an independent responsibility. The technical reasons are recorded on the findings. These are accepted tradeoffs, not fixed findings, and no rule/profile/threshold changed. Fresh analysis must independently establish the new head's result.Worker-departure follow-through
Separate preparation
3b3018a56f0fad5c953a6b02aceb5b626c624cb8/ treec8175dc77e93f70352781ff89c3bf773462a1b65adds original-worker revocation, recorded retry settlement and optionallease_revoked_athistory diagnostics, then integrates current test-only main22b2651a. Production bytes match the tested departure component52650566/ docs successor16e5f2ab. Actual broad Core verification again passes 257 / 1,825; the explicit Server component passes SQLite and real PostgreSQL/MySQL deregistration, stale receipt, exhaustion, deadline-priority, rollback and two-connection contention controls. Exact counts, failed originals and limits are recorded in #878's component receipt.This source is now merged at the identity above after its own build
39556756587, quality39556745148, Sonar39556756560and boundary39556756515pass. Fresh main must qualify separately. The separate Server preparation contains qualified tracing main and its published Core 2.9.0 lock; it has not adopted the new lease API from a published package. Source overlay tests remain distinct. All substantive Core/Server component resources and dependencies are verified absent at 06:10:14 UTC.