Reduce worker execution allocation and copying

[vigoo]

Resolves GOL-482

Summary

Reduce avoidable allocation and copying in worker execution while preserving retry, replay, ordering, authorization, and durability semantics:

• Broadcast completion events through private Arc<Event> storage; unrelated subscribers inspect borrowed events and only matching consumers clone their result.
• Copy S3 upload bytes once and share them across retries; borrow blob mutation inputs across durable retries.
• Retain oplog status snapshots, construct plugin metadata only when sending, and allocate plugin oplog buffers only while an active plugin or outstanding delivery needs them. Preserve checkpoint/retry behavior and whole-range storage fallback.
• Consume claimed scheduler actions instead of cloning them, and serialize scheduled actions once before metric observation and storage retries. Exercise memory, SQLite, and PostgreSQL storage.
• Move disposable oplog request/response payloads into their caches and cache primary-oplog metric labels at state construction.
• Add focused regressions for replay recovery, retry input identity, cache activation/deactivation, serialized storage, payload thresholds, and event consumers.

The implementation and review follow-ups are retained as 14 separate commits. Every implementation step and the combined integration received an Oracle PASS; the final bounded bug-finder run reported no bugs found.

Measurements

An isolated event-broadcast harness compares the previous owned-event design with shared events. The harness was optimized with -C opt-level=3 (dependencies used the dev profile), counts allocations and requested bytes, and makes received events and matching cloned outputs observable with black_box. Results were identical across three repetitions.

Workload	Owned events: allocations / requested bytes	Shared events: allocations / requested bytes
1 MiB payload, 16 unrelated subscribers	48 / 16,777,632	1 / 304
1 MiB payload, one matching subscriber	4 / 2,097,178	2 / 1,048,880
64-byte payload, one unrelated subscriber	3 / 90	1 / 304
No consumers	0 / 0	1 / 304

Shared events have a fixed allocation cost: they can request more bytes for tiny payloads or zero consumers. These are allocation measurements, not end-to-end latency or fully release-optimized benchmark claims.

The 10,000-append plugin control verified no cache without delivery demand versus 10,000 cached entries with an active plugin. Debug timings are not evidence of a reliable speedup. Other copy removals are supported by source inspection and ownership/pointer-preservation tests, not runtime performance claims.

Reproduce the event measurement:

CARGO_BUILD_JOBS=4 CARGO_PROFILE_DEV_DEBUG=0 CARGO_INCREMENTAL=0 \
  cargo rustc -p golem-worker-executor --profile dev \
  --bench event_broadcast_allocations -- -C opt-level=3
# Execute the resulting event_broadcast_allocations binary in target/debug/deps.

Local verification

• Executor library filters services::events services::oplog services::scheduler services::blob_store: 154 passed, 0 failed.
• Service-base library suite, including S3 retry tests: 45 passed, 0 failed.
• 13 distinct integration tests passed: blob mutation/read retries, completed and incomplete replay, UTF-8 authorization, protected storage/config/secret imports, PostgreSQL scheduler storage, blob/oplog metrics, and P3 blob reads.
• All-target Clippy for golem-worker-executor, golem-service-base, and golem-worker-executor-test-utils, with --no-deps -Dwarnings: passed.
• Scoped formatting and git diff --check: passed.
• Built the host-api and Rust SDK test fixtures. An initial integration setup failure from a missing fixture was resolved by building it; subsequent affected runs passed.

CI follow-up

Merged current main without rewriting the implementation commits. Initial CI exposed an upstream test initializer missed by the shard-manager leader-election change: the local-server memory-override test omitted the new RunDetails.leadership field. A separate one-line fix sets it to None, matching embedded SQLite startup. Oracle reviewed that fix and returned PASS; the targeted local-server test passed.

The next run passed 51 jobs and exposed an intermittent reconstruction-barrier test failure, also reproduced locally. A separate test/fixture-only fix replaces competing pre-custom HTTP checkpoints with the existing host-side entity completion gate. It explicitly establishes a completed entity and unterminated custom call at crash, then verifies repair remains blocked until reconstruction validation, proceeds after release, and writes exactly once. Oracle reviewed the actual fix and returned PASS. The rebuilt fixture and corrected test passed 11 consecutive local runs. Production durability code and test timeouts were not changed for this follow-up.

The five related tool-reconstruction tests also passed locally, and scoped integration-test Clippy passed with -Dwarnings.

Previous verified CI: PASS. Run 34377368603, attempt 2 completed successfully before the latest main update: all 52 runnable CI jobs passed, with 7 non-applicable jobs skipped. The preceding attempt was blocked by cache-service HTTP 500/404 errors during binary uploads/restores, before affected tests ran; a full workflow retry succeeded without further source changes.

Latest main update

Merged main again and resolved the plugin comment and scheduler test-helper conflicts, retaining both the upstream behavior changes and GOL-482 optimizations. Oracle identified interference between the retained exact-count metric test and two new upstream Resume tests. A separate test-only commit adds a shared async mutex around those three tests without weakening their assertions. Oracle re-review: PASS.

The first updated CI run exposed a missing DbPostgresConfig.acquire_timeout field in the PostgreSQL scheduler test fixture. The fixture now sets it to None, matching the test framework and preserving SQLx's default timeout. Oracle review: PASS. The corrected head passed the local scheduler/plugin filters with --nocapture --test-threads=8: 44 passed, 0 failed. Scoped executor library/test Clippy with --no-deps -Dwarnings, formatting, and diff checks also passed.

Final updated CI: PASS. Run 34628100705 completed successfully: all 52 runnable jobs passed, with 7 non-applicable jobs skipped. The PostgreSQL scheduler integration test and all three Resume metric tests passed in this run. All PR checks are completed with success, neutral, or skipped conclusions; the Netlify preview status is successful. The branch includes the latest main and GitHub reports no merge conflicts.

[netlify]

✅ Deploy Preview for golemcloud canceled.

Name	Link
🔨 Latest commit	42e9b74
🔍 Latest deploy log	https://app.netlify.com/projects/golemcloud/deploys/6aa43ac498e7210008b90ec1