Skip to content

Fix snapshot recovery for component initializer host calls - #3856

Merged
vigoo merged 12 commits into
mainfrom
fix/snapshot-recovery-pr
Sep 11, 2026
Merged

vigoo merged 12 commits into
mainfrom
fix/snapshot-recovery-pr

Conversation

@vigoo

@vigoo vigoo commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Fix

Core initializers execute before load-snapshot, after the executor has already skipped the history replaced by the snapshot. An initializer clock read therefore attempts to replay a call whose original record was skipped and fails component instantiation.

Apply the existing snapshot durability suppression during primary component instantiation when recovering from a snapshot. End suppression before preparation and replay of the remaining invocation history. Fresh creation and full replay retain their existing behavior.

Regression evidence

The Rust reactor _initialize calls Instant::now during core instantiation (verified in compiled WASM, not a lazy constructor). The test leaves increment 11 outside the last snapshot, asserts the snapshot position did not change, then requires successful snapshot loading and recovery of value 11.

  • Original executor + revised Rust reproducer: fails at _initialize with expected recorded monotonic_clock::now Start ... continuation ... valid only for an incomplete entity (1.254s).
  • Fixed executor + revised Rust reproducer: passes (1.175s).
  • Seven relevant non-Go snapshot/repeated-restart/disabled-snapshot/memory-reconciliation tests: 7 passed, 0 failed (19.466s).
  • Original Go snapshot test reproduces the same initialization error. The unchanged Go test passes with this executor fix and a separately isolated Go SDK loader prerequisite. Correcting only the Go SDK still reproduces the original clock error, establishing executor-fix causality. No Go source or test is included in this PR.

Dependency and validation scope

Clock prerequisite #3855 is merged. This branch includes that prerequisite. Its current diff against main contains seven files: the original three-file snapshot fix, three test files corrected for fixture-search expectations and asynchronous snapshot timing, and debugger test-client error diagnostics. No Go code is included. This snapshot PR remains unmerged.

The initial CI run failed 87 tests: 86 reported cannot block a synchronous task before returning, including the core-start clock fixture, because the clock prerequisite was absent. The remaining HTTP poll-loop timeout occurred when the test attempted interruption in the idle gap between construction and start_polling; its readiness correction is owned by #3855. The genuine reactor initializer and explicit unsnapshotted increment-11 regression remain unchanged.

After including the clock readiness follow-up, the exact combined branch passed all eight poll-loop tests plus eight snapshot/recovery tests: 16 passed, 0 failed (43.522s). Package formatting and diff checks also passed. CI is validating the pushed combination.

That CI run resolved the original failures but exposed three oplog-search assertions expecting two imported calls rather than three (the additional call is the core initializer clock), plus an immediate snapshot-count assertion racing queued snapshot publication. Expectations now include the clock call, and the hot-update test waits with a 30-second bound while retaining exact count and recovered-revision assertions. The worker search, hot-update, and original tail regression passed locally (3/3); hot-update passed five additional runs. PostgreSQL/SQLite variants are validated by CI. Formatting and diff checks passed.

Executor all-targets Clippy with warnings denied passed before the final test-only strengthening. Revised tests, formatting, fixture validation, fixture Clippy, and diff checks passed. Full suites remain for CI. The independent Go SDK fresh-instance restoration correction is preserved separately and is not proposed for main.

CI run 34471956153 passed all worker groups, CLI streaming, and PostgreSQL/SQLite search tests. Its only failure was an intermittent debugger connection closing, reported as Stream ended unexpectedly. Local investigation passed 41 isolated tests and 90 tests in concurrent debugger runs. The test client now preserves WebSocket close reasons and transport errors; no retry or relaxed assertion was added, and the underlying intermittent cause is not claimed fixed.

Final CI: run 34475464292 passed on the latest head. All 66 checks are complete: 53 successful, 10 intentionally skipped, and 3 neutral; the Netlify status also succeeded. The previously failing debugger test passed, and the group-5 report contains 525 passed, 0 failed tests. No checks remain pending or failed.

Investigation and evidence review: https://ampcode.com/threads/T-01a08615-324b-76fb-883a-9fa3472a5e3a

@vigoo
vigoo requested a review from a team September 10, 2026 09:56
@netlify

netlify Bot commented Sep 10, 2026

Copy link
Copy Markdown

Deploy Preview for golemcloud canceled.

Name Link
🔨 Latest commit 63abbac
🔍 Latest deploy log https://app.netlify.com/projects/golemcloud/deploys/6aa427b0d28c240008a061b3

@vigoo
vigoo merged commit 52b23c9 into main Sep 11, 2026
67 checks passed
@vigoo
vigoo deleted the fix/snapshot-recovery-pr branch September 11, 2026 16:40
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 11, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants