Skip to content

fix: account for runtime copy during bundling - #2568

Merged
chris-olszewski merged 2 commits into
mainfrom
olszewski/fix_shutdown_ref
Oct 21, 2026
Merged

chris-olszewski merged 2 commits into
mainfrom
olszewski/fix_shutdown_ref

Conversation

@chris-olszewski

@chris-olszewski chris-olszewski commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

What was changed

For worker creation, take a new prepare hook that will prevent the shutdown of a Runtime while active. Move workflow bundling and workflow creator creation to happen in the prepare hook.

Why?

Observed some replay tests failing in CI with the following:

not ok 14 - Can replay v2 smorgasbord history from 1.18.1
  ---
    name: AssertionError
    assertion: notThrowsAsync
    values:
      'Returned promise rejected with:': |-
        TypeError {
          message: 'fn replay_worker_new.args[0]: failed to downcast any to neon::types_impl::boxed::JsBox<alloc::rc::Rc<temporal_sdk_typescript_bridge::helpers::handles::FinalizeWrapper<temporal_sdk_typescript_bridge::runtime::Runtime>>>',
        }
    at: '<anonymous> (src/__tests__/test-otel.ts:1065:11)'
  ...
not ok 15 - Can replay v2 signal workflow history from 1.18.1
  ---
    name: AssertionError
    assertion: notThrowsAsync
    values:
      'Returned promise rejected with:': |-
        TypeError {
          message: 'fn replay_worker_new.args[0]: failed to downcast any to neon::types_impl::boxed::JsBox<alloc::rc::Rc<temporal_sdk_typescript_bridge::helpers::handles::FinalizeWrapper<temporal_sdk_typescript_bridge::runtime::Runtime>>>',
        }
    at: '<anonymous> (src/__tests__/test-otel.ts:1080:11)'
  ...

A replay worker creation grabs the shared Runtime.instance(), then asynchronously bundles its workflow code. During that gap, another worker can finish and shut down the Runtime because the replay isn’t yet tracked. Replay creation then receives an undefined native handle, causing the downcast error.

Checklist

  1. Closes N/A

  2. How was this tested:
    Added a unit tests that verifies we cleanup runtime on prepare failure and a red/green regression test where a replay worker shutsdown while another one is bundling.

  3. Any docs updates needed?
    N/A

@chris-olszewski
chris-olszewski marked this pull request as ready for review October 21, 2026 19:02
@chris-olszewski
chris-olszewski requested a review from a team as a code owner October 21, 2026 19:02
@cconstable

Copy link
Copy Markdown
Contributor

This lgtm.

I had claude take a quick pass too and it pointed out an issue that is unrelated but is in the code you modified and seems reasonable. Do with this what you will:

If Worker.create fails after building the WorkflowCreator, the creator is never destroyed. Its workflow threads keep running and stop the process from exiting. This happens when connecting fails or the namespace doesn't exist. The fix is to declare workflowCreator outside the prepare callback, as the PR already does for connection, and call await workflowCreator?.destroy() in the catch.

Worker.create:
- Line 576: add let workflowCreator: WorkflowCreator | undefined; next to connection.
- Line 582: remove the local declaration so line 584 assigns to the outer variable.
- Lines 595–600: in the catch, add await workflowCreator?.destroy(). Use try/finally so the connection still closes if destroy() throws.

runReplayHistory (optional):
- Lines 788–798: same pattern. Bring workflowCreator out of the callback (line 793) and wrap the call in try/catch. This one matters less. Replay runs the workflow code in-process with no threads, and native replay-worker creation rarely fails.

@chris-olszewski
chris-olszewski merged commit d98735e into main Oct 21, 2026
27 checks passed
@chris-olszewski
chris-olszewski deleted the olszewski/fix_shutdown_ref branch October 21, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants