Repository navigation
computer: Restart or replay interrupted isolate JavaScript runs - #238
Merged
Merged
Conversation
馃 Changeset detectedLatest commit: 1616d60 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
scuffi
force-pushed
the
computer/durable-js-exec
branch
from
October 15, 2026 09:30
9c57075 to
656da63
Compare
A run can now say what should happen to it if the Durable Object restarts while it runs, through an onInterrupt option on exec that module backends receive with the rest of the execution input. The policy is fail, restart, or replay. Backends that can't resume a run ignore it, so nothing changes until a backend acts on it. WorkspaceRuntime.recover() connects every module backend and asks each to settle the runs a restart interrupted. A backend that does this when it connects needs no more than that, so recover() gives a host a way to resume runs from its own alarm instead of waiting for the next runtime call.
A deploy or crash ended every isolate JavaScript run in flight, and the next connection marked it failed. WorkerJavaScriptBackend now takes an onInterrupt option, overridable per run, that can start such a run again instead. A run under restart or replay saves its module graph, input, context and limits when it starts. When the backend next connects, it finds the runs still marked running and launches each again from that saved state, within the run's original timeout and for at most three attempts. A run whose generated runtime modules, compatibility date or flags changed in between fails instead, since it would resume on different code. Restart runs the code again from the top. Replay does the same, but answers the calls the run already made from a journal. Every capability call already passes through the host bridge, so the bridge journals each one by name and arguments, writing the response to SQLite before the isolate sees it. On replay a matching call gets its recorded response without running again, in whatever order the calls arrive. A call still in flight at the restart runs again. A call the journal doesn't hold while recorded calls remain, or a run that finishes without making them, fails the run with a message saying it took a different path. Output already recorded is skipped as the replay produces it again. To keep a replay on the same path, a replayable run gets a seeded Math.random and a virtual clock that reads the time the run started, the time each journaled call completed, and a timer's due time when it fires. That makes the clock deterministic without freezing a run that waits on timers. While journaled responses remain, timers fire at once, so a replay doesn't spend the time left in its timeout waiting out sleeps it already waited. The user's code doesn't change. Network requests are not journaled and run again on replay.
Explain the onInterrupt policies, when interrupted runs resume, what a replay journals and how it detects a run taking a different path, the seeded random numbers and virtual clock that keep replays on the same path, and the limits: three attempts, the original timeout, the same runtime code, and network requests that run again.
A replay that diverged named only the calls involved, so two writes to different paths read as "it called fs.writeFile, but the next call it made before was fs.writeFile". The journal now labels each entry with the call's name and a short form of its arguments, with byte arrays summarized by length and long values cut, and the divergence message uses that label for both calls.
A replayed run used to get its journaled responses as fast as the host could send them, in the order it asked for them. Code whose path depends on which call finishes first, such as Promise.race, could take a different branch on replay than it did the first time. Timers fired at once while replaying, so a timer that lost a race against a call the first time could win it on replay. The clock moved only when a journaled call completed, so a loop polling the network until a deadline never reached it, and fetch wasn't journaled at all. A replayable run now logs the order it sees its events in: each response by an ordinal the host assigns when it commits it, and each timer by the order it was set in. A new generated module, workspace-durable.js, keeps the log and sends it with the next capability call or output frame. Each chunk says where in the log it goes, since calls and output reach the host separately, and the host saves it in the same transaction as the call or output it came with. Whatever the run did after the last thing it sent, nobody saw, so the log covers everything a replay has to repeat. On replay the host answers journaled calls at once, and the isolate holds each response and timer back until the log reaches it. Once the log runs out, responses it never placed are released and pending timers get their remaining real delay. Ordering by recorded events rather than by comparing timestamps matters because the Workers clock doesn't move while code computes: a timer set after a stretch of computation is due earlier by the clock than it fires in fact, and a timestamp comparison puts it ahead of responses it lost to. The clock is virtual: it moves to each response's commit time and to a timer's due time when it fires. Math.random, crypto.getRandomValues and crypto.randomUUID draw from a ChaCha20 stream over a 256-bit seed saved with the run, and performance.now and AbortSignal.timeout follow the virtual clock. A replayable run's fetch goes through the bridge as a net.fetch call. The host makes the request through the backend's egress policy and buffers the response within the per-call payload limit, so the journal holds it like any other response. Replayed output is now compared byte for byte against what the first attempt recorded instead of being skipped by length, and a mismatch counts as divergence. A replay that waits for an event it never reaches is reported to the host after a second with nothing else to do, rather than hanging until its timeout.
A replayable run interrupted while a host module call was in flight runs that call again when it resumes, since its response never reached the code. That suits reads and idempotent writes, but a function that charges a card or sends a message did it twice, and had no way to tell the second call from a new one. Every host module call now gets a callId in its context. A journaled call keeps the same id when it runs again, built from the execution id and its journal position, so a function can pass it to an external service as an idempotency key. Calls outside a journal get a unique id. A function that can't deduplicate can be wrapped in unsafeToRepeat(). The journal flags a call it's running again after an interruption, and the bridge fails the run when that call goes to a marked function, instead of making it a second time.
A Durable Object runs at most ten Dynamic Workers at once. After a restart, the backend relaunched every interrupted run as soon as it connected, so a Workspace with many replayable runs in flight could hit that limit, fail some of the relaunches, and leave no room for new executions. Interrupted runs now wait in a queue, and at most four restart at once. Each one that settles lets the next start. Checks that don't need a Worker, such as the attempt cap and the runtime fingerprint, still run at once, so a run that can't resume fails without waiting its turn. A queued run's remaining timeout is computed when it starts, and its attempt counter moves only then, so time spent waiting counts against its timeout but waiting doesn't use up an attempt. Killing a queued run settles it without starting it.
The interrupted-executions section described replay as answering calls in any order, with timers firing at once and fetch going out again, and listed network requests among the things a replay repeats. Replays now follow the order the run saw its events in, on a virtual clock, with seeded random numbers and journaled fetch, so the section says so. It also covers what a run can't count on: crypto.subtle keys, node:crypto, node:timers and sockets stay nondeterministic, and a fetch response is buffered within the call size limit. It documents callId and unsafeToRepeat(), the four-at-a-time restart, and that a filesystem change and its journal entry commit together.
WorkspaceModuleCallContext now requires a callId, so the test helpers that build a context by hand stopped typechecking, and the rlm example's build failed on its capability test. Give each helper a fixed callId.
A replayable run's fetch goes through the bridge so it can be journaled, and that made it count toward every capability budget: 256 calls, 8 MiB of responses, 8 MiB of requests and 32 calls at once by default. None of those apply to fetch in a run that isn't replayable, so switching a run to replay could make it fail partway through, and a long agent session would hit the call or response budget easily. fetch now skips those budgets. Instead of refusing a call past the concurrency limit, the bridge keeps at most six requests open and queues the rest, as a Worker does with its connections, which also bounds how many buffered bodies the object holds at once. Each request and response body still has to fit within maxCapabilityBytes. A journaled response larger than 256K characters is now split across rows. A Durable Object's SQLite holds at most 2 MB in a row, so raising maxCapabilityBytes would otherwise have made the journal write fail.
A replayable run's event log rode only on its next capability call or output frame, so a run that set many timers between them kept the whole log in memory until then. Past 65,536 events the host refused the chunk as malformed and failed a run that had done nothing wrong. Once 4,096 events are waiting, the isolate now sends them on their own through a new recordLog bridge method. Each chunk says where in the log it goes, as before, so it doesn't matter whether it arrives before or after the call or output that follows it.
The interrupted-executions section said a host could call recover() from an alarm, without saying when that matters. A caller polling a run or reconnecting to its output resumes it by connecting, so most runs need nothing more. A run nobody is waiting on stays interrupted after a deploy until something calls the object, and fails if its timeout passes first. Say so, and why the Workspace leaves the alarm to the host.
A replayable run swaps in its own setTimeout, Date and fetch, and each differed from the real one in a way code could trip over. Under nodejs_compat, the global timers hand back Node Timeout objects, so code calling unref() or refresh() on one worked in an ordinary run and threw in a replayable one. A class extending Date lost its prototype, since the replacement constructor ignored new.target. fetch dropped the redirect mode, so a request made with redirect: "manual" silently followed the redirect. The timers now return Timeout-like handles when the runtime's own do, detected once at install, and clearTimeout maps a handle back to its timer. The Date replacement constructs through Reflect.construct, and fetch passes the redirect mode to the host's request.
Every end-to-end test interrupted a run once. A run interrupted while it replays resumes from a journal and event log written across two attempts, which only reasoning covered. The test runner's host module gains hold(), a gate a restart opens by name, so a test can restart the object once at the first gate and again at a second. The run completes with its output once, and each call ran once except the two that were in flight at an interruption.
The journal format, event log and fingerprint check will change as replay develops, and a run interrupted across such a change fails rather than resuming. WorkerJavaScriptBackend and workspace.runtime are now listed as stable, so give resuming its own experimental row in the package's status table, and say so in the docs, the onInterrupt JSDoc and the changeset. The changeset also notes the new required callId on host module call contexts. A comment on the durable tables records why a change to their shape should rename them rather than migrate: every backend creates them when it connects, and CREATE TABLE IF NOT EXISTS leaves an older table as it was.
scuffi
force-pushed
the
computer/durable-js-exec
branch
from
October 15, 2026 14:17
7dd6fd8 to
67ce22f
Compare
Say what was added and leave the details to the docs.
scuffi
marked this pull request as ready for review
October 15, 2026 15:21
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A call whose response would take the run past its cumulative response budget gets a short budget error instead, and that error isn't counted. The journal records the error, and a replay counted it, so later live calls saw less budget than the first attempt had and could fail where it succeeded. A replay now recognizes the budget error and skips it. A comment at the journal commit also records why a filesystem change and its entry can't come apart: the change is synchronous SQL on the same database with no I/O before the commit, so both land in the same implicit transaction.
The replayable scheduler.wait took only a delay, so an aborted wait still resolved and the code after it ran. It now rejects with the signal's reason when the signal is already aborted or aborts while it waits, as the runtime's own does. The abort comes from a timer or a response the event log already orders, so it replays the same way.
A replay recognized the uncounted budget error by its message, but a host function can throw the same text, and the first attempt counted that one. The budget error now carries an uncounted mark, which the journal stores with the response, and a replay skips only a response with the mark.
A replayable scheduler.wait or fetch given a signal added an abort listener and left it attached after finishing, so code that reused one signal across many waits or requests kept every listener until the signal went away. Each now removes its listener when it settles.
Keep what a user needs to choose a policy and live with it: the policies, when a run resumes, what replay keeps the same, calls that mustn't run twice and the limits. Leave how the event log travels and how the clock moves to the code.
The docs and JSDoc said ready() resumes interrupted runs, but ready()
without arguments only indexes mounts and connects no backend.
ready({ all: true }) or a backend's id does, as does any runtime call.
An interrupted run resumed only if a digest of the code the runtime loads into the isolate matched, so any release that touched that code failed every interrupted run, even when it replayed runs exactly as before. Runs that wait a long time make that likely. The fingerprint now covers REPLAY_VERSION in place of the code itself, along with the options that change how a run replays. The constant is bumped by hand when a change would replay a run differently. A test pins a digest of the runtime's code and fails when it changes, so whoever changed it has to decide whether to bump the version.
A run got three attempts in total, so a long run could be failed by interruptions spread over its life even though it made progress between each. The durable row now records how many calls were journaled when an attempt started, and recovery starts the count again when the run has journaled more since. The run's timeout still bounds how long it can keep going, so one that crashes the same way after a little progress each time still ends.
| execution_id TEXT NOT NULL, | ||
| policy TEXT NOT NULL, | ||
| attempt INTEGER NOT NULL, | ||
| progress INTEGER NOT NULL DEFAULT 0, |
There was a problem hiding this comment.
馃敶 Existing workspaces cannot resume interrupted runs
On an existing workspace, createDurableTables leaves the durable table without progress. loadDurable then fails on recovery, and recordAttempt cannot track new runs after their first interruption.
Learn more
The backend creates its tables on connection using CREATE TABLE IF NOT EXISTS, which leaves an existing table unchanged. The prior table had no progress column. loadDurable selects that column, so recovery catches the SQLite error and fails the interrupted run. Even a run started after deployment cannot resume because recordAttempt updates the absent column.
Example: A workspace created before the upgrade retains its old durable table. Its interrupted replayable run fails on reconnection, and a newly started replayable run fails at its first recovery too.
Recommended fix: Migrate workspace_runtime_durable by adding progress INTEGER NOT NULL DEFAULT 0 when the column is missing, before reading or updating it. Verify recovery against storage with the old table schema.
Was this helpful? React with 馃憤 or 馃憥 to provide feedback.
Comment on lines
+612
to
+617
| return fingerprint([ | ||
| String(REPLAY_VERSION), | ||
| String(this.#options.maxStdioBytes), | ||
| String(this.#options.maxCapabilityBytes), | ||
| this.#options.compatibilityDate, | ||
| this.#options.compatibilityFlags.join(","), |
There was a problem hiding this comment.
馃煛 Unchanged runtime upgrade rejects pending runs
When a run spans an upgrade, #fingerprint rejects its saved runtime-code hash even if the runtime is unchanged. Existing interrupted runs fail instead of resuming.
Learn more
An interrupted run saves its fingerprint at startup and recovery requires an exact match. The prior fingerprint was a digest of generated runtime modules and compatibility settings; the new one is a digest of REPLAY_VERSION and limits/settings. Those two formats yield different hashes even for identical runtime code. Migrating the durable table alone does not permit these in-flight runs to resume.
Example: A run started with the prior package build and compatibility date 2026-05-23 is interrupted by a deploy with identical generated runtime code and settings. The saved code digest differs from the new version digest, so recovery marks it failed.
Recommended fix: On recovery, recognize the prior fingerprint format and compare it with a digest computed from the saved entry name and the current generated runtime modules; accept it only when equivalent. Continue writing the new versioned fingerprint for new runs, and test an upgrade with unchanged code.
Was this helpful? React with 馃憤 or 馃憥 to provide feedback.
Comment on lines
+2089
to
+2093
| it("asks for a REPLAY_VERSION check when the runtime's code changes", () => { | ||
| // The code the runtime loads into the isolate changed. If a run that | ||
| // started on the old code could replay differently on the new code, | ||
| // bump REPLAY_VERSION in journal.ts. Either way, update this digest. | ||
| expect(fingerprint(runtimeCode("main.js", 1024, 1024))).toBe("v1:0fb1pjqz8570ofg1opq2fb:31550"); |
There was a problem hiding this comment.
馃攳 Replay-version test misses host-side changes
The digest covers isolate code but not host-side replay logic. Changes to journal matching or bridge responses can alter replay without failing this test; review those changes when deciding whether to bump REPLAY_VERSION.
Was this helpful? React with 馃憤 or 馃憥 to provide feedback.
scuffi
added this pull request to stack #253
October 20, 2026 10:34
The REPLAY_VERSION comment only mentioned the code the runtime loads into the isolate, which is all the digest test watches. A change to how the journal matches calls or when the bridge answers them can replay a run differently too, and nothing flags it, so the comment now says so.
The tables were named for an idea the API never mentions, "durable", and for a schedule that is really the isolate's event log, beside an older events table that holds output. Some columns didn't say what they held either. workspace_runtime_durable is now workspace_runtime_resume, its chunks table workspace_runtime_resume_start, and workspace_runtime_schedule workspace_runtime_event_log. policy is on_interrupt, after the option, and progress is calls_at_attempt. The journal's at and ordinal are journaled_at and journal_order. The internal helpers follow the new names. None of these tables have been released, so they change in place.
Comment on lines
+111
to
+117
| export const RESUME_TABLES = [ | ||
| "workspace_runtime_resume", | ||
| "workspace_runtime_resume_start", | ||
| "workspace_runtime_journal", | ||
| "workspace_runtime_journal_chunks", | ||
| "workspace_runtime_event_log", | ||
| ] as const; |
There was a problem hiding this comment.
馃煛 Old replay state never gets pruned
After an upgrade, RESUME_TABLES omits the old start and event-log tables. Interrupted runs fail recovery, but their saved graphs and event logs remain in SQLite indefinitely.
Learn more
The old workspace_runtime_durable, workspace_runtime_durable_chunks, and workspace_runtime_schedule tables persist across reconnects. Runs from those tables cannot load under the new schema, but retention cleanup iterates only RESUME_TABLES. Neither completion nor retention clears the obsolete rows. This leaves potentially large saved module graphs and event logs in the Durable Object's SQLite allocation.
Example: An interrupted run has a saved module graph split into chunks and thousands of logged events. On upgrade the run fails, and retention later deletes its execution record, but the old graph and event-log rows remain.
Recommended fix: Add a one-time cleanup for obsolete resume tables after handling interrupted records, or migrate their state. Account for old journal tables and rows separately if they are retained for compatibility.
Was this helpful? React with 馃憤 or 馃憥 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Right now a deploy or crash kills every isolate JavaScript run in flight, and the next connection marks it failed. Now that runs can go on for a while, that's the main way they die: after a deploy, the old instance gets shut down about five minutes later.
This adds an experimental
onInterruptoption. With"restart", an interrupted run starts again from the top. With"replay", it starts again but gets everything it already did answered from a journal, so it catches up to where it was without repeating side effects, then carries on live. The code being run doesn't change. There's nostep()or anything like it.Changes
onInterruptgoes on the backend and can be overridden per run."fail"is still the default.A replayable run saves what it needs to start over when it starts: its module graph, input, limits and a random seed. The next time the backend connects, it relaunches any run still marked running, four at a time, since a Durable Object only gets ten Dynamic Workers. The next runtime call or
ready({ all: true })triggers that connection, or the host can call the newworkspace.runtime.recover()from its own alarm.The journal. Filesystem calls, host module calls and, in a replayable run,
fetchall go through the host bridge, so that's where the journal sits. Each response is written to SQLite before the isolate sees it, keyed by call name and arguments, and a filesystem change commits in the same transaction as its entry. On replay:flowchart LR call[call from isolate] --> match{in journal?} match -- "recorded response" --> answer[answer from journal] match -- "was in flight" --> live[run it again] match -- "no, journal empty" --> live2[run live, journal it] match -- "no, entries left" --> fail[fail: took a different path]Keeping the order. Replaying responses isn't enough on its own. The replay gets them all at once, so a
Promise.raceor a timer racing a call could go differently. While a run is live, the isolate logs the order responses arrive and timers fire in, and sends that log along with its next call or output. On replay, it holds each response and timer back until the log says it's their turn.I tried timestamps first, but the Workers clock doesn't move while code computes, so a timer set after some work looks due earlier than it actually fired. There's a test for that case.
Everything else that varies. A replayable run gets a virtual clock for
Dateandperformance.now(), and seeded randomness forMath.random(),crypto.getRandomValues()andcrypto.randomUUID(). Itsfetchgoes through the host's egress policy and is journaled like any other call. Output is compared byte for byte with the first attempt, and any difference counts as divergence. A replay that gets stuck before reaching where it was interrupted fails after a second rather than hanging until its timeout.Calls that shouldn't run twice. A call in flight at the interruption runs again, since the code never saw its result. Host module calls now get a
context.callIdthat stays the same across that rerun, so it works as an idempotency key. A function that can't deduplicate can be wrapped inunsafeToRepeat(), and an interruption during it fails the run instead:Guards. A run gets three attempts, counted again each time it journals a call, all within its original timeout. It only resumes on the same runtime: the same
REPLAY_VERSION, compatibility date and flags, and output limits. Anything else fails with a message saying why.REPLAY_VERSIONis bumped by hand when a change would replay runs differently, so upgrades that don't touch replay keep interrupted runs alive. A test pins a digest of the runtime's generated code and fails when it changes, so that decision can't be skipped.restartandreplayare marked experimental in the package's status table and the docs, so the journal format and behavior can keep changing.Trying it
Start a run, redeploy, wait for the old instance to go (about five minutes), then make any runtime call or call
recover().Testing
The end-to-end tests run the real runner in workerd and restart the object with
ctx.abort()mid-run. They cover replay and restart, response order underPromise.race, timers racing calls (including after a stretch of computation),fetchnot being sent again, random values repeating, a replay that calls something different or prints something different,unsafeToRepeat(), a second interruption during a replay, and the defaultfailpolicy. The scheduler also has its own tests in anode:vmrealm, including the ChaCha20 stream against the RFC 8439 test vector. Each replay test was checked to fail with its feature turned off.I also ran it in production with
ctx.abort()mid-run on compatibility date 2026-10-12:fetch, crashed at 4 sfetchcalls and a timer, crashed at 6 srandomUUID()printed each second, crashed at 3.5 sfetchin a loop untilDate.now()passes 8 s, no crashAnd an earlier version through a real redeploy, with a run that ticked every 5 seconds for 12 minutes, each tick incrementing a counter file. The recovery path hasn't changed since:
restartCaveats
fail, except that host module call contexts now have a requiredcallId.crypto.subtle.generateKey(),node:cryptorandomness,node:timersandcloudflare:socketsaren't covered. A run that prints their results or passes them to a call diverges on replay.fetchis buffered rather than streamed, each body has to fit inmaxCapabilityBytes(1 MiB by default), and it's slower, since every request goes through the Durable Object: a sequential loop got through about 40% as many requests as underfail. It doesn't count toward the capability budgets, and at most six requests are open at once, with the rest queued. A request in flight at the interruption is sent again. Aborting a request stops the code waiting for it, but the host's request runs on until it finishes or hits the host call deadline, holding one of the six slots.recover(). If that happens after the run's timeout, the run fails. The Workspace doesn't set an alarm itself, since an object only gets one.Follow-ups
node:timersandnode:timers/promisesin replay.fetchabort through to the host's request.computerdexecution.