[eersnington]

An actor alarm that comes due while no envoy is connected is lost. The actor never wakes.

This happens when the engine starts with overdue alarms, before envoys reconnect, or when an alarm comes due during shutdown.

Sleeping + Timer(AlarmTimeout)
  Executor::prepare → select_allocation → None    # no envoy connected yet
  wake → allocate_actor(None)
    None => phase = Sleeping                      # alarm consumed, nothing re-arms it

• When the alarm wakes a sleeping actor and no envoy can start it, the actor state machine re-arms the alarm 1 s later (plus the stop-resend jitter) instead of dropping it.
• The re-armed timer commits with the alarm's acknowledgement, so a crash in between can't lose it.

Scenario	Before	After
Alarm due while the engine is down	never fires	fires about 1 s after restart
Alarm due during shutdown	never fires	fires after restart
Alarm still in the future across a restart	on time	on time

[paulbettner]

Title

Overdue alarm is consumed before an envoy reconnects after an engine restart (3.0.0-preview.5)

Summary

In case it's helpful:

A one-shot schedule due while the standalone engine is down does not run automatically after the engine restarts, although its worker process stays alive and the actor's schedule survives. The same minimal engine-only outage succeeds with the existing PR #5951 retry change applied to the exact preview.5 source. This report is specifically about Sleeping + due alarm + no eligible envoy, not every startup/reallocation failure.

Versions and topology

• rivetkit@3.0.0-preview.5, native Linux x64 GNU N-API binding; standalone Linux x64 engine, not the SDK-local control plane.
• SDK/source base: 83442222dbfd57abff08cd945262df0fbbc87dcd.
• Control: rivetkit@2.3.31, source f6926c3103809c0119f06c85f0c38ccc4671073c.
• Node 24.19.0; local builds use Rust 1.99.0, Cargo.lock unchanged, dev profile. Same SDK/N-API package is retained when substituting the patched engine.
• One engine and one separately managed persistent worker; loopback listeners only. Same persistent filesystem objects and engine log directory are reused across restart; a fresh directory is used for each trial. Stock 40-second node-lost threshold; no shortened recovery threshold.

Engine executable	SHA-256
npm preview.5	ed0d578cbdc77e604b9e7f7ef97cab01f04336bcbf41d759467c7558688aaa9f
preview.5 + PR5951 production change	ab99739441efe4d2f469de82d2fca4fd10996bb77f88a703b1fdd8a8940840e4
locally built unmodified preview.5 production code, same toolchain/profile	19bfcbe8aae83e13676cf52c27eed63eb336f9f2d2b3360b955f65e3bf9bf229
npm 2.3.31	469ff09c041ebcd2c0e6f04f347c18aa9494f544fbc11faf34e6ee993aa921eb

The npm engine hash is the executable pin; the npm SDK gitHead is not independent proof of the released executable's build commit. The local source/build controls remove that ambiguity for the patched/unpatched comparison.

Minimal reproduction

Use a normal persistent worker pool named outage-repro and a single actor with sleepTimeout: 5000. The essential actions are:

arm: async (c, token) => {
  const armedAt = Date.now();
  const dueAt = armedAt + 60_000;
  const scheduleId = await c.schedule.at(dueAt, "fire", token, dueAt);
  c.state.arm = { token, dueAt, scheduleId };
  await c.saveState({ immediate: true }); // normal request, not scheduled handler
  return { actorId: c.actorId, armedAt, dueAt, scheduleId };
},
fire: (c, token, dueAt, scheduledFire) => {
  // Append and fsync an external JSONL record here, including Date.now(),
  // actor ID, process PID, token, dueAt and native scheduledFire metadata.
  // Do not use an actor query to observe the result.
}

The observer records native onSleep before the outage. There is no immediate state save inside a scheduled handler, no JS scheduling timer, no onWake re-registration and no keep-awake option. The worker uses an explicit engine endpoint, runtime: "native", startEngine: false, startServices: false, node: {poolName: "outage-repro", version: 1}, and RIVETKIT_ENGINE_SPAWN=never. The worker must not manage or restart the engine.

Install the pinned packages with npm install --ignore-scripts --save-exact rivetkit@3.0.0-preview.5. Start the packaged engine with:

./node_modules/@rivetkit/engine-cli-linux-x64-musl/rivet-engine --config engine.json

The tested single-engine configuration binds guard, peer and metrics to 127.0.0.1, supplies a one-datacenter loopback topology, and sets these persistence/recovery variables (paths below are relative example directories, not machine-specific paths):

export RIVET__STORAGE__LOG_DIRECTORY="$PWD/engine-data/log"
export RIVET__OBJECT_STORAGE__PRIMARY__FILE_SYSTEM__PATH="$PWD/engine-data/objects"
export RIVET__STORAGE__AUTO_PROVISION_PARTITIONS=1
export RIVET__STORAGE__BALANCING__ENABLED=false
export RIVET__STORAGE__WRITE_TO_COLD_AFTER_MS=5000
export RIVET__STORAGE__LOG_SHARD__MAX_DISK_BYTES=4294967296
export RIVET__STORAGE__COLD_STORE__DISK_CACHE_BYTES=0
export RIVET__STORAGE__STOP_TIMEOUT_MS=6000
export RIVET__STORAGE__STOP_TEARDOWN_MS=2000
export RIVET__PEGBOARD__NODE_LOST_THRESHOLD=40000

These are the preview.5 launcher's accepted environment names; current docs use some newer depot/envoy names. Persistent objects alone are not sufficient: retain the local log/instance directory too. Do not reuse a 2.x store as a 3.x store.

1. Start the engine, create the normal pool, then start the separate worker. Wait for initial storage routing and worker readiness before arming. /health alone can become ready before the initial pool PUT succeeds.
2. From a separate short-lived caller, create the actor and invoke arm once. Take its armedAt as T0. Dispose the client and require the caller process to exit successfully before T+25 seconds.
3. Confirm native onSleep for this actor after arm, with no later wake. At T+30 seconds, SIGKILL only the recorded engine PID; wait for actual process exit. Keep the worker process alive and keep both storage directories.
4. At T+60, the schedule becomes due while the engine is absent. Verify that the worker still has the same PID/start identity.
5. At T+120, restart the same executable/config/data directory. Do not restart the worker.
6. Make no actor, metadata or health calls after arm for the entire 600 seconds after engine restart (approximately through T+720). Count only the fsynced handler-side effects and local process/listener/lifecycle evidence.
7. Freeze that result before any diagnostic request. Count a recovery only if exactly one correctly attributed kind: "at" effect occurs after engine restart; an outage-time local callback does not qualify. Repeat with fresh actors/data three times per engine.
8. In a separate matched control, enable a different actor with c.cron.every({name: "resume", interval: 60_000, action: "tick", args: [token]}); use another effect file. Require at least two distinct post-restart scheduled tick times without a caller. This does not assert replay of every missed interval.

After the frozen window, an explicitly separate caller may invoke a no-op/inspect action on the original actor to establish engine/worker readiness and test whether the original overdue schedule is retained. Such a caller-triggered fire must never turn the preceding negative result into a pass.

Expected and observed

Expected: once the engine and worker reconnect, the overdue alarm autonomously starts the sleeping actor and invokes the retained schedule; recurring work resumes without a request to that actor.

Observed full 600-second windows: stock preview.5 0/3 autonomous fires; patched preview.5 3/3; 2.3.31 3/3. The same-toolchain/profile local unpatched engine also produced 0/3. A separate stock readiness cohort reproduced 0/3 through its frozen windows; a later caller then successfully started the original actor and caused its original overdue schedule to fire in 3/3, proving engine/worker usability and retention without upgrading any no-caller result. The matched recurrence control produced 0 ticks on stock and 11 distinct post-restart ticks on patched, with no caller and no outage-time ticks. All trial engines/workers were stopped with terminal/ownership read-back. One initial pre-arm storage-routing failure is excluded, not counted as an alarm regression. These are bounded observations, not a claim that a zero-fire run could never recover later.

The three patched minimal trials fired 3.990, 4.010 and 4.019 seconds after engine restart. Example on 2026-11-01 UTC: arm 06:21:06.720, stop 06:21:36.819, due 06:22:06.720, restart 06:23:06.892, fire 06:23:10.882, observation cutoff 06:33:06.990. A stock readiness trial armed at 06:36:23.648, stopped at 06:36:53.647, was due at 06:37:23.648, restarted at 06:38:23.721, and had no fire through 06:48:23.820; only its separately recorded post-cutoff caller triggered a fire.

Suspected cause and existing fix

At the pinned preview.5 base:

• actor_machine/mod.rs:489–498: the sleeping alarm calls wake.
• mod.rs:958–960 and 1137–1140: allocation None leaves it sleeping without a retry timer.
• executor.rs:812–843: an empty eligible-envoy set returns None; a live worker process is not yet necessarily an eligible reconnected envoy.
• state-machine-durable/runner.rs:217–303: the fired timer is removed and its revision acknowledged with the next remaining timer in the same commit. With no replacement timer, the wake is consumed.

PR #5951 reuses the existing durable timer mechanism to rearm this case after 1000 ms plus the existing 0–1999 ms jitter. It does not add a scanner, schema or protocol. The direct regression test fails on original code and passes with this change; immediate available-allocation startup remains passing. Runtime results support this mechanism, but the original incident's exact allocation result was not instrumented, so its causal attribution remains a root-cause guess, not a captured transition trace.

Known limits — not observed in these trials and not introduced by this patch

• Due while Stopping → deferred reallocate → allocation=None: preview.5 actor_machine/mod.rs:390–413,1073–1101,1137–1140; the unchanged Stopping/stop-completion paths bypass the patched Sleeping branch and can still settle without a replacement alarm.
• Selection succeeds → native startup fails before overdue drain: preview.5 actor_machine/mod.rs:155–167,1073–1101 and engine/sdks/rust/envoy-client/src/actor.rs:298–325; the patch retries absent allocation, not an already-selected start that fails and returns to Sleeping.

This is distinct from #5833's claimed SQL action loss, scheduled-handler immediate-save failures, and general fenced-storage restart failures. No broader recovery guarantee or exactly-once side-effect guarantee is asserted.