Implement Oplog epoch fencing: oplog_metadata, epoch-checked appends, ShardLost relinquish

.agents/skills/understanding-durable-execution/SKILL.md

@@ -33,9 +33,15 @@ and entity bodies) and `retries.md` (in-function versus trap-based retries).
 
 2. **The resident runtime is disposable.** The Wasmtime `Store`, the worker task, the executor
    process, sockets, channels and subscriptions can vanish while work is pending. Suspension,
-   eviction, resharding, restart and crash must all be recoverable the same way: throw the
-   instance away, build a new `Store`, replay the oplog, continue. Lifecycle hints (`Suspend`,
+   eviction, restart and crash must all be recoverable the same way, on the same executor: throw
+   the instance away, build a new `Store`, replay the oplog, continue. Lifecycle hints (`Suspend`,
    `Interrupted`, `Restart`) change status and scheduling policy, not the recovery mechanism.
+   Losing the shard does not reconstruct here at all: every oplog write asserts this executor's
+   shard epoch inside the storage transaction, and once another executor holds the shard, the
+   write is refused (`OplogError::Fenced`) instead of accepted. The agent then *relinquishes*
+   (`InterruptKind::ShardLost`) — stopped with nothing more written, dropped from this executor,
+   never restarted in place — and it is the new owner that builds the `Store` and replays. See
+   "Resharding, revocation and the oplog epoch fence" below and `crash-matrix.md`.
    Owner: `worker/invocation_loop.rs::run` (outer loop: create instance → recover → run →
    suspend/retry) and `durable_host/mod.rs::prepare_instance`.
 
@@ -93,6 +99,14 @@ says how strict that commit is: `Always` waits for durable storage; `DurableOnly
 for durable agents (`PrimaryOplog::commit` flushes everything; `EphemeralOplog` honours the
 level). Guarantees such as "accepted only after commit" refer to the commit, not the append.
 
+A commit can also be *refused*: every append and commit asserts this executor's shard epoch
+inside the storage transaction, and storage that has recorded a newer epoch returns
+`OplogError::Fenced` instead of writing anything. `commit_oplog_and_update_state` and
+`add_and_commit_oplog` surface that refusal rather than swallowing it, so a refused
+`PendingAgentInvocation` commit is not acknowledged as accepted and a refused
+`AgentInvocationFinished` commit is not published to waiters. The first refusal latches — every
+later add on that oplog is refused too — and the agent relinquishes instead of retrying.
+
 `worker/state_actor.rs::commit_and_update_state` samples the appended tip before its explicit
 commit and ignores receipt entries already folded into the published status. Primary/ephemeral
 threshold flushes and replica waits can commit outside the status actor, so even an empty receipt
@@ -142,8 +156,13 @@ each recorded invocation in `InvocationMode::Replay`, and when there is no furth
 `AgentInvocationStarted` it switches to live. Replay starts from the chosen snapshot baseline
 (see Snapshots and updates), not necessarily from `OplogIndex::INITIAL`. Interruption kinds
 (`Worker::set_interrupting`): `Interrupt` stays interrupted, `Restart` is a simulated crash with
-automatic recovery, `Suspend` unloads and resumes on demand; all three end in the same
-reconstruction path. Eviction (`EvictionClass::{LoadedIdle, WarmRunnable}`) never unloads a
+automatic recovery, `Suspend` unloads and resumes on demand — each of these three reconstructs on
+this same executor. `ShardLost` does not: it means this executor lost the agent's shard (a
+revoked/reassigned shard, or an oplog write refused on the shard epoch), and instead of
+reconstructing, the agent relinquishes — stopped without writing to its oplog or status, dropped
+from this executor, and left for the new owner to reconstruct.
+
+Eviction (`EvictionClass::{LoadedIdle, WarmRunnable}`) never unloads a
 worker that is executing or holds non-durable in-memory work. Ephemeral agents are fail-stop:
 `reconstructed_ephemeral` rebuilds only for observation and result lookup, "but the instance must
 never be started again" (`worker/mod.rs`, `INACTIVE_EPHEMERAL_AGENT_ERROR`).
@@ -257,6 +276,42 @@ in a pending p3 wait). `Resumed` does not enqueue an invocation: the existing
 does not use this marker and retains its normal `Idle`/automatic-recovery semantics; ephemeral
 agents retain their clean fail-stop lifecycle and never append it.
 
+### Resharding, revocation and the oplog epoch fence
+
+Two triggers give an agent up rather than reconstructing it here, and both end in the same place,
+`RelinquishReason` and `InterruptKind::ShardLost`:
+
+- **Assignment change.** The shard manager's `RevokeShards` and `AssignShards` gRPC calls
+  (`grpc/mod.rs::revoke_shards_internal`, `::assign_shards_internal` via
+  `apply_shard_assignment_effects`) relinquish every agent whose shard this executor no longer
+  holds, or whose held epoch fell behind the delivered one (another executor may have written to
+  it meanwhile) — `RelinquishReason::ShardRevoked` for the former, `ShardNotAssigned` for the
+  latter. `apply_shard_assignment_effects` then calls the *other* `on_shard_assignment_changed`
+  (`durable_host/mod.rs`, the `WorkerCtx` hook) to recover agents on shards newly held, the
+  opposite direction from relinquish.
+- **Oplog epoch fence.** Every indexed-storage oplog write asserts this executor's current shard
+  epoch inside the storage transaction (`storage/indexed/{postgres,sqlite}.rs`, surfaced through
+  `services/oplog/primary.rs`); a shard manager mints a new, higher epoch for the new owner when it
+  takes over, so a write from an executor that has lost the shard is refused rather than written
+  (`OplogError::Fenced` / `OplogFence`, carrying the asserted and, when known, the actual epoch).
+  This is what protects an assignment change this executor has not yet heard about, and a revoked
+  lease it is still trying to renew: `RelinquishReason::Fenced`. Once one write is refused the
+  fence *latches* — every later append or commit on that oplog is refused too, without a second
+  round trip to storage — so nothing further is ever written by this executor for that agent. Only
+  Postgres and the SQLite-backed indexed storages can fence a write this way
+  (`IndexedStorage::supports_epoch_fencing`); an executor configured with Redis and a real shard
+  manager refuses to start rather than run unfenced.
+
+Either way, `Worker::relinquish` (`worker/mod.rs`) stops the agent without writing to its oplog or
+status, drops it from this executor's `ActiveAgents`, and hands its invocation waiters a retriable
+error (`ShardingNotReady`, or the fenced-specific variant) rather than an in-place restart — the
+oplog is left exactly as it was, for the shard's new owner to reconstruct from when the worker
+service routes a request there. An invocation still pending in this executor's queue when it
+relinquishes is failed the same way: with a retriable error and no cached result, never with a
+result the queue happened to already hold, so a client retry runs it exactly once, on the new
+owner. See `crash-matrix.md` for the fence's failure modes and `services/active_agents/mod.rs` for
+the sweep that relinquishes on an assignment change.
+
 ## Oplog model
 
 Entries are positional or hints (`OplogEntry::is_hint()`). Replay consumes positional entries in
@@ -599,6 +654,7 @@ satisfies one does not imply the others.
 | "Cursor reached the end, so I can do the live effect now." | Liveness is `store_is_live(...)`: the primary needs `switch_to_live` to publish after reconstruction fences; an entity Store needs its own `local_live_tail`. Cursor exhaustion is neither. | `pending_replay_to_live_is_fail_closed_until_finished`, `entity_store_liveness_is_scoped_to_its_invocation_mode` |
 | "The voluntary-suspension predicate gates interruption or recovery." | It only defers proactive yielding while live work progresses; explicit interruption and arbitrary Store loss still use ordinary reconstruction. | Simulated-crash tests at arbitrary points (`simulated_crash`, `interrupt`) |
 | "Restart differs from suspend." | Both discard the `Store` and reconstruct. | `counter_resource_test_2_with_restart` (state continues across an executor restart), `reacquire_permits_restart_preserves_accepted_queued_live_invocation` |
+| "Losing a shard reconstructs the agent, like a restart." | It relinquishes instead: stopped here without writing to its oplog or status, dropped, never rebuilt on this executor. Only the new owner reconstructs. | `oplog_fencing_guard_tests` (`lib.rs`), fence tests in `services/oplog/{primary,tests}.rs` |
 | "A retried RPC attempt executed the target again." | Same key ⇒ same target invocation; count target mutations, not attempts. | Provider-side counter tests in `tests/rpc.rs` |
 | "Atomic rollback should generate a fresh RPC key." | Logical counter is owned by the outermost atomic region; keys survive `Jump`. | `tests/transactions.rs`, `tests/revert.rs` |
 | "Equal return values prove deduplication." | Deterministic echoes are equal even with duplicate execution; count side effects. | Counter-based RPC tests |

.agents/skills/understanding-durable-execution/reference/crash-matrix.md

@@ -1,10 +1,16 @@
 # Crash-window matrix
 
-"Crash" here means any loss of the resident runtime: process death, `Restart` (simulated crash),
-`Suspend`, eviction, resharding (`on_shard_assignment_changed`), or an executor drop in a test.
-Reconstruction is identical in every case: new `Store`, `prepare_instance`, `resume_replay`,
-publish Live. The matrix says what the next incarnation does for a crash inside each window and
-which durable fact makes that safe.
+"Crash" here means any loss of the resident runtime *on the same executor*: process death,
+`Restart` (simulated crash), `Suspend`, eviction, or an executor drop in a test. Reconstruction is
+identical in every case: new `Store`, `prepare_instance`, `resume_replay`, publish Live. The
+matrix says what the next incarnation does for a crash inside each window and which durable fact
+makes that safe.
+
+Resharding and the oplog epoch fence are different: this executor does not reconstruct at all. It
+relinquishes the agent (`InterruptKind::ShardLost`) — stopped without writing to its oplog or
+status, dropped here — and the shard's new owner is the one that runs `prepare_instance` /
+`resume_replay`, on its own copy of the same oplog. See "Resharding and the oplog epoch fence"
+below for what that leaves behind.
 
 ## Durable host call (`concurrent/call.rs`, `concurrent/delivery.rs`)
 
@@ -92,6 +98,23 @@ which durable fact makes that safe.
 | Body traps | no entity terminal | Owner invocation fails; owner group drains; siblings blocked on the lane are fenced | `guest_trap_fences_a_blocked_sibling_and_drains_the_owner_group` |
 | Owner reaches replay tail while a body is still reconstructing | — | `HistoricalReconstruction` fences keep `PendingReplayToLive` closed until every active body validates | `completed_reconstruction_claim_blocks_concurrent_replay_to_live` |
 
+## Resharding and the oplog epoch fence (`worker/mod.rs::relinquish`, `services/oplog/primary.rs`)
+
+Two triggers relinquish an agent instead of reconstructing it here: the shard manager revoking or
+reassigning the shard (`grpc/mod.rs::revoke_shards_internal` / `assign_shards_internal`,
+`RelinquishReason::ShardRevoked` / `ShardNotAssigned`), and a write refused because the epoch this
+executor asserted no longer matches storage (`OplogError::Fenced`, `RelinquishReason::Fenced`).
+Only Postgres and the SQLite-backed indexed storages can refuse a write this way; an executor
+configured with Redis and a real shard manager refuses to start rather than run unfenced.
+
+| Crash window | Oplog shape left behind | What happens here | Durable fact relied on |
+|---|---|---|---|
+| Assignment revoked/reassigned, before any write is attempted | whatever was already committed | `relinquish_matching` stops matching agents directly; no write is attempted or refused | `ShardService::check_worker` / the delivered assignment, not the oplog |
+| A write is attempted after the shard actually moved | nothing new; the attempted entry is refused, not partially written | The refusal is returned (`OplogError::Fenced`), not retried or swallowed; the agent relinquishes | Epoch asserted inside the storage transaction |
+| Any later write on the same oplog handle | still nothing new | The fence latches: every later add/commit is refused immediately, without a second storage round trip | The oplog's own latched `OplogFence` |
+| An invocation still queued when relinquish runs | unaffected | Failed with a retriable error (`ShardingNotReady` / the fenced variant), never a cached result | `PendingLiveInvocationDisposition::Fail` |
+| The new owner opens the same agent | the fenced executor's last accepted entries | Ordinary `prepare_instance` / `resume_replay`, from committed history exactly as it was left | Nothing was written after the fence latched |
+
 ## Oplog-processor plugins (`services/oplog/plugin.rs`)
 
 | Crash window | Recovery |

Makefile.toml

@@ -725,6 +725,7 @@ cargo-test-r run --package golem-registry-service --test '*' -- --nocapture --re
 cargo-test-r run --package golem-worker-service --test '*' -- --nocapture --report-time $JUNIT_OPTS
 RUST_LOG=debug cargo-test-r run --package golem-debugging-service --test '*' -- --report-time $JUNIT_OPTS
 cargo-test-r run --package golem-shard-manager --test integration -- --nocapture --report-time $JUNIT_OPTS
+cargo-test-r run --package golem-test-framework --test signal_unreaped_child -- --nocapture --report-time $JUNIT_OPTS
 '''
 
 [tasks.integration-tests-group6]