diff --git a/docs/deployment-reference.md b/docs/deployment-reference.md index 05e399ac..09c1f786 100644 --- a/docs/deployment-reference.md +++ b/docs/deployment-reference.md @@ -305,6 +305,15 @@ Decommission the whole physical resource set after revoking traffic and credenti When `init()` or `createD1Storage()` receives `tablePrefix`, pass the same value as `storageTablePrefix` to `createFlowsafeWorker()`. The shared rule accepts an empty prefix or requires an ASCII letter or underscore first, followed by ASCII letters, numbers, or underscores. The prefix can contain at most 39 characters so every final Mastra table identifier stays within its 63-character limit. Maintenance applies it to workflow snapshots, threads/messages, background tasks and their snapshots, notifications, thread state, and schedule-trigger history. All six exported low-level purge functions validate the same contract before preparing D1 statements. Approval, resource-owner, deployment-sentinel, subscription, and other fixed-schema tables remain unprefixed. A valid but mismatched value targets a different table family; Flowsafe does not auto-discover it. +Host-owned notification effects register one receipt-writing table prefix and one receipt limit in the deployment's D1 database. The default limit is 100,000 retained receipts; a deployment can set a lower limit. Existing receipts remain available for replay at the limit. Ordinary notification storage can still use other prefixes, but those prefixes cannot write operation receipts in the same deployment. + +To upgrade a deployment with retained operation receipts: + +1. Lock its execution fence and wait for earlier receipt writers to finish. Keep every Worker and Durable Object that can write the D1 database under exclusive deployment control. +2. Inspect the D1 tables ending in `flowsafe_notification_operation_receipts`. If more than one prefix has retained rows, leave the fence locked and resolve the histories before enabling effects. Do not delete receipts to meet the limit. +3. Deploy the version that registers receipt scope to every writer of that D1 database. Set one prefix and one limit across those writers. Check the active versions before unlocking. An older writer can bypass the registration, so do not run mixed versions with the fence open. +4. Unlock the fence. The first new receipt operation registers the prefix and limit; later writes with a different value fail closed. Check the `flowsafe_notification_receipt_scope` row after the first operation. Fence the deployment again before any rollback to an older writer. + Idle-thread retention does not release thread or resource ownership. A thread can still have an agent binding, schedule, subscription, goal, or other standing wake source after its idle memory rows expire. An explicit host teardown must remove every authoritative standing record before it calls `ActorContext.releaseResource()` for the thread and resource claims. ## Egress and process execution diff --git a/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md b/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md new file mode 100644 index 00000000..90599140 --- /dev/null +++ b/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md @@ -0,0 +1,333 @@ +# Recover a remote-step resume without repeating an external effect + +> Content type: design plan. Audience: FlowSafe maintainers and deployment hosts. Status: proposed, not implemented. + +The dated amendment at the end is the current execution plan. The preceding sections preserve the decision and code baseline from 2026-10-12. + +This plan records the limit of safe cross-instance resume recovery on pinned `@mastra/core@1.73.0`. Its `run.resume` can execute arbitrary tool closures, nested agents, background work, model calls and direct network effects outside a FlowSafe-owned adapter. General agent continuation is therefore ineligible for automatic replay. A narrow host-owned operation with an explicit, closed effect set may use a downstream idempotent executor; the subsequent Mastra continuation remains single-admission. An uncertain Mastra continuation stays unresolved and blocks drain until independent evidence or a future runtime contract resolves it. + +## Baseline and evidence + +- Branch: `plan/flowsafe-remote-step-broker`; base commit: `4096fbd` (2026-10-12). The settlement checkpoint at that commit leaves expired `resuming` rows unresolved and in drain inventory. Architecture pass 3 cleared the lease-reclaim MUST in `/tmp/asp-flowsafe-settlement-arch-pass3-20261012.md`. Reconfirm the appendix against the checked-out commit before editing. +- Runtime: Node and pnpm versions in the root manifest; Cloudflare Workers, D1 and Durable Objects in production. Use a linked worktree with the shared pnpm store. Keep checks in `/tmp` logs. The full FlowSafe suite passed 6,648/6,648 for the committed settlement content. Automatic cross-isolate replay remains a release gate. +- Cloudflare [Durable Object lifecycle](https://developers.cloudflare.com/durable-objects/concepts/durable-object-lifecycle/) says shutdown can route new requests to a new instance while an old HTTP/RPC request continues if it avoids object storage. A later old-instance storage access fails. Cloudflare [alarms](https://developers.cloudflare.com/durable-objects/api/alarms/) run at least once and have bounded automatic retries. Neither an in-memory lock, a lease timeout, an alarm, nor a Worker deploy proves that an old continuation or its external effect stopped. +- FlowSafe's [README runtime section](../../packages/flowsafe/README.md) and [interrupted-leg design](../do-runner-design.md#interrupted-legs) say a deploy can leave an outgoing leg running and that an aborted external call may already have taken effect. This plan treats that as part of the supported execution model. +- The installed `@mastra/core@1.73.0` source executes `tool.execute` directly in `dist/agent/durable/index.js` and `dist/workflow-event-processor-LHwOlFwA.js`; `dist/agent-D-eTzWxF.js` enters `run.resume` and nested agent paths. Its engine persistence includes `persist`, `replace` and `patch` paths without a remote-step attempt generation. The appendix anchors the exact installed source. The proposed generic effect adapter and runtime generation seam fail this version's capability check. + +## Decision and invariant + +**Recommend a host-owned closed-operation profile.** Its declared operations run outside arbitrary resumed Mastra closures, through a concrete downstream executor. The host enumerates every operation and its inputs before admission. Each effect uses an operation key that stays identical across retries of that logical operation. The downstream system must atomically deduplicate the key with the effect and return the first operation's durable outcome. A preliminary read, best-effort cancellation or FlowSafe-only fence does not establish this property. If the external system supports a fencing generation instead, it must reject stale generations in the same atomic operation as the effect and expose a durable outcome lookup. The profile never authorizes automatic replay of `run.resume` on the pinned core. + +No downstream executor is selected in this repository yet. The profile remains a gated proposal until an owner names one and proves its exact idempotency and receipt API. The existing general agent path stays fail closed. + +A `resumeAttemptId` identifies an execution attempt and helps attribute logs. It must not become the external idempotency key, because retries would use a new attempt ID. The stable effect key binds deployment, run generation, remote invocation and an enumerated host operation. The host persists that operation and its request digest before calling the executor. The executor returns a durable receipt or an explicit unresolved state. It must never mark a timed-out request as not executed because the response was lost. + +One retained result remains bound to one exact suspension. On resume, preserve the stored principal, thread/resource binding, start token, run ID, step path, suspension timestamp/count, mutation epoch, deadline, result digest and current deployment fence. Host operations may be retried through their downstream idempotency contract. Mastra `run.resume` may be admitted once; a newer instance must not invoke it again merely because a lease expired or the snapshot still says suspended. `RunnerRuntime.resume` receives no remote-step attempt generation through `resumeViaRuntime`, and the pinned engine does not fence competing snapshot writes. An unresolved first call remains `resuming`/`recovery-blocked` inventory work. + +An advanced snapshot alone does not prove which attempt or retained result produced it. A crash after the snapshot write loses the in-memory `resumeViaRuntime` witness. The pinned engine has no proven attempt-linked receipt stored atomically with every advancing snapshot write. Without that receipt, classify the advanced state as unresolved rather than `applied`; an operator needs independent evidence. A future runtime may lift this limit only after its entire persistence path atomically binds generation, exact suspension and result digest to the advancing write. + +### Considered options + +| Option | Decision | +| --- | --- | +| Extend the in-instance `withDispatchLock` and reclaim expired leases | Reject. It cannot serialize an outgoing object instance after deploy. | +| Wait a fixed grace period or rely on a Durable Object alarm | Reject. Time and alarm delivery do not prove quiescence or external outcome. | +| Check the authoritative snapshot and retry when it still says suspended | Reject for arbitrary effects. The old continuation can be in flight without a completed snapshot write. | +| Require a universally transactional external operation | Reject as a generic FlowSafe feature. FlowSafe does not control downstream systems. | +| Generic resumed Mastra tool adapter | Reject on pinned 1.73.0. Direct tool closures and nested work bypass it, and snapshot writes lack attempt generation. | +| Host-owned closed operations with stable, atomically enforced effect keys and durable receipts | Recommend for the external operation before single-admission continuation. Hosts lacking a concrete executor retain a visible unresolved obligation. | + +No new dependency is proposed. Reuse the existing journal, D1 transaction and conditional-write patterns, runtime status reader, host authentication, execution fence, and inventory. Any later package addition needs the repository package-first and supply-chain checks. + +## Recovery protocol + +1. Before calling the host-owned executor, the trusted host verifies the retained result and current authoritative run. In one D1 transaction, it records the closed operation list, full run/suspension identity, result digest and stable executor keys. A callback payload cannot select the profile or add an operation. A row whose previous host operation is unresolved can be retried only under the executor's atomic idempotency contract. A row whose Mastra resume was admitted cannot be resumed again automatically. +2. For each host operation, persist its key and request digest before sending. The downstream executor atomically records the effect and first result under that key. A repeated key and identical request returns its durable receipt; a changed request fails closed. After a lost response, query by key or retry with the same key. If the executor cannot establish the effect outcome, mark the row unresolved and block drain. +3. After all host receipts are durable, atomically record a one-time `resume-admitted` decision bound to the exact suspension and retained result. Then call the existing trusted `resumeViaRuntime` path once. A crash between admission and the call is intentionally ambiguous and blocks retry. If the call returns, `settleAfterResume` may use its returned witness and authoritative status to settle; a missing witness, unreadable run or ambiguous advancement remains unresolved. No host receipt proves that an arbitrary Mastra tool executed once. +4. On crash or alarm, scan bounded active work and route inspection to the owning thread host. Reconcile only host operations whose downstream executor enforces stable keys. If Mastra resume was admitted, read the journal and authoritative snapshot for diagnosis but do not call `run.resume` again or infer `applied` from an advanced snapshot. Record independent operator evidence before resolving a missing witness. A future automatic continuation protocol requires a separately reviewed runtime version that fences every competing snapshot write and persists attempt-linked provenance atomically with advancement; the pinned version fails this gate. +5. Keep host receipts and uncertain resume records in drain inventory across deploy, rollback, deadline, retention cleanup and migration. An automatic purge cannot erase the evidence that prevents replay. An operator resolution records the independent external evidence and must not invent `applied` from elapsed time. + +The protocol needs one host-only recovery method and a versioned journal migration. Its endpoint or method must use the same internal authentication and stored-principal checks as the current remote-step host route. It must not be a public result route. Versioned schema migration preserves old rows; a legacy `resuming` row without durable attempt/effect evidence is ambiguous and remains blocked. Refuse unknown schema versions. During mixed-version rollout, old code must not claim or settle new-version attempts; stage code and migration so the old version fails closed or keep remote-step dispatch disabled until cutover. Prove that with an actual old/new Worker-version exercise, not an inferred migration rule. + +### Before and target contracts + +The current `RemoteStepJournal.claimResume` writes an expiring lease for the initial result-retained claim, then the host calls the runtime: + +```typescript +const leaseOwner = await reader.claimResume(invocationId, 60_000); +const resumed = await withExecution(execution, () => + durable.resumeViaRuntime({ + runId: ref.runId, + requestedBy: stored.principal.id, +``` + +The current `RemoteStepJournal.reconcileResult` can settle a retained result but returns `pending-quiescence` for `resuming`: + +```typescript +if (invocation.status === 'resuming') return 'pending-quiescence'; +const owner = await this.claimResume(invocationId, 60_000); +return this.settleAfterResume(invocationId, owner); +``` + +The target is an **internal contract sketch**, not a claim that these methods exist. It shows where the host-owned operation can safely retry and where Mastra continuation cannot: + +```typescript +const operation = await reader.prepareHostOperation({ + invocationId, + trustedClosedProfileId, +}); +const receipt = await executor.executeIdempotent(operation); +await reader.recordHostReceipt(operation.id, receipt); +await reader.admitSingleResume(invocationId, operation.id); +const resumed = await withExecution(execution, () => + durable.resumeViaRuntime(boundResumeRequest), +); +await reader.settleAfterResume(invocationId, operation.id, resumed); +``` + +The exact method names and types require implementation design. `admitSingleResume` must persist before `resumeViaRuntime`; any error or crash after admission leaves an unresolved record. The closed executor is host-owned, not a wrapper around arbitrary `tool.execute`. Do not weaken existing approval resume or ordinary agent run APIs to fit this sketch. + +The pinned runtime lacks the generation and provenance path required for automatic `run.resume` replay. `FencedWorkflowsStorageD1.persistWorkflowSnapshot` alone cannot fence all engine writes. A future protocol must inspect `RunnerRuntime.resume` and every installed engine persistence path, including persist, replace and patch, then prove atomic generation refusal and provenance. This plan does not implement that future protocol. + +## Implementation checkpoints + +1. **Capability result and host contract:** The pinned `@mastra/core@1.73.0` audit failed the generic adapter and generation proof. Do not re-run the same proof expecting an arbitrary Mastra continuation to become safe. Identify a concrete host-owned closed operation and executor, list every effect it can perform, and prove atomic idempotency and receipt lookup for that executor. If no concrete operation meets the contract, leave the existing fail-closed result path in place and defer this profile. +2. **Journal and host operation state:** Add a versioned migration, stable operation keys/receipts, one-time `resume-admitted` marker, active index/inventory, and host-only inspection method. Keep old rows intact and ambiguous rows blocked. Authenticate the method; re-read principal, binding, start token, epoch, deadline and authoritative suspension. Keep the current host lock for same-instance ordering, but do not use it as cross-instance evidence. +3. **Closed executor and recovery duty:** Add the host-owned executor only after its downstream idempotency contract is proven. Wire bounded scan and alarm/maintenance wake-up to inspect and reconcile host operations. The duty never re-invokes Mastra after `resume-admitted`. Keep alarm scheduling compatible with the host's existing alarm owner, because a Durable Object has one alarm slot. +4. **Cutover and proof:** Exercise old/new versions, crash and lost acknowledgement on real D1/workerd storage. Keep feature disabled for customer dispatch until the proof matrix below passes. Merge to `dev` through reviewed checkpoints and publish through the Anchorage release process only after the release gates pass. + +## Verification and release gates + +Run each checkpoint's narrowest changed-file checks once, in `/tmp` logs with the content ID and final `EXIT_STATUS`. For the shared journal schema/protocol checkpoint, the quality gate's shared-contract exception selects the owning FlowSafe package suite. Candidate commands, adjusted to the changed-file list at execution time: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run \ + src/do-runner/remote-step-journal.test.ts \ + src/agent-host/thread-host.test.ts \ + src/agent-host/thread-topology.test.ts +pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts \ + packages/flowsafe/src/agent-host/thread-host.ts +pnpm --filter @proofoftech/flowsafe typecheck:test +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe test +``` + +The gate owner redirects each command's complete output to a distinct `/tmp` log, appends its exit status, and gives reviewers the paths. After all checkpoints, run the repository's full suite once, including direct scenarios where the changed contract reaches them. Do not substitute mocked D1 or a second object in one process for the real fresh-isolate proof. + +| Proof | Required observation | +| --- | --- | +| Two resumes in one object instance | One attempt begins; the other sees the durable state, with no duplicate effect. | +| Deploy while old instance is inside a host-owned external operation beyond the 60s lease | New instance never treats lease expiry as quiescence. A repeated host operation uses the stable key and returns one durable executor outcome. | +| Crash before host effect, after effect before receipt, and after receipt before Mastra admission | The executor receipt resolves the host operation or the row stays blocked; no ambiguous effect is repeated under a new key. | +| Crash after `resume-admitted` before call, during call, or after advancing snapshot before journal settlement | No automatic second `run.resume` occurs. A missing returned witness stays unresolved even if the snapshot advanced; operator evidence is required. | +| Identical and conflicting retries | Same operation key plus same request yields the original receipt; changed request fails closed. | +| Old Mastra attempt remains in flight after deploy | No newer Mastra attempt starts; journal and drain inventory preserve the uncertainty. | +| Changed principal, resource mapping, start token, suspension, epoch, deadline, or fence | Recovery refuses before any effect. | +| Legacy and unknown schema rows | Known rows survive migration; ambiguous `resuming` rows stay visible and blocked; unknown versions refuse. | +| Alarm duplicate, exhausted retries, process restart, and no traffic | Maintenance remains scheduled and scans bounded active work without relying on a single alarm delivery. | +| Migration drain and retention | Active/blocked rows remain inventory work; cleanup cannot erase unresolved effect evidence. | +| Closed-profile bypass | An operation outside the enumerated host contract is rejected; arbitrary `tool.execute` remains ineligible for automatic replay. | + +Every implementation checkpoint is T2 for concurrency, authentication, migration and data loss: independent clean-code, QA, architecture and security lanes. This plan received one design-level review when added; that review does not run or simulate its commands. No MUST may remain when a checkpoint closes. A live release requires the concrete executor's idempotency proof and a real old/new Cloudflare rollout exercise. Without those proofs, retain the fail-closed state. Even with them, advertise recovery only for the host-owned operation, not automatic Mastra continuation. + +## Reuse map and excluded work + +- `packages/flowsafe/src/do-runner/remote-step-journal.ts`: canonical suspension identity, retained result digest, D1 migration, conditional settlement and bounded recovery scan +- `packages/flowsafe/src/agent-host/thread-host.ts`: trusted internal route, stored-principal authorization, bound runtime resume and same-instance dispatch lock +- `packages/flowsafe/src/agent-host/thread-topology.ts`: owning thread routing and trusted host context +- `packages/flowsafe/src/do-runner/execution-fence.ts`: deployment epoch and drain boundary +- `packages/flowsafe/src/do-runner/inventory.ts`: active obligation visibility +- `packages/flowsafe/src/agent-runner/durable-agent-runner.ts`: pinned runtime continuation and authoritative status path +- `packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.resume`: runtime re-entry that currently lacks a remote-step attempt generation +- `packages/flowsafe/src/do-runner/fenced-workflows-d1.ts`, `FencedWorkflowsStorageD1.persistWorkflowSnapshot`: partial storage seam; its existence does not authorize replay + +The plan does not change the public result-intake API, approval decision semantics, arbitrary agent tools, customer signing, Fleet Control ownership, payment policy, or external executor implementation. Those are separate owners or cannot meet this replay contract without their own proof. It also does not turn an unknown effect outcome into a terminal success or failure; doing so would hide an unresolved obligation. + +## Appendix: current-state anchors + +The FlowSafe excerpts come from `4096fbd`; the installed Mastra excerpts come from the locked `@mastra/core@1.73.0` package. Reconfirm symbols and semantics, not line numbers. + +- `packages/flowsafe/src/agent-host/thread-host.ts`, `createThreadAgentHost` dispatch and remote-step route: + +```typescript +const withDispatchLock = createFifoLock(); +``` + +The route later calls `withDispatchLock`; the lock lives in one host object instance. The remote-step route currently refuses an active `resuming` row: + +```typescript +if (invocation.status === 'resuming') + throw new AgentHostRequestError( + 503, + 'remote step resume awaits executor quiescence', + ); +``` + +- `packages/flowsafe/src/do-runner/remote-step-journal.ts`, `claimResume` and `settleAfterResume`: + +```typescript +SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ? +``` + +```typescript +AND resume_lease_owner = ? AND permit_generation = ? +AND permit_epoch IS ? AND start_token = ? +``` + +These predicates protect the journal row but do not prove the external effect or the outgoing runtime leg stopped. + +- `packages/flowsafe/src/do-runner/remote-step-journal.ts`, `recoverableResults` and `reconcileResult`: the current scan selects retained/resuming work through the v3 partial index, while `reconcileResult` returns `pending-quiescence` for an active `resuming` row. The journal migration must preserve those records and add durable attempt evidence before any automatic reclaim. + +- `packages/flowsafe/src/agent-runner/durable-agent-runner.ts`, `FlowsafeDurableAgent.resumeViaRuntime`, and `packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.resume`: the current call passes no attempt token: + +```typescript +const summary = await this.#runtime.resume( + this.getWorkflow().id, + options.runId, + resumeOptions, +); +``` + +`RunnerRuntime.resume` then enters the pinned engine with `run.resume({ step: options.step, resumeData: options.resumeData, requestContext })`. It has no remote-step attempt generation. Concurrent replay is excluded from this plan. + +- `packages/flowsafe/src/do-runner/fenced-workflows-d1.ts`, `FencedWorkflowsStorageD1.persistWorkflowSnapshot`: the current storage seam accepts `PersistInput` without a remote-step attempt generation: + +```typescript +override async persistWorkflowSnapshot(args: PersistInput): Promise { + const scope = this.#scopes.getStore(); +``` + +The installed `node_modules/.pnpm/@mastra+core@1.73.0_express@5.2.1_zod@4.4.3/node_modules/@mastra/core/dist/agent/durable/index.js`, `executeDurableToolCalls`, calls the supplied tool closure: + +```javascript +const result = await tool.execute(toolCall.args, { +``` + +The installed `dist/workflow-event-processor-LHwOlFwA.js` also invokes a tool closure from its workflow tool execution path: + +```javascript +return tool.execute(inputData, toolContext); +``` + +The installed `dist/agent-D-eTzWxF.js` enters a resumed agent run and can resume nested work: + +```javascript +const executionPromise = resumeContext ? run.resume({ +``` + +```javascript +else if (resumeNested) res = await run.resume({ +``` + +The pinned engine's workflow writes include persist, replace and patch paths without a remote-step generation. The closure calls above are enough to rule out an effect adapter that wraps only the FlowSafe host route. A future automatic-continuation proposal must inspect all engine write paths and bind its advancing write to durable resume provenance; a separate post-write journal update cannot serve as the receipt. + +## Amendment: execute the host-owned notification operation (2026-10-13) + +**Goal and status.** Implement recovery of a selected host-owned notification enqueue after a lost acknowledgement or host restart. `D1NotificationsStorage.createNotificationOnce` is the selected executor; it inserts the notification and immutable request receipt in one D1 batch. A native two-Durable-Object contention fixture has proved local atomicity. No production caller or deployed D1 proof exists. This amendment supersedes the earlier statement that no executor is selected; it does not authorize automatic Mastra `run.resume` replay. + +### Current baseline and constraints + +- Branch: `plan/flowsafe-remote-step-broker`; amendment base: `97237039350137f42bdc603303f3496f3e7f1431` (2026-10-13). Reconfirm each appendix anchor against the checked-out commit before editing. The historical `4096fbd` baseline above remains a record, not an edit target. +- Use the repository's locked Node, pnpm, `@mastra/core@1.73.0`, Cloudflare Workers, Durable Objects and D1. Work in a linked worktree. Put checks and build artifacts under `/tmp`, and record an exit status in each log. ASP signing, dispatch policy and customer UI remain in ASP. +- The 100,000 retained notification receipt cap is approved and implemented per configured D1 table prefix. One D1/prefix per deployment is the supported topology; a multi-prefix deployment must refuse this profile until deployment-wide accounting is designed. Measure D1 `rows_read` and latency near the cap before release; do not silently purge receipts. +- A workflow-target v4 journal and private workflow route exist, but the selected executor and target-thread recovery below apply to an **agent** invocation. The workflow-target continuation remains a separate gate. The existing `recoverableResults` deliberately selects `target_kind = 'agent'` while drain inventory retains both kinds. + +### Operation identity and state contract + +The host selects the notification profile from trusted configuration, never from a result body. The profile fixes the notification source, kind, recipient thread/resource, identity and payload construction. Name and version one canonical projection, `notification-record-values-v1`, inside `notifications-d1.ts`; both the journal preparation and `createNotificationOnce` use its `JSON.stringify(recordValues(record))` UTF-8 bytes after defaults and field conversion. Persist the exact `CreateNotificationInput`, projection version, projected bytes and their SHA-256 digest before any effect attempt. The executor computes its receipt `request_digest` from those same bytes and returns or exposes that digest for exact comparison with the journal. Unknown projection versions refuse. Its `operationKey` binds deployment ID, journal invocation ID, run start token, suspension identity and the profile's operation ID; it is stable across host restarts and retries. The journal owns the mapping from this identity to one notification ID. A new resume lease owner does not change either ID. The executor's existing digest check refuses a reused key with changed notification input; the journal compares its stored digest to the executor receipt on every retry. + +Use a versioned journal record with states for prepared operation, effect admitted, receipt recorded, resume admitted and unresolved continuation. Keep terminal settlement and retained drain inventory. Preparation stores identity and request but grants no effect authority. Immediately before the first executor call, one conditional D1 write admits that exact host operation. Its SQL predicate checks the current fence row, deadline, stored principal and binding, start token, result digest, journal revision and authoritative suspension identity in the same D1 database transaction; the host's preceding run read is diagnostic, not the authority check. Require a schema path that can compare the authoritative snapshot's current suspension and generation in that transaction. If it cannot, leave the profile disabled. A zero-row or unreadable admission result refuses the effect. The successful write is the effect's linearization point: a later deadline or fence change cannot erase an already admitted operation, whose receipt remains drain work. If the admission write's outcome is ambiguous, reread the durable operation state before any executor call; never infer success from a timeout. After `createNotificationOnce` returns, persist its digest-bearing receipt with an equally bound compare-and-swap. A lost response is recoverable by calling the same method with the exact key and input only for a durably admitted operation; it returns the original receipt or refuses a changed request. Add a read-only receipt lookup inside `D1NotificationsStorage` so recovery can observe an old effect without creating a new one. If no receipt exists and admission was refused or cannot be proved, leave the row unresolved. A cap error before insertion leaves an admitted operation visible; existing-key retry still works at the cap. + +Persist a **one-time resume-admitted marker** before entering Mastra. It binds the exact retained result, suspension and operation receipt. The journal must permit one transition into admission and refuse every later attempt, including after lease expiry, deploy and restart. A crash after admission but before the call leaves an unresolved continuation; neither the receipt nor an advanced snapshot proves a second `run.resume` safe. The selected notification can recover automatically, but Mastra continuation cannot until its snapshot writes have generation fencing and attempt-linked provenance. This limit is an explicit release behavior, not a pending implementation shortcut. + +### Edit anchors and before → after + +The following excerpts are verbatim at the amendment base. Method names in the target sketches specify the contract to implement; inspect the actual types and line positions before editing. + +1. **Executor:** `packages/flowsafe/src/signals/notifications-d1.ts`, `D1NotificationsStorage.createNotificationOnce` currently enters with: + + ```typescript + async createNotificationOnce( + operationKey: string, + input: CreateNotificationInput & { id: string; createdAt: Date }, + ): Promise<{ + operationKey: string; + notificationId: string; + threadId: string; + }> { + ``` + + The existing D1 batch inserts a receipt first and gates the inbox insert on `changes() = 1`. Reuse this direct method. Extract its `JSON.stringify(recordValues(record))` projection into one versioned function that the trusted host and executor both call, without changing the bytes or receipt behavior. The adapter constructs and freezes arguments from the prepared journal row. Expose the receipt digest and a read-only lookup under the same storage boundary; do not replace the D1 effect transaction. + +2. **Journal:** `packages/flowsafe/src/do-runner/remote-step-journal.ts`, `reconcileResult` currently starts: + + ```typescript + if (invocation.status === 'resuming') return 'pending-quiescence'; + const owner = await this.claimResume(invocationId, 60_000); + return this.settleAfterResume(invocationId, owner); + ``` + + Keep this fail-closed path for ordinary remote steps. Add a separate notification operation path whose order is `prepare` → `admitEffect` → `createNotificationOnce` → `recordReceipt` → `admitResume` → one trusted runtime call → witnessed settlement. `admitEffect` owns the atomic authority predicate described above. Recovery may repeat the executor with the stored key and request after proved effect admission and before `admitResume`; it must never use `claimResume` to reclaim a previously admitted Mastra call. + +3. **Host:** `packages/flowsafe/src/agent-host/thread-host.ts`, the retained-result route currently enters Mastra after: + + ```typescript + const leaseOwner = await reader.claimResume(invocationId, 60_000); + const resumed = await withExecution(execution, () => + durable.resumeViaRuntime({ + ``` + + A notification-profile branch runs inside the same target thread's `withDispatchLock`. It re-reads the authoritative run and stored principal/binding, then requires the conditional `admitEffect` D1 write before calling `createNotificationOnce` from trusted host configuration. It records the receipt, rechecks authority for continuation, then commits the one-time marker before this runtime call. The ordinary branch retains its current fail-closed behavior. A host-only recovery method uses the same lock, authentication and stored-principal checks; it exposes no public result route. + +4. **Maintenance:** `createThreadAgentHost.recoverOwnership` currently uses: + + ```typescript + return withDispatchLock(async () => { + const storage = options.stateStorage(); + ``` + + The recovery method shares this serialization point. A scheduled maintenance Worker advances a durable, bounded D1 cursor over active notification-profile operations and routes each selected item to its owning thread Durable Object. The thread method shares the existing dispatch lock. Its alarm owner also wakes pending local work during no traffic; extend its existing `recoverOwnership` scheduling and delete conditions so notification work cannot be erased by owner recovery. Process a fixed batch, persist the next cursor, wrap at the end and rearm while work remains. Do not install a second independent alarm owner or scan retained terminal history. A failed item stays visible; one item cannot prevent progress through later IDs. A lease expiry schedules inspection but never authorizes another Mastra call. + +5. **Migration and mixed writers:** `remote-step-journal.ts` currently recognizes version 4 with: + + ```typescript + if (record(rows[0])?.id === 1 && record(rows[0])?.version === 4) { + ``` + + Its current update trigger is: + + ```typescript + const REMOTE_STEP_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; + ``` + + Migrate known v1–v4 rows into a validated v5 layout without losing permits, retained results or unsettled obligations. Add a required `write_generation` initialized to a positive value for migrated rows. Require v5 inserts to supply it and v5 updates to increase it through a D1 trigger; an already-initialized v4 writer preserves the value and is refused. Validate the table, indexes, trigger and exact schema version before reads or writes. Unknown or malformed layouts refuse. An old Worker that initializes after migration sees unknown version; prove both old-writer cases in an old/new Worker exercise before enabling dispatch. Legacy `resuming` rows without the new admission evidence remain unresolved. + +### Ordered implementation checkpoints + +1. **Journal contract and migration:** Add the v5 operation fields, conditional effect-admission transition, active index and drain inventory projection. Preserve existing v4 behavior and data. Refuse stale writers with the generation trigger and test both old-initialized and fresh old versions. Prove that the admission SQL can compare current fence and authoritative suspension in the same D1 transaction; otherwise keep the profile disabled. Use `remote-step-journal.ts` migration and conditional-write patterns rather than a parallel journal. +2. **Trusted host executor:** Add the closed notification profile to the target thread host. Derive all effect fields from trusted stored context, share the executor's versioned projection and use `D1NotificationsStorage.createNotificationOnce` directly after proved effect admission. Save its digest-bearing receipt. Add the single-admission transition before runtime entry. Keep `resumeViaRuntime` one-shot and report ambiguous outcomes as unresolved. +3. **Recovery duty:** Add host-only inspection, a scheduled D1 cursor scan and target-thread alarm wake through the existing owner recovery scheduler. Repeat a previously admitted effect with its stable key; after a refused or ambiguous admission, use read-only receipt inspection and keep missing evidence unresolved. Preserve progress after restart, duplicated alarms and an item that repeatedly fails. Never retry admitted continuation. +4. **Cutover proof:** Keep the profile default off until native fresh-isolate and deployed D1 proof, old/new Worker refusal, authenticated host route negatives, cap latency measurement and drain inventory checks pass. Review and release via the Anchorage `dev` process; ASP integration follows a published, release-age-eligible FlowSafe version. + +No new package is needed. The existing direct D1 notification storage is the smallest integration boundary; a new executor package would duplicate its atomic receipt contract. An external notifier lacks this repository's D1 transaction and cannot satisfy this plan without a separate proof. General Mastra replay, workflow-target continuation, arbitrary tools, payment, customer dispatch and ASP policy are excluded for the reasons above. + +### Scoped verification and proof + +Select tests from the checkpoint's changed paths and shared contracts. The v5 schema is a shared contract, so run the owning FlowSafe package suite once for that checkpoint under the quality-gate exception. For other checkpoints use the named files below, then changed-file Biome, package typecheck/build when the tool cannot retain config with file targets. Redirect each command to a distinct `/tmp` log with the content ID and `EXIT_STATUS` footer. Run the repository full suite once after the final checkpoint, not at intermediate checkpoints. + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run \ + src/do-runner/remote-step-journal.test.ts \ + src/agent-host/thread-host.test.ts \ + src/signals/notifications-d1.test.ts +pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts \ + packages/flowsafe/src/agent-host/thread-host.ts \ + packages/flowsafe/src/signals/notifications-d1.ts +pnpm --filter @proofoftech/flowsafe typecheck:test +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe spike:verify +``` + +The gate owner narrows these candidates to the changed-file list and records the exact commands in its logs. All implementation checkpoints are T2: independent clean-code, QA, architecture and security lanes review the checkpoint diff and the called journal, host, D1 notification and alarm contracts. Review this amendment once at design level before implementation. + +The live proof must show one D1 notification and receipt when two fresh object instances race, including a lost acknowledgement, same-key retry at the last receipt slot, changed-request refusal and crash before/after receipt. Change the fence, deadline and suspension after preparation but before effect admission: the D1 write must refuse and the inbox must stay empty. Interrupt the admission response: recovery must read its durable result before any effect. Compare the journal's projected request bytes and digest with the executor's receipt digest, including defaulted and date fields. The proof must also show a persisted cursor eventually reaching work beyond one batch, duplicate alarm delivery, restart and no-traffic wake, an old Worker unable to write v5 rows, and blocked drain while Mastra admission is uncertain. Inspect the authoritative D1 receipt and inbox after each fault. Local workerd evidence does not substitute for a deployed D1 exercise. Successful notification recovery never implies automatic continuation; a test that calls `run.resume` twice fails this plan even if it observes one notification. diff --git a/docs/plans/2026-10-13-workflow-remote-step-journal.md b/docs/plans/2026-10-13-workflow-remote-step-journal.md new file mode 100644 index 00000000..e4e9cb8c --- /dev/null +++ b/docs/plans/2026-10-13-workflow-remote-step-journal.md @@ -0,0 +1,81 @@ +# Workflow-target remote-step journal + +> Content type: implementation plan. Audience: FlowSafe maintainers. Status: planned; customer dispatch remains disabled. + +## Baseline and environment + +- Branch `plan/flowsafe-remote-step-broker` at `ee4003a42882c0d1282004d5fed0274a5b4e2939` (2026-10-13). Reconfirm the appendix anchors against the checked-out commit before editing. The concurrent state-egress checkpoint owns `src/host-kit/state-egress.ts` and `flowsafe-worker.ts`; this plan does not edit them. +- Node and pnpm come from the root manifest. Work in the linked worktree with the shared package store. Capture checks in `/tmp` with a final `EXIT_STATUS` line. D1 `batch()` is transactional according to [Cloudflare's API documentation](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch). +- This is a default-off journal contract. No workflow host route, ASP permit authority, customer dispatch, or automatic `run.resume` is enabled by this checkpoint. + +## Decision + +Extend the existing `RemoteStepJournal` D1 authority to distinguish `agent` and `workflow` target rows. Preserve every v3 agent row and its permit/result state. A workflow row uses the trusted start provenance `{kind:'workflow', id: workflowId}` and has no agent, thread, or resource ID. Its suspension carries a versioned marker and exact JSON request bytes; the journal stores a SHA-256 digest of those bytes. Registration and permit consumption compare the same authoritative snapshot, principal, deployment, start token, fence generation, suspension identity, request bytes, and deadline as the agent path. The first retained result remains immutable under the consumed permit digest. + +The workflow suspension wire shape is `suspendPayload[stepPath.join('.')].remoteStep = { marker: 'flowsafe.remote-step.workflow/v1', request: }`, alongside `type: 'suspension'`. The trusted workflow host must emit this shape; user code cannot obtain authority merely by constructing the payload. The marker is a versioned parser selector, not proof of ownership. Confirm it against the ASP v2 author before wiring a caller. + +Reject synthetic agent/thread/resource IDs: they would make a workflow row appear to be an agent row and let agent-host routing or cleanup misinterpret its authority. Reject a second workflow permit table: two authorities would need a new cross-table uniqueness and replay protocol. Reject a preflight-only public API: it could be mistaken for dispatch authority while storing no permit or result. Keep general Mastra `run.resume` and automatic recovery fail-closed: the pinned `@mastra/core@1.73.0` does not fence all snapshot write paths by remote attempt generation (see [durable-resume plan](2026-10-12-remote-step-durable-resume-recovery.md)). + +## Reuse and package decision + +Reuse `RemoteStepJournal`, `ExecutionFenceStore`, `assertExecutionPrincipal`, `isPathSafeId`, authoritative runtime status/start state, D1 conditional updates, and Web Crypto SHA-256 in `src/do-runner/remote-step-journal.ts`. The existing direct dependencies are `@mastra/core@1.73.0`, `@mastra/cloudflare-d1@1.4.0`, and `jose@6.2.8`; none provides this host-specific journal schema and CAS contract. A new package adds no useful boundary. A separate custom store duplicates the existing permit protocol. The narrow custom change is a discriminated target row and suspension selector in the existing journal. No dependency changes. + +## Checkpoints + +1. **Schema v4.** Rebuild the journal table transactionally because SQLite cannot remove `NOT NULL` from agent identity columns in place. Add `target_kind`, `suspension_marker`, `request_digest`, and `write_revision`; make agent/thread/resource nullable. Copy v3 rows as `agent`, with their original marker, `NULL` request digest, and revision zero; preserve their status, permit, result, lease, and settlement fields. The agent branch continues to compare the exact stored `request_json`; it does not require a digest on migrated rows. A workflow row requires a digest. Add separate partial unique indexes for agent and workflow suspension identities, plus the recovery index. Validate exact schema/index/trigger definitions and refuse unknown versions. A v4 update increments `write_revision` in the same statement as its state transition. A D1 trigger rejects updates that do not advance it; v3 code omits that column, so an old in-flight isolate cannot mutate a row after migration. `target_kind NOT NULL` rejects its inserts. Test a v3 initialized journal instance attempting both operations after another instance migrates. A v4 binary reads migrated agent rows unchanged. Keep rollback to a v4-aware binary; an old v3 binary fails closed. +2. **Workflow storage port.** Add a workflow-specific registration method, a workflow getter, and typed workflow invocation. Registration accepts a trusted workflow address and original principal, verifies start provenance and exact suspension, then performs a snapshot-guarded insert/update. Existing agent-host `get` and `register` remain agent-only. Shared permit, consume, result and retained-result methods dispatch by target kind inside the same journal authority. No caller is wired yet. +3. **Proof.** Use real SQLite-backed D1 semantics for old-row preservation, conflicting registration, restart/reopen, permit replay, wrong request/principal/start token/fence/deadline, and result conflict. Add a workerd fresh-isolate scenario before claiming a live persistence proof. The default-off package may merge without a customer route; it may not be described as a working WFP dispatch path until the authenticated host route and workerd proof land. + +## Verification + +After each code checkpoint, derive scoped checks from changed paths. The journal schema is a shared contract, so the owning package's full test suite applies once under the quality-gate exception. Candidate commands: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run src/do-runner/remote-step-journal.test.ts +pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts packages/flowsafe/src/do-runner/remote-step-journal.test.ts +pnpm --filter @proofoftech/flowsafe typecheck:test +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe test +``` + +Each command needs its own `/tmp` log and actual exit status. Done for this checkpoint means schema upgrade preserves v3 rows, workflow register/permit/result behavior passes, agent-host behavior passes, and no public workflow dispatch or resume route is enabled. The top-level quality-gate owner runs T2 clean-code, QA, architecture, and security lanes, then commits. Live enablement additionally requires a real D1/workerd restart and replay proof, authenticated host binding, and ASP grant verification. + +## Appendix: current-state anchors + +At the baseline SHA, `src/do-runner/remote-step-journal.ts` declares an agent-only address and row: + +```ts +export interface RemoteStepAddress { + deploymentId: string; + tablePrefix: string; + workflowId: string; + agentId: string; + threadId: string; + resourceId: string; + runId: string; +} +``` + +The `RemoteStepJournal.initialize` table has `agent_id TEXT NOT NULL`, `thread_id TEXT NOT NULL`, and `resource_id TEXT NOT NULL`, with a unique key containing those columns. Its `readSchema` accepts version 3 or 2 and exact column/index definitions. Before → after: v3 agent-only fields and uniqueness → v4 discriminated target with nullable workflow agent fields and partial target-specific unique indexes. A simple nullable-column alteration is insufficient because the existing unique key cannot represent workflow identity. + +`RemoteStepJournal.#execution` currently refuses workflow starts: + +```ts +if ( + state?.storage !== 'd1' || + target?.kind !== 'agent' || + !isPathSafeId(target.resourceId) +) + throw new RemoteStepRefusal( + 503, + 'remote step run identity is unreadable', + ); +``` + +Before → after: the agent branch retains its exact checks; the workflow branch requires `target.kind === 'workflow'`, `target.id === address.workflowId`, no agent-start provenance, matching original principal, deployment/table prefix/run ID/start token, and the current execution fence. It never fills agent fields from workflow IDs. + +`selectedSuspension` currently matches `toolCallSuspended.remoteStep === REMOTE_STEP_MARKER` and reads `tool.request`. Before → after: keep that agent selector and add a separate workflow selector for the versioned `remoteStep.marker` and `remoteStep.request`, with one selected path and exact request bytes/digest. + +`RemoteStepJournal.issuePermit` and `consumePermit` compare `suspension.requestJson` against `JSON.stringify(invocation.request)` and conditionally update the journal with `request_json = ?`, snapshot equality, and `executionFenceAdmissionSql`. Before → after: retain those CAS predicates and add the target kind and marker. The workflow branch also compares its stored request digest; legacy agent rows continue to use exact request bytes without a digest. Every v4 state update advances `write_revision` under a D1 trigger, which rejects a previously initialized v3 isolate's updates. `recordResult` keeps `permit_digest` and immutable result comparison. No code path retries `run.resume` from a retained result. + +`src/do-runner/execution-admission.ts` defines `StartIdentity.target` as a union with `{kind:'workflow',id}`. `src/do-runner/runtime.ts` already checks that a recovered workflow target ID equals its execution workflow ID. The new journal branch uses that trusted provenance rather than creating another identity format. diff --git a/docs/proposals/README.md b/docs/proposals/README.md index cce08c8c..cf75d3ff 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -4,6 +4,7 @@ Documents in this directory are design records. Read each document's status befo - [Breakwater improvement roadmap](breakwater-improvement-roadmap.md): mixed shipped and planned work - [Durable result gate](durable-result-gate.md): proposal; not implemented +- [Remote step broker](flowsafe-remote-step-broker.md): implementation plan; not implemented - [Mature package adoption plan](mature-package-adoption-plan.md): implemented and verified - [Model gateway policy](model-gateway-policy.md): partially implemented; provider, budget, model-egress, structured-output, and scoring gates remain planned diff --git a/docs/proposals/flowsafe-remote-step-broker.md b/docs/proposals/flowsafe-remote-step-broker.md new file mode 100644 index 00000000..291c2108 --- /dev/null +++ b/docs/proposals/flowsafe-remote-step-broker.md @@ -0,0 +1,298 @@ +# Proposal: resume a durable agent after a remote step + +> Status: implementation plan, not shipped. Content type: design proposal. Audience: FlowSafe maintainers and hosts that execute a remote agent tool. + +This plan adds a generic durable handoff for a tool that waits for an external result. FlowSafe records the suspended run and an invocation, accepts one bound result, then resumes through its existing runtime. The host supplies its own authentication, dispatch transport, and policy; this proposal adds no customer, Workers for Platforms, signing, or capability policy to Anchorage. + +## Baseline and outcome + +Baseline: `plan/flowsafe-remote-step-broker` from `origin/dev` at `7d2ad352b0e4422811faf4ef4240a9172b360cce`, read 2026-10-11. Reconfirm the anchors below before editing. Use Node and pnpm versions from the root manifest. Work in a linked worktree with the shared pnpm store. The production Worker uses D1 and Durable Objects; in-memory Vitest results alone cannot prove restart behavior. The existing release is `@proofoftech/flowsafe@0.27.0` with `@mastra/core@1.73.0` as an exact peer. + +Done means a remote-step tool can suspend, persist a dispatchable invocation, survive Durable Object eviction, accept a result bound to that exact suspension, and resume through `RunnerRuntime`. Duplicate dispatch and duplicate result delivery converge on the same invocation. A changed suspension or unrelated run cannot resume. Migration cutover waits for remote dispatch claims to finish or be confirmed stopped; changing the FlowSafe fence alone cannot stop an external callback already in flight. “Once” applies to accepted result and resume admission; an external side effect still needs the host's stable invocation idempotency key. + +The first implementation checkpoint must prove that core 1.73.0 persists a tool's `workflow.suspend()` shape and resumes it with `workflow.resumeData` after a fresh host instance. If that probe fails, stop before publishing a broker API. A workflow-only suspension would not satisfy this agent-tool objective. + +## Ownership and package choice + +FlowSafe owns the durable run, suspension and resume contract. The deployment host owns authenticated result intake and the external executor. Breakwater continues to own connector authorization; Fleet Control owns deployment and migration; a customer platform owns artifact validation and grants. The generic broker never treats a caller's `runId`, `principal`, tool name, or result as authority by itself. + +Package-first check, 2026-10-11: + +| Candidate | Fit and decision | +| --- | --- | +| Existing FlowSafe `AgentThreadTopology`, `RunnerRuntime`, D1 storage, and execution fence | Reuse. They already provide authenticated internal Durable Object routing, exact suspension comparison, storage, and restart-aware resume. The broker needs a small state table and host adapter at this boundary. | +| Declared peer `@mastra/core@1.73.0` | Use its durable tool suspension, subject to the pinned-version probe. Do not call its direct resume API; FlowSafe's wrapper rebuilds the registry and enforces runtime admission. [Mastra durable agent guide](https://mastra.ai/docs/harness/durable-agents) and [Mastra tool suspension source guide](https://github.com/mastra-ai/mastra/blob/main/packages/core/src/tools/hitl.md) describe the general behavior, not the exact pinned implementation. | +| Cloudflare Durable Object alarms | Use the existing host's alarm and D1 patterns for retry and recovery. Alarms are at least once, so the journal and executor must tolerate repetition. [Cloudflare alarm reference](https://developers.cloudflare.com/durable-objects/api/alarms/). | +| Separate queue/workflow broker package | Reject for this seam: it cannot compare the run's authoritative suspension or execution fence without routing back through FlowSafe. It adds a second owner for state that must be correlated with the existing D1 snapshot. No new package is justified. | + +No dependency or package-manager change is proposed. An adapter with a real job remains custom: it validates the broker envelope and joins D1 state to the authoritative FlowSafe run. It is narrower than another broker runtime. The source links describe current upstream behavior; repeat the pinned-core probe because current upstream code may differ from 1.73.0. License and install exposure stay at the declared FlowSafe dependency set. The candidate has no added native component or lifecycle script; the broker adds one D1 table, two bounded recovery scans, and migration/cutover obligations. Search security advisories and the lockfile again if implementation later changes dependencies. + +## Contract and state transitions + +Expose a host-only `RemoteStepBroker` interface from the FlowSafe agent-host subpath. The host supplies a trusted dispatch callback and a trusted result-intake adapter; FlowSafe never mounts a public result route. The unique suspension key is `{deploymentIdentity, tablePrefix, workflowId, agentId, threadId, resourceId, runId, stepPath, suspendedAt, resumeCount}`. These fields come from a storage-backed run read and verified host binding, not from the executor. `invocationId` is a separate opaque identifier generated once for the inserted key and supplied to the executor as its idempotency key. The database's unique constraint excludes `invocationId`; a concurrent insert conflict reads and returns the winning row. The deployment/table fields prevent a reused run id in another generation from claiming an earlier invocation. + +Persist states `pending`, `dispatched`, `result-recorded`, `resuming`, and `settled`, with a terminal `failed` record for a permanent refusal. A D1 uniqueness constraint on the suspension key admits one invocation. A `resuming` row carries a lease owner and expiry, not permanent ownership. The result record stores a bounded, JSON-safe value or a bounded external result reference, a digest, and the exact invocation identity; it does not store arbitrary headers or credentials. Its configured retention must outlive the run's retry window and follow the deployment's purge policy. Limit size and shape at the host ingress and again in FlowSafe before D1 writes. Set the byte limit from existing D1/run bounds after the tool probe; reject oversized input rather than truncate it. + +1. The tool suspends with a reserved, versioned remote-step marker and a static host-owned step key. The host uses `RunnerRuntime.authoritativeStatus()` or the thread host's storage-backed equivalent and creates the invocation only after the suspension is durable. A periodic, bounded scan of the owning deployment's `mastra_workflow_snapshot` rows selects durable agent runs suspended on that marker, then re-reads each through the authoritative host before inserting any missing row. The scan has a persistent cursor and runs even with no broker rows or armed broker alarm. An unreadable row stops that candidate without dispatch and remains visible for retry. Do not use `RunnerRuntime.status()`, which can fall back to memory, for a dispatch decision. +2. A D1 insert or read returns the same invocation for a repeated observation of the same key. The broker transactionally claims a short-lived dispatch permit against the current fence state and mutation epoch before calling the host dispatch adapter. The adapter receives the stable `invocationId` and permit generation. The executor calls the trusted host gate before its effect; that gate atomically consumes the permit once under the same deployment epoch and records its acceptance. The fence-row read, epoch comparison, and permit transition must share one D1 transaction with the fence mutation protocol; a read followed by a separate permit write is insufficient. A repeated gate call returns an `already-consumed` receipt without authorizing another effect. After a lost acknowledgement or expired permit, the broker may issue a new generation for the same invocation only after the prior claim is closed or its executor has confirmed quiescence; a new permit cannot reuse a prior epoch. The executor must deduplicate the effect by `invocationId` across permit generations. If it cannot, the host refuses remote-step dispatch. +3. Trusted result intake checks the invocation id and the complete suspension identity against the journal and current authoritative run. It inserts the first result by compare-and-swap. Identical redelivery returns the stored outcome. Different bytes or digest for the same id refuse with conflict. A result cannot select another step. +4. A recovery duty claims `result-recorded` or an expired `resuming` lease, re-reads the execution fence and `authoritativeStatus()`, and sends an internal resume with the exact `expectedSuspension`. It restores the original stored execution principal and current permissions, never the result sender's identity. It marks `settled` only after the authoritative run state proves the resume advanced or ended. If the resume response is lost, an expired lease triggers an authoritative status read: unchanged suspension retries the same recorded result; advanced or terminal state settles without replay; unreadable state retains the row for retry. A new suspension gets a new fingerprint and invocation. +5. `migration-locked` and an unreadable fence stop new permit claims and resume admission. `draining` permits completion of an existing run under FlowSafe's current fence contract. `proof-only` admits only its nominated run. Entering migration lock revokes unconsumed permits by epoch, then waits for consumed permits and in-flight executor callbacks to report settled or confirmed stopped before the migration inventory can be empty. If the executor cannot report quiescence, cutover blocks; it does not infer safety from permit expiry. Include pending and active broker rows in FlowSafe's read-only drain inventory and table census. A terminal run, canceled run, expired result, or changed fingerprint refuses the old result without starting work. + +The broker deadline is the earlier of a stored run deadline, when one exists, and a required host-configured remote-step deadline. The dispatch permit's shorter expiry never extends that result deadline. At deadline or cancellation, stop issuing permits and resume retries, mark the journal terminal after an authoritative run read, and request best-effort executor cancellation. A cancel request does not prove an external effect stopped. Keep its active claim in drain inventory until the executor confirms completion or stop; otherwise cutover remains blocked. Retain the terminal journal long enough to refuse late result delivery and permit replay. + +Do not claim atomicity between Mastra's snapshot write and the broker's D1 row. The periodic snapshot scan closes that gap by creating an invocation from a durable suspension; it never dispatches on a transient in-memory event. Do not claim exactly-once external execution without an idempotent executor. A lost resume acknowledgement leaves `result-recorded` or `resuming` for a later authoritative status read. + +### Risky edit: resume authority + +Current `AgentThreadTopology.resume` accepts an `ApprovalRecord` and `ApprovalDecision`, then sends `defaultResumeData(record, decision)` with `expectedSuspensionFor(record)`. The target host reauthorizes the stored principal and calls `durable.resumeViaRuntime()`. A remote result is not an approval and must not fabricate either record. Add a distinct host-only result-resume method whose request carries the journaled invocation and result. It must reuse the target host's run ownership, catalog authorization, `withExecution`, and `resumeViaRuntime` path, with mandatory `expectedSuspension`. Do not broaden the existing public agent router or optionalize the fingerprint for this method. + +Before (`thread-topology.ts`, `AgentThreadTopology.resume`): + +```typescript +resume( + context: ActorContext, + record: ApprovalRecord, + decision: ApprovalDecision, +): Promise; +``` + +After, as a separate interface rather than changing approval semantics: + +```typescript +resumeRemoteStep( + context: ActorContext, + invocation: StoredRemoteStepInvocation, + result: RecordedRemoteStepResult, +): Promise; +``` + +The implementation must not accept an arbitrary caller-constructed `StoredRemoteStepInvocation`; the topology loads or validates its D1 row by id before sending. The thread host independently reads the same row and authoritative snapshot before execution. If adding a separate entry path to `AGENT_ENTRY_PATHS`, treat it as a continuation like `approval.resume`: a principal kind that could start the run may resume unless it has been removed from the catalog. Never classify the result sender as the run principal. + +## Implementation checkpoints + +### 1. Pinned-core suspension probe + +Use a deterministic model and a tool with `suspendSchema`/`resumeSchema`. Assert that the tool's `workflow.suspend()` reaches `RunSummary` with a distinguishable payload, survives a new FlowSafe host instance on persisted storage, and receives only the recorded result after `resumeViaRuntime`. Exercise refusal for malformed resume data and a second suspension. Add this to the existing real-loop agent tests; do not add a production export for the test. If the pinned core cannot safely resume that tool, document the failure and choose a supported workflow step boundary in a new reviewed plan. + +### 2. Journal and dispatch admission + +Add the D1 broker table through the existing FlowSafe provisioning/migration path. Use a versioned canonical envelope, uniqueness on the suspension key excluding `invocationId`, CAS result writes, bounded payloads, and a stable invocation id. Add the row to the inventory work category and schema census. Reuse the current D1 error and transaction patterns; no new parser or cryptographic primitive. Reconcile a suspension after a crash before insertion from the periodic snapshot scan, including when no broker row exists. Recover dispatch after acknowledgement loss through the permit protocol. Dispatch requires an authoritative status and a transactionally claimed permit at the current fence epoch. The executor contract requires idempotency by `invocationId` and an authenticated acknowledgement of quiescence. + +### 3. Result and runtime continuation + +Add a host-only topology method and target route. Validate deployment identity, internal request authentication, owner, thread/resource/run correlation, journal identity, and current suspension before `resumeViaRuntime`. Restore the original principal and current catalog policy. Bind the result as resume data to the exact tool step. Preserve the existing approval resume route. Record a durable outcome after reading authoritative status, including when the resume response is lost. + +### 4. Recovery and public documentation + +Connect two bounded scans to the existing maintenance alarm pattern: the periodic snapshot scan finds orphaned remote suspensions, and the broker-row scan recovers `pending`, `dispatched`, `result-recorded`, and expired `resuming` leases. Neither depends on an existing broker row to schedule its successor. Recover from isolate loss; do not retry a permanent refusal as execution. A timeout or cancellation requests best-effort executor cancel and keeps active claims visible until confirmed quiescent. Document the generic host contract, permit generations and epoch revocation, the requirement for external idempotency, fence/cutover behavior, retention, and the absence of a public result route. Add a FlowSafe minor changeset for the new API after behavior passes the real-worker proof. + +## Reuse map and exclusions + +| Existing code | Reuse | +| --- | --- | +| `packages/flowsafe/src/agent-host/thread-topology.ts`, `createAgentThreadTopology` | Authenticated internal routing and owner-checked status | +| `packages/flowsafe/src/agent-host/thread-host.ts`, `createAgentThreadHost` | Stored principal, current catalog permission check, serialized execution | +| `packages/flowsafe/src/agent-runner/durable-agent-runner.ts`, `resumeViaRuntime` | Fresh-host registry rebuild and runtime resume | +| `packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.authoritativeStatus` and `resume` | Storage-backed state, fence and exact suspension check | +| `packages/flowsafe/src/do-runner/execution-fence.ts` | Deployment admission and migration lock | +| `packages/flowsafe/src/do-runner/d1-storage.ts`, deadline snapshot scan | Bounded, persisted-cursor D1 scan pattern for orphan discovery | +| `packages/flowsafe/src/do-runner/inventory.ts` and `mastra-schema-guard.test.ts` | Read-only drain classification and table census | +| `packages/flowsafe/src/agent-runner/agent-gate-round-trip.test.ts` | Real runtime suspension and grant round-trip test pattern | + +Out of scope: WFP source or build admission, artifact signatures, capability grants, user/org quotas, customer billing, and public authentication. Those are platform policy. Approval decisions remain approval decisions; the broker does not mint Breakwater approval grants from a remote result. The durable-result confidentiality proposal is separate: this broker must not claim its stored result is output-policy approved, and hosts should avoid sensitive result content until that gate is implemented. + +## Verification and release gate + +Run each changed test file once per checkpoint, then the owning package checks allowed by the repository quality gate. Send complete output to `/tmp` logs and record exit status. Candidate commands for the implementation worktree: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run \ + src/agent-runner/durable-agent-runner.test.ts \ + src/agent-runner/agent-gate-round-trip.test.ts \ + src/agent-host/thread-topology.test.ts \ + src/agent-host/thread-host.test.ts +pnpm --filter @proofoftech/flowsafe typecheck +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe spike:verify +``` + +For the shared protocol and migration checkpoint, run the owning package's whole test suite once because the exported contract and D1 schema have broad consumers. Verify a real D1/workerd path with a fresh isolate and duplicate callback. After all checkpoints, run the repository's full suite once under the quality-gate rule, including the direct scenario filters that exercise changed contracts. Require these observations: one persisted invocation per suspension key under concurrent observations; orphan suspension discovery with no broker row; no dispatch before durable suspension; permit replay/renewal across an epoch change; identical result replay; conflicting result refusal; stale or foreign result refusal; cutover blocked while a callback remains active; deadline and cancellation stop retries but do not falsely claim executor stop; successful original-principal resume after eviction; recovery of an expired `resuming` lease; no repeated external side effect with a conforming idempotent executor; one terminal run state after lost acknowledgements. + +The implementation is auth, migration, data-retention, and concurrency work. Each code checkpoint is T2: clean-code, QA, architecture, and security lanes. This plan gets one design-level review when added; reviewers do not simulate its commands. A change to this plan after implementation starts records the departure once at completion. Merge a reviewed feature branch into `dev`, not `main`; publish only through Anchorage's version workflow. + +## Addendum: bind remote steps to the started resource + +Baseline for this addendum: `plan/flowsafe-remote-step-broker` at `3c286b84a25564e2ae604da70dbae3a27f1ffe3e`, read 2026-10-11. Reconfirm the appendix anchors before editing. A host's `ActorContext.resourceIdFromKey(threadId)` may return a resource distinct from the thread ID. The current remote-step journal compares its address with the package default `resourceIdFromKey(threadId)`, while start provenance records only the thread ID. This prevents a valid host mapping from dispatching and leaves no independent start-time resource proof for a journal row. + +Implement this before enabling remote-step dispatch or result continuation: + +1. Record the effective `resourceId` at agent start in versioned, durable start provenance. The topology resolves its trusted `ActorContext.resourceIdFromKey(threadId)` once for the winning start and sends that value with the internal start. Threaded starts use the selected thread; unthreaded starts use the newly minted thread. Persist the value in the reservation claim and result, runtime snapshot, and host recovery record. A caller-supplied address does not establish this binding. +2. Inject a host-owned resource resolver into `ThreadAgentHostOptions`, wired from the same deployment mapping as the topology. The target host has no `ActorContext`; it must independently resolve and compare the received resource at start and its binding read, `selectedAgentState`, owner recovery, normal status, termination, and approval or remote resume paths. Update `FlowsafeDurableAgent.authoritativeAgentStartState` to select and compare the persisted start resource rather than deriving it from the thread. Update topology replay and `publicApproval`/`publicAgentRunEnvelope` projections to compare the persisted target resource with the expected run resource. Audit the same start identity checks in `StartIdempotencyStore`, fenced workflow admission, and runtime reconstruction. A changed mapping or missing proof refuses broker authority without silently rewriting the run's binding. +3. Make the authoritative start state expose the recorded resource alongside agent, thread, owner, workflow, run, and start token. At journal registration, permit claim and consume, result recording, and recovery, compare the complete persisted binding against the invocation. The thread host independently checks that same binding before resume. Require the currently trusted topology mapping and stored host binding to agree with the start-time value. Do not substitute the package default mapping for the host's mapping. +4. Upgrade existing snapshots, reservations, and journal state through the repository's versioned D1 and durable-state patterns. A legacy run may proceed only when an independently persisted, run-specific host binding proves its resource ID and agrees with its start token, owner, thread, and run. Otherwise keep it unreadable for remote-step authority and visible to recovery or drain inventory. Do not synthesize a missing resource ID from a thread ID, current mapper, or caller address; do not add a permissive schema default. A mixed old/new deployment must refuse remote-step dispatch until both sides understand the new binding. Existing non-broker run status and ordinary approval resume retain their current compatibility behavior. +5. Test a host mapping such as `thread-1 → account-7` across both threaded and unthreaded starts, fresh-host restart, registration, permit consume, result intake, and resume. Test a changed mapper, a mismatched journal resource, a forged caller resource, a legacy start without independent proof, and a corrupted stored binding. Each refusal must occur before an external effect or resume. Use a real D1/workerd path for the upgrade and fresh-isolate case; in-memory mocks alone do not prove it. + +The chosen authority is the persisted start binding checked again by the trusted host. Reject using `resourceIdFromKey(threadId)` inside the journal: it is the package's identity validator, not the host's resource mapping. Reject trusting the journal's resource column or current mapping alone: either can differ from the resource at start. Reject deriving legacy values from thread IDs: a valid host can map them to different resources. This extends the existing start identity and journal boundary; it adds no dependency, public route, or new authentication scheme. + +Reuse `createAgentThreadTopology`'s `expectedResource` and `sendStart`, the thread host's `snapshotExecutionFor` and owner recovery, `RunnerRuntime.authoritativeStartState`, `StartReservationStore`'s staged schema migration, and `RemoteStepJournal.#execution`. Existing `packages/flowsafe/scripts/spike-verify.mjs` drives `packages/flowsafe/spike/worker.ts` on real workerd and D1, including process kill and restart. Extend that pair with an `FR1` resource-binding scenario; the current script has no such proof. Keep WFP grant policy, customer identities, external executor credentials, and ordinary approval semantics out of this change. Do not change the host's resource mapping for existing users as a migration shortcut. + +Scoped verification for the binding checkpoint, with each command's full output and exit status recorded in a new `/tmp` log: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run src/agent-host/thread-topology.test.ts src/agent-host/thread-host.test.ts src/do-runner/remote-step-journal.test.ts src/do-runner/start-idempotency.test.ts src/do-runner/execution-admission.test.ts +pnpm --filter @proofoftech/flowsafe typecheck +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe spike:verify +``` + +The `spike:verify` command must execute the new `FR1` scenario in `spike-verify.mjs`, backed by the spike worker's thread host and journal. `FR1` starts a remote-step run under a non-identity mapping, kills workerd, verifies the port is closed, restarts on the same D1, then registers, consumes a permit, records a result, and resumes through the target host. It also seeds a pre-upgrade start without independent resource proof and asserts refusal before permit or resume. Check the named `FR1` pass and refusal observations in that command's log; an unchanged spike run does not satisfy this gate. Because the start identity and D1 schema are shared contracts, run the FlowSafe package suite once at that checkpoint under the quality-gate exception. Run T2 clean-code, QA, architecture, and security lanes on its actual diff and in-reach start, host, runtime, journal, and migration contracts. The final repository suite and release gate remain as stated above. + +## Appendix: current-state anchors + +### Resource authority + +These snippets are verbatim at the addendum baseline. They identify risky auth and migration edits; re-read the containing symbols before changing them. + +`packages/flowsafe/src/do-runner/execution-admission.ts`, `StartIdentity` currently records an agent target without its resource: + +```typescript + | { + readonly kind: 'agent'; + readonly id: string; + readonly threadId: string; + }; +``` + +After: add a required, path-safe `resourceId` to the versioned agent target and propagate it through normalization, serialization, replay, and authoritative start reads. Legacy records without separately proved binding refuse broker authority. + +`packages/flowsafe/src/agent-host/thread-topology.ts`, `sendStart` currently resolves the resource again when sending to the target host: + +```typescript + threadId: targetThreadId, + resourceId: context.resourceIdFromKey(targetThreadId), + runId: targetRunId, +``` + +After: capture the trusted mapping with the start decision and pass that exact resource into the host, reservation, and persisted start provenance. On a replay, verify the winning reservation's recorded resource before sending; a new mapping must not silently change the run's resource. + +`packages/flowsafe/src/agent-host/thread-host.ts`, normal start, selection, and recovery currently compare against the package default instead of a host-owned mapping: + +```typescript +if (ref.resourceId !== resourceIdFromKey(scope.threadId)) + throw new AgentHostRequestError(404, 'run not found'); +``` + +The selector uses the same condition; owner recovery compares `stored.resourceId !== resourceIdFromKey(threadId)`. After: inject the deployment's trusted resolver through `ThreadAgentHostOptions`, validate and capture its output, and compare it with the persisted start identity and stored binding at each entry. Keep this check independent of the received internal request. + +`packages/flowsafe/src/agent-runner/durable-agent-runner.ts`, `authoritativeAgentStartState` currently derives the observed resource: + +```typescript +const observedAgentId = identity.target.id; +const observedThreadId = identity.target.threadId; +const observedResourceId = resourceIdFromKey(observedThreadId); +``` + +After: read `observedResourceId` from the validated, versioned start identity and compare the snapshot's context and memory against it. Ordinary legacy reads retain their explicit compatibility path; a legacy record cannot obtain broker authority without independent proof. + +`packages/flowsafe/src/agent-host/thread-topology.ts`, `publicApproval` currently rejects a resource that differs from the default mapping: + +```typescript +target.agentId !== execution.target.id || +target.threadId !== execution.target.threadId || +target.resourceId !== resourceIdFromKey(target.threadId) +``` + +After: compare the approval target and replay envelope with `execution.target.resourceId`, then separately compare that persisted value with the topology's current trusted mapping before admission. + +`packages/flowsafe/src/do-runner/remote-step-journal.ts`, `RemoteStepJournal.#execution` currently checks: + +```typescript + target.id !== address.agentId || + target.threadId !== address.threadId || + (principal !== undefined && + (state.provenance.startIdentity?.owner.kind !== principal.kind || + state.provenance.startIdentity.owner.id !== principal.id)) || + state.execution.tablePrefix !== address.tablePrefix || + address.resourceId !== resourceIdFromKey(address.threadId) +``` + +After: compare `address.resourceId` with the resource recorded in authoritative start provenance. Reject absent or conflicting provenance before an invocation, permit, result, or resume obtains authority. `RemoteStepJournal.register` currently accepts the address and principal as inputs; they remain claims until this check succeeds. + +`packages/flowsafe/src/agent-host/thread-host.ts`, the remote-result route currently checks the start target but not its resource: + +```typescript + state.execution.target.kind !== 'agent' || + state.execution.target.id !== invocation.agentId || + state.execution.target.threadId !== invocation.threadId || + state.execution.owner.kind !== invocation.principal.kind || +``` + +After: compare the persisted start resource, invocation resource, stored host binding, and trusted mapping before `withExecution` and `resumeViaRuntime`. Refuse on any missing or unequal value. Keep the stored principal and exact suspension checks that follow. + +### Earlier proposal anchors + +These are verbatim at the baseline commit. Reconfirm them before applying the plan. + +### Thread status and resume (`packages/flowsafe/src/agent-host/thread-topology.ts`) + +```typescript +status( + context: ActorContext, + input: AgentThreadRunRef, +): Promise; +``` + +`createAgentThreadTopology` routes status through `threads.send()` after `requireResourceAccess` on thread, resource, and run. Its approval resume currently sends: + +```typescript +resumeData: defaultResumeData(record, decision), +...expectedSuspensionFor(record), +entryPath: 'approval.resume', +requestedBy: record.decidedBy, +``` + +### Current suspension correlation (`packages/flowsafe/src/agent-host/thread-host.ts`, `currentApprovals`) + +```typescript +runRecord.suspendedAt === summary.suspendedAt?.[key] && +runRecord.resumeCount === summary.resumeCount?.[key] +``` + +`thread-host.ts`'s resume route calls `authorize()` on the stored principal and then: + +```typescript +return durable.resumeViaRuntime({ + runId: ref.runId, + requestedBy: requesterId, + ...(step !== undefined ? { step } : {}), + ...resumeFields, +``` + +### Runtime check (`packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.resume`) + +```typescript +const proof = await this.#assertResumeFence(workflowId, runId); +return this.#withRunLock(workflowId, runId, async () => { +``` + +Later in the same resume, the exact suspension check is: + +```typescript +if ( + leg.suspendedAt !== expectedSuspension.suspendedAt || + leg.resumeCount !== expectedSuspension.resumeCount +) + throw new SuspensionChangedError(workflowId, runId, stepKey); +``` + +### Guarded entry grammar (`packages/flowsafe/src/agent-runner/agent-run-state.ts`) + +```typescript +export const AGENT_ENTRY_PATHS = [ + 'http.start', + 'approval.resume', + 'signal.message', +``` + +The array continues with other existing entry paths. `packages/flowsafe/src/agent-host/catalog.ts` treats `approval.resume` as a continuation for automated principals. A new broker continuation must preserve that distinction. diff --git a/packages/agent-starter/CHANGELOG.md b/packages/agent-starter/CHANGELOG.md index dfa67860..7c483f58 100644 --- a/packages/agent-starter/CHANGELOG.md +++ b/packages/agent-starter/CHANGELOG.md @@ -1,5 +1,12 @@ # anchorage-agent-starter +## 0.0.27 + +### Patch Changes + +- Updated dependencies [ee4003a] + - @proofoftech/flowsafe@0.29.0 + ## 0.0.26 ### Patch Changes diff --git a/packages/agent-starter/package.json b/packages/agent-starter/package.json index 00625d3d..ae58c9b6 100644 --- a/packages/agent-starter/package.json +++ b/packages/agent-starter/package.json @@ -1,6 +1,6 @@ { "name": "anchorage-agent-starter", - "version": "0.0.26", + "version": "0.0.27", "private": true, "description": "Production-shaped Cloudflare Workers starter for durable, approval-gated Anchorage agents", "type": "module", diff --git a/packages/flowsafe/CHANGELOG.md b/packages/flowsafe/CHANGELOG.md index 33aaf677..097bc5d0 100644 --- a/packages/flowsafe/CHANGELOG.md +++ b/packages/flowsafe/CHANGELOG.md @@ -1,5 +1,11 @@ # @proofoftech/flowsafe +## 0.29.0 + +### Minor Changes + +- ee4003a: Add a D1-backed remote-step journal for agent and workflow suspensions, one-time host dispatch permits, retained results, and a host context derived from the stored invocation. Versioned private state egress carries internal dispatch metadata without changing the legacy route. A host-owned notification operation stores its effect and replay receipt in one D1 transaction with a configurable retained-receipt cap. General Mastra continuation remains fail-closed without generation-fenced snapshot writes; hosts must supply trusted dispatch transport and downstream effect receipts before enabling remote execution. + ## 0.28.0 ### Minor Changes diff --git a/packages/flowsafe/deploy/README.md b/packages/flowsafe/deploy/README.md index 2aa8c65a..e6820090 100644 --- a/packages/flowsafe/deploy/README.md +++ b/packages/flowsafe/deploy/README.md @@ -215,3 +215,57 @@ pnpm --filter @proofoftech/flowsafe spike:verify `spike:verify` adds the workerd boundary: suspend, kill, restart on persisted state, resume from an approved record, and refuse a mismatched deployment sentinel. See the repository [deployment reference](../../../docs/deployment-reference.md), [operations runbook](../../../docs/operations-runbook.md), and [security threat model](../../../docs/security-threat-model.md) before exposing traffic. + +## Run the isolated notification staging control + +Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The initial run requires a `migration-locked` fence at epoch and revision zero, an empty v5 journal, and an armed recovery alarm. It acquires a Linux `flock` keyed by Cloudflare account and Worker before reading the enabled version. Its detached watchdog inherits the same locked open file descriptor, seals or relocks the fence after controller death or its 15-minute deadline, and performs the tracked off deployment on controller request or recovery. Sealing advances the locked fence's revision, so a delayed controller unlock cannot succeed after watchdog shutdown starts. The controller never deploys off itself. + +A failed preflight may leave the fence sealed at `migration-locked`, epoch and revision `1`, with `requireMutationEpoch=true`, even though no effect ran. An operator may retry this exact state once: verify the prior run's tracked off Worker version, confirm the remote-step journal is empty and the notification and receipt tables are absent, deploy a fresh enabled version under the same lock with `--var STAGING_MUTATION_EPOCH:2`, and set `FLOWSAFE_STAGING_RETRY_OFF_VERSION` to that immediately prior 100% off deployment ID. The retry opens at epoch and revision `2`; the watchdog seals a still-locked fence at `2` or relocks an open fence at `3`. A different fence or a second sealed retry requires operator review; do not update the D1 fence row directly. + +Use the same lock for **every** command that can upload, deploy, promote, or change traffic for `anchorage-flowsafe-remote-step-staging`. From `packages/flowsafe`, define the wrapper below and use it for the initial enabled deployment and any other same-Worker deployment command: + +```bash +export FLOWSAFE_CF_ACCOUNT_ID=replace-with-32-lowercase-hex-account-id +flowsafe_locked_deploy() { + [[ "$FLOWSAFE_CF_ACCOUNT_ID" =~ ^[0-9a-f]{32}$ ]] || return 2 + local lock_dir="/tmp/flowsafe-deploy-locks-$(id -u)" + install -d -m 700 "$lock_dir" || return + (umask 077; flock -n -E 75 "$lock_dir/${FLOWSAFE_CF_ACCOUNT_ID}-anchorage-flowsafe-remote-step-staging.lock" "$@") +} +flowsafe_locked_deploy pnpm exec wrangler deploy \ + --config deploy/notification-staging.wrangler.jsonc \ + --var NOTIFICATION_PROOF_ENABLED:true \ + --var STAGING_MUTATION_EPOCH:1 \ + --outdir /tmp/flowsafe-notification-enable +``` + +The wrapper exits `75` if another local deployment owns the lock. Run the control on the same host and Unix user as the wrapper; the lock directory is fixed under `/tmp`, regardless of `TMPDIR`. Set `FLOWSAFE_STAGING_ENABLED_VERSION` from the exact enabled deployment ID before starting the control. The lock serializes cooperating local processes only. A deployment from another host, the Cloudflare dashboard, or an actor that skips this wrapper can still race the version check and deploy. Confirm exclusive deployment control separately before treating the live proof as safe. + +Pin `STAGING_MUTATION_EPOCH` in the enabled version to the epoch its controller will open: `1` for an initial run or `2` for the permitted retry. Keep the off config at `0`. The epoch is deployment configuration; never derive it from a request or adopt the current D1 epoch. + +For a fresh isolated proof, create a new Worker and D1 database. Copy `notification-staging.wrangler.jsonc` to an absolute `0600` path outside the repository. Change its Worker name, D1 name and ID, and deployment tenant to unique values. Resolve `main` and each `alias` target to their absolute paths in this checkout. Keep every other field unchanged, including the off flag, disabled public route, version metadata binding, and Durable Object migration. Do not reuse a sealed D1 database for an initial run. + +Copy `notification-staging-probe.wrangler.jsonc` to a second absolute `0600` path. Set its `services[0].service` to the new Worker name and resolve its `main` path; keep its other fields unchanged. Set `FLOWSAFE_STAGING_CONFIG_FILE` and `FLOWSAFE_STAGING_PROBE_CONFIG_FILE` to the private paths, and set `FLOWSAFE_STAGING_PROBE_PORT` to the local port. Start the probe before enabling the subject Worker: + +```bash +pnpm exec wrangler dev \ + --config "$FLOWSAFE_STAGING_PROBE_CONFIG_FILE" \ + --port "$FLOWSAFE_STAGING_PROBE_PORT" +``` + +Use the new Worker name and subject config path in the deployment wrapper. The controller checks the enabled version's D1 and tenant bindings and the probe subject's authenticated version identity before unlocking. It compiles a private off bundle before arming the watchdog; rollback uses that bundle. + +Set these environment variables before running `node deploy/notification-staging-live.mjs --run` from `packages/flowsafe`: + +| Variable | Value | +| --- | --- | +| `FLOWSAFE_STAGING_PROBE_PORT` | Port of the local Service-binding probe | +| `FLOWSAFE_STAGING_PROBE_SECRET_FILE` | Absolute path to the probe credential in a `0600` file | +| `FLOWSAFE_CF_TOKEN_FILE` | Absolute path to the Cloudflare API token in a `0600` file | +| `FLOWSAFE_CF_ACCOUNT_ID` | Cloudflare account ID as 32 lowercase hexadecimal characters; the same value used by the deployment wrapper | +| `FLOWSAFE_STAGING_ENABLED_VERSION` | Exact 100% enabled subject Worker version ID | +| `FLOWSAFE_STAGING_RETRY_OFF_VERSION` | Retry only: immediately prior 100% off Worker version ID; omit for an initial run | +| `FLOWSAFE_STAGING_CONFIG_FILE` | Fresh isolated proof only: absolute path to the private `0600` off config; omit to use the tracked staging Worker | +| `FLOWSAFE_STAGING_PROBE_CONFIG_FILE` | Fresh isolated proof only: absolute path to the private `0600` local probe config targeting the selected Worker | + +The controller writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the probe becomes unavailable, the watchdog still attempts the version-checked off deployment and reads the fence through the D1 API. An open or unreadable fence leaves completion unconfirmed. Inspect that private directory's `watchdog.log` and verify the fence and current Worker version before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. diff --git a/packages/flowsafe/deploy/notification-staging-control.mjs b/packages/flowsafe/deploy/notification-staging-control.mjs new file mode 100644 index 00000000..1b2118eb --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-control.mjs @@ -0,0 +1,320 @@ +import { readFileSync, statSync } from 'node:fs'; + +const initialFence = { + state: 'migration-locked', + mutationEpoch: 0, + requireMutationEpoch: false, + transitionRevision: 0, +}; +export const sealedRetryFence = { + state: 'migration-locked', + mutationEpoch: 1, + requireMutationEpoch: true, + transitionRevision: 1, +}; + +function assert(condition, message) { + if (!condition) throw new Error(message); +} + +function exactFence(value, expected) { + assert(value && typeof value === 'object', 'fence reading is missing'); + for (const [key, expectedValue] of Object.entries(expected)) + assert(value[key] === expectedValue, `fence ${key} differs`); + assert(value.proofKey === undefined, 'fence has a proof key'); + assert(value.proofRunId === undefined, 'fence has a proof run'); +} + +function nextFence(value, state) { + assert(Number.isSafeInteger(value.mutationEpoch), 'invalid fence epoch'); + assert( + Number.isSafeInteger(value.transitionRevision), + 'invalid fence revision', + ); + assert( + value.mutationEpoch < Number.MAX_SAFE_INTEGER, + 'fence epoch exhausted', + ); + assert( + value.transitionRevision < Number.MAX_SAFE_INTEGER, + 'fence revision exhausted', + ); + return { + state, + mutationEpoch: value.mutationEpoch + 1, + requireMutationEpoch: true, + transitionRevision: value.transitionRevision + 1, + }; +} + +export function readSecretFile(path) { + assert( + typeof path === 'string' && path.startsWith('/'), + 'absolute secret path required', + ); + const info = statSync(path); + assert( + info.isFile() && (info.mode & 0o077) === 0, + 'secret file must be private', + ); + const secret = readFileSync(path, 'utf8').trim(); + assert(/^[\x21-\x7e]{32,256}$/.test(secret), 'invalid probe secret'); + return secret; +} + +export function localProbe({ port, secret, fetcher = fetch }) { + assert( + Number.isInteger(port) && port > 0 && port < 65536, + 'invalid local port', + ); + return async (method, path, body) => { + const response = await fetcher(`http://127.0.0.1:${port}${path}`, { + method, + headers: { + authorization: `Bearer ${secret}`, + ...(body === undefined ? {} : { 'content-type': 'application/json' }), + }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + signal: AbortSignal.timeout(90_000), + }); + assert( + response.status === 200, + `local probe ${path} returned ${response.status}`, + ); + return response.json(); + }; +} + +export function trustedD1Checks(query) { + let baseline; + const count = async (sql) => { + const rows = await query(sql); + assert( + rows.length === 1 && Number.isSafeInteger(rows[0].count), + 'notification baseline is unreadable', + ); + return rows[0].count; + }; + const counts = async ({ + notificationTableExists = true, + receiptTableExists = true, + } = {}) => { + return { + notifications: notificationTableExists + ? await count( + "SELECT COUNT(*) AS count FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification'", + ) + : 0, + receipts: receiptTableExists + ? await count( + 'SELECT COUNT(*) AS count FROM flowsafe_notification_operation_receipts', + ) + : 0, + }; + }; + return { + preflightD1: async ({ requireAbsentEffectTables = false } = {}) => { + const schema = await query( + "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'flowsafe_remote_step_invocations'", + ); + assert( + schema.length === 1 && + typeof schema[0].sql === 'string' && + schema[0].sql.includes('write_generation INTEGER NOT NULL') && + schema[0].sql.includes('operation_state TEXT'), + 'remote step v5 schema is missing', + ); + const rows = await query( + 'SELECT COUNT(*) AS count FROM flowsafe_remote_step_invocations', + ); + assert( + rows.length === 1 && rows[0].count === 0, + 'remote step journal is not empty', + ); + const notificationSchemas = await query( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')", + ); + const present = new Set(notificationSchemas.map((row) => row.name)); + assert( + notificationSchemas.length === present.size && + [...present].every( + (name) => + name === 'mastra_notifications' || + name === 'flowsafe_notification_operation_receipts', + ), + 'notification schema is unreadable', + ); + assert( + !requireAbsentEffectTables || present.size === 0, + 'retry effect tables are present', + ); + baseline = await counts({ + notificationTableExists: present.has('mastra_notifications'), + receiptTableExists: present.has( + 'flowsafe_notification_operation_receipts', + ), + }); + }, + verifyD1: async (invocationId) => { + assert( + /^(?!\.{1,2}$)[A-Za-z0-9._~-]{1,200}$/.test(invocationId), + 'invalid invocation ID', + ); + const rows = await query( + `SELECT i.status, i.settlement_outcome, i.operation_state, i.thread_id, i.notification_id, ` + + `r.operation_key AS receipt_key, r.request_digest AS receipt_digest, ` + + `r.thread_id AS receipt_thread_id, r.notification_id AS receipt_notification_id, ` + + `n.id AS stored_notification_id, ` + + `n.source, n.kind FROM flowsafe_remote_step_invocations i ` + + `LEFT JOIN flowsafe_notification_operation_receipts r ON r.operation_key = i.operation_key ` + + `AND r.thread_id = i.thread_id AND r.notification_id = i.notification_id ` + + `AND r.request_digest = i.notification_request_digest ` + + `LEFT JOIN mastra_notifications n ON n.id = i.notification_id AND n.thread_id = i.thread_id ` + + `WHERE i.invocation_id = '${invocationId}'`, + ); + assert(rows.length === 1, 'expected exactly one staging invocation'); + const row = rows[0]; + assert( + row.status === 'settled' && + row.settlement_outcome === 'applied' && + row.operation_state === 'resume-admitted' && + typeof row.notification_id === 'string' && + row.notification_id === row.stored_notification_id && + typeof row.receipt_key === 'string' && + typeof row.receipt_thread_id === 'string' && + row.receipt_thread_id === row.thread_id && + row.receipt_notification_id === row.notification_id && + /^[a-f0-9]{64}$/.test(row.receipt_digest) && + row.source === 'flowsafe-staging' && + row.kind === 'host-owned-notification', + 'staging notification or receipt is unconfirmed', + ); + const after = await counts(); + assert( + baseline && + after.notifications === baseline.notifications + 1 && + after.receipts === baseline.receipts + 1, + 'staging notification or receipt count differs', + ); + }, + }; +} + +export async function runNotificationStagingControl({ + probe, + preflightD1, + verifyD1, + requestOffDeploy, + onBeforeUnlock = async () => {}, + expectedInitialFence = initialFence, +}) { + assert( + typeof requestOffDeploy === 'function', + 'off deploy watchdog is required', + ); + assert( + expectedInitialFence === initialFence || + Object.entries(sealedRetryFence).every( + ([key, value]) => expectedInitialFence?.[key] === value, + ), + 'unsupported staging fence baseline', + ); + let initial; + let unlockAttempted = false; + let outcome; + let failure; + try { + initial = await probe('GET', '/admin/execution-fence'); + exactFence(initial, expectedInitialFence); + const maintenance = await probe('GET', '/admin/maintenance-status'); + assert( + Number.isSafeInteger(maintenance.alarmAt) && + Number.isSafeInteger(maintenance.nextRemoteStepRecoveryAt) && + maintenance.alarmAt <= + Math.max(Date.now(), maintenance.nextRemoteStepRecoveryAt) && + maintenance.lastRemoteStepRecoveryError === undefined, + 'notification recovery duty is not armed', + ); + await preflightD1({ + requireAbsentEffectTables: expectedInitialFence.mutationEpoch === 1, + }); + await onBeforeUnlock(); + unlockAttempted = true; + const opened = await probe('POST', '/admin/execution-fence', { + expected: 'migration-locked', + next: 'open', + expectedMutationEpoch: initial.mutationEpoch, + expectedRevision: initial.transitionRevision, + advanceMutationEpoch: true, + }); + exactFence(opened, nextFence(initial, 'open')); + const started = await probe('POST', '/start'); + assert( + started?.status === 'suspended' && + typeof started.invocationId === 'string' && + started.invocationId.length > 0, + 'staging start did not suspend', + ); + const completed = await probe('POST', '/complete', { + invocationId: started.invocationId, + }); + assert( + completed?.invocationId === started.invocationId && + completed.status === 'success', + 'staging completion did not settle', + ); + await verifyD1(started.invocationId); + outcome = { + invocationId: started.invocationId, + verdict: 'notification_staging_passed', + }; + } catch (error) { + failure = error; + } finally { + if (unlockAttempted) { + try { + const current = await probe('GET', '/admin/execution-fence'); + if (current.state === 'open') { + assert( + current.requireMutationEpoch === true, + 'open fence has wrong epoch policy', + ); + assert( + current.mutationEpoch === initial.mutationEpoch + 1, + 'open fence epoch differs', + ); + assert( + current.transitionRevision === initial.transitionRevision + 1, + 'open fence revision differs', + ); + const locked = await probe('POST', '/admin/execution-fence', { + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: current.mutationEpoch, + expectedRevision: current.transitionRevision, + advanceMutationEpoch: true, + }); + exactFence(locked, nextFence(current, 'migration-locked')); + } else { + exactFence(current, expectedInitialFence); + assert(!outcome, 'successful proof lost its open fence'); + } + } catch (error) { + failure = new AggregateError( + failure ? [failure, error] : [error], + 'staging fence relock is unconfirmed', + ); + } + } + try { + await requestOffDeploy(); + } catch (error) { + failure = new AggregateError( + failure ? [failure, error] : [error], + 'staging off deploy is unconfirmed', + ); + } + } + if (failure) throw failure; + return outcome; +} diff --git a/packages/flowsafe/deploy/notification-staging-control.node-test.mjs b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs new file mode 100644 index 00000000..78bb8677 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs @@ -0,0 +1,296 @@ +import assert from 'node:assert/strict'; +import { chmodSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { DatabaseSync } from 'node:sqlite'; +import { test } from 'node:test'; +import { + localProbe, + readSecretFile, + runNotificationStagingControl, + trustedD1Checks, +} from './notification-staging-control.mjs'; + +function fixture(failAt, initialState = 'migration-locked', initialEpoch = 0) { + const calls = []; + let fence = { + state: initialState, + mutationEpoch: initialEpoch, + requireMutationEpoch: initialEpoch !== 0, + transitionRevision: initialEpoch, + }; + const probe = async (method, path, body) => { + calls.push({ method, path, body }); + if (method === 'GET' && path === '/admin/execution-fence') + return { ...fence }; + if (method === 'GET' && path === '/admin/maintenance-status') { + return { alarmAt: 1, nextRemoteStepRecoveryAt: 1 }; + } + if (path === '/admin/execution-fence') { + assert.equal(body.expected, fence.state); + assert.equal(body.expectedMutationEpoch, fence.mutationEpoch); + assert.equal(body.expectedRevision, fence.transitionRevision); + fence = { + state: body.next, + mutationEpoch: fence.mutationEpoch + 1, + requireMutationEpoch: true, + transitionRevision: fence.transitionRevision + 1, + }; + if (failAt === 'unlock-response' && body.next === 'open') + throw new Error('response lost after accepted unlock'); + return { ...fence }; + } + if (path === '/start') { + if (failAt === 'start') throw new Error('start failed'); + return { status: 'suspended', invocationId: 'invocation-1' }; + } + if (path === '/complete') { + if (failAt === 'complete') throw new Error('complete failed'); + return { status: 'success', invocationId: body.invocationId }; + } + throw new Error('unexpected request'); + }; + let offRequests = 0; + const options = { + probe, + preflightD1: async () => { + if (failAt === 'preflight') throw new Error('preflight failed'); + }, + verifyD1: async () => { + if (failAt === 'verify') throw new Error('verification failed'); + }, + requestOffDeploy: async () => { + offRequests++; + }, + }; + return { + options, + calls, + fence: () => fence, + offRequests: () => offRequests, + }; +} + +test('preflight mismatch and read-only failure never unlock', async () => { + for (const scenario of [fixture(null, 'open'), fixture('preflight')]) { + await assert.rejects(runNotificationStagingControl(scenario.options)); + assert.equal( + scenario.calls.filter((call) => call.method === 'POST').length, + 0, + ); + assert.equal(scenario.offRequests(), 1); + } +}); + +test('each post-unlock failure requests a fresh CAS relock and off deploy', async () => { + for (const step of ['unlock-response', 'start', 'complete', 'verify']) { + const scenario = fixture(step); + await assert.rejects(runNotificationStagingControl(scenario.options)); + assert.deepEqual(scenario.fence(), { + state: 'migration-locked', + mutationEpoch: 2, + requireMutationEpoch: true, + transitionRevision: 2, + }); + assert.equal(scenario.offRequests(), 1); + const writes = scenario.calls.filter( + (call) => + call.method === 'POST' && call.path === '/admin/execution-fence', + ); + assert.deepEqual( + writes.map((call) => call.body.next), + ['open', 'migration-locked'], + ); + assert.deepEqual( + [writes[0].body.expectedRevision, writes[1].body.expectedRevision], + [0, 1], + ); + } +}); + +test('a successful proof verifies before relock and requests off deploy', async () => { + const scenario = fixture(); + assert.deepEqual(await runNotificationStagingControl(scenario.options), { + invocationId: 'invocation-1', + verdict: 'notification_staging_passed', + }); + assert.equal(scenario.fence().state, 'migration-locked'); + assert.equal(scenario.offRequests(), 1); +}); + +test('explicit sealed retry advances the fence monotonically', async () => { + const scenario = fixture(null, 'migration-locked', 1); + const expectedInitialFence = { + state: 'migration-locked', + mutationEpoch: 1, + requireMutationEpoch: true, + transitionRevision: 1, + }; + assert.deepEqual( + await runNotificationStagingControl({ + ...scenario.options, + expectedInitialFence, + }), + { invocationId: 'invocation-1', verdict: 'notification_staging_passed' }, + ); + assert.deepEqual(scenario.fence(), { + state: 'migration-locked', + mutationEpoch: 3, + requireMutationEpoch: true, + transitionRevision: 3, + }); + assert.deepEqual( + scenario.calls + .filter( + (call) => + call.method === 'POST' && call.path === '/admin/execution-fence', + ) + .map((call) => call.body.expectedRevision), + [1, 2], + ); +}); + +test('a conflicting fence is not overwritten while off deploy is still requested', async () => { + const scenario = fixture(); + scenario.options.verifyD1 = async () => { + throw new Error('verification failed'); + }; + const original = scenario.options.probe; + scenario.options.probe = async (method, path, body) => { + if ( + method === 'GET' && + path === '/admin/execution-fence' && + scenario.calls.some((call) => call.path === '/start') + ) { + return { + state: 'draining', + mutationEpoch: 1, + requireMutationEpoch: true, + transitionRevision: 2, + }; + } + return original(method, path, body); + }; + await assert.rejects( + runNotificationStagingControl(scenario.options), + /relock is unconfirmed/, + ); + assert.equal(scenario.fence().state, 'open'); + assert.equal(scenario.offRequests(), 1); +}); + +test('probe reads a private secret file and calls only loopback', async () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-probe-secret-test-')); + try { + const path = join(directory, 'secret'); + const secret = 'a'.repeat(32); + writeFileSync(path, secret, { mode: 0o600 }); + const requests = []; + const probe = localProbe({ + port: 8897, + secret: readSecretFile(path), + fetcher: async (url, init) => { + requests.push({ url, init }); + return Response.json({ state: 'migration-locked' }); + }, + }); + assert.deepEqual(await probe('GET', '/admin/execution-fence'), { + state: 'migration-locked', + }); + assert.equal( + requests[0].url, + 'http://127.0.0.1:8897/admin/execution-fence', + ); + assert.equal(requests[0].init.headers.authorization, `Bearer ${secret}`); + chmodSync(path, 0o644); + assert.throws(() => readSecretFile(path), /private/); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + +test('D1 proof requires the receipt tuple and one new notification and receipt', async () => { + const db = new DatabaseSync(':memory:'); + try { + db.exec(`CREATE TABLE flowsafe_remote_step_invocations ( + invocation_id TEXT PRIMARY KEY, status TEXT, settlement_outcome TEXT, + operation_state TEXT, thread_id TEXT, notification_id TEXT, + operation_key TEXT, notification_request_digest TEXT, + write_generation INTEGER NOT NULL + ); + CREATE TABLE flowsafe_notification_operation_receipts ( + operation_key TEXT PRIMARY KEY, request_digest TEXT, + thread_id TEXT, notification_id TEXT + ); + CREATE TABLE mastra_notifications ( + id TEXT, thread_id TEXT, source TEXT, kind TEXT + );`); + const checks = trustedD1Checks((sql) => db.prepare(sql).all()); + await assert.rejects( + checks.preflightD1({ requireAbsentEffectTables: true }), + /retry effect tables are present/, + ); + await checks.preflightD1(); + db.exec(`INSERT INTO flowsafe_remote_step_invocations VALUES + ('invocation-1', 'settled', 'applied', 'resume-admitted', + 'thread-1', 'notification-1', 'operation-1', '${'a'.repeat(64)}', 1); + INSERT INTO mastra_notifications VALUES + ('notification-1', 'thread-1', 'flowsafe-staging', 'host-owned-notification'); + INSERT INTO flowsafe_notification_operation_receipts VALUES + ('operation-1', '${'a'.repeat(64)}', 'other-thread', 'notification-1');`); + await assert.rejects(checks.verifyD1('invocation-1'), /unconfirmed/); + db.exec( + "UPDATE flowsafe_notification_operation_receipts SET thread_id = 'thread-1', notification_id = 'other-notification'", + ); + await assert.rejects(checks.verifyD1('invocation-1'), /unconfirmed/); + db.exec( + "UPDATE flowsafe_notification_operation_receipts SET notification_id = 'notification-1'", + ); + await checks.verifyD1('invocation-1'); + db.exec( + `UPDATE flowsafe_notification_operation_receipts SET request_digest = '${'b'.repeat(64)}'`, + ); + await assert.rejects(checks.verifyD1('invocation-1'), /unconfirmed/); + db.exec( + `UPDATE flowsafe_notification_operation_receipts SET request_digest = '${'a'.repeat(64)}'`, + ); + db.exec(`INSERT INTO mastra_notifications VALUES + ('notification-2', 'thread-2', 'flowsafe-staging', 'host-owned-notification');`); + await assert.rejects(checks.verifyD1('invocation-1'), /count differs/); + } finally { + db.close(); + } +}); + +test('D1 preflight accepts absent notification tables but proof requires the effect', async () => { + const db = new DatabaseSync(':memory:'); + try { + db.exec(`CREATE TABLE flowsafe_remote_step_invocations ( + invocation_id TEXT PRIMARY KEY, status TEXT, settlement_outcome TEXT, + operation_state TEXT, thread_id TEXT, notification_id TEXT, + operation_key TEXT, notification_request_digest TEXT, + write_generation INTEGER NOT NULL + );`); + const checks = trustedD1Checks((sql) => db.prepare(sql).all()); + await checks.preflightD1({ requireAbsentEffectTables: true }); + await checks.preflightD1(); + db.exec(`INSERT INTO flowsafe_remote_step_invocations VALUES + ('invocation-1', 'settled', 'applied', 'resume-admitted', + 'thread-1', 'notification-1', 'operation-1', '${'a'.repeat(64)}', 1);`); + await assert.rejects(checks.verifyD1('invocation-1'), /no such table/); + db.exec(`CREATE TABLE mastra_notifications ( + id TEXT, thread_id TEXT, source TEXT, kind TEXT + ); + INSERT INTO mastra_notifications VALUES + ('notification-1', 'thread-1', 'flowsafe-staging', 'host-owned-notification'); + CREATE TABLE flowsafe_notification_operation_receipts ( + operation_key TEXT PRIMARY KEY, request_digest TEXT, + thread_id TEXT, notification_id TEXT + ); + INSERT INTO flowsafe_notification_operation_receipts VALUES + ('operation-1', '${'a'.repeat(64)}', 'thread-1', 'notification-1');`); + await checks.verifyD1('invocation-1'); + } finally { + db.close(); + } +}); diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs new file mode 100644 index 00000000..60897ba9 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -0,0 +1,750 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { + appendFileSync, + closeSync, + constants, + existsSync, + fstatSync, + mkdirSync, + openSync, + readFileSync, + statSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve as resolvePath } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { isDeepStrictEqual } from 'node:util'; +import { + decodeExecutionFenceMutationMetadata, + EXECUTION_FENCE_ROW_ID, + EXECUTION_FENCE_TABLE, +} from '#deployment-identity-protocol'; +import { + localProbe, + readSecretFile, + runNotificationStagingControl, + sealedRetryFence, + trustedD1Checks, +} from './notification-staging-control.mjs'; + +const here = dirname(fileURLToPath(import.meta.url)); +const flowsafe = dirname(here); +const wrangler = join(flowsafe, '../../node_modules/.bin/wrangler'); +const trackedConfigPath = join(here, 'notification-staging.wrangler.jsonc'); +const trackedConfig = JSON.parse(readFileSync(trackedConfigPath, 'utf8')); +const trackedProbeConfigPath = join( + here, + 'notification-staging-probe.wrangler.jsonc', +); +const trackedProbeConfig = JSON.parse( + readFileSync(trackedProbeConfigPath, 'utf8'), +); +const versionPattern = + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; +const accountPattern = /^[0-9a-f]{32}$/; +const workerNamePattern = /^[a-z0-9-]+$/; + +function assert(condition, message) { + if (!condition) throw new Error(message); +} + +function privateDiagnostic(error, secrets = []) { + const parts = []; + const visit = (value) => { + if (!(value instanceof Error)) return; + parts.push(value.message); + if (value instanceof AggregateError) value.errors.forEach(visit); + if (value.cause) visit(value.cause); + }; + visit(error); + return parts + .join(' | ') + .replaceAll(secrets[0] ?? '\u0000', '[redacted]') + .replaceAll(secrets[1] ?? '\u0000', '[redacted]') + .slice(0, 4_096); +} + +function privateFile(path) { + assert( + typeof path === 'string' && path.startsWith('/'), + 'absolute private path required', + ); + const info = statSync(path); + assert( + info.isFile() && (info.mode & 0o077) === 0, + 'private file permissions required', + ); + return readFileSync(path, 'utf8').trim(); +} + +function stagingConfig(configPath) { + const source = + configPath === trackedConfigPath + ? readFileSync(configPath, 'utf8') + : privateFile(configPath); + const selected = JSON.parse(source); + const expected = structuredClone(trackedConfig); + if (configPath !== trackedConfigPath) { + assert( + workerNamePattern.test(selected.name) && + selected.name !== trackedConfig.name && + /^[a-z0-9-]+$/.test(selected.d1_databases?.[0]?.database_name) && + selected.d1_databases[0].database_name !== + trackedConfig.d1_databases[0].database_name && + versionPattern.test(selected.d1_databases[0].database_id) && + selected.d1_databases[0].database_id !== + trackedConfig.d1_databases[0].database_id && + /^[a-z0-9]{1,32}$/.test(selected.vars?.DEPLOYMENT_TENANT) && + selected.vars.DEPLOYMENT_TENANT !== + trackedConfig.vars.DEPLOYMENT_TENANT, + 'private staging config resource identity differs', + ); + expected.name = selected.name; + expected.main = resolvePath(here, trackedConfig.main); + expected.alias = Object.fromEntries( + Object.entries(trackedConfig.alias).map(([name, path]) => [ + name, + resolvePath(here, path), + ]), + ); + expected.d1_databases[0].database_name = + selected.d1_databases[0].database_name; + expected.d1_databases[0].database_id = selected.d1_databases[0].database_id; + expected.vars.DEPLOYMENT_TENANT = selected.vars.DEPLOYMENT_TENANT; + } + assert( + isDeepStrictEqual(selected, expected), + 'private staging config differs from the off template', + ); + return { + source, + name: selected.name, + database: selected.d1_databases[0].database_name, + databaseId: selected.d1_databases[0].database_id, + tenant: selected.vars.DEPLOYMENT_TENANT, + }; +} + +function verifyProbeConfig(configPath, workerName, fresh) { + assert( + fresh === (configPath !== trackedProbeConfigPath), + 'private probe config is required for the selected Worker', + ); + const selected = JSON.parse( + configPath === trackedProbeConfigPath + ? readFileSync(configPath, 'utf8') + : privateFile(configPath), + ); + const expected = structuredClone(trackedProbeConfig); + if (fresh) { + expected.main = resolvePath(here, trackedProbeConfig.main); + expected.services[0].service = workerName; + } + assert( + isDeepStrictEqual(selected, expected), + 'private probe config differs from the selected Worker', + ); +} + +export function createWranglerControl({ + command = spawnSync, + token, + configPath = trackedConfigPath, +}) { + const selected = stagingConfig(configPath); + function run(args, timeout = 30_000) { + assert( + stagingConfig(configPath).source === selected.source, + 'staging config changed since preflight', + ); + const result = command(wrangler, args, { + cwd: flowsafe, + encoding: 'utf8', + timeout, + env: { + ...process.env, + CLOUDFLARE_API_TOKEN: token, + NO_COLOR: '1', + WRANGLER_SEND_METRICS: 'false', + }, + }); + if (result.status !== 0) { + const details = [ + `exit=${result.status ?? 'none'}`, + `signal=${result.signal ?? 'none'}`, + result.error?.message, + result.stderr?.slice(-2_048), + ] + .filter(Boolean) + .join(' | '); + throw new Error(`Wrangler ${args[0]} ${args[1]} failed`, { + cause: new Error(details), + }); + } + return result.stdout; + } + function json(args) { + const value = JSON.parse(run(args)); + assert(value !== null, 'Wrangler returned null'); + return value; + } + function deployments() { + const list = json([ + 'deployments', + 'list', + '--config', + configPath, + '--json', + ]); + assert(Array.isArray(list) && list.length > 0, 'deployment list is empty'); + const ordered = list + .map((deployment) => { + const createdAt = Date.parse(deployment.created_on); + assert( + Number.isFinite(createdAt), + 'deployment timestamp is unreadable', + ); + return { deployment, createdAt }; + }) + .sort((a, b) => b.createdAt - a.createdAt); + assert( + ordered.length === 1 || ordered[0].createdAt > ordered[1].createdAt, + 'latest deployment is ambiguous', + ); + return ordered.map(({ deployment }) => deployment); + } + function fullVersion(deployment) { + assert( + deployment.versions?.length === 1 && + deployment.versions[0].percentage === 100, + 'deployment is not a single full version', + ); + const id = deployment.versions[0].version_id; + assert(versionPattern.test(id), 'invalid deployment version'); + return id; + } + function currentVersion() { + return fullVersion(deployments()[0]); + } + function proofEnabled(version, expectedMutationEpoch) { + assert(versionPattern.test(version), 'invalid version'); + const detail = json([ + 'versions', + 'view', + version, + '--config', + configPath, + '--json', + ]); + const bindings = detail.resources?.bindings; + assert(Array.isArray(bindings), 'version bindings are unreadable'); + const flag = bindings.filter( + (binding) => binding.name === 'NOTIFICATION_PROOF_ENABLED', + ); + const d1 = bindings.filter((binding) => binding.name === 'DB'); + const tenant = bindings.filter( + (binding) => binding.name === 'DEPLOYMENT_TENANT', + ); + assert( + flag.length === 1 && flag[0].type === 'plain_text', + 'proof flag binding differs', + ); + assert( + d1.length === 1 && + d1[0].type === 'd1' && + d1[0].database_id === selected.databaseId, + 'enabled D1 binding differs', + ); + assert( + tenant.length === 1 && + tenant[0].type === 'plain_text' && + tenant[0].text === selected.tenant, + 'enabled deployment tenant differs', + ); + if (expectedMutationEpoch !== undefined) { + assert( + expectedMutationEpoch === 1 || expectedMutationEpoch === 2, + 'invalid enabled epoch', + ); + const epoch = bindings.filter( + (binding) => binding.name === 'STAGING_MUTATION_EPOCH', + ); + assert( + epoch.length === 1 && + epoch[0].type === 'plain_text' && + epoch[0].text === String(expectedMutationEpoch), + 'enabled epoch binding differs', + ); + } + return flag[0].text === 'true'; + } + return { + currentVersion, + proofEnabled, + priorOffVersion(expectedEnabled) { + const ordered = deployments(); + assert( + ordered.length > 1 && fullVersion(ordered[0]) === expectedEnabled, + 'retry deployment history differs', + ); + const prior = fullVersion(ordered[1]); + assert(!proofEnabled(prior), 'prior deployment is not off'); + return prior; + }, + query(sql) { + const output = json([ + 'd1', + 'execute', + selected.database, + '--remote', + '--yes', + '--json', + '--config', + configPath, + '--command', + sql, + ]); + assert( + Array.isArray(output) && + output.length === 1 && + output[0].success === true && + Array.isArray(output[0].results), + 'D1 read failed', + ); + return output[0].results; + }, + prepareOffBundle(directory) { + assert(directory.startsWith('/'), 'absolute off bundle path required'); + mkdirSync(directory, { mode: 0o700 }); + run( + ['deploy', '--dry-run', '--config', configPath, '--outdir', directory], + 180_000, + ); + const bundlePath = join(directory, 'notification-staging.worker.js'); + assert(statSync(bundlePath).isFile(), 'off bundle is missing'); + run( + [ + 'deploy', + bundlePath, + '--dry-run', + '--config', + configPath, + '--outdir', + join(directory, 'bundle-rehearsal'), + ], + 180_000, + ); + return bundlePath; + }, + deployOffIfCurrent(expectedVersion, outdir, bundlePath) { + assert(versionPattern.test(expectedVersion), 'invalid expected version'); + assert( + typeof bundlePath === 'string' && + bundlePath.startsWith('/') && + statSync(bundlePath).isFile(), + 'pinned off bundle is required', + ); + assert( + currentVersion() === expectedVersion, + 'enabled deployment changed', + ); + assert( + proofEnabled(expectedVersion), + 'expected deployment is not enabled', + ); + run( + ['deploy', bundlePath, '--config', configPath, '--outdir', outdir], + 180_000, + ); + const next = currentVersion(); + assert( + next !== expectedVersion && proofEnabled(next) === false, + 'off deploy is unconfirmed', + ); + return next; + }, + }; +} + +export async function sealFence(probe, baselineEpoch = 0) { + assert( + baselineEpoch === 0 || baselineEpoch === 1, + 'invalid watchdog baseline', + ); + for (let attempt = 0; attempt < 3; attempt++) { + const current = await probe('GET', '/admin/execution-fence'); + assert( + current.state === 'migration-locked' || current.state === 'open', + 'watchdog found an unexpected fence state', + ); + const initial = + current.state === 'migration-locked' && + current.mutationEpoch === baselineEpoch && + current.transitionRevision === baselineEpoch && + current.requireMutationEpoch === (baselineEpoch !== 0); + const open = + current.state === 'open' && + current.mutationEpoch === baselineEpoch + 1 && + current.transitionRevision === baselineEpoch + 1 && + current.requireMutationEpoch === true; + if (current.state === 'migration-locked' && !initial) { + confirmedFence(current, baselineEpoch); + return current; + } + assert(initial || open, 'watchdog fence metadata differs'); + try { + await probe('POST', '/admin/execution-fence', { + expected: current.state, + next: 'migration-locked', + expectedMutationEpoch: current.mutationEpoch, + expectedRevision: current.transitionRevision, + advanceMutationEpoch: true, + }); + } catch { + // The authoritative read resolves a lost response or a competing open CAS. + } + } + const current = await probe('GET', '/admin/execution-fence'); + confirmedFence(current, baselineEpoch); + return current; +} + +function settingsFromEnv(env) { + const port = Number(env.FLOWSAFE_STAGING_PROBE_PORT); + const expectedVersion = env.FLOWSAFE_STAGING_ENABLED_VERSION; + assert( + Number.isInteger(port) && port > 0 && port < 65536, + 'invalid local probe port', + ); + assert( + versionPattern.test(expectedVersion), + 'expected enabled version required', + ); + const probeSecretPath = env.FLOWSAFE_STAGING_PROBE_SECRET_FILE; + const cfTokenPath = env.FLOWSAFE_CF_TOKEN_FILE; + const secret = readSecretFile(probeSecretPath); + const token = privateFile(cfTokenPath); + assert(token.length > 0, 'Cloudflare token is empty'); + const accountId = env.FLOWSAFE_CF_ACCOUNT_ID; + assert(accountPattern.test(accountId), 'Cloudflare account ID is required'); + const priorOffVersion = env.FLOWSAFE_STAGING_RETRY_OFF_VERSION; + assert( + priorOffVersion === undefined || versionPattern.test(priorOffVersion), + 'invalid prior off version', + ); + const configPath = env.FLOWSAFE_STAGING_CONFIG_FILE ?? trackedConfigPath; + const selected = stagingConfig(configPath); + verifyProbeConfig( + env.FLOWSAFE_STAGING_PROBE_CONFIG_FILE ?? trackedProbeConfigPath, + selected.name, + configPath !== trackedConfigPath, + ); + return { + port, + expectedVersion, + probeSecretPath, + cfTokenPath, + secret, + token, + accountId, + priorOffVersion, + configPath, + selected, + }; +} + +function acquireDeploymentLock(accountId, workerName) { + const directory = join('/tmp', `flowsafe-deploy-locks-${process.getuid()}`); + mkdirSync(directory, { mode: 0o700, recursive: true }); + const directoryInfo = statSync(directory); + assert( + directoryInfo.isDirectory() && + directoryInfo.uid === process.getuid() && + (directoryInfo.mode & 0o077) === 0, + 'deployment lock directory is not private', + ); + const path = join(directory, `${accountId}-${workerName}.lock`); + const fd = openSync( + path, + constants.O_CREAT | constants.O_RDWR | constants.O_NOFOLLOW, + 0o600, + ); + const info = fstatSync(fd); + assert( + info.isFile() && info.uid === process.getuid() && (info.mode & 0o077) === 0, + 'deployment lock file is not private', + ); + const result = spawnSync('flock', ['--nonblock', '--exclusive', '3'], { + stdio: ['ignore', 'pipe', 'pipe', fd], + }); + assert(result.status === 0, 'deployment lock is unavailable'); + return fd; +} + +function confirmedFence(current, baselineEpoch = 0) { + assert( + current.state === 'migration-locked', + 'staging fence relock is unconfirmed', + ); + const sealed = + current.mutationEpoch === baselineEpoch + 1 && + current.transitionRevision === baselineEpoch + 1 && + current.requireMutationEpoch === true; + const relocked = + current.mutationEpoch === baselineEpoch + 2 && + current.transitionRevision === baselineEpoch + 2 && + current.requireMutationEpoch === true; + assert(sealed || relocked, 'staging fence metadata differs'); +} + +async function watchdog(stateFile) { + const lockFd = 3; + assert(fstatSync(lockFd).isFile(), 'inherited deployment lock is missing'); + const state = JSON.parse(privateFile(stateFile)); + const { + port, + expectedVersion, + probeSecretPath, + cfTokenPath, + parentPid, + deadlineAt, + outdir, + baselineEpoch, + configPath, + workerName, + bundlePath, + } = state; + assert( + baselineEpoch === 0 || baselineEpoch === 1, + 'invalid watchdog baseline', + ); + const secret = readSecretFile(probeSecretPath); + const token = privateFile(cfTokenPath); + const probe = localProbe({ port, secret }); + const selected = stagingConfig(configPath); + assert(selected.name === workerName, 'watchdog Worker identity differs'); + assert( + bundlePath === + join( + dirname(stateFile), + 'pinned-off-bundle', + 'notification-staging.worker.js', + ) && statSync(bundlePath).isFile(), + 'watchdog pinned off bundle is missing', + ); + const cli = createWranglerControl({ + token, + configPath, + }); + assert( + cli.currentVersion() === expectedVersion && + cli.proofEnabled(expectedVersion, baselineEpoch + 1), + 'watchdog enabled version differs', + ); + writeFileSync(`${stateFile}.ready`, '', { mode: 0o600, flag: 'wx' }); + while (Date.now() < deadlineAt && !existsSync(`${stateFile}.off-requested`)) { + try { + process.kill(parentPid, 0); + } catch { + break; + } + await new Promise((resolve) => setTimeout(resolve, 1_000)); + } + let sealFailure; + try { + await sealFence(probe, baselineEpoch); + } catch (error) { + sealFailure = error; + } + cli.deployOffIfCurrent(expectedVersion, outdir, bundlePath); + try { + const rows = cli.query(`SELECT * FROM ${EXECUTION_FENCE_TABLE}`); + assert( + rows.length === 1 && rows[0].id === EXECUTION_FENCE_ROW_ID, + 'authoritative fence row differs', + ); + const row = rows[0]; + const metadata = decodeExecutionFenceMutationMetadata(row); + assert( + row.proof_key === null && + row.proof_run_id === null && + metadata.proofTablePrefix === null && + metadata.proofWorkflowId === null && + metadata.proofStartToken === null, + 'authoritative fence proof differs', + ); + confirmedFence({ state: row.state, ...metadata }, baselineEpoch); + } catch (error) { + const failure = new AggregateError( + sealFailure ? [sealFailure, error] : [error], + 'staging fence relock is unconfirmed', + ); + throw new Error(privateDiagnostic(failure, [secret, token])); + } + writeFileSync(`${stateFile}.off-confirmed`, '', { mode: 0o600, flag: 'wx' }); +} + +async function controller() { + const settings = settingsFromEnv(process.env); + const lockFd = acquireDeploymentLock( + settings.accountId, + settings.selected.name, + ); + const directory = join( + tmpdir(), + `flowsafe-notification-control-${process.pid}`, + ); + mkdirSync(directory, { mode: 0o700 }); + const configPath = + settings.configPath === trackedConfigPath + ? trackedConfigPath + : join(directory, 'off.wrangler.jsonc'); + if (configPath !== trackedConfigPath) + writeFileSync(configPath, settings.selected.source, { + mode: 0o600, + flag: 'wx', + }); + const stateFile = join(directory, 'watchdog.json'); + const outdir = join(directory, 'off-deploy'); + let stage = 'enabled-version-preflight'; + try { + const cli = createWranglerControl({ token: settings.token, configPath }); + const baselineEpoch = settings.priorOffVersion === undefined ? 0 : 1; + assert( + cli.currentVersion() === settings.expectedVersion && + cli.proofEnabled(settings.expectedVersion, baselineEpoch + 1), + 'enabled version preflight differs', + ); + const probe = localProbe({ port: settings.port, secret: settings.secret }); + if (baselineEpoch === 1) { + assert( + settings.priorOffVersion !== settings.expectedVersion && + cli.priorOffVersion(settings.expectedVersion) === + settings.priorOffVersion, + 'retry prior off version is unconfirmed', + ); + const fence = await probe('GET', '/admin/execution-fence'); + assert( + Object.entries(sealedRetryFence).every( + ([key, value]) => fence[key] === value, + ) && + fence.proofKey === undefined && + fence.proofRunId === undefined, + 'retry fence is not the exact sealed baseline', + ); + } + stage = 'off-bundle-preparation'; + const bundlePath = cli.prepareOffBundle( + join(directory, 'pinned-off-bundle'), + ); + stage = 'watchdog-start'; + const watcherLog = openSync(join(directory, 'watchdog.log'), 'wx', 0o600); + writeFileSync( + stateFile, + JSON.stringify({ + port: settings.port, + expectedVersion: settings.expectedVersion, + probeSecretPath: settings.probeSecretPath, + cfTokenPath: settings.cfTokenPath, + parentPid: process.pid, + deadlineAt: Date.now() + 900_000, + outdir, + baselineEpoch, + configPath, + workerName: settings.selected.name, + bundlePath, + }), + { mode: 0o600, flag: 'wx' }, + ); + const watcher = spawn( + process.execPath, + [fileURLToPath(import.meta.url), '--watchdog', stateFile], + { + detached: true, + stdio: ['ignore', watcherLog, watcherLog, lockFd], + env: process.env, + }, + ); + closeSync(watcherLog); + watcher.unref(); + assert(Number.isInteger(watcher.pid), 'watchdog did not start'); + const readyDeadline = Date.now() + 20_000; + while (!existsSync(`${stateFile}.ready`) && Date.now() < readyDeadline) { + await new Promise((resolve) => setTimeout(resolve, 100)); + } + assert( + existsSync(`${stateFile}.ready`) && + watcher.exitCode === null && + watcher.signalCode === null, + 'watchdog readiness is unconfirmed', + ); + process.kill(watcher.pid, 0); + let offConfirmed = false; + const requestOffDeploy = async () => { + if (offConfirmed) return; + writeFileSync(`${stateFile}.off-requested`, '', { + mode: 0o600, + flag: 'wx', + }); + while (!existsSync(`${stateFile}.off-confirmed`)) { + assert( + watcher.exitCode === null && watcher.signalCode === null, + 'watchdog exited before off confirmation', + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + } + offConfirmed = true; + }; + stage = 'proof'; + const result = await runNotificationStagingControl({ + probe, + ...trustedD1Checks((sql) => cli.query(sql)), + expectedInitialFence: baselineEpoch === 1 ? sealedRetryFence : undefined, + requestOffDeploy, + onBeforeUnlock: async () => { + assert( + watcher.exitCode === null && watcher.signalCode === null, + 'watchdog exited before fence unlock', + ); + process.kill(watcher.pid, 0); + const identity = await probe('GET', '/identity'); + assert( + identity.versionId === settings.expectedVersion && + identity.deploymentTenant === settings.selected.tenant, + 'authenticated probe subject differs from enabled Worker', + ); + }, + }); + assert(offConfirmed, 'off deploy was not confirmed'); + return result; + } catch (error) { + appendFileSync( + join(directory, 'watchdog.log'), + `controller ${stage}: ${privateDiagnostic(error, [settings.secret, settings.token])}\n`, + { mode: 0o600 }, + ); + throw new Error( + `staging controller failed at ${stage}; inspect private watchdog log`, + ); + } +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const action = process.argv[2]; + const execute = + action === '--watchdog' + ? watchdog(process.argv[3]) + : action === '--run' && process.argv.length === 3 + ? controller() + : Promise.reject(new Error('usage: --run')); + execute.then( + (result) => { + if (result) process.stdout.write(`${JSON.stringify(result)}\n`); + }, + (error) => { + process.stderr.write(`${error.message}\n`); + process.exitCode = 1; + }, + ); +} diff --git a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs new file mode 100644 index 00000000..dcc0400a --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs @@ -0,0 +1,971 @@ +import assert from 'node:assert/strict'; +import { spawn, spawnSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { createServer } from 'node:http'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { after, test } from 'node:test'; +import { fileURLToPath } from 'node:url'; +import { + createWranglerControl, + sealFence, +} from './notification-staging-live.mjs'; + +const enabled = 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa'; +const disabled = 'bbbbbbbb-bbbb-4bbb-bbbb-bbbbbbbbbbbb'; +const bundleDirectory = mkdtempSync( + join(tmpdir(), 'flowsafe-test-off-bundle-'), +); +const bundlePath = join(bundleDirectory, 'notification-staging.worker.js'); +writeFileSync(bundlePath, 'export default {};', { mode: 0o600 }); +after(() => rmSync(bundleDirectory, { recursive: true, force: true })); + +function freshConfig() { + const source = JSON.parse( + readFileSync( + new URL('./notification-staging.wrangler.jsonc', import.meta.url), + 'utf8', + ), + ); + source.name = 'anchorage-flowsafe-remote-step-staging-fresh'; + source.main = fileURLToPath( + new URL('./notification-staging.worker.ts', import.meta.url), + ); + source.alias = Object.fromEntries( + Object.entries(source.alias).map(([key, value]) => [ + key, + fileURLToPath(new URL(value, import.meta.url)), + ]), + ); + source.d1_databases[0].database_name = 'anchorage-flowsafe-fresh-db'; + source.d1_databases[0].database_id = 'cccccccc-cccc-4ccc-cccc-cccccccccccc'; + source.vars.DEPLOYMENT_TENANT = 'flowsafenotificationfresh'; + return source; +} + +function versionBindings( + source, + proofEnabled, + mutationEpoch = proofEnabled ? 1 : 0, +) { + return [ + { + name: 'NOTIFICATION_PROOF_ENABLED', + type: 'plain_text', + text: String(proofEnabled), + }, + ...(mutationEpoch === null + ? [] + : [ + { + name: 'STAGING_MUTATION_EPOCH', + type: 'plain_text', + text: String(mutationEpoch), + }, + ]), + { + name: 'DEPLOYMENT_TENANT', + type: 'plain_text', + text: source.vars.DEPLOYMENT_TENANT, + }, + { + name: 'DB', + type: 'd1', + database_id: source.d1_databases[0].database_id, + }, + ]; +} + +function fakeCli({ enabledFlag = true, priorVersion, mutationEpoch } = {}) { + const calls = []; + let version = enabled; + const command = (_executable, args) => { + calls.push(args); + if (args[0] === 'deployments') { + return { + status: 0, + stdout: JSON.stringify([ + { + created_on: '2026-10-15T00:00:00.000Z', + versions: [{ version_id: version, percentage: 100 }], + }, + ...(priorVersion + ? [ + { + created_on: '2026-10-14T23:59:00.000Z', + versions: [{ version_id: priorVersion, percentage: 100 }], + }, + ] + : []), + ]), + }; + } + if (args[0] === 'versions') { + const source = JSON.parse( + readFileSync( + new URL('./notification-staging.wrangler.jsonc', import.meta.url), + 'utf8', + ), + ); + return { + status: 0, + stdout: JSON.stringify({ + resources: { + bindings: versionBindings( + source, + args[2] === enabled && enabledFlag, + mutationEpoch, + ), + }, + }), + }; + } + if (args[0] === 'deploy') { + version = disabled; + return { status: 0, stdout: 'deployed' }; + } + if (args[0] === 'd1') { + return { + status: 0, + stdout: JSON.stringify([{ success: true, results: [{ count: 0 }] }]), + }; + } + throw new Error('unexpected CLI'); + }; + const cli = createWranglerControl({ command, token: 'fake-token' }); + return { + cli, + calls, + change() { + version = disabled; + }, + }; +} + +test('version-checked off deploy uses the tracked false config', () => { + const fixture = fakeCli(); + assert.equal(fixture.cli.currentVersion(), enabled); + assert.equal(fixture.cli.proofEnabled(enabled), true); + assert.equal( + fixture.cli.deployOffIfCurrent( + enabled, + '/tmp/staging-off-test', + bundlePath, + ), + disabled, + ); + assert.equal(fixture.cli.proofEnabled(disabled), false); + assert.equal(fixture.calls.filter((args) => args[0] === 'deploy').length, 1); +}); + +test('changed version or disabled flag refuses before deploy', () => { + const changed = fakeCli(); + changed.change(); + assert.throws( + () => + changed.cli.deployOffIfCurrent( + enabled, + '/tmp/staging-off-test', + bundlePath, + ), + /changed/, + ); + assert.equal(changed.calls.filter((args) => args[0] === 'deploy').length, 0); + const disabledFlag = fakeCli({ enabledFlag: false }); + assert.throws( + () => + disabledFlag.cli.deployOffIfCurrent( + enabled, + '/tmp/staging-off-test', + bundlePath, + ), + /not enabled/, + ); + assert.equal( + disabledFlag.calls.filter((args) => args[0] === 'deploy').length, + 0, + ); +}); + +test('retry evidence requires the immediately prior full off deployment', () => { + const prior = fakeCli({ priorVersion: disabled }); + assert.equal(prior.cli.priorOffVersion(enabled), disabled); + const absent = fakeCli(); + assert.throws(() => absent.cli.priorOffVersion(enabled), /history differs/); + const enabledPrior = fakeCli({ priorVersion: enabled }); + assert.throws(() => enabledPrior.cli.priorOffVersion(enabled), /not off/); +}); + +test('enabled version must pin the epoch the controller will open', () => { + for (const epoch of [null, '01', '0', '2']) { + const fixture = fakeCli({ mutationEpoch: epoch }); + assert.throws( + () => fixture.cli.proofEnabled(enabled, 1), + /epoch binding differs/, + ); + } + assert.equal( + fakeCli({ mutationEpoch: 1 }).cli.proofEnabled(enabled, 1), + true, + ); + assert.equal( + fakeCli({ mutationEpoch: 2 }).cli.proofEnabled(enabled, 2), + true, + ); +}); + +test('enabled version requires exact D1 and tenant bindings', () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-d1-mismatch-test-')); + try { + const source = freshConfig(); + const path = join(directory, 'private.wrangler.jsonc'); + writeFileSync(path, JSON.stringify(source), { mode: 0o600 }); + let boundDatabaseId = 'dddddddd-dddd-4ddd-dddd-dddddddddddd'; + let boundTenant = source.vars.DEPLOYMENT_TENANT; + const cli = createWranglerControl({ + configPath: path, + token: 'fake-token', + command: (_executable, args) => { + assert.equal(args[0], 'versions'); + return { + status: 0, + stdout: JSON.stringify({ + resources: { + bindings: [ + { + name: 'NOTIFICATION_PROOF_ENABLED', + type: 'plain_text', + text: 'true', + }, + { + name: 'DEPLOYMENT_TENANT', + type: 'plain_text', + text: boundTenant, + }, + { + name: 'DB', + type: 'd1', + database_id: boundDatabaseId, + }, + ], + }, + }), + }; + }, + }); + assert.throws( + () => cli.proofEnabled(enabled), + /enabled D1 binding differs/, + ); + boundDatabaseId = source.d1_databases[0].database_id; + boundTenant = 'wrongtenant'; + assert.throws( + () => cli.proofEnabled(enabled), + /enabled deployment tenant differs/, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + +test('private staging config scopes version, D1, and off deployment to one fresh resource', () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-fresh-config-test-')); + try { + const source = freshConfig(); + const path = join(directory, 'private.wrangler.jsonc'); + writeFileSync(path, JSON.stringify(source), { mode: 0o600 }); + const calls = []; + let currentVersion = enabled; + const command = (_executable, args) => { + calls.push(args); + if (args[0] === 'deployments') + return { + status: 0, + stdout: JSON.stringify([ + { + created_on: '2026-10-15T00:00:00.000Z', + versions: [{ version_id: currentVersion, percentage: 100 }], + }, + ]), + }; + if (args[0] === 'versions') + return { + status: 0, + stdout: JSON.stringify({ + resources: { + bindings: versionBindings(source, args[2] === enabled), + }, + }), + }; + if (args[0] === 'd1') + return { + status: 0, + stdout: JSON.stringify([{ success: true, results: [] }]), + }; + if (args[0] === 'deploy') { + currentVersion = disabled; + return { status: 0, stdout: 'deployed' }; + } + throw new Error('unexpected CLI'); + }; + const cli = createWranglerControl({ + command, + token: 'fake-token', + configPath: path, + }); + assert.equal(cli.currentVersion(), enabled); + assert.deepEqual(cli.query('SELECT 1'), []); + assert.equal( + cli.deployOffIfCurrent(enabled, join(directory, 'off'), bundlePath), + disabled, + ); + assert.equal(calls.filter((args) => args[0] === 'deploy').length, 1); + for (const args of calls) { + assert.equal(args[args.indexOf('--config') + 1], path); + if (args[0] === 'd1') + assert.equal(args[2], source.d1_databases[0].database_name); + } + source.routes = [{ pattern: 'example.com/*', zone_name: 'example.com' }]; + writeFileSync(path, JSON.stringify(source), { mode: 0o600 }); + assert.throws( + () => cli.query('SELECT 1'), + /private staging config differs/, + ); + assert.throws( + () => + createWranglerControl({ + command, + token: 'fake-token', + configPath: path, + }), + /private staging config differs/, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + +test('Wrangler failure preserves exit and stderr in its diagnostic cause', () => { + const cli = createWranglerControl({ + token: 'fake-token', + command: () => ({ status: 1, signal: null, stderr: 'deployment rejected' }), + }); + assert.throws( + () => cli.currentVersion(), + (error) => { + assert.match(error.message, /Wrangler deployments list failed/); + assert.match(error.cause?.message, /exit=1/); + assert.match(error.cause?.message, /deployment rejected/); + return true; + }, + ); +}); + +test('watchdog seals the initial fence against a delayed controller open', async () => { + const calls = []; + let fence = { + state: 'migration-locked', + mutationEpoch: 0, + transitionRevision: 0, + requireMutationEpoch: false, + }; + const probe = async (method, path, body) => { + calls.push({ method, path, body }); + if (method === 'GET') return fence; + assert.equal(body.expected, 'migration-locked'); + assert.equal(body.next, 'migration-locked'); + assert.equal(body.expectedMutationEpoch, 0); + assert.equal(body.expectedRevision, 0); + fence = { + state: 'migration-locked', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + return fence; + }; + assert.deepEqual(await sealFence(probe), fence); + assert.equal(calls.filter(({ method }) => method === 'POST').length, 1); + assert.equal(fence.mutationEpoch, 1); + assert.equal(fence.transitionRevision, 1); +}); + +test('watchdog seals the explicit retry baseline against a delayed controller open', async () => { + let fence = { + state: 'migration-locked', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + const probe = async (method, _path, body) => { + if (method === 'GET') return fence; + assert.deepEqual(body, { + expected: 'migration-locked', + next: 'migration-locked', + expectedMutationEpoch: 1, + expectedRevision: 1, + advanceMutationEpoch: true, + }); + fence = { + state: 'migration-locked', + mutationEpoch: 2, + transitionRevision: 2, + requireMutationEpoch: true, + }; + return fence; + }; + assert.deepEqual(await sealFence(probe, 1), fence); +}); + +test('watchdog relocks when controller open wins the deadline CAS', async () => { + let fence = { + state: 'migration-locked', + mutationEpoch: 0, + transitionRevision: 0, + requireMutationEpoch: false, + }; + const probe = async (method, _path, body) => { + if (method === 'GET') return fence; + if (body.expected === 'migration-locked') { + fence = { + state: 'open', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + throw new Error('controller open won'); + } + assert.deepEqual(body, { + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: 1, + expectedRevision: 1, + advanceMutationEpoch: true, + }); + fence = { + state: 'migration-locked', + mutationEpoch: 2, + transitionRevision: 2, + requireMutationEpoch: true, + }; + return fence; + }; + assert.deepEqual(await sealFence(probe), fence); + assert.equal(fence.state, 'migration-locked'); + assert.equal(fence.transitionRevision, 2); +}); + +test('watchdog resolves a lost relock response from the authoritative fence', async () => { + let fence = { + state: 'open', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + const probe = async (method, _path, body) => { + if (method === 'GET') return fence; + assert.equal(body.expected, 'open'); + fence = { + state: 'migration-locked', + mutationEpoch: 2, + transitionRevision: 2, + requireMutationEpoch: true, + }; + throw new Error('response lost after commit'); + }; + assert.deepEqual(await sealFence(probe), fence); +}); + +async function exerciseWatchdog(outcome) { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-live-watchdog-test-')); + const secret = 's'.repeat(32); + const tokenFile = join(directory, 'cf-token'); + const secretFile = join(directory, 'probe-secret'); + const preloadFile = join(directory, 'fake-wrangler.mjs'); + const offFile = join(directory, 'off-deploy'); + const offConfirmedFile = join(directory, 'off-confirmed'); + const watcherPidFile = join(directory, 'watchdog-pid'); + const deployStartedFile = join(directory, 'deploy-started'); + const releaseDeployFile = join(directory, 'release-deploy'); + const effectStartedFile = join(directory, 'effect-started'); + const fenceFile = join(directory, 'fence.json'); + const deadProbe = + outcome === 'dead probe after relock' || + outcome === 'dead probe while open'; + const accountId = + outcome === 'parent death' + ? 'c'.repeat(32) + : outcome === 'sealed retry' + ? 'e'.repeat(32) + : outcome === 'fresh config' + ? 'f'.repeat(32) + : outcome === 'mismatched probe' + ? '1'.repeat(32) + : 'd'.repeat(32); + const workerName = + outcome === 'fresh config' + ? 'anchorage-flowsafe-remote-step-staging-fresh' + : 'anchorage-flowsafe-remote-step-staging'; + const lockFile = join( + '/tmp', + `flowsafe-deploy-locks-${process.getuid()}`, + `${accountId}-${workerName}.lock`, + ); + writeFileSync(tokenFile, 'fake-token', { mode: 0o600 }); + writeFileSync(secretFile, secret, { mode: 0o600 }); + const privateConfigPath = join(directory, 'fresh.wrangler.jsonc'); + const privateProbeConfigPath = join(directory, 'fresh-probe.wrangler.jsonc'); + if (outcome === 'fresh config') { + writeFileSync(privateConfigPath, JSON.stringify(freshConfig()), { + mode: 0o600, + }); + const probeConfig = JSON.parse( + readFileSync( + new URL('./notification-staging-probe.wrangler.jsonc', import.meta.url), + 'utf8', + ), + ); + probeConfig.main = fileURLToPath( + new URL('./notification-staging-probe.worker.ts', import.meta.url), + ); + probeConfig.services[0].service = workerName; + writeFileSync(privateProbeConfigPath, JSON.stringify(probeConfig), { + mode: 0o600, + }); + } + writeFileSync( + preloadFile, + `import childProcess from 'node:child_process'; +import { syncBuiltinESMExports } from 'node:module'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +const originalSpawn = childProcess.spawn; +const originalSpawnSync = childProcess.spawnSync; +const originalNow = Date.now; +Date.now = () => originalNow() + + (process.env.TEST_CLOCK_JUMP === 'true' && + !process.argv.includes('--watchdog') && + existsSync(process.env.TEST_DEPLOY_STARTED_FILE) ? 300_000 : 0); +childProcess.spawn = (...args) => { + const child = originalSpawn(...args); + if (args[1]?.includes('--watchdog')) + writeFileSync(process.env.TEST_WATCHDOG_PID_FILE, String(child.pid)); + return child; +}; +childProcess.spawnSync = (executable, args, options) => { + if (executable === 'flock') return originalSpawnSync(executable, args, options); + if (!executable.endsWith('/node_modules/.bin/wrangler')) + throw new Error('unexpected synchronous command'); + if (process.env.TEST_FRESH_CONFIG === 'true') { + const configPath = args[args.indexOf('--config') + 1]; + if (!configPath.endsWith('/off.wrangler.jsonc')) + throw new Error('fresh config snapshot was not used'); + if (args[0] === 'd1' && args[2] !== 'anchorage-flowsafe-fresh-db') + throw new Error('fresh D1 was not selected'); + } + const off = existsSync(process.env.TEST_OFF_FILE); + const command = args[0]; + if (off && command === 'deployments') + writeFileSync(process.env.TEST_OFF_CONFIRMED_FILE, 'checked'); + const sql = args.at(-1); + const effectStarted = existsSync(process.env.TEST_EFFECT_STARTED_FILE); + if (command === 'd1' && !effectStarted && + (sql.includes('FROM flowsafe_notification_operation_receipts') || + sql.includes('FROM mastra_notifications'))) + return { status: 1, stderr: 'notification table is absent' }; + const response = command === 'deployments' ? [{ + created_on: '2026-10-15T00:00:00.000Z', + versions: [{ version_id: off ? '${disabled}' : '${enabled}', percentage: 100 }], + }, ...(process.env.TEST_RETRY === 'true' ? [{ + created_on: '2026-10-14T23:59:00.000Z', + versions: [{ version_id: '${disabled}', percentage: 100 }], + }] : [])] : command === 'versions' ? { + resources: { bindings: [{ + name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', + text: off || args[2] === '${disabled}' ? 'false' : 'true', + }, { + name: 'STAGING_MUTATION_EPOCH', type: 'plain_text', + text: off || args[2] === '${disabled}' ? '0' : process.env.TEST_RETRY === 'true' ? '2' : '1', + }, { + name: 'DEPLOYMENT_TENANT', type: 'plain_text', + text: process.env.TEST_FRESH_CONFIG === 'true' + ? 'flowsafenotificationfresh' : 'flowsafenotificationstaging', + }, { + name: 'DB', type: 'd1', + database_id: process.env.TEST_FRESH_CONFIG === 'true' + ? 'cccccccc-cccc-4ccc-cccc-cccccccccccc' + : 'b863a219-2096-4e13-92fe-4db592b67944', + }] }, + } : command === 'd1' ? [{ success: true, results: + sql.includes('FROM flowsafe_execution_fence') + ? [JSON.parse(readFileSync(process.env.TEST_FENCE_FILE, 'utf8'))] + : sql.includes("name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')") + ? effectStarted ? [{ name: 'mastra_notifications' }, + { name: 'flowsafe_notification_operation_receipts' }] : [] + : sql.includes('sqlite_master') + ? [{ sql: 'write_generation INTEGER NOT NULL, operation_state TEXT' }] + : sql.includes('COUNT(*) AS count FROM flowsafe_remote_step_invocations') + ? [{ count: 0 }] + : sql.includes('COUNT(*) AS count FROM mastra_notifications') || + sql.includes('COUNT(*) AS count FROM flowsafe_notification_operation_receipts') + ? [{ count: effectStarted ? 1 : 0 }] + : sql.includes('FROM flowsafe_remote_step_invocations i') + ? [{ status: 'settled', settlement_outcome: 'applied', + operation_state: 'resume-admitted', thread_id: 'thread', + notification_id: 'notification', receipt_key: 'key', + receipt_digest: 'a'.repeat(64), receipt_thread_id: 'thread', + receipt_notification_id: 'notification', + stored_notification_id: 'notification', source: 'flowsafe-staging', + kind: 'host-owned-notification' }] + : [], + }] : null; + if (command === 'deploy') { + if (args.includes('--dry-run')) { + const outdir = args[args.indexOf('--outdir') + 1]; + mkdirSync(outdir, { recursive: true, mode: 0o700 }); + writeFileSync(outdir + '/notification-staging.worker.js', 'export default {};'); + return { status: 0, stdout: 'bundled' }; + } + if (!args[1]?.endsWith('/pinned-off-bundle/notification-staging.worker.js') || + !existsSync(args[1])) + throw new Error('pinned off bundle was not used'); + writeFileSync(process.env.TEST_DEPLOY_STARTED_FILE, 'started'); + while (!existsSync(process.env.TEST_RELEASE_DEPLOY_FILE)) + originalSpawnSync('sleep', ['0.02']); + writeFileSync(process.env.TEST_OFF_FILE, 'deployed'); + return { status: 0, stdout: 'deployed' }; + } + if (response === null) throw new Error('unexpected Wrangler command'); + return { status: 0, stdout: JSON.stringify(response) }; +}; +syncBuiltinESMExports(); +`, + ); + + const requests = []; + let fence = { + state: 'migration-locked', + mutationEpoch: outcome === 'sealed retry' ? 1 : 0, + requireMutationEpoch: outcome === 'sealed retry', + transitionRevision: outcome === 'sealed retry' ? 1 : 0, + }; + let startReceived = false; + let probeUnavailable = false; + const recordFence = () => + writeFileSync( + fenceFile, + JSON.stringify({ + id: 'deployment', + state: fence.state, + proof_key: null, + proof_run_id: null, + updated_at: 1, + last_transition_request: + fence.mutationEpoch === 0 ? null : 'test-transition', + transition_revision: fence.transitionRevision, + mutation_epoch: fence.mutationEpoch, + require_mutation_epoch: Number(fence.requireMutationEpoch), + proof_table_prefix: null, + proof_workflow_id: null, + proof_start_token: null, + }), + ); + recordFence(); + const server = createServer(async (request, response) => { + assert.equal(request.headers.authorization, `Bearer ${secret}`); + const path = request.url; + const method = request.method; + requests.push(`${method} ${path}`); + if (probeUnavailable) { + response.destroy(); + return; + } + if (method === 'POST' && path === '/start') { + startReceived = true; + writeFileSync(effectStartedFile, 'started'); + if (outcome === 'dead probe while open') { + probeUnavailable = true; + response.destroy(); + return; + } + if (outcome === 'parent death' || outcome === 'fresh config') return; + response.writeHead(200, { 'content-type': 'application/json' }); + response.end( + JSON.stringify({ status: 'suspended', invocationId: 'invocation' }), + ); + return; + } + let value; + if (path === '/complete') { + value = { invocationId: 'invocation', status: 'success' }; + } else if (path === '/identity') { + value = { + versionId: outcome === 'mismatched probe' ? disabled : enabled, + deploymentTenant: + outcome === 'fresh config' + ? 'flowsafenotificationfresh' + : 'flowsafenotificationstaging', + }; + } else if (path === '/admin/maintenance-status') { + value = { alarmAt: 1, nextRemoteStepRecoveryAt: 1 }; + } else if (path === '/admin/execution-fence' && method === 'POST') { + const chunks = []; + for await (const chunk of request) chunks.push(chunk); + const body = JSON.parse(Buffer.concat(chunks).toString('utf8')); + assert.equal(body.expected, fence.state); + assert.equal(body.expectedMutationEpoch, fence.mutationEpoch); + assert.equal(body.expectedRevision, fence.transitionRevision); + assert.equal(body.advanceMutationEpoch, true); + fence = { + state: body.next, + mutationEpoch: fence.mutationEpoch + 1, + requireMutationEpoch: true, + transitionRevision: fence.transitionRevision + 1, + }; + recordFence(); + if ( + outcome === 'dead probe after relock' && + fence.state === 'migration-locked' + ) + probeUnavailable = true; + value = fence; + } else { + assert.equal(path, '/admin/execution-fence'); + value = fence; + } + response.writeHead(200, { 'content-type': 'application/json' }); + response.end(JSON.stringify(value)); + }); + let controller; + let watcherPid; + try { + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const port = server.address().port; + controller = spawn( + process.execPath, + [ + fileURLToPath( + new URL('./notification-staging-live.mjs', import.meta.url), + ), + '--run', + ], + { + env: { + ...process.env, + NODE_OPTIONS: `--import=${preloadFile}`, + TMPDIR: directory, + FLOWSAFE_STAGING_PROBE_PORT: String(port), + FLOWSAFE_STAGING_ENABLED_VERSION: enabled, + ...(outcome === 'sealed retry' + ? { FLOWSAFE_STAGING_RETRY_OFF_VERSION: disabled } + : {}), + FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, + FLOWSAFE_CF_TOKEN_FILE: tokenFile, + FLOWSAFE_CF_ACCOUNT_ID: accountId, + ...(outcome === 'fresh config' + ? { + FLOWSAFE_STAGING_CONFIG_FILE: privateConfigPath, + FLOWSAFE_STAGING_PROBE_CONFIG_FILE: privateProbeConfigPath, + } + : {}), + TEST_OFF_FILE: offFile, + TEST_OFF_CONFIRMED_FILE: offConfirmedFile, + TEST_WATCHDOG_PID_FILE: watcherPidFile, + TEST_DEPLOY_STARTED_FILE: deployStartedFile, + TEST_RELEASE_DEPLOY_FILE: releaseDeployFile, + TEST_EFFECT_STARTED_FILE: effectStartedFile, + TEST_FENCE_FILE: fenceFile, + TEST_CLOCK_JUMP: outcome === 'controller request' ? 'true' : 'false', + TEST_RETRY: outcome === 'sealed retry' ? 'true' : 'false', + TEST_FRESH_CONFIG: outcome === 'fresh config' ? 'true' : 'false', + }, + stdio: 'ignore', + }, + ); + const stateFile = join( + directory, + `flowsafe-notification-control-${controller.pid}`, + 'watchdog.json', + ); + const deadline = Date.now() + 8_000; + const waitFor = async (condition) => { + while (!condition()) { + assert(Date.now() < deadline, 'watchdog process timed out'); + await new Promise((resolve) => setTimeout(resolve, 20)); + } + }; + await waitFor(() => existsSync(`${stateFile}.ready`)); + if (outcome === 'fresh config') { + const state = JSON.parse(readFileSync(stateFile, 'utf8')); + assert.equal(state.workerName, workerName); + assert.equal( + state.configPath, + join(dirname(stateFile), 'off.wrangler.jsonc'), + ); + assert.equal( + readFileSync(state.configPath, 'utf8'), + readFileSync(privateConfigPath, 'utf8'), + ); + } + if (outcome === 'sealed retry') + assert.equal( + JSON.parse(readFileSync(stateFile, 'utf8')).baselineEpoch, + 1, + ); + await waitFor(() => existsSync(watcherPidFile)); + watcherPid = Number(readFileSync(watcherPidFile, 'utf8')); + if (outcome !== 'mismatched probe') await waitFor(() => startReceived); + assert.equal(existsSync(offFile), false); + if (outcome === 'parent death' || outcome === 'fresh config') { + assert.equal(fence.state, 'open'); + controller.kill('SIGKILL'); + } + await waitFor(() => existsSync(deployStartedFile)); + assert.equal( + spawnSync('flock', ['--nonblock', lockFile, 'true']).status, + 1, + ); + writeFileSync(releaseDeployFile, 'release'); + await waitFor(() => existsSync(offConfirmedFile)); + assert.equal(readFileSync(offFile, 'utf8'), 'deployed'); + assert.deepEqual(fence, { + state: outcome === 'dead probe while open' ? 'open' : 'migration-locked', + mutationEpoch: + outcome === 'sealed retry' + ? 3 + : outcome === 'mismatched probe' || + outcome === 'dead probe while open' + ? 1 + : 2, + requireMutationEpoch: true, + transitionRevision: + outcome === 'sealed retry' + ? 3 + : outcome === 'mismatched probe' || + outcome === 'dead probe while open' + ? 1 + : 2, + }); + assert.deepEqual( + requests.filter((request) => request === 'POST /admin/execution-fence'), + outcome === 'mismatched probe' || outcome === 'dead probe while open' + ? ['POST /admin/execution-fence'] + : ['POST /admin/execution-fence', 'POST /admin/execution-fence'], + ); + if (outcome === 'mismatched probe') assert.equal(startReceived, false); + if (outcome === 'controller request') { + await waitFor(() => controller.exitCode !== null); + assert.equal(controller.exitCode, 0); + } + await waitFor( + () => spawnSync('flock', ['--nonblock', lockFile, 'true']).status === 0, + ); + if (deadProbe) { + await waitFor(() => controller.exitCode !== null); + assert.equal( + controller.exitCode, + outcome === 'dead probe after relock' ? 0 : 1, + ); + assert.equal( + existsSync(`${stateFile}.off-confirmed`), + outcome === 'dead probe after relock', + ); + } + } finally { + controller?.kill('SIGKILL'); + if (Number.isInteger(watcherPid)) { + try { + process.kill(watcherPid, 'SIGKILL'); + } catch { + // The watchdog may have exited after the off deployment. + } + } + server.closeAllConnections(); + await new Promise((resolve) => server.close(resolve)); + rmSync(directory, { recursive: true, force: true }); + } +} + +test('watchdog owns off deployment after parent death', () => + exerciseWatchdog('parent death')); + +test('fresh Worker watchdog uses the private off config snapshot after parent death', () => + exerciseWatchdog('fresh config')); + +test('mismatched authenticated probe subject prevents fence unlock and effect', () => + exerciseWatchdog('mismatched probe')); + +test('watchdog owns off deployment after controller request', () => + exerciseWatchdog('controller request')); + +test('sealed retry persists watchdog baseline before unlocking and relocks at three', () => + exerciseWatchdog('sealed retry')); + +test('watchdog deploys off through probe failure after the controller relocks', () => + exerciseWatchdog('dead probe after relock')); + +test('watchdog deploys off but refuses confirmation when the dead probe leaves an open fence', () => + exerciseWatchdog('dead probe while open')); + +test('controller refuses a contended deployment lock before Wrangler preflight', async () => { + const directory = mkdtempSync( + join(tmpdir(), 'flowsafe-lock-contention-test-'), + ); + const accountId = 'e'.repeat(32); + const lockDirectory = join( + '/tmp', + `flowsafe-deploy-locks-${process.getuid()}`, + ); + const lockFile = join( + lockDirectory, + `${accountId}-anchorage-flowsafe-remote-step-staging.lock`, + ); + mkdirSync(lockDirectory, { recursive: true, mode: 0o700 }); + writeFileSync(lockFile, '', { mode: 0o600 }); + const secretFile = join(directory, 'probe-secret'); + const tokenFile = join(directory, 'cf-token'); + writeFileSync(secretFile, 's'.repeat(32), { mode: 0o600 }); + writeFileSync(tokenFile, 'fake-token', { mode: 0o600 }); + const holder = spawn('flock', ['--exclusive', lockFile, 'sleep', '5'], { + stdio: 'ignore', + }); + try { + const deadline = Date.now() + 3_000; + while (spawnSync('flock', ['--nonblock', lockFile, 'true']).status !== 1) { + assert(Date.now() < deadline, 'lock holder did not acquire lock'); + await new Promise((resolve) => setTimeout(resolve, 20)); + } + const result = spawnSync( + process.execPath, + [ + fileURLToPath( + new URL('./notification-staging-live.mjs', import.meta.url), + ), + '--run', + ], + { + encoding: 'utf8', + env: { + ...process.env, + FLOWSAFE_STAGING_PROBE_PORT: '12345', + FLOWSAFE_STAGING_ENABLED_VERSION: enabled, + FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, + FLOWSAFE_CF_TOKEN_FILE: tokenFile, + FLOWSAFE_CF_ACCOUNT_ID: accountId, + }, + }, + ); + assert.equal(result.status, 1); + assert.match(result.stderr, /deployment lock is unavailable/); + assert.doesNotMatch(result.stderr, /Wrangler/); + } finally { + holder.kill('SIGKILL'); + rmSync(directory, { recursive: true, force: true }); + } +}); diff --git a/packages/flowsafe/deploy/notification-staging-native.mjs b/packages/flowsafe/deploy/notification-staging-native.mjs new file mode 100644 index 00000000..182a741a --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-native.mjs @@ -0,0 +1,225 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + EXECUTION_FENCE_DDL, + EXECUTION_FENCE_ROW_ID, + EXECUTION_FENCE_TABLE, +} from '#deployment-identity-protocol'; +import { + createWorkerdServerLifecycle, + parsePort, +} from '../../../scripts/workerd-server-lifecycle.mjs'; + +const flowsafe = dirname(dirname(fileURLToPath(import.meta.url))); +const wrangler = join(flowsafe, '../../node_modules/.bin/wrangler'); +const config = join(flowsafe, 'deploy/notification-staging.wrangler.jsonc'); +const port = parsePort( + process.env.NOTIFICATION_STAGING_NATIVE_PORT ?? 8897, + 'NOTIFICATION_STAGING_NATIVE_PORT', +); +const tag = 'flowsafenotificationstaging'; +const database = 'anchorage-flowsafe-remote-step-staging-db'; +const secret = () => randomBytes(32).toString('hex'); +const deploymentSecret = secret(); +const maintenanceSecret = secret(); +const probeSecret = secret(); +const lifecycle = createWorkerdServerLifecycle({ port }); +let stateDir; +let serverOutput = ''; + +function workerdDiagnostics() { + return serverOutput + .replaceAll(deploymentSecret, '[redacted]') + .replaceAll(maintenanceSecret, '[redacted]') + .replaceAll(probeSecret, '[redacted]'); +} + +function expect(condition, message, detail) { + if (!condition) + throw new Error( + `${message}${detail === undefined ? '' : `: ${JSON.stringify(detail)}`}`, + ); +} + +function localD1(command) { + const result = spawnSync( + wrangler, + [ + 'd1', + 'execute', + database, + '--local', + '--yes', + '--json', + '--config', + config, + '--persist-to', + stateDir, + '--command', + command, + ], + { cwd: flowsafe, encoding: 'utf8', timeout: 30_000 }, + ); + expect( + result.status === 0, + 'local D1 query failed', + result.stderr?.slice(-800), + ); + return JSON.parse(result.stdout); +} + +async function post(path, authorization, body) { + const response = await fetch(`http://127.0.0.1:${port}${path}`, { + method: 'POST', + headers: { + ...(authorization ? { 'x-flowsafe-staging-probe': authorization } : {}), + ...(body ? { 'content-type': 'application/json' } : {}), + }, + ...(body ? { body: JSON.stringify(body) } : {}), + signal: AbortSignal.timeout(90_000), + }); + const text = await response.text(); + let value; + try { + value = JSON.parse(text); + } catch { + value = text.slice(0, 1000); + } + return { status: response.status, value }; +} + +async function run() { + await lifecycle.preflight(); + stateDir = mkdtempSync(join(tmpdir(), 'flowsafe-notification-native-')); + localD1( + 'CREATE TABLE IF NOT EXISTS flowsafe_deployment (id INTEGER PRIMARY KEY CHECK (id = 1), tenant_tag TEXT NOT NULL, provisioned_at TEXT NOT NULL); ' + + `INSERT OR IGNORE INTO flowsafe_deployment (id, tenant_tag, provisioned_at) VALUES (1, '${tag}', datetime('now')); ` + + `${EXECUTION_FENCE_DDL}; ` + + `INSERT OR IGNORE INTO ${EXECUTION_FENCE_TABLE} (id, state, proof_key, proof_run_id, updated_at) ` + + `VALUES ('${EXECUTION_FENCE_ROW_ID}', 'migration-locked', NULL, NULL, ${Date.now()});`, + ); + await lifecycle.start('notification-staging-native', () => { + const child = spawn( + wrangler, + [ + 'dev', + '--config', + config, + '--local', + '--port', + String(port), + '--ip', + '127.0.0.1', + '--inspector-port', + '0', + '--persist-to', + stateDir, + '--var', + `DEPLOYMENT_TENANT:${tag}`, + '--var', + `DEPLOYMENT_IDENTITY_SECRET:${deploymentSecret}`, + '--var', + `MAINTENANCE_ADMIN_SECRET:${maintenanceSecret}`, + '--var', + `STAGING_PROBE_SECRET:${probeSecret}`, + '--var', + 'NOTIFICATION_PROOF_ENABLED:true', + '--var', + 'STAGING_MUTATION_EPOCH:1', + ], + { + cwd: flowsafe, + detached: true, + stdio: ['ignore', 'pipe', 'pipe'], + env: { ...process.env, WRANGLER_SEND_METRICS: 'false', NO_COLOR: '1' }, + }, + ); + const capture = (chunk) => { + serverOutput = `${serverOutput}${chunk.toString()}`.slice(-30000); + }; + child.stdout.on('data', capture); + child.stderr.on('data', capture); + return { child, generation: 'notification-staging-native' }; + }); + const refused = await post('/__staging/start'); + expect( + refused.status === 403, + 'private start admitted an unauthenticated call', + refused, + ); + const opened = await fetch(`http://127.0.0.1:${port}/admin/execution-fence`, { + method: 'POST', + headers: { + authorization: `Bearer ${maintenanceSecret}`, + 'content-type': 'application/json', + }, + body: JSON.stringify({ + expected: 'migration-locked', + next: 'open', + expectedMutationEpoch: 0, + expectedRevision: 0, + advanceMutationEpoch: true, + }), + signal: AbortSignal.timeout(90_000), + }); + expect(opened.status === 200, 'native execution fence did not open'); + const fence = await opened.json(); + expect( + fence.mutationEpoch === 1 && fence.requireMutationEpoch === true, + 'native execution fence did not require the enabled epoch', + fence, + ); + const started = await post('/__staging/start', probeSecret); + expect(started.status === 200, 'guarded start/register failed', { + response: started, + workerd: workerdDiagnostics(), + }); + expect( + started.value.status === 'suspended', + 'guarded agent did not suspend', + started, + ); + expect( + typeof started.value.invocationId === 'string', + 'journal registration did not return an invocation', + started, + ); + const completed = await post('/__staging/complete', probeSecret, { + invocationId: started.value.invocationId, + }); + expect(completed.status === 200, 'selected notification completion failed', { + response: completed, + workerd: workerdDiagnostics(), + }); + const rows = localD1( + `SELECT COUNT(*) AS count FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification'`, + ); + expect( + rows?.[0]?.results?.[0]?.count === 1, + 'selected notification was not retained exactly once', + rows, + ); + console.log( + JSON.stringify({ + verdict: 'native_notification_staging_passed', + runId: started.value.runId, + invocationId: started.value.invocationId, + completionStatus: completed.value.status, + notificationCount: 1, + }), + ); +} + +try { + await run(); +} catch (error) { + console.error(workerdDiagnostics()); + throw error; +} finally { + await lifecycle.stop(); + if (stateDir) rmSync(stateDir, { recursive: true, force: true }); +} diff --git a/packages/flowsafe/deploy/notification-staging-no-rest.ts b/packages/flowsafe/deploy/notification-staging-no-rest.ts new file mode 100644 index 00000000..697fcf6a --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-no-rest.ts @@ -0,0 +1,7 @@ +export default class UnavailableStagingRestClient { + constructor() { + throw new Error( + 'Cloudflare REST client is unavailable in notification staging', + ); + } +} diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts new file mode 100644 index 00000000..2deb0079 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts @@ -0,0 +1,174 @@ +import type { Fetcher } from '@cloudflare/workers-types'; +import { expect, it, vi } from 'vitest'; +import handler from './notification-staging-probe.worker.js'; + +it('forwards a staging request only after the local caller supplies the probe credential', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 204 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + }; + const request = (credential?: string) => + new Request('http://127.0.0.1/ensure', { + method: 'POST', + ...(credential + ? { headers: { authorization: `Bearer ${credential}` } } + : {}), + }); + + expect((await handler.fetch(request(), env)).status).toBe(403); + expect((await handler.fetch(request('wrong'), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + + expect((await handler.fetch(request(secret), env)).status).toBe(204); + expect(fetch).toHaveBeenCalledOnce(); + const forwarded = fetch.mock.calls[0]; + expect(forwarded?.[0]).toBe('http://subject/__staging/ensure'); + expect(forwarded?.[1]).toMatchObject({ method: 'POST' }); + expect(forwarded?.[1]?.headers).toMatchObject({ + 'x-flowsafe-staging-probe': secret, + }); +}); + +it('returns the authenticated subject version through the private binding', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => + Response.json({ + versionId: 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa', + deploymentTenant: 'flowsafenotificationfresh', + }), + ); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + }; + const request = (credential: string) => + new Request('http://127.0.0.1/identity', { + headers: { authorization: `Bearer ${credential}` }, + }); + expect((await handler.fetch(request('wrong'), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + const response = await handler.fetch(request(secret), env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ + versionId: 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa', + deploymentTenant: 'flowsafenotificationfresh', + }); + expect(fetch.mock.calls[0]?.[0]).toBe('http://subject/__staging/identity'); +}); + +it('keeps start and completion on the authenticated private service binding', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 204 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + }; + for (const path of ['/start', '/complete']) { + const body = + path === '/complete' ? '{"invocationId":"invocation-1"}' : undefined; + const request = (credential?: string) => + new Request(`http://127.0.0.1${path}`, { + method: 'POST', + ...(credential + ? { headers: { authorization: `Bearer ${credential}` } } + : {}), + ...(body ? { body } : {}), + }); + expect((await handler.fetch(request(), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + expect((await handler.fetch(request(secret), env)).status).toBe(204); + const forwarded = fetch.mock.calls.at(-1); + expect(forwarded?.[0]).toBe(`http://subject/__staging${path}`); + if (body) expect(forwarded?.[1]?.body).toBe(body); + fetch.mockClear(); + } +}); + +it('authenticates local admin requests and supplies only the proxy-owned admin credential', async () => { + const probeSecret = 'staging-local-probe-secret-32-bytes-minimum'; + const adminSecret = 'staging-admin-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 200 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: probeSecret, + MAINTENANCE_ADMIN_SECRET: adminSecret, + }; + for (const [method, path, body] of [ + ['GET', '/admin/execution-fence', undefined], + ['POST', '/admin/execution-fence', '{"expected":"open","next":"draining"}'], + ['GET', '/admin/maintenance-status', undefined], + ] as const) { + const request = (credential: string) => + new Request(`http://127.0.0.1${path}`, { + method, + headers: { authorization: `Bearer ${credential}` }, + ...(body ? { body } : {}), + }); + expect((await handler.fetch(request('wrong'), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + expect((await handler.fetch(request(probeSecret), env)).status).toBe(200); + expect(fetch).toHaveBeenCalledOnce(); + expect(fetch.mock.calls[0]?.[0]).toBe(`http://subject${path}`); + expect(fetch.mock.calls[0]?.[1]).toMatchObject({ + method, + headers: { authorization: `Bearer ${adminSecret}` }, + }); + expect(fetch.mock.calls[0]?.[1]?.body).toBe(body); + fetch.mockClear(); + } +}); + +it('rejects unlisted admin operations, query strings, and oversized writes', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 200 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + MAINTENANCE_ADMIN_SECRET: 'staging-admin-secret-32-bytes-minimum', + }; + const headers = { authorization: `Bearer ${secret}` }; + for (const request of [ + new Request('http://127.0.0.1/admin/inventory', { headers }), + new Request('http://127.0.0.1/admin/execution-fence', { + method: 'DELETE', + headers, + }), + new Request('http://127.0.0.1/admin/execution-fence?state=open', { + headers, + }), + ]) { + expect((await handler.fetch(request, env)).status).toBe(404); + } + expect( + ( + await handler.fetch( + new Request('http://127.0.0.1/admin/execution-fence', { + method: 'POST', + headers, + body: 'x'.repeat(4_097), + }), + env, + ) + ).status, + ).toBe(413); + expect(fetch).not.toHaveBeenCalled(); +}); + +it('fails closed when the admin credential is missing or reused as the caller credential', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 200 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + MAINTENANCE_ADMIN_SECRET: undefined as string | undefined, + }; + const request = new Request('http://127.0.0.1/admin/execution-fence', { + headers: { authorization: `Bearer ${secret}` }, + }); + expect((await handler.fetch(request, env)).status).toBe(503); + env.MAINTENANCE_ADMIN_SECRET = secret; + expect((await handler.fetch(request, env)).status).toBe(503); + expect(fetch).not.toHaveBeenCalled(); +}); diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.ts new file mode 100644 index 00000000..1a7934e3 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.ts @@ -0,0 +1,92 @@ +import type { Fetcher } from '@cloudflare/workers-types'; +import { + assertDeploymentIdentitySecret, + credentialsMatch, +} from '../src/do-runner/deployment-identity.js'; +import { readBoundedBody } from '../src/http-body.js'; + +const MAX_FORWARD_BODY_BYTES = 4_096; + +interface Env { + SUBJECT: Fetcher; + STAGING_PROBE_SECRET: string; + MAINTENANCE_ADMIN_SECRET?: string; +} + +export default { + async fetch(request: Request, env: Env): Promise { + const url = new URL(request.url); + if (url.hostname !== '127.0.0.1' && url.hostname !== 'localhost') + return new Response(null, { status: 404 }); + if (url.search || url.hash) return new Response(null, { status: 404 }); + const staging = + (request.method === 'POST' && + ['/ensure', '/start', '/complete'].includes(url.pathname)) || + (request.method === 'GET' && url.pathname === '/identity'); + const admin = + (request.method === 'GET' && + ['/admin/execution-fence', '/admin/maintenance-status'].includes( + url.pathname, + )) || + (request.method === 'POST' && url.pathname === '/admin/execution-fence'); + if (!staging && !admin) return new Response(null, { status: 404 }); + try { + assertDeploymentIdentitySecret( + env.STAGING_PROBE_SECRET, + 'staging probe credential', + ); + } catch { + return new Response(null, { status: 503 }); + } + const authorization = request.headers.get('authorization'); + if ( + !authorization?.startsWith('Bearer ') || + !(await credentialsMatch( + authorization.slice('Bearer '.length), + env.STAGING_PROBE_SECRET, + )) + ) + return new Response(null, { status: 403 }); + if (admin) { + try { + assertDeploymentIdentitySecret( + env.MAINTENANCE_ADMIN_SECRET, + 'staging maintenance admin credential', + ); + } catch { + return new Response(null, { status: 503 }); + } + if ( + await credentialsMatch( + env.STAGING_PROBE_SECRET, + env.MAINTENANCE_ADMIN_SECRET as string, + ) + ) + return new Response(null, { status: 503 }); + } + const hasBody = + (staging && url.pathname === '/complete') || + (admin && request.method === 'POST'); + const raw = await readBoundedBody( + request, + hasBody ? MAX_FORWARD_BODY_BYTES : 0, + ); + if (!raw.ok) + return new Response(null, { + status: raw.reason === 'payload-too-large' ? 413 : 400, + }); + return env.SUBJECT.fetch( + `http://subject${admin ? '' : '/__staging'}${url.pathname}`, + { + method: request.method, + headers: { + ...(admin + ? { authorization: `Bearer ${env.MAINTENANCE_ADMIN_SECRET}` } + : { 'x-flowsafe-staging-probe': env.STAGING_PROBE_SECRET }), + ...(hasBody ? { 'content-type': 'application/json' } : {}), + }, + ...(hasBody ? { body: raw.text } : {}), + }, + ) as Promise; + }, +}; diff --git a/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc new file mode 100644 index 00000000..a121acfe --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc @@ -0,0 +1,16 @@ +{ + "$schema": "https://unpkg.com/wrangler/config-schema.json", + "name": "anchorage-flowsafe-remote-step-staging-probe-local", + "main": "./notification-staging-probe.worker.ts", + "compatibility_date": "2026-09-10", + "compatibility_flags": ["nodejs_compat"], + "workers_dev": false, + "dev": { "ip": "127.0.0.1" }, + "services": [ + { + "binding": "SUBJECT", + "service": "anchorage-flowsafe-remote-step-staging", + "remote": true + } + ] +} diff --git a/packages/flowsafe/deploy/notification-staging.worker.ts b/packages/flowsafe/deploy/notification-staging.worker.ts new file mode 100644 index 00000000..206bbd2b --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging.worker.ts @@ -0,0 +1,475 @@ +import type { + Request as CfRequest, + D1Database, + DurableObjectNamespace, + DurableObjectState, + ExecutionContext, +} from '@cloudflare/workers-types'; +import type { Agent } from '@mastra/core/agent'; +import type { MastraModelConfig } from '@mastra/core/llm'; +import { Mastra } from '@mastra/core/mastra'; +import { createTool } from '@mastra/core/tools'; +import { createGuardedAgent } from '@proofoftech/breakwater/agent'; +import { AuditLogger } from '@proofoftech/breakwater/audit'; +import { z } from 'zod'; +import { + type AgentModule, + type AgentThreadStateStorage, + createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, + createThreadAgentHost, + type ThreadAgentHost, +} from '../src/agent-host/index.js'; +import { createFlowsafeDurableAgent } from '../src/agent-runner/index.js'; +import { + ApprovalService, + createPrincipalActorContext, +} from '../src/approval-api/index.js'; +import { credentialsMatch } from '../src/do-runner/deployment-identity.js'; +import { + assertDeploymentIdentitySecret, + createD1Storage, + createHostPubSub, + DurableObjectRunner, + doErrorResponse, + ensureDeploymentIdentityBindings, + executionFenceFor, + type InitResult, + init, + mutationEpochFromHeader, + type RunnerRuntime, + startIdempotencyFor, + ThreadDurableObject, + type ThreadScope, +} from '../src/do-runner/index.js'; +import { + REMOTE_STEP_MARKER, + RemoteStepJournal, +} from '../src/do-runner/remote-step-journal.js'; +import { + approvalStoreFactoryFor, + createFlowsafeMaintenanceDurableObject, + createFlowsafeWorker, + ensureRemoteStepNotificationRecoveryDuty, + type FlowsafeWorkerConfig, + staticTokenVerifier, +} from '../src/host-kit/index.js'; + +interface Env { + DB: D1Database; + DEPLOYMENT_TENANT: string; + DEPLOYMENT_IDENTITY_SECRET: string; + MAINTENANCE_ADMIN_SECRET: string; + STAGING_PROBE_SECRET: string; + STAGING_MUTATION_EPOCH: string; + NOTIFICATION_PROOF_ENABLED?: string; + CF_VERSION_METADATA: { id: string }; + RUNNER: DurableObjectNamespace; + THREAD: DurableObjectNamespace; + MAINTENANCE: DurableObjectNamespace; +} + +const OPERATION_ID = 'staging-notification-enqueue-v1'; +const AGENT_ID = 'staging-notification-agent'; +const TOOL_ID = 'staging_notification'; +const SYSTEM_ID = 'staging-notification-worker'; +const RESULT_JSON = '{"completed":true}'; +const EVIDENCE_JSON = '{"operation":"no-side-effect-staging-dispatch-v1"}'; + +function stagingAgent(): AgentModule { + const audit = new AuditLogger({ + sink: (event) => console.log(JSON.stringify(event)), + }); + const tool = createTool({ + id: TOOL_ID, + description: 'Suspend for the host-owned staging notification', + inputSchema: z.object({}), + outputSchema: z.object({ completed: z.boolean() }), + resumeSchema: z.object({ completed: z.boolean() }), + execute: async (_input, context) => { + if (context.agent?.resumeData) return { completed: true }; + if (!context.agent?.suspend) + throw new Error('staging tool suspension is unavailable'); + await context.agent.suspend({ + remoteStep: REMOTE_STEP_MARKER, + request: { notificationOperationId: OPERATION_ID }, + }); + throw new Error('staging tool suspension returned'); + }, + }); + const model: MastraModelConfig = { + specificationVersion: 'v2', + provider: 'flowsafe-staging', + modelId: 'fixed-notification-tool', + supportedUrls: {}, + doGenerate: async () => ({ + content: [ + { + type: 'tool-call', + toolCallId: `call_${crypto.randomUUID()}`, + toolName: TOOL_ID, + input: '{}', + }, + ], + finishReason: 'tool-calls', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + warnings: [], + }), + doStream: async () => ({ + stream: new ReadableStream({ + start(controller) { + controller.enqueue({ type: 'stream-start', warnings: [] }); + controller.enqueue({ + type: 'tool-call', + toolCallId: `call_${crypto.randomUUID()}`, + toolName: TOOL_ID, + input: '{}', + }); + controller.enqueue({ + type: 'finish', + finishReason: 'tool-calls', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + }); + controller.close(); + }, + }), + }), + }; + return { + meta: { + id: AGENT_ID, + title: 'Staging notification agent', + description: 'Produces a selected host-owned notification suspension', + allowedRoles: ['operator'], + allowedAutomation: [ + { kind: 'system', entryPaths: ['notification.dispatch'] }, + ], + }, + agent: createGuardedAgent({ + id: AGENT_ID, + name: 'Staging notification agent', + instructions: 'Call staging_notification once.', + model, + tools: { [TOOL_ID]: tool }, + allowedRoles: ['operator'], + allowedPrincipalKinds: ['system'], + policies: [], + audit, + maxSteps: 1, + toolChoice: 'required', + }), + }; +} + +export class NotificationStagingRunner extends DurableObjectRunner { + protected build(env: Env): RunnerRuntime { + return init(env as never).runtime; + } + + protected runOwnership(env: Env) { + return approvalStoreFactoryFor(env.DB).resources(); + } +} + +function journalFor(env: Env, runtime: ReturnType['runtime']) { + return new RemoteStepJournal({ + db: env.DB as unknown as never, + fence: executionFenceFor(env.DB as unknown as never), + runtime, + deploymentId: env.DEPLOYMENT_TENANT, + tablePrefix: '', + allowedToolIds: [TOOL_ID], + verifyCompletion: async ({ invocation, resultJson, evidenceJson }) => { + if ( + invocation.request === null || + typeof invocation.request !== 'object' || + (invocation.request as { notificationOperationId?: unknown }) + .notificationOperationId !== OPERATION_ID || + resultJson !== RESULT_JSON || + evidenceJson !== EVIDENCE_JSON + ) + throw new Error( + 'staging no-op completion does not match the operation', + ); + }, + }); +} + +function stagingRuntime(env: Env): { + runtime: ReturnType['runtime']; + workflowId: string; +} { + const storage = createD1Storage({ binding: env.DB as unknown as never }); + const app = init( + { storage }, + { + executionFence: executionFenceFor(env.DB as unknown as never), + startIdempotency: startIdempotencyFor(env.DB as unknown as never), + }, + ); + const agents = { + [AGENT_ID]: stagingAgent().agent as unknown as Agent, + } as Record; + const mastra = new Mastra({ storage, agents }); + const durable = createFlowsafeDurableAgent({ + agent: mastra.getAgentById(AGENT_ID), + runtime: app.runtime, + threadRuntime: mastra.agentThreadStreamRuntime, + }); + return { runtime: app.runtime, workflowId: durable.getWorkflow().id }; +} + +export class NotificationStagingThread extends ThreadDurableObject { + #host?: ThreadAgentHost; + + constructor(state: DurableObjectState, env: Env) { + super(state as never, env); + } + + protected build(env: Env): InitResult { + const storage = createD1Storage({ binding: env.DB as unknown as never }); + const app = init( + { storage }, + { + executionFence: executionFenceFor(env.DB as unknown as never), + startIdempotency: startIdempotencyFor(env.DB as unknown as never), + pubsub: createHostPubSub(), + requestContextForRun: (workflowId, runId, leg) => { + if (!this.#host) throw new Error('thread agent host is unavailable'); + return this.#host.requestContextForRun()(workflowId, runId, leg); + }, + }, + ); + const journal = journalFor(env, app.runtime); + this.#host = createThreadAgentHost({ + buildModules: () => [stagingAgent()], + storage: () => storage, + stateStorage: () => { + if (!this.state?.storage) + throw new Error('thread storage is unavailable'); + return this.state.storage as AgentThreadStateStorage; + }, + resourceAccess: () => approvalStoreFactoryFor(env.DB).resources(), + approvalService: () => + new ApprovalService({ + store: approvalStoreFactoryFor(env.DB).store(), + executionFence: executionFenceFor(env.DB as unknown as never), + workflowTablePrefix: '', + }), + ...(env.NOTIFICATION_PROOF_ENABLED === 'true' + ? { + remoteSteps: journal, + remoteStepNotification: { + journal, + agentId: AGENT_ID, + operationId: OPERATION_ID, + source: 'flowsafe-staging', + kind: 'host-owned-notification', + summary: 'Staging notification recovery', + ensureScheduledRecovery: () => + ensureRemoteStepNotificationRecoveryDuty(env), + }, + } + : {}), + }); + return app; + } + + protected async route( + request: Request, + scope: ThreadScope, + ): Promise { + return ( + (await this.#host?.route(request, scope)) ?? + new Response(null, { status: 404 }) + ); + } + + protected async onAlarm( + _env: Env, + threadId: string, + app: InitResult, + deploymentTag?: string, + ): Promise { + if (!this.#host) throw new Error('thread host is unavailable'); + await this.#host.recoverOwnership({ threadId, init: app, deploymentTag }); + } +} + +const config = { + workflows: [], + mutationEpoch: (env: Env) => + mutationEpochFromHeader(env.STAGING_MUTATION_EPOCH), + systemPrincipalId: SYSTEM_ID, + buildVerifier: () => staticTokenVerifier(new Map()), + maintenance: { + sweepIntervalMs: 60_000, + purgeIntervalMs: 60_000, + remoteStepRecoveryIntervalMs: 60_000, + }, + remoteStepRecoveryTick: (env: Env) => async () => { + if (env.NOTIFICATION_PROOF_ENABLED !== 'true') + return { scanned: 0, recovered: 0, unresolved: 0 }; + const journal = journalFor(env, stagingRuntime(env).runtime); + await journal.initialize(); + return createRemoteStepNotificationRecoveryTick({ + journal, + namespace: env.THREAD, + deploymentIdentitySecret: env.DEPLOYMENT_IDENTITY_SECRET, + })(); + }, + preRoutes: async (request: Request, env: Env, _ctx, kit) => { + const url = new URL(request.url); + if (!url.pathname.startsWith('/__staging/')) return null; + if (env.NOTIFICATION_PROOF_ENABLED !== 'true') + return new Response(null, { status: 404 }); + assertDeploymentIdentitySecret( + env.STAGING_PROBE_SECRET, + 'staging probe credential', + ); + const supplied = request.headers.get('x-flowsafe-staging-probe'); + if ( + !supplied || + !(await credentialsMatch(supplied, env.STAGING_PROBE_SECRET)) + ) + return new Response(null, { status: 403 }); + if (request.method === 'GET' && url.pathname === '/__staging/identity') { + if ( + !/^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/.test( + env.CF_VERSION_METADATA?.id, + ) + ) + return new Response(null, { status: 503 }); + return Response.json({ + versionId: env.CF_VERSION_METADATA.id, + deploymentTenant: env.DEPLOYMENT_TENANT, + }); + } + if (request.method === 'POST' && url.pathname === '/__staging/ensure') { + await ensureRemoteStepNotificationRecoveryDuty(env); + return Response.json({ armed: true }); + } + if ( + request.method !== 'POST' || + (url.pathname !== '/__staging/start' && + url.pathname !== '/__staging/complete') + ) + return new Response(null, { status: 404 }); + const { runtime, workflowId } = stagingRuntime(env); + const journal = journalFor(env, runtime); + await journal.initialize(); + const storeFactory = approvalStoreFactoryFor(env.DB); + const context = createPrincipalActorContext({ + principal: { + kind: 'system', + id: env.DEPLOYMENT_TENANT, + purpose: 'private-notification-staging-proof', + }, + storeFactory, + deploymentTag: env.DEPLOYMENT_TENANT, + mutationEpoch: mutationEpochFromHeader(env.STAGING_MUTATION_EPOCH), + buildService: (store) => + new ApprovalService({ + store, + executionFence: executionFenceFor(env.DB as unknown as never), + workflowTablePrefix: '', + }), + }); + const topology = createAgentThreadTopology( + env.THREAD, + env.DEPLOYMENT_IDENTITY_SECRET, + { + startIdempotency: startIdempotencyFor(env.DB as unknown as never), + executionFence: executionFenceFor(env.DB as unknown as never), + remoteStepDispatch: journal, + remoteSteps: journal, + }, + ); + if (url.pathname === '/__staging/start') { + const threadId = context.newThreadId(); + const resourceId = context.resourceIdFromKey(threadId); + await context.claimResource('thread', threadId); + await context.claimResource('resource', resourceId); + const started = await topology.start(context, { + agentId: AGENT_ID, + prompt: 'Produce the host-owned staging notification.', + entryPath: 'notification.dispatch', + threadId, + resourceId, + threaded: true, + }); + if (started.summary.status !== 'suspended') + throw new Error('staging agent did not suspend'); + const invocation = await journal.register( + { + deploymentId: env.DEPLOYMENT_TENANT, + tablePrefix: '', + workflowId, + agentId: started.agentId, + threadId: started.threadId, + resourceId: started.resourceId, + runId: started.runId, + }, + Date.now() + 120_000, + context.principal, + ); + return Response.json({ + invocationId: invocation.invocationId, + runId: started.runId, + threadId: started.threadId, + status: started.summary.status, + }); + } + const body = (await request.json()) as { invocationId?: unknown }; + if (typeof body.invocationId !== 'string') + return new Response(null, { status: 400 }); + const stored = await kit.contextForStoredInvocation( + journal, + body.invocationId, + ); + if ( + stored?.principal.kind !== 'system' || + stored.principal.id !== context.principal.id + ) + return new Response(null, { status: 404 }); + const ticket = await topology.issueRemoteStepDispatch( + stored, + body.invocationId, + 30_000, + ); + const consumed = await topology.consumeRemoteStepDispatch( + stored, + ticket.permit, + ); + if (consumed.kind !== 'accepted') + throw new Error('staging no-op dispatch was already consumed'); + await journal.recordResult( + body.invocationId, + consumed.rawPermit, + RESULT_JSON, + EVIDENCE_JSON, + ); + const resumed = await topology.resumeRemoteStep(stored, body.invocationId); + return Response.json({ + invocationId: body.invocationId, + status: resumed.summary.status, + }); + }, +} satisfies FlowsafeWorkerConfig; + +export class NotificationStagingMaintenance extends createFlowsafeMaintenanceDurableObject( + config, +) {} + +const worker = createFlowsafeWorker(config); + +export default { + async fetch(request: CfRequest, env: Env, ctx: ExecutionContext) { + try { + await ensureDeploymentIdentityBindings(env); + return await worker.fetch(request as unknown as Request, env, ctx); + } catch (error) { + return doErrorResponse(error); + } + }, +}; diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc new file mode 100644 index 00000000..4996282c --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -0,0 +1,50 @@ +{ + "$schema": "https://unpkg.com/wrangler/config-schema.json", + "name": "anchorage-flowsafe-remote-step-staging", + "main": "./notification-staging.worker.ts", + "minify": true, + "compatibility_date": "2026-09-10", + "compatibility_flags": ["nodejs_compat"], + "alias": { + "@proofoftech/breakwater/agent": "../../breakwater/src/agent/index.ts", + "cloudflare": "./notification-staging-no-rest.ts" + }, + "workers_dev": false, + "version_metadata": { "binding": "CF_VERSION_METADATA" }, + "secrets": { + "required": [ + "DEPLOYMENT_IDENTITY_SECRET", + "MAINTENANCE_ADMIN_SECRET", + "STAGING_PROBE_SECRET" + ] + }, + "durable_objects": { + "bindings": [ + { "name": "RUNNER", "class_name": "NotificationStagingRunner" }, + { "name": "THREAD", "class_name": "NotificationStagingThread" }, + { "name": "MAINTENANCE", "class_name": "NotificationStagingMaintenance" } + ] + }, + "migrations": [ + { + "tag": "notification-staging-v1", + "new_sqlite_classes": [ + "NotificationStagingRunner", + "NotificationStagingThread", + "NotificationStagingMaintenance" + ] + } + ], + "d1_databases": [ + { + "binding": "DB", + "database_name": "anchorage-flowsafe-remote-step-staging-db", + "database_id": "b863a219-2096-4e13-92fe-4db592b67944" + } + ], + "vars": { + "DEPLOYMENT_TENANT": "flowsafenotificationstaging", + "STAGING_MUTATION_EPOCH": "0", + "NOTIFICATION_PROOF_ENABLED": "false" + } +} diff --git a/packages/flowsafe/deploy/tsconfig.json b/packages/flowsafe/deploy/tsconfig.json index 4263bb90..aa6aaf52 100644 --- a/packages/flowsafe/deploy/tsconfig.json +++ b/packages/flowsafe/deploy/tsconfig.json @@ -17,6 +17,8 @@ }, "include": [ "./worker.ts", + "./notification-staging.worker.ts", + "./notification-staging-probe.worker.ts", "../src/**/*.ts", "../test-support/maintenance-alarm-harness-worker.ts" ], diff --git a/packages/flowsafe/package.json b/packages/flowsafe/package.json index 5373f10d..eea6389c 100644 --- a/packages/flowsafe/package.json +++ b/packages/flowsafe/package.json @@ -1,6 +1,6 @@ { "name": "@proofoftech/flowsafe", - "version": "0.28.0", + "version": "0.29.0", "publishConfig": { "access": "public" }, diff --git a/packages/flowsafe/scripts/agent-host-pack-test.mjs b/packages/flowsafe/scripts/agent-host-pack-test.mjs index 7008df81..c4d7dd7c 100644 --- a/packages/flowsafe/scripts/agent-host-pack-test.mjs +++ b/packages/flowsafe/scripts/agent-host-pack-test.mjs @@ -717,7 +717,7 @@ import { type SignalDatabase, type SignalRouterOptions, } from '@proofoftech/flowsafe/signals'; import { - type RunnerRuntime, type RunExecutionIdentity, type StartRunOptions, + type RunnerRuntime, type RunExecutionIdentity, type StartRunOptions, type RemoteStepJournal, type ThreadScope, type StartReservationReading, type PersistedStartResult, type RunSummary, } from '@proofoftech/flowsafe/do-runner'; @@ -729,6 +729,9 @@ import { import { type AgentStartAuthority, type FlowsafeDurableAgent, } from '@proofoftech/flowsafe/agent-runner'; +import { + createWorkflowRemoteStepPreRoute, type AgentThreadDispatchTopology, +} from '@proofoftech/flowsafe/agent-host'; // @ts-expect-error internal context capture is not a root export import type { captureActorContext as RootCapture } from '@proofoftech/flowsafe'; // @ts-expect-error internal context capture is not an approval export @@ -839,6 +842,9 @@ const workerConfig: FlowsafeWorkerConfig = { createFlowsafeWorker(workerConfig); createFlowsafeWorker({ ...workerConfig, mutationEpoch: 0 }); createFlowsafeWorker({ ...workerConfig, beforeStart: legacyWorkerHook }); +declare const remoteStepJournal: RemoteStepJournal; +declare const remoteStepDispatch: AgentThreadDispatchTopology; +createFlowsafeWorker({ ...workerConfig, preRoutes: createWorkflowRemoteStepPreRoute(remoteStepJournal, remoteStepDispatch) }); void [contextInput, doContextInput, legacyDoInput, legacyRouterHook, routerPolicyResult, workerPolicyResult]; const onPrepared = (execution: RunExecutionIdentity): void => { void execution.startToken; }; const legacyOptions: StartRunOptions = { runId: 'legacy' }; @@ -1369,6 +1375,18 @@ const repair = await capability.withInitialAdmission({ execution: repairExecutio }, () => workflow.createRun({ runId: repairExecution.runId })); const repairRequest = { expected: repair.witness.row, execution: repairExecution, attemptToken: 'repair-correlation', nowMs: 1700000000123 }; const observer = doRunner.init({ storage }, { executionFence: new doRunner.ExecutionFenceStore(binding), startIdempotency: 'none' }).runtime; +const remoteStepBinding = sqliteUnitDatabase(openSqlite()); +const remoteStepJournal = new doRunner.RemoteStepJournal({ + db: remoteStepBinding, fence: new doRunner.ExecutionFenceStore(remoteStepBinding), runtime: observer, + deploymentId: 'packed-remote-step', tablePrefix: '', allowedToolIds: ['remote-tool'], + verifyCompletion: async () => { throw new Error('no remote completion is configured'); }, +}); +await remoteStepJournal.initialize(); +assert.equal(await remoteStepJournal.getWorkflow('absent-invocation'), undefined); +const privateRemoteRoute = host.createWorkflowRemoteStepPreRoute(remoteStepJournal, {}); +const privateRemoteMethod = await privateRemoteRoute(new Request('https://packed.test/_flowsafe/workflow-remote-step/ticket'), {}, {}, {}); +assert.equal(privateRemoteMethod.status, 405); +assert.deepEqual(await privateRemoteMethod.json(), { error: 'method not allowed' }); observer.register(workflow); const initial = await observer.authoritativeStartState(workflow.id, repairExecution.runId); assert.equal(initial.kind, 'initial'); diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index 1eaeb9ec..4c73f068 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -111,6 +111,7 @@ const AUTH = { const STREAM_TICKET_SECRET = 'spike-local-stream-secret-do-not-deploy'; const DEPLOYMENT_TENANT = 'spike'; const DEPLOYMENT_IDENTITY_SECRET = 'spike-local-deployment-identity-secret'; +const PRIVATE_REMOTE_SECRET = 'spike-local-private-admin-secret-2026'; const nowSec = () => Math.floor(Date.now() / 1000); // Track E (M-007): the github webhook signing secret. MUST equal @@ -168,6 +169,8 @@ function startServer(generation, stateDir, logPath) { `DEPLOYMENT_TENANT:${DEPLOYMENT_TENANT}`, '--var', `DEPLOYMENT_IDENTITY_SECRET:${DEPLOYMENT_IDENTITY_SECRET}`, + '--var', + `MAINTENANCE_ADMIN_SECRET:${PRIVATE_REMOTE_SECRET}`, // Sign stream tickets with the SAME key spike-verify forges with, so the // ticket fail-closed probes exercise CLAIM rejection, not signature drift. '--var', @@ -176,6 +179,11 @@ function startServer(generation, stateDir, logPath) { // with, so a valid webhook is ACCEPTED and a forged one REJECTED (E-S2). '--var', `GITHUB_WEBHOOK_SECRET:${GITHUB_WEBHOOK_SECRET}`, + ...(generation.startsWith('gen-remote-') || + generation.startsWith('gen-workflow-remote-') || + generation.startsWith('gen-private-remote-') + ? ['--var', 'SPIKE_REMOTE_STEP_PROOF:enabled'] + : []), ], { cwd: FLOWSAFE, @@ -437,7 +445,7 @@ async function readInventoryIndex() { } // One category, PAGED TO EXHAUSTION through the inventory's own cursors — the -// sweep an operator's drain proof is actually made of, not a single page that +// sweep an operator's drain proof needs, not a single page that // could hide the rows behind it. `limit: 3` forces several continuations even // on the spike's small data, so the keyset is exercised rather than skipped. async function sweepInventoryCategory(category) { @@ -466,7 +474,7 @@ async function sweepInventoryCategory(category) { throw new Error(`inventory paging did not terminate for ${category}`); } -// Every WORK category, swept. The drain proof is defined over exactly these. +// Every WORK category, swept for the seeded-row visibility check. async function sweepInventoryWork(index) { const work = index.categories.filter((entry) => entry.class === 'work'); const swept = {}; @@ -766,6 +774,18 @@ async function main() { launchServer('gen-1', stateDir, join(tmpDir, 'gen1.log')), ); + await step( + 'RS0 remote-step fixture is absent unless explicitly enabled', + async () => { + const absent = await http('GET', '/spike/remote-step/scan'); + assert( + absent.status === 404, + 'remote-step fixture stays unmounted', + absent, + ); + }, + ); + await step( 'PA0 boundary: authenticated catalog, role gates, and forged context/header rejection', async () => { @@ -3183,13 +3203,13 @@ async function main() { }, ); - // --- Drain inventory (F2): the migration proof, on real workerd ----------- + // --- Drain inventory (F2): seeded-row visibility on real workerd ---------- // Unit tests can prove the queries are pure SELECTs against the real schemas. // What they cannot prove is that those queries read the SAME D1 that real // HTTP traffic on real workerd wrote its runs, approvals, reservations, // notifications, and tasks into — and that the numbers MOVE as that work // drains. An inventory that reported the right shape over the wrong database - // would pass every unit test and certify a deployment that still owed work. + // would pass every unit test while missing this work. await step( 'FV1 inventory index: every category, its class, the states no query can ' + @@ -3203,7 +3223,9 @@ async function main() { .filter((entry) => entry.class === 'standing') .map((entry) => entry.category); assert( - work.length === 7 && standing.length === 2, + work.length === 8 && + work.includes('remote-steps') && + standing.length === 2, 'the index splits work from standing configuration', { work, standing }, ); @@ -3225,16 +3247,13 @@ async function main() { ); await step( - 'FV2 drain proof: seeded work appears while draining, leaves as it is ' + - 'finished, and reads empty across TWO consecutive sweeps before the lock', + 'FV2 seeded inventory: work appears while draining, leaves as it is ' + + 'finished, and stays absent under the lock', async () => { const index = await readInventoryIndex(); - // A BASELINE, because this deployment has been running scenarios for the - // whole spike and legitimately still holds work from them. The proof - // below is over the seeded DELTA: these exact rows appear, and these - // exact rows are gone — which is the same claim an empty sweep makes, - // stated about rows whose lifecycle this step controls. + // A baseline accounts for work left by earlier spike scenarios. This + // check follows the seeded rows; it does not prove that all work drained. const before = await sweepInventoryWork(index); // Seed: a suspended run with a pending approval, and a keyed start @@ -3322,9 +3341,8 @@ async function main() { keyedDecided, ); - // The proof: the seeded rows are gone, and STILL gone on a second full - // sweep. One sweep is a point-in-time reading: traffic can still create - // or complete work while it runs, so the contract asks for two. + // The seeded rows are gone on repeated reads. A migration drain proof + // also requires an empty work inventory across the recovery interval. const gone = (sweep) => !sweep.runs.keys.some((key) => key.endsWith(`/${drained.runId}`)) && !sweep.runs.keys.some((key) => key.endsWith(`/${keyed.body.runId}`)) && @@ -3340,7 +3358,7 @@ async function main() { const second = await sweepInventoryWork(index); assert( gone(second), - 'the second consecutive sweep agrees — the drain proof holds', + 'the repeated sweep no longer reports the seeded work', { runs: second.runs.keys, approvals: second['approvals-waiting'].keys, @@ -3357,18 +3375,16 @@ async function main() { ); assert( schedules.class === 'standing' && schedules.keys.length >= 1, - 'schedules are still reported while the drain proof passes', + 'schedules are still reported while seeded work clears', schedules, ); assert( subscriptions.class === 'standing' && subscriptions.keys.length >= 1, - 'provider subscriptions are still reported while the proof passes', + 'provider subscriptions are still reported while seeded work clears', subscriptions, ); - // Lock: the proof held, so the deployment may stop executing. The - // inventory keeps answering — reads are ungated in every state, which is - // what lets an operator verify the lock they just took. + // The lock keeps inventory reads available while refusing new execution. await moveFence('draining', 'migration-locked'); const locked = await sweepInventoryWork(index); assert( @@ -3381,7 +3397,7 @@ async function main() { headers: AUTH.operator, }); assertFenced( - 'a run start after the drain proof locked the deployment', + 'a run start under the migration lock', refusedStart, 'migration-locked', ); @@ -3389,6 +3405,441 @@ async function main() { }, ); + await step( + 'RS1 remote-step journal: real D1 permit winner and retained result survive process death', + async () => { + await killServer(currentServer); + // The fixture status is stale so a summary detached from the persisted + // snapshot cannot register the still-suspended run. + await executeLocalD1( + stateDir, + 'CREATE TABLE spike_remote_step_fixture (id INTEGER PRIMARY KEY, resource_id TEXT NOT NULL, status TEXT NOT NULL); ' + + "INSERT INTO spike_remote_step_fixture VALUES (1, 'account-7', 'success'); " + + 'CREATE TABLE spike_remote_mastra_workflow_snapshot (workflow_name TEXT NOT NULL, run_id TEXT NOT NULL, snapshot TEXT NOT NULL, PRIMARY KEY (workflow_name, run_id)); ' + + "INSERT INTO spike_remote_mastra_workflow_snapshot VALUES ('durable-agentic-loop', 'spike-remote-run', '{\"status\":\"suspended\"}');", + ); + await launchServer( + 'gen-remote-1', + stateDir, + join(tmpDir, 'gen-remote-1.log'), + ); + const wrong = await http('POST', '/spike/remote-step/register', { + body: { resourceId: 'spike-remote-thread' }, + }); + assert( + wrong.status === 409, + 'the thread id cannot stand in for the mapped resource', + wrong, + ); + const registered = await http('POST', '/spike/remote-step/register', { + body: { resourceId: 'account-7' }, + }); + assert( + registered.status === 200 && + typeof registered.body.invocationId === 'string', + 'the mapped-resource invocation was registered', + registered, + ); + const invocationId = registered.body.invocationId; + const issued = await http('POST', '/spike/remote-step/issue', { + body: { invocationId }, + }); + assert( + issued.status === 200 && issued.body.generation === 1, + 'the dispatch permit was issued', + issued, + ); + const attempts = await Promise.all( + [0, 1].map(() => + http('POST', '/spike/remote-step/consume', { + body: { permit: issued.body }, + }), + ), + ); + assert( + attempts.filter((attempt) => attempt.body.kind === 'accepted') + .length === 1 && + attempts.filter((attempt) => attempt.body.kind === 'already-consumed') + .length === 1, + 'concurrent permit consumption has one winner', + attempts, + ); + const rawPermit = attempts.find( + (attempt) => attempt.body.kind === 'accepted', + ).body.rawPermit; + const recorded = await http('POST', '/spike/remote-step/record', { + body: { invocationId, rawPermit }, + }); + assert( + recorded.status === 200 && recorded.body.kind === 'recorded', + 'the result was retained', + recorded, + ); + await killServer(currentServer); + await executeLocalD1( + stateDir, + "UPDATE spike_remote_step_fixture SET resource_id = 'account-8' WHERE id = 1;", + ); + await launchServer( + 'gen-remote-2', + stateDir, + join(tmpDir, 'gen-remote-2.log'), + ); + const retained = await http( + 'GET', + `/spike/remote-step/read?invocationId=${invocationId}`, + ); + assert( + retained.status === 200 && + retained.body.invocation.status === 'result-retained' && + retained.body.result.resultJson === '{"answer":1}', + 'fresh workerd process reads the persisted result', + retained, + ); + const permitReplay = await http('POST', '/spike/remote-step/consume', { + body: { permit: issued.body }, + }); + assert( + permitReplay.status === 200 && + permitReplay.body.kind === 'already-consumed', + 'fresh process refuses to grant the consumed permit again', + permitReplay, + ); + const resultReplay = await http('POST', '/spike/remote-step/record', { + body: { invocationId, rawPermit }, + }); + assert( + resultReplay.status === 200 && resultReplay.body.kind === 'replay', + 'fresh process reads the retained result without a second effect', + resultReplay, + ); + const displaced = await http('POST', '/spike/remote-step/claim', { + body: { invocationId }, + }); + assert( + displaced.status === 409, + 'changed mapped-resource provenance refuses a resume claim', + displaced, + ); + await killServer(currentServer); + await executeLocalD1( + stateDir, + "UPDATE spike_remote_step_fixture SET resource_id = 'account-7' WHERE id = 1;", + ); + await launchServer( + 'gen-remote-3', + stateDir, + join(tmpDir, 'gen-remote-3.log'), + ); + const scan = await http('GET', '/spike/remote-step/scan'); + assert( + scan.status === 200 && scan.body.invocationIds.includes(invocationId), + 'fresh process recovery scan finds the retained result', + scan, + ); + const claimed = await http('POST', '/spike/remote-step/claim', { + body: { invocationId }, + }); + assert( + claimed.status === 200 && typeof claimed.body.owner === 'string', + 'one resume claim was persisted', + claimed, + ); + await killServer(currentServer); + await executeLocalD1( + stateDir, + 'UPDATE spike_remote_mastra_workflow_snapshot SET snapshot = \'{"status":"success"}\' WHERE run_id = \'spike-remote-run\';', + ); + await launchServer( + 'gen-remote-4', + stateDir, + join(tmpDir, 'gen-remote-4.log'), + ); + const ambiguous = await http('POST', '/spike/remote-step/reconcile', { + body: { invocationId }, + }); + assert( + ambiguous.status === 200 && + ambiguous.body.outcome === 'pending-quiescence', + 'fresh process does not replay an ambiguous resume', + ambiguous, + ); + const settled = await http('POST', '/spike/remote-step/settle', { + body: { invocationId, owner: claimed.body.owner }, + }); + assert( + settled.status === 200 && + settled.body.outcome === 'advanced-without-proof', + 'advanced snapshot without a returned witness remains unresolved', + settled, + ); + const final = await http( + 'GET', + `/spike/remote-step/read?invocationId=${invocationId}`, + ); + assert( + final.status === 200 && + final.body.invocation.status === 'failed' && + final.body.invocation.settlementOutcome === 'advanced-without-proof', + 'the unresolved outcome remains durable', + final, + ); + const drain = await sweepInventoryCategory('remote-steps'); + assert( + drain.class === 'work' && drain.keys.includes(invocationId), + 'the unresolved result remains in operator drain inventory', + drain, + ); + }, + ); + + await step( + 'RS2 workflow-target journal: D1 permit and retained result survive process death without resume', + async () => { + await killServer(currentServer); + await executeLocalD1( + stateDir, + "INSERT INTO spike_remote_mastra_workflow_snapshot VALUES ('durable-agentic-loop', 'spike-workflow-run', '{\"status\":\"suspended\"}');", + ); + await launchServer( + 'gen-workflow-remote-1', + stateDir, + join(tmpDir, 'gen-workflow-remote-1.log'), + ); + const registered = await http( + 'POST', + '/spike/remote-step/workflow-register', + { + body: {}, + }, + ); + assert( + registered.status === 200 && + registered.body.targetKind === 'workflow' && + typeof registered.body.invocationId === 'string', + 'trusted workflow provenance registered a workflow row', + registered, + ); + const invocationId = registered.body.invocationId; + const issued = await http('POST', '/spike/remote-step/workflow-issue', { + body: { invocationId }, + }); + assert( + issued.status === 200 && issued.body.generation === 1, + 'workflow dispatch permit was issued', + issued, + ); + const attempts = await Promise.all( + [0, 1].map(() => + http('POST', '/spike/remote-step/workflow-consume', { + body: { permit: issued.body }, + }), + ), + ); + assert( + attempts.filter((attempt) => attempt.body.kind === 'accepted') + .length === 1 && + attempts.filter((attempt) => attempt.body.kind === 'already-consumed') + .length === 1, + 'workflow permit consumption has one winner', + attempts, + ); + const rawPermit = attempts.find( + (attempt) => attempt.body.kind === 'accepted', + ).body.rawPermit; + const recorded = await http( + 'POST', + '/spike/remote-step/workflow-record', + { + body: { invocationId, rawPermit }, + }, + ); + assert( + recorded.status === 200 && recorded.body.kind === 'recorded', + 'workflow result was retained', + recorded, + ); + await killServer(currentServer); + await launchServer( + 'gen-workflow-remote-2', + stateDir, + join(tmpDir, 'gen-workflow-remote-2.log'), + ); + const retained = await http( + 'GET', + `/spike/remote-step/workflow-read?invocationId=${invocationId}`, + ); + assert( + retained.status === 200 && + retained.body.invocation.status === 'result-retained' && + retained.body.result.resultJson === '{"answer":1}', + 'fresh workerd process reads the workflow result', + retained, + ); + const replay = await http('POST', '/spike/remote-step/workflow-consume', { + body: { permit: issued.body }, + }); + assert( + replay.status === 200 && replay.body.kind === 'already-consumed', + 'fresh process does not grant workflow effect authority twice', + replay, + ); + const claim = await http('POST', '/spike/remote-step/workflow-claim', { + body: { invocationId }, + }); + assert( + claim.status === 404, + 'workflow continuation remains unavailable', + claim, + ); + }, + ); + + await step( + 'RS3 private workflow route: trusted start, one permit winner, retained result and restart refusal', + async () => { + await killServer(currentServer); + await launchServer( + 'gen-private-remote-1', + stateDir, + join(tmpDir, 'gen-private-remote-1.log'), + ); + const route = '/_flowsafe/workflow-remote-step'; + const privateHeaders = { + authorization: `Bearer ${PRIVATE_REMOTE_SECRET}`, + }; + const absent = await http('POST', `${route}/register`, { + body: { + workflowId: 'spike-private-remote', + runId: 'unknown-run', + hostDeadlineAt: Date.now() + 120_000, + }, + }); + assert( + absent.status === 401, + 'missing private credential is refused', + absent, + ); + const wrong = await http('POST', `${route}/register`, { + headers: AUTH.admin, + body: { + workflowId: 'spike-private-remote', + runId: 'unknown-run', + hostDeadlineAt: Date.now() + 120_000, + }, + }); + assert( + wrong.status === 401, + 'ordinary admin bearer cannot delegate private authority', + wrong, + ); + const started = await http('POST', '/runs', { + headers: AUTH.operator, + body: { + workflowId: 'spike-private-remote', + inputData: { query: 'x' }, + }, + }); + assert( + started.status === 200 && + started.body.status === 'suspended' && + typeof started.body.runId === 'string', + 'real workflow suspends under its Durable Object', + started, + ); + const runId = started.body.runId; + const registered = await http('POST', `${route}/register`, { + headers: privateHeaders, + body: { + workflowId: 'spike-private-remote', + runId, + hostDeadlineAt: Date.now() + 120_000, + }, + }); + assert( + registered.status === 200 && + typeof registered.body.invocationId === 'string', + 'verified stored start registers the workflow invocation', + registered, + ); + const invocationId = registered.body.invocationId; + const issued = await http('POST', `${route}/ticket`, { + headers: privateHeaders, + body: { invocationId, lifetimeMs: 60_000 }, + }); + assert( + issued.status === 200 && + issued.body.runId === runId && + issued.body.requestJson === '{"version":2,"input":{"query":"x"}}' && + issued.body.permit?.generation === 1, + 'private ticket binds the real suspension and dispatch request', + issued, + ); + const attempts = await Promise.all( + [0, 1].map(() => + http('POST', `${route}/consume`, { + headers: privateHeaders, + body: { permit: issued.body.permit }, + }), + ), + ); + assert( + attempts.filter((attempt) => attempt.body.kind === 'accepted') + .length === 1 && + attempts.filter((attempt) => attempt.body.kind === 'already-consumed') + .length === 1, + 'private workflow permit has one consumer', + attempts, + ); + const rawPermit = attempts.find( + (attempt) => attempt.body.kind === 'accepted', + ).body.rawPermit; + const recorded = await http('POST', `${route}/result`, { + headers: privateHeaders, + body: { + invocationId, + rawPermit, + resultJson: '{"answer":1}', + evidenceJson: '{"receipt":"complete"}', + }, + }); + assert( + recorded.status === 200 && recorded.body.kind === 'recorded', + 'private workflow result is retained', + recorded, + ); + await killServer(currentServer); + await launchServer( + 'gen-private-remote-2', + stateDir, + join(tmpDir, 'gen-private-remote-2.log'), + ); + const replay = await http('POST', `${route}/consume`, { + headers: privateHeaders, + body: { permit: issued.body.permit }, + }); + assert( + replay.status === 200 && replay.body.kind === 'already-consumed', + 'fresh process refuses a second effect authority', + replay, + ); + const resultReplay = await http('POST', `${route}/result`, { + headers: privateHeaders, + body: { + invocationId, + rawPermit, + resultJson: '{"answer":1}', + evidenceJson: '{"receipt":"complete"}', + }, + }); + assert( + resultReplay.status === 200 && resultReplay.body.kind === 'replay', + 'fresh process returns the retained result on replay', + resultReplay, + ); + }, + ); + await step( 'S deployment sentinel mismatch: a freshly started Worker refuses a D1 ' + 'provisioned for another deployment', @@ -3484,11 +3935,15 @@ try { 'class, the Durable Object journal window it cannot see, and the rule an ' + 'empty answer means something under; seeded runs, an approval, and a ' + 'start reservation appeared as outstanding work while draining, left as ' + - 'each was finished, and read absent across TWO consecutive full sweeps ' + - 'paged through the keyset — after which the deployment locked, still ' + + 'each was finished, and stayed absent on repeated paged reads ' + + 'through the keyset — after which the deployment locked, still ' + 'answered the same inventory, and refused to mint, while its schedules ' + 'and provider subscriptions kept being reported as standing ' + 'configuration a migration carries rather than drains. ' + + 'Remote-step journal (RS1): a single permit winner retained a result ' + + 'across process death, changed mapped-resource provenance refused a ' + + 'resume claim, and an advanced snapshot without a returned witness ' + + 'remained in operator drain inventory after another restart. ' + 'Finally, a fresh Worker refused a D1 sentinel stamped for ' + 'another deployment with 503 before authentication or routing.', ); diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index 64bfed83..fbc15e12 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -87,7 +87,9 @@ import { createAgentRouter, createAgentThreadTopology, createThreadAgentHost, + createWorkflowRemoteStepPreRoute, type ThreadAgentHost, + WORKFLOW_REMOTE_STEP_ROUTE, } from '../src/agent-host/index.js'; import { readAgentThreadBinding } from '../src/agent-runner/index.js'; import { @@ -139,6 +141,7 @@ import { isSuspensionTimeoutResumeData, mintThreadId, type RunnerRuntime, + type RunSummary, resourceIdFromKey, type StartIdempotencyStore, SUSPENSION_DEADLINE_PAYLOAD_KEY, @@ -150,6 +153,12 @@ import { verifyDurableObjectDeploymentIdentity, verifyDurableObjectDeploymentRequest, } from '../src/do-runner/index.js'; +import { + REMOTE_STEP_MARKER, + RemoteStepJournal, + type RemoteStepPermit, + WORKFLOW_REMOTE_STEP_MARKER, +} from '../src/do-runner/remote-step-journal.js'; import { createObjectiveRouter, type ObjectiveAuditEvent, @@ -158,11 +167,14 @@ import { abandonApprovalsForRun, bearerActorAuthenticator, createDoRunTopology, + createFlowsafeMaintenanceDurableObject, + createFlowsafeWorker, createHubTopology, createRunRouter, createStreamRouter, createThreadTopology, doSummary, + type FlowsafeWorkerConfig, RunRouteError, staticTokenVerifier, type WorkflowMeta, @@ -208,6 +220,9 @@ interface Env { /** Singleton deployment signal-provider host DO; see DemoSignalProviderHost (Track E). */ SIGNAL_PROVIDER_HOST: DurableObjectNamespace; BACKGROUND_TASKS: DurableObjectNamespace; + MAINTENANCE: DurableObjectNamespace; + MAINTENANCE_ADMIN_SECRET: string; + SPIKE_REMOTE_STEP_PROOF?: string; SPIKE_LLM_MODEL_ID?: string; SPIKE_LLM_API_KEY?: string; SPIKE_LLM_BASE_URL?: string; @@ -661,6 +676,12 @@ async function countExecution(db: D1Database, counterId: string) { } const WORKFLOWS: ReadonlyArray = [ + { + id: 'spike-private-remote', + title: 'Private remote-step proof', + description: 'Suspends a real D1-backed workflow for private dispatch', + sampleInput: { query: 'x' }, + }, { id: 'demo-approval', title: 'Demo approval', @@ -851,6 +872,28 @@ function defineWorkflows(env: Env): RunnerRuntime { .then(publish) .commit(); + createWorkflow({ + id: 'spike-private-remote', + inputSchema: z.object({ query: z.string() }), + outputSchema: z.object({ query: z.string() }), + }) + .then( + createStep({ + id: 'execute', + inputSchema: z.object({ query: z.string() }), + outputSchema: z.object({ query: z.string() }), + execute: async ({ inputData, suspend }) => + suspend({ + type: 'suspension', + remoteStep: { + marker: WORKFLOW_REMOTE_STEP_MARKER, + request: { version: 2, input: { query: inputData.query } }, + }, + }), + }), + ) + .commit(); + // FI1/FI2: demo-approval's shape with a COUNTING first step. The step is // ordinary paid work as far as the runtime is concerned — it writes to D1 // before the run suspends — so a second execution of this key would show up @@ -3073,6 +3116,245 @@ async function handleInventoryProbe( } } +// Local workerd fixture: the journal reads a persisted suspension and start +// identity without entering Mastra. It proves D1 recovery, not agent replay. +async function handleRemoteStepProbe( + request: Request, + env: Env, +): Promise { + const path = new URL(request.url).pathname; + if (!path.startsWith('/spike/remote-step/')) return null; + if (env.SPIKE_REMOTE_STEP_PROOF !== 'enabled') + return json({ error: 'not found' }, 404); + const operation = path.slice('/spike/remote-step/'.length); + const workflow = operation.startsWith('workflow-'); + const action = workflow ? operation.slice('workflow-'.length) : operation; + const runId = workflow ? 'spike-workflow-run' : 'spike-remote-run'; + if ( + request.method !== (action === 'read' || action === 'scan' ? 'GET' : 'POST') + ) + return json({ error: 'method not allowed' }, 405); + try { + const fixture = await env.DB.prepare( + 'SELECT resource_id FROM spike_remote_step_fixture WHERE id = 1', + ).first<{ resource_id: string }>(); + const snapshot = await env.DB.prepare( + 'SELECT snapshot FROM spike_remote_mastra_workflow_snapshot WHERE workflow_name = ? AND run_id = ?', + ) + .bind('durable-agentic-loop', runId) + .first<{ snapshot: string }>(); + if (!fixture || !snapshot) + throw new Error('remote-step fixture is unavailable'); + const persisted = JSON.parse(snapshot.snapshot) as { status?: unknown }; + if (persisted.status !== 'suspended' && persisted.status !== 'success') + throw new Error('remote-step snapshot status is unavailable'); + const summary: RunSummary = + persisted.status === 'success' + ? { + runId, + status: 'success', + result: { answer: 1 }, + } + : workflow + ? { + runId, + status: 'suspended', + suspended: [['execute']], + suspendedAt: { execute: 1000 }, + suspendPayload: { + execute: { + type: 'suspension', + remoteStep: { + marker: WORKFLOW_REMOTE_STEP_MARKER, + request: { version: 2, input: { query: 'x' } }, + }, + }, + }, + } + : { + runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 1000 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }; + const runtime = { + authoritativeStatus: async () => summary, + authoritativeStartState: async () => ({ + storage: 'd1', + snapshot: persisted, + execution: { + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + runId, + startToken: 'spike-start-token', + }, + provenance: { + startIdentity: { + owner: { kind: 'human', id: 'operator-1' }, + target: workflow + ? { kind: 'workflow', id: 'durable-agentic-loop' } + : { + kind: 'agent', + id: 'writer', + threadId: 'spike-remote-thread', + resourceId: fixture.resource_id, + }, + }, + }, + }), + } as unknown as Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + const journal = new RemoteStepJournal({ + db: env.DB as unknown as never, + fence: executionFenceForEnv(env), + runtime, + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + ...(workflow ? { verifyWorkflowCompletion: async () => undefined } : {}), + }); + await journal.initialize(); + const body = + request.method === 'POST' + ? ((await request.json()) as Record) + : {}; + const invocationId = + request.method === 'GET' + ? (new URL(request.url).searchParams.get('invocationId') ?? '') + : String(body.invocationId ?? ''); + switch (action) { + case 'register': + return json( + await (workflow + ? journal.registerWorkflow( + { + targetKind: 'workflow', + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + runId, + }, + Date.now() + 120_000, + { kind: 'human', id: 'operator-1', role: 'operator' }, + ) + : journal.register( + { + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'spike-remote-thread', + resourceId: String(body.resourceId ?? ''), + runId, + }, + Date.now() + 120_000, + { kind: 'human', id: 'operator-1', role: 'operator' }, + )), + ); + case 'issue': + return json(await journal.issuePermit(invocationId, 60_000)); + case 'consume': + return json( + await journal.consumePermit(body.permit as RemoteStepPermit), + ); + case 'record': + return json( + await journal.recordResult( + invocationId, + String(body.rawPermit), + '{"answer":1}', + '{"receipt":"complete"}', + ), + ); + case 'read': + return json({ + invocation: workflow + ? await journal.getWorkflow(invocationId) + : await journal.get(invocationId), + result: await journal.readRetainedResult(invocationId), + }); + case 'scan': + return json(await journal.recoverableResults(10)); + case 'claim': + return json({ owner: await journal.claimResume(invocationId, 60_000) }); + case 'reconcile': + return json({ outcome: await journal.reconcileResult(invocationId) }); + case 'settle': + return json({ + outcome: await journal.settleAfterResume( + invocationId, + String(body.owner), + ), + }); + default: + return json({ error: 'not found' }, 404); + } + } catch (error) { + return doErrorResponse(error); + } +} + +const privateRemoteConfig: FlowsafeWorkerConfig = { + systemPrincipalId: SYSTEM_PRINCIPAL_ID, + workflows: WORKFLOWS, + maintenance: { sweepIntervalMs: 60_000, purgeIntervalMs: 60_000 }, + buildVerifier: () => staticTokenVerifier(SPIKE_ACTORS), + preRoutes: async (request, env, ctx, kit) => { + if ( + !new URL(request.url).pathname.startsWith( + `${WORKFLOW_REMOTE_STEP_ROUTE}/`, + ) + ) + return null; + const unauthorized = await kit.authorizePrivateWorkflowDispatch(request); + if (unauthorized) return unauthorized; + const journal = new RemoteStepJournal({ + db: env.DB as unknown as never, + fence: executionFenceForEnv(env), + runtime: defineWorkflows(env), + deploymentId: env.DEPLOYMENT_TENANT, + tablePrefix: '', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + verifyWorkflowCompletion: async () => undefined, + }); + await journal.initialize(); + const dispatch = createAgentThreadTopology( + env.THREAD, + env.DEPLOYMENT_IDENTITY_SECRET, + { + startIdempotency: startIdempotencyForEnv(env), + executionFence: executionFenceForEnv(env), + workflowRemoteStepDispatch: journal, + }, + ); + return createWorkflowRemoteStepPreRoute(journal, dispatch)( + request, + env, + ctx, + kit, + ); + }, +}; + +const privateRemoteWorker = createFlowsafeWorker(privateRemoteConfig); +export const DemoPrivateMaintenance = + createFlowsafeMaintenanceDurableObject(privateRemoteConfig); + const handler: ExportedHandler = { async fetch( request: CfRequest, @@ -3109,6 +3391,13 @@ const handler: ExportedHandler = { buildService: (store) => buildApprovalService(store, env, streamSink), }); const routed = request as unknown as Request; + if ( + new URL(routed.url).pathname.startsWith(`${WORKFLOW_REMOTE_STEP_ROUTE}/`) + ) { + if (env.SPIKE_REMOTE_STEP_PROOF !== 'enabled') + return json({ error: 'not found' }, 404); + return privateRemoteWorker.fetch(routed, env, ctx); + } const runTopology = createDoRunTopology( env.RUNNER, env.DEPLOYMENT_IDENTITY_SECRET, @@ -3125,6 +3414,9 @@ const handler: ExportedHandler = { const inventoryProbe = await handleInventoryProbe(routed, env); if (inventoryProbe) return inventoryProbe; + const remoteStepProbe = await handleRemoteStepProbe(routed, env); + if (remoteStepProbe) return remoteStepProbe; + const agentProbeResponse = await handleLiveAgentRoute(routed, env, resolve); if (agentProbeResponse) return agentProbeResponse; diff --git a/packages/flowsafe/spike/wrangler.jsonc b/packages/flowsafe/spike/wrangler.jsonc index b4670513..9ee877e9 100644 --- a/packages/flowsafe/spike/wrangler.jsonc +++ b/packages/flowsafe/spike/wrangler.jsonc @@ -16,6 +16,7 @@ { "name": "HUB", "class_name": "DemoHub" }, { "name": "THREAD", "class_name": "DemoThread" }, { "name": "BACKGROUND_TASKS", "class_name": "DemoBackgroundTasks" }, + { "name": "MAINTENANCE", "class_name": "DemoPrivateMaintenance" }, { "name": "SIGNAL_PROVIDER_HOST", "class_name": "DemoSignalProviderHost" @@ -29,7 +30,8 @@ { "tag": "v2", "new_sqlite_classes": ["DemoHub"] }, { "tag": "v3", "new_sqlite_classes": ["DemoThread"] }, { "tag": "v4", "new_sqlite_classes": ["DemoSignalProviderHost"] }, - { "tag": "v5", "new_sqlite_classes": ["DemoBackgroundTasks"] } + { "tag": "v5", "new_sqlite_classes": ["DemoBackgroundTasks"] }, + { "tag": "v6", "new_sqlite_classes": ["DemoPrivateMaintenance"] } ], "d1_databases": [ { @@ -46,6 +48,7 @@ "vars": { "DEPLOYMENT_TENANT": "spike", "DEPLOYMENT_IDENTITY_SECRET": "spike-local-deployment-identity-secret", + "MAINTENANCE_ADMIN_SECRET": "spike-local-private-admin-secret-2026", "STREAM_TICKET_SECRET": "spike-local-stream-secret-do-not-deploy", // Track E: the github webhook signing secret. spike-verify.mjs re-passes it // via `--var` and signs its webhook probes with it (byte-equal). Never a real diff --git a/packages/flowsafe/src/agent-host/catalog.ts b/packages/flowsafe/src/agent-host/catalog.ts index 20243b4c..0b9997da 100644 --- a/packages/flowsafe/src/agent-host/catalog.ts +++ b/packages/flowsafe/src/agent-host/catalog.ts @@ -159,7 +159,7 @@ function automationCheckFor( // human approves it — a decided approval and a stranded run. The narrowing // that matters is still enforced: a kind removed from the declaration // entirely can no longer resume. - if (entryPath === 'approval.resume') { + if (entryPath === 'approval.resume' || entryPath === 'remote-step.resume') { return rules.some((rule) => rule.kind === principal.kind); } return rules.some( diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index af52ff3a..a04faed2 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -10,6 +10,14 @@ export { ReservedExecutionContextError, stripReservedExecutionContext, } from '../do-runner/index.js'; +export type { + RecordedRemoteStepResult, + RemoteStepInvocation, + RemoteStepNotificationOperation, + RemoteStepPermit, + WorkflowRemoteStepAddress, + WorkflowRemoteStepInvocation, +} from '../do-runner/remote-step-journal.js'; export { type AgentApprovalResumer, type AgentApprovalResumerOptions, @@ -50,6 +58,12 @@ export { type AgentThreadTopology, type AgentThreadTopologyOptions, createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, + type RemoteStepContinuationReader, + type RemoteStepDispatchReader, + type RemoteStepDispatchTicket, + type WorkflowRemoteStepDispatchReader, + type WorkflowRemoteStepDispatchTicket, } from './thread-topology.js'; export { AGENT_AUDIT_CONTEXT_KEY, @@ -70,3 +84,8 @@ export type { TrustedAgentExecution, } from './types.js'; export { isPermissionIdentifier } from './types.js'; +export type { WorkflowDispatchPort } from './workflow-remote-step-route.js'; +export { + createWorkflowRemoteStepPreRoute, + WORKFLOW_REMOTE_STEP_ROUTE, +} from './workflow-remote-step-route.js'; diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 1c1a1115..95043e22 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -9,6 +9,7 @@ import { nestedArray } from '../../test-support/deep-json.js'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; import { AgentRunSelectorMismatchError, + type AgentStartAuthority, FlowsafeDurableAgent, } from '../agent-runner/durable-agent-runner.js'; import { @@ -22,6 +23,12 @@ import { type RecoverableResourceOwnershipStore, type ResourceOwnershipDatabase, } from '../approval-api/index.js'; +import { decodeExecutionPrincipal } from '../approval-api/principal.js'; +import { + credentialsMatch, + DEPLOYMENT_IDENTITY_HEADER, +} from '../do-runner/deployment-identity.js'; +import { EXECUTION_PRINCIPAL_HEADER } from '../do-runner/execution-principal-header.js'; import { FENCED_WORKFLOW_STORAGE, type FencedWorkflowAdmissionCapability, @@ -49,12 +56,30 @@ import { resourceIdFromKey, SUSPENSION_TIMEOUT_RESUME_KEY, } from '../do-runner/index.js'; +import { + REMOTE_STEP_MARKER, + RemoteStepJournal, +} from '../do-runner/remote-step-journal.js'; import { runInputDepthMessage } from '../do-runner/run-input-depth.js'; import { lifecycleFromRequestContext } from '../do-runner/run-lifecycle.js'; +import { + createFlowsafeMaintenanceDurableObject, + ensureRemoteStepNotificationRecoveryDuty, + type FlowsafeWorkerConfig, + type FlowsafeWorkerEnv, + MAINTENANCE_INSTANCE_NAME, + type MaintenanceDurableObjectState, +} from '../host-kit/flowsafe-worker.js'; +import type { + ThreadNamespaceLike, + ThreadRequestInit, +} from '../host-kit/thread-topology.js'; +import { staticTokenVerifier } from '../host-kit/verifier.js'; import { D1SchedulesStorage, type ScheduleDatabase, } from '../schedules/schedules-d1.js'; +import { D1NotificationsStorage } from '../signals/notifications-d1.js'; import { type AgentThreadInstanceScope, type AgentThreadStateStorage, @@ -63,7 +88,11 @@ import { type PrincipalPermissionResolver, type ThreadAgentStartInput, } from './thread-host.js'; -import { createAgentThreadTopology } from './thread-topology.js'; +import { + createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, + type RemoteStepContinuationReader, +} from './thread-topology.js'; import type { AgentAutomationRule, Permission } from './types.js'; const mocked = vi.hoisted(() => ({ @@ -264,6 +293,12 @@ function harness( dispatchId: string, runId: string, ) => Promise; + remoteSteps?: RemoteStepContinuationReader; + remoteStepNotification?: NonNullable< + import('./thread-host.js').ThreadAgentHostOptions['remoteStepNotification'] + >; + stateStorage?: AgentThreadStateStorage; + resourceIdFromKey?: (threadId: string) => string; } = {}, ): Harness { const state = new Map(); @@ -274,9 +309,24 @@ function harness( state.set(key, structuredClone(value)); }, delete: async (key) => state.delete(key), - list: async ({ prefix }: { prefix: string }) => + list: async ({ + prefix, + startAfter, + limit, + }: { + prefix: string; + startAfter?: string; + limit?: number; + }) => new Map( - [...state.entries()].filter(([key]) => key.startsWith(prefix)), + [...state.entries()] + .filter( + ([key]) => + key.startsWith(prefix) && + (startAfter === undefined || key > startAfter), + ) + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .slice(0, limit), ) as Map, getAlarm: async () => alarm === undefined @@ -373,7 +423,12 @@ function harness( kind: options.principal?.kind ?? 'human', id: options.principal?.id ?? 'operator-1', }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }; const current = snapshot as { requestContext: Record; @@ -425,13 +480,13 @@ function harness( : undefined; return { kind: summary.status === 'pending' ? 'initial' : 'result', - storage: 'unfenced', + storage: options.remoteSteps ? 'd1' : 'unfenced', execution: ( state.get(OWNER_RECOVERY_PREFIX + runId) as | { execution?: unknown } | undefined )?.execution ?? { - tablePrefix: null, + tablePrefix: options.remoteSteps?.tablePrefix ?? null, workflowId: 'durable-agentic-loop', runId, startToken: 'test-generation', @@ -492,6 +547,9 @@ function harness( }, }; const host = createThreadAgentHost({ + ...(options.resourceIdFromKey + ? { resourceIdFromKey: options.resourceIdFromKey } + : {}), ...(options.authorizeAutomatedEntry ? { authorizeAutomatedEntry: options.authorizeAutomatedEntry } : {}), @@ -523,12 +581,16 @@ function harness( storageScopes.push(instanceScope); return storage; }, - stateStorage: () => stateStorage, + stateStorage: () => options.stateStorage ?? stateStorage, resourceAccess: () => resourceAccess, scheduleSource: () => scheduleSource, ...(options.discardScheduleDispatch ? { discardScheduleDispatch: options.discardScheduleDispatch } : {}), + ...(options.remoteSteps ? { remoteSteps: options.remoteSteps } : {}), + ...(options.remoteStepNotification + ? { remoteStepNotification: options.remoteStepNotification } + : {}), approvalService: (instanceScope) => { approvalScopes.push(instanceScope); if (options.approvalService) return options.approvalService; @@ -1419,7 +1481,12 @@ describe('direct thread host capture', () => { mutationEpoch: 2, startIdentity: { owner: { kind: principal.kind, id: principal.id }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded }, onPreparedStartIdentity: expect.any(Function), @@ -1572,6 +1639,49 @@ describe('direct thread host capture', () => { }); describe('createThreadAgentHost owner recovery', () => { + it('advances a bounded notification marker cursor past repeatedly failing early items', async () => { + vi.spyOn(console, 'error').mockImplementation(() => {}); + onTestFinished(() => { + vi.restoreAllMocks(); + }); + const seen: string[] = []; + const journal = { + deploymentId: 'deployment', + tablePrefix: '', + get: async (invocationId: string) => { + seen.push(invocationId); + throw new Error('recovery item unavailable'); + }, + } as unknown as RemoteStepJournal; + const fixture = harness(['writer'], { + remoteSteps: journal, + remoteStepNotification: { + journal, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + ensureScheduledRecovery: async () => undefined, + }, + }); + const ids = Array.from( + { length: 18 }, + (_, index) => `operation-${String(index).padStart(2, '0')}`, + ); + for (const id of ids) + await fixture.stateStorage.put( + `flowsafe:agent-notification-recovery:v1:${id}`, + id, + ); + const recover = () => fixture.host.recoverOwnership(fixture.scope); + await expect(recover()).rejects.toThrow('recovery item unavailable'); + expect(seen).toEqual(ids.slice(0, 16)); + await expect(recover()).rejects.toThrow('recovery item unavailable'); + expect(seen).toEqual(ids); + await expect(recover()).rejects.toThrow('recovery item unavailable'); + expect(seen).toEqual([...ids, ...ids.slice(0, 16)]); + }); async function seedPreparingJournal( state: Map, resources: InMemoryResourceOwnershipStore, @@ -2106,6 +2216,23 @@ describe('createThreadAgentHost owner recovery', () => { }); describe('createThreadAgentHost', () => { + it('starts under a trusted nonidentity resource mapping', async () => { + const f = harness(['writer'], { + resourceIdFromKey: () => 'account-7', + }); + + await f.host.start(f.scope, { + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'account-7', + runId: 'account-run', + prompt: 'go', + entryPath: 'http.start', + }); + + const authority = mocked.stream.mock.calls[0]?.[7] as AgentStartAuthority; + expect(authority.startIdentity.target.resourceId).toBe('account-7'); + }); it.each([ ['top-level', [{ role: 'system', content: 'x' }]], ['once-nested', [[{ role: 'system', content: 'x' }]]], @@ -3131,7 +3258,12 @@ describe('createThreadAgentHost', () => { { startIdentity: { owner: HUMAN_OWNER, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded: false }, onPreparedStartIdentity: expect.any(Function), @@ -3455,7 +3587,12 @@ describe('createThreadAgentHost', () => { { startIdentity: { owner: { kind: 'service', id: 'webhook-dispatcher' }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded: true }, onPreparedStartIdentity: expect.any(Function), @@ -3700,7 +3837,12 @@ describe('createThreadAgentHost', () => { { startIdentity: { owner: { kind: 'system', id: 'signal-dispatcher' }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded: true }, onPreparedStartIdentity: expect.any(Function), @@ -3954,6 +4096,406 @@ describe('createThreadAgentHost', () => { ]); }); + it('resumes a retained remote result for its stored run and suspension', async () => { + const principal: ExecutionPrincipal = { + kind: 'service', + id: 'executor-service', + purpose: 'remote-step-run', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: vi.fn(async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained' as const, + permitGeneration: 1, + })), + readRetainedResult: vi.fn(async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + })), + claimResume: vi.fn(async () => 'lease-1'), + settleAfterResume: vi.fn(async () => 'applied' as const), + }; + const fixture = harness(['writer'], { + remoteSteps: reader, + principal, + allowedAutomation: [{ kind: 'service', entryPaths: ['http.start'] }], + }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + const resumed = { + runId: 'acme_run', + status: 'suspended' as const, + suspended: [['next']], + }; + mocked.resumeViaRuntime.mockResolvedValue(resumed); + const request = new Request( + 'https://thread/_flowsafe/agent-host/remote-step-resume', + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + invocationId: 'invocation_1', + resultJson: '{"answer":999}', + }), + }, + ); + const response = await fixture.host.route(request, fixture.scope); + expect(response?.status).toBe(200); + expect(mocked.resumeViaRuntime).toHaveBeenCalledWith( + expect.objectContaining({ + runId: 'acme_run', + requestedBy: 'executor-service', + requestedByKind: 'service', + step: ['tool'], + resumeData: { answer: 1 }, + expectedSuspension: { suspendedAt: 1000 }, + }), + ); + expect(reader.claimResume).toHaveBeenCalledWith('invocation_1', 60_000); + expect(reader.settleAfterResume).toHaveBeenCalledWith( + 'invocation_1', + 'lease-1', + resumed, + ); + + mocked.resumeViaRuntime.mockClear(); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1001 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: 409 }); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + + it.each([ + ['terminal-not-success', 409], + ['deadline-expired', 409], + ['retry', 503], + ['advanced-without-proof', 503], + ] as const)('refuses a stale successful resume response when settlement is %s', async (settlement, expectedStatus) => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained', + permitGeneration: 1, + }), + readRetainedResult: async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + }), + claimResume: vi.fn(async () => 'lease-1'), + settleAfterResume: vi.fn(async () => settlement), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + const stale = { runId: 'acme_run', status: 'success' as const }; + mocked.resumeViaRuntime.mockResolvedValue(stale); + + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: expectedStatus }); + expect(reader.settleAfterResume).toHaveBeenCalledWith( + 'invocation_1', + 'lease-1', + stale, + ); + }); + + it('waits for an in-flight resume before considering its expired lease', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const entered = deferredSignal(); + const release = deferredSignal(); + let status: 'result-retained' | 'resuming' | 'settled' = 'result-retained'; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status, + permitGeneration: 1, + ...(status === 'resuming' + ? { resumeLeaseExpiresAt: Date.now() - 1 } + : {}), + }), + readRetainedResult: async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + }), + claimResume: vi.fn(async () => { + status = 'resuming'; + return 'lease-1'; + }), + settleAfterResume: vi.fn(async () => { + status = 'settled'; + return 'applied' as const; + }), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + mocked.resumeViaRuntime.mockImplementation(async () => { + entered.resolve(); + await release.promise; + return { runId: 'acme_run', status: 'suspended' }; + }); + const request = () => + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }); + const first = fixture.host.route(request(), fixture.scope); + await entered.promise; + const second = fixture.host + .route(request(), fixture.scope) + .catch((error) => error); + try { + await new Promise((resolve) => setImmediate(resolve)); + expect(reader.claimResume).toHaveBeenCalledTimes(1); + } finally { + release.resolve(); + } + expect((await first)?.status).toBe(200); + expect(await second).toMatchObject({ status: 404 }); + expect(reader.claimResume).toHaveBeenCalledTimes(1); + expect(mocked.resumeViaRuntime).toHaveBeenCalledTimes(1); + }); + + it('refuses an expired lease even when the run advanced', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'resuming', + resumeLeaseExpiresAt: Date.now() - 1, + permitGeneration: 1, + }), + readRetainedResult: vi.fn(async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + })), + claimResume: vi.fn(async () => 'lease-2'), + settleAfterResume: vi.fn(async () => 'applied' as const), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ runId: 'acme_run', status: 'success' }); + + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: 503 }); + expect(reader.claimResume).not.toHaveBeenCalled(); + expect(reader.readRetainedResult).not.toHaveBeenCalled(); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + + it('refuses a reserved timeout envelope in a retained remote result', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained', + permitGeneration: 1, + }), + readRetainedResult: async () => ({ + invocationId: 'invocation_1', + resultJson: JSON.stringify({ [SUSPENSION_TIMEOUT_RESUME_KEY]: true }), + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + }), + claimResume: async () => 'lease-1', + settleAfterResume: async () => 'applied' as const, + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: 400 }); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + it('accepts an approval-resume requester exactly at the principal-id bound', async () => { const fixture = harness(); seedSuspendedApprovalRun(fixture); @@ -7047,11 +7589,14 @@ async function hostAgentLifecycleFixture( journal?: boolean; lifecycle?: boolean; snapshotTarget?: { agentId: string; threadId: string }; + resourceIdFromKey?: (threadId: string) => string; + startResourceId?: string; } = {}, ) { const core = await vi.importActual( '@mastra/core/mastra', ); + const { InMemoryStore } = await import('@mastra/core/storage'); const breakwater = await vi.importActual< typeof import('@proofoftech/breakwater') @@ -7075,6 +7620,7 @@ async function hostAgentLifecycleFixture( const threaded = input.threaded ?? true; const status = input.status ?? 'suspended'; const version = input.provenance ?? 'v1'; + const startResourceId = input.startResourceId ?? RESOURCE_ID; const sql = openSqlite() as ReturnType & { close(): void }; const binding = sqliteUnitDatabase(sql) as ExecutionFenceDatabase & ResourceOwnershipDatabase; @@ -7108,6 +7654,9 @@ async function hostAgentLifecycleFixture( }; const dispatch = vi.fn(async () => {}); const fixture = harness(['writer'], { + ...(input.resourceIdFromKey + ? { resourceIdFromKey: input.resourceIdFromKey } + : {}), storage, init: app, resourceAccess: resources, @@ -7137,7 +7686,12 @@ async function hostAgentLifecycleFixture( await reservations.bindPreparedStart(claim, { ...execution, owner: HUMAN_OWNER, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: startResourceId, + }, }); } const recovery = ownerRecovery('acme_run', { @@ -7145,15 +7699,22 @@ async function hostAgentLifecycleFixture( token: 'host-lifecycle-attempt', execution, threaded, + resourceId: startResourceId, ...(claim ? { startReservation: claim } : {}), }); seedRecoveryState(fixture.state, 'acme_run', recovery, threaded); + if (threaded && startResourceId !== RESOURCE_ID) + fixture.state.set(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: startResourceId, + }); if (!input.journal) fixture.state.delete(TEST_OWNER_RECOVERY_KEY); await resources.reserveAll( [ { kind: 'run', resourceId: 'acme_run' }, { kind: 'thread', resourceId: 'acme_thread' }, - { kind: 'resource', resourceId: RESOURCE_ID }, + { kind: 'resource', resourceId: startResourceId }, ], HUMAN_OWNER, recovery.token as string, @@ -7164,7 +7725,10 @@ async function hostAgentLifecycleFixture( if (!workflows) throw new Error('missing host workflow domain'); const snapshotAgentId = input.snapshotTarget?.agentId ?? 'writer'; const snapshotThreadId = input.snapshotTarget?.threadId ?? 'acme_thread'; - const snapshotResourceId = resourceIdFromKey(snapshotThreadId); + const snapshotResourceId = + snapshotThreadId === 'acme_thread' + ? startResourceId + : resourceIdFromKey(snapshotThreadId); const snapshot: import('@mastra/core/workflows').WorkflowRunState = { runId: 'acme_run', status: @@ -7221,6 +7785,7 @@ async function hostAgentLifecycleFixture( kind: 'agent', id: snapshotAgentId, threadId: snapshotThreadId, + resourceId: snapshotResourceId, }, }, agentStart: { threaded }, @@ -7266,6 +7831,7 @@ async function hostAgentLifecycleFixture( .all(); return { ...fixture, + binding, storage, app, sql, @@ -7285,14 +7851,848 @@ async function hostAgentLifecycleFixture( }; } -function hostAgentLifecycleRequest(suffix = '', query = '') { +function hostAgentLifecycleRequest( + suffix = '', + query = '', + resourceId = RESOURCE_ID, +) { return new Request( - `https://thread/_flowsafe/agent-host/runs/writer/acme_run${suffix}?resourceId=${RESOURCE_ID}${query}`, + `https://thread/_flowsafe/agent-host/runs/writer/acme_run${suffix}?resourceId=${resourceId}${query}`, suffix ? { method: 'POST' } : undefined, ); } +it('resumes a retained mapped-resource result from D1 after the agent host is recreated', async () => { + const mappedResourceId = 'account-7'; + const mappedResource = () => mappedResourceId; + const fixture = await hostAgentLifecycleFixture({ + mode: 'fenced', + provenance: 'modern', + resourceIdFromKey: mappedResource, + startResourceId: mappedResourceId, + }); + onTestFinished(() => { + vi.restoreAllMocks(); + fixture.sql.close(); + }); + fixture.snapshot.suspendedPaths = { tool: [0] }; + Object.assign(fixture.snapshot.context, { + tool: { + status: 'suspended', + payload: {}, + startedAt: 50, + suspendedAt: 1000, + suspendPayload: { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + await fixture.persist(); + await fixture.host.route( + hostAgentLifecycleRequest('', '', mappedResourceId), + fixture.scope, + ); + const journal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: fixture.app.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + workflowId: fixture.execution.workflowId, + agentId: 'writer', + threadId: 'acme_thread', + resourceId: mappedResourceId, + runId: 'acme_run', + }, + Date.now() + 60_000, + fixture.scope.principal, + ); + const permit = await journal.issuePermit(invocation.invocationId, 5_000); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"answer":1}', + '{"receipt":"complete"}', + ); + + const freshApp = init( + { storage: fixture.storage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const reopened = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: freshApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const fresh = harness(['writer'], { + resourceIdFromKey: mappedResource, + init: freshApp, + storage: fixture.storage, + stateStorage: fixture.stateStorage, + resourceAccess: fixture.resources, + remoteSteps: { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + get: reopened.get.bind(reopened), + readRetainedResult: reopened.readRetainedResult.bind(reopened), + claimResume: reopened.claimResume.bind(reopened), + settleAfterResume: async () => 'applied' as const, + }, + }); + const resume = vi + .spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime') + .mockResolvedValue({ runId: 'acme_run', status: 'suspended' }); + await fixture.stateStorage.put(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: 'other-resource', + }); + const displaced = await fresh.host + .route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + fresh.scope, + ) + .catch((error) => error); + expect(displaced).toMatchObject({ status: 404 }); + expect(resume).not.toHaveBeenCalled(); + await fixture.stateStorage.put(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: mappedResourceId, + }); + const response = await fresh.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + fresh.scope, + ); + expect(response?.status).toBe(200); + expect(resume).toHaveBeenCalledWith( + expect.objectContaining({ + runId: 'acme_run', + requestedBy: 'operator-1', + requestedByKind: 'human', + step: ['tool'], + resumeData: { answer: 1 }, + expectedSuspension: { suspendedAt: 1000 }, + }), + ); + const leaseBefore = await fixture.binding + .prepare( + 'SELECT resume_lease_owner FROM flowsafe_remote_step_invocations WHERE invocation_id = ?', + ) + .bind(invocation.invocationId) + .all(); + expect(leaseBefore.results).toMatchObject([ + { resume_lease_owner: expect.any(String) }, + ]); + await fixture.binding + .prepare( + 'UPDATE flowsafe_remote_step_invocations SET resume_lease_expires_at = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ?', + ) + .bind(Date.now() - 1, invocation.invocationId) + .run(); + await expect( + fresh.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + fresh.scope, + ), + ).rejects.toMatchObject({ status: 503 }); + expect( + await fixture.binding + .prepare( + 'SELECT resume_lease_owner FROM flowsafe_remote_step_invocations WHERE invocation_id = ?', + ) + .bind(invocation.invocationId) + .all(), + ).toMatchObject({ results: leaseBefore.results }); + expect(resume).toHaveBeenCalledTimes(1); +}); + +it('admits a configured notification effect once before agent continuation', async () => { + const fixture = await hostAgentLifecycleFixture({ + mode: 'fenced', + provenance: 'modern', + }); + onTestFinished(() => { + vi.restoreAllMocks(); + fixture.sql.close(); + }); + fixture.snapshot.suspendedPaths = { tool: [0] }; + Object.assign(fixture.snapshot.context, { + tool: { + status: 'suspended', + payload: {}, + startedAt: 50, + suspendedAt: 1000, + suspendPayload: { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { notificationOperationId: 'enqueue-v1' }, + }, + }, + }, + }); + await fixture.persist(); + await fixture.host.route(hostAgentLifecycleRequest(), fixture.scope); + const journal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: fixture.app.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + workflowId: fixture.execution.workflowId, + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + }, + Date.now() + 60_000, + fixture.scope.principal, + ); + const permit = await journal.issuePermit(invocation.invocationId, 5_000); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"untrusted":"result"}', + '{"receipt":"complete"}', + ); + const freshApp = init( + { storage: fixture.storage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const reopened = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: freshApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const readiness = vi.fn(async () => undefined); + const host = harness(['writer'], { + init: freshApp, + storage: fixture.storage, + stateStorage: fixture.stateStorage, + resourceAccess: fixture.resources, + remoteSteps: reopened, + remoteStepNotification: { + journal: reopened, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + payload: { fixed: true }, + ensureScheduledRecovery: readiness, + }, + }); + const resume = vi + .spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime') + .mockResolvedValue({ runId: 'acme_run', status: 'suspended' }); + const request = () => + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }); + readiness.mockRejectedValueOnce(new Error('recovery duty unavailable')); + await expect(host.host.route(request(), host.scope)).rejects.toThrow( + 'recovery duty unavailable', + ); + expect( + (await reopened.readNotificationOperation(invocation.invocationId))?.state, + ).toBe('prepared'); + const admission = vi + .spyOn(reopened, 'admitNotificationEffect') + .mockRejectedValueOnce(new Error('effect admission refused')); + await expect(host.host.route(request(), host.scope)).rejects.toThrow( + 'effect admission refused', + ); + expect( + fixture.sql + .prepare( + "SELECT COUNT(*) AS count FROM sqlite_master WHERE name = 'host_lifecycle_mastra_notifications'", + ) + .get(), + ).toMatchObject({ count: 0 }); + expect(resume).not.toHaveBeenCalled(); + admission.mockRestore(); + const recordReceipt = reopened.recordNotificationReceipt.bind(reopened); + const lostReceipt = vi + .spyOn(reopened, 'recordNotificationReceipt') + .mockImplementationOnce(async (...args) => { + await recordReceipt(...args); + throw new Error('receipt acknowledgement lost'); + }); + await expect(host.host.route(request(), host.scope)).rejects.toThrow( + 'receipt acknowledgement lost', + ); + expect(resume).not.toHaveBeenCalled(); + lostReceipt.mockRestore(); + const restartedStorage = createD1Storage({ + binding: fixture.binding, + tablePrefix: 'host_lifecycle_', + }); + await restartedStorage.init(); + const restartedApp = init( + { storage: restartedStorage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const restartedJournal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: restartedApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const restartedHost = harness(['writer'], { + init: restartedApp, + storage: restartedStorage, + stateStorage: { ...fixture.stateStorage }, + resourceAccess: fixture.resources, + remoteSteps: restartedJournal, + remoteStepNotification: { + journal: restartedJournal, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Changed after deployment', + payload: { fixed: false }, + ensureScheduledRecovery: async () => undefined, + }, + }); + await expect( + restartedHost.host.route(request(), restartedHost.scope), + ).rejects.toMatchObject({ + status: 503, + }); + const operation = await restartedJournal.readNotificationOperation( + invocation.invocationId, + ); + expect(operation?.state).toBe('resume-admitted'); + expect( + await restartedJournal + .notificationStorage() + .readNotificationOperationReceipt(operation?.operationKey ?? ''), + ).toMatchObject({ + requestDigest: operation?.requestDigest, + notificationId: operation?.notificationId, + }); + expect( + fixture.sql + .prepare( + 'SELECT source, kind, summary, payload FROM host_lifecycle_mastra_notifications', + ) + .get(), + ).toMatchObject({ + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + payload: '{"fixed":true}', + }); + expect(resume).toHaveBeenCalledTimes(1); + await expect( + restartedHost.host.route(request(), restartedHost.scope), + ).rejects.toMatchObject({ + status: 503, + }); + expect(resume).toHaveBeenCalledTimes(1); + expect( + fixture.sql + .prepare( + 'SELECT COUNT(*) AS count FROM host_lifecycle_mastra_notifications', + ) + .get(), + ).toMatchObject({ count: 1 }); +}); + +it.each([ + 'before write', + 'after write', + 'scheduled tick', +] as const)('recovers an admitted notification after its deadline without resuming Mastra (%s)', async (failurePoint) => { + const fixture = await hostAgentLifecycleFixture({ + mode: 'fenced', + provenance: 'modern', + }); + onTestFinished(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + fixture.sql.close(); + }); + fixture.snapshot.suspendedPaths = { tool: [0] }; + Object.assign(fixture.snapshot.context, { + tool: { + status: 'suspended', + payload: {}, + startedAt: 50, + suspendedAt: 1000, + suspendPayload: { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { notificationOperationId: 'enqueue-v1' }, + }, + }, + }, + }); + await fixture.persist(); + await fixture.host.route(hostAgentLifecycleRequest(), fixture.scope); + const journal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: fixture.app.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + workflowId: fixture.execution.workflowId, + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + }, + Date.now() + 60_000, + fixture.scope.principal, + ); + const permit = await journal.issuePermit(invocation.invocationId, 5_000); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"untrusted":"result"}', + '{"receipt":"complete"}', + ); + const freshApp = init( + { storage: fixture.storage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const reopened = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: freshApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const host = harness(['writer'], { + init: freshApp, + storage: fixture.storage, + stateStorage: fixture.stateStorage, + resourceAccess: fixture.resources, + remoteSteps: reopened, + remoteStepNotification: { + journal: reopened, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + ensureScheduledRecovery: async () => undefined, + }, + }); + const retained = await reopened.readRetainedResult(invocation.invocationId); + if (!retained) throw new Error('retained result is missing'); + await reopened.prepareNotificationOperation( + invocation.invocationId, + 'enqueue-v1', + { + threadId: invocation.threadId, + resourceId: invocation.resourceId, + agentId: invocation.agentId, + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + sourceId: invocation.invocationId, + createdAt: new Date(retained.executorQuiescentAt), + }, + ); + expect( + await host.host.recoverNotificationOperation( + host.scope, + invocation.invocationId, + ), + ).toBe('unresolved'); + const resume = vi.spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime'); + const create = D1NotificationsStorage.prototype.createNotificationOnce; + const lostCall = vi + .spyOn(D1NotificationsStorage.prototype, 'createNotificationOnce') + .mockImplementationOnce(async function ( + this: D1NotificationsStorage, + ...args: Parameters + ) { + if (failurePoint === 'after write') await create.apply(this, args); + throw new Error('executor unavailable after admission'); + }); + await expect( + host.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + host.scope, + ), + ).rejects.toThrow('executor unavailable after admission'); + lostCall.mockRestore(); + expect( + (await journal.readNotificationOperation(invocation.invocationId))?.state, + ).toBe('effect-admitted'); + expect(fixture.alarmAt()).toBeDefined(); + await fixture.binding + .prepare( + 'UPDATE flowsafe_remote_step_invocations SET deadline_at = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ?', + ) + .bind(Date.now() - 1, invocation.invocationId) + .run(); + const restartedStorage = createD1Storage({ + binding: fixture.binding, + tablePrefix: 'host_lifecycle_', + }); + await restartedStorage.init(); + const restartedApp = init( + { storage: restartedStorage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const restartedJournal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: restartedApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const restarted = harness(['writer'], { + init: restartedApp, + storage: restartedStorage, + stateStorage: { ...fixture.stateStorage }, + resourceAccess: fixture.resources, + remoteSteps: restartedJournal, + remoteStepNotification: { + journal: restartedJournal, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + ensureScheduledRecovery: async () => undefined, + }, + }); + const recoveryRequest = () => + new Request( + 'https://thread/_flowsafe/agent-host/remote-step-notification-recover', + { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }, + ); + await expect( + restarted.host.route(recoveryRequest(), { + ...restarted.scope, + principal: { ...restarted.scope.principal, id: 'different-user' }, + }), + ).rejects.toMatchObject({ status: 404 }); + if (failurePoint === 'before write') + expect( + fixture.sql + .prepare( + "SELECT COUNT(*) AS count FROM sqlite_master WHERE name = 'host_lifecycle_mastra_notifications'", + ) + .get(), + ).toMatchObject({ count: 0 }); + if (failurePoint === 'before write') { + await restarted.host.recoverOwnership({ + threadId: restarted.scope.threadId, + init: restarted.scope.init, + deploymentTag: restarted.scope.deploymentTag, + }); + } else if (failurePoint === 'after write') { + const recovered = await restarted.host.route( + recoveryRequest(), + restarted.scope, + ); + expect(recovered?.status).toBe(200); + expect(await recovered?.json()).toEqual({ state: 'receipt-recorded' }); + } else { + fixture.sql.exec(`CREATE TABLE flowsafe_deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + tenant_tag TEXT NOT NULL, + provisioned_at TEXT NOT NULL + ); INSERT INTO flowsafe_deployment VALUES + (1, 'acme', '2026-10-14T00:00:00.000Z')`); + const secret = 'test-deployment-identity-secret-0001'; + let failFirst = true; + const namespace: ThreadNamespaceLike = { + idFromName: (name) => name, + get: () => ({ + fetch: (async ( + url: Request | string, + requestInit?: ThreadRequestInit, + ) => { + const request = + typeof url === 'string' ? new Request(url, requestInit) : url; + if ( + !(await credentialsMatch( + request.headers.get(DEPLOYMENT_IDENTITY_HEADER) ?? '', + secret, + )) + ) + return new Response(null, { status: 403 }); + const principal = decodeExecutionPrincipal( + request.headers.get(EXECUTION_PRINCIPAL_HEADER) ?? '', + ); + if (!principal) return new Response(null, { status: 403 }); + if (failFirst) { + failFirst = false; + return new Response(null, { status: 503 }); + } + try { + return ( + (await restarted.host.route(request, { + ...restarted.scope, + principal, + })) ?? new Response(null, { status: 404 }) + ); + } catch (error) { + return doErrorResponse(error); + } + }) as ReturnType['get']>['fetch'], + }), + }; + const tick = createRemoteStepNotificationRecoveryTick({ + journal: restartedJournal, + namespace, + deploymentIdentitySecret: secret, + limit: 1, + }); + const values = new Map(); + let alarmAt: number | null = null; + const maintenanceState: MaintenanceDurableObjectState = { + id: { name: MAINTENANCE_INSTANCE_NAME }, + storage: { + get: async (key: string) => values.get(key) as T | undefined, + getAlarm: async () => alarmAt, + setAlarm: async (time) => { + alarmAt = Number(time); + }, + transaction: async (work) => { + const writes = new Map(); + let planned = alarmAt; + const result = await work({ + get: async (key: string) => values.get(key) as T | undefined, + put: async (key, value) => { + writes.set(key, structuredClone(value)); + }, + setAlarm: async (time) => { + planned = Number(time); + }, + }); + for (const [key, value] of writes) values.set(key, value); + alarmAt = planned; + return result; + }, + }, + }; + const config: FlowsafeWorkerConfig = { + workflows: [], + systemPrincipalId: 'maintenance-test', + buildVerifier: () => staticTokenVerifier(new Map()), + maintenance: { + sweepIntervalMs: 60_000, + purgeIntervalMs: 60_000, + remoteStepRecoveryIntervalMs: 1_000, + }, + remoteStepRecoveryTick: () => tick, + }; + const Maintenance = createFlowsafeMaintenanceDurableObject(config); + let maintenance: { + fetch(request: Request): Promise; + alarm(): Promise; + }; + const env: FlowsafeWorkerEnv = { + DB: fixture.binding as FlowsafeWorkerEnv['DB'], + DEPLOYMENT_TENANT: 'acme', + DEPLOYMENT_IDENTITY_SECRET: secret, + RUNNER: { + idFromName: (name) => name, + get: () => ({ fetch: async () => new Response(null, { status: 503 }) }), + }, + MAINTENANCE: { + idFromName: (name) => name, + get: () => ({ + fetch: (url, requestInit) => + maintenance.fetch(new Request(url, requestInit)), + }), + }, + }; + maintenance = new Maintenance(maintenanceState, env); + const now = Date.now(); + vi.useFakeTimers(); + vi.setSystemTime(now); + await ensureRemoteStepNotificationRecoveryDuty(env); + for (let index = 0; index < 4; index += 1) await maintenance.alarm(); + expect( + ( + await restartedJournal.readNotificationOperation( + invocation.invocationId, + ) + )?.state, + ).toBe('effect-admitted'); + expect( + fixture.sql + .prepare( + 'SELECT after_invocation_id FROM flowsafe_remote_step_notification_cursor', + ) + .get(), + ).toMatchObject({ after_invocation_id: invocation.invocationId }); + maintenance = new Maintenance(maintenanceState, env); + vi.setSystemTime(now + 1_000); + await maintenance.alarm(); + await maintenance.alarm(); + expect( + ( + await restartedJournal.readNotificationOperation( + invocation.invocationId, + ) + )?.state, + ).toBe('receipt-recorded'); + } + expect( + await restarted.host.recoverNotificationOperation( + restarted.scope, + invocation.invocationId, + ), + ).toBe('receipt-recorded'); + expect( + (await journal.readNotificationOperation(invocation.invocationId))?.state, + ).toBe('receipt-recorded'); + await restarted.host.recoverOwnership({ + threadId: restarted.scope.threadId, + init: restarted.scope.init, + deploymentTag: restarted.scope.deploymentTag, + }); + await restarted.host.recoverOwnership({ + threadId: restarted.scope.threadId, + init: restarted.scope.init, + deploymentTag: restarted.scope.deploymentTag, + }); + expect(fixture.alarmAt()).toBeUndefined(); + expect( + fixture.sql + .prepare( + 'SELECT COUNT(*) AS count FROM host_lifecycle_mastra_notifications', + ) + .get(), + ).toMatchObject({ count: 1 }); + await expect( + restarted.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + restarted.scope, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(resume).not.toHaveBeenCalled(); + expect(D1NotificationsStorage.prototype.createNotificationOnce).toBe(create); +}); + describe('agent host selector lookup isolation', () => { + it.each([ + ['modern', 'status'], + ['modern', 'resume'], + ['v1', 'status'], + ] as const)('refuses an unthreaded %s run started under an older resource mapping on %s', async (provenance, route) => { + const f = await hostAgentLifecycleFixture({ + provenance, + threaded: false, + resourceIdFromKey: () => 'account-7', + }); + onTestFinished(() => { + vi.restoreAllMocks(); + f.sql.close(); + }); + const resume = vi.spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime'); + const request = + route === 'status' + ? new Request( + 'https://thread/_flowsafe/agent-host/runs/writer/acme_run?resourceId=account-7', + ) + : new Request('https://thread/_flowsafe/agent-host/resume', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'account-7', + runId: 'acme_run', + requestedBy: 'reviewer-2', + entryPath: 'approval.resume', + resumeData: {}, + }), + }); + const outcome = await f.host + .route(request, f.scope) + .catch((error) => error); + if (route === 'status') expect(outcome).toMatchObject({ status: 404 }); + else expect(outcome).toBeInstanceOf(AgentRunSelectorMismatchError); + expect(resume).not.toHaveBeenCalled(); + }); + it.each([ ['modern', true, 'status'], ['modern', false, 'stream'], @@ -7406,6 +8806,8 @@ describe('agent host selector lookup isolation', () => { if (!context) throw new Error('missing snapshot context'); context['flowsafe.runProvenance'].startIdentity.target.threadId = 'other-thread'; + context['flowsafe.runProvenance'].startIdentity.target.resourceId = + 'other-thread'; Object.assign(context, { threadId: 'other-thread', resourceId: 'other-thread', diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index c0d54045..7e5368d7 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -82,6 +82,12 @@ import { } from '../do-runner/index.js'; import { isDefinitiveInitialAdmissionRefusal } from '../do-runner/initial-admission-refusal.js'; import { mastraRegistryEntries } from '../do-runner/mastra-registry.js'; +import { + REMOTE_STEP_MARKER, + type RemoteStepInvocation, + type RemoteStepJournal, + type RemoteStepNotificationOperation, +} from '../do-runner/remote-step-journal.js'; import { exceedsRunInputDepth, runInputDepthMessage, @@ -109,10 +115,15 @@ import { abandonApprovalsForRun, reconcileApprovalsForSummary, } from '../host-kit/index.js'; +import { + type NotificationOperationInput, + projectNotificationOperation, +} from '../signals/notifications-d1.js'; import { createAgentModuleCatalog } from './catalog.js'; import { AGENT_HOST_ROUTE_PREFIX, publicAgentRunEnvelope, + type RemoteStepContinuationReader, } from './thread-topology.js'; import { createTrustedAgentRequestContext, @@ -132,7 +143,11 @@ export interface AgentThreadStateStorage { get(key: string): Promise; put(key: string, value: T): Promise; delete(key: string): Promise; - list(options: { prefix: string }): Promise>; + list(options: { + prefix: string; + startAfter?: string; + limit?: number; + }): Promise>; getAlarm(): Promise; setAlarm(scheduledTime: number | Date): Promise; deleteAlarm(): Promise; @@ -171,6 +186,19 @@ export type { } from './types.js'; export interface ThreadAgentHostOptions { + remoteSteps?: RemoteStepContinuationReader; + remoteStepNotification?: { + journal: RemoteStepJournal; + agentId: string; + operationId: string; + source: string; + kind: string; + summary: string; + payload?: unknown; + receiptLimit?: number; + ensureScheduledRecovery: () => Promise; + }; + resourceIdFromKey?: (threadId: string) => string; /** Narrows automated entry beyond what each agent's metadata declares. */ authorizeAutomatedEntry?: AutomatedEntryAuthorizer; /** @@ -298,6 +326,10 @@ export interface ThreadAgentHost { input: { agentId: string; resourceId: string; runId: string }, ): Promise; recoverOwnership(scope: AgentThreadInstanceScope): Promise; + recoverNotificationOperation( + scope: ThreadScope, + invocationId: string, + ): Promise<'receipt-recorded' | 'unresolved'>; route(request: Request, scope: ThreadScope): Promise; } @@ -345,6 +377,11 @@ async function objectBody(request: Request): Promise> { } const AGENT_OWNER_RECOVERY_PREFIX = 'flowsafe:agent-owner-recovery:v1:'; +const AGENT_NOTIFICATION_RECOVERY_PREFIX = + 'flowsafe:agent-notification-recovery:v1:'; +const AGENT_NOTIFICATION_RECOVERY_CURSOR_KEY = + 'flowsafe:agent-notification-recovery-cursor:v1'; +const AGENT_NOTIFICATION_RECOVERY_BATCH = 16; const AGENT_OWNER_RECOVERY_DELAY_MS = 60_000; type AgentOwnerRecovery = { @@ -465,6 +502,45 @@ function record(value: unknown): Record | undefined { : undefined; } +async function retainedNotificationInput( + invocation: RemoteStepInvocation, + operation: RemoteStepNotificationOperation, + executorQuiescentAt: number, +): Promise { + let savedInput: Record | undefined; + try { + savedInput = record(JSON.parse(operation.requestJson)); + } catch { + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + } + if (typeof savedInput?.createdAt !== 'string') + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + const stableInput = { + ...savedInput, + createdAt: new Date(savedInput.createdAt), + } as NotificationOperationInput; + let projection: Awaited>; + try { + projection = await projectNotificationOperation(stableInput); + } catch { + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + } + if ( + JSON.stringify(stableInput) !== operation.requestJson || + stableInput.id !== operation.notificationId || + stableInput.threadId !== invocation.threadId || + stableInput.resourceId !== invocation.resourceId || + stableInput.agentId !== invocation.agentId || + stableInput.sourceId !== invocation.invocationId || + savedInput.createdAt !== new Date(executorQuiescentAt).toISOString() || + projection.version !== operation.projectionVersion || + projection.requestJson !== operation.projectionJson || + projection.requestDigest !== operation.requestDigest + ) + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + return stableInput; +} + function scheduleAgentContext( target: ScheduleSourceAgentTarget, ): Record { @@ -620,6 +696,36 @@ function ndjson( export function createThreadAgentHost( options: ThreadAgentHostOptions, ): ThreadAgentHost { + const notificationProfile = options.remoteStepNotification; + if ( + notificationProfile && + (options.remoteSteps !== notificationProfile.journal || + !isPathSafeId(notificationProfile.agentId) || + !isPathSafeId(notificationProfile.operationId) || + !notificationProfile.source || + !notificationProfile.kind || + !notificationProfile.summary || + typeof notificationProfile.ensureScheduledRecovery !== 'function' || + (notificationProfile.receiptLimit !== undefined && + (!Number.isSafeInteger(notificationProfile.receiptLimit) || + notificationProfile.receiptLimit < 0))) + ) + throw new TypeError('remote step notification profile is malformed'); + const notificationReceiptLimit = notificationProfile?.receiptLimit; + let notificationStorage: + | ReturnType + | undefined; + const resourceForThread = (threadId: string): string => { + let resourceId: string; + try { + resourceId = (options.resourceIdFromKey ?? resourceIdFromKey)(threadId); + } catch { + throw new AgentHostRequestError(503, 'resource mapping is unavailable'); + } + if (!isPathSafeId(resourceId)) + throw new AgentHostRequestError(503, 'resource mapping is unavailable'); + return resourceId; + }; let catalogPromise: Promise | undefined; let runtime: | { @@ -883,7 +989,7 @@ export function createThreadAgentHost( scope, { agentId: runRecord.agentId, - resourceId: resourceIdFromKey(scope.threadId), + resourceId: resourceForThread(scope.threadId), runId, }, { includeLegacy: true }, @@ -1056,7 +1162,7 @@ export function createThreadAgentHost( !isPathSafeId(stored.agentId) || !isPathSafeId(stored.threadId) || !isPathSafeId(stored.resourceId) || - stored.resourceId !== resourceIdFromKey(threadId) || + stored.resourceId !== resourceForThread(threadId) || !isPathSafeId(stored.runId) || !isPathSafeId(stored.token) || ownerRecoveryKey(stored.runId) !== key || @@ -1203,6 +1309,14 @@ export function createThreadAgentHost( }; type NormalAgentRunState = AuthoritativeAgentStartState | LegacyAgentRunState; + const matchesStartResource = ( + threadId: string, + resourceId: string, + persistedResourceId: string | undefined, + ): boolean => + persistedResourceId === undefined + ? resourceId === resourceIdFromKey(threadId) + : persistedResourceId === resourceId; async function selectedAgentState( scope: AgentThreadInstanceScope, @@ -1219,23 +1333,47 @@ export function createThreadAgentHost( readOptions?: { readonly includeLegacy: true }, ): Promise { const includeLegacy = readOptions?.includeLegacy === true; - if (ref.resourceId !== resourceIdFromKey(scope.threadId)) + if (ref.resourceId !== resourceForThread(scope.threadId)) throw new AgentHostRequestError(404, 'run not found'); const current = await runtimeFor(scope); const durable = current.agents.get(ref.agentId); if (!current.catalog.get(ref.agentId) || !durable) throw new AgentHostRequestError(404, 'agent not found'); + const assertSelectedResource = ( + state: T, + ): T => { + const target = + state && state.kind !== 'legacy' ? state.execution.target : undefined; + if ( + state && + (target?.kind === 'workflow' || + !matchesStartResource( + scope.threadId, + ref.resourceId, + target?.kind === 'agent' ? target.resourceId : undefined, + )) + ) + throw new AgentRunSelectorMismatchError( + durable.getWorkflow().id, + ref.runId, + ); + return state; + }; if (includeLegacy) - return durable.authoritativeAgentStartState( + return assertSelectedResource( + await durable.authoritativeAgentStartState( + scope.init.runtime, + scope.threadId, + ref.runId, + { includeLegacy: true }, + ), + ); + return assertSelectedResource( + await durable.authoritativeAgentStartState( scope.init.runtime, scope.threadId, ref.runId, - { includeLegacy: true }, - ); - return durable.authoritativeAgentStartState( - scope.init.runtime, - scope.threadId, - ref.runId, + ), ); } @@ -1268,6 +1406,11 @@ export function createThreadAgentHost( state.execution.target.kind !== 'agent' || state.execution.target.id !== recovery.agentId || state.execution.target.threadId !== recovery.threadId || + !matchesStartResource( + recovery.threadId, + recovery.resourceId, + state.execution.target.resourceId, + ) || state.execution.owner.kind !== recovery.runRecord.principal.kind || state.execution.owner.id !== recovery.runRecord.principal.id ) @@ -1362,7 +1505,7 @@ export function createThreadAgentHost( if (state.kind === 'legacy') { const ref = { agentId: expected.agentId, - resourceId: resourceIdFromKey(scope.threadId), + resourceId: resourceForThread(scope.threadId), runId, }; await assertLegacyTerminalCurrent(scope, ref, state, expected); @@ -1377,6 +1520,11 @@ export function createThreadAgentHost( state.execution.target.kind !== 'agent' || state.execution.target.id !== expected.agentId || state.execution.target.threadId !== scope.threadId || + !matchesStartResource( + scope.threadId, + resourceForThread(scope.threadId), + state.execution.target.resourceId, + ) || state.execution.owner.kind !== expected.principal.kind || state.execution.owner.id !== expected.principal.id ) @@ -1724,6 +1872,49 @@ export function createThreadAgentHost( }; }; + const readRemoteStepHostAuthority = async ( + scope: ThreadScope, + invocation: RemoteStepInvocation, + ) => { + const ref = { + agentId: invocation.agentId, + resourceId: invocation.resourceId, + runId: invocation.runId, + }; + const selected = await snapshotExecutionFor(scope, ref); + const state = selected.state; + if ( + state.kind !== 'result' || + state.storage !== 'd1' || + state.execution.tablePrefix !== invocation.tablePrefix || + state.execution.workflowId !== invocation.workflowId || + state.execution.startToken !== invocation.startToken || + state.execution.target.kind !== 'agent' || + state.execution.target.id !== invocation.agentId || + state.execution.target.threadId !== invocation.threadId || + state.execution.target.resourceId !== invocation.resourceId || + state.execution.owner.kind !== invocation.principal.kind || + state.execution.owner.id !== invocation.principal.id + ) + throw new AgentHostRequestError(409, 'remote step run changed'); + const stored = await readRun(ref.runId); + if ( + !stored || + stored.agentId !== ref.agentId || + !samePrincipal(stored.principal, invocation.principal) + ) + throw new AgentHostRequestError(404, 'run not found'); + const binding = await readBinding(); + if ( + (selected.threaded && + (binding?.agentId !== ref.agentId || + binding.resourceId !== ref.resourceId)) || + (!selected.threaded && binding) + ) + throw new AgentHostRequestError(404, 'run not found'); + return { ref, selected, state, stored }; + }; + const statusFor = async ( scope: ThreadScope, ref: { agentId: string; resourceId: string; runId: string }, @@ -1955,6 +2146,73 @@ export function createThreadAgentHost( }), ); + const recoverNotificationOperation = async ( + scope: ThreadScope, + invocationId: string, + ): Promise<'receipt-recorded' | 'unresolved'> => { + instanceScopeFor(scope); + const profile = notificationProfile; + if (!profile || !isPathSafeId(invocationId)) + throw new AgentHostRequestError(404, 'invocation not found'); + const invocation = await profile.journal.get(invocationId); + if ( + !invocation || + invocation.deploymentId !== profile.journal.deploymentId || + invocation.tablePrefix !== profile.journal.tablePrefix || + invocation.threadId !== scope.threadId || + invocation.agentId !== profile.agentId || + !samePrincipal(invocation.principal, scope.principal) || + record(invocation.request)?.notificationOperationId !== + profile.operationId + ) + throw new AgentHostRequestError(404, 'invocation not found'); + const operation = + await profile.journal.readNotificationOperation(invocationId); + if (!operation || operation.operationId !== profile.operationId) + throw new AgentHostRequestError(409, 'remote step operation changed'); + if ( + operation.state === 'resume-admitted' || + operation.state === 'unresolved' || + operation.state === 'prepared' + ) + return 'unresolved'; + await readRemoteStepHostAuthority(scope, invocation); + const retained = await profile.journal.readRetainedResult(invocationId); + if (!retained || invocation.status !== 'result-retained') + throw new AgentHostRequestError(409, 'remote step result is unavailable'); + const stableInput = await retainedNotificationInput( + invocation, + operation, + retained.executorQuiescentAt, + ); + if (!notificationStorage) + notificationStorage = profile.journal.notificationStorage( + notificationReceiptLimit, + ); + const storage = notificationStorage; + if (operation.state === 'effect-admitted') { + const receipt = await storage.createNotificationOnce( + operation.operationKey, + stableInput, + ); + await profile.journal.recordNotificationReceipt(invocationId, receipt); + return 'receipt-recorded'; + } + const receipt = await storage.readNotificationOperationReceipt( + operation.operationKey, + ); + if ( + receipt?.requestDigest !== operation.requestDigest || + receipt.notificationId !== operation.notificationId || + receipt.threadId !== invocation.threadId + ) + throw new AgentHostRequestError( + 503, + 'remote step notification receipt is unreadable', + ); + return 'receipt-recorded'; + }; + const host: ThreadAgentHost = { requestContextForRun: (base) => async (workflowId, runId, leg) => { const values = base ? await base(workflowId, runId, leg) : undefined; @@ -1969,6 +2227,8 @@ export function createThreadAgentHost( : values; }, serializeDispatch: withDispatchLock, + recoverNotificationOperation: (scope, invocationId) => + withDispatchLock(() => recoverNotificationOperation(scope, invocationId)), notificationDispatchAllowed: async (scope, agentId) => { instanceScopeFor(scope); return (await catalogFor(scope)).automationAllowed( @@ -2040,11 +2300,111 @@ export function createThreadAgentHost( firstFailure ??= { error }; } } + if (notificationProfile) { + const cursor = + (await storage.get( + AGENT_NOTIFICATION_RECOVERY_CURSOR_KEY, + )) ?? ''; + if ( + typeof cursor !== 'string' || + (cursor !== '' && + (!cursor.startsWith(AGENT_NOTIFICATION_RECOVERY_PREFIX) || + !isPathSafeId( + cursor.slice(AGENT_NOTIFICATION_RECOVERY_PREFIX.length), + ))) + ) + throw new AgentHostRequestError( + 503, + 'remote step recovery cursor is unreadable', + ); + let notifications = await storage.list({ + prefix: AGENT_NOTIFICATION_RECOVERY_PREFIX, + ...(cursor === '' ? {} : { startAfter: cursor }), + limit: AGENT_NOTIFICATION_RECOVERY_BATCH, + }); + if (notifications.size === 0 && cursor !== '') + notifications = await storage.list({ + prefix: AGENT_NOTIFICATION_RECOVERY_PREFIX, + limit: AGENT_NOTIFICATION_RECOVERY_BATCH, + }); + for (const [key, stored] of notifications) { + const invocationId = key.slice( + AGENT_NOTIFICATION_RECOVERY_PREFIX.length, + ); + try { + if (stored !== invocationId || !isPathSafeId(invocationId)) + throw new AgentHostRequestError( + 503, + 'remote step recovery marker is unreadable', + ); + const invocation = + await notificationProfile.journal.get(invocationId); + if ( + !invocation || + invocation.threadId !== scope.threadId || + invocation.deploymentId !== + notificationProfile.journal.deploymentId || + invocation.tablePrefix !== + notificationProfile.journal.tablePrefix + ) + throw new AgentHostRequestError( + 503, + 'remote step recovery identity changed', + ); + const operation = + await notificationProfile.journal.readNotificationOperation( + invocationId, + ); + if ( + operation?.state === 'effect-admitted' || + operation?.state === 'receipt-recorded' + ) { + const recovered = await recoverNotificationOperation( + { ...scope, principal: invocation.principal }, + invocationId, + ); + if (recovered !== 'receipt-recorded') + throw new AgentHostRequestError( + 503, + 'remote step notification recovery is unresolved', + ); + } else if ( + operation?.state !== 'resume-admitted' && + operation?.state !== 'unresolved' + ) { + throw new AgentHostRequestError( + 503, + 'remote step notification recovery is unresolved', + ); + } + await storage.delete(key); + } catch (error) { + console.error( + JSON.stringify({ + type: 'remote-step-notification-recovery-failed', + threadId: scope.threadId, + invocationId, + error: errorMessageOf(error), + }), + ); + firstFailure ??= { error }; + } + } + const last = [...notifications.keys()].at(-1); + await storage.put( + AGENT_NOTIFICATION_RECOVERY_CURSOR_KEY, + last ?? '', + ); + } await withRecoveryLock(async () => { const remaining = await storage.list({ prefix: AGENT_OWNER_RECOVERY_PREFIX, }); - if (remaining.size > 0) { + const remainingNotifications = await storage.list({ + prefix: AGENT_NOTIFICATION_RECOVERY_PREFIX, + limit: 1, + }); + if (remaining.size > 0 || remainingNotifications.size > 0) { await ensureOwnerRecoveryAlarm(storage); } else { await storage.deleteAlarm(); @@ -2148,6 +2508,7 @@ export function createThreadAgentHost( kind: 'agent', id: ref.agentId, threadId: ref.threadId, + resourceId: ref.resourceId, }), }, ); @@ -2216,7 +2577,7 @@ export function createThreadAgentHost( entry, scope.principal, ); - if (ref.resourceId !== resourceIdFromKey(scope.threadId)) { + if (ref.resourceId !== resourceForThread(scope.threadId)) { throw new AgentHostRequestError(404, 'run not found'); } const execution: TrustedAgentExecution = { @@ -2465,7 +2826,7 @@ export function createThreadAgentHost( resolveBoundAgent: async (scope, input) => { const binding = await readBinding(); if (!binding) throw new AgentHostRequestError(404, 'agent not found'); - if (binding.resourceId !== resourceIdFromKey(scope.threadId)) { + if (binding.resourceId !== resourceForThread(scope.threadId)) { throw new AgentHostRequestError(404, 'agent not found'); } if (input.agentId && input.agentId !== binding.agentId) { @@ -2588,6 +2949,18 @@ export function createThreadAgentHost( preflightUrl.pathname === `${AGENT_HOST_ROUTE_PREFIX}/resume` ? await objectBody(request) : undefined; + const remoteStepBody = + request.method === 'POST' && + preflightUrl.pathname === + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-resume` + ? await objectBody(request) + : undefined; + const notificationRecoveryBody = + request.method === 'POST' && + preflightUrl.pathname === + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-notification-recover` + ? await objectBody(request) + : undefined; return withDispatchLock(async () => { const url = new URL(request.url); @@ -2600,7 +2973,7 @@ export function createThreadAgentHost( const binding = await readBinding(); if ( !resourceId || - resourceId !== resourceIdFromKey(scope.threadId) || + resourceId !== resourceForThread(scope.threadId) || binding?.resourceId !== resourceId || (agentId !== null && binding.agentId !== agentId) ) { @@ -2693,6 +3066,272 @@ export function createThreadAgentHost( ); } + if (notificationRecoveryBody) { + const invocationId = notificationRecoveryBody.invocationId; + if (!isPathSafeId(invocationId)) + throw new AgentHostRequestError(404, 'invocation not found'); + return json({ + state: await recoverNotificationOperation(scope, invocationId), + }); + } + + if (remoteStepBody) { + const reader = options.remoteSteps; + if (!reader) + throw new AgentHostRequestError( + 503, + 'remote steps are unavailable', + ); + const invocationId = remoteStepBody.invocationId; + if (!isPathSafeId(invocationId)) + throw new AgentHostRequestError(404, 'invocation not found'); + const invocation = await reader.get(invocationId); + if ( + !invocation || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + invocation.threadId !== scope.threadId || + !samePrincipal(invocation.principal, scope.principal) + ) + throw new AgentHostRequestError(404, 'invocation not found'); + if (invocation.status === 'resuming') + throw new AgentHostRequestError( + 503, + 'remote step resume awaits executor quiescence', + ); + const retained = await reader.readRetainedResult(invocationId); + if ( + !invocation || + !retained || + retained.invocationId !== invocationId || + invocation.status !== 'result-retained' || + invocation.deadlineAt <= Date.now() + ) + throw new AgentHostRequestError(404, 'invocation not found'); + const { ref, selected, state, stored } = + await readRemoteStepHostAuthority(scope, invocation); + const key = invocation.stepPath.join('.'); + const summary = state.summary; + const payload = + summary.suspendPayload !== null && + typeof summary.suspendPayload === 'object' && + !Array.isArray(summary.suspendPayload) + ? (summary.suspendPayload as Record)[key] + : undefined; + const suspended = + payload !== null && typeof payload === 'object' + ? (payload as Record) + : undefined; + const tool = + suspended?.toolCallSuspended !== null && + typeof suspended?.toolCallSuspended === 'object' + ? (suspended.toolCallSuspended as Record) + : undefined; + if ( + summary.status !== 'suspended' || + !summary.suspended?.some( + (path) => + JSON.stringify(path) === JSON.stringify(invocation.stepPath), + ) || + summary.suspendedAt?.[key] !== invocation.suspendedAt || + summary.resumeCount?.[key] !== invocation.resumeCount || + suspended?.type !== 'suspension' || + tool?.remoteStep !== REMOTE_STEP_MARKER + ) + throw new AgentHostRequestError( + 409, + 'remote step suspension changed', + ); + let recordedResult: unknown; + try { + recordedResult = JSON.parse(retained.resultJson); + } catch { + throw new AgentHostRequestError( + 503, + 'remote step result is unreadable', + ); + } + const { current, module, principalPermissions } = await authorize( + scope, + ref.agentId, + 'remote-step.resume', + stored.principal, + ); + const durable = current.agents.get(module.meta.id); + if (!durable) throw new Error('guarded agent was not registered'); + const execution: TrustedAgentExecution = { + agentId: ref.agentId, + deploymentTag: scope.deploymentTag, + principal: stored.principal, + threadId: scope.threadId, + resourceId: ref.resourceId, + runId: ref.runId, + entryPath: 'remote-step.resume', + principalPermissions, + safeContext: selected.safeContext, + }; + let leaseOwner: string; + if ( + notificationProfile && + invocation.agentId === notificationProfile.agentId && + record(invocation.request)?.notificationOperationId === + notificationProfile.operationId + ) { + const journal = notificationProfile.journal; + const existing = + await journal.readNotificationOperation(invocationId); + if ( + existing && + (existing.invocationId !== invocationId || + existing.operationId !== notificationProfile.operationId) + ) + throw new AgentHostRequestError( + 409, + 'remote step operation changed', + ); + const operation = + existing ?? + (await journal.prepareNotificationOperation( + invocationId, + notificationProfile.operationId, + { + threadId: invocation.threadId, + resourceId: invocation.resourceId, + agentId: invocation.agentId, + source: notificationProfile.source, + kind: notificationProfile.kind, + summary: notificationProfile.summary, + sourceId: invocation.invocationId, + createdAt: new Date(retained.executorQuiescentAt), + ...(notificationProfile.payload === undefined + ? {} + : { payload: notificationProfile.payload }), + }, + )); + const stableInput = await retainedNotificationInput( + invocation, + operation, + retained.executorQuiescentAt, + ); + if (!notificationStorage) + notificationStorage = journal.notificationStorage( + notificationReceiptLimit, + ); + const notifications = notificationStorage; + if ( + operation.state === 'prepared' || + operation.state === 'effect-admitted' + ) { + if (operation.state === 'prepared') + await notificationProfile.ensureScheduledRecovery(); + const admitted = + await journal.admitNotificationEffect(invocationId); + if ( + admitted.operationKey !== operation.operationKey || + admitted.requestDigest !== operation.requestDigest + ) + throw new AgentHostRequestError( + 409, + 'remote step operation changed', + ); + await withRecoveryLock(async () => { + const storage = options.stateStorage(); + await ensureOwnerRecoveryAlarm(storage); + await storage.put( + `${AGENT_NOTIFICATION_RECOVERY_PREFIX}${invocationId}`, + invocationId, + ); + await ensureOwnerRecoveryAlarm(storage); + }); + const receipt = await notifications.createNotificationOnce( + admitted.operationKey, + stableInput, + ); + await journal.recordNotificationReceipt(invocationId, receipt); + await options + .stateStorage() + .delete(`${AGENT_NOTIFICATION_RECOVERY_PREFIX}${invocationId}`); + } else if (operation.state === 'receipt-recorded') { + const receipt = + await notifications.readNotificationOperationReceipt( + operation.operationKey, + ); + if ( + receipt?.requestDigest !== operation.requestDigest || + receipt.notificationId !== operation.notificationId || + receipt.threadId !== invocation.threadId + ) + throw new AgentHostRequestError( + 503, + 'remote step notification receipt is unreadable', + ); + } else { + throw new AgentHostRequestError( + 503, + 'remote step continuation is unresolved', + ); + } + execution.principalPermissions = ( + await authorize( + scope, + ref.agentId, + 'remote-step.resume', + stored.principal, + ) + ).principalPermissions; + leaseOwner = await journal.admitNotificationResume(invocationId); + } else { + leaseOwner = await reader.claimResume(invocationId, 60_000); + } + const resumed = await withExecution(execution, () => + durable.resumeViaRuntime({ + runId: ref.runId, + requestedBy: stored.principal.id, + step: [...invocation.stepPath], + resumeData: resumeData(recordedResult), + requestedByKind: stored.principal.kind, + expectedSuspension: { + suspendedAt: invocation.suspendedAt, + ...(invocation.resumeCount === undefined + ? {} + : { resumeCount: invocation.resumeCount }), + }, + ...(selected.threaded + ? { + memory: { + thread: scope.threadId, + resource: ref.resourceId, + }, + } + : {}), + }), + ); + const settlement = await reader.settleAfterResume( + invocationId, + leaseOwner, + resumed, + ); + if (settlement === 'retry' || settlement === 'advanced-without-proof') + throw new AgentHostRequestError( + 503, + 'remote step resume outcome is unresolved', + ); + if (settlement !== 'applied') + throw new AgentHostRequestError( + 409, + 'remote step result was not applied', + ); + const result = await envelopeFor( + scope, + ref, + stored.principal, + resumed, + ); + if (isTerminalRunStatus(resumed.status)) + await finalizeTerminalAgentState(scope, ref, undefined, stored); + return json(result); + } + if (resumeBody) { const body = resumeBody; const ref = runRef(scope, body); diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 1a0909c3..bc3b5603 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -19,6 +19,9 @@ import { RunRouteError } from '../host-kit/run-route-error.js'; import { type AgentThreadDispatchTopology, createAgentThreadTopology, + type RemoteStepContinuationReader, + type RemoteStepDispatchReader, + type WorkflowRemoteStepDispatchReader, } from './thread-topology.js'; const DEPLOYMENT_IDENTITY_SECRET = 'test-deployment-identity-secret-0001'; @@ -29,7 +32,11 @@ interface Hit { init?: ThreadRequestInit; } -function harness(): { +function harness( + remoteSteps?: RemoteStepContinuationReader, + remoteStepDispatch?: RemoteStepDispatchReader, + workflowRemoteStepDispatch?: WorkflowRemoteStepDispatchReader, +): { topology: AgentThreadDispatchTopology; hits: Hit[]; } { @@ -50,7 +57,7 @@ function harness(): { summary: { runId: body.runId, status: 'success' }, }); } - if (url.endsWith('/resume')) { + if (url.endsWith('/resume') || url.endsWith('/remote-step-resume')) { const body = JSON.parse(init?.body ?? '{}') as Record; return Response.json({ agentId: body.agentId, @@ -92,11 +99,398 @@ function harness(): { topology: createAgentThreadTopology(namespace, DEPLOYMENT_IDENTITY_SECRET, { startIdempotency: 'none', executionFence: 'none', + ...(remoteSteps ? { remoteSteps } : {}), + ...(remoteStepDispatch ? { remoteStepDispatch } : {}), + ...(workflowRemoteStepDispatch ? { workflowRemoteStepDispatch } : {}), }), hits, }; } +it('keeps workflow remote-step dispatch within the stored owner and deployment', async () => { + const owner = context().value; + const permit = { + invocationId: 'workflow-invocation', + generation: 2, + mutationEpoch: 7, + transitionRevision: 4, + expiresAt: 9_999_999_999_999, + }; + const invocation = { + targetKind: 'workflow' as const, + invocationId: permit.invocationId, + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'customer-workflow', + runId: 'workflow-run', + principal: owner.principal, + startToken: 'stored-start-token', + stepPath: ['remote-step'], + suspendedAt: 1000, + request: { script: 'script-1' }, + requestDigest: 'stored-request-digest', + deadlineAt: permit.expiresAt, + status: 'pending' as const, + permitGeneration: 1, + }; + const dispatched = { + ...invocation, + status: 'dispatched' as const, + permitGeneration: permit.generation, + }; + const reader: WorkflowRemoteStepDispatchReader = { + deploymentId: invocation.deploymentId, + tablePrefix: invocation.tablePrefix, + registerWorkflow: vi.fn(async () => invocation), + getWorkflow: vi + .fn() + .mockResolvedValueOnce(invocation) + .mockResolvedValue(dispatched), + readWorkflowDispatchRequestJson: vi.fn(async () => '{"script":"script-1"}'), + issuePermit: vi.fn(async () => permit), + consumePermit: vi + .fn() + .mockResolvedValueOnce({ kind: 'accepted', rawPermit: 'winner-secret' }) + .mockResolvedValueOnce({ kind: 'already-consumed' }), + recordResult: vi.fn(async () => ({ + kind: 'recorded' as const, + resultJson: '{"answer":42}', + resultDigest: 'stored-result-digest', + })), + }; + const { topology } = harness(undefined, undefined, reader); + expect( + await topology.registerWorkflowRemoteStep( + owner, + invocation.workflowId, + invocation.runId, + 10_000, + ), + ).toEqual(invocation); + expect(reader.registerWorkflow).toHaveBeenCalledWith( + { + targetKind: 'workflow', + deploymentId: invocation.deploymentId, + tablePrefix: invocation.tablePrefix, + workflowId: invocation.workflowId, + runId: invocation.runId, + }, + 10_000, + owner.principal, + ); + const ticket = await topology.issueWorkflowRemoteStepDispatch( + owner, + invocation.invocationId, + 60_000, + ); + expect(ticket).toMatchObject({ + invocationId: invocation.invocationId, + requestJson: '{"script":"script-1"}', + requestDigest: invocation.requestDigest, + permit, + }); + expect(ticket).not.toHaveProperty('principal'); + expect(ticket).not.toHaveProperty('startToken'); + expect(reader.issuePermit).toHaveBeenCalledWith( + invocation.invocationId, + 60_000, + ); + const workflowReadsBeforeGet = vi.mocked(reader.getWorkflow).mock.calls + .length; + expect( + await topology.getWorkflowRemoteStep(owner, invocation.invocationId), + ).toEqual(dispatched); + expect(vi.mocked(reader.getWorkflow).mock.calls.length).toBe( + workflowReadsBeforeGet + 1, + ); + expect( + await topology.consumeWorkflowRemoteStepDispatch(owner, permit), + ).toEqual({ kind: 'accepted', rawPermit: 'winner-secret' }); + expect( + await topology.consumeWorkflowRemoteStepDispatch(owner, permit), + ).toEqual({ kind: 'already-consumed' }); + expect(reader.consumePermit).toHaveBeenNthCalledWith(1, permit); + expect(reader.consumePermit).toHaveBeenNthCalledWith(2, permit); + expect( + await topology.recordWorkflowRemoteStepResult( + owner, + invocation.invocationId, + 'winner-secret', + '{"answer":42}', + '{"quiescent":true}', + ), + ).toMatchObject({ kind: 'recorded', resultJson: '{"answer":42}' }); + expect(reader.recordResult).toHaveBeenCalledWith( + invocation.invocationId, + 'winner-secret', + '{"answer":42}', + '{"quiescent":true}', + ); + const otherPrincipal = { + ...owner, + principal: { + kind: 'human' as const, + id: 'other', + role: 'operator' as const, + }, + }; + await expect( + topology.issueWorkflowRemoteStepDispatch( + otherPrincipal, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.getWorkflowRemoteStep(otherPrincipal, invocation.invocationId), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.consumeWorkflowRemoteStepDispatch(otherPrincipal, permit), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.recordWorkflowRemoteStepResult( + otherPrincipal, + invocation.invocationId, + 'winner-secret', + '{"answer":42}', + '{"quiescent":true}', + ), + ).rejects.toMatchObject({ status: 404 }); + const deniedRun = { ...owner, canAccessResource: async () => false }; + await expect( + topology.registerWorkflowRemoteStep( + deniedRun, + invocation.workflowId, + invocation.runId, + 10_000, + ), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.issueWorkflowRemoteStepDispatch( + deniedRun, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.getWorkflowRemoteStep(deniedRun, invocation.invocationId), + ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.getWorkflow) + .mockResolvedValueOnce(dispatched) + .mockResolvedValueOnce({ + ...dispatched, + permitGeneration: permit.generation + 1, + }); + await expect( + topology.issueWorkflowRemoteStepDispatch( + owner, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 409 }); + vi.mocked(reader.getWorkflow).mockResolvedValueOnce({ + ...dispatched, + deploymentId: 'another-deployment', + }); + await expect( + topology.issueWorkflowRemoteStepDispatch( + owner, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.getWorkflow).mockResolvedValueOnce({ + ...dispatched, + deploymentId: 'another-deployment', + }); + await expect( + topology.getWorkflowRemoteStep(owner, invocation.invocationId), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.issuePermit).toHaveBeenCalledTimes(2); + expect(reader.consumePermit).toHaveBeenCalledTimes(2); + expect(reader.recordResult).toHaveBeenCalledTimes(1); +}); + +it('issues a remote dispatch ticket and exposes a raw permit to the consume winner', async () => { + const permit = { + invocationId: 'invocation_1', + generation: 2, + mutationEpoch: 7, + transitionRevision: 4, + expiresAt: 9_999_999_999_999, + }; + const invocation = { + invocationId: permit.invocationId, + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'acme_resource_acme_thread', + runId: 'acme_run', + principal: context().value.principal, + startToken: 'private-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: { script: 'script-1' }, + deadlineAt: permit.expiresAt, + status: 'dispatched' as const, + permitGeneration: permit.generation, + }; + const reader: RemoteStepDispatchReader = { + deploymentId: invocation.deploymentId, + tablePrefix: invocation.tablePrefix, + get: vi.fn(async () => invocation), + readDispatchRequestJson: vi.fn(async () => '{"script":"script-1"}'), + issuePermit: vi.fn(async () => permit), + consumePermit: vi + .fn() + .mockResolvedValueOnce({ kind: 'accepted', rawPermit: 'secret' }) + .mockResolvedValueOnce({ kind: 'already-consumed' }), + }; + const { topology } = harness(undefined, reader); + const ticket = await topology.issueRemoteStepDispatch( + context().value, + permit.invocationId, + 60_000, + ); + expect(ticket).toMatchObject({ + invocationId: permit.invocationId, + requestJson: '{"script":"script-1"}', + permit, + }); + expect(ticket).not.toHaveProperty('principal'); + expect(ticket).not.toHaveProperty('startToken'); + expect(reader.issuePermit).toHaveBeenCalledWith(permit.invocationId, 60_000); + expect( + await topology.consumeRemoteStepDispatch(context().value, permit), + ).toEqual({ kind: 'accepted', rawPermit: 'secret' }); + expect( + await topology.consumeRemoteStepDispatch(context().value, permit), + ).toEqual({ kind: 'already-consumed' }); + const displaced = context().value; + await expect( + topology.issueRemoteStepDispatch( + { + ...displaced, + principal: { kind: 'human', id: 'other', role: 'operator' }, + }, + permit.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.issuePermit).toHaveBeenCalledTimes(1); + await expect( + topology.consumeRemoteStepDispatch( + { + ...displaced, + principal: { kind: 'human', id: 'other', role: 'operator' }, + }, + permit, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.consumePermit).toHaveBeenCalledTimes(2); + const deniedAccess = { + ...displaced, + canAccessResource: async () => false, + }; + await expect( + topology.issueRemoteStepDispatch(deniedAccess, permit.invocationId, 60_000), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.consumeRemoteStepDispatch(deniedAccess, permit), + ).rejects.toMatchObject({ status: 404 }); + const displacedResource = { + ...displaced, + resourceIdFromKey: () => 'another-resource', + }; + await expect( + topology.issueRemoteStepDispatch( + displacedResource, + permit.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.get).mockResolvedValueOnce({ + ...invocation, + deploymentId: 'other-deployment', + }); + await expect( + topology.issueRemoteStepDispatch(displaced, permit.invocationId, 60_000), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.issuePermit).toHaveBeenCalledTimes(1); + expect(reader.readDispatchRequestJson).toHaveBeenCalledTimes(1); + expect(reader.consumePermit).toHaveBeenCalledTimes(2); +}); + +it('routes a retained result under the journaled principal and refuses a changed resource mapping', async () => { + const principal = { + kind: 'human' as const, + id: 'original', + role: 'operator' as const, + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: vi.fn(async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'acme_resource_acme_thread', + runId: 'acme_run', + principal, + startToken: 'generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained' as const, + permitGeneration: 1, + })), + readRetainedResult: vi.fn(async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + })), + claimResume: vi.fn(async () => 'lease-1'), + settleAfterResume: vi.fn(async () => 'applied' as const), + }; + const { topology, hits } = harness(reader); + await topology.resumeRemoteStep(context().value, 'invocation_1'); + const sent = hits[0]; + expect(sent?.url).toContain('/_flowsafe/agent-host/remote-step-resume'); + expect(JSON.parse(sent?.init?.body ?? '{}')).toEqual({ + invocationId: 'invocation_1', + }); + expect(sent?.init?.headers?.['x-flowsafe-principal']).toBe( + '{"kind":"human","id":"original","role":"operator"}', + ); + const stored = await reader.get('invocation_1'); + if (!stored) throw new Error('invocation was not found'); + vi.mocked(reader.get).mockResolvedValue({ + ...stored, + resourceId: 'displaced-resource', + }); + await expect( + topology.resumeRemoteStep(context().value, 'invocation_1'), + ).rejects.toMatchObject({ status: 404 }); + expect(hits).toHaveLength(1); + vi.mocked(reader.get).mockResolvedValue({ + ...stored, + status: 'resuming', + resumeLeaseExpiresAt: Date.now() - 1, + }); + await expect( + topology.resumeRemoteStep(context().value, 'invocation_1'), + ).rejects.toMatchObject({ status: 503 }); + expect(hits).toHaveLength(1); +}); + function context() { let runMints = 0; let threadMints = 0; diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 23cb8b43..8d47a7dc 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -11,6 +11,7 @@ import { import { isExecutionPrincipalId, isExecutionPrincipalKind, + samePrincipal, } from '../approval-api/principal.js'; import { APPROVAL_PRIORITIES, @@ -30,10 +31,18 @@ import { type StartReservation, type StartReservationReading, } from '../do-runner/index.js'; +import type { + RecordedRemoteStepResult, + RemoteStepInvocation, + RemoteStepJournal, + RemoteStepPermit, + WorkflowRemoteStepInvocation, +} from '../do-runner/remote-step-journal.js'; import { isTerminalRunStatus, type RunStatus, } from '../do-runner/run-terminal-state.js'; +import type { RunSummary } from '../do-runner/runtime.js'; import { captureReservation, type StartReservationOwner, @@ -120,6 +129,88 @@ export interface AgentThreadTopologyOptions { * for a host with no fence. */ executionFence: ExecutionFenceWiring; + remoteSteps?: RemoteStepContinuationReader; + remoteStepDispatch?: RemoteStepDispatchReader; + workflowRemoteStepDispatch?: WorkflowRemoteStepDispatchReader; +} + +export type WorkflowRemoteStepDispatchReader = Pick< + RemoteStepJournal, + | 'deploymentId' + | 'tablePrefix' + | 'registerWorkflow' + | 'getWorkflow' + | 'readWorkflowDispatchRequestJson' + | 'issuePermit' + | 'consumePermit' + | 'recordResult' +>; + +export interface WorkflowRemoteStepDispatchTicket { + invocationId: string; + deploymentId: string; + workflowId: string; + runId: string; + stepPath: readonly string[]; + suspendedAt: number; + resumeCount?: number; + deadlineAt: number; + requestJson: string; + requestDigest: string; + permit: RemoteStepPermit; +} + +export interface RemoteStepDispatchReader { + deploymentId: string; + tablePrefix: string; + get(invocationId: string): Promise; + readDispatchRequestJson(invocationId: string): Promise; + issuePermit( + invocationId: string, + lifetimeMs: number, + ): Promise; + consumePermit( + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + >; +} + +export interface RemoteStepDispatchTicket { + invocationId: string; + deploymentId: string; + workflowId: string; + agentId: string; + threadId: string; + resourceId: string; + runId: string; + stepPath: readonly string[]; + suspendedAt: number; + resumeCount?: number; + deadlineAt: number; + requestJson: string; + permit: RemoteStepPermit; +} + +export interface RemoteStepContinuationReader { + deploymentId: string; + tablePrefix: string; + get(invocationId: string): Promise; + readRetainedResult( + invocationId: string, + ): Promise; + claimResume(invocationId: string, leaseMs: number): Promise; + settleAfterResume( + invocationId: string, + leaseOwner: string, + witnessed?: RunSummary, + ): Promise< + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof' + >; } export interface AgentThreadRunRef { @@ -181,6 +272,49 @@ export interface AgentThreadDispatchTopology extends AgentThreadBoundTopology { context: ActorContext, input: AgentThreadRunRef, ): Promise; + resumeRemoteStep( + context: ActorContext, + invocationId: string, + ): Promise; + issueRemoteStepDispatch( + context: ActorContext, + invocationId: string, + lifetimeMs: number, + ): Promise; + consumeRemoteStepDispatch( + context: ActorContext, + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + >; + registerWorkflowRemoteStep( + context: ActorContext, + workflowId: string, + runId: string, + hostDeadlineAt: number, + ): Promise; + getWorkflowRemoteStep( + context: ActorContext, + invocationId: string, + ): Promise; + issueWorkflowRemoteStepDispatch( + context: ActorContext, + invocationId: string, + lifetimeMs: number, + ): Promise; + consumeWorkflowRemoteStepDispatch( + context: ActorContext, + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + >; + recordWorkflowRemoteStepResult( + context: ActorContext, + invocationId: string, + rawPermit: string, + resultJson: string, + evidenceJson: string, + ): ReturnType; } function errorFrom(response: Response): Promise { @@ -230,6 +364,75 @@ export function createAgentThreadTopology( namespace, deploymentIdentitySecret, ); + const dispatchInvocation = async ( + context: ActorContext, + invocationId: string, + ) => { + const reader = options.remoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.get(invocationId); + if ( + !invocation || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + !samePrincipal(invocation.principal, context.principal) + ) + throw new RunRouteError(404, 'invocation not found'); + expectedResource(context, invocation.threadId, invocation.resourceId); + await requireResourceAccess( + context, + 'thread', + invocation.threadId, + 'write', + 'run', + ); + await requireResourceAccess( + context, + 'resource', + invocation.resourceId, + 'write', + 'run', + ); + await requireResourceAccess( + context, + 'run', + invocation.runId, + 'write', + 'run', + ); + return { reader, invocation }; + }; + const workflowDispatchInvocation = async ( + context: ActorContext, + invocationId: string, + access: 'read' | 'write', + loadedInvocation?: WorkflowRemoteStepInvocation, + ) => { + const reader = options.workflowRemoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = + loadedInvocation ?? (await reader.getWorkflow(invocationId)); + if ( + invocation?.targetKind !== 'workflow' || + invocation.invocationId !== invocationId || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + !samePrincipal(invocation.principal, context.principal) + ) + throw new RunRouteError(404, 'invocation not found'); + await requireResourceAccess( + context, + 'run', + invocation.runId, + access, + 'run', + ); + return { reader, invocation }; + }; const fetchStatus = async ( context: ActorContext, input: AgentThreadRunRef, @@ -386,6 +589,147 @@ export function createAgentThreadTopology( } }; return { + registerWorkflowRemoteStep: async ( + context, + workflowId, + runId, + hostDeadlineAt, + ) => { + const reader = options.workflowRemoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(workflowId)) + throw new RunRouteError(404, 'workflow not found'); + await requireResourceAccess(context, 'run', runId, 'write', 'run'); + return reader.registerWorkflow( + { + targetKind: 'workflow', + deploymentId: reader.deploymentId, + tablePrefix: reader.tablePrefix, + workflowId, + runId, + }, + hostDeadlineAt, + context.principal, + ); + }, + getWorkflowRemoteStep: async (context, invocationId) => { + const reader = options.workflowRemoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.getWorkflow(invocationId); + if (!invocation) return undefined; + return ( + await workflowDispatchInvocation( + context, + invocationId, + 'read', + invocation, + ) + ).invocation; + }, + issueWorkflowRemoteStepDispatch: async ( + context, + invocationId, + lifetimeMs, + ) => { + const { reader } = await workflowDispatchInvocation( + context, + invocationId, + 'write', + ); + const permit = await reader.issuePermit(invocationId, lifetimeMs); + const { invocation } = await workflowDispatchInvocation( + context, + invocationId, + 'write', + ); + if ( + invocation.status !== 'dispatched' || + invocation.permitGeneration !== permit.generation || + invocation.deadlineAt < permit.expiresAt + ) + throw new RunRouteError(409, 'remote step dispatch changed'); + const requestJson = + await reader.readWorkflowDispatchRequestJson(invocationId); + return { + invocationId, + deploymentId: invocation.deploymentId, + workflowId: invocation.workflowId, + runId: invocation.runId, + stepPath: invocation.stepPath, + suspendedAt: invocation.suspendedAt, + ...(invocation.resumeCount === undefined + ? {} + : { resumeCount: invocation.resumeCount }), + deadlineAt: invocation.deadlineAt, + requestJson, + requestDigest: invocation.requestDigest, + permit, + }; + }, + consumeWorkflowRemoteStepDispatch: async (context, permit) => { + const { reader } = await workflowDispatchInvocation( + context, + permit.invocationId, + 'write', + ); + return reader.consumePermit(permit); + }, + recordWorkflowRemoteStepResult: async ( + context, + invocationId, + rawPermit, + resultJson, + evidenceJson, + ) => { + const { reader } = await workflowDispatchInvocation( + context, + invocationId, + 'write', + ); + return reader.recordResult( + invocationId, + rawPermit, + resultJson, + evidenceJson, + ); + }, + issueRemoteStepDispatch: async (context, invocationId, lifetimeMs) => { + await dispatchInvocation(context, invocationId); + const reader = options.remoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + const permit = await reader.issuePermit(invocationId, lifetimeMs); + const { invocation } = await dispatchInvocation(context, invocationId); + if ( + invocation.status !== 'dispatched' || + invocation.permitGeneration !== permit.generation || + invocation.deadlineAt < permit.expiresAt + ) + throw new RunRouteError(409, 'remote step dispatch changed'); + const requestJson = await reader.readDispatchRequestJson(invocationId); + return { + invocationId, + deploymentId: invocation.deploymentId, + workflowId: invocation.workflowId, + agentId: invocation.agentId, + threadId: invocation.threadId, + resourceId: invocation.resourceId, + runId: invocation.runId, + stepPath: invocation.stepPath, + suspendedAt: invocation.suspendedAt, + ...(invocation.resumeCount === undefined + ? {} + : { resumeCount: invocation.resumeCount }), + deadlineAt: invocation.deadlineAt, + requestJson, + permit, + }; + }, + consumeRemoteStepDispatch: async (context, permit) => { + const { reader } = await dispatchInvocation(context, permit.invocationId); + return reader.consumePermit(permit); + }, requireBoundThread: async (context, target: BoundThreadTarget) => { const threadId = validThread(target.threadId); const resourceId = expectedResource(context, threadId, target.resourceId); @@ -737,6 +1081,118 @@ export function createAgentThreadTopology( ), ); }, + resumeRemoteStep: async (context, invocationId) => { + const reader = options.remoteSteps; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.get(invocationId); + if (invocation?.status === 'resuming') + throw new RunRouteError( + 503, + 'remote step resume awaits executor quiescence', + ); + if ( + invocation?.status !== 'result-retained' || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + !isPathSafeId(invocation.agentId) || + !isPathSafeId(invocation.runId) || + !(await reader.readRetainedResult(invocationId)) + ) + throw new RunRouteError(404, 'invocation not found'); + expectedResource(context, invocation.threadId, invocation.resourceId); + return envelope( + await threads.send( + { + principal: invocation.principal, + mutationEpoch: context.mutationEpoch, + }, + invocation.threadId, + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-resume`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ invocationId }), + }, + ), + ); + }, + }; +} + +export function createRemoteStepNotificationRecoveryTick(options: { + journal: RemoteStepJournal; + namespace: ThreadNamespaceLike; + deploymentIdentitySecret: string; + limit?: number; +}): () => Promise<{ scanned: number; recovered: number; unresolved: number }> { + const limit = options.limit ?? 16; + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) + throw new TypeError('notification recovery limit is malformed'); + const threads = createThreadTopology( + options.namespace, + options.deploymentIdentitySecret, + ); + return async () => { + const ids = await options.journal.nextNotificationRecoveryBatch(limit); + let recovered = 0; + let unresolved = 0; + let firstFailure: { error: unknown } | undefined; + for (const invocationId of ids) { + try { + const invocation = await options.journal.get(invocationId); + if ( + !invocation || + !['result-retained', 'resuming'].includes(invocation.status) + ) + continue; + if ( + invocation.invocationId !== invocationId || + invocation.deploymentId !== options.journal.deploymentId || + invocation.tablePrefix !== options.journal.tablePrefix + ) + throw new RunRouteError(409, 'remote step recovery identity changed'); + const operation = + await options.journal.readNotificationOperation(invocationId); + if (!operation || operation.invocationId !== invocationId) + throw new RunRouteError( + 409, + 'remote step recovery operation changed', + ); + if (operation.state !== 'effect-admitted') { + unresolved += 1; + continue; + } + const response = await threads.send( + { principal: invocation.principal }, + invocation.threadId, + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-notification-recover`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ invocationId }), + }, + ); + if (!response.ok) + throw new RunRouteError( + response.status, + 'remote step notification recovery failed', + ); + const after = + await options.journal.readNotificationOperation(invocationId); + if (after?.state !== 'receipt-recorded') + throw new RunRouteError( + 503, + 'remote step notification receipt is unresolved', + ); + recovered += 1; + } catch (error) { + firstFailure ??= { error }; + } + } + if (firstFailure) throw firstFailure.error; + return { scanned: ids.length, recovered, unresolved }; }; } diff --git a/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts b/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts new file mode 100644 index 00000000..e1a7d48a --- /dev/null +++ b/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts @@ -0,0 +1,193 @@ +// SPDX-License-Identifier: Apache-2.0 + +import { doErrorResponse } from '../do-runner/do-error-response.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; +import type { RemoteStepPermit } from '../do-runner/remote-step-journal.js'; +import type { + FlowsafeWorkerConfig, + FlowsafeWorkerEnv, +} from '../host-kit/flowsafe-worker.js'; +import { + RunRouteError, + runRouteErrorBody, +} from '../host-kit/run-route-error.js'; +import { readBoundedBody } from '../http-body.js'; +import type { + AgentThreadDispatchTopology, + WorkflowRemoteStepDispatchReader, +} from './thread-topology.js'; + +export const WORKFLOW_REMOTE_STEP_ROUTE = '/_flowsafe/workflow-remote-step'; + +export type WorkflowDispatchPort = Pick< + AgentThreadDispatchTopology, + | 'registerWorkflowRemoteStep' + | 'issueWorkflowRemoteStepDispatch' + | 'consumeWorkflowRemoteStepDispatch' + | 'recordWorkflowRemoteStepResult' +>; + +type Operation = 'register' | 'ticket' | 'consume' | 'result'; + +const fields: Record = { + register: ['workflowId', 'runId', 'hostDeadlineAt'], + ticket: ['invocationId', 'lifetimeMs'], + consume: ['permit'], + result: ['invocationId', 'rawPermit', 'resultJson', 'evidenceJson'], +}; + +function json(value: unknown, status = 200): Response { + return Response.json(value, { + status, + headers: { 'cache-control': 'no-store' }, + }); +} + +function exactBody( + value: unknown, + operation: Operation, +): Record | undefined { + if (value === null || typeof value !== 'object' || Array.isArray(value)) + return undefined; + const body = value as Record; + const keys = Object.keys(body); + const expected = fields[operation]; + return keys.length === expected.length && + keys.every((key) => expected.includes(key)) + ? body + : undefined; +} + +function permit(value: unknown): RemoteStepPermit | undefined { + if (value === null || typeof value !== 'object' || Array.isArray(value)) + return undefined; + const candidate = value as Record; + if ( + Object.keys(candidate).sort().join(',') !== + 'expiresAt,generation,invocationId,mutationEpoch,transitionRevision' || + typeof candidate.invocationId !== 'string' || + !isPathSafeId(candidate.invocationId) || + !['generation', 'mutationEpoch', 'transitionRevision', 'expiresAt'].every( + (key) => + Number.isSafeInteger(candidate[key]) && (candidate[key] as number) >= 0, + ) + ) + return undefined; + return candidate as unknown as RemoteStepPermit; +} + +/** A host mounts this route through preRoutes on its private state-script Worker. */ +export function createWorkflowRemoteStepPreRoute( + journal: WorkflowRemoteStepDispatchReader, + dispatch: WorkflowDispatchPort, +): NonNullable['preRoutes']> { + return async (request, _env, _ctx, kit) => { + const url = new URL(request.url); + if (!url.pathname.startsWith(`${WORKFLOW_REMOTE_STEP_ROUTE}/`)) return null; + const operation = url.pathname.slice(WORKFLOW_REMOTE_STEP_ROUTE.length + 1); + if (!Object.hasOwn(fields, operation) || url.search) + return json({ error: 'not found' }, 404); + if (request.method !== 'POST') + return json({ error: 'method not allowed' }, 405); + const unauthorized = await kit.authorizePrivateWorkflowDispatch(request); + if (unauthorized) return unauthorized; + const raw = await readBoundedBody(request, 65_536); + if (!raw.ok) + return json( + { error: 'invalid request body' }, + raw.reason === 'payload-too-large' ? 413 : 400, + ); + let parsed: unknown; + try { + parsed = JSON.parse(raw.text); + } catch { + return json({ error: 'invalid request body' }, 400); + } + const body = exactBody(parsed, operation as Operation); + if (!body) return json({ error: 'invalid request body' }, 400); + try { + if (operation === 'register') { + if ( + typeof body.workflowId !== 'string' || + !isPathSafeId(body.workflowId) || + typeof body.runId !== 'string' || + !isPathSafeId(body.runId) || + !Number.isSafeInteger(body.hostDeadlineAt) || + (body.hostDeadlineAt as number) <= 0 + ) + return json({ error: 'invalid request body' }, 400); + const context = await kit.contextForStoredWorkflowStart( + body.workflowId, + body.runId, + ); + if (!context) return json({ error: 'run not found' }, 404); + const invocation = await dispatch.registerWorkflowRemoteStep( + context, + body.workflowId, + body.runId, + body.hostDeadlineAt as number, + ); + return json({ invocationId: invocation.invocationId }); + } + const invocationId = + operation === 'consume' + ? permit(body.permit)?.invocationId + : body.invocationId; + if (typeof invocationId !== 'string' || !isPathSafeId(invocationId)) + return json({ error: 'invalid request body' }, 400); + const context = await kit.contextForStoredWorkflowInvocation( + journal, + invocationId, + ); + if (!context) return json({ error: 'invocation not found' }, 404); + if (operation === 'ticket') { + if ( + !Number.isSafeInteger(body.lifetimeMs) || + (body.lifetimeMs as number) <= 0 + ) + return json({ error: 'invalid request body' }, 400); + return json( + await dispatch.issueWorkflowRemoteStepDispatch( + context, + invocationId, + body.lifetimeMs as number, + ), + ); + } + if (operation === 'consume') { + const submitted = permit(body.permit); + if (!submitted) return json({ error: 'invalid request body' }, 400); + return json( + await dispatch.consumeWorkflowRemoteStepDispatch(context, submitted), + ); + } + if ( + typeof body.rawPermit !== 'string' || + typeof body.resultJson !== 'string' || + typeof body.evidenceJson !== 'string' + ) + return json({ error: 'invalid request body' }, 400); + return json( + await dispatch.recordWorkflowRemoteStepResult( + context, + invocationId, + body.rawPermit, + body.resultJson, + body.evidenceJson, + ), + ); + } catch (error) { + if (error instanceof RunRouteError) { + const refusal = runRouteErrorBody(error); + return refusal + ? json(refusal, error.status) + : json({ error: 'private workflow dispatch failed' }, 500); + } + const response = doErrorResponse(error); + if (response.status === 500) + return json({ error: 'private workflow dispatch failed' }, 500); + response.headers.set('cache-control', 'no-store'); + return response; + } + }; +} diff --git a/packages/flowsafe/src/agent-runner/agent-run-state.ts b/packages/flowsafe/src/agent-runner/agent-run-state.ts index e878045d..0ab65c24 100644 --- a/packages/flowsafe/src/agent-runner/agent-run-state.ts +++ b/packages/flowsafe/src/agent-runner/agent-run-state.ts @@ -15,6 +15,7 @@ export const AGENT_RUN_STORAGE_KEY_PREFIX = 'flowsafe:agent-run:v1:'; export const AGENT_ENTRY_PATHS = [ 'http.start', 'approval.resume', + 'remote-step.resume', 'signal.message', 'signal.queue', 'signal.reactive', diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts index be88ba1e..ecdef4de 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts @@ -156,6 +156,7 @@ function fakeRuntime( workflowIds, start, resume, + authoritativeStatus: async () => null, ...(overrides.pubsub !== undefined ? { pubsub: overrides.pubsub } : {}), } as unknown as RunnerRuntime; return { @@ -642,9 +643,223 @@ async function realAgentBridgeFixture( : {}), }); const start = vi.spyOn(runtime, 'start'); - return { sql, fence, workflows, counts, runtime, agent, start }; + return { sql, binding, fence, workflows, counts, runtime, agent, start }; } +describe('durable tool suspension after host restart', () => { + it.each([ + ['object', { key: 'account-7' }], + ['scalar', 'account-7'], + ['array', ['account-7']], + ])('restores a %s tool payload and applies the result to the suspended call', async (_shape, suspendedPayload) => { + const runId = 'remote-tool-resume'; + const toolId = 'remote_lookup'; + const received: unknown[] = []; + const tool = createTool({ + id: toolId, + description: 'Wait for a remote lookup result', + inputSchema: z.object({ key: z.string() }), + suspendSchema: z.unknown(), + resumeSchema: z.object({ value: z.string() }), + execute: async ({ key }, context) => { + const result = context?.agent?.resumeData; + if (result) { + received.push({ + key, + result, + payload: context.agent?.suspendPayload, + }); + return { value: result.value }; + } + if (!context?.agent?.suspend) + throw new Error('tool suspension unavailable'); + await context.agent.suspend(suspendedPayload); + }, + }); + const text = localModelFixture(() => undefined); + const textStream = ( + text as unknown as { + doStream: (options: { abortSignal?: AbortSignal }) => Promise; + } + ).doStream; + let modelCalls = 0; + const model = { + ...(text as object), + doStream: async (options: Parameters[0]) => { + modelCalls++; + if (modelCalls > 2) return textStream(options); + return { + stream: new ReadableStream({ + start(controller) { + controller.enqueue({ type: 'stream-start', warnings: [] }); + controller.enqueue({ + type: 'tool-call', + toolCallId: `remote-call-${modelCalls}`, + toolName: toolId, + input: JSON.stringify({ key: 'account-7' }), + }); + controller.enqueue({ + type: 'finish', + finishReason: 'tool-calls', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + }); + controller.close(); + }, + }), + }; + }, + } as unknown as MastraModelConfig; + const first = await realAgentBridgeFixture( + undefined, + undefined, + false, + undefined, + { model, tools: { [toolId]: tool } }, + ); + let output: + | Awaited> + | undefined; + try { + output = await first.agent.streamUntilPersisted( + 'Look up account-7.', + { runId, maxSteps: 3, disableBackgroundTasks: true }, + 'operator-1', + 'human', + `${runId}-attempt`, + undefined, + undefined, + startAuthority(), + ); + const workflowId = first.agent.getWorkflow().id; + const suspended = await first.runtime.authoritativeStatus( + workflowId, + runId, + ); + expect(suspended).toMatchObject({ status: 'suspended' }); + const step = suspended?.suspended?.[0]; + if (!step) throw new Error('tool did not suspend'); + const stepKey = step.join('.'); + expect( + (suspended?.suspendPayload as Record | undefined)?.[ + stepKey + ], + ).toMatchObject({ + type: 'suspension', + toolCallId: 'remote-call-1', + toolCallSuspended: suspendedPayload, + }); + expect(received).toEqual([]); + + output.cleanup(); + registryFor(first.agent).clear(); + globalRunRegistry.delete(runId); + const storage = createD1Storage({ binding: first.binding }); + await storage.init(); + const { runtime } = init( + { storage }, + { + executionFence: first.fence, + startIdempotency: 'none', + pubsub: createHostPubSub(), + }, + ); + const restored = createFlowsafeDurableAgent({ + agent: new Agent({ + id: 'writer', + name: 'Writer', + instructions: 'Return done.', + model, + tools: { [toolId]: tool }, + }), + runtime, + cache: false, + maxSteps: 1, + }); + const suspendedAt = suspended.suspendedAt?.[stepKey]; + if (typeof suspendedAt !== 'number') { + throw new Error('tool suspension has no timestamp'); + } + const firstResume = { + runId, + requestedBy: 'operator-1', + step, + expectedSuspension: { + suspendedAt, + ...(suspended.resumeCount?.[stepKey] === undefined + ? {} + : { resumeCount: suspended.resumeCount[stepKey] }), + }, + }; + await expect( + restored.resumeViaRuntime({ + ...firstResume, + resumeData: { value: 7 }, + }), + ).rejects.toThrow(); + expect(received).toEqual([]); + expect( + await runtime.authoritativeStatus(workflowId, runId), + ).toMatchObject({ + status: 'suspended', + }); + const resumed = await restored.resumeViaRuntime({ + ...firstResume, + resumeData: { value: 'found' }, + }); + expect(resumed.status).toBe('suspended'); + const nextStep = resumed.suspended?.[0]; + if (!nextStep) throw new Error('second tool call did not suspend'); + const nextKey = nextStep.join('.'); + expect( + (resumed.suspendPayload as Record | undefined)?.[ + nextKey + ], + ).toMatchObject({ + type: 'suspension', + toolCallId: 'remote-call-2', + toolCallSuspended: suspendedPayload, + }); + const nextTimestamp = resumed.suspendedAt?.[nextKey]; + if (typeof nextTimestamp !== 'number') { + throw new Error('second tool suspension has no timestamp'); + } + const settled = await restored.resumeViaRuntime({ + runId, + requestedBy: 'operator-1', + step: nextStep, + resumeData: { value: 'again' }, + expectedSuspension: { + suspendedAt: nextTimestamp, + ...(resumed.resumeCount?.[nextKey] === undefined + ? {} + : { resumeCount: resumed.resumeCount[nextKey] }), + }, + }); + expect(settled.status).toBe('success'); + expect(received).toEqual([ + { + key: 'account-7', + result: { value: 'found' }, + payload: suspendedPayload, + }, + { + key: 'account-7', + result: { value: 'again' }, + payload: suspendedPayload, + }, + ]); + registryFor(restored).clear(); + globalRunRegistry.delete(runId); + } finally { + output?.cleanup(); + registryFor(first.agent).clear(); + globalRunRegistry.delete(runId); + first.start.mockRestore(); + first.sql.close(); + } + }); +}); + const OUTCOME_MARKER = 'MKCLIENTOUTCOME'; const CLEAN_OUTCOME = 'clean client result'; const APPROVAL = { id: 'approval_1' }; @@ -3743,6 +3958,32 @@ describe('FlowsafeDurableAgent thread runtime registration and rehydration', () expect(agent.getPubSub()).toBe(pubsub); }); + it('passes an explicit automated requester kind to the runtime resume', async () => { + const { runtime, resume } = fakeRuntime(); + const agent = createFlowsafeDurableAgent({ agent: testAgent(), runtime }); + await agent.resumeViaRuntime({ + runId: 'run-1', + requestedBy: 'service-1', + requestedByKind: 'service', + }); + expect(resume).toHaveBeenCalledWith( + expect.anything(), + 'run-1', + expect.objectContaining({ + requestedBy: 'service-1', + requestedByKind: 'service', + }), + ); + await expect( + agent.resumeViaRuntime({ + runId: 'run-2', + requestedBy: 'service-1', + requestedByKind: 'forged' as never, + }), + ).rejects.toBeInstanceOf(InvalidRunRequestError); + expect(resume).toHaveBeenCalledTimes(1); + }); + it('resumes observation after retained earlier events on the observer bus', async () => { const pubsub = new EventEmitterPubSub(); const { runtime, resume } = fakeRuntime({ diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts index 4b8a0f1c..bd473311 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts @@ -2022,12 +2022,17 @@ export class FlowsafeDurableAgent< async resumeViaRuntime(options: { runId: string; requestedBy: string; + requestedByKind?: ExecutionPrincipalKind; step?: string | string[]; resumeData?: unknown; memory?: DurableAgentStreamOptions['memory']; expectedSuspension?: ResumeRunOptions['expectedSuspension']; }): Promise { this.#assertCallerRunId(options.runId); + const requestedByKind = + options.requestedByKind === undefined ? 'human' : options.requestedByKind; + if (!isExecutionPrincipalKind(requestedByKind)) + throw new InvalidRunRequestError('requestedByKind is malformed'); const memory = this.#guardedCallOptionMapper ? (this.#guardedCallOptionMapper( 'memory', @@ -2037,53 +2042,109 @@ export class FlowsafeDurableAgent< let rehydrated = false; let leg: ResumedThreadLeg | undefined; const legAbort = new AbortController(); + const resumeOptions: ResumeRunOptions = { + ...(options.step !== undefined ? { step: options.step } : {}), + ...(options.resumeData !== undefined + ? { resumeData: options.resumeData } + : {}), + ...(options.expectedSuspension !== undefined + ? { expectedSuspension: options.expectedSuspension } + : {}), + requestedBy: options.requestedBy, + requestedByKind, + legAbort, + prepareExecution: async (requestContext, workflowInput) => { + const current = await this.#runtime.authoritativeStatus( + this.getWorkflow().id, + options.runId, + ); + const step = Array.isArray(options.step) + ? options.step.join('.') + : (options.step ?? + (current?.suspended?.length === 1 + ? current.suspended[0]?.join('.') + : undefined)); + const payload = isRecord(current?.suspendPayload) + ? current.suspendPayload + : undefined; + const suspension = + step && isRecord(payload?.[step]) ? payload[step] : undefined; + if (suspension?.type === 'suspension') { + const toolName = suspension.toolName; + if ( + typeof toolName !== 'string' || + typeof suspension.toolCallId !== 'string' + ) + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + const tools = (await this.listTools({ + requestContext, + })) as Record; + const tool = tools[toolName]; + const schema = isRecord(tool) ? tool.resumeSchema : undefined; + const standard = isRecord(schema) ? schema['~standard'] : undefined; + if (!isRecord(standard) || typeof standard.validate !== 'function') + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + const validated: unknown = await standard.validate( + options.resumeData, + ); + if (!isRecord(validated)) + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + if (validated.issues !== undefined) + throw new InvalidRunRequestError( + 'resume data does not match suspended tool', + ); + if (!Object.hasOwn(validated, 'value')) + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + resumeOptions.resumeData = validated.value; + } + const offset = ( + await this.pubsub.getHistory(AGENT_STREAM_TOPIC(options.runId)) + ).length; + const modelSettings = persistedModelSettings(workflowInput); + await this.#rehydrateRegistry({ + runId: options.runId, + requestContext, + abortController: legAbort, + ...(memory !== undefined ? { memory } : {}), + ...(modelSettings !== undefined ? { modelSettings } : {}), + }); + rehydrated = true; + const observed = await this.observe(options.runId, { offset }); + if (this.#threadRuntime) { + leg = this.#trackResumedLeg(options.runId); + await this.#threadRuntime.registerRun( + this as unknown as Parameters< + Mastra['agentThreadStreamRuntime']['registerRun'] + >[0], + bindResumedLegLifecycle(observed.output, leg), + { + runId: options.runId, + ...(memory !== undefined ? { memory } : {}), + } as Parameters< + Mastra['agentThreadStreamRuntime']['registerRun'] + >[2], + this.getPubSub(), + ); + } + }, + }; try { const summary = await this.#runtime.resume( this.getWorkflow().id, options.runId, - { - ...(options.step !== undefined ? { step: options.step } : {}), - ...(options.resumeData !== undefined - ? { resumeData: options.resumeData } - : {}), - ...(options.expectedSuspension !== undefined - ? { expectedSuspension: options.expectedSuspension } - : {}), - requestedBy: options.requestedBy, - requestedByKind: 'human', - legAbort, - prepareExecution: async (requestContext, workflowInput) => { - const offset = ( - await this.pubsub.getHistory(AGENT_STREAM_TOPIC(options.runId)) - ).length; - const modelSettings = persistedModelSettings(workflowInput); - await this.#rehydrateRegistry({ - runId: options.runId, - requestContext, - abortController: legAbort, - ...(memory !== undefined ? { memory } : {}), - ...(modelSettings !== undefined ? { modelSettings } : {}), - }); - rehydrated = true; - const observed = await this.observe(options.runId, { offset }); - if (this.#threadRuntime) { - leg = this.#trackResumedLeg(options.runId); - await this.#threadRuntime.registerRun( - this as unknown as Parameters< - Mastra['agentThreadStreamRuntime']['registerRun'] - >[0], - bindResumedLegLifecycle(observed.output, leg), - { - runId: options.runId, - ...(memory !== undefined ? { memory } : {}), - } as Parameters< - Mastra['agentThreadStreamRuntime']['registerRun'] - >[2], - this.getPubSub(), - ); - } - }, - }, + resumeOptions, ); const terminalError = legTerminalError( summary, @@ -2483,7 +2544,8 @@ export class FlowsafeDurableAgent< throw new Error('agent observation identity is malformed'); const observedAgentId = identity.target.id; const observedThreadId = identity.target.threadId; - const observedResourceId = resourceIdFromKey(observedThreadId); + const observedResourceId = + identity.target.resourceId ?? resourceIdFromKey(observedThreadId); const context = state.snapshot.requestContext; const record = (value: unknown): Record => { if (value === null || typeof value !== 'object' || Array.isArray(value)) diff --git a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts index 1b17de66..c9110211 100644 --- a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts +++ b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts @@ -13,12 +13,53 @@ import { readDeploymentIdentity, seedDeploymentIdentity, } from './do-runner/deployment-identity.js'; +import { D1NotificationsStorage } from './signals/notifications-d1.js'; interface TestBindings { DB: D1Database; + SIGNAL_PROVIDER_HOST: DurableObjectNamespace; } const db = (): D1Database => (env as unknown as TestBindings).DB; +const notificationRequestDigests = { + 'operation-1': + '69d0b9cae1fecec25e3fdad599bc1d4b441761843e4c399563c43b4f8e27699c', + 'operation-2': + '96bdec0f9c3160a3bdd190c24543168f4a4bcc5b7041507c996b042fc593f1dc', +} as const; + +function receiptActor(name: string) { + const namespace = (env as unknown as TestBindings).SIGNAL_PROVIDER_HOST; + return namespace.get(namespace.idFromName(name)); +} + +function createReceipt( + actor: ReturnType, + operationKey: string, + notificationId: string, + receiptLimit: number, + barrier?: { + id: string; + participant: string; + phase?: 'receipt-read' | 'scope-census'; + }, + tablePrefix = '', +) { + return actor.fetch( + new Request('http://localhost/receipt', { + method: 'POST', + body: JSON.stringify({ + operationKey, + notificationId, + receiptLimit, + tablePrefix, + barrierId: barrier?.id, + participant: barrier?.participant, + barrierPhase: barrier?.phase, + }), + }), + ); +} describe('FlowSafe Cloudflare fidelity', () => { beforeEach(reset); @@ -60,4 +101,256 @@ describe('FlowSafe Cloudflare fidelity', () => { expect(winners).toHaveLength(1); expect(await store.owner('run', 'run-cas')).toEqual(winners[0]?.owner); }); + + it('commits one notification receipt and effect when distinct durable objects submit the same operation', async () => { + await new D1NotificationsStorage(db()).init(); + await db() + .prepare(`CREATE TABLE notification_test_barrier ( + barrier_id TEXT NOT NULL, + participant TEXT NOT NULL, + PRIMARY KEY (barrier_id, participant) + )`) + .run(); + const left = receiptActor('receipt-left'); + const right = receiptActor('receipt-right'); + const responses = await Promise.all([ + createReceipt(left, 'operation-1', 'notification-1', 1, { + id: 'same-key', + participant: 'left', + }), + createReceipt(right, 'operation-1', 'notification-1', 1, { + id: 'same-key', + participant: 'right', + }), + ]); + + const ready = await db() + .prepare( + "SELECT participant FROM notification_test_barrier WHERE barrier_id = 'same-key' ORDER BY participant", + ) + .all<{ participant: string }>(); + expect(ready.results).toEqual([ + { participant: 'left' }, + { participant: 'right' }, + ]); + + expect(responses.map((response) => response.status)).toEqual([200, 200]); + expect( + await Promise.all(responses.map((response) => response.json())), + ).toEqual([ + { + operationKey: 'operation-1', + notificationId: 'notification-1', + threadId: 'thread', + requestDigest: notificationRequestDigests['operation-1'], + }, + { + operationKey: 'operation-1', + notificationId: 'notification-1', + threadId: 'thread', + requestDigest: notificationRequestDigests['operation-1'], + }, + ]); + const receipts = await db() + .prepare( + 'SELECT operation_key, notification_id FROM flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string; notification_id: string }>(); + expect(receipts.results).toEqual([ + { operation_key: 'operation-1', notification_id: 'notification-1' }, + ]); + const inbox = await db() + .prepare('SELECT id, thread_id FROM mastra_notifications') + .all<{ id: string; thread_id: string }>(); + expect(inbox.results).toEqual([ + { id: 'notification-1', thread_id: 'thread' }, + ]); + }); + + it('admits one distinct durable object operation at the last receipt slot', async () => { + await new D1NotificationsStorage(db()).init(); + await db() + .prepare(`CREATE TABLE notification_test_barrier ( + barrier_id TEXT NOT NULL, + participant TEXT NOT NULL, + PRIMARY KEY (barrier_id, participant) + )`) + .run(); + await db() + .prepare(`WITH RECURSIVE sequence(n) AS ( + VALUES (1) UNION ALL SELECT n + 1 FROM sequence WHERE n < 99999 + ) + INSERT INTO flowsafe_notification_operation_receipts + (operation_key, request_digest, thread_id, notification_id) + SELECT 'seed-' || n, 'seed', 'seed-thread', 'seed-' || n FROM sequence`) + .run(); + const left = receiptActor('slot-left'); + const right = receiptActor('slot-right'); + const responses = await Promise.all([ + createReceipt(left, 'operation-1', 'notification-1', 100_000, { + id: 'last-slot', + participant: 'left', + }), + createReceipt(right, 'operation-2', 'notification-2', 100_000, { + id: 'last-slot', + participant: 'right', + }), + ]); + + const ready = await db() + .prepare( + "SELECT participant FROM notification_test_barrier WHERE barrier_id = 'last-slot' ORDER BY participant", + ) + .all<{ participant: string }>(); + expect(ready.results).toEqual([ + { participant: 'left' }, + { participant: 'right' }, + ]); + + expect(responses.map((response) => response.status).sort()).toEqual([ + 200, 409, + ]); + const receipts = await db() + .prepare(`SELECT operation_key, notification_id, request_digest + FROM flowsafe_notification_operation_receipts + WHERE operation_key IN ('operation-1', 'operation-2')`) + .all<{ + operation_key: string; + notification_id: string; + request_digest: string; + }>(); + const inbox = await db() + .prepare('SELECT id FROM mastra_notifications') + .all<{ id: string }>(); + expect([ + [ + { + operation_key: 'operation-1', + notification_id: 'notification-1', + request_digest: notificationRequestDigests['operation-1'], + }, + ], + [ + { + operation_key: 'operation-2', + notification_id: 'notification-2', + request_digest: notificationRequestDigests['operation-2'], + }, + ], + ]).toContainEqual(receipts.results); + const winner = receipts.results[0]; + if (!winner) throw new Error('receipt is missing'); + expect(inbox.results).toEqual([{ id: winner.notification_id }]); + const count = await db() + .prepare( + 'SELECT COUNT(*) AS total FROM flowsafe_notification_operation_receipts', + ) + .first<{ total: number }>(); + expect(count?.total).toBe(100_000); + const loserResponse = responses.find((response) => response.status === 409); + expect(await loserResponse?.json()).toEqual({ + error: 'notification operation receipt cap reached', + }); + const replay = await createReceipt( + winner.operation_key === 'operation-1' ? right : left, + winner.operation_key, + winner.notification_id, + 100_000, + ); + expect(replay.status).toBe(200); + expect(await replay.json()).toEqual({ + operationKey: winner.operation_key, + notificationId: winner.notification_id, + threadId: 'thread', + requestDigest: winner.request_digest, + }); + }, 30_000); + + it('admits one receipt prefix across distinct durable objects', async () => { + await Promise.all([ + new D1NotificationsStorage(db(), 'left_').init(), + new D1NotificationsStorage(db(), 'right_').init(), + ]); + await db() + .prepare(`CREATE TABLE notification_test_barrier ( + barrier_id TEXT NOT NULL, + participant TEXT NOT NULL, + PRIMARY KEY (barrier_id, participant) + )`) + .run(); + const responses = await Promise.all([ + createReceipt( + receiptActor('prefix-left'), + 'operation-1', + 'notification-1', + 1, + { id: 'prefix-claim', participant: 'left', phase: 'scope-census' }, + 'left_', + ), + createReceipt( + receiptActor('prefix-right'), + 'operation-2', + 'notification-2', + 1, + { id: 'prefix-claim', participant: 'right', phase: 'scope-census' }, + 'right_', + ), + ]); + const ready = await db() + .prepare(`SELECT participant FROM notification_test_barrier + WHERE barrier_id = 'prefix-claim' ORDER BY participant`) + .all<{ participant: string }>(); + expect(ready.results).toEqual([ + { participant: 'left' }, + { participant: 'right' }, + ]); + expect(responses.map((response) => response.status).sort()).toEqual([ + 200, 409, + ]); + const loser = responses.find((response) => response.status === 409); + expect(await loser?.json()).toEqual({ + error: 'notification receipt scope differs from this deployment', + }); + const [leftReceipts, rightReceipts, leftInbox, rightInbox] = + await Promise.all([ + db() + .prepare( + 'SELECT operation_key FROM left_flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string }>(), + db() + .prepare( + 'SELECT operation_key FROM right_flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string }>(), + db() + .prepare('SELECT id FROM left_mastra_notifications') + .all<{ id: string }>(), + db() + .prepare('SELECT id FROM right_mastra_notifications') + .all<{ id: string }>(), + ]); + const expected = + responses[0]?.status === 200 + ? { + receipts: [[{ operation_key: 'operation-1' }], []], + inbox: [[{ id: 'notification-1' }], []], + prefix: 'left_', + } + : { + receipts: [[], [{ operation_key: 'operation-2' }]], + inbox: [[], [{ id: 'notification-2' }]], + prefix: 'right_', + }; + expect([leftReceipts.results, rightReceipts.results]).toEqual( + expected.receipts, + ); + expect([leftInbox.results, rightInbox.results]).toEqual(expected.inbox); + const scope = await db() + .prepare( + 'SELECT table_prefix, receipt_limit FROM flowsafe_notification_receipt_scope', + ) + .first<{ table_prefix: string; receipt_limit: number }>(); + expect(scope).toEqual({ table_prefix: expected.prefix, receipt_limit: 1 }); + }); }); diff --git a/packages/flowsafe/src/do-runner/durable-object.test.ts b/packages/flowsafe/src/do-runner/durable-object.test.ts index 1befaef7..f90ed2a3 100644 --- a/packages/flowsafe/src/do-runner/durable-object.test.ts +++ b/packages/flowsafe/src/do-runner/durable-object.test.ts @@ -637,6 +637,328 @@ async function requireSecondMutationEpoch( } describe('workflow ingress capture', () => { + const principalPath = 'http://do/runs/gated/c-run/start-principal'; + const receiptKey = 'flowsafe:start-principal:v1'; + + function namedReader(fixture: ReturnType) { + const state = { + ...fixture.journal.state, + id: { name: 'gated:c-run' }, + } as DurableObjectState; + return new TestRunner(state, fixture.env); + } + + it('returns the verified full principal from the named run after D1 admission', async () => { + const fixture = workflowIngressFixture(); + const principal: ExecutionPrincipal = { + kind: 'agent', + id: 'agent-1', + purpose: 'customer dispatch', + delegatedBy: 'owner-1', + }; + expect( + ( + await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY, principal), + ) + ).status, + ).toBe(200); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(200); + expect(await response.json()).toEqual( + JSON.parse(fixture.journal.values.get(receiptKey) as string), + ); + }); + + it('refuses an unnamed or wrongly named object without reading the receipt', async () => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const read = vi.spyOn(fixture.journal.storage, 'get'); + for (const state of [ + fixture.journal.state, + { ...fixture.journal.state, id: { name: 'gated:other' } }, + ]) { + const response = await new TestRunner( + state as DurableObjectState, + fixture.env, + ).fetch(deploymentIdentityRequest(principalPath)); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + } + expect(read).not.toHaveBeenCalled(); + }); + + it.each([ + ['missing', undefined], + ['malformed JSON', '{'], + ['unknown version', JSON.stringify({ version: 2 })], + ])('refuses a %s receipt without disclosing a principal', async (_case, raw) => { + const fixture = workflowIngressFixture(); + if (raw !== undefined) fixture.journal.values.set(receiptKey, raw); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it.each([ + [ + 'extra field', + (receipt: Record) => ({ ...receipt, extra: true }), + ], + [ + 'changed principal', + (receipt: Record) => ({ + ...receipt, + principal: REPLACEMENT_PRINCIPAL, + }), + ], + [ + 'changed owner', + (receipt: Record) => ({ + ...receipt, + owner: { kind: 'human', id: 'other' }, + }), + ], + [ + 'changed generation', + (receipt: Record) => ({ + ...receipt, + execution: { ...(receipt.execution as object), startToken: 'other' }, + }), + ], + ])('refuses a receipt with %s', async (_case, change) => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const original = JSON.parse( + fixture.journal.values.get(receiptKey) as string, + ); + fixture.journal.values.set(receiptKey, JSON.stringify(change(original))); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it('refuses unreadable D1 state and wrong workflow provenance', async () => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const state = await fixture.runtime.authoritativeStartState( + 'gated', + 'c-run', + ); + if (!state) throw new Error('missing admitted workflow'); + const read = vi.spyOn(fixture.runtime, 'authoritativeStartState'); + for (const result of [ + null, + { + ...state, + execution: { ...state.execution, startToken: 'other-generation' }, + }, + { + ...state, + provenance: { ...state.provenance, startToken: 'other-generation' }, + }, + { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + owner: { kind: 'human', id: 'other-owner' }, + target: { kind: 'workflow', id: 'gated' }, + }, + }, + }, + { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + owner: { kind: 'human', id: 'owner-1' }, + target: { kind: 'agent', id: 'gated', threadId: 'thread-1' }, + }, + }, + }, + ]) { + read.mockResolvedValueOnce(result as typeof state); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + } + read.mockRejectedValueOnce(new Error('D1 unavailable')); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it.each([ + 'unfenced', + 'legacy', + ])('refuses a %s authoritative read beside a valid receipt', async (storage) => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const state = await fixture.runtime.authoritativeStartState( + 'gated', + 'c-run', + ); + if (!state) throw new Error('missing admitted workflow'); + vi.spyOn(fixture.runtime, 'authoritativeStartState').mockResolvedValueOnce({ + ...state, + storage, + } as typeof state); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it('persists the trusted principal and prepared identity for a direct start', async () => { + const fixture = workflowIngressFixture(); + const principal: ExecutionPrincipal = { + kind: 'agent', + id: 'agent-1', + purpose: 'customer dispatch', + delegatedBy: 'owner-1', + }; + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY, principal), + ); + expect(response.status).toBe(200); + const stored = fixture.journal.values.get('flowsafe:start-principal:v1'); + expect(typeof stored).toBe('string'); + const authoritative = await fixture.runtime.authoritativeStartState( + 'gated', + 'c-run', + ); + if (!authoritative || authoritative.kind === 'initial') + throw new Error('missing persisted start identity'); + expect(JSON.parse(stored as string)).toEqual({ + version: 1, + execution: { + tablePrefix: '', + workflowId: 'gated', + runId: 'c-run', + startToken: authoritative.execution.startToken, + }, + owner: { kind: 'agent', id: 'agent-1' }, + principal, + }); + }); + + it('does not record a start whose D1 admission fails', async () => { + const fixture = workflowIngressFixture(true); + await fixture.storage.init(); + const workflows = (await fixture.storage.getStore( + 'workflows', + )) as FencedWorkflowsStorageD1; + const native = workflows[FENCED_WORKFLOW_STORAGE]; + if (!native) throw new Error('missing managed owned workflow capability'); + Object.defineProperty(workflows, FENCED_WORKFLOW_STORAGE, { + value: { + ...native, + withInitialAdmission: () => { + throw new Error('admission failed'); + }, + } satisfies FencedWorkflowAdmissionCapability, + configurable: true, + }); + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).not.toBe(200); + expect( + await workflows.loadWorkflowSnapshot({ + workflowName: 'gated', + runId: 'c-run', + }), + ).toBeNull(); + expect(fixture.journal.values.has('flowsafe:start-principal:v1')).toBe( + false, + ); + }); + + it.each([ + 'not-json', + JSON.stringify({ version: 2 }), + JSON.stringify({ version: 1, principal: REPLACEMENT_PRINCIPAL }), + ])('keeps a conflicting start receipt unchanged (%s)', async (existing) => { + const fixture = workflowIngressFixture(); + fixture.journal.values.set('flowsafe:start-principal:v1', existing); + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).toBe(200); + expect(fixture.journal.values.get('flowsafe:start-principal:v1')).toBe( + existing, + ); + }); + + it('keeps a committed start receipt when its write acknowledgment is lost', async () => { + const fixture = workflowIngressFixture(); + const nativePut = fixture.journal.storage.put.bind(fixture.journal.storage); + vi.spyOn(fixture.journal.storage, 'put').mockImplementation( + async (key, value) => { + await nativePut(key, value); + if (key === 'flowsafe:start-principal:v1') + throw new Error('write acknowledgment lost'); + }, + ); + const logged = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).toBe(200); + expect( + JSON.parse( + fixture.journal.values.get('flowsafe:start-principal:v1') as string, + ), + ).toMatchObject({ + version: 1, + owner: { kind: 'human', id: 'owner-1' }, + principal: OWNER_PRINCIPAL, + }); + } finally { + logged.mockRestore(); + } + }); + + it('starts normally when the principal receipt cannot be stored', async () => { + const fixture = workflowIngressFixture(); + const nativePut = fixture.journal.storage.put.bind(fixture.journal.storage); + vi.spyOn(fixture.journal.storage, 'put').mockImplementation( + async (key, value) => { + if (key === 'flowsafe:start-principal:v1') + throw new Error('receipt storage unavailable'); + await nativePut(key, value); + }, + ); + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).toBe(200); + expect(fixture.journal.values.has('flowsafe:start-principal:v1')).toBe( + false, + ); + }); + it('starts a queued run after its sender disconnects', async () => { const fixture = workflowIngressFixture(); const hold = await holdWorkflowFifo(fixture); @@ -861,6 +1183,9 @@ describe('workflow ingress capture', () => { } const response = await pending; expect(response.status).toBe(200); + expect(fixture.journal.values.has('flowsafe:start-principal:v1')).toBe( + !scheduled, + ); expect(fixture.start).toHaveBeenCalledOnce(); expect(fixture.start.mock.calls[0]?.[0]).toBe('gated'); const options = fixture.start.mock.calls[0]?.[1]; diff --git a/packages/flowsafe/src/do-runner/durable-object.ts b/packages/flowsafe/src/do-runner/durable-object.ts index 13863b80..2f54da06 100644 --- a/packages/flowsafe/src/do-runner/durable-object.ts +++ b/packages/flowsafe/src/do-runner/durable-object.ts @@ -183,6 +183,7 @@ export interface DurableObjectRunLifecycleHooks { } const RUN_OWNER_RECOVERY_KEY = 'flowsafe:run-owner-recovery:v1'; +const START_PRINCIPAL_RECEIPT_KEY = 'flowsafe:start-principal:v1'; export const RUN_OWNER_RECOVERY_DELAY_MS = 60_000; const SUSPENSION_DEADLINE_RETRY_MS = 60_000; // How long a run's state may stay unreadable before the entries due under it @@ -336,6 +337,12 @@ function sameRunRecovery( ); } +function preparedRunExecution( + recovery: RunOwnerRecovery, +): RunExecutionIdentity | undefined { + return recovery.phase === 'preparing' ? undefined : recovery.execution; +} + interface StartBody { startReservation?: unknown; workflowId?: string; @@ -1009,6 +1016,115 @@ export abstract class DurableObjectRunner { return prepared; } + async #recordStartPrincipal( + execution: RunExecutionIdentity, + owner: DurableObjectRunOwner, + principal: ExecutionPrincipal, + ): Promise { + const storage = this.state?.storage; + if (!storage) return; + const receipt = JSON.stringify({ + version: 1, + execution: normalizeRunExecutionIdentity(execution), + owner, + principal, + }); + try { + const existing = await storage.get(START_PRINCIPAL_RECEIPT_KEY); + if (existing !== undefined && existing !== receipt) { + console.error('start principal receipt conflicts with prepared start'); + return; + } + if (existing === undefined) { + try { + await storage.put(START_PRINCIPAL_RECEIPT_KEY, receipt); + } catch { + // A lost acknowledgment is resolved by the read below. + } + } + if ((await storage.get(START_PRINCIPAL_RECEIPT_KEY)) !== receipt) + console.error('start principal receipt could not be verified'); + } catch (error) { + console.error('start principal receipt could not be verified', error); + } + } + + async #startPrincipalReceipt( + workflowId: string, + runId: string, + ): Promise { + const unavailable = () => json({ error: 'run not found' }, 404); + const storage = this.state?.storage; + if (!storage || this.state?.id?.name !== `${workflowId}:${runId}`) + return unavailable(); + try { + const raw = await storage.get(START_PRINCIPAL_RECEIPT_KEY); + if (typeof raw !== 'string') return unavailable(); + const parsed: unknown = JSON.parse(raw); + if ( + parsed === null || + typeof parsed !== 'object' || + Array.isArray(parsed) + ) + return unavailable(); + const receipt = parsed as Record; + if ( + receipt.version !== 1 || + receipt.owner === null || + typeof receipt.owner !== 'object' || + Array.isArray(receipt.owner) || + receipt.principal === null || + typeof receipt.principal !== 'object' || + Array.isArray(receipt.principal) + ) + return unavailable(); + const execution = normalizeD1RunExecutionIdentity(receipt.execution); + const owner = receipt.owner as Record; + const principal = decodeExecutionPrincipal( + JSON.stringify(receipt.principal), + ); + if ( + !isExecutionPrincipalKind(owner.kind) || + !isExecutionPrincipalId(owner.id) || + !principal || + execution.workflowId !== workflowId || + execution.runId !== runId || + owner.kind !== principal.kind || + owner.id !== principal.id + ) + return unavailable(); + const canonical = { + version: 1, + execution, + owner: { kind: owner.kind, id: owner.id }, + principal, + }; + if (JSON.stringify(canonical) !== raw) return unavailable(); + const state = await this.#ensureRuntime().authoritativeStartState( + workflowId, + runId, + ); + const identity = state?.provenance.startIdentity; + if ( + state?.storage !== 'd1' || + state.execution.tablePrefix !== execution.tablePrefix || + state.execution.workflowId !== execution.workflowId || + state.execution.runId !== execution.runId || + state.execution.startToken !== execution.startToken || + state.provenance.startToken !== execution.startToken || + identity?.target.kind !== 'workflow' || + identity.target.id !== workflowId || + identity.owner.kind !== owner.kind || + identity.owner.id !== owner.id || + state.provenance.agentStart !== undefined + ) + return unavailable(); + return json(canonical); + } catch { + return unavailable(); + } + } + #matchingRunState( recovery: RunOwnerRecovery, state: AuthoritativeStartState | null, @@ -2590,6 +2706,13 @@ export abstract class DurableObjectRunner { frame.unwound = true; } summary = await this.#finishRunOwner(recovery, summary); + const preparedExecution = preparedRunExecution(recovery); + if (scheduleId === undefined && preparedExecution) + await this.#recordStartPrincipal( + preparedExecution, + owner, + principal, + ); this.#broadcastRunSummary(summary); return json( await this.#withSuspensionTimers(startWorkflowId, summary), @@ -2639,6 +2762,17 @@ export abstract class DurableObjectRunner { this.#assertRunIdentity(workflowId, runId); return json({ live: this.#isStartLive(workflowId, runId) }); } + if ( + request.method === 'GET' && + segments.length === 4 && + action === 'start-principal' && + workflowId && + runId + ) { + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) + return json({ error: 'run not found' }, 404); + return this.#startPrincipalReceipt(workflowId, runId); + } if ( request.method === 'GET' && segments.length === 3 && diff --git a/packages/flowsafe/src/do-runner/execution-admission.test.ts b/packages/flowsafe/src/do-runner/execution-admission.test.ts index 48187f00..54eaac8b 100644 --- a/packages/flowsafe/src/do-runner/execution-admission.test.ts +++ b/packages/flowsafe/src/do-runner/execution-admission.test.ts @@ -71,6 +71,21 @@ describe('execution identity and epoch helpers', () => { .tablePrefix, ).toBe('a'.repeat(39)); expect(normalizeStartIdentity(START)).toEqual(START); + expect( + normalizeStartIdentity({ + ...START, + target: { ...START.target, resourceId: 'account-7' }, + }), + ).toEqual({ + ...START, + target: { ...START.target, resourceId: 'account-7' }, + }); + expect(() => + normalizeStartIdentity({ + ...START, + target: { ...START.target, resourceId: 'bad/resource' }, + }), + ).toThrow(InvalidExecutionIdentityError); const workflow = { owner: START.owner, target: { kind: 'workflow', id: 'logical-parent' }, diff --git a/packages/flowsafe/src/do-runner/execution-admission.ts b/packages/flowsafe/src/do-runner/execution-admission.ts index 331b1716..6cad53d1 100644 --- a/packages/flowsafe/src/do-runner/execution-admission.ts +++ b/packages/flowsafe/src/do-runner/execution-admission.ts @@ -87,6 +87,7 @@ export interface StartIdentity { readonly kind: 'agent'; readonly id: string; readonly threadId: string; + readonly resourceId?: string; }; } @@ -112,6 +113,8 @@ const IDENTITY_ERRORS = { 'target.id': 'target.id must be a URL-path-safe identifier', 'target.threadId': 'target.threadId is required only for an agent target and must be URL-path-safe', + 'target.resourceId': + 'target.resourceId is valid only for an agent target and must be URL-path-safe', } as const; export class InvalidExecutionIdentityError extends DoStatusError { @@ -226,7 +229,10 @@ export function normalizeStartIdentity(value: unknown): StartIdentity { 'identity', ); const { kind: ownerKind, id: ownerId } = identityObject(rawOwner, 'owner'); - const { kind, id, threadId } = identityObject(rawTarget, 'target'); + const { kind, id, threadId, resourceId } = identityObject( + rawTarget, + 'target', + ); if (!isExecutionPrincipalKind(ownerKind)) throw new InvalidExecutionIdentityError('owner.kind'); if (!isExecutionPrincipalId(ownerId)) @@ -238,13 +244,22 @@ export function normalizeStartIdentity(value: unknown): StartIdentity { if (kind === 'agent') { if (!isPathSafeId(threadId)) throw new InvalidExecutionIdentityError('target.threadId'); + if (resourceId !== undefined && !isPathSafeId(resourceId)) + throw new InvalidExecutionIdentityError('target.resourceId'); return Object.freeze({ owner, - target: Object.freeze({ kind, id, threadId }), + target: Object.freeze({ + kind, + id, + threadId, + ...(resourceId === undefined ? {} : { resourceId }), + }), }); } if (threadId !== undefined) throw new InvalidExecutionIdentityError('target.threadId'); + if (resourceId !== undefined) + throw new InvalidExecutionIdentityError('target.resourceId'); return Object.freeze({ owner, target: Object.freeze({ kind, id }) }); } diff --git a/packages/flowsafe/src/do-runner/index.ts b/packages/flowsafe/src/do-runner/index.ts index a9340b38..f5d2a449 100644 --- a/packages/flowsafe/src/do-runner/index.ts +++ b/packages/flowsafe/src/do-runner/index.ts @@ -224,6 +224,14 @@ export { mintThreadId, resourceIdFromKey } from './memory-id.js'; export { isPathSafeId, PATH_SAFE_ID_PATTERN } from './path-safe-id.js'; export type { HostPubSub } from './pubsub.js'; export { createHostPubSub } from './pubsub.js'; +export { + type RemoteStepAddress, + type RemoteStepInvocation, + RemoteStepJournal, + type RemoteStepNotificationOperation, + type RemoteStepPermit, + type WorkflowRemoteStepAddress, +} from './remote-step-journal.js'; export type { RunEconomicOperation, RunLifecyclePrincipal, diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index 14ef2a8c..e9488a93 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -70,6 +70,11 @@ import { DUE_NOTIFICATION_SQL, notificationTimestampMillis, } from './notification-predicate.js'; +import { + REMOTE_STEP_NOTIFICATION_CURSOR_TABLE, + REMOTE_STEP_SCHEMA_TABLE, + REMOTE_STEP_TABLE, +} from './remote-step-journal.js'; import { START_IDEMPOTENCY_TABLE, START_RESERVATION_STATES, @@ -122,6 +127,7 @@ export type InventoryCategoryClass = 'work' | 'standing'; */ export const INVENTORY_CATEGORIES = [ 'runs', + 'remote-steps', 'approvals-waiting', 'schedule-deferred-dispatches', 'pending-notifications', @@ -290,6 +296,25 @@ export interface FlowsafeTableEntry { * characters. */ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ + { + table: REMOTE_STEP_NOTIFICATION_CURSOR_TABLE, + purpose: 'position of the scheduled notification operation recovery scan', + accounting: { + excluded: 'the invocation journal holds the work this cursor revisits', + }, + }, + { + table: REMOTE_STEP_SCHEMA_TABLE, + purpose: 'remote invocation journal schema version', + accounting: { + excluded: 'schema metadata contains no run or external dispatch work', + }, + }, + { + table: REMOTE_STEP_TABLE, + purpose: 'durable remote tool invocations and their dispatch permits', + accounting: { category: 'remote-steps' }, + }, { table: APPROVALS_TABLE, purpose: 'the durable approval queue — one record per suspended decision', @@ -320,6 +345,23 @@ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ 'a single counter row, not a unit of work. The notifications it orders are inventoried under pending-notifications; the counter itself has nothing to finish and is re-derivable from the inbox.', }, }, + { + table: 'flowsafe_notification_operation_receipts', + purpose: 'immutable receipts for host-owned inbox notification effects', + accounting: { + excluded: + 'a receipt records an effect already committed with its inbox row; replay must retain it after the inbox row is settled or removed, so it is historical evidence rather than active work', + }, + }, + { + table: 'flowsafe_notification_receipt_scope', + purpose: + 'the deployment receipt prefix and retained receipt admission limit', + accounting: { + excluded: + 'the singleton configures receipt admission and contains no active notification effect', + }, + }, { table: RESOURCE_OWNER_TABLE, purpose: @@ -479,6 +521,13 @@ export const INVENTORY_CATEGORY_DESCRIPTORS: readonly InventoryCategoryDescripto holds: 'a workflow or agent run that has not reached a terminal state (cancelled and timed_out count as live until their lifecycle cleanup completes)', }, + { + category: 'remote-steps', + class: 'work', + table: REMOTE_STEP_TABLE, + holds: + 'a remote invocation without a recorded result or an external dispatch not confirmed stopped', + }, { category: 'approvals-waiting', class: 'work', @@ -861,7 +910,10 @@ export class DeploymentInventory { `unknown inventory category '${String(category)}'`, ); } - const query = this.#queryFor(category); + const query = + category === 'remote-steps' + ? await this.#remoteStepQuery() + : this.#queryFor(category); const table = this.#tableFor(descriptor); const cursor = parseCursor(options.cursor, category, query.key.length); const limit = clampLimit(options.limit); @@ -927,6 +979,31 @@ export class DeploymentInventory { : descriptor.table; } + async #remoteStepQuery(): Promise { + const base = this.#queryFor('remote-steps'); + const { results } = await this.#db + .prepare('SELECT name FROM pragma_table_xinfo(?)') + .bind(REMOTE_STEP_TABLE) + .all<{ name: unknown }>(); + const columns = new Set(results.map((row) => row.name)); + if (!columns.has('operation_state')) return base; + if (!columns.has('resume_admitted_at')) + throw new Error('remote step inventory schema is unreadable'); + return { + ...base, + detail: [ + ...base.detail, + 'operation_state', + 'resume_admitted_at', + 'settlement_outcome', + ], + where: `((${base.where}) OR + (operation_state IN ('resume-admitted', 'unresolved') + AND status IN ('failed', 'settled') + AND settlement_outcome IS NOT 'applied'))`, + }; + } + async #page( table: string, query: CategoryQuery, @@ -1005,6 +1082,23 @@ export class DeploymentInventory { binds: RUN_TERMINAL_STATUSES, annotate: annotateRunOwners, }; + case 'remote-steps': + return { + key: ['invocation_id'], + detail: [ + 'workflow_id', + 'run_id', + 'agent_id', + 'thread_id', + 'status', + 'deadline_at', + 'permit_generation', + 'permit_consumed_at', + ], + where: + "status NOT IN ('failed', 'settled') OR settlement_outcome IN ('advanced-without-proof', 'deadline-expired') OR (status = 'settled' AND (settlement_outcome IS NOT 'applied' OR result_json IS NULL OR result_digest IS NULL OR executor_quiescent_at IS NULL)) OR (permit_consumed_at IS NOT NULL AND executor_quiescent_at IS NULL)", + binds: [], + }; case 'approvals-waiting': // OPEN_STATUSES, not just 'pending': a claimed or escalated approval is // still undecided, and its run is still suspended behind it. The same diff --git a/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts b/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts index 301ae961..dcbe5f70 100644 --- a/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts +++ b/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts @@ -76,6 +76,7 @@ import { type InventoryTableAccounting, } from './inventory.js'; import { mintThreadId, resourceIdFromKey } from './memory-id.js'; +import { RemoteStepJournal } from './remote-step-journal.js'; import type { RunnerRuntime } from './runtime.js'; import { START_IDEMPOTENCY_TABLE, @@ -485,7 +486,17 @@ describe('Mastra persistence guards (D1Store SQL over node:sqlite)', () => { threadId: 'thr-1', resourceId: 'res-1', }); - await new ExecutionFenceStore(binding as never).seed('open'); + const fence = new ExecutionFenceStore(binding as never); + await fence.seed('open'); + await new RemoteStepJournal({ + db: binding as never, + fence, + runtime, + deploymentId: 'census-deployment', + tablePrefix: '', + allowedToolIds: ['census-tool'], + verifyCompletion: async () => undefined, + }).initialize(); sqlite.exec( `CREATE TABLE IF NOT EXISTS ${DEPLOYMENT_SENTINEL_TABLE} ( id INTEGER PRIMARY KEY CHECK (id = 1), diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts new file mode 100644 index 00000000..aea156d3 --- /dev/null +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -0,0 +1,2354 @@ +// SPDX-License-Identifier: Apache-2.0 + +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { z } from 'zod'; +import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; +import { + createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, +} from '../agent-host/thread-topology.js'; +import { decodeExecutionPrincipal } from '../approval-api/principal.js'; +import type { FlowsafeWorkerEnv } from '../host-kit/flowsafe-worker.js'; +import { createFlowsafeWorker } from '../host-kit/flowsafe-worker.js'; +import { approvalStoreFactoryFor } from '../host-kit/host-approval-service.js'; +import type { + ThreadNamespaceLike, + ThreadRequestInit, +} from '../host-kit/thread-topology.js'; +import { staticTokenVerifier } from '../host-kit/verifier.js'; +import { + D1NotificationsStorage, + type NotificationOperationInput, + projectNotificationOperation, +} from '../signals/notifications-d1.js'; +import { createD1Storage } from './d1-storage.js'; +import { + credentialsMatch, + DEPLOYMENT_IDENTITY_HEADER, +} from './deployment-identity.js'; +import { ExecutionFenceStore } from './execution-fence.js'; +import { EXECUTION_PRINCIPAL_HEADER } from './execution-principal-header.js'; +import { init } from './init.js'; +import { DeploymentInventory } from './inventory.js'; +import { + REMOTE_STEP_MARKER, + REMOTE_STEP_NOTIFICATION_CURSOR_TABLE, + REMOTE_STEP_TABLE, + RemoteStepJournal, + WORKFLOW_REMOTE_STEP_MARKER, +} from './remote-step-journal.js'; +import type { RunnerRuntime, RunSummary } from './runtime.js'; + +const ADDRESS = { + deploymentId: 'tenant-1', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'thread-1', + resourceId: 'thread-1', + runId: 'run-1', +}; +const PRINCIPAL = { + kind: 'human', + id: 'operator-1', + role: 'operator', +} as const; + +describe('remote step journal dispatch admission', () => { + const databases: Array<{ close(): void }> = []; + + afterEach(() => { + for (const database of databases.splice(0)) database.close(); + }); + + async function fixture() { + const sqlite = openSqlite() as ReturnType & { + close(): void; + }; + databases.push(sqlite); + const db = sqliteUnitDatabase(sqlite); + sqlite.exec(`CREATE TABLE mastra_workflow_snapshot ( + workflow_name TEXT NOT NULL, run_id TEXT NOT NULL, + snapshot TEXT NOT NULL, PRIMARY KEY (workflow_name, run_id))`); + sqlite + .prepare(`INSERT INTO mastra_workflow_snapshot VALUES (?, ?, ?)`) + .run(ADDRESS.workflowId, ADDRESS.runId, '{"status":"suspended"}'); + const fence = new ExecutionFenceStore(db); + await fence.seed('open'); + let now = 1_000; + let summary: RunSummary = { + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 500 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }; + const runtime = { + authoritativeStatus: async () => summary, + authoritativeStartState: async () => ({ + storage: 'd1', + snapshot: JSON.parse( + ( + sqlite + .prepare('SELECT snapshot FROM mastra_workflow_snapshot') + .get() as { snapshot: string } + ).snapshot, + ), + execution: { + tablePrefix: '', + workflowId: ADDRESS.workflowId, + runId: ADDRESS.runId, + startToken: 'start-token', + }, + provenance: { + startIdentity: { + owner: { kind: PRINCIPAL.kind, id: PRINCIPAL.id }, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + resourceId: ADDRESS.resourceId, + }, + }, + }, + }), + } as unknown as Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + const verifyCompletion = vi.fn(async () => undefined); + const journal = new RemoteStepJournal({ + db, + fence, + runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + verifyCompletion, + now: () => now, + }); + await journal.initialize(); + return { + db, + fence, + journal, + verifyCompletion, + runtime, + reopen: () => + new RemoteStepJournal({ + db, + fence, + runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + verifyCompletion, + now: () => now, + }), + sqlite, + setNow: (value: number) => { + now = value; + }, + setSummary: (value: RunSummary) => { + summary = value; + }, + retainResult: async () => { + const invocation = await journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await journal.issuePermit( + invocation.invocationId, + 1_000, + ); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ); + return { invocation, rawPermit: consumed.rawPermit }; + }, + setRequest: (account: string) => { + summary = { + ...summary, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account }, + }, + }, + }, + }; + sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run(JSON.stringify({ status: 'suspended', request: { account } })); + }, + }; + } + + function restoreV3Schema(h: Awaited>) { + const columns = ( + h.sqlite + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all() as Array<{ name: string }> + ) + .slice( + 0, + ( + h.sqlite + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all() as Array<{ name: string }> + ).findIndex(({ name }) => name === 'target_kind'), + ) + .map(({ name }) => name) + .join(', '); + h.sqlite.exec(`CREATE TABLE ${REMOTE_STEP_TABLE}_v3 ( + invocation_id TEXT PRIMARY KEY, principal_json TEXT NOT NULL, + start_token TEXT NOT NULL, deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, workflow_id TEXT NOT NULL, + agent_id TEXT NOT NULL, thread_id TEXT NOT NULL, + resource_id TEXT NOT NULL, run_id TEXT NOT NULL, + step_path TEXT NOT NULL, suspended_at INTEGER NOT NULL, + resume_count INTEGER NOT NULL, request_json TEXT NOT NULL, + deadline_at INTEGER NOT NULL, status TEXT NOT NULL, + permit_generation INTEGER NOT NULL DEFAULT 0, permit_epoch INTEGER, + permit_revision INTEGER, permit_expires_at INTEGER, + permit_consumed_at INTEGER, permit_digest TEXT, result_json TEXT, + result_digest TEXT, quiescence_digest TEXT, executor_quiescent_at INTEGER, + resume_lease_owner TEXT, resume_lease_expires_at INTEGER, + settled_at INTEGER, settlement_outcome TEXT, + UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, + thread_id, resource_id, run_id, step_path, suspended_at, resume_count) + )`); + h.sqlite.exec( + `INSERT INTO ${REMOTE_STEP_TABLE}_v3 (${columns}) SELECT ${columns} FROM ${REMOTE_STEP_TABLE}`, + ); + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec( + `ALTER TABLE ${REMOTE_STEP_TABLE}_v3 RENAME TO ${REMOTE_STEP_TABLE}`, + ); + h.sqlite.exec( + `CREATE INDEX flowsafe_remote_step_recovery ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE status IN ('result-retained', 'resuming')`, + ); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 3'); + } + + async function workflowFixture() { + const h = await fixture(); + const original = h.runtime.authoritativeStartState.bind(h.runtime); + h.runtime.authoritativeStartState = (async ( + workflowId: string, + runId: string, + ) => { + const state = await original(workflowId, runId); + if (!state) return state; + return { + ...state, + provenance: { + startIdentity: { + owner: { kind: PRINCIPAL.kind, id: PRINCIPAL.id }, + target: { kind: 'workflow', id: ADDRESS.workflowId }, + }, + }, + } as typeof state; + }) as typeof h.runtime.authoritativeStartState; + const request = { version: 2, input: { query: 'x' } }; + const summary = (value: unknown): RunSummary => ({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['execute']], + suspendedAt: { execute: 500 }, + suspendPayload: { + execute: { + type: 'suspension', + remoteStep: { + marker: WORKFLOW_REMOTE_STEP_MARKER, + request: value, + }, + }, + }, + }); + h.setSummary(summary(request)); + const verifyWorkflowCompletion = vi.fn(async () => undefined); + const journal = () => + new RemoteStepJournal({ + db: h.db, + fence: h.fence, + runtime: h.runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + verifyCompletion: h.verifyCompletion, + verifyWorkflowCompletion, + now: () => 1_000, + }); + const address = { + targetKind: 'workflow', + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + workflowId: ADDRESS.workflowId, + runId: ADDRESS.runId, + } as const; + return { ...h, address, journal, summary, verifyWorkflowCompletion }; + } + + it('converges repeated observation on one invocation without a permit', async () => { + const h = await fixture(); + const [first, replay] = await Promise.all([ + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + ]); + expect(replay.invocationId).toBe(first.invocationId); + expect(await h.journal.readDispatchRequestJson(first.invocationId)).toBe( + '{"account":"acme"}', + ); + expect(first).toMatchObject({ + status: 'pending', + permitGeneration: 0, + request: { account: 'acme' }, + stepPath: ['durable-agentic-execution'], + suspendedAt: 500, + }); + expect( + h.sqlite + .prepare(`SELECT count(*) AS count FROM ${REMOTE_STEP_TABLE}`) + .get(), + ).toMatchObject({ count: 1 }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).entries, + ).toEqual([ + { + key: [first.invocationId], + detail: expect.objectContaining({ + status: 'pending', + run_id: ADDRESS.runId, + }), + }, + ]); + }); + + it('refuses unauthorized dispatch and replay before the trusted effect', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const owner = { kind: PRINCIPAL.kind, id: PRINCIPAL.id }; + const resources = approvalStoreFactoryFor(h.db).resources(); + for (const kind of ['thread', 'resource', 'run'] as const) { + const id = kind === 'run' ? ADDRESS.runId : ADDRESS.threadId; + expect(await resources.claim(kind, id, owner)).toBe(true); + } + const topology = createAgentThreadTopology( + { + idFromName: (name) => name, + get: () => ({ + fetch: async () => { + throw new Error('dispatch admission must not address a thread'); + }, + }), + }, + 'test-deployment-identity-secret-0001', + { + startIdempotency: 'none', + executionFence: 'none', + remoteStepDispatch: h.journal, + }, + ); + type Env = FlowsafeWorkerEnv & { DISPATCH_FIXTURE_SECRET: string }; + const effect = vi.fn(async (_rawPermit: string) => undefined); + const worker = createFlowsafeWorker({ + workflows: [], + systemPrincipalId: 'test-system', + buildVerifier: () => staticTokenVerifier(new Map()), + maintenance: { sweepIntervalMs: 60_000, purgeIntervalMs: 60_000 }, + preRoutes: async (request, env, _ctx, kit) => { + if (new URL(request.url).pathname !== '/__fixture/remote-dispatch') + return null; + const credential = request.headers.get('x-dispatch-fixture-secret'); + if ( + !credential || + !(await credentialsMatch(credential, env.DISPATCH_FIXTURE_SECRET)) + ) + return new Response(null, { status: 401 }); + const input = (await request.json()) as Record; + const keys = Object.keys(input).sort().join(','); + const invocationId = + input.action === 'issue' && keys === 'action,invocationId' + ? input.invocationId + : input.action === 'consume' && keys === 'action,permit' + ? (input.permit as { invocationId?: unknown } | null) + ?.invocationId + : undefined; + if (typeof invocationId !== 'string') + return new Response(null, { status: 400 }); + const context = await kit.contextForStoredInvocation( + h.journal, + invocationId, + ); + if (!context) return new Response(null, { status: 404 }); + if (input.action === 'issue') { + const ticket = await topology.issueRemoteStepDispatch( + context, + invocationId, + 1_000, + ); + return Response.json(ticket); + } + const result = await topology.consumeRemoteStepDispatch( + context, + input.permit as Parameters< + typeof topology.consumeRemoteStepDispatch + >[1], + ); + if (result.kind === 'already-consumed') + return new Response(null, { status: 409 }); + await effect(result.rawPermit); + return Response.json({ kind: 'accepted' }); + }, + }); + h.sqlite.exec(`CREATE TABLE flowsafe_deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + tenant_tag TEXT NOT NULL, + provisioned_at TEXT NOT NULL + )`); + h.sqlite + .prepare( + `INSERT INTO flowsafe_deployment VALUES (1, 'acme', '2026-08-10T00:00:00.000Z')`, + ) + .run(); + const fixtureEnv: Env = { + DB: h.db, + DEPLOYMENT_TENANT: 'acme', + DEPLOYMENT_IDENTITY_SECRET: 'test-deployment-identity-secret-0001', + DISPATCH_FIXTURE_SECRET: 'separate-test-dispatch-secret-0001', + RUNNER: { + idFromName: (name) => name, + get: () => ({ fetch: async () => new Response(null, { status: 503 }) }), + }, + MAINTENANCE: { + idFromName: (name) => name, + get: () => ({ fetch: async () => new Response(null, { status: 503 }) }), + }, + }; + const ctx = { waitUntil: () => undefined }; + const send = (body: unknown, secret?: string) => + worker.fetch( + new Request('https://host/__fixture/remote-dispatch', { + method: 'POST', + headers: { + 'content-type': 'application/json', + ...(secret ? { 'x-dispatch-fixture-secret': secret } : {}), + }, + body: JSON.stringify(body), + }), + fixtureEnv, + ctx, + ); + + expect( + (await send({ action: 'issue', invocationId: invocation.invocationId })) + .status, + ).toBe(401); + expect( + ( + await send( + { action: 'issue', invocationId: invocation.invocationId }, + 'wrong-secret', + ) + ).status, + ).toBe(401); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'pending', + ); + expect(effect).not.toHaveBeenCalled(); + expect( + ( + await send( + { + action: 'issue', + invocationId: invocation.invocationId, + lifetimeMs: 60_000, + }, + fixtureEnv.DISPATCH_FIXTURE_SECRET, + ) + ).status, + ).toBe(400); + expect( + ( + await send( + { + action: 'issue', + invocationId: invocation.invocationId, + principal: PRINCIPAL, + }, + fixtureEnv.DISPATCH_FIXTURE_SECRET, + ) + ).status, + ).toBe(400); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'pending', + ); + + const issued = await send( + { action: 'issue', invocationId: invocation.invocationId }, + fixtureEnv.DISPATCH_FIXTURE_SECRET, + ); + expect(issued.status).toBe(200); + const ticket = (await issued.json()) as { + permit: Parameters[1]; + }; + expect(ticket).not.toHaveProperty('rawPermit'); + const consume = { action: 'consume', permit: ticket.permit }; + expect((await send(consume)).status).toBe(401); + expect((await send(consume, 'wrong-secret')).status).toBe(401); + expect(effect).not.toHaveBeenCalled(); + const accepted = await send(consume, fixtureEnv.DISPATCH_FIXTURE_SECRET); + expect(accepted.status).toBe(200); + expect(await accepted.json()).toEqual({ kind: 'accepted' }); + expect(effect).toHaveBeenCalledTimes(1); + expect(effect).toHaveBeenCalledWith(expect.stringMatching(/^[\w-]{43}$/)); + expect( + (await send(consume, fixtureEnv.DISPATCH_FIXTURE_SECRET)).status, + ).toBe(409); + expect(effect).toHaveBeenCalledTimes(1); + }); + + it('registers the resource recorded in start provenance independently of the thread id', async () => { + const h = await fixture(); + const original = h.runtime.authoritativeStartState.bind(h.runtime); + h.runtime.authoritativeStartState = (async ( + workflowId: string, + runId: string, + ) => { + const state = await original(workflowId, runId); + if (!state) return state; + return { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + ...state.provenance.startIdentity, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + resourceId: 'account-7', + }, + }, + }, + } as typeof state; + }) as typeof h.runtime.authoritativeStartState; + + const invocation = await h.journal.register( + { ...ADDRESS, resourceId: 'account-7' }, + 10_000, + PRINCIPAL, + ); + expect(invocation.resourceId).toBe('account-7'); + }); + + it('refuses remote dispatch when legacy start provenance lacks a resource', async () => { + const h = await fixture(); + const original = h.runtime.authoritativeStartState.bind(h.runtime); + h.runtime.authoritativeStartState = (async ( + workflowId: string, + runId: string, + ) => { + const state = await original(workflowId, runId); + if (!state) return state; + return { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + ...state.provenance.startIdentity, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + }, + }, + }, + } as typeof state; + }) as typeof h.runtime.authoritativeStartState; + + await expect( + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + ).rejects.toMatchObject({ + status: 503, + }); + expect( + h.sqlite + .prepare(`SELECT count(*) AS count FROM ${REMOTE_STEP_TABLE}`) + .get(), + ).toEqual({ count: 0 }); + }); + + it('upgrades a prior journal while preserving an existing invocation', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + restoreV3Schema(h); + h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); + for (const column of [ + 'settlement_outcome', + 'settled_at', + 'resume_lease_expires_at', + 'resume_lease_owner', + 'executor_quiescent_at', + 'quiescence_digest', + 'result_digest', + 'result_json', + ]) + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN ${column}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + + await h.reopen().initialize(); + + expect(await h.reopen().get(invocation.invocationId)).toMatchObject({ + invocationId: invocation.invocationId, + status: 'pending', + }); + const columns = h.sqlite + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all() as Array<{ name: string }>; + expect(columns.map((column) => column.name)).toContain( + 'executor_quiescent_at', + ); + }); + + it('upgrades a live v2 result without losing its retained bytes', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"kept"}', + '{"receipt":"verified"}', + ); + restoreV3Schema(h); + h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); + for (const column of [ + 'settlement_outcome', + 'settled_at', + 'resume_lease_expires_at', + 'resume_lease_owner', + ]) + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN ${column}`); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 2'); + + await h.reopen().initialize(); + + expect( + await h.reopen().readRetainedResult(invocation.invocationId), + ).toMatchObject({ + resultJson: '{"value":"kept"}', + }); + expect( + h.sqlite.prepare('SELECT version FROM flowsafe_remote_step_schema').get(), + ).toEqual({ version: 5 }); + expect(await h.reopen().recoverableResults(1)).toEqual({ + invocationIds: [invocation.invocationId], + nextCursor: invocation.invocationId, + }); + }); + + it('preserves a live v3 resume lease while upgrading the journal', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 1_000); + restoreV3Schema(h); + const reopened = h.reopen(); + await reopened.initialize(); + expect(await reopened.get(invocation.invocationId)).toMatchObject({ + status: 'resuming', + resumeLeaseExpiresAt: 2_000, + }); + expect( + await reopened.readRetainedResult(invocation.invocationId), + ).toMatchObject({ + resultJson: '{"value":"done"}', + }); + expect( + await reopened.settleAfterResume(invocation.invocationId, owner), + ).toBe('retry'); + }); + + it('refuses an unknown journal schema version before dispatch', async () => { + const h = await fixture(); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 99'); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses a partial journal schema without changing its rows', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN result_digest`); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + expect( + h.sqlite.prepare(`SELECT invocation_id FROM ${REMOTE_STEP_TABLE}`).get(), + ).toMatchObject({ invocation_id: invocation.invocationId }); + }); + + it('refuses an unversioned journal without its suspension uniqueness constraint', async () => { + const h = await fixture(); + restoreV3Schema(h); + const ddl = ( + h.sqlite + .prepare( + `SELECT sql FROM sqlite_schema WHERE name = '${REMOTE_STEP_TABLE}'`, + ) + .get() as { sql: string } + ).sql; + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec(ddl.replace(/,\s*UNIQUE \(deployment_id,[^)]*\)/, '')); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses an unversioned journal without the permit-generation default', async () => { + const h = await fixture(); + const ddl = ( + h.sqlite + .prepare( + `SELECT sql FROM sqlite_schema WHERE name = '${REMOTE_STEP_TABLE}'`, + ) + .get() as { sql: string } + ).sql; + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec( + ddl.replace( + 'permit_generation INTEGER NOT NULL DEFAULT 0', + 'permit_generation INTEGER NOT NULL', + ), + ); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses an unversioned workflow journal with a changed target constraint', async () => { + const h = await fixture(); + const ddl = ( + h.sqlite + .prepare( + `SELECT sql FROM sqlite_schema WHERE name = '${REMOTE_STEP_TABLE}'`, + ) + .get() as { sql: string } + ).sql; + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec( + ddl.replace( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + "CHECK ((target_kind = 'agent' AND agent_id IS NULL", + ), + ); + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('converges when another initializer wins the same journal upgrade', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + restoreV3Schema(h); + h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); + for (const column of [ + 'settlement_outcome', + 'settled_at', + 'resume_lease_expires_at', + 'resume_lease_owner', + 'executor_quiescent_at', + 'quiescence_digest', + 'result_digest', + 'result_json', + ]) + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN ${column}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + const originalBatch = h.db.batch.bind(h.db); + h.db.batch = async (statements) => { + h.db.batch = originalBatch; + await h.reopen().initialize(); + return originalBatch(statements); + }; + + await h.reopen().initialize(); + + expect(await h.reopen().get(invocation.invocationId)).toMatchObject({ + invocationId: invocation.invocationId, + status: 'pending', + }); + }); + + it('binds registration to the original execution principal', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + expect(invocation.principal).toEqual(PRINCIPAL); + await expect( + h.journal.register(ADDRESS, 10_000, { + kind: 'human', + id: 'other', + role: 'admin', + }), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('returns raw effect authority only to the winning consume call', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + expect(permit).toMatchObject({ generation: 1, expiresAt: 2_000 }); + const winner = await h.journal.consumePermit(permit); + expect(winner).toMatchObject({ + kind: 'accepted', + rawPermit: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/), + }); + expect(await h.journal.consumePermit(permit)).toEqual({ + kind: 'already-consumed', + }); + const stored = h.sqlite + .prepare(`SELECT permit_digest FROM ${REMOTE_STEP_TABLE}`) + .get() as { permit_digest: string }; + expect(stored.permit_digest).toMatch(/^[a-f0-9]{64}$/); + expect(JSON.stringify(stored)).not.toContain( + (winner as { rawPermit: string }).rawPermit, + ); + await expect( + h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + expect( + (await h.journal.get(invocation.invocationId))?.permitConsumedAt, + ).toBeDefined(); + }); + + it('grants raw authority to one concurrent D1 consume winner', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const outcomes = await Promise.all([ + h.journal.consumePermit(permit), + h.reopen().consumePermit(permit), + ]); + expect(outcomes.filter((item) => item.kind === 'accepted')).toHaveLength(1); + expect(outcomes.filter((item) => item.kind === 'already-consumed')).toEqual( + [{ kind: 'already-consumed' }], + ); + const winner = outcomes.find((item) => item.kind === 'accepted'); + expect(winner).toMatchObject({ rawPermit: expect.any(String) }); + }); + + it('recovers one permit after a journal instance is replaced', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const recovered = h.reopen(); + expect(await recovered.issuePermit(invocation.invocationId, 1_000)).toEqual( + permit, + ); + expect(await recovered.consumePermit(permit)).toMatchObject({ + kind: 'accepted', + rawPermit: expect.any(String), + }); + expect(await h.reopen().consumePermit(permit)).toEqual({ + kind: 'already-consumed', + }); + }); + + it('stores one verified JSON result and replays only the same bytes', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + const bytes = '{ "value": "done" }'; + const evidence = '{"receipt":"verified-completion"}'; + expect( + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + bytes, + evidence, + ), + ).toMatchObject({ kind: 'recorded', resultJson: bytes }); + expect(h.verifyCompletion).toHaveBeenCalledOnce(); + expect(h.verifyCompletion).toHaveBeenCalledWith( + expect.objectContaining({ + invocation: expect.objectContaining({ + invocationId: invocation.invocationId, + }), + rawPermit: consumed.rawPermit, + resultJson: bytes, + evidenceJson: evidence, + }), + ); + expect( + await h + .reopen() + .recordResult( + invocation.invocationId, + consumed.rawPermit, + bytes, + evidence, + ), + ).toMatchObject({ kind: 'replay', resultJson: bytes }); + expect(h.verifyCompletion).toHaveBeenCalledOnce(); + await expect( + h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + evidence, + ), + ).rejects.toMatchObject({ status: 409 }); + await expect( + h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + bytes, + '{"receipt":"different"}', + ), + ).rejects.toMatchObject({ status: 409 }); + const stored = h.sqlite + .prepare( + `SELECT result_json, result_digest, executor_quiescent_at + FROM ${REMOTE_STEP_TABLE}`, + ) + .get() as Record; + expect(stored).toMatchObject({ + result_json: bytes, + result_digest: expect.stringMatching(/^[a-f0-9]{64}$/), + executor_quiescent_at: expect.any(Number), + }); + expect( + await h.journal.readRetainedResult(invocation.invocationId), + ).toMatchObject({ + resultJson: bytes, + resultDigest: stored.result_digest, + }); + }); + + it('retains drain work after an unproven run advancement', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); + h.setSummary({ runId: ADDRESS.runId, status: 'success' }); + h.sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run('{"status":"success"}'); + expect((await h.reopen().get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('settles a witnessed new suspension and keeps callback replay idempotent', async () => { + const h = await fixture(); + const { invocation, rawPermit } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 1_000); + await expect( + h.reopen().claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + const next: RunSummary = { + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['next']], + suspendedAt: { next: 1_100 }, + suspendPayload: { next: { type: 'suspension' } }, + }; + expect( + await h.reopen().settleAfterResume(invocation.invocationId, owner, next), + ).toBe('retry'); + const nextOwner = await h.journal.claimResume( + invocation.invocationId, + 1_000, + ); + h.setSummary(next); + expect( + await h + .reopen() + .settleAfterResume(invocation.invocationId, nextOwner, next), + ).toBe('applied'); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'settled', + settlementOutcome: 'applied', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(0); + expect(await h.journal.recoverableResults(1)).toEqual({ + invocationIds: [], + }); + expect( + await h.journal.recoverableResults(1, invocation.invocationId), + ).toEqual({ invocationIds: [] }); + expect( + await h.journal.recordResult( + invocation.invocationId, + rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'replay' }); + expect( + await h.journal.readRetainedResult(invocation.invocationId), + ).toBeUndefined(); + h.sqlite + .prepare( + `UPDATE ${REMOTE_STEP_TABLE} SET result_digest = NULL, write_revision = write_revision + 1, write_generation = write_generation + 1`, + ) + .run(); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('retains an expired claim across another journal instance until its owner settles', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 100); + expect(await h.reopen().recoverableResults(10)).toEqual({ + invocationIds: [], + }); + h.setNow(1_101); + expect(await h.reopen().recoverableResults(10)).toEqual({ + invocationIds: [invocation.invocationId], + }); + await expect( + h.journal.claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + expect(await h.reopen().reconcileResult(invocation.invocationId)).toBe( + 'pending-quiescence', + ); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'resuming', + }); + expect( + h.sqlite + .prepare(`SELECT resume_lease_owner FROM ${REMOTE_STEP_TABLE}`) + .get(), + ).toEqual({ resume_lease_owner: owner }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + h.setSummary({ runId: ADDRESS.runId, status: 'success' }); + expect( + await h.journal.settleAfterResume(invocation.invocationId, owner, { + runId: ADDRESS.runId, + status: 'success', + }), + ).toBe('applied'); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'settled', + settlementOutcome: 'applied', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(0); + }); + + it('retains uncertain advancement when no resume lease was claimed', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + h.setSummary({ runId: ADDRESS.runId, status: 'success' }); + expect(await h.reopen().reconcileResult(invocation.invocationId)).toBe( + 'advanced-without-proof', + ); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'failed', + settlementOutcome: 'advanced-without-proof', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + await expect( + h.journal.claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('records cancellation separately from a successful result application', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 1_000); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + expect( + await h.journal.settleAfterResume(invocation.invocationId, owner), + ).toBe('terminal-not-success'); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'failed', + settlementOutcome: 'terminal-not-success', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(0); + }); + + it('records verified completion from a prior epoch without resuming a locked run', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + expect( + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'recorded' }); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('retains a verified late result as drain work after cancellation', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + h.sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run('{"status":"cancelled"}'); + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('keeps a consumed claim active if completion verification fails', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + h.verifyCompletion.mockRejectedValueOnce(new Error('unverified executor')); + await expect( + h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"unverified"}', + ), + ).rejects.toThrow('unverified executor'); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'dispatched', + ); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('refuses a wrong raw permit before calling the completion verifier', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + const wrong = + consumed.rawPermit === 'A'.repeat(43) ? 'B'.repeat(43) : 'A'.repeat(43); + await expect( + h.journal.recordResult( + invocation.invocationId, + wrong, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).rejects.toMatchObject({ status: 403 }); + expect(h.verifyCompletion).not.toHaveBeenCalled(); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'dispatched', + ); + }); + + it.each([ + 'issue', + 'consume', + ] as const)('refuses %s when the stored suspension request changes', async (operation) => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = + operation === 'consume' + ? await h.journal.issuePermit(invocation.invocationId, 1_000) + : undefined; + h.setRequest('other'); + await expect( + permit + ? h.journal.consumePermit(permit) + : h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('tightens a repeated registration to the earlier host deadline', async () => { + const h = await fixture(); + const first = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const repeated = await h.journal.register(ADDRESS, 1_500, PRINCIPAL); + expect(repeated.invocationId).toBe(first.invocationId); + expect(repeated.deadlineAt).toBe(1_500); + h.setNow(1_501); + await expect( + h.journal.issuePermit(first.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('caps a permit when the host deadline tightens after preflight', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("status = 'dispatched', permit_generation")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + await h.journal.register(ADDRESS, 1_500, PRINCIPAL); + return bound.all(); + }, + }; + }, + }; + }; + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + expect(permit.expiresAt).toBe(1_500); + h.setNow(1_501); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + }); + + it.each([ + 'issue', + 'consume', + ] as const)('refuses %s while a cancellation intent is persisted', async (operation) => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = + operation === 'consume' + ? await h.journal.issuePermit(invocation.invocationId, 1_000) + : undefined; + h.sqlite.prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?').run( + JSON.stringify({ + status: 'suspended', + requestContext: { + 'flowsafe.runLifecycle': { + version: 1, + revision: 2, + transitionIntent: { + status: 'cancelled', + requestedAt: 1_000, + replayPrincipals: [{ kind: 'human', id: 'operator-1' }], + }, + }, + }, + }), + ); + await expect( + permit + ? h.journal.consumePermit(permit) + : h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('rejects a remote request larger than its stored payload limit', async () => { + const h = await fixture(); + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 500 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: 'x'.repeat(16_384), + }, + }, + }, + }); + await expect( + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + ).rejects.toMatchObject({ + status: 400, + }); + }); + + it('refuses a marker from a tool the host did not designate', async () => { + const h = await fixture(); + const restricted = new RemoteStepJournal({ + db: h.db, + fence: h.fence, + runtime: h.runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['other_tool'], + verifyCompletion: h.verifyCompletion, + }); + await expect( + restricted.register(ADDRESS, 10_000, PRINCIPAL), + ).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses a corrupted persisted table identity before building a query', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + h.sqlite + .prepare( + `UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?, write_revision = write_revision + 1, write_generation = write_generation + 1`, + ) + .run('bad"_prefix'); + await expect( + h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses an unconsumed permit after the fence moves to migration lock', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 503, + }); + expect( + (await h.journal.get(invocation.invocationId))?.permitConsumedAt, + ).toBeUndefined(); + }); + + it('refuses a permit when the fence moves after preflight but before the write', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes('SET permit_consumed_at = ?')) return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + return bound.all(); + }, + }; + }, + }; + }; + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + expect( + (await h.journal.get(invocation.invocationId))?.permitConsumedAt, + ).toBeUndefined(); + }); + + it('refuses dispatch after cancellation or deadline', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + h.sqlite + .prepare(`UPDATE mastra_workflow_snapshot SET snapshot = ?`) + .run('{"status":"cancelled"}'); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + h.setNow(10_001); + await expect( + h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 409, + }); + }); + + it('retains a workflow result across journal reopen and rejects permit replay', async () => { + const h = await workflowFixture(); + const journal = h.journal(); + const first = await journal.registerWorkflow(h.address, 10_000, PRINCIPAL); + const replay = await journal.registerWorkflow(h.address, 10_000, PRINCIPAL); + expect(replay.invocationId).toBe(first.invocationId); + expect(first.requestDigest).toBe( + 'f908a6c18fa0b52b0f1295fc8f1cb5db083020cb18dab24ebf26637f55a9af49', + ); + expect(await journal.get(first.invocationId)).toBeUndefined(); + expect( + await journal.readWorkflowDispatchRequestJson(first.invocationId), + ).toBe('{"version":2,"input":{"query":"x"}}'); + await expect( + journal.readDispatchRequestJson(first.invocationId), + ).rejects.toMatchObject({ status: 404 }); + expect(await journal.getWorkflow(first.invocationId)).toMatchObject({ + targetKind: 'workflow', + status: 'pending', + stepPath: ['execute'], + }); + await expect( + h.reopen().issuePermit(first.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 503, + }); + const permit = await journal.issuePermit(first.invocationId, 1_000); + const winner = await journal.consumePermit(permit); + if (winner.kind !== 'accepted') throw new Error('permit was not won'); + expect(await h.journal().consumePermit(permit)).toEqual({ + kind: 'already-consumed', + }); + await expect( + journal.recordResult( + first.invocationId, + 'A'.repeat(43), + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).rejects.toMatchObject({ status: 403 }); + expect(h.verifyWorkflowCompletion).not.toHaveBeenCalled(); + expect( + await journal.recordResult( + first.invocationId, + winner.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'recorded' }); + expect(h.verifyWorkflowCompletion).toHaveBeenCalledExactlyOnceWith({ + invocation: expect.objectContaining({ + invocationId: first.invocationId, + targetKind: 'workflow', + requestDigest: first.requestDigest, + status: 'dispatched', + }), + rawPermit: winner.rawPermit, + resultJson: '{"value":"done"}', + evidenceJson: '{"receipt":"verified"}', + }); + const reopened = h.journal(); + await reopened.initialize(); + expect(await reopened.readRetainedResult(first.invocationId)).toMatchObject( + { + resultJson: '{"value":"done"}', + }, + ); + expect(await reopened.recoverableResults(10)).toEqual({ + invocationIds: [], + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).entries, + ).toEqual([expect.objectContaining({ key: [first.invocationId] })]); + expect( + await reopened.recordResult( + first.invocationId, + winner.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'replay' }); + await expect( + reopened.recordResult( + first.invocationId, + winner.rawPermit, + '{"value":"changed"}', + '{"receipt":"verified"}', + ), + ).rejects.toMatchObject({ status: 409 }); + expect(h.verifyWorkflowCompletion).toHaveBeenCalledTimes(1); + await expect( + reopened.claimResume(first.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 404, + }); + }); + + it('refuses workflow dispatch when trusted provenance or suspension changes', async () => { + const h = await workflowFixture(); + const journal = h.journal(); + await expect( + journal.registerWorkflow(h.address, 10_000, { + ...PRINCIPAL, + id: 'another-operator', + }), + ).rejects.toMatchObject({ status: 409 }); + const first = await journal.registerWorkflow(h.address, 10_000, PRINCIPAL); + h.setSummary(h.summary({ version: 2, input: { query: 'changed' } })); + await expect( + journal.issuePermit(first.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 409, + }); + h.setSummary(h.summary({ version: 2, input: { query: 'x' } })); + const permit = await journal.issuePermit(first.invocationId, 1_000); + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + await expect(journal.consumePermit(permit)).rejects.toMatchObject({ + status: 503, + }); + }); + + it('rejects stale v4-style writes after v5 migration', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + restoreV3Schema(h); + await h.reopen().initialize(); + expect(() => + h.sqlite + .prepare( + `UPDATE ${REMOTE_STEP_TABLE} SET deadline_at = ?, write_revision = write_revision + 1 WHERE invocation_id = ?`, + ) + .run(9_000, invocation.invocationId), + ).toThrow('remote step writer version changed'); + expect(() => + h.sqlite + .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( + invocation_id, principal_json, start_token, deployment_id, + table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, + step_path, suspended_at, resume_count, request_json, deadline_at, status) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending')`) + .run( + 'old-writer', + JSON.stringify(PRINCIPAL), + 'start-token', + ADDRESS.deploymentId, + ADDRESS.tablePrefix, + ADDRESS.workflowId, + ADDRESS.agentId, + ADDRESS.threadId, + ADDRESS.resourceId, + ADDRESS.runId, + '["execute"]', + 500, + -1, + '{}', + 10_000, + ), + ).toThrow(); + expect(await h.reopen().get(invocation.invocationId)).toMatchObject({ + deadlineAt: 10_000, + }); + }); + + it('binds a notification effect and one continuation admission to the retained result', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const input = { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + payload: { value: 'done' }, + createdAt: new Date('2026-10-13T00:00:00.000Z'), + deliverAt: new Date('2026-10-14T00:00:00.000Z'), + }; + const prepared = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + input, + ); + const projected = await projectNotificationOperation({ + ...input, + id: prepared.notificationId, + }); + expect(prepared).toMatchObject({ + state: 'prepared', + projectionVersion: 'notification-record-values-v1', + projectionJson: projected.requestJson, + requestDigest: projected.requestDigest, + }); + expect(await h.reopen().recoverableResults(10)).toEqual({ + invocationIds: [], + }); + await expect( + h.journal.claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + expect( + (await h.reopen().admitNotificationEffect(invocation.invocationId)).state, + ).toBe('effect-admitted'); + const notifications = new D1NotificationsStorage(h.db); + const receipt = await notifications.createNotificationOnce( + prepared.operationKey, + { ...input, id: prepared.notificationId }, + ); + expect(receipt.requestDigest).toBe(prepared.requestDigest); + expect( + ( + await h.journal.recordNotificationReceipt( + invocation.invocationId, + receipt, + ) + ).state, + ).toBe('receipt-recorded'); + expect( + await notifications.readNotificationOperationReceipt( + prepared.operationKey, + ), + ).toEqual(receipt); + const owner = await h + .reopen() + .admitNotificationResume(invocation.invocationId); + expect(owner).toMatch(/^[a-f0-9-]{36}$/); + await expect( + h.journal.admitNotificationResume(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ + state: 'resume-admitted', + requestDigest: receipt.requestDigest, + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('retains an unwitnessed notification continuation in drain inventory after terminal failure', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const input = { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + input, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + const receipt = await new D1NotificationsStorage( + h.db, + ).createNotificationOnce(operation.operationKey, { + ...input, + id: operation.notificationId, + }); + await h.journal.recordNotificationReceipt(invocation.invocationId, receipt); + const owner = await h.journal.admitNotificationResume( + invocation.invocationId, + ); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + expect( + await h.journal.settleAfterResume(invocation.invocationId, owner), + ).toBe('terminal-not-success'); + const inventory = await new DeploymentInventory(h.db).read('remote-steps'); + expect(inventory.count).toBe(1); + expect(inventory.entries[0]).toMatchObject({ + key: [invocation.invocationId], + detail: { + status: 'failed', + operation_state: 'resume-admitted', + settlement_outcome: 'terminal-not-success', + }, + }); + }); + + it('persists a bounded notification recovery cursor across instances and wraps past the final active row', async () => { + const h = await fixture(); + const ids: string[] = []; + for (const suspendedAt of [500, 600, 700]) { + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': suspendedAt }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: `call-${suspendedAt}`, + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + ids.push(invocation.invocationId); + } + const expected = ids.sort(); + expect(await h.journal.nextNotificationRecoveryBatch(1)).toEqual([ + expected[0], + ]); + expect(await h.reopen().nextNotificationRecoveryBatch(1)).toEqual([ + expected[1], + ]); + expect(await h.reopen().nextNotificationRecoveryBatch(1)).toEqual([ + expected[2], + ]); + expect(await h.reopen().nextNotificationRecoveryBatch(1)).toEqual([ + expected[0], + ]); + h.sqlite + .prepare( + `UPDATE ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} SET version = 2`, + ) + .run(); + await expect( + h.reopen().nextNotificationRecoveryBatch(1), + ).rejects.toMatchObject({ status: 503 }); + }); + + it('routes recovery with the stored principal and continues past a failed item', async () => { + const h = await fixture(); + const ids: string[] = []; + for (const suspendedAt of [500, 600]) { + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': suspendedAt }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: `call-${suspendedAt}`, + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + ids.push(invocation.invocationId); + } + const sorted = ids.sort(); + const firstId = sorted[0]; + const secondId = sorted[1]; + if (!firstId || !secondId) throw new Error('recovery items are missing'); + const secret = 'test-deployment-identity-secret-0001'; + const sent: string[] = []; + let failFirst = true; + const namespace: ThreadNamespaceLike = { + idFromName: (name) => name, + get: (threadId) => ({ + fetch: (async ( + url: Request | string, + requestInit?: ThreadRequestInit, + ) => { + const request = + typeof url === 'string' ? new Request(url, requestInit) : url; + if ( + threadId !== ADDRESS.threadId || + !(await credentialsMatch( + request.headers.get(DEPLOYMENT_IDENTITY_HEADER) ?? '', + secret, + )) || + JSON.stringify( + decodeExecutionPrincipal( + request.headers.get(EXECUTION_PRINCIPAL_HEADER) ?? '', + ), + ) !== JSON.stringify(PRINCIPAL) + ) + return new Response(null, { status: 403 }); + const body = (await request.json()) as { invocationId: string }; + sent.push(body.invocationId); + if (body.invocationId === firstId && failFirst) + return new Response(null, { status: 503 }); + const operation = await h.journal.readNotificationOperation( + body.invocationId, + ); + if (!operation) return new Response(null, { status: 404 }); + if (operation.state === 'effect-admitted') { + const saved = JSON.parse(operation.requestJson) as Omit< + NotificationOperationInput, + 'createdAt' + > & { + createdAt: string; + }; + const receipt = await h.journal + .notificationStorage() + .createNotificationOnce(operation.operationKey, { + ...saved, + createdAt: new Date(saved.createdAt), + }); + await h.journal.recordNotificationReceipt( + body.invocationId, + receipt, + ); + } + return Response.json({ state: 'receipt-recorded' }); + }) as ReturnType['get']>['fetch'], + }), + }; + const tick = createRemoteStepNotificationRecoveryTick({ + journal: h.journal, + namespace, + deploymentIdentitySecret: secret, + limit: 2, + }); + await expect(tick()).rejects.toMatchObject({ status: 503 }); + expect(sent).toEqual(sorted); + expect((await h.journal.readNotificationOperation(secondId))?.state).toBe( + 'receipt-recorded', + ); + failFirst = false; + expect(await tick()).toEqual({ + scanned: 2, + recovered: 1, + unresolved: 1, + }); + expect(sent).toEqual([...sorted, firstId]); + }); + + it('advances a limit-one remote-step inventory cursor past active rows', async () => { + const h = await fixture(); + const first = await h.retainResult(); + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 600 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-2', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + const second = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + expect(second.invocationId).not.toBe(first.invocation.invocationId); + const inventory = new DeploymentInventory(h.db); + const page1 = await inventory.read('remote-steps', { limit: 1 }); + const page2 = await inventory.read('remote-steps', { + limit: 1, + cursor: page1.cursor, + }); + expect(page1.count).toBe(2); + expect(page2.entries).toHaveLength(1); + expect(page2.entries[0]?.key).not.toEqual(page1.entries[0]?.key); + expect(page2.count).toBeUndefined(); + expect( + ( + await inventory.read('remote-steps', { + limit: 1, + cursor: page2.cursor, + }) + ).entries, + ).toEqual([]); + }); + + it('refuses a second continuation admission after a committed response is lost and the lease expires', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + h.sqlite + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET deadline_at = ?, write_revision = write_revision + 1, + write_generation = write_generation + 1 WHERE invocation_id = ?`) + .run(100_000, invocation.invocationId); + const input = { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + input, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + const receipt = await new D1NotificationsStorage( + h.db, + ).createNotificationOnce(operation.operationKey, { + ...input, + id: operation.notificationId, + }); + await h.journal.recordNotificationReceipt(invocation.invocationId, receipt); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("operation_state = 'resume-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + await bound.all(); + throw new Error('lost continuation admission response'); + }, + }; + }, + }; + }; + await expect( + h.journal.admitNotificationResume(invocation.invocationId), + ).rejects.toThrow('lost continuation admission response'); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ + state: 'resume-admitted', + requestDigest: receipt.requestDigest, + }); + h.setNow(61_001); + await expect( + h.reopen().admitNotificationResume(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('refuses effect admission when the authoritative snapshot changes at the D1 write', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("SET operation_state = 'effect-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + h.sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run('{"status":"cancelled"}'); + return bound.all(); + }, + }; + }, + }; + }; + await expect( + h.journal.admitNotificationEffect(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'prepared' }); + }); + + it('refuses a changed notification receipt after effect admission', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + await expect( + h.journal.recordNotificationReceipt(invocation.invocationId, { + operationKey: operation.operationKey, + notificationId: operation.notificationId, + threadId: ADDRESS.threadId, + requestDigest: 'f'.repeat(64), + }), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'effect-admitted' }); + }); + + it('refuses a matching claimed receipt when D1 has no notification effect', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + await new D1NotificationsStorage(h.db).init(); + await h.journal.admitNotificationEffect(invocation.invocationId); + await expect( + h.journal.recordNotificationReceipt(invocation.invocationId, { + operationKey: operation.operationKey, + notificationId: operation.notificationId, + threadId: ADDRESS.threadId, + requestDigest: operation.requestDigest, + }), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'effect-admitted' }); + }); + + it('refuses stale principal, start token, result and deadline before effect admission', async () => { + const cases = [ + { + column: 'principal_json', + value: JSON.stringify({ kind: 'human', id: 'other', role: 'operator' }), + status: 409, + }, + { column: 'start_token', value: 'other-start', status: 409 }, + { column: 'result_digest', value: 'f'.repeat(64), status: 503 }, + { column: 'deadline_at', value: 1_000, status: 409 }, + ] as const; + for (const changed of cases) { + const h = await fixture(); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + h.sqlite + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET ${changed.column} = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1`) + .run(changed.value); + await expect( + h.journal.admitNotificationEffect(invocation.invocationId), + ).rejects.toMatchObject({ status: changed.status }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'prepared' }); + } + }); + + it('refuses effect admission when the fence moves after its read', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("SET operation_state = 'effect-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + return bound.all(); + }, + }; + }, + }; + }; + await expect( + h.journal.admitNotificationEffect(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'prepared' }); + }); + + it('reads durable effect admission after its write response is lost', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const prepared = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + const prepare = h.db.prepare.bind(h.db); + let interrupted = false; + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("SET operation_state = 'effect-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + const result = await bound.all(); + if (!interrupted) { + interrupted = true; + throw new Error('lost admission response'); + } + return result; + }, + }; + }, + }; + }; + expect( + await h.journal.admitNotificationEffect(invocation.invocationId), + ).toMatchObject({ + state: 'effect-admitted', + operationKey: prepared.operationKey, + }); + expect( + await h.reopen().admitNotificationEffect(invocation.invocationId), + ).toMatchObject({ + state: 'effect-admitted', + operationKey: prepared.operationKey, + }); + }); + + it('refuses a real suspended snapshot after runtime persists cancellation intent', async () => { + const sqlite = openSqlite() as ReturnType & { + close(): void; + }; + databases.push(sqlite); + const db = sqliteUnitDatabase(sqlite); + const storage = createD1Storage({ binding: db }); + await storage.init(); + const fence = new ExecutionFenceStore(db); + await fence.seed('open'); + const { createWorkflow, createStep, runtime } = init( + { storage }, + { executionFence: fence, startIdempotency: 'none' }, + ); + const gate = createStep({ + id: 'durable-agentic-execution', + inputSchema: z.object({}), + outputSchema: z.object({}), + suspendSchema: z.object({ + type: z.literal('suspension'), + toolCallId: z.string(), + toolName: z.string(), + toolCallSuspended: z.object({ + remoteStep: z.literal(REMOTE_STEP_MARKER), + request: z.object({ account: z.string() }), + }), + }), + execute: async ({ suspend }) => + suspend({ + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }), + }); + createWorkflow({ + id: ADDRESS.workflowId, + inputSchema: z.object({}), + outputSchema: z.object({}), + }) + .then(gate) + .commit(); + const started = await runtime.start(ADDRESS.workflowId, { + runId: ADDRESS.runId, + inputData: {}, + requestedBy: 'operator-1', + requestedByKind: 'human', + startIdentity: { + owner: { kind: 'human', id: 'operator-1' }, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + resourceId: ADDRESS.resourceId, + }, + }, + agentStart: { threaded: false }, + }); + expect(started.status).toBe('suspended'); + const journal = new RemoteStepJournal({ + db, + fence, + runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + ADDRESS, + Date.now() + 60_000, + PRINCIPAL, + ); + const permit = await journal.issuePermit(invocation.invocationId, 30_000); + expect(permit.generation).toBe(1); + expect( + await runtime.cancelActiveExecution( + ADDRESS.workflowId, + ADDRESS.runId, + 'cancelled', + [{ kind: 'human', id: 'operator-1' }], + ), + ).toBe(false); + expect( + await runtime.authoritativeStatus(ADDRESS.workflowId, ADDRESS.runId), + ).toMatchObject({ status: 'suspended' }); + const stored = sqlite + .prepare( + `SELECT json_type(snapshot, + '$.requestContext."flowsafe.runLifecycle".transitionIntent') AS intent + FROM mastra_workflow_snapshot WHERE workflow_name = ? AND run_id = ?`, + ) + .get(ADDRESS.workflowId, ADDRESS.runId) as { intent: string }; + expect(stored.intent).toBe('object'); + await expect(journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + }); +}); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts new file mode 100644 index 00000000..249c9483 --- /dev/null +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -0,0 +1,2848 @@ +// SPDX-License-Identifier: Apache-2.0 + +import { EXECUTION_FENCE_TABLE } from '#deployment-identity-protocol'; +import { + assertExecutionPrincipal, + type ExecutionPrincipal, +} from '../approval-api/principal.js'; +import type { SignalDatabase } from '../signals/d1-shared.js'; +import { + D1NotificationsStorage, + DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, + NOTIFICATION_OPERATION_PROJECTION_VERSION, + NOTIFICATION_OPERATION_RECEIPTS_TABLE, + type NotificationOperationInput, + projectNotificationOperation, +} from '../signals/notifications-d1.js'; +import { credentialsMatch } from './deployment-identity.js'; +import { DoStatusError } from './do-status-error.js'; +import { + admitsExistingRun, + captureExecutionFenceAdmissionSchema, + type ExecutionFenceDatabase, + type ExecutionFenceStore, + executionFenceAdmissionSql, + executionFenceAdmissionValues, +} from './execution-fence.js'; +import { isPathSafeId } from './path-safe-id.js'; +import { + lifecycleFromRequestContext, + RUN_LIFECYCLE_CONTEXT_KEY, +} from './run-lifecycle.js'; +import type { RunnerRuntime, RunSummary } from './runtime.js'; +import { validateTablePrefix } from './table-prefix.js'; + +export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; +export const REMOTE_STEP_SCHEMA_TABLE = 'flowsafe_remote_step_schema'; +export const REMOTE_STEP_NOTIFICATION_CURSOR_TABLE = + 'flowsafe_remote_step_notification_cursor'; +const REMOTE_STEP_NOTIFICATION_CURSOR_SQL = `CREATE TABLE ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} ( + deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, + version INTEGER NOT NULL, + after_invocation_id TEXT NOT NULL, + PRIMARY KEY (deployment_id, table_prefix) +)`; +const REMOTE_STEP_RECOVERY_INDEX = 'flowsafe_remote_step_recovery'; +const REMOTE_STEP_RECOVERY_INDEX_SQL = `CREATE INDEX ${REMOTE_STEP_RECOVERY_INDEX} ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE status IN ('result-retained', 'resuming')`; +const REMOTE_STEP_AGENT_INDEX = 'flowsafe_remote_step_agent_suspension'; +const REMOTE_STEP_WORKFLOW_INDEX = 'flowsafe_remote_step_workflow_suspension'; +const REMOTE_STEP_AGENT_INDEX_SQL = `CREATE UNIQUE INDEX ${REMOTE_STEP_AGENT_INDEX} ON ${REMOTE_STEP_TABLE} (deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count) WHERE target_kind = 'agent'`; +const REMOTE_STEP_WORKFLOW_INDEX_SQL = `CREATE UNIQUE INDEX ${REMOTE_STEP_WORKFLOW_INDEX} ON ${REMOTE_STEP_TABLE} (deployment_id, table_prefix, workflow_id, run_id, step_path, suspended_at, resume_count) WHERE target_kind = 'workflow'`; +const REMOTE_STEP_WRITE_TRIGGER = 'flowsafe_remote_step_write_version'; +const REMOTE_STEP_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; +const REMOTE_STEP_V5_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_generation <= OLD.write_generation OR NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; +const REMOTE_STEP_OPERATION_INDEX = + 'flowsafe_remote_step_notification_recovery'; +const REMOTE_STEP_OPERATION_INDEX_SQL = `CREATE INDEX ${REMOTE_STEP_OPERATION_INDEX} ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved')`; +const REMOTE_STEP_RESULT_COLUMNS = [ + 'result_json', + 'result_digest', + 'quiescence_digest', + 'executor_quiescent_at', +] as const; +const REMOTE_STEP_SETTLEMENT_COLUMNS = [ + 'resume_lease_owner', + 'resume_lease_expires_at', + 'settled_at', + 'settlement_outcome', +] as const; +const REMOTE_STEP_COLUMNS = [ + 'invocation_id', + 'principal_json', + 'start_token', + 'deployment_id', + 'table_prefix', + 'workflow_id', + 'agent_id', + 'thread_id', + 'resource_id', + 'run_id', + 'step_path', + 'suspended_at', + 'resume_count', + 'request_json', + 'deadline_at', + 'status', + 'permit_generation', + 'permit_epoch', + 'permit_revision', + 'permit_expires_at', + 'permit_consumed_at', + 'permit_digest', + ...REMOTE_STEP_RESULT_COLUMNS, + ...REMOTE_STEP_SETTLEMENT_COLUMNS, + 'target_kind', + 'suspension_marker', + 'request_digest', + 'write_revision', +] as const; +const REMOTE_STEP_V5_COLUMNS = [ + ...REMOTE_STEP_COLUMNS, + 'write_generation', + 'operation_key', + 'operation_id', + 'notification_id', + 'notification_request_json', + 'notification_projection_version', + 'notification_projection_json', + 'notification_request_digest', + 'operation_state', + 'effect_admitted_at', + 'receipt_recorded_at', + 'resume_admitted_at', +] as const; +const REMOTE_STEP_INTEGER_COLUMNS = new Set([ + 'suspended_at', + 'resume_count', + 'deadline_at', + 'permit_generation', + 'permit_epoch', + 'permit_revision', + 'permit_expires_at', + 'permit_consumed_at', + 'executor_quiescent_at', + 'resume_lease_expires_at', + 'settled_at', + 'write_revision', + 'write_generation', + 'effect_admitted_at', + 'receipt_recorded_at', + 'resume_admitted_at', +]); +export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; +export const WORKFLOW_REMOTE_STEP_MARKER = 'flowsafe.remote-step.workflow/v1'; +export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; +export const MAX_REMOTE_STEP_RESULT_BYTES = 16_384; + +const SHA256_HEX = /^[a-f0-9]{64}$/; +const RAW_PERMIT = /^[A-Za-z0-9_-]{43}$/; + +function remoteStepTableSql(name: string, version: 4 | 5 = 5): string { + return `CREATE TABLE IF NOT EXISTS ${name} ( + invocation_id TEXT PRIMARY KEY, + principal_json TEXT NOT NULL, + start_token TEXT NOT NULL, + deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, + workflow_id TEXT NOT NULL, + agent_id TEXT, + thread_id TEXT, + resource_id TEXT, + run_id TEXT NOT NULL, + step_path TEXT NOT NULL, + suspended_at INTEGER NOT NULL, + resume_count INTEGER NOT NULL, + request_json TEXT NOT NULL, + deadline_at INTEGER NOT NULL, + status TEXT NOT NULL, + permit_generation INTEGER NOT NULL DEFAULT 0, + permit_epoch INTEGER, + permit_revision INTEGER, + permit_expires_at INTEGER, + permit_consumed_at INTEGER, + permit_digest TEXT, + result_json TEXT, + result_digest TEXT, + quiescence_digest TEXT, + executor_quiescent_at INTEGER, + resume_lease_owner TEXT, + resume_lease_expires_at INTEGER, + settled_at INTEGER, + settlement_outcome TEXT, + target_kind TEXT NOT NULL CHECK (target_kind IN ('agent', 'workflow')), + suspension_marker TEXT NOT NULL, + request_digest TEXT, + write_revision INTEGER NOT NULL DEFAULT 0${ + version === 5 + ? `, + write_generation INTEGER NOT NULL CHECK (write_generation > 0), + operation_key TEXT UNIQUE, + operation_id TEXT, + notification_id TEXT, + notification_request_json TEXT, + notification_projection_version TEXT, + notification_projection_json TEXT, + notification_request_digest TEXT, + operation_state TEXT CHECK (operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved')), + effect_admitted_at INTEGER, + receipt_recorded_at INTEGER, + resume_admitted_at INTEGER, + CHECK ((operation_state IS NULL AND operation_key IS NULL AND operation_id IS NULL + AND notification_id IS NULL AND notification_request_json IS NULL + AND notification_projection_version IS NULL AND notification_projection_json IS NULL + AND notification_request_digest IS NULL AND effect_admitted_at IS NULL + AND receipt_recorded_at IS NULL AND resume_admitted_at IS NULL) OR + (target_kind = 'agent' AND operation_key IS NOT NULL AND operation_id IS NOT NULL + AND notification_id IS NOT NULL AND notification_request_json IS NOT NULL + AND notification_projection_version = 'notification-record-values-v1' + AND notification_projection_json IS NOT NULL AND notification_request_digest IS NOT NULL))` + : '' + }, + CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL + AND thread_id IS NOT NULL AND resource_id IS NOT NULL + AND suspension_marker = '${REMOTE_STEP_MARKER}') OR + (target_kind = 'workflow' AND agent_id IS NULL + AND thread_id IS NULL AND resource_id IS NULL + AND suspension_marker = '${WORKFLOW_REMOTE_STEP_MARKER}' + AND request_digest IS NOT NULL)) + )`; +} + +function boundedJsonBytes(value: string): void { + if ( + typeof value !== 'string' || + new TextEncoder().encode(value).length > MAX_REMOTE_STEP_RESULT_BYTES + ) + throw new RemoteStepRefusal(400, 'remote step JSON exceeds its limit'); + try { + JSON.parse(value); + } catch { + throw new RemoteStepRefusal(400, 'remote step JSON is malformed'); + } +} + +function rawDispatchPermit(): string { + const bytes = crypto.getRandomValues(new Uint8Array(32)); + let binary = ''; + for (const byte of bytes) binary += String.fromCharCode(byte); + return btoa(binary) + .replace(/\+/g, '-') + .replace(/\//g, '_') + .replace(/=+$/, ''); +} + +async function sha256Hex(value: string): Promise { + const digest = await crypto.subtle.digest( + 'SHA-256', + new TextEncoder().encode(value), + ); + return Array.from(new Uint8Array(digest), (byte) => + byte.toString(16).padStart(2, '0'), + ).join(''); +} + +const SNAPSHOT_EXECUTION_OPEN_SQL = `json_valid(snapshot) + AND json_type(snapshot, '$.requestContext."${RUN_LIFECYCLE_CONTEXT_KEY}".transitionIntent') IS NULL + AND json_type(snapshot, '$.requestContext."${RUN_LIFECYCLE_CONTEXT_KEY}".terminal') IS NULL`; + +export interface RemoteStepAddress { + deploymentId: string; + tablePrefix: string; + workflowId: string; + agentId: string; + threadId: string; + resourceId: string; + runId: string; +} + +export interface WorkflowRemoteStepAddress { + targetKind: 'workflow'; + deploymentId: string; + tablePrefix: string; + workflowId: string; + runId: string; +} + +export interface RemoteStepInvocation extends RemoteStepAddress { + invocationId: string; + principal: ExecutionPrincipal; + startToken: string; + stepPath: readonly string[]; + suspendedAt: number; + resumeCount?: number; + request: unknown; + deadlineAt: number; + status: + | 'pending' + | 'dispatched' + | 'result-retained' + | 'resuming' + | 'settled' + | 'failed'; + permitGeneration: number; + permitEpoch?: number; + permitRevision?: number; + permitExpiresAt?: number; + permitConsumedAt?: number; + resumeLeaseExpiresAt?: number; + settlementOutcome?: + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof'; +} + +export interface WorkflowRemoteStepInvocation + extends Omit { + targetKind: 'workflow'; + requestDigest: string; +} + +export interface RemoteStepPermit { + invocationId: string; + generation: number; + mutationEpoch: number; + transitionRevision: number; + expiresAt: number; +} + +export interface RecordedRemoteStepResult { + invocationId: string; + resultJson: string; + resultDigest: string; + quiescenceDigest: string; + executorQuiescentAt: number; +} + +export interface RemoteStepNotificationOperation { + invocationId: string; + operationId: string; + operationKey: string; + notificationId: string; + requestJson: string; + projectionVersion: typeof NOTIFICATION_OPERATION_PROJECTION_VERSION; + projectionJson: string; + requestDigest: string; + state: + | 'prepared' + | 'effect-admitted' + | 'receipt-recorded' + | 'resume-admitted' + | 'unresolved'; + effectAdmittedAt?: number; + receiptRecordedAt?: number; + resumeAdmittedAt?: number; +} + +class RemoteStepRefusal extends DoStatusError { + readonly status: number; + + constructor(status: number, message: string) { + super(message); + this.name = 'RemoteStepRefusal'; + this.status = status; + } +} + +type JournalRow = { + invocation_id: string; + principal_json: string; + start_token: string; + deployment_id: string; + table_prefix: string; + workflow_id: string; + agent_id: string; + thread_id: string; + resource_id: string; + run_id: string; + step_path: string; + suspended_at: number; + resume_count: number; + request_json: string; + deadline_at: number; + status: string; + permit_generation: number; + permit_epoch: number | null; + permit_revision: number | null; + permit_expires_at: number | null; + permit_consumed_at: number | null; + permit_digest: string | null; + result_json: string | null; + result_digest: string | null; + quiescence_digest: string | null; + executor_quiescent_at: number | null; + resume_lease_owner: string | null; + resume_lease_expires_at: number | null; + settled_at: number | null; + settlement_outcome: string | null; + target_kind: string; + suspension_marker: string; + request_digest: string | null; + write_revision: number; + write_generation: number; + operation_key: string | null; + operation_id: string | null; + notification_id: string | null; + notification_request_json: string | null; + notification_projection_version: string | null; + notification_projection_json: string | null; + notification_request_digest: string | null; + operation_state: string | null; + effect_admitted_at: number | null; + receipt_recorded_at: number | null; + resume_admitted_at: number | null; +}; + +function record(value: unknown): Record | undefined { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +function positiveTime(value: unknown): value is number { + return Number.isSafeInteger(value) && (value as number) > 0; +} + +function rowResult(value: unknown): JournalRow | undefined { + const rows = record(value)?.results; + return Array.isArray(rows) && rows.length === 1 + ? (rows[0] as JournalRow) + : undefined; +} + +function validLifecycle(row: JournalRow): boolean { + return ( + Number.isSafeInteger(row.resume_count) && + (row.resume_count === -1 || row.resume_count >= 1) && + Number.isSafeInteger(row.permit_generation) && + row.permit_generation >= 0 && + [ + 'pending', + 'dispatched', + 'result-retained', + 'resuming', + 'settled', + 'failed', + ].includes(row.status) && + (row.settlement_outcome === null || + [ + 'applied', + 'terminal-not-success', + 'deadline-expired', + 'advanced-without-proof', + ].includes(row.settlement_outcome)) && + (row.status !== 'resuming' || + (isPathSafeId(row.resume_lease_owner) && + positiveTime(row.resume_lease_expires_at) && + row.result_json !== null)) && + (row.status === 'resuming' || + (row.resume_lease_owner === null && + row.resume_lease_expires_at === null)) && + (row.status !== 'settled' || + (row.settlement_outcome === 'applied' && + row.result_json !== null && + row.result_digest !== null && + positiveTime(row.executor_quiescent_at))) && + (row.settlement_outcome === null || + ((row.status === 'settled' || row.status === 'failed') && + positiveTime(row.settled_at))) + ); +} + +function validPayloadBounds(row: JournalRow): boolean { + return ( + typeof row.step_path === 'string' && + row.step_path.length <= 1_024 && + typeof row.request_json === 'string' && + new TextEncoder().encode(row.request_json).length <= + MAX_REMOTE_STEP_REQUEST_BYTES + ); +} + +function validStepPath(value: unknown): value is string[] { + return ( + Array.isArray(value) && + value.length > 0 && + value.every((part) => typeof part === 'string' && part !== '') + ); +} + +function decode(row: JournalRow): RemoteStepInvocation { + if ( + row.target_kind !== 'agent' || + row.suspension_marker !== REMOTE_STEP_MARKER || + (row.request_digest !== null && !SHA256_HEX.test(row.request_digest)) || + !Number.isSafeInteger(row.write_revision) || + row.write_revision < 0 || + !positiveTime(row.write_generation) || + !validPayloadBounds(row) || + typeof row.principal_json !== 'string' || + !isPathSafeId(row.deployment_id) || + !isPathSafeId(row.workflow_id) || + !isPathSafeId(row.agent_id) || + !isPathSafeId(row.thread_id) || + !isPathSafeId(row.resource_id) || + !isPathSafeId(row.run_id) || + !isPathSafeId(row.start_token) + ) + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + try { + validateTablePrefix(row.table_prefix); + } catch { + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + } + let stepPath: unknown; + let request: unknown; + let principal: ExecutionPrincipal; + try { + stepPath = JSON.parse(row.step_path); + request = JSON.parse(row.request_json); + principal = assertExecutionPrincipal( + JSON.parse(row.principal_json), + 'stored remote step principal', + ); + } catch { + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + } + if ( + !validStepPath(stepPath) || + !isPathSafeId(row.invocation_id) || + !positiveTime(row.suspended_at) || + !positiveTime(row.deadline_at) || + !validLifecycle(row) + ) + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + return { + invocationId: row.invocation_id, + principal, + startToken: row.start_token, + deploymentId: row.deployment_id, + tablePrefix: row.table_prefix, + workflowId: row.workflow_id, + agentId: row.agent_id, + threadId: row.thread_id, + resourceId: row.resource_id, + runId: row.run_id, + stepPath, + suspendedAt: row.suspended_at, + ...(row.resume_count < 0 ? {} : { resumeCount: row.resume_count }), + request, + deadlineAt: row.deadline_at, + status: row.status as RemoteStepInvocation['status'], + permitGeneration: row.permit_generation, + ...(row.permit_epoch === null ? {} : { permitEpoch: row.permit_epoch }), + ...(row.permit_revision === null + ? {} + : { permitRevision: row.permit_revision }), + ...(row.permit_expires_at === null + ? {} + : { permitExpiresAt: row.permit_expires_at }), + ...(row.permit_consumed_at === null + ? {} + : { permitConsumedAt: row.permit_consumed_at }), + ...(row.resume_lease_expires_at === null + ? {} + : { resumeLeaseExpiresAt: row.resume_lease_expires_at }), + ...(row.settlement_outcome === null + ? {} + : { + settlementOutcome: row.settlement_outcome as + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof', + }), + }; +} + +function decodeWorkflow(row: JournalRow): WorkflowRemoteStepInvocation { + if ( + row.target_kind !== 'workflow' || + row.suspension_marker !== WORKFLOW_REMOTE_STEP_MARKER || + row.agent_id !== null || + row.thread_id !== null || + row.resource_id !== null || + !isPathSafeId(row.invocation_id) || + !isPathSafeId(row.deployment_id) || + !isPathSafeId(row.workflow_id) || + !isPathSafeId(row.run_id) || + !isPathSafeId(row.start_token) || + typeof row.request_digest !== 'string' || + !SHA256_HEX.test(row.request_digest) || + !Number.isSafeInteger(row.write_revision) || + row.write_revision < 0 || + !positiveTime(row.write_generation) || + !positiveTime(row.suspended_at) || + !positiveTime(row.deadline_at) || + !validLifecycle(row) || + !validPayloadBounds(row) + ) + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + try { + validateTablePrefix(row.table_prefix); + const stepPath: unknown = JSON.parse(row.step_path); + const request: unknown = JSON.parse(row.request_json); + const principal = assertExecutionPrincipal( + JSON.parse(row.principal_json), + 'stored remote step principal', + ); + if (!validStepPath(stepPath)) throw new Error('invalid workflow row'); + return { + targetKind: 'workflow', + invocationId: row.invocation_id, + principal, + startToken: row.start_token, + deploymentId: row.deployment_id, + tablePrefix: row.table_prefix, + workflowId: row.workflow_id, + runId: row.run_id, + stepPath, + suspendedAt: row.suspended_at, + ...(row.resume_count < 0 ? {} : { resumeCount: row.resume_count }), + request, + requestDigest: row.request_digest, + deadlineAt: row.deadline_at, + status: row.status as WorkflowRemoteStepInvocation['status'], + permitGeneration: row.permit_generation, + ...(row.permit_epoch === null ? {} : { permitEpoch: row.permit_epoch }), + ...(row.permit_revision === null + ? {} + : { permitRevision: row.permit_revision }), + ...(row.permit_expires_at === null + ? {} + : { permitExpiresAt: row.permit_expires_at }), + ...(row.permit_consumed_at === null + ? {} + : { permitConsumedAt: row.permit_consumed_at }), + ...(row.resume_lease_expires_at === null + ? {} + : { resumeLeaseExpiresAt: row.resume_lease_expires_at }), + ...(row.settlement_outcome === null + ? {} + : { + settlementOutcome: + row.settlement_outcome as WorkflowRemoteStepInvocation['settlementOutcome'], + }), + }; + } catch { + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + } +} + +function decodeAny( + row: JournalRow, +): RemoteStepInvocation | WorkflowRemoteStepInvocation { + if (row.target_kind === 'agent') return decode(row); + if (row.target_kind === 'workflow') return decodeWorkflow(row); + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); +} + +function decodeNotificationOperation( + row: JournalRow, +): RemoteStepNotificationOperation | undefined { + if (row.operation_state === null) { + if ( + row.operation_key !== null || + row.operation_id !== null || + row.notification_id !== null || + row.notification_request_json !== null || + row.notification_projection_version !== null || + row.notification_projection_json !== null || + row.notification_request_digest !== null || + row.effect_admitted_at !== null || + row.receipt_recorded_at !== null || + row.resume_admitted_at !== null + ) + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + return undefined; + } + if ( + row.target_kind !== 'agent' || + !isPathSafeId(row.operation_key) || + !isPathSafeId(row.operation_id) || + !isPathSafeId(row.notification_id) || + typeof row.notification_request_json !== 'string' || + typeof row.notification_projection_json !== 'string' || + row.notification_projection_version !== + NOTIFICATION_OPERATION_PROJECTION_VERSION || + typeof row.notification_request_digest !== 'string' || + !SHA256_HEX.test(row.notification_request_digest) || + ![ + 'prepared', + 'effect-admitted', + 'receipt-recorded', + 'resume-admitted', + 'unresolved', + ].includes(row.operation_state) || + new TextEncoder().encode(row.notification_request_json).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + new TextEncoder().encode(row.notification_projection_json).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + (row.operation_state === 'prepared' && + (row.effect_admitted_at !== null || + row.receipt_recorded_at !== null || + row.resume_admitted_at !== null)) || + (row.operation_state !== 'prepared' && + !positiveTime(row.effect_admitted_at)) || + (row.operation_state === 'effect-admitted' && + (row.receipt_recorded_at !== null || row.resume_admitted_at !== null)) || + (['receipt-recorded', 'resume-admitted'].includes(row.operation_state) && + !positiveTime(row.receipt_recorded_at)) || + (row.operation_state === 'receipt-recorded' && + row.resume_admitted_at !== null) || + (row.operation_state === 'resume-admitted' && + !positiveTime(row.resume_admitted_at)) + ) + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + try { + JSON.parse(row.notification_request_json); + JSON.parse(row.notification_projection_json); + } catch { + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + } + return { + invocationId: row.invocation_id, + operationId: row.operation_id, + operationKey: row.operation_key, + notificationId: row.notification_id, + requestJson: row.notification_request_json, + projectionVersion: NOTIFICATION_OPERATION_PROJECTION_VERSION, + projectionJson: row.notification_projection_json, + requestDigest: row.notification_request_digest, + state: row.operation_state as RemoteStepNotificationOperation['state'], + ...(row.effect_admitted_at === null + ? {} + : { effectAdmittedAt: row.effect_admitted_at }), + ...(row.receipt_recorded_at === null + ? {} + : { receiptRecordedAt: row.receipt_recorded_at }), + ...(row.resume_admitted_at === null + ? {} + : { resumeAdmittedAt: row.resume_admitted_at }), + }; +} + +function selectedSuspension( + summary: RunSummary, + allowedToolIds: ReadonlySet, +): { + stepPath: string[]; + suspendedAt: number; + resumeCount?: number; + requestJson: string; +} { + if (summary.status !== 'suspended') + throw new RemoteStepRefusal(409, 'run is not suspended'); + const payloads = record(summary.suspendPayload); + const matches = (summary.suspended ?? []).flatMap((path) => { + const key = path.join('.'); + const suspended = record(payloads?.[key]); + const tool = record(suspended?.toolCallSuspended); + return suspended?.type === 'suspension' && + tool?.remoteStep === REMOTE_STEP_MARKER + ? [{ stepPath: path, key, tool, suspended }] + : []; + }); + if (matches.length !== 1) + throw new RemoteStepRefusal(409, 'remote step suspension is ambiguous'); + const selected = matches[0]; + if (!selected) + throw new RemoteStepRefusal(409, 'remote step suspension is ambiguous'); + const { stepPath, key, tool, suspended } = selected; + const suspendedAt = summary.suspendedAt?.[key]; + const resumeCount = summary.resumeCount?.[key]; + if ( + !positiveTime(suspendedAt) || + typeof suspended.toolName !== 'string' || + !allowedToolIds.has(suspended.toolName) || + typeof suspended.toolCallId !== 'string' || + suspended.toolCallId === '' || + (resumeCount !== undefined && + (!Number.isSafeInteger(resumeCount) || resumeCount < 1)) || + !Object.hasOwn(tool, 'request') + ) + throw new RemoteStepRefusal(503, 'remote step suspension is unreadable'); + const requestJson = JSON.stringify(tool.request); + if ( + typeof requestJson !== 'string' || + new TextEncoder().encode(requestJson).length > MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal(400, 'remote step request exceeds its limit'); + return { + stepPath, + suspendedAt, + ...(resumeCount === undefined ? {} : { resumeCount }), + requestJson, + }; +} + +function selectedWorkflowSuspension(summary: RunSummary): { + stepPath: string[]; + suspendedAt: number; + resumeCount?: number; + requestJson: string; +} { + if (summary.status !== 'suspended') + throw new RemoteStepRefusal(409, 'run is not suspended'); + const payloads = record(summary.suspendPayload); + const matches = (summary.suspended ?? []).flatMap((path) => { + const key = path.join('.'); + const suspended = record(payloads?.[key]); + const remoteStep = record(suspended?.remoteStep); + return suspended?.type === 'suspension' && + remoteStep?.marker === WORKFLOW_REMOTE_STEP_MARKER + ? [{ path, key, remoteStep }] + : []; + }); + if (matches.length !== 1) + throw new RemoteStepRefusal(409, 'remote step suspension is ambiguous'); + const selected = matches[0]; + if (!selected || !Object.hasOwn(selected.remoteStep, 'request')) + throw new RemoteStepRefusal(503, 'remote step suspension is unreadable'); + const suspendedAt = summary.suspendedAt?.[selected.key]; + const resumeCount = summary.resumeCount?.[selected.key]; + const requestJson = JSON.stringify(selected.remoteStep.request); + if ( + !positiveTime(suspendedAt) || + (resumeCount !== undefined && + (!Number.isSafeInteger(resumeCount) || resumeCount < 1)) || + typeof requestJson !== 'string' || + new TextEncoder().encode(requestJson).length > MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal(503, 'remote step suspension is unreadable'); + return { + stepPath: selected.path, + suspendedAt, + ...(resumeCount === undefined ? {} : { resumeCount }), + requestJson, + }; +} + +export class RemoteStepJournal { + readonly #db: ExecutionFenceDatabase; + readonly #fence: ExecutionFenceStore; + readonly #runtime: Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + readonly #now: () => number; + readonly #deploymentId: string; + readonly #tablePrefix: string; + readonly #allowedToolIds: ReadonlySet; + readonly #verifyCompletion: (input: { + invocation: RemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; + readonly #verifyWorkflowCompletion?: (input: { + invocation: WorkflowRemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; + + get deploymentId(): string { + return this.#deploymentId; + } + + get tablePrefix(): string { + return this.#tablePrefix; + } + + notificationStorage( + operationReceiptLimit = DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, + ): D1NotificationsStorage { + const db = this.#db as ExecutionFenceDatabase & SignalDatabase; + if (typeof db.batch !== 'function') + throw new Error('notification operation requires database.batch()'); + return new D1NotificationsStorage( + db, + this.#tablePrefix, + operationReceiptLimit, + ); + } + + constructor(input: { + db: ExecutionFenceDatabase; + fence: ExecutionFenceStore; + runtime: Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + deploymentId: string; + tablePrefix: string; + allowedToolIds: readonly string[]; + verifyCompletion: (input: { + invocation: RemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; + verifyWorkflowCompletion?: (input: { + invocation: WorkflowRemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; + now?: () => number; + }) { + if (!input.fence.usesDatabase(input.db)) + throw new Error('remote step fence uses another database'); + if (!isPathSafeId(input.deploymentId)) + throw new Error('remote step deployment id is malformed'); + validateTablePrefix(input.tablePrefix); + if ( + !Array.isArray(input.allowedToolIds) || + input.allowedToolIds.length === 0 || + input.allowedToolIds.some((id) => !isPathSafeId(id)) || + new Set(input.allowedToolIds).size !== input.allowedToolIds.length + ) + throw new Error('remote step tool ids are malformed'); + if (typeof input.verifyCompletion !== 'function') + throw new Error('remote step completion verifier is required'); + this.#db = input.db; + this.#fence = input.fence; + this.#runtime = input.runtime; + this.#deploymentId = input.deploymentId; + this.#tablePrefix = input.tablePrefix; + this.#allowedToolIds = new Set(input.allowedToolIds); + this.#verifyCompletion = input.verifyCompletion; + this.#verifyWorkflowCompletion = input.verifyWorkflowCompletion; + this.#now = input.now ?? Date.now; + } + + async initialize(): Promise { + await this.#db.prepare(remoteStepTableSql(REMOTE_STEP_TABLE)).run(); + await this.#db + .prepare(`CREATE TABLE IF NOT EXISTS ${REMOTE_STEP_SCHEMA_TABLE} ( + id INTEGER PRIMARY KEY CHECK (id = 1), + version INTEGER NOT NULL + )`) + .run(); + await this.#db + .prepare( + REMOTE_STEP_NOTIFICATION_CURSOR_SQL.replace( + 'CREATE TABLE ', + 'CREATE TABLE IF NOT EXISTS ', + ), + ) + .run(); + const readSchema = async () => { + const schema = await this.#db + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all(); + const columns = record(schema)?.results; + if (!Array.isArray(columns)) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + const samePrefix = ( + expected: readonly string[], + v4 = false, + v5 = false, + ) => + columns.length === expected.length && + columns.every((value, index) => { + const column = record(value); + const name = column?.name; + return ( + name === expected[index] && + column?.type === + (REMOTE_STEP_INTEGER_COLUMNS.has(name as string) + ? 'INTEGER' + : 'TEXT') && + column?.notnull === + ((index > 0 && + index <= 16 && + !(v4 && index >= 6 && index <= 8)) || + ((v4 || v5) && + ['target_kind', 'suspension_marker', 'write_revision'].includes( + name as string, + )) || + (v5 && name === 'write_generation') + ? 1 + : 0) && + column?.pk === (index === 0 ? 1 : 0) && + column?.hidden === 0 && + column?.dflt_value === + (name === 'permit_generation' || name === 'write_revision' + ? '0' + : null) + ); + }); + const indexes = record( + await this.#db.prepare(`PRAGMA index_list(${REMOTE_STEP_TABLE})`).all(), + )?.results; + if (!Array.isArray(indexes)) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + const suspensionKey = [ + 'deployment_id', + 'table_prefix', + 'workflow_id', + 'agent_id', + 'thread_id', + 'resource_id', + 'run_id', + 'step_path', + 'suspended_at', + 'resume_count', + ]; + let uniqueSuspension = false; + for (const value of indexes) { + const indexRow = record(value); + const name = indexRow?.name; + if ( + indexRow?.unique !== 1 || + indexRow?.origin !== 'u' || + indexRow?.partial !== 0 || + typeof name !== 'string' || + !/^sqlite_autoindex_flowsafe_remote_step_invocations_[1-9][0-9]*$/.test( + name, + ) + ) + continue; + const details = record( + await this.#db.prepare(`PRAGMA index_info('${name}')`).all(), + )?.results; + if ( + Array.isArray(details) && + details.length === suspensionKey.length && + details.every( + (part, index) => + record(part)?.seqno === index && + record(part)?.name === suspensionKey[index], + ) + ) + uniqueSuspension = true; + } + const recoveryIndex = indexes.find( + (value) => record(value)?.name === REMOTE_STEP_RECOVERY_INDEX, + ); + const recoveryDefinition = record( + await this.#db + .prepare('SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?') + .bind('index', REMOTE_STEP_RECOVERY_INDEX) + .all(), + )?.results; + const validRecoveryIndex = + record(recoveryIndex)?.unique === 0 && + record(recoveryIndex)?.origin === 'c' && + record(recoveryIndex)?.partial === 1 && + Array.isArray(recoveryDefinition) && + recoveryDefinition.length === 1 && + record(recoveryDefinition[0])?.sql === REMOTE_STEP_RECOVERY_INDEX_SQL; + const validV4Index = async (name: string, sql: string) => { + const entry = indexes.find((value) => record(value)?.name === name); + const definition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('index', name) + .all(), + )?.results; + return ( + record(entry)?.unique === 1 && + record(entry)?.origin === 'c' && + record(entry)?.partial === 1 && + Array.isArray(definition) && + definition.length === 1 && + record(definition[0])?.sql === sql + ); + }; + const validV4Table = async () => { + const tableDefinition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('table', REMOTE_STEP_TABLE) + .all(), + )?.results; + const tableSql = + Array.isArray(tableDefinition) && tableDefinition.length === 1 + ? record(tableDefinition[0])?.sql + : undefined; + return ( + typeof tableSql === 'string' && + tableSql.includes( + "target_kind TEXT NOT NULL CHECK (target_kind IN ('agent', 'workflow'))", + ) && + tableSql.includes( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + ) && + tableSql.includes("(target_kind = 'workflow' AND agent_id IS NULL") && + tableSql.includes('write_revision INTEGER NOT NULL DEFAULT 0') + ); + }; + const validV4 = async () => { + const trigger = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('trigger', REMOTE_STEP_WRITE_TRIGGER) + .all(), + )?.results; + return ( + (await validV4Table()) && + validRecoveryIndex && + (await validV4Index( + REMOTE_STEP_AGENT_INDEX, + REMOTE_STEP_AGENT_INDEX_SQL, + )) && + (await validV4Index( + REMOTE_STEP_WORKFLOW_INDEX, + REMOTE_STEP_WORKFLOW_INDEX_SQL, + )) && + Array.isArray(trigger) && + trigger.length === 1 && + record(trigger[0])?.sql === REMOTE_STEP_WRITE_TRIGGER_SQL + ); + }; + const validV5 = async () => { + const tableDefinition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('table', REMOTE_STEP_TABLE) + .all(), + )?.results; + const tableSql = Array.isArray(tableDefinition) + ? record(tableDefinition[0])?.sql + : undefined; + const operationIndex = indexes.find( + (value) => record(value)?.name === REMOTE_STEP_OPERATION_INDEX, + ); + const operationDefinition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('index', REMOTE_STEP_OPERATION_INDEX) + .all(), + )?.results; + const trigger = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('trigger', REMOTE_STEP_WRITE_TRIGGER) + .all(), + )?.results; + return ( + typeof tableSql === 'string' && + tableSql.includes( + 'write_generation INTEGER NOT NULL CHECK (write_generation > 0)', + ) && + tableSql.includes('operation_key TEXT UNIQUE') && + tableSql.includes( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + ) && + tableSql.includes("(target_kind = 'workflow' AND agent_id IS NULL") && + tableSql.includes( + "operation_state TEXT CHECK (operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved'))", + ) && + tableSql.includes( + 'CHECK ((operation_state IS NULL AND operation_key IS NULL', + ) && + tableSql.includes( + "(target_kind = 'agent' AND operation_key IS NOT NULL", + ) && + validRecoveryIndex && + (await validV4Index( + REMOTE_STEP_AGENT_INDEX, + REMOTE_STEP_AGENT_INDEX_SQL, + )) && + (await validV4Index( + REMOTE_STEP_WORKFLOW_INDEX, + REMOTE_STEP_WORKFLOW_INDEX_SQL, + )) && + record(operationIndex)?.unique === 0 && + record(operationIndex)?.origin === 'c' && + record(operationIndex)?.partial === 1 && + Array.isArray(operationDefinition) && + operationDefinition.length === 1 && + record(operationDefinition[0])?.sql === + REMOTE_STEP_OPERATION_INDEX_SQL && + Array.isArray(trigger) && + trigger.length === 1 && + record(trigger[0])?.sql === REMOTE_STEP_V5_WRITE_TRIGGER_SQL + ); + }; + const versions = await this.#db + .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) + .all(); + const rows = record(versions)?.results; + if (!Array.isArray(rows) || rows.length > 1) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + if (rows.length === 1) { + if (record(rows[0])?.id === 1 && record(rows[0])?.version === 5) { + if ( + !samePrefix(REMOTE_STEP_V5_COLUMNS, true, true) || + !(await validV5()) + ) + throw new RemoteStepRefusal( + 503, + 'remote step schema is unreadable', + ); + return 'current-v5'; + } + if (record(rows[0])?.id === 1 && record(rows[0])?.version === 4) { + if (!samePrefix(REMOTE_STEP_COLUMNS, true) || !(await validV4())) + throw new RemoteStepRefusal( + 503, + 'remote step schema is unreadable', + ); + return 'current-v4'; + } + if ( + record(rows[0])?.id !== 1 || + (record(rows[0])?.version !== 3 && record(rows[0])?.version !== 2) || + !uniqueSuspension || + !samePrefix( + record(rows[0])?.version === 3 + ? REMOTE_STEP_COLUMNS.slice(0, -4) + : REMOTE_STEP_COLUMNS.slice(0, -8), + ) || + (record(rows[0])?.version === 3 && !validRecoveryIndex) + ) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + return record(rows[0])?.version === 3 ? 'prior-v3' : 'prior-v2'; + } + if (uniqueSuspension && samePrefix(REMOTE_STEP_COLUMNS.slice(0, -12))) + return 'prior-v1'; + if (uniqueSuspension && samePrefix(REMOTE_STEP_COLUMNS.slice(0, -8))) + return 'unversioned-v2'; + if (uniqueSuspension && samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) + return 'unversioned-v3'; + if (samePrefix(REMOTE_STEP_COLUMNS, true) && (await validV4Table())) + return 'unversioned-v4'; + if (samePrefix(REMOTE_STEP_V5_COLUMNS, true, true)) { + const definition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('table', REMOTE_STEP_TABLE) + .all(), + )?.results; + const sql = Array.isArray(definition) + ? record(definition[0])?.sql + : undefined; + if ( + typeof sql === 'string' && + sql.includes( + 'write_generation INTEGER NOT NULL CHECK (write_generation > 0)', + ) && + sql.includes('operation_key TEXT UNIQUE') && + sql.includes( + "operation_state TEXT CHECK (operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved'))", + ) && + sql.includes( + 'CHECK ((operation_state IS NULL AND operation_key IS NULL', + ) && + sql.includes( + "(target_kind = 'agent' AND operation_key IS NOT NULL", + ) && + sql.includes( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + ) && + sql.includes("(target_kind = 'workflow' AND agent_id IS NULL") + ) + return 'unversioned-v5'; + } + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + }; + const stage = await readSchema(); + if (stage === 'current-v5') return; + if ( + stage === 'prior-v1' || + stage === 'prior-v2' || + stage === 'unversioned-v2' + ) { + const transactional = this.#db as ExecutionFenceDatabase & { + batch?: ( + statements: ReturnType[], + ) => Promise; + }; + if (typeof transactional.batch !== 'function') + throw new RemoteStepRefusal( + 503, + 'remote step schema upgrade is unavailable', + ); + try { + await transactional.batch([ + ...(stage === 'prior-v1' ? REMOTE_STEP_RESULT_COLUMNS : []).map( + (column) => + this.#db.prepare( + `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${column === 'executor_quiescent_at' ? 'INTEGER' : 'TEXT'}`, + ), + ), + ...REMOTE_STEP_SETTLEMENT_COLUMNS.map((column) => + this.#db.prepare( + `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${REMOTE_STEP_INTEGER_COLUMNS.has(column) ? 'INTEGER' : 'TEXT'}`, + ), + ), + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare( + stage === 'prior-v2' + ? `UPDATE ${REMOTE_STEP_SCHEMA_TABLE} SET version = 3 WHERE id = 1 AND version = 2` + : `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 3)`, + ), + ]); + } catch (error) { + const current = await readSchema(); + if ( + current !== 'prior-v3' && + current !== 'current-v4' && + current !== 'current-v5' + ) + throw error; + } + return this.initialize(); + } + const transactional = this.#db as ExecutionFenceDatabase & { + batch?: ( + statements: ReturnType[], + ) => Promise; + }; + if (typeof transactional.batch !== 'function') + throw new RemoteStepRefusal( + 503, + 'remote step schema upgrade is unavailable', + ); + if (stage === 'unversioned-v3') { + try { + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 3)`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'prior-v3') throw error; + } + return this.initialize(); + } + if (stage === 'prior-v3') { + const replacement = `${REMOTE_STEP_TABLE}_v4`; + try { + await transactional.batch([ + this.#db.prepare(remoteStepTableSql(replacement, 4)), + this.#db.prepare(`INSERT INTO ${replacement} (${REMOTE_STEP_COLUMNS.join(', ')}) + SELECT ${REMOTE_STEP_COLUMNS.slice(0, -4).join(', ')}, 'agent', '${REMOTE_STEP_MARKER}', NULL, 0 FROM ${REMOTE_STEP_TABLE}`), + this.#db.prepare(`DROP TABLE ${REMOTE_STEP_TABLE}`), + this.#db.prepare( + `ALTER TABLE ${replacement} RENAME TO ${REMOTE_STEP_TABLE}`, + ), + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WRITE_TRIGGER_SQL), + this.#db.prepare( + `UPDATE ${REMOTE_STEP_SCHEMA_TABLE} SET version = 4 WHERE id = 1 AND version = 3`, + ), + ]); + } catch (error) { + if (!['current-v4', 'current-v5'].includes(await readSchema())) + throw error; + } + return this.initialize(); + } + if (stage === 'unversioned-v4') { + try { + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WRITE_TRIGGER_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 4)`, + ), + ]); + } catch (error) { + if (!['current-v4', 'current-v5'].includes(await readSchema())) + throw error; + } + return this.initialize(); + } + if (stage === 'current-v4') { + const replacement = `${REMOTE_STEP_TABLE}_v5`; + try { + await transactional.batch([ + this.#db.prepare(remoteStepTableSql(replacement)), + this.#db.prepare(`INSERT INTO ${replacement} (${REMOTE_STEP_V5_COLUMNS.join(', ')}) + SELECT ${REMOTE_STEP_COLUMNS.join(', ')}, 1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL + FROM ${REMOTE_STEP_TABLE}`), + this.#db.prepare(`DROP TABLE ${REMOTE_STEP_TABLE}`), + this.#db.prepare( + `ALTER TABLE ${replacement} RENAME TO ${REMOTE_STEP_TABLE}`, + ), + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_OPERATION_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_V5_WRITE_TRIGGER_SQL), + this.#db.prepare( + `UPDATE ${REMOTE_STEP_SCHEMA_TABLE} SET version = 5 WHERE id = 1 AND version = 4`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'current-v5') throw error; + } + return; + } + if (stage === 'unversioned-v5') { + try { + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_OPERATION_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_V5_WRITE_TRIGGER_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 5)`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'current-v5') throw error; + } + return; + } + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + } + + async #snapshot( + address: RemoteStepAddress | WorkflowRemoteStepAddress, + ): Promise { + const result = await this.#db + .prepare( + `SELECT snapshot FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? LIMIT 2`, + ) + .bind(address.workflowId, address.runId) + .all(); + const rows = record(result)?.results; + if ( + !Array.isArray(rows) || + rows.length !== 1 || + typeof record(rows[0])?.snapshot !== 'string' + ) + throw new RemoteStepRefusal(503, 'remote step snapshot is unreadable'); + return record(rows[0])?.snapshot as string; + } + + async #current(address: RemoteStepAddress | WorkflowRemoteStepAddress) { + const summary = await this.#runtime.authoritativeStatus( + address.workflowId, + address.runId, + ); + if (!summary) throw new RemoteStepRefusal(404, 'run not found'); + return 'targetKind' in address + ? selectedWorkflowSuspension(summary) + : selectedSuspension(summary, this.#allowedToolIds); + } + + async #stableSnapshot( + address: RemoteStepAddress | WorkflowRemoteStepAddress, + ) { + const before = await this.#current(address); + const snapshot = await this.#snapshot(address); + const state = await this.#execution(address); + const lifecycle = lifecycleFromRequestContext( + state.snapshot.requestContext, + ); + if (lifecycle?.transitionIntent || lifecycle?.terminal) + throw new RemoteStepRefusal(409, 'remote step run is ending'); + const after = await this.#current(address); + if ( + before.suspendedAt !== after.suspendedAt || + before.resumeCount !== after.resumeCount || + JSON.stringify(before.stepPath) !== JSON.stringify(after.stepPath) || + before.requestJson !== after.requestJson + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + return { suspension: after, snapshot }; + } + + async register( + address: RemoteStepAddress, + hostDeadlineAt: number, + originalPrincipal: ExecutionPrincipal, + ): Promise { + if ( + address.deploymentId !== this.#deploymentId || + address.tablePrefix !== this.#tablePrefix || + !isPathSafeId(address.workflowId) || + !isPathSafeId(address.agentId) || + !isPathSafeId(address.threadId) || + !isPathSafeId(address.resourceId) || + !isPathSafeId(address.runId) || + !positiveTime(hostDeadlineAt) + ) + throw new RemoteStepRefusal(400, 'remote step address is malformed'); + validateTablePrefix(address.tablePrefix); + let principal: ExecutionPrincipal; + try { + principal = assertExecutionPrincipal( + originalPrincipal, + 'remote step principal', + ); + } catch { + throw new RemoteStepRefusal(400, 'remote step principal is malformed'); + } + const execution = await this.#execution(address, principal); + const { suspension, snapshot } = await this.#stableSnapshot(address); + const summary = await this.#runtime.authoritativeStatus( + address.workflowId, + address.runId, + ); + const deadlineAt = Math.min( + hostDeadlineAt, + summary?.deadlineAt ?? hostDeadlineAt, + ); + if (deadlineAt <= this.#now()) + throw new RemoteStepRefusal(409, 'remote step deadline elapsed'); + const invocationId = crypto.randomUUID(); + const stepPath = JSON.stringify(suspension.stepPath); + await this.#db + .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( + invocation_id, principal_json, start_token, deployment_id, table_prefix, workflow_id, agent_id, + thread_id, resource_id, run_id, step_path, suspended_at, resume_count, + request_json, deadline_at, status, target_kind, suspension_marker, write_generation) + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'agent', '${REMOTE_STEP_MARKER}', 1 + WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + ON CONFLICT DO NOTHING`) + .bind( + invocationId, + JSON.stringify(principal), + execution.execution.startToken, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.agentId, + address.threadId, + address.resourceId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + deadlineAt, + address.workflowId, + address.runId, + snapshot, + ) + .run(); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE deployment_id = ? AND table_prefix = ? AND workflow_id = ? + AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? + AND step_path = ? AND suspended_at = ? AND resume_count = ? + AND request_json = ? AND principal_json = ? AND start_token = ? + AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + RETURNING *`) + .bind( + deadlineAt, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.agentId, + address.threadId, + address.resourceId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + JSON.stringify(principal), + execution.execution.startToken, + address.workflowId, + address.runId, + snapshot, + ) + .all(); + const row = rowResult(result); + if (!row || row.request_json !== suspension.requestJson) + throw new RemoteStepRefusal( + 409, + 'remote step registration did not settle', + ); + return decode(row); + } + + async registerWorkflow( + address: WorkflowRemoteStepAddress, + hostDeadlineAt: number, + originalPrincipal: ExecutionPrincipal, + ): Promise { + if ( + address.targetKind !== 'workflow' || + address.deploymentId !== this.#deploymentId || + address.tablePrefix !== this.#tablePrefix || + !isPathSafeId(address.workflowId) || + !isPathSafeId(address.runId) || + !positiveTime(hostDeadlineAt) + ) + throw new RemoteStepRefusal(400, 'remote step address is malformed'); + validateTablePrefix(address.tablePrefix); + let principal: ExecutionPrincipal; + try { + principal = assertExecutionPrincipal( + originalPrincipal, + 'remote step principal', + ); + } catch { + throw new RemoteStepRefusal(400, 'remote step principal is malformed'); + } + const execution = await this.#execution(address, principal); + const { suspension, snapshot } = await this.#stableSnapshot(address); + const summary = await this.#runtime.authoritativeStatus( + address.workflowId, + address.runId, + ); + const deadlineAt = Math.min( + hostDeadlineAt, + summary?.deadlineAt ?? hostDeadlineAt, + ); + if (deadlineAt <= this.#now()) + throw new RemoteStepRefusal(409, 'remote step deadline elapsed'); + const requestDigest = await sha256Hex(suspension.requestJson); + const stepPath = JSON.stringify(suspension.stepPath); + const invocationId = crypto.randomUUID(); + await this.#db + .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( + invocation_id, principal_json, start_token, deployment_id, table_prefix, + workflow_id, run_id, step_path, suspended_at, resume_count, request_json, + deadline_at, status, target_kind, suspension_marker, request_digest, write_generation) + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'workflow', + '${WORKFLOW_REMOTE_STEP_MARKER}', ?, 1 + WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + ON CONFLICT DO NOTHING`) + .bind( + invocationId, + JSON.stringify(principal), + execution.execution.startToken, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + deadlineAt, + requestDigest, + address.workflowId, + address.runId, + snapshot, + ) + .run(); + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE target_kind = 'workflow' AND deployment_id = ? + AND table_prefix = ? AND workflow_id = ? AND run_id = ? + AND step_path = ? AND suspended_at = ? AND resume_count = ? + AND request_json = ? AND request_digest = ? + AND principal_json = ? AND start_token = ? + AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + RETURNING *`) + .bind( + deadlineAt, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + requestDigest, + JSON.stringify(principal), + execution.execution.startToken, + address.workflowId, + address.runId, + snapshot, + ) + .all(), + ); + if (!row) + throw new RemoteStepRefusal( + 409, + 'remote step registration did not settle', + ); + return decodeWorkflow(row); + } + + async get(invocationId: string): Promise { + const row = await this.#getRow(invocationId); + return row?.target_kind === 'agent' ? decode(row) : undefined; + } + + async getWorkflow( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + return row?.target_kind === 'workflow' ? decodeWorkflow(row) : undefined; + } + + async #getRow(invocationId: string): Promise { + if (!isPathSafeId(invocationId)) + throw new RemoteStepRefusal(400, 'invocation id is malformed'); + const row = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + return row; + } + + async #getAny( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + return row ? decodeAny(row) : undefined; + } + + async readDispatchRequestJson(invocationId: string): Promise { + return this.#readDispatchRequestJson(invocationId, 'agent'); + } + + async readWorkflowDispatchRequestJson(invocationId: string): Promise { + return this.#readDispatchRequestJson(invocationId, 'workflow'); + } + + async #readDispatchRequestJson( + invocationId: string, + targetKind: 'agent' | 'workflow', + ): Promise { + if (!isPathSafeId(invocationId)) + throw new RemoteStepRefusal(400, 'invocation id is malformed'); + const row = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!row) throw new RemoteStepRefusal(404, 'invocation not found'); + if (row.target_kind !== targetKind) + throw new RemoteStepRefusal(404, 'invocation not found'); + decodeAny(row); + return row.request_json; + } + + async #execution( + address: RemoteStepAddress | WorkflowRemoteStepAddress, + principal?: ExecutionPrincipal, + ) { + const state = await this.#runtime.authoritativeStartState( + address.workflowId, + address.runId, + ); + const target = state?.provenance.startIdentity?.target; + if (state?.storage !== 'd1' || !target) + throw new RemoteStepRefusal( + 503, + 'remote step run identity is unreadable', + ); + if ( + (principal !== undefined && + (state.provenance.startIdentity?.owner.kind !== principal.kind || + state.provenance.startIdentity.owner.id !== principal.id)) || + state.execution.tablePrefix !== address.tablePrefix || + state.execution.workflowId !== address.workflowId || + state.execution.runId !== address.runId + ) + throw new RemoteStepRefusal(409, 'remote step run identity changed'); + if ('targetKind' in address) { + if ( + target.kind !== 'workflow' || + target.id !== address.workflowId || + state.provenance.agentStart !== undefined + ) + throw new RemoteStepRefusal(409, 'remote step run identity changed'); + } else if (target.kind !== 'agent' || !isPathSafeId(target.resourceId)) { + throw new RemoteStepRefusal( + 503, + 'remote step run identity is unreadable', + ); + } else if ( + target.id !== address.agentId || + target.threadId !== address.threadId || + address.resourceId !== target.resourceId + ) + throw new RemoteStepRefusal(409, 'remote step run identity changed'); + return state; + } + + async #admission( + invocation: RemoteStepInvocation | WorkflowRemoteStepInvocation, + priorBindings: number, + ) { + const observed = await this.#fence.readForAdmission(); + const proof = (await this.#execution(invocation, invocation.principal)) + .execution; + if (proof.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); + if (!admitsExistingRun(observed.reading, proof)) + throw new RemoteStepRefusal(503, 'remote step execution is fenced'); + const schema = await captureExecutionFenceAdmissionSchema( + await this.#db + .prepare(`PRAGMA table_xinfo(${EXECUTION_FENCE_TABLE})`) + .all(), + ); + const values: unknown[] = []; + const bind = (value: unknown) => { + values.push(value); + return `?${priorBindings + values.length}`; + }; + const predicate = executionFenceAdmissionSql({ + callerEpoch: bind(observed.reading.mutationEpoch ?? null), + semantic: executionFenceAdmissionValues(observed).map(bind), + schema: bind(schema), + statePredicate: `f.state IN ('open', 'draining') OR + (f.state = 'proof-only' AND f.proof_table_prefix IS ${bind(proof.tablePrefix)} + AND f.proof_workflow_id IS ${bind(proof.workflowId)} + AND f.proof_run_id IS ${bind(proof.runId)} + AND f.proof_start_token IS ${bind(proof.startToken)})`, + }); + return { predicate, values, bind, observed: observed.reading }; + } + + async issuePermit( + invocationId: string, + lifetimeMs: number, + ): Promise { + const invocation = await this.#getAny(invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if ('targetKind' in invocation && !this.#verifyWorkflowCompletion) + throw new RemoteStepRefusal( + 503, + 'workflow remote step completion is unavailable', + ); + if (!positiveTime(lifetimeMs)) + throw new RemoteStepRefusal(400, 'permit lifetime is malformed'); + const now = this.#now(); + const expiresAt = Math.min(invocation.deadlineAt, now + lifetimeMs); + if (!positiveTime(expiresAt) || expiresAt <= now) + throw new RemoteStepRefusal(409, 'remote step deadline elapsed'); + const address = invocation; + const { suspension, snapshot } = await this.#stableSnapshot(address); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) || + ('targetKind' in invocation && + (await sha256Hex(suspension.requestJson)) !== invocation.requestDigest) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const gate = await this.#admission(invocation, 13); + if ( + invocation.status === 'dispatched' && + invocation.permitConsumedAt === undefined && + invocation.permitExpiresAt !== undefined && + invocation.permitExpiresAt > now && + invocation.permitEpoch === gate.observed.mutationEpoch && + invocation.permitRevision === gate.observed.transitionRevision + ) + return { + invocationId, + generation: invocation.permitGeneration, + mutationEpoch: invocation.permitEpoch, + transitionRevision: invocation.permitRevision, + expiresAt: invocation.permitExpiresAt, + }; + if ( + invocation.status === 'failed' || + invocation.permitConsumedAt !== undefined + ) + throw new RemoteStepRefusal(409, 'remote step dispatch is active'); + const epoch = gate.observed.mutationEpoch; + const revision = gate.observed.transitionRevision; + if (epoch === undefined || revision === undefined) + throw new RemoteStepRefusal(503, 'remote step fence is unversioned'); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET + status = 'dispatched', permit_generation = permit_generation + 1, + permit_epoch = ?, permit_revision = ?, + permit_expires_at = MIN(?, deadline_at), permit_consumed_at = NULL, + permit_digest = NULL, write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND permit_generation = ? AND deadline_at > ? + AND status IN ('pending', 'dispatched') + AND permit_consumed_at IS NULL AND request_json = ? + AND target_kind = ? AND suspension_marker = ? + AND request_digest IS ? + AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + AND ${gate.predicate} RETURNING *`) + .bind( + epoch, + revision, + expiresAt, + invocationId, + invocation.permitGeneration, + now, + suspension.requestJson, + 'targetKind' in invocation ? 'workflow' : 'agent', + 'targetKind' in invocation + ? WORKFLOW_REMOTE_STEP_MARKER + : REMOTE_STEP_MARKER, + 'targetKind' in invocation ? invocation.requestDigest : null, + invocation.workflowId, + invocation.runId, + snapshot, + ...gate.values, + ) + .all(); + const row = rowResult(result); + if (!row) throw new RemoteStepRefusal(409, 'remote step dispatch changed'); + return { + invocationId, + generation: row.permit_generation, + mutationEpoch: epoch, + transitionRevision: revision, + expiresAt: row.permit_expires_at as number, + }; + } + + async consumePermit( + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + > { + const invocation = await this.#getAny(permit.invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if ('targetKind' in invocation && !this.#verifyWorkflowCompletion) + throw new RemoteStepRefusal( + 503, + 'workflow remote step completion is unavailable', + ); + if ( + invocation.permitGeneration !== permit.generation || + invocation.permitEpoch !== permit.mutationEpoch || + invocation.permitRevision !== permit.transitionRevision || + invocation.permitExpiresAt !== permit.expiresAt + ) + throw new RemoteStepRefusal(409, 'remote step permit changed'); + if (invocation.permitConsumedAt !== undefined) + return { kind: 'already-consumed' }; + const now = this.#now(); + const { suspension, snapshot } = await this.#stableSnapshot(invocation); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) || + ('targetKind' in invocation && + (await sha256Hex(suspension.requestJson)) !== invocation.requestDigest) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const rawPermit = rawDispatchPermit(); + const permitDigest = await sha256Hex(rawPermit); + const gate = await this.#admission(invocation, 16); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET permit_consumed_at = ?, permit_digest = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND permit_generation = ? + AND permit_epoch = ? AND permit_revision = ? + AND permit_expires_at = ? AND permit_expires_at > ? + AND deadline_at > ? AND permit_consumed_at IS NULL + AND status = 'dispatched' AND request_json = ? + AND target_kind = ? AND suspension_marker = ? + AND request_digest IS ? + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + AND ${gate.predicate} RETURNING invocation_id`) + .bind( + now, + permitDigest, + permit.invocationId, + permit.generation, + permit.mutationEpoch, + permit.transitionRevision, + permit.expiresAt, + now, + now, + suspension.requestJson, + 'targetKind' in invocation ? 'workflow' : 'agent', + 'targetKind' in invocation + ? WORKFLOW_REMOTE_STEP_MARKER + : REMOTE_STEP_MARKER, + 'targetKind' in invocation ? invocation.requestDigest : null, + invocation.workflowId, + invocation.runId, + snapshot, + ...gate.values, + ) + .all(); + if (rowResult(result)) return { kind: 'accepted', rawPermit }; + const after = await this.#getAny(permit.invocationId); + if ( + after?.permitGeneration === permit.generation && + after.permitConsumedAt !== undefined + ) + return { kind: 'already-consumed' }; + throw new RemoteStepRefusal(409, 'remote step permit is unavailable'); + } + + async recordResult( + invocationId: string, + rawPermit: string, + resultJson: string, + evidenceJson: string, + ): Promise< + | { kind: 'recorded'; resultJson: string; resultDigest: string } + | { kind: 'replay'; resultJson: string; resultDigest: string } + > { + if (!isPathSafeId(invocationId) || !RAW_PERMIT.test(rawPermit)) + throw new RemoteStepRefusal( + 400, + 'remote step result identity is malformed', + ); + boundedJsonBytes(resultJson); + boundedJsonBytes(evidenceJson); + const stored = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + const invocation = decodeAny(stored); + const permitDigest = stored.permit_digest; + if ( + typeof permitDigest !== 'string' || + !SHA256_HEX.test(permitDigest) || + invocation.permitConsumedAt === undefined + ) + throw new RemoteStepRefusal(409, 'remote step permit was not consumed'); + const presentedDigest = await sha256Hex(rawPermit); + if (!(await credentialsMatch(presentedDigest, permitDigest))) + throw new RemoteStepRefusal(403, 'remote step permit is invalid'); + const resultDigest = await sha256Hex(resultJson); + const quiescenceDigest = await sha256Hex(evidenceJson); + const replay = (row: JournalRow) => { + if ( + row.result_json !== resultJson || + row.result_digest !== resultDigest || + row.quiescence_digest !== quiescenceDigest || + !positiveTime(row.executor_quiescent_at) + ) + throw new RemoteStepRefusal(409, 'remote step result conflicts'); + return { kind: 'replay' as const, resultJson, resultDigest }; + }; + if (stored.result_json !== null) return replay(stored); + if ('targetKind' in invocation) { + if (!this.#verifyWorkflowCompletion) + throw new RemoteStepRefusal( + 503, + 'workflow remote step completion is unavailable', + ); + await this.#verifyWorkflowCompletion({ + invocation, + rawPermit, + resultJson, + evidenceJson, + }); + } else { + await this.#verifyCompletion({ + invocation, + rawPermit, + resultJson, + evidenceJson, + }); + } + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET result_json = ?, result_digest = ?, quiescence_digest = ?, + executor_quiescent_at = ?, status = 'result-retained', + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND permit_digest = ? + AND permit_consumed_at IS NOT NULL AND result_json IS NULL + AND status = 'dispatched' + RETURNING *`) + .bind( + resultJson, + resultDigest, + quiescenceDigest, + this.#now(), + invocationId, + permitDigest, + ) + .all(); + if (rowResult(result)) + return { kind: 'recorded', resultJson, resultDigest }; + const after = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!after) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + return replay(after); + } + + async readRetainedResult( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + if (!row) return undefined; + decodeAny(row); + return this.#retainedResultFromRow(row); + } + + async #retainedResultFromRow( + row: JournalRow, + ): Promise { + const resultJson = row.result_json; + const resultDigest = row.result_digest; + const quiescenceDigest = row.quiescence_digest; + const executorQuiescentAt = row.executor_quiescent_at; + if (resultJson === null) return undefined; + if (row.status === 'settled' || row.status === 'failed') return undefined; + if ( + !['result-retained', 'resuming'].includes(row.status) || + row.permit_consumed_at === null || + typeof resultDigest !== 'string' || + !SHA256_HEX.test(resultDigest) || + typeof quiescenceDigest !== 'string' || + !SHA256_HEX.test(quiescenceDigest) || + !positiveTime(executorQuiescentAt) + ) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + try { + boundedJsonBytes(resultJson); + } catch { + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + } + if ((await sha256Hex(resultJson)) !== resultDigest) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + return { + invocationId: row.invocation_id, + resultJson, + resultDigest, + quiescenceDigest, + executorQuiescentAt, + }; + } + + async readNotificationOperation( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + if (!row) return undefined; + decodeAny(row); + return this.#notificationOperationFromRow(row); + } + + async #notificationOperationFromRow( + row: JournalRow, + ): Promise { + const operation = decodeNotificationOperation(row); + if ( + operation && + (await sha256Hex(operation.projectionJson)) !== operation.requestDigest + ) + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + return operation; + } + + async prepareNotificationOperation( + invocationId: string, + operationId: string, + input: Omit, + ): Promise { + if (!isPathSafeId(operationId)) + throw new RemoteStepRefusal(400, 'remote step operation id is malformed'); + const stored = await this.#getRow(invocationId); + const invocation = + stored?.target_kind === 'agent' ? decode(stored) : undefined; + if (stored && !invocation) decodeAny(stored); + const retained = stored + ? await this.#retainedResultFromRow(stored) + : undefined; + if ( + !stored || + !invocation || + invocation.status !== 'result-retained' || + !retained + ) + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const operationKey = await sha256Hex( + JSON.stringify([ + invocation.deploymentId, + invocation.invocationId, + invocation.startToken, + invocation.stepPath, + invocation.suspendedAt, + invocation.resumeCount ?? -1, + operationId, + ]), + ); + const notificationId = operationKey; + const request = { ...input, id: notificationId }; + const projection = await projectNotificationOperation(request); + const requestJson = JSON.stringify(request); + if ( + new TextEncoder().encode(requestJson).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + new TextEncoder().encode(projection.requestJson).length > + MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal( + 400, + 'remote step operation exceeds its limit', + ); + const existing = await this.#notificationOperationFromRow(stored); + if (existing) { + if ( + existing.operationKey !== operationKey || + existing.requestJson !== requestJson || + existing.projectionJson !== projection.requestJson || + existing.requestDigest !== projection.requestDigest + ) + throw new RemoteStepRefusal(409, 'remote step operation changed'); + return existing; + } + const result = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET operation_key = ?, operation_id = ?, notification_id = ?, + notification_request_json = ?, notification_projection_version = ?, + notification_projection_json = ?, notification_request_digest = ?, + operation_state = 'prepared', + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND target_kind = 'agent' + AND status = 'result-retained' AND operation_state IS NULL + AND principal_json = ? AND start_token = ? AND result_digest = ? + AND write_revision = ? AND write_generation = ? + RETURNING *`) + .bind( + operationKey, + operationId, + notificationId, + requestJson, + projection.version, + projection.requestJson, + projection.requestDigest, + invocationId, + JSON.stringify(invocation.principal), + invocation.startToken, + retained.resultDigest, + stored.write_revision, + stored.write_generation, + ) + .all(), + ); + if (result) + return decodeNotificationOperation( + result, + ) as RemoteStepNotificationOperation; + const after = await this.readNotificationOperation(invocationId); + if ( + after?.operationKey === operationKey && + after.requestDigest === projection.requestDigest + ) + return after; + throw new RemoteStepRefusal(409, 'remote step operation changed'); + } + + async admitNotificationEffect( + invocationId: string, + ): Promise { + const stored = await this.#getRow(invocationId); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + const invocation = decode(stored); + const operation = await this.#notificationOperationFromRow(stored); + if ( + !operation || + !['prepared', 'effect-admitted'].includes(operation.state) + ) + throw new RemoteStepRefusal(409, 'remote step operation is unavailable'); + if (operation.state === 'effect-admitted') return operation; + const retained = await this.#retainedResultFromRow(stored); + if (!retained || invocation.status !== 'result-retained') + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const { suspension, snapshot } = await this.#stableSnapshot(invocation); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const now = this.#now(); + const base = [ + now, + invocationId, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + invocation.startToken, + JSON.stringify(invocation.principal), + retained.resultDigest, + operation.operationKey, + operation.requestDigest, + stored.write_revision, + stored.write_generation, + now, + invocation.workflowId, + invocation.runId, + snapshot, + ]; + const gate = await this.#admission(invocation, base.length); + try { + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET operation_state = 'effect-admitted', effect_admitted_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? + AND workflow_id = ? AND agent_id = ? AND thread_id = ? + AND resource_id = ? AND run_id = ? AND start_token = ? + AND principal_json = ? AND result_digest = ? + AND operation_key = ? AND notification_request_digest = ? + AND write_revision = ? AND write_generation = ? AND deadline_at > ? + AND target_kind = 'agent' AND status = 'result-retained' + AND operation_state = 'prepared' + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + AND ${gate.predicate} RETURNING *`) + .bind(...base, ...gate.values) + .all(), + ); + if (row) + return decodeNotificationOperation( + row, + ) as RemoteStepNotificationOperation; + } catch (error) { + const after = await this.readNotificationOperation(invocationId); + if ( + after?.state === 'effect-admitted' && + after.operationKey === operation.operationKey && + after.requestDigest === operation.requestDigest + ) + return after; + throw error; + } + const after = await this.readNotificationOperation(invocationId); + if ( + after?.state === 'effect-admitted' && + after.operationKey === operation.operationKey && + after.requestDigest === operation.requestDigest + ) + return after; + throw new RemoteStepRefusal(409, 'remote step effect admission changed'); + } + + async recordNotificationReceipt( + invocationId: string, + receipt: { + operationKey: string; + notificationId: string; + threadId: string; + requestDigest: string; + }, + ): Promise { + const stored = await this.#getRow(invocationId); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + decode(stored); + const operation = await this.#notificationOperationFromRow(stored); + if ( + operation?.state !== 'effect-admitted' || + receipt.operationKey !== operation.operationKey || + receipt.notificationId !== operation.notificationId || + receipt.threadId !== stored.thread_id || + receipt.requestDigest !== operation.requestDigest + ) + throw new RemoteStepRefusal( + 409, + 'remote step notification receipt changed', + ); + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET operation_state = 'receipt-recorded', receipt_recorded_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND status = 'result-retained' + AND operation_state = 'effect-admitted' AND operation_key = ? + AND notification_id = ? AND notification_request_digest = ? + AND result_digest = ? AND write_revision = ? AND write_generation = ? + AND EXISTS (SELECT 1 FROM "${stored.table_prefix}${NOTIFICATION_OPERATION_RECEIPTS_TABLE}" + WHERE operation_key = ? AND request_digest = ? + AND thread_id = ? AND notification_id = ?) + RETURNING *`) + .bind( + this.#now(), + invocationId, + operation.operationKey, + operation.notificationId, + operation.requestDigest, + stored.result_digest, + stored.write_revision, + stored.write_generation, + operation.operationKey, + operation.requestDigest, + stored.thread_id, + operation.notificationId, + ) + .all(), + ); + if (row) + return decodeNotificationOperation( + row, + ) as RemoteStepNotificationOperation; + throw new RemoteStepRefusal( + 409, + 'remote step notification receipt changed', + ); + } + + async admitNotificationResume(invocationId: string): Promise { + const stored = await this.#getRow(invocationId); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + const invocation = decode(stored); + const operation = await this.#notificationOperationFromRow(stored); + if ( + operation?.state !== 'receipt-recorded' || + invocation.status !== 'result-retained' + ) + throw new RemoteStepRefusal( + 409, + 'remote step continuation is unavailable', + ); + const retained = await this.#retainedResultFromRow(stored); + if (!retained) + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const { suspension, snapshot } = await this.#stableSnapshot(invocation); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const now = this.#now(); + const owner = crypto.randomUUID(); + const base = [ + owner, + now + 60_000, + now, + invocationId, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + invocation.startToken, + JSON.stringify(invocation.principal), + retained.resultDigest, + operation.operationKey, + operation.requestDigest, + stored.write_revision, + stored.write_generation, + now, + invocation.workflowId, + invocation.runId, + snapshot, + ]; + const gate = await this.#admission(invocation, base.length); + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ?, + operation_state = 'resume-admitted', resume_admitted_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? + AND workflow_id = ? AND agent_id = ? AND thread_id = ? + AND resource_id = ? AND run_id = ? AND start_token = ? + AND principal_json = ? AND result_digest = ? AND operation_key = ? + AND notification_request_digest = ? AND write_revision = ? + AND write_generation = ? AND deadline_at > ? + AND operation_state = 'receipt-recorded' AND status = 'result-retained' + AND receipt_recorded_at IS NOT NULL AND resume_admitted_at IS NULL + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + AND ${gate.predicate} RETURNING invocation_id`) + .bind(...base, ...gate.values) + .all(), + ); + if (!row) + throw new RemoteStepRefusal(409, 'remote step continuation changed'); + return owner; + } + + async claimResume(invocationId: string, leaseMs: number): Promise { + if ( + !isPathSafeId(invocationId) || + !Number.isSafeInteger(leaseMs) || + leaseMs < 1 || + leaseMs > 60_000 + ) + throw new RemoteStepRefusal(400, 'remote step resume claim is malformed'); + const invocation = await this.get(invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if (invocation.status !== 'result-retained') + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const start = await this.#execution(invocation, invocation.principal); + if (start.execution.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); + if (!(await this.readRetainedResult(invocationId))) + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const now = this.#now(); + const owner = crypto.randomUUID(); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND operation_state IS NULL + AND deployment_id = ? AND table_prefix = ? + AND workflow_id = ? AND agent_id = ? AND thread_id = ? + AND resource_id = ? AND run_id = ? AND start_token = ? + AND permit_generation = ? AND permit_epoch IS ? + AND result_digest IS NOT NULL + AND status = 'result-retained' + RETURNING invocation_id`) + .bind( + owner, + now + leaseMs, + invocationId, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + invocation.startToken, + invocation.permitGeneration, + invocation.permitEpoch ?? null, + ) + .all(); + if (!rowResult(result)) + throw new RemoteStepRefusal(409, 'remote step resume is claimed'); + return owner; + } + + async settleAfterResume( + invocationId: string, + leaseOwner: string, + witnessed?: RunSummary, + ): Promise< + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof' + > { + if (!isPathSafeId(invocationId) || !isPathSafeId(leaseOwner)) + throw new RemoteStepRefusal(400, 'remote step resume claim is malformed'); + const invocation = await this.get(invocationId); + if (invocation?.status !== 'resuming') + throw new RemoteStepRefusal(409, 'remote step resume claim changed'); + const start = await this.#execution(invocation, invocation.principal); + if (start.execution.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); + const current = await this.#runtime.authoritativeStatus( + invocation.workflowId, + invocation.runId, + ); + if (!current || current.runId !== invocation.runId) + throw new RemoteStepRefusal(503, 'remote step outcome is unreadable'); + const same = (summary: RunSummary) => { + if (summary.status !== 'suspended') return false; + const key = invocation.stepPath.join('.'); + const pathPresent = summary.suspended?.some( + (path) => JSON.stringify(path) === JSON.stringify(invocation.stepPath), + ); + const suspended = record(record(summary.suspendPayload)?.[key]); + const tool = record(suspended?.toolCallSuspended); + return ( + pathPresent === true && + summary.suspendedAt?.[key] === invocation.suspendedAt && + summary.resumeCount?.[key] === invocation.resumeCount && + suspended?.type === 'suspension' && + tool?.remoteStep === REMOTE_STEP_MARKER && + JSON.stringify(tool.request) === JSON.stringify(invocation.request) + ); + }; + const sameObservedOutcome = (left: RunSummary, right: RunSummary) => + left.runId === right.runId && + left.status === right.status && + ((left.status === 'success' && + JSON.stringify(left.result) === JSON.stringify(right.result)) || + (left.status === 'suspended' && + JSON.stringify(left.suspended) === JSON.stringify(right.suspended) && + JSON.stringify(left.suspendedAt) === + JSON.stringify(right.suspendedAt) && + JSON.stringify(left.resumeCount) === + JSON.stringify(right.resumeCount) && + JSON.stringify(left.suspendPayload) === + JSON.stringify(right.suspendPayload))); + let outcome: + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof'; + if (same(current)) + outcome = + this.#now() >= invocation.deadlineAt ? 'deadline-expired' : 'retry'; + else if (current.status === 'suspended' || current.status === 'success') + outcome = + witnessed && sameObservedOutcome(witnessed, current) && !same(witnessed) + ? 'applied' + : 'advanced-without-proof'; + else if ( + [ + 'failed', + 'tripwire', + 'canceled', + 'bailed', + 'skipped', + 'cancelled', + 'timed_out', + ].includes(current.status) + ) + outcome = 'terminal-not-success'; + else + throw new RemoteStepRefusal(503, 'remote step outcome is still changing'); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET status = ?, resume_lease_owner = NULL, resume_lease_expires_at = NULL, + settled_at = ?, settlement_outcome = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND status = 'resuming' + AND resume_lease_owner = ? AND permit_generation = ? + AND permit_epoch IS ? AND start_token = ? + AND deployment_id = ? AND table_prefix = ? AND workflow_id = ? + AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? + RETURNING invocation_id`) + .bind( + outcome === 'retry' + ? 'result-retained' + : outcome === 'applied' + ? 'settled' + : 'failed', + outcome === 'retry' ? null : this.#now(), + outcome === 'retry' ? null : outcome, + invocationId, + leaseOwner, + invocation.permitGeneration, + invocation.permitEpoch ?? null, + invocation.startToken, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + ) + .all(); + if (!rowResult(result)) + throw new RemoteStepRefusal(409, 'remote step resume claim changed'); + return outcome; + } + + async nextNotificationRecoveryBatch(limit: number): Promise { + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) + throw new RemoteStepRefusal( + 400, + 'notification recovery limit is malformed', + ); + await this.initialize(); + const definition = record( + await this.#db + .prepare('SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?') + .bind('table', REMOTE_STEP_NOTIFICATION_CURSOR_TABLE) + .all(), + )?.results; + if ( + !Array.isArray(definition) || + definition.length !== 1 || + record(definition[0])?.sql !== REMOTE_STEP_NOTIFICATION_CURSOR_SQL + ) + throw new RemoteStepRefusal( + 503, + 'notification recovery cursor is unreadable', + ); + await this.#db + .prepare(`INSERT OR IGNORE INTO ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} + (deployment_id, table_prefix, version, after_invocation_id) + VALUES (?, ?, 1, '')`) + .bind(this.#deploymentId, this.#tablePrefix) + .run(); + const cursorRows = record( + await this.#db + .prepare(`SELECT version, after_invocation_id + FROM ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} + WHERE deployment_id = ? AND table_prefix = ?`) + .bind(this.#deploymentId, this.#tablePrefix) + .all(), + )?.results; + const cursor = + Array.isArray(cursorRows) && cursorRows.length === 1 + ? record(cursorRows[0]) + : undefined; + const after = cursor?.after_invocation_id; + if ( + cursor?.version !== 1 || + typeof after !== 'string' || + (after !== '' && !isPathSafeId(after)) + ) + throw new RemoteStepRefusal( + 503, + 'notification recovery cursor is unreadable', + ); + const scan = async (afterInvocationId: string) => { + const values = record( + await this.#db + .prepare(`SELECT invocation_id FROM ${REMOTE_STEP_TABLE} + INDEXED BY ${REMOTE_STEP_OPERATION_INDEX} + WHERE invocation_id > ? AND deployment_id = ? AND table_prefix = ? + AND target_kind = 'agent' + AND status IN ('result-retained', 'resuming') + AND operation_state IN ('prepared', 'effect-admitted', + 'receipt-recorded', 'resume-admitted', 'unresolved') + ORDER BY invocation_id LIMIT ?`) + .bind(afterInvocationId, this.#deploymentId, this.#tablePrefix, limit) + .all(), + )?.results; + if ( + !Array.isArray(values) || + values.some((value) => !isPathSafeId(record(value)?.invocation_id)) + ) + throw new RemoteStepRefusal( + 503, + 'notification recovery scan is unreadable', + ); + return values.map((value) => record(value)?.invocation_id as string); + }; + let ids = await scan(after); + if (ids.length === 0 && after !== '') ids = await scan(''); + const next = ids.at(-1) ?? ''; + const advanced = record( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} + SET after_invocation_id = ? + WHERE deployment_id = ? AND table_prefix = ? AND version = 1 + AND after_invocation_id = ? + RETURNING after_invocation_id`) + .bind(next, this.#deploymentId, this.#tablePrefix, after) + .all(), + )?.results; + if (!Array.isArray(advanced) || advanced.length !== 1) + throw new RemoteStepRefusal(409, 'notification recovery cursor changed'); + return ids; + } + + async recoverableResults( + limit: number, + afterInvocationId = '', + ): Promise<{ invocationIds: string[]; nextCursor?: string }> { + if ( + !Number.isSafeInteger(limit) || + limit < 1 || + limit > 100 || + (afterInvocationId !== '' && !isPathSafeId(afterInvocationId)) + ) + throw new RemoteStepRefusal( + 400, + 'remote step recovery limit is malformed', + ); + const now = this.#now(); + const rows = record( + await this.#db + .prepare(`SELECT invocation_id, status, resume_lease_expires_at + FROM ${REMOTE_STEP_TABLE} INDEXED BY ${REMOTE_STEP_RECOVERY_INDEX} + WHERE invocation_id > ? + AND target_kind = 'agent' + AND operation_state IS NULL + AND status IN ('result-retained', 'resuming') + ORDER BY invocation_id LIMIT ?`) + .bind(afterInvocationId, limit) + .all(), + )?.results; + if ( + !Array.isArray(rows) || + rows.some( + (row) => + !isPathSafeId(record(row)?.invocation_id) || + !['result-retained', 'resuming'].includes( + record(row)?.status as string, + ) || + (record(row)?.status === 'resuming' && + !positiveTime(record(row)?.resume_lease_expires_at)), + ) + ) + throw new RemoteStepRefusal( + 503, + 'remote step recovery scan is unreadable', + ); + const invocationIds = rows + .filter((row) => { + const value = record(row); + return ( + value?.status === 'result-retained' || + (value?.status === 'resuming' && + positiveTime(value.resume_lease_expires_at) && + value.resume_lease_expires_at <= now) + ); + }) + .map((row) => record(row)?.invocation_id as string); + const last = rows.at(-1); + return { + invocationIds, + ...(rows.length === limit && last + ? { nextCursor: record(last)?.invocation_id as string } + : {}), + }; + } + + async reconcileResult( + invocationId: string, + ): Promise< + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof' + | 'pending-quiescence' + > { + const invocation = await this.get(invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if (invocation.status === 'resuming') return 'pending-quiescence'; + const owner = await this.claimResume(invocationId, 60_000); + return this.settleAfterResume(invocationId, owner); + } +} diff --git a/packages/flowsafe/src/execution-entry-matrix.test.ts b/packages/flowsafe/src/execution-entry-matrix.test.ts index 616306ca..f5679c2e 100644 --- a/packages/flowsafe/src/execution-entry-matrix.test.ts +++ b/packages/flowsafe/src/execution-entry-matrix.test.ts @@ -1290,6 +1290,11 @@ type GateSite = { }; const GATE_SITES: ReadonlyArray = [ + { + file: 'do-runner/remote-step-journal.ts', + predicate: 'admitsExistingRun', + drivenBy: 'do-runner/remote-step-journal.test.ts', + }, { file: 'do-runner/execution-fence.ts', predicate: 'admitsExistingRun', @@ -1466,7 +1471,11 @@ function sourceFileSystem(): SourceFileSystem { ).process?.getBuiltinModule?.('node:fs') as SourceFileSystem; } -type SourceFile = { readonly file: string; readonly source: string }; +type SourceFile = { + readonly file: string; + readonly source: string; + readonly parsed?: ts.SourceFile; +}; type SourceUrlModule = { fileURLToPath(url: URL): string }; @@ -1511,14 +1520,14 @@ function walkSourceFiles( * Every actual predicate/delegation call. Definitions, re-exports and comments * are not calls; the defining module's own nomination gates remain visible. */ -function predicateCallSites({ file, source }: SourceFile): GateSite[] { +function predicateCallSites({ + file, + source, + parsed: cached, +}: SourceFile): GateSite[] { const found: GateSite[] = []; - const parsed = ts.createSourceFile( - file, - source, - ts.ScriptTarget.Latest, - true, - ); + const parsed = + cached ?? ts.createSourceFile(file, source, ts.ScriptTarget.Latest, true); const visit = (node: ts.Node): void => { if (ts.isCallExpression(node)) { const expression = node.expression; @@ -1555,13 +1564,13 @@ function predicateCallSites({ file, source }: SourceFile): GateSite[] { return found; } -function sqlAdmissionSites({ file, source }: SourceFile): GateSite[] { - const parsed = ts.createSourceFile( - file, - source, - ts.ScriptTarget.Latest, - true, - ); +function sqlAdmissionSites({ + file, + source, + parsed: cached, +}: SourceFile): GateSite[] { + const parsed = + cached ?? ts.createSourceFile(file, source, ts.ScriptTarget.Latest, true); const found: GateSite[] = []; const admissionBindings = new Set(); const collectBindings = (node: ts.Node): void => { @@ -1617,9 +1626,15 @@ function sqlAdmissionSites({ file, source }: SourceFile): GateSite[] { function gateCallSites(): GateSite[] { const found: GateSite[] = []; walkSourceFiles(sourceRoot(), (sourceFile) => { + const parsed = ts.createSourceFile( + sourceFile.file, + sourceFile.source, + ts.ScriptTarget.Latest, + true, + ); found.push( - ...predicateCallSites(sourceFile), - ...sqlAdmissionSites(sourceFile), + ...predicateCallSites({ ...sourceFile, parsed }), + ...sqlAdmissionSites({ ...sourceFile, parsed }), ); }); return found; diff --git a/packages/flowsafe/src/host-kit/do-response.ts b/packages/flowsafe/src/host-kit/do-response.ts index e0a7af0b..823a8764 100644 --- a/packages/flowsafe/src/host-kit/do-response.ts +++ b/packages/flowsafe/src/host-kit/do-response.ts @@ -7,11 +7,15 @@ // router's error mapping rather than collapse into a generic 500. import { + assertExecutionPrincipal, + type ExecutionPrincipal, isExecutionPrincipalId, isExecutionPrincipalKind, } from '../approval-api/principal.js'; import { isPublishedRefusal } from '../do-runner/do-status-error.js'; import { + type D1RunExecutionIdentity, + normalizeD1RunExecutionIdentity, normalizeStartExecutionIdentity, type StartExecutionIdentity, } from '../do-runner/execution-admission.js'; @@ -315,3 +319,75 @@ export async function doPersistedStart( throw new RunRouteError(503, 'persisted start is not readable'); } } + +export interface StartPrincipalReceipt { + readonly version: 1; + readonly execution: D1RunExecutionIdentity; + readonly owner: { + readonly kind: ExecutionPrincipal['kind']; + readonly id: string; + }; + readonly principal: ExecutionPrincipal; +} + +/** @internal A private DO receipt is trusted only after exact selector and owner checks. */ +export async function doStartPrincipal( + response: DoResponseLike, + expected: { workflowId: string; runId: string }, +): Promise { + if (!response.ok) { + throw await objectAnswerError('run-object', response, 'run'); + } + try { + if (response.status !== 200) throw new Error('invalid status'); + const payload = persistedStartRecord(await response.json()); + if ( + Object.keys(payload).sort().join(',') !== + 'execution,owner,principal,version' || + payload.version !== 1 + ) + throw new Error('invalid receipt'); + const rawExecution = persistedStartRecord(payload.execution); + if ( + Object.keys(rawExecution).sort().join(',') !== + 'runId,startToken,tablePrefix,workflowId' + ) + throw new Error('invalid execution'); + const execution = normalizeD1RunExecutionIdentity(rawExecution); + if ( + rawExecution.tablePrefix !== execution.tablePrefix || + execution.workflowId !== expected.workflowId || + execution.runId !== expected.runId + ) + throw new Error('selector mismatch'); + const owner = persistedStartRecord(payload.owner); + if ( + Object.keys(owner).sort().join(',') !== 'id,kind' || + !isExecutionPrincipalKind(owner.kind) || + !isExecutionPrincipalId(owner.id) + ) + throw new Error('invalid owner'); + const rawPrincipal = persistedStartRecord(payload.principal); + const principal = assertExecutionPrincipal(rawPrincipal, 'start receipt'); + const principalKeys = + principal.kind === 'human' + ? 'id,kind,role' + : principal.kind === 'agent' && principal.delegatedBy !== undefined + ? 'delegatedBy,id,kind,purpose' + : 'id,kind,purpose'; + if ( + Object.keys(rawPrincipal).sort().join(',') !== principalKeys || + principal.kind !== owner.kind || + principal.id !== owner.id + ) + throw new Error('principal mismatch'); + return Object.freeze({ + version: 1, + execution, + owner: Object.freeze({ kind: owner.kind, id: owner.id }), + principal, + }); + } catch { + throw new RunRouteError(503, 'run start principal is not readable'); + } +} diff --git a/packages/flowsafe/src/host-kit/do-run-topology.test.ts b/packages/flowsafe/src/host-kit/do-run-topology.test.ts index 54a099a3..75d64707 100644 --- a/packages/flowsafe/src/host-kit/do-run-topology.test.ts +++ b/packages/flowsafe/src/host-kit/do-run-topology.test.ts @@ -726,3 +726,111 @@ describe('protected replay workflow transport', () => { ); }); }); + +describe('private start-principal receipt transport', () => { + const receipt = { + version: 1, + execution: { + tablePrefix: 'tenant_', + workflowId: 'workflow-1', + runId: 'run-1', + startToken: 'generation-1', + }, + owner: { kind: 'agent', id: 'agent-1' }, + principal: { + kind: 'agent', + id: 'agent-1', + purpose: 'customer.dispatch', + delegatedBy: 'operator-1', + }, + }; + + function topologyFor(payload: unknown, status = 200) { + const fetch = vi.fn( + async ( + _url: string, + _init?: Parameters< + ReturnType['get']>['fetch'] + >[1], + ) => Response.json(payload, { status }), + ); + const idFromName = vi.fn((name: string) => name); + return { + fetch, + idFromName, + topology: createDoRunTopology( + { idFromName, get: () => ({ fetch }) }, + DEPLOYMENT_IDENTITY_SECRET, + ), + }; + } + + it('reads the exact run receipt over the authenticated private DO route', async () => { + const { topology, fetch, idFromName } = topologyFor(receipt); + await expect( + topology.startPrincipal('workflow-1', 'run-1'), + ).resolves.toEqual(receipt); + expect(idFromName).toHaveBeenCalledWith('workflow-1:run-1'); + expect(fetch).toHaveBeenCalledWith( + 'http://do/runs/workflow-1/run-1/start-principal', + expect.objectContaining({ headers: expect.any(Object) }), + ); + const headers = new Headers(fetch.mock.calls[0]?.[1]?.headers); + expect(headers.get('x-flowsafe-deployment-identity')).toBe( + DEPLOYMENT_IDENTITY_SECRET, + ); + expect(headers.has(EXECUTION_PRINCIPAL_HEADER)).toBe(false); + }); + + it('treats a missing receipt as absence and refuses path-unsafe selectors locally', async () => { + await expect( + topologyFor({ error: 'run not found' }, 404).topology.startPrincipal( + 'workflow-1', + 'run-1', + ), + ).resolves.toBeUndefined(); + const { topology, fetch, idFromName } = topologyFor(receipt); + await expect( + topology.startPrincipal('workflow/1', 'run-1'), + ).rejects.toMatchObject({ + status: 400, + }); + expect(idFromName).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }); + + it.each([ + [ + 'foreign run', + { ...receipt, execution: { ...receipt.execution, runId: 'other' } }, + ], + [ + 'foreign workflow', + { ...receipt, execution: { ...receipt.execution, workflowId: 'other' } }, + ], + ['owner mismatch', { ...receipt, owner: { kind: 'agent', id: 'other' } }], + ['unknown version', { ...receipt, version: 2 }], + [ + 'noncanonical prefix', + { + ...receipt, + execution: { ...receipt.execution, tablePrefix: 'TENANT_' }, + }, + ], + [ + 'unknown authority field', + { ...receipt, callerPrincipal: receipt.principal }, + ], + [ + 'malformed principal', + { ...receipt, principal: { ...receipt.principal, purpose: '' } }, + ], + ])('refuses a %s private receipt', async (_label, payload) => { + await expect( + topologyFor(payload).topology.startPrincipal('workflow-1', 'run-1'), + ).rejects.toMatchObject({ + status: 503, + message: 'run start principal is not readable', + }); + }); +}); diff --git a/packages/flowsafe/src/host-kit/do-run-topology.ts b/packages/flowsafe/src/host-kit/do-run-topology.ts index 34c0b8c5..6f2c25ca 100644 --- a/packages/flowsafe/src/host-kit/do-run-topology.ts +++ b/packages/flowsafe/src/host-kit/do-run-topology.ts @@ -25,14 +25,18 @@ import { type RunSummary, stampMutationEpoch, } from '../do-runner/index.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; import type { PersistedStartResult } from '../do-runner/start-idempotency.js'; import { captureReservation } from '../do-runner/start-reservation-contract.js'; import { type DoResponseLike, doPersistedStart, doStartLiveness, + doStartPrincipal, doSummary, + type StartPrincipalReceipt, } from './do-response.js'; +import { RunRouteError } from './run-route-error.js'; import type { RunStartInput } from './run-router.js'; /** The subset of a DurableObjectStub the topology uses. */ @@ -91,6 +95,11 @@ export interface DoRunTopology { /** Additive run-lifecycle operations returned by createDoRunTopology. */ export interface DoRunLifecycleTopology extends DoRunTopology { + /** Reads the authenticated run object's stored start authority. */ + startPrincipal( + workflowId: string, + runId: string, + ): Promise; terminate( workflowId: string, runId: string, @@ -146,6 +155,17 @@ export function createDoRunTopology( ); return { + startPrincipal: async (workflowId, runId) => { + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) { + throw new RunRouteError(400, 'invalid run identity'); + } + const response = await stub(workflowId, runId).fetch( + `http://do/runs/${workflowId}/${runId}/start-principal`, + { headers: deploymentIdentityHeaders(deploymentIdentitySecret) }, + ); + if (response.status === 404) return undefined; + return doStartPrincipal(response, { workflowId, runId }); + }, start: async ({ workflowId, runId, diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts index 73d82598..9f75ea3f 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts @@ -3,6 +3,11 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; +import type { WorkflowRemoteStepDispatchReader } from '../agent-host/thread-topology.js'; +import { + createWorkflowRemoteStepPreRoute, + WORKFLOW_REMOTE_STEP_ROUTE, +} from '../agent-host/workflow-remote-step-route.js'; import type { ApprovalNotificationEvent, ApprovalRecord, @@ -936,6 +941,332 @@ describe('createFlowsafeWorker fetch pipeline', () => { expect(await list.json()).toEqual([]); }); + it('authenticates private workflow dispatch before reading stored authority', async () => { + const { env, ctx } = makeEnv(); + const startRead = vi.fn(async () => + Response.json({ + version: 1, + execution: { + tablePrefix: '', + workflowId: 'wf', + runId: 'run-1', + startToken: 'start-1', + }, + owner: { kind: 'human', id: 'ada' }, + principal: { kind: 'human', id: 'ada', role: 'operator' }, + }), + ); + env.RUNNER = { + idFromName: (name: string) => name, + get: () => ({ fetch: startRead }), + }; + const invocation = { + targetKind: 'workflow' as const, + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + tablePrefix: '', + workflowId: 'wf', + runId: 'run-1', + principal: { + kind: 'human' as const, + id: 'ada', + role: 'operator' as const, + }, + startToken: 'start-1', + stepPath: ['remote-step'], + suspendedAt: 1000, + request: { script: 'script-1' }, + requestDigest: 'stored-request-digest', + deadlineAt: 9000, + status: 'pending' as const, + permitGeneration: 0, + }; + const getWorkflow = vi.fn(async () => invocation); + const journal = { + deploymentId: 'deployment-1', + tablePrefix: '', + getWorkflow, + } as unknown as WorkflowRemoteStepDispatchReader; + const registerWorkflowRemoteStep = vi.fn(async () => invocation); + const issueWorkflowRemoteStepDispatch = vi.fn(async () => ({ + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + workflowId: 'wf', + runId: 'run-1', + stepPath: ['remote-step'], + suspendedAt: 1000, + deadlineAt: 9000, + requestJson: '{"script":"script-1"}', + requestDigest: 'stored-request-digest', + permit: { + invocationId: 'invocation-1', + generation: 1, + mutationEpoch: 1, + transitionRevision: 1, + expiresAt: 9000, + }, + })); + const consumeWorkflowRemoteStepDispatch = vi.fn(async () => ({ + kind: 'accepted' as const, + rawPermit: 'raw-secret', + })); + const recordWorkflowRemoteStepResult = vi.fn(async () => ({ + kind: 'recorded' as const, + resultJson: '{"answer":42}', + resultDigest: 'stored-result-digest', + })); + const worker = makeWorker({ + preRoutes: createWorkflowRemoteStepPreRoute(journal, { + registerWorkflowRemoteStep, + issueWorkflowRemoteStepDispatch, + consumeWorkflowRemoteStepDispatch, + recordWorkflowRemoteStepResult, + }), + }); + const request = (operation: string, body: unknown, credential?: string) => + new Request(`http://host${WORKFLOW_REMOTE_STEP_ROUTE}/${operation}`, { + method: 'POST', + headers: { + 'content-type': 'application/json', + ...(credential ? { authorization: `Bearer ${credential}` } : {}), + }, + body: JSON.stringify(body), + }); + const register = { workflowId: 'wf', runId: 'run-1', hostDeadlineAt: 9000 }; + expect( + (await worker.fetch(request('register', register), env, ctx)).status, + ).toBe(503); + env.MAINTENANCE_ADMIN_SECRET = 'private-admin-secret-00000000000001'; + expect( + (await worker.fetch(request('register', register, 'wrong'), env, ctx)) + .status, + ).toBe(401); + expect(startRead).not.toHaveBeenCalled(); + expect(getWorkflow).not.toHaveBeenCalled(); + const token = env.MAINTENANCE_ADMIN_SECRET; + expect( + ( + await worker.fetch( + request( + 'register', + { ...register, principal: invocation.principal }, + token, + ), + env, + ctx, + ) + ).status, + ).toBe(400); + expect(startRead).not.toHaveBeenCalled(); + const registered = await worker.fetch( + request('register', register, token), + env, + ctx, + ); + expect(registered.status).toBe(200); + expect(await registered.json()).toEqual({ invocationId: 'invocation-1' }); + expect(registerWorkflowRemoteStep).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + 'wf', + 'run-1', + 9000, + ); + const ticket = await worker.fetch( + request( + 'ticket', + { invocationId: 'invocation-1', lifetimeMs: 60000 }, + token, + ), + env, + ctx, + ); + expect(ticket.status).toBe(200); + expect(issueWorkflowRemoteStepDispatch).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + 'invocation-1', + 60000, + ); + const issued = (await ticket.json()) as { + permit: { + invocationId: string; + generation: number; + mutationEpoch: number; + transitionRevision: number; + expiresAt: number; + }; + }; + const consumed = await worker.fetch( + request('consume', { permit: issued.permit }, token), + env, + ctx, + ); + expect(consumed.status).toBe(200); + expect(await consumed.json()).toEqual({ + kind: 'accepted', + rawPermit: 'raw-secret', + }); + expect(consumed.headers.get('cache-control')).toBe('no-store'); + expect(consumeWorkflowRemoteStepDispatch).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + issued.permit, + ); + const result = await worker.fetch( + request( + 'result', + { + invocationId: 'invocation-1', + rawPermit: 'raw-secret', + resultJson: '{"answer":42}', + evidenceJson: '{"quiescent":true}', + }, + token, + ), + env, + ctx, + ); + expect(result.status).toBe(200); + expect(recordWorkflowRemoteStepResult).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + 'invocation-1', + 'raw-secret', + '{"answer":42}', + '{"quiescent":true}', + ); + issueWorkflowRemoteStepDispatch.mockRejectedValueOnce( + new RunRouteError(409, 'remote step dispatch changed'), + ); + const changed = await worker.fetch( + request( + 'ticket', + { invocationId: 'invocation-1', lifetimeMs: 60000 }, + token, + ), + env, + ctx, + ); + expect(changed.status).toBe(409); + }); + + it('binds a pre-route context to the stored invocation principal', async () => { + const journal = { + deploymentId: 'deployment-1', + tablePrefix: '', + get: vi.fn(async () => ({ + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + tablePrefix: '', + principal: { + kind: 'service' as const, + id: 'trusted-dispatcher', + purpose: 'remote-step', + }, + })), + }; + const worker = makeWorker({ + preRoutes: async (request, _env, _ctx, kit) => { + if (new URL(request.url).pathname !== '/test/stored-principal') + return null; + const context = await kit.contextForStoredInvocation( + journal, + 'invocation-1', + ); + return Response.json(context?.principal ?? null); + }, + }); + const { env, ctx } = makeEnv(); + + const response = await worker.fetch( + new Request('http://host/test/stored-principal'), + env, + ctx, + ); + + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ + kind: 'service', + id: 'trusted-dispatcher', + purpose: 'remote-step', + }); + expect(journal.get).toHaveBeenCalledWith('invocation-1'); + }); + + it.each([ + { + name: 'missing invocation', + invocationId: 'invocation-1', + row: undefined, + reads: 1, + }, + { + name: 'mismatched invocation id', + invocationId: 'invocation-1', + row: { invocationId: 'another-invocation' }, + reads: 1, + }, + { + name: 'mismatched deployment', + invocationId: 'invocation-1', + row: { deploymentId: 'another-deployment' }, + reads: 1, + }, + { + name: 'mismatched table prefix', + invocationId: 'invocation-1', + row: { tablePrefix: 'another_' }, + reads: 1, + }, + { + name: 'invalid invocation id', + invocationId: '../invocation-1', + row: {}, + reads: 0, + }, + ])('refuses a stored-principal context for $name', async ({ + invocationId, + row, + reads, + }) => { + const stored = { + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + tablePrefix: '', + principal: { + kind: 'service' as const, + id: 'trusted-dispatcher', + purpose: 'remote-step', + }, + }; + const journal = { + deploymentId: 'deployment-1', + tablePrefix: '', + get: vi.fn(async () => + row === undefined ? undefined : { ...stored, ...row }, + ), + }; + const worker = makeWorker({ + preRoutes: async (request, _env, _ctx, kit) => { + if (new URL(request.url).pathname !== '/test/stored-principal') + return null; + const context = await kit.contextForStoredInvocation( + journal, + invocationId, + ); + return Response.json(context?.principal ?? null); + }, + }); + const { env, ctx } = makeEnv(); + + const response = await worker.fetch( + new Request('http://host/test/stored-principal'), + env, + ctx, + ); + + expect(response.status).toBe(200); + expect(await response.json()).toBeNull(); + expect(journal.get).toHaveBeenCalledTimes(reads); + }); + it('mounts the agent router after preRoutes and before the signal router', async () => { const order: string[] = []; const worker = makeWorker({ @@ -1516,6 +1847,35 @@ describe('createFlowsafeWorker maintenance duties', () => { expect(outboundFetch).not.toHaveBeenCalled(); }); + it('admits a trusted private state egress context without a route hostname', async () => { + const worker = makeWorker(); + const { env, ctx } = makeEnv(); + const outboundFetch = vi.fn(async () => new Response(null)); + env.FLEET_RESOURCE_ROLE = 'platform-state'; + env.FLEET_ENVIRONMENT = 'production'; + env.FLEET_RESOURCE_GROUP = '0123456789abcdefabcd'; + env.OUTBOUND_PROXY = { fetch: outboundFetch }; + env.OUTBOUND_PROXY_CREDENTIAL = 'outbound-state-credential-0000000001'; + env.OUTBOUND_TENANT_ID = 'acme'; + env.OUTBOUND_ENVIRONMENT = 'production'; + env.OUTBOUND_RESOURCE_GROUP_ID = '0123456789abcdefabcd'; + env.OUTBOUND_STATE_SCRIPT_NAME = + 'acme-production-state-0123456789abcdefabcd'; + env.OUTBOUND_ACTIVATION_VERSION = '2'; + env.OUTBOUND_ACTIVATION_KIND = 'internal-dispatch'; + env.OUTBOUND_DISPATCH_KEY = 'acme:production'; + env.OUTBOUND_POLICY_ID = '0123456789abcdefabcd'; + + const response = await worker.fetch( + new Request('http://host/healthz'), + env, + ctx, + ); + + expect(response.status).toBe(200); + expect(outboundFetch).not.toHaveBeenCalled(); + }); + it('rejects candidate access to trusted state egress before service use', async () => { const worker = makeWorker(); const { env, ctx } = makeEnv(); @@ -1532,6 +1892,20 @@ describe('createFlowsafeWorker maintenance duties', () => { expect(outboundFetch).not.toHaveBeenCalled(); }); + it('rejects a private dispatch key on a candidate Worker', async () => { + const worker = makeWorker(); + const { env, ctx } = makeEnv(); + env.OUTBOUND_DISPATCH_KEY = 'acme:production'; + + const response = await worker.fetch( + new Request('http://host/healthz'), + env, + ctx, + ); + + expect(response.status).toBe(500); + }); + it('the purge duty runs ONLY the purge duties', async () => { // #given const logs = capturedLogs(); @@ -3300,7 +3674,9 @@ describe('createFlowsafeWorker drain inventory', () => { unenumerable: Array<{ name: string }>; drainProof: { reading: string; reachableFrom: string[] }; }; - expect(body.categories.filter((c) => c.class === 'work')).toHaveLength(7); + expect(body.categories).toContainEqual( + expect.objectContaining({ category: 'remote-steps', class: 'work' }), + ); expect(body.categories.filter((c) => c.class === 'standing')).toHaveLength( 2, ); diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.ts index b0471fc4..599e7c19 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.ts @@ -15,10 +15,14 @@ import { APPROVAL_ROLES, createActorResolver, createApprovalRouter, + createPrincipalActorContext, createResourceOwnershipSchema, + DECIDER_ROLES, RESOURCE_OWNERSHIP_TABLE, + selfDecisionExempts, trustAutomationPrincipal, } from '../approval-api/index.js'; +import type { ExecutionPrincipal } from '../approval-api/principal.js'; import { type AuditProxyNamespaceLike, type AuditQueue, @@ -68,6 +72,7 @@ import { verifyDurableObjectDeploymentIdentity, verifyDurableObjectDeploymentRequest, } from '../do-runner/index.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; import { validateTablePrefix } from '../do-runner/table-prefix.js'; import { readBoundedBody } from '../http-body.js'; import { @@ -124,6 +129,39 @@ export interface FlowsafeWorkerContext { /** Fixed instance name used by every deployment maintenance namespace. */ export const MAINTENANCE_INSTANCE_NAME = 'deployment-maintenance'; +export async function ensureRemoteStepNotificationRecoveryDuty( + env: FlowsafeWorkerEnv, +): Promise { + const id = env.MAINTENANCE.idFromName(MAINTENANCE_INSTANCE_NAME); + const response = await env.MAINTENANCE.get(id).fetch( + 'http://maintenance/ensure', + { + method: 'POST', + headers: deploymentIdentityHeaders(env.DEPLOYMENT_IDENTITY_SECRET), + }, + ); + if (!response.ok) + throw new Error('remote step notification recovery duty is unavailable'); + let value: unknown; + try { + value = await response.json(); + } catch { + throw new Error('remote step notification recovery duty is unreadable'); + } + const health = + value !== null && typeof value === 'object' + ? (value as Record) + : undefined; + const next = validTime(health?.nextRemoteStepRecoveryAt); + const alarmAt = validTime(health?.alarmAt); + if ( + next === undefined || + alarmAt === undefined || + alarmAt > Math.max(Date.now(), next) + ) + throw new Error('remote step notification recovery duty is not armed'); +} + export interface MaintenanceStubLike { fetch( url: string, @@ -142,20 +180,24 @@ export interface MaintenanceHealth { nextPurgeAt: number; nextTickAt?: number; nextNotificationAt?: number; + nextRemoteStepRecoveryAt?: number; lastSweepAt?: number; lastPurgeAt?: number; lastTickAt?: number; lastNotificationAt?: number; + lastRemoteStepRecoveryAt?: number; lastDeadlineAt?: number; lastSweepAttemptAt?: number; lastPurgeAttemptAt?: number; lastTickAttemptAt?: number; lastNotificationAttemptAt?: number; + lastRemoteStepRecoveryAttemptAt?: number; lastDeadlineAttemptAt?: number; lastSweepError?: string; lastPurgeError?: string; lastTickError?: string; lastNotificationError?: string; + lastRemoteStepRecoveryError?: string; lastDeadlineError?: string; } @@ -301,6 +343,9 @@ export interface FlowsafeWorkerEnv { OUTBOUND_RESOURCE_GROUP_ID?: string; OUTBOUND_STATE_SCRIPT_NAME?: string; OUTBOUND_ROUTE_HOSTNAME?: string; + OUTBOUND_ACTIVATION_VERSION?: string; + OUTBOUND_ACTIVATION_KIND?: string; + OUTBOUND_DISPATCH_KEY?: string; OUTBOUND_POLICY_ID?: string; /** Dedicated control-plane credential for maintenance administration. */ MAINTENANCE_ADMIN_SECRET?: string; @@ -361,13 +406,15 @@ export interface FlowsafeWorkerConfig * Recurrence intervals for the singleton maintenance Durable Object. Every * duty receives its own alarm invocation; `tickIntervalMs` is required when * `scheduleTick` is configured and omitted otherwise, and - * `notificationIntervalMs` likewise with `notificationTick`. + * `notificationIntervalMs` likewise with `notificationTick`, and + * `remoteStepRecoveryIntervalMs` with `remoteStepRecoveryTick`. */ maintenance: { sweepIntervalMs: number; purgeIntervalMs: number; tickIntervalMs?: number; notificationIntervalMs?: number; + remoteStepRecoveryIntervalMs?: number; /** Deadline duty cadence. Defaults to sweepIntervalMs. */ deadlineIntervalMs?: number; /** Maximum runs per deadline pass. Default 100. */ @@ -383,7 +430,49 @@ export interface FlowsafeWorkerConfig request: Request, env: Env, ctx: FlowsafeWorkerContext, - kit: { resolve: ActorResolver; topology: DoRunTopology }, + kit: { + resolve: ActorResolver; + topology: DoRunTopology; + authorizePrivateWorkflowDispatch: ( + request: Request, + ) => Promise; + contextForStoredWorkflowStart: ( + workflowId: string, + runId: string, + ) => Promise; + contextForStoredWorkflowInvocation: ( + journal: { + deploymentId: string; + tablePrefix: string; + getWorkflow: (invocationId: string) => Promise< + | { + invocationId: string; + deploymentId: string; + tablePrefix: string; + principal: ExecutionPrincipal; + } + | undefined + >; + }, + invocationId: string, + ) => Promise; + contextForStoredInvocation: ( + journal: { + deploymentId: string; + tablePrefix: string; + get: (invocationId: string) => Promise< + | { + invocationId: string; + deploymentId: string; + tablePrefix: string; + principal: ExecutionPrincipal; + } + | undefined + >; + }, + invocationId: string, + ) => Promise; + }, ) => Promise; /** Host policy immediately before a validated start reaches the run DO. */ beforeStart?: ( @@ -501,6 +590,7 @@ export interface FlowsafeWorkerConfig * notification duty. */ notificationTick?: (env: Env) => () => Promise; + remoteStepRecoveryTick?: (env: Env) => () => Promise; /** * Extra purge duties. The * returned fields fold into the ONE combined `{type:'maintenance'}` log @@ -647,6 +737,9 @@ const STATE_EGRESS_ENV_KEYS = [ 'OUTBOUND_RESOURCE_GROUP_ID', 'OUTBOUND_STATE_SCRIPT_NAME', 'OUTBOUND_ROUTE_HOSTNAME', + 'OUTBOUND_ACTIVATION_VERSION', + 'OUTBOUND_ACTIVATION_KIND', + 'OUTBOUND_DISPATCH_KEY', 'OUTBOUND_POLICY_ID', ] as const; @@ -710,7 +803,8 @@ export type MaintenanceDuty = | 'sweep' | 'purge' | 'tick' - | 'notification'; + | 'notification' + | 'remoteStepRecovery'; /** * The cursor seam a duty resumes from and advances. Each field is optional @@ -761,6 +855,11 @@ const INJECTED_TICK_DUTIES = { interval: 'notificationIntervalMs', logType: 'notification-dispatch', }, + remoteStepRecovery: { + builder: 'remoteStepRecoveryTick', + interval: 'remoteStepRecoveryIntervalMs', + logType: 'remote-step-notification-recovery', + }, } as const; type InjectedTickDuty = keyof typeof INJECTED_TICK_DUTIES; @@ -1177,7 +1276,7 @@ export function createFlowsafeWorker( 'storageTablePrefix', ); // Authenticate before constructing request-scoped services. - const buildResolve = ( + const buildContexts = ( env: Env, topology: DoRunTopology, waitUntil: (promise: Promise) => void, @@ -1185,28 +1284,41 @@ export function createFlowsafeWorker( selfDecision: SelfDecisionPolicy, stream: ApprovalStreamSink | undefined, mutationEpoch: number | undefined, - ): ActorResolver => { - const base = createActorResolver({ + ) => { + const storeFactory = approvalStoreFactoryFor(env.DB, storageTablePrefix); + const buildService = (store: ApprovalStore) => + buildConfiguredApprovalService(config, env, topology, { + store, + executionFence: executionFenceForEnv(env), + workflowTablePrefix: (storageTablePrefix ?? '').toLowerCase(), + waitUntil, + notify, + stream, + allowSelfDecision: selfDecision, + }); + const resolve = createActorResolver({ mutationEpoch, authenticate: bearerActorAuthenticator(config.buildVerifier(env)), - storeFactory: approvalStoreFactoryFor(env.DB, storageTablePrefix), + storeFactory, deploymentTag: env.DEPLOYMENT_TENANT, - buildService: (store) => - buildConfiguredApprovalService(config, env, topology, { - store, - executionFence: executionFenceForEnv(env), - workflowTablePrefix: (storageTablePrefix ?? '').toLowerCase(), - waitUntil, - notify, - stream, - allowSelfDecision: selfDecision, - }), + buildService, // The resolver's canSelfDecide display hint reads the SAME policy the // service enforces (passed to the shared service builder above), so the // /workflows echo can never contradict decide(). allowSelfDecision: selfDecision, }); - return base; + const contextForPrincipal = (principal: ExecutionPrincipal): ActorContext => + createPrincipalActorContext({ + principal, + mutationEpoch, + storeFactory, + deploymentTag: env.DEPLOYMENT_TENANT, + buildService, + canSelfDecide: (role) => + DECIDER_ROLES.includes(role) && + selfDecisionExempts(selfDecision, role), + }); + return { resolve, contextForPrincipal }; }; // Parse ONCE per deployment value, memoized by the RAW @@ -1595,7 +1707,7 @@ export function createFlowsafeWorker( ), ); } - const resolve = buildResolve( + const { resolve, contextForPrincipal } = buildContexts( env, topology, waitUntil, @@ -1606,9 +1718,67 @@ export function createFlowsafeWorker( ); if (config.preRoutes) { + const contextForStoredInvocation = async ( + journal: { + deploymentId: string; + tablePrefix: string; + get: (invocationId: string) => Promise< + | { + invocationId: string; + deploymentId: string; + tablePrefix: string; + principal: ExecutionPrincipal; + } + | undefined + >; + }, + invocationId: string, + ) => { + if (!isPathSafeId(invocationId)) return undefined; + const invocation = await journal.get(invocationId); + if ( + !invocation || + invocation.invocationId !== invocationId || + invocation.deploymentId !== journal.deploymentId || + invocation.tablePrefix !== journal.tablePrefix || + invocation.tablePrefix !== + (storageTablePrefix ?? '').toLowerCase() + ) + return undefined; + return contextForPrincipal(invocation.principal); + }; const preResponse = await config.preRoutes(request, env, ctx, { resolve, topology, + authorizePrivateWorkflowDispatch: async (privateRequest) => { + const gate = await authorizeAdminCredential(privateRequest, env, { + surface: 'private workflow dispatch', + delegateCapability: false, + }); + return gate.authorized ? undefined : gate.response; + }, + contextForStoredWorkflowStart: async (workflowId, runId) => { + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) + return undefined; + const receipt = await topology.startPrincipal(workflowId, runId); + if ( + !receipt || + receipt.execution.tablePrefix !== + (storageTablePrefix ?? '').toLowerCase() + ) + return undefined; + return contextForPrincipal(receipt.principal); + }, + contextForStoredWorkflowInvocation: (journal, invocationId) => + contextForStoredInvocation( + { + deploymentId: journal.deploymentId, + tablePrefix: journal.tablePrefix, + get: journal.getWorkflow.bind(journal), + }, + invocationId, + ), + contextForStoredInvocation, }); if (preResponse) return preResponse; } @@ -1816,6 +1986,7 @@ const DUTY_ORDER = [ 'purge', 'tick', 'notification', + 'remoteStepRecovery', ] as const; export type MaintenanceDurableObjectConstructor = @@ -1839,6 +2010,7 @@ export function createFlowsafeMaintenanceDurableObject< purge: config.maintenance.purgeIntervalMs, tick: config.maintenance.tickIntervalMs, notification: config.maintenance.notificationIntervalMs, + remoteStepRecovery: config.maintenance.remoteStepRecoveryIntervalMs, }; for (const duty of DUTY_ORDER) { const interval = intervals[duty]; @@ -2222,6 +2394,12 @@ const DUTY_HEALTH_FIELDS = { attempt: 'lastNotificationAttemptAt', error: 'lastNotificationError', }, + remoteStepRecovery: { + next: 'nextRemoteStepRecoveryAt', + last: 'lastRemoteStepRecoveryAt', + attempt: 'lastRemoteStepRecoveryAttemptAt', + error: 'lastRemoteStepRecoveryError', + }, } as const satisfies Record< MaintenanceDuty, { diff --git a/packages/flowsafe/src/host-kit/index.ts b/packages/flowsafe/src/host-kit/index.ts index 4dcfa192..9a0f4fb5 100644 --- a/packages/flowsafe/src/host-kit/index.ts +++ b/packages/flowsafe/src/host-kit/index.ts @@ -74,7 +74,11 @@ export { resumeRunWithRequeue, } from './approval-bridge.js'; export { bearerActorAuthenticator, parseActorTokens } from './bearer-auth.js'; -export { type DoResponseLike, doSummary } from './do-response.js'; +export { + type DoResponseLike, + doSummary, + type StartPrincipalReceipt, +} from './do-response.js'; export { createDoRunTopology, type DoRunLifecycleTopology, @@ -108,6 +112,7 @@ export { createFlowsafeMaintenanceDurableObject, createFlowsafeRunnerLifecycle, createFlowsafeWorker, + ensureRemoteStepNotificationRecoveryDuty, MAINTENANCE_INSTANCE_NAME, } from './flowsafe-worker.js'; // hostAuditSink stays module-internal for the same reason as @@ -174,15 +179,21 @@ export type { export { createRunRouter } from './run-router.js'; export type { StateEgressBinding, + StateEgressCommonContext, + StateEgressCommonEnv, StateEgressContext, StateEgressEnv, } from './state-egress.js'; export { createStateEgressFetch, + STATE_EGRESS_ACTIVATION_KIND_HEADER, + STATE_EGRESS_ACTIVATION_VERSION_HEADER, STATE_EGRESS_CREDENTIAL_HEADER, + STATE_EGRESS_DISPATCH_KEY_HEADER, STATE_EGRESS_ENVIRONMENT_HEADER, STATE_EGRESS_HEADERS, STATE_EGRESS_POLICY_ID_HEADER, + STATE_EGRESS_PRIVATE_HEADERS, STATE_EGRESS_RESERVED_HEADERS, STATE_EGRESS_RESOURCE_GROUP_HEADER, STATE_EGRESS_ROUTE_HOSTNAME_HEADER, diff --git a/packages/flowsafe/src/host-kit/maintenance-do.test.ts b/packages/flowsafe/src/host-kit/maintenance-do.test.ts index 3f7fe9ce..a573a33e 100644 --- a/packages/flowsafe/src/host-kit/maintenance-do.test.ts +++ b/packages/flowsafe/src/host-kit/maintenance-do.test.ts @@ -12,6 +12,7 @@ import { } from '../do-runner/start-reservation-contract.js'; import { createFlowsafeMaintenanceDurableObject, + ensureRemoteStepNotificationRecoveryDuty, type FlowsafeWorkerConfig, type FlowsafeWorkerEnv, MAINTENANCE_INSTANCE_NAME, @@ -1166,6 +1167,68 @@ describe('alarm-driven deployment maintenance', () => { ); }); + it('wakes the optional remote-step recovery duty again after restart without traffic', async () => { + vi.useFakeTimers(); + vi.setSystemTime(NOW); + vi.spyOn(console, 'log').mockImplementation(() => {}); + const calls: number[] = []; + const { instance, internalRequest, reconstruct } = harness({ + config: { + maintenance: { + sweepIntervalMs: 15 * 60_000, + purgeIntervalMs: 60 * 60_000, + remoteStepRecoveryIntervalMs: 1_000, + }, + remoteStepRecoveryTick: () => async () => { + calls.push(Date.now()); + return { scanned: 0, recovered: 0, unresolved: 0 }; + }, + }, + }); + await healthOf(instance, internalRequest('/ensure', 'POST')); + for (let index = 0; index < 4; index += 1) await instance.alarm(); + expect(calls).toEqual([NOW]); + const restarted = reconstruct(); + await restarted.alarm(); + expect(calls).toEqual([NOW]); + vi.advanceTimersByTime(1_000); + await restarted.alarm(); + expect(calls).toEqual([NOW, NOW + 1_000]); + }); + + it('arms and verifies the configured remote-step recovery duty before admission', async () => { + vi.useFakeTimers(); + vi.setSystemTime(NOW); + const active = harness({ + config: { + maintenance: { + sweepIntervalMs: 15 * 60_000, + purgeIntervalMs: 60 * 60_000, + remoteStepRecoveryIntervalMs: 1_000, + }, + remoteStepRecoveryTick: () => async () => ({ scanned: 0 }), + }, + }); + const linked = ( + fixture: ReturnType, + ): FlowsafeWorkerEnv => ({ + ...fixture.env, + MAINTENANCE: { + idFromName: (name) => name, + get: () => ({ + fetch: (url, init) => fixture.instance.fetch(new Request(url, init)), + }), + }, + }); + await expect( + ensureRemoteStepNotificationRecoveryDuty(linked(active)), + ).resolves.toBeUndefined(); + expect(active.storage.alarmAt).toBe(NOW); + await expect( + ensureRemoteStepNotificationRecoveryDuty(linked(harness())), + ).rejects.toThrow('not armed'); + }); + it('rejects non-singleton instances before touching maintenance state', async () => { vi.useFakeTimers(); vi.setSystemTime(NOW); diff --git a/packages/flowsafe/src/host-kit/state-egress.test.ts b/packages/flowsafe/src/host-kit/state-egress.test.ts index 572b0ad3..2a65e9ca 100644 --- a/packages/flowsafe/src/host-kit/state-egress.test.ts +++ b/packages/flowsafe/src/host-kit/state-egress.test.ts @@ -17,6 +17,9 @@ import { const OUTBOUND_CREDENTIAL = 'outbound-state-credential-0000000001'; const DEPLOYMENT_IDENTITY = 'deployment-identity-credential-00001'; +const PRIVATE_VERSION_HEADER = 'x-anchorage-state-activation-version'; +const PRIVATE_KIND_HEADER = 'x-anchorage-state-activation-kind'; +const PRIVATE_KEY_HEADER = 'x-anchorage-state-dispatch-key'; function stateEnv( fetch = vi.fn(async (_request: Request) => new Response('proxied')), @@ -35,6 +38,87 @@ function stateEnv( } describe('trusted state egress adapter', () => { + it('uses a private dispatch key without a host route and removes the spoofed host header', async () => { + const bindingFetch = vi.fn( + async (_request: Request) => new Response('private'), + ); + const { OUTBOUND_ROUTE_HOSTNAME: _route, ...legacy } = + stateEnv(bindingFetch); + const env = { + ...legacy, + OUTBOUND_ACTIVATION_VERSION: '2', + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch', + OUTBOUND_DISPATCH_KEY: 'acme:production', + } as unknown as StateEgressEnv; + + await expect(validateStateEgressEnv(env)).resolves.toMatchObject({ + activationVersion: '2', + activationKind: 'internal-dispatch', + dispatchKey: 'acme:production', + }); + const response = await createStateEgressFetch(env)( + new Request('https://api.vendor.test/write', { + headers: { + [STATE_EGRESS_ROUTE_HOSTNAME_HEADER]: 'victim.example.test', + [PRIVATE_KEY_HEADER]: 'victim:production', + }, + }), + ); + expect(await response.text()).toBe('private'); + const forwarded = bindingFetch.mock.calls[0]?.[0]; + expect( + forwarded?.headers.get(STATE_EGRESS_ROUTE_HOSTNAME_HEADER), + ).toBeNull(); + expect(forwarded?.headers.get(PRIVATE_VERSION_HEADER)).toBe('2'); + expect(forwarded?.headers.get(PRIVATE_KIND_HEADER)).toBe( + 'internal-dispatch', + ); + expect(forwarded?.headers.get(PRIVATE_KEY_HEADER)).toBe('acme:production'); + }); + + it('rejects mixed host and private activation bindings', async () => { + const env = { + ...stateEnv(), + OUTBOUND_ACTIVATION_VERSION: '2', + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch', + OUTBOUND_DISPATCH_KEY: 'acme:production', + } as StateEgressEnv; + await expect(validateStateEgressEnv(env)).rejects.toThrow(); + }); + + it.each([ + ['unknown version', { OUTBOUND_ACTIVATION_VERSION: '3' }], + ['unknown kind', { OUTBOUND_ACTIVATION_KIND: 'host' }], + ['foreign key', { OUTBOUND_DISPATCH_KEY: 'other:production' }], + ['host value', { OUTBOUND_ROUTE_HOSTNAME: 'acme.example.test' }], + ])('rejects a private activation with %s', async (_case, patch) => { + const { OUTBOUND_ROUTE_HOSTNAME: _route, ...legacy } = stateEnv(); + const env = { + ...legacy, + OUTBOUND_ACTIVATION_VERSION: '2', + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch', + OUTBOUND_DISPATCH_KEY: 'acme:production', + ...patch, + } as unknown as StateEgressEnv; + await expect(validateStateEgressEnv(env)).rejects.toThrow(); + }); + + it('strips private authority headers from a legacy host request', async () => { + const bindingFetch = vi.fn(async (_request: Request) => new Response()); + await createStateEgressFetch(stateEnv(bindingFetch))( + new Request('https://api.vendor.test/write', { + headers: { + [PRIVATE_VERSION_HEADER]: '2', + [PRIVATE_KIND_HEADER]: 'internal-dispatch', + [PRIVATE_KEY_HEADER]: 'victim:production', + }, + }), + ); + const forwarded = bindingFetch.mock.calls[0]?.[0]; + expect(forwarded?.headers.get(PRIVATE_VERSION_HEADER)).toBeNull(); + expect(forwarded?.headers.get(PRIVATE_KIND_HEADER)).toBeNull(); + expect(forwarded?.headers.get(PRIVATE_KEY_HEADER)).toBeNull(); + }); it('matches a presented credential against its canonical SHA-256 digest', async () => { const digest = await crypto.subtle.digest( 'SHA-256', diff --git a/packages/flowsafe/src/host-kit/state-egress.ts b/packages/flowsafe/src/host-kit/state-egress.ts index 02d73e6d..45907be0 100644 --- a/packages/flowsafe/src/host-kit/state-egress.ts +++ b/packages/flowsafe/src/host-kit/state-egress.ts @@ -16,6 +16,12 @@ export const STATE_EGRESS_RESOURCE_GROUP_HEADER = export const STATE_EGRESS_SCRIPT_HEADER = 'x-anchorage-state-script' as const; export const STATE_EGRESS_ROUTE_HOSTNAME_HEADER = 'x-anchorage-state-route-hostname' as const; +export const STATE_EGRESS_ACTIVATION_VERSION_HEADER = + 'x-anchorage-state-activation-version' as const; +export const STATE_EGRESS_ACTIVATION_KIND_HEADER = + 'x-anchorage-state-activation-kind' as const; +export const STATE_EGRESS_DISPATCH_KEY_HEADER = + 'x-anchorage-state-dispatch-key' as const; export const STATE_EGRESS_POLICY_ID_HEADER = 'x-anchorage-state-policy-id' as const; @@ -29,36 +35,74 @@ export const STATE_EGRESS_HEADERS = { policyId: STATE_EGRESS_POLICY_ID_HEADER, } as const; -export const STATE_EGRESS_RESERVED_HEADERS = - Object.values(STATE_EGRESS_HEADERS); +export const STATE_EGRESS_PRIVATE_HEADERS = { + activationVersion: STATE_EGRESS_ACTIVATION_VERSION_HEADER, + activationKind: STATE_EGRESS_ACTIVATION_KIND_HEADER, + dispatchKey: STATE_EGRESS_DISPATCH_KEY_HEADER, +} as const; + +export const STATE_EGRESS_RESERVED_HEADERS = [ + ...Object.values(STATE_EGRESS_HEADERS), + ...Object.values(STATE_EGRESS_PRIVATE_HEADERS), +]; export interface StateEgressBinding { fetch(request: Request): Promise; } -export interface StateEgressContext { +export interface StateEgressCommonContext { readonly credential: string; readonly tenantTag: string; readonly environment: string; readonly resourceGroupId: string; readonly stateScriptName: string; - readonly routeHostname: string; readonly policyId: string; } -/** Exact trusted-state bindings consumed by createStateEgressFetch. */ -export interface StateEgressEnv { +export type StateEgressContext = StateEgressCommonContext & + ( + | Readonly<{ + routeHostname: string; + activationVersion?: never; + activationKind?: never; + dispatchKey?: never; + }> + | Readonly<{ + routeHostname?: never; + activationVersion: '2'; + activationKind: 'internal-dispatch'; + dispatchKey: string; + }> + ); + +export interface StateEgressCommonEnv { readonly OUTBOUND_PROXY: StateEgressBinding; readonly OUTBOUND_PROXY_CREDENTIAL: string; readonly OUTBOUND_TENANT_ID: string; readonly OUTBOUND_ENVIRONMENT: string; readonly OUTBOUND_RESOURCE_GROUP_ID: string; readonly OUTBOUND_STATE_SCRIPT_NAME: string; - readonly OUTBOUND_ROUTE_HOSTNAME: string; readonly OUTBOUND_POLICY_ID: string; readonly DEPLOYMENT_IDENTITY_SECRET: string; } +/** Exact trusted-state bindings consumed by createStateEgressFetch. */ +export type StateEgressEnv = StateEgressCommonEnv & + ( + | Readonly<{ + OUTBOUND_ROUTE_HOSTNAME: string; + OUTBOUND_ACTIVATION_VERSION?: never; + OUTBOUND_ACTIVATION_KIND?: never; + OUTBOUND_DISPATCH_KEY?: never; + }> + | Readonly<{ + OUTBOUND_ROUTE_HOSTNAME?: never; + OUTBOUND_ACTIVATION_VERSION: '2'; + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch'; + OUTBOUND_DISPATCH_KEY: string; + }> + ); + const CREDENTIAL_PATTERN = /^[\x21-\x7e]{32,256}$/; const RESOURCE_GROUP_PATTERN = /^[a-f0-9]{20}$/; const SCRIPT_NAME_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/; @@ -122,22 +166,48 @@ function stateEgressContext(env: StateEgressEnv): StateEgressContext { if (!SCRIPT_NAME_PATTERN.test(env.OUTBOUND_STATE_SCRIPT_NAME)) { throw new Error('trusted state outbound script is invalid'); } - const routeHostname = canonicalRouteHostname(env.OUTBOUND_ROUTE_HOSTNAME); - if (!routeHostname) { - throw new Error('trusted state outbound route hostname is invalid'); - } if (!POLICY_ID_PATTERN.test(env.OUTBOUND_POLICY_ID)) { throw new Error('trusted state outbound policy is invalid'); } - return { + const common = { credential: env.OUTBOUND_PROXY_CREDENTIAL, tenantTag: env.OUTBOUND_TENANT_ID, environment: env.OUTBOUND_ENVIRONMENT, resourceGroupId: env.OUTBOUND_RESOURCE_GROUP_ID, stateScriptName: env.OUTBOUND_STATE_SCRIPT_NAME, - routeHostname, policyId: env.OUTBOUND_POLICY_ID, }; + if (env.OUTBOUND_ACTIVATION_VERSION === undefined) { + if ( + env.OUTBOUND_ACTIVATION_KIND !== undefined || + env.OUTBOUND_DISPATCH_KEY !== undefined + ) { + throw new Error('trusted state outbound activation is mixed'); + } + const routeHostname = + typeof env.OUTBOUND_ROUTE_HOSTNAME === 'string' + ? canonicalRouteHostname(env.OUTBOUND_ROUTE_HOSTNAME) + : undefined; + if (!routeHostname) { + throw new Error('trusted state outbound route hostname is invalid'); + } + return { ...common, routeHostname }; + } + if ( + env.OUTBOUND_ACTIVATION_VERSION !== '2' || + env.OUTBOUND_ACTIVATION_KIND !== 'internal-dispatch' || + env.OUTBOUND_ROUTE_HOSTNAME !== undefined || + env.OUTBOUND_DISPATCH_KEY !== + `${env.OUTBOUND_TENANT_ID}:${env.OUTBOUND_ENVIRONMENT}` + ) { + throw new Error('trusted state outbound private activation is invalid'); + } + return { + ...common, + activationVersion: '2', + activationKind: 'internal-dispatch', + dispatchKey: env.OUTBOUND_DISPATCH_KEY, + }; } /** Validate the complete trusted-state egress binding group. */ @@ -171,12 +241,22 @@ export function createStateEgressFetch(env: StateEgressEnv): typeof fetch { } const request = new Request(input, init); const headers = new Headers(request.headers); + for (const name of STATE_EGRESS_RESERVED_HEADERS) headers.delete(name); headers.set(STATE_EGRESS_CREDENTIAL_HEADER, context.credential); headers.set(STATE_EGRESS_TENANT_HEADER, context.tenantTag); headers.set(STATE_EGRESS_ENVIRONMENT_HEADER, context.environment); headers.set(STATE_EGRESS_RESOURCE_GROUP_HEADER, context.resourceGroupId); headers.set(STATE_EGRESS_SCRIPT_HEADER, context.stateScriptName); - headers.set(STATE_EGRESS_ROUTE_HOSTNAME_HEADER, context.routeHostname); + if (context.activationVersion === '2') { + headers.set( + STATE_EGRESS_ACTIVATION_VERSION_HEADER, + context.activationVersion, + ); + headers.set(STATE_EGRESS_ACTIVATION_KIND_HEADER, context.activationKind); + headers.set(STATE_EGRESS_DISPATCH_KEY_HEADER, context.dispatchKey); + } else { + headers.set(STATE_EGRESS_ROUTE_HOSTNAME_HEADER, context.routeHostname); + } headers.set(STATE_EGRESS_POLICY_ID_HEADER, context.policyId); return service.fetch(new Request(request, { headers })); }; diff --git a/packages/flowsafe/src/signals/index.ts b/packages/flowsafe/src/signals/index.ts index 11d9e3ca..ea8bdafc 100644 --- a/packages/flowsafe/src/signals/index.ts +++ b/packages/flowsafe/src/signals/index.ts @@ -28,6 +28,7 @@ export { type NotificationDispatchTickOptions, type NotificationDispatchTickResult, } from './notification-dispatch.js'; +export type { NotificationOperationInput } from './notifications-d1.js'; export { D1NotificationsStorage } from './notifications-d1.js'; // The P6 ingestion trust boundary router (CI-M-004-004). export { diff --git a/packages/flowsafe/src/signals/notification-dispatch.ts b/packages/flowsafe/src/signals/notification-dispatch.ts index 13939c8a..672f20ef 100644 --- a/packages/flowsafe/src/signals/notification-dispatch.ts +++ b/packages/flowsafe/src/signals/notification-dispatch.ts @@ -18,71 +18,18 @@ import { positiveSafeInteger, } from '../numeric-config.js'; import { jsonOrNull } from './d1-shared.js'; +import type { + NotificationDeliveryFailure, + NotificationDeliveryObservation, + NotificationDeliveryStorage, +} from './notifications-d1.js'; -/** Scalar values preserve the attempted observation across asynchronous writes. */ -export type NotificationDeliveryObservation = Readonly<{ - id: string; - threadId: string; - source: string; - kind: string; - priority: NotificationRecord['priority']; - status: NotificationRecord['status']; - summary: string; - payload: string | null; - resourceId: string | null; - agentId: string | null; - sourceId: string | null; - dedupeKey: string | null; - coalesceKey: string | null; - coalescedCount: number; - attributes: string | null; - createdAt: string; - updatedAt: string; - deliverAt: string | null; - summaryAt: string | null; - deliveryReason: string | null; - deliveryAttempts: number; - lastDeliveryAttemptAt: string | null; - lastDeliveryError: string | null; - deliveredSignalId: string | null; - summarySignalId: string | null; - deliveredAt: string | null; - seenAt: string | null; - dismissedAt: string | null; - archivedAt: string | null; - discardedAt: string | null; - metadata: string | null; -}>; - -export type NotificationDeliveryFailure = - | Readonly<{ - type: 'retry'; - updatedAt: string; - deliveryAttempts: number; - lastDeliveryAttemptAt: string; - lastDeliveryError: string; - deliverAt?: string; - summaryAt?: string; - }> - | Readonly<{ - type: 'discard'; - updatedAt: string; - deliveryAttempts: number; - lastDeliveryAttemptAt: string; - lastDeliveryError: string; - }> - | Readonly<{ type: 'exhausted'; updatedAt: string }>; - -export type NotificationDeliveryUpdateResult = - | { applied: true; record: NotificationRecord } - | { applied: false }; - -export interface NotificationDeliveryStorage extends NotificationsStorage { - updateNotificationDeliveryIfUnchanged(input: { - expected: NotificationDeliveryObservation; - failure: NotificationDeliveryFailure; - }): Promise; -} +export type { + NotificationDeliveryFailure, + NotificationDeliveryObservation, + NotificationDeliveryStorage, + NotificationDeliveryUpdateResult, +} from './notifications-d1.js'; /** Maximum route-valid ids in one trusted thread-DO dispatch request. */ export const MAX_NOTIFICATION_DISPATCH_IDS = 100; diff --git a/packages/flowsafe/src/signals/notifications-d1.test.ts b/packages/flowsafe/src/signals/notifications-d1.test.ts index 7404f4ec..3f909926 100644 --- a/packages/flowsafe/src/signals/notifications-d1.test.ts +++ b/packages/flowsafe/src/signals/notifications-d1.test.ts @@ -38,6 +38,462 @@ function sharedStores(): [D1NotificationsStorage, D1NotificationsStorage] { ]; } +describe('host-owned notification operation receipt', () => { + it('admits only one receipt prefix in a D1 deployment', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, 'left_', 1); + const right = new D1NotificationsStorage(db, 'right_', 1); + await Promise.all([left.init(), right.init()]); + const input = { + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const outcomes = await Promise.allSettled([ + left.createNotificationOnce('operation-left', { + ...input, + id: 'notification-left', + }), + right.createNotificationOnce('operation-right', { + ...input, + id: 'notification-right', + }), + ]); + expect( + outcomes.filter((outcome) => outcome.status === 'fulfilled'), + ).toHaveLength(1); + expect( + outcomes.filter((outcome) => outcome.status === 'rejected'), + ).toHaveLength(1); + const rejected = outcomes.find((outcome) => outcome.status === 'rejected'); + expect(rejected).toMatchObject({ + reason: { + message: 'notification receipt scope differs from this deployment', + }, + }); + const loserPrefix = outcomes[0]?.status === 'rejected' ? 'left_' : 'right_'; + const loserOperation = + outcomes[0]?.status === 'rejected' ? 'operation-left' : 'operation-right'; + const loserNotification = + outcomes[0]?.status === 'rejected' + ? 'notification-left' + : 'notification-right'; + expect( + await db + .prepare( + `SELECT operation_key FROM ${loserPrefix}flowsafe_notification_operation_receipts WHERE operation_key = ?`, + ) + .bind(loserOperation) + .first(), + ).toBeNull(); + expect( + await db + .prepare( + `SELECT id FROM ${loserPrefix}mastra_notifications WHERE id = ?`, + ) + .bind(loserNotification) + .first(), + ).toBeNull(); + }); + + it('retains a prefix and limit across fresh storage instances', async () => { + const db = database(); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const first = new D1NotificationsStorage(db, 'agent_', 1); + const receipt = await first.createNotificationOnce('operation-1', input); + const fresh = new D1NotificationsStorage(db, 'agent_', 1); + expect(await fresh.createNotificationOnce('operation-1', input)).toEqual( + receipt, + ); + await expect( + new D1NotificationsStorage(db, 'agent_', 2).createNotificationOnce( + 'operation-2', + { ...input, id: 'notification-2' }, + ), + ).rejects.toThrow( + 'notification receipt scope differs from this deployment', + ); + expect( + await fresh.getNotification({ threadId: 'thread', id: 'notification-2' }), + ).toBeNull(); + }); + + it('refuses ambiguous nonempty legacy receipt histories', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, 'left_'); + const right = new D1NotificationsStorage(db, 'right_'); + await Promise.all([left.init(), right.init()]); + for (const prefix of ['left_', 'right_']) { + await db + .prepare(`INSERT INTO ${prefix}flowsafe_notification_operation_receipts + (operation_key, request_digest, thread_id, notification_id) + VALUES (?, ?, ?, ?)`) + .bind( + `legacy-${prefix}`, + 'a'.repeat(64), + 'thread', + `notification-${prefix}`, + ) + .run(); + } + await expect( + left.createNotificationOnce('operation-new', { + id: 'notification-new', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }), + ).rejects.toThrow('notification receipt history spans another prefix'); + const receipt = await db + .prepare( + 'SELECT operation_key FROM left_flowsafe_notification_operation_receipts WHERE operation_key = ?', + ) + .bind('operation-new') + .first(); + expect(receipt).toBeNull(); + }); + + it('adopts the single nonempty legacy prefix', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, 'left_'); + const right = new D1NotificationsStorage(db, 'right_'); + await Promise.all([left.init(), right.init()]); + await db + .prepare(`INSERT INTO left_flowsafe_notification_operation_receipts + (operation_key, request_digest, thread_id, notification_id) + VALUES ('legacy', ?, 'thread', 'notification-legacy')`) + .bind('a'.repeat(64)) + .run(); + const input = { + id: 'notification-new', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + await expect( + right.createNotificationOnce('operation-new', input), + ).rejects.toThrow('notification receipt history spans another prefix'); + await expect( + left.createNotificationOnce('operation-new', input), + ).resolves.toMatchObject({ + operationKey: 'operation-new', + notificationId: 'notification-new', + }); + }); + + it('refuses an unknown receipt scope version', async () => { + const db = database(); + await db + .prepare(`CREATE TABLE flowsafe_notification_receipt_scope ( + id INTEGER PRIMARY KEY, + version INTEGER NOT NULL, + table_prefix TEXT NOT NULL, + receipt_limit INTEGER NOT NULL + )`) + .run(); + await db + .prepare(`INSERT INTO flowsafe_notification_receipt_scope + (id, version, table_prefix, receipt_limit) VALUES (1, 2, '', 100000)`) + .run(); + const notifications = new D1NotificationsStorage(db); + await expect( + notifications.createNotificationOnce('operation-new', { + id: 'notification-new', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }), + ).rejects.toThrow( + 'notification receipt scope differs from this deployment', + ); + }); + + it('refuses a new operation at the retained receipt cap while replay still succeeds', async () => { + const db = database(); + const notifications = new D1NotificationsStorage(db, '', 1); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const first = await notifications.createNotificationOnce( + 'operation-1', + input, + ); + await expect( + notifications.createNotificationOnce('operation-2', { + ...input, + id: 'notification-2', + }), + ).rejects.toThrow('notification operation receipt cap reached'); + expect( + await notifications.createNotificationOnce('operation-1', input), + ).toEqual(first); + expect( + await notifications.getNotification({ + threadId: 'thread', + id: 'notification-2', + }), + ).toBeNull(); + }); + + it('admits at most one concurrent new operation at the last receipt slot', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, '', 1); + const right = new D1NotificationsStorage(db, '', 1); + const input = { + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const results = await Promise.allSettled([ + left.createNotificationOnce('operation-1', { + ...input, + id: 'notification-1', + }), + right.createNotificationOnce('operation-2', { + ...input, + id: 'notification-2', + }), + ]); + expect( + results.filter((result) => result.status === 'fulfilled'), + ).toHaveLength(1); + expect( + results.filter((result) => result.status === 'rejected'), + ).toHaveLength(1); + const receipts = await db + .prepare( + 'SELECT operation_key FROM flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string }>(); + expect(receipts.results).toHaveLength(1); + }); + + it('does not reset a delivered notification when a stale same-key retry reaches the cap', async () => { + const db = database(); + const first = new D1NotificationsStorage(db, '', 1); + let signalRead: () => void = () => undefined; + let releaseRead: () => void = () => undefined; + const readPaused = new Promise((resolve) => { + signalRead = resolve; + }); + const readReleased = new Promise((resolve) => { + releaseRead = resolve; + }); + let intercepted = false; + const retry = new D1NotificationsStorage( + interceptFirst(db, async (query, read) => { + const result = await read(); + if ( + !intercepted && + result === null && + query.includes( + 'FROM flowsafe_notification_operation_receipts WHERE operation_key = ?', + ) + ) { + intercepted = true; + signalRead(); + await readReleased; + } + return result; + }), + '', + 1, + ); + await Promise.all([first.init(), retry.init()]); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const pendingRetry = retry.createNotificationOnce('operation-1', input); + await readPaused; + const receipt = await first.createNotificationOnce('operation-1', input); + await first.updateNotification({ + threadId: input.threadId, + id: input.id, + status: 'delivered', + }); + releaseRead(); + expect(await pendingRetry).toEqual(receipt); + expect( + ( + await first.getNotification({ + threadId: input.threadId, + id: input.id, + }) + )?.status, + ).toBe('delivered'); + }); + + it('returns the first durable receipt after a lost acknowledgement without overwriting the inbox', async () => { + const db = database(); + const atomicBatch = db.batch?.bind(db); + if (!atomicBatch) throw new Error('D1 batch is unavailable'); + let loseAcknowledgement = false; + const first = new D1NotificationsStorage({ + prepare: db.prepare.bind(db), + async batch(statements) { + const results = await atomicBatch(statements); + if (loseAcknowledgement) { + loseAcknowledgement = false; + throw new Error('lost acknowledgement'); + } + return results; + }, + }); + const retry = new D1NotificationsStorage(db); + const input = { + id: 'remote-step-notification', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + payload: { value: 'private-receipt-canary' }, + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + await first.init(); + loseAcknowledgement = true; + const receipt = await first.createNotificationOnce( + 'remote-invocation', + input, + ); + expect(receipt).toMatchObject({ + operationKey: 'remote-invocation', + notificationId: input.id, + threadId: input.threadId, + }); + const retained = await db + .prepare('SELECT * FROM flowsafe_notification_operation_receipts') + .first>(); + expect(retained?.request_digest).toMatch(/^[a-f0-9]{64}$/); + expect(receipt.requestDigest).toBe(retained?.request_digest); + expect(JSON.stringify(retained)).not.toContain(input.summary); + expect(JSON.stringify(retained)).not.toContain('private-receipt-canary'); + const delivered = await retry.getNotification({ + threadId: input.threadId, + id: input.id, + }); + expect(delivered?.summary).toBe(input.summary); + expect( + await retry.createNotificationOnce('remote-invocation', input), + ).toEqual(receipt); + await expect( + retry.createNotificationOnce('remote-invocation', { + ...input, + summary: 'Different result', + }), + ).rejects.toThrow('different request'); + expect( + (await retry.listNotifications({ threadId: input.threadId })).length, + ).toBe(1); + }); + + it('commits one effect under concurrent retry and rejects inbox identity collisions', async () => { + const db = database(); + const left = new D1NotificationsStorage(db); + const right = new D1NotificationsStorage(db); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const [first, second] = await Promise.all([ + left.createNotificationOnce('operation-1', input), + right.createNotificationOnce('operation-1', input), + ]); + expect(first).toEqual(second); + expect((await left.listNotifications({ threadId: 'thread' })).length).toBe( + 1, + ); + await expect( + right.createNotificationOnce('operation-2', input), + ).rejects.toThrow(); + const receipts = await db + .prepare( + 'SELECT operation_key FROM flowsafe_notification_operation_receipts ORDER BY operation_key', + ) + .all<{ operation_key: string }>(); + expect(receipts.results).toEqual([{ operation_key: 'operation-1' }]); + await left.createNotification({ ...input, id: 'ordinary-notification' }); + await expect( + right.createNotificationOnce('operation-3', { + ...input, + id: 'ordinary-notification', + summary: 'Overwrite attempt', + }), + ).rejects.toThrow('not committed'); + expect( + ( + await left.getNotification({ + threadId: 'thread', + id: 'ordinary-notification', + }) + )?.summary, + ).toBe('Completed'); + }); + + it('rolls back a receipt when the inbox insert fails after reservation', async () => { + const db = database(); + const notifications = new D1NotificationsStorage(db); + await notifications.init(); + await db + .prepare(`CREATE TRIGGER fail_notification_effect + BEFORE INSERT ON mastra_notifications + WHEN NEW.id = 'blocked-notification' + BEGIN SELECT RAISE(ABORT, 'blocked inbox insert'); END`) + .run(); + await expect( + notifications.createNotificationOnce('blocked-operation', { + id: 'blocked-notification', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'This effect does not commit', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }), + ).rejects.toThrow('blocked inbox insert'); + const receipt = await db + .prepare(`SELECT operation_key FROM flowsafe_notification_operation_receipts + WHERE operation_key = 'blocked-operation'`) + .first(); + expect(receipt).toBeNull(); + expect( + await notifications.getNotification({ + threadId: 'thread', + id: 'blocked-notification', + }), + ).toBeNull(); + }); +}); + function interceptFirst( db: SignalDatabase, intercept: (query: string, read: () => Promise) => Promise, diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index 6eccd141..0fc1def5 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -29,6 +29,7 @@ import { notificationTimestampMillis, notificationTimestampSql, } from '../do-runner/notification-predicate.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; import { validateTablePrefix } from '../do-runner/table-prefix.js'; import { nonnegativeSafeInteger } from '../numeric-config.js'; import { @@ -40,12 +41,71 @@ import { type SignalDatabase, type SignalStatement, } from './d1-shared.js'; -import type { - NotificationDeliveryFailure, - NotificationDeliveryObservation, - NotificationDeliveryStorage, - NotificationDeliveryUpdateResult, -} from './notification-dispatch.js'; + +/** Scalar values preserve the attempted observation across asynchronous writes. */ +export type NotificationDeliveryObservation = Readonly<{ + id: string; + threadId: string; + source: string; + kind: string; + priority: NotificationRecord['priority']; + status: NotificationRecord['status']; + summary: string; + payload: string | null; + resourceId: string | null; + agentId: string | null; + sourceId: string | null; + dedupeKey: string | null; + coalesceKey: string | null; + coalescedCount: number; + attributes: string | null; + createdAt: string; + updatedAt: string; + deliverAt: string | null; + summaryAt: string | null; + deliveryReason: string | null; + deliveryAttempts: number; + lastDeliveryAttemptAt: string | null; + lastDeliveryError: string | null; + deliveredSignalId: string | null; + summarySignalId: string | null; + deliveredAt: string | null; + seenAt: string | null; + dismissedAt: string | null; + archivedAt: string | null; + discardedAt: string | null; + metadata: string | null; +}>; + +export type NotificationDeliveryFailure = + | Readonly<{ + type: 'retry'; + updatedAt: string; + deliveryAttempts: number; + lastDeliveryAttemptAt: string; + lastDeliveryError: string; + deliverAt?: string; + summaryAt?: string; + }> + | Readonly<{ + type: 'discard'; + updatedAt: string; + deliveryAttempts: number; + lastDeliveryAttemptAt: string; + lastDeliveryError: string; + }> + | Readonly<{ type: 'exhausted'; updatedAt: string }>; + +export type NotificationDeliveryUpdateResult = + | { applied: true; record: NotificationRecord } + | { applied: false }; + +export interface NotificationDeliveryStorage extends NotificationsStorage { + updateNotificationDeliveryIfUnchanged(input: { + expected: NotificationDeliveryObservation; + failure: NotificationDeliveryFailure; + }): Promise; +} /** The raw row shape `mastra_notifications` stores and `rowToRecord` reads. */ interface NotificationRow { @@ -332,6 +392,76 @@ function isDuplicateColumn(error: unknown): boolean { * a template literal is one the census can only match by restating it. */ export const NOTIFICATION_SEQUENCE_TABLE = 'flowsafe_notification_sequence'; +export const NOTIFICATION_OPERATION_PROJECTION_VERSION = + 'notification-record-values-v1'; +export const NOTIFICATION_OPERATION_RECEIPTS_TABLE = + 'flowsafe_notification_operation_receipts'; +export const NOTIFICATION_RECEIPT_SCOPE_TABLE = + 'flowsafe_notification_receipt_scope'; +export const DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT = 100_000; + +export type NotificationOperationInput = CreateNotificationInput & { + id: string; + createdAt: Date; +}; + +export async function projectNotificationOperation( + input: NotificationOperationInput, +): Promise<{ + version: typeof NOTIFICATION_OPERATION_PROJECTION_VERSION; + values: unknown[]; + requestJson: string; + requestDigest: string; +}> { + if ( + !isPathSafeId(input.id) || + !isPathSafeId(input.threadId) || + !(input.createdAt instanceof Date) || + !Number.isFinite(input.createdAt.getTime()) + ) + throw new TypeError('notification operation identity is malformed'); + const record: NotificationRecord = { + id: input.id, + threadId: input.threadId, + source: input.source, + kind: input.kind, + priority: input.priority ?? 'medium', + status: 'pending', + summary: input.summary, + payload: input.payload, + resourceId: input.resourceId, + agentId: input.agentId, + sourceId: input.sourceId, + dedupeKey: input.dedupeKey, + coalesceKey: input.coalesceKey, + coalescedCount: 1, + attributes: input.attributes, + createdAt: input.createdAt, + updatedAt: input.createdAt, + deliverAt: input.deliverAt, + summaryAt: input.summaryAt, + deliveryReason: input.deliveryReason, + deliveryAttempts: 0, + metadata: input.metadata, + }; + const values = recordValues(record); + validateNotificationDeliveryValues(values, true); + const requestJson = JSON.stringify(values); + const digestBytes = new Uint8Array( + await crypto.subtle.digest( + 'SHA-256', + new TextEncoder().encode(requestJson), + ), + ); + return { + version: NOTIFICATION_OPERATION_PROJECTION_VERSION, + values, + requestJson, + requestDigest: Array.from(digestBytes, (byte) => + byte.toString(16).padStart(2, '0'), + ).join(''), + }; +} export class D1NotificationsStorage extends NotificationsStorage @@ -340,17 +470,37 @@ export class D1NotificationsStorage readonly #db: SignalDatabase; readonly #table: string; readonly #sequenceTable: string; + readonly #operationReceiptTable: string; + readonly #operationReceiptLimit: number; + readonly #prefix: string; readonly #ordinalIndex: string; readonly #legacyReplaceTrigger: string; readonly #legacyInsertTrigger: string; #ready?: Promise; + #receiptScopeReady?: Promise; - constructor(db: SignalDatabase, tablePrefix = '') { + constructor( + db: SignalDatabase, + tablePrefix = '', + operationReceiptLimit = DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, + ) { super(); const prefix = validateTablePrefix(tablePrefix) ?? ''; this.#db = db; + this.#prefix = prefix; this.#table = `${prefix}mastra_notifications`; this.#sequenceTable = `${prefix}${NOTIFICATION_SEQUENCE_TABLE}`; + this.#operationReceiptTable = `${prefix}${NOTIFICATION_OPERATION_RECEIPTS_TABLE}`; + this.#operationReceiptLimit = nonnegativeSafeInteger( + operationReceiptLimit, + 'operationReceiptLimit', + ); + if ( + this.#operationReceiptLimit > DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT + ) + throw new RangeError( + 'operationReceiptLimit exceeds the deployment maximum', + ); this.#ordinalIndex = `idx_${this.#table}_insertion_ordinal`; this.#legacyReplaceTrigger = `trg_${this.#table}_preserve_insertion_ordinal`; this.#legacyInsertTrigger = `trg_${this.#table}_allocate_insertion_ordinal`; @@ -413,6 +563,25 @@ export class D1NotificationsStorage ) .run(); await this.#migrateOrdinalSchema(true); + await this.#db + .prepare( + `CREATE TABLE IF NOT EXISTS ${this.#operationReceiptTable} ( + operation_key TEXT PRIMARY KEY, + request_digest TEXT NOT NULL, + thread_id TEXT NOT NULL, + notification_id TEXT NOT NULL, + UNIQUE (thread_id, notification_id) + )`, + ) + .run(); + await this.#db + .prepare(`CREATE TABLE IF NOT EXISTS ${NOTIFICATION_RECEIPT_SCOPE_TABLE} ( + id INTEGER PRIMARY KEY CHECK (id = 1), + version INTEGER NOT NULL CHECK (version = 1), + table_prefix TEXT NOT NULL, + receipt_limit INTEGER NOT NULL CHECK (receipt_limit BETWEEN 0 AND ${DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT}) + )`) + .run(); } async #migrateOrdinalSchema(retryDuplicateColumn: boolean): Promise { @@ -547,6 +716,240 @@ export class D1NotificationsStorage await this.#ensureSchema(); } + #ensureReceiptScope(): Promise { + if (!this.#receiptScopeReady) { + this.#receiptScopeReady = this.#registerReceiptScope().catch( + (error: unknown) => { + this.#receiptScopeReady = undefined; + throw error; + }, + ); + } + return this.#receiptScopeReady; + } + + async #registerReceiptScope(): Promise { + const legacyTables = await this.#db + .prepare(`SELECT name FROM sqlite_master + WHERE type = 'table' AND substr(name, -length(?)) = ?`) + .bind( + NOTIFICATION_OPERATION_RECEIPTS_TABLE, + NOTIFICATION_OPERATION_RECEIPTS_TABLE, + ) + .all<{ name: string }>(); + for (const { name } of legacyTables.results) { + const prefix = name.slice( + 0, + -NOTIFICATION_OPERATION_RECEIPTS_TABLE.length, + ); + if (validateTablePrefix(prefix) !== prefix) + throw new Error('notification receipt history has an invalid prefix'); + const retained = await this.#db + .prepare(`SELECT 1 AS retained FROM "${name}" LIMIT 1`) + .first<{ retained: number }>(); + if (retained && prefix !== this.#prefix) + throw new Error('notification receipt history spans another prefix'); + } + const columns = await this.#db + .prepare(`PRAGMA table_info(${NOTIFICATION_RECEIPT_SCOPE_TABLE})`) + .all<{ name: string; type: string; notnull: number; pk: number }>(); + if ( + JSON.stringify( + columns.results.map(({ name, type, notnull, pk }) => [ + name, + type, + notnull, + pk, + ]), + ) !== + JSON.stringify([ + ['id', 'INTEGER', 0, 1], + ['version', 'INTEGER', 1, 0], + ['table_prefix', 'TEXT', 1, 0], + ['receipt_limit', 'INTEGER', 1, 0], + ]) + ) + throw new Error('notification receipt scope schema is unknown'); + await this.#db + .prepare(`INSERT INTO ${NOTIFICATION_RECEIPT_SCOPE_TABLE} + (id, version, table_prefix, receipt_limit) + VALUES (1, 1, ?, ?) ON CONFLICT(id) DO NOTHING`) + .bind(this.#prefix, this.#operationReceiptLimit) + .run(); + const scopes = await this.#db + .prepare(`SELECT id, version, table_prefix, receipt_limit + FROM ${NOTIFICATION_RECEIPT_SCOPE_TABLE}`) + .all<{ + id: number; + version: number; + table_prefix: string; + receipt_limit: number; + }>(); + if ( + scopes.results.length !== 1 || + scopes.results[0]?.id !== 1 || + scopes.results[0]?.version !== 1 || + scopes.results[0]?.table_prefix !== this.#prefix || + scopes.results[0]?.receipt_limit !== this.#operationReceiptLimit + ) + throw new Error( + 'notification receipt scope differs from this deployment', + ); + } + + async createNotificationOnce( + operationKey: string, + input: NotificationOperationInput, + ): Promise<{ + operationKey: string; + notificationId: string; + threadId: string; + requestDigest: string; + }> { + if ( + !isPathSafeId(operationKey) || + !isPathSafeId(input.id) || + !isPathSafeId(input.threadId) || + !(input.createdAt instanceof Date) || + !Number.isFinite(input.createdAt.getTime()) + ) { + throw new TypeError('notification operation identity is malformed'); + } + await this.#ensureSchema(); + await this.#ensureReceiptScope(); + const batch = this.#db.batch?.bind(this.#db); + if (!batch) + throw new Error('notification operation requires database.batch()'); + const { values, requestDigest } = await projectNotificationOperation(input); + const receipt = { + operationKey, + notificationId: input.id, + threadId: input.threadId, + requestDigest, + }; + const readReceipt = async () => + this.#db + .prepare( + `SELECT request_digest, thread_id, notification_id + FROM ${this.#operationReceiptTable} WHERE operation_key = ?`, + ) + .bind(operationKey) + .first<{ + request_digest: string; + thread_id: string; + notification_id: string; + }>(); + const existing = await readReceipt(); + if (existing) { + if (existing.request_digest !== requestDigest) + throw new Error('notification operation has a different request'); + return { + operationKey, + notificationId: existing.notification_id, + threadId: existing.thread_id, + requestDigest: existing.request_digest, + }; + } + try { + // The inbox's legacy insert trigger updates existing IDs, so the receipt + // reserves both the operation key and inbox identity before insertion. + await batch([ + this.#db + .prepare( + `INSERT INTO ${this.#operationReceiptTable} + (operation_key, request_digest, thread_id, notification_id) + SELECT ?, ?, ?, ? + WHERE NOT EXISTS ( + SELECT 1 FROM ${this.#table} + WHERE thread_id = ? AND id = ? + ) AND ( + SELECT COUNT(*) FROM ${this.#operationReceiptTable} + ) < ?`, + ) + .bind( + operationKey, + requestDigest, + input.threadId, + input.id, + input.threadId, + input.id, + this.#operationReceiptLimit, + ), + this.#db + .prepare( + `INSERT INTO ${this.#table} (${NOTIFICATION_COLUMNS.join(', ')}) + SELECT ${NOTIFICATION_COLUMNS.map(() => '?').join(', ')} + WHERE changes() = 1 AND EXISTS ( + SELECT 1 FROM ${this.#operationReceiptTable} + WHERE operation_key = ? AND request_digest = ? + )`, + ) + .bind(...values, operationKey, requestDigest), + ]); + const committed = await readReceipt(); + if (!committed) { + const count = await this.#db + .prepare( + `SELECT COUNT(*) AS count FROM ${this.#operationReceiptTable}`, + ) + .first<{ count: number }>(); + if (count && count.count >= this.#operationReceiptLimit) + throw new Error('notification operation receipt cap reached'); + throw new Error('notification operation was not committed'); + } + return receipt; + } catch (error) { + const committed = await readReceipt(); + if (!committed) throw error; + if (committed.request_digest !== requestDigest) + throw new Error('notification operation has a different request'); + return { + operationKey, + notificationId: committed.notification_id, + threadId: committed.thread_id, + requestDigest: committed.request_digest, + }; + } + } + + async readNotificationOperationReceipt(operationKey: string): Promise< + | { + operationKey: string; + notificationId: string; + threadId: string; + requestDigest: string; + } + | undefined + > { + if (!isPathSafeId(operationKey)) + throw new TypeError('notification operation identity is malformed'); + await this.#ensureSchema(); + const row = await this.#db + .prepare( + `SELECT request_digest, thread_id, notification_id + FROM ${this.#operationReceiptTable} WHERE operation_key = ?`, + ) + .bind(operationKey) + .first<{ + request_digest: string; + thread_id: string; + notification_id: string; + }>(); + if (!row) return undefined; + if ( + !/^[a-f0-9]{64}$/.test(row.request_digest) || + !isPathSafeId(row.thread_id) || + !isPathSafeId(row.notification_id) + ) + throw new Error('notification operation receipt is unreadable'); + return { + operationKey, + notificationId: row.notification_id, + threadId: row.thread_id, + requestDigest: row.request_digest, + }; + } + async createNotification( input: CreateNotificationInput, ): Promise { diff --git a/packages/flowsafe/src/signals/storage.ts b/packages/flowsafe/src/signals/storage.ts index f3ca2439..480c3d26 100644 --- a/packages/flowsafe/src/signals/storage.ts +++ b/packages/flowsafe/src/signals/storage.ts @@ -12,7 +12,10 @@ import type { MastraStorageDomains } from '@mastra/core/storage'; import type { D1DatabaseBinding } from '../do-runner/index.js'; import { validateTablePrefix } from '../do-runner/table-prefix.js'; import type { SignalDatabase } from './d1-shared.js'; -import { D1NotificationsStorage } from './notifications-d1.js'; +import { + D1NotificationsStorage, + DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, +} from './notifications-d1.js'; import { D1ThreadStateStorage } from './thread-state-d1.js'; /** @@ -25,13 +28,18 @@ import { D1ThreadStateStorage } from './thread-state-d1.js'; export function createSignalStorageDomains( binding: D1DatabaseBinding, tablePrefix = '', + notificationOperationReceiptLimit = DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, ): MastraStorageDomains { const prefix = validateTablePrefix(tablePrefix) ?? ''; // The structural SignalDatabase subset (prepare→bind/first/all/run) is exactly // what a real D1Database exposes; the cast bridges the workers-types-free seam. const db = binding as unknown as SignalDatabase; return { - notifications: new D1NotificationsStorage(db, prefix), + notifications: new D1NotificationsStorage( + db, + prefix, + notificationOperationReceiptLimit, + ), threadState: new D1ThreadStateStorage(db, prefix), }; } diff --git a/packages/flowsafe/test-support/cloudflare-test-worker.ts b/packages/flowsafe/test-support/cloudflare-test-worker.ts index bc46ee9a..ce97d6ca 100644 --- a/packages/flowsafe/test-support/cloudflare-test-worker.ts +++ b/packages/flowsafe/test-support/cloudflare-test-worker.ts @@ -1,16 +1,184 @@ // SPDX-License-Identifier: Apache-2.0 +import type { + SignalDatabase, + SignalStatement, +} from '../src/signals/d1-shared.js'; +import { D1NotificationsStorage } from '../src/signals/notifications-d1.js'; + +async function waitForReceiptReaders( + db: D1Database, + hub: DurableObjectNamespace, + barrierId: string, + participant: string, +): Promise { + await db + .prepare( + 'INSERT INTO notification_test_barrier (barrier_id, participant) VALUES (?, ?)', + ) + .bind(barrierId, participant) + .run(); + const response = await hub.get(hub.idFromName(barrierId)).fetch( + new Request('http://localhost/barrier', { + method: 'POST', + body: JSON.stringify({ barrierId }), + }), + ); + if (!response.ok) throw new Error('notification receipt barrier timed out'); +} + +function receiptBarrierDatabase( + db: D1Database, + hub: DurableObjectNamespace, + barrierId: string, + participant: string, + phase: 'receipt-read' | 'scope-census', +): SignalDatabase { + let barrierEntered = false; + function intercept(statement: SignalStatement): SignalStatement { + return { + bind(...values) { + return intercept(statement.bind(...values)); + }, + async first() { + const row = await statement.first(); + if (phase === 'receipt-read' && row === null && !barrierEntered) { + barrierEntered = true; + await waitForReceiptReaders(db, hub, barrierId, participant); + } + return row; + }, + async all() { + const rows = await statement.all(); + if (phase === 'scope-census' && !barrierEntered) { + barrierEntered = true; + await waitForReceiptReaders(db, hub, barrierId, participant); + } + return rows; + }, + run() { + return statement.run(); + }, + }; + } + return { + prepare(query) { + const statement = db.prepare(query); + return ( + phase === 'scope-census' + ? query.includes('SELECT name FROM sqlite_master') + : query.includes('SELECT request_digest, thread_id, notification_id') + ) + ? intercept(statement) + : statement; + }, + batch: db.batch.bind(db) as SignalDatabase['batch'], + }; +} + class TestDurableObject { - fetch(): Response { + fetch(_request: Request): Response | Promise { return new Response(null, { status: 204 }); } } export class DemoRunner extends TestDurableObject {} -export class DemoHub extends TestDurableObject {} +export class DemoHub extends TestDurableObject { + #waiting?: { + resolve: (response: Response) => void; + timeout: ReturnType; + }; + + constructor( + _state: DurableObjectState, + private readonly bindings: { DB: D1Database }, + ) { + super(); + } + + async fetch(request: Request): Promise { + const { barrierId } = (await request.json()) as { barrierId: string }; + const ready = await this.bindings.DB.prepare( + 'SELECT COUNT(*) AS count FROM notification_test_barrier WHERE barrier_id = ?', + ) + .bind(barrierId) + .first<{ count: number }>(); + if (this.#waiting) { + const waiting = this.#waiting; + this.#waiting = undefined; + clearTimeout(waiting.timeout); + const status = ready?.count === 2 ? 200 : 409; + waiting.resolve(Response.json({ ready: ready?.count }, { status })); + return Response.json({ ready: ready?.count }, { status }); + } + return new Promise((resolve) => { + const timeout = setTimeout(() => { + this.#waiting = undefined; + resolve(new Response(null, { status: 504 })); + }, 10_000); + this.#waiting = { resolve, timeout }; + }); + } +} export class DemoThread extends TestDurableObject {} export class DemoBackgroundTasks extends TestDurableObject {} -export class DemoSignalProviderHost extends TestDurableObject {} +export class DemoSignalProviderHost extends TestDurableObject { + constructor( + _state: DurableObjectState, + private readonly bindings: { + DB: D1Database; + HUB: DurableObjectNamespace; + }, + ) { + super(); + } + + async fetch(request: Request): Promise { + const input = (await request.json()) as { + operationKey: string; + notificationId: string; + receiptLimit: number; + tablePrefix?: string; + barrierId?: string; + participant?: string; + barrierPhase?: 'receipt-read' | 'scope-census'; + }; + const database = + input.barrierId && input.participant + ? receiptBarrierDatabase( + this.bindings.DB, + this.bindings.HUB, + input.barrierId, + input.participant, + input.barrierPhase ?? 'receipt-read', + ) + : this.bindings.DB; + const notifications = new D1NotificationsStorage( + database, + input.tablePrefix ?? '', + input.receiptLimit, + ); + try { + const receipt = await notifications.createNotificationOnce( + input.operationKey, + { + id: input.notificationId, + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + return Response.json(receipt); + } catch (error) { + return Response.json( + { error: error instanceof Error ? error.message : String(error) }, + { status: 409 }, + ); + } + } +} export default { fetch(): Response { diff --git a/packages/showcase/CHANGELOG.md b/packages/showcase/CHANGELOG.md index 74610893..dbac3fb1 100644 --- a/packages/showcase/CHANGELOG.md +++ b/packages/showcase/CHANGELOG.md @@ -1,5 +1,12 @@ # showcase +## 0.0.33 + +### Patch Changes + +- Updated dependencies [ee4003a] + - @proofoftech/flowsafe@0.29.0 + ## 0.0.32 ### Patch Changes diff --git a/packages/showcase/package.json b/packages/showcase/package.json index b53cc076..9902f688 100644 --- a/packages/showcase/package.json +++ b/packages/showcase/package.json @@ -1,6 +1,6 @@ { "name": "showcase", - "version": "0.0.32", + "version": "0.0.33", "private": true, "description": "Anchorage showcase — six runnable workflows and seven guardrail scenarios behind one React frontend", "type": "module", diff --git a/scripts/docs-check.mjs b/scripts/docs-check.mjs index 22177488..7170bba0 100644 --- a/scripts/docs-check.mjs +++ b/scripts/docs-check.mjs @@ -861,6 +861,7 @@ function checkOrphanedGuides(root, markdownFiles) { const relativePath = toPosix(relative(root, file)); return ( relativePath.startsWith('docs/') && + !relativePath.startsWith('docs/plans/') && basename(file).toLowerCase() !== 'claude.md' && !reachable.has(file) ); diff --git a/scripts/docs-check.test.mjs b/scripts/docs-check.test.mjs index a7968832..25a253a2 100644 --- a/scripts/docs-check.test.mjs +++ b/scripts/docs-check.test.mjs @@ -389,6 +389,31 @@ test('reachability follows absolute repository links into the docs tree', () => ); }); +test('design plans do not require a public documentation index link', () => { + const root = fixture({ + 'docs/README.md': '# Documentation\n', + 'docs/plans/internal-recovery.md': '# Recovery design plan\n', + 'docs/unlinked.md': '# Unlinked guide\n', + }); + + const result = checkRepository({ + root, + markdownFiles: markdownFiles(root, [ + 'docs/README.md', + 'docs/plans/internal-recovery.md', + 'docs/unlinked.md', + ]), + packageChecks: false, + }); + + assert.deepEqual( + result.errors.map((error) => `${error.file}: ${error.message}`), + [ + 'docs/unlinked.md: guide is not reachable from a public documentation index', + ], + ); +}); + test('reachability skips ignored schemes, off-repository URLs, and missing blob targets', () => { const root = fixture({ 'docs/README.md': `# Documentation