From 617a6a2760ab6c2ebe623ba39e0208fbdefc8e7d Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 11:14:23 +0400 Subject: [PATCH 01/73] Plan durable FlowSafe remote-step broker --- docs/proposals/README.md | 1 + docs/proposals/flowsafe-remote-step-broker.md | 189 ++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 docs/proposals/flowsafe-remote-step-broker.md diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 59f4d2be..7d422553 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -4,6 +4,7 @@ Documents in this directory are design records. Read each document's status befo - [Breakwater improvement roadmap](breakwater-improvement-roadmap.md): mixed shipped and planned work - [Durable result gate](durable-result-gate.md): proposal; not implemented +- [Remote step broker](flowsafe-remote-step-broker.md): implementation plan; not implemented - [Mature package adoption plan](mature-package-adoption-plan.md): implemented and verified - [Model gateway policy](model-gateway-policy.md): partially implemented; provider, budget, model-egress, structured-output, and scoring gates remain planned - [Single-tenant packages with physical tenant isolation](single-tenant-physical-isolation.md): implemented; credentialed Workers for Platforms conformance remains an operator-run release gate diff --git a/docs/proposals/flowsafe-remote-step-broker.md b/docs/proposals/flowsafe-remote-step-broker.md new file mode 100644 index 00000000..accdd242 --- /dev/null +++ b/docs/proposals/flowsafe-remote-step-broker.md @@ -0,0 +1,189 @@ +# Proposal: resume a durable agent after a remote step + +> Status: implementation plan, not shipped. Content type: design proposal. Audience: FlowSafe maintainers and hosts that execute a remote agent tool. + +This plan adds a generic durable handoff for a tool that waits for an external result. FlowSafe records the suspended run and an invocation, accepts one bound result, then resumes through its existing runtime. The host supplies its own authentication, dispatch transport, and policy; this proposal adds no customer, Workers for Platforms, signing, or capability policy to Anchorage. + +## Baseline and outcome + +Baseline: `plan/flowsafe-remote-step-broker` from `origin/dev` at `7d2ad352b0e4422811faf4ef4240a9172b360cce`, read 2026-10-11. Reconfirm the anchors below before editing. Use Node and pnpm versions from the root manifest. Work in a linked worktree with the shared pnpm store. The production Worker uses D1 and Durable Objects; in-memory Vitest results alone cannot prove restart behavior. The existing release is `@proofoftech/flowsafe@0.27.0` with `@mastra/core@1.73.0` as an exact peer. + +Done means a remote-step tool can suspend, persist a dispatchable invocation, survive Durable Object eviction, accept a result bound to that exact suspension, and resume through `RunnerRuntime`. Duplicate dispatch and duplicate result delivery converge on the same invocation. A changed suspension or unrelated run cannot resume. Migration cutover waits for remote dispatch claims to finish or be confirmed stopped; changing the FlowSafe fence alone cannot stop an external callback already in flight. “Once” applies to accepted result and resume admission; an external side effect still needs the host's stable invocation idempotency key. + +The first implementation checkpoint must prove that core 1.73.0 persists a tool's `workflow.suspend()` shape and resumes it with `workflow.resumeData` after a fresh host instance. If that probe fails, stop before publishing a broker API. A workflow-only suspension would not satisfy this agent-tool objective. + +## Ownership and package choice + +FlowSafe owns the durable run, suspension and resume contract. The deployment host owns authenticated result intake and the external executor. Breakwater continues to own connector authorization; Fleet Control owns deployment and migration; a customer platform owns artifact validation and grants. The generic broker never treats a caller's `runId`, `principal`, tool name, or result as authority by itself. + +Package-first check, 2026-10-11: + +| Candidate | Fit and decision | +| --- | --- | +| Existing FlowSafe `AgentThreadTopology`, `RunnerRuntime`, D1 storage, and execution fence | Reuse. They already provide authenticated internal Durable Object routing, exact suspension comparison, storage, and restart-aware resume. The broker needs a small state table and host adapter at this boundary. | +| Declared peer `@mastra/core@1.73.0` | Use its durable tool suspension, subject to the pinned-version probe. Do not call its direct resume API; FlowSafe's wrapper rebuilds the registry and enforces runtime admission. [Mastra durable agent guide](https://mastra.ai/docs/harness/durable-agents) and [Mastra tool suspension source guide](https://github.com/mastra-ai/mastra/blob/main/packages/core/src/tools/hitl.md) describe the general behavior, not the exact pinned implementation. | +| Cloudflare Durable Object alarms | Use the existing host's alarm and D1 patterns for retry and recovery. Alarms are at least once, so the journal and executor must tolerate repetition. [Cloudflare alarm reference](https://developers.cloudflare.com/durable-objects/api/alarms/). | +| Separate queue/workflow broker package | Reject for this seam: it cannot compare the run's authoritative suspension or execution fence without routing back through FlowSafe. It adds a second owner for state that must be correlated with the existing D1 snapshot. No new package is justified. | + +No dependency or package-manager change is proposed. An adapter with a real job remains custom: it validates the broker envelope and joins D1 state to the authoritative FlowSafe run. It is narrower than another broker runtime. The source links describe current upstream behavior; repeat the pinned-core probe because current upstream code may differ from 1.73.0. License and install exposure stay at the declared FlowSafe dependency set. The candidate has no added native component or lifecycle script; the broker adds one D1 table, two bounded recovery scans, and migration/cutover obligations. Search security advisories and the lockfile again if implementation later changes dependencies. + +## Contract and state transitions + +Expose a host-only `RemoteStepBroker` interface from the FlowSafe agent-host subpath. The host supplies a trusted dispatch callback and a trusted result-intake adapter; FlowSafe never mounts a public result route. The unique suspension key is `{deploymentIdentity, tablePrefix, workflowId, agentId, threadId, resourceId, runId, stepPath, suspendedAt, resumeCount}`. These fields come from a storage-backed run read and verified host binding, not from the executor. `invocationId` is a separate opaque identifier generated once for the inserted key and supplied to the executor as its idempotency key. The database's unique constraint excludes `invocationId`; a concurrent insert conflict reads and returns the winning row. The deployment/table fields prevent a reused run id in another generation from claiming an earlier invocation. + +Persist states `pending`, `dispatched`, `result-recorded`, `resuming`, and `settled`, with a terminal `failed` record for a permanent refusal. A D1 uniqueness constraint on the suspension key admits one invocation. A `resuming` row carries a lease owner and expiry, not permanent ownership. The result record stores a bounded, JSON-safe value or a bounded external result reference, a digest, and the exact invocation identity; it does not store arbitrary headers or credentials. Its configured retention must outlive the run's retry window and follow the deployment's purge policy. Limit size and shape at the host ingress and again in FlowSafe before D1 writes. Set the byte limit from existing D1/run bounds after the tool probe; reject oversized input rather than truncate it. + +1. The tool suspends with a reserved, versioned remote-step marker and a static host-owned step key. The host uses `RunnerRuntime.authoritativeStatus()` or the thread host's storage-backed equivalent and creates the invocation only after the suspension is durable. A periodic, bounded scan of the owning deployment's `mastra_workflow_snapshot` rows selects durable agent runs suspended on that marker, then re-reads each through the authoritative host before inserting any missing row. The scan has a persistent cursor and runs even with no broker rows or armed broker alarm. An unreadable row stops that candidate without dispatch and remains visible for retry. Do not use `RunnerRuntime.status()`, which can fall back to memory, for a dispatch decision. +2. A D1 insert or read returns the same invocation for a repeated observation of the same key. The broker transactionally claims a short-lived dispatch permit against the current fence state and mutation epoch before calling the host dispatch adapter. The adapter receives the stable `invocationId` and permit generation. The executor calls the trusted host gate before its effect; that gate atomically consumes the permit once under the same deployment epoch and records its acceptance. The fence-row read, epoch comparison, and permit transition must share one D1 transaction with the fence mutation protocol; a read followed by a separate permit write is insufficient. A repeated gate call returns an `already-consumed` receipt without authorizing another effect. After a lost acknowledgement or expired permit, the broker may issue a new generation for the same invocation only after the prior claim is closed or its executor has confirmed quiescence; a new permit cannot reuse a prior epoch. The executor must deduplicate the effect by `invocationId` across permit generations. If it cannot, the host refuses remote-step dispatch. +3. Trusted result intake checks the invocation id and the complete suspension identity against the journal and current authoritative run. It inserts the first result by compare-and-swap. Identical redelivery returns the stored outcome. Different bytes or digest for the same id refuse with conflict. A result cannot select another step. +4. A recovery duty claims `result-recorded` or an expired `resuming` lease, re-reads the execution fence and `authoritativeStatus()`, and sends an internal resume with the exact `expectedSuspension`. It restores the original stored execution principal and current permissions, never the result sender's identity. It marks `settled` only after the authoritative run state proves the resume advanced or ended. If the resume response is lost, an expired lease triggers an authoritative status read: unchanged suspension retries the same recorded result; advanced or terminal state settles without replay; unreadable state retains the row for retry. A new suspension gets a new fingerprint and invocation. +5. `migration-locked` and an unreadable fence stop new permit claims and resume admission. `draining` permits completion of an existing run under FlowSafe's current fence contract. `proof-only` admits only its nominated run. Entering migration lock revokes unconsumed permits by epoch, then waits for consumed permits and in-flight executor callbacks to report settled or confirmed stopped before the migration inventory can be empty. If the executor cannot report quiescence, cutover blocks; it does not infer safety from permit expiry. Include pending and active broker rows in FlowSafe's read-only drain inventory and table census. A terminal run, canceled run, expired result, or changed fingerprint refuses the old result without starting work. + +The broker deadline is the earlier of a stored run deadline, when one exists, and a required host-configured remote-step deadline. The dispatch permit's shorter expiry never extends that result deadline. At deadline or cancellation, stop issuing permits and resume retries, mark the journal terminal after an authoritative run read, and request best-effort executor cancellation. A cancel request does not prove an external effect stopped. Keep its active claim in drain inventory until the executor confirms completion or stop; otherwise cutover remains blocked. Retain the terminal journal long enough to refuse late result delivery and permit replay. + +Do not claim atomicity between Mastra's snapshot write and the broker's D1 row. The periodic snapshot scan closes that gap by creating an invocation from a durable suspension; it never dispatches on a transient in-memory event. Do not claim exactly-once external execution without an idempotent executor. A lost resume acknowledgement leaves `result-recorded` or `resuming` for a later authoritative status read. + +### Risky edit: resume authority + +Current `AgentThreadTopology.resume` accepts an `ApprovalRecord` and `ApprovalDecision`, then sends `defaultResumeData(record, decision)` with `expectedSuspensionFor(record)`. The target host reauthorizes the stored principal and calls `durable.resumeViaRuntime()`. A remote result is not an approval and must not fabricate either record. Add a distinct host-only result-resume method whose request carries the journaled invocation and result. It must reuse the target host's run ownership, catalog authorization, `withExecution`, and `resumeViaRuntime` path, with mandatory `expectedSuspension`. Do not broaden the existing public agent router or optionalize the fingerprint for this method. + +Before (`thread-topology.ts`, `AgentThreadTopology.resume`): + +```typescript +resume( + context: ActorContext, + record: ApprovalRecord, + decision: ApprovalDecision, +): Promise; +``` + +After, as a separate interface rather than changing approval semantics: + +```typescript +resumeRemoteStep( + context: ActorContext, + invocation: StoredRemoteStepInvocation, + result: RecordedRemoteStepResult, +): Promise; +``` + +The implementation must not accept an arbitrary caller-constructed `StoredRemoteStepInvocation`; the topology loads or validates its D1 row by id before sending. The thread host independently reads the same row and authoritative snapshot before execution. If adding a separate entry path to `AGENT_ENTRY_PATHS`, treat it as a continuation like `approval.resume`: a principal kind that could start the run may resume unless it has been removed from the catalog. Never classify the result sender as the run principal. + +## Implementation checkpoints + +### 1. Pinned-core suspension probe + +Use a deterministic model and a tool with `suspendSchema`/`resumeSchema`. Assert that the tool's `workflow.suspend()` reaches `RunSummary` with a distinguishable payload, survives a new FlowSafe host instance on persisted storage, and receives only the recorded result after `resumeViaRuntime`. Exercise refusal for malformed resume data and a second suspension. Add this to the existing real-loop agent tests; do not add a production export for the test. If the pinned core cannot safely resume that tool, document the failure and choose a supported workflow step boundary in a new reviewed plan. + +### 2. Journal and dispatch admission + +Add the D1 broker table through the existing FlowSafe provisioning/migration path. Use a versioned canonical envelope, uniqueness on the suspension key excluding `invocationId`, CAS result writes, bounded payloads, and a stable invocation id. Add the row to the inventory work category and schema census. Reuse the current D1 error and transaction patterns; no new parser or cryptographic primitive. Reconcile a suspension after a crash before insertion from the periodic snapshot scan, including when no broker row exists. Recover dispatch after acknowledgement loss through the permit protocol. Dispatch requires an authoritative status and a transactionally claimed permit at the current fence epoch. The executor contract requires idempotency by `invocationId` and an authenticated acknowledgement of quiescence. + +### 3. Result and runtime continuation + +Add a host-only topology method and target route. Validate deployment identity, internal request authentication, owner, thread/resource/run correlation, journal identity, and current suspension before `resumeViaRuntime`. Restore the original principal and current catalog policy. Bind the result as resume data to the exact tool step. Preserve the existing approval resume route. Record a durable outcome after reading authoritative status, including when the resume response is lost. + +### 4. Recovery and public documentation + +Connect two bounded scans to the existing maintenance alarm pattern: the periodic snapshot scan finds orphaned remote suspensions, and the broker-row scan recovers `pending`, `dispatched`, `result-recorded`, and expired `resuming` leases. Neither depends on an existing broker row to schedule its successor. Recover from isolate loss; do not retry a permanent refusal as execution. A timeout or cancellation requests best-effort executor cancel and keeps active claims visible until confirmed quiescent. Document the generic host contract, permit generations and epoch revocation, the requirement for external idempotency, fence/cutover behavior, retention, and the absence of a public result route. Add a FlowSafe minor changeset for the new API after behavior passes the real-worker proof. + +## Reuse map and exclusions + +| Existing code | Reuse | +| --- | --- | +| `packages/flowsafe/src/agent-host/thread-topology.ts`, `createAgentThreadTopology` | Authenticated internal routing and owner-checked status | +| `packages/flowsafe/src/agent-host/thread-host.ts`, `createAgentThreadHost` | Stored principal, current catalog permission check, serialized execution | +| `packages/flowsafe/src/agent-runner/durable-agent-runner.ts`, `resumeViaRuntime` | Fresh-host registry rebuild and runtime resume | +| `packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.authoritativeStatus` and `resume` | Storage-backed state, fence and exact suspension check | +| `packages/flowsafe/src/do-runner/execution-fence.ts` | Deployment admission and migration lock | +| `packages/flowsafe/src/do-runner/d1-storage.ts`, deadline snapshot scan | Bounded, persisted-cursor D1 scan pattern for orphan discovery | +| `packages/flowsafe/src/do-runner/inventory.ts` and `mastra-schema-guard.test.ts` | Read-only drain classification and table census | +| `packages/flowsafe/src/agent-runner/agent-gate-round-trip.test.ts` | Real runtime suspension and grant round-trip test pattern | + +Out of scope: WFP source or build admission, artifact signatures, capability grants, user/org quotas, customer billing, and public authentication. Those are platform policy. Approval decisions remain approval decisions; the broker does not mint Breakwater approval grants from a remote result. The durable-result confidentiality proposal is separate: this broker must not claim its stored result is output-policy approved, and hosts should avoid sensitive result content until that gate is implemented. + +## Verification and release gate + +Run each changed test file once per checkpoint, then the owning package checks allowed by the repository quality gate. Send complete output to `/tmp` logs and record exit status. Candidate commands for the implementation worktree: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run \ + src/agent-runner/durable-agent-runner.test.ts \ + src/agent-runner/agent-gate-round-trip.test.ts \ + src/agent-host/thread-topology.test.ts \ + src/agent-host/thread-host.test.ts +pnpm --filter @proofoftech/flowsafe typecheck +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe spike:verify +``` + +For the shared protocol and migration checkpoint, run the owning package's whole test suite once because the exported contract and D1 schema have broad consumers. Verify a real D1/workerd path with a fresh isolate and duplicate callback. After all checkpoints, run the repository's full suite once under the quality-gate rule, including the direct scenario filters that exercise changed contracts. Require these observations: one persisted invocation per suspension key under concurrent observations; orphan suspension discovery with no broker row; no dispatch before durable suspension; permit replay/renewal across an epoch change; identical result replay; conflicting result refusal; stale or foreign result refusal; cutover blocked while a callback remains active; deadline and cancellation stop retries but do not falsely claim executor stop; successful original-principal resume after eviction; recovery of an expired `resuming` lease; no repeated external side effect with a conforming idempotent executor; one terminal run state after lost acknowledgements. + +The implementation is auth, migration, data-retention, and concurrency work. Each code checkpoint is T2: clean-code, QA, architecture, and security lanes. This plan gets one design-level review when added; reviewers do not simulate its commands. A change to this plan after implementation starts records the departure once at completion. Merge a reviewed feature branch into `dev`, not `main`; publish only through Anchorage's version workflow. + +## Appendix: current-state anchors + +These are verbatim at the baseline commit. Reconfirm them before applying the plan. + +### Thread status and resume (`packages/flowsafe/src/agent-host/thread-topology.ts`) + +```typescript +status( + context: ActorContext, + input: AgentThreadRunRef, +): Promise; +``` + +`createAgentThreadTopology` routes status through `threads.send()` after `requireResourceAccess` on thread, resource, and run. Its approval resume currently sends: + +```typescript +resumeData: defaultResumeData(record, decision), +...expectedSuspensionFor(record), +entryPath: 'approval.resume', +requestedBy: record.decidedBy, +``` + +### Current suspension correlation (`packages/flowsafe/src/agent-host/thread-host.ts`, `currentApprovals`) + +```typescript +runRecord.suspendedAt === summary.suspendedAt?.[key] && +runRecord.resumeCount === summary.resumeCount?.[key] +``` + +`thread-host.ts`'s resume route calls `authorize()` on the stored principal and then: + +```typescript +return durable.resumeViaRuntime({ + runId: ref.runId, + requestedBy: requesterId, + ...(step !== undefined ? { step } : {}), + ...resumeFields, +``` + +### Runtime check (`packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.resume`) + +```typescript +const proof = await this.#assertResumeFence(workflowId, runId); +return this.#withRunLock(workflowId, runId, async () => { +``` + +Later in the same resume, the exact suspension check is: + +```typescript +if ( + leg.suspendedAt !== expectedSuspension.suspendedAt || + leg.resumeCount !== expectedSuspension.resumeCount +) + throw new SuspensionChangedError(workflowId, runId, stepKey); +``` + +### Guarded entry grammar (`packages/flowsafe/src/agent-runner/agent-run-state.ts`) + +```typescript +export const AGENT_ENTRY_PATHS = [ + 'http.start', + 'approval.resume', + 'signal.message', +``` + +The array continues with other existing entry paths. `packages/flowsafe/src/agent-host/catalog.ts` treats `approval.resume` as a continuation for automated principals. A new broker continuation must preserve that distinction. From 02d597535658b44b7b61f531d5df5103f7ce853d Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 11:32:56 +0400 Subject: [PATCH 02/73] Validate persisted tool resume data before runtime continuation Use the pinned tool resume schema after exact suspension admission so malformed results cannot advance a durable agent. A fresh-host regression failed before this check and passes afterward. --- .../agent-runner/durable-agent-runner.test.ts | 213 +++++++++++++++++- .../src/agent-runner/durable-agent-runner.ts | 143 ++++++++---- 2 files changed, 312 insertions(+), 44 deletions(-) diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts index be88ba1e..dbe8904d 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts @@ -156,6 +156,7 @@ function fakeRuntime( workflowIds, start, resume, + authoritativeStatus: async () => null, ...(overrides.pubsub !== undefined ? { pubsub: overrides.pubsub } : {}), } as unknown as RunnerRuntime; return { @@ -642,9 +643,219 @@ async function realAgentBridgeFixture( : {}), }); const start = vi.spyOn(runtime, 'start'); - return { sql, fence, workflows, counts, runtime, agent, start }; + return { sql, binding, fence, workflows, counts, runtime, agent, start }; } +describe('durable tool suspension after host restart', () => { + it('restores the tool payload and applies the result to the suspended call', async () => { + const runId = 'remote-tool-resume'; + const toolId = 'remote_lookup'; + const received: unknown[] = []; + const tool = createTool({ + id: toolId, + description: 'Wait for a remote lookup result', + inputSchema: z.object({ key: z.string() }), + suspendSchema: z.object({ key: z.string() }), + resumeSchema: z.object({ value: z.string() }), + execute: async ({ key }, context) => { + const result = context?.agent?.resumeData; + if (result) { + received.push({ + key, + result, + payload: context.agent?.suspendPayload, + }); + return { value: result.value }; + } + if (!context?.agent?.suspend) + throw new Error('tool suspension unavailable'); + await context.agent.suspend({ key }); + }, + }); + const text = localModelFixture(() => undefined); + const textStream = ( + text as unknown as { + doStream: (options: { abortSignal?: AbortSignal }) => Promise; + } + ).doStream; + let modelCalls = 0; + const model = { + ...(text as object), + doStream: async (options: Parameters[0]) => { + modelCalls++; + if (modelCalls > 2) return textStream(options); + return { + stream: new ReadableStream({ + start(controller) { + controller.enqueue({ type: 'stream-start', warnings: [] }); + controller.enqueue({ + type: 'tool-call', + toolCallId: `remote-call-${modelCalls}`, + toolName: toolId, + input: JSON.stringify({ key: 'account-7' }), + }); + controller.enqueue({ + type: 'finish', + finishReason: 'tool-calls', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + }); + controller.close(); + }, + }), + }; + }, + } as unknown as MastraModelConfig; + const first = await realAgentBridgeFixture( + undefined, + undefined, + false, + undefined, + { model, tools: { [toolId]: tool } }, + ); + let output: + | Awaited> + | undefined; + try { + output = await first.agent.streamUntilPersisted( + 'Look up account-7.', + { runId, maxSteps: 3, disableBackgroundTasks: true }, + 'operator-1', + 'human', + `${runId}-attempt`, + undefined, + undefined, + startAuthority(), + ); + const workflowId = first.agent.getWorkflow().id; + const suspended = await first.runtime.authoritativeStatus( + workflowId, + runId, + ); + expect(suspended).toMatchObject({ status: 'suspended' }); + const step = suspended?.suspended?.[0]; + if (!step) throw new Error('tool did not suspend'); + const stepKey = step.join('.'); + expect( + (suspended?.suspendPayload as Record | undefined)?.[ + stepKey + ], + ).toMatchObject({ + type: 'suspension', + toolCallId: 'remote-call-1', + toolCallSuspended: { key: 'account-7' }, + }); + expect(received).toEqual([]); + + output.cleanup(); + registryFor(first.agent).clear(); + globalRunRegistry.delete(runId); + const storage = createD1Storage({ binding: first.binding }); + await storage.init(); + const { runtime } = init( + { storage }, + { + executionFence: first.fence, + startIdempotency: 'none', + pubsub: createHostPubSub(), + }, + ); + const restored = createFlowsafeDurableAgent({ + agent: new Agent({ + id: 'writer', + name: 'Writer', + instructions: 'Return done.', + model, + tools: { [toolId]: tool }, + }), + runtime, + cache: false, + maxSteps: 1, + }); + const suspendedAt = suspended.suspendedAt?.[stepKey]; + if (typeof suspendedAt !== 'number') { + throw new Error('tool suspension has no timestamp'); + } + const firstResume = { + runId, + requestedBy: 'operator-1', + step, + expectedSuspension: { + suspendedAt, + ...(suspended.resumeCount?.[stepKey] === undefined + ? {} + : { resumeCount: suspended.resumeCount[stepKey] }), + }, + }; + await expect( + restored.resumeViaRuntime({ + ...firstResume, + resumeData: { value: 7 }, + }), + ).rejects.toThrow(); + expect(received).toEqual([]); + expect( + await runtime.authoritativeStatus(workflowId, runId), + ).toMatchObject({ + status: 'suspended', + }); + const resumed = await restored.resumeViaRuntime({ + ...firstResume, + resumeData: { value: 'found' }, + }); + expect(resumed.status).toBe('suspended'); + const nextStep = resumed.suspended?.[0]; + if (!nextStep) throw new Error('second tool call did not suspend'); + const nextKey = nextStep.join('.'); + expect( + (resumed.suspendPayload as Record | undefined)?.[ + nextKey + ], + ).toMatchObject({ + type: 'suspension', + toolCallId: 'remote-call-2', + toolCallSuspended: { key: 'account-7' }, + }); + const nextTimestamp = resumed.suspendedAt?.[nextKey]; + if (typeof nextTimestamp !== 'number') { + throw new Error('second tool suspension has no timestamp'); + } + const settled = await restored.resumeViaRuntime({ + runId, + requestedBy: 'operator-1', + step: nextStep, + resumeData: { value: 'again' }, + expectedSuspension: { + suspendedAt: nextTimestamp, + ...(resumed.resumeCount?.[nextKey] === undefined + ? {} + : { resumeCount: resumed.resumeCount[nextKey] }), + }, + }); + expect(settled.status).toBe('success'); + expect(received).toEqual([ + { + key: 'account-7', + result: { value: 'found' }, + payload: { key: 'account-7' }, + }, + { + key: 'account-7', + result: { value: 'again' }, + payload: { key: 'account-7' }, + }, + ]); + registryFor(restored).clear(); + globalRunRegistry.delete(runId); + } finally { + output?.cleanup(); + registryFor(first.agent).clear(); + globalRunRegistry.delete(runId); + first.start.mockRestore(); + first.sql.close(); + } + }); +}); + const OUTCOME_MARKER = 'MKCLIENTOUTCOME'; const CLEAN_OUTCOME = 'clean client result'; const APPROVAL = { id: 'approval_1' }; diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts index 4b8a0f1c..1211c8ae 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts @@ -2037,53 +2037,110 @@ export class FlowsafeDurableAgent< let rehydrated = false; let leg: ResumedThreadLeg | undefined; const legAbort = new AbortController(); + const resumeOptions: ResumeRunOptions = { + ...(options.step !== undefined ? { step: options.step } : {}), + ...(options.resumeData !== undefined + ? { resumeData: options.resumeData } + : {}), + ...(options.expectedSuspension !== undefined + ? { expectedSuspension: options.expectedSuspension } + : {}), + requestedBy: options.requestedBy, + requestedByKind: 'human', + legAbort, + prepareExecution: async (requestContext, workflowInput) => { + const current = await this.#runtime.authoritativeStatus( + this.getWorkflow().id, + options.runId, + ); + const step = Array.isArray(options.step) + ? options.step.join('.') + : (options.step ?? + (current?.suspended?.length === 1 + ? current.suspended[0]?.join('.') + : undefined)); + const payload = isRecord(current?.suspendPayload) + ? current.suspendPayload + : undefined; + const suspension = + step && isRecord(payload?.[step]) ? payload[step] : undefined; + if (suspension?.type === 'suspension') { + const toolName = suspension.toolName; + if ( + typeof toolName !== 'string' || + typeof suspension.toolCallId !== 'string' || + !isRecord(suspension.toolCallSuspended) + ) + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + const tools = (await this.listTools({ + requestContext, + })) as Record; + const tool = tools[toolName]; + const schema = isRecord(tool) ? tool.resumeSchema : undefined; + const standard = isRecord(schema) ? schema['~standard'] : undefined; + if (!isRecord(standard) || typeof standard.validate !== 'function') + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + const validated: unknown = await standard.validate( + options.resumeData, + ); + if (!isRecord(validated)) + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + if (validated.issues !== undefined) + throw new InvalidRunRequestError( + 'resume data does not match suspended tool', + ); + if (!Object.hasOwn(validated, 'value')) + throw new RunStateUnreadableError( + this.getWorkflow().id, + options.runId, + ); + resumeOptions.resumeData = validated.value; + } + const offset = ( + await this.pubsub.getHistory(AGENT_STREAM_TOPIC(options.runId)) + ).length; + const modelSettings = persistedModelSettings(workflowInput); + await this.#rehydrateRegistry({ + runId: options.runId, + requestContext, + abortController: legAbort, + ...(memory !== undefined ? { memory } : {}), + ...(modelSettings !== undefined ? { modelSettings } : {}), + }); + rehydrated = true; + const observed = await this.observe(options.runId, { offset }); + if (this.#threadRuntime) { + leg = this.#trackResumedLeg(options.runId); + await this.#threadRuntime.registerRun( + this as unknown as Parameters< + Mastra['agentThreadStreamRuntime']['registerRun'] + >[0], + bindResumedLegLifecycle(observed.output, leg), + { + runId: options.runId, + ...(memory !== undefined ? { memory } : {}), + } as Parameters< + Mastra['agentThreadStreamRuntime']['registerRun'] + >[2], + this.getPubSub(), + ); + } + }, + }; try { const summary = await this.#runtime.resume( this.getWorkflow().id, options.runId, - { - ...(options.step !== undefined ? { step: options.step } : {}), - ...(options.resumeData !== undefined - ? { resumeData: options.resumeData } - : {}), - ...(options.expectedSuspension !== undefined - ? { expectedSuspension: options.expectedSuspension } - : {}), - requestedBy: options.requestedBy, - requestedByKind: 'human', - legAbort, - prepareExecution: async (requestContext, workflowInput) => { - const offset = ( - await this.pubsub.getHistory(AGENT_STREAM_TOPIC(options.runId)) - ).length; - const modelSettings = persistedModelSettings(workflowInput); - await this.#rehydrateRegistry({ - runId: options.runId, - requestContext, - abortController: legAbort, - ...(memory !== undefined ? { memory } : {}), - ...(modelSettings !== undefined ? { modelSettings } : {}), - }); - rehydrated = true; - const observed = await this.observe(options.runId, { offset }); - if (this.#threadRuntime) { - leg = this.#trackResumedLeg(options.runId); - await this.#threadRuntime.registerRun( - this as unknown as Parameters< - Mastra['agentThreadStreamRuntime']['registerRun'] - >[0], - bindResumedLegLifecycle(observed.output, leg), - { - runId: options.runId, - ...(memory !== undefined ? { memory } : {}), - } as Parameters< - Mastra['agentThreadStreamRuntime']['registerRun'] - >[2], - this.getPubSub(), - ); - } - }, - }, + resumeOptions, ); const terminalError = legTerminalError( summary, From 0615cd87d0f9961fd229146b38333eb55db310a7 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 11:40:17 +0400 Subject: [PATCH 03/73] Accept scalar tool suspension payloads on resume --- .../agent-runner/durable-agent-runner.test.ts | 18 +++++++++++------- .../src/agent-runner/durable-agent-runner.ts | 3 +-- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts index dbe8904d..225a2a50 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts @@ -647,7 +647,11 @@ async function realAgentBridgeFixture( } describe('durable tool suspension after host restart', () => { - it('restores the tool payload and applies the result to the suspended call', async () => { + it.each([ + ['object', { key: 'account-7' }], + ['scalar', 'account-7'], + ['array', ['account-7']], + ])('restores a %s tool payload and applies the result to the suspended call', async (_shape, suspendedPayload) => { const runId = 'remote-tool-resume'; const toolId = 'remote_lookup'; const received: unknown[] = []; @@ -655,7 +659,7 @@ describe('durable tool suspension after host restart', () => { id: toolId, description: 'Wait for a remote lookup result', inputSchema: z.object({ key: z.string() }), - suspendSchema: z.object({ key: z.string() }), + suspendSchema: z.unknown(), resumeSchema: z.object({ value: z.string() }), execute: async ({ key }, context) => { const result = context?.agent?.resumeData; @@ -669,7 +673,7 @@ describe('durable tool suspension after host restart', () => { } if (!context?.agent?.suspend) throw new Error('tool suspension unavailable'); - await context.agent.suspend({ key }); + await context.agent.suspend(suspendedPayload); }, }); const text = localModelFixture(() => undefined); @@ -742,7 +746,7 @@ describe('durable tool suspension after host restart', () => { ).toMatchObject({ type: 'suspension', toolCallId: 'remote-call-1', - toolCallSuspended: { key: 'account-7' }, + toolCallSuspended: suspendedPayload, }); expect(received).toEqual([]); @@ -813,7 +817,7 @@ describe('durable tool suspension after host restart', () => { ).toMatchObject({ type: 'suspension', toolCallId: 'remote-call-2', - toolCallSuspended: { key: 'account-7' }, + toolCallSuspended: suspendedPayload, }); const nextTimestamp = resumed.suspendedAt?.[nextKey]; if (typeof nextTimestamp !== 'number') { @@ -836,12 +840,12 @@ describe('durable tool suspension after host restart', () => { { key: 'account-7', result: { value: 'found' }, - payload: { key: 'account-7' }, + payload: suspendedPayload, }, { key: 'account-7', result: { value: 'again' }, - payload: { key: 'account-7' }, + payload: suspendedPayload, }, ]); registryFor(restored).clear(); diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts index 1211c8ae..3b670343 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts @@ -2068,8 +2068,7 @@ export class FlowsafeDurableAgent< const toolName = suspension.toolName; if ( typeof toolName !== 'string' || - typeof suspension.toolCallId !== 'string' || - !isRecord(suspension.toolCallSuspended) + typeof suspension.toolCallId !== 'string' ) throw new RunStateUnreadableError( this.getWorkflow().id, From fa06037cb99f6da9c51ab11441eaa9c46725d6d9 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 12:08:59 +0400 Subject: [PATCH 04/73] Journal remote-step suspensions and gated dispatch permits Persist exact suspension identity and request, serialize permit issue/consume with snapshot and fence state, and expose bounded inventory. Host dispatch and result continuation remain disabled pending permit hardening and live proof. --- packages/flowsafe/src/do-runner/inventory.ts | 32 + .../src/do-runner/mastra-schema-guard.test.ts | 12 +- .../src/do-runner/remote-step-journal.test.ts | 314 +++++++++ .../src/do-runner/remote-step-journal.ts | 651 ++++++++++++++++++ 4 files changed, 1008 insertions(+), 1 deletion(-) create mode 100644 packages/flowsafe/src/do-runner/remote-step-journal.test.ts create mode 100644 packages/flowsafe/src/do-runner/remote-step-journal.ts diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index 14ef2a8c..c9751802 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -70,6 +70,7 @@ import { DUE_NOTIFICATION_SQL, notificationTimestampMillis, } from './notification-predicate.js'; +import { REMOTE_STEP_TABLE } from './remote-step-journal.js'; import { START_IDEMPOTENCY_TABLE, START_RESERVATION_STATES, @@ -122,6 +123,7 @@ export type InventoryCategoryClass = 'work' | 'standing'; */ export const INVENTORY_CATEGORIES = [ 'runs', + 'remote-steps', 'approvals-waiting', 'schedule-deferred-dispatches', 'pending-notifications', @@ -290,6 +292,11 @@ export interface FlowsafeTableEntry { * characters. */ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ + { + table: REMOTE_STEP_TABLE, + purpose: 'durable remote tool invocations and their dispatch permits', + accounting: { category: 'remote-steps' }, + }, { table: APPROVALS_TABLE, purpose: 'the durable approval queue — one record per suspended decision', @@ -479,6 +486,13 @@ export const INVENTORY_CATEGORY_DESCRIPTORS: readonly InventoryCategoryDescripto holds: 'a workflow or agent run that has not reached a terminal state (cancelled and timed_out count as live until their lifecycle cleanup completes)', }, + { + category: 'remote-steps', + class: 'work', + table: REMOTE_STEP_TABLE, + holds: + 'a remote invocation without a recorded result or an external dispatch not confirmed stopped', + }, { category: 'approvals-waiting', class: 'work', @@ -1005,6 +1019,24 @@ export class DeploymentInventory { binds: RUN_TERMINAL_STATUSES, annotate: annotateRunOwners, }; + case 'remote-steps': + return { + key: ['invocation_id'], + detail: [ + 'workflow_id', + 'run_id', + 'agent_id', + 'thread_id', + 'status', + 'deadline_at', + 'permit_generation', + 'permit_consumed_at', + 'permit_closed_at', + ], + where: + "status <> 'failed' OR (permit_consumed_at IS NOT NULL AND permit_closed_at IS NULL)", + binds: [], + }; case 'approvals-waiting': // OPEN_STATUSES, not just 'pending': a claimed or escalated approval is // still undecided, and its run is still suspended behind it. The same diff --git a/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts b/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts index 301ae961..bf7768c8 100644 --- a/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts +++ b/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts @@ -76,6 +76,7 @@ import { type InventoryTableAccounting, } from './inventory.js'; import { mintThreadId, resourceIdFromKey } from './memory-id.js'; +import { RemoteStepJournal } from './remote-step-journal.js'; import type { RunnerRuntime } from './runtime.js'; import { START_IDEMPOTENCY_TABLE, @@ -485,7 +486,16 @@ describe('Mastra persistence guards (D1Store SQL over node:sqlite)', () => { threadId: 'thr-1', resourceId: 'res-1', }); - await new ExecutionFenceStore(binding as never).seed('open'); + const fence = new ExecutionFenceStore(binding as never); + await fence.seed('open'); + await new RemoteStepJournal({ + db: binding as never, + fence, + runtime, + deploymentId: 'census-deployment', + tablePrefix: '', + allowedToolIds: ['census-tool'], + }).initialize(); sqlite.exec( `CREATE TABLE IF NOT EXISTS ${DEPLOYMENT_SENTINEL_TABLE} ( id INTEGER PRIMARY KEY CHECK (id = 1), diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts new file mode 100644 index 00000000..a460c410 --- /dev/null +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -0,0 +1,314 @@ +// SPDX-License-Identifier: Apache-2.0 + +import { afterEach, describe, expect, it } from 'vitest'; +import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; +import { ExecutionFenceStore } from './execution-fence.js'; +import { DeploymentInventory } from './inventory.js'; +import { + REMOTE_STEP_MARKER, + REMOTE_STEP_TABLE, + RemoteStepJournal, +} from './remote-step-journal.js'; +import type { RunnerRuntime, RunSummary } from './runtime.js'; + +const ADDRESS = { + deploymentId: 'tenant-1', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'thread-1', + resourceId: 'thread-1', + runId: 'run-1', +}; + +describe('remote step journal dispatch admission', () => { + const databases: Array<{ close(): void }> = []; + + afterEach(() => { + for (const database of databases.splice(0)) database.close(); + }); + + async function fixture() { + const sqlite = openSqlite() as ReturnType & { + close(): void; + }; + databases.push(sqlite); + const db = sqliteUnitDatabase(sqlite); + sqlite.exec(`CREATE TABLE mastra_workflow_snapshot ( + workflow_name TEXT NOT NULL, run_id TEXT NOT NULL, + snapshot TEXT NOT NULL, PRIMARY KEY (workflow_name, run_id))`); + sqlite + .prepare(`INSERT INTO mastra_workflow_snapshot VALUES (?, ?, ?)`) + .run(ADDRESS.workflowId, ADDRESS.runId, '{"status":"suspended"}'); + const fence = new ExecutionFenceStore(db); + await fence.seed('open'); + let now = 1_000; + let summary: RunSummary = { + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 500 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }; + const runtime = { + authoritativeStatus: async () => summary, + authoritativeStartState: async () => ({ + storage: 'd1', + execution: { + tablePrefix: '', + workflowId: ADDRESS.workflowId, + runId: ADDRESS.runId, + startToken: 'start-token', + }, + provenance: { + startIdentity: { + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + }, + }, + }, + }), + } as unknown as Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + const journal = new RemoteStepJournal({ + db, + fence, + runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + now: () => now, + }); + await journal.initialize(); + return { + db, + fence, + journal, + runtime, + reopen: () => + new RemoteStepJournal({ + db, + fence, + runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + now: () => now, + }), + sqlite, + setNow: (value: number) => { + now = value; + }, + setSummary: (value: RunSummary) => { + summary = value; + }, + }; + } + + it('converges repeated observation on one invocation without a permit', async () => { + const h = await fixture(); + const [first, replay] = await Promise.all([ + h.journal.register(ADDRESS, 10_000), + h.journal.register(ADDRESS, 10_000), + ]); + expect(replay.invocationId).toBe(first.invocationId); + expect(first).toMatchObject({ + status: 'pending', + permitGeneration: 0, + request: { account: 'acme' }, + stepPath: ['durable-agentic-execution'], + suspendedAt: 500, + }); + expect( + h.sqlite + .prepare(`SELECT count(*) AS count FROM ${REMOTE_STEP_TABLE}`) + .get(), + ).toMatchObject({ count: 1 }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).entries, + ).toEqual([ + { + key: [first.invocationId], + detail: expect.objectContaining({ + status: 'pending', + run_id: ADDRESS.runId, + }), + }, + ]); + }); + + it('accepts one permit consumption and refuses replay as effect authority', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + expect(permit).toMatchObject({ generation: 1, expiresAt: 2_000 }); + expect(await h.journal.consumePermit(permit)).toBe('accepted'); + expect(await h.journal.consumePermit(permit)).toBe('already-consumed'); + await expect( + h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 409, + }); + await h.journal.closePermit(permit); + const renewed = await h.journal.issuePermit(invocation.invocationId, 1_000); + expect(renewed.generation).toBe(2); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + expect(await h.journal.consumePermit(renewed)).toBe('accepted'); + }); + + it('recovers one permit after a journal instance is replaced', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const recovered = h.reopen(); + expect(await recovered.issuePermit(invocation.invocationId, 1_000)).toEqual( + permit, + ); + expect(await recovered.consumePermit(permit)).toBe('accepted'); + expect(await h.reopen().consumePermit(permit)).toBe('already-consumed'); + }); + + it('rejects a remote request larger than its stored payload limit', async () => { + const h = await fixture(); + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 500 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: 'x'.repeat(16_384), + }, + }, + }, + }); + await expect(h.journal.register(ADDRESS, 10_000)).rejects.toMatchObject({ + status: 400, + }); + }); + + it('refuses a marker from a tool the host did not designate', async () => { + const h = await fixture(); + const restricted = new RemoteStepJournal({ + db: h.db, + fence: h.fence, + runtime: h.runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['other_tool'], + }); + await expect(restricted.register(ADDRESS, 10_000)).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses a corrupted persisted table identity before building a query', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + h.sqlite + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?`) + .run('bad"_prefix'); + await expect( + h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses an unconsumed permit after the fence moves to migration lock', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 503, + }); + expect( + (await h.journal.get(invocation.invocationId))?.permitConsumedAt, + ).toBeUndefined(); + }); + + it('refuses a permit when the fence moves after preflight but before the write', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes('SET permit_consumed_at = ?')) return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + return bound.all(); + }, + }; + }, + }; + }; + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + expect( + (await h.journal.get(invocation.invocationId))?.permitConsumedAt, + ).toBeUndefined(); + }); + + it('refuses dispatch after cancellation or deadline', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + h.sqlite + .prepare(`UPDATE mastra_workflow_snapshot SET snapshot = ?`) + .run('{"status":"cancelled"}'); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + h.setNow(10_001); + await expect( + h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 409, + }); + }); +}); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts new file mode 100644 index 00000000..9ae204f2 --- /dev/null +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -0,0 +1,651 @@ +// SPDX-License-Identifier: Apache-2.0 + +import { EXECUTION_FENCE_TABLE } from '#deployment-identity-protocol'; +import { DoStatusError } from './do-status-error.js'; +import { + admitsExistingRun, + captureExecutionFenceAdmissionSchema, + type ExecutionFenceDatabase, + type ExecutionFenceStore, + executionFenceAdmissionSql, + executionFenceAdmissionValues, +} from './execution-fence.js'; +import { resourceIdFromKey } from './memory-id.js'; +import { isPathSafeId } from './path-safe-id.js'; +import type { RunnerRuntime, RunSummary } from './runtime.js'; +import { validateTablePrefix } from './table-prefix.js'; + +export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; +export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; +export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; + +export interface RemoteStepAddress { + deploymentId: string; + tablePrefix: string; + workflowId: string; + agentId: string; + threadId: string; + resourceId: string; + runId: string; +} + +export interface RemoteStepInvocation extends RemoteStepAddress { + invocationId: string; + stepPath: readonly string[]; + suspendedAt: number; + resumeCount?: number; + request: unknown; + deadlineAt: number; + status: 'pending' | 'dispatched' | 'failed'; + permitGeneration: number; + permitEpoch?: number; + permitRevision?: number; + permitExpiresAt?: number; + permitConsumedAt?: number; + permitClosedAt?: number; +} + +export interface RemoteStepPermit { + invocationId: string; + generation: number; + mutationEpoch: number; + transitionRevision: number; + expiresAt: number; +} + +class RemoteStepRefusal extends DoStatusError { + readonly status: number; + + constructor(status: number, message: string) { + super(message); + this.name = 'RemoteStepRefusal'; + this.status = status; + } +} + +type JournalRow = { + invocation_id: string; + deployment_id: string; + table_prefix: string; + workflow_id: string; + agent_id: string; + thread_id: string; + resource_id: string; + run_id: string; + step_path: string; + suspended_at: number; + resume_count: number; + request_json: string; + deadline_at: number; + status: string; + permit_generation: number; + permit_epoch: number | null; + permit_revision: number | null; + permit_expires_at: number | null; + permit_consumed_at: number | null; + permit_closed_at: number | null; +}; + +function record(value: unknown): Record | undefined { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +function positiveTime(value: unknown): value is number { + return Number.isSafeInteger(value) && (value as number) > 0; +} + +function rowResult(value: unknown): JournalRow | undefined { + const rows = record(value)?.results; + return Array.isArray(rows) && rows.length === 1 + ? (rows[0] as JournalRow) + : undefined; +} + +function decode(row: JournalRow): RemoteStepInvocation { + if ( + typeof row.step_path !== 'string' || + row.step_path.length > 1_024 || + typeof row.request_json !== 'string' || + new TextEncoder().encode(row.request_json).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + !isPathSafeId(row.deployment_id) || + !isPathSafeId(row.workflow_id) || + !isPathSafeId(row.agent_id) || + !isPathSafeId(row.thread_id) || + !isPathSafeId(row.resource_id) || + !isPathSafeId(row.run_id) + ) + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + try { + validateTablePrefix(row.table_prefix); + } catch { + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + } + let stepPath: unknown; + let request: unknown; + try { + stepPath = JSON.parse(row.step_path); + request = JSON.parse(row.request_json); + } catch { + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + } + if ( + !Array.isArray(stepPath) || + stepPath.length === 0 || + stepPath.some((part) => typeof part !== 'string' || part === '') || + !isPathSafeId(row.invocation_id) || + !positiveTime(row.suspended_at) || + !positiveTime(row.deadline_at) || + !Number.isSafeInteger(row.resume_count) || + (row.resume_count !== -1 && row.resume_count < 1) || + !Number.isSafeInteger(row.permit_generation) || + row.permit_generation < 0 || + !['pending', 'dispatched', 'failed'].includes(row.status) + ) + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + return { + invocationId: row.invocation_id, + deploymentId: row.deployment_id, + tablePrefix: row.table_prefix, + workflowId: row.workflow_id, + agentId: row.agent_id, + threadId: row.thread_id, + resourceId: row.resource_id, + runId: row.run_id, + stepPath, + suspendedAt: row.suspended_at, + ...(row.resume_count < 0 ? {} : { resumeCount: row.resume_count }), + request, + deadlineAt: row.deadline_at, + status: row.status as RemoteStepInvocation['status'], + permitGeneration: row.permit_generation, + ...(row.permit_epoch === null ? {} : { permitEpoch: row.permit_epoch }), + ...(row.permit_revision === null + ? {} + : { permitRevision: row.permit_revision }), + ...(row.permit_expires_at === null + ? {} + : { permitExpiresAt: row.permit_expires_at }), + ...(row.permit_consumed_at === null + ? {} + : { permitConsumedAt: row.permit_consumed_at }), + ...(row.permit_closed_at === null + ? {} + : { permitClosedAt: row.permit_closed_at }), + }; +} + +function selectedSuspension( + summary: RunSummary, + allowedToolIds: ReadonlySet, +): { + stepPath: string[]; + suspendedAt: number; + resumeCount?: number; + requestJson: string; +} { + if (summary.status !== 'suspended') + throw new RemoteStepRefusal(409, 'run is not suspended'); + const payloads = record(summary.suspendPayload); + const matches = (summary.suspended ?? []).flatMap((path) => { + const key = path.join('.'); + const suspended = record(payloads?.[key]); + const tool = record(suspended?.toolCallSuspended); + return suspended?.type === 'suspension' && + tool?.remoteStep === REMOTE_STEP_MARKER + ? [{ stepPath: path, key, tool, suspended }] + : []; + }); + if (matches.length !== 1) + throw new RemoteStepRefusal(409, 'remote step suspension is ambiguous'); + const selected = matches[0]; + if (!selected) + throw new RemoteStepRefusal(409, 'remote step suspension is ambiguous'); + const { stepPath, key, tool, suspended } = selected; + const suspendedAt = summary.suspendedAt?.[key]; + const resumeCount = summary.resumeCount?.[key]; + if ( + !positiveTime(suspendedAt) || + typeof suspended.toolName !== 'string' || + !allowedToolIds.has(suspended.toolName) || + typeof suspended.toolCallId !== 'string' || + suspended.toolCallId === '' || + (resumeCount !== undefined && + (!Number.isSafeInteger(resumeCount) || resumeCount < 1)) || + !Object.hasOwn(tool, 'request') + ) + throw new RemoteStepRefusal(503, 'remote step suspension is unreadable'); + const requestJson = JSON.stringify(tool.request); + if ( + typeof requestJson !== 'string' || + new TextEncoder().encode(requestJson).length > MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal(400, 'remote step request exceeds its limit'); + return { + stepPath, + suspendedAt, + ...(resumeCount === undefined ? {} : { resumeCount }), + requestJson, + }; +} + +export class RemoteStepJournal { + readonly #db: ExecutionFenceDatabase; + readonly #fence: ExecutionFenceStore; + readonly #runtime: Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + readonly #now: () => number; + readonly #deploymentId: string; + readonly #tablePrefix: string; + readonly #allowedToolIds: ReadonlySet; + + constructor(input: { + db: ExecutionFenceDatabase; + fence: ExecutionFenceStore; + runtime: Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + deploymentId: string; + tablePrefix: string; + allowedToolIds: readonly string[]; + now?: () => number; + }) { + if (!input.fence.usesDatabase(input.db)) + throw new Error('remote step fence uses another database'); + if (!isPathSafeId(input.deploymentId)) + throw new Error('remote step deployment id is malformed'); + validateTablePrefix(input.tablePrefix); + if ( + !Array.isArray(input.allowedToolIds) || + input.allowedToolIds.length === 0 || + input.allowedToolIds.some((id) => !isPathSafeId(id)) || + new Set(input.allowedToolIds).size !== input.allowedToolIds.length + ) + throw new Error('remote step tool ids are malformed'); + this.#db = input.db; + this.#fence = input.fence; + this.#runtime = input.runtime; + this.#deploymentId = input.deploymentId; + this.#tablePrefix = input.tablePrefix; + this.#allowedToolIds = new Set(input.allowedToolIds); + this.#now = input.now ?? Date.now; + } + + async initialize(): Promise { + await this.#db + .prepare(`CREATE TABLE IF NOT EXISTS ${REMOTE_STEP_TABLE} ( + invocation_id TEXT PRIMARY KEY, + deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, + workflow_id TEXT NOT NULL, + agent_id TEXT NOT NULL, + thread_id TEXT NOT NULL, + resource_id TEXT NOT NULL, + run_id TEXT NOT NULL, + step_path TEXT NOT NULL, + suspended_at INTEGER NOT NULL, + resume_count INTEGER NOT NULL, + request_json TEXT NOT NULL, + deadline_at INTEGER NOT NULL, + status TEXT NOT NULL, + permit_generation INTEGER NOT NULL DEFAULT 0, + permit_epoch INTEGER, + permit_revision INTEGER, + permit_expires_at INTEGER, + permit_consumed_at INTEGER, + permit_closed_at INTEGER, + UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, + thread_id, resource_id, run_id, step_path, suspended_at, resume_count) + )`) + .run(); + } + + async #snapshot(address: RemoteStepAddress): Promise { + const result = await this.#db + .prepare( + `SELECT snapshot FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? LIMIT 2`, + ) + .bind(address.workflowId, address.runId) + .all(); + const rows = record(result)?.results; + if ( + !Array.isArray(rows) || + rows.length !== 1 || + typeof record(rows[0])?.snapshot !== 'string' + ) + throw new RemoteStepRefusal(503, 'remote step snapshot is unreadable'); + return record(rows[0])?.snapshot as string; + } + + async #current(address: RemoteStepAddress) { + const summary = await this.#runtime.authoritativeStatus( + address.workflowId, + address.runId, + ); + if (!summary) throw new RemoteStepRefusal(404, 'run not found'); + return selectedSuspension(summary, this.#allowedToolIds); + } + + async #stableSnapshot(address: RemoteStepAddress) { + const before = await this.#current(address); + const snapshot = await this.#snapshot(address); + const after = await this.#current(address); + if ( + before.suspendedAt !== after.suspendedAt || + before.resumeCount !== after.resumeCount || + JSON.stringify(before.stepPath) !== JSON.stringify(after.stepPath) || + before.requestJson !== after.requestJson + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + return { suspension: after, snapshot }; + } + + async register( + address: RemoteStepAddress, + hostDeadlineAt: number, + ): Promise { + if ( + address.deploymentId !== this.#deploymentId || + address.tablePrefix !== this.#tablePrefix || + !isPathSafeId(address.workflowId) || + !isPathSafeId(address.agentId) || + !isPathSafeId(address.threadId) || + !isPathSafeId(address.resourceId) || + !isPathSafeId(address.runId) || + !positiveTime(hostDeadlineAt) + ) + throw new RemoteStepRefusal(400, 'remote step address is malformed'); + validateTablePrefix(address.tablePrefix); + await this.#execution(address); + const { suspension, snapshot } = await this.#stableSnapshot(address); + const summary = await this.#runtime.authoritativeStatus( + address.workflowId, + address.runId, + ); + const deadlineAt = Math.min( + hostDeadlineAt, + summary?.deadlineAt ?? hostDeadlineAt, + ); + if (deadlineAt <= this.#now()) + throw new RemoteStepRefusal(409, 'remote step deadline elapsed'); + const invocationId = crypto.randomUUID(); + const stepPath = JSON.stringify(suspension.stepPath); + await this.#db + .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( + invocation_id, deployment_id, table_prefix, workflow_id, agent_id, + thread_id, resource_id, run_id, step_path, suspended_at, resume_count, + request_json, deadline_at, status) + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending' + WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + ON CONFLICT DO NOTHING`) + .bind( + invocationId, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.agentId, + address.threadId, + address.resourceId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + deadlineAt, + address.workflowId, + address.runId, + snapshot, + ) + .run(); + const result = await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} + WHERE deployment_id = ? AND table_prefix = ? AND workflow_id = ? + AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? + AND step_path = ? AND suspended_at = ? AND resume_count = ?`) + .bind( + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.agentId, + address.threadId, + address.resourceId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + ) + .all(); + const row = rowResult(result); + if (!row || row.request_json !== suspension.requestJson) + throw new RemoteStepRefusal( + 409, + 'remote step registration did not settle', + ); + return decode(row); + } + + async get(invocationId: string): Promise { + if (!isPathSafeId(invocationId)) + throw new RemoteStepRefusal(400, 'invocation id is malformed'); + const row = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + return row ? decode(row) : undefined; + } + + async #execution(address: RemoteStepAddress) { + const state = await this.#runtime.authoritativeStartState( + address.workflowId, + address.runId, + ); + const target = state?.provenance.startIdentity?.target; + if (state?.storage !== 'd1' || target?.kind !== 'agent') + throw new RemoteStepRefusal( + 503, + 'remote step run identity is unreadable', + ); + if ( + target.id !== address.agentId || + target.threadId !== address.threadId || + state.execution.tablePrefix !== address.tablePrefix || + address.resourceId !== resourceIdFromKey(address.threadId) + ) + throw new RemoteStepRefusal(409, 'remote step run identity changed'); + return state.execution; + } + + async #admission(invocation: RemoteStepInvocation, priorBindings: number) { + const observed = await this.#fence.readForAdmission(); + const proof = await this.#execution(invocation); + if (!admitsExistingRun(observed.reading, proof)) + throw new RemoteStepRefusal(503, 'remote step execution is fenced'); + const schema = await captureExecutionFenceAdmissionSchema( + await this.#db + .prepare(`PRAGMA table_xinfo(${EXECUTION_FENCE_TABLE})`) + .all(), + ); + const values: unknown[] = []; + const bind = (value: unknown) => { + values.push(value); + return `?${priorBindings + values.length}`; + }; + const predicate = executionFenceAdmissionSql({ + callerEpoch: bind(observed.reading.mutationEpoch ?? null), + semantic: executionFenceAdmissionValues(observed).map(bind), + schema: bind(schema), + statePredicate: `f.state IN ('open', 'draining') OR + (f.state = 'proof-only' AND f.proof_table_prefix IS ${bind(proof.tablePrefix)} + AND f.proof_workflow_id IS ${bind(proof.workflowId)} + AND f.proof_run_id IS ${bind(proof.runId)} + AND f.proof_start_token IS ${bind(proof.startToken)})`, + }); + return { predicate, values, bind, observed: observed.reading }; + } + + async issuePermit( + invocationId: string, + lifetimeMs: number, + ): Promise { + const invocation = await this.get(invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if (!positiveTime(lifetimeMs)) + throw new RemoteStepRefusal(400, 'permit lifetime is malformed'); + const now = this.#now(); + const expiresAt = Math.min(invocation.deadlineAt, now + lifetimeMs); + if (!positiveTime(expiresAt) || expiresAt <= now) + throw new RemoteStepRefusal(409, 'remote step deadline elapsed'); + const address = invocation; + const { suspension, snapshot } = await this.#stableSnapshot(address); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const gate = await this.#admission(invocation, 9); + if ( + invocation.status === 'dispatched' && + invocation.permitConsumedAt === undefined && + invocation.permitExpiresAt !== undefined && + invocation.permitExpiresAt > now && + invocation.permitEpoch === gate.observed.mutationEpoch && + invocation.permitRevision === gate.observed.transitionRevision + ) + return { + invocationId, + generation: invocation.permitGeneration, + mutationEpoch: invocation.permitEpoch, + transitionRevision: invocation.permitRevision, + expiresAt: invocation.permitExpiresAt, + }; + if ( + invocation.status === 'failed' || + (invocation.permitConsumedAt !== undefined && + invocation.permitClosedAt === undefined) + ) + throw new RemoteStepRefusal(409, 'remote step dispatch is active'); + const epoch = gate.observed.mutationEpoch; + const revision = gate.observed.transitionRevision; + if (epoch === undefined || revision === undefined) + throw new RemoteStepRefusal(503, 'remote step fence is unversioned'); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET + status = 'dispatched', permit_generation = permit_generation + 1, + permit_epoch = ?, permit_revision = ?, permit_expires_at = ?, + permit_consumed_at = NULL, permit_closed_at = NULL + WHERE invocation_id = ? AND permit_generation = ? AND deadline_at > ? + AND status IN ('pending', 'dispatched') + AND (permit_consumed_at IS NULL OR permit_closed_at IS NOT NULL) + AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + AND ${gate.predicate} RETURNING *`) + .bind( + epoch, + revision, + expiresAt, + invocationId, + invocation.permitGeneration, + now, + invocation.workflowId, + invocation.runId, + snapshot, + ...gate.values, + ) + .all(); + const row = rowResult(result); + if (!row) throw new RemoteStepRefusal(409, 'remote step dispatch changed'); + return { + invocationId, + generation: row.permit_generation, + mutationEpoch: epoch, + transitionRevision: revision, + expiresAt, + }; + } + + async consumePermit( + permit: RemoteStepPermit, + ): Promise<'accepted' | 'already-consumed'> { + const invocation = await this.get(permit.invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if ( + invocation.permitGeneration !== permit.generation || + invocation.permitEpoch !== permit.mutationEpoch || + invocation.permitRevision !== permit.transitionRevision || + invocation.permitExpiresAt !== permit.expiresAt + ) + throw new RemoteStepRefusal(409, 'remote step permit changed'); + if (invocation.permitConsumedAt !== undefined) return 'already-consumed'; + const now = this.#now(); + const { suspension, snapshot } = await this.#stableSnapshot(invocation); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const gate = await this.#admission(invocation, 11); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET permit_consumed_at = ? + WHERE invocation_id = ? AND permit_generation = ? + AND permit_epoch = ? AND permit_revision = ? + AND permit_expires_at = ? AND permit_expires_at > ? + AND deadline_at > ? AND permit_consumed_at IS NULL + AND status = 'dispatched' + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + AND ${gate.predicate} RETURNING invocation_id`) + .bind( + now, + permit.invocationId, + permit.generation, + permit.mutationEpoch, + permit.transitionRevision, + permit.expiresAt, + now, + now, + invocation.workflowId, + invocation.runId, + snapshot, + ...gate.values, + ) + .all(); + if (rowResult(result)) return 'accepted'; + const after = await this.get(permit.invocationId); + if ( + after?.permitGeneration === permit.generation && + after.permitConsumedAt !== undefined + ) + return 'already-consumed'; + throw new RemoteStepRefusal(409, 'remote step permit is unavailable'); + } + + async closePermit(permit: RemoteStepPermit): Promise { + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET permit_closed_at = ? + WHERE invocation_id = ? AND permit_generation = ? + AND permit_consumed_at IS NOT NULL AND permit_closed_at IS NULL + RETURNING invocation_id`) + .bind(this.#now(), permit.invocationId, permit.generation) + .all(); + if (rowResult(result)) return; + const current = await this.get(permit.invocationId); + if ( + current?.permitGeneration !== permit.generation || + current.permitClosedAt === undefined + ) + throw new RemoteStepRefusal(409, 'remote step permit close conflicts'); + } +} From 667c07c6ab8860289b8d652d16c7aaadbe5803e1 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 12:20:32 +0400 Subject: [PATCH 05/73] Bind remote permits to current run authority Refuse pending cancellation, bind effects to the current request and tightened deadline, and keep consumed permits closed until a trusted executor completion contract exists. --- packages/flowsafe/src/do-runner/inventory.ts | 4 +- .../src/do-runner/remote-step-journal.test.ts | 229 +++++++++++++++++- .../src/do-runner/remote-step-journal.ts | 87 ++++--- 3 files changed, 270 insertions(+), 50 deletions(-) diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index c9751802..8b985482 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -1031,10 +1031,8 @@ export class DeploymentInventory { 'deadline_at', 'permit_generation', 'permit_consumed_at', - 'permit_closed_at', ], - where: - "status <> 'failed' OR (permit_consumed_at IS NOT NULL AND permit_closed_at IS NULL)", + where: "status <> 'failed' OR permit_consumed_at IS NOT NULL", binds: [], }; case 'approvals-waiting': diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index a460c410..dfa5d5fb 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -1,8 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 import { afterEach, describe, expect, it } from 'vitest'; +import { z } from 'zod'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; +import { createD1Storage } from './d1-storage.js'; import { ExecutionFenceStore } from './execution-fence.js'; +import { init } from './init.js'; import { DeploymentInventory } from './inventory.js'; import { REMOTE_STEP_MARKER, @@ -64,6 +67,13 @@ describe('remote step journal dispatch admission', () => { authoritativeStatus: async () => summary, authoritativeStartState: async () => ({ storage: 'd1', + snapshot: JSON.parse( + ( + sqlite + .prepare('SELECT snapshot FROM mastra_workflow_snapshot') + .get() as { snapshot: string } + ).snapshot, + ), execution: { tablePrefix: '', workflowId: ADDRESS.workflowId, @@ -116,6 +126,25 @@ describe('remote step journal dispatch admission', () => { setSummary: (value: RunSummary) => { summary = value; }, + setRequest: (account: string) => { + summary = { + ...summary, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account }, + }, + }, + }, + }; + sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run(JSON.stringify({ status: 'suspended', request: { account } })); + }, }; } @@ -163,13 +192,9 @@ describe('remote step journal dispatch admission', () => { ).rejects.toMatchObject({ status: 409, }); - await h.journal.closePermit(permit); - const renewed = await h.journal.issuePermit(invocation.invocationId, 1_000); - expect(renewed.generation).toBe(2); - await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ - status: 409, - }); - expect(await h.journal.consumePermit(renewed)).toBe('accepted'); + expect( + (await h.journal.get(invocation.invocationId))?.permitConsumedAt, + ).toBeDefined(); }); it('recovers one permit after a journal instance is replaced', async () => { @@ -184,6 +209,99 @@ describe('remote step journal dispatch admission', () => { expect(await h.reopen().consumePermit(permit)).toBe('already-consumed'); }); + it.each([ + 'issue', + 'consume', + ] as const)('refuses %s when the stored suspension request changes', async (operation) => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = + operation === 'consume' + ? await h.journal.issuePermit(invocation.invocationId, 1_000) + : undefined; + h.setRequest('other'); + await expect( + permit + ? h.journal.consumePermit(permit) + : h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('tightens a repeated registration to the earlier host deadline', async () => { + const h = await fixture(); + const first = await h.journal.register(ADDRESS, 10_000); + const repeated = await h.journal.register(ADDRESS, 1_500); + expect(repeated.invocationId).toBe(first.invocationId); + expect(repeated.deadlineAt).toBe(1_500); + h.setNow(1_501); + await expect( + h.journal.issuePermit(first.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('caps a permit when the host deadline tightens after preflight', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("status = 'dispatched', permit_generation")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + await h.journal.register(ADDRESS, 1_500); + return bound.all(); + }, + }; + }, + }; + }; + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + expect(permit.expiresAt).toBe(1_500); + h.setNow(1_501); + await expect(h.journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + }); + + it.each([ + 'issue', + 'consume', + ] as const)('refuses %s while a cancellation intent is persisted', async (operation) => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000); + const permit = + operation === 'consume' + ? await h.journal.issuePermit(invocation.invocationId, 1_000) + : undefined; + h.sqlite.prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?').run( + JSON.stringify({ + status: 'suspended', + requestContext: { + 'flowsafe.runLifecycle': { + version: 1, + revision: 2, + transitionIntent: { + status: 'cancelled', + requestedAt: 1_000, + replayPrincipals: [{ kind: 'human', id: 'operator-1' }], + }, + }, + }, + }), + ); + await expect( + permit + ? h.journal.consumePermit(permit) + : h.journal.issuePermit(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + it('rejects a remote request larger than its stored payload limit', async () => { const h = await fixture(); h.setSummary({ @@ -311,4 +429,101 @@ describe('remote step journal dispatch admission', () => { status: 409, }); }); + + it('refuses a real suspended snapshot after runtime persists cancellation intent', async () => { + const sqlite = openSqlite() as ReturnType & { + close(): void; + }; + databases.push(sqlite); + const db = sqliteUnitDatabase(sqlite); + const storage = createD1Storage({ binding: db }); + await storage.init(); + const fence = new ExecutionFenceStore(db); + await fence.seed('open'); + const { createWorkflow, createStep, runtime } = init( + { storage }, + { executionFence: fence, startIdempotency: 'none' }, + ); + const gate = createStep({ + id: 'durable-agentic-execution', + inputSchema: z.object({}), + outputSchema: z.object({}), + suspendSchema: z.object({ + type: z.literal('suspension'), + toolCallId: z.string(), + toolName: z.string(), + toolCallSuspended: z.object({ + remoteStep: z.literal(REMOTE_STEP_MARKER), + request: z.object({ account: z.string() }), + }), + }), + execute: async ({ suspend }) => + suspend({ + type: 'suspension', + toolCallId: 'call-1', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }), + }); + createWorkflow({ + id: ADDRESS.workflowId, + inputSchema: z.object({}), + outputSchema: z.object({}), + }) + .then(gate) + .commit(); + const started = await runtime.start(ADDRESS.workflowId, { + runId: ADDRESS.runId, + inputData: {}, + requestedBy: 'operator-1', + requestedByKind: 'human', + startIdentity: { + owner: { kind: 'human', id: 'operator-1' }, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + }, + }, + agentStart: { threaded: false }, + }); + expect(started.status).toBe('suspended'); + const journal = new RemoteStepJournal({ + db, + fence, + runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + }); + await journal.initialize(); + const invocation = await journal.register(ADDRESS, Date.now() + 60_000); + const permit = await journal.issuePermit(invocation.invocationId, 30_000); + expect(permit.generation).toBe(1); + expect( + await runtime.cancelActiveExecution( + ADDRESS.workflowId, + ADDRESS.runId, + 'cancelled', + [{ kind: 'human', id: 'operator-1' }], + ), + ).toBe(false); + expect( + await runtime.authoritativeStatus(ADDRESS.workflowId, ADDRESS.runId), + ).toMatchObject({ status: 'suspended' }); + const stored = sqlite + .prepare( + `SELECT json_type(snapshot, + '$.requestContext."flowsafe.runLifecycle".transitionIntent') AS intent + FROM mastra_workflow_snapshot WHERE workflow_name = ? AND run_id = ?`, + ) + .get(ADDRESS.workflowId, ADDRESS.runId) as { intent: string }; + expect(stored.intent).toBe('object'); + await expect(journal.consumePermit(permit)).rejects.toMatchObject({ + status: 409, + }); + }); }); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 9ae204f2..bcd04c21 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -12,6 +12,10 @@ import { } from './execution-fence.js'; import { resourceIdFromKey } from './memory-id.js'; import { isPathSafeId } from './path-safe-id.js'; +import { + lifecycleFromRequestContext, + RUN_LIFECYCLE_CONTEXT_KEY, +} from './run-lifecycle.js'; import type { RunnerRuntime, RunSummary } from './runtime.js'; import { validateTablePrefix } from './table-prefix.js'; @@ -19,6 +23,10 @@ export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; +const SNAPSHOT_EXECUTION_OPEN_SQL = `json_valid(snapshot) + AND json_type(snapshot, '$.requestContext."${RUN_LIFECYCLE_CONTEXT_KEY}".transitionIntent') IS NULL + AND json_type(snapshot, '$.requestContext."${RUN_LIFECYCLE_CONTEXT_KEY}".terminal') IS NULL`; + export interface RemoteStepAddress { deploymentId: string; tablePrefix: string; @@ -42,7 +50,6 @@ export interface RemoteStepInvocation extends RemoteStepAddress { permitRevision?: number; permitExpiresAt?: number; permitConsumedAt?: number; - permitClosedAt?: number; } export interface RemoteStepPermit { @@ -83,7 +90,6 @@ type JournalRow = { permit_revision: number | null; permit_expires_at: number | null; permit_consumed_at: number | null; - permit_closed_at: number | null; }; function record(value: unknown): Record | undefined { @@ -171,9 +177,6 @@ function decode(row: JournalRow): RemoteStepInvocation { ...(row.permit_consumed_at === null ? {} : { permitConsumedAt: row.permit_consumed_at }), - ...(row.permit_closed_at === null - ? {} - : { permitClosedAt: row.permit_closed_at }), }; } @@ -298,7 +301,6 @@ export class RemoteStepJournal { permit_revision INTEGER, permit_expires_at INTEGER, permit_consumed_at INTEGER, - permit_closed_at INTEGER, UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count) )`) @@ -335,6 +337,12 @@ export class RemoteStepJournal { async #stableSnapshot(address: RemoteStepAddress) { const before = await this.#current(address); const snapshot = await this.#snapshot(address); + const state = await this.#execution(address); + const lifecycle = lifecycleFromRequestContext( + state.snapshot.requestContext, + ); + if (lifecycle?.transitionIntent || lifecycle?.terminal) + throw new RemoteStepRefusal(409, 'remote step run is ending'); const after = await this.#current(address); if ( before.suspendedAt !== after.suspendedAt || @@ -362,7 +370,6 @@ export class RemoteStepJournal { ) throw new RemoteStepRefusal(400, 'remote step address is malformed'); validateTablePrefix(address.tablePrefix); - await this.#execution(address); const { suspension, snapshot } = await this.#stableSnapshot(address); const summary = await this.#runtime.authoritativeStatus( address.workflowId, @@ -383,7 +390,8 @@ export class RemoteStepJournal { request_json, deadline_at, status) SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending' WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" - WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) ON CONFLICT DO NOTHING`) .bind( invocationId, @@ -405,11 +413,18 @@ export class RemoteStepJournal { ) .run(); const result = await this.#db - .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET deadline_at = MIN(deadline_at, ?) WHERE deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? - AND step_path = ? AND suspended_at = ? AND resume_count = ?`) + AND step_path = ? AND suspended_at = ? AND resume_count = ? + AND request_json = ? + AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + RETURNING *`) .bind( + deadlineAt, address.deploymentId, address.tablePrefix, address.workflowId, @@ -420,6 +435,10 @@ export class RemoteStepJournal { stepPath, suspension.suspendedAt, suspension.resumeCount ?? -1, + suspension.requestJson, + address.workflowId, + address.runId, + snapshot, ) .all(); const row = rowResult(result); @@ -461,12 +480,12 @@ export class RemoteStepJournal { address.resourceId !== resourceIdFromKey(address.threadId) ) throw new RemoteStepRefusal(409, 'remote step run identity changed'); - return state.execution; + return state; } async #admission(invocation: RemoteStepInvocation, priorBindings: number) { const observed = await this.#fence.readForAdmission(); - const proof = await this.#execution(invocation); + const proof = (await this.#execution(invocation)).execution; if (!admitsExistingRun(observed.reading, proof)) throw new RemoteStepRefusal(503, 'remote step execution is fenced'); const schema = await captureExecutionFenceAdmissionSchema( @@ -509,11 +528,12 @@ export class RemoteStepJournal { if ( suspension.suspendedAt !== invocation.suspendedAt || suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || JSON.stringify(suspension.stepPath) !== JSON.stringify(invocation.stepPath) ) throw new RemoteStepRefusal(409, 'remote step suspension changed'); - const gate = await this.#admission(invocation, 9); + const gate = await this.#admission(invocation, 10); if ( invocation.status === 'dispatched' && invocation.permitConsumedAt === undefined && @@ -531,8 +551,7 @@ export class RemoteStepJournal { }; if ( invocation.status === 'failed' || - (invocation.permitConsumedAt !== undefined && - invocation.permitClosedAt === undefined) + invocation.permitConsumedAt !== undefined ) throw new RemoteStepRefusal(409, 'remote step dispatch is active'); const epoch = gate.observed.mutationEpoch; @@ -542,13 +561,14 @@ export class RemoteStepJournal { const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = 'dispatched', permit_generation = permit_generation + 1, - permit_epoch = ?, permit_revision = ?, permit_expires_at = ?, - permit_consumed_at = NULL, permit_closed_at = NULL + permit_epoch = ?, permit_revision = ?, + permit_expires_at = MIN(?, deadline_at), permit_consumed_at = NULL WHERE invocation_id = ? AND permit_generation = ? AND deadline_at > ? AND status IN ('pending', 'dispatched') - AND (permit_consumed_at IS NULL OR permit_closed_at IS NOT NULL) + AND permit_consumed_at IS NULL AND request_json = ? AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" - WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) AND ${gate.predicate} RETURNING *`) .bind( epoch, @@ -557,6 +577,7 @@ export class RemoteStepJournal { invocationId, invocation.permitGeneration, now, + suspension.requestJson, invocation.workflowId, invocation.runId, snapshot, @@ -570,7 +591,7 @@ export class RemoteStepJournal { generation: row.permit_generation, mutationEpoch: epoch, transitionRevision: revision, - expiresAt, + expiresAt: row.permit_expires_at as number, }; } @@ -592,20 +613,22 @@ export class RemoteStepJournal { if ( suspension.suspendedAt !== invocation.suspendedAt || suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || JSON.stringify(suspension.stepPath) !== JSON.stringify(invocation.stepPath) ) throw new RemoteStepRefusal(409, 'remote step suspension changed'); - const gate = await this.#admission(invocation, 11); + const gate = await this.#admission(invocation, 12); const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET permit_consumed_at = ? WHERE invocation_id = ? AND permit_generation = ? AND permit_epoch = ? AND permit_revision = ? AND permit_expires_at = ? AND permit_expires_at > ? AND deadline_at > ? AND permit_consumed_at IS NULL - AND status = 'dispatched' + AND status = 'dispatched' AND request_json = ? AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" - WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) AND ${gate.predicate} RETURNING invocation_id`) .bind( now, @@ -616,6 +639,7 @@ export class RemoteStepJournal { permit.expiresAt, now, now, + suspension.requestJson, invocation.workflowId, invocation.runId, snapshot, @@ -631,21 +655,4 @@ export class RemoteStepJournal { return 'already-consumed'; throw new RemoteStepRefusal(409, 'remote step permit is unavailable'); } - - async closePermit(permit: RemoteStepPermit): Promise { - const result = await this.#db - .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET permit_closed_at = ? - WHERE invocation_id = ? AND permit_generation = ? - AND permit_consumed_at IS NOT NULL AND permit_closed_at IS NULL - RETURNING invocation_id`) - .bind(this.#now(), permit.invocationId, permit.generation) - .all(); - if (rowResult(result)) return; - const current = await this.get(permit.invocationId); - if ( - current?.permitGeneration !== permit.generation || - current.permitClosedAt === undefined - ) - throw new RemoteStepRefusal(409, 'remote step permit close conflicts'); - } } From 13f426dbc9748b41a4cb041aecc4ec88467e2394 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 12:49:13 +0400 Subject: [PATCH 06/73] Align direct proof bounds with remote-step inventory Accept the new FlowSafe work category in Fleet scenario evidence and keep gate and inventory tests aligned with the expanded drain surface. --- .../scripts/direct-credentialed-run-state.mjs | 2 +- .../test/direct-reference-fence.harness.test.ts | 7 ++++--- packages/flowsafe/src/execution-entry-matrix.test.ts | 5 +++++ packages/flowsafe/src/host-kit/flowsafe-worker.test.ts | 4 +++- 4 files changed, 13 insertions(+), 5 deletions(-) diff --git a/packages/fleet-control/scripts/direct-credentialed-run-state.mjs b/packages/fleet-control/scripts/direct-credentialed-run-state.mjs index bdb4c2de..25d31e46 100644 --- a/packages/fleet-control/scripts/direct-credentialed-run-state.mjs +++ b/packages/fleet-control/scripts/direct-credentialed-run-state.mjs @@ -85,7 +85,7 @@ export const DIRECT_SCENARIO_ARRAY_MAXIMA = Object.freeze({ auditFindings: 16, footprintVersionIds: 8, deploymentVersions: 2, - inventoryCategories: 9, + inventoryCategories: 10, inventory: Object.freeze({ databaseIds: 2, namespaceIds: 4, diff --git a/packages/fleet-control/test/direct-reference-fence.harness.test.ts b/packages/fleet-control/test/direct-reference-fence.harness.test.ts index e83883b7..fff7b894 100644 --- a/packages/fleet-control/test/direct-reference-fence.harness.test.ts +++ b/packages/fleet-control/test/direct-reference-fence.harness.test.ts @@ -260,15 +260,16 @@ describe.sequential('reference fence composition at release 1', { }); }); - it('counts the fence, index and category reads as eleven maintenance subrequests', async () => { + it('counts the fence, index and category reads as maintenance subrequests', async () => { const { response, value } = await fixture.call({ kind: 'tenant-fence', role: 'a', operation: 'inventory', }); expect(value.ok).toBe(true); - expect(2 + INVENTORY_CATEGORIES.length).toBe(11); - expect(response.headers.get('X-Direct-Maintenance-Attempts')).toBe('11'); + expect(response.headers.get('X-Direct-Maintenance-Attempts')).toBe( + String(2 + INVENTORY_CATEGORIES.length), + ); }); it('replays numeric and null fence bindings through the database clone', async () => { diff --git a/packages/flowsafe/src/execution-entry-matrix.test.ts b/packages/flowsafe/src/execution-entry-matrix.test.ts index 616306ca..ca60b182 100644 --- a/packages/flowsafe/src/execution-entry-matrix.test.ts +++ b/packages/flowsafe/src/execution-entry-matrix.test.ts @@ -1290,6 +1290,11 @@ type GateSite = { }; const GATE_SITES: ReadonlyArray = [ + { + file: 'do-runner/remote-step-journal.ts', + predicate: 'admitsExistingRun', + drivenBy: 'do-runner/remote-step-journal.test.ts', + }, { file: 'do-runner/execution-fence.ts', predicate: 'admitsExistingRun', diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts index 73d82598..02904ef0 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts @@ -3300,7 +3300,9 @@ describe('createFlowsafeWorker drain inventory', () => { unenumerable: Array<{ name: string }>; drainProof: { reading: string; reachableFrom: string[] }; }; - expect(body.categories.filter((c) => c.class === 'work')).toHaveLength(7); + expect(body.categories).toContainEqual( + expect.objectContaining({ category: 'remote-steps', class: 'work' }), + ); expect(body.categories.filter((c) => c.class === 'standing')).toHaveLength( 2, ); From 2afc8fff1083f7f21dc4a0272973474fb0204847 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 13:11:34 +0400 Subject: [PATCH 07/73] Bind remote results to one consumed dispatch permit Return opaque authority only to the consume winner, stage bounded exact-byte results after trusted completion proof, and retain unresolved work for drain. Host continuation remains separate. --- packages/flowsafe/src/do-runner/inventory.ts | 3 +- .../src/do-runner/mastra-schema-guard.test.ts | 1 + .../src/do-runner/remote-step-journal.test.ts | 281 +++++++++++++-- .../src/do-runner/remote-step-journal.ts | 337 +++++++++++++++++- 4 files changed, 580 insertions(+), 42 deletions(-) diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index 8b985482..d31436a9 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -1032,7 +1032,8 @@ export class DeploymentInventory { 'permit_generation', 'permit_consumed_at', ], - where: "status <> 'failed' OR permit_consumed_at IS NOT NULL", + where: + "status NOT IN ('failed', 'settled') OR (permit_consumed_at IS NOT NULL AND executor_quiescent_at IS NULL)", binds: [], }; case 'approvals-waiting': diff --git a/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts b/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts index bf7768c8..dcbe5f70 100644 --- a/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts +++ b/packages/flowsafe/src/do-runner/mastra-schema-guard.test.ts @@ -495,6 +495,7 @@ describe('Mastra persistence guards (D1Store SQL over node:sqlite)', () => { deploymentId: 'census-deployment', tablePrefix: '', allowedToolIds: ['census-tool'], + verifyCompletion: async () => undefined, }).initialize(); sqlite.exec( `CREATE TABLE IF NOT EXISTS ${DEPLOYMENT_SENTINEL_TABLE} ( diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index dfa5d5fb..c25ea7ff 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -1,6 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 -import { afterEach, describe, expect, it } from 'vitest'; +import { afterEach, describe, expect, it, vi } from 'vitest'; import { z } from 'zod'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; import { createD1Storage } from './d1-storage.js'; @@ -23,6 +23,11 @@ const ADDRESS = { resourceId: 'thread-1', runId: 'run-1', }; +const PRINCIPAL = { + kind: 'human', + id: 'operator-1', + role: 'operator', +} as const; describe('remote step journal dispatch admission', () => { const databases: Array<{ close(): void }> = []; @@ -82,6 +87,7 @@ describe('remote step journal dispatch admission', () => { }, provenance: { startIdentity: { + owner: { kind: PRINCIPAL.kind, id: PRINCIPAL.id }, target: { kind: 'agent', id: ADDRESS.agentId, @@ -94,6 +100,7 @@ describe('remote step journal dispatch admission', () => { RunnerRuntime, 'authoritativeStatus' | 'authoritativeStartState' >; + const verifyCompletion = vi.fn(async () => undefined); const journal = new RemoteStepJournal({ db, fence, @@ -101,6 +108,7 @@ describe('remote step journal dispatch admission', () => { deploymentId: ADDRESS.deploymentId, tablePrefix: ADDRESS.tablePrefix, allowedToolIds: ['remote_lookup'], + verifyCompletion, now: () => now, }); await journal.initialize(); @@ -108,6 +116,7 @@ describe('remote step journal dispatch admission', () => { db, fence, journal, + verifyCompletion, runtime, reopen: () => new RemoteStepJournal({ @@ -117,6 +126,7 @@ describe('remote step journal dispatch admission', () => { deploymentId: ADDRESS.deploymentId, tablePrefix: ADDRESS.tablePrefix, allowedToolIds: ['remote_lookup'], + verifyCompletion, now: () => now, }), sqlite, @@ -151,8 +161,8 @@ describe('remote step journal dispatch admission', () => { it('converges repeated observation on one invocation without a permit', async () => { const h = await fixture(); const [first, replay] = await Promise.all([ - h.journal.register(ADDRESS, 10_000), - h.journal.register(ADDRESS, 10_000), + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + h.journal.register(ADDRESS, 10_000, PRINCIPAL), ]); expect(replay.invocationId).toBe(first.invocationId); expect(first).toMatchObject({ @@ -180,18 +190,42 @@ describe('remote step journal dispatch admission', () => { ]); }); - it('accepts one permit consumption and refuses replay as effect authority', async () => { + it('binds registration to the original execution principal', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + expect(invocation.principal).toEqual(PRINCIPAL); + await expect( + h.journal.register(ADDRESS, 10_000, { + kind: 'human', + id: 'other', + role: 'admin', + }), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('returns raw effect authority only to the winning consume call', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); expect(permit).toMatchObject({ generation: 1, expiresAt: 2_000 }); - expect(await h.journal.consumePermit(permit)).toBe('accepted'); - expect(await h.journal.consumePermit(permit)).toBe('already-consumed'); + const winner = await h.journal.consumePermit(permit); + expect(winner).toMatchObject({ + kind: 'accepted', + rawPermit: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/), + }); + expect(await h.journal.consumePermit(permit)).toEqual({ + kind: 'already-consumed', + }); + const stored = h.sqlite + .prepare(`SELECT permit_digest FROM ${REMOTE_STEP_TABLE}`) + .get() as { permit_digest: string }; + expect(stored.permit_digest).toMatch(/^[a-f0-9]{64}$/); + expect(JSON.stringify(stored)).not.toContain( + (winner as { rawPermit: string }).rawPermit, + ); await expect( h.journal.issuePermit(invocation.invocationId, 1_000), - ).rejects.toMatchObject({ - status: 409, - }); + ).rejects.toMatchObject({ status: 409 }); expect( (await h.journal.get(invocation.invocationId))?.permitConsumedAt, ).toBeDefined(); @@ -199,14 +233,199 @@ describe('remote step journal dispatch admission', () => { it('recovers one permit after a journal instance is replaced', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); const recovered = h.reopen(); expect(await recovered.issuePermit(invocation.invocationId, 1_000)).toEqual( permit, ); - expect(await recovered.consumePermit(permit)).toBe('accepted'); - expect(await h.reopen().consumePermit(permit)).toBe('already-consumed'); + expect(await recovered.consumePermit(permit)).toMatchObject({ + kind: 'accepted', + rawPermit: expect.any(String), + }); + expect(await h.reopen().consumePermit(permit)).toEqual({ + kind: 'already-consumed', + }); + }); + + it('stores one verified JSON result and replays only the same bytes', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + const bytes = '{ "value": "done" }'; + const evidence = '{"receipt":"verified-completion"}'; + expect( + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + bytes, + evidence, + ), + ).toMatchObject({ kind: 'recorded', resultJson: bytes }); + expect(h.verifyCompletion).toHaveBeenCalledOnce(); + expect(h.verifyCompletion).toHaveBeenCalledWith( + expect.objectContaining({ + invocation: expect.objectContaining({ + invocationId: invocation.invocationId, + }), + rawPermit: consumed.rawPermit, + resultJson: bytes, + evidenceJson: evidence, + }), + ); + expect( + await h + .reopen() + .recordResult( + invocation.invocationId, + consumed.rawPermit, + bytes, + evidence, + ), + ).toMatchObject({ kind: 'replay', resultJson: bytes }); + expect(h.verifyCompletion).toHaveBeenCalledOnce(); + await expect( + h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + evidence, + ), + ).rejects.toMatchObject({ status: 409 }); + await expect( + h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + bytes, + '{"receipt":"different"}', + ), + ).rejects.toMatchObject({ status: 409 }); + const stored = h.sqlite + .prepare( + `SELECT result_json, result_digest, executor_quiescent_at + FROM ${REMOTE_STEP_TABLE}`, + ) + .get() as Record; + expect(stored).toMatchObject({ + result_json: bytes, + result_digest: expect.stringMatching(/^[a-f0-9]{64}$/), + executor_quiescent_at: expect.any(Number), + }); + expect( + await h.journal.readRecordedResult(invocation.invocationId), + ).toMatchObject({ + resultJson: bytes, + resultDigest: stored.result_digest, + }); + }); + + it('settles a verified result only after the physical run advances', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ); + await expect( + h.journal.settleAfterResume(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + h.setSummary({ runId: ADDRESS.runId, status: 'success' }); + h.sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run('{"status":"success"}'); + expect( + await h.journal.settleAfterResume(invocation.invocationId), + ).toMatchObject({ + status: 'settled', + }); + expect( + (await h.reopen().settleAfterResume(invocation.invocationId)).status, + ).toBe('settled'); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).entries, + ).toEqual([]); + }); + + it('records verified completion from a prior epoch without resuming a locked run', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + expect( + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'recorded' }); + await expect( + h.journal.settleAfterResume(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('keeps a consumed claim active if completion verification fails', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + h.verifyCompletion.mockRejectedValueOnce(new Error('unverified executor')); + await expect( + h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"unverified"}', + ), + ).rejects.toThrow('unverified executor'); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'dispatched', + ); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('refuses a wrong raw permit before calling the completion verifier', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + const wrong = + consumed.rawPermit === 'A'.repeat(43) ? 'B'.repeat(43) : 'A'.repeat(43); + await expect( + h.journal.recordResult( + invocation.invocationId, + wrong, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).rejects.toMatchObject({ status: 403 }); + expect(h.verifyCompletion).not.toHaveBeenCalled(); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'dispatched', + ); }); it.each([ @@ -214,7 +433,7 @@ describe('remote step journal dispatch admission', () => { 'consume', ] as const)('refuses %s when the stored suspension request changes', async (operation) => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = operation === 'consume' ? await h.journal.issuePermit(invocation.invocationId, 1_000) @@ -229,8 +448,8 @@ describe('remote step journal dispatch admission', () => { it('tightens a repeated registration to the earlier host deadline', async () => { const h = await fixture(); - const first = await h.journal.register(ADDRESS, 10_000); - const repeated = await h.journal.register(ADDRESS, 1_500); + const first = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const repeated = await h.journal.register(ADDRESS, 1_500, PRINCIPAL); expect(repeated.invocationId).toBe(first.invocationId); expect(repeated.deadlineAt).toBe(1_500); h.setNow(1_501); @@ -241,7 +460,7 @@ describe('remote step journal dispatch admission', () => { it('caps a permit when the host deadline tightens after preflight', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const prepare = h.db.prepare.bind(h.db); h.db.prepare = (sql) => { const statement = prepare(sql); @@ -254,7 +473,7 @@ describe('remote step journal dispatch admission', () => { return { ...bound, all: async () => { - await h.journal.register(ADDRESS, 1_500); + await h.journal.register(ADDRESS, 1_500, PRINCIPAL); return bound.all(); }, }; @@ -274,7 +493,7 @@ describe('remote step journal dispatch admission', () => { 'consume', ] as const)('refuses %s while a cancellation intent is persisted', async (operation) => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = operation === 'consume' ? await h.journal.issuePermit(invocation.invocationId, 1_000) @@ -321,7 +540,9 @@ describe('remote step journal dispatch admission', () => { }, }, }); - await expect(h.journal.register(ADDRESS, 10_000)).rejects.toMatchObject({ + await expect( + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + ).rejects.toMatchObject({ status: 400, }); }); @@ -335,15 +556,18 @@ describe('remote step journal dispatch admission', () => { deploymentId: ADDRESS.deploymentId, tablePrefix: ADDRESS.tablePrefix, allowedToolIds: ['other_tool'], + verifyCompletion: h.verifyCompletion, }); - await expect(restricted.register(ADDRESS, 10_000)).rejects.toMatchObject({ + await expect( + restricted.register(ADDRESS, 10_000, PRINCIPAL), + ).rejects.toMatchObject({ status: 503, }); }); it('refuses a corrupted persisted table identity before building a query', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); h.sqlite .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?`) .run('bad"_prefix'); @@ -356,7 +580,7 @@ describe('remote step journal dispatch admission', () => { it('refuses an unconsumed permit after the fence moves to migration lock', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); const before = await h.fence.read(); await h.fence.transition({ @@ -376,7 +600,7 @@ describe('remote step journal dispatch admission', () => { it('refuses a permit when the fence moves after preflight but before the write', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); const prepare = h.db.prepare.bind(h.db); h.db.prepare = (sql) => { @@ -413,7 +637,7 @@ describe('remote step journal dispatch admission', () => { it('refuses dispatch after cancellation or deadline', async () => { const h = await fixture(); - const invocation = await h.journal.register(ADDRESS, 10_000); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); h.sqlite @@ -498,9 +722,14 @@ describe('remote step journal dispatch admission', () => { deploymentId: ADDRESS.deploymentId, tablePrefix: ADDRESS.tablePrefix, allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, }); await journal.initialize(); - const invocation = await journal.register(ADDRESS, Date.now() + 60_000); + const invocation = await journal.register( + ADDRESS, + Date.now() + 60_000, + PRINCIPAL, + ); const permit = await journal.issuePermit(invocation.invocationId, 30_000); expect(permit.generation).toBe(1); expect( diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index bcd04c21..7feddd8b 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -1,6 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 import { EXECUTION_FENCE_TABLE } from '#deployment-identity-protocol'; +import { + assertExecutionPrincipal, + type ExecutionPrincipal, +} from '../approval-api/principal.js'; +import { credentialsMatch } from './deployment-identity.js'; import { DoStatusError } from './do-status-error.js'; import { admitsExistingRun, @@ -16,12 +21,50 @@ import { lifecycleFromRequestContext, RUN_LIFECYCLE_CONTEXT_KEY, } from './run-lifecycle.js'; +import { isTerminalRunStatus } from './run-terminal-state.js'; import type { RunnerRuntime, RunSummary } from './runtime.js'; import { validateTablePrefix } from './table-prefix.js'; export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; +export const MAX_REMOTE_STEP_RESULT_BYTES = 16_384; + +const SHA256_HEX = /^[a-f0-9]{64}$/; +const RAW_PERMIT = /^[A-Za-z0-9_-]{43}$/; + +function boundedJsonBytes(value: string): void { + if ( + typeof value !== 'string' || + new TextEncoder().encode(value).length > MAX_REMOTE_STEP_RESULT_BYTES + ) + throw new RemoteStepRefusal(400, 'remote step JSON exceeds its limit'); + try { + JSON.parse(value); + } catch { + throw new RemoteStepRefusal(400, 'remote step JSON is malformed'); + } +} + +function rawDispatchPermit(): string { + const bytes = crypto.getRandomValues(new Uint8Array(32)); + let binary = ''; + for (const byte of bytes) binary += String.fromCharCode(byte); + return btoa(binary) + .replace(/\+/g, '-') + .replace(/\//g, '_') + .replace(/=+$/, ''); +} + +async function sha256Hex(value: string): Promise { + const digest = await crypto.subtle.digest( + 'SHA-256', + new TextEncoder().encode(value), + ); + return Array.from(new Uint8Array(digest), (byte) => + byte.toString(16).padStart(2, '0'), + ).join(''); +} const SNAPSHOT_EXECUTION_OPEN_SQL = `json_valid(snapshot) AND json_type(snapshot, '$.requestContext."${RUN_LIFECYCLE_CONTEXT_KEY}".transitionIntent') IS NULL @@ -39,12 +82,14 @@ export interface RemoteStepAddress { export interface RemoteStepInvocation extends RemoteStepAddress { invocationId: string; + principal: ExecutionPrincipal; + startToken: string; stepPath: readonly string[]; suspendedAt: number; resumeCount?: number; request: unknown; deadlineAt: number; - status: 'pending' | 'dispatched' | 'failed'; + status: 'pending' | 'dispatched' | 'result-recorded' | 'settled' | 'failed'; permitGeneration: number; permitEpoch?: number; permitRevision?: number; @@ -60,6 +105,14 @@ export interface RemoteStepPermit { expiresAt: number; } +export interface RecordedRemoteStepResult { + invocationId: string; + resultJson: string; + resultDigest: string; + quiescenceDigest: string; + executorQuiescentAt: number; +} + class RemoteStepRefusal extends DoStatusError { readonly status: number; @@ -72,6 +125,8 @@ class RemoteStepRefusal extends DoStatusError { type JournalRow = { invocation_id: string; + principal_json: string; + start_token: string; deployment_id: string; table_prefix: string; workflow_id: string; @@ -90,6 +145,11 @@ type JournalRow = { permit_revision: number | null; permit_expires_at: number | null; permit_consumed_at: number | null; + permit_digest: string | null; + result_json: string | null; + result_digest: string | null; + quiescence_digest: string | null; + executor_quiescent_at: number | null; }; function record(value: unknown): Record | undefined { @@ -114,6 +174,7 @@ function decode(row: JournalRow): RemoteStepInvocation { typeof row.step_path !== 'string' || row.step_path.length > 1_024 || typeof row.request_json !== 'string' || + typeof row.principal_json !== 'string' || new TextEncoder().encode(row.request_json).length > MAX_REMOTE_STEP_REQUEST_BYTES || !isPathSafeId(row.deployment_id) || @@ -121,7 +182,8 @@ function decode(row: JournalRow): RemoteStepInvocation { !isPathSafeId(row.agent_id) || !isPathSafeId(row.thread_id) || !isPathSafeId(row.resource_id) || - !isPathSafeId(row.run_id) + !isPathSafeId(row.run_id) || + !isPathSafeId(row.start_token) ) throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); try { @@ -131,9 +193,14 @@ function decode(row: JournalRow): RemoteStepInvocation { } let stepPath: unknown; let request: unknown; + let principal: ExecutionPrincipal; try { stepPath = JSON.parse(row.step_path); request = JSON.parse(row.request_json); + principal = assertExecutionPrincipal( + JSON.parse(row.principal_json), + 'stored remote step principal', + ); } catch { throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); } @@ -148,11 +215,15 @@ function decode(row: JournalRow): RemoteStepInvocation { (row.resume_count !== -1 && row.resume_count < 1) || !Number.isSafeInteger(row.permit_generation) || row.permit_generation < 0 || - !['pending', 'dispatched', 'failed'].includes(row.status) + !['pending', 'dispatched', 'result-recorded', 'settled', 'failed'].includes( + row.status, + ) ) throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); return { invocationId: row.invocation_id, + principal, + startToken: row.start_token, deploymentId: row.deployment_id, tablePrefix: row.table_prefix, workflowId: row.workflow_id, @@ -245,6 +316,12 @@ export class RemoteStepJournal { readonly #deploymentId: string; readonly #tablePrefix: string; readonly #allowedToolIds: ReadonlySet; + readonly #verifyCompletion: (input: { + invocation: RemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; constructor(input: { db: ExecutionFenceDatabase; @@ -256,6 +333,12 @@ export class RemoteStepJournal { deploymentId: string; tablePrefix: string; allowedToolIds: readonly string[]; + verifyCompletion: (input: { + invocation: RemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; now?: () => number; }) { if (!input.fence.usesDatabase(input.db)) @@ -270,12 +353,15 @@ export class RemoteStepJournal { new Set(input.allowedToolIds).size !== input.allowedToolIds.length ) throw new Error('remote step tool ids are malformed'); + if (typeof input.verifyCompletion !== 'function') + throw new Error('remote step completion verifier is required'); this.#db = input.db; this.#fence = input.fence; this.#runtime = input.runtime; this.#deploymentId = input.deploymentId; this.#tablePrefix = input.tablePrefix; this.#allowedToolIds = new Set(input.allowedToolIds); + this.#verifyCompletion = input.verifyCompletion; this.#now = input.now ?? Date.now; } @@ -283,6 +369,8 @@ export class RemoteStepJournal { await this.#db .prepare(`CREATE TABLE IF NOT EXISTS ${REMOTE_STEP_TABLE} ( invocation_id TEXT PRIMARY KEY, + principal_json TEXT NOT NULL, + start_token TEXT NOT NULL, deployment_id TEXT NOT NULL, table_prefix TEXT NOT NULL, workflow_id TEXT NOT NULL, @@ -301,6 +389,11 @@ export class RemoteStepJournal { permit_revision INTEGER, permit_expires_at INTEGER, permit_consumed_at INTEGER, + permit_digest TEXT, + result_json TEXT, + result_digest TEXT, + quiescence_digest TEXT, + executor_quiescent_at INTEGER, UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count) )`) @@ -357,6 +450,7 @@ export class RemoteStepJournal { async register( address: RemoteStepAddress, hostDeadlineAt: number, + originalPrincipal: ExecutionPrincipal, ): Promise { if ( address.deploymentId !== this.#deploymentId || @@ -370,6 +464,16 @@ export class RemoteStepJournal { ) throw new RemoteStepRefusal(400, 'remote step address is malformed'); validateTablePrefix(address.tablePrefix); + let principal: ExecutionPrincipal; + try { + principal = assertExecutionPrincipal( + originalPrincipal, + 'remote step principal', + ); + } catch { + throw new RemoteStepRefusal(400, 'remote step principal is malformed'); + } + const execution = await this.#execution(address, principal); const { suspension, snapshot } = await this.#stableSnapshot(address); const summary = await this.#runtime.authoritativeStatus( address.workflowId, @@ -385,16 +489,18 @@ export class RemoteStepJournal { const stepPath = JSON.stringify(suspension.stepPath); await this.#db .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( - invocation_id, deployment_id, table_prefix, workflow_id, agent_id, + invocation_id, principal_json, start_token, deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count, request_json, deadline_at, status) - SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending' + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending' WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) ON CONFLICT DO NOTHING`) .bind( invocationId, + JSON.stringify(principal), + execution.execution.startToken, address.deploymentId, address.tablePrefix, address.workflowId, @@ -418,7 +524,7 @@ export class RemoteStepJournal { WHERE deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? AND step_path = ? AND suspended_at = ? AND resume_count = ? - AND request_json = ? + AND request_json = ? AND principal_json = ? AND start_token = ? AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) @@ -436,6 +542,8 @@ export class RemoteStepJournal { suspension.suspendedAt, suspension.resumeCount ?? -1, suspension.requestJson, + JSON.stringify(principal), + execution.execution.startToken, address.workflowId, address.runId, snapshot, @@ -462,7 +570,7 @@ export class RemoteStepJournal { return row ? decode(row) : undefined; } - async #execution(address: RemoteStepAddress) { + async #execution(address: RemoteStepAddress, principal?: ExecutionPrincipal) { const state = await this.#runtime.authoritativeStartState( address.workflowId, address.runId, @@ -476,6 +584,9 @@ export class RemoteStepJournal { if ( target.id !== address.agentId || target.threadId !== address.threadId || + (principal !== undefined && + (state.provenance.startIdentity?.owner.kind !== principal.kind || + state.provenance.startIdentity.owner.id !== principal.id)) || state.execution.tablePrefix !== address.tablePrefix || address.resourceId !== resourceIdFromKey(address.threadId) ) @@ -485,7 +596,10 @@ export class RemoteStepJournal { async #admission(invocation: RemoteStepInvocation, priorBindings: number) { const observed = await this.#fence.readForAdmission(); - const proof = (await this.#execution(invocation)).execution; + const proof = (await this.#execution(invocation, invocation.principal)) + .execution; + if (proof.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); if (!admitsExistingRun(observed.reading, proof)) throw new RemoteStepRefusal(503, 'remote step execution is fenced'); const schema = await captureExecutionFenceAdmissionSchema( @@ -562,7 +676,8 @@ export class RemoteStepJournal { .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = 'dispatched', permit_generation = permit_generation + 1, permit_epoch = ?, permit_revision = ?, - permit_expires_at = MIN(?, deadline_at), permit_consumed_at = NULL + permit_expires_at = MIN(?, deadline_at), permit_consumed_at = NULL, + permit_digest = NULL WHERE invocation_id = ? AND permit_generation = ? AND deadline_at > ? AND status IN ('pending', 'dispatched') AND permit_consumed_at IS NULL AND request_json = ? @@ -597,7 +712,9 @@ export class RemoteStepJournal { async consumePermit( permit: RemoteStepPermit, - ): Promise<'accepted' | 'already-consumed'> { + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + > { const invocation = await this.get(permit.invocationId); if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); if ( @@ -607,7 +724,8 @@ export class RemoteStepJournal { invocation.permitExpiresAt !== permit.expiresAt ) throw new RemoteStepRefusal(409, 'remote step permit changed'); - if (invocation.permitConsumedAt !== undefined) return 'already-consumed'; + if (invocation.permitConsumedAt !== undefined) + return { kind: 'already-consumed' }; const now = this.#now(); const { suspension, snapshot } = await this.#stableSnapshot(invocation); if ( @@ -618,9 +736,12 @@ export class RemoteStepJournal { JSON.stringify(invocation.stepPath) ) throw new RemoteStepRefusal(409, 'remote step suspension changed'); - const gate = await this.#admission(invocation, 12); + const rawPermit = rawDispatchPermit(); + const permitDigest = await sha256Hex(rawPermit); + const gate = await this.#admission(invocation, 13); const result = await this.#db - .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET permit_consumed_at = ? + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET permit_consumed_at = ?, permit_digest = ? WHERE invocation_id = ? AND permit_generation = ? AND permit_epoch = ? AND permit_revision = ? AND permit_expires_at = ? AND permit_expires_at > ? @@ -632,6 +753,7 @@ export class RemoteStepJournal { AND ${gate.predicate} RETURNING invocation_id`) .bind( now, + permitDigest, permit.invocationId, permit.generation, permit.mutationEpoch, @@ -646,13 +768,198 @@ export class RemoteStepJournal { ...gate.values, ) .all(); - if (rowResult(result)) return 'accepted'; + if (rowResult(result)) return { kind: 'accepted', rawPermit }; const after = await this.get(permit.invocationId); if ( after?.permitGeneration === permit.generation && after.permitConsumedAt !== undefined ) - return 'already-consumed'; + return { kind: 'already-consumed' }; throw new RemoteStepRefusal(409, 'remote step permit is unavailable'); } + + async recordResult( + invocationId: string, + rawPermit: string, + resultJson: string, + evidenceJson: string, + ): Promise< + | { kind: 'recorded'; resultJson: string; resultDigest: string } + | { kind: 'replay'; resultJson: string; resultDigest: string } + > { + if (!isPathSafeId(invocationId) || !RAW_PERMIT.test(rawPermit)) + throw new RemoteStepRefusal( + 400, + 'remote step result identity is malformed', + ); + boundedJsonBytes(resultJson); + boundedJsonBytes(evidenceJson); + const stored = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + const invocation = decode(stored); + const permitDigest = stored.permit_digest; + if ( + typeof permitDigest !== 'string' || + !SHA256_HEX.test(permitDigest) || + invocation.permitConsumedAt === undefined + ) + throw new RemoteStepRefusal(409, 'remote step permit was not consumed'); + const presentedDigest = await sha256Hex(rawPermit); + if (!(await credentialsMatch(presentedDigest, permitDigest))) + throw new RemoteStepRefusal(403, 'remote step permit is invalid'); + const resultDigest = await sha256Hex(resultJson); + const quiescenceDigest = await sha256Hex(evidenceJson); + const replay = (row: JournalRow) => { + if ( + row.result_json !== resultJson || + row.result_digest !== resultDigest || + row.quiescence_digest !== quiescenceDigest || + !positiveTime(row.executor_quiescent_at) + ) + throw new RemoteStepRefusal(409, 'remote step result conflicts'); + return { kind: 'replay' as const, resultJson, resultDigest }; + }; + if (stored.result_json !== null) return replay(stored); + await this.#verifyCompletion({ + invocation, + rawPermit, + resultJson, + evidenceJson, + }); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET result_json = ?, result_digest = ?, quiescence_digest = ?, + executor_quiescent_at = ?, status = 'result-recorded' + WHERE invocation_id = ? AND permit_digest = ? + AND permit_consumed_at IS NOT NULL AND result_json IS NULL + AND status = 'dispatched' + RETURNING *`) + .bind( + resultJson, + resultDigest, + quiescenceDigest, + this.#now(), + invocationId, + permitDigest, + ) + .all(); + if (rowResult(result)) + return { kind: 'recorded', resultJson, resultDigest }; + const after = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!after) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + return replay(after); + } + + async readRecordedResult( + invocationId: string, + ): Promise { + if (!isPathSafeId(invocationId)) + throw new RemoteStepRefusal(400, 'invocation id is malformed'); + const row = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!row) return undefined; + decode(row); + const resultJson = row.result_json; + const resultDigest = row.result_digest; + const quiescenceDigest = row.quiescence_digest; + const executorQuiescentAt = row.executor_quiescent_at; + if (resultJson === null) return undefined; + if ( + !['result-recorded', 'settled'].includes(row.status) || + row.permit_consumed_at === null || + typeof resultDigest !== 'string' || + !SHA256_HEX.test(resultDigest) || + typeof quiescenceDigest !== 'string' || + !SHA256_HEX.test(quiescenceDigest) || + !positiveTime(executorQuiescentAt) + ) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + try { + boundedJsonBytes(resultJson); + } catch { + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + } + if ((await sha256Hex(resultJson)) !== resultDigest) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + return { + invocationId, + resultJson, + resultDigest, + quiescenceDigest, + executorQuiescentAt, + }; + } + + async settleAfterResume(invocationId: string): Promise { + const invocation = await this.get(invocationId); + if ( + !invocation || + !['result-recorded', 'settled'].includes(invocation.status) + ) + throw new RemoteStepRefusal(409, 'remote step has no result to settle'); + const result = await this.readRecordedResult(invocationId); + if (!result) + throw new RemoteStepRefusal(503, 'remote step result is unreadable'); + const state = await this.#execution(invocation, invocation.principal); + if (state.execution.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); + const summary = await this.#runtime.authoritativeStatus( + invocation.workflowId, + invocation.runId, + ); + if (!summary) + throw new RemoteStepRefusal(503, 'remote step run is unreadable'); + if (summary.status === 'suspended') { + const key = invocation.stepPath.join('.'); + const stillCurrent = + (summary.suspended ?? []).some( + (path) => + JSON.stringify(path) === JSON.stringify(invocation.stepPath), + ) && + summary.suspendedAt?.[key] === invocation.suspendedAt && + summary.resumeCount?.[key] === invocation.resumeCount; + if (stillCurrent) + throw new RemoteStepRefusal(409, 'remote step has not resumed'); + } else if (!isTerminalRunStatus(summary.status)) { + throw new RemoteStepRefusal(409, 'remote step resume is still active'); + } + if (invocation.status === 'settled') return invocation; + const snapshot = await this.#snapshot(invocation); + const written = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = 'settled' + WHERE invocation_id = ? AND status = 'result-recorded' + AND result_digest = ? AND executor_quiescent_at IS NOT NULL + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) + RETURNING *`) + .bind( + invocationId, + result.resultDigest, + invocation.workflowId, + invocation.runId, + snapshot, + ) + .all(), + ); + if (written) return decode(written); + const after = await this.get(invocationId); + if (after?.status === 'settled') return after; + throw new RemoteStepRefusal(409, 'remote step settlement changed'); + } } From d1090721205c70ac56e17cb2f3244eef7613e8c3 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 13:24:25 +0400 Subject: [PATCH 08/73] Retain remote results until exact continuation proof Store verified results as drain work, remove unproven settlement, and exercise concurrent permit consumption. A later host continuation must prove this result advanced the run. --- packages/flowsafe/src/do-runner/inventory.ts | 2 +- .../src/do-runner/remote-step-journal.test.ts | 71 ++++++++++++++----- .../src/do-runner/remote-step-journal.ts | 71 ++----------------- 3 files changed, 59 insertions(+), 85 deletions(-) diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index d31436a9..a082d656 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -1033,7 +1033,7 @@ export class DeploymentInventory { 'permit_consumed_at', ], where: - "status NOT IN ('failed', 'settled') OR (permit_consumed_at IS NOT NULL AND executor_quiescent_at IS NULL)", + "status <> 'failed' OR (permit_consumed_at IS NOT NULL AND executor_quiescent_at IS NULL)", binds: [], }; case 'approvals-waiting': diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index c25ea7ff..e457e8e5 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -231,6 +231,22 @@ describe('remote step journal dispatch admission', () => { ).toBeDefined(); }); + it('grants raw authority to one concurrent D1 consume winner', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const outcomes = await Promise.all([ + h.journal.consumePermit(permit), + h.reopen().consumePermit(permit), + ]); + expect(outcomes.filter((item) => item.kind === 'accepted')).toHaveLength(1); + expect(outcomes.filter((item) => item.kind === 'already-consumed')).toEqual( + [{ kind: 'already-consumed' }], + ); + const winner = outcomes.find((item) => item.kind === 'accepted'); + expect(winner).toMatchObject({ rawPermit: expect.any(String) }); + }); + it('recovers one permit after a journal instance is replaced', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); @@ -314,14 +330,14 @@ describe('remote step journal dispatch admission', () => { executor_quiescent_at: expect.any(Number), }); expect( - await h.journal.readRecordedResult(invocation.invocationId), + await h.journal.readRetainedResult(invocation.invocationId), ).toMatchObject({ resultJson: bytes, resultDigest: stored.result_digest, }); }); - it('settles a verified result only after the physical run advances', async () => { + it('retains drain work after an unproven run advancement', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); @@ -333,24 +349,19 @@ describe('remote step journal dispatch admission', () => { '{"value":"done"}', '{"receipt":"verified"}', ); - await expect( - h.journal.settleAfterResume(invocation.invocationId), - ).rejects.toMatchObject({ status: 409 }); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); h.setSummary({ runId: ADDRESS.runId, status: 'success' }); h.sqlite .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') .run('{"status":"success"}'); + expect((await h.reopen().get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); expect( - await h.journal.settleAfterResume(invocation.invocationId), - ).toMatchObject({ - status: 'settled', - }); - expect( - (await h.reopen().settleAfterResume(invocation.invocationId)).status, - ).toBe('settled'); - expect( - (await new DeploymentInventory(h.db).read('remote-steps')).entries, - ).toEqual([]); + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); }); it('records verified completion from a prior epoch without resuming a locked run', async () => { @@ -375,9 +386,33 @@ describe('remote step journal dispatch admission', () => { '{"receipt":"verified"}', ), ).toMatchObject({ kind: 'recorded' }); - await expect( - h.journal.settleAfterResume(invocation.invocationId), - ).rejects.toMatchObject({ status: 409 }); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('retains a verified late result as drain work after cancellation', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + h.sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run('{"status":"cancelled"}'); + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'result-retained', + ); expect( (await new DeploymentInventory(h.db).read('remote-steps')).count, ).toBe(1); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 7feddd8b..ef488941 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -21,7 +21,6 @@ import { lifecycleFromRequestContext, RUN_LIFECYCLE_CONTEXT_KEY, } from './run-lifecycle.js'; -import { isTerminalRunStatus } from './run-terminal-state.js'; import type { RunnerRuntime, RunSummary } from './runtime.js'; import { validateTablePrefix } from './table-prefix.js'; @@ -89,7 +88,7 @@ export interface RemoteStepInvocation extends RemoteStepAddress { resumeCount?: number; request: unknown; deadlineAt: number; - status: 'pending' | 'dispatched' | 'result-recorded' | 'settled' | 'failed'; + status: 'pending' | 'dispatched' | 'result-retained' | 'failed'; permitGeneration: number; permitEpoch?: number; permitRevision?: number; @@ -215,9 +214,7 @@ function decode(row: JournalRow): RemoteStepInvocation { (row.resume_count !== -1 && row.resume_count < 1) || !Number.isSafeInteger(row.permit_generation) || row.permit_generation < 0 || - !['pending', 'dispatched', 'result-recorded', 'settled', 'failed'].includes( - row.status, - ) + !['pending', 'dispatched', 'result-retained', 'failed'].includes(row.status) ) throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); return { @@ -834,7 +831,7 @@ export class RemoteStepJournal { const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET result_json = ?, result_digest = ?, quiescence_digest = ?, - executor_quiescent_at = ?, status = 'result-recorded' + executor_quiescent_at = ?, status = 'result-retained' WHERE invocation_id = ? AND permit_digest = ? AND permit_consumed_at IS NOT NULL AND result_json IS NULL AND status = 'dispatched' @@ -861,7 +858,7 @@ export class RemoteStepJournal { return replay(after); } - async readRecordedResult( + async readRetainedResult( invocationId: string, ): Promise { if (!isPathSafeId(invocationId)) @@ -880,7 +877,7 @@ export class RemoteStepJournal { const executorQuiescentAt = row.executor_quiescent_at; if (resultJson === null) return undefined; if ( - !['result-recorded', 'settled'].includes(row.status) || + row.status !== 'result-retained' || row.permit_consumed_at === null || typeof resultDigest !== 'string' || !SHA256_HEX.test(resultDigest) || @@ -904,62 +901,4 @@ export class RemoteStepJournal { executorQuiescentAt, }; } - - async settleAfterResume(invocationId: string): Promise { - const invocation = await this.get(invocationId); - if ( - !invocation || - !['result-recorded', 'settled'].includes(invocation.status) - ) - throw new RemoteStepRefusal(409, 'remote step has no result to settle'); - const result = await this.readRecordedResult(invocationId); - if (!result) - throw new RemoteStepRefusal(503, 'remote step result is unreadable'); - const state = await this.#execution(invocation, invocation.principal); - if (state.execution.startToken !== invocation.startToken) - throw new RemoteStepRefusal(409, 'remote step run generation changed'); - const summary = await this.#runtime.authoritativeStatus( - invocation.workflowId, - invocation.runId, - ); - if (!summary) - throw new RemoteStepRefusal(503, 'remote step run is unreadable'); - if (summary.status === 'suspended') { - const key = invocation.stepPath.join('.'); - const stillCurrent = - (summary.suspended ?? []).some( - (path) => - JSON.stringify(path) === JSON.stringify(invocation.stepPath), - ) && - summary.suspendedAt?.[key] === invocation.suspendedAt && - summary.resumeCount?.[key] === invocation.resumeCount; - if (stillCurrent) - throw new RemoteStepRefusal(409, 'remote step has not resumed'); - } else if (!isTerminalRunStatus(summary.status)) { - throw new RemoteStepRefusal(409, 'remote step resume is still active'); - } - if (invocation.status === 'settled') return invocation; - const snapshot = await this.#snapshot(invocation); - const written = rowResult( - await this.#db - .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = 'settled' - WHERE invocation_id = ? AND status = 'result-recorded' - AND result_digest = ? AND executor_quiescent_at IS NOT NULL - AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" - WHERE workflow_name = ? AND run_id = ? AND snapshot = ?) - RETURNING *`) - .bind( - invocationId, - result.resultDigest, - invocation.workflowId, - invocation.runId, - snapshot, - ) - .all(), - ); - if (written) return decode(written); - const after = await this.get(invocationId); - if (after?.status === 'settled') return after; - throw new RemoteStepRefusal(409, 'remote step settlement changed'); - } } From ebe133e95fde29c351577fdee869b84a2b600cef Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 13:31:05 +0400 Subject: [PATCH 09/73] Align FlowSafe inventory spike with remote-step work Recognize the new work category and describe FV2 as seeded-row visibility. Full migration drain still requires an empty inventory across the recovery interval. --- packages/flowsafe/scripts/spike-verify.mjs | 44 ++++++++++------------ 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index 1eaeb9ec..47c73c45 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -437,7 +437,7 @@ async function readInventoryIndex() { } // One category, PAGED TO EXHAUSTION through the inventory's own cursors — the -// sweep an operator's drain proof is actually made of, not a single page that +// sweep an operator's drain proof needs, not a single page that // could hide the rows behind it. `limit: 3` forces several continuations even // on the spike's small data, so the keyset is exercised rather than skipped. async function sweepInventoryCategory(category) { @@ -466,7 +466,7 @@ async function sweepInventoryCategory(category) { throw new Error(`inventory paging did not terminate for ${category}`); } -// Every WORK category, swept. The drain proof is defined over exactly these. +// Every WORK category, swept for the seeded-row visibility check. async function sweepInventoryWork(index) { const work = index.categories.filter((entry) => entry.class === 'work'); const swept = {}; @@ -3183,13 +3183,13 @@ async function main() { }, ); - // --- Drain inventory (F2): the migration proof, on real workerd ----------- + // --- Drain inventory (F2): seeded-row visibility on real workerd ---------- // Unit tests can prove the queries are pure SELECTs against the real schemas. // What they cannot prove is that those queries read the SAME D1 that real // HTTP traffic on real workerd wrote its runs, approvals, reservations, // notifications, and tasks into — and that the numbers MOVE as that work // drains. An inventory that reported the right shape over the wrong database - // would pass every unit test and certify a deployment that still owed work. + // would pass every unit test while missing this work. await step( 'FV1 inventory index: every category, its class, the states no query can ' + @@ -3203,7 +3203,9 @@ async function main() { .filter((entry) => entry.class === 'standing') .map((entry) => entry.category); assert( - work.length === 7 && standing.length === 2, + work.length === 8 && + work.includes('remote-steps') && + standing.length === 2, 'the index splits work from standing configuration', { work, standing }, ); @@ -3225,16 +3227,13 @@ async function main() { ); await step( - 'FV2 drain proof: seeded work appears while draining, leaves as it is ' + - 'finished, and reads empty across TWO consecutive sweeps before the lock', + 'FV2 seeded inventory: work appears while draining, leaves as it is ' + + 'finished, and stays absent under the lock', async () => { const index = await readInventoryIndex(); - // A BASELINE, because this deployment has been running scenarios for the - // whole spike and legitimately still holds work from them. The proof - // below is over the seeded DELTA: these exact rows appear, and these - // exact rows are gone — which is the same claim an empty sweep makes, - // stated about rows whose lifecycle this step controls. + // A baseline accounts for work left by earlier spike scenarios. This + // check follows the seeded rows; it does not prove that all work drained. const before = await sweepInventoryWork(index); // Seed: a suspended run with a pending approval, and a keyed start @@ -3322,9 +3321,8 @@ async function main() { keyedDecided, ); - // The proof: the seeded rows are gone, and STILL gone on a second full - // sweep. One sweep is a point-in-time reading: traffic can still create - // or complete work while it runs, so the contract asks for two. + // The seeded rows are gone on repeated reads. A migration drain proof + // also requires an empty work inventory across the recovery interval. const gone = (sweep) => !sweep.runs.keys.some((key) => key.endsWith(`/${drained.runId}`)) && !sweep.runs.keys.some((key) => key.endsWith(`/${keyed.body.runId}`)) && @@ -3340,7 +3338,7 @@ async function main() { const second = await sweepInventoryWork(index); assert( gone(second), - 'the second consecutive sweep agrees — the drain proof holds', + 'the repeated sweep no longer reports the seeded work', { runs: second.runs.keys, approvals: second['approvals-waiting'].keys, @@ -3357,18 +3355,16 @@ async function main() { ); assert( schedules.class === 'standing' && schedules.keys.length >= 1, - 'schedules are still reported while the drain proof passes', + 'schedules are still reported while seeded work clears', schedules, ); assert( subscriptions.class === 'standing' && subscriptions.keys.length >= 1, - 'provider subscriptions are still reported while the proof passes', + 'provider subscriptions are still reported while seeded work clears', subscriptions, ); - // Lock: the proof held, so the deployment may stop executing. The - // inventory keeps answering — reads are ungated in every state, which is - // what lets an operator verify the lock they just took. + // The lock keeps inventory reads available while refusing new execution. await moveFence('draining', 'migration-locked'); const locked = await sweepInventoryWork(index); assert( @@ -3381,7 +3377,7 @@ async function main() { headers: AUTH.operator, }); assertFenced( - 'a run start after the drain proof locked the deployment', + 'a run start under the migration lock', refusedStart, 'migration-locked', ); @@ -3484,8 +3480,8 @@ try { 'class, the Durable Object journal window it cannot see, and the rule an ' + 'empty answer means something under; seeded runs, an approval, and a ' + 'start reservation appeared as outstanding work while draining, left as ' + - 'each was finished, and read absent across TWO consecutive full sweeps ' + - 'paged through the keyset — after which the deployment locked, still ' + + 'each was finished, and stayed absent on repeated paged reads ' + + 'through the keyset — after which the deployment locked, still ' + 'answered the same inventory, and refused to mint, while its schedules ' + 'and provider subscriptions kept being reported as standing ' + 'configuration a migration carries rather than drains. ' + From 3c286b84a25564e2ae604da70dbae3a27f1ffe3e Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 13:46:01 +0400 Subject: [PATCH 10/73] Add optional FlowSafe remote-step host continuation Route retained results through the stored principal and exact suspension on the internal thread host. Keep wiring optional; schema upgrade, resource binding, and durable settlement remain release gates. --- packages/flowsafe/src/agent-host/catalog.ts | 2 +- .../src/agent-host/thread-host.test.ts | 114 ++++++++++++- .../flowsafe/src/agent-host/thread-host.ts | 153 ++++++++++++++++++ .../src/agent-host/thread-topology.test.ts | 53 +++++- .../src/agent-host/thread-topology.ts | 49 ++++++ .../src/agent-runner/agent-run-state.ts | 1 + 6 files changed, 366 insertions(+), 6 deletions(-) diff --git a/packages/flowsafe/src/agent-host/catalog.ts b/packages/flowsafe/src/agent-host/catalog.ts index 20243b4c..0b9997da 100644 --- a/packages/flowsafe/src/agent-host/catalog.ts +++ b/packages/flowsafe/src/agent-host/catalog.ts @@ -159,7 +159,7 @@ function automationCheckFor( // human approves it — a decided approval and a stranded run. The narrowing // that matters is still enforced: a kind removed from the declaration // entirely can no longer resume. - if (entryPath === 'approval.resume') { + if (entryPath === 'approval.resume' || entryPath === 'remote-step.resume') { return rules.some((rule) => rule.kind === principal.kind); } return rules.some( diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 1c1a1115..51b65032 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -49,6 +49,7 @@ import { resourceIdFromKey, SUSPENSION_TIMEOUT_RESUME_KEY, } from '../do-runner/index.js'; +import { REMOTE_STEP_MARKER } from '../do-runner/remote-step-journal.js'; import { runInputDepthMessage } from '../do-runner/run-input-depth.js'; import { lifecycleFromRequestContext } from '../do-runner/run-lifecycle.js'; import { @@ -63,7 +64,10 @@ import { type PrincipalPermissionResolver, type ThreadAgentStartInput, } from './thread-host.js'; -import { createAgentThreadTopology } from './thread-topology.js'; +import { + createAgentThreadTopology, + type RemoteStepContinuationReader, +} from './thread-topology.js'; import type { AgentAutomationRule, Permission } from './types.js'; const mocked = vi.hoisted(() => ({ @@ -264,6 +268,7 @@ function harness( dispatchId: string, runId: string, ) => Promise; + remoteSteps?: RemoteStepContinuationReader; } = {}, ): Harness { const state = new Map(); @@ -425,13 +430,13 @@ function harness( : undefined; return { kind: summary.status === 'pending' ? 'initial' : 'result', - storage: 'unfenced', + storage: options.remoteSteps ? 'd1' : 'unfenced', execution: ( state.get(OWNER_RECOVERY_PREFIX + runId) as | { execution?: unknown } | undefined )?.execution ?? { - tablePrefix: null, + tablePrefix: options.remoteSteps?.tablePrefix ?? null, workflowId: 'durable-agentic-loop', runId, startToken: 'test-generation', @@ -529,6 +534,7 @@ function harness( ...(options.discardScheduleDispatch ? { discardScheduleDispatch: options.discardScheduleDispatch } : {}), + ...(options.remoteSteps ? { remoteSteps: options.remoteSteps } : {}), approvalService: (instanceScope) => { approvalScopes.push(instanceScope); if (options.approvalService) return options.approvalService; @@ -3954,6 +3960,108 @@ describe('createThreadAgentHost', () => { ]); }); + it('resumes a retained remote result for its stored run and suspension', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: vi.fn(async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained' as const, + permitGeneration: 1, + })), + readRetainedResult: vi.fn(async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + })), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + mocked.resumeViaRuntime.mockResolvedValue({ + runId: 'acme_run', + status: 'suspended', + suspended: [['next']], + }); + const request = new Request( + 'https://thread/_flowsafe/agent-host/remote-step-resume', + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + invocationId: 'invocation_1', + resultJson: '{"answer":999}', + }), + }, + ); + const response = await fixture.host.route(request, fixture.scope); + expect(response?.status).toBe(200); + expect(mocked.resumeViaRuntime).toHaveBeenCalledWith( + expect.objectContaining({ + runId: 'acme_run', + requestedBy: 'operator-1', + step: ['tool'], + resumeData: { answer: 1 }, + expectedSuspension: { suspendedAt: 1000 }, + }), + ); + + mocked.resumeViaRuntime.mockClear(); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1001 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: 409 }); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + it('accepts an approval-resume requester exactly at the principal-id bound', async () => { const fixture = harness(); seedSuspendedApprovalRun(fixture); diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index c0d54045..066989e1 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -82,6 +82,7 @@ import { } from '../do-runner/index.js'; import { isDefinitiveInitialAdmissionRefusal } from '../do-runner/initial-admission-refusal.js'; import { mastraRegistryEntries } from '../do-runner/mastra-registry.js'; +import { REMOTE_STEP_MARKER } from '../do-runner/remote-step-journal.js'; import { exceedsRunInputDepth, runInputDepthMessage, @@ -113,6 +114,7 @@ import { createAgentModuleCatalog } from './catalog.js'; import { AGENT_HOST_ROUTE_PREFIX, publicAgentRunEnvelope, + type RemoteStepContinuationReader, } from './thread-topology.js'; import { createTrustedAgentRequestContext, @@ -171,6 +173,7 @@ export type { } from './types.js'; export interface ThreadAgentHostOptions { + remoteSteps?: RemoteStepContinuationReader; /** Narrows automated entry beyond what each agent's metadata declares. */ authorizeAutomatedEntry?: AutomatedEntryAuthorizer; /** @@ -2588,6 +2591,12 @@ export function createThreadAgentHost( preflightUrl.pathname === `${AGENT_HOST_ROUTE_PREFIX}/resume` ? await objectBody(request) : undefined; + const remoteStepBody = + request.method === 'POST' && + preflightUrl.pathname === + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-resume` + ? await objectBody(request) + : undefined; return withDispatchLock(async () => { const url = new URL(request.url); @@ -2693,6 +2702,150 @@ export function createThreadAgentHost( ); } + if (remoteStepBody) { + const reader = options.remoteSteps; + if (!reader) + throw new AgentHostRequestError( + 503, + 'remote steps are unavailable', + ); + const invocationId = remoteStepBody.invocationId; + if (!isPathSafeId(invocationId)) + throw new AgentHostRequestError(404, 'invocation not found'); + const invocation = await reader.get(invocationId); + const retained = await reader.readRetainedResult(invocationId); + if ( + !invocation || + !retained || + retained.invocationId !== invocationId || + invocation.status !== 'result-retained' || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + invocation.threadId !== scope.threadId || + !samePrincipal(invocation.principal, scope.principal) || + invocation.deadlineAt <= Date.now() + ) + throw new AgentHostRequestError(404, 'invocation not found'); + const ref = { + agentId: invocation.agentId, + resourceId: invocation.resourceId, + runId: invocation.runId, + }; + const selected = await snapshotExecutionFor(scope, ref); + const state = selected.state; + if ( + state.kind !== 'result' || + state.storage !== 'd1' || + state.execution.tablePrefix !== invocation.tablePrefix || + state.execution.workflowId !== invocation.workflowId || + state.execution.startToken !== invocation.startToken || + state.execution.target.kind !== 'agent' || + state.execution.target.id !== invocation.agentId || + state.execution.target.threadId !== invocation.threadId || + state.execution.owner.kind !== invocation.principal.kind || + state.execution.owner.id !== invocation.principal.id + ) + throw new AgentHostRequestError(409, 'remote step run changed'); + const stored = await readRun(ref.runId); + if ( + !stored || + stored.agentId !== ref.agentId || + !samePrincipal(stored.principal, invocation.principal) + ) + throw new AgentHostRequestError(404, 'run not found'); + const key = invocation.stepPath.join('.'); + const summary = state.summary; + const payload = + summary.suspendPayload !== null && + typeof summary.suspendPayload === 'object' && + !Array.isArray(summary.suspendPayload) + ? (summary.suspendPayload as Record)[key] + : undefined; + const suspended = + payload !== null && typeof payload === 'object' + ? (payload as Record) + : undefined; + const tool = + suspended?.toolCallSuspended !== null && + typeof suspended?.toolCallSuspended === 'object' + ? (suspended.toolCallSuspended as Record) + : undefined; + if ( + summary.status !== 'suspended' || + !summary.suspended?.some( + (path) => + JSON.stringify(path) === JSON.stringify(invocation.stepPath), + ) || + summary.suspendedAt?.[key] !== invocation.suspendedAt || + summary.resumeCount?.[key] !== invocation.resumeCount || + suspended?.type !== 'suspension' || + tool?.remoteStep !== REMOTE_STEP_MARKER + ) + throw new AgentHostRequestError( + 409, + 'remote step suspension changed', + ); + let recordedResult: unknown; + try { + recordedResult = JSON.parse(retained.resultJson); + } catch { + throw new AgentHostRequestError( + 503, + 'remote step result is unreadable', + ); + } + const { current, module, principalPermissions } = await authorize( + scope, + ref.agentId, + 'remote-step.resume', + stored.principal, + ); + const durable = current.agents.get(module.meta.id); + if (!durable) throw new Error('guarded agent was not registered'); + const execution: TrustedAgentExecution = { + agentId: ref.agentId, + deploymentTag: scope.deploymentTag, + principal: stored.principal, + threadId: scope.threadId, + resourceId: ref.resourceId, + runId: ref.runId, + entryPath: 'remote-step.resume', + principalPermissions, + safeContext: selected.safeContext, + }; + const resumed = await withExecution(execution, () => + durable.resumeViaRuntime({ + runId: ref.runId, + requestedBy: stored.principal.id, + step: [...invocation.stepPath], + resumeData: recordedResult, + expectedSuspension: { + suspendedAt: invocation.suspendedAt, + ...(invocation.resumeCount === undefined + ? {} + : { resumeCount: invocation.resumeCount }), + }, + ...(selected.threaded + ? { + memory: { + thread: scope.threadId, + resource: ref.resourceId, + }, + } + : {}), + }), + ); + const result = await envelopeFor( + scope, + ref, + stored.principal, + resumed, + ); + if (isTerminalRunStatus(resumed.status)) + await finalizeTerminalAgentState(scope, ref, undefined, stored); + return json(result); + } + if (resumeBody) { const body = resumeBody; const ref = runRef(scope, body); diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 1a0909c3..25cdc84c 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -19,6 +19,7 @@ import { RunRouteError } from '../host-kit/run-route-error.js'; import { type AgentThreadDispatchTopology, createAgentThreadTopology, + type RemoteStepContinuationReader, } from './thread-topology.js'; const DEPLOYMENT_IDENTITY_SECRET = 'test-deployment-identity-secret-0001'; @@ -29,7 +30,7 @@ interface Hit { init?: ThreadRequestInit; } -function harness(): { +function harness(remoteSteps?: RemoteStepContinuationReader): { topology: AgentThreadDispatchTopology; hits: Hit[]; } { @@ -50,7 +51,7 @@ function harness(): { summary: { runId: body.runId, status: 'success' }, }); } - if (url.endsWith('/resume')) { + if (url.endsWith('/resume') || url.endsWith('/remote-step-resume')) { const body = JSON.parse(init?.body ?? '{}') as Record; return Response.json({ agentId: body.agentId, @@ -92,11 +93,59 @@ function harness(): { topology: createAgentThreadTopology(namespace, DEPLOYMENT_IDENTITY_SECRET, { startIdempotency: 'none', executionFence: 'none', + ...(remoteSteps ? { remoteSteps } : {}), }), hits, }; } +it('routes a retained result under the journaled principal and sends only its id', async () => { + const principal = { + kind: 'human' as const, + id: 'original', + role: 'operator' as const, + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: vi.fn(async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'acme_resource_acme_thread', + runId: 'acme_run', + principal, + startToken: 'generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained' as const, + permitGeneration: 1, + })), + readRetainedResult: vi.fn(async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + })), + }; + const { topology, hits } = harness(reader); + await topology.resumeRemoteStep(context().value, 'invocation_1'); + const sent = hits[0]; + expect(sent?.url).toContain('/_flowsafe/agent-host/remote-step-resume'); + expect(JSON.parse(sent?.init?.body ?? '{}')).toEqual({ + invocationId: 'invocation_1', + }); + expect(sent?.init?.headers?.['x-flowsafe-principal']).toBe( + '{"kind":"human","id":"original","role":"operator"}', + ); +}); + function context() { let runMints = 0; let threadMints = 0; diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 23cb8b43..eb01a237 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -30,6 +30,10 @@ import { type StartReservation, type StartReservationReading, } from '../do-runner/index.js'; +import type { + RecordedRemoteStepResult, + RemoteStepInvocation, +} from '../do-runner/remote-step-journal.js'; import { isTerminalRunStatus, type RunStatus, @@ -120,6 +124,16 @@ export interface AgentThreadTopologyOptions { * for a host with no fence. */ executionFence: ExecutionFenceWiring; + remoteSteps?: RemoteStepContinuationReader; +} + +export interface RemoteStepContinuationReader { + deploymentId: string; + tablePrefix: string; + get(invocationId: string): Promise; + readRetainedResult( + invocationId: string, + ): Promise; } export interface AgentThreadRunRef { @@ -181,6 +195,10 @@ export interface AgentThreadDispatchTopology extends AgentThreadBoundTopology { context: ActorContext, input: AgentThreadRunRef, ): Promise; + resumeRemoteStep( + context: ActorContext, + invocationId: string, + ): Promise; } function errorFrom(response: Response): Promise { @@ -737,6 +755,37 @@ export function createAgentThreadTopology( ), ); }, + resumeRemoteStep: async (context, invocationId) => { + const reader = options.remoteSteps; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.get(invocationId); + if ( + invocation?.status !== 'result-retained' || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + !isPathSafeId(invocation.agentId) || + !isPathSafeId(invocation.runId) || + !(await reader.readRetainedResult(invocationId)) + ) + throw new RunRouteError(404, 'invocation not found'); + return envelope( + await threads.send( + { + principal: invocation.principal, + mutationEpoch: context.mutationEpoch, + }, + invocation.threadId, + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-resume`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ invocationId }), + }, + ), + ); + }, }; } diff --git a/packages/flowsafe/src/agent-runner/agent-run-state.ts b/packages/flowsafe/src/agent-runner/agent-run-state.ts index e878045d..0ab65c24 100644 --- a/packages/flowsafe/src/agent-runner/agent-run-state.ts +++ b/packages/flowsafe/src/agent-runner/agent-run-state.ts @@ -15,6 +15,7 @@ export const AGENT_RUN_STORAGE_KEY_PREFIX = 'flowsafe:agent-run:v1:'; export const AGENT_ENTRY_PATHS = [ 'http.start', 'approval.resume', + 'remote-step.resume', 'signal.message', 'signal.queue', 'signal.reactive', From f7db3e6fa44c78eebdf2b1cb5dfdbf5ffdb486aa Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 13:50:45 +0400 Subject: [PATCH 11/73] Account for remote-step inventory in Fleet scenario The resumed proof performs two inventory sweeps. Each now reads the remote-step work category, so the request-count guard expects two additional reads. --- .../fleet-control/test/direct-credentialed-scenario.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/fleet-control/test/direct-credentialed-scenario.test.ts b/packages/fleet-control/test/direct-credentialed-scenario.test.ts index c7d81e1b..f2b3721a 100644 --- a/packages/fleet-control/test/direct-credentialed-scenario.test.ts +++ b/packages/fleet-control/test/direct-credentialed-scenario.test.ts @@ -1084,7 +1084,7 @@ describe.sequential('fence composition through native reference dispatch', { expected: 'migration-locked', next: 'open', }); - expect(inventoryCount()).toBe(beforeInventory + 20); + expect(inventoryCount()).toBe(beforeInventory + 22); expect(proof?.reopen.a?.after).toMatchObject({ state: 'open', mutationEpoch: 1, From 8888fa9cc723e72852025d5d5a82a258cb38b27c Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 13:55:55 +0400 Subject: [PATCH 12/73] Plan authoritative resource binding for remote steps Persist the started resource across topology, host, runtime, journal and recovery. Require nonidentity and legacy refusal proof on real workerd before broker dispatch is enabled. --- docs/proposals/flowsafe-remote-step-broker.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/docs/proposals/flowsafe-remote-step-broker.md b/docs/proposals/flowsafe-remote-step-broker.md index accdd242..291c2108 100644 --- a/docs/proposals/flowsafe-remote-step-broker.md +++ b/docs/proposals/flowsafe-remote-step-broker.md @@ -121,8 +121,117 @@ For the shared protocol and migration checkpoint, run the owning package's whole The implementation is auth, migration, data-retention, and concurrency work. Each code checkpoint is T2: clean-code, QA, architecture, and security lanes. This plan gets one design-level review when added; reviewers do not simulate its commands. A change to this plan after implementation starts records the departure once at completion. Merge a reviewed feature branch into `dev`, not `main`; publish only through Anchorage's version workflow. +## Addendum: bind remote steps to the started resource + +Baseline for this addendum: `plan/flowsafe-remote-step-broker` at `3c286b84a25564e2ae604da70dbae3a27f1ffe3e`, read 2026-10-11. Reconfirm the appendix anchors before editing. A host's `ActorContext.resourceIdFromKey(threadId)` may return a resource distinct from the thread ID. The current remote-step journal compares its address with the package default `resourceIdFromKey(threadId)`, while start provenance records only the thread ID. This prevents a valid host mapping from dispatching and leaves no independent start-time resource proof for a journal row. + +Implement this before enabling remote-step dispatch or result continuation: + +1. Record the effective `resourceId` at agent start in versioned, durable start provenance. The topology resolves its trusted `ActorContext.resourceIdFromKey(threadId)` once for the winning start and sends that value with the internal start. Threaded starts use the selected thread; unthreaded starts use the newly minted thread. Persist the value in the reservation claim and result, runtime snapshot, and host recovery record. A caller-supplied address does not establish this binding. +2. Inject a host-owned resource resolver into `ThreadAgentHostOptions`, wired from the same deployment mapping as the topology. The target host has no `ActorContext`; it must independently resolve and compare the received resource at start and its binding read, `selectedAgentState`, owner recovery, normal status, termination, and approval or remote resume paths. Update `FlowsafeDurableAgent.authoritativeAgentStartState` to select and compare the persisted start resource rather than deriving it from the thread. Update topology replay and `publicApproval`/`publicAgentRunEnvelope` projections to compare the persisted target resource with the expected run resource. Audit the same start identity checks in `StartIdempotencyStore`, fenced workflow admission, and runtime reconstruction. A changed mapping or missing proof refuses broker authority without silently rewriting the run's binding. +3. Make the authoritative start state expose the recorded resource alongside agent, thread, owner, workflow, run, and start token. At journal registration, permit claim and consume, result recording, and recovery, compare the complete persisted binding against the invocation. The thread host independently checks that same binding before resume. Require the currently trusted topology mapping and stored host binding to agree with the start-time value. Do not substitute the package default mapping for the host's mapping. +4. Upgrade existing snapshots, reservations, and journal state through the repository's versioned D1 and durable-state patterns. A legacy run may proceed only when an independently persisted, run-specific host binding proves its resource ID and agrees with its start token, owner, thread, and run. Otherwise keep it unreadable for remote-step authority and visible to recovery or drain inventory. Do not synthesize a missing resource ID from a thread ID, current mapper, or caller address; do not add a permissive schema default. A mixed old/new deployment must refuse remote-step dispatch until both sides understand the new binding. Existing non-broker run status and ordinary approval resume retain their current compatibility behavior. +5. Test a host mapping such as `thread-1 → account-7` across both threaded and unthreaded starts, fresh-host restart, registration, permit consume, result intake, and resume. Test a changed mapper, a mismatched journal resource, a forged caller resource, a legacy start without independent proof, and a corrupted stored binding. Each refusal must occur before an external effect or resume. Use a real D1/workerd path for the upgrade and fresh-isolate case; in-memory mocks alone do not prove it. + +The chosen authority is the persisted start binding checked again by the trusted host. Reject using `resourceIdFromKey(threadId)` inside the journal: it is the package's identity validator, not the host's resource mapping. Reject trusting the journal's resource column or current mapping alone: either can differ from the resource at start. Reject deriving legacy values from thread IDs: a valid host can map them to different resources. This extends the existing start identity and journal boundary; it adds no dependency, public route, or new authentication scheme. + +Reuse `createAgentThreadTopology`'s `expectedResource` and `sendStart`, the thread host's `snapshotExecutionFor` and owner recovery, `RunnerRuntime.authoritativeStartState`, `StartReservationStore`'s staged schema migration, and `RemoteStepJournal.#execution`. Existing `packages/flowsafe/scripts/spike-verify.mjs` drives `packages/flowsafe/spike/worker.ts` on real workerd and D1, including process kill and restart. Extend that pair with an `FR1` resource-binding scenario; the current script has no such proof. Keep WFP grant policy, customer identities, external executor credentials, and ordinary approval semantics out of this change. Do not change the host's resource mapping for existing users as a migration shortcut. + +Scoped verification for the binding checkpoint, with each command's full output and exit status recorded in a new `/tmp` log: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run src/agent-host/thread-topology.test.ts src/agent-host/thread-host.test.ts src/do-runner/remote-step-journal.test.ts src/do-runner/start-idempotency.test.ts src/do-runner/execution-admission.test.ts +pnpm --filter @proofoftech/flowsafe typecheck +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe spike:verify +``` + +The `spike:verify` command must execute the new `FR1` scenario in `spike-verify.mjs`, backed by the spike worker's thread host and journal. `FR1` starts a remote-step run under a non-identity mapping, kills workerd, verifies the port is closed, restarts on the same D1, then registers, consumes a permit, records a result, and resumes through the target host. It also seeds a pre-upgrade start without independent resource proof and asserts refusal before permit or resume. Check the named `FR1` pass and refusal observations in that command's log; an unchanged spike run does not satisfy this gate. Because the start identity and D1 schema are shared contracts, run the FlowSafe package suite once at that checkpoint under the quality-gate exception. Run T2 clean-code, QA, architecture, and security lanes on its actual diff and in-reach start, host, runtime, journal, and migration contracts. The final repository suite and release gate remain as stated above. + ## Appendix: current-state anchors +### Resource authority + +These snippets are verbatim at the addendum baseline. They identify risky auth and migration edits; re-read the containing symbols before changing them. + +`packages/flowsafe/src/do-runner/execution-admission.ts`, `StartIdentity` currently records an agent target without its resource: + +```typescript + | { + readonly kind: 'agent'; + readonly id: string; + readonly threadId: string; + }; +``` + +After: add a required, path-safe `resourceId` to the versioned agent target and propagate it through normalization, serialization, replay, and authoritative start reads. Legacy records without separately proved binding refuse broker authority. + +`packages/flowsafe/src/agent-host/thread-topology.ts`, `sendStart` currently resolves the resource again when sending to the target host: + +```typescript + threadId: targetThreadId, + resourceId: context.resourceIdFromKey(targetThreadId), + runId: targetRunId, +``` + +After: capture the trusted mapping with the start decision and pass that exact resource into the host, reservation, and persisted start provenance. On a replay, verify the winning reservation's recorded resource before sending; a new mapping must not silently change the run's resource. + +`packages/flowsafe/src/agent-host/thread-host.ts`, normal start, selection, and recovery currently compare against the package default instead of a host-owned mapping: + +```typescript +if (ref.resourceId !== resourceIdFromKey(scope.threadId)) + throw new AgentHostRequestError(404, 'run not found'); +``` + +The selector uses the same condition; owner recovery compares `stored.resourceId !== resourceIdFromKey(threadId)`. After: inject the deployment's trusted resolver through `ThreadAgentHostOptions`, validate and capture its output, and compare it with the persisted start identity and stored binding at each entry. Keep this check independent of the received internal request. + +`packages/flowsafe/src/agent-runner/durable-agent-runner.ts`, `authoritativeAgentStartState` currently derives the observed resource: + +```typescript +const observedAgentId = identity.target.id; +const observedThreadId = identity.target.threadId; +const observedResourceId = resourceIdFromKey(observedThreadId); +``` + +After: read `observedResourceId` from the validated, versioned start identity and compare the snapshot's context and memory against it. Ordinary legacy reads retain their explicit compatibility path; a legacy record cannot obtain broker authority without independent proof. + +`packages/flowsafe/src/agent-host/thread-topology.ts`, `publicApproval` currently rejects a resource that differs from the default mapping: + +```typescript +target.agentId !== execution.target.id || +target.threadId !== execution.target.threadId || +target.resourceId !== resourceIdFromKey(target.threadId) +``` + +After: compare the approval target and replay envelope with `execution.target.resourceId`, then separately compare that persisted value with the topology's current trusted mapping before admission. + +`packages/flowsafe/src/do-runner/remote-step-journal.ts`, `RemoteStepJournal.#execution` currently checks: + +```typescript + target.id !== address.agentId || + target.threadId !== address.threadId || + (principal !== undefined && + (state.provenance.startIdentity?.owner.kind !== principal.kind || + state.provenance.startIdentity.owner.id !== principal.id)) || + state.execution.tablePrefix !== address.tablePrefix || + address.resourceId !== resourceIdFromKey(address.threadId) +``` + +After: compare `address.resourceId` with the resource recorded in authoritative start provenance. Reject absent or conflicting provenance before an invocation, permit, result, or resume obtains authority. `RemoteStepJournal.register` currently accepts the address and principal as inputs; they remain claims until this check succeeds. + +`packages/flowsafe/src/agent-host/thread-host.ts`, the remote-result route currently checks the start target but not its resource: + +```typescript + state.execution.target.kind !== 'agent' || + state.execution.target.id !== invocation.agentId || + state.execution.target.threadId !== invocation.threadId || + state.execution.owner.kind !== invocation.principal.kind || +``` + +After: compare the persisted start resource, invocation resource, stored host binding, and trusted mapping before `withExecution` and `resumeViaRuntime`. Refuse on any missing or unequal value. Keep the stored principal and exact suspension checks that follow. + +### Earlier proposal anchors + These are verbatim at the baseline commit. Reconfirm them before applying the plan. ### Thread status and resume (`packages/flowsafe/src/agent-host/thread-topology.ts`) From 97404800cbf86eb530cb0ed4cb1b12a7caee7e41 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 14:07:40 +0400 Subject: [PATCH 13/73] Preserve remote resume principal kind and timeout authority The host forwards the stored principal kind and rejects reserved timeout data before runtime resume. Approval resume keeps its human default; focused and package suites pass. --- .../src/agent-host/thread-host.test.ts | 76 +++++++++++++++++-- .../flowsafe/src/agent-host/thread-host.ts | 3 +- .../agent-runner/durable-agent-runner.test.ts | 26 +++++++ .../src/agent-runner/durable-agent-runner.ts | 7 +- 4 files changed, 105 insertions(+), 7 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 51b65032..3c7a810e 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -3962,9 +3962,9 @@ describe('createThreadAgentHost', () => { it('resumes a retained remote result for its stored run and suspension', async () => { const principal: ExecutionPrincipal = { - kind: 'human', - id: 'operator-1', - role: 'operator', + kind: 'service', + id: 'executor-service', + purpose: 'remote-step-run', }; const reader: RemoteStepContinuationReader = { deploymentId: 'deployment', @@ -3995,7 +3995,11 @@ describe('createThreadAgentHost', () => { executorQuiescentAt: 1000, })), }; - const fixture = harness(['writer'], { remoteSteps: reader }); + const fixture = harness(['writer'], { + remoteSteps: reader, + principal, + allowedAutomation: [{ kind: 'service', entryPaths: ['http.start'] }], + }); seedSuspendedApprovalRun(fixture); fixture.setSummary({ runId: 'acme_run', @@ -4030,7 +4034,8 @@ describe('createThreadAgentHost', () => { expect(mocked.resumeViaRuntime).toHaveBeenCalledWith( expect.objectContaining({ runId: 'acme_run', - requestedBy: 'operator-1', + requestedBy: 'executor-service', + requestedByKind: 'service', step: ['tool'], resumeData: { answer: 1 }, expectedSuspension: { suspendedAt: 1000 }, @@ -4062,6 +4067,67 @@ describe('createThreadAgentHost', () => { expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); }); + it('refuses a reserved timeout envelope in a retained remote result', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained', + permitGeneration: 1, + }), + readRetainedResult: async () => ({ + invocationId: 'invocation_1', + resultJson: JSON.stringify({ [SUSPENSION_TIMEOUT_RESUME_KEY]: true }), + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + }), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: 400 }); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + it('accepts an approval-resume requester exactly at the principal-id bound', async () => { const fixture = harness(); seedSuspendedApprovalRun(fixture); diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index 066989e1..e9a5cb34 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -2818,7 +2818,8 @@ export function createThreadAgentHost( runId: ref.runId, requestedBy: stored.principal.id, step: [...invocation.stepPath], - resumeData: recordedResult, + resumeData: resumeData(recordedResult), + requestedByKind: stored.principal.kind, expectedSuspension: { suspendedAt: invocation.suspendedAt, ...(invocation.resumeCount === undefined diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts index 225a2a50..ecdef4de 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.test.ts @@ -3958,6 +3958,32 @@ describe('FlowsafeDurableAgent thread runtime registration and rehydration', () expect(agent.getPubSub()).toBe(pubsub); }); + it('passes an explicit automated requester kind to the runtime resume', async () => { + const { runtime, resume } = fakeRuntime(); + const agent = createFlowsafeDurableAgent({ agent: testAgent(), runtime }); + await agent.resumeViaRuntime({ + runId: 'run-1', + requestedBy: 'service-1', + requestedByKind: 'service', + }); + expect(resume).toHaveBeenCalledWith( + expect.anything(), + 'run-1', + expect.objectContaining({ + requestedBy: 'service-1', + requestedByKind: 'service', + }), + ); + await expect( + agent.resumeViaRuntime({ + runId: 'run-2', + requestedBy: 'service-1', + requestedByKind: 'forged' as never, + }), + ).rejects.toBeInstanceOf(InvalidRunRequestError); + expect(resume).toHaveBeenCalledTimes(1); + }); + it('resumes observation after retained earlier events on the observer bus', async () => { const pubsub = new EventEmitterPubSub(); const { runtime, resume } = fakeRuntime({ diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts index 3b670343..aaee0de8 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts @@ -2022,12 +2022,17 @@ export class FlowsafeDurableAgent< async resumeViaRuntime(options: { runId: string; requestedBy: string; + requestedByKind?: ExecutionPrincipalKind; step?: string | string[]; resumeData?: unknown; memory?: DurableAgentStreamOptions['memory']; expectedSuspension?: ResumeRunOptions['expectedSuspension']; }): Promise { this.#assertCallerRunId(options.runId); + const requestedByKind = + options.requestedByKind === undefined ? 'human' : options.requestedByKind; + if (!isExecutionPrincipalKind(requestedByKind)) + throw new InvalidRunRequestError('requestedByKind is malformed'); const memory = this.#guardedCallOptionMapper ? (this.#guardedCallOptionMapper( 'memory', @@ -2046,7 +2051,7 @@ export class FlowsafeDurableAgent< ? { expectedSuspension: options.expectedSuspension } : {}), requestedBy: options.requestedBy, - requestedByKind: 'human', + requestedByKind, legAbort, prepareExecution: async (requestContext, workflowInput) => { const current = await this.#runtime.authoritativeStatus( From 6943aa67e80de2ed91f9db29cfd35a98aa3f39b1 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 14:30:39 +0400 Subject: [PATCH 14/73] Export remote-step host types for API docs Document the host reader's referenced types and parse source once in the fence matrix census so the full suite fits its existing timeout under load. --- packages/flowsafe/src/agent-host/index.ts | 5 +++ .../src/execution-entry-matrix.test.ts | 44 ++++++++++++------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index af52ff3a..1d26f49b 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -10,6 +10,10 @@ export { ReservedExecutionContextError, stripReservedExecutionContext, } from '../do-runner/index.js'; +export type { + RecordedRemoteStepResult, + RemoteStepInvocation, +} from '../do-runner/remote-step-journal.js'; export { type AgentApprovalResumer, type AgentApprovalResumerOptions, @@ -50,6 +54,7 @@ export { type AgentThreadTopology, type AgentThreadTopologyOptions, createAgentThreadTopology, + type RemoteStepContinuationReader, } from './thread-topology.js'; export { AGENT_AUDIT_CONTEXT_KEY, diff --git a/packages/flowsafe/src/execution-entry-matrix.test.ts b/packages/flowsafe/src/execution-entry-matrix.test.ts index ca60b182..f5679c2e 100644 --- a/packages/flowsafe/src/execution-entry-matrix.test.ts +++ b/packages/flowsafe/src/execution-entry-matrix.test.ts @@ -1471,7 +1471,11 @@ function sourceFileSystem(): SourceFileSystem { ).process?.getBuiltinModule?.('node:fs') as SourceFileSystem; } -type SourceFile = { readonly file: string; readonly source: string }; +type SourceFile = { + readonly file: string; + readonly source: string; + readonly parsed?: ts.SourceFile; +}; type SourceUrlModule = { fileURLToPath(url: URL): string }; @@ -1516,14 +1520,14 @@ function walkSourceFiles( * Every actual predicate/delegation call. Definitions, re-exports and comments * are not calls; the defining module's own nomination gates remain visible. */ -function predicateCallSites({ file, source }: SourceFile): GateSite[] { +function predicateCallSites({ + file, + source, + parsed: cached, +}: SourceFile): GateSite[] { const found: GateSite[] = []; - const parsed = ts.createSourceFile( - file, - source, - ts.ScriptTarget.Latest, - true, - ); + const parsed = + cached ?? ts.createSourceFile(file, source, ts.ScriptTarget.Latest, true); const visit = (node: ts.Node): void => { if (ts.isCallExpression(node)) { const expression = node.expression; @@ -1560,13 +1564,13 @@ function predicateCallSites({ file, source }: SourceFile): GateSite[] { return found; } -function sqlAdmissionSites({ file, source }: SourceFile): GateSite[] { - const parsed = ts.createSourceFile( - file, - source, - ts.ScriptTarget.Latest, - true, - ); +function sqlAdmissionSites({ + file, + source, + parsed: cached, +}: SourceFile): GateSite[] { + const parsed = + cached ?? ts.createSourceFile(file, source, ts.ScriptTarget.Latest, true); const found: GateSite[] = []; const admissionBindings = new Set(); const collectBindings = (node: ts.Node): void => { @@ -1622,9 +1626,15 @@ function sqlAdmissionSites({ file, source }: SourceFile): GateSite[] { function gateCallSites(): GateSite[] { const found: GateSite[] = []; walkSourceFiles(sourceRoot(), (sourceFile) => { + const parsed = ts.createSourceFile( + sourceFile.file, + sourceFile.source, + ts.ScriptTarget.Latest, + true, + ); found.push( - ...predicateCallSites(sourceFile), - ...sqlAdmissionSites(sourceFile), + ...predicateCallSites({ ...sourceFile, parsed }), + ...sqlAdmissionSites({ ...sourceFile, parsed }), ); }); return found; From 82fed07b6d361b7385062c97b9df0a960c95ddb8 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 14:40:58 +0400 Subject: [PATCH 15/73] Prove retained remote result survives a fresh host Exercise D1 journal persistence and exact host continuation after recreating the host. The runtime resume itself remains covered by the pinned-core probe. --- .../src/agent-host/thread-host.test.ts | 119 +++++++++++++++++- 1 file changed, 117 insertions(+), 2 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 3c7a810e..17206517 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -49,7 +49,10 @@ import { resourceIdFromKey, SUSPENSION_TIMEOUT_RESUME_KEY, } from '../do-runner/index.js'; -import { REMOTE_STEP_MARKER } from '../do-runner/remote-step-journal.js'; +import { + REMOTE_STEP_MARKER, + RemoteStepJournal, +} from '../do-runner/remote-step-journal.js'; import { runInputDepthMessage } from '../do-runner/run-input-depth.js'; import { lifecycleFromRequestContext } from '../do-runner/run-lifecycle.js'; import { @@ -269,6 +272,7 @@ function harness( runId: string, ) => Promise; remoteSteps?: RemoteStepContinuationReader; + stateStorage?: AgentThreadStateStorage; } = {}, ): Harness { const state = new Map(); @@ -528,7 +532,7 @@ function harness( storageScopes.push(instanceScope); return storage; }, - stateStorage: () => stateStorage, + stateStorage: () => options.stateStorage ?? stateStorage, resourceAccess: () => resourceAccess, scheduleSource: () => scheduleSource, ...(options.discardScheduleDispatch @@ -7440,6 +7444,7 @@ async function hostAgentLifecycleFixture( .all(); return { ...fixture, + binding, storage, app, sql, @@ -7466,6 +7471,116 @@ function hostAgentLifecycleRequest(suffix = '', query = '') { ); } +it('reads a retained remote result from D1 after the agent host is recreated', async () => { + const fixture = await hostAgentLifecycleFixture({ + mode: 'fenced', + provenance: 'modern', + }); + onTestFinished(() => { + vi.restoreAllMocks(); + fixture.sql.close(); + }); + fixture.snapshot.suspendedPaths = { tool: [0] }; + Object.assign(fixture.snapshot.context, { + tool: { + status: 'suspended', + payload: {}, + startedAt: 50, + suspendedAt: 1000, + suspendPayload: { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + await fixture.persist(); + await fixture.host.route(hostAgentLifecycleRequest(), fixture.scope); + const journal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: fixture.app.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + workflowId: fixture.execution.workflowId, + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + }, + Date.now() + 60_000, + fixture.scope.principal, + ); + const permit = await journal.issuePermit(invocation.invocationId, 5_000); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"answer":1}', + '{"receipt":"complete"}', + ); + + const freshApp = init( + { storage: fixture.storage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const reopened = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: freshApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const fresh = harness(['writer'], { + init: freshApp, + storage: fixture.storage, + stateStorage: fixture.stateStorage, + resourceAccess: fixture.resources, + remoteSteps: { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + get: reopened.get.bind(reopened), + readRetainedResult: reopened.readRetainedResult.bind(reopened), + }, + }); + const resume = vi + .spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime') + .mockResolvedValue({ runId: 'acme_run', status: 'suspended' }); + const response = await fresh.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + fresh.scope, + ); + expect(response?.status).toBe(200); + expect(resume).toHaveBeenCalledWith( + expect.objectContaining({ + runId: 'acme_run', + requestedBy: 'operator-1', + requestedByKind: 'human', + step: ['tool'], + resumeData: { answer: 1 }, + expectedSuspension: { suspendedAt: 1000 }, + }), + ); +}); + describe('agent host selector lookup isolation', () => { it.each([ ['modern', true, 'status'], From 469f76606c93296c7331556181ce9b19c55dd617 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 23:18:00 +0400 Subject: [PATCH 16/73] feat(flowsafe): version remote step journal schema Upgrade the known prior journal layout transactionally, preserve permit/result records, and refuse unknown or partial schemas before remote-step authority is used. --- packages/flowsafe/src/do-runner/inventory.ts | 12 +- .../src/do-runner/remote-step-journal.test.ts | 49 +++++++ .../src/do-runner/remote-step-journal.ts | 123 ++++++++++++++++++ 3 files changed, 183 insertions(+), 1 deletion(-) diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index a082d656..a796bff6 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -70,7 +70,10 @@ import { DUE_NOTIFICATION_SQL, notificationTimestampMillis, } from './notification-predicate.js'; -import { REMOTE_STEP_TABLE } from './remote-step-journal.js'; +import { + REMOTE_STEP_SCHEMA_TABLE, + REMOTE_STEP_TABLE, +} from './remote-step-journal.js'; import { START_IDEMPOTENCY_TABLE, START_RESERVATION_STATES, @@ -292,6 +295,13 @@ export interface FlowsafeTableEntry { * characters. */ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ + { + table: REMOTE_STEP_SCHEMA_TABLE, + purpose: 'remote invocation journal schema version', + accounting: { + excluded: 'schema metadata contains no run or external dispatch work', + }, + }, { table: REMOTE_STEP_TABLE, purpose: 'durable remote tool invocations and their dispatch permits', diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index e457e8e5..96f62e94 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -190,6 +190,55 @@ describe('remote step journal dispatch admission', () => { ]); }); + it('upgrades a prior journal while preserving an existing invocation', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + for (const column of [ + 'executor_quiescent_at', + 'quiescence_digest', + 'result_digest', + 'result_json', + ]) + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN ${column}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + + await h.reopen().initialize(); + + expect(await h.reopen().get(invocation.invocationId)).toMatchObject({ + invocationId: invocation.invocationId, + status: 'pending', + }); + const columns = h.sqlite + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all() as Array<{ name: string }>; + expect(columns.map((column) => column.name)).toContain( + 'executor_quiescent_at', + ); + }); + + it('refuses an unknown journal schema version before dispatch', async () => { + const h = await fixture(); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 3'); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses a partial journal schema without changing its rows', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN result_digest`); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + expect( + h.sqlite.prepare(`SELECT invocation_id FROM ${REMOTE_STEP_TABLE}`).get(), + ).toMatchObject({ invocation_id: invocation.invocationId }); + }); + it('binds registration to the original execution principal', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index ef488941..cbbdc1e2 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -25,6 +25,49 @@ import type { RunnerRuntime, RunSummary } from './runtime.js'; import { validateTablePrefix } from './table-prefix.js'; export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; +export const REMOTE_STEP_SCHEMA_TABLE = 'flowsafe_remote_step_schema'; +const REMOTE_STEP_RESULT_COLUMNS = [ + 'result_json', + 'result_digest', + 'quiescence_digest', + 'executor_quiescent_at', +] as const; +const REMOTE_STEP_COLUMNS = [ + 'invocation_id', + 'principal_json', + 'start_token', + 'deployment_id', + 'table_prefix', + 'workflow_id', + 'agent_id', + 'thread_id', + 'resource_id', + 'run_id', + 'step_path', + 'suspended_at', + 'resume_count', + 'request_json', + 'deadline_at', + 'status', + 'permit_generation', + 'permit_epoch', + 'permit_revision', + 'permit_expires_at', + 'permit_consumed_at', + 'permit_digest', + ...REMOTE_STEP_RESULT_COLUMNS, +] as const; +const REMOTE_STEP_INTEGER_COLUMNS = new Set([ + 'suspended_at', + 'resume_count', + 'deadline_at', + 'permit_generation', + 'permit_epoch', + 'permit_revision', + 'permit_expires_at', + 'permit_consumed_at', + 'executor_quiescent_at', +]); export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; export const MAX_REMOTE_STEP_RESULT_BYTES = 16_384; @@ -395,6 +438,86 @@ export class RemoteStepJournal { thread_id, resource_id, run_id, step_path, suspended_at, resume_count) )`) .run(); + await this.#db + .prepare(`CREATE TABLE IF NOT EXISTS ${REMOTE_STEP_SCHEMA_TABLE} ( + id INTEGER PRIMARY KEY CHECK (id = 1), + version INTEGER NOT NULL + )`) + .run(); + const schema = await this.#db + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all(); + const columns = record(schema)?.results; + if ( + !Array.isArray(columns) || + columns.some( + (column) => + typeof record(column)?.name !== 'string' || + record(column)?.hidden !== 0, + ) + ) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + const names = columns.map((column) => record(column)?.name); + const samePrefix = (expected: readonly string[]) => + names.length === expected.length && + names.every((name, index) => { + const column = record(columns[index]); + return ( + name === expected[index] && + column?.type === + (REMOTE_STEP_INTEGER_COLUMNS.has(name as string) + ? 'INTEGER' + : 'TEXT') && + column?.notnull === (index > 0 && index <= 16 ? 1 : 0) && + column?.pk === (index === 0 ? 1 : 0) + ); + }); + const versions = await this.#db + .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) + .all(); + const rows = record(versions)?.results; + if (!Array.isArray(rows) || rows.length > 1) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + if (rows.length === 1) { + if ( + record(rows[0])?.id !== 1 || + record(rows[0])?.version !== 2 || + !samePrefix(REMOTE_STEP_COLUMNS) + ) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + return; + } + if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) { + const transactional = this.#db as ExecutionFenceDatabase & { + batch?: ( + statements: ReturnType[], + ) => Promise; + }; + if (typeof transactional.batch !== 'function') + throw new RemoteStepRefusal( + 503, + 'remote step schema upgrade is unavailable', + ); + await transactional.batch([ + ...REMOTE_STEP_RESULT_COLUMNS.map((column) => + this.#db.prepare( + `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${column === 'executor_quiescent_at' ? 'INTEGER' : 'TEXT'}`, + ), + ), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, + ), + ]); + return; + } + if (!samePrefix(REMOTE_STEP_COLUMNS)) { + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + } + await this.#db + .prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, + ) + .run(); } async #snapshot(address: RemoteStepAddress): Promise { From 940c0ceaf21120c5ed4370b11b85fc35ed508b13 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 23:25:25 +0400 Subject: [PATCH 17/73] fix(flowsafe): validate journal schema before migration Require the suspension key constraint and permit default before accepting legacy rows. Recheck the completed schema after concurrent migration races. --- .../src/do-runner/remote-step-journal.test.ts | 67 +++++++ .../src/do-runner/remote-step-journal.ts | 170 ++++++++++++------ 2 files changed, 179 insertions(+), 58 deletions(-) diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 96f62e94..27164721 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -239,6 +239,73 @@ describe('remote step journal dispatch admission', () => { ).toMatchObject({ invocation_id: invocation.invocationId }); }); + it('refuses an unversioned journal without its suspension uniqueness constraint', async () => { + const h = await fixture(); + const ddl = ( + h.sqlite + .prepare( + `SELECT sql FROM sqlite_schema WHERE name = '${REMOTE_STEP_TABLE}'`, + ) + .get() as { sql: string } + ).sql; + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec(ddl.replace(/,\s*UNIQUE \(deployment_id,[^)]*\)/, '')); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('refuses an unversioned journal without the permit-generation default', async () => { + const h = await fixture(); + const ddl = ( + h.sqlite + .prepare( + `SELECT sql FROM sqlite_schema WHERE name = '${REMOTE_STEP_TABLE}'`, + ) + .get() as { sql: string } + ).sql; + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec( + ddl.replace( + 'permit_generation INTEGER NOT NULL DEFAULT 0', + 'permit_generation INTEGER NOT NULL', + ), + ); + + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + + it('converges when another initializer wins the same journal upgrade', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + for (const column of [ + 'executor_quiescent_at', + 'quiescence_digest', + 'result_digest', + 'result_json', + ]) + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN ${column}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + const originalBatch = h.db.batch.bind(h.db); + h.db.batch = async (statements) => { + h.db.batch = originalBatch; + await h.reopen().initialize(); + return originalBatch(statements); + }; + + await h.reopen().initialize(); + + expect(await h.reopen().get(invocation.invocationId)).toMatchObject({ + invocationId: invocation.invocationId, + status: 'pending', + }); + }); + it('binds registration to the original execution principal', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index cbbdc1e2..613f9e68 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -444,50 +444,99 @@ export class RemoteStepJournal { version INTEGER NOT NULL )`) .run(); - const schema = await this.#db - .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) - .all(); - const columns = record(schema)?.results; - if ( - !Array.isArray(columns) || - columns.some( - (column) => - typeof record(column)?.name !== 'string' || - record(column)?.hidden !== 0, - ) - ) - throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - const names = columns.map((column) => record(column)?.name); - const samePrefix = (expected: readonly string[]) => - names.length === expected.length && - names.every((name, index) => { - const column = record(columns[index]); - return ( - name === expected[index] && - column?.type === - (REMOTE_STEP_INTEGER_COLUMNS.has(name as string) - ? 'INTEGER' - : 'TEXT') && - column?.notnull === (index > 0 && index <= 16 ? 1 : 0) && - column?.pk === (index === 0 ? 1 : 0) - ); - }); - const versions = await this.#db - .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) - .all(); - const rows = record(versions)?.results; - if (!Array.isArray(rows) || rows.length > 1) - throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - if (rows.length === 1) { - if ( - record(rows[0])?.id !== 1 || - record(rows[0])?.version !== 2 || - !samePrefix(REMOTE_STEP_COLUMNS) - ) + const readSchema = async () => { + const schema = await this.#db + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all(); + const columns = record(schema)?.results; + if (!Array.isArray(columns)) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - return; - } - if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) { + const samePrefix = (expected: readonly string[]) => + columns.length === expected.length && + columns.every((value, index) => { + const column = record(value); + const name = column?.name; + return ( + name === expected[index] && + column?.type === + (REMOTE_STEP_INTEGER_COLUMNS.has(name as string) + ? 'INTEGER' + : 'TEXT') && + column?.notnull === (index > 0 && index <= 16 ? 1 : 0) && + column?.pk === (index === 0 ? 1 : 0) && + column?.hidden === 0 && + column?.dflt_value === (name === 'permit_generation' ? '0' : null) + ); + }); + const indexes = record( + await this.#db.prepare(`PRAGMA index_list(${REMOTE_STEP_TABLE})`).all(), + )?.results; + if (!Array.isArray(indexes)) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + const suspensionKey = [ + 'deployment_id', + 'table_prefix', + 'workflow_id', + 'agent_id', + 'thread_id', + 'resource_id', + 'run_id', + 'step_path', + 'suspended_at', + 'resume_count', + ]; + let uniqueSuspension = false; + for (const value of indexes) { + const indexRow = record(value); + const name = indexRow?.name; + if ( + indexRow?.unique !== 1 || + indexRow?.origin !== 'u' || + indexRow?.partial !== 0 || + typeof name !== 'string' || + !/^sqlite_autoindex_flowsafe_remote_step_invocations_[1-9][0-9]*$/.test( + name, + ) + ) + continue; + const details = record( + await this.#db.prepare(`PRAGMA index_info('${name}')`).all(), + )?.results; + if ( + Array.isArray(details) && + details.length === suspensionKey.length && + details.every( + (part, index) => + record(part)?.seqno === index && + record(part)?.name === suspensionKey[index], + ) + ) + uniqueSuspension = true; + } + if (!uniqueSuspension) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + const versions = await this.#db + .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) + .all(); + const rows = record(versions)?.results; + if (!Array.isArray(rows) || rows.length > 1) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + if (rows.length === 1) { + if ( + record(rows[0])?.id !== 1 || + record(rows[0])?.version !== 2 || + !samePrefix(REMOTE_STEP_COLUMNS) + ) + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + return 'current'; + } + if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) return 'prior'; + if (samePrefix(REMOTE_STEP_COLUMNS)) return 'unversioned-current'; + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + }; + const stage = await readSchema(); + if (stage === 'current') return; + if (stage === 'prior') { const transactional = this.#db as ExecutionFenceDatabase & { batch?: ( statements: ReturnType[], @@ -498,26 +547,31 @@ export class RemoteStepJournal { 503, 'remote step schema upgrade is unavailable', ); - await transactional.batch([ - ...REMOTE_STEP_RESULT_COLUMNS.map((column) => + try { + await transactional.batch([ + ...REMOTE_STEP_RESULT_COLUMNS.map((column) => + this.#db.prepare( + `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${column === 'executor_quiescent_at' ? 'INTEGER' : 'TEXT'}`, + ), + ), this.#db.prepare( - `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${column === 'executor_quiescent_at' ? 'INTEGER' : 'TEXT'}`, + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, ), - ), - this.#db.prepare( - `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, - ), - ]); + ]); + } catch (error) { + if ((await readSchema()) !== 'current') throw error; + } return; } - if (!samePrefix(REMOTE_STEP_COLUMNS)) { - throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + try { + await this.#db + .prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, + ) + .run(); + } catch (error) { + if ((await readSchema()) !== 'current') throw error; } - await this.#db - .prepare( - `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, - ) - .run(); } async #snapshot(address: RemoteStepAddress): Promise { From 17896ecdfe7e1858c76135abe7ba79ed45bca5cc Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Sun, 11 Oct 2026 23:35:21 +0400 Subject: [PATCH 18/73] feat(flowsafe): bind remote journal to start resource Persist the host's start resource and require journal addresses to match that provenance. Ambiguous legacy starts refuse broker authority; host topology support follows separately. --- .../src/agent-host/thread-host.test.ts | 38 ++++++++-- .../flowsafe/src/agent-host/thread-host.ts | 1 + .../src/agent-runner/durable-agent-runner.ts | 3 +- .../src/do-runner/execution-admission.test.ts | 15 ++++ .../src/do-runner/execution-admission.ts | 19 ++++- .../src/do-runner/remote-step-journal.test.ts | 73 +++++++++++++++++++ .../src/do-runner/remote-step-journal.ts | 9 ++- 7 files changed, 147 insertions(+), 11 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 17206517..532ff57f 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -1429,7 +1429,12 @@ describe('direct thread host capture', () => { mutationEpoch: 2, startIdentity: { owner: { kind: principal.kind, id: principal.id }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded }, onPreparedStartIdentity: expect.any(Function), @@ -3141,7 +3146,12 @@ describe('createThreadAgentHost', () => { { startIdentity: { owner: HUMAN_OWNER, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded: false }, onPreparedStartIdentity: expect.any(Function), @@ -3465,7 +3475,12 @@ describe('createThreadAgentHost', () => { { startIdentity: { owner: { kind: 'service', id: 'webhook-dispatcher' }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded: true }, onPreparedStartIdentity: expect.any(Function), @@ -3710,7 +3725,12 @@ describe('createThreadAgentHost', () => { { startIdentity: { owner: { kind: 'system', id: 'signal-dispatcher' }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }, agentStart: { threaded: true }, onPreparedStartIdentity: expect.any(Function), @@ -7315,7 +7335,12 @@ async function hostAgentLifecycleFixture( await reservations.bindPreparedStart(claim, { ...execution, owner: HUMAN_OWNER, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }); } const recovery = ownerRecovery('acme_run', { @@ -7399,6 +7424,7 @@ async function hostAgentLifecycleFixture( kind: 'agent', id: snapshotAgentId, threadId: snapshotThreadId, + resourceId: snapshotResourceId, }, }, agentStart: { threaded }, @@ -7695,6 +7721,8 @@ describe('agent host selector lookup isolation', () => { if (!context) throw new Error('missing snapshot context'); context['flowsafe.runProvenance'].startIdentity.target.threadId = 'other-thread'; + context['flowsafe.runProvenance'].startIdentity.target.resourceId = + 'other-thread'; Object.assign(context, { threadId: 'other-thread', resourceId: 'other-thread', diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index e9a5cb34..896eb32e 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -2151,6 +2151,7 @@ export function createThreadAgentHost( kind: 'agent', id: ref.agentId, threadId: ref.threadId, + resourceId: ref.resourceId, }), }, ); diff --git a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts index aaee0de8..bd473311 100644 --- a/packages/flowsafe/src/agent-runner/durable-agent-runner.ts +++ b/packages/flowsafe/src/agent-runner/durable-agent-runner.ts @@ -2544,7 +2544,8 @@ export class FlowsafeDurableAgent< throw new Error('agent observation identity is malformed'); const observedAgentId = identity.target.id; const observedThreadId = identity.target.threadId; - const observedResourceId = resourceIdFromKey(observedThreadId); + const observedResourceId = + identity.target.resourceId ?? resourceIdFromKey(observedThreadId); const context = state.snapshot.requestContext; const record = (value: unknown): Record => { if (value === null || typeof value !== 'object' || Array.isArray(value)) diff --git a/packages/flowsafe/src/do-runner/execution-admission.test.ts b/packages/flowsafe/src/do-runner/execution-admission.test.ts index 48187f00..54eaac8b 100644 --- a/packages/flowsafe/src/do-runner/execution-admission.test.ts +++ b/packages/flowsafe/src/do-runner/execution-admission.test.ts @@ -71,6 +71,21 @@ describe('execution identity and epoch helpers', () => { .tablePrefix, ).toBe('a'.repeat(39)); expect(normalizeStartIdentity(START)).toEqual(START); + expect( + normalizeStartIdentity({ + ...START, + target: { ...START.target, resourceId: 'account-7' }, + }), + ).toEqual({ + ...START, + target: { ...START.target, resourceId: 'account-7' }, + }); + expect(() => + normalizeStartIdentity({ + ...START, + target: { ...START.target, resourceId: 'bad/resource' }, + }), + ).toThrow(InvalidExecutionIdentityError); const workflow = { owner: START.owner, target: { kind: 'workflow', id: 'logical-parent' }, diff --git a/packages/flowsafe/src/do-runner/execution-admission.ts b/packages/flowsafe/src/do-runner/execution-admission.ts index 331b1716..6cad53d1 100644 --- a/packages/flowsafe/src/do-runner/execution-admission.ts +++ b/packages/flowsafe/src/do-runner/execution-admission.ts @@ -87,6 +87,7 @@ export interface StartIdentity { readonly kind: 'agent'; readonly id: string; readonly threadId: string; + readonly resourceId?: string; }; } @@ -112,6 +113,8 @@ const IDENTITY_ERRORS = { 'target.id': 'target.id must be a URL-path-safe identifier', 'target.threadId': 'target.threadId is required only for an agent target and must be URL-path-safe', + 'target.resourceId': + 'target.resourceId is valid only for an agent target and must be URL-path-safe', } as const; export class InvalidExecutionIdentityError extends DoStatusError { @@ -226,7 +229,10 @@ export function normalizeStartIdentity(value: unknown): StartIdentity { 'identity', ); const { kind: ownerKind, id: ownerId } = identityObject(rawOwner, 'owner'); - const { kind, id, threadId } = identityObject(rawTarget, 'target'); + const { kind, id, threadId, resourceId } = identityObject( + rawTarget, + 'target', + ); if (!isExecutionPrincipalKind(ownerKind)) throw new InvalidExecutionIdentityError('owner.kind'); if (!isExecutionPrincipalId(ownerId)) @@ -238,13 +244,22 @@ export function normalizeStartIdentity(value: unknown): StartIdentity { if (kind === 'agent') { if (!isPathSafeId(threadId)) throw new InvalidExecutionIdentityError('target.threadId'); + if (resourceId !== undefined && !isPathSafeId(resourceId)) + throw new InvalidExecutionIdentityError('target.resourceId'); return Object.freeze({ owner, - target: Object.freeze({ kind, id, threadId }), + target: Object.freeze({ + kind, + id, + threadId, + ...(resourceId === undefined ? {} : { resourceId }), + }), }); } if (threadId !== undefined) throw new InvalidExecutionIdentityError('target.threadId'); + if (resourceId !== undefined) + throw new InvalidExecutionIdentityError('target.resourceId'); return Object.freeze({ owner, target: Object.freeze({ kind, id }) }); } diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 27164721..cbff5515 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -92,6 +92,7 @@ describe('remote step journal dispatch admission', () => { kind: 'agent', id: ADDRESS.agentId, threadId: ADDRESS.threadId, + resourceId: ADDRESS.resourceId, }, }, }, @@ -190,6 +191,77 @@ describe('remote step journal dispatch admission', () => { ]); }); + it('accepts the resource recorded at start when the host maps a thread elsewhere', async () => { + const h = await fixture(); + const original = h.runtime.authoritativeStartState.bind(h.runtime); + h.runtime.authoritativeStartState = (async ( + workflowId: string, + runId: string, + ) => { + const state = await original(workflowId, runId); + if (!state) return state; + return { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + ...state.provenance.startIdentity, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + resourceId: 'account-7', + }, + }, + }, + } as typeof state; + }) as typeof h.runtime.authoritativeStartState; + + const invocation = await h.journal.register( + { ...ADDRESS, resourceId: 'account-7' }, + 10_000, + PRINCIPAL, + ); + expect(invocation.resourceId).toBe('account-7'); + }); + + it('refuses remote dispatch when legacy start provenance lacks a resource', async () => { + const h = await fixture(); + const original = h.runtime.authoritativeStartState.bind(h.runtime); + h.runtime.authoritativeStartState = (async ( + workflowId: string, + runId: string, + ) => { + const state = await original(workflowId, runId); + if (!state) return state; + return { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + ...state.provenance.startIdentity, + target: { + kind: 'agent', + id: ADDRESS.agentId, + threadId: ADDRESS.threadId, + }, + }, + }, + } as typeof state; + }) as typeof h.runtime.authoritativeStartState; + + await expect( + h.journal.register(ADDRESS, 10_000, PRINCIPAL), + ).rejects.toMatchObject({ + status: 503, + }); + expect( + h.sqlite + .prepare(`SELECT count(*) AS count FROM ${REMOTE_STEP_TABLE}`) + .get(), + ).toEqual({ count: 0 }); + }); + it('upgrades a prior journal while preserving an existing invocation', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); @@ -861,6 +933,7 @@ describe('remote step journal dispatch admission', () => { kind: 'agent', id: ADDRESS.agentId, threadId: ADDRESS.threadId, + resourceId: ADDRESS.resourceId, }, }, agentStart: { threaded: false }, diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 613f9e68..01b2b845 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -15,7 +15,6 @@ import { executionFenceAdmissionSql, executionFenceAdmissionValues, } from './execution-fence.js'; -import { resourceIdFromKey } from './memory-id.js'; import { isPathSafeId } from './path-safe-id.js'; import { lifecycleFromRequestContext, @@ -750,7 +749,11 @@ export class RemoteStepJournal { address.runId, ); const target = state?.provenance.startIdentity?.target; - if (state?.storage !== 'd1' || target?.kind !== 'agent') + if ( + state?.storage !== 'd1' || + target?.kind !== 'agent' || + !isPathSafeId(target.resourceId) + ) throw new RemoteStepRefusal( 503, 'remote step run identity is unreadable', @@ -762,7 +765,7 @@ export class RemoteStepJournal { (state.provenance.startIdentity?.owner.kind !== principal.kind || state.provenance.startIdentity.owner.id !== principal.id)) || state.execution.tablePrefix !== address.tablePrefix || - address.resourceId !== resourceIdFromKey(address.threadId) + address.resourceId !== target.resourceId ) throw new RemoteStepRefusal(409, 'remote step run identity changed'); return state; From 3c207d40548eba78070a3a2cb6e5aae48c0e1a30 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Mon, 12 Oct 2026 00:37:15 +0400 Subject: [PATCH 19/73] fix(flowsafe): bind host resume to start resource Resolve trusted resource identity for host starts and callbacks. Reject resumed and recovered runs under a different persisted start resource; fail closed for ambiguous legacy mappings. Focused host tests, Biome, and test typecheck pass. --- .../src/agent-host/thread-host.test.ts | 90 +++++++++++++++++- .../flowsafe/src/agent-host/thread-host.ts | 91 ++++++++++++++++--- 2 files changed, 166 insertions(+), 15 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 532ff57f..3bcac182 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -9,6 +9,7 @@ import { nestedArray } from '../../test-support/deep-json.js'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; import { AgentRunSelectorMismatchError, + type AgentStartAuthority, FlowsafeDurableAgent, } from '../agent-runner/durable-agent-runner.js'; import { @@ -273,6 +274,7 @@ function harness( ) => Promise; remoteSteps?: RemoteStepContinuationReader; stateStorage?: AgentThreadStateStorage; + resourceIdFromKey?: (threadId: string) => string; } = {}, ): Harness { const state = new Map(); @@ -382,7 +384,12 @@ function harness( kind: options.principal?.kind ?? 'human', id: options.principal?.id ?? 'operator-1', }, - target: { kind: 'agent', id: 'writer', threadId: 'acme_thread' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + }, }; const current = snapshot as { requestContext: Record; @@ -501,6 +508,9 @@ function harness( }, }; const host = createThreadAgentHost({ + ...(options.resourceIdFromKey + ? { resourceIdFromKey: options.resourceIdFromKey } + : {}), ...(options.authorizeAutomatedEntry ? { authorizeAutomatedEntry: options.authorizeAutomatedEntry } : {}), @@ -2121,6 +2131,23 @@ describe('createThreadAgentHost owner recovery', () => { }); describe('createThreadAgentHost', () => { + it('starts under a trusted nonidentity resource mapping', async () => { + const f = harness(['writer'], { + resourceIdFromKey: () => 'account-7', + }); + + await f.host.start(f.scope, { + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'account-7', + runId: 'account-run', + prompt: 'go', + entryPath: 'http.start', + }); + + const authority = mocked.stream.mock.calls[0]?.[7] as AgentStartAuthority; + expect(authority.startIdentity.target.resourceId).toBe('account-7'); + }); it.each([ ['top-level', [{ role: 'system', content: 'x' }]], ['once-nested', [[{ role: 'system', content: 'x' }]]], @@ -7245,6 +7272,7 @@ async function hostAgentLifecycleFixture( journal?: boolean; lifecycle?: boolean; snapshotTarget?: { agentId: string; threadId: string }; + resourceIdFromKey?: (threadId: string) => string; } = {}, ) { const core = await vi.importActual( @@ -7306,6 +7334,9 @@ async function hostAgentLifecycleFixture( }; const dispatch = vi.fn(async () => {}); const fixture = harness(['writer'], { + ...(input.resourceIdFromKey + ? { resourceIdFromKey: input.resourceIdFromKey } + : {}), storage, init: app, resourceAccess: resources, @@ -7605,9 +7636,66 @@ it('reads a retained remote result from D1 after the agent host is recreated', a expectedSuspension: { suspendedAt: 1000 }, }), ); + await fixture.stateStorage.put(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: 'other-resource', + }); + const displaced = await fresh.host + .route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + fresh.scope, + ) + .catch((error) => error); + expect(displaced).toMatchObject({ status: 404 }); + expect(resume).toHaveBeenCalledTimes(1); }); describe('agent host selector lookup isolation', () => { + it.each([ + ['modern', 'status'], + ['modern', 'resume'], + ['v1', 'status'], + ] as const)('refuses an unthreaded %s run started under an older resource mapping on %s', async (provenance, route) => { + const f = await hostAgentLifecycleFixture({ + provenance, + threaded: false, + resourceIdFromKey: () => 'account-7', + }); + onTestFinished(() => { + vi.restoreAllMocks(); + f.sql.close(); + }); + const resume = vi.spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime'); + const request = + route === 'status' + ? new Request( + 'https://thread/_flowsafe/agent-host/runs/writer/acme_run?resourceId=account-7', + ) + : new Request('https://thread/_flowsafe/agent-host/resume', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'account-7', + runId: 'acme_run', + requestedBy: 'reviewer-2', + entryPath: 'approval.resume', + resumeData: {}, + }), + }); + const outcome = await f.host + .route(request, f.scope) + .catch((error) => error); + if (route === 'status') expect(outcome).toMatchObject({ status: 404 }); + else expect(outcome).toBeInstanceOf(AgentRunSelectorMismatchError); + expect(resume).not.toHaveBeenCalled(); + }); + it.each([ ['modern', true, 'status'], ['modern', false, 'stream'], diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index 896eb32e..3ac5c93a 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -174,6 +174,7 @@ export type { export interface ThreadAgentHostOptions { remoteSteps?: RemoteStepContinuationReader; + resourceIdFromKey?: (threadId: string) => string; /** Narrows automated entry beyond what each agent's metadata declares. */ authorizeAutomatedEntry?: AutomatedEntryAuthorizer; /** @@ -623,6 +624,17 @@ function ndjson( export function createThreadAgentHost( options: ThreadAgentHostOptions, ): ThreadAgentHost { + const resourceForThread = (threadId: string): string => { + let resourceId: string; + try { + resourceId = (options.resourceIdFromKey ?? resourceIdFromKey)(threadId); + } catch { + throw new AgentHostRequestError(503, 'resource mapping is unavailable'); + } + if (!isPathSafeId(resourceId)) + throw new AgentHostRequestError(503, 'resource mapping is unavailable'); + return resourceId; + }; let catalogPromise: Promise | undefined; let runtime: | { @@ -886,7 +898,7 @@ export function createThreadAgentHost( scope, { agentId: runRecord.agentId, - resourceId: resourceIdFromKey(scope.threadId), + resourceId: resourceForThread(scope.threadId), runId, }, { includeLegacy: true }, @@ -1059,7 +1071,7 @@ export function createThreadAgentHost( !isPathSafeId(stored.agentId) || !isPathSafeId(stored.threadId) || !isPathSafeId(stored.resourceId) || - stored.resourceId !== resourceIdFromKey(threadId) || + stored.resourceId !== resourceForThread(threadId) || !isPathSafeId(stored.runId) || !isPathSafeId(stored.token) || ownerRecoveryKey(stored.runId) !== key || @@ -1206,6 +1218,14 @@ export function createThreadAgentHost( }; type NormalAgentRunState = AuthoritativeAgentStartState | LegacyAgentRunState; + const matchesStartResource = ( + threadId: string, + resourceId: string, + persistedResourceId: string | undefined, + ): boolean => + persistedResourceId === undefined + ? resourceId === resourceIdFromKey(threadId) + : persistedResourceId === resourceId; async function selectedAgentState( scope: AgentThreadInstanceScope, @@ -1222,23 +1242,47 @@ export function createThreadAgentHost( readOptions?: { readonly includeLegacy: true }, ): Promise { const includeLegacy = readOptions?.includeLegacy === true; - if (ref.resourceId !== resourceIdFromKey(scope.threadId)) + if (ref.resourceId !== resourceForThread(scope.threadId)) throw new AgentHostRequestError(404, 'run not found'); const current = await runtimeFor(scope); const durable = current.agents.get(ref.agentId); if (!current.catalog.get(ref.agentId) || !durable) throw new AgentHostRequestError(404, 'agent not found'); + const assertSelectedResource = ( + state: T, + ): T => { + const target = + state && state.kind !== 'legacy' ? state.execution.target : undefined; + if ( + state && + (target?.kind === 'workflow' || + !matchesStartResource( + scope.threadId, + ref.resourceId, + target?.kind === 'agent' ? target.resourceId : undefined, + )) + ) + throw new AgentRunSelectorMismatchError( + durable.getWorkflow().id, + ref.runId, + ); + return state; + }; if (includeLegacy) - return durable.authoritativeAgentStartState( + return assertSelectedResource( + await durable.authoritativeAgentStartState( + scope.init.runtime, + scope.threadId, + ref.runId, + { includeLegacy: true }, + ), + ); + return assertSelectedResource( + await durable.authoritativeAgentStartState( scope.init.runtime, scope.threadId, ref.runId, - { includeLegacy: true }, - ); - return durable.authoritativeAgentStartState( - scope.init.runtime, - scope.threadId, - ref.runId, + ), ); } @@ -1271,6 +1315,11 @@ export function createThreadAgentHost( state.execution.target.kind !== 'agent' || state.execution.target.id !== recovery.agentId || state.execution.target.threadId !== recovery.threadId || + !matchesStartResource( + recovery.threadId, + recovery.resourceId, + state.execution.target.resourceId, + ) || state.execution.owner.kind !== recovery.runRecord.principal.kind || state.execution.owner.id !== recovery.runRecord.principal.id ) @@ -1365,7 +1414,7 @@ export function createThreadAgentHost( if (state.kind === 'legacy') { const ref = { agentId: expected.agentId, - resourceId: resourceIdFromKey(scope.threadId), + resourceId: resourceForThread(scope.threadId), runId, }; await assertLegacyTerminalCurrent(scope, ref, state, expected); @@ -1380,6 +1429,11 @@ export function createThreadAgentHost( state.execution.target.kind !== 'agent' || state.execution.target.id !== expected.agentId || state.execution.target.threadId !== scope.threadId || + !matchesStartResource( + scope.threadId, + resourceForThread(scope.threadId), + state.execution.target.resourceId, + ) || state.execution.owner.kind !== expected.principal.kind || state.execution.owner.id !== expected.principal.id ) @@ -2220,7 +2274,7 @@ export function createThreadAgentHost( entry, scope.principal, ); - if (ref.resourceId !== resourceIdFromKey(scope.threadId)) { + if (ref.resourceId !== resourceForThread(scope.threadId)) { throw new AgentHostRequestError(404, 'run not found'); } const execution: TrustedAgentExecution = { @@ -2469,7 +2523,7 @@ export function createThreadAgentHost( resolveBoundAgent: async (scope, input) => { const binding = await readBinding(); if (!binding) throw new AgentHostRequestError(404, 'agent not found'); - if (binding.resourceId !== resourceIdFromKey(scope.threadId)) { + if (binding.resourceId !== resourceForThread(scope.threadId)) { throw new AgentHostRequestError(404, 'agent not found'); } if (input.agentId && input.agentId !== binding.agentId) { @@ -2610,7 +2664,7 @@ export function createThreadAgentHost( const binding = await readBinding(); if ( !resourceId || - resourceId !== resourceIdFromKey(scope.threadId) || + resourceId !== resourceForThread(scope.threadId) || binding?.resourceId !== resourceId || (agentId !== null && binding.agentId !== agentId) ) { @@ -2743,6 +2797,7 @@ export function createThreadAgentHost( state.execution.target.kind !== 'agent' || state.execution.target.id !== invocation.agentId || state.execution.target.threadId !== invocation.threadId || + state.execution.target.resourceId !== invocation.resourceId || state.execution.owner.kind !== invocation.principal.kind || state.execution.owner.id !== invocation.principal.id ) @@ -2754,6 +2809,14 @@ export function createThreadAgentHost( !samePrincipal(stored.principal, invocation.principal) ) throw new AgentHostRequestError(404, 'run not found'); + const binding = await readBinding(); + if ( + (selected.threaded && + (binding?.agentId !== ref.agentId || + binding.resourceId !== ref.resourceId)) || + (!selected.threaded && binding) + ) + throw new AgentHostRequestError(404, 'run not found'); const key = invocation.stepPath.join('.'); const summary = state.summary; const payload = From 4096fbddc028f8b321ac687adfa02c7b3faa27b2 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Mon, 12 Oct 2026 03:31:06 +0400 Subject: [PATCH 20/73] feat(flowsafe): settle retained remote results durably Version the journal, preserve known rows, index active results, and bind witnessed settlement to the exact resume owner. Expired uncertain resumes stay fail-closed across overlapping isolates; durable replay requires a separate effect-fencing protocol. --- .../src/agent-host/thread-host.test.ts | 299 ++++++++++++- .../flowsafe/src/agent-host/thread-host.ts | 33 +- .../src/agent-host/thread-topology.test.ts | 13 + .../src/agent-host/thread-topology.ts | 18 + packages/flowsafe/src/do-runner/inventory.ts | 2 +- .../src/do-runner/remote-step-journal.test.ts | 205 ++++++++- .../src/do-runner/remote-step-journal.ts | 401 +++++++++++++++++- 7 files changed, 936 insertions(+), 35 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 3bcac182..d8ee2518 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -4045,6 +4045,8 @@ describe('createThreadAgentHost', () => { quiescenceDigest: 'evidence', executorQuiescentAt: 1000, })), + claimResume: vi.fn(async () => 'lease-1'), + settleAfterResume: vi.fn(async () => 'applied' as const), }; const fixture = harness(['writer'], { remoteSteps: reader, @@ -4064,11 +4066,12 @@ describe('createThreadAgentHost', () => { }, }, }); - mocked.resumeViaRuntime.mockResolvedValue({ + const resumed = { runId: 'acme_run', - status: 'suspended', + status: 'suspended' as const, suspended: [['next']], - }); + }; + mocked.resumeViaRuntime.mockResolvedValue(resumed); const request = new Request( 'https://thread/_flowsafe/agent-host/remote-step-resume', { @@ -4092,6 +4095,12 @@ describe('createThreadAgentHost', () => { expectedSuspension: { suspendedAt: 1000 }, }), ); + expect(reader.claimResume).toHaveBeenCalledWith('invocation_1', 60_000); + expect(reader.settleAfterResume).toHaveBeenCalledWith( + 'invocation_1', + 'lease-1', + resumed, + ); mocked.resumeViaRuntime.mockClear(); fixture.setSummary({ @@ -4118,6 +4127,227 @@ describe('createThreadAgentHost', () => { expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); }); + it.each([ + ['terminal-not-success', 409], + ['deadline-expired', 409], + ['retry', 503], + ['advanced-without-proof', 503], + ] as const)('refuses a stale successful resume response when settlement is %s', async (settlement, expectedStatus) => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'result-retained', + permitGeneration: 1, + }), + readRetainedResult: async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + }), + claimResume: vi.fn(async () => 'lease-1'), + settleAfterResume: vi.fn(async () => settlement), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + const stale = { runId: 'acme_run', status: 'success' as const }; + mocked.resumeViaRuntime.mockResolvedValue(stale); + + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: expectedStatus }); + expect(reader.settleAfterResume).toHaveBeenCalledWith( + 'invocation_1', + 'lease-1', + stale, + ); + }); + + it('waits for an in-flight resume before considering its expired lease', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const entered = deferredSignal(); + const release = deferredSignal(); + let status: 'result-retained' | 'resuming' | 'settled' = 'result-retained'; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status, + permitGeneration: 1, + ...(status === 'resuming' + ? { resumeLeaseExpiresAt: Date.now() - 1 } + : {}), + }), + readRetainedResult: async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + }), + claimResume: vi.fn(async () => { + status = 'resuming'; + return 'lease-1'; + }), + settleAfterResume: vi.fn(async () => { + status = 'settled'; + return 'applied' as const; + }), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ + runId: 'acme_run', + status: 'suspended', + suspended: [['tool']], + suspendedAt: { tool: 1000 }, + suspendPayload: { + tool: { + type: 'suspension', + toolCallSuspended: { remoteStep: REMOTE_STEP_MARKER }, + }, + }, + }); + mocked.resumeViaRuntime.mockImplementation(async () => { + entered.resolve(); + await release.promise; + return { runId: 'acme_run', status: 'suspended' }; + }); + const request = () => + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }); + const first = fixture.host.route(request(), fixture.scope); + await entered.promise; + const second = fixture.host + .route(request(), fixture.scope) + .catch((error) => error); + try { + await new Promise((resolve) => setImmediate(resolve)); + expect(reader.claimResume).toHaveBeenCalledTimes(1); + } finally { + release.resolve(); + } + expect((await first)?.status).toBe(200); + expect(await second).toMatchObject({ status: 404 }); + expect(reader.claimResume).toHaveBeenCalledTimes(1); + expect(mocked.resumeViaRuntime).toHaveBeenCalledTimes(1); + }); + + it('refuses an expired lease even when the run advanced', async () => { + const principal: ExecutionPrincipal = { + kind: 'human', + id: 'operator-1', + role: 'operator', + }; + const reader: RemoteStepContinuationReader = { + deploymentId: 'deployment', + tablePrefix: '', + get: async () => ({ + invocationId: 'invocation_1', + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + principal, + startToken: 'test-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: {}, + deadlineAt: 9999999999999, + status: 'resuming', + resumeLeaseExpiresAt: Date.now() - 1, + permitGeneration: 1, + }), + readRetainedResult: vi.fn(async () => ({ + invocationId: 'invocation_1', + resultJson: '{"answer":1}', + resultDigest: 'digest', + quiescenceDigest: 'evidence', + executorQuiescentAt: 1000, + })), + claimResume: vi.fn(async () => 'lease-2'), + settleAfterResume: vi.fn(async () => 'applied' as const), + }; + const fixture = harness(['writer'], { remoteSteps: reader }); + seedSuspendedApprovalRun(fixture); + fixture.setSummary({ runId: 'acme_run', status: 'success' }); + + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: 'invocation_1' }), + }), + fixture.scope, + ), + ).rejects.toMatchObject({ status: 503 }); + expect(reader.claimResume).not.toHaveBeenCalled(); + expect(reader.readRetainedResult).not.toHaveBeenCalled(); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + it('refuses a reserved timeout envelope in a retained remote result', async () => { const principal: ExecutionPrincipal = { kind: 'human', @@ -4152,6 +4382,8 @@ describe('createThreadAgentHost', () => { quiescenceDigest: 'evidence', executorQuiescentAt: 1000, }), + claimResume: async () => 'lease-1', + settleAfterResume: async () => 'applied' as const, }; const fixture = harness(['writer'], { remoteSteps: reader }); seedSuspendedApprovalRun(fixture); @@ -7278,6 +7510,7 @@ async function hostAgentLifecycleFixture( const core = await vi.importActual( '@mastra/core/mastra', ); + const { InMemoryStore } = await import('@mastra/core/storage'); const breakwater = await vi.importActual< typeof import('@proofoftech/breakwater') @@ -7613,11 +7846,34 @@ it('reads a retained remote result from D1 after the agent host is recreated', a tablePrefix: 'host_lifecycle_', get: reopened.get.bind(reopened), readRetainedResult: reopened.readRetainedResult.bind(reopened), + claimResume: reopened.claimResume.bind(reopened), + settleAfterResume: async () => 'applied' as const, }, }); const resume = vi .spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime') .mockResolvedValue({ runId: 'acme_run', status: 'suspended' }); + await fixture.stateStorage.put(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: 'other-resource', + }); + const displaced = await fresh.host + .route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + fresh.scope, + ) + .catch((error) => error); + expect(displaced).toMatchObject({ status: 404 }); + expect(resume).not.toHaveBeenCalled(); + await fixture.stateStorage.put(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: RESOURCE_ID, + }); const response = await fresh.host.route( new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { method: 'POST', @@ -7636,21 +7892,38 @@ it('reads a retained remote result from D1 after the agent host is recreated', a expectedSuspension: { suspendedAt: 1000 }, }), ); - await fixture.stateStorage.put(THREAD_BINDING_KEY, { - version: 1, - agentId: 'writer', - resourceId: 'other-resource', - }); - const displaced = await fresh.host - .route( + const leaseBefore = await fixture.binding + .prepare( + 'SELECT resume_lease_owner FROM flowsafe_remote_step_invocations WHERE invocation_id = ?', + ) + .bind(invocation.invocationId) + .all(); + expect(leaseBefore.results).toMatchObject([ + { resume_lease_owner: expect.any(String) }, + ]); + fixture.binding + .prepare( + 'UPDATE flowsafe_remote_step_invocations SET resume_lease_expires_at = ? WHERE invocation_id = ?', + ) + .bind(Date.now() - 1, invocation.invocationId) + .run(); + await expect( + fresh.host.route( new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { method: 'POST', body: JSON.stringify({ invocationId: invocation.invocationId }), }), fresh.scope, - ) - .catch((error) => error); - expect(displaced).toMatchObject({ status: 404 }); + ), + ).rejects.toMatchObject({ status: 503 }); + expect( + await fixture.binding + .prepare( + 'SELECT resume_lease_owner FROM flowsafe_remote_step_invocations WHERE invocation_id = ?', + ) + .bind(invocation.invocationId) + .all(), + ).toMatchObject({ results: leaseBefore.results }); expect(resume).toHaveBeenCalledTimes(1); }); diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index 3ac5c93a..178ba949 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -2768,16 +2768,25 @@ export function createThreadAgentHost( if (!isPathSafeId(invocationId)) throw new AgentHostRequestError(404, 'invocation not found'); const invocation = await reader.get(invocationId); + if ( + !invocation || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + invocation.threadId !== scope.threadId || + !samePrincipal(invocation.principal, scope.principal) + ) + throw new AgentHostRequestError(404, 'invocation not found'); + if (invocation.status === 'resuming') + throw new AgentHostRequestError( + 503, + 'remote step resume awaits executor quiescence', + ); const retained = await reader.readRetainedResult(invocationId); if ( !invocation || !retained || retained.invocationId !== invocationId || invocation.status !== 'result-retained' || - invocation.deploymentId !== reader.deploymentId || - invocation.tablePrefix !== reader.tablePrefix || - invocation.threadId !== scope.threadId || - !samePrincipal(invocation.principal, scope.principal) || invocation.deadlineAt <= Date.now() ) throw new AgentHostRequestError(404, 'invocation not found'); @@ -2877,6 +2886,7 @@ export function createThreadAgentHost( principalPermissions, safeContext: selected.safeContext, }; + const leaseOwner = await reader.claimResume(invocationId, 60_000); const resumed = await withExecution(execution, () => durable.resumeViaRuntime({ runId: ref.runId, @@ -2900,6 +2910,21 @@ export function createThreadAgentHost( : {}), }), ); + const settlement = await reader.settleAfterResume( + invocationId, + leaseOwner, + resumed, + ); + if (settlement === 'retry' || settlement === 'advanced-without-proof') + throw new AgentHostRequestError( + 503, + 'remote step resume outcome is unresolved', + ); + if (settlement !== 'applied') + throw new AgentHostRequestError( + 409, + 'remote step result was not applied', + ); const result = await envelopeFor( scope, ref, diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 25cdc84c..f76e3300 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -133,6 +133,8 @@ it('routes a retained result under the journaled principal and sends only its id quiescenceDigest: 'evidence', executorQuiescentAt: 1000, })), + claimResume: vi.fn(async () => 'lease-1'), + settleAfterResume: vi.fn(async () => 'applied' as const), }; const { topology, hits } = harness(reader); await topology.resumeRemoteStep(context().value, 'invocation_1'); @@ -144,6 +146,17 @@ it('routes a retained result under the journaled principal and sends only its id expect(sent?.init?.headers?.['x-flowsafe-principal']).toBe( '{"kind":"human","id":"original","role":"operator"}', ); + const stored = await reader.get('invocation_1'); + if (!stored) throw new Error('invocation was not found'); + vi.mocked(reader.get).mockResolvedValue({ + ...stored, + status: 'resuming', + resumeLeaseExpiresAt: Date.now() - 1, + }); + await expect( + topology.resumeRemoteStep(context().value, 'invocation_1'), + ).rejects.toMatchObject({ status: 503 }); + expect(hits).toHaveLength(1); }); function context() { diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index eb01a237..44fc33fd 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -38,6 +38,7 @@ import { isTerminalRunStatus, type RunStatus, } from '../do-runner/run-terminal-state.js'; +import type { RunSummary } from '../do-runner/runtime.js'; import { captureReservation, type StartReservationOwner, @@ -134,6 +135,18 @@ export interface RemoteStepContinuationReader { readRetainedResult( invocationId: string, ): Promise; + claimResume(invocationId: string, leaseMs: number): Promise; + settleAfterResume( + invocationId: string, + leaseOwner: string, + witnessed?: RunSummary, + ): Promise< + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof' + >; } export interface AgentThreadRunRef { @@ -761,6 +774,11 @@ export function createAgentThreadTopology( if (!isPathSafeId(invocationId)) throw new RunRouteError(404, 'invocation not found'); const invocation = await reader.get(invocationId); + if (invocation?.status === 'resuming') + throw new RunRouteError( + 503, + 'remote step resume awaits executor quiescence', + ); if ( invocation?.status !== 'result-retained' || invocation.deploymentId !== reader.deploymentId || diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index a796bff6..95185917 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -1043,7 +1043,7 @@ export class DeploymentInventory { 'permit_consumed_at', ], where: - "status <> 'failed' OR (permit_consumed_at IS NOT NULL AND executor_quiescent_at IS NULL)", + "status NOT IN ('failed', 'settled') OR settlement_outcome IN ('advanced-without-proof', 'deadline-expired') OR (status = 'settled' AND (settlement_outcome IS NOT 'applied' OR result_json IS NULL OR result_digest IS NULL OR executor_quiescent_at IS NULL)) OR (permit_consumed_at IS NOT NULL AND executor_quiescent_at IS NULL)", binds: [], }; case 'approvals-waiting': diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index cbff5515..05d7ba9a 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -137,6 +137,22 @@ describe('remote step journal dispatch admission', () => { setSummary: (value: RunSummary) => { summary = value; }, + retainResult: async () => { + const invocation = await journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await journal.issuePermit( + invocation.invocationId, + 1_000, + ); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ); + return { invocation, rawPermit: consumed.rawPermit }; + }, setRequest: (account: string) => { summary = { ...summary, @@ -265,7 +281,12 @@ describe('remote step journal dispatch admission', () => { it('upgrades a prior journal while preserving an existing invocation', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); for (const column of [ + 'settlement_outcome', + 'settled_at', + 'resume_lease_expires_at', + 'resume_lease_owner', 'executor_quiescent_at', 'quiescence_digest', 'result_digest', @@ -288,9 +309,47 @@ describe('remote step journal dispatch admission', () => { ); }); + it('upgrades a live v2 result without losing its retained bytes', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const permit = await h.journal.issuePermit(invocation.invocationId, 1_000); + const consumed = await h.journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await h.journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"value":"kept"}', + '{"receipt":"verified"}', + ); + h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); + for (const column of [ + 'settlement_outcome', + 'settled_at', + 'resume_lease_expires_at', + 'resume_lease_owner', + ]) + h.sqlite.exec(`ALTER TABLE ${REMOTE_STEP_TABLE} DROP COLUMN ${column}`); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 2'); + + await h.reopen().initialize(); + + expect( + await h.reopen().readRetainedResult(invocation.invocationId), + ).toMatchObject({ + resultJson: '{"value":"kept"}', + }); + expect( + h.sqlite.prepare('SELECT version FROM flowsafe_remote_step_schema').get(), + ).toEqual({ version: 3 }); + expect(await h.reopen().recoverableResults(1)).toEqual({ + invocationIds: [invocation.invocationId], + nextCursor: invocation.invocationId, + }); + }); + it('refuses an unknown journal schema version before dispatch', async () => { const h = await fixture(); - h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 3'); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 4'); await expect(h.reopen().initialize()).rejects.toMatchObject({ status: 503, @@ -355,7 +414,12 @@ describe('remote step journal dispatch admission', () => { it('converges when another initializer wins the same journal upgrade', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); for (const column of [ + 'settlement_outcome', + 'settled_at', + 'resume_lease_expires_at', + 'resume_lease_owner', 'executor_quiescent_at', 'quiescence_digest', 'result_digest', @@ -552,6 +616,145 @@ describe('remote step journal dispatch admission', () => { ).toBe(1); }); + it('settles a witnessed new suspension and keeps callback replay idempotent', async () => { + const h = await fixture(); + const { invocation, rawPermit } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 1_000); + await expect( + h.reopen().claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + const next: RunSummary = { + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['next']], + suspendedAt: { next: 1_100 }, + suspendPayload: { next: { type: 'suspension' } }, + }; + expect( + await h.reopen().settleAfterResume(invocation.invocationId, owner, next), + ).toBe('retry'); + const nextOwner = await h.journal.claimResume( + invocation.invocationId, + 1_000, + ); + h.setSummary(next); + expect( + await h + .reopen() + .settleAfterResume(invocation.invocationId, nextOwner, next), + ).toBe('applied'); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'settled', + settlementOutcome: 'applied', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(0); + expect(await h.journal.recoverableResults(1)).toEqual({ + invocationIds: [], + }); + expect( + await h.journal.recoverableResults(1, invocation.invocationId), + ).toEqual({ invocationIds: [] }); + expect( + await h.journal.recordResult( + invocation.invocationId, + rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'replay' }); + expect( + await h.journal.readRetainedResult(invocation.invocationId), + ).toBeUndefined(); + h.sqlite + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET result_digest = NULL`) + .run(); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('retains an expired claim across another journal instance until its owner settles', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 100); + expect(await h.reopen().recoverableResults(10)).toEqual({ + invocationIds: [], + }); + h.setNow(1_101); + expect(await h.reopen().recoverableResults(10)).toEqual({ + invocationIds: [invocation.invocationId], + }); + await expect( + h.journal.claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + expect(await h.reopen().reconcileResult(invocation.invocationId)).toBe( + 'pending-quiescence', + ); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'resuming', + }); + expect( + h.sqlite + .prepare(`SELECT resume_lease_owner FROM ${REMOTE_STEP_TABLE}`) + .get(), + ).toEqual({ resume_lease_owner: owner }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + h.setSummary({ runId: ADDRESS.runId, status: 'success' }); + expect( + await h.journal.settleAfterResume(invocation.invocationId, owner, { + runId: ADDRESS.runId, + status: 'success', + }), + ).toBe('applied'); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'settled', + settlementOutcome: 'applied', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(0); + }); + + it('retains uncertain advancement when no resume lease was claimed', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + h.setSummary({ runId: ADDRESS.runId, status: 'success' }); + expect(await h.reopen().reconcileResult(invocation.invocationId)).toBe( + 'advanced-without-proof', + ); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'failed', + settlementOutcome: 'advanced-without-proof', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + await expect( + h.journal.claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + }); + + it('records cancellation separately from a successful result application', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 1_000); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + expect( + await h.journal.settleAfterResume(invocation.invocationId, owner), + ).toBe('terminal-not-success'); + expect(await h.journal.get(invocation.invocationId)).toMatchObject({ + status: 'failed', + settlementOutcome: 'terminal-not-success', + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(0); + }); + it('records verified completion from a prior epoch without resuming a locked run', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 01b2b845..2631daae 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -25,12 +25,20 @@ import { validateTablePrefix } from './table-prefix.js'; export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; export const REMOTE_STEP_SCHEMA_TABLE = 'flowsafe_remote_step_schema'; +const REMOTE_STEP_RECOVERY_INDEX = 'flowsafe_remote_step_recovery'; +const REMOTE_STEP_RECOVERY_INDEX_SQL = `CREATE INDEX ${REMOTE_STEP_RECOVERY_INDEX} ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE status IN ('result-retained', 'resuming')`; const REMOTE_STEP_RESULT_COLUMNS = [ 'result_json', 'result_digest', 'quiescence_digest', 'executor_quiescent_at', ] as const; +const REMOTE_STEP_SETTLEMENT_COLUMNS = [ + 'resume_lease_owner', + 'resume_lease_expires_at', + 'settled_at', + 'settlement_outcome', +] as const; const REMOTE_STEP_COLUMNS = [ 'invocation_id', 'principal_json', @@ -55,6 +63,7 @@ const REMOTE_STEP_COLUMNS = [ 'permit_consumed_at', 'permit_digest', ...REMOTE_STEP_RESULT_COLUMNS, + ...REMOTE_STEP_SETTLEMENT_COLUMNS, ] as const; const REMOTE_STEP_INTEGER_COLUMNS = new Set([ 'suspended_at', @@ -66,6 +75,8 @@ const REMOTE_STEP_INTEGER_COLUMNS = new Set([ 'permit_expires_at', 'permit_consumed_at', 'executor_quiescent_at', + 'resume_lease_expires_at', + 'settled_at', ]); export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; @@ -130,12 +141,24 @@ export interface RemoteStepInvocation extends RemoteStepAddress { resumeCount?: number; request: unknown; deadlineAt: number; - status: 'pending' | 'dispatched' | 'result-retained' | 'failed'; + status: + | 'pending' + | 'dispatched' + | 'result-retained' + | 'resuming' + | 'settled' + | 'failed'; permitGeneration: number; permitEpoch?: number; permitRevision?: number; permitExpiresAt?: number; permitConsumedAt?: number; + resumeLeaseExpiresAt?: number; + settlementOutcome?: + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof'; } export interface RemoteStepPermit { @@ -191,6 +214,10 @@ type JournalRow = { result_digest: string | null; quiescence_digest: string | null; executor_quiescent_at: number | null; + resume_lease_owner: string | null; + resume_lease_expires_at: number | null; + settled_at: number | null; + settlement_outcome: string | null; }; function record(value: unknown): Record | undefined { @@ -256,7 +283,36 @@ function decode(row: JournalRow): RemoteStepInvocation { (row.resume_count !== -1 && row.resume_count < 1) || !Number.isSafeInteger(row.permit_generation) || row.permit_generation < 0 || - !['pending', 'dispatched', 'result-retained', 'failed'].includes(row.status) + ![ + 'pending', + 'dispatched', + 'result-retained', + 'resuming', + 'settled', + 'failed', + ].includes(row.status) || + (row.settlement_outcome !== null && + ![ + 'applied', + 'terminal-not-success', + 'deadline-expired', + 'advanced-without-proof', + ].includes(row.settlement_outcome)) || + (row.status === 'resuming' && + (!isPathSafeId(row.resume_lease_owner) || + !positiveTime(row.resume_lease_expires_at) || + row.result_json === null)) || + (row.status !== 'resuming' && + (row.resume_lease_owner !== null || + row.resume_lease_expires_at !== null)) || + (row.status === 'settled' && row.settlement_outcome !== 'applied') || + (row.status === 'settled' && + (row.result_json === null || + row.result_digest === null || + !positiveTime(row.executor_quiescent_at))) || + (row.settlement_outcome !== null && + ((row.status !== 'settled' && row.status !== 'failed') || + !positiveTime(row.settled_at))) ) throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); return { @@ -287,6 +343,18 @@ function decode(row: JournalRow): RemoteStepInvocation { ...(row.permit_consumed_at === null ? {} : { permitConsumedAt: row.permit_consumed_at }), + ...(row.resume_lease_expires_at === null + ? {} + : { resumeLeaseExpiresAt: row.resume_lease_expires_at }), + ...(row.settlement_outcome === null + ? {} + : { + settlementOutcome: row.settlement_outcome as + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof', + }), }; } @@ -433,6 +501,10 @@ export class RemoteStepJournal { result_digest TEXT, quiescence_digest TEXT, executor_quiescent_at INTEGER, + resume_lease_owner TEXT, + resume_lease_expires_at INTEGER, + settled_at INTEGER, + settlement_outcome TEXT, UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count) )`) @@ -514,6 +586,22 @@ export class RemoteStepJournal { } if (!uniqueSuspension) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); + const recoveryIndex = indexes.find( + (value) => record(value)?.name === REMOTE_STEP_RECOVERY_INDEX, + ); + const recoveryDefinition = record( + await this.#db + .prepare('SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?') + .bind('index', REMOTE_STEP_RECOVERY_INDEX) + .all(), + )?.results; + const validRecoveryIndex = + record(recoveryIndex)?.unique === 0 && + record(recoveryIndex)?.origin === 'c' && + record(recoveryIndex)?.partial === 1 && + Array.isArray(recoveryDefinition) && + recoveryDefinition.length === 1 && + record(recoveryDefinition[0])?.sql === REMOTE_STEP_RECOVERY_INDEX_SQL; const versions = await this.#db .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) .all(); @@ -523,19 +611,29 @@ export class RemoteStepJournal { if (rows.length === 1) { if ( record(rows[0])?.id !== 1 || - record(rows[0])?.version !== 2 || - !samePrefix(REMOTE_STEP_COLUMNS) + (record(rows[0])?.version !== 3 && record(rows[0])?.version !== 2) || + !samePrefix( + record(rows[0])?.version === 3 + ? REMOTE_STEP_COLUMNS + : REMOTE_STEP_COLUMNS.slice(0, -4), + ) || + (record(rows[0])?.version === 3 && !validRecoveryIndex) ) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - return 'current'; + return record(rows[0])?.version === 3 ? 'current' : 'prior-v2'; } - if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) return 'prior'; + if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -8))) return 'prior-v1'; + if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) return 'unversioned-v2'; if (samePrefix(REMOTE_STEP_COLUMNS)) return 'unversioned-current'; throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); }; const stage = await readSchema(); if (stage === 'current') return; - if (stage === 'prior') { + if ( + stage === 'prior-v1' || + stage === 'prior-v2' || + stage === 'unversioned-v2' + ) { const transactional = this.#db as ExecutionFenceDatabase & { batch?: ( statements: ReturnType[], @@ -548,13 +646,22 @@ export class RemoteStepJournal { ); try { await transactional.batch([ - ...REMOTE_STEP_RESULT_COLUMNS.map((column) => + ...(stage === 'prior-v1' ? REMOTE_STEP_RESULT_COLUMNS : []).map( + (column) => + this.#db.prepare( + `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${column === 'executor_quiescent_at' ? 'INTEGER' : 'TEXT'}`, + ), + ), + ...REMOTE_STEP_SETTLEMENT_COLUMNS.map((column) => this.#db.prepare( - `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${column === 'executor_quiescent_at' ? 'INTEGER' : 'TEXT'}`, + `ALTER TABLE ${REMOTE_STEP_TABLE} ADD COLUMN ${column} ${REMOTE_STEP_INTEGER_COLUMNS.has(column) ? 'INTEGER' : 'TEXT'}`, ), ), + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), this.#db.prepare( - `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, + stage === 'prior-v2' + ? `UPDATE ${REMOTE_STEP_SCHEMA_TABLE} SET version = 3 WHERE id = 1 AND version = 2` + : `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 3)`, ), ]); } catch (error) { @@ -562,12 +669,23 @@ export class RemoteStepJournal { } return; } + const transactional = this.#db as ExecutionFenceDatabase & { + batch?: ( + statements: ReturnType[], + ) => Promise; + }; + if (typeof transactional.batch !== 'function') + throw new RemoteStepRefusal( + 503, + 'remote step schema upgrade is unavailable', + ); try { - await this.#db - .prepare( - `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 2)`, - ) - .run(); + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 3)`, + ), + ]); } catch (error) { if ((await readSchema()) !== 'current') throw error; } @@ -765,6 +883,8 @@ export class RemoteStepJournal { (state.provenance.startIdentity?.owner.kind !== principal.kind || state.provenance.startIdentity.owner.id !== principal.id)) || state.execution.tablePrefix !== address.tablePrefix || + state.execution.workflowId !== address.workflowId || + state.execution.runId !== address.runId || address.resourceId !== target.resourceId ) throw new RemoteStepRefusal(409, 'remote step run identity changed'); @@ -1056,8 +1176,9 @@ export class RemoteStepJournal { const quiescenceDigest = row.quiescence_digest; const executorQuiescentAt = row.executor_quiescent_at; if (resultJson === null) return undefined; + if (row.status === 'settled' || row.status === 'failed') return undefined; if ( - row.status !== 'result-retained' || + !['result-retained', 'resuming'].includes(row.status) || row.permit_consumed_at === null || typeof resultDigest !== 'string' || !SHA256_HEX.test(resultDigest) || @@ -1081,4 +1202,252 @@ export class RemoteStepJournal { executorQuiescentAt, }; } + + async claimResume(invocationId: string, leaseMs: number): Promise { + if ( + !isPathSafeId(invocationId) || + !Number.isSafeInteger(leaseMs) || + leaseMs < 1 || + leaseMs > 60_000 + ) + throw new RemoteStepRefusal(400, 'remote step resume claim is malformed'); + const invocation = await this.get(invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if (invocation.status !== 'result-retained') + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const start = await this.#execution(invocation, invocation.principal); + if (start.execution.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); + if (!(await this.readRetainedResult(invocationId))) + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const now = this.#now(); + const owner = crypto.randomUUID(); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ? + WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? + AND workflow_id = ? AND agent_id = ? AND thread_id = ? + AND resource_id = ? AND run_id = ? AND start_token = ? + AND permit_generation = ? AND permit_epoch IS ? + AND result_digest IS NOT NULL + AND status = 'result-retained' + RETURNING invocation_id`) + .bind( + owner, + now + leaseMs, + invocationId, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + invocation.startToken, + invocation.permitGeneration, + invocation.permitEpoch ?? null, + ) + .all(); + if (!rowResult(result)) + throw new RemoteStepRefusal(409, 'remote step resume is claimed'); + return owner; + } + + async settleAfterResume( + invocationId: string, + leaseOwner: string, + witnessed?: RunSummary, + ): Promise< + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof' + > { + if (!isPathSafeId(invocationId) || !isPathSafeId(leaseOwner)) + throw new RemoteStepRefusal(400, 'remote step resume claim is malformed'); + const invocation = await this.get(invocationId); + if (invocation?.status !== 'resuming') + throw new RemoteStepRefusal(409, 'remote step resume claim changed'); + const start = await this.#execution(invocation, invocation.principal); + if (start.execution.startToken !== invocation.startToken) + throw new RemoteStepRefusal(409, 'remote step run generation changed'); + const current = await this.#runtime.authoritativeStatus( + invocation.workflowId, + invocation.runId, + ); + if (!current || current.runId !== invocation.runId) + throw new RemoteStepRefusal(503, 'remote step outcome is unreadable'); + const same = (summary: RunSummary) => { + if (summary.status !== 'suspended') return false; + const key = invocation.stepPath.join('.'); + const pathPresent = summary.suspended?.some( + (path) => JSON.stringify(path) === JSON.stringify(invocation.stepPath), + ); + const suspended = record(record(summary.suspendPayload)?.[key]); + const tool = record(suspended?.toolCallSuspended); + return ( + pathPresent === true && + summary.suspendedAt?.[key] === invocation.suspendedAt && + summary.resumeCount?.[key] === invocation.resumeCount && + suspended?.type === 'suspension' && + tool?.remoteStep === REMOTE_STEP_MARKER && + JSON.stringify(tool.request) === JSON.stringify(invocation.request) + ); + }; + const sameObservedOutcome = (left: RunSummary, right: RunSummary) => + left.runId === right.runId && + left.status === right.status && + ((left.status === 'success' && + JSON.stringify(left.result) === JSON.stringify(right.result)) || + (left.status === 'suspended' && + JSON.stringify(left.suspended) === JSON.stringify(right.suspended) && + JSON.stringify(left.suspendedAt) === + JSON.stringify(right.suspendedAt) && + JSON.stringify(left.resumeCount) === + JSON.stringify(right.resumeCount) && + JSON.stringify(left.suspendPayload) === + JSON.stringify(right.suspendPayload))); + let outcome: + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof'; + if (same(current)) + outcome = + this.#now() >= invocation.deadlineAt ? 'deadline-expired' : 'retry'; + else if (current.status === 'suspended' || current.status === 'success') + outcome = + witnessed && sameObservedOutcome(witnessed, current) && !same(witnessed) + ? 'applied' + : 'advanced-without-proof'; + else if ( + [ + 'failed', + 'tripwire', + 'canceled', + 'bailed', + 'skipped', + 'cancelled', + 'timed_out', + ].includes(current.status) + ) + outcome = 'terminal-not-success'; + else + throw new RemoteStepRefusal(503, 'remote step outcome is still changing'); + const result = await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET status = ?, resume_lease_owner = NULL, resume_lease_expires_at = NULL, + settled_at = ?, settlement_outcome = ? + WHERE invocation_id = ? AND status = 'resuming' + AND resume_lease_owner = ? AND permit_generation = ? + AND permit_epoch IS ? AND start_token = ? + AND deployment_id = ? AND table_prefix = ? AND workflow_id = ? + AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? + RETURNING invocation_id`) + .bind( + outcome === 'retry' + ? 'result-retained' + : outcome === 'applied' + ? 'settled' + : 'failed', + outcome === 'retry' ? null : this.#now(), + outcome === 'retry' ? null : outcome, + invocationId, + leaseOwner, + invocation.permitGeneration, + invocation.permitEpoch ?? null, + invocation.startToken, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + ) + .all(); + if (!rowResult(result)) + throw new RemoteStepRefusal(409, 'remote step resume claim changed'); + return outcome; + } + + async recoverableResults( + limit: number, + afterInvocationId = '', + ): Promise<{ invocationIds: string[]; nextCursor?: string }> { + if ( + !Number.isSafeInteger(limit) || + limit < 1 || + limit > 100 || + (afterInvocationId !== '' && !isPathSafeId(afterInvocationId)) + ) + throw new RemoteStepRefusal( + 400, + 'remote step recovery limit is malformed', + ); + const now = this.#now(); + const rows = record( + await this.#db + .prepare(`SELECT invocation_id, status, resume_lease_expires_at + FROM ${REMOTE_STEP_TABLE} INDEXED BY ${REMOTE_STEP_RECOVERY_INDEX} + WHERE invocation_id > ? + AND status IN ('result-retained', 'resuming') + ORDER BY invocation_id LIMIT ?`) + .bind(afterInvocationId, limit) + .all(), + )?.results; + if ( + !Array.isArray(rows) || + rows.some( + (row) => + !isPathSafeId(record(row)?.invocation_id) || + !['result-retained', 'resuming'].includes( + record(row)?.status as string, + ) || + (record(row)?.status === 'resuming' && + !positiveTime(record(row)?.resume_lease_expires_at)), + ) + ) + throw new RemoteStepRefusal( + 503, + 'remote step recovery scan is unreadable', + ); + const invocationIds = rows + .filter((row) => { + const value = record(row); + return ( + value?.status === 'result-retained' || + (value?.status === 'resuming' && + positiveTime(value.resume_lease_expires_at) && + value.resume_lease_expires_at <= now) + ); + }) + .map((row) => record(row)?.invocation_id as string); + const last = rows.at(-1); + return { + invocationIds, + ...(rows.length === limit && last + ? { nextCursor: record(last)?.invocation_id as string } + : {}), + }; + } + + async reconcileResult( + invocationId: string, + ): Promise< + | 'retry' + | 'applied' + | 'terminal-not-success' + | 'deadline-expired' + | 'advanced-without-proof' + | 'pending-quiescence' + > { + const invocation = await this.get(invocationId); + if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if (invocation.status === 'resuming') return 'pending-quiescence'; + const owner = await this.claimResume(invocationId, 60_000); + return this.settleAfterResume(invocationId, owner); + } } From ff64a6a09be6dd74a85530f0b76ab26bbf97534e Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Mon, 12 Oct 2026 03:31:28 +0400 Subject: [PATCH 21/73] docs: plan fenced remote resume recovery Keep general Mastra continuation replay blocked; specify a host-owned restricted profile only with durable effect receipts, generation-fenced snapshots, and real cross-isolate proof. The plan records the pinned runtime limitation. --- ...-12-remote-step-durable-resume-recovery.md | 220 ++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 docs/plans/2026-10-12-remote-step-durable-resume-recovery.md diff --git a/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md b/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md new file mode 100644 index 00000000..f01b6090 --- /dev/null +++ b/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md @@ -0,0 +1,220 @@ +# Recover a remote-step resume without repeating an external effect + +> Content type: design plan. Audience: FlowSafe maintainers and deployment hosts. Status: proposed, not implemented. + +This plan records the limit of safe cross-instance resume recovery on pinned `@mastra/core@1.73.0`. Its `run.resume` can execute arbitrary tool closures, nested agents, background work, model calls and direct network effects outside a FlowSafe-owned adapter. General agent continuation is therefore ineligible for automatic replay. A narrow host-owned operation with an explicit, closed effect set may use a downstream idempotent executor; the subsequent Mastra continuation remains single-admission. An uncertain Mastra continuation stays unresolved and blocks drain until independent evidence or a future runtime contract resolves it. + +## Baseline and evidence + +- Branch: `plan/flowsafe-remote-step-broker`; base commit: `4096fbd` (2026-10-12). The settlement checkpoint at that commit leaves expired `resuming` rows unresolved and in drain inventory. Architecture pass 3 cleared the lease-reclaim MUST in `/tmp/asp-flowsafe-settlement-arch-pass3-20261012.md`. Reconfirm the appendix against the checked-out commit before editing. +- Runtime: Node and pnpm versions in the root manifest; Cloudflare Workers, D1 and Durable Objects in production. Use a linked worktree with the shared pnpm store. Keep checks in `/tmp` logs. The full FlowSafe suite passed 6,648/6,648 for the committed settlement content. Automatic cross-isolate replay remains a release gate. +- Cloudflare [Durable Object lifecycle](https://developers.cloudflare.com/durable-objects/concepts/durable-object-lifecycle/) says shutdown can route new requests to a new instance while an old HTTP/RPC request continues if it avoids object storage. A later old-instance storage access fails. Cloudflare [alarms](https://developers.cloudflare.com/durable-objects/api/alarms/) run at least once and have bounded automatic retries. Neither an in-memory lock, a lease timeout, an alarm, nor a Worker deploy proves that an old continuation or its external effect stopped. +- FlowSafe's [README runtime section](../../packages/flowsafe/README.md) and [interrupted-leg design](../do-runner-design.md#interrupted-legs) say a deploy can leave an outgoing leg running and that an aborted external call may already have taken effect. This plan treats that as part of the supported execution model. +- The installed `@mastra/core@1.73.0` source executes `tool.execute` directly in `dist/agent/durable/index.js` and `dist/workflow-event-processor-LHwOlFwA.js`; `dist/agent-D-eTzWxF.js` enters `run.resume` and nested agent paths. Its engine persistence includes `persist`, `replace` and `patch` paths without a remote-step attempt generation. The appendix anchors the exact installed source. The proposed generic effect adapter and runtime generation seam fail this version's capability check. + +## Decision and invariant + +**Recommend a host-owned closed-operation profile.** Its declared operations run outside arbitrary resumed Mastra closures, through a concrete downstream executor. The host enumerates every operation and its inputs before admission. Each effect uses an operation key that stays identical across retries of that logical operation. The downstream system must atomically deduplicate the key with the effect and return the first operation's durable outcome. A preliminary read, best-effort cancellation or FlowSafe-only fence does not establish this property. If the external system supports a fencing generation instead, it must reject stale generations in the same atomic operation as the effect and expose a durable outcome lookup. The profile never authorizes automatic replay of `run.resume` on the pinned core. + +No downstream executor is selected in this repository yet. The profile remains a gated proposal until an owner names one and proves its exact idempotency and receipt API. The existing general agent path stays fail closed. + +A `resumeAttemptId` identifies an execution attempt and helps attribute logs. It must not become the external idempotency key, because retries would use a new attempt ID. The stable effect key binds deployment, run generation, remote invocation and an enumerated host operation. The host persists that operation and its request digest before calling the executor. The executor returns a durable receipt or an explicit unresolved state. It must never mark a timed-out request as not executed because the response was lost. + +One retained result remains bound to one exact suspension. On resume, preserve the stored principal, thread/resource binding, start token, run ID, step path, suspension timestamp/count, mutation epoch, deadline, result digest and current deployment fence. Host operations may be retried through their downstream idempotency contract. Mastra `run.resume` may be admitted once; a newer instance must not invoke it again merely because a lease expired or the snapshot still says suspended. `RunnerRuntime.resume` receives no remote-step attempt generation through `resumeViaRuntime`, and the pinned engine does not fence competing snapshot writes. An unresolved first call remains `resuming`/`recovery-blocked` inventory work. + +An advanced snapshot alone does not prove which attempt or retained result produced it. A crash after the snapshot write loses the in-memory `resumeViaRuntime` witness. The pinned engine has no proven attempt-linked receipt stored atomically with every advancing snapshot write. Without that receipt, classify the advanced state as unresolved rather than `applied`; an operator needs independent evidence. A future runtime may lift this limit only after its entire persistence path atomically binds generation, exact suspension and result digest to the advancing write. + +### Considered options + +| Option | Decision | +| --- | --- | +| Extend the in-instance `withDispatchLock` and reclaim expired leases | Reject. It cannot serialize an outgoing object instance after deploy. | +| Wait a fixed grace period or rely on a Durable Object alarm | Reject. Time and alarm delivery do not prove quiescence or external outcome. | +| Check the authoritative snapshot and retry when it still says suspended | Reject for arbitrary effects. The old continuation can be in flight without a completed snapshot write. | +| Require a universally transactional external operation | Reject as a generic FlowSafe feature. FlowSafe does not control downstream systems. | +| Generic resumed Mastra tool adapter | Reject on pinned 1.73.0. Direct tool closures and nested work bypass it, and snapshot writes lack attempt generation. | +| Host-owned closed operations with stable, atomically enforced effect keys and durable receipts | Recommend for the external operation before single-admission continuation. Hosts lacking a concrete executor retain a visible unresolved obligation. | + +No new dependency is proposed. Reuse the existing journal, D1 transaction and conditional-write patterns, runtime status reader, host authentication, execution fence, and inventory. Any later package addition needs the repository package-first and supply-chain checks. + +## Recovery protocol + +1. Before calling the host-owned executor, the trusted host verifies the retained result and current authoritative run. In one D1 transaction, it records the closed operation list, full run/suspension identity, result digest and stable executor keys. A callback payload cannot select the profile or add an operation. A row whose previous host operation is unresolved can be retried only under the executor's atomic idempotency contract. A row whose Mastra resume was admitted cannot be resumed again automatically. +2. For each host operation, persist its key and request digest before sending. The downstream executor atomically records the effect and first result under that key. A repeated key and identical request returns its durable receipt; a changed request fails closed. After a lost response, query by key or retry with the same key. If the executor cannot establish the effect outcome, mark the row unresolved and block drain. +3. After all host receipts are durable, atomically record a one-time `resume-admitted` decision bound to the exact suspension and retained result. Then call the existing trusted `resumeViaRuntime` path once. A crash between admission and the call is intentionally ambiguous and blocks retry. If the call returns, `settleAfterResume` may use its returned witness and authoritative status to settle; a missing witness, unreadable run or ambiguous advancement remains unresolved. No host receipt proves that an arbitrary Mastra tool executed once. +4. On crash or alarm, scan bounded active work and route inspection to the owning thread host. Reconcile only host operations whose downstream executor enforces stable keys. If Mastra resume was admitted, read the journal and authoritative snapshot for diagnosis but do not call `run.resume` again or infer `applied` from an advanced snapshot. Record independent operator evidence before resolving a missing witness. A future automatic continuation protocol requires a separately reviewed runtime version that fences every competing snapshot write and persists attempt-linked provenance atomically with advancement; the pinned version fails this gate. +5. Keep host receipts and uncertain resume records in drain inventory across deploy, rollback, deadline, retention cleanup and migration. An automatic purge cannot erase the evidence that prevents replay. An operator resolution records the independent external evidence and must not invent `applied` from elapsed time. + +The protocol needs one host-only recovery method and a versioned journal migration. Its endpoint or method must use the same internal authentication and stored-principal checks as the current remote-step host route. It must not be a public result route. Versioned schema migration preserves old rows; a legacy `resuming` row without durable attempt/effect evidence is ambiguous and remains blocked. Refuse unknown schema versions. During mixed-version rollout, old code must not claim or settle new-version attempts; stage code and migration so the old version fails closed or keep remote-step dispatch disabled until cutover. Prove that with an actual old/new Worker-version exercise, not an inferred migration rule. + +### Before and target contracts + +The current `RemoteStepJournal.claimResume` writes an expiring lease for the initial result-retained claim, then the host calls the runtime: + +```typescript +const leaseOwner = await reader.claimResume(invocationId, 60_000); +const resumed = await withExecution(execution, () => + durable.resumeViaRuntime({ + runId: ref.runId, + requestedBy: stored.principal.id, +``` + +The current `RemoteStepJournal.reconcileResult` can settle a retained result but returns `pending-quiescence` for `resuming`: + +```typescript +if (invocation.status === 'resuming') return 'pending-quiescence'; +const owner = await this.claimResume(invocationId, 60_000); +return this.settleAfterResume(invocationId, owner); +``` + +The target is an **internal contract sketch**, not a claim that these methods exist. It shows where the host-owned operation can safely retry and where Mastra continuation cannot: + +```typescript +const operation = await reader.prepareHostOperation({ + invocationId, + trustedClosedProfileId, +}); +const receipt = await executor.executeIdempotent(operation); +await reader.recordHostReceipt(operation.id, receipt); +await reader.admitSingleResume(invocationId, operation.id); +const resumed = await withExecution(execution, () => + durable.resumeViaRuntime(boundResumeRequest), +); +await reader.settleAfterResume(invocationId, operation.id, resumed); +``` + +The exact method names and types require implementation design. `admitSingleResume` must persist before `resumeViaRuntime`; any error or crash after admission leaves an unresolved record. The closed executor is host-owned, not a wrapper around arbitrary `tool.execute`. Do not weaken existing approval resume or ordinary agent run APIs to fit this sketch. + +The pinned runtime lacks the generation and provenance path required for automatic `run.resume` replay. `FencedWorkflowsStorageD1.persistWorkflowSnapshot` alone cannot fence all engine writes. A future protocol must inspect `RunnerRuntime.resume` and every installed engine persistence path, including persist, replace and patch, then prove atomic generation refusal and provenance. This plan does not implement that future protocol. + +## Implementation checkpoints + +1. **Capability result and host contract:** The pinned `@mastra/core@1.73.0` audit failed the generic adapter and generation proof. Do not re-run the same proof expecting an arbitrary Mastra continuation to become safe. Identify a concrete host-owned closed operation and executor, list every effect it can perform, and prove atomic idempotency and receipt lookup for that executor. If no concrete operation meets the contract, leave the existing fail-closed result path in place and defer this profile. +2. **Journal and host operation state:** Add a versioned migration, stable operation keys/receipts, one-time `resume-admitted` marker, active index/inventory, and host-only inspection method. Keep old rows intact and ambiguous rows blocked. Authenticate the method; re-read principal, binding, start token, epoch, deadline and authoritative suspension. Keep the current host lock for same-instance ordering, but do not use it as cross-instance evidence. +3. **Closed executor and recovery duty:** Add the host-owned executor only after its downstream idempotency contract is proven. Wire bounded scan and alarm/maintenance wake-up to inspect and reconcile host operations. The duty never re-invokes Mastra after `resume-admitted`. Keep alarm scheduling compatible with the host's existing alarm owner, because a Durable Object has one alarm slot. +4. **Cutover and proof:** Exercise old/new versions, crash and lost acknowledgement on real D1/workerd storage. Keep feature disabled for customer dispatch until the proof matrix below passes. Merge to `dev` through reviewed checkpoints and publish through the Anchorage release process only after the release gates pass. + +## Verification and release gates + +Run each checkpoint's narrowest changed-file checks once, in `/tmp` logs with the content ID and final `EXIT_STATUS`. For the shared journal schema/protocol checkpoint, the quality gate's shared-contract exception selects the owning FlowSafe package suite. Candidate commands, adjusted to the changed-file list at execution time: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run \ + src/do-runner/remote-step-journal.test.ts \ + src/agent-host/thread-host.test.ts \ + src/agent-host/thread-topology.test.ts +pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts \ + packages/flowsafe/src/agent-host/thread-host.ts +pnpm --filter @proofoftech/flowsafe typecheck:test +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe test +``` + +The gate owner redirects each command's complete output to a distinct `/tmp` log, appends its exit status, and gives reviewers the paths. After all checkpoints, run the repository's full suite once, including direct scenarios where the changed contract reaches them. Do not substitute mocked D1 or a second object in one process for the real fresh-isolate proof. + +| Proof | Required observation | +| --- | --- | +| Two resumes in one object instance | One attempt begins; the other sees the durable state, with no duplicate effect. | +| Deploy while old instance is inside a host-owned external operation beyond the 60s lease | New instance never treats lease expiry as quiescence. A repeated host operation uses the stable key and returns one durable executor outcome. | +| Crash before host effect, after effect before receipt, and after receipt before Mastra admission | The executor receipt resolves the host operation or the row stays blocked; no ambiguous effect is repeated under a new key. | +| Crash after `resume-admitted` before call, during call, or after advancing snapshot before journal settlement | No automatic second `run.resume` occurs. A missing returned witness stays unresolved even if the snapshot advanced; operator evidence is required. | +| Identical and conflicting retries | Same operation key plus same request yields the original receipt; changed request fails closed. | +| Old Mastra attempt remains in flight after deploy | No newer Mastra attempt starts; journal and drain inventory preserve the uncertainty. | +| Changed principal, resource mapping, start token, suspension, epoch, deadline, or fence | Recovery refuses before any effect. | +| Legacy and unknown schema rows | Known rows survive migration; ambiguous `resuming` rows stay visible and blocked; unknown versions refuse. | +| Alarm duplicate, exhausted retries, process restart, and no traffic | Maintenance remains scheduled and scans bounded active work without relying on a single alarm delivery. | +| Migration drain and retention | Active/blocked rows remain inventory work; cleanup cannot erase unresolved effect evidence. | +| Closed-profile bypass | An operation outside the enumerated host contract is rejected; arbitrary `tool.execute` remains ineligible for automatic replay. | + +Every implementation checkpoint is T2 for concurrency, authentication, migration and data loss: independent clean-code, QA, architecture and security lanes. This plan received one design-level review when added; that review does not run or simulate its commands. No MUST may remain when a checkpoint closes. A live release requires the concrete executor's idempotency proof and a real old/new Cloudflare rollout exercise. Without those proofs, retain the fail-closed state. Even with them, advertise recovery only for the host-owned operation, not automatic Mastra continuation. + +## Reuse map and excluded work + +- `packages/flowsafe/src/do-runner/remote-step-journal.ts`: canonical suspension identity, retained result digest, D1 migration, conditional settlement and bounded recovery scan +- `packages/flowsafe/src/agent-host/thread-host.ts`: trusted internal route, stored-principal authorization, bound runtime resume and same-instance dispatch lock +- `packages/flowsafe/src/agent-host/thread-topology.ts`: owning thread routing and trusted host context +- `packages/flowsafe/src/do-runner/execution-fence.ts`: deployment epoch and drain boundary +- `packages/flowsafe/src/do-runner/inventory.ts`: active obligation visibility +- `packages/flowsafe/src/agent-runner/durable-agent-runner.ts`: pinned runtime continuation and authoritative status path +- `packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.resume`: runtime re-entry that currently lacks a remote-step attempt generation +- `packages/flowsafe/src/do-runner/fenced-workflows-d1.ts`, `FencedWorkflowsStorageD1.persistWorkflowSnapshot`: partial storage seam; its existence does not authorize replay + +The plan does not change the public result-intake API, approval decision semantics, arbitrary agent tools, customer signing, Fleet Control ownership, payment policy, or external executor implementation. Those are separate owners or cannot meet this replay contract without their own proof. It also does not turn an unknown effect outcome into a terminal success or failure; doing so would hide an unresolved obligation. + +## Appendix: current-state anchors + +The FlowSafe excerpts come from `4096fbd`; the installed Mastra excerpts come from the locked `@mastra/core@1.73.0` package. Reconfirm symbols and semantics, not line numbers. + +- `packages/flowsafe/src/agent-host/thread-host.ts`, `createThreadAgentHost` dispatch and remote-step route: + +```typescript +const withDispatchLock = createFifoLock(); +``` + +The route later calls `withDispatchLock`; the lock lives in one host object instance. The remote-step route currently refuses an active `resuming` row: + +```typescript +if (invocation.status === 'resuming') + throw new AgentHostRequestError( + 503, + 'remote step resume awaits executor quiescence', + ); +``` + +- `packages/flowsafe/src/do-runner/remote-step-journal.ts`, `claimResume` and `settleAfterResume`: + +```typescript +SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ? +``` + +```typescript +AND resume_lease_owner = ? AND permit_generation = ? +AND permit_epoch IS ? AND start_token = ? +``` + +These predicates protect the journal row but do not prove the external effect or the outgoing runtime leg stopped. + +- `packages/flowsafe/src/do-runner/remote-step-journal.ts`, `recoverableResults` and `reconcileResult`: the current scan selects retained/resuming work through the v3 partial index, while `reconcileResult` returns `pending-quiescence` for an active `resuming` row. The journal migration must preserve those records and add durable attempt evidence before any automatic reclaim. + +- `packages/flowsafe/src/agent-runner/durable-agent-runner.ts`, `FlowsafeDurableAgent.resumeViaRuntime`, and `packages/flowsafe/src/do-runner/runtime.ts`, `RunnerRuntime.resume`: the current call passes no attempt token: + +```typescript +const summary = await this.#runtime.resume( + this.getWorkflow().id, + options.runId, + resumeOptions, +); +``` + +`RunnerRuntime.resume` then enters the pinned engine with `run.resume({ step: options.step, resumeData: options.resumeData, requestContext })`. It has no remote-step attempt generation. Concurrent replay is excluded from this plan. + +- `packages/flowsafe/src/do-runner/fenced-workflows-d1.ts`, `FencedWorkflowsStorageD1.persistWorkflowSnapshot`: the current storage seam accepts `PersistInput` without a remote-step attempt generation: + +```typescript +override async persistWorkflowSnapshot(args: PersistInput): Promise { + const scope = this.#scopes.getStore(); +``` + +The installed `node_modules/.pnpm/@mastra+core@1.73.0_express@5.2.1_zod@4.4.3/node_modules/@mastra/core/dist/agent/durable/index.js`, `executeDurableToolCalls`, calls the supplied tool closure: + +```javascript +const result = await tool.execute(toolCall.args, { +``` + +The installed `dist/workflow-event-processor-LHwOlFwA.js` also invokes a tool closure from its workflow tool execution path: + +```javascript +return tool.execute(inputData, toolContext); +``` + +The installed `dist/agent-D-eTzWxF.js` enters a resumed agent run and can resume nested work: + +```javascript +const executionPromise = resumeContext ? run.resume({ +``` + +```javascript +else if (resumeNested) res = await run.resume({ +``` + +The pinned engine's workflow writes include persist, replace and patch paths without a remote-step generation. The closure calls above are enough to rule out an effect adapter that wraps only the FlowSafe host route. A future automatic-continuation proposal must inspect all engine write paths and bind its advancing write to durable resume provenance; a separate post-write journal update cannot serve as the receipt. From d8c54b68b3647c4da1235c54b8cfc8edbaed9738 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Mon, 12 Oct 2026 05:33:44 +0400 Subject: [PATCH 22/73] fix(flowsafe): bind remote resume to topology resource Refuse a retained result when the current trusted resource mapping differs from the journaled start resource. Exercise nonidentity mapping and changed-binding refusal across host rehydration. --- .../src/agent-host/thread-host.test.ts | 42 +++++++++++++++---- .../src/agent-host/thread-topology.test.ts | 10 ++++- .../src/agent-host/thread-topology.ts | 1 + .../src/do-runner/remote-step-journal.test.ts | 2 +- 4 files changed, 44 insertions(+), 11 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index d8ee2518..49f825e3 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -7505,6 +7505,7 @@ async function hostAgentLifecycleFixture( lifecycle?: boolean; snapshotTarget?: { agentId: string; threadId: string }; resourceIdFromKey?: (threadId: string) => string; + startResourceId?: string; } = {}, ) { const core = await vi.importActual( @@ -7534,6 +7535,7 @@ async function hostAgentLifecycleFixture( const threaded = input.threaded ?? true; const status = input.status ?? 'suspended'; const version = input.provenance ?? 'v1'; + const startResourceId = input.startResourceId ?? RESOURCE_ID; const sql = openSqlite() as ReturnType & { close(): void }; const binding = sqliteUnitDatabase(sql) as ExecutionFenceDatabase & ResourceOwnershipDatabase; @@ -7603,7 +7605,7 @@ async function hostAgentLifecycleFixture( kind: 'agent', id: 'writer', threadId: 'acme_thread', - resourceId: RESOURCE_ID, + resourceId: startResourceId, }, }); } @@ -7612,15 +7614,22 @@ async function hostAgentLifecycleFixture( token: 'host-lifecycle-attempt', execution, threaded, + resourceId: startResourceId, ...(claim ? { startReservation: claim } : {}), }); seedRecoveryState(fixture.state, 'acme_run', recovery, threaded); + if (threaded && startResourceId !== RESOURCE_ID) + fixture.state.set(THREAD_BINDING_KEY, { + version: 1, + agentId: 'writer', + resourceId: startResourceId, + }); if (!input.journal) fixture.state.delete(TEST_OWNER_RECOVERY_KEY); await resources.reserveAll( [ { kind: 'run', resourceId: 'acme_run' }, { kind: 'thread', resourceId: 'acme_thread' }, - { kind: 'resource', resourceId: RESOURCE_ID }, + { kind: 'resource', resourceId: startResourceId }, ], HUMAN_OWNER, recovery.token as string, @@ -7631,7 +7640,10 @@ async function hostAgentLifecycleFixture( if (!workflows) throw new Error('missing host workflow domain'); const snapshotAgentId = input.snapshotTarget?.agentId ?? 'writer'; const snapshotThreadId = input.snapshotTarget?.threadId ?? 'acme_thread'; - const snapshotResourceId = resourceIdFromKey(snapshotThreadId); + const snapshotResourceId = + snapshotThreadId === 'acme_thread' + ? startResourceId + : resourceIdFromKey(snapshotThreadId); const snapshot: import('@mastra/core/workflows').WorkflowRunState = { runId: 'acme_run', status: @@ -7754,17 +7766,25 @@ async function hostAgentLifecycleFixture( }; } -function hostAgentLifecycleRequest(suffix = '', query = '') { +function hostAgentLifecycleRequest( + suffix = '', + query = '', + resourceId = RESOURCE_ID, +) { return new Request( - `https://thread/_flowsafe/agent-host/runs/writer/acme_run${suffix}?resourceId=${RESOURCE_ID}${query}`, + `https://thread/_flowsafe/agent-host/runs/writer/acme_run${suffix}?resourceId=${resourceId}${query}`, suffix ? { method: 'POST' } : undefined, ); } -it('reads a retained remote result from D1 after the agent host is recreated', async () => { +it('resumes a retained mapped-resource result from D1 after the agent host is recreated', async () => { + const mappedResourceId = 'account-7'; + const mappedResource = () => mappedResourceId; const fixture = await hostAgentLifecycleFixture({ mode: 'fenced', provenance: 'modern', + resourceIdFromKey: mappedResource, + startResourceId: mappedResourceId, }); onTestFinished(() => { vi.restoreAllMocks(); @@ -7789,7 +7809,10 @@ it('reads a retained remote result from D1 after the agent host is recreated', a }, }); await fixture.persist(); - await fixture.host.route(hostAgentLifecycleRequest(), fixture.scope); + await fixture.host.route( + hostAgentLifecycleRequest('', '', mappedResourceId), + fixture.scope, + ); const journal = new RemoteStepJournal({ db: fixture.binding, fence: fixture.fence, @@ -7807,7 +7830,7 @@ it('reads a retained remote result from D1 after the agent host is recreated', a workflowId: fixture.execution.workflowId, agentId: 'writer', threadId: 'acme_thread', - resourceId: RESOURCE_ID, + resourceId: mappedResourceId, runId: 'acme_run', }, Date.now() + 60_000, @@ -7837,6 +7860,7 @@ it('reads a retained remote result from D1 after the agent host is recreated', a verifyCompletion: async () => undefined, }); const fresh = harness(['writer'], { + resourceIdFromKey: mappedResource, init: freshApp, storage: fixture.storage, stateStorage: fixture.stateStorage, @@ -7872,7 +7896,7 @@ it('reads a retained remote result from D1 after the agent host is recreated', a await fixture.stateStorage.put(THREAD_BINDING_KEY, { version: 1, agentId: 'writer', - resourceId: RESOURCE_ID, + resourceId: mappedResourceId, }); const response = await fresh.host.route( new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index f76e3300..00a65481 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -99,7 +99,7 @@ function harness(remoteSteps?: RemoteStepContinuationReader): { }; } -it('routes a retained result under the journaled principal and sends only its id', async () => { +it('routes a retained result under the journaled principal and refuses a changed resource mapping', async () => { const principal = { kind: 'human' as const, id: 'original', @@ -148,6 +148,14 @@ it('routes a retained result under the journaled principal and sends only its id ); const stored = await reader.get('invocation_1'); if (!stored) throw new Error('invocation was not found'); + vi.mocked(reader.get).mockResolvedValue({ + ...stored, + resourceId: 'displaced-resource', + }); + await expect( + topology.resumeRemoteStep(context().value, 'invocation_1'), + ).rejects.toMatchObject({ status: 404 }); + expect(hits).toHaveLength(1); vi.mocked(reader.get).mockResolvedValue({ ...stored, status: 'resuming', diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 44fc33fd..6ac23e2f 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -788,6 +788,7 @@ export function createAgentThreadTopology( !(await reader.readRetainedResult(invocationId)) ) throw new RunRouteError(404, 'invocation not found'); + expectedResource(context, invocation.threadId, invocation.resourceId); return envelope( await threads.send( { diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 05d7ba9a..a3932d32 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -207,7 +207,7 @@ describe('remote step journal dispatch admission', () => { ]); }); - it('accepts the resource recorded at start when the host maps a thread elsewhere', async () => { + it('registers the resource recorded in start provenance independently of the thread id', async () => { const h = await fixture(); const original = h.runtime.authoritativeStartState.bind(h.runtime); h.runtime.authoritativeStartState = (async ( From 61b157b7b3d06065bfbfd0d61bd8586255a5a863 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Mon, 12 Oct 2026 14:25:27 +0400 Subject: [PATCH 23/73] test(flowsafe): exercise remote step journal across workerd restarts Use the existing local spike to prove permit/result retention, resource refusal and operator drain inventory after a fresh process. The fixture uses a synthetic runtime and does not claim Mastra continuation. --- packages/flowsafe/scripts/spike-verify.mjs | 174 +++++++++++++++++++++ packages/flowsafe/spike/worker.ts | 161 +++++++++++++++++++ 2 files changed, 335 insertions(+) diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index 47c73c45..b0ae5058 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -3385,6 +3385,176 @@ async function main() { }, ); + await step( + 'RS1 remote-step journal: real D1 permit winner and retained result survive process death', + async () => { + await killServer(currentServer); + await executeLocalD1( + stateDir, + 'CREATE TABLE spike_remote_step_fixture (id INTEGER PRIMARY KEY, resource_id TEXT NOT NULL, status TEXT NOT NULL); ' + + "INSERT INTO spike_remote_step_fixture VALUES (1, 'account-7', 'suspended'); " + + 'CREATE TABLE spike_remote_mastra_workflow_snapshot (workflow_name TEXT NOT NULL, run_id TEXT NOT NULL, snapshot TEXT NOT NULL, PRIMARY KEY (workflow_name, run_id)); ' + + "INSERT INTO spike_remote_mastra_workflow_snapshot VALUES ('durable-agentic-loop', 'spike-remote-run', '{\"status\":\"suspended\"}');", + ); + await launchServer( + 'gen-remote-1', + stateDir, + join(tmpDir, 'gen-remote-1.log'), + ); + const wrong = await http('POST', '/spike/remote-step/register', { + body: { resourceId: 'spike-remote-thread' }, + }); + assert( + wrong.status === 409, + 'the thread id cannot stand in for the mapped resource', + wrong, + ); + const registered = await http('POST', '/spike/remote-step/register', { + body: { resourceId: 'account-7' }, + }); + assert( + registered.status === 200 && + typeof registered.body.invocationId === 'string', + 'the mapped-resource invocation was registered', + registered, + ); + const invocationId = registered.body.invocationId; + const issued = await http('POST', '/spike/remote-step/issue', { + body: { invocationId }, + }); + assert( + issued.status === 200 && issued.body.generation === 1, + 'the dispatch permit was issued', + issued, + ); + const attempts = await Promise.all( + [0, 1].map(() => + http('POST', '/spike/remote-step/consume', { + body: { permit: issued.body }, + }), + ), + ); + assert( + attempts.filter((attempt) => attempt.body.kind === 'accepted') + .length === 1 && + attempts.filter((attempt) => attempt.body.kind === 'already-consumed') + .length === 1, + 'concurrent permit consumption has one winner', + attempts, + ); + const rawPermit = attempts.find( + (attempt) => attempt.body.kind === 'accepted', + ).body.rawPermit; + const recorded = await http('POST', '/spike/remote-step/record', { + body: { invocationId, rawPermit }, + }); + assert( + recorded.status === 200 && recorded.body.kind === 'recorded', + 'the result was retained', + recorded, + ); + await killServer(currentServer); + await executeLocalD1( + stateDir, + "UPDATE spike_remote_step_fixture SET resource_id = 'account-8' WHERE id = 1;", + ); + await launchServer( + 'gen-remote-2', + stateDir, + join(tmpDir, 'gen-remote-2.log'), + ); + const retained = await http( + 'GET', + `/spike/remote-step/read?invocationId=${invocationId}`, + ); + assert( + retained.status === 200 && + retained.body.invocation.status === 'result-retained' && + retained.body.result.resultJson === '{"answer":1}', + 'fresh workerd process reads the persisted result', + retained, + ); + const displaced = await http('POST', '/spike/remote-step/claim', { + body: { invocationId }, + }); + assert( + displaced.status === 409, + 'changed mapped-resource provenance refuses a resume claim', + displaced, + ); + await killServer(currentServer); + await executeLocalD1( + stateDir, + "UPDATE spike_remote_step_fixture SET resource_id = 'account-7' WHERE id = 1;", + ); + await launchServer( + 'gen-remote-3', + stateDir, + join(tmpDir, 'gen-remote-3.log'), + ); + const scan = await http('GET', '/spike/remote-step/scan'); + assert( + scan.status === 200 && scan.body.invocationIds.includes(invocationId), + 'fresh process recovery scan finds the retained result', + scan, + ); + const claimed = await http('POST', '/spike/remote-step/claim', { + body: { invocationId }, + }); + assert( + claimed.status === 200 && typeof claimed.body.owner === 'string', + 'one resume claim was persisted', + claimed, + ); + await killServer(currentServer); + await executeLocalD1( + stateDir, + "UPDATE spike_remote_step_fixture SET status = 'success' WHERE id = 1; " + + 'UPDATE spike_remote_mastra_workflow_snapshot SET snapshot = \'{"status":"success"}\' WHERE run_id = \'spike-remote-run\';', + ); + await launchServer( + 'gen-remote-4', + stateDir, + join(tmpDir, 'gen-remote-4.log'), + ); + const ambiguous = await http('POST', '/spike/remote-step/reconcile', { + body: { invocationId }, + }); + assert( + ambiguous.status === 200 && + ambiguous.body.outcome === 'pending-quiescence', + 'fresh process does not replay an ambiguous resume', + ambiguous, + ); + const settled = await http('POST', '/spike/remote-step/settle', { + body: { invocationId, owner: claimed.body.owner }, + }); + assert( + settled.status === 200 && + settled.body.outcome === 'advanced-without-proof', + 'advanced snapshot without a returned witness remains unresolved', + settled, + ); + const final = await http( + 'GET', + `/spike/remote-step/read?invocationId=${invocationId}`, + ); + assert( + final.status === 200 && + final.body.invocation.status === 'failed' && + final.body.invocation.settlementOutcome === 'advanced-without-proof', + 'the unresolved outcome remains durable', + final, + ); + const drain = await sweepInventoryCategory('remote-steps'); + assert( + drain.class === 'work' && drain.keys.includes(invocationId), + 'the unresolved result remains in operator drain inventory', + drain, + ); + }, + ); + await step( 'S deployment sentinel mismatch: a freshly started Worker refuses a D1 ' + 'provisioned for another deployment', @@ -3485,6 +3655,10 @@ try { 'answered the same inventory, and refused to mint, while its schedules ' + 'and provider subscriptions kept being reported as standing ' + 'configuration a migration carries rather than drains. ' + + 'Remote-step journal (RS1): a single permit winner retained a result ' + + 'across process death, changed mapped-resource provenance refused a ' + + 'resume claim, and an advanced snapshot without a returned witness ' + + 'remained in operator drain inventory after another restart. ' + 'Finally, a fresh Worker refused a D1 sentinel stamped for ' + 'another deployment with 503 before authentication or routing.', ); diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index 64bfed83..c5a905d3 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -139,6 +139,7 @@ import { isSuspensionTimeoutResumeData, mintThreadId, type RunnerRuntime, + type RunSummary, resourceIdFromKey, type StartIdempotencyStore, SUSPENSION_DEADLINE_PAYLOAD_KEY, @@ -150,6 +151,11 @@ import { verifyDurableObjectDeploymentIdentity, verifyDurableObjectDeploymentRequest, } from '../src/do-runner/index.js'; +import { + REMOTE_STEP_MARKER, + RemoteStepJournal, + type RemoteStepPermit, +} from '../src/do-runner/remote-step-journal.js'; import { createObjectiveRouter, type ObjectiveAuditEvent, @@ -3073,6 +3079,158 @@ async function handleInventoryProbe( } } +// Local workerd fixture: the journal reads a persisted suspension and start +// identity without entering Mastra. It proves D1 recovery, not agent replay. +async function handleRemoteStepProbe( + request: Request, + env: Env, +): Promise { + const path = new URL(request.url).pathname; + if (!path.startsWith('/spike/remote-step/')) return null; + const operation = path.slice('/spike/remote-step/'.length); + if ( + request.method !== + (operation === 'read' || operation === 'scan' ? 'GET' : 'POST') + ) + return json({ error: 'method not allowed' }, 405); + try { + const fixture = await env.DB.prepare( + 'SELECT resource_id, status FROM spike_remote_step_fixture WHERE id = 1', + ).first<{ resource_id: string; status: string }>(); + const snapshot = await env.DB.prepare( + 'SELECT snapshot FROM spike_remote_mastra_workflow_snapshot WHERE workflow_name = ? AND run_id = ?', + ) + .bind('durable-agentic-loop', 'spike-remote-run') + .first<{ snapshot: string }>(); + if (!fixture || !snapshot) + throw new Error('remote-step fixture is unavailable'); + const summary: RunSummary = + fixture.status === 'success' + ? { + runId: 'spike-remote-run', + status: 'success', + result: { answer: 1 }, + } + : { + runId: 'spike-remote-run', + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 1000 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }; + const runtime = { + authoritativeStatus: async () => summary, + authoritativeStartState: async () => ({ + storage: 'd1', + snapshot: JSON.parse(snapshot.snapshot), + execution: { + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + runId: 'spike-remote-run', + startToken: 'spike-start-token', + }, + provenance: { + startIdentity: { + owner: { kind: 'human', id: 'operator-1' }, + target: { + kind: 'agent', + id: 'writer', + threadId: 'spike-remote-thread', + resourceId: fixture.resource_id, + }, + }, + }, + }), + } as unknown as Pick< + RunnerRuntime, + 'authoritativeStatus' | 'authoritativeStartState' + >; + const journal = new RemoteStepJournal({ + db: env.DB as unknown as never, + fence: executionFenceForEnv(env), + runtime, + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const body = + request.method === 'POST' + ? ((await request.json()) as Record) + : {}; + const invocationId = + request.method === 'GET' + ? (new URL(request.url).searchParams.get('invocationId') ?? '') + : String(body.invocationId ?? ''); + switch (operation) { + case 'register': + return json( + await journal.register( + { + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'spike-remote-thread', + resourceId: String(body.resourceId ?? ''), + runId: 'spike-remote-run', + }, + Date.now() + 120_000, + { kind: 'human', id: 'operator-1', role: 'operator' }, + ), + ); + case 'issue': + return json(await journal.issuePermit(invocationId, 60_000)); + case 'consume': + return json( + await journal.consumePermit(body.permit as RemoteStepPermit), + ); + case 'record': + return json( + await journal.recordResult( + invocationId, + String(body.rawPermit), + '{"answer":1}', + '{"receipt":"complete"}', + ), + ); + case 'read': + return json({ + invocation: await journal.get(invocationId), + result: await journal.readRetainedResult(invocationId), + }); + case 'scan': + return json(await journal.recoverableResults(10)); + case 'claim': + return json({ owner: await journal.claimResume(invocationId, 60_000) }); + case 'reconcile': + return json({ outcome: await journal.reconcileResult(invocationId) }); + case 'settle': + return json({ + outcome: await journal.settleAfterResume( + invocationId, + String(body.owner), + ), + }); + default: + return json({ error: 'not found' }, 404); + } + } catch (error) { + return doErrorResponse(error); + } +} + const handler: ExportedHandler = { async fetch( request: CfRequest, @@ -3125,6 +3283,9 @@ const handler: ExportedHandler = { const inventoryProbe = await handleInventoryProbe(routed, env); if (inventoryProbe) return inventoryProbe; + const remoteStepProbe = await handleRemoteStepProbe(routed, env); + if (remoteStepProbe) return remoteStepProbe; + const agentProbeResponse = await handleLiveAgentRoute(routed, env, resolve); if (agentProbeResponse) return agentProbeResponse; From 839f46237cdc38035d84f1f2e71df9b8f3fba79d Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Mon, 12 Oct 2026 14:35:35 +0400 Subject: [PATCH 24/73] test(flowsafe): derive spike status from persisted snapshot The restart proof now fails when the workflow snapshot does not advance, even if the separate fixture status says success. --- packages/flowsafe/scripts/spike-verify.mjs | 7 ++++--- packages/flowsafe/spike/worker.ts | 11 +++++++---- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index b0ae5058..520f81f3 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -3389,10 +3389,12 @@ async function main() { 'RS1 remote-step journal: real D1 permit winner and retained result survive process death', async () => { await killServer(currentServer); + // The fixture status is stale so a summary detached from the persisted + // snapshot cannot register the still-suspended run. await executeLocalD1( stateDir, 'CREATE TABLE spike_remote_step_fixture (id INTEGER PRIMARY KEY, resource_id TEXT NOT NULL, status TEXT NOT NULL); ' + - "INSERT INTO spike_remote_step_fixture VALUES (1, 'account-7', 'suspended'); " + + "INSERT INTO spike_remote_step_fixture VALUES (1, 'account-7', 'success'); " + 'CREATE TABLE spike_remote_mastra_workflow_snapshot (workflow_name TEXT NOT NULL, run_id TEXT NOT NULL, snapshot TEXT NOT NULL, PRIMARY KEY (workflow_name, run_id)); ' + "INSERT INTO spike_remote_mastra_workflow_snapshot VALUES ('durable-agentic-loop', 'spike-remote-run', '{\"status\":\"suspended\"}');", ); @@ -3509,8 +3511,7 @@ async function main() { await killServer(currentServer); await executeLocalD1( stateDir, - "UPDATE spike_remote_step_fixture SET status = 'success' WHERE id = 1; " + - 'UPDATE spike_remote_mastra_workflow_snapshot SET snapshot = \'{"status":"success"}\' WHERE run_id = \'spike-remote-run\';', + 'UPDATE spike_remote_mastra_workflow_snapshot SET snapshot = \'{"status":"success"}\' WHERE run_id = \'spike-remote-run\';', ); await launchServer( 'gen-remote-4', diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index c5a905d3..a72c4dab 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -3095,8 +3095,8 @@ async function handleRemoteStepProbe( return json({ error: 'method not allowed' }, 405); try { const fixture = await env.DB.prepare( - 'SELECT resource_id, status FROM spike_remote_step_fixture WHERE id = 1', - ).first<{ resource_id: string; status: string }>(); + 'SELECT resource_id FROM spike_remote_step_fixture WHERE id = 1', + ).first<{ resource_id: string }>(); const snapshot = await env.DB.prepare( 'SELECT snapshot FROM spike_remote_mastra_workflow_snapshot WHERE workflow_name = ? AND run_id = ?', ) @@ -3104,8 +3104,11 @@ async function handleRemoteStepProbe( .first<{ snapshot: string }>(); if (!fixture || !snapshot) throw new Error('remote-step fixture is unavailable'); + const persisted = JSON.parse(snapshot.snapshot) as { status?: unknown }; + if (persisted.status !== 'suspended' && persisted.status !== 'success') + throw new Error('remote-step snapshot status is unavailable'); const summary: RunSummary = - fixture.status === 'success' + persisted.status === 'success' ? { runId: 'spike-remote-run', status: 'success', @@ -3132,7 +3135,7 @@ async function handleRemoteStepProbe( authoritativeStatus: async () => summary, authoritativeStartState: async () => ({ storage: 'd1', - snapshot: JSON.parse(snapshot.snapshot), + snapshot: persisted, execution: { tablePrefix: 'spike_remote_', workflowId: 'durable-agentic-loop', From 66e239ec3b471fafe0c048f3e0f8e10f8b16a3d5 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:37:59 +0400 Subject: [PATCH 25/73] feat(flowsafe): record atomic notification effects Bind host-owned notification enqueues to an immutable request digest and a D1 transaction receipt. Replays return the first effect without overwriting the inbox. --- packages/flowsafe/src/do-runner/inventory.ts | 8 + .../src/signals/notifications-d1.test.ts | 144 +++++++++++++++++ .../flowsafe/src/signals/notifications-d1.ts | 150 ++++++++++++++++++ 3 files changed, 302 insertions(+) diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index 95185917..21283641 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -337,6 +337,14 @@ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ 'a single counter row, not a unit of work. The notifications it orders are inventoried under pending-notifications; the counter itself has nothing to finish and is re-derivable from the inbox.', }, }, + { + table: 'flowsafe_notification_operation_receipts', + purpose: 'immutable receipts for host-owned inbox notification effects', + accounting: { + excluded: + 'a receipt records an effect already committed with its inbox row; replay must retain it after the inbox row is settled or removed, so it is historical evidence rather than active work', + }, + }, { table: RESOURCE_OWNER_TABLE, purpose: diff --git a/packages/flowsafe/src/signals/notifications-d1.test.ts b/packages/flowsafe/src/signals/notifications-d1.test.ts index 7404f4ec..461acb9e 100644 --- a/packages/flowsafe/src/signals/notifications-d1.test.ts +++ b/packages/flowsafe/src/signals/notifications-d1.test.ts @@ -38,6 +38,150 @@ function sharedStores(): [D1NotificationsStorage, D1NotificationsStorage] { ]; } +describe('host-owned notification operation receipt', () => { + it('returns the first durable receipt after a lost acknowledgement without overwriting the inbox', async () => { + const db = database(); + const atomicBatch = db.batch?.bind(db); + if (!atomicBatch) throw new Error('D1 batch is unavailable'); + let loseAcknowledgement = false; + const first = new D1NotificationsStorage({ + prepare: db.prepare.bind(db), + async batch(statements) { + const results = await atomicBatch(statements); + if (loseAcknowledgement) { + loseAcknowledgement = false; + throw new Error('lost acknowledgement'); + } + return results; + }, + }); + const retry = new D1NotificationsStorage(db); + const input = { + id: 'remote-step-notification', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + payload: { value: 'private-receipt-canary' }, + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + await first.init(); + loseAcknowledgement = true; + const receipt = await first.createNotificationOnce( + 'remote-invocation', + input, + ); + expect(receipt).toEqual({ + operationKey: 'remote-invocation', + notificationId: input.id, + threadId: input.threadId, + }); + const retained = await db + .prepare('SELECT * FROM flowsafe_notification_operation_receipts') + .first>(); + expect(retained?.request_digest).toMatch(/^[a-f0-9]{64}$/); + expect(JSON.stringify(retained)).not.toContain(input.summary); + expect(JSON.stringify(retained)).not.toContain('private-receipt-canary'); + const delivered = await retry.getNotification({ + threadId: input.threadId, + id: input.id, + }); + expect(delivered?.summary).toBe(input.summary); + expect( + await retry.createNotificationOnce('remote-invocation', input), + ).toEqual(receipt); + await expect( + retry.createNotificationOnce('remote-invocation', { + ...input, + summary: 'Different result', + }), + ).rejects.toThrow('different request'); + expect( + (await retry.listNotifications({ threadId: input.threadId })).length, + ).toBe(1); + }); + + it('commits one effect under concurrent retry and rejects inbox identity collisions', async () => { + const db = database(); + const left = new D1NotificationsStorage(db); + const right = new D1NotificationsStorage(db); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const [first, second] = await Promise.all([ + left.createNotificationOnce('operation-1', input), + right.createNotificationOnce('operation-1', input), + ]); + expect(first).toEqual(second); + expect((await left.listNotifications({ threadId: 'thread' })).length).toBe( + 1, + ); + await expect( + right.createNotificationOnce('operation-2', input), + ).rejects.toThrow(); + const receipts = await db + .prepare( + 'SELECT operation_key FROM flowsafe_notification_operation_receipts ORDER BY operation_key', + ) + .all<{ operation_key: string }>(); + expect(receipts.results).toEqual([{ operation_key: 'operation-1' }]); + await left.createNotification({ ...input, id: 'ordinary-notification' }); + await expect( + right.createNotificationOnce('operation-3', { + ...input, + id: 'ordinary-notification', + summary: 'Overwrite attempt', + }), + ).rejects.toThrow('not committed'); + expect( + ( + await left.getNotification({ + threadId: 'thread', + id: 'ordinary-notification', + }) + )?.summary, + ).toBe('Completed'); + }); + + it('rolls back a receipt when the inbox insert fails after reservation', async () => { + const db = database(); + const notifications = new D1NotificationsStorage(db); + await notifications.init(); + await db + .prepare(`CREATE TRIGGER fail_notification_effect + BEFORE INSERT ON mastra_notifications + WHEN NEW.id = 'blocked-notification' + BEGIN SELECT RAISE(ABORT, 'blocked inbox insert'); END`) + .run(); + await expect( + notifications.createNotificationOnce('blocked-operation', { + id: 'blocked-notification', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'This effect does not commit', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }), + ).rejects.toThrow('blocked inbox insert'); + const receipt = await db + .prepare(`SELECT operation_key FROM flowsafe_notification_operation_receipts + WHERE operation_key = 'blocked-operation'`) + .first(); + expect(receipt).toBeNull(); + expect( + await notifications.getNotification({ + threadId: 'thread', + id: 'blocked-notification', + }), + ).toBeNull(); + }); +}); + function interceptFirst( db: SignalDatabase, intercept: (query: string, read: () => Promise) => Promise, diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index 6eccd141..3ac9ff02 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -29,6 +29,7 @@ import { notificationTimestampMillis, notificationTimestampSql, } from '../do-runner/notification-predicate.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; import { validateTablePrefix } from '../do-runner/table-prefix.js'; import { nonnegativeSafeInteger } from '../numeric-config.js'; import { @@ -332,6 +333,8 @@ function isDuplicateColumn(error: unknown): boolean { * a template literal is one the census can only match by restating it. */ export const NOTIFICATION_SEQUENCE_TABLE = 'flowsafe_notification_sequence'; +const NOTIFICATION_OPERATION_RECEIPTS_TABLE = + 'flowsafe_notification_operation_receipts'; export class D1NotificationsStorage extends NotificationsStorage @@ -340,6 +343,7 @@ export class D1NotificationsStorage readonly #db: SignalDatabase; readonly #table: string; readonly #sequenceTable: string; + readonly #operationReceiptTable: string; readonly #ordinalIndex: string; readonly #legacyReplaceTrigger: string; readonly #legacyInsertTrigger: string; @@ -351,6 +355,7 @@ export class D1NotificationsStorage this.#db = db; this.#table = `${prefix}mastra_notifications`; this.#sequenceTable = `${prefix}${NOTIFICATION_SEQUENCE_TABLE}`; + this.#operationReceiptTable = `${prefix}${NOTIFICATION_OPERATION_RECEIPTS_TABLE}`; this.#ordinalIndex = `idx_${this.#table}_insertion_ordinal`; this.#legacyReplaceTrigger = `trg_${this.#table}_preserve_insertion_ordinal`; this.#legacyInsertTrigger = `trg_${this.#table}_allocate_insertion_ordinal`; @@ -413,6 +418,17 @@ export class D1NotificationsStorage ) .run(); await this.#migrateOrdinalSchema(true); + await this.#db + .prepare( + `CREATE TABLE IF NOT EXISTS ${this.#operationReceiptTable} ( + operation_key TEXT PRIMARY KEY, + request_digest TEXT NOT NULL, + thread_id TEXT NOT NULL, + notification_id TEXT NOT NULL, + UNIQUE (thread_id, notification_id) + )`, + ) + .run(); } async #migrateOrdinalSchema(retryDuplicateColumn: boolean): Promise { @@ -547,6 +563,140 @@ export class D1NotificationsStorage await this.#ensureSchema(); } + async createNotificationOnce( + operationKey: string, + input: CreateNotificationInput & { id: string; createdAt: Date }, + ): Promise<{ + operationKey: string; + notificationId: string; + threadId: string; + }> { + if ( + !isPathSafeId(operationKey) || + !isPathSafeId(input.id) || + !isPathSafeId(input.threadId) || + !(input.createdAt instanceof Date) || + !Number.isFinite(input.createdAt.getTime()) + ) { + throw new TypeError('notification operation identity is malformed'); + } + await this.#ensureSchema(); + const batch = this.#db.batch?.bind(this.#db); + if (!batch) + throw new Error('notification operation requires database.batch()'); + const record: NotificationRecord = { + id: input.id, + threadId: input.threadId, + source: input.source, + kind: input.kind, + priority: input.priority ?? 'medium', + status: 'pending', + summary: input.summary, + payload: input.payload, + resourceId: input.resourceId, + agentId: input.agentId, + sourceId: input.sourceId, + dedupeKey: input.dedupeKey, + coalesceKey: input.coalesceKey, + coalescedCount: 1, + attributes: input.attributes, + createdAt: input.createdAt, + updatedAt: input.createdAt, + deliverAt: input.deliverAt, + summaryAt: input.summaryAt, + deliveryReason: input.deliveryReason, + deliveryAttempts: 0, + metadata: input.metadata, + }; + const values = recordValues(record); + validateNotificationDeliveryValues(values, true); + const requestJson = JSON.stringify(values); + const digestBytes = new Uint8Array( + await crypto.subtle.digest( + 'SHA-256', + new TextEncoder().encode(requestJson), + ), + ); + const requestDigest = Array.from(digestBytes, (byte) => + byte.toString(16).padStart(2, '0'), + ).join(''); + const receipt = { + operationKey, + notificationId: input.id, + threadId: input.threadId, + }; + const readReceipt = async () => + this.#db + .prepare( + `SELECT request_digest, thread_id, notification_id + FROM ${this.#operationReceiptTable} WHERE operation_key = ?`, + ) + .bind(operationKey) + .first<{ + request_digest: string; + thread_id: string; + notification_id: string; + }>(); + const existing = await readReceipt(); + if (existing) { + if (existing.request_digest !== requestDigest) + throw new Error('notification operation has a different request'); + return { + operationKey, + notificationId: existing.notification_id, + threadId: existing.thread_id, + }; + } + try { + // The inbox's legacy insert trigger updates existing IDs, so the receipt + // reserves both the operation key and inbox identity before insertion. + await batch([ + this.#db + .prepare( + `INSERT INTO ${this.#operationReceiptTable} + (operation_key, request_digest, thread_id, notification_id) + SELECT ?, ?, ?, ? + WHERE NOT EXISTS ( + SELECT 1 FROM ${this.#table} + WHERE thread_id = ? AND id = ? + )`, + ) + .bind( + operationKey, + requestDigest, + input.threadId, + input.id, + input.threadId, + input.id, + ), + this.#db + .prepare( + `INSERT INTO ${this.#table} (${NOTIFICATION_COLUMNS.join(', ')}) + SELECT ${NOTIFICATION_COLUMNS.map(() => '?').join(', ')} + WHERE EXISTS ( + SELECT 1 FROM ${this.#operationReceiptTable} + WHERE operation_key = ? AND request_digest = ? + )`, + ) + .bind(...values, operationKey, requestDigest), + ]); + const committed = await readReceipt(); + if (!committed) + throw new Error('notification operation was not committed'); + return receipt; + } catch (error) { + const committed = await readReceipt(); + if (!committed) throw error; + if (committed.request_digest !== requestDigest) + throw new Error('notification operation has a different request'); + return { + operationKey, + notificationId: committed.notification_id, + threadId: committed.thread_id, + }; + } + } + async createNotification( input: CreateNotificationInput, ): Promise { From ca2937cd29285258debe4e40bb3898392a5d65ab Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:47:58 +0400 Subject: [PATCH 26/73] feat(flowsafe): expose host-owned dispatch permit port Bind issue and winner-only consume to stored principal, resource, deployment and invocation identity. Keep transport and general runtime continuation closed. --- packages/flowsafe/src/agent-host/index.ts | 3 + .../src/agent-host/thread-topology.test.ts | 119 ++++++++++++++++- .../src/agent-host/thread-topology.ts | 121 ++++++++++++++++++ packages/flowsafe/src/do-runner/index.ts | 6 + .../src/do-runner/remote-step-journal.test.ts | 3 + .../src/do-runner/remote-step-journal.ts | 22 ++++ 6 files changed, 273 insertions(+), 1 deletion(-) diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index 1d26f49b..04cb83c9 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -13,6 +13,7 @@ export { export type { RecordedRemoteStepResult, RemoteStepInvocation, + RemoteStepPermit, } from '../do-runner/remote-step-journal.js'; export { type AgentApprovalResumer, @@ -55,6 +56,8 @@ export { type AgentThreadTopologyOptions, createAgentThreadTopology, type RemoteStepContinuationReader, + type RemoteStepDispatchReader, + type RemoteStepDispatchTicket, } from './thread-topology.js'; export { AGENT_AUDIT_CONTEXT_KEY, diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 00a65481..9b5a9b66 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -20,6 +20,7 @@ import { type AgentThreadDispatchTopology, createAgentThreadTopology, type RemoteStepContinuationReader, + type RemoteStepDispatchReader, } from './thread-topology.js'; const DEPLOYMENT_IDENTITY_SECRET = 'test-deployment-identity-secret-0001'; @@ -30,7 +31,10 @@ interface Hit { init?: ThreadRequestInit; } -function harness(remoteSteps?: RemoteStepContinuationReader): { +function harness( + remoteSteps?: RemoteStepContinuationReader, + remoteStepDispatch?: RemoteStepDispatchReader, +): { topology: AgentThreadDispatchTopology; hits: Hit[]; } { @@ -94,11 +98,124 @@ function harness(remoteSteps?: RemoteStepContinuationReader): { startIdempotency: 'none', executionFence: 'none', ...(remoteSteps ? { remoteSteps } : {}), + ...(remoteStepDispatch ? { remoteStepDispatch } : {}), }), hits, }; } +it('issues a remote dispatch ticket and exposes a raw permit to the consume winner', async () => { + const permit = { + invocationId: 'invocation_1', + generation: 2, + mutationEpoch: 7, + transitionRevision: 4, + expiresAt: 9_999_999_999_999, + }; + const invocation = { + invocationId: permit.invocationId, + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'acme_thread', + resourceId: 'acme_resource_acme_thread', + runId: 'acme_run', + principal: context().value.principal, + startToken: 'private-generation', + stepPath: ['tool'], + suspendedAt: 1000, + request: { script: 'script-1' }, + deadlineAt: permit.expiresAt, + status: 'dispatched' as const, + permitGeneration: permit.generation, + }; + const reader: RemoteStepDispatchReader = { + deploymentId: invocation.deploymentId, + tablePrefix: invocation.tablePrefix, + get: vi.fn(async () => invocation), + readDispatchRequestJson: vi.fn(async () => '{"script":"script-1"}'), + issuePermit: vi.fn(async () => permit), + consumePermit: vi + .fn() + .mockResolvedValueOnce({ kind: 'accepted', rawPermit: 'secret' }) + .mockResolvedValueOnce({ kind: 'already-consumed' }), + }; + const { topology } = harness(undefined, reader); + const ticket = await topology.issueRemoteStepDispatch( + context().value, + permit.invocationId, + 60_000, + ); + expect(ticket).toMatchObject({ + invocationId: permit.invocationId, + requestJson: '{"script":"script-1"}', + permit, + }); + expect(ticket).not.toHaveProperty('principal'); + expect(ticket).not.toHaveProperty('startToken'); + expect(reader.issuePermit).toHaveBeenCalledWith(permit.invocationId, 60_000); + expect( + await topology.consumeRemoteStepDispatch(context().value, permit), + ).toEqual({ kind: 'accepted', rawPermit: 'secret' }); + expect( + await topology.consumeRemoteStepDispatch(context().value, permit), + ).toEqual({ kind: 'already-consumed' }); + const displaced = context().value; + await expect( + topology.issueRemoteStepDispatch( + { + ...displaced, + principal: { kind: 'human', id: 'other', role: 'operator' }, + }, + permit.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.issuePermit).toHaveBeenCalledTimes(1); + await expect( + topology.consumeRemoteStepDispatch( + { + ...displaced, + principal: { kind: 'human', id: 'other', role: 'operator' }, + }, + permit, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.consumePermit).toHaveBeenCalledTimes(2); + const deniedAccess = { + ...displaced, + canAccessResource: async () => false, + }; + await expect( + topology.issueRemoteStepDispatch(deniedAccess, permit.invocationId, 60_000), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.consumeRemoteStepDispatch(deniedAccess, permit), + ).rejects.toMatchObject({ status: 404 }); + const displacedResource = { + ...displaced, + resourceIdFromKey: () => 'another-resource', + }; + await expect( + topology.issueRemoteStepDispatch( + displacedResource, + permit.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.get).mockResolvedValueOnce({ + ...invocation, + deploymentId: 'other-deployment', + }); + await expect( + topology.issueRemoteStepDispatch(displaced, permit.invocationId, 60_000), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.issuePermit).toHaveBeenCalledTimes(1); + expect(reader.readDispatchRequestJson).toHaveBeenCalledTimes(1); + expect(reader.consumePermit).toHaveBeenCalledTimes(2); +}); + it('routes a retained result under the journaled principal and refuses a changed resource mapping', async () => { const principal = { kind: 'human' as const, diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 6ac23e2f..0532fb1c 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -11,6 +11,7 @@ import { import { isExecutionPrincipalId, isExecutionPrincipalKind, + samePrincipal, } from '../approval-api/principal.js'; import { APPROVAL_PRIORITIES, @@ -33,6 +34,7 @@ import { import type { RecordedRemoteStepResult, RemoteStepInvocation, + RemoteStepPermit, } from '../do-runner/remote-step-journal.js'; import { isTerminalRunStatus, @@ -126,6 +128,39 @@ export interface AgentThreadTopologyOptions { */ executionFence: ExecutionFenceWiring; remoteSteps?: RemoteStepContinuationReader; + remoteStepDispatch?: RemoteStepDispatchReader; +} + +export interface RemoteStepDispatchReader { + deploymentId: string; + tablePrefix: string; + get(invocationId: string): Promise; + readDispatchRequestJson(invocationId: string): Promise; + issuePermit( + invocationId: string, + lifetimeMs: number, + ): Promise; + consumePermit( + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + >; +} + +export interface RemoteStepDispatchTicket { + invocationId: string; + deploymentId: string; + workflowId: string; + agentId: string; + threadId: string; + resourceId: string; + runId: string; + stepPath: readonly string[]; + suspendedAt: number; + resumeCount?: number; + deadlineAt: number; + requestJson: string; + permit: RemoteStepPermit; } export interface RemoteStepContinuationReader { @@ -212,6 +247,17 @@ export interface AgentThreadDispatchTopology extends AgentThreadBoundTopology { context: ActorContext, invocationId: string, ): Promise; + issueRemoteStepDispatch( + context: ActorContext, + invocationId: string, + lifetimeMs: number, + ): Promise; + consumeRemoteStepDispatch( + context: ActorContext, + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + >; } function errorFrom(response: Response): Promise { @@ -261,6 +307,46 @@ export function createAgentThreadTopology( namespace, deploymentIdentitySecret, ); + const dispatchInvocation = async ( + context: ActorContext, + invocationId: string, + ) => { + const reader = options.remoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.get(invocationId); + if ( + !invocation || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + !samePrincipal(invocation.principal, context.principal) + ) + throw new RunRouteError(404, 'invocation not found'); + expectedResource(context, invocation.threadId, invocation.resourceId); + await requireResourceAccess( + context, + 'thread', + invocation.threadId, + 'write', + 'run', + ); + await requireResourceAccess( + context, + 'resource', + invocation.resourceId, + 'write', + 'run', + ); + await requireResourceAccess( + context, + 'run', + invocation.runId, + 'write', + 'run', + ); + return { reader, invocation }; + }; const fetchStatus = async ( context: ActorContext, input: AgentThreadRunRef, @@ -417,6 +503,41 @@ export function createAgentThreadTopology( } }; return { + issueRemoteStepDispatch: async (context, invocationId, lifetimeMs) => { + await dispatchInvocation(context, invocationId); + const reader = options.remoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + const permit = await reader.issuePermit(invocationId, lifetimeMs); + const { invocation } = await dispatchInvocation(context, invocationId); + if ( + invocation.status !== 'dispatched' || + invocation.permitGeneration !== permit.generation || + invocation.deadlineAt < permit.expiresAt + ) + throw new RunRouteError(409, 'remote step dispatch changed'); + const requestJson = await reader.readDispatchRequestJson(invocationId); + return { + invocationId, + deploymentId: invocation.deploymentId, + workflowId: invocation.workflowId, + agentId: invocation.agentId, + threadId: invocation.threadId, + resourceId: invocation.resourceId, + runId: invocation.runId, + stepPath: invocation.stepPath, + suspendedAt: invocation.suspendedAt, + ...(invocation.resumeCount === undefined + ? {} + : { resumeCount: invocation.resumeCount }), + deadlineAt: invocation.deadlineAt, + requestJson, + permit, + }; + }, + consumeRemoteStepDispatch: async (context, permit) => { + const { reader } = await dispatchInvocation(context, permit.invocationId); + return reader.consumePermit(permit); + }, requireBoundThread: async (context, target: BoundThreadTarget) => { const threadId = validThread(target.threadId); const resourceId = expectedResource(context, threadId, target.resourceId); diff --git a/packages/flowsafe/src/do-runner/index.ts b/packages/flowsafe/src/do-runner/index.ts index 55055e18..2290f5ce 100644 --- a/packages/flowsafe/src/do-runner/index.ts +++ b/packages/flowsafe/src/do-runner/index.ts @@ -224,6 +224,12 @@ export { mintThreadId, resourceIdFromKey } from './memory-id.js'; export { isPathSafeId, PATH_SAFE_ID_PATTERN } from './path-safe-id.js'; export type { HostPubSub } from './pubsub.js'; export { createHostPubSub } from './pubsub.js'; +export { + type RemoteStepAddress, + type RemoteStepInvocation, + RemoteStepJournal, + type RemoteStepPermit, +} from './remote-step-journal.js'; export type { RunEconomicOperation, RunLifecyclePrincipal, diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index a3932d32..9b44d964 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -182,6 +182,9 @@ describe('remote step journal dispatch admission', () => { h.journal.register(ADDRESS, 10_000, PRINCIPAL), ]); expect(replay.invocationId).toBe(first.invocationId); + expect(await h.journal.readDispatchRequestJson(first.invocationId)).toBe( + '{"account":"acme"}', + ); expect(first).toMatchObject({ status: 'pending', permitGeneration: 0, diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 2631daae..c1f678be 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -430,6 +430,14 @@ export class RemoteStepJournal { evidenceJson: string; }) => Promise; + get deploymentId(): string { + return this.#deploymentId; + } + + get tablePrefix(): string { + return this.#tablePrefix; + } + constructor(input: { db: ExecutionFenceDatabase; fence: ExecutionFenceStore; @@ -861,6 +869,20 @@ export class RemoteStepJournal { return row ? decode(row) : undefined; } + async readDispatchRequestJson(invocationId: string): Promise { + if (!isPathSafeId(invocationId)) + throw new RemoteStepRefusal(400, 'invocation id is malformed'); + const row = rowResult( + await this.#db + .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) + .bind(invocationId) + .all(), + ); + if (!row) throw new RemoteStepRefusal(404, 'invocation not found'); + decode(row); + return row.request_json; + } + async #execution(address: RemoteStepAddress, principal?: ExecutionPrincipal) { const state = await this.#runtime.authoritativeStartState( address.workflowId, From c3b86e3e0266035f2ece045aee301c9bba0d574c Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:23:13 +0400 Subject: [PATCH 27/73] feat(flowsafe): derive host context from stored invocation Expose a pre-route host helper that reconstructs the persisted principal with the configured approval and service wiring. Reject mismatched invocation identity before dispatch. --- .../src/host-kit/flowsafe-worker.test.ts | 120 ++++++++++++++++++ .../flowsafe/src/host-kit/flowsafe-worker.ts | 85 ++++++++++--- 2 files changed, 188 insertions(+), 17 deletions(-) diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts index 02904ef0..ae8b8764 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts @@ -936,6 +936,126 @@ describe('createFlowsafeWorker fetch pipeline', () => { expect(await list.json()).toEqual([]); }); + it('binds a pre-route context to the stored invocation principal', async () => { + const journal = { + deploymentId: 'deployment-1', + tablePrefix: '', + get: vi.fn(async () => ({ + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + tablePrefix: '', + principal: { + kind: 'service' as const, + id: 'trusted-dispatcher', + purpose: 'remote-step', + }, + })), + }; + const worker = makeWorker({ + preRoutes: async (request, _env, _ctx, kit) => { + if (new URL(request.url).pathname !== '/test/stored-principal') + return null; + const context = await kit.contextForStoredInvocation( + journal, + 'invocation-1', + ); + return Response.json(context?.principal ?? null); + }, + }); + const { env, ctx } = makeEnv(); + + const response = await worker.fetch( + new Request('http://host/test/stored-principal'), + env, + ctx, + ); + + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ + kind: 'service', + id: 'trusted-dispatcher', + purpose: 'remote-step', + }); + expect(journal.get).toHaveBeenCalledWith('invocation-1'); + }); + + it.each([ + { + name: 'missing invocation', + invocationId: 'invocation-1', + row: undefined, + reads: 1, + }, + { + name: 'mismatched invocation id', + invocationId: 'invocation-1', + row: { invocationId: 'another-invocation' }, + reads: 1, + }, + { + name: 'mismatched deployment', + invocationId: 'invocation-1', + row: { deploymentId: 'another-deployment' }, + reads: 1, + }, + { + name: 'mismatched table prefix', + invocationId: 'invocation-1', + row: { tablePrefix: 'another_' }, + reads: 1, + }, + { + name: 'invalid invocation id', + invocationId: '../invocation-1', + row: {}, + reads: 0, + }, + ])('refuses a stored-principal context for $name', async ({ + invocationId, + row, + reads, + }) => { + const stored = { + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + tablePrefix: '', + principal: { + kind: 'service' as const, + id: 'trusted-dispatcher', + purpose: 'remote-step', + }, + }; + const journal = { + deploymentId: 'deployment-1', + tablePrefix: '', + get: vi.fn(async () => + row === undefined ? undefined : { ...stored, ...row }, + ), + }; + const worker = makeWorker({ + preRoutes: async (request, _env, _ctx, kit) => { + if (new URL(request.url).pathname !== '/test/stored-principal') + return null; + const context = await kit.contextForStoredInvocation( + journal, + invocationId, + ); + return Response.json(context?.principal ?? null); + }, + }); + const { env, ctx } = makeEnv(); + + const response = await worker.fetch( + new Request('http://host/test/stored-principal'), + env, + ctx, + ); + + expect(response.status).toBe(200); + expect(await response.json()).toBeNull(); + expect(journal.get).toHaveBeenCalledTimes(reads); + }); + it('mounts the agent router after preRoutes and before the signal router', async () => { const order: string[] = []; const worker = makeWorker({ diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.ts index b0471fc4..2daff5ed 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.ts @@ -15,10 +15,14 @@ import { APPROVAL_ROLES, createActorResolver, createApprovalRouter, + createPrincipalActorContext, createResourceOwnershipSchema, + DECIDER_ROLES, RESOURCE_OWNERSHIP_TABLE, + selfDecisionExempts, trustAutomationPrincipal, } from '../approval-api/index.js'; +import type { ExecutionPrincipal } from '../approval-api/principal.js'; import { type AuditProxyNamespaceLike, type AuditQueue, @@ -68,6 +72,7 @@ import { verifyDurableObjectDeploymentIdentity, verifyDurableObjectDeploymentRequest, } from '../do-runner/index.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; import { validateTablePrefix } from '../do-runner/table-prefix.js'; import { readBoundedBody } from '../http-body.js'; import { @@ -383,7 +388,26 @@ export interface FlowsafeWorkerConfig request: Request, env: Env, ctx: FlowsafeWorkerContext, - kit: { resolve: ActorResolver; topology: DoRunTopology }, + kit: { + resolve: ActorResolver; + topology: DoRunTopology; + contextForStoredInvocation: ( + journal: { + deploymentId: string; + tablePrefix: string; + get: (invocationId: string) => Promise< + | { + invocationId: string; + deploymentId: string; + tablePrefix: string; + principal: ExecutionPrincipal; + } + | undefined + >; + }, + invocationId: string, + ) => Promise; + }, ) => Promise; /** Host policy immediately before a validated start reaches the run DO. */ beforeStart?: ( @@ -1177,7 +1201,7 @@ export function createFlowsafeWorker( 'storageTablePrefix', ); // Authenticate before constructing request-scoped services. - const buildResolve = ( + const buildContexts = ( env: Env, topology: DoRunTopology, waitUntil: (promise: Promise) => void, @@ -1185,28 +1209,41 @@ export function createFlowsafeWorker( selfDecision: SelfDecisionPolicy, stream: ApprovalStreamSink | undefined, mutationEpoch: number | undefined, - ): ActorResolver => { - const base = createActorResolver({ + ) => { + const storeFactory = approvalStoreFactoryFor(env.DB, storageTablePrefix); + const buildService = (store: ApprovalStore) => + buildConfiguredApprovalService(config, env, topology, { + store, + executionFence: executionFenceForEnv(env), + workflowTablePrefix: (storageTablePrefix ?? '').toLowerCase(), + waitUntil, + notify, + stream, + allowSelfDecision: selfDecision, + }); + const resolve = createActorResolver({ mutationEpoch, authenticate: bearerActorAuthenticator(config.buildVerifier(env)), - storeFactory: approvalStoreFactoryFor(env.DB, storageTablePrefix), + storeFactory, deploymentTag: env.DEPLOYMENT_TENANT, - buildService: (store) => - buildConfiguredApprovalService(config, env, topology, { - store, - executionFence: executionFenceForEnv(env), - workflowTablePrefix: (storageTablePrefix ?? '').toLowerCase(), - waitUntil, - notify, - stream, - allowSelfDecision: selfDecision, - }), + buildService, // The resolver's canSelfDecide display hint reads the SAME policy the // service enforces (passed to the shared service builder above), so the // /workflows echo can never contradict decide(). allowSelfDecision: selfDecision, }); - return base; + const contextForPrincipal = (principal: ExecutionPrincipal): ActorContext => + createPrincipalActorContext({ + principal, + mutationEpoch, + storeFactory, + deploymentTag: env.DEPLOYMENT_TENANT, + buildService, + canSelfDecide: (role) => + DECIDER_ROLES.includes(role) && + selfDecisionExempts(selfDecision, role), + }); + return { resolve, contextForPrincipal }; }; // Parse ONCE per deployment value, memoized by the RAW @@ -1595,7 +1632,7 @@ export function createFlowsafeWorker( ), ); } - const resolve = buildResolve( + const { resolve, contextForPrincipal } = buildContexts( env, topology, waitUntil, @@ -1609,6 +1646,20 @@ export function createFlowsafeWorker( const preResponse = await config.preRoutes(request, env, ctx, { resolve, topology, + contextForStoredInvocation: async (journal, invocationId) => { + if (!isPathSafeId(invocationId)) return undefined; + const invocation = await journal.get(invocationId); + if ( + !invocation || + invocation.invocationId !== invocationId || + invocation.deploymentId !== journal.deploymentId || + invocation.tablePrefix !== journal.tablePrefix || + invocation.tablePrefix !== + (storageTablePrefix ?? '').toLowerCase() + ) + return undefined; + return contextForPrincipal(invocation.principal); + }, }); if (preResponse) return preResponse; } From 912639512ec4314330d899b6221463eb83ee9d46 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:30:49 +0400 Subject: [PATCH 28/73] test(flowsafe): verify remote journal replay after restart Mount the spike remote-step fixture only for its proof generation and check consumed-permit and retained-result replay after a fresh workerd process starts. --- packages/flowsafe/scripts/spike-verify.mjs | 32 ++++++++++++++++++++++ packages/flowsafe/spike/worker.ts | 3 ++ 2 files changed, 35 insertions(+) diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index 520f81f3..30e7377d 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -176,6 +176,9 @@ function startServer(generation, stateDir, logPath) { // with, so a valid webhook is ACCEPTED and a forged one REJECTED (E-S2). '--var', `GITHUB_WEBHOOK_SECRET:${GITHUB_WEBHOOK_SECRET}`, + ...(generation.startsWith('gen-remote-') + ? ['--var', 'SPIKE_REMOTE_STEP_PROOF:enabled'] + : []), ], { cwd: FLOWSAFE, @@ -766,6 +769,18 @@ async function main() { launchServer('gen-1', stateDir, join(tmpDir, 'gen1.log')), ); + await step( + 'RS0 remote-step fixture is absent unless explicitly enabled', + async () => { + const absent = await http('GET', '/spike/remote-step/scan'); + assert( + absent.status === 404, + 'remote-step fixture stays unmounted', + absent, + ); + }, + ); + await step( 'PA0 boundary: authenticated catalog, role gates, and forged context/header rejection', async () => { @@ -3476,6 +3491,23 @@ async function main() { 'fresh workerd process reads the persisted result', retained, ); + const permitReplay = await http('POST', '/spike/remote-step/consume', { + body: { permit: issued.body }, + }); + assert( + permitReplay.status === 200 && + permitReplay.body.kind === 'already-consumed', + 'fresh process refuses to grant the consumed permit again', + permitReplay, + ); + const resultReplay = await http('POST', '/spike/remote-step/record', { + body: { invocationId, rawPermit }, + }); + assert( + resultReplay.status === 200 && resultReplay.body.kind === 'replay', + 'fresh process reads the retained result without a second effect', + resultReplay, + ); const displaced = await http('POST', '/spike/remote-step/claim', { body: { invocationId }, }); diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index a72c4dab..2646fa48 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -214,6 +214,7 @@ interface Env { /** Singleton deployment signal-provider host DO; see DemoSignalProviderHost (Track E). */ SIGNAL_PROVIDER_HOST: DurableObjectNamespace; BACKGROUND_TASKS: DurableObjectNamespace; + SPIKE_REMOTE_STEP_PROOF?: string; SPIKE_LLM_MODEL_ID?: string; SPIKE_LLM_API_KEY?: string; SPIKE_LLM_BASE_URL?: string; @@ -3087,6 +3088,8 @@ async function handleRemoteStepProbe( ): Promise { const path = new URL(request.url).pathname; if (!path.startsWith('/spike/remote-step/')) return null; + if (env.SPIKE_REMOTE_STEP_PROOF !== 'enabled') + return json({ error: 'not found' }, 404); const operation = path.slice('/spike/remote-step/'.length); if ( request.method !== From f1fd6d98be6316a6b4f3d525aad87dba09570e31 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:18:16 +0400 Subject: [PATCH 29/73] feat(flowsafe): cap retained notification receipts Bound first-effect admission at 100,000 receipts in the D1 transaction, keep identical replay available at capacity, and prevent a losing batch from mutating the inbox. --- .../src/signals/notifications-d1.test.ts | 127 ++++++++++++++++++ .../flowsafe/src/signals/notifications-d1.ts | 29 +++- packages/flowsafe/src/signals/storage.ts | 12 +- 3 files changed, 162 insertions(+), 6 deletions(-) diff --git a/packages/flowsafe/src/signals/notifications-d1.test.ts b/packages/flowsafe/src/signals/notifications-d1.test.ts index 461acb9e..ba50b9f6 100644 --- a/packages/flowsafe/src/signals/notifications-d1.test.ts +++ b/packages/flowsafe/src/signals/notifications-d1.test.ts @@ -39,6 +39,133 @@ function sharedStores(): [D1NotificationsStorage, D1NotificationsStorage] { } describe('host-owned notification operation receipt', () => { + it('refuses a new operation at the retained receipt cap while replay still succeeds', async () => { + const db = database(); + const notifications = new D1NotificationsStorage(db, '', 1); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const first = await notifications.createNotificationOnce( + 'operation-1', + input, + ); + await expect( + notifications.createNotificationOnce('operation-2', { + ...input, + id: 'notification-2', + }), + ).rejects.toThrow('notification operation receipt cap reached'); + expect( + await notifications.createNotificationOnce('operation-1', input), + ).toEqual(first); + expect( + await notifications.getNotification({ + threadId: 'thread', + id: 'notification-2', + }), + ).toBeNull(); + }); + + it('admits at most one concurrent new operation at the last receipt slot', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, '', 1); + const right = new D1NotificationsStorage(db, '', 1); + const input = { + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const results = await Promise.allSettled([ + left.createNotificationOnce('operation-1', { + ...input, + id: 'notification-1', + }), + right.createNotificationOnce('operation-2', { + ...input, + id: 'notification-2', + }), + ]); + expect( + results.filter((result) => result.status === 'fulfilled'), + ).toHaveLength(1); + expect( + results.filter((result) => result.status === 'rejected'), + ).toHaveLength(1); + const receipts = await db + .prepare( + 'SELECT operation_key FROM flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string }>(); + expect(receipts.results).toHaveLength(1); + }); + + it('does not reset a delivered notification when a stale same-key retry reaches the cap', async () => { + const db = database(); + const first = new D1NotificationsStorage(db, '', 1); + let signalRead: () => void = () => undefined; + let releaseRead: () => void = () => undefined; + const readPaused = new Promise((resolve) => { + signalRead = resolve; + }); + const readReleased = new Promise((resolve) => { + releaseRead = resolve; + }); + let intercepted = false; + const retry = new D1NotificationsStorage( + interceptFirst(db, async (query, read) => { + const result = await read(); + if ( + !intercepted && + result === null && + query.includes( + 'FROM flowsafe_notification_operation_receipts WHERE operation_key = ?', + ) + ) { + intercepted = true; + signalRead(); + await readReleased; + } + return result; + }), + '', + 1, + ); + await Promise.all([first.init(), retry.init()]); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const pendingRetry = retry.createNotificationOnce('operation-1', input); + await readPaused; + const receipt = await first.createNotificationOnce('operation-1', input); + await first.updateNotification({ + threadId: input.threadId, + id: input.id, + status: 'delivered', + }); + releaseRead(); + expect(await pendingRetry).toEqual(receipt); + expect( + ( + await first.getNotification({ + threadId: input.threadId, + id: input.id, + }) + )?.status, + ).toBe('delivered'); + }); + it('returns the first durable receipt after a lost acknowledgement without overwriting the inbox', async () => { const db = database(); const atomicBatch = db.batch?.bind(db); diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index 3ac9ff02..2553dd16 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -335,6 +335,7 @@ function isDuplicateColumn(error: unknown): boolean { export const NOTIFICATION_SEQUENCE_TABLE = 'flowsafe_notification_sequence'; const NOTIFICATION_OPERATION_RECEIPTS_TABLE = 'flowsafe_notification_operation_receipts'; +export const DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT = 100_000; export class D1NotificationsStorage extends NotificationsStorage @@ -344,18 +345,27 @@ export class D1NotificationsStorage readonly #table: string; readonly #sequenceTable: string; readonly #operationReceiptTable: string; + readonly #operationReceiptLimit: number; readonly #ordinalIndex: string; readonly #legacyReplaceTrigger: string; readonly #legacyInsertTrigger: string; #ready?: Promise; - constructor(db: SignalDatabase, tablePrefix = '') { + constructor( + db: SignalDatabase, + tablePrefix = '', + operationReceiptLimit = DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, + ) { super(); const prefix = validateTablePrefix(tablePrefix) ?? ''; this.#db = db; this.#table = `${prefix}mastra_notifications`; this.#sequenceTable = `${prefix}${NOTIFICATION_SEQUENCE_TABLE}`; this.#operationReceiptTable = `${prefix}${NOTIFICATION_OPERATION_RECEIPTS_TABLE}`; + this.#operationReceiptLimit = nonnegativeSafeInteger( + operationReceiptLimit, + 'operationReceiptLimit', + ); this.#ordinalIndex = `idx_${this.#table}_insertion_ordinal`; this.#legacyReplaceTrigger = `trg_${this.#table}_preserve_insertion_ordinal`; this.#legacyInsertTrigger = `trg_${this.#table}_allocate_insertion_ordinal`; @@ -659,7 +669,9 @@ export class D1NotificationsStorage WHERE NOT EXISTS ( SELECT 1 FROM ${this.#table} WHERE thread_id = ? AND id = ? - )`, + ) AND ( + SELECT COUNT(*) FROM ${this.#operationReceiptTable} + ) < ?`, ) .bind( operationKey, @@ -668,12 +680,13 @@ export class D1NotificationsStorage input.id, input.threadId, input.id, + this.#operationReceiptLimit, ), this.#db .prepare( `INSERT INTO ${this.#table} (${NOTIFICATION_COLUMNS.join(', ')}) SELECT ${NOTIFICATION_COLUMNS.map(() => '?').join(', ')} - WHERE EXISTS ( + WHERE changes() = 1 AND EXISTS ( SELECT 1 FROM ${this.#operationReceiptTable} WHERE operation_key = ? AND request_digest = ? )`, @@ -681,8 +694,16 @@ export class D1NotificationsStorage .bind(...values, operationKey, requestDigest), ]); const committed = await readReceipt(); - if (!committed) + if (!committed) { + const count = await this.#db + .prepare( + `SELECT COUNT(*) AS count FROM ${this.#operationReceiptTable}`, + ) + .first<{ count: number }>(); + if (count && count.count >= this.#operationReceiptLimit) + throw new Error('notification operation receipt cap reached'); throw new Error('notification operation was not committed'); + } return receipt; } catch (error) { const committed = await readReceipt(); diff --git a/packages/flowsafe/src/signals/storage.ts b/packages/flowsafe/src/signals/storage.ts index f3ca2439..480c3d26 100644 --- a/packages/flowsafe/src/signals/storage.ts +++ b/packages/flowsafe/src/signals/storage.ts @@ -12,7 +12,10 @@ import type { MastraStorageDomains } from '@mastra/core/storage'; import type { D1DatabaseBinding } from '../do-runner/index.js'; import { validateTablePrefix } from '../do-runner/table-prefix.js'; import type { SignalDatabase } from './d1-shared.js'; -import { D1NotificationsStorage } from './notifications-d1.js'; +import { + D1NotificationsStorage, + DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, +} from './notifications-d1.js'; import { D1ThreadStateStorage } from './thread-state-d1.js'; /** @@ -25,13 +28,18 @@ import { D1ThreadStateStorage } from './thread-state-d1.js'; export function createSignalStorageDomains( binding: D1DatabaseBinding, tablePrefix = '', + notificationOperationReceiptLimit = DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, ): MastraStorageDomains { const prefix = validateTablePrefix(tablePrefix) ?? ''; // The structural SignalDatabase subset (prepare→bind/first/all/run) is exactly // what a real D1Database exposes; the cast bridges the workers-types-free seam. const db = binding as unknown as SignalDatabase; return { - notifications: new D1NotificationsStorage(db, prefix), + notifications: new D1NotificationsStorage( + db, + prefix, + notificationOperationReceiptLimit, + ), threadState: new D1ThreadStateStorage(db, prefix), }; } From 61d4150546da0a6acfe329bd415bcb929d231581 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:25:54 +0400 Subject: [PATCH 30/73] test(flowsafe): prove local authenticated dispatch admission Exercise a dedicated pre-route credential, stored principal, one-time journal consume and replay refusal without adding a production route. --- .../src/do-runner/remote-step-journal.test.ts | 190 ++++++++++++++++++ 1 file changed, 190 insertions(+) diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 9b44d964..66b8e5a5 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -3,7 +3,13 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { z } from 'zod'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; +import { createAgentThreadTopology } from '../agent-host/thread-topology.js'; +import type { FlowsafeWorkerEnv } from '../host-kit/flowsafe-worker.js'; +import { createFlowsafeWorker } from '../host-kit/flowsafe-worker.js'; +import { approvalStoreFactoryFor } from '../host-kit/host-approval-service.js'; +import { staticTokenVerifier } from '../host-kit/verifier.js'; import { createD1Storage } from './d1-storage.js'; +import { credentialsMatch } from './deployment-identity.js'; import { ExecutionFenceStore } from './execution-fence.js'; import { init } from './init.js'; import { DeploymentInventory } from './inventory.js'; @@ -210,6 +216,190 @@ describe('remote step journal dispatch admission', () => { ]); }); + it('refuses unauthorized dispatch and replay before the trusted effect', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + const owner = { kind: PRINCIPAL.kind, id: PRINCIPAL.id }; + const resources = approvalStoreFactoryFor(h.db).resources(); + for (const kind of ['thread', 'resource', 'run'] as const) { + const id = kind === 'run' ? ADDRESS.runId : ADDRESS.threadId; + expect(await resources.claim(kind, id, owner)).toBe(true); + } + const topology = createAgentThreadTopology( + { + idFromName: (name) => name, + get: () => ({ + fetch: async () => { + throw new Error('dispatch admission must not address a thread'); + }, + }), + }, + 'test-deployment-identity-secret-0001', + { + startIdempotency: 'none', + executionFence: 'none', + remoteStepDispatch: h.journal, + }, + ); + type Env = FlowsafeWorkerEnv & { DISPATCH_FIXTURE_SECRET: string }; + const effect = vi.fn(async (_rawPermit: string) => undefined); + const worker = createFlowsafeWorker({ + workflows: [], + systemPrincipalId: 'test-system', + buildVerifier: () => staticTokenVerifier(new Map()), + maintenance: { sweepIntervalMs: 60_000, purgeIntervalMs: 60_000 }, + preRoutes: async (request, env, _ctx, kit) => { + if (new URL(request.url).pathname !== '/__fixture/remote-dispatch') + return null; + const credential = request.headers.get('x-dispatch-fixture-secret'); + if ( + !credential || + !(await credentialsMatch(credential, env.DISPATCH_FIXTURE_SECRET)) + ) + return new Response(null, { status: 401 }); + const input = (await request.json()) as Record; + const keys = Object.keys(input).sort().join(','); + const invocationId = + input.action === 'issue' && keys === 'action,invocationId' + ? input.invocationId + : input.action === 'consume' && keys === 'action,permit' + ? (input.permit as { invocationId?: unknown } | null) + ?.invocationId + : undefined; + if (typeof invocationId !== 'string') + return new Response(null, { status: 400 }); + const context = await kit.contextForStoredInvocation( + h.journal, + invocationId, + ); + if (!context) return new Response(null, { status: 404 }); + if (input.action === 'issue') { + const ticket = await topology.issueRemoteStepDispatch( + context, + invocationId, + 1_000, + ); + return Response.json(ticket); + } + const result = await topology.consumeRemoteStepDispatch( + context, + input.permit as Parameters< + typeof topology.consumeRemoteStepDispatch + >[1], + ); + if (result.kind === 'already-consumed') + return new Response(null, { status: 409 }); + await effect(result.rawPermit); + return Response.json({ kind: 'accepted' }); + }, + }); + h.sqlite.exec(`CREATE TABLE flowsafe_deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + tenant_tag TEXT NOT NULL, + provisioned_at TEXT NOT NULL + )`); + h.sqlite + .prepare( + `INSERT INTO flowsafe_deployment VALUES (1, 'acme', '2026-08-10T00:00:00.000Z')`, + ) + .run(); + const fixtureEnv: Env = { + DB: h.db, + DEPLOYMENT_TENANT: 'acme', + DEPLOYMENT_IDENTITY_SECRET: 'test-deployment-identity-secret-0001', + DISPATCH_FIXTURE_SECRET: 'separate-test-dispatch-secret-0001', + RUNNER: { + idFromName: (name) => name, + get: () => ({ fetch: async () => new Response(null, { status: 503 }) }), + }, + MAINTENANCE: { + idFromName: (name) => name, + get: () => ({ fetch: async () => new Response(null, { status: 503 }) }), + }, + }; + const ctx = { waitUntil: () => undefined }; + const send = (body: unknown, secret?: string) => + worker.fetch( + new Request('https://host/__fixture/remote-dispatch', { + method: 'POST', + headers: { + 'content-type': 'application/json', + ...(secret ? { 'x-dispatch-fixture-secret': secret } : {}), + }, + body: JSON.stringify(body), + }), + fixtureEnv, + ctx, + ); + + expect( + (await send({ action: 'issue', invocationId: invocation.invocationId })) + .status, + ).toBe(401); + expect( + ( + await send( + { action: 'issue', invocationId: invocation.invocationId }, + 'wrong-secret', + ) + ).status, + ).toBe(401); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'pending', + ); + expect(effect).not.toHaveBeenCalled(); + expect( + ( + await send( + { + action: 'issue', + invocationId: invocation.invocationId, + lifetimeMs: 60_000, + }, + fixtureEnv.DISPATCH_FIXTURE_SECRET, + ) + ).status, + ).toBe(400); + expect( + ( + await send( + { + action: 'issue', + invocationId: invocation.invocationId, + principal: PRINCIPAL, + }, + fixtureEnv.DISPATCH_FIXTURE_SECRET, + ) + ).status, + ).toBe(400); + expect((await h.journal.get(invocation.invocationId))?.status).toBe( + 'pending', + ); + + const issued = await send( + { action: 'issue', invocationId: invocation.invocationId }, + fixtureEnv.DISPATCH_FIXTURE_SECRET, + ); + expect(issued.status).toBe(200); + const ticket = (await issued.json()) as { + permit: Parameters[1]; + }; + expect(ticket).not.toHaveProperty('rawPermit'); + const consume = { action: 'consume', permit: ticket.permit }; + expect((await send(consume)).status).toBe(401); + expect((await send(consume, 'wrong-secret')).status).toBe(401); + expect(effect).not.toHaveBeenCalled(); + const accepted = await send(consume, fixtureEnv.DISPATCH_FIXTURE_SECRET); + expect(accepted.status).toBe(200); + expect(await accepted.json()).toEqual({ kind: 'accepted' }); + expect(effect).toHaveBeenCalledTimes(1); + expect(effect).toHaveBeenCalledWith(expect.stringMatching(/^[\w-]{43}$/)); + expect( + (await send(consume, fixtureEnv.DISPATCH_FIXTURE_SECRET)).status, + ).toBe(409); + expect(effect).toHaveBeenCalledTimes(1); + }); + it('registers the resource recorded in start provenance independently of the thread id', async () => { const h = await fixture(); const original = h.runtime.authoritativeStartState.bind(h.runtime); From ee4003a42882c0d1282004d5fed0274a5b4e2939 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:42:40 +0400 Subject: [PATCH 31/73] chore(flowsafe): record remote-step broker release Declare a minor FlowSafe release for the gated host dispatch and notification receipt APIs while keeping general Mastra continuation closed. --- .changeset/flowsafe-remote-step-broker.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changeset/flowsafe-remote-step-broker.md diff --git a/.changeset/flowsafe-remote-step-broker.md b/.changeset/flowsafe-remote-step-broker.md new file mode 100644 index 00000000..9ecb2380 --- /dev/null +++ b/.changeset/flowsafe-remote-step-broker.md @@ -0,0 +1,5 @@ +--- +'@proofoftech/flowsafe': minor +--- + +Add a D1-backed remote-step journal, one-time host dispatch permits, retained results, and a host context derived from the stored invocation. A host-owned notification operation stores its effect and replay receipt in one D1 transaction with a configurable retained-receipt cap. General Mastra continuation remains fail-closed without generation-fenced snapshot writes; hosts must supply trusted dispatch transport and downstream effect receipts before enabling remote execution. From 06ef2498488380370c948df0c3a90247cc0c78ea Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:00:00 +0400 Subject: [PATCH 32/73] docs(flowsafe): plan workflow-target remote-step journal Specify v4 row migration, mixed-version write refusal, legacy permit continuity and fail-closed workflow result retention before implementation. --- ...2026-10-13-workflow-remote-step-journal.md | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 docs/plans/2026-10-13-workflow-remote-step-journal.md diff --git a/docs/plans/2026-10-13-workflow-remote-step-journal.md b/docs/plans/2026-10-13-workflow-remote-step-journal.md new file mode 100644 index 00000000..e4e9cb8c --- /dev/null +++ b/docs/plans/2026-10-13-workflow-remote-step-journal.md @@ -0,0 +1,81 @@ +# Workflow-target remote-step journal + +> Content type: implementation plan. Audience: FlowSafe maintainers. Status: planned; customer dispatch remains disabled. + +## Baseline and environment + +- Branch `plan/flowsafe-remote-step-broker` at `ee4003a42882c0d1282004d5fed0274a5b4e2939` (2026-10-13). Reconfirm the appendix anchors against the checked-out commit before editing. The concurrent state-egress checkpoint owns `src/host-kit/state-egress.ts` and `flowsafe-worker.ts`; this plan does not edit them. +- Node and pnpm come from the root manifest. Work in the linked worktree with the shared package store. Capture checks in `/tmp` with a final `EXIT_STATUS` line. D1 `batch()` is transactional according to [Cloudflare's API documentation](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch). +- This is a default-off journal contract. No workflow host route, ASP permit authority, customer dispatch, or automatic `run.resume` is enabled by this checkpoint. + +## Decision + +Extend the existing `RemoteStepJournal` D1 authority to distinguish `agent` and `workflow` target rows. Preserve every v3 agent row and its permit/result state. A workflow row uses the trusted start provenance `{kind:'workflow', id: workflowId}` and has no agent, thread, or resource ID. Its suspension carries a versioned marker and exact JSON request bytes; the journal stores a SHA-256 digest of those bytes. Registration and permit consumption compare the same authoritative snapshot, principal, deployment, start token, fence generation, suspension identity, request bytes, and deadline as the agent path. The first retained result remains immutable under the consumed permit digest. + +The workflow suspension wire shape is `suspendPayload[stepPath.join('.')].remoteStep = { marker: 'flowsafe.remote-step.workflow/v1', request: }`, alongside `type: 'suspension'`. The trusted workflow host must emit this shape; user code cannot obtain authority merely by constructing the payload. The marker is a versioned parser selector, not proof of ownership. Confirm it against the ASP v2 author before wiring a caller. + +Reject synthetic agent/thread/resource IDs: they would make a workflow row appear to be an agent row and let agent-host routing or cleanup misinterpret its authority. Reject a second workflow permit table: two authorities would need a new cross-table uniqueness and replay protocol. Reject a preflight-only public API: it could be mistaken for dispatch authority while storing no permit or result. Keep general Mastra `run.resume` and automatic recovery fail-closed: the pinned `@mastra/core@1.73.0` does not fence all snapshot write paths by remote attempt generation (see [durable-resume plan](2026-10-12-remote-step-durable-resume-recovery.md)). + +## Reuse and package decision + +Reuse `RemoteStepJournal`, `ExecutionFenceStore`, `assertExecutionPrincipal`, `isPathSafeId`, authoritative runtime status/start state, D1 conditional updates, and Web Crypto SHA-256 in `src/do-runner/remote-step-journal.ts`. The existing direct dependencies are `@mastra/core@1.73.0`, `@mastra/cloudflare-d1@1.4.0`, and `jose@6.2.8`; none provides this host-specific journal schema and CAS contract. A new package adds no useful boundary. A separate custom store duplicates the existing permit protocol. The narrow custom change is a discriminated target row and suspension selector in the existing journal. No dependency changes. + +## Checkpoints + +1. **Schema v4.** Rebuild the journal table transactionally because SQLite cannot remove `NOT NULL` from agent identity columns in place. Add `target_kind`, `suspension_marker`, `request_digest`, and `write_revision`; make agent/thread/resource nullable. Copy v3 rows as `agent`, with their original marker, `NULL` request digest, and revision zero; preserve their status, permit, result, lease, and settlement fields. The agent branch continues to compare the exact stored `request_json`; it does not require a digest on migrated rows. A workflow row requires a digest. Add separate partial unique indexes for agent and workflow suspension identities, plus the recovery index. Validate exact schema/index/trigger definitions and refuse unknown versions. A v4 update increments `write_revision` in the same statement as its state transition. A D1 trigger rejects updates that do not advance it; v3 code omits that column, so an old in-flight isolate cannot mutate a row after migration. `target_kind NOT NULL` rejects its inserts. Test a v3 initialized journal instance attempting both operations after another instance migrates. A v4 binary reads migrated agent rows unchanged. Keep rollback to a v4-aware binary; an old v3 binary fails closed. +2. **Workflow storage port.** Add a workflow-specific registration method, a workflow getter, and typed workflow invocation. Registration accepts a trusted workflow address and original principal, verifies start provenance and exact suspension, then performs a snapshot-guarded insert/update. Existing agent-host `get` and `register` remain agent-only. Shared permit, consume, result and retained-result methods dispatch by target kind inside the same journal authority. No caller is wired yet. +3. **Proof.** Use real SQLite-backed D1 semantics for old-row preservation, conflicting registration, restart/reopen, permit replay, wrong request/principal/start token/fence/deadline, and result conflict. Add a workerd fresh-isolate scenario before claiming a live persistence proof. The default-off package may merge without a customer route; it may not be described as a working WFP dispatch path until the authenticated host route and workerd proof land. + +## Verification + +After each code checkpoint, derive scoped checks from changed paths. The journal schema is a shared contract, so the owning package's full test suite applies once under the quality-gate exception. Candidate commands: + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run src/do-runner/remote-step-journal.test.ts +pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts packages/flowsafe/src/do-runner/remote-step-journal.test.ts +pnpm --filter @proofoftech/flowsafe typecheck:test +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe test +``` + +Each command needs its own `/tmp` log and actual exit status. Done for this checkpoint means schema upgrade preserves v3 rows, workflow register/permit/result behavior passes, agent-host behavior passes, and no public workflow dispatch or resume route is enabled. The top-level quality-gate owner runs T2 clean-code, QA, architecture, and security lanes, then commits. Live enablement additionally requires a real D1/workerd restart and replay proof, authenticated host binding, and ASP grant verification. + +## Appendix: current-state anchors + +At the baseline SHA, `src/do-runner/remote-step-journal.ts` declares an agent-only address and row: + +```ts +export interface RemoteStepAddress { + deploymentId: string; + tablePrefix: string; + workflowId: string; + agentId: string; + threadId: string; + resourceId: string; + runId: string; +} +``` + +The `RemoteStepJournal.initialize` table has `agent_id TEXT NOT NULL`, `thread_id TEXT NOT NULL`, and `resource_id TEXT NOT NULL`, with a unique key containing those columns. Its `readSchema` accepts version 3 or 2 and exact column/index definitions. Before → after: v3 agent-only fields and uniqueness → v4 discriminated target with nullable workflow agent fields and partial target-specific unique indexes. A simple nullable-column alteration is insufficient because the existing unique key cannot represent workflow identity. + +`RemoteStepJournal.#execution` currently refuses workflow starts: + +```ts +if ( + state?.storage !== 'd1' || + target?.kind !== 'agent' || + !isPathSafeId(target.resourceId) +) + throw new RemoteStepRefusal( + 503, + 'remote step run identity is unreadable', + ); +``` + +Before → after: the agent branch retains its exact checks; the workflow branch requires `target.kind === 'workflow'`, `target.id === address.workflowId`, no agent-start provenance, matching original principal, deployment/table prefix/run ID/start token, and the current execution fence. It never fills agent fields from workflow IDs. + +`selectedSuspension` currently matches `toolCallSuspended.remoteStep === REMOTE_STEP_MARKER` and reads `tool.request`. Before → after: keep that agent selector and add a separate workflow selector for the versioned `remoteStep.marker` and `remoteStep.request`, with one selected path and exact request bytes/digest. + +`RemoteStepJournal.issuePermit` and `consumePermit` compare `suspension.requestJson` against `JSON.stringify(invocation.request)` and conditionally update the journal with `request_json = ?`, snapshot equality, and `executionFenceAdmissionSql`. Before → after: retain those CAS predicates and add the target kind and marker. The workflow branch also compares its stored request digest; legacy agent rows continue to use exact request bytes without a digest. Every v4 state update advances `write_revision` under a D1 trigger, which rejects a previously initialized v3 isolate's updates. `recordResult` keeps `permit_digest` and immutable result comparison. No code path retries `run.resume` from a retained result. + +`src/do-runner/execution-admission.ts` defines `StartIdentity.target` as a union with `{kind:'workflow',id}`. `src/do-runner/runtime.ts` already checks that a recovered workflow target ID equals its execution workflow ID. The new journal branch uses that trusted provenance rather than creating another identity format. From 937c12ac16ff8f7f8a08313d6aaa44cf87eb44d3 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:07:56 +0400 Subject: [PATCH 33/73] feat(flowsafe): support private state-egress identity Preserve legacy host headers and add a versioned dispatch-key context with strict mixed-mode refusal and trusted header stamping. --- .../src/host-kit/flowsafe-worker.test.ts | 43 +++++++ .../flowsafe/src/host-kit/flowsafe-worker.ts | 6 + packages/flowsafe/src/host-kit/index.ts | 4 + .../src/host-kit/state-egress.test.ts | 84 ++++++++++++++ .../flowsafe/src/host-kit/state-egress.ts | 108 +++++++++++++++--- 5 files changed, 231 insertions(+), 14 deletions(-) diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts index ae8b8764..cffa6ec9 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts @@ -1636,6 +1636,35 @@ describe('createFlowsafeWorker maintenance duties', () => { expect(outboundFetch).not.toHaveBeenCalled(); }); + it('admits a trusted private state egress context without a route hostname', async () => { + const worker = makeWorker(); + const { env, ctx } = makeEnv(); + const outboundFetch = vi.fn(async () => new Response(null)); + env.FLEET_RESOURCE_ROLE = 'platform-state'; + env.FLEET_ENVIRONMENT = 'production'; + env.FLEET_RESOURCE_GROUP = '0123456789abcdefabcd'; + env.OUTBOUND_PROXY = { fetch: outboundFetch }; + env.OUTBOUND_PROXY_CREDENTIAL = 'outbound-state-credential-0000000001'; + env.OUTBOUND_TENANT_ID = 'acme'; + env.OUTBOUND_ENVIRONMENT = 'production'; + env.OUTBOUND_RESOURCE_GROUP_ID = '0123456789abcdefabcd'; + env.OUTBOUND_STATE_SCRIPT_NAME = + 'acme-production-state-0123456789abcdefabcd'; + env.OUTBOUND_ACTIVATION_VERSION = '2'; + env.OUTBOUND_ACTIVATION_KIND = 'internal-dispatch'; + env.OUTBOUND_DISPATCH_KEY = 'acme:production'; + env.OUTBOUND_POLICY_ID = '0123456789abcdefabcd'; + + const response = await worker.fetch( + new Request('http://host/healthz'), + env, + ctx, + ); + + expect(response.status).toBe(200); + expect(outboundFetch).not.toHaveBeenCalled(); + }); + it('rejects candidate access to trusted state egress before service use', async () => { const worker = makeWorker(); const { env, ctx } = makeEnv(); @@ -1652,6 +1681,20 @@ describe('createFlowsafeWorker maintenance duties', () => { expect(outboundFetch).not.toHaveBeenCalled(); }); + it('rejects a private dispatch key on a candidate Worker', async () => { + const worker = makeWorker(); + const { env, ctx } = makeEnv(); + env.OUTBOUND_DISPATCH_KEY = 'acme:production'; + + const response = await worker.fetch( + new Request('http://host/healthz'), + env, + ctx, + ); + + expect(response.status).toBe(500); + }); + it('the purge duty runs ONLY the purge duties', async () => { // #given const logs = capturedLogs(); diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.ts index 2daff5ed..8fbcc23d 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.ts @@ -306,6 +306,9 @@ export interface FlowsafeWorkerEnv { OUTBOUND_RESOURCE_GROUP_ID?: string; OUTBOUND_STATE_SCRIPT_NAME?: string; OUTBOUND_ROUTE_HOSTNAME?: string; + OUTBOUND_ACTIVATION_VERSION?: string; + OUTBOUND_ACTIVATION_KIND?: string; + OUTBOUND_DISPATCH_KEY?: string; OUTBOUND_POLICY_ID?: string; /** Dedicated control-plane credential for maintenance administration. */ MAINTENANCE_ADMIN_SECRET?: string; @@ -671,6 +674,9 @@ const STATE_EGRESS_ENV_KEYS = [ 'OUTBOUND_RESOURCE_GROUP_ID', 'OUTBOUND_STATE_SCRIPT_NAME', 'OUTBOUND_ROUTE_HOSTNAME', + 'OUTBOUND_ACTIVATION_VERSION', + 'OUTBOUND_ACTIVATION_KIND', + 'OUTBOUND_DISPATCH_KEY', 'OUTBOUND_POLICY_ID', ] as const; diff --git a/packages/flowsafe/src/host-kit/index.ts b/packages/flowsafe/src/host-kit/index.ts index 4dcfa192..9bf1bc96 100644 --- a/packages/flowsafe/src/host-kit/index.ts +++ b/packages/flowsafe/src/host-kit/index.ts @@ -179,10 +179,14 @@ export type { } from './state-egress.js'; export { createStateEgressFetch, + STATE_EGRESS_ACTIVATION_KIND_HEADER, + STATE_EGRESS_ACTIVATION_VERSION_HEADER, STATE_EGRESS_CREDENTIAL_HEADER, + STATE_EGRESS_DISPATCH_KEY_HEADER, STATE_EGRESS_ENVIRONMENT_HEADER, STATE_EGRESS_HEADERS, STATE_EGRESS_POLICY_ID_HEADER, + STATE_EGRESS_PRIVATE_HEADERS, STATE_EGRESS_RESERVED_HEADERS, STATE_EGRESS_RESOURCE_GROUP_HEADER, STATE_EGRESS_ROUTE_HOSTNAME_HEADER, diff --git a/packages/flowsafe/src/host-kit/state-egress.test.ts b/packages/flowsafe/src/host-kit/state-egress.test.ts index 572b0ad3..2a65e9ca 100644 --- a/packages/flowsafe/src/host-kit/state-egress.test.ts +++ b/packages/flowsafe/src/host-kit/state-egress.test.ts @@ -17,6 +17,9 @@ import { const OUTBOUND_CREDENTIAL = 'outbound-state-credential-0000000001'; const DEPLOYMENT_IDENTITY = 'deployment-identity-credential-00001'; +const PRIVATE_VERSION_HEADER = 'x-anchorage-state-activation-version'; +const PRIVATE_KIND_HEADER = 'x-anchorage-state-activation-kind'; +const PRIVATE_KEY_HEADER = 'x-anchorage-state-dispatch-key'; function stateEnv( fetch = vi.fn(async (_request: Request) => new Response('proxied')), @@ -35,6 +38,87 @@ function stateEnv( } describe('trusted state egress adapter', () => { + it('uses a private dispatch key without a host route and removes the spoofed host header', async () => { + const bindingFetch = vi.fn( + async (_request: Request) => new Response('private'), + ); + const { OUTBOUND_ROUTE_HOSTNAME: _route, ...legacy } = + stateEnv(bindingFetch); + const env = { + ...legacy, + OUTBOUND_ACTIVATION_VERSION: '2', + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch', + OUTBOUND_DISPATCH_KEY: 'acme:production', + } as unknown as StateEgressEnv; + + await expect(validateStateEgressEnv(env)).resolves.toMatchObject({ + activationVersion: '2', + activationKind: 'internal-dispatch', + dispatchKey: 'acme:production', + }); + const response = await createStateEgressFetch(env)( + new Request('https://api.vendor.test/write', { + headers: { + [STATE_EGRESS_ROUTE_HOSTNAME_HEADER]: 'victim.example.test', + [PRIVATE_KEY_HEADER]: 'victim:production', + }, + }), + ); + expect(await response.text()).toBe('private'); + const forwarded = bindingFetch.mock.calls[0]?.[0]; + expect( + forwarded?.headers.get(STATE_EGRESS_ROUTE_HOSTNAME_HEADER), + ).toBeNull(); + expect(forwarded?.headers.get(PRIVATE_VERSION_HEADER)).toBe('2'); + expect(forwarded?.headers.get(PRIVATE_KIND_HEADER)).toBe( + 'internal-dispatch', + ); + expect(forwarded?.headers.get(PRIVATE_KEY_HEADER)).toBe('acme:production'); + }); + + it('rejects mixed host and private activation bindings', async () => { + const env = { + ...stateEnv(), + OUTBOUND_ACTIVATION_VERSION: '2', + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch', + OUTBOUND_DISPATCH_KEY: 'acme:production', + } as StateEgressEnv; + await expect(validateStateEgressEnv(env)).rejects.toThrow(); + }); + + it.each([ + ['unknown version', { OUTBOUND_ACTIVATION_VERSION: '3' }], + ['unknown kind', { OUTBOUND_ACTIVATION_KIND: 'host' }], + ['foreign key', { OUTBOUND_DISPATCH_KEY: 'other:production' }], + ['host value', { OUTBOUND_ROUTE_HOSTNAME: 'acme.example.test' }], + ])('rejects a private activation with %s', async (_case, patch) => { + const { OUTBOUND_ROUTE_HOSTNAME: _route, ...legacy } = stateEnv(); + const env = { + ...legacy, + OUTBOUND_ACTIVATION_VERSION: '2', + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch', + OUTBOUND_DISPATCH_KEY: 'acme:production', + ...patch, + } as unknown as StateEgressEnv; + await expect(validateStateEgressEnv(env)).rejects.toThrow(); + }); + + it('strips private authority headers from a legacy host request', async () => { + const bindingFetch = vi.fn(async (_request: Request) => new Response()); + await createStateEgressFetch(stateEnv(bindingFetch))( + new Request('https://api.vendor.test/write', { + headers: { + [PRIVATE_VERSION_HEADER]: '2', + [PRIVATE_KIND_HEADER]: 'internal-dispatch', + [PRIVATE_KEY_HEADER]: 'victim:production', + }, + }), + ); + const forwarded = bindingFetch.mock.calls[0]?.[0]; + expect(forwarded?.headers.get(PRIVATE_VERSION_HEADER)).toBeNull(); + expect(forwarded?.headers.get(PRIVATE_KIND_HEADER)).toBeNull(); + expect(forwarded?.headers.get(PRIVATE_KEY_HEADER)).toBeNull(); + }); it('matches a presented credential against its canonical SHA-256 digest', async () => { const digest = await crypto.subtle.digest( 'SHA-256', diff --git a/packages/flowsafe/src/host-kit/state-egress.ts b/packages/flowsafe/src/host-kit/state-egress.ts index 02d73e6d..846ddcea 100644 --- a/packages/flowsafe/src/host-kit/state-egress.ts +++ b/packages/flowsafe/src/host-kit/state-egress.ts @@ -16,6 +16,12 @@ export const STATE_EGRESS_RESOURCE_GROUP_HEADER = export const STATE_EGRESS_SCRIPT_HEADER = 'x-anchorage-state-script' as const; export const STATE_EGRESS_ROUTE_HOSTNAME_HEADER = 'x-anchorage-state-route-hostname' as const; +export const STATE_EGRESS_ACTIVATION_VERSION_HEADER = + 'x-anchorage-state-activation-version' as const; +export const STATE_EGRESS_ACTIVATION_KIND_HEADER = + 'x-anchorage-state-activation-kind' as const; +export const STATE_EGRESS_DISPATCH_KEY_HEADER = + 'x-anchorage-state-dispatch-key' as const; export const STATE_EGRESS_POLICY_ID_HEADER = 'x-anchorage-state-policy-id' as const; @@ -29,36 +35,74 @@ export const STATE_EGRESS_HEADERS = { policyId: STATE_EGRESS_POLICY_ID_HEADER, } as const; -export const STATE_EGRESS_RESERVED_HEADERS = - Object.values(STATE_EGRESS_HEADERS); +export const STATE_EGRESS_PRIVATE_HEADERS = { + activationVersion: STATE_EGRESS_ACTIVATION_VERSION_HEADER, + activationKind: STATE_EGRESS_ACTIVATION_KIND_HEADER, + dispatchKey: STATE_EGRESS_DISPATCH_KEY_HEADER, +} as const; + +export const STATE_EGRESS_RESERVED_HEADERS = [ + ...Object.values(STATE_EGRESS_HEADERS), + ...Object.values(STATE_EGRESS_PRIVATE_HEADERS), +]; export interface StateEgressBinding { fetch(request: Request): Promise; } -export interface StateEgressContext { +interface StateEgressCommonContext { readonly credential: string; readonly tenantTag: string; readonly environment: string; readonly resourceGroupId: string; readonly stateScriptName: string; - readonly routeHostname: string; readonly policyId: string; } -/** Exact trusted-state bindings consumed by createStateEgressFetch. */ -export interface StateEgressEnv { +export type StateEgressContext = StateEgressCommonContext & + ( + | Readonly<{ + routeHostname: string; + activationVersion?: never; + activationKind?: never; + dispatchKey?: never; + }> + | Readonly<{ + routeHostname?: never; + activationVersion: '2'; + activationKind: 'internal-dispatch'; + dispatchKey: string; + }> + ); + +interface StateEgressCommonEnv { readonly OUTBOUND_PROXY: StateEgressBinding; readonly OUTBOUND_PROXY_CREDENTIAL: string; readonly OUTBOUND_TENANT_ID: string; readonly OUTBOUND_ENVIRONMENT: string; readonly OUTBOUND_RESOURCE_GROUP_ID: string; readonly OUTBOUND_STATE_SCRIPT_NAME: string; - readonly OUTBOUND_ROUTE_HOSTNAME: string; readonly OUTBOUND_POLICY_ID: string; readonly DEPLOYMENT_IDENTITY_SECRET: string; } +/** Exact trusted-state bindings consumed by createStateEgressFetch. */ +export type StateEgressEnv = StateEgressCommonEnv & + ( + | Readonly<{ + OUTBOUND_ROUTE_HOSTNAME: string; + OUTBOUND_ACTIVATION_VERSION?: never; + OUTBOUND_ACTIVATION_KIND?: never; + OUTBOUND_DISPATCH_KEY?: never; + }> + | Readonly<{ + OUTBOUND_ROUTE_HOSTNAME?: never; + OUTBOUND_ACTIVATION_VERSION: '2'; + OUTBOUND_ACTIVATION_KIND: 'internal-dispatch'; + OUTBOUND_DISPATCH_KEY: string; + }> + ); + const CREDENTIAL_PATTERN = /^[\x21-\x7e]{32,256}$/; const RESOURCE_GROUP_PATTERN = /^[a-f0-9]{20}$/; const SCRIPT_NAME_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/; @@ -122,22 +166,48 @@ function stateEgressContext(env: StateEgressEnv): StateEgressContext { if (!SCRIPT_NAME_PATTERN.test(env.OUTBOUND_STATE_SCRIPT_NAME)) { throw new Error('trusted state outbound script is invalid'); } - const routeHostname = canonicalRouteHostname(env.OUTBOUND_ROUTE_HOSTNAME); - if (!routeHostname) { - throw new Error('trusted state outbound route hostname is invalid'); - } if (!POLICY_ID_PATTERN.test(env.OUTBOUND_POLICY_ID)) { throw new Error('trusted state outbound policy is invalid'); } - return { + const common = { credential: env.OUTBOUND_PROXY_CREDENTIAL, tenantTag: env.OUTBOUND_TENANT_ID, environment: env.OUTBOUND_ENVIRONMENT, resourceGroupId: env.OUTBOUND_RESOURCE_GROUP_ID, stateScriptName: env.OUTBOUND_STATE_SCRIPT_NAME, - routeHostname, policyId: env.OUTBOUND_POLICY_ID, }; + if (env.OUTBOUND_ACTIVATION_VERSION === undefined) { + if ( + env.OUTBOUND_ACTIVATION_KIND !== undefined || + env.OUTBOUND_DISPATCH_KEY !== undefined + ) { + throw new Error('trusted state outbound activation is mixed'); + } + const routeHostname = + typeof env.OUTBOUND_ROUTE_HOSTNAME === 'string' + ? canonicalRouteHostname(env.OUTBOUND_ROUTE_HOSTNAME) + : undefined; + if (!routeHostname) { + throw new Error('trusted state outbound route hostname is invalid'); + } + return { ...common, routeHostname }; + } + if ( + env.OUTBOUND_ACTIVATION_VERSION !== '2' || + env.OUTBOUND_ACTIVATION_KIND !== 'internal-dispatch' || + env.OUTBOUND_ROUTE_HOSTNAME !== undefined || + env.OUTBOUND_DISPATCH_KEY !== + `${env.OUTBOUND_TENANT_ID}:${env.OUTBOUND_ENVIRONMENT}` + ) { + throw new Error('trusted state outbound private activation is invalid'); + } + return { + ...common, + activationVersion: '2', + activationKind: 'internal-dispatch', + dispatchKey: env.OUTBOUND_DISPATCH_KEY, + }; } /** Validate the complete trusted-state egress binding group. */ @@ -171,12 +241,22 @@ export function createStateEgressFetch(env: StateEgressEnv): typeof fetch { } const request = new Request(input, init); const headers = new Headers(request.headers); + for (const name of STATE_EGRESS_RESERVED_HEADERS) headers.delete(name); headers.set(STATE_EGRESS_CREDENTIAL_HEADER, context.credential); headers.set(STATE_EGRESS_TENANT_HEADER, context.tenantTag); headers.set(STATE_EGRESS_ENVIRONMENT_HEADER, context.environment); headers.set(STATE_EGRESS_RESOURCE_GROUP_HEADER, context.resourceGroupId); headers.set(STATE_EGRESS_SCRIPT_HEADER, context.stateScriptName); - headers.set(STATE_EGRESS_ROUTE_HOSTNAME_HEADER, context.routeHostname); + if (context.activationVersion === '2') { + headers.set( + STATE_EGRESS_ACTIVATION_VERSION_HEADER, + context.activationVersion, + ); + headers.set(STATE_EGRESS_ACTIVATION_KIND_HEADER, context.activationKind); + headers.set(STATE_EGRESS_DISPATCH_KEY_HEADER, context.dispatchKey); + } else { + headers.set(STATE_EGRESS_ROUTE_HOSTNAME_HEADER, context.routeHostname); + } headers.set(STATE_EGRESS_POLICY_ID_HEADER, context.policyId); return service.fetch(new Request(request, { headers })); }; From ed9bfad3c20f70a90e120e5f8376e2fac3f448a7 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:37:32 +0400 Subject: [PATCH 34/73] Add versioned workflow remote-step journal Persist workflow suspension bindings and request digests with migration and writer fencing. Keep agent recovery scans separate while verifying workflow permit consumption and process restart in the workerd spike. --- packages/flowsafe/scripts/spike-verify.mjs | 105 ++- packages/flowsafe/spike/worker.ts | 118 ++- .../src/agent-host/thread-host.test.ts | 4 +- .../src/do-runner/remote-step-journal.test.ts | 323 +++++++- .../src/do-runner/remote-step-journal.ts | 752 +++++++++++++++--- 5 files changed, 1157 insertions(+), 145 deletions(-) diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index 30e7377d..77a73b45 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -176,7 +176,8 @@ function startServer(generation, stateDir, logPath) { // with, so a valid webhook is ACCEPTED and a forged one REJECTED (E-S2). '--var', `GITHUB_WEBHOOK_SECRET:${GITHUB_WEBHOOK_SECRET}`, - ...(generation.startsWith('gen-remote-') + ...(generation.startsWith('gen-remote-') || + generation.startsWith('gen-workflow-remote-') ? ['--var', 'SPIKE_REMOTE_STEP_PROOF:enabled'] : []), ], @@ -3588,6 +3589,108 @@ async function main() { }, ); + await step( + 'RS2 workflow-target journal: D1 permit and retained result survive process death without resume', + async () => { + await killServer(currentServer); + await executeLocalD1( + stateDir, + "INSERT INTO spike_remote_mastra_workflow_snapshot VALUES ('durable-agentic-loop', 'spike-workflow-run', '{\"status\":\"suspended\"}');", + ); + await launchServer( + 'gen-workflow-remote-1', + stateDir, + join(tmpDir, 'gen-workflow-remote-1.log'), + ); + const registered = await http( + 'POST', + '/spike/remote-step/workflow-register', + { + body: {}, + }, + ); + assert( + registered.status === 200 && + registered.body.targetKind === 'workflow' && + typeof registered.body.invocationId === 'string', + 'trusted workflow provenance registered a workflow row', + registered, + ); + const invocationId = registered.body.invocationId; + const issued = await http('POST', '/spike/remote-step/workflow-issue', { + body: { invocationId }, + }); + assert( + issued.status === 200 && issued.body.generation === 1, + 'workflow dispatch permit was issued', + issued, + ); + const attempts = await Promise.all( + [0, 1].map(() => + http('POST', '/spike/remote-step/workflow-consume', { + body: { permit: issued.body }, + }), + ), + ); + assert( + attempts.filter((attempt) => attempt.body.kind === 'accepted') + .length === 1 && + attempts.filter((attempt) => attempt.body.kind === 'already-consumed') + .length === 1, + 'workflow permit consumption has one winner', + attempts, + ); + const rawPermit = attempts.find( + (attempt) => attempt.body.kind === 'accepted', + ).body.rawPermit; + const recorded = await http( + 'POST', + '/spike/remote-step/workflow-record', + { + body: { invocationId, rawPermit }, + }, + ); + assert( + recorded.status === 200 && recorded.body.kind === 'recorded', + 'workflow result was retained', + recorded, + ); + await killServer(currentServer); + await launchServer( + 'gen-workflow-remote-2', + stateDir, + join(tmpDir, 'gen-workflow-remote-2.log'), + ); + const retained = await http( + 'GET', + `/spike/remote-step/workflow-read?invocationId=${invocationId}`, + ); + assert( + retained.status === 200 && + retained.body.invocation.status === 'result-retained' && + retained.body.result.resultJson === '{"answer":1}', + 'fresh workerd process reads the workflow result', + retained, + ); + const replay = await http('POST', '/spike/remote-step/workflow-consume', { + body: { permit: issued.body }, + }); + assert( + replay.status === 200 && replay.body.kind === 'already-consumed', + 'fresh process does not grant workflow effect authority twice', + replay, + ); + const claim = await http('POST', '/spike/remote-step/workflow-claim', { + body: { invocationId }, + }); + assert( + claim.status === 404, + 'workflow continuation remains unavailable', + claim, + ); + }, + ); + await step( 'S deployment sentinel mismatch: a freshly started Worker refuses a D1 ' + 'provisioned for another deployment', diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index 2646fa48..8ef13dfa 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -155,6 +155,7 @@ import { REMOTE_STEP_MARKER, RemoteStepJournal, type RemoteStepPermit, + WORKFLOW_REMOTE_STEP_MARKER, } from '../src/do-runner/remote-step-journal.js'; import { createObjectiveRouter, @@ -3091,9 +3092,11 @@ async function handleRemoteStepProbe( if (env.SPIKE_REMOTE_STEP_PROOF !== 'enabled') return json({ error: 'not found' }, 404); const operation = path.slice('/spike/remote-step/'.length); + const workflow = operation.startsWith('workflow-'); + const action = workflow ? operation.slice('workflow-'.length) : operation; + const runId = workflow ? 'spike-workflow-run' : 'spike-remote-run'; if ( - request.method !== - (operation === 'read' || operation === 'scan' ? 'GET' : 'POST') + request.method !== (action === 'read' || action === 'scan' ? 'GET' : 'POST') ) return json({ error: 'method not allowed' }, 405); try { @@ -3103,7 +3106,7 @@ async function handleRemoteStepProbe( const snapshot = await env.DB.prepare( 'SELECT snapshot FROM spike_remote_mastra_workflow_snapshot WHERE workflow_name = ? AND run_id = ?', ) - .bind('durable-agentic-loop', 'spike-remote-run') + .bind('durable-agentic-loop', runId) .first<{ snapshot: string }>(); if (!fixture || !snapshot) throw new Error('remote-step fixture is unavailable'); @@ -3113,27 +3116,43 @@ async function handleRemoteStepProbe( const summary: RunSummary = persisted.status === 'success' ? { - runId: 'spike-remote-run', + runId, status: 'success', result: { answer: 1 }, } - : { - runId: 'spike-remote-run', - status: 'suspended', - suspended: [['durable-agentic-execution']], - suspendedAt: { 'durable-agentic-execution': 1000 }, - suspendPayload: { - 'durable-agentic-execution': { - type: 'suspension', - toolName: 'remote_lookup', - toolCallId: 'call-1', - toolCallSuspended: { - remoteStep: REMOTE_STEP_MARKER, - request: { account: 'acme' }, + : workflow + ? { + runId, + status: 'suspended', + suspended: [['execute']], + suspendedAt: { execute: 1000 }, + suspendPayload: { + execute: { + type: 'suspension', + remoteStep: { + marker: WORKFLOW_REMOTE_STEP_MARKER, + request: { version: 2, input: { query: 'x' } }, + }, }, }, - }, - }; + } + : { + runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 1000 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }; const runtime = { authoritativeStatus: async () => summary, authoritativeStartState: async () => ({ @@ -3142,18 +3161,20 @@ async function handleRemoteStepProbe( execution: { tablePrefix: 'spike_remote_', workflowId: 'durable-agentic-loop', - runId: 'spike-remote-run', + runId, startToken: 'spike-start-token', }, provenance: { startIdentity: { owner: { kind: 'human', id: 'operator-1' }, - target: { - kind: 'agent', - id: 'writer', - threadId: 'spike-remote-thread', - resourceId: fixture.resource_id, - }, + target: workflow + ? { kind: 'workflow', id: 'durable-agentic-loop' } + : { + kind: 'agent', + id: 'writer', + threadId: 'spike-remote-thread', + resourceId: fixture.resource_id, + }, }, }, }), @@ -3169,6 +3190,7 @@ async function handleRemoteStepProbe( tablePrefix: 'spike_remote_', allowedToolIds: ['remote_lookup'], verifyCompletion: async () => undefined, + ...(workflow ? { verifyWorkflowCompletion: async () => undefined } : {}), }); await journal.initialize(); const body = @@ -3179,22 +3201,34 @@ async function handleRemoteStepProbe( request.method === 'GET' ? (new URL(request.url).searchParams.get('invocationId') ?? '') : String(body.invocationId ?? ''); - switch (operation) { + switch (action) { case 'register': return json( - await journal.register( - { - deploymentId: 'spike', - tablePrefix: 'spike_remote_', - workflowId: 'durable-agentic-loop', - agentId: 'writer', - threadId: 'spike-remote-thread', - resourceId: String(body.resourceId ?? ''), - runId: 'spike-remote-run', - }, - Date.now() + 120_000, - { kind: 'human', id: 'operator-1', role: 'operator' }, - ), + await (workflow + ? journal.registerWorkflow( + { + targetKind: 'workflow', + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + runId, + }, + Date.now() + 120_000, + { kind: 'human', id: 'operator-1', role: 'operator' }, + ) + : journal.register( + { + deploymentId: 'spike', + tablePrefix: 'spike_remote_', + workflowId: 'durable-agentic-loop', + agentId: 'writer', + threadId: 'spike-remote-thread', + resourceId: String(body.resourceId ?? ''), + runId, + }, + Date.now() + 120_000, + { kind: 'human', id: 'operator-1', role: 'operator' }, + )), ); case 'issue': return json(await journal.issuePermit(invocationId, 60_000)); @@ -3213,7 +3247,9 @@ async function handleRemoteStepProbe( ); case 'read': return json({ - invocation: await journal.get(invocationId), + invocation: workflow + ? await journal.getWorkflow(invocationId) + : await journal.get(invocationId), result: await journal.readRetainedResult(invocationId), }); case 'scan': diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 49f825e3..13e73650 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -7925,9 +7925,9 @@ it('resumes a retained mapped-resource result from D1 after the agent host is re expect(leaseBefore.results).toMatchObject([ { resume_lease_owner: expect.any(String) }, ]); - fixture.binding + await fixture.binding .prepare( - 'UPDATE flowsafe_remote_step_invocations SET resume_lease_expires_at = ? WHERE invocation_id = ?', + 'UPDATE flowsafe_remote_step_invocations SET resume_lease_expires_at = ?, write_revision = write_revision + 1 WHERE invocation_id = ?', ) .bind(Date.now() - 1, invocation.invocationId) .run(); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 66b8e5a5..7dbb6b62 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -17,6 +17,7 @@ import { REMOTE_STEP_MARKER, REMOTE_STEP_TABLE, RemoteStepJournal, + WORKFLOW_REMOTE_STEP_MARKER, } from './remote-step-journal.js'; import type { RunnerRuntime, RunSummary } from './runtime.js'; @@ -181,6 +182,105 @@ describe('remote step journal dispatch admission', () => { }; } + function restoreV3Schema(h: Awaited>) { + const columns = ( + h.sqlite + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all() as Array<{ name: string }> + ) + .slice(0, -4) + .map(({ name }) => name) + .join(', '); + h.sqlite.exec(`CREATE TABLE ${REMOTE_STEP_TABLE}_v3 ( + invocation_id TEXT PRIMARY KEY, principal_json TEXT NOT NULL, + start_token TEXT NOT NULL, deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, workflow_id TEXT NOT NULL, + agent_id TEXT NOT NULL, thread_id TEXT NOT NULL, + resource_id TEXT NOT NULL, run_id TEXT NOT NULL, + step_path TEXT NOT NULL, suspended_at INTEGER NOT NULL, + resume_count INTEGER NOT NULL, request_json TEXT NOT NULL, + deadline_at INTEGER NOT NULL, status TEXT NOT NULL, + permit_generation INTEGER NOT NULL DEFAULT 0, permit_epoch INTEGER, + permit_revision INTEGER, permit_expires_at INTEGER, + permit_consumed_at INTEGER, permit_digest TEXT, result_json TEXT, + result_digest TEXT, quiescence_digest TEXT, executor_quiescent_at INTEGER, + resume_lease_owner TEXT, resume_lease_expires_at INTEGER, + settled_at INTEGER, settlement_outcome TEXT, + UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, + thread_id, resource_id, run_id, step_path, suspended_at, resume_count) + )`); + h.sqlite.exec( + `INSERT INTO ${REMOTE_STEP_TABLE}_v3 (${columns}) SELECT ${columns} FROM ${REMOTE_STEP_TABLE}`, + ); + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec( + `ALTER TABLE ${REMOTE_STEP_TABLE}_v3 RENAME TO ${REMOTE_STEP_TABLE}`, + ); + h.sqlite.exec( + `CREATE INDEX flowsafe_remote_step_recovery ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE status IN ('result-retained', 'resuming')`, + ); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 3'); + } + + async function workflowFixture() { + const h = await fixture(); + const original = h.runtime.authoritativeStartState.bind(h.runtime); + h.runtime.authoritativeStartState = (async ( + workflowId: string, + runId: string, + ) => { + const state = await original(workflowId, runId); + if (!state) return state; + return { + ...state, + provenance: { + startIdentity: { + owner: { kind: PRINCIPAL.kind, id: PRINCIPAL.id }, + target: { kind: 'workflow', id: ADDRESS.workflowId }, + }, + }, + } as typeof state; + }) as typeof h.runtime.authoritativeStartState; + const request = { version: 2, input: { query: 'x' } }; + const summary = (value: unknown): RunSummary => ({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['execute']], + suspendedAt: { execute: 500 }, + suspendPayload: { + execute: { + type: 'suspension', + remoteStep: { + marker: WORKFLOW_REMOTE_STEP_MARKER, + request: value, + }, + }, + }, + }); + h.setSummary(summary(request)); + const verifyWorkflowCompletion = vi.fn(async () => undefined); + const journal = () => + new RemoteStepJournal({ + db: h.db, + fence: h.fence, + runtime: h.runtime, + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + allowedToolIds: ['remote_lookup'], + verifyCompletion: h.verifyCompletion, + verifyWorkflowCompletion, + now: () => 1_000, + }); + const address = { + targetKind: 'workflow', + deploymentId: ADDRESS.deploymentId, + tablePrefix: ADDRESS.tablePrefix, + workflowId: ADDRESS.workflowId, + runId: ADDRESS.runId, + } as const; + return { ...h, address, journal, summary, verifyWorkflowCompletion }; + } + it('converges repeated observation on one invocation without a permit', async () => { const h = await fixture(); const [first, replay] = await Promise.all([ @@ -474,6 +574,7 @@ describe('remote step journal dispatch admission', () => { it('upgrades a prior journal while preserving an existing invocation', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + restoreV3Schema(h); h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); for (const column of [ 'settlement_outcome', @@ -514,6 +615,7 @@ describe('remote step journal dispatch admission', () => { '{"value":"kept"}', '{"receipt":"verified"}', ); + restoreV3Schema(h); h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); for (const column of [ 'settlement_outcome', @@ -533,16 +635,37 @@ describe('remote step journal dispatch admission', () => { }); expect( h.sqlite.prepare('SELECT version FROM flowsafe_remote_step_schema').get(), - ).toEqual({ version: 3 }); + ).toEqual({ version: 4 }); expect(await h.reopen().recoverableResults(1)).toEqual({ invocationIds: [invocation.invocationId], nextCursor: invocation.invocationId, }); }); + it('preserves a live v3 resume lease while upgrading the journal', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const owner = await h.journal.claimResume(invocation.invocationId, 1_000); + restoreV3Schema(h); + const reopened = h.reopen(); + await reopened.initialize(); + expect(await reopened.get(invocation.invocationId)).toMatchObject({ + status: 'resuming', + resumeLeaseExpiresAt: 2_000, + }); + expect( + await reopened.readRetainedResult(invocation.invocationId), + ).toMatchObject({ + resultJson: '{"value":"done"}', + }); + expect( + await reopened.settleAfterResume(invocation.invocationId, owner), + ).toBe('retry'); + }); + it('refuses an unknown journal schema version before dispatch', async () => { const h = await fixture(); - h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 4'); + h.sqlite.exec('UPDATE flowsafe_remote_step_schema SET version = 99'); await expect(h.reopen().initialize()).rejects.toMatchObject({ status: 503, @@ -565,6 +688,7 @@ describe('remote step journal dispatch admission', () => { it('refuses an unversioned journal without its suspension uniqueness constraint', async () => { const h = await fixture(); + restoreV3Schema(h); const ddl = ( h.sqlite .prepare( @@ -604,9 +728,32 @@ describe('remote step journal dispatch admission', () => { }); }); + it('refuses an unversioned workflow journal with a changed target constraint', async () => { + const h = await fixture(); + const ddl = ( + h.sqlite + .prepare( + `SELECT sql FROM sqlite_schema WHERE name = '${REMOTE_STEP_TABLE}'`, + ) + .get() as { sql: string } + ).sql; + h.sqlite.exec(`DROP TABLE ${REMOTE_STEP_TABLE}`); + h.sqlite.exec('DROP TABLE flowsafe_remote_step_schema'); + h.sqlite.exec( + ddl.replace( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + "CHECK ((target_kind = 'agent' AND agent_id IS NULL", + ), + ); + await expect(h.reopen().initialize()).rejects.toMatchObject({ + status: 503, + }); + }); + it('converges when another initializer wins the same journal upgrade', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + restoreV3Schema(h); h.sqlite.exec('DROP INDEX flowsafe_remote_step_recovery'); for (const column of [ 'settlement_outcome', @@ -861,7 +1008,9 @@ describe('remote step journal dispatch admission', () => { await h.journal.readRetainedResult(invocation.invocationId), ).toBeUndefined(); h.sqlite - .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET result_digest = NULL`) + .prepare( + `UPDATE ${REMOTE_STEP_TABLE} SET result_digest = NULL, write_revision = write_revision + 1`, + ) .run(); expect( (await new DeploymentInventory(h.db).read('remote-steps')).count, @@ -1188,7 +1337,9 @@ describe('remote step journal dispatch admission', () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); h.sqlite - .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?`) + .prepare( + `UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?, write_revision = write_revision + 1`, + ) .run('bad"_prefix'); await expect( h.journal.issuePermit(invocation.invocationId, 1_000), @@ -1273,6 +1424,170 @@ describe('remote step journal dispatch admission', () => { }); }); + it('retains a workflow result across journal reopen and rejects permit replay', async () => { + const h = await workflowFixture(); + const journal = h.journal(); + const first = await journal.registerWorkflow(h.address, 10_000, PRINCIPAL); + const replay = await journal.registerWorkflow(h.address, 10_000, PRINCIPAL); + expect(replay.invocationId).toBe(first.invocationId); + expect(first.requestDigest).toBe( + 'f908a6c18fa0b52b0f1295fc8f1cb5db083020cb18dab24ebf26637f55a9af49', + ); + expect(await journal.get(first.invocationId)).toBeUndefined(); + expect(await journal.getWorkflow(first.invocationId)).toMatchObject({ + targetKind: 'workflow', + status: 'pending', + stepPath: ['execute'], + }); + await expect( + h.reopen().issuePermit(first.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 503, + }); + const permit = await journal.issuePermit(first.invocationId, 1_000); + const winner = await journal.consumePermit(permit); + if (winner.kind !== 'accepted') throw new Error('permit was not won'); + expect(await h.journal().consumePermit(permit)).toEqual({ + kind: 'already-consumed', + }); + await expect( + journal.recordResult( + first.invocationId, + 'A'.repeat(43), + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).rejects.toMatchObject({ status: 403 }); + expect(h.verifyWorkflowCompletion).not.toHaveBeenCalled(); + expect( + await journal.recordResult( + first.invocationId, + winner.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'recorded' }); + expect(h.verifyWorkflowCompletion).toHaveBeenCalledExactlyOnceWith({ + invocation: expect.objectContaining({ + invocationId: first.invocationId, + targetKind: 'workflow', + requestDigest: first.requestDigest, + status: 'dispatched', + }), + rawPermit: winner.rawPermit, + resultJson: '{"value":"done"}', + evidenceJson: '{"receipt":"verified"}', + }); + const reopened = h.journal(); + await reopened.initialize(); + expect(await reopened.readRetainedResult(first.invocationId)).toMatchObject( + { + resultJson: '{"value":"done"}', + }, + ); + expect(await reopened.recoverableResults(10)).toEqual({ + invocationIds: [], + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).entries, + ).toEqual([expect.objectContaining({ key: [first.invocationId] })]); + expect( + await reopened.recordResult( + first.invocationId, + winner.rawPermit, + '{"value":"done"}', + '{"receipt":"verified"}', + ), + ).toMatchObject({ kind: 'replay' }); + await expect( + reopened.recordResult( + first.invocationId, + winner.rawPermit, + '{"value":"changed"}', + '{"receipt":"verified"}', + ), + ).rejects.toMatchObject({ status: 409 }); + expect(h.verifyWorkflowCompletion).toHaveBeenCalledTimes(1); + await expect( + reopened.claimResume(first.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 404, + }); + }); + + it('refuses workflow dispatch when trusted provenance or suspension changes', async () => { + const h = await workflowFixture(); + const journal = h.journal(); + await expect( + journal.registerWorkflow(h.address, 10_000, { + ...PRINCIPAL, + id: 'another-operator', + }), + ).rejects.toMatchObject({ status: 409 }); + const first = await journal.registerWorkflow(h.address, 10_000, PRINCIPAL); + h.setSummary(h.summary({ version: 2, input: { query: 'changed' } })); + await expect( + journal.issuePermit(first.invocationId, 1_000), + ).rejects.toMatchObject({ + status: 409, + }); + h.setSummary(h.summary({ version: 2, input: { query: 'x' } })); + const permit = await journal.issuePermit(first.invocationId, 1_000); + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + await expect(journal.consumePermit(permit)).rejects.toMatchObject({ + status: 503, + }); + }); + + it('rejects stale v3-style journal writes after v4 migration', async () => { + const h = await fixture(); + const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + restoreV3Schema(h); + await h.reopen().initialize(); + expect(() => + h.sqlite + .prepare( + `UPDATE ${REMOTE_STEP_TABLE} SET deadline_at = ? WHERE invocation_id = ?`, + ) + .run(9_000, invocation.invocationId), + ).toThrow('remote step writer version changed'); + expect(() => + h.sqlite + .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( + invocation_id, principal_json, start_token, deployment_id, + table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, + step_path, suspended_at, resume_count, request_json, deadline_at, status) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending')`) + .run( + 'old-writer', + JSON.stringify(PRINCIPAL), + 'start-token', + ADDRESS.deploymentId, + ADDRESS.tablePrefix, + ADDRESS.workflowId, + ADDRESS.agentId, + ADDRESS.threadId, + ADDRESS.resourceId, + ADDRESS.runId, + '["execute"]', + 500, + -1, + '{}', + 10_000, + ), + ).toThrow(); + expect(await h.reopen().get(invocation.invocationId)).toMatchObject({ + deadlineAt: 10_000, + }); + }); + it('refuses a real suspended snapshot after runtime persists cancellation intent', async () => { const sqlite = openSqlite() as ReturnType & { close(): void; diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index c1f678be..e3030144 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -27,6 +27,12 @@ export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; export const REMOTE_STEP_SCHEMA_TABLE = 'flowsafe_remote_step_schema'; const REMOTE_STEP_RECOVERY_INDEX = 'flowsafe_remote_step_recovery'; const REMOTE_STEP_RECOVERY_INDEX_SQL = `CREATE INDEX ${REMOTE_STEP_RECOVERY_INDEX} ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE status IN ('result-retained', 'resuming')`; +const REMOTE_STEP_AGENT_INDEX = 'flowsafe_remote_step_agent_suspension'; +const REMOTE_STEP_WORKFLOW_INDEX = 'flowsafe_remote_step_workflow_suspension'; +const REMOTE_STEP_AGENT_INDEX_SQL = `CREATE UNIQUE INDEX ${REMOTE_STEP_AGENT_INDEX} ON ${REMOTE_STEP_TABLE} (deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count) WHERE target_kind = 'agent'`; +const REMOTE_STEP_WORKFLOW_INDEX_SQL = `CREATE UNIQUE INDEX ${REMOTE_STEP_WORKFLOW_INDEX} ON ${REMOTE_STEP_TABLE} (deployment_id, table_prefix, workflow_id, run_id, step_path, suspended_at, resume_count) WHERE target_kind = 'workflow'`; +const REMOTE_STEP_WRITE_TRIGGER = 'flowsafe_remote_step_write_version'; +const REMOTE_STEP_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; const REMOTE_STEP_RESULT_COLUMNS = [ 'result_json', 'result_digest', @@ -64,6 +70,10 @@ const REMOTE_STEP_COLUMNS = [ 'permit_digest', ...REMOTE_STEP_RESULT_COLUMNS, ...REMOTE_STEP_SETTLEMENT_COLUMNS, + 'target_kind', + 'suspension_marker', + 'request_digest', + 'write_revision', ] as const; const REMOTE_STEP_INTEGER_COLUMNS = new Set([ 'suspended_at', @@ -77,14 +87,62 @@ const REMOTE_STEP_INTEGER_COLUMNS = new Set([ 'executor_quiescent_at', 'resume_lease_expires_at', 'settled_at', + 'write_revision', ]); export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; +export const WORKFLOW_REMOTE_STEP_MARKER = 'flowsafe.remote-step.workflow/v1'; export const MAX_REMOTE_STEP_REQUEST_BYTES = 16_384; export const MAX_REMOTE_STEP_RESULT_BYTES = 16_384; const SHA256_HEX = /^[a-f0-9]{64}$/; const RAW_PERMIT = /^[A-Za-z0-9_-]{43}$/; +function remoteStepTableSql(name: string): string { + return `CREATE TABLE IF NOT EXISTS ${name} ( + invocation_id TEXT PRIMARY KEY, + principal_json TEXT NOT NULL, + start_token TEXT NOT NULL, + deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, + workflow_id TEXT NOT NULL, + agent_id TEXT, + thread_id TEXT, + resource_id TEXT, + run_id TEXT NOT NULL, + step_path TEXT NOT NULL, + suspended_at INTEGER NOT NULL, + resume_count INTEGER NOT NULL, + request_json TEXT NOT NULL, + deadline_at INTEGER NOT NULL, + status TEXT NOT NULL, + permit_generation INTEGER NOT NULL DEFAULT 0, + permit_epoch INTEGER, + permit_revision INTEGER, + permit_expires_at INTEGER, + permit_consumed_at INTEGER, + permit_digest TEXT, + result_json TEXT, + result_digest TEXT, + quiescence_digest TEXT, + executor_quiescent_at INTEGER, + resume_lease_owner TEXT, + resume_lease_expires_at INTEGER, + settled_at INTEGER, + settlement_outcome TEXT, + target_kind TEXT NOT NULL CHECK (target_kind IN ('agent', 'workflow')), + suspension_marker TEXT NOT NULL, + request_digest TEXT, + write_revision INTEGER NOT NULL DEFAULT 0, + CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL + AND thread_id IS NOT NULL AND resource_id IS NOT NULL + AND suspension_marker = '${REMOTE_STEP_MARKER}') OR + (target_kind = 'workflow' AND agent_id IS NULL + AND thread_id IS NULL AND resource_id IS NULL + AND suspension_marker = '${WORKFLOW_REMOTE_STEP_MARKER}' + AND request_digest IS NOT NULL)) + )`; +} + function boundedJsonBytes(value: string): void { if ( typeof value !== 'string' || @@ -132,6 +190,14 @@ export interface RemoteStepAddress { runId: string; } +export interface WorkflowRemoteStepAddress { + targetKind: 'workflow'; + deploymentId: string; + tablePrefix: string; + workflowId: string; + runId: string; +} + export interface RemoteStepInvocation extends RemoteStepAddress { invocationId: string; principal: ExecutionPrincipal; @@ -161,6 +227,12 @@ export interface RemoteStepInvocation extends RemoteStepAddress { | 'advanced-without-proof'; } +export interface WorkflowRemoteStepInvocation + extends Omit { + targetKind: 'workflow'; + requestDigest: string; +} + export interface RemoteStepPermit { invocationId: string; generation: number; @@ -218,6 +290,10 @@ type JournalRow = { resume_lease_expires_at: number | null; settled_at: number | null; settlement_outcome: string | null; + target_kind: string; + suspension_marker: string; + request_digest: string | null; + write_revision: number; }; function record(value: unknown): Record | undefined { @@ -239,6 +315,11 @@ function rowResult(value: unknown): JournalRow | undefined { function decode(row: JournalRow): RemoteStepInvocation { if ( + row.target_kind !== 'agent' || + row.suspension_marker !== REMOTE_STEP_MARKER || + (row.request_digest !== null && !SHA256_HEX.test(row.request_digest)) || + !Number.isSafeInteger(row.write_revision) || + row.write_revision < 0 || typeof row.step_path !== 'string' || row.step_path.length > 1_024 || typeof row.request_json !== 'string' || @@ -358,6 +439,129 @@ function decode(row: JournalRow): RemoteStepInvocation { }; } +function decodeWorkflow(row: JournalRow): WorkflowRemoteStepInvocation { + if ( + row.target_kind !== 'workflow' || + row.suspension_marker !== WORKFLOW_REMOTE_STEP_MARKER || + row.agent_id !== null || + row.thread_id !== null || + row.resource_id !== null || + !isPathSafeId(row.invocation_id) || + !isPathSafeId(row.deployment_id) || + !isPathSafeId(row.workflow_id) || + !isPathSafeId(row.run_id) || + !isPathSafeId(row.start_token) || + typeof row.request_digest !== 'string' || + !SHA256_HEX.test(row.request_digest) || + !Number.isSafeInteger(row.write_revision) || + row.write_revision < 0 || + !positiveTime(row.suspended_at) || + !positiveTime(row.deadline_at) || + !Number.isSafeInteger(row.resume_count) || + (row.resume_count !== -1 && row.resume_count < 1) || + !Number.isSafeInteger(row.permit_generation) || + row.permit_generation < 0 || + ![ + 'pending', + 'dispatched', + 'result-retained', + 'resuming', + 'settled', + 'failed', + ].includes(row.status) || + (row.status === 'resuming' && + (!isPathSafeId(row.resume_lease_owner) || + !positiveTime(row.resume_lease_expires_at) || + row.result_json === null)) || + (row.status !== 'resuming' && + (row.resume_lease_owner !== null || + row.resume_lease_expires_at !== null)) || + typeof row.step_path !== 'string' || + row.step_path.length > 1_024 || + typeof row.request_json !== 'string' || + new TextEncoder().encode(row.request_json).length > + MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + try { + validateTablePrefix(row.table_prefix); + const stepPath: unknown = JSON.parse(row.step_path); + const request: unknown = JSON.parse(row.request_json); + const principal = assertExecutionPrincipal( + JSON.parse(row.principal_json), + 'stored remote step principal', + ); + if ( + !Array.isArray(stepPath) || + stepPath.length === 0 || + stepPath.some((part) => typeof part !== 'string' || part === '') || + (row.settlement_outcome !== null && + ![ + 'applied', + 'terminal-not-success', + 'deadline-expired', + 'advanced-without-proof', + ].includes(row.settlement_outcome)) || + (row.status === 'settled' && row.settlement_outcome !== 'applied') || + (row.status === 'settled' && + (row.result_json === null || + row.result_digest === null || + !positiveTime(row.executor_quiescent_at))) || + (row.settlement_outcome !== null && + ((row.status !== 'settled' && row.status !== 'failed') || + !positiveTime(row.settled_at))) + ) + throw new Error('invalid workflow row'); + return { + targetKind: 'workflow', + invocationId: row.invocation_id, + principal, + startToken: row.start_token, + deploymentId: row.deployment_id, + tablePrefix: row.table_prefix, + workflowId: row.workflow_id, + runId: row.run_id, + stepPath, + suspendedAt: row.suspended_at, + ...(row.resume_count < 0 ? {} : { resumeCount: row.resume_count }), + request, + requestDigest: row.request_digest, + deadlineAt: row.deadline_at, + status: row.status as WorkflowRemoteStepInvocation['status'], + permitGeneration: row.permit_generation, + ...(row.permit_epoch === null ? {} : { permitEpoch: row.permit_epoch }), + ...(row.permit_revision === null + ? {} + : { permitRevision: row.permit_revision }), + ...(row.permit_expires_at === null + ? {} + : { permitExpiresAt: row.permit_expires_at }), + ...(row.permit_consumed_at === null + ? {} + : { permitConsumedAt: row.permit_consumed_at }), + ...(row.resume_lease_expires_at === null + ? {} + : { resumeLeaseExpiresAt: row.resume_lease_expires_at }), + ...(row.settlement_outcome === null + ? {} + : { + settlementOutcome: + row.settlement_outcome as WorkflowRemoteStepInvocation['settlementOutcome'], + }), + }; + } catch { + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); + } +} + +function decodeAny( + row: JournalRow, +): RemoteStepInvocation | WorkflowRemoteStepInvocation { + if (row.target_kind === 'agent') return decode(row); + if (row.target_kind === 'workflow') return decodeWorkflow(row); + throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); +} + function selectedSuspension( summary: RunSummary, allowedToolIds: ReadonlySet, @@ -412,6 +616,48 @@ function selectedSuspension( }; } +function selectedWorkflowSuspension(summary: RunSummary): { + stepPath: string[]; + suspendedAt: number; + resumeCount?: number; + requestJson: string; +} { + if (summary.status !== 'suspended') + throw new RemoteStepRefusal(409, 'run is not suspended'); + const payloads = record(summary.suspendPayload); + const matches = (summary.suspended ?? []).flatMap((path) => { + const key = path.join('.'); + const suspended = record(payloads?.[key]); + const remoteStep = record(suspended?.remoteStep); + return suspended?.type === 'suspension' && + remoteStep?.marker === WORKFLOW_REMOTE_STEP_MARKER + ? [{ path, key, remoteStep }] + : []; + }); + if (matches.length !== 1) + throw new RemoteStepRefusal(409, 'remote step suspension is ambiguous'); + const selected = matches[0]; + if (!selected || !Object.hasOwn(selected.remoteStep, 'request')) + throw new RemoteStepRefusal(503, 'remote step suspension is unreadable'); + const suspendedAt = summary.suspendedAt?.[selected.key]; + const resumeCount = summary.resumeCount?.[selected.key]; + const requestJson = JSON.stringify(selected.remoteStep.request); + if ( + !positiveTime(suspendedAt) || + (resumeCount !== undefined && + (!Number.isSafeInteger(resumeCount) || resumeCount < 1)) || + typeof requestJson !== 'string' || + new TextEncoder().encode(requestJson).length > MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal(503, 'remote step suspension is unreadable'); + return { + stepPath: selected.path, + suspendedAt, + ...(resumeCount === undefined ? {} : { resumeCount }), + requestJson, + }; +} + export class RemoteStepJournal { readonly #db: ExecutionFenceDatabase; readonly #fence: ExecutionFenceStore; @@ -429,6 +675,12 @@ export class RemoteStepJournal { resultJson: string; evidenceJson: string; }) => Promise; + readonly #verifyWorkflowCompletion?: (input: { + invocation: WorkflowRemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; get deploymentId(): string { return this.#deploymentId; @@ -454,6 +706,12 @@ export class RemoteStepJournal { resultJson: string; evidenceJson: string; }) => Promise; + verifyWorkflowCompletion?: (input: { + invocation: WorkflowRemoteStepInvocation; + rawPermit: string; + resultJson: string; + evidenceJson: string; + }) => Promise; now?: () => number; }) { if (!input.fence.usesDatabase(input.db)) @@ -477,46 +735,12 @@ export class RemoteStepJournal { this.#tablePrefix = input.tablePrefix; this.#allowedToolIds = new Set(input.allowedToolIds); this.#verifyCompletion = input.verifyCompletion; + this.#verifyWorkflowCompletion = input.verifyWorkflowCompletion; this.#now = input.now ?? Date.now; } async initialize(): Promise { - await this.#db - .prepare(`CREATE TABLE IF NOT EXISTS ${REMOTE_STEP_TABLE} ( - invocation_id TEXT PRIMARY KEY, - principal_json TEXT NOT NULL, - start_token TEXT NOT NULL, - deployment_id TEXT NOT NULL, - table_prefix TEXT NOT NULL, - workflow_id TEXT NOT NULL, - agent_id TEXT NOT NULL, - thread_id TEXT NOT NULL, - resource_id TEXT NOT NULL, - run_id TEXT NOT NULL, - step_path TEXT NOT NULL, - suspended_at INTEGER NOT NULL, - resume_count INTEGER NOT NULL, - request_json TEXT NOT NULL, - deadline_at INTEGER NOT NULL, - status TEXT NOT NULL, - permit_generation INTEGER NOT NULL DEFAULT 0, - permit_epoch INTEGER, - permit_revision INTEGER, - permit_expires_at INTEGER, - permit_consumed_at INTEGER, - permit_digest TEXT, - result_json TEXT, - result_digest TEXT, - quiescence_digest TEXT, - executor_quiescent_at INTEGER, - resume_lease_owner TEXT, - resume_lease_expires_at INTEGER, - settled_at INTEGER, - settlement_outcome TEXT, - UNIQUE (deployment_id, table_prefix, workflow_id, agent_id, - thread_id, resource_id, run_id, step_path, suspended_at, resume_count) - )`) - .run(); + await this.#db.prepare(remoteStepTableSql(REMOTE_STEP_TABLE)).run(); await this.#db .prepare(`CREATE TABLE IF NOT EXISTS ${REMOTE_STEP_SCHEMA_TABLE} ( id INTEGER PRIMARY KEY CHECK (id = 1), @@ -530,7 +754,7 @@ export class RemoteStepJournal { const columns = record(schema)?.results; if (!Array.isArray(columns)) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - const samePrefix = (expected: readonly string[]) => + const samePrefix = (expected: readonly string[], v4 = false) => columns.length === expected.length && columns.every((value, index) => { const column = record(value); @@ -541,10 +765,22 @@ export class RemoteStepJournal { (REMOTE_STEP_INTEGER_COLUMNS.has(name as string) ? 'INTEGER' : 'TEXT') && - column?.notnull === (index > 0 && index <= 16 ? 1 : 0) && + column?.notnull === + ((index > 0 && + index <= 16 && + !(v4 && index >= 6 && index <= 8)) || + (v4 && + ['target_kind', 'suspension_marker', 'write_revision'].includes( + name as string, + )) + ? 1 + : 0) && column?.pk === (index === 0 ? 1 : 0) && column?.hidden === 0 && - column?.dflt_value === (name === 'permit_generation' ? '0' : null) + column?.dflt_value === + (name === 'permit_generation' || name === 'write_revision' + ? '0' + : null) ); }); const indexes = record( @@ -592,8 +828,6 @@ export class RemoteStepJournal { ) uniqueSuspension = true; } - if (!uniqueSuspension) - throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); const recoveryIndex = indexes.find( (value) => record(value)?.name === REMOTE_STEP_RECOVERY_INDEX, ); @@ -610,6 +844,75 @@ export class RemoteStepJournal { Array.isArray(recoveryDefinition) && recoveryDefinition.length === 1 && record(recoveryDefinition[0])?.sql === REMOTE_STEP_RECOVERY_INDEX_SQL; + const validV4Index = async (name: string, sql: string) => { + const entry = indexes.find((value) => record(value)?.name === name); + const definition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('index', name) + .all(), + )?.results; + return ( + record(entry)?.unique === 1 && + record(entry)?.origin === 'c' && + record(entry)?.partial === 1 && + Array.isArray(definition) && + definition.length === 1 && + record(definition[0])?.sql === sql + ); + }; + const validV4Table = async () => { + const tableDefinition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('table', REMOTE_STEP_TABLE) + .all(), + )?.results; + const tableSql = + Array.isArray(tableDefinition) && tableDefinition.length === 1 + ? record(tableDefinition[0])?.sql + : undefined; + return ( + typeof tableSql === 'string' && + tableSql.includes( + "target_kind TEXT NOT NULL CHECK (target_kind IN ('agent', 'workflow'))", + ) && + tableSql.includes( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + ) && + tableSql.includes("(target_kind = 'workflow' AND agent_id IS NULL") && + tableSql.includes('write_revision INTEGER NOT NULL DEFAULT 0') + ); + }; + const validV4 = async () => { + const trigger = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('trigger', REMOTE_STEP_WRITE_TRIGGER) + .all(), + )?.results; + return ( + (await validV4Table()) && + validRecoveryIndex && + (await validV4Index( + REMOTE_STEP_AGENT_INDEX, + REMOTE_STEP_AGENT_INDEX_SQL, + )) && + (await validV4Index( + REMOTE_STEP_WORKFLOW_INDEX, + REMOTE_STEP_WORKFLOW_INDEX_SQL, + )) && + Array.isArray(trigger) && + trigger.length === 1 && + record(trigger[0])?.sql === REMOTE_STEP_WRITE_TRIGGER_SQL + ); + }; const versions = await this.#db .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) .all(); @@ -617,26 +920,40 @@ export class RemoteStepJournal { if (!Array.isArray(rows) || rows.length > 1) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); if (rows.length === 1) { + if (record(rows[0])?.id === 1 && record(rows[0])?.version === 4) { + if (!samePrefix(REMOTE_STEP_COLUMNS, true) || !(await validV4())) + throw new RemoteStepRefusal( + 503, + 'remote step schema is unreadable', + ); + return 'current-v4'; + } if ( record(rows[0])?.id !== 1 || (record(rows[0])?.version !== 3 && record(rows[0])?.version !== 2) || + !uniqueSuspension || !samePrefix( record(rows[0])?.version === 3 - ? REMOTE_STEP_COLUMNS - : REMOTE_STEP_COLUMNS.slice(0, -4), + ? REMOTE_STEP_COLUMNS.slice(0, -4) + : REMOTE_STEP_COLUMNS.slice(0, -8), ) || (record(rows[0])?.version === 3 && !validRecoveryIndex) ) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - return record(rows[0])?.version === 3 ? 'current' : 'prior-v2'; + return record(rows[0])?.version === 3 ? 'prior-v3' : 'prior-v2'; } - if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -8))) return 'prior-v1'; - if (samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) return 'unversioned-v2'; - if (samePrefix(REMOTE_STEP_COLUMNS)) return 'unversioned-current'; + if (uniqueSuspension && samePrefix(REMOTE_STEP_COLUMNS.slice(0, -12))) + return 'prior-v1'; + if (uniqueSuspension && samePrefix(REMOTE_STEP_COLUMNS.slice(0, -8))) + return 'unversioned-v2'; + if (uniqueSuspension && samePrefix(REMOTE_STEP_COLUMNS.slice(0, -4))) + return 'unversioned-v3'; + if (samePrefix(REMOTE_STEP_COLUMNS, true) && (await validV4Table())) + return 'unversioned-v4'; throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); }; const stage = await readSchema(); - if (stage === 'current') return; + if (stage === 'current-v4') return; if ( stage === 'prior-v1' || stage === 'prior-v2' || @@ -673,9 +990,10 @@ export class RemoteStepJournal { ), ]); } catch (error) { - if ((await readSchema()) !== 'current') throw error; + const current = await readSchema(); + if (current !== 'prior-v3' && current !== 'current-v4') throw error; } - return; + return this.initialize(); } const transactional = this.#db as ExecutionFenceDatabase & { batch?: ( @@ -687,19 +1005,65 @@ export class RemoteStepJournal { 503, 'remote step schema upgrade is unavailable', ); - try { - await transactional.batch([ - this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), - this.#db.prepare( - `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 3)`, - ), - ]); - } catch (error) { - if ((await readSchema()) !== 'current') throw error; + if (stage === 'unversioned-v3') { + try { + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 3)`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'prior-v3') throw error; + } + return this.initialize(); + } + if (stage === 'prior-v3') { + const replacement = `${REMOTE_STEP_TABLE}_v4`; + try { + await transactional.batch([ + this.#db.prepare(remoteStepTableSql(replacement)), + this.#db.prepare(`INSERT INTO ${replacement} (${REMOTE_STEP_COLUMNS.join(', ')}) + SELECT ${REMOTE_STEP_COLUMNS.slice(0, -4).join(', ')}, 'agent', '${REMOTE_STEP_MARKER}', NULL, 0 FROM ${REMOTE_STEP_TABLE}`), + this.#db.prepare(`DROP TABLE ${REMOTE_STEP_TABLE}`), + this.#db.prepare( + `ALTER TABLE ${replacement} RENAME TO ${REMOTE_STEP_TABLE}`, + ), + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WRITE_TRIGGER_SQL), + this.#db.prepare( + `UPDATE ${REMOTE_STEP_SCHEMA_TABLE} SET version = 4 WHERE id = 1 AND version = 3`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'current-v4') throw error; + } + return; + } + if (stage === 'unversioned-v4') { + try { + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WRITE_TRIGGER_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 4)`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'current-v4') throw error; + } + return; } + throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); } - async #snapshot(address: RemoteStepAddress): Promise { + async #snapshot( + address: RemoteStepAddress | WorkflowRemoteStepAddress, + ): Promise { const result = await this.#db .prepare( `SELECT snapshot FROM "${address.tablePrefix}mastra_workflow_snapshot" @@ -717,16 +1081,20 @@ export class RemoteStepJournal { return record(rows[0])?.snapshot as string; } - async #current(address: RemoteStepAddress) { + async #current(address: RemoteStepAddress | WorkflowRemoteStepAddress) { const summary = await this.#runtime.authoritativeStatus( address.workflowId, address.runId, ); if (!summary) throw new RemoteStepRefusal(404, 'run not found'); - return selectedSuspension(summary, this.#allowedToolIds); + return 'targetKind' in address + ? selectedWorkflowSuspension(summary) + : selectedSuspension(summary, this.#allowedToolIds); } - async #stableSnapshot(address: RemoteStepAddress) { + async #stableSnapshot( + address: RemoteStepAddress | WorkflowRemoteStepAddress, + ) { const before = await this.#current(address); const snapshot = await this.#snapshot(address); const state = await this.#execution(address); @@ -790,8 +1158,8 @@ export class RemoteStepJournal { .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( invocation_id, principal_json, start_token, deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count, - request_json, deadline_at, status) - SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending' + request_json, deadline_at, status, target_kind, suspension_marker) + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'agent', '${REMOTE_STEP_MARKER}' WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) @@ -819,7 +1187,7 @@ export class RemoteStepJournal { .run(); const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} - SET deadline_at = MIN(deadline_at, ?) + SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1 WHERE deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? AND step_path = ? AND suspended_at = ? AND resume_count = ? @@ -857,7 +1225,128 @@ export class RemoteStepJournal { return decode(row); } + async registerWorkflow( + address: WorkflowRemoteStepAddress, + hostDeadlineAt: number, + originalPrincipal: ExecutionPrincipal, + ): Promise { + if ( + address.targetKind !== 'workflow' || + address.deploymentId !== this.#deploymentId || + address.tablePrefix !== this.#tablePrefix || + !isPathSafeId(address.workflowId) || + !isPathSafeId(address.runId) || + !positiveTime(hostDeadlineAt) + ) + throw new RemoteStepRefusal(400, 'remote step address is malformed'); + validateTablePrefix(address.tablePrefix); + let principal: ExecutionPrincipal; + try { + principal = assertExecutionPrincipal( + originalPrincipal, + 'remote step principal', + ); + } catch { + throw new RemoteStepRefusal(400, 'remote step principal is malformed'); + } + const execution = await this.#execution(address, principal); + const { suspension, snapshot } = await this.#stableSnapshot(address); + const summary = await this.#runtime.authoritativeStatus( + address.workflowId, + address.runId, + ); + const deadlineAt = Math.min( + hostDeadlineAt, + summary?.deadlineAt ?? hostDeadlineAt, + ); + if (deadlineAt <= this.#now()) + throw new RemoteStepRefusal(409, 'remote step deadline elapsed'); + const requestDigest = await sha256Hex(suspension.requestJson); + const stepPath = JSON.stringify(suspension.stepPath); + const invocationId = crypto.randomUUID(); + await this.#db + .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( + invocation_id, principal_json, start_token, deployment_id, table_prefix, + workflow_id, run_id, step_path, suspended_at, resume_count, request_json, + deadline_at, status, target_kind, suspension_marker, request_digest) + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'workflow', + '${WORKFLOW_REMOTE_STEP_MARKER}', ? + WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + ON CONFLICT DO NOTHING`) + .bind( + invocationId, + JSON.stringify(principal), + execution.execution.startToken, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + deadlineAt, + requestDigest, + address.workflowId, + address.runId, + snapshot, + ) + .run(); + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1 + WHERE target_kind = 'workflow' AND deployment_id = ? + AND table_prefix = ? AND workflow_id = ? AND run_id = ? + AND step_path = ? AND suspended_at = ? AND resume_count = ? + AND request_json = ? AND request_digest = ? + AND principal_json = ? AND start_token = ? + AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + RETURNING *`) + .bind( + deadlineAt, + address.deploymentId, + address.tablePrefix, + address.workflowId, + address.runId, + stepPath, + suspension.suspendedAt, + suspension.resumeCount ?? -1, + suspension.requestJson, + requestDigest, + JSON.stringify(principal), + execution.execution.startToken, + address.workflowId, + address.runId, + snapshot, + ) + .all(), + ); + if (!row) + throw new RemoteStepRefusal( + 409, + 'remote step registration did not settle', + ); + return decodeWorkflow(row); + } + async get(invocationId: string): Promise { + const row = await this.#getRow(invocationId); + return row?.target_kind === 'agent' ? decode(row) : undefined; + } + + async getWorkflow( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + return row?.target_kind === 'workflow' ? decodeWorkflow(row) : undefined; + } + + async #getRow(invocationId: string): Promise { if (!isPathSafeId(invocationId)) throw new RemoteStepRefusal(400, 'invocation id is malformed'); const row = rowResult( @@ -866,7 +1355,14 @@ export class RemoteStepJournal { .bind(invocationId) .all(), ); - return row ? decode(row) : undefined; + return row; + } + + async #getAny( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + return row ? decodeAny(row) : undefined; } async readDispatchRequestJson(invocationId: string): Promise { @@ -883,37 +1379,54 @@ export class RemoteStepJournal { return row.request_json; } - async #execution(address: RemoteStepAddress, principal?: ExecutionPrincipal) { + async #execution( + address: RemoteStepAddress | WorkflowRemoteStepAddress, + principal?: ExecutionPrincipal, + ) { const state = await this.#runtime.authoritativeStartState( address.workflowId, address.runId, ); const target = state?.provenance.startIdentity?.target; - if ( - state?.storage !== 'd1' || - target?.kind !== 'agent' || - !isPathSafeId(target.resourceId) - ) + if (state?.storage !== 'd1' || !target) throw new RemoteStepRefusal( 503, 'remote step run identity is unreadable', ); if ( - target.id !== address.agentId || - target.threadId !== address.threadId || (principal !== undefined && (state.provenance.startIdentity?.owner.kind !== principal.kind || state.provenance.startIdentity.owner.id !== principal.id)) || state.execution.tablePrefix !== address.tablePrefix || state.execution.workflowId !== address.workflowId || - state.execution.runId !== address.runId || + state.execution.runId !== address.runId + ) + throw new RemoteStepRefusal(409, 'remote step run identity changed'); + if ('targetKind' in address) { + if ( + target.kind !== 'workflow' || + target.id !== address.workflowId || + state.provenance.agentStart !== undefined + ) + throw new RemoteStepRefusal(409, 'remote step run identity changed'); + } else if (target.kind !== 'agent' || !isPathSafeId(target.resourceId)) { + throw new RemoteStepRefusal( + 503, + 'remote step run identity is unreadable', + ); + } else if ( + target.id !== address.agentId || + target.threadId !== address.threadId || address.resourceId !== target.resourceId ) throw new RemoteStepRefusal(409, 'remote step run identity changed'); return state; } - async #admission(invocation: RemoteStepInvocation, priorBindings: number) { + async #admission( + invocation: RemoteStepInvocation | WorkflowRemoteStepInvocation, + priorBindings: number, + ) { const observed = await this.#fence.readForAdmission(); const proof = (await this.#execution(invocation, invocation.principal)) .execution; @@ -948,8 +1461,13 @@ export class RemoteStepJournal { invocationId: string, lifetimeMs: number, ): Promise { - const invocation = await this.get(invocationId); + const invocation = await this.#getAny(invocationId); if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if ('targetKind' in invocation && !this.#verifyWorkflowCompletion) + throw new RemoteStepRefusal( + 503, + 'workflow remote step completion is unavailable', + ); if (!positiveTime(lifetimeMs)) throw new RemoteStepRefusal(400, 'permit lifetime is malformed'); const now = this.#now(); @@ -963,10 +1481,12 @@ export class RemoteStepJournal { suspension.resumeCount !== invocation.resumeCount || suspension.requestJson !== JSON.stringify(invocation.request) || JSON.stringify(suspension.stepPath) !== - JSON.stringify(invocation.stepPath) + JSON.stringify(invocation.stepPath) || + ('targetKind' in invocation && + (await sha256Hex(suspension.requestJson)) !== invocation.requestDigest) ) throw new RemoteStepRefusal(409, 'remote step suspension changed'); - const gate = await this.#admission(invocation, 10); + const gate = await this.#admission(invocation, 13); if ( invocation.status === 'dispatched' && invocation.permitConsumedAt === undefined && @@ -996,10 +1516,12 @@ export class RemoteStepJournal { status = 'dispatched', permit_generation = permit_generation + 1, permit_epoch = ?, permit_revision = ?, permit_expires_at = MIN(?, deadline_at), permit_consumed_at = NULL, - permit_digest = NULL + permit_digest = NULL, write_revision = write_revision + 1 WHERE invocation_id = ? AND permit_generation = ? AND deadline_at > ? AND status IN ('pending', 'dispatched') AND permit_consumed_at IS NULL AND request_json = ? + AND target_kind = ? AND suspension_marker = ? + AND request_digest IS ? AND EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) @@ -1012,6 +1534,11 @@ export class RemoteStepJournal { invocation.permitGeneration, now, suspension.requestJson, + 'targetKind' in invocation ? 'workflow' : 'agent', + 'targetKind' in invocation + ? WORKFLOW_REMOTE_STEP_MARKER + : REMOTE_STEP_MARKER, + 'targetKind' in invocation ? invocation.requestDigest : null, invocation.workflowId, invocation.runId, snapshot, @@ -1034,8 +1561,13 @@ export class RemoteStepJournal { ): Promise< { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } > { - const invocation = await this.get(permit.invocationId); + const invocation = await this.#getAny(permit.invocationId); if (!invocation) throw new RemoteStepRefusal(404, 'invocation not found'); + if ('targetKind' in invocation && !this.#verifyWorkflowCompletion) + throw new RemoteStepRefusal( + 503, + 'workflow remote step completion is unavailable', + ); if ( invocation.permitGeneration !== permit.generation || invocation.permitEpoch !== permit.mutationEpoch || @@ -1052,20 +1584,24 @@ export class RemoteStepJournal { suspension.resumeCount !== invocation.resumeCount || suspension.requestJson !== JSON.stringify(invocation.request) || JSON.stringify(suspension.stepPath) !== - JSON.stringify(invocation.stepPath) + JSON.stringify(invocation.stepPath) || + ('targetKind' in invocation && + (await sha256Hex(suspension.requestJson)) !== invocation.requestDigest) ) throw new RemoteStepRefusal(409, 'remote step suspension changed'); const rawPermit = rawDispatchPermit(); const permitDigest = await sha256Hex(rawPermit); - const gate = await this.#admission(invocation, 13); + const gate = await this.#admission(invocation, 16); const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} - SET permit_consumed_at = ?, permit_digest = ? + SET permit_consumed_at = ?, permit_digest = ?, write_revision = write_revision + 1 WHERE invocation_id = ? AND permit_generation = ? AND permit_epoch = ? AND permit_revision = ? AND permit_expires_at = ? AND permit_expires_at > ? AND deadline_at > ? AND permit_consumed_at IS NULL AND status = 'dispatched' AND request_json = ? + AND target_kind = ? AND suspension_marker = ? + AND request_digest IS ? AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) @@ -1081,6 +1617,11 @@ export class RemoteStepJournal { now, now, suspension.requestJson, + 'targetKind' in invocation ? 'workflow' : 'agent', + 'targetKind' in invocation + ? WORKFLOW_REMOTE_STEP_MARKER + : REMOTE_STEP_MARKER, + 'targetKind' in invocation ? invocation.requestDigest : null, invocation.workflowId, invocation.runId, snapshot, @@ -1088,7 +1629,7 @@ export class RemoteStepJournal { ) .all(); if (rowResult(result)) return { kind: 'accepted', rawPermit }; - const after = await this.get(permit.invocationId); + const after = await this.#getAny(permit.invocationId); if ( after?.permitGeneration === permit.generation && after.permitConsumedAt !== undefined @@ -1120,7 +1661,7 @@ export class RemoteStepJournal { .all(), ); if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); - const invocation = decode(stored); + const invocation = decodeAny(stored); const permitDigest = stored.permit_digest; if ( typeof permitDigest !== 'string' || @@ -1144,16 +1685,31 @@ export class RemoteStepJournal { return { kind: 'replay' as const, resultJson, resultDigest }; }; if (stored.result_json !== null) return replay(stored); - await this.#verifyCompletion({ - invocation, - rawPermit, - resultJson, - evidenceJson, - }); + if ('targetKind' in invocation) { + if (!this.#verifyWorkflowCompletion) + throw new RemoteStepRefusal( + 503, + 'workflow remote step completion is unavailable', + ); + await this.#verifyWorkflowCompletion({ + invocation, + rawPermit, + resultJson, + evidenceJson, + }); + } else { + await this.#verifyCompletion({ + invocation, + rawPermit, + resultJson, + evidenceJson, + }); + } const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET result_json = ?, result_digest = ?, quiescence_digest = ?, - executor_quiescent_at = ?, status = 'result-retained' + executor_quiescent_at = ?, status = 'result-retained', + write_revision = write_revision + 1 WHERE invocation_id = ? AND permit_digest = ? AND permit_consumed_at IS NOT NULL AND result_json IS NULL AND status = 'dispatched' @@ -1192,7 +1748,7 @@ export class RemoteStepJournal { .all(), ); if (!row) return undefined; - decode(row); + decodeAny(row); const resultJson = row.result_json; const resultDigest = row.result_digest; const quiescenceDigest = row.quiescence_digest; @@ -1246,7 +1802,8 @@ export class RemoteStepJournal { const owner = crypto.randomUUID(); const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} - SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ? + SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ?, + write_revision = write_revision + 1 WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? AND start_token = ? @@ -1361,7 +1918,7 @@ export class RemoteStepJournal { const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = ?, resume_lease_owner = NULL, resume_lease_expires_at = NULL, - settled_at = ?, settlement_outcome = ? + settled_at = ?, settlement_outcome = ?, write_revision = write_revision + 1 WHERE invocation_id = ? AND status = 'resuming' AND resume_lease_owner = ? AND permit_generation = ? AND permit_epoch IS ? AND start_token = ? @@ -1415,6 +1972,7 @@ export class RemoteStepJournal { .prepare(`SELECT invocation_id, status, resume_lease_expires_at FROM ${REMOTE_STEP_TABLE} INDEXED BY ${REMOTE_STEP_RECOVERY_INDEX} WHERE invocation_id > ? + AND target_kind = 'agent' AND status IN ('result-retained', 'resuming') ORDER BY invocation_id LIMIT ?`) .bind(afterInvocationId, limit) From e176092eb3ae88988e9382e6a18fdd844999490f Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:39:36 +0400 Subject: [PATCH 35/73] Describe workflow journal and private state egress in release note --- .changeset/flowsafe-remote-step-broker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/flowsafe-remote-step-broker.md b/.changeset/flowsafe-remote-step-broker.md index 9ecb2380..ecfbe192 100644 --- a/.changeset/flowsafe-remote-step-broker.md +++ b/.changeset/flowsafe-remote-step-broker.md @@ -2,4 +2,4 @@ '@proofoftech/flowsafe': minor --- -Add a D1-backed remote-step journal, one-time host dispatch permits, retained results, and a host context derived from the stored invocation. A host-owned notification operation stores its effect and replay receipt in one D1 transaction with a configurable retained-receipt cap. General Mastra continuation remains fail-closed without generation-fenced snapshot writes; hosts must supply trusted dispatch transport and downstream effect receipts before enabling remote execution. +Add a D1-backed remote-step journal for agent and workflow suspensions, one-time host dispatch permits, retained results, and a host context derived from the stored invocation. Versioned private state egress carries internal dispatch metadata without changing the legacy route. A host-owned notification operation stores its effect and replay receipt in one D1 transaction with a configurable retained-receipt cap. General Mastra continuation remains fail-closed without generation-fenced snapshot writes; hosts must supply trusted dispatch transport and downstream effect receipts before enabling remote execution. From 592d9a6238b75bd6d9825319042a4570abd474ba Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:53:49 +0400 Subject: [PATCH 36/73] Expose host-owned workflow remote-step port Bind workflow registration and permit operations to stored principal, deployment and run access through the host topology. Keep the port opt-in and route-free while private transport and verified continuation remain pending. --- packages/flowsafe/src/agent-host/index.ts | 3 + .../src/agent-host/thread-topology.test.ts | 170 ++++++++++++++++ .../src/agent-host/thread-topology.ts | 184 ++++++++++++++++++ .../src/do-runner/remote-step-journal.test.ts | 6 + .../src/do-runner/remote-step-journal.ts | 15 +- 5 files changed, 377 insertions(+), 1 deletion(-) diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index 04cb83c9..38397f37 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -14,6 +14,7 @@ export type { RecordedRemoteStepResult, RemoteStepInvocation, RemoteStepPermit, + WorkflowRemoteStepInvocation, } from '../do-runner/remote-step-journal.js'; export { type AgentApprovalResumer, @@ -58,6 +59,8 @@ export { type RemoteStepContinuationReader, type RemoteStepDispatchReader, type RemoteStepDispatchTicket, + type WorkflowRemoteStepDispatchReader, + type WorkflowRemoteStepDispatchTicket, } from './thread-topology.js'; export { AGENT_AUDIT_CONTEXT_KEY, diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 9b5a9b66..77843730 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -21,6 +21,7 @@ import { createAgentThreadTopology, type RemoteStepContinuationReader, type RemoteStepDispatchReader, + type WorkflowRemoteStepDispatchReader, } from './thread-topology.js'; const DEPLOYMENT_IDENTITY_SECRET = 'test-deployment-identity-secret-0001'; @@ -34,6 +35,7 @@ interface Hit { function harness( remoteSteps?: RemoteStepContinuationReader, remoteStepDispatch?: RemoteStepDispatchReader, + workflowRemoteStepDispatch?: WorkflowRemoteStepDispatchReader, ): { topology: AgentThreadDispatchTopology; hits: Hit[]; @@ -99,11 +101,179 @@ function harness( executionFence: 'none', ...(remoteSteps ? { remoteSteps } : {}), ...(remoteStepDispatch ? { remoteStepDispatch } : {}), + ...(workflowRemoteStepDispatch ? { workflowRemoteStepDispatch } : {}), }), hits, }; } +it('keeps workflow remote-step dispatch within the stored owner and deployment', async () => { + const owner = context().value; + const permit = { + invocationId: 'workflow-invocation', + generation: 2, + mutationEpoch: 7, + transitionRevision: 4, + expiresAt: 9_999_999_999_999, + }; + const invocation = { + targetKind: 'workflow' as const, + invocationId: permit.invocationId, + deploymentId: 'deployment', + tablePrefix: '', + workflowId: 'customer-workflow', + runId: 'workflow-run', + principal: owner.principal, + startToken: 'stored-start-token', + stepPath: ['remote-step'], + suspendedAt: 1000, + request: { script: 'script-1' }, + requestDigest: 'stored-request-digest', + deadlineAt: permit.expiresAt, + status: 'pending' as const, + permitGeneration: 1, + }; + const dispatched = { + ...invocation, + status: 'dispatched' as const, + permitGeneration: permit.generation, + }; + const reader: WorkflowRemoteStepDispatchReader = { + deploymentId: invocation.deploymentId, + tablePrefix: invocation.tablePrefix, + registerWorkflow: vi.fn(async () => invocation), + getWorkflow: vi + .fn() + .mockResolvedValueOnce(invocation) + .mockResolvedValue(dispatched), + readWorkflowDispatchRequestJson: vi.fn(async () => '{"script":"script-1"}'), + issuePermit: vi.fn(async () => permit), + consumePermit: vi + .fn() + .mockResolvedValueOnce({ kind: 'accepted', rawPermit: 'winner-secret' }) + .mockResolvedValueOnce({ kind: 'already-consumed' }), + recordResult: vi.fn(async () => ({ + kind: 'recorded' as const, + resultJson: '{"answer":42}', + resultDigest: 'stored-result-digest', + })), + }; + const { topology } = harness(undefined, undefined, reader); + expect( + await topology.registerWorkflowRemoteStep( + owner, + invocation.workflowId, + invocation.runId, + 10_000, + ), + ).toEqual(invocation); + expect(reader.registerWorkflow).toHaveBeenCalledWith( + { + targetKind: 'workflow', + deploymentId: invocation.deploymentId, + tablePrefix: invocation.tablePrefix, + workflowId: invocation.workflowId, + runId: invocation.runId, + }, + 10_000, + owner.principal, + ); + const ticket = await topology.issueWorkflowRemoteStepDispatch( + owner, + invocation.invocationId, + 60_000, + ); + expect(ticket).toMatchObject({ + invocationId: invocation.invocationId, + requestJson: '{"script":"script-1"}', + requestDigest: invocation.requestDigest, + permit, + }); + expect(ticket).not.toHaveProperty('principal'); + expect(ticket).not.toHaveProperty('startToken'); + expect( + await topology.getWorkflowRemoteStep(owner, invocation.invocationId), + ).toEqual(dispatched); + expect( + await topology.consumeWorkflowRemoteStepDispatch(owner, permit), + ).toEqual({ kind: 'accepted', rawPermit: 'winner-secret' }); + expect( + await topology.consumeWorkflowRemoteStepDispatch(owner, permit), + ).toEqual({ kind: 'already-consumed' }); + expect( + await topology.recordWorkflowRemoteStepResult( + owner, + invocation.invocationId, + 'winner-secret', + '{"answer":42}', + '{"quiescent":true}', + ), + ).toMatchObject({ kind: 'recorded', resultJson: '{"answer":42}' }); + expect(reader.recordResult).toHaveBeenCalledWith( + invocation.invocationId, + 'winner-secret', + '{"answer":42}', + '{"quiescent":true}', + ); + const otherPrincipal = { + ...owner, + principal: { + kind: 'human' as const, + id: 'other', + role: 'operator' as const, + }, + }; + await expect( + topology.issueWorkflowRemoteStepDispatch( + otherPrincipal, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.consumeWorkflowRemoteStepDispatch(otherPrincipal, permit), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.recordWorkflowRemoteStepResult( + otherPrincipal, + invocation.invocationId, + 'winner-secret', + '{"answer":42}', + '{"quiescent":true}', + ), + ).rejects.toMatchObject({ status: 404 }); + const deniedRun = { ...owner, canAccessResource: async () => false }; + await expect( + topology.registerWorkflowRemoteStep( + deniedRun, + invocation.workflowId, + invocation.runId, + 10_000, + ), + ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.issueWorkflowRemoteStepDispatch( + deniedRun, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.getWorkflow).mockResolvedValueOnce({ + ...dispatched, + deploymentId: 'another-deployment', + }); + await expect( + topology.issueWorkflowRemoteStepDispatch( + owner, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(reader.issuePermit).toHaveBeenCalledTimes(1); + expect(reader.consumePermit).toHaveBeenCalledTimes(2); + expect(reader.recordResult).toHaveBeenCalledTimes(1); +}); + it('issues a remote dispatch ticket and exposes a raw permit to the consume winner', async () => { const permit = { invocationId: 'invocation_1', diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 0532fb1c..356a07bc 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -34,7 +34,9 @@ import { import type { RecordedRemoteStepResult, RemoteStepInvocation, + RemoteStepJournal, RemoteStepPermit, + WorkflowRemoteStepInvocation, } from '../do-runner/remote-step-journal.js'; import { isTerminalRunStatus, @@ -129,6 +131,33 @@ export interface AgentThreadTopologyOptions { executionFence: ExecutionFenceWiring; remoteSteps?: RemoteStepContinuationReader; remoteStepDispatch?: RemoteStepDispatchReader; + workflowRemoteStepDispatch?: WorkflowRemoteStepDispatchReader; +} + +export type WorkflowRemoteStepDispatchReader = Pick< + RemoteStepJournal, + | 'deploymentId' + | 'tablePrefix' + | 'registerWorkflow' + | 'getWorkflow' + | 'readWorkflowDispatchRequestJson' + | 'issuePermit' + | 'consumePermit' + | 'recordResult' +>; + +export interface WorkflowRemoteStepDispatchTicket { + invocationId: string; + deploymentId: string; + workflowId: string; + runId: string; + stepPath: readonly string[]; + suspendedAt: number; + resumeCount?: number; + deadlineAt: number; + requestJson: string; + requestDigest: string; + permit: RemoteStepPermit; } export interface RemoteStepDispatchReader { @@ -258,6 +287,34 @@ export interface AgentThreadDispatchTopology extends AgentThreadBoundTopology { ): Promise< { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } >; + registerWorkflowRemoteStep( + context: ActorContext, + workflowId: string, + runId: string, + hostDeadlineAt: number, + ): Promise; + getWorkflowRemoteStep( + context: ActorContext, + invocationId: string, + ): Promise; + issueWorkflowRemoteStepDispatch( + context: ActorContext, + invocationId: string, + lifetimeMs: number, + ): Promise; + consumeWorkflowRemoteStepDispatch( + context: ActorContext, + permit: RemoteStepPermit, + ): Promise< + { kind: 'accepted'; rawPermit: string } | { kind: 'already-consumed' } + >; + recordWorkflowRemoteStepResult( + context: ActorContext, + invocationId: string, + rawPermit: string, + resultJson: string, + evidenceJson: string, + ): ReturnType; } function errorFrom(response: Response): Promise { @@ -347,6 +404,33 @@ export function createAgentThreadTopology( ); return { reader, invocation }; }; + const workflowDispatchInvocation = async ( + context: ActorContext, + invocationId: string, + access: 'read' | 'write', + ) => { + const reader = options.workflowRemoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.getWorkflow(invocationId); + if ( + invocation?.targetKind !== 'workflow' || + invocation.invocationId !== invocationId || + invocation.deploymentId !== reader.deploymentId || + invocation.tablePrefix !== reader.tablePrefix || + !samePrincipal(invocation.principal, context.principal) + ) + throw new RunRouteError(404, 'invocation not found'); + await requireResourceAccess( + context, + 'run', + invocation.runId, + access, + 'run', + ); + return { reader, invocation }; + }; const fetchStatus = async ( context: ActorContext, input: AgentThreadRunRef, @@ -503,6 +587,106 @@ export function createAgentThreadTopology( } }; return { + registerWorkflowRemoteStep: async ( + context, + workflowId, + runId, + hostDeadlineAt, + ) => { + const reader = options.workflowRemoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(workflowId)) + throw new RunRouteError(404, 'workflow not found'); + await requireResourceAccess(context, 'run', runId, 'write', 'run'); + return reader.registerWorkflow( + { + targetKind: 'workflow', + deploymentId: reader.deploymentId, + tablePrefix: reader.tablePrefix, + workflowId, + runId, + }, + hostDeadlineAt, + context.principal, + ); + }, + getWorkflowRemoteStep: async (context, invocationId) => { + const reader = options.workflowRemoteStepDispatch; + if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); + if (!isPathSafeId(invocationId)) + throw new RunRouteError(404, 'invocation not found'); + const invocation = await reader.getWorkflow(invocationId); + if (!invocation) return undefined; + return (await workflowDispatchInvocation(context, invocationId, 'read')) + .invocation; + }, + issueWorkflowRemoteStepDispatch: async ( + context, + invocationId, + lifetimeMs, + ) => { + const { reader } = await workflowDispatchInvocation( + context, + invocationId, + 'write', + ); + const permit = await reader.issuePermit(invocationId, lifetimeMs); + const { invocation } = await workflowDispatchInvocation( + context, + invocationId, + 'write', + ); + if ( + invocation.status !== 'dispatched' || + invocation.permitGeneration !== permit.generation || + invocation.deadlineAt < permit.expiresAt + ) + throw new RunRouteError(409, 'remote step dispatch changed'); + const requestJson = + await reader.readWorkflowDispatchRequestJson(invocationId); + return { + invocationId, + deploymentId: invocation.deploymentId, + workflowId: invocation.workflowId, + runId: invocation.runId, + stepPath: invocation.stepPath, + suspendedAt: invocation.suspendedAt, + ...(invocation.resumeCount === undefined + ? {} + : { resumeCount: invocation.resumeCount }), + deadlineAt: invocation.deadlineAt, + requestJson, + requestDigest: invocation.requestDigest, + permit, + }; + }, + consumeWorkflowRemoteStepDispatch: async (context, permit) => { + const { reader } = await workflowDispatchInvocation( + context, + permit.invocationId, + 'write', + ); + return reader.consumePermit(permit); + }, + recordWorkflowRemoteStepResult: async ( + context, + invocationId, + rawPermit, + resultJson, + evidenceJson, + ) => { + const { reader } = await workflowDispatchInvocation( + context, + invocationId, + 'write', + ); + return reader.recordResult( + invocationId, + rawPermit, + resultJson, + evidenceJson, + ); + }, issueRemoteStepDispatch: async (context, invocationId, lifetimeMs) => { await dispatchInvocation(context, invocationId); const reader = options.remoteStepDispatch; diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 7dbb6b62..34ab3c74 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -1434,6 +1434,12 @@ describe('remote step journal dispatch admission', () => { 'f908a6c18fa0b52b0f1295fc8f1cb5db083020cb18dab24ebf26637f55a9af49', ); expect(await journal.get(first.invocationId)).toBeUndefined(); + expect( + await journal.readWorkflowDispatchRequestJson(first.invocationId), + ).toBe('{"version":2,"input":{"query":"x"}}'); + await expect( + journal.readDispatchRequestJson(first.invocationId), + ).rejects.toMatchObject({ status: 404 }); expect(await journal.getWorkflow(first.invocationId)).toMatchObject({ targetKind: 'workflow', status: 'pending', diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index e3030144..ed4bf398 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -1366,6 +1366,17 @@ export class RemoteStepJournal { } async readDispatchRequestJson(invocationId: string): Promise { + return this.#readDispatchRequestJson(invocationId, 'agent'); + } + + async readWorkflowDispatchRequestJson(invocationId: string): Promise { + return this.#readDispatchRequestJson(invocationId, 'workflow'); + } + + async #readDispatchRequestJson( + invocationId: string, + targetKind: 'agent' | 'workflow', + ): Promise { if (!isPathSafeId(invocationId)) throw new RemoteStepRefusal(400, 'invocation id is malformed'); const row = rowResult( @@ -1375,7 +1386,9 @@ export class RemoteStepJournal { .all(), ); if (!row) throw new RemoteStepRefusal(404, 'invocation not found'); - decode(row); + if (row.target_kind !== targetKind) + throw new RemoteStepRefusal(404, 'invocation not found'); + decodeAny(row); return row.request_json; } From c5387937b444d61bb60b48b66c07a9e3de2d10d4 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:26:39 +0400 Subject: [PATCH 37/73] Persist verified workflow start principal after admission Record an immutable per-run DO receipt only after confirmed run-owner completion. Leave scheduled, legacy, failed and ambiguous starts remote-ineligible while preserving ordinary start behavior. --- .../src/do-runner/durable-object.test.ts | 133 ++++++++++++++++++ .../flowsafe/src/do-runner/durable-object.ts | 47 +++++++ 2 files changed, 180 insertions(+) diff --git a/packages/flowsafe/src/do-runner/durable-object.test.ts b/packages/flowsafe/src/do-runner/durable-object.test.ts index 1befaef7..fec6b813 100644 --- a/packages/flowsafe/src/do-runner/durable-object.test.ts +++ b/packages/flowsafe/src/do-runner/durable-object.test.ts @@ -637,6 +637,136 @@ async function requireSecondMutationEpoch( } describe('workflow ingress capture', () => { + it('persists the trusted principal and prepared identity for a direct start', async () => { + const fixture = workflowIngressFixture(); + const principal: ExecutionPrincipal = { + kind: 'agent', + id: 'agent-1', + purpose: 'customer dispatch', + delegatedBy: 'owner-1', + }; + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY, principal), + ); + expect(response.status).toBe(200); + const stored = fixture.journal.values.get('flowsafe:start-principal:v1'); + expect(typeof stored).toBe('string'); + const authoritative = await fixture.runtime.authoritativeStartState( + 'gated', + 'c-run', + ); + if (!authoritative || authoritative.kind === 'initial') + throw new Error('missing persisted start identity'); + expect(JSON.parse(stored as string)).toEqual({ + version: 1, + execution: { + tablePrefix: '', + workflowId: 'gated', + runId: 'c-run', + startToken: authoritative.execution.startToken, + }, + owner: { kind: 'agent', id: 'agent-1' }, + principal, + }); + }); + + it('does not record a start whose D1 admission fails', async () => { + const fixture = workflowIngressFixture(true); + await fixture.storage.init(); + const workflows = (await fixture.storage.getStore( + 'workflows', + )) as FencedWorkflowsStorageD1; + const native = workflows[FENCED_WORKFLOW_STORAGE]; + if (!native) throw new Error('missing managed owned workflow capability'); + Object.defineProperty(workflows, FENCED_WORKFLOW_STORAGE, { + value: { + ...native, + withInitialAdmission: () => { + throw new Error('admission failed'); + }, + } satisfies FencedWorkflowAdmissionCapability, + configurable: true, + }); + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).not.toBe(200); + expect( + await workflows.loadWorkflowSnapshot({ + workflowName: 'gated', + runId: 'c-run', + }), + ).toBeNull(); + expect(fixture.journal.values.has('flowsafe:start-principal:v1')).toBe( + false, + ); + }); + + it.each([ + 'not-json', + JSON.stringify({ version: 2 }), + JSON.stringify({ version: 1, principal: REPLACEMENT_PRINCIPAL }), + ])('keeps a conflicting start receipt unchanged (%s)', async (existing) => { + const fixture = workflowIngressFixture(); + fixture.journal.values.set('flowsafe:start-principal:v1', existing); + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).toBe(200); + expect(fixture.journal.values.get('flowsafe:start-principal:v1')).toBe( + existing, + ); + }); + + it('keeps a committed start receipt when its write acknowledgment is lost', async () => { + const fixture = workflowIngressFixture(); + const nativePut = fixture.journal.storage.put.bind(fixture.journal.storage); + vi.spyOn(fixture.journal.storage, 'put').mockImplementation( + async (key, value) => { + await nativePut(key, value); + if (key === 'flowsafe:start-principal:v1') + throw new Error('write acknowledgment lost'); + }, + ); + const logged = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).toBe(200); + expect( + JSON.parse( + fixture.journal.values.get('flowsafe:start-principal:v1') as string, + ), + ).toMatchObject({ + version: 1, + owner: { kind: 'human', id: 'owner-1' }, + principal: OWNER_PRINCIPAL, + }); + } finally { + logged.mockRestore(); + } + }); + + it('starts normally when the principal receipt cannot be stored', async () => { + const fixture = workflowIngressFixture(); + const nativePut = fixture.journal.storage.put.bind(fixture.journal.storage); + vi.spyOn(fixture.journal.storage, 'put').mockImplementation( + async (key, value) => { + if (key === 'flowsafe:start-principal:v1') + throw new Error('receipt storage unavailable'); + await nativePut(key, value); + }, + ); + const response = await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY), + ); + expect(response.status).toBe(200); + expect(fixture.journal.values.has('flowsafe:start-principal:v1')).toBe( + false, + ); + }); + it('starts a queued run after its sender disconnects', async () => { const fixture = workflowIngressFixture(); const hold = await holdWorkflowFifo(fixture); @@ -861,6 +991,9 @@ describe('workflow ingress capture', () => { } const response = await pending; expect(response.status).toBe(200); + expect(fixture.journal.values.has('flowsafe:start-principal:v1')).toBe( + !scheduled, + ); expect(fixture.start).toHaveBeenCalledOnce(); expect(fixture.start.mock.calls[0]?.[0]).toBe('gated'); const options = fixture.start.mock.calls[0]?.[1]; diff --git a/packages/flowsafe/src/do-runner/durable-object.ts b/packages/flowsafe/src/do-runner/durable-object.ts index 13863b80..5af32699 100644 --- a/packages/flowsafe/src/do-runner/durable-object.ts +++ b/packages/flowsafe/src/do-runner/durable-object.ts @@ -183,6 +183,7 @@ export interface DurableObjectRunLifecycleHooks { } const RUN_OWNER_RECOVERY_KEY = 'flowsafe:run-owner-recovery:v1'; +const START_PRINCIPAL_RECEIPT_KEY = 'flowsafe:start-principal:v1'; export const RUN_OWNER_RECOVERY_DELAY_MS = 60_000; const SUSPENSION_DEADLINE_RETRY_MS = 60_000; // How long a run's state may stay unreadable before the entries due under it @@ -336,6 +337,12 @@ function sameRunRecovery( ); } +function preparedRunExecution( + recovery: RunOwnerRecovery, +): RunExecutionIdentity | undefined { + return recovery.phase === 'preparing' ? undefined : recovery.execution; +} + interface StartBody { startReservation?: unknown; workflowId?: string; @@ -1009,6 +1016,39 @@ export abstract class DurableObjectRunner { return prepared; } + async #recordStartPrincipal( + execution: RunExecutionIdentity, + owner: DurableObjectRunOwner, + principal: ExecutionPrincipal, + ): Promise { + const storage = this.state?.storage; + if (!storage) return; + const receipt = JSON.stringify({ + version: 1, + execution: normalizeRunExecutionIdentity(execution), + owner, + principal, + }); + try { + const existing = await storage.get(START_PRINCIPAL_RECEIPT_KEY); + if (existing !== undefined && existing !== receipt) { + console.error('start principal receipt conflicts with prepared start'); + return; + } + if (existing === undefined) { + try { + await storage.put(START_PRINCIPAL_RECEIPT_KEY, receipt); + } catch { + // A lost acknowledgment is resolved by the read below. + } + } + if ((await storage.get(START_PRINCIPAL_RECEIPT_KEY)) !== receipt) + console.error('start principal receipt could not be verified'); + } catch (error) { + console.error('start principal receipt could not be verified', error); + } + } + #matchingRunState( recovery: RunOwnerRecovery, state: AuthoritativeStartState | null, @@ -2590,6 +2630,13 @@ export abstract class DurableObjectRunner { frame.unwound = true; } summary = await this.#finishRunOwner(recovery, summary); + const preparedExecution = preparedRunExecution(recovery); + if (scheduleId === undefined && preparedExecution) + await this.#recordStartPrincipal( + preparedExecution, + owner, + principal, + ); this.#broadcastRunSummary(summary); return json( await this.#withSuspensionTimers(startWorkflowId, summary), From 25a527761ccdbcf8676e86109dc32705ed5c4142 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:40:08 +0400 Subject: [PATCH 38/73] Read workflow start principal through private DO authority Require exact receipt, named object and admitted D1 provenance before returning a full principal to host topology. Keep missing, legacy and mismatched starts opaque and expose no public route. --- .../src/do-runner/durable-object.test.ts | 192 ++++++++++++++++++ .../flowsafe/src/do-runner/durable-object.ts | 87 ++++++++ packages/flowsafe/src/host-kit/do-response.ts | 76 +++++++ .../src/host-kit/do-run-topology.test.ts | 108 ++++++++++ .../flowsafe/src/host-kit/do-run-topology.ts | 20 ++ packages/flowsafe/src/host-kit/index.ts | 6 +- 6 files changed, 488 insertions(+), 1 deletion(-) diff --git a/packages/flowsafe/src/do-runner/durable-object.test.ts b/packages/flowsafe/src/do-runner/durable-object.test.ts index fec6b813..f90ed2a3 100644 --- a/packages/flowsafe/src/do-runner/durable-object.test.ts +++ b/packages/flowsafe/src/do-runner/durable-object.test.ts @@ -637,6 +637,198 @@ async function requireSecondMutationEpoch( } describe('workflow ingress capture', () => { + const principalPath = 'http://do/runs/gated/c-run/start-principal'; + const receiptKey = 'flowsafe:start-principal:v1'; + + function namedReader(fixture: ReturnType) { + const state = { + ...fixture.journal.state, + id: { name: 'gated:c-run' }, + } as DurableObjectState; + return new TestRunner(state, fixture.env); + } + + it('returns the verified full principal from the named run after D1 admission', async () => { + const fixture = workflowIngressFixture(); + const principal: ExecutionPrincipal = { + kind: 'agent', + id: 'agent-1', + purpose: 'customer dispatch', + delegatedBy: 'owner-1', + }; + expect( + ( + await fixture.runner.fetch( + post('/runs', WORKFLOW_START_BODY, principal), + ) + ).status, + ).toBe(200); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(200); + expect(await response.json()).toEqual( + JSON.parse(fixture.journal.values.get(receiptKey) as string), + ); + }); + + it('refuses an unnamed or wrongly named object without reading the receipt', async () => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const read = vi.spyOn(fixture.journal.storage, 'get'); + for (const state of [ + fixture.journal.state, + { ...fixture.journal.state, id: { name: 'gated:other' } }, + ]) { + const response = await new TestRunner( + state as DurableObjectState, + fixture.env, + ).fetch(deploymentIdentityRequest(principalPath)); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + } + expect(read).not.toHaveBeenCalled(); + }); + + it.each([ + ['missing', undefined], + ['malformed JSON', '{'], + ['unknown version', JSON.stringify({ version: 2 })], + ])('refuses a %s receipt without disclosing a principal', async (_case, raw) => { + const fixture = workflowIngressFixture(); + if (raw !== undefined) fixture.journal.values.set(receiptKey, raw); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it.each([ + [ + 'extra field', + (receipt: Record) => ({ ...receipt, extra: true }), + ], + [ + 'changed principal', + (receipt: Record) => ({ + ...receipt, + principal: REPLACEMENT_PRINCIPAL, + }), + ], + [ + 'changed owner', + (receipt: Record) => ({ + ...receipt, + owner: { kind: 'human', id: 'other' }, + }), + ], + [ + 'changed generation', + (receipt: Record) => ({ + ...receipt, + execution: { ...(receipt.execution as object), startToken: 'other' }, + }), + ], + ])('refuses a receipt with %s', async (_case, change) => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const original = JSON.parse( + fixture.journal.values.get(receiptKey) as string, + ); + fixture.journal.values.set(receiptKey, JSON.stringify(change(original))); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it('refuses unreadable D1 state and wrong workflow provenance', async () => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const state = await fixture.runtime.authoritativeStartState( + 'gated', + 'c-run', + ); + if (!state) throw new Error('missing admitted workflow'); + const read = vi.spyOn(fixture.runtime, 'authoritativeStartState'); + for (const result of [ + null, + { + ...state, + execution: { ...state.execution, startToken: 'other-generation' }, + }, + { + ...state, + provenance: { ...state.provenance, startToken: 'other-generation' }, + }, + { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + owner: { kind: 'human', id: 'other-owner' }, + target: { kind: 'workflow', id: 'gated' }, + }, + }, + }, + { + ...state, + provenance: { + ...state.provenance, + startIdentity: { + owner: { kind: 'human', id: 'owner-1' }, + target: { kind: 'agent', id: 'gated', threadId: 'thread-1' }, + }, + }, + }, + ]) { + read.mockResolvedValueOnce(result as typeof state); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + } + read.mockRejectedValueOnce(new Error('D1 unavailable')); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + + it.each([ + 'unfenced', + 'legacy', + ])('refuses a %s authoritative read beside a valid receipt', async (storage) => { + const fixture = workflowIngressFixture(); + expect( + (await fixture.runner.fetch(post('/runs', WORKFLOW_START_BODY))).status, + ).toBe(200); + const state = await fixture.runtime.authoritativeStartState( + 'gated', + 'c-run', + ); + if (!state) throw new Error('missing admitted workflow'); + vi.spyOn(fixture.runtime, 'authoritativeStartState').mockResolvedValueOnce({ + ...state, + storage, + } as typeof state); + const response = await namedReader(fixture).fetch( + deploymentIdentityRequest(principalPath), + ); + expect(response.status).toBe(404); + expect(await response.json()).toEqual({ error: 'run not found' }); + }); + it('persists the trusted principal and prepared identity for a direct start', async () => { const fixture = workflowIngressFixture(); const principal: ExecutionPrincipal = { diff --git a/packages/flowsafe/src/do-runner/durable-object.ts b/packages/flowsafe/src/do-runner/durable-object.ts index 5af32699..2f54da06 100644 --- a/packages/flowsafe/src/do-runner/durable-object.ts +++ b/packages/flowsafe/src/do-runner/durable-object.ts @@ -1049,6 +1049,82 @@ export abstract class DurableObjectRunner { } } + async #startPrincipalReceipt( + workflowId: string, + runId: string, + ): Promise { + const unavailable = () => json({ error: 'run not found' }, 404); + const storage = this.state?.storage; + if (!storage || this.state?.id?.name !== `${workflowId}:${runId}`) + return unavailable(); + try { + const raw = await storage.get(START_PRINCIPAL_RECEIPT_KEY); + if (typeof raw !== 'string') return unavailable(); + const parsed: unknown = JSON.parse(raw); + if ( + parsed === null || + typeof parsed !== 'object' || + Array.isArray(parsed) + ) + return unavailable(); + const receipt = parsed as Record; + if ( + receipt.version !== 1 || + receipt.owner === null || + typeof receipt.owner !== 'object' || + Array.isArray(receipt.owner) || + receipt.principal === null || + typeof receipt.principal !== 'object' || + Array.isArray(receipt.principal) + ) + return unavailable(); + const execution = normalizeD1RunExecutionIdentity(receipt.execution); + const owner = receipt.owner as Record; + const principal = decodeExecutionPrincipal( + JSON.stringify(receipt.principal), + ); + if ( + !isExecutionPrincipalKind(owner.kind) || + !isExecutionPrincipalId(owner.id) || + !principal || + execution.workflowId !== workflowId || + execution.runId !== runId || + owner.kind !== principal.kind || + owner.id !== principal.id + ) + return unavailable(); + const canonical = { + version: 1, + execution, + owner: { kind: owner.kind, id: owner.id }, + principal, + }; + if (JSON.stringify(canonical) !== raw) return unavailable(); + const state = await this.#ensureRuntime().authoritativeStartState( + workflowId, + runId, + ); + const identity = state?.provenance.startIdentity; + if ( + state?.storage !== 'd1' || + state.execution.tablePrefix !== execution.tablePrefix || + state.execution.workflowId !== execution.workflowId || + state.execution.runId !== execution.runId || + state.execution.startToken !== execution.startToken || + state.provenance.startToken !== execution.startToken || + identity?.target.kind !== 'workflow' || + identity.target.id !== workflowId || + identity.owner.kind !== owner.kind || + identity.owner.id !== owner.id || + state.provenance.agentStart !== undefined + ) + return unavailable(); + return json(canonical); + } catch { + return unavailable(); + } + } + #matchingRunState( recovery: RunOwnerRecovery, state: AuthoritativeStartState | null, @@ -2686,6 +2762,17 @@ export abstract class DurableObjectRunner { this.#assertRunIdentity(workflowId, runId); return json({ live: this.#isStartLive(workflowId, runId) }); } + if ( + request.method === 'GET' && + segments.length === 4 && + action === 'start-principal' && + workflowId && + runId + ) { + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) + return json({ error: 'run not found' }, 404); + return this.#startPrincipalReceipt(workflowId, runId); + } if ( request.method === 'GET' && segments.length === 3 && diff --git a/packages/flowsafe/src/host-kit/do-response.ts b/packages/flowsafe/src/host-kit/do-response.ts index e0a7af0b..823a8764 100644 --- a/packages/flowsafe/src/host-kit/do-response.ts +++ b/packages/flowsafe/src/host-kit/do-response.ts @@ -7,11 +7,15 @@ // router's error mapping rather than collapse into a generic 500. import { + assertExecutionPrincipal, + type ExecutionPrincipal, isExecutionPrincipalId, isExecutionPrincipalKind, } from '../approval-api/principal.js'; import { isPublishedRefusal } from '../do-runner/do-status-error.js'; import { + type D1RunExecutionIdentity, + normalizeD1RunExecutionIdentity, normalizeStartExecutionIdentity, type StartExecutionIdentity, } from '../do-runner/execution-admission.js'; @@ -315,3 +319,75 @@ export async function doPersistedStart( throw new RunRouteError(503, 'persisted start is not readable'); } } + +export interface StartPrincipalReceipt { + readonly version: 1; + readonly execution: D1RunExecutionIdentity; + readonly owner: { + readonly kind: ExecutionPrincipal['kind']; + readonly id: string; + }; + readonly principal: ExecutionPrincipal; +} + +/** @internal A private DO receipt is trusted only after exact selector and owner checks. */ +export async function doStartPrincipal( + response: DoResponseLike, + expected: { workflowId: string; runId: string }, +): Promise { + if (!response.ok) { + throw await objectAnswerError('run-object', response, 'run'); + } + try { + if (response.status !== 200) throw new Error('invalid status'); + const payload = persistedStartRecord(await response.json()); + if ( + Object.keys(payload).sort().join(',') !== + 'execution,owner,principal,version' || + payload.version !== 1 + ) + throw new Error('invalid receipt'); + const rawExecution = persistedStartRecord(payload.execution); + if ( + Object.keys(rawExecution).sort().join(',') !== + 'runId,startToken,tablePrefix,workflowId' + ) + throw new Error('invalid execution'); + const execution = normalizeD1RunExecutionIdentity(rawExecution); + if ( + rawExecution.tablePrefix !== execution.tablePrefix || + execution.workflowId !== expected.workflowId || + execution.runId !== expected.runId + ) + throw new Error('selector mismatch'); + const owner = persistedStartRecord(payload.owner); + if ( + Object.keys(owner).sort().join(',') !== 'id,kind' || + !isExecutionPrincipalKind(owner.kind) || + !isExecutionPrincipalId(owner.id) + ) + throw new Error('invalid owner'); + const rawPrincipal = persistedStartRecord(payload.principal); + const principal = assertExecutionPrincipal(rawPrincipal, 'start receipt'); + const principalKeys = + principal.kind === 'human' + ? 'id,kind,role' + : principal.kind === 'agent' && principal.delegatedBy !== undefined + ? 'delegatedBy,id,kind,purpose' + : 'id,kind,purpose'; + if ( + Object.keys(rawPrincipal).sort().join(',') !== principalKeys || + principal.kind !== owner.kind || + principal.id !== owner.id + ) + throw new Error('principal mismatch'); + return Object.freeze({ + version: 1, + execution, + owner: Object.freeze({ kind: owner.kind, id: owner.id }), + principal, + }); + } catch { + throw new RunRouteError(503, 'run start principal is not readable'); + } +} diff --git a/packages/flowsafe/src/host-kit/do-run-topology.test.ts b/packages/flowsafe/src/host-kit/do-run-topology.test.ts index 54a099a3..75d64707 100644 --- a/packages/flowsafe/src/host-kit/do-run-topology.test.ts +++ b/packages/flowsafe/src/host-kit/do-run-topology.test.ts @@ -726,3 +726,111 @@ describe('protected replay workflow transport', () => { ); }); }); + +describe('private start-principal receipt transport', () => { + const receipt = { + version: 1, + execution: { + tablePrefix: 'tenant_', + workflowId: 'workflow-1', + runId: 'run-1', + startToken: 'generation-1', + }, + owner: { kind: 'agent', id: 'agent-1' }, + principal: { + kind: 'agent', + id: 'agent-1', + purpose: 'customer.dispatch', + delegatedBy: 'operator-1', + }, + }; + + function topologyFor(payload: unknown, status = 200) { + const fetch = vi.fn( + async ( + _url: string, + _init?: Parameters< + ReturnType['get']>['fetch'] + >[1], + ) => Response.json(payload, { status }), + ); + const idFromName = vi.fn((name: string) => name); + return { + fetch, + idFromName, + topology: createDoRunTopology( + { idFromName, get: () => ({ fetch }) }, + DEPLOYMENT_IDENTITY_SECRET, + ), + }; + } + + it('reads the exact run receipt over the authenticated private DO route', async () => { + const { topology, fetch, idFromName } = topologyFor(receipt); + await expect( + topology.startPrincipal('workflow-1', 'run-1'), + ).resolves.toEqual(receipt); + expect(idFromName).toHaveBeenCalledWith('workflow-1:run-1'); + expect(fetch).toHaveBeenCalledWith( + 'http://do/runs/workflow-1/run-1/start-principal', + expect.objectContaining({ headers: expect.any(Object) }), + ); + const headers = new Headers(fetch.mock.calls[0]?.[1]?.headers); + expect(headers.get('x-flowsafe-deployment-identity')).toBe( + DEPLOYMENT_IDENTITY_SECRET, + ); + expect(headers.has(EXECUTION_PRINCIPAL_HEADER)).toBe(false); + }); + + it('treats a missing receipt as absence and refuses path-unsafe selectors locally', async () => { + await expect( + topologyFor({ error: 'run not found' }, 404).topology.startPrincipal( + 'workflow-1', + 'run-1', + ), + ).resolves.toBeUndefined(); + const { topology, fetch, idFromName } = topologyFor(receipt); + await expect( + topology.startPrincipal('workflow/1', 'run-1'), + ).rejects.toMatchObject({ + status: 400, + }); + expect(idFromName).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }); + + it.each([ + [ + 'foreign run', + { ...receipt, execution: { ...receipt.execution, runId: 'other' } }, + ], + [ + 'foreign workflow', + { ...receipt, execution: { ...receipt.execution, workflowId: 'other' } }, + ], + ['owner mismatch', { ...receipt, owner: { kind: 'agent', id: 'other' } }], + ['unknown version', { ...receipt, version: 2 }], + [ + 'noncanonical prefix', + { + ...receipt, + execution: { ...receipt.execution, tablePrefix: 'TENANT_' }, + }, + ], + [ + 'unknown authority field', + { ...receipt, callerPrincipal: receipt.principal }, + ], + [ + 'malformed principal', + { ...receipt, principal: { ...receipt.principal, purpose: '' } }, + ], + ])('refuses a %s private receipt', async (_label, payload) => { + await expect( + topologyFor(payload).topology.startPrincipal('workflow-1', 'run-1'), + ).rejects.toMatchObject({ + status: 503, + message: 'run start principal is not readable', + }); + }); +}); diff --git a/packages/flowsafe/src/host-kit/do-run-topology.ts b/packages/flowsafe/src/host-kit/do-run-topology.ts index 34c0b8c5..6f2c25ca 100644 --- a/packages/flowsafe/src/host-kit/do-run-topology.ts +++ b/packages/flowsafe/src/host-kit/do-run-topology.ts @@ -25,14 +25,18 @@ import { type RunSummary, stampMutationEpoch, } from '../do-runner/index.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; import type { PersistedStartResult } from '../do-runner/start-idempotency.js'; import { captureReservation } from '../do-runner/start-reservation-contract.js'; import { type DoResponseLike, doPersistedStart, doStartLiveness, + doStartPrincipal, doSummary, + type StartPrincipalReceipt, } from './do-response.js'; +import { RunRouteError } from './run-route-error.js'; import type { RunStartInput } from './run-router.js'; /** The subset of a DurableObjectStub the topology uses. */ @@ -91,6 +95,11 @@ export interface DoRunTopology { /** Additive run-lifecycle operations returned by createDoRunTopology. */ export interface DoRunLifecycleTopology extends DoRunTopology { + /** Reads the authenticated run object's stored start authority. */ + startPrincipal( + workflowId: string, + runId: string, + ): Promise; terminate( workflowId: string, runId: string, @@ -146,6 +155,17 @@ export function createDoRunTopology( ); return { + startPrincipal: async (workflowId, runId) => { + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) { + throw new RunRouteError(400, 'invalid run identity'); + } + const response = await stub(workflowId, runId).fetch( + `http://do/runs/${workflowId}/${runId}/start-principal`, + { headers: deploymentIdentityHeaders(deploymentIdentitySecret) }, + ); + if (response.status === 404) return undefined; + return doStartPrincipal(response, { workflowId, runId }); + }, start: async ({ workflowId, runId, diff --git a/packages/flowsafe/src/host-kit/index.ts b/packages/flowsafe/src/host-kit/index.ts index 9bf1bc96..354cfdfc 100644 --- a/packages/flowsafe/src/host-kit/index.ts +++ b/packages/flowsafe/src/host-kit/index.ts @@ -74,7 +74,11 @@ export { resumeRunWithRequeue, } from './approval-bridge.js'; export { bearerActorAuthenticator, parseActorTokens } from './bearer-auth.js'; -export { type DoResponseLike, doSummary } from './do-response.js'; +export { + type DoResponseLike, + doSummary, + type StartPrincipalReceipt, +} from './do-response.js'; export { createDoRunTopology, type DoRunLifecycleTopology, From c5d946439b9084cc5b66b0b27512f46e3b0e7a82 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:02:13 +0400 Subject: [PATCH 39/73] Add authenticated private workflow remote-step route Authenticate a non-delegating host credential before invocation reads, bind registration to the verified start principal and later operations to the stored invocation. Keep the route opt-in with no ASP caller or general resume. --- packages/flowsafe/src/agent-host/index.ts | 4 + .../src/agent-host/thread-topology.test.ts | 21 +- .../agent-host/workflow-remote-step-route.ts | 193 ++++++++++++++++ .../src/host-kit/flowsafe-worker.test.ts | 211 ++++++++++++++++++ .../flowsafe/src/host-kit/flowsafe-worker.ts | 85 ++++++- 5 files changed, 504 insertions(+), 10 deletions(-) create mode 100644 packages/flowsafe/src/agent-host/workflow-remote-step-route.ts diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index 38397f37..ce0b547c 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -81,3 +81,7 @@ export type { TrustedAgentExecution, } from './types.js'; export { isPermissionIdentifier } from './types.js'; +export { + createWorkflowRemoteStepPreRoute, + WORKFLOW_REMOTE_STEP_ROUTE, +} from './workflow-remote-step-route.js'; diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 77843730..977183d8 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -191,6 +191,10 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', }); expect(ticket).not.toHaveProperty('principal'); expect(ticket).not.toHaveProperty('startToken'); + expect(reader.issuePermit).toHaveBeenCalledWith( + invocation.invocationId, + 60_000, + ); expect( await topology.getWorkflowRemoteStep(owner, invocation.invocationId), ).toEqual(dispatched); @@ -200,6 +204,8 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', expect( await topology.consumeWorkflowRemoteStepDispatch(owner, permit), ).toEqual({ kind: 'already-consumed' }); + expect(reader.consumePermit).toHaveBeenNthCalledWith(1, permit); + expect(reader.consumePermit).toHaveBeenNthCalledWith(2, permit); expect( await topology.recordWorkflowRemoteStepResult( owner, @@ -258,6 +264,19 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', 60_000, ), ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.getWorkflow) + .mockResolvedValueOnce(dispatched) + .mockResolvedValueOnce({ + ...dispatched, + permitGeneration: permit.generation + 1, + }); + await expect( + topology.issueWorkflowRemoteStepDispatch( + owner, + invocation.invocationId, + 60_000, + ), + ).rejects.toMatchObject({ status: 409 }); vi.mocked(reader.getWorkflow).mockResolvedValueOnce({ ...dispatched, deploymentId: 'another-deployment', @@ -269,7 +288,7 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', 60_000, ), ).rejects.toMatchObject({ status: 404 }); - expect(reader.issuePermit).toHaveBeenCalledTimes(1); + expect(reader.issuePermit).toHaveBeenCalledTimes(2); expect(reader.consumePermit).toHaveBeenCalledTimes(2); expect(reader.recordResult).toHaveBeenCalledTimes(1); }); diff --git a/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts b/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts new file mode 100644 index 00000000..a07a7877 --- /dev/null +++ b/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts @@ -0,0 +1,193 @@ +// SPDX-License-Identifier: Apache-2.0 + +import { doErrorResponse } from '../do-runner/do-error-response.js'; +import { isPathSafeId } from '../do-runner/path-safe-id.js'; +import type { RemoteStepPermit } from '../do-runner/remote-step-journal.js'; +import type { + FlowsafeWorkerConfig, + FlowsafeWorkerEnv, +} from '../host-kit/flowsafe-worker.js'; +import { + RunRouteError, + runRouteErrorBody, +} from '../host-kit/run-route-error.js'; +import { readBoundedBody } from '../http-body.js'; +import type { + AgentThreadDispatchTopology, + WorkflowRemoteStepDispatchReader, +} from './thread-topology.js'; + +export const WORKFLOW_REMOTE_STEP_ROUTE = '/_flowsafe/workflow-remote-step'; + +type WorkflowDispatchPort = Pick< + AgentThreadDispatchTopology, + | 'registerWorkflowRemoteStep' + | 'issueWorkflowRemoteStepDispatch' + | 'consumeWorkflowRemoteStepDispatch' + | 'recordWorkflowRemoteStepResult' +>; + +type Operation = 'register' | 'ticket' | 'consume' | 'result'; + +const fields: Record = { + register: ['workflowId', 'runId', 'hostDeadlineAt'], + ticket: ['invocationId', 'lifetimeMs'], + consume: ['permit'], + result: ['invocationId', 'rawPermit', 'resultJson', 'evidenceJson'], +}; + +function json(value: unknown, status = 200): Response { + return Response.json(value, { + status, + headers: { 'cache-control': 'no-store' }, + }); +} + +function exactBody( + value: unknown, + operation: Operation, +): Record | undefined { + if (value === null || typeof value !== 'object' || Array.isArray(value)) + return undefined; + const body = value as Record; + const keys = Object.keys(body); + const expected = fields[operation]; + return keys.length === expected.length && + keys.every((key) => expected.includes(key)) + ? body + : undefined; +} + +function permit(value: unknown): RemoteStepPermit | undefined { + if (value === null || typeof value !== 'object' || Array.isArray(value)) + return undefined; + const candidate = value as Record; + if ( + Object.keys(candidate).sort().join(',') !== + 'expiresAt,generation,invocationId,mutationEpoch,transitionRevision' || + typeof candidate.invocationId !== 'string' || + !isPathSafeId(candidate.invocationId) || + !['generation', 'mutationEpoch', 'transitionRevision', 'expiresAt'].every( + (key) => + Number.isSafeInteger(candidate[key]) && (candidate[key] as number) >= 0, + ) + ) + return undefined; + return candidate as unknown as RemoteStepPermit; +} + +/** A host mounts this route through preRoutes on its private state-script Worker. */ +export function createWorkflowRemoteStepPreRoute( + journal: WorkflowRemoteStepDispatchReader, + dispatch: WorkflowDispatchPort, +): NonNullable['preRoutes']> { + return async (request, _env, _ctx, kit) => { + const url = new URL(request.url); + if (!url.pathname.startsWith(`${WORKFLOW_REMOTE_STEP_ROUTE}/`)) return null; + const operation = url.pathname.slice(WORKFLOW_REMOTE_STEP_ROUTE.length + 1); + if (!Object.hasOwn(fields, operation) || url.search) + return json({ error: 'not found' }, 404); + if (request.method !== 'POST') + return json({ error: 'method not allowed' }, 405); + const unauthorized = await kit.authorizePrivateWorkflowDispatch(request); + if (unauthorized) return unauthorized; + const raw = await readBoundedBody(request, 65_536); + if (!raw.ok) + return json( + { error: 'invalid request body' }, + raw.reason === 'payload-too-large' ? 413 : 400, + ); + let parsed: unknown; + try { + parsed = JSON.parse(raw.text); + } catch { + return json({ error: 'invalid request body' }, 400); + } + const body = exactBody(parsed, operation as Operation); + if (!body) return json({ error: 'invalid request body' }, 400); + try { + if (operation === 'register') { + if ( + typeof body.workflowId !== 'string' || + !isPathSafeId(body.workflowId) || + typeof body.runId !== 'string' || + !isPathSafeId(body.runId) || + !Number.isSafeInteger(body.hostDeadlineAt) || + (body.hostDeadlineAt as number) <= 0 + ) + return json({ error: 'invalid request body' }, 400); + const context = await kit.contextForStoredWorkflowStart( + body.workflowId, + body.runId, + ); + if (!context) return json({ error: 'run not found' }, 404); + const invocation = await dispatch.registerWorkflowRemoteStep( + context, + body.workflowId, + body.runId, + body.hostDeadlineAt as number, + ); + return json({ invocationId: invocation.invocationId }); + } + const invocationId = + operation === 'consume' + ? permit(body.permit)?.invocationId + : body.invocationId; + if (typeof invocationId !== 'string' || !isPathSafeId(invocationId)) + return json({ error: 'invalid request body' }, 400); + const context = await kit.contextForStoredWorkflowInvocation( + journal, + invocationId, + ); + if (!context) return json({ error: 'invocation not found' }, 404); + if (operation === 'ticket') { + if ( + !Number.isSafeInteger(body.lifetimeMs) || + (body.lifetimeMs as number) <= 0 + ) + return json({ error: 'invalid request body' }, 400); + return json( + await dispatch.issueWorkflowRemoteStepDispatch( + context, + invocationId, + body.lifetimeMs as number, + ), + ); + } + if (operation === 'consume') { + const submitted = permit(body.permit); + if (!submitted) return json({ error: 'invalid request body' }, 400); + return json( + await dispatch.consumeWorkflowRemoteStepDispatch(context, submitted), + ); + } + if ( + typeof body.rawPermit !== 'string' || + typeof body.resultJson !== 'string' || + typeof body.evidenceJson !== 'string' + ) + return json({ error: 'invalid request body' }, 400); + return json( + await dispatch.recordWorkflowRemoteStepResult( + context, + invocationId, + body.rawPermit, + body.resultJson, + body.evidenceJson, + ), + ); + } catch (error) { + if (error instanceof RunRouteError) { + const refusal = runRouteErrorBody(error); + return refusal + ? json(refusal, error.status) + : json({ error: 'private workflow dispatch failed' }, 500); + } + const response = doErrorResponse(error); + if (response.status === 500) + return json({ error: 'private workflow dispatch failed' }, 500); + response.headers.set('cache-control', 'no-store'); + return response; + } + }; +} diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts index cffa6ec9..9f75ea3f 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.test.ts @@ -3,6 +3,11 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; +import type { WorkflowRemoteStepDispatchReader } from '../agent-host/thread-topology.js'; +import { + createWorkflowRemoteStepPreRoute, + WORKFLOW_REMOTE_STEP_ROUTE, +} from '../agent-host/workflow-remote-step-route.js'; import type { ApprovalNotificationEvent, ApprovalRecord, @@ -936,6 +941,212 @@ describe('createFlowsafeWorker fetch pipeline', () => { expect(await list.json()).toEqual([]); }); + it('authenticates private workflow dispatch before reading stored authority', async () => { + const { env, ctx } = makeEnv(); + const startRead = vi.fn(async () => + Response.json({ + version: 1, + execution: { + tablePrefix: '', + workflowId: 'wf', + runId: 'run-1', + startToken: 'start-1', + }, + owner: { kind: 'human', id: 'ada' }, + principal: { kind: 'human', id: 'ada', role: 'operator' }, + }), + ); + env.RUNNER = { + idFromName: (name: string) => name, + get: () => ({ fetch: startRead }), + }; + const invocation = { + targetKind: 'workflow' as const, + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + tablePrefix: '', + workflowId: 'wf', + runId: 'run-1', + principal: { + kind: 'human' as const, + id: 'ada', + role: 'operator' as const, + }, + startToken: 'start-1', + stepPath: ['remote-step'], + suspendedAt: 1000, + request: { script: 'script-1' }, + requestDigest: 'stored-request-digest', + deadlineAt: 9000, + status: 'pending' as const, + permitGeneration: 0, + }; + const getWorkflow = vi.fn(async () => invocation); + const journal = { + deploymentId: 'deployment-1', + tablePrefix: '', + getWorkflow, + } as unknown as WorkflowRemoteStepDispatchReader; + const registerWorkflowRemoteStep = vi.fn(async () => invocation); + const issueWorkflowRemoteStepDispatch = vi.fn(async () => ({ + invocationId: 'invocation-1', + deploymentId: 'deployment-1', + workflowId: 'wf', + runId: 'run-1', + stepPath: ['remote-step'], + suspendedAt: 1000, + deadlineAt: 9000, + requestJson: '{"script":"script-1"}', + requestDigest: 'stored-request-digest', + permit: { + invocationId: 'invocation-1', + generation: 1, + mutationEpoch: 1, + transitionRevision: 1, + expiresAt: 9000, + }, + })); + const consumeWorkflowRemoteStepDispatch = vi.fn(async () => ({ + kind: 'accepted' as const, + rawPermit: 'raw-secret', + })); + const recordWorkflowRemoteStepResult = vi.fn(async () => ({ + kind: 'recorded' as const, + resultJson: '{"answer":42}', + resultDigest: 'stored-result-digest', + })); + const worker = makeWorker({ + preRoutes: createWorkflowRemoteStepPreRoute(journal, { + registerWorkflowRemoteStep, + issueWorkflowRemoteStepDispatch, + consumeWorkflowRemoteStepDispatch, + recordWorkflowRemoteStepResult, + }), + }); + const request = (operation: string, body: unknown, credential?: string) => + new Request(`http://host${WORKFLOW_REMOTE_STEP_ROUTE}/${operation}`, { + method: 'POST', + headers: { + 'content-type': 'application/json', + ...(credential ? { authorization: `Bearer ${credential}` } : {}), + }, + body: JSON.stringify(body), + }); + const register = { workflowId: 'wf', runId: 'run-1', hostDeadlineAt: 9000 }; + expect( + (await worker.fetch(request('register', register), env, ctx)).status, + ).toBe(503); + env.MAINTENANCE_ADMIN_SECRET = 'private-admin-secret-00000000000001'; + expect( + (await worker.fetch(request('register', register, 'wrong'), env, ctx)) + .status, + ).toBe(401); + expect(startRead).not.toHaveBeenCalled(); + expect(getWorkflow).not.toHaveBeenCalled(); + const token = env.MAINTENANCE_ADMIN_SECRET; + expect( + ( + await worker.fetch( + request( + 'register', + { ...register, principal: invocation.principal }, + token, + ), + env, + ctx, + ) + ).status, + ).toBe(400); + expect(startRead).not.toHaveBeenCalled(); + const registered = await worker.fetch( + request('register', register, token), + env, + ctx, + ); + expect(registered.status).toBe(200); + expect(await registered.json()).toEqual({ invocationId: 'invocation-1' }); + expect(registerWorkflowRemoteStep).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + 'wf', + 'run-1', + 9000, + ); + const ticket = await worker.fetch( + request( + 'ticket', + { invocationId: 'invocation-1', lifetimeMs: 60000 }, + token, + ), + env, + ctx, + ); + expect(ticket.status).toBe(200); + expect(issueWorkflowRemoteStepDispatch).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + 'invocation-1', + 60000, + ); + const issued = (await ticket.json()) as { + permit: { + invocationId: string; + generation: number; + mutationEpoch: number; + transitionRevision: number; + expiresAt: number; + }; + }; + const consumed = await worker.fetch( + request('consume', { permit: issued.permit }, token), + env, + ctx, + ); + expect(consumed.status).toBe(200); + expect(await consumed.json()).toEqual({ + kind: 'accepted', + rawPermit: 'raw-secret', + }); + expect(consumed.headers.get('cache-control')).toBe('no-store'); + expect(consumeWorkflowRemoteStepDispatch).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + issued.permit, + ); + const result = await worker.fetch( + request( + 'result', + { + invocationId: 'invocation-1', + rawPermit: 'raw-secret', + resultJson: '{"answer":42}', + evidenceJson: '{"quiescent":true}', + }, + token, + ), + env, + ctx, + ); + expect(result.status).toBe(200); + expect(recordWorkflowRemoteStepResult).toHaveBeenCalledWith( + expect.objectContaining({ principal: invocation.principal }), + 'invocation-1', + 'raw-secret', + '{"answer":42}', + '{"quiescent":true}', + ); + issueWorkflowRemoteStepDispatch.mockRejectedValueOnce( + new RunRouteError(409, 'remote step dispatch changed'), + ); + const changed = await worker.fetch( + request( + 'ticket', + { invocationId: 'invocation-1', lifetimeMs: 60000 }, + token, + ), + env, + ctx, + ); + expect(changed.status).toBe(409); + }); + it('binds a pre-route context to the stored invocation principal', async () => { const journal = { deploymentId: 'deployment-1', diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.ts index 8fbcc23d..75b1dd87 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.ts @@ -394,6 +394,29 @@ export interface FlowsafeWorkerConfig kit: { resolve: ActorResolver; topology: DoRunTopology; + authorizePrivateWorkflowDispatch: ( + request: Request, + ) => Promise; + contextForStoredWorkflowStart: ( + workflowId: string, + runId: string, + ) => Promise; + contextForStoredWorkflowInvocation: ( + journal: { + deploymentId: string; + tablePrefix: string; + getWorkflow: (invocationId: string) => Promise< + | { + invocationId: string; + deploymentId: string; + tablePrefix: string; + principal: ExecutionPrincipal; + } + | undefined + >; + }, + invocationId: string, + ) => Promise; contextForStoredInvocation: ( journal: { deploymentId: string; @@ -1649,23 +1672,67 @@ export function createFlowsafeWorker( ); if (config.preRoutes) { + const contextForStoredInvocation = async ( + journal: { + deploymentId: string; + tablePrefix: string; + get: (invocationId: string) => Promise< + | { + invocationId: string; + deploymentId: string; + tablePrefix: string; + principal: ExecutionPrincipal; + } + | undefined + >; + }, + invocationId: string, + ) => { + if (!isPathSafeId(invocationId)) return undefined; + const invocation = await journal.get(invocationId); + if ( + !invocation || + invocation.invocationId !== invocationId || + invocation.deploymentId !== journal.deploymentId || + invocation.tablePrefix !== journal.tablePrefix || + invocation.tablePrefix !== + (storageTablePrefix ?? '').toLowerCase() + ) + return undefined; + return contextForPrincipal(invocation.principal); + }; const preResponse = await config.preRoutes(request, env, ctx, { resolve, topology, - contextForStoredInvocation: async (journal, invocationId) => { - if (!isPathSafeId(invocationId)) return undefined; - const invocation = await journal.get(invocationId); + authorizePrivateWorkflowDispatch: async (privateRequest) => { + const gate = await authorizeAdminCredential(privateRequest, env, { + surface: 'private workflow dispatch', + delegateCapability: false, + }); + return gate.authorized ? undefined : gate.response; + }, + contextForStoredWorkflowStart: async (workflowId, runId) => { + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) + return undefined; + const receipt = await topology.startPrincipal(workflowId, runId); if ( - !invocation || - invocation.invocationId !== invocationId || - invocation.deploymentId !== journal.deploymentId || - invocation.tablePrefix !== journal.tablePrefix || - invocation.tablePrefix !== + !receipt || + receipt.execution.tablePrefix !== (storageTablePrefix ?? '').toLowerCase() ) return undefined; - return contextForPrincipal(invocation.principal); + return contextForPrincipal(receipt.principal); }, + contextForStoredWorkflowInvocation: (journal, invocationId) => + contextForStoredInvocation( + { + deploymentId: journal.deploymentId, + tablePrefix: journal.tablePrefix, + get: journal.getWorkflow.bind(journal), + }, + invocationId, + ), + contextForStoredInvocation, }); if (preResponse) return preResponse; } From 785ef555411ae06bef9d1a18bf94f2e5cdfb1dea Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:45:09 +0400 Subject: [PATCH 40/73] Prove notification receipt contention in native D1 Synchronize two Durable Object actors after real missing-receipt reads, then run the unchanged atomic batch against shared D1. Verify one effect, last-slot cap and exact replay. --- .../src/cloudflare-fidelity.workerd.test.ts | 173 ++++++++++++++++++ .../test-support/cloudflare-test-worker.ts | 161 +++++++++++++++- 2 files changed, 331 insertions(+), 3 deletions(-) diff --git a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts index 1b17de66..9cc15365 100644 --- a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts +++ b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts @@ -13,13 +13,41 @@ import { readDeploymentIdentity, seedDeploymentIdentity, } from './do-runner/deployment-identity.js'; +import { D1NotificationsStorage } from './signals/notifications-d1.js'; interface TestBindings { DB: D1Database; + SIGNAL_PROVIDER_HOST: DurableObjectNamespace; } const db = (): D1Database => (env as unknown as TestBindings).DB; +function receiptActor(name: string) { + const namespace = (env as unknown as TestBindings).SIGNAL_PROVIDER_HOST; + return namespace.get(namespace.idFromName(name)); +} + +function createReceipt( + actor: ReturnType, + operationKey: string, + notificationId: string, + receiptLimit: number, + barrier?: { id: string; participant: string }, +) { + return actor.fetch( + new Request('http://localhost/receipt', { + method: 'POST', + body: JSON.stringify({ + operationKey, + notificationId, + receiptLimit, + barrierId: barrier?.id, + participant: barrier?.participant, + }), + }), + ); +} + describe('FlowSafe Cloudflare fidelity', () => { beforeEach(reset); @@ -60,4 +88,149 @@ describe('FlowSafe Cloudflare fidelity', () => { expect(winners).toHaveLength(1); expect(await store.owner('run', 'run-cas')).toEqual(winners[0]?.owner); }); + + it('commits one notification receipt and effect when distinct durable objects submit the same operation', async () => { + await new D1NotificationsStorage(db()).init(); + await db() + .prepare(`CREATE TABLE notification_test_barrier ( + barrier_id TEXT NOT NULL, + participant TEXT NOT NULL, + PRIMARY KEY (barrier_id, participant) + )`) + .run(); + const left = receiptActor('receipt-left'); + const right = receiptActor('receipt-right'); + const responses = await Promise.all([ + createReceipt(left, 'operation-1', 'notification-1', 1, { + id: 'same-key', + participant: 'left', + }), + createReceipt(right, 'operation-1', 'notification-1', 1, { + id: 'same-key', + participant: 'right', + }), + ]); + + const ready = await db() + .prepare( + "SELECT participant FROM notification_test_barrier WHERE barrier_id = 'same-key' ORDER BY participant", + ) + .all<{ participant: string }>(); + expect(ready.results).toEqual([ + { participant: 'left' }, + { participant: 'right' }, + ]); + + expect(responses.map((response) => response.status)).toEqual([200, 200]); + expect( + await Promise.all(responses.map((response) => response.json())), + ).toEqual([ + { + operationKey: 'operation-1', + notificationId: 'notification-1', + threadId: 'thread', + }, + { + operationKey: 'operation-1', + notificationId: 'notification-1', + threadId: 'thread', + }, + ]); + const receipts = await db() + .prepare( + 'SELECT operation_key, notification_id FROM flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string; notification_id: string }>(); + expect(receipts.results).toEqual([ + { operation_key: 'operation-1', notification_id: 'notification-1' }, + ]); + const inbox = await db() + .prepare('SELECT id, thread_id FROM mastra_notifications') + .all<{ id: string; thread_id: string }>(); + expect(inbox.results).toEqual([ + { id: 'notification-1', thread_id: 'thread' }, + ]); + }); + + it('admits one distinct durable object operation at the last receipt slot', async () => { + await new D1NotificationsStorage(db()).init(); + await db() + .prepare(`CREATE TABLE notification_test_barrier ( + barrier_id TEXT NOT NULL, + participant TEXT NOT NULL, + PRIMARY KEY (barrier_id, participant) + )`) + .run(); + await db() + .prepare(`WITH RECURSIVE sequence(n) AS ( + VALUES (1) UNION ALL SELECT n + 1 FROM sequence WHERE n < 99999 + ) + INSERT INTO flowsafe_notification_operation_receipts + (operation_key, request_digest, thread_id, notification_id) + SELECT 'seed-' || n, 'seed', 'seed-thread', 'seed-' || n FROM sequence`) + .run(); + const left = receiptActor('slot-left'); + const right = receiptActor('slot-right'); + const responses = await Promise.all([ + createReceipt(left, 'operation-1', 'notification-1', 100_000, { + id: 'last-slot', + participant: 'left', + }), + createReceipt(right, 'operation-2', 'notification-2', 100_000, { + id: 'last-slot', + participant: 'right', + }), + ]); + + const ready = await db() + .prepare( + "SELECT participant FROM notification_test_barrier WHERE barrier_id = 'last-slot' ORDER BY participant", + ) + .all<{ participant: string }>(); + expect(ready.results).toEqual([ + { participant: 'left' }, + { participant: 'right' }, + ]); + + expect(responses.map((response) => response.status).sort()).toEqual([ + 200, 409, + ]); + const receipts = await db() + .prepare(`SELECT operation_key, notification_id + FROM flowsafe_notification_operation_receipts + WHERE operation_key IN ('operation-1', 'operation-2')`) + .all<{ operation_key: string; notification_id: string }>(); + const inbox = await db() + .prepare('SELECT id FROM mastra_notifications') + .all<{ id: string }>(); + expect([ + [{ operation_key: 'operation-1', notification_id: 'notification-1' }], + [{ operation_key: 'operation-2', notification_id: 'notification-2' }], + ]).toContainEqual(receipts.results); + const winner = receipts.results[0]; + if (!winner) throw new Error('receipt is missing'); + expect(inbox.results).toEqual([{ id: winner.notification_id }]); + const count = await db() + .prepare( + 'SELECT COUNT(*) AS total FROM flowsafe_notification_operation_receipts', + ) + .first<{ total: number }>(); + expect(count?.total).toBe(100_000); + const loserResponse = responses.find((response) => response.status === 409); + expect(await loserResponse?.json()).toEqual({ + error: 'notification operation receipt cap reached', + }); + const replay = await createReceipt( + winner.operation_key === 'operation-1' ? right : left, + winner.operation_key, + winner.notification_id, + 100_000, + ); + expect(replay.status).toBe(200); + expect(await replay.json()).toEqual({ + operationKey: winner.operation_key, + notificationId: winner.notification_id, + threadId: 'thread', + }); + }, 30_000); }); diff --git a/packages/flowsafe/test-support/cloudflare-test-worker.ts b/packages/flowsafe/test-support/cloudflare-test-worker.ts index bc46ee9a..23171b69 100644 --- a/packages/flowsafe/test-support/cloudflare-test-worker.ts +++ b/packages/flowsafe/test-support/cloudflare-test-worker.ts @@ -1,16 +1,171 @@ // SPDX-License-Identifier: Apache-2.0 +import type { + SignalDatabase, + SignalStatement, +} from '../src/signals/d1-shared.js'; +import { D1NotificationsStorage } from '../src/signals/notifications-d1.js'; + +async function waitForReceiptReaders( + db: D1Database, + hub: DurableObjectNamespace, + barrierId: string, + participant: string, +): Promise { + await db + .prepare( + 'INSERT INTO notification_test_barrier (barrier_id, participant) VALUES (?, ?)', + ) + .bind(barrierId, participant) + .run(); + const response = await hub.get(hub.idFromName(barrierId)).fetch( + new Request('http://localhost/barrier', { + method: 'POST', + body: JSON.stringify({ barrierId }), + }), + ); + if (!response.ok) throw new Error('notification receipt barrier timed out'); +} + +function receiptBarrierDatabase( + db: D1Database, + hub: DurableObjectNamespace, + barrierId: string, + participant: string, +): SignalDatabase { + let observedMissing = false; + function intercept(statement: SignalStatement): SignalStatement { + return { + bind(...values) { + return intercept(statement.bind(...values)); + }, + async first() { + const row = await statement.first(); + if (row === null && !observedMissing) { + observedMissing = true; + await waitForReceiptReaders(db, hub, barrierId, participant); + } + return row; + }, + all() { + return statement.all(); + }, + run() { + return statement.run(); + }, + }; + } + return { + prepare(query) { + const statement = db.prepare(query); + return query.includes('SELECT request_digest, thread_id, notification_id') + ? intercept(statement) + : statement; + }, + batch: db.batch.bind(db) as SignalDatabase['batch'], + }; +} + class TestDurableObject { - fetch(): Response { + fetch(_request: Request): Response | Promise { return new Response(null, { status: 204 }); } } export class DemoRunner extends TestDurableObject {} -export class DemoHub extends TestDurableObject {} +export class DemoHub extends TestDurableObject { + #waiting?: { + resolve: (response: Response) => void; + timeout: ReturnType; + }; + + constructor( + _state: DurableObjectState, + private readonly bindings: { DB: D1Database }, + ) { + super(); + } + + async fetch(request: Request): Promise { + const { barrierId } = (await request.json()) as { barrierId: string }; + const ready = await this.bindings.DB.prepare( + 'SELECT COUNT(*) AS count FROM notification_test_barrier WHERE barrier_id = ?', + ) + .bind(barrierId) + .first<{ count: number }>(); + if (this.#waiting) { + const waiting = this.#waiting; + this.#waiting = undefined; + clearTimeout(waiting.timeout); + const status = ready?.count === 2 ? 200 : 409; + waiting.resolve(Response.json({ ready: ready?.count }, { status })); + return Response.json({ ready: ready?.count }, { status }); + } + return new Promise((resolve) => { + const timeout = setTimeout(() => { + this.#waiting = undefined; + resolve(new Response(null, { status: 504 })); + }, 10_000); + this.#waiting = { resolve, timeout }; + }); + } +} export class DemoThread extends TestDurableObject {} export class DemoBackgroundTasks extends TestDurableObject {} -export class DemoSignalProviderHost extends TestDurableObject {} +export class DemoSignalProviderHost extends TestDurableObject { + constructor( + _state: DurableObjectState, + private readonly bindings: { + DB: D1Database; + HUB: DurableObjectNamespace; + }, + ) { + super(); + } + + async fetch(request: Request): Promise { + const input = (await request.json()) as { + operationKey: string; + notificationId: string; + receiptLimit: number; + barrierId?: string; + participant?: string; + }; + const database = + input.barrierId && input.participant + ? receiptBarrierDatabase( + this.bindings.DB, + this.bindings.HUB, + input.barrierId, + input.participant, + ) + : this.bindings.DB; + const notifications = new D1NotificationsStorage( + database, + '', + input.receiptLimit, + ); + try { + const receipt = await notifications.createNotificationOnce( + input.operationKey, + { + id: input.notificationId, + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + return Response.json(receipt); + } catch (error) { + return Response.json( + { error: error instanceof Error ? error.message : String(error) }, + { status: 409 }, + ); + } + } +} export default { fetch(): Response { From 97237039350137f42bdc603303f3496f3e7f1431 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:45:22 +0400 Subject: [PATCH 41/73] Prove private workflow route across workerd restart Start a real suspended workflow through the composed host, then verify authenticated registration, one permit winner and retained-result replay after process restart. Keep the fixture default-off and result authenticity unclaimed. --- packages/flowsafe/scripts/spike-verify.mjs | 151 ++++++++++++++++++++- packages/flowsafe/spike/worker.ts | 89 ++++++++++++ packages/flowsafe/spike/wrangler.jsonc | 5 +- 3 files changed, 243 insertions(+), 2 deletions(-) diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index 77a73b45..4c73f068 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -111,6 +111,7 @@ const AUTH = { const STREAM_TICKET_SECRET = 'spike-local-stream-secret-do-not-deploy'; const DEPLOYMENT_TENANT = 'spike'; const DEPLOYMENT_IDENTITY_SECRET = 'spike-local-deployment-identity-secret'; +const PRIVATE_REMOTE_SECRET = 'spike-local-private-admin-secret-2026'; const nowSec = () => Math.floor(Date.now() / 1000); // Track E (M-007): the github webhook signing secret. MUST equal @@ -168,6 +169,8 @@ function startServer(generation, stateDir, logPath) { `DEPLOYMENT_TENANT:${DEPLOYMENT_TENANT}`, '--var', `DEPLOYMENT_IDENTITY_SECRET:${DEPLOYMENT_IDENTITY_SECRET}`, + '--var', + `MAINTENANCE_ADMIN_SECRET:${PRIVATE_REMOTE_SECRET}`, // Sign stream tickets with the SAME key spike-verify forges with, so the // ticket fail-closed probes exercise CLAIM rejection, not signature drift. '--var', @@ -177,7 +180,8 @@ function startServer(generation, stateDir, logPath) { '--var', `GITHUB_WEBHOOK_SECRET:${GITHUB_WEBHOOK_SECRET}`, ...(generation.startsWith('gen-remote-') || - generation.startsWith('gen-workflow-remote-') + generation.startsWith('gen-workflow-remote-') || + generation.startsWith('gen-private-remote-') ? ['--var', 'SPIKE_REMOTE_STEP_PROOF:enabled'] : []), ], @@ -3691,6 +3695,151 @@ async function main() { }, ); + await step( + 'RS3 private workflow route: trusted start, one permit winner, retained result and restart refusal', + async () => { + await killServer(currentServer); + await launchServer( + 'gen-private-remote-1', + stateDir, + join(tmpDir, 'gen-private-remote-1.log'), + ); + const route = '/_flowsafe/workflow-remote-step'; + const privateHeaders = { + authorization: `Bearer ${PRIVATE_REMOTE_SECRET}`, + }; + const absent = await http('POST', `${route}/register`, { + body: { + workflowId: 'spike-private-remote', + runId: 'unknown-run', + hostDeadlineAt: Date.now() + 120_000, + }, + }); + assert( + absent.status === 401, + 'missing private credential is refused', + absent, + ); + const wrong = await http('POST', `${route}/register`, { + headers: AUTH.admin, + body: { + workflowId: 'spike-private-remote', + runId: 'unknown-run', + hostDeadlineAt: Date.now() + 120_000, + }, + }); + assert( + wrong.status === 401, + 'ordinary admin bearer cannot delegate private authority', + wrong, + ); + const started = await http('POST', '/runs', { + headers: AUTH.operator, + body: { + workflowId: 'spike-private-remote', + inputData: { query: 'x' }, + }, + }); + assert( + started.status === 200 && + started.body.status === 'suspended' && + typeof started.body.runId === 'string', + 'real workflow suspends under its Durable Object', + started, + ); + const runId = started.body.runId; + const registered = await http('POST', `${route}/register`, { + headers: privateHeaders, + body: { + workflowId: 'spike-private-remote', + runId, + hostDeadlineAt: Date.now() + 120_000, + }, + }); + assert( + registered.status === 200 && + typeof registered.body.invocationId === 'string', + 'verified stored start registers the workflow invocation', + registered, + ); + const invocationId = registered.body.invocationId; + const issued = await http('POST', `${route}/ticket`, { + headers: privateHeaders, + body: { invocationId, lifetimeMs: 60_000 }, + }); + assert( + issued.status === 200 && + issued.body.runId === runId && + issued.body.requestJson === '{"version":2,"input":{"query":"x"}}' && + issued.body.permit?.generation === 1, + 'private ticket binds the real suspension and dispatch request', + issued, + ); + const attempts = await Promise.all( + [0, 1].map(() => + http('POST', `${route}/consume`, { + headers: privateHeaders, + body: { permit: issued.body.permit }, + }), + ), + ); + assert( + attempts.filter((attempt) => attempt.body.kind === 'accepted') + .length === 1 && + attempts.filter((attempt) => attempt.body.kind === 'already-consumed') + .length === 1, + 'private workflow permit has one consumer', + attempts, + ); + const rawPermit = attempts.find( + (attempt) => attempt.body.kind === 'accepted', + ).body.rawPermit; + const recorded = await http('POST', `${route}/result`, { + headers: privateHeaders, + body: { + invocationId, + rawPermit, + resultJson: '{"answer":1}', + evidenceJson: '{"receipt":"complete"}', + }, + }); + assert( + recorded.status === 200 && recorded.body.kind === 'recorded', + 'private workflow result is retained', + recorded, + ); + await killServer(currentServer); + await launchServer( + 'gen-private-remote-2', + stateDir, + join(tmpDir, 'gen-private-remote-2.log'), + ); + const replay = await http('POST', `${route}/consume`, { + headers: privateHeaders, + body: { permit: issued.body.permit }, + }); + assert( + replay.status === 200 && replay.body.kind === 'already-consumed', + 'fresh process refuses a second effect authority', + replay, + ); + const resultReplay = await http('POST', `${route}/result`, { + headers: privateHeaders, + body: { + invocationId, + rawPermit, + resultJson: '{"answer":1}', + evidenceJson: '{"receipt":"complete"}', + }, + }); + assert( + resultReplay.status === 200 && resultReplay.body.kind === 'replay', + 'fresh process returns the retained result on replay', + resultReplay, + ); + }, + ); + await step( 'S deployment sentinel mismatch: a freshly started Worker refuses a D1 ' + 'provisioned for another deployment', diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index 8ef13dfa..fbc15e12 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -87,7 +87,9 @@ import { createAgentRouter, createAgentThreadTopology, createThreadAgentHost, + createWorkflowRemoteStepPreRoute, type ThreadAgentHost, + WORKFLOW_REMOTE_STEP_ROUTE, } from '../src/agent-host/index.js'; import { readAgentThreadBinding } from '../src/agent-runner/index.js'; import { @@ -165,11 +167,14 @@ import { abandonApprovalsForRun, bearerActorAuthenticator, createDoRunTopology, + createFlowsafeMaintenanceDurableObject, + createFlowsafeWorker, createHubTopology, createRunRouter, createStreamRouter, createThreadTopology, doSummary, + type FlowsafeWorkerConfig, RunRouteError, staticTokenVerifier, type WorkflowMeta, @@ -215,6 +220,8 @@ interface Env { /** Singleton deployment signal-provider host DO; see DemoSignalProviderHost (Track E). */ SIGNAL_PROVIDER_HOST: DurableObjectNamespace; BACKGROUND_TASKS: DurableObjectNamespace; + MAINTENANCE: DurableObjectNamespace; + MAINTENANCE_ADMIN_SECRET: string; SPIKE_REMOTE_STEP_PROOF?: string; SPIKE_LLM_MODEL_ID?: string; SPIKE_LLM_API_KEY?: string; @@ -669,6 +676,12 @@ async function countExecution(db: D1Database, counterId: string) { } const WORKFLOWS: ReadonlyArray = [ + { + id: 'spike-private-remote', + title: 'Private remote-step proof', + description: 'Suspends a real D1-backed workflow for private dispatch', + sampleInput: { query: 'x' }, + }, { id: 'demo-approval', title: 'Demo approval', @@ -859,6 +872,28 @@ function defineWorkflows(env: Env): RunnerRuntime { .then(publish) .commit(); + createWorkflow({ + id: 'spike-private-remote', + inputSchema: z.object({ query: z.string() }), + outputSchema: z.object({ query: z.string() }), + }) + .then( + createStep({ + id: 'execute', + inputSchema: z.object({ query: z.string() }), + outputSchema: z.object({ query: z.string() }), + execute: async ({ inputData, suspend }) => + suspend({ + type: 'suspension', + remoteStep: { + marker: WORKFLOW_REMOTE_STEP_MARKER, + request: { version: 2, input: { query: inputData.query } }, + }, + }), + }), + ) + .commit(); + // FI1/FI2: demo-approval's shape with a COUNTING first step. The step is // ordinary paid work as far as the runtime is concerned — it writes to D1 // before the run suspends — so a second execution of this key would show up @@ -3273,6 +3308,53 @@ async function handleRemoteStepProbe( } } +const privateRemoteConfig: FlowsafeWorkerConfig = { + systemPrincipalId: SYSTEM_PRINCIPAL_ID, + workflows: WORKFLOWS, + maintenance: { sweepIntervalMs: 60_000, purgeIntervalMs: 60_000 }, + buildVerifier: () => staticTokenVerifier(SPIKE_ACTORS), + preRoutes: async (request, env, ctx, kit) => { + if ( + !new URL(request.url).pathname.startsWith( + `${WORKFLOW_REMOTE_STEP_ROUTE}/`, + ) + ) + return null; + const unauthorized = await kit.authorizePrivateWorkflowDispatch(request); + if (unauthorized) return unauthorized; + const journal = new RemoteStepJournal({ + db: env.DB as unknown as never, + fence: executionFenceForEnv(env), + runtime: defineWorkflows(env), + deploymentId: env.DEPLOYMENT_TENANT, + tablePrefix: '', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + verifyWorkflowCompletion: async () => undefined, + }); + await journal.initialize(); + const dispatch = createAgentThreadTopology( + env.THREAD, + env.DEPLOYMENT_IDENTITY_SECRET, + { + startIdempotency: startIdempotencyForEnv(env), + executionFence: executionFenceForEnv(env), + workflowRemoteStepDispatch: journal, + }, + ); + return createWorkflowRemoteStepPreRoute(journal, dispatch)( + request, + env, + ctx, + kit, + ); + }, +}; + +const privateRemoteWorker = createFlowsafeWorker(privateRemoteConfig); +export const DemoPrivateMaintenance = + createFlowsafeMaintenanceDurableObject(privateRemoteConfig); + const handler: ExportedHandler = { async fetch( request: CfRequest, @@ -3309,6 +3391,13 @@ const handler: ExportedHandler = { buildService: (store) => buildApprovalService(store, env, streamSink), }); const routed = request as unknown as Request; + if ( + new URL(routed.url).pathname.startsWith(`${WORKFLOW_REMOTE_STEP_ROUTE}/`) + ) { + if (env.SPIKE_REMOTE_STEP_PROOF !== 'enabled') + return json({ error: 'not found' }, 404); + return privateRemoteWorker.fetch(routed, env, ctx); + } const runTopology = createDoRunTopology( env.RUNNER, env.DEPLOYMENT_IDENTITY_SECRET, diff --git a/packages/flowsafe/spike/wrangler.jsonc b/packages/flowsafe/spike/wrangler.jsonc index b4670513..9ee877e9 100644 --- a/packages/flowsafe/spike/wrangler.jsonc +++ b/packages/flowsafe/spike/wrangler.jsonc @@ -16,6 +16,7 @@ { "name": "HUB", "class_name": "DemoHub" }, { "name": "THREAD", "class_name": "DemoThread" }, { "name": "BACKGROUND_TASKS", "class_name": "DemoBackgroundTasks" }, + { "name": "MAINTENANCE", "class_name": "DemoPrivateMaintenance" }, { "name": "SIGNAL_PROVIDER_HOST", "class_name": "DemoSignalProviderHost" @@ -29,7 +30,8 @@ { "tag": "v2", "new_sqlite_classes": ["DemoHub"] }, { "tag": "v3", "new_sqlite_classes": ["DemoThread"] }, { "tag": "v4", "new_sqlite_classes": ["DemoSignalProviderHost"] }, - { "tag": "v5", "new_sqlite_classes": ["DemoBackgroundTasks"] } + { "tag": "v5", "new_sqlite_classes": ["DemoBackgroundTasks"] }, + { "tag": "v6", "new_sqlite_classes": ["DemoPrivateMaintenance"] } ], "d1_databases": [ { @@ -46,6 +48,7 @@ "vars": { "DEPLOYMENT_TENANT": "spike", "DEPLOYMENT_IDENTITY_SECRET": "spike-local-deployment-identity-secret", + "MAINTENANCE_ADMIN_SECRET": "spike-local-private-admin-secret-2026", "STREAM_TICKET_SECRET": "spike-local-stream-secret-do-not-deploy", // Track E: the github webhook signing secret. spike-verify.mjs re-passes it // via `--var` and signs its webhook probes with it (byte-equal). Never a real From 9a455ff1c9ddd54348a35b1b2cf540f8f717f763 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:52:38 +0400 Subject: [PATCH 42/73] Plan restricted notification recovery authority Define conditional D1 effect admission, stable executor receipt bytes and host-owned maintenance reconciliation while ambiguous Mastra continuation stays blocked. --- ...-12-remote-step-durable-resume-recovery.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md b/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md index f01b6090..90599140 100644 --- a/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md +++ b/docs/plans/2026-10-12-remote-step-durable-resume-recovery.md @@ -2,6 +2,8 @@ > Content type: design plan. Audience: FlowSafe maintainers and deployment hosts. Status: proposed, not implemented. +The dated amendment at the end is the current execution plan. The preceding sections preserve the decision and code baseline from 2026-10-12. + This plan records the limit of safe cross-instance resume recovery on pinned `@mastra/core@1.73.0`. Its `run.resume` can execute arbitrary tool closures, nested agents, background work, model calls and direct network effects outside a FlowSafe-owned adapter. General agent continuation is therefore ineligible for automatic replay. A narrow host-owned operation with an explicit, closed effect set may use a downstream idempotent executor; the subsequent Mastra continuation remains single-admission. An uncertain Mastra continuation stays unresolved and blocks drain until independent evidence or a future runtime contract resolves it. ## Baseline and evidence @@ -218,3 +220,114 @@ else if (resumeNested) res = await run.resume({ ``` The pinned engine's workflow writes include persist, replace and patch paths without a remote-step generation. The closure calls above are enough to rule out an effect adapter that wraps only the FlowSafe host route. A future automatic-continuation proposal must inspect all engine write paths and bind its advancing write to durable resume provenance; a separate post-write journal update cannot serve as the receipt. + +## Amendment: execute the host-owned notification operation (2026-10-13) + +**Goal and status.** Implement recovery of a selected host-owned notification enqueue after a lost acknowledgement or host restart. `D1NotificationsStorage.createNotificationOnce` is the selected executor; it inserts the notification and immutable request receipt in one D1 batch. A native two-Durable-Object contention fixture has proved local atomicity. No production caller or deployed D1 proof exists. This amendment supersedes the earlier statement that no executor is selected; it does not authorize automatic Mastra `run.resume` replay. + +### Current baseline and constraints + +- Branch: `plan/flowsafe-remote-step-broker`; amendment base: `97237039350137f42bdc603303f3496f3e7f1431` (2026-10-13). Reconfirm each appendix anchor against the checked-out commit before editing. The historical `4096fbd` baseline above remains a record, not an edit target. +- Use the repository's locked Node, pnpm, `@mastra/core@1.73.0`, Cloudflare Workers, Durable Objects and D1. Work in a linked worktree. Put checks and build artifacts under `/tmp`, and record an exit status in each log. ASP signing, dispatch policy and customer UI remain in ASP. +- The 100,000 retained notification receipt cap is approved and implemented per configured D1 table prefix. One D1/prefix per deployment is the supported topology; a multi-prefix deployment must refuse this profile until deployment-wide accounting is designed. Measure D1 `rows_read` and latency near the cap before release; do not silently purge receipts. +- A workflow-target v4 journal and private workflow route exist, but the selected executor and target-thread recovery below apply to an **agent** invocation. The workflow-target continuation remains a separate gate. The existing `recoverableResults` deliberately selects `target_kind = 'agent'` while drain inventory retains both kinds. + +### Operation identity and state contract + +The host selects the notification profile from trusted configuration, never from a result body. The profile fixes the notification source, kind, recipient thread/resource, identity and payload construction. Name and version one canonical projection, `notification-record-values-v1`, inside `notifications-d1.ts`; both the journal preparation and `createNotificationOnce` use its `JSON.stringify(recordValues(record))` UTF-8 bytes after defaults and field conversion. Persist the exact `CreateNotificationInput`, projection version, projected bytes and their SHA-256 digest before any effect attempt. The executor computes its receipt `request_digest` from those same bytes and returns or exposes that digest for exact comparison with the journal. Unknown projection versions refuse. Its `operationKey` binds deployment ID, journal invocation ID, run start token, suspension identity and the profile's operation ID; it is stable across host restarts and retries. The journal owns the mapping from this identity to one notification ID. A new resume lease owner does not change either ID. The executor's existing digest check refuses a reused key with changed notification input; the journal compares its stored digest to the executor receipt on every retry. + +Use a versioned journal record with states for prepared operation, effect admitted, receipt recorded, resume admitted and unresolved continuation. Keep terminal settlement and retained drain inventory. Preparation stores identity and request but grants no effect authority. Immediately before the first executor call, one conditional D1 write admits that exact host operation. Its SQL predicate checks the current fence row, deadline, stored principal and binding, start token, result digest, journal revision and authoritative suspension identity in the same D1 database transaction; the host's preceding run read is diagnostic, not the authority check. Require a schema path that can compare the authoritative snapshot's current suspension and generation in that transaction. If it cannot, leave the profile disabled. A zero-row or unreadable admission result refuses the effect. The successful write is the effect's linearization point: a later deadline or fence change cannot erase an already admitted operation, whose receipt remains drain work. If the admission write's outcome is ambiguous, reread the durable operation state before any executor call; never infer success from a timeout. After `createNotificationOnce` returns, persist its digest-bearing receipt with an equally bound compare-and-swap. A lost response is recoverable by calling the same method with the exact key and input only for a durably admitted operation; it returns the original receipt or refuses a changed request. Add a read-only receipt lookup inside `D1NotificationsStorage` so recovery can observe an old effect without creating a new one. If no receipt exists and admission was refused or cannot be proved, leave the row unresolved. A cap error before insertion leaves an admitted operation visible; existing-key retry still works at the cap. + +Persist a **one-time resume-admitted marker** before entering Mastra. It binds the exact retained result, suspension and operation receipt. The journal must permit one transition into admission and refuse every later attempt, including after lease expiry, deploy and restart. A crash after admission but before the call leaves an unresolved continuation; neither the receipt nor an advanced snapshot proves a second `run.resume` safe. The selected notification can recover automatically, but Mastra continuation cannot until its snapshot writes have generation fencing and attempt-linked provenance. This limit is an explicit release behavior, not a pending implementation shortcut. + +### Edit anchors and before → after + +The following excerpts are verbatim at the amendment base. Method names in the target sketches specify the contract to implement; inspect the actual types and line positions before editing. + +1. **Executor:** `packages/flowsafe/src/signals/notifications-d1.ts`, `D1NotificationsStorage.createNotificationOnce` currently enters with: + + ```typescript + async createNotificationOnce( + operationKey: string, + input: CreateNotificationInput & { id: string; createdAt: Date }, + ): Promise<{ + operationKey: string; + notificationId: string; + threadId: string; + }> { + ``` + + The existing D1 batch inserts a receipt first and gates the inbox insert on `changes() = 1`. Reuse this direct method. Extract its `JSON.stringify(recordValues(record))` projection into one versioned function that the trusted host and executor both call, without changing the bytes or receipt behavior. The adapter constructs and freezes arguments from the prepared journal row. Expose the receipt digest and a read-only lookup under the same storage boundary; do not replace the D1 effect transaction. + +2. **Journal:** `packages/flowsafe/src/do-runner/remote-step-journal.ts`, `reconcileResult` currently starts: + + ```typescript + if (invocation.status === 'resuming') return 'pending-quiescence'; + const owner = await this.claimResume(invocationId, 60_000); + return this.settleAfterResume(invocationId, owner); + ``` + + Keep this fail-closed path for ordinary remote steps. Add a separate notification operation path whose order is `prepare` → `admitEffect` → `createNotificationOnce` → `recordReceipt` → `admitResume` → one trusted runtime call → witnessed settlement. `admitEffect` owns the atomic authority predicate described above. Recovery may repeat the executor with the stored key and request after proved effect admission and before `admitResume`; it must never use `claimResume` to reclaim a previously admitted Mastra call. + +3. **Host:** `packages/flowsafe/src/agent-host/thread-host.ts`, the retained-result route currently enters Mastra after: + + ```typescript + const leaseOwner = await reader.claimResume(invocationId, 60_000); + const resumed = await withExecution(execution, () => + durable.resumeViaRuntime({ + ``` + + A notification-profile branch runs inside the same target thread's `withDispatchLock`. It re-reads the authoritative run and stored principal/binding, then requires the conditional `admitEffect` D1 write before calling `createNotificationOnce` from trusted host configuration. It records the receipt, rechecks authority for continuation, then commits the one-time marker before this runtime call. The ordinary branch retains its current fail-closed behavior. A host-only recovery method uses the same lock, authentication and stored-principal checks; it exposes no public result route. + +4. **Maintenance:** `createThreadAgentHost.recoverOwnership` currently uses: + + ```typescript + return withDispatchLock(async () => { + const storage = options.stateStorage(); + ``` + + The recovery method shares this serialization point. A scheduled maintenance Worker advances a durable, bounded D1 cursor over active notification-profile operations and routes each selected item to its owning thread Durable Object. The thread method shares the existing dispatch lock. Its alarm owner also wakes pending local work during no traffic; extend its existing `recoverOwnership` scheduling and delete conditions so notification work cannot be erased by owner recovery. Process a fixed batch, persist the next cursor, wrap at the end and rearm while work remains. Do not install a second independent alarm owner or scan retained terminal history. A failed item stays visible; one item cannot prevent progress through later IDs. A lease expiry schedules inspection but never authorizes another Mastra call. + +5. **Migration and mixed writers:** `remote-step-journal.ts` currently recognizes version 4 with: + + ```typescript + if (record(rows[0])?.id === 1 && record(rows[0])?.version === 4) { + ``` + + Its current update trigger is: + + ```typescript + const REMOTE_STEP_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; + ``` + + Migrate known v1–v4 rows into a validated v5 layout without losing permits, retained results or unsettled obligations. Add a required `write_generation` initialized to a positive value for migrated rows. Require v5 inserts to supply it and v5 updates to increase it through a D1 trigger; an already-initialized v4 writer preserves the value and is refused. Validate the table, indexes, trigger and exact schema version before reads or writes. Unknown or malformed layouts refuse. An old Worker that initializes after migration sees unknown version; prove both old-writer cases in an old/new Worker exercise before enabling dispatch. Legacy `resuming` rows without the new admission evidence remain unresolved. + +### Ordered implementation checkpoints + +1. **Journal contract and migration:** Add the v5 operation fields, conditional effect-admission transition, active index and drain inventory projection. Preserve existing v4 behavior and data. Refuse stale writers with the generation trigger and test both old-initialized and fresh old versions. Prove that the admission SQL can compare current fence and authoritative suspension in the same D1 transaction; otherwise keep the profile disabled. Use `remote-step-journal.ts` migration and conditional-write patterns rather than a parallel journal. +2. **Trusted host executor:** Add the closed notification profile to the target thread host. Derive all effect fields from trusted stored context, share the executor's versioned projection and use `D1NotificationsStorage.createNotificationOnce` directly after proved effect admission. Save its digest-bearing receipt. Add the single-admission transition before runtime entry. Keep `resumeViaRuntime` one-shot and report ambiguous outcomes as unresolved. +3. **Recovery duty:** Add host-only inspection, a scheduled D1 cursor scan and target-thread alarm wake through the existing owner recovery scheduler. Repeat a previously admitted effect with its stable key; after a refused or ambiguous admission, use read-only receipt inspection and keep missing evidence unresolved. Preserve progress after restart, duplicated alarms and an item that repeatedly fails. Never retry admitted continuation. +4. **Cutover proof:** Keep the profile default off until native fresh-isolate and deployed D1 proof, old/new Worker refusal, authenticated host route negatives, cap latency measurement and drain inventory checks pass. Review and release via the Anchorage `dev` process; ASP integration follows a published, release-age-eligible FlowSafe version. + +No new package is needed. The existing direct D1 notification storage is the smallest integration boundary; a new executor package would duplicate its atomic receipt contract. An external notifier lacks this repository's D1 transaction and cannot satisfy this plan without a separate proof. General Mastra replay, workflow-target continuation, arbitrary tools, payment, customer dispatch and ASP policy are excluded for the reasons above. + +### Scoped verification and proof + +Select tests from the checkpoint's changed paths and shared contracts. The v5 schema is a shared contract, so run the owning FlowSafe package suite once for that checkpoint under the quality-gate exception. For other checkpoints use the named files below, then changed-file Biome, package typecheck/build when the tool cannot retain config with file targets. Redirect each command to a distinct `/tmp` log with the content ID and `EXIT_STATUS` footer. Run the repository full suite once after the final checkpoint, not at intermediate checkpoints. + +```bash +pnpm --filter @proofoftech/flowsafe exec vitest run \ + src/do-runner/remote-step-journal.test.ts \ + src/agent-host/thread-host.test.ts \ + src/signals/notifications-d1.test.ts +pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts \ + packages/flowsafe/src/agent-host/thread-host.ts \ + packages/flowsafe/src/signals/notifications-d1.ts +pnpm --filter @proofoftech/flowsafe typecheck:test +pnpm --filter @proofoftech/flowsafe build +pnpm --filter @proofoftech/flowsafe spike:verify +``` + +The gate owner narrows these candidates to the changed-file list and records the exact commands in its logs. All implementation checkpoints are T2: independent clean-code, QA, architecture and security lanes review the checkpoint diff and the called journal, host, D1 notification and alarm contracts. Review this amendment once at design level before implementation. + +The live proof must show one D1 notification and receipt when two fresh object instances race, including a lost acknowledgement, same-key retry at the last receipt slot, changed-request refusal and crash before/after receipt. Change the fence, deadline and suspension after preparation but before effect admission: the D1 write must refuse and the inbox must stay empty. Interrupt the admission response: recovery must read its durable result before any effect. Compare the journal's projected request bytes and digest with the executor's receipt digest, including defaulted and date fields. The proof must also show a persisted cursor eventually reaching work beyond one batch, duplicate alarm delivery, restart and no-traffic wake, an old Worker unable to write v5 rows, and blocked drain while Mastra admission is uncertain. Inspect the authoritative D1 receipt and inbox after each fault. Local workerd evidence does not substitute for a deployed D1 exercise. Successful notification recovery never implies automatic continuation; a test that calls `run.resume` twice fails this plan even if it observes one notification. From 648dee903367533fbdcf0de8f271baf67897b2b8 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:14:29 +0400 Subject: [PATCH 43/73] Fence restricted notification recovery in FlowSafe journal Add versioned v5 state, exact notification effect admission and one-time continuation admission. Preserve v4 records and block stale writers; host executor and authoritative receipt readback remain default-off follow-up gates. --- .../src/agent-host/thread-host.test.ts | 2 +- .../src/do-runner/remote-step-journal.test.ts | 305 +++++++- .../src/do-runner/remote-step-journal.ts | 731 +++++++++++++++++- .../src/signals/notifications-d1.test.ts | 3 +- .../flowsafe/src/signals/notifications-d1.ts | 146 +++- 5 files changed, 1119 insertions(+), 68 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 13e73650..fda21e66 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -7927,7 +7927,7 @@ it('resumes a retained mapped-resource result from D1 after the agent host is re ]); await fixture.binding .prepare( - 'UPDATE flowsafe_remote_step_invocations SET resume_lease_expires_at = ?, write_revision = write_revision + 1 WHERE invocation_id = ?', + 'UPDATE flowsafe_remote_step_invocations SET resume_lease_expires_at = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ?', ) .bind(Date.now() - 1, invocation.invocationId) .run(); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index 34ab3c74..ae91e982 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -8,6 +8,10 @@ import type { FlowsafeWorkerEnv } from '../host-kit/flowsafe-worker.js'; import { createFlowsafeWorker } from '../host-kit/flowsafe-worker.js'; import { approvalStoreFactoryFor } from '../host-kit/host-approval-service.js'; import { staticTokenVerifier } from '../host-kit/verifier.js'; +import { + D1NotificationsStorage, + projectNotificationOperation, +} from '../signals/notifications-d1.js'; import { createD1Storage } from './d1-storage.js'; import { credentialsMatch } from './deployment-identity.js'; import { ExecutionFenceStore } from './execution-fence.js'; @@ -188,7 +192,14 @@ describe('remote step journal dispatch admission', () => { .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) .all() as Array<{ name: string }> ) - .slice(0, -4) + .slice( + 0, + ( + h.sqlite + .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) + .all() as Array<{ name: string }> + ).findIndex(({ name }) => name === 'target_kind'), + ) .map(({ name }) => name) .join(', '); h.sqlite.exec(`CREATE TABLE ${REMOTE_STEP_TABLE}_v3 ( @@ -635,7 +646,7 @@ describe('remote step journal dispatch admission', () => { }); expect( h.sqlite.prepare('SELECT version FROM flowsafe_remote_step_schema').get(), - ).toEqual({ version: 4 }); + ).toEqual({ version: 5 }); expect(await h.reopen().recoverableResults(1)).toEqual({ invocationIds: [invocation.invocationId], nextCursor: invocation.invocationId, @@ -1009,7 +1020,7 @@ describe('remote step journal dispatch admission', () => { ).toBeUndefined(); h.sqlite .prepare( - `UPDATE ${REMOTE_STEP_TABLE} SET result_digest = NULL, write_revision = write_revision + 1`, + `UPDATE ${REMOTE_STEP_TABLE} SET result_digest = NULL, write_revision = write_revision + 1, write_generation = write_generation + 1`, ) .run(); expect( @@ -1338,7 +1349,7 @@ describe('remote step journal dispatch admission', () => { const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); h.sqlite .prepare( - `UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?, write_revision = write_revision + 1`, + `UPDATE ${REMOTE_STEP_TABLE} SET table_prefix = ?, write_revision = write_revision + 1, write_generation = write_generation + 1`, ) .run('bad"_prefix'); await expect( @@ -1552,7 +1563,7 @@ describe('remote step journal dispatch admission', () => { }); }); - it('rejects stale v3-style journal writes after v4 migration', async () => { + it('rejects stale v4-style writes after v5 migration', async () => { const h = await fixture(); const invocation = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); restoreV3Schema(h); @@ -1560,7 +1571,7 @@ describe('remote step journal dispatch admission', () => { expect(() => h.sqlite .prepare( - `UPDATE ${REMOTE_STEP_TABLE} SET deadline_at = ? WHERE invocation_id = ?`, + `UPDATE ${REMOTE_STEP_TABLE} SET deadline_at = ?, write_revision = write_revision + 1 WHERE invocation_id = ?`, ) .run(9_000, invocation.invocationId), ).toThrow('remote step writer version changed'); @@ -1594,6 +1605,288 @@ describe('remote step journal dispatch admission', () => { }); }); + it('binds a notification effect and one continuation admission to the retained result', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const input = { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + payload: { value: 'done' }, + createdAt: new Date('2026-10-13T00:00:00.000Z'), + deliverAt: new Date('2026-10-14T00:00:00.000Z'), + }; + const prepared = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + input, + ); + const projected = await projectNotificationOperation({ + ...input, + id: prepared.notificationId, + }); + expect(prepared).toMatchObject({ + state: 'prepared', + projectionVersion: 'notification-record-values-v1', + projectionJson: projected.requestJson, + requestDigest: projected.requestDigest, + }); + expect(await h.reopen().recoverableResults(10)).toEqual({ + invocationIds: [], + }); + await expect( + h.journal.claimResume(invocation.invocationId, 1_000), + ).rejects.toMatchObject({ status: 409 }); + expect( + (await h.reopen().admitNotificationEffect(invocation.invocationId)).state, + ).toBe('effect-admitted'); + const notifications = new D1NotificationsStorage(h.db); + const receipt = await notifications.createNotificationOnce( + prepared.operationKey, + { ...input, id: prepared.notificationId }, + ); + expect(receipt.requestDigest).toBe(prepared.requestDigest); + expect( + ( + await h.journal.recordNotificationReceipt( + invocation.invocationId, + receipt, + ) + ).state, + ).toBe('receipt-recorded'); + expect( + await notifications.readNotificationOperationReceipt( + prepared.operationKey, + ), + ).toEqual(receipt); + const owner = await h + .reopen() + .admitNotificationResume(invocation.invocationId); + expect(owner).toMatch(/^[a-f0-9-]{36}$/); + await expect( + h.journal.admitNotificationResume(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ + state: 'resume-admitted', + requestDigest: receipt.requestDigest, + }); + expect( + (await new DeploymentInventory(h.db).read('remote-steps')).count, + ).toBe(1); + }); + + it('refuses effect admission when the authoritative snapshot changes at the D1 write', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("SET operation_state = 'effect-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + h.sqlite + .prepare('UPDATE mastra_workflow_snapshot SET snapshot = ?') + .run('{"status":"cancelled"}'); + return bound.all(); + }, + }; + }, + }; + }; + await expect( + h.journal.admitNotificationEffect(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'prepared' }); + }); + + it('refuses a changed notification receipt after effect admission', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + await expect( + h.journal.recordNotificationReceipt(invocation.invocationId, { + operationKey: operation.operationKey, + notificationId: operation.notificationId, + threadId: ADDRESS.threadId, + requestDigest: 'f'.repeat(64), + }), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'effect-admitted' }); + }); + + it('refuses stale principal, start token, result and deadline before effect admission', async () => { + const cases = [ + { + column: 'principal_json', + value: JSON.stringify({ kind: 'human', id: 'other', role: 'operator' }), + status: 409, + }, + { column: 'start_token', value: 'other-start', status: 409 }, + { column: 'result_digest', value: 'f'.repeat(64), status: 503 }, + { column: 'deadline_at', value: 1_000, status: 409 }, + ] as const; + for (const changed of cases) { + const h = await fixture(); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + h.sqlite + .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET ${changed.column} = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1`) + .run(changed.value); + await expect( + h.journal.admitNotificationEffect(invocation.invocationId), + ).rejects.toMatchObject({ status: changed.status }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'prepared' }); + } + }); + + it('refuses effect admission when the fence moves after its read', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("SET operation_state = 'effect-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + const before = await h.fence.read(); + await h.fence.transition({ + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: before.mutationEpoch, + expectedRevision: before.transitionRevision, + advanceMutationEpoch: true, + }); + return bound.all(); + }, + }; + }, + }; + }; + await expect( + h.journal.admitNotificationEffect(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'prepared' }); + }); + + it('reads durable effect admission after its write response is lost', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const prepared = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + const prepare = h.db.prepare.bind(h.db); + let interrupted = false; + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("SET operation_state = 'effect-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + const result = await bound.all(); + if (!interrupted) { + interrupted = true; + throw new Error('lost admission response'); + } + return result; + }, + }; + }, + }; + }; + expect( + await h.journal.admitNotificationEffect(invocation.invocationId), + ).toMatchObject({ + state: 'effect-admitted', + operationKey: prepared.operationKey, + }); + expect( + await h.reopen().admitNotificationEffect(invocation.invocationId), + ).toMatchObject({ + state: 'effect-admitted', + operationKey: prepared.operationKey, + }); + }); + it('refuses a real suspended snapshot after runtime persists cancellation intent', async () => { const sqlite = openSqlite() as ReturnType & { close(): void; diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index ed4bf398..6e1f1fe0 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -5,6 +5,11 @@ import { assertExecutionPrincipal, type ExecutionPrincipal, } from '../approval-api/principal.js'; +import { + NOTIFICATION_OPERATION_PROJECTION_VERSION, + type NotificationOperationInput, + projectNotificationOperation, +} from '../signals/notifications-d1.js'; import { credentialsMatch } from './deployment-identity.js'; import { DoStatusError } from './do-status-error.js'; import { @@ -33,6 +38,10 @@ const REMOTE_STEP_AGENT_INDEX_SQL = `CREATE UNIQUE INDEX ${REMOTE_STEP_AGENT_IND const REMOTE_STEP_WORKFLOW_INDEX_SQL = `CREATE UNIQUE INDEX ${REMOTE_STEP_WORKFLOW_INDEX} ON ${REMOTE_STEP_TABLE} (deployment_id, table_prefix, workflow_id, run_id, step_path, suspended_at, resume_count) WHERE target_kind = 'workflow'`; const REMOTE_STEP_WRITE_TRIGGER = 'flowsafe_remote_step_write_version'; const REMOTE_STEP_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; +const REMOTE_STEP_V5_WRITE_TRIGGER_SQL = `CREATE TRIGGER ${REMOTE_STEP_WRITE_TRIGGER} BEFORE UPDATE ON ${REMOTE_STEP_TABLE} WHEN NEW.write_generation <= OLD.write_generation OR NEW.write_revision <= OLD.write_revision BEGIN SELECT RAISE(ABORT, 'remote step writer version changed'); END`; +const REMOTE_STEP_OPERATION_INDEX = + 'flowsafe_remote_step_notification_recovery'; +const REMOTE_STEP_OPERATION_INDEX_SQL = `CREATE INDEX ${REMOTE_STEP_OPERATION_INDEX} ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved')`; const REMOTE_STEP_RESULT_COLUMNS = [ 'result_json', 'result_digest', @@ -75,6 +84,21 @@ const REMOTE_STEP_COLUMNS = [ 'request_digest', 'write_revision', ] as const; +const REMOTE_STEP_V5_COLUMNS = [ + ...REMOTE_STEP_COLUMNS, + 'write_generation', + 'operation_key', + 'operation_id', + 'notification_id', + 'notification_request_json', + 'notification_projection_version', + 'notification_projection_json', + 'notification_request_digest', + 'operation_state', + 'effect_admitted_at', + 'receipt_recorded_at', + 'resume_admitted_at', +] as const; const REMOTE_STEP_INTEGER_COLUMNS = new Set([ 'suspended_at', 'resume_count', @@ -88,6 +112,10 @@ const REMOTE_STEP_INTEGER_COLUMNS = new Set([ 'resume_lease_expires_at', 'settled_at', 'write_revision', + 'write_generation', + 'effect_admitted_at', + 'receipt_recorded_at', + 'resume_admitted_at', ]); export const REMOTE_STEP_MARKER = 'flowsafe.remote-step/v1'; export const WORKFLOW_REMOTE_STEP_MARKER = 'flowsafe.remote-step.workflow/v1'; @@ -97,7 +125,7 @@ export const MAX_REMOTE_STEP_RESULT_BYTES = 16_384; const SHA256_HEX = /^[a-f0-9]{64}$/; const RAW_PERMIT = /^[A-Za-z0-9_-]{43}$/; -function remoteStepTableSql(name: string): string { +function remoteStepTableSql(name: string, version: 4 | 5 = 5): string { return `CREATE TABLE IF NOT EXISTS ${name} ( invocation_id TEXT PRIMARY KEY, principal_json TEXT NOT NULL, @@ -132,7 +160,32 @@ function remoteStepTableSql(name: string): string { target_kind TEXT NOT NULL CHECK (target_kind IN ('agent', 'workflow')), suspension_marker TEXT NOT NULL, request_digest TEXT, - write_revision INTEGER NOT NULL DEFAULT 0, + write_revision INTEGER NOT NULL DEFAULT 0${ + version === 5 + ? `, + write_generation INTEGER NOT NULL CHECK (write_generation > 0), + operation_key TEXT UNIQUE, + operation_id TEXT, + notification_id TEXT, + notification_request_json TEXT, + notification_projection_version TEXT, + notification_projection_json TEXT, + notification_request_digest TEXT, + operation_state TEXT CHECK (operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved')), + effect_admitted_at INTEGER, + receipt_recorded_at INTEGER, + resume_admitted_at INTEGER, + CHECK ((operation_state IS NULL AND operation_key IS NULL AND operation_id IS NULL + AND notification_id IS NULL AND notification_request_json IS NULL + AND notification_projection_version IS NULL AND notification_projection_json IS NULL + AND notification_request_digest IS NULL AND effect_admitted_at IS NULL + AND receipt_recorded_at IS NULL AND resume_admitted_at IS NULL) OR + (target_kind = 'agent' AND operation_key IS NOT NULL AND operation_id IS NOT NULL + AND notification_id IS NOT NULL AND notification_request_json IS NOT NULL + AND notification_projection_version = 'notification-record-values-v1' + AND notification_projection_json IS NOT NULL AND notification_request_digest IS NOT NULL))` + : '' + }, CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL AND thread_id IS NOT NULL AND resource_id IS NOT NULL AND suspension_marker = '${REMOTE_STEP_MARKER}') OR @@ -249,6 +302,26 @@ export interface RecordedRemoteStepResult { executorQuiescentAt: number; } +export interface RemoteStepNotificationOperation { + invocationId: string; + operationId: string; + operationKey: string; + notificationId: string; + requestJson: string; + projectionVersion: typeof NOTIFICATION_OPERATION_PROJECTION_VERSION; + projectionJson: string; + requestDigest: string; + state: + | 'prepared' + | 'effect-admitted' + | 'receipt-recorded' + | 'resume-admitted' + | 'unresolved'; + effectAdmittedAt?: number; + receiptRecordedAt?: number; + resumeAdmittedAt?: number; +} + class RemoteStepRefusal extends DoStatusError { readonly status: number; @@ -294,6 +367,18 @@ type JournalRow = { suspension_marker: string; request_digest: string | null; write_revision: number; + write_generation: number; + operation_key: string | null; + operation_id: string | null; + notification_id: string | null; + notification_request_json: string | null; + notification_projection_version: string | null; + notification_projection_json: string | null; + notification_request_digest: string | null; + operation_state: string | null; + effect_admitted_at: number | null; + receipt_recorded_at: number | null; + resume_admitted_at: number | null; }; function record(value: unknown): Record | undefined { @@ -320,6 +405,7 @@ function decode(row: JournalRow): RemoteStepInvocation { (row.request_digest !== null && !SHA256_HEX.test(row.request_digest)) || !Number.isSafeInteger(row.write_revision) || row.write_revision < 0 || + !positiveTime(row.write_generation) || typeof row.step_path !== 'string' || row.step_path.length > 1_024 || typeof row.request_json !== 'string' || @@ -455,6 +541,7 @@ function decodeWorkflow(row: JournalRow): WorkflowRemoteStepInvocation { !SHA256_HEX.test(row.request_digest) || !Number.isSafeInteger(row.write_revision) || row.write_revision < 0 || + !positiveTime(row.write_generation) || !positiveTime(row.suspended_at) || !positiveTime(row.deadline_at) || !Number.isSafeInteger(row.resume_count) || @@ -562,6 +649,91 @@ function decodeAny( throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); } +function decodeNotificationOperation( + row: JournalRow, +): RemoteStepNotificationOperation | undefined { + if (row.operation_state === null) { + if ( + row.operation_key !== null || + row.operation_id !== null || + row.notification_id !== null || + row.notification_request_json !== null || + row.notification_projection_version !== null || + row.notification_projection_json !== null || + row.notification_request_digest !== null || + row.effect_admitted_at !== null || + row.receipt_recorded_at !== null || + row.resume_admitted_at !== null + ) + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + return undefined; + } + if ( + row.target_kind !== 'agent' || + !isPathSafeId(row.operation_key) || + !isPathSafeId(row.operation_id) || + !isPathSafeId(row.notification_id) || + typeof row.notification_request_json !== 'string' || + typeof row.notification_projection_json !== 'string' || + row.notification_projection_version !== + NOTIFICATION_OPERATION_PROJECTION_VERSION || + typeof row.notification_request_digest !== 'string' || + !SHA256_HEX.test(row.notification_request_digest) || + ![ + 'prepared', + 'effect-admitted', + 'receipt-recorded', + 'resume-admitted', + 'unresolved', + ].includes(row.operation_state) || + new TextEncoder().encode(row.notification_request_json).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + new TextEncoder().encode(row.notification_projection_json).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + (row.operation_state === 'prepared' && + (row.effect_admitted_at !== null || + row.receipt_recorded_at !== null || + row.resume_admitted_at !== null)) || + (row.operation_state !== 'prepared' && + !positiveTime(row.effect_admitted_at)) || + (row.operation_state === 'effect-admitted' && + (row.receipt_recorded_at !== null || row.resume_admitted_at !== null)) || + (['receipt-recorded', 'resume-admitted'].includes(row.operation_state) && + !positiveTime(row.receipt_recorded_at)) || + (row.operation_state === 'receipt-recorded' && + row.resume_admitted_at !== null) || + (row.operation_state === 'resume-admitted' && + !positiveTime(row.resume_admitted_at)) + ) + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + try { + JSON.parse(row.notification_request_json); + JSON.parse(row.notification_projection_json); + } catch { + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + } + return { + invocationId: row.invocation_id, + operationId: row.operation_id, + operationKey: row.operation_key, + notificationId: row.notification_id, + requestJson: row.notification_request_json, + projectionVersion: NOTIFICATION_OPERATION_PROJECTION_VERSION, + projectionJson: row.notification_projection_json, + requestDigest: row.notification_request_digest, + state: row.operation_state as RemoteStepNotificationOperation['state'], + ...(row.effect_admitted_at === null + ? {} + : { effectAdmittedAt: row.effect_admitted_at }), + ...(row.receipt_recorded_at === null + ? {} + : { receiptRecordedAt: row.receipt_recorded_at }), + ...(row.resume_admitted_at === null + ? {} + : { resumeAdmittedAt: row.resume_admitted_at }), + }; +} + function selectedSuspension( summary: RunSummary, allowedToolIds: ReadonlySet, @@ -754,7 +926,11 @@ export class RemoteStepJournal { const columns = record(schema)?.results; if (!Array.isArray(columns)) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); - const samePrefix = (expected: readonly string[], v4 = false) => + const samePrefix = ( + expected: readonly string[], + v4 = false, + v5 = false, + ) => columns.length === expected.length && columns.every((value, index) => { const column = record(value); @@ -769,10 +945,11 @@ export class RemoteStepJournal { ((index > 0 && index <= 16 && !(v4 && index >= 6 && index <= 8)) || - (v4 && + ((v4 || v5) && ['target_kind', 'suspension_marker', 'write_revision'].includes( name as string, - )) + )) || + (v5 && name === 'write_generation') ? 1 : 0) && column?.pk === (index === 0 ? 1 : 0) && @@ -913,6 +1090,77 @@ export class RemoteStepJournal { record(trigger[0])?.sql === REMOTE_STEP_WRITE_TRIGGER_SQL ); }; + const validV5 = async () => { + const tableDefinition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('table', REMOTE_STEP_TABLE) + .all(), + )?.results; + const tableSql = Array.isArray(tableDefinition) + ? record(tableDefinition[0])?.sql + : undefined; + const operationIndex = indexes.find( + (value) => record(value)?.name === REMOTE_STEP_OPERATION_INDEX, + ); + const operationDefinition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('index', REMOTE_STEP_OPERATION_INDEX) + .all(), + )?.results; + const trigger = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('trigger', REMOTE_STEP_WRITE_TRIGGER) + .all(), + )?.results; + return ( + typeof tableSql === 'string' && + tableSql.includes( + 'write_generation INTEGER NOT NULL CHECK (write_generation > 0)', + ) && + tableSql.includes('operation_key TEXT UNIQUE') && + tableSql.includes( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + ) && + tableSql.includes("(target_kind = 'workflow' AND agent_id IS NULL") && + tableSql.includes( + "operation_state TEXT CHECK (operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved'))", + ) && + tableSql.includes( + 'CHECK ((operation_state IS NULL AND operation_key IS NULL', + ) && + tableSql.includes( + "(target_kind = 'agent' AND operation_key IS NOT NULL", + ) && + validRecoveryIndex && + (await validV4Index( + REMOTE_STEP_AGENT_INDEX, + REMOTE_STEP_AGENT_INDEX_SQL, + )) && + (await validV4Index( + REMOTE_STEP_WORKFLOW_INDEX, + REMOTE_STEP_WORKFLOW_INDEX_SQL, + )) && + record(operationIndex)?.unique === 0 && + record(operationIndex)?.origin === 'c' && + record(operationIndex)?.partial === 1 && + Array.isArray(operationDefinition) && + operationDefinition.length === 1 && + record(operationDefinition[0])?.sql === + REMOTE_STEP_OPERATION_INDEX_SQL && + Array.isArray(trigger) && + trigger.length === 1 && + record(trigger[0])?.sql === REMOTE_STEP_V5_WRITE_TRIGGER_SQL + ); + }; const versions = await this.#db .prepare(`SELECT id, version FROM ${REMOTE_STEP_SCHEMA_TABLE} LIMIT 2`) .all(); @@ -920,6 +1168,17 @@ export class RemoteStepJournal { if (!Array.isArray(rows) || rows.length > 1) throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); if (rows.length === 1) { + if (record(rows[0])?.id === 1 && record(rows[0])?.version === 5) { + if ( + !samePrefix(REMOTE_STEP_V5_COLUMNS, true, true) || + !(await validV5()) + ) + throw new RemoteStepRefusal( + 503, + 'remote step schema is unreadable', + ); + return 'current-v5'; + } if (record(rows[0])?.id === 1 && record(rows[0])?.version === 4) { if (!samePrefix(REMOTE_STEP_COLUMNS, true) || !(await validV4())) throw new RemoteStepRefusal( @@ -950,10 +1209,44 @@ export class RemoteStepJournal { return 'unversioned-v3'; if (samePrefix(REMOTE_STEP_COLUMNS, true) && (await validV4Table())) return 'unversioned-v4'; + if (samePrefix(REMOTE_STEP_V5_COLUMNS, true, true)) { + const definition = record( + await this.#db + .prepare( + 'SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?', + ) + .bind('table', REMOTE_STEP_TABLE) + .all(), + )?.results; + const sql = Array.isArray(definition) + ? record(definition[0])?.sql + : undefined; + if ( + typeof sql === 'string' && + sql.includes( + 'write_generation INTEGER NOT NULL CHECK (write_generation > 0)', + ) && + sql.includes('operation_key TEXT UNIQUE') && + sql.includes( + "operation_state TEXT CHECK (operation_state IN ('prepared', 'effect-admitted', 'receipt-recorded', 'resume-admitted', 'unresolved'))", + ) && + sql.includes( + 'CHECK ((operation_state IS NULL AND operation_key IS NULL', + ) && + sql.includes( + "(target_kind = 'agent' AND operation_key IS NOT NULL", + ) && + sql.includes( + "CHECK ((target_kind = 'agent' AND agent_id IS NOT NULL", + ) && + sql.includes("(target_kind = 'workflow' AND agent_id IS NULL") + ) + return 'unversioned-v5'; + } throw new RemoteStepRefusal(503, 'remote step schema is unreadable'); }; const stage = await readSchema(); - if (stage === 'current-v4') return; + if (stage === 'current-v5') return; if ( stage === 'prior-v1' || stage === 'prior-v2' || @@ -991,7 +1284,12 @@ export class RemoteStepJournal { ]); } catch (error) { const current = await readSchema(); - if (current !== 'prior-v3' && current !== 'current-v4') throw error; + if ( + current !== 'prior-v3' && + current !== 'current-v4' && + current !== 'current-v5' + ) + throw error; } return this.initialize(); } @@ -1022,7 +1320,7 @@ export class RemoteStepJournal { const replacement = `${REMOTE_STEP_TABLE}_v4`; try { await transactional.batch([ - this.#db.prepare(remoteStepTableSql(replacement)), + this.#db.prepare(remoteStepTableSql(replacement, 4)), this.#db.prepare(`INSERT INTO ${replacement} (${REMOTE_STEP_COLUMNS.join(', ')}) SELECT ${REMOTE_STEP_COLUMNS.slice(0, -4).join(', ')}, 'agent', '${REMOTE_STEP_MARKER}', NULL, 0 FROM ${REMOTE_STEP_TABLE}`), this.#db.prepare(`DROP TABLE ${REMOTE_STEP_TABLE}`), @@ -1038,9 +1336,10 @@ export class RemoteStepJournal { ), ]); } catch (error) { - if ((await readSchema()) !== 'current-v4') throw error; + if (!['current-v4', 'current-v5'].includes(await readSchema())) + throw error; } - return; + return this.initialize(); } if (stage === 'unversioned-v4') { try { @@ -1054,7 +1353,51 @@ export class RemoteStepJournal { ), ]); } catch (error) { - if ((await readSchema()) !== 'current-v4') throw error; + if (!['current-v4', 'current-v5'].includes(await readSchema())) + throw error; + } + return this.initialize(); + } + if (stage === 'current-v4') { + const replacement = `${REMOTE_STEP_TABLE}_v5`; + try { + await transactional.batch([ + this.#db.prepare(remoteStepTableSql(replacement)), + this.#db.prepare(`INSERT INTO ${replacement} (${REMOTE_STEP_V5_COLUMNS.join(', ')}) + SELECT ${REMOTE_STEP_COLUMNS.join(', ')}, 1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL + FROM ${REMOTE_STEP_TABLE}`), + this.#db.prepare(`DROP TABLE ${REMOTE_STEP_TABLE}`), + this.#db.prepare( + `ALTER TABLE ${replacement} RENAME TO ${REMOTE_STEP_TABLE}`, + ), + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_OPERATION_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_V5_WRITE_TRIGGER_SQL), + this.#db.prepare( + `UPDATE ${REMOTE_STEP_SCHEMA_TABLE} SET version = 5 WHERE id = 1 AND version = 4`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'current-v5') throw error; + } + return; + } + if (stage === 'unversioned-v5') { + try { + await transactional.batch([ + this.#db.prepare(REMOTE_STEP_RECOVERY_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_AGENT_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_WORKFLOW_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_OPERATION_INDEX_SQL), + this.#db.prepare(REMOTE_STEP_V5_WRITE_TRIGGER_SQL), + this.#db.prepare( + `INSERT INTO ${REMOTE_STEP_SCHEMA_TABLE} (id, version) VALUES (1, 5)`, + ), + ]); + } catch (error) { + if ((await readSchema()) !== 'current-v5') throw error; } return; } @@ -1158,8 +1501,8 @@ export class RemoteStepJournal { .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( invocation_id, principal_json, start_token, deployment_id, table_prefix, workflow_id, agent_id, thread_id, resource_id, run_id, step_path, suspended_at, resume_count, - request_json, deadline_at, status, target_kind, suspension_marker) - SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'agent', '${REMOTE_STEP_MARKER}' + request_json, deadline_at, status, target_kind, suspension_marker, write_generation) + SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'agent', '${REMOTE_STEP_MARKER}', 1 WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) @@ -1187,7 +1530,7 @@ export class RemoteStepJournal { .run(); const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} - SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1 + SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? AND step_path = ? AND suspended_at = ? AND resume_count = ? @@ -1268,9 +1611,9 @@ export class RemoteStepJournal { .prepare(`INSERT INTO ${REMOTE_STEP_TABLE} ( invocation_id, principal_json, start_token, deployment_id, table_prefix, workflow_id, run_id, step_path, suspended_at, resume_count, request_json, - deadline_at, status, target_kind, suspension_marker, request_digest) + deadline_at, status, target_kind, suspension_marker, request_digest, write_generation) SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 'workflow', - '${WORKFLOW_REMOTE_STEP_MARKER}', ? + '${WORKFLOW_REMOTE_STEP_MARKER}', ?, 1 WHERE EXISTS (SELECT 1 FROM "${address.tablePrefix}mastra_workflow_snapshot" WHERE workflow_name = ? AND run_id = ? AND snapshot = ? AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) @@ -1297,7 +1640,7 @@ export class RemoteStepJournal { const row = rowResult( await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} - SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1 + SET deadline_at = MIN(deadline_at, ?), write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE target_kind = 'workflow' AND deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND run_id = ? AND step_path = ? AND suspended_at = ? AND resume_count = ? @@ -1529,7 +1872,7 @@ export class RemoteStepJournal { status = 'dispatched', permit_generation = permit_generation + 1, permit_epoch = ?, permit_revision = ?, permit_expires_at = MIN(?, deadline_at), permit_consumed_at = NULL, - permit_digest = NULL, write_revision = write_revision + 1 + permit_digest = NULL, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ? AND permit_generation = ? AND deadline_at > ? AND status IN ('pending', 'dispatched') AND permit_consumed_at IS NULL AND request_json = ? @@ -1607,7 +1950,7 @@ export class RemoteStepJournal { const gate = await this.#admission(invocation, 16); const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} - SET permit_consumed_at = ?, permit_digest = ?, write_revision = write_revision + 1 + SET permit_consumed_at = ?, permit_digest = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ? AND permit_generation = ? AND permit_epoch = ? AND permit_revision = ? AND permit_expires_at = ? AND permit_expires_at > ? @@ -1722,7 +2065,7 @@ export class RemoteStepJournal { .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET result_json = ?, result_digest = ?, quiescence_digest = ?, executor_quiescent_at = ?, status = 'result-retained', - write_revision = write_revision + 1 + write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ? AND permit_digest = ? AND permit_consumed_at IS NOT NULL AND result_json IS NULL AND status = 'dispatched' @@ -1794,6 +2137,346 @@ export class RemoteStepJournal { }; } + async readNotificationOperation( + invocationId: string, + ): Promise { + const row = await this.#getRow(invocationId); + if (!row) return undefined; + decodeAny(row); + const operation = decodeNotificationOperation(row); + if ( + operation && + (await sha256Hex(operation.projectionJson)) !== operation.requestDigest + ) + throw new RemoteStepRefusal(503, 'remote step operation is unreadable'); + return operation; + } + + async prepareNotificationOperation( + invocationId: string, + operationId: string, + input: Omit, + ): Promise { + if (!isPathSafeId(operationId)) + throw new RemoteStepRefusal(400, 'remote step operation id is malformed'); + const stored = await this.#getRow(invocationId); + const invocation = + stored?.target_kind === 'agent' ? decode(stored) : undefined; + const retained = await this.readRetainedResult(invocationId); + if ( + !stored || + !invocation || + invocation.status !== 'result-retained' || + !retained + ) + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const operationKey = await sha256Hex( + JSON.stringify([ + invocation.deploymentId, + invocation.invocationId, + invocation.startToken, + invocation.stepPath, + invocation.suspendedAt, + invocation.resumeCount ?? -1, + operationId, + ]), + ); + const notificationId = operationKey; + const request = { ...input, id: notificationId }; + const projection = await projectNotificationOperation(request); + const requestJson = JSON.stringify(request); + if ( + new TextEncoder().encode(requestJson).length > + MAX_REMOTE_STEP_REQUEST_BYTES || + new TextEncoder().encode(projection.requestJson).length > + MAX_REMOTE_STEP_REQUEST_BYTES + ) + throw new RemoteStepRefusal( + 400, + 'remote step operation exceeds its limit', + ); + const existing = await this.readNotificationOperation(invocationId); + if (existing) { + if ( + existing.operationKey !== operationKey || + existing.requestJson !== requestJson || + existing.projectionJson !== projection.requestJson || + existing.requestDigest !== projection.requestDigest + ) + throw new RemoteStepRefusal(409, 'remote step operation changed'); + return existing; + } + const result = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET operation_key = ?, operation_id = ?, notification_id = ?, + notification_request_json = ?, notification_projection_version = ?, + notification_projection_json = ?, notification_request_digest = ?, + operation_state = 'prepared', + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND target_kind = 'agent' + AND status = 'result-retained' AND operation_state IS NULL + AND principal_json = ? AND start_token = ? AND result_digest = ? + AND write_revision = ? AND write_generation = ? + RETURNING *`) + .bind( + operationKey, + operationId, + notificationId, + requestJson, + projection.version, + projection.requestJson, + projection.requestDigest, + invocationId, + JSON.stringify(invocation.principal), + invocation.startToken, + retained.resultDigest, + stored.write_revision, + stored.write_generation, + ) + .all(), + ); + if (result) + return decodeNotificationOperation( + result, + ) as RemoteStepNotificationOperation; + const after = await this.readNotificationOperation(invocationId); + if ( + after?.operationKey === operationKey && + after.requestDigest === projection.requestDigest + ) + return after; + throw new RemoteStepRefusal(409, 'remote step operation changed'); + } + + async admitNotificationEffect( + invocationId: string, + ): Promise { + const stored = await this.#getRow(invocationId); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + const invocation = decode(stored); + const operation = await this.readNotificationOperation(invocationId); + if ( + !operation || + !['prepared', 'effect-admitted'].includes(operation.state) + ) + throw new RemoteStepRefusal(409, 'remote step operation is unavailable'); + if (operation.state === 'effect-admitted') return operation; + const retained = await this.readRetainedResult(invocationId); + if (!retained || invocation.status !== 'result-retained') + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const { suspension, snapshot } = await this.#stableSnapshot(invocation); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const now = this.#now(); + const base = [ + now, + invocationId, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + invocation.startToken, + JSON.stringify(invocation.principal), + retained.resultDigest, + operation.operationKey, + operation.requestDigest, + stored.write_revision, + stored.write_generation, + now, + invocation.workflowId, + invocation.runId, + snapshot, + ]; + const gate = await this.#admission(invocation, base.length); + try { + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET operation_state = 'effect-admitted', effect_admitted_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? + AND workflow_id = ? AND agent_id = ? AND thread_id = ? + AND resource_id = ? AND run_id = ? AND start_token = ? + AND principal_json = ? AND result_digest = ? + AND operation_key = ? AND notification_request_digest = ? + AND write_revision = ? AND write_generation = ? AND deadline_at > ? + AND target_kind = 'agent' AND status = 'result-retained' + AND operation_state = 'prepared' + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + AND ${gate.predicate} RETURNING *`) + .bind(...base, ...gate.values) + .all(), + ); + if (row) + return decodeNotificationOperation( + row, + ) as RemoteStepNotificationOperation; + } catch (error) { + const after = await this.readNotificationOperation(invocationId); + if ( + after?.state === 'effect-admitted' && + after.operationKey === operation.operationKey && + after.requestDigest === operation.requestDigest + ) + return after; + throw error; + } + const after = await this.readNotificationOperation(invocationId); + if ( + after?.state === 'effect-admitted' && + after.operationKey === operation.operationKey && + after.requestDigest === operation.requestDigest + ) + return after; + throw new RemoteStepRefusal(409, 'remote step effect admission changed'); + } + + async recordNotificationReceipt( + invocationId: string, + receipt: { + operationKey: string; + notificationId: string; + threadId: string; + requestDigest: string; + }, + ): Promise { + const stored = await this.#getRow(invocationId); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + decode(stored); + const operation = await this.readNotificationOperation(invocationId); + if ( + operation?.state !== 'effect-admitted' || + receipt.operationKey !== operation.operationKey || + receipt.notificationId !== operation.notificationId || + receipt.threadId !== stored.thread_id || + receipt.requestDigest !== operation.requestDigest + ) + throw new RemoteStepRefusal( + 409, + 'remote step notification receipt changed', + ); + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET operation_state = 'receipt-recorded', receipt_recorded_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND status = 'result-retained' + AND operation_state = 'effect-admitted' AND operation_key = ? + AND notification_id = ? AND notification_request_digest = ? + AND result_digest = ? AND write_revision = ? AND write_generation = ? + RETURNING *`) + .bind( + this.#now(), + invocationId, + operation.operationKey, + operation.notificationId, + operation.requestDigest, + stored.result_digest, + stored.write_revision, + stored.write_generation, + ) + .all(), + ); + if (row) + return decodeNotificationOperation( + row, + ) as RemoteStepNotificationOperation; + throw new RemoteStepRefusal( + 409, + 'remote step notification receipt changed', + ); + } + + async admitNotificationResume(invocationId: string): Promise { + const stored = await this.#getRow(invocationId); + if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); + const invocation = decode(stored); + const operation = await this.readNotificationOperation(invocationId); + if ( + operation?.state !== 'receipt-recorded' || + invocation.status !== 'result-retained' + ) + throw new RemoteStepRefusal( + 409, + 'remote step continuation is unavailable', + ); + const retained = await this.readRetainedResult(invocationId); + if (!retained) + throw new RemoteStepRefusal(409, 'remote step result is unavailable'); + const { suspension, snapshot } = await this.#stableSnapshot(invocation); + if ( + suspension.suspendedAt !== invocation.suspendedAt || + suspension.resumeCount !== invocation.resumeCount || + suspension.requestJson !== JSON.stringify(invocation.request) || + JSON.stringify(suspension.stepPath) !== + JSON.stringify(invocation.stepPath) + ) + throw new RemoteStepRefusal(409, 'remote step suspension changed'); + const now = this.#now(); + const owner = crypto.randomUUID(); + const base = [ + owner, + now + 60_000, + now, + invocationId, + invocation.deploymentId, + invocation.tablePrefix, + invocation.workflowId, + invocation.agentId, + invocation.threadId, + invocation.resourceId, + invocation.runId, + invocation.startToken, + JSON.stringify(invocation.principal), + retained.resultDigest, + operation.operationKey, + operation.requestDigest, + stored.write_revision, + stored.write_generation, + now, + invocation.workflowId, + invocation.runId, + snapshot, + ]; + const gate = await this.#admission(invocation, base.length); + const row = rowResult( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ?, + operation_state = 'resume-admitted', resume_admitted_at = ?, + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? + AND workflow_id = ? AND agent_id = ? AND thread_id = ? + AND resource_id = ? AND run_id = ? AND start_token = ? + AND principal_json = ? AND result_digest = ? AND operation_key = ? + AND notification_request_digest = ? AND write_revision = ? + AND write_generation = ? AND deadline_at > ? + AND operation_state = 'receipt-recorded' AND status = 'result-retained' + AND receipt_recorded_at IS NOT NULL AND resume_admitted_at IS NULL + AND EXISTS (SELECT 1 FROM "${invocation.tablePrefix}mastra_workflow_snapshot" + WHERE workflow_name = ? AND run_id = ? AND snapshot = ? + AND ${SNAPSHOT_EXECUTION_OPEN_SQL}) + AND ${gate.predicate} RETURNING invocation_id`) + .bind(...base, ...gate.values) + .all(), + ); + if (!row) + throw new RemoteStepRefusal(409, 'remote step continuation changed'); + return owner; + } + async claimResume(invocationId: string, leaseMs: number): Promise { if ( !isPathSafeId(invocationId) || @@ -1816,8 +2499,9 @@ export class RemoteStepJournal { const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = 'resuming', resume_lease_owner = ?, resume_lease_expires_at = ?, - write_revision = write_revision + 1 - WHERE invocation_id = ? AND deployment_id = ? AND table_prefix = ? + write_revision = write_revision + 1, write_generation = write_generation + 1 + WHERE invocation_id = ? AND operation_state IS NULL + AND deployment_id = ? AND table_prefix = ? AND workflow_id = ? AND agent_id = ? AND thread_id = ? AND resource_id = ? AND run_id = ? AND start_token = ? AND permit_generation = ? AND permit_epoch IS ? @@ -1931,7 +2615,7 @@ export class RemoteStepJournal { const result = await this.#db .prepare(`UPDATE ${REMOTE_STEP_TABLE} SET status = ?, resume_lease_owner = NULL, resume_lease_expires_at = NULL, - settled_at = ?, settlement_outcome = ?, write_revision = write_revision + 1 + settled_at = ?, settlement_outcome = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ? AND status = 'resuming' AND resume_lease_owner = ? AND permit_generation = ? AND permit_epoch IS ? AND start_token = ? @@ -1986,6 +2670,7 @@ export class RemoteStepJournal { FROM ${REMOTE_STEP_TABLE} INDEXED BY ${REMOTE_STEP_RECOVERY_INDEX} WHERE invocation_id > ? AND target_kind = 'agent' + AND operation_state IS NULL AND status IN ('result-retained', 'resuming') ORDER BY invocation_id LIMIT ?`) .bind(afterInvocationId, limit) diff --git a/packages/flowsafe/src/signals/notifications-d1.test.ts b/packages/flowsafe/src/signals/notifications-d1.test.ts index ba50b9f6..9416f671 100644 --- a/packages/flowsafe/src/signals/notifications-d1.test.ts +++ b/packages/flowsafe/src/signals/notifications-d1.test.ts @@ -198,7 +198,7 @@ describe('host-owned notification operation receipt', () => { 'remote-invocation', input, ); - expect(receipt).toEqual({ + expect(receipt).toMatchObject({ operationKey: 'remote-invocation', notificationId: input.id, threadId: input.threadId, @@ -207,6 +207,7 @@ describe('host-owned notification operation receipt', () => { .prepare('SELECT * FROM flowsafe_notification_operation_receipts') .first>(); expect(retained?.request_digest).toMatch(/^[a-f0-9]{64}$/); + expect(receipt.requestDigest).toBe(retained?.request_digest); expect(JSON.stringify(retained)).not.toContain(input.summary); expect(JSON.stringify(retained)).not.toContain('private-receipt-canary'); const delivered = await retry.getNotification({ diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index 2553dd16..f144b807 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -333,10 +333,75 @@ function isDuplicateColumn(error: unknown): boolean { * a template literal is one the census can only match by restating it. */ export const NOTIFICATION_SEQUENCE_TABLE = 'flowsafe_notification_sequence'; +export const NOTIFICATION_OPERATION_PROJECTION_VERSION = + 'notification-record-values-v1'; const NOTIFICATION_OPERATION_RECEIPTS_TABLE = 'flowsafe_notification_operation_receipts'; export const DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT = 100_000; +export type NotificationOperationInput = CreateNotificationInput & { + id: string; + createdAt: Date; +}; + +export async function projectNotificationOperation( + input: NotificationOperationInput, +): Promise<{ + version: typeof NOTIFICATION_OPERATION_PROJECTION_VERSION; + values: unknown[]; + requestJson: string; + requestDigest: string; +}> { + if ( + !isPathSafeId(input.id) || + !isPathSafeId(input.threadId) || + !(input.createdAt instanceof Date) || + !Number.isFinite(input.createdAt.getTime()) + ) + throw new TypeError('notification operation identity is malformed'); + const record: NotificationRecord = { + id: input.id, + threadId: input.threadId, + source: input.source, + kind: input.kind, + priority: input.priority ?? 'medium', + status: 'pending', + summary: input.summary, + payload: input.payload, + resourceId: input.resourceId, + agentId: input.agentId, + sourceId: input.sourceId, + dedupeKey: input.dedupeKey, + coalesceKey: input.coalesceKey, + coalescedCount: 1, + attributes: input.attributes, + createdAt: input.createdAt, + updatedAt: input.createdAt, + deliverAt: input.deliverAt, + summaryAt: input.summaryAt, + deliveryReason: input.deliveryReason, + deliveryAttempts: 0, + metadata: input.metadata, + }; + const values = recordValues(record); + validateNotificationDeliveryValues(values, true); + const requestJson = JSON.stringify(values); + const digestBytes = new Uint8Array( + await crypto.subtle.digest( + 'SHA-256', + new TextEncoder().encode(requestJson), + ), + ); + return { + version: NOTIFICATION_OPERATION_PROJECTION_VERSION, + values, + requestJson, + requestDigest: Array.from(digestBytes, (byte) => + byte.toString(16).padStart(2, '0'), + ).join(''), + }; +} + export class D1NotificationsStorage extends NotificationsStorage implements NotificationDeliveryStorage @@ -575,11 +640,12 @@ export class D1NotificationsStorage async createNotificationOnce( operationKey: string, - input: CreateNotificationInput & { id: string; createdAt: Date }, + input: NotificationOperationInput, ): Promise<{ operationKey: string; notificationId: string; threadId: string; + requestDigest: string; }> { if ( !isPathSafeId(operationKey) || @@ -594,46 +660,12 @@ export class D1NotificationsStorage const batch = this.#db.batch?.bind(this.#db); if (!batch) throw new Error('notification operation requires database.batch()'); - const record: NotificationRecord = { - id: input.id, - threadId: input.threadId, - source: input.source, - kind: input.kind, - priority: input.priority ?? 'medium', - status: 'pending', - summary: input.summary, - payload: input.payload, - resourceId: input.resourceId, - agentId: input.agentId, - sourceId: input.sourceId, - dedupeKey: input.dedupeKey, - coalesceKey: input.coalesceKey, - coalescedCount: 1, - attributes: input.attributes, - createdAt: input.createdAt, - updatedAt: input.createdAt, - deliverAt: input.deliverAt, - summaryAt: input.summaryAt, - deliveryReason: input.deliveryReason, - deliveryAttempts: 0, - metadata: input.metadata, - }; - const values = recordValues(record); - validateNotificationDeliveryValues(values, true); - const requestJson = JSON.stringify(values); - const digestBytes = new Uint8Array( - await crypto.subtle.digest( - 'SHA-256', - new TextEncoder().encode(requestJson), - ), - ); - const requestDigest = Array.from(digestBytes, (byte) => - byte.toString(16).padStart(2, '0'), - ).join(''); + const { values, requestDigest } = await projectNotificationOperation(input); const receipt = { operationKey, notificationId: input.id, threadId: input.threadId, + requestDigest, }; const readReceipt = async () => this.#db @@ -655,6 +687,7 @@ export class D1NotificationsStorage operationKey, notificationId: existing.notification_id, threadId: existing.thread_id, + requestDigest: existing.request_digest, }; } try { @@ -714,10 +747,49 @@ export class D1NotificationsStorage operationKey, notificationId: committed.notification_id, threadId: committed.thread_id, + requestDigest: committed.request_digest, }; } } + async readNotificationOperationReceipt(operationKey: string): Promise< + | { + operationKey: string; + notificationId: string; + threadId: string; + requestDigest: string; + } + | undefined + > { + if (!isPathSafeId(operationKey)) + throw new TypeError('notification operation identity is malformed'); + await this.#ensureSchema(); + const row = await this.#db + .prepare( + `SELECT request_digest, thread_id, notification_id + FROM ${this.#operationReceiptTable} WHERE operation_key = ?`, + ) + .bind(operationKey) + .first<{ + request_digest: string; + thread_id: string; + notification_id: string; + }>(); + if (!row) return undefined; + if ( + !/^[a-f0-9]{64}$/.test(row.request_digest) || + !isPathSafeId(row.thread_id) || + !isPathSafeId(row.notification_id) + ) + throw new Error('notification operation receipt is unreadable'); + return { + operationKey, + notificationId: row.notification_id, + threadId: row.thread_id, + requestDigest: row.request_digest, + }; + } + async createNotification( input: CreateNotificationInput, ): Promise { From c0ede41d6622e58ce25c7fc19cd63ecb44c237f9 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Wed, 14 Oct 2026 22:26:44 +0400 Subject: [PATCH 44/73] Verify notification effects before FlowSafe resume admission Require the atomic D1 notification receipt in the journal transition and keep unresolved continuation rows in drain inventory. Apply the keyset cursor to both inventory branches. --- packages/flowsafe/src/do-runner/inventory.ts | 30 ++- .../src/do-runner/remote-step-journal.test.ts | 175 ++++++++++++++++++ .../src/do-runner/remote-step-journal.ts | 8 + .../flowsafe/src/signals/notifications-d1.ts | 2 +- 4 files changed, 213 insertions(+), 2 deletions(-) diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index 21283641..70765d1a 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -893,7 +893,10 @@ export class DeploymentInventory { `unknown inventory category '${String(category)}'`, ); } - const query = this.#queryFor(category); + const query = + category === 'remote-steps' + ? await this.#remoteStepQuery() + : this.#queryFor(category); const table = this.#tableFor(descriptor); const cursor = parseCursor(options.cursor, category, query.key.length); const limit = clampLimit(options.limit); @@ -959,6 +962,31 @@ export class DeploymentInventory { : descriptor.table; } + async #remoteStepQuery(): Promise { + const base = this.#queryFor('remote-steps'); + const { results } = await this.#db + .prepare('SELECT name FROM pragma_table_xinfo(?)') + .bind(REMOTE_STEP_TABLE) + .all<{ name: unknown }>(); + const columns = new Set(results.map((row) => row.name)); + if (!columns.has('operation_state')) return base; + if (!columns.has('resume_admitted_at')) + throw new Error('remote step inventory schema is unreadable'); + return { + ...base, + detail: [ + ...base.detail, + 'operation_state', + 'resume_admitted_at', + 'settlement_outcome', + ], + where: `((${base.where}) OR + (operation_state IN ('resume-admitted', 'unresolved') + AND status IN ('failed', 'settled') + AND settlement_outcome IS NOT 'applied'))`, + }; + } + async #page( table: string, query: CategoryQuery, diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index ae91e982..ac57a834 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -1678,6 +1678,152 @@ describe('remote step journal dispatch admission', () => { ).toBe(1); }); + it('retains an unwitnessed notification continuation in drain inventory after terminal failure', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const input = { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + input, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + const receipt = await new D1NotificationsStorage( + h.db, + ).createNotificationOnce(operation.operationKey, { + ...input, + id: operation.notificationId, + }); + await h.journal.recordNotificationReceipt(invocation.invocationId, receipt); + const owner = await h.journal.admitNotificationResume( + invocation.invocationId, + ); + h.setSummary({ runId: ADDRESS.runId, status: 'cancelled' }); + expect( + await h.journal.settleAfterResume(invocation.invocationId, owner), + ).toBe('terminal-not-success'); + const inventory = await new DeploymentInventory(h.db).read('remote-steps'); + expect(inventory.count).toBe(1); + expect(inventory.entries[0]).toMatchObject({ + key: [invocation.invocationId], + detail: { + status: 'failed', + operation_state: 'resume-admitted', + settlement_outcome: 'terminal-not-success', + }, + }); + }); + + it('advances a limit-one remote-step inventory cursor past active rows', async () => { + const h = await fixture(); + const first = await h.retainResult(); + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': 600 }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: 'call-2', + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + const second = await h.journal.register(ADDRESS, 10_000, PRINCIPAL); + expect(second.invocationId).not.toBe(first.invocation.invocationId); + const inventory = new DeploymentInventory(h.db); + const page1 = await inventory.read('remote-steps', { limit: 1 }); + const page2 = await inventory.read('remote-steps', { + limit: 1, + cursor: page1.cursor, + }); + expect(page1.count).toBe(2); + expect(page2.entries).toHaveLength(1); + expect(page2.entries[0]?.key).not.toEqual(page1.entries[0]?.key); + expect(page2.count).toBeUndefined(); + expect( + ( + await inventory.read('remote-steps', { + limit: 1, + cursor: page2.cursor, + }) + ).entries, + ).toEqual([]); + }); + + it('refuses a second continuation admission after a committed response is lost and the lease expires', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + h.sqlite + .prepare(`UPDATE ${REMOTE_STEP_TABLE} + SET deadline_at = ?, write_revision = write_revision + 1, + write_generation = write_generation + 1 WHERE invocation_id = ?`) + .run(100_000, invocation.invocationId); + const input = { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + input, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + const receipt = await new D1NotificationsStorage( + h.db, + ).createNotificationOnce(operation.operationKey, { + ...input, + id: operation.notificationId, + }); + await h.journal.recordNotificationReceipt(invocation.invocationId, receipt); + const prepare = h.db.prepare.bind(h.db); + h.db.prepare = (sql) => { + const statement = prepare(sql); + if (!sql.includes("operation_state = 'resume-admitted'")) + return statement; + return { + ...statement, + bind: (...values) => { + const bound = statement.bind(...values); + return { + ...bound, + all: async () => { + await bound.all(); + throw new Error('lost continuation admission response'); + }, + }; + }, + }; + }; + await expect( + h.journal.admitNotificationResume(invocation.invocationId), + ).rejects.toThrow('lost continuation admission response'); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ + state: 'resume-admitted', + requestDigest: receipt.requestDigest, + }); + h.setNow(61_001); + await expect( + h.reopen().admitNotificationResume(invocation.invocationId), + ).rejects.toMatchObject({ status: 409 }); + }); + it('refuses effect admission when the authoritative snapshot changes at the D1 write', async () => { const h = await fixture(); const { invocation } = await h.retainResult(); @@ -1749,6 +1895,35 @@ describe('remote step journal dispatch admission', () => { ).toMatchObject({ state: 'effect-admitted' }); }); + it('refuses a matching claimed receipt when D1 has no notification effect', async () => { + const h = await fixture(); + const { invocation } = await h.retainResult(); + const operation = await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + await new D1NotificationsStorage(h.db).init(); + await h.journal.admitNotificationEffect(invocation.invocationId); + await expect( + h.journal.recordNotificationReceipt(invocation.invocationId, { + operationKey: operation.operationKey, + notificationId: operation.notificationId, + threadId: ADDRESS.threadId, + requestDigest: operation.requestDigest, + }), + ).rejects.toMatchObject({ status: 409 }); + expect( + await h.reopen().readNotificationOperation(invocation.invocationId), + ).toMatchObject({ state: 'effect-admitted' }); + }); + it('refuses stale principal, start token, result and deadline before effect admission', async () => { const cases = [ { diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 6e1f1fe0..17833837 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -7,6 +7,7 @@ import { } from '../approval-api/principal.js'; import { NOTIFICATION_OPERATION_PROJECTION_VERSION, + NOTIFICATION_OPERATION_RECEIPTS_TABLE, type NotificationOperationInput, projectNotificationOperation, } from '../signals/notifications-d1.js'; @@ -2376,6 +2377,9 @@ export class RemoteStepJournal { AND operation_state = 'effect-admitted' AND operation_key = ? AND notification_id = ? AND notification_request_digest = ? AND result_digest = ? AND write_revision = ? AND write_generation = ? + AND EXISTS (SELECT 1 FROM "${stored.table_prefix}${NOTIFICATION_OPERATION_RECEIPTS_TABLE}" + WHERE operation_key = ? AND request_digest = ? + AND thread_id = ? AND notification_id = ?) RETURNING *`) .bind( this.#now(), @@ -2386,6 +2390,10 @@ export class RemoteStepJournal { stored.result_digest, stored.write_revision, stored.write_generation, + operation.operationKey, + operation.requestDigest, + stored.thread_id, + operation.notificationId, ) .all(), ); diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index f144b807..d37b8438 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -335,7 +335,7 @@ function isDuplicateColumn(error: unknown): boolean { export const NOTIFICATION_SEQUENCE_TABLE = 'flowsafe_notification_sequence'; export const NOTIFICATION_OPERATION_PROJECTION_VERSION = 'notification-record-values-v1'; -const NOTIFICATION_OPERATION_RECEIPTS_TABLE = +export const NOTIFICATION_OPERATION_RECEIPTS_TABLE = 'flowsafe_notification_operation_receipts'; export const DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT = 100_000; From a3af9451beb1fada6eb727b0ca385edfbf69deb9 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Wed, 14 Oct 2026 23:07:20 +0400 Subject: [PATCH 45/73] Run selected notification effects under FlowSafe host authority Use same-D1 effect admission and receipts before one-time continuation. Retry from frozen journal arguments after profile changes; keep general Mastra replay and scheduled recovery closed. --- .../src/agent-host/thread-host.test.ts | 210 ++++++++++++++++++ .../flowsafe/src/agent-host/thread-host.ts | 163 +++++++++++++- .../src/do-runner/remote-step-journal.ts | 16 ++ 3 files changed, 387 insertions(+), 2 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index fda21e66..a41da822 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -273,6 +273,9 @@ function harness( runId: string, ) => Promise; remoteSteps?: RemoteStepContinuationReader; + remoteStepNotification?: NonNullable< + import('./thread-host.js').ThreadAgentHostOptions['remoteStepNotification'] + >; stateStorage?: AgentThreadStateStorage; resourceIdFromKey?: (threadId: string) => string; } = {}, @@ -549,6 +552,9 @@ function harness( ? { discardScheduleDispatch: options.discardScheduleDispatch } : {}), ...(options.remoteSteps ? { remoteSteps: options.remoteSteps } : {}), + ...(options.remoteStepNotification + ? { remoteStepNotification: options.remoteStepNotification } + : {}), approvalService: (instanceScope) => { approvalScopes.push(instanceScope); if (options.approvalService) return options.approvalService; @@ -7951,6 +7957,210 @@ it('resumes a retained mapped-resource result from D1 after the agent host is re expect(resume).toHaveBeenCalledTimes(1); }); +it('admits a configured notification effect once before agent continuation', async () => { + const fixture = await hostAgentLifecycleFixture({ + mode: 'fenced', + provenance: 'modern', + }); + onTestFinished(() => { + vi.restoreAllMocks(); + fixture.sql.close(); + }); + fixture.snapshot.suspendedPaths = { tool: [0] }; + Object.assign(fixture.snapshot.context, { + tool: { + status: 'suspended', + payload: {}, + startedAt: 50, + suspendedAt: 1000, + suspendPayload: { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { notificationOperationId: 'enqueue-v1' }, + }, + }, + }, + }); + await fixture.persist(); + await fixture.host.route(hostAgentLifecycleRequest(), fixture.scope); + const journal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: fixture.app.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + workflowId: fixture.execution.workflowId, + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + }, + Date.now() + 60_000, + fixture.scope.principal, + ); + const permit = await journal.issuePermit(invocation.invocationId, 5_000); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"untrusted":"result"}', + '{"receipt":"complete"}', + ); + const freshApp = init( + { storage: fixture.storage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const reopened = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: freshApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const host = harness(['writer'], { + init: freshApp, + storage: fixture.storage, + stateStorage: fixture.stateStorage, + resourceAccess: fixture.resources, + remoteSteps: reopened, + remoteStepNotification: { + journal: reopened, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + payload: { fixed: true }, + }, + }); + const resume = vi + .spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime') + .mockResolvedValue({ runId: 'acme_run', status: 'suspended' }); + const request = () => + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }); + const admission = vi + .spyOn(reopened, 'admitNotificationEffect') + .mockRejectedValueOnce(new Error('effect admission refused')); + await expect(host.host.route(request(), host.scope)).rejects.toThrow( + 'effect admission refused', + ); + expect( + fixture.sql + .prepare( + "SELECT COUNT(*) AS count FROM sqlite_master WHERE name = 'host_lifecycle_mastra_notifications'", + ) + .get(), + ).toMatchObject({ count: 0 }); + expect(resume).not.toHaveBeenCalled(); + admission.mockRestore(); + const recordReceipt = reopened.recordNotificationReceipt.bind(reopened); + const lostReceipt = vi + .spyOn(reopened, 'recordNotificationReceipt') + .mockImplementationOnce(async (...args) => { + await recordReceipt(...args); + throw new Error('receipt acknowledgement lost'); + }); + await expect(host.host.route(request(), host.scope)).rejects.toThrow( + 'receipt acknowledgement lost', + ); + expect(resume).not.toHaveBeenCalled(); + lostReceipt.mockRestore(); + const restartedStorage = createD1Storage({ + binding: fixture.binding, + tablePrefix: 'host_lifecycle_', + }); + await restartedStorage.init(); + const restartedApp = init( + { storage: restartedStorage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const restartedJournal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: restartedApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const restartedHost = harness(['writer'], { + init: restartedApp, + storage: restartedStorage, + stateStorage: { ...fixture.stateStorage }, + resourceAccess: fixture.resources, + remoteSteps: restartedJournal, + remoteStepNotification: { + journal: restartedJournal, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Changed after deployment', + payload: { fixed: false }, + }, + }); + await expect( + restartedHost.host.route(request(), restartedHost.scope), + ).rejects.toMatchObject({ + status: 503, + }); + const operation = await restartedJournal.readNotificationOperation( + invocation.invocationId, + ); + expect(operation?.state).toBe('resume-admitted'); + expect( + await restartedJournal + .notificationStorage() + .readNotificationOperationReceipt(operation?.operationKey ?? ''), + ).toMatchObject({ + requestDigest: operation?.requestDigest, + notificationId: operation?.notificationId, + }); + expect( + fixture.sql + .prepare( + 'SELECT source, kind, summary, payload FROM host_lifecycle_mastra_notifications', + ) + .get(), + ).toMatchObject({ + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + payload: '{"fixed":true}', + }); + expect(resume).toHaveBeenCalledTimes(1); + await expect( + restartedHost.host.route(request(), restartedHost.scope), + ).rejects.toMatchObject({ + status: 503, + }); + expect(resume).toHaveBeenCalledTimes(1); + expect( + fixture.sql + .prepare( + 'SELECT COUNT(*) AS count FROM host_lifecycle_mastra_notifications', + ) + .get(), + ).toMatchObject({ count: 1 }); +}); + describe('agent host selector lookup isolation', () => { it.each([ ['modern', 'status'], diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index 178ba949..506333a3 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -82,7 +82,10 @@ import { } from '../do-runner/index.js'; import { isDefinitiveInitialAdmissionRefusal } from '../do-runner/initial-admission-refusal.js'; import { mastraRegistryEntries } from '../do-runner/mastra-registry.js'; -import { REMOTE_STEP_MARKER } from '../do-runner/remote-step-journal.js'; +import { + REMOTE_STEP_MARKER, + type RemoteStepJournal, +} from '../do-runner/remote-step-journal.js'; import { exceedsRunInputDepth, runInputDepthMessage, @@ -110,6 +113,10 @@ import { abandonApprovalsForRun, reconcileApprovalsForSummary, } from '../host-kit/index.js'; +import { + type NotificationOperationInput, + projectNotificationOperation, +} from '../signals/notifications-d1.js'; import { createAgentModuleCatalog } from './catalog.js'; import { AGENT_HOST_ROUTE_PREFIX, @@ -174,6 +181,16 @@ export type { export interface ThreadAgentHostOptions { remoteSteps?: RemoteStepContinuationReader; + remoteStepNotification?: { + journal: RemoteStepJournal; + agentId: string; + operationId: string; + source: string; + kind: string; + summary: string; + payload?: unknown; + receiptLimit?: number; + }; resourceIdFromKey?: (threadId: string) => string; /** Narrows automated entry beyond what each agent's metadata declares. */ authorizeAutomatedEntry?: AutomatedEntryAuthorizer; @@ -624,6 +641,24 @@ function ndjson( export function createThreadAgentHost( options: ThreadAgentHostOptions, ): ThreadAgentHost { + const notificationProfile = options.remoteStepNotification; + if ( + notificationProfile && + (options.remoteSteps !== notificationProfile.journal || + !isPathSafeId(notificationProfile.agentId) || + !isPathSafeId(notificationProfile.operationId) || + !notificationProfile.source || + !notificationProfile.kind || + !notificationProfile.summary || + (notificationProfile.receiptLimit !== undefined && + (!Number.isSafeInteger(notificationProfile.receiptLimit) || + notificationProfile.receiptLimit < 0))) + ) + throw new TypeError('remote step notification profile is malformed'); + const notificationReceiptLimit = notificationProfile?.receiptLimit; + let notificationStorage: + | ReturnType + | undefined; const resourceForThread = (threadId: string): string => { let resourceId: string; try { @@ -2886,7 +2921,131 @@ export function createThreadAgentHost( principalPermissions, safeContext: selected.safeContext, }; - const leaseOwner = await reader.claimResume(invocationId, 60_000); + let leaseOwner: string; + if ( + notificationProfile && + invocation.agentId === notificationProfile.agentId && + record(invocation.request)?.notificationOperationId === + notificationProfile.operationId + ) { + const journal = notificationProfile.journal; + const existing = + await journal.readNotificationOperation(invocationId); + if ( + existing && + (existing.invocationId !== invocationId || + existing.operationId !== notificationProfile.operationId) + ) + throw new AgentHostRequestError( + 409, + 'remote step operation changed', + ); + const operation = + existing ?? + (await journal.prepareNotificationOperation( + invocationId, + notificationProfile.operationId, + { + threadId: invocation.threadId, + resourceId: invocation.resourceId, + agentId: invocation.agentId, + source: notificationProfile.source, + kind: notificationProfile.kind, + summary: notificationProfile.summary, + sourceId: invocation.invocationId, + createdAt: new Date(retained.executorQuiescentAt), + ...(notificationProfile.payload === undefined + ? {} + : { payload: notificationProfile.payload }), + }, + )); + const savedInput = JSON.parse(operation.requestJson) as Record< + string, + unknown + >; + if (typeof savedInput?.createdAt !== 'string') + throw new AgentHostRequestError( + 503, + 'remote step operation is unreadable', + ); + const stableInput = { + ...savedInput, + createdAt: new Date(savedInput.createdAt), + } as NotificationOperationInput; + const projection = await projectNotificationOperation(stableInput); + if ( + JSON.stringify(stableInput) !== operation.requestJson || + stableInput.id !== operation.notificationId || + stableInput.threadId !== invocation.threadId || + stableInput.resourceId !== invocation.resourceId || + stableInput.agentId !== invocation.agentId || + stableInput.sourceId !== invocation.invocationId || + savedInput.createdAt !== + new Date(retained.executorQuiescentAt).toISOString() || + projection.version !== operation.projectionVersion || + projection.requestJson !== operation.projectionJson || + projection.requestDigest !== operation.requestDigest + ) + throw new AgentHostRequestError( + 503, + 'remote step operation is unreadable', + ); + if (!notificationStorage) + notificationStorage = journal.notificationStorage( + notificationReceiptLimit, + ); + const notifications = notificationStorage; + if ( + operation.state === 'prepared' || + operation.state === 'effect-admitted' + ) { + const admitted = + await journal.admitNotificationEffect(invocationId); + if ( + admitted.operationKey !== operation.operationKey || + admitted.requestDigest !== operation.requestDigest + ) + throw new AgentHostRequestError( + 409, + 'remote step operation changed', + ); + const receipt = await notifications.createNotificationOnce( + admitted.operationKey, + stableInput, + ); + await journal.recordNotificationReceipt(invocationId, receipt); + } else if (operation.state === 'receipt-recorded') { + const receipt = + await notifications.readNotificationOperationReceipt( + operation.operationKey, + ); + if ( + receipt?.requestDigest !== operation.requestDigest || + receipt.notificationId !== operation.notificationId || + receipt.threadId !== invocation.threadId + ) + throw new AgentHostRequestError( + 503, + 'remote step notification receipt is unreadable', + ); + } else { + throw new AgentHostRequestError( + 503, + 'remote step continuation is unresolved', + ); + } + execution.principalPermissions = ( + await authorize( + scope, + ref.agentId, + 'remote-step.resume', + stored.principal, + ) + ).principalPermissions; + leaseOwner = await journal.admitNotificationResume(invocationId); + } else { + leaseOwner = await reader.claimResume(invocationId, 60_000); + } const resumed = await withExecution(execution, () => durable.resumeViaRuntime({ runId: ref.runId, diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index 17833837..a40ee9a3 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -5,7 +5,10 @@ import { assertExecutionPrincipal, type ExecutionPrincipal, } from '../approval-api/principal.js'; +import type { SignalDatabase } from '../signals/d1-shared.js'; import { + D1NotificationsStorage, + DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, NOTIFICATION_OPERATION_PROJECTION_VERSION, NOTIFICATION_OPERATION_RECEIPTS_TABLE, type NotificationOperationInput, @@ -863,6 +866,19 @@ export class RemoteStepJournal { return this.#tablePrefix; } + notificationStorage( + operationReceiptLimit = DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT, + ): D1NotificationsStorage { + const db = this.#db as ExecutionFenceDatabase & SignalDatabase; + if (typeof db.batch !== 'function') + throw new Error('notification operation requires database.batch()'); + return new D1NotificationsStorage( + db, + this.#tablePrefix, + operationReceiptLimit, + ); + } + constructor(input: { db: ExecutionFenceDatabase; fence: ExecutionFenceStore; From ddcf49707a6d23cbe13fe2a1d871a3491510da4f Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Wed, 14 Oct 2026 23:40:31 +0400 Subject: [PATCH 46/73] Recover admitted notification effects through trusted host Retry the frozen same-key D1 effect after deadline under shared host authority checks without retrying Mastra. Scheduled cursor and alarm routing remain a separate default-off checkpoint. --- .../src/agent-host/thread-host.test.ts | 230 +++++++++++++++++ .../flowsafe/src/agent-host/thread-host.ts | 231 +++++++++++++----- 2 files changed, 394 insertions(+), 67 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index a41da822..7454ac39 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -60,6 +60,7 @@ import { D1SchedulesStorage, type ScheduleDatabase, } from '../schedules/schedules-d1.js'; +import { D1NotificationsStorage } from '../signals/notifications-d1.js'; import { type AgentThreadInstanceScope, type AgentThreadStateStorage, @@ -8161,6 +8162,235 @@ it('admits a configured notification effect once before agent continuation', asy ).toMatchObject({ count: 1 }); }); +it.each([ + 'before write', + 'after write', +] as const)('recovers an admitted notification after its deadline without resuming Mastra (%s)', async (failurePoint) => { + const fixture = await hostAgentLifecycleFixture({ + mode: 'fenced', + provenance: 'modern', + }); + onTestFinished(() => { + vi.restoreAllMocks(); + fixture.sql.close(); + }); + fixture.snapshot.suspendedPaths = { tool: [0] }; + Object.assign(fixture.snapshot.context, { + tool: { + status: 'suspended', + payload: {}, + startedAt: 50, + suspendedAt: 1000, + suspendPayload: { + type: 'suspension', + toolName: 'remote_lookup', + toolCallId: 'call-1', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { notificationOperationId: 'enqueue-v1' }, + }, + }, + }, + }); + await fixture.persist(); + await fixture.host.route(hostAgentLifecycleRequest(), fixture.scope); + const journal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: fixture.app.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + await journal.initialize(); + const invocation = await journal.register( + { + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + workflowId: fixture.execution.workflowId, + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + }, + Date.now() + 60_000, + fixture.scope.principal, + ); + const permit = await journal.issuePermit(invocation.invocationId, 5_000); + const consumed = await journal.consumePermit(permit); + if (consumed.kind !== 'accepted') throw new Error('permit was not won'); + await journal.recordResult( + invocation.invocationId, + consumed.rawPermit, + '{"untrusted":"result"}', + '{"receipt":"complete"}', + ); + const freshApp = init( + { storage: fixture.storage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const reopened = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: freshApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const host = harness(['writer'], { + init: freshApp, + storage: fixture.storage, + stateStorage: fixture.stateStorage, + resourceAccess: fixture.resources, + remoteSteps: reopened, + remoteStepNotification: { + journal: reopened, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + }, + }); + const retained = await reopened.readRetainedResult(invocation.invocationId); + if (!retained) throw new Error('retained result is missing'); + await reopened.prepareNotificationOperation( + invocation.invocationId, + 'enqueue-v1', + { + threadId: invocation.threadId, + resourceId: invocation.resourceId, + agentId: invocation.agentId, + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + sourceId: invocation.invocationId, + createdAt: new Date(retained.executorQuiescentAt), + }, + ); + expect( + await host.host.recoverNotificationOperation( + host.scope, + invocation.invocationId, + ), + ).toBe('unresolved'); + const resume = vi.spyOn(FlowsafeDurableAgent.prototype, 'resumeViaRuntime'); + const create = D1NotificationsStorage.prototype.createNotificationOnce; + const lostCall = vi + .spyOn(D1NotificationsStorage.prototype, 'createNotificationOnce') + .mockImplementationOnce(async function ( + this: D1NotificationsStorage, + ...args: Parameters + ) { + if (failurePoint === 'after write') await create.apply(this, args); + throw new Error('executor unavailable after admission'); + }); + await expect( + host.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + host.scope, + ), + ).rejects.toThrow('executor unavailable after admission'); + lostCall.mockRestore(); + expect( + (await journal.readNotificationOperation(invocation.invocationId))?.state, + ).toBe('effect-admitted'); + await fixture.binding + .prepare( + 'UPDATE flowsafe_remote_step_invocations SET deadline_at = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ?', + ) + .bind(Date.now() - 1, invocation.invocationId) + .run(); + const restartedStorage = createD1Storage({ + binding: fixture.binding, + tablePrefix: 'host_lifecycle_', + }); + await restartedStorage.init(); + const restartedApp = init( + { storage: restartedStorage }, + { executionFence: fixture.fence, startIdempotency: fixture.reservations }, + ); + const restartedJournal = new RemoteStepJournal({ + db: fixture.binding, + fence: fixture.fence, + runtime: restartedApp.runtime, + deploymentId: 'deployment', + tablePrefix: 'host_lifecycle_', + allowedToolIds: ['remote_lookup'], + verifyCompletion: async () => undefined, + }); + const restarted = harness(['writer'], { + init: restartedApp, + storage: restartedStorage, + stateStorage: { ...fixture.stateStorage }, + resourceAccess: fixture.resources, + remoteSteps: restartedJournal, + remoteStepNotification: { + journal: restartedJournal, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + }, + }); + await expect( + restarted.host.recoverNotificationOperation( + { + ...restarted.scope, + principal: { ...restarted.scope.principal, id: 'different-user' }, + }, + invocation.invocationId, + ), + ).rejects.toMatchObject({ status: 404 }); + if (failurePoint === 'before write') + expect( + fixture.sql + .prepare( + "SELECT COUNT(*) AS count FROM sqlite_master WHERE name = 'host_lifecycle_mastra_notifications'", + ) + .get(), + ).toMatchObject({ count: 0 }); + expect( + await restarted.host.recoverNotificationOperation( + restarted.scope, + invocation.invocationId, + ), + ).toBe('receipt-recorded'); + expect( + await restarted.host.recoverNotificationOperation( + restarted.scope, + invocation.invocationId, + ), + ).toBe('receipt-recorded'); + expect( + (await journal.readNotificationOperation(invocation.invocationId))?.state, + ).toBe('receipt-recorded'); + expect( + fixture.sql + .prepare( + 'SELECT COUNT(*) AS count FROM host_lifecycle_mastra_notifications', + ) + .get(), + ).toMatchObject({ count: 1 }); + await expect( + restarted.host.route( + new Request('https://thread/_flowsafe/agent-host/remote-step-resume', { + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), + }), + restarted.scope, + ), + ).rejects.toMatchObject({ status: 404 }); + expect(resume).not.toHaveBeenCalled(); + expect(D1NotificationsStorage.prototype.createNotificationOnce).toBe(create); +}); + describe('agent host selector lookup isolation', () => { it.each([ ['modern', 'status'], diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index 506333a3..ab3bb205 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -84,7 +84,9 @@ import { isDefinitiveInitialAdmissionRefusal } from '../do-runner/initial-admiss import { mastraRegistryEntries } from '../do-runner/mastra-registry.js'; import { REMOTE_STEP_MARKER, + type RemoteStepInvocation, type RemoteStepJournal, + type RemoteStepNotificationOperation, } from '../do-runner/remote-step-journal.js'; import { exceedsRunInputDepth, @@ -319,6 +321,10 @@ export interface ThreadAgentHost { input: { agentId: string; resourceId: string; runId: string }, ): Promise; recoverOwnership(scope: AgentThreadInstanceScope): Promise; + recoverNotificationOperation( + scope: ThreadScope, + invocationId: string, + ): Promise<'receipt-recorded' | 'unresolved'>; route(request: Request, scope: ThreadScope): Promise; } @@ -486,6 +492,45 @@ function record(value: unknown): Record | undefined { : undefined; } +async function retainedNotificationInput( + invocation: RemoteStepInvocation, + operation: RemoteStepNotificationOperation, + executorQuiescentAt: number, +): Promise { + let savedInput: Record | undefined; + try { + savedInput = record(JSON.parse(operation.requestJson)); + } catch { + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + } + if (typeof savedInput?.createdAt !== 'string') + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + const stableInput = { + ...savedInput, + createdAt: new Date(savedInput.createdAt), + } as NotificationOperationInput; + let projection: Awaited>; + try { + projection = await projectNotificationOperation(stableInput); + } catch { + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + } + if ( + JSON.stringify(stableInput) !== operation.requestJson || + stableInput.id !== operation.notificationId || + stableInput.threadId !== invocation.threadId || + stableInput.resourceId !== invocation.resourceId || + stableInput.agentId !== invocation.agentId || + stableInput.sourceId !== invocation.invocationId || + savedInput.createdAt !== new Date(executorQuiescentAt).toISOString() || + projection.version !== operation.projectionVersion || + projection.requestJson !== operation.projectionJson || + projection.requestDigest !== operation.requestDigest + ) + throw new AgentHostRequestError(503, 'remote step operation is unreadable'); + return stableInput; +} + function scheduleAgentContext( target: ScheduleSourceAgentTarget, ): Record { @@ -1816,6 +1861,49 @@ export function createThreadAgentHost( }; }; + const readRemoteStepHostAuthority = async ( + scope: ThreadScope, + invocation: RemoteStepInvocation, + ) => { + const ref = { + agentId: invocation.agentId, + resourceId: invocation.resourceId, + runId: invocation.runId, + }; + const selected = await snapshotExecutionFor(scope, ref); + const state = selected.state; + if ( + state.kind !== 'result' || + state.storage !== 'd1' || + state.execution.tablePrefix !== invocation.tablePrefix || + state.execution.workflowId !== invocation.workflowId || + state.execution.startToken !== invocation.startToken || + state.execution.target.kind !== 'agent' || + state.execution.target.id !== invocation.agentId || + state.execution.target.threadId !== invocation.threadId || + state.execution.target.resourceId !== invocation.resourceId || + state.execution.owner.kind !== invocation.principal.kind || + state.execution.owner.id !== invocation.principal.id + ) + throw new AgentHostRequestError(409, 'remote step run changed'); + const stored = await readRun(ref.runId); + if ( + !stored || + stored.agentId !== ref.agentId || + !samePrincipal(stored.principal, invocation.principal) + ) + throw new AgentHostRequestError(404, 'run not found'); + const binding = await readBinding(); + if ( + (selected.threaded && + (binding?.agentId !== ref.agentId || + binding.resourceId !== ref.resourceId)) || + (!selected.threaded && binding) + ) + throw new AgentHostRequestError(404, 'run not found'); + return { ref, selected, state, stored }; + }; + const statusFor = async ( scope: ThreadScope, ref: { agentId: string; resourceId: string; runId: string }, @@ -2047,6 +2135,73 @@ export function createThreadAgentHost( }), ); + const recoverNotificationOperation = async ( + scope: ThreadScope, + invocationId: string, + ): Promise<'receipt-recorded' | 'unresolved'> => { + instanceScopeFor(scope); + const profile = notificationProfile; + if (!profile || !isPathSafeId(invocationId)) + throw new AgentHostRequestError(404, 'invocation not found'); + const invocation = await profile.journal.get(invocationId); + if ( + !invocation || + invocation.deploymentId !== profile.journal.deploymentId || + invocation.tablePrefix !== profile.journal.tablePrefix || + invocation.threadId !== scope.threadId || + invocation.agentId !== profile.agentId || + !samePrincipal(invocation.principal, scope.principal) || + record(invocation.request)?.notificationOperationId !== + profile.operationId + ) + throw new AgentHostRequestError(404, 'invocation not found'); + const operation = + await profile.journal.readNotificationOperation(invocationId); + if (!operation || operation.operationId !== profile.operationId) + throw new AgentHostRequestError(409, 'remote step operation changed'); + if ( + operation.state === 'resume-admitted' || + operation.state === 'unresolved' || + operation.state === 'prepared' + ) + return 'unresolved'; + await readRemoteStepHostAuthority(scope, invocation); + const retained = await profile.journal.readRetainedResult(invocationId); + if (!retained || invocation.status !== 'result-retained') + throw new AgentHostRequestError(409, 'remote step result is unavailable'); + const stableInput = await retainedNotificationInput( + invocation, + operation, + retained.executorQuiescentAt, + ); + if (!notificationStorage) + notificationStorage = profile.journal.notificationStorage( + notificationReceiptLimit, + ); + const storage = notificationStorage; + if (operation.state === 'effect-admitted') { + const receipt = await storage.createNotificationOnce( + operation.operationKey, + stableInput, + ); + await profile.journal.recordNotificationReceipt(invocationId, receipt); + return 'receipt-recorded'; + } + const receipt = await storage.readNotificationOperationReceipt( + operation.operationKey, + ); + if ( + receipt?.requestDigest !== operation.requestDigest || + receipt.notificationId !== operation.notificationId || + receipt.threadId !== invocation.threadId + ) + throw new AgentHostRequestError( + 503, + 'remote step notification receipt is unreadable', + ); + return 'receipt-recorded'; + }; + const host: ThreadAgentHost = { requestContextForRun: (base) => async (workflowId, runId, leg) => { const values = base ? await base(workflowId, runId, leg) : undefined; @@ -2061,6 +2216,8 @@ export function createThreadAgentHost( : values; }, serializeDispatch: withDispatchLock, + recoverNotificationOperation: (scope, invocationId) => + withDispatchLock(() => recoverNotificationOperation(scope, invocationId)), notificationDispatchAllowed: async (scope, agentId) => { instanceScopeFor(scope); return (await catalogFor(scope)).automationAllowed( @@ -2825,42 +2982,8 @@ export function createThreadAgentHost( invocation.deadlineAt <= Date.now() ) throw new AgentHostRequestError(404, 'invocation not found'); - const ref = { - agentId: invocation.agentId, - resourceId: invocation.resourceId, - runId: invocation.runId, - }; - const selected = await snapshotExecutionFor(scope, ref); - const state = selected.state; - if ( - state.kind !== 'result' || - state.storage !== 'd1' || - state.execution.tablePrefix !== invocation.tablePrefix || - state.execution.workflowId !== invocation.workflowId || - state.execution.startToken !== invocation.startToken || - state.execution.target.kind !== 'agent' || - state.execution.target.id !== invocation.agentId || - state.execution.target.threadId !== invocation.threadId || - state.execution.target.resourceId !== invocation.resourceId || - state.execution.owner.kind !== invocation.principal.kind || - state.execution.owner.id !== invocation.principal.id - ) - throw new AgentHostRequestError(409, 'remote step run changed'); - const stored = await readRun(ref.runId); - if ( - !stored || - stored.agentId !== ref.agentId || - !samePrincipal(stored.principal, invocation.principal) - ) - throw new AgentHostRequestError(404, 'run not found'); - const binding = await readBinding(); - if ( - (selected.threaded && - (binding?.agentId !== ref.agentId || - binding.resourceId !== ref.resourceId)) || - (!selected.threaded && binding) - ) - throw new AgentHostRequestError(404, 'run not found'); + const { ref, selected, state, stored } = + await readRemoteStepHostAuthority(scope, invocation); const key = invocation.stepPath.join('.'); const summary = state.summary; const payload = @@ -2959,37 +3082,11 @@ export function createThreadAgentHost( : { payload: notificationProfile.payload }), }, )); - const savedInput = JSON.parse(operation.requestJson) as Record< - string, - unknown - >; - if (typeof savedInput?.createdAt !== 'string') - throw new AgentHostRequestError( - 503, - 'remote step operation is unreadable', - ); - const stableInput = { - ...savedInput, - createdAt: new Date(savedInput.createdAt), - } as NotificationOperationInput; - const projection = await projectNotificationOperation(stableInput); - if ( - JSON.stringify(stableInput) !== operation.requestJson || - stableInput.id !== operation.notificationId || - stableInput.threadId !== invocation.threadId || - stableInput.resourceId !== invocation.resourceId || - stableInput.agentId !== invocation.agentId || - stableInput.sourceId !== invocation.invocationId || - savedInput.createdAt !== - new Date(retained.executorQuiescentAt).toISOString() || - projection.version !== operation.projectionVersion || - projection.requestJson !== operation.projectionJson || - projection.requestDigest !== operation.requestDigest - ) - throw new AgentHostRequestError( - 503, - 'remote step operation is unreadable', - ); + const stableInput = await retainedNotificationInput( + invocation, + operation, + retained.executorQuiescentAt, + ); if (!notificationStorage) notificationStorage = journal.notificationStorage( notificationReceiptLimit, From cf1f2c83782031afc9a5c4ce32c3fd846e27fc9c Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 00:21:30 +0400 Subject: [PATCH 47/73] Schedule bounded notification effect recovery Persist a versioned recovery cursor and call the authenticated target thread through the singleton maintenance duty. Require an armed duty before selected effect admission and bound local alarm work across wakes. --- packages/flowsafe/src/agent-host/index.ts | 1 + .../src/agent-host/thread-host.test.ts | 286 +++++++++++++++++- .../flowsafe/src/agent-host/thread-host.ts | 144 ++++++++- .../src/agent-host/thread-topology.ts | 75 +++++ packages/flowsafe/src/do-runner/inventory.ts | 8 + .../src/do-runner/remote-step-journal.test.ts | 191 +++++++++++- .../src/do-runner/remote-step-journal.ts | 109 +++++++ .../flowsafe/src/host-kit/flowsafe-worker.ts | 58 +++- packages/flowsafe/src/host-kit/index.ts | 1 + .../src/host-kit/maintenance-do.test.ts | 63 ++++ 10 files changed, 917 insertions(+), 19 deletions(-) diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index ce0b547c..847d51b9 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -56,6 +56,7 @@ export { type AgentThreadTopology, type AgentThreadTopologyOptions, createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, type RemoteStepContinuationReader, type RemoteStepDispatchReader, type RemoteStepDispatchTicket, diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index 7454ac39..95043e22 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -23,6 +23,12 @@ import { type RecoverableResourceOwnershipStore, type ResourceOwnershipDatabase, } from '../approval-api/index.js'; +import { decodeExecutionPrincipal } from '../approval-api/principal.js'; +import { + credentialsMatch, + DEPLOYMENT_IDENTITY_HEADER, +} from '../do-runner/deployment-identity.js'; +import { EXECUTION_PRINCIPAL_HEADER } from '../do-runner/execution-principal-header.js'; import { FENCED_WORKFLOW_STORAGE, type FencedWorkflowAdmissionCapability, @@ -56,6 +62,19 @@ import { } from '../do-runner/remote-step-journal.js'; import { runInputDepthMessage } from '../do-runner/run-input-depth.js'; import { lifecycleFromRequestContext } from '../do-runner/run-lifecycle.js'; +import { + createFlowsafeMaintenanceDurableObject, + ensureRemoteStepNotificationRecoveryDuty, + type FlowsafeWorkerConfig, + type FlowsafeWorkerEnv, + MAINTENANCE_INSTANCE_NAME, + type MaintenanceDurableObjectState, +} from '../host-kit/flowsafe-worker.js'; +import type { + ThreadNamespaceLike, + ThreadRequestInit, +} from '../host-kit/thread-topology.js'; +import { staticTokenVerifier } from '../host-kit/verifier.js'; import { D1SchedulesStorage, type ScheduleDatabase, @@ -71,6 +90,7 @@ import { } from './thread-host.js'; import { createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, type RemoteStepContinuationReader, } from './thread-topology.js'; import type { AgentAutomationRule, Permission } from './types.js'; @@ -289,9 +309,24 @@ function harness( state.set(key, structuredClone(value)); }, delete: async (key) => state.delete(key), - list: async ({ prefix }: { prefix: string }) => + list: async ({ + prefix, + startAfter, + limit, + }: { + prefix: string; + startAfter?: string; + limit?: number; + }) => new Map( - [...state.entries()].filter(([key]) => key.startsWith(prefix)), + [...state.entries()] + .filter( + ([key]) => + key.startsWith(prefix) && + (startAfter === undefined || key > startAfter), + ) + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .slice(0, limit), ) as Map, getAlarm: async () => alarm === undefined @@ -1604,6 +1639,49 @@ describe('direct thread host capture', () => { }); describe('createThreadAgentHost owner recovery', () => { + it('advances a bounded notification marker cursor past repeatedly failing early items', async () => { + vi.spyOn(console, 'error').mockImplementation(() => {}); + onTestFinished(() => { + vi.restoreAllMocks(); + }); + const seen: string[] = []; + const journal = { + deploymentId: 'deployment', + tablePrefix: '', + get: async (invocationId: string) => { + seen.push(invocationId); + throw new Error('recovery item unavailable'); + }, + } as unknown as RemoteStepJournal; + const fixture = harness(['writer'], { + remoteSteps: journal, + remoteStepNotification: { + journal, + agentId: 'writer', + operationId: 'enqueue-v1', + source: 'flowsafe', + kind: 'host-owned', + summary: 'Agent notification', + ensureScheduledRecovery: async () => undefined, + }, + }); + const ids = Array.from( + { length: 18 }, + (_, index) => `operation-${String(index).padStart(2, '0')}`, + ); + for (const id of ids) + await fixture.stateStorage.put( + `flowsafe:agent-notification-recovery:v1:${id}`, + id, + ); + const recover = () => fixture.host.recoverOwnership(fixture.scope); + await expect(recover()).rejects.toThrow('recovery item unavailable'); + expect(seen).toEqual(ids.slice(0, 16)); + await expect(recover()).rejects.toThrow('recovery item unavailable'); + expect(seen).toEqual(ids); + await expect(recover()).rejects.toThrow('recovery item unavailable'); + expect(seen).toEqual([...ids, ...ids.slice(0, 16)]); + }); async function seedPreparingJournal( state: Map, resources: InMemoryResourceOwnershipStore, @@ -8032,6 +8110,7 @@ it('admits a configured notification effect once before agent continuation', asy allowedToolIds: ['remote_lookup'], verifyCompletion: async () => undefined, }); + const readiness = vi.fn(async () => undefined); const host = harness(['writer'], { init: freshApp, storage: fixture.storage, @@ -8046,6 +8125,7 @@ it('admits a configured notification effect once before agent continuation', asy kind: 'host-owned', summary: 'Agent notification', payload: { fixed: true }, + ensureScheduledRecovery: readiness, }, }); const resume = vi @@ -8056,6 +8136,13 @@ it('admits a configured notification effect once before agent continuation', asy method: 'POST', body: JSON.stringify({ invocationId: invocation.invocationId }), }); + readiness.mockRejectedValueOnce(new Error('recovery duty unavailable')); + await expect(host.host.route(request(), host.scope)).rejects.toThrow( + 'recovery duty unavailable', + ); + expect( + (await reopened.readNotificationOperation(invocation.invocationId))?.state, + ).toBe('prepared'); const admission = vi .spyOn(reopened, 'admitNotificationEffect') .mockRejectedValueOnce(new Error('effect admission refused')); @@ -8115,6 +8202,7 @@ it('admits a configured notification effect once before agent continuation', asy kind: 'host-owned', summary: 'Changed after deployment', payload: { fixed: false }, + ensureScheduledRecovery: async () => undefined, }, }); await expect( @@ -8165,12 +8253,14 @@ it('admits a configured notification effect once before agent continuation', asy it.each([ 'before write', 'after write', + 'scheduled tick', ] as const)('recovers an admitted notification after its deadline without resuming Mastra (%s)', async (failurePoint) => { const fixture = await hostAgentLifecycleFixture({ mode: 'fenced', provenance: 'modern', }); onTestFinished(() => { + vi.useRealTimers(); vi.restoreAllMocks(); fixture.sql.close(); }); @@ -8252,6 +8342,7 @@ it.each([ source: 'flowsafe', kind: 'host-owned', summary: 'Agent notification', + ensureScheduledRecovery: async () => undefined, }, }); const retained = await reopened.readRetainedResult(invocation.invocationId); @@ -8300,6 +8391,7 @@ it.each([ expect( (await journal.readNotificationOperation(invocation.invocationId))?.state, ).toBe('effect-admitted'); + expect(fixture.alarmAt()).toBeDefined(); await fixture.binding .prepare( 'UPDATE flowsafe_remote_step_invocations SET deadline_at = ?, write_revision = write_revision + 1, write_generation = write_generation + 1 WHERE invocation_id = ?', @@ -8337,16 +8429,22 @@ it.each([ source: 'flowsafe', kind: 'host-owned', summary: 'Agent notification', + ensureScheduledRecovery: async () => undefined, }, }); - await expect( - restarted.host.recoverNotificationOperation( + const recoveryRequest = () => + new Request( + 'https://thread/_flowsafe/agent-host/remote-step-notification-recover', { - ...restarted.scope, - principal: { ...restarted.scope.principal, id: 'different-user' }, + method: 'POST', + body: JSON.stringify({ invocationId: invocation.invocationId }), }, - invocation.invocationId, - ), + ); + await expect( + restarted.host.route(recoveryRequest(), { + ...restarted.scope, + principal: { ...restarted.scope.principal, id: 'different-user' }, + }), ).rejects.toMatchObject({ status: 404 }); if (failurePoint === 'before write') expect( @@ -8356,12 +8454,163 @@ it.each([ ) .get(), ).toMatchObject({ count: 0 }); - expect( - await restarted.host.recoverNotificationOperation( + if (failurePoint === 'before write') { + await restarted.host.recoverOwnership({ + threadId: restarted.scope.threadId, + init: restarted.scope.init, + deploymentTag: restarted.scope.deploymentTag, + }); + } else if (failurePoint === 'after write') { + const recovered = await restarted.host.route( + recoveryRequest(), restarted.scope, - invocation.invocationId, - ), - ).toBe('receipt-recorded'); + ); + expect(recovered?.status).toBe(200); + expect(await recovered?.json()).toEqual({ state: 'receipt-recorded' }); + } else { + fixture.sql.exec(`CREATE TABLE flowsafe_deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + tenant_tag TEXT NOT NULL, + provisioned_at TEXT NOT NULL + ); INSERT INTO flowsafe_deployment VALUES + (1, 'acme', '2026-10-14T00:00:00.000Z')`); + const secret = 'test-deployment-identity-secret-0001'; + let failFirst = true; + const namespace: ThreadNamespaceLike = { + idFromName: (name) => name, + get: () => ({ + fetch: (async ( + url: Request | string, + requestInit?: ThreadRequestInit, + ) => { + const request = + typeof url === 'string' ? new Request(url, requestInit) : url; + if ( + !(await credentialsMatch( + request.headers.get(DEPLOYMENT_IDENTITY_HEADER) ?? '', + secret, + )) + ) + return new Response(null, { status: 403 }); + const principal = decodeExecutionPrincipal( + request.headers.get(EXECUTION_PRINCIPAL_HEADER) ?? '', + ); + if (!principal) return new Response(null, { status: 403 }); + if (failFirst) { + failFirst = false; + return new Response(null, { status: 503 }); + } + try { + return ( + (await restarted.host.route(request, { + ...restarted.scope, + principal, + })) ?? new Response(null, { status: 404 }) + ); + } catch (error) { + return doErrorResponse(error); + } + }) as ReturnType['get']>['fetch'], + }), + }; + const tick = createRemoteStepNotificationRecoveryTick({ + journal: restartedJournal, + namespace, + deploymentIdentitySecret: secret, + limit: 1, + }); + const values = new Map(); + let alarmAt: number | null = null; + const maintenanceState: MaintenanceDurableObjectState = { + id: { name: MAINTENANCE_INSTANCE_NAME }, + storage: { + get: async (key: string) => values.get(key) as T | undefined, + getAlarm: async () => alarmAt, + setAlarm: async (time) => { + alarmAt = Number(time); + }, + transaction: async (work) => { + const writes = new Map(); + let planned = alarmAt; + const result = await work({ + get: async (key: string) => values.get(key) as T | undefined, + put: async (key, value) => { + writes.set(key, structuredClone(value)); + }, + setAlarm: async (time) => { + planned = Number(time); + }, + }); + for (const [key, value] of writes) values.set(key, value); + alarmAt = planned; + return result; + }, + }, + }; + const config: FlowsafeWorkerConfig = { + workflows: [], + systemPrincipalId: 'maintenance-test', + buildVerifier: () => staticTokenVerifier(new Map()), + maintenance: { + sweepIntervalMs: 60_000, + purgeIntervalMs: 60_000, + remoteStepRecoveryIntervalMs: 1_000, + }, + remoteStepRecoveryTick: () => tick, + }; + const Maintenance = createFlowsafeMaintenanceDurableObject(config); + let maintenance: { + fetch(request: Request): Promise; + alarm(): Promise; + }; + const env: FlowsafeWorkerEnv = { + DB: fixture.binding as FlowsafeWorkerEnv['DB'], + DEPLOYMENT_TENANT: 'acme', + DEPLOYMENT_IDENTITY_SECRET: secret, + RUNNER: { + idFromName: (name) => name, + get: () => ({ fetch: async () => new Response(null, { status: 503 }) }), + }, + MAINTENANCE: { + idFromName: (name) => name, + get: () => ({ + fetch: (url, requestInit) => + maintenance.fetch(new Request(url, requestInit)), + }), + }, + }; + maintenance = new Maintenance(maintenanceState, env); + const now = Date.now(); + vi.useFakeTimers(); + vi.setSystemTime(now); + await ensureRemoteStepNotificationRecoveryDuty(env); + for (let index = 0; index < 4; index += 1) await maintenance.alarm(); + expect( + ( + await restartedJournal.readNotificationOperation( + invocation.invocationId, + ) + )?.state, + ).toBe('effect-admitted'); + expect( + fixture.sql + .prepare( + 'SELECT after_invocation_id FROM flowsafe_remote_step_notification_cursor', + ) + .get(), + ).toMatchObject({ after_invocation_id: invocation.invocationId }); + maintenance = new Maintenance(maintenanceState, env); + vi.setSystemTime(now + 1_000); + await maintenance.alarm(); + await maintenance.alarm(); + expect( + ( + await restartedJournal.readNotificationOperation( + invocation.invocationId, + ) + )?.state, + ).toBe('receipt-recorded'); + } expect( await restarted.host.recoverNotificationOperation( restarted.scope, @@ -8371,6 +8620,17 @@ it.each([ expect( (await journal.readNotificationOperation(invocation.invocationId))?.state, ).toBe('receipt-recorded'); + await restarted.host.recoverOwnership({ + threadId: restarted.scope.threadId, + init: restarted.scope.init, + deploymentTag: restarted.scope.deploymentTag, + }); + await restarted.host.recoverOwnership({ + threadId: restarted.scope.threadId, + init: restarted.scope.init, + deploymentTag: restarted.scope.deploymentTag, + }); + expect(fixture.alarmAt()).toBeUndefined(); expect( fixture.sql .prepare( diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index ab3bb205..7e5368d7 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -143,7 +143,11 @@ export interface AgentThreadStateStorage { get(key: string): Promise; put(key: string, value: T): Promise; delete(key: string): Promise; - list(options: { prefix: string }): Promise>; + list(options: { + prefix: string; + startAfter?: string; + limit?: number; + }): Promise>; getAlarm(): Promise; setAlarm(scheduledTime: number | Date): Promise; deleteAlarm(): Promise; @@ -192,6 +196,7 @@ export interface ThreadAgentHostOptions { summary: string; payload?: unknown; receiptLimit?: number; + ensureScheduledRecovery: () => Promise; }; resourceIdFromKey?: (threadId: string) => string; /** Narrows automated entry beyond what each agent's metadata declares. */ @@ -372,6 +377,11 @@ async function objectBody(request: Request): Promise> { } const AGENT_OWNER_RECOVERY_PREFIX = 'flowsafe:agent-owner-recovery:v1:'; +const AGENT_NOTIFICATION_RECOVERY_PREFIX = + 'flowsafe:agent-notification-recovery:v1:'; +const AGENT_NOTIFICATION_RECOVERY_CURSOR_KEY = + 'flowsafe:agent-notification-recovery-cursor:v1'; +const AGENT_NOTIFICATION_RECOVERY_BATCH = 16; const AGENT_OWNER_RECOVERY_DELAY_MS = 60_000; type AgentOwnerRecovery = { @@ -695,6 +705,7 @@ export function createThreadAgentHost( !notificationProfile.source || !notificationProfile.kind || !notificationProfile.summary || + typeof notificationProfile.ensureScheduledRecovery !== 'function' || (notificationProfile.receiptLimit !== undefined && (!Number.isSafeInteger(notificationProfile.receiptLimit) || notificationProfile.receiptLimit < 0))) @@ -2289,11 +2300,111 @@ export function createThreadAgentHost( firstFailure ??= { error }; } } + if (notificationProfile) { + const cursor = + (await storage.get( + AGENT_NOTIFICATION_RECOVERY_CURSOR_KEY, + )) ?? ''; + if ( + typeof cursor !== 'string' || + (cursor !== '' && + (!cursor.startsWith(AGENT_NOTIFICATION_RECOVERY_PREFIX) || + !isPathSafeId( + cursor.slice(AGENT_NOTIFICATION_RECOVERY_PREFIX.length), + ))) + ) + throw new AgentHostRequestError( + 503, + 'remote step recovery cursor is unreadable', + ); + let notifications = await storage.list({ + prefix: AGENT_NOTIFICATION_RECOVERY_PREFIX, + ...(cursor === '' ? {} : { startAfter: cursor }), + limit: AGENT_NOTIFICATION_RECOVERY_BATCH, + }); + if (notifications.size === 0 && cursor !== '') + notifications = await storage.list({ + prefix: AGENT_NOTIFICATION_RECOVERY_PREFIX, + limit: AGENT_NOTIFICATION_RECOVERY_BATCH, + }); + for (const [key, stored] of notifications) { + const invocationId = key.slice( + AGENT_NOTIFICATION_RECOVERY_PREFIX.length, + ); + try { + if (stored !== invocationId || !isPathSafeId(invocationId)) + throw new AgentHostRequestError( + 503, + 'remote step recovery marker is unreadable', + ); + const invocation = + await notificationProfile.journal.get(invocationId); + if ( + !invocation || + invocation.threadId !== scope.threadId || + invocation.deploymentId !== + notificationProfile.journal.deploymentId || + invocation.tablePrefix !== + notificationProfile.journal.tablePrefix + ) + throw new AgentHostRequestError( + 503, + 'remote step recovery identity changed', + ); + const operation = + await notificationProfile.journal.readNotificationOperation( + invocationId, + ); + if ( + operation?.state === 'effect-admitted' || + operation?.state === 'receipt-recorded' + ) { + const recovered = await recoverNotificationOperation( + { ...scope, principal: invocation.principal }, + invocationId, + ); + if (recovered !== 'receipt-recorded') + throw new AgentHostRequestError( + 503, + 'remote step notification recovery is unresolved', + ); + } else if ( + operation?.state !== 'resume-admitted' && + operation?.state !== 'unresolved' + ) { + throw new AgentHostRequestError( + 503, + 'remote step notification recovery is unresolved', + ); + } + await storage.delete(key); + } catch (error) { + console.error( + JSON.stringify({ + type: 'remote-step-notification-recovery-failed', + threadId: scope.threadId, + invocationId, + error: errorMessageOf(error), + }), + ); + firstFailure ??= { error }; + } + } + const last = [...notifications.keys()].at(-1); + await storage.put( + AGENT_NOTIFICATION_RECOVERY_CURSOR_KEY, + last ?? '', + ); + } await withRecoveryLock(async () => { const remaining = await storage.list({ prefix: AGENT_OWNER_RECOVERY_PREFIX, }); - if (remaining.size > 0) { + const remainingNotifications = await storage.list({ + prefix: AGENT_NOTIFICATION_RECOVERY_PREFIX, + limit: 1, + }); + if (remaining.size > 0 || remainingNotifications.size > 0) { await ensureOwnerRecoveryAlarm(storage); } else { await storage.deleteAlarm(); @@ -2844,6 +2955,12 @@ export function createThreadAgentHost( `${AGENT_HOST_ROUTE_PREFIX}/remote-step-resume` ? await objectBody(request) : undefined; + const notificationRecoveryBody = + request.method === 'POST' && + preflightUrl.pathname === + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-notification-recover` + ? await objectBody(request) + : undefined; return withDispatchLock(async () => { const url = new URL(request.url); @@ -2949,6 +3066,15 @@ export function createThreadAgentHost( ); } + if (notificationRecoveryBody) { + const invocationId = notificationRecoveryBody.invocationId; + if (!isPathSafeId(invocationId)) + throw new AgentHostRequestError(404, 'invocation not found'); + return json({ + state: await recoverNotificationOperation(scope, invocationId), + }); + } + if (remoteStepBody) { const reader = options.remoteSteps; if (!reader) @@ -3096,6 +3222,8 @@ export function createThreadAgentHost( operation.state === 'prepared' || operation.state === 'effect-admitted' ) { + if (operation.state === 'prepared') + await notificationProfile.ensureScheduledRecovery(); const admitted = await journal.admitNotificationEffect(invocationId); if ( @@ -3106,11 +3234,23 @@ export function createThreadAgentHost( 409, 'remote step operation changed', ); + await withRecoveryLock(async () => { + const storage = options.stateStorage(); + await ensureOwnerRecoveryAlarm(storage); + await storage.put( + `${AGENT_NOTIFICATION_RECOVERY_PREFIX}${invocationId}`, + invocationId, + ); + await ensureOwnerRecoveryAlarm(storage); + }); const receipt = await notifications.createNotificationOnce( admitted.operationKey, stableInput, ); await journal.recordNotificationReceipt(invocationId, receipt); + await options + .stateStorage() + .delete(`${AGENT_NOTIFICATION_RECOVERY_PREFIX}${invocationId}`); } else if (operation.state === 'receipt-recorded') { const receipt = await notifications.readNotificationOperationReceipt( diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 356a07bc..98f74538 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -1113,6 +1113,81 @@ export function createAgentThreadTopology( }; } +export function createRemoteStepNotificationRecoveryTick(options: { + journal: RemoteStepJournal; + namespace: ThreadNamespaceLike; + deploymentIdentitySecret: string; + limit?: number; +}): () => Promise<{ scanned: number; recovered: number; unresolved: number }> { + const limit = options.limit ?? 16; + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) + throw new TypeError('notification recovery limit is malformed'); + const threads = createThreadTopology( + options.namespace, + options.deploymentIdentitySecret, + ); + return async () => { + const ids = await options.journal.nextNotificationRecoveryBatch(limit); + let recovered = 0; + let unresolved = 0; + let firstFailure: { error: unknown } | undefined; + for (const invocationId of ids) { + try { + const invocation = await options.journal.get(invocationId); + if ( + !invocation || + !['result-retained', 'resuming'].includes(invocation.status) + ) + continue; + if ( + invocation.invocationId !== invocationId || + invocation.deploymentId !== options.journal.deploymentId || + invocation.tablePrefix !== options.journal.tablePrefix + ) + throw new RunRouteError(409, 'remote step recovery identity changed'); + const operation = + await options.journal.readNotificationOperation(invocationId); + if (!operation || operation.invocationId !== invocationId) + throw new RunRouteError( + 409, + 'remote step recovery operation changed', + ); + if (operation.state !== 'effect-admitted') { + unresolved += 1; + continue; + } + const response = await threads.send( + { principal: invocation.principal }, + invocation.threadId, + `${AGENT_HOST_ROUTE_PREFIX}/remote-step-notification-recover`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ invocationId }), + }, + ); + if (!response.ok) + throw new RunRouteError( + response.status, + 'remote step notification recovery failed', + ); + const after = + await options.journal.readNotificationOperation(invocationId); + if (after?.state !== 'receipt-recorded') + throw new RunRouteError( + 503, + 'remote step notification receipt is unresolved', + ); + recovered += 1; + } catch (error) { + firstFailure ??= { error }; + } + } + if (firstFailure) throw firstFailure.error; + return { scanned: ids.length, recovered, unresolved }; + }; +} + function publicApproval( value: unknown, execution: StartExecutionIdentity, diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index 70765d1a..bd16bf71 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -71,6 +71,7 @@ import { notificationTimestampMillis, } from './notification-predicate.js'; import { + REMOTE_STEP_NOTIFICATION_CURSOR_TABLE, REMOTE_STEP_SCHEMA_TABLE, REMOTE_STEP_TABLE, } from './remote-step-journal.js'; @@ -295,6 +296,13 @@ export interface FlowsafeTableEntry { * characters. */ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ + { + table: REMOTE_STEP_NOTIFICATION_CURSOR_TABLE, + purpose: 'position of the scheduled notification operation recovery scan', + accounting: { + excluded: 'the invocation journal holds the work this cursor revisits', + }, + }, { table: REMOTE_STEP_SCHEMA_TABLE, purpose: 'remote invocation journal schema version', diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts index ac57a834..aea156d3 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.test.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.test.ts @@ -3,22 +3,36 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { z } from 'zod'; import { openSqlite, sqliteUnitDatabase } from '../../test-support/sqlite.js'; -import { createAgentThreadTopology } from '../agent-host/thread-topology.js'; +import { + createAgentThreadTopology, + createRemoteStepNotificationRecoveryTick, +} from '../agent-host/thread-topology.js'; +import { decodeExecutionPrincipal } from '../approval-api/principal.js'; import type { FlowsafeWorkerEnv } from '../host-kit/flowsafe-worker.js'; import { createFlowsafeWorker } from '../host-kit/flowsafe-worker.js'; import { approvalStoreFactoryFor } from '../host-kit/host-approval-service.js'; +import type { + ThreadNamespaceLike, + ThreadRequestInit, +} from '../host-kit/thread-topology.js'; import { staticTokenVerifier } from '../host-kit/verifier.js'; import { D1NotificationsStorage, + type NotificationOperationInput, projectNotificationOperation, } from '../signals/notifications-d1.js'; import { createD1Storage } from './d1-storage.js'; -import { credentialsMatch } from './deployment-identity.js'; +import { + credentialsMatch, + DEPLOYMENT_IDENTITY_HEADER, +} from './deployment-identity.js'; import { ExecutionFenceStore } from './execution-fence.js'; +import { EXECUTION_PRINCIPAL_HEADER } from './execution-principal-header.js'; import { init } from './init.js'; import { DeploymentInventory } from './inventory.js'; import { REMOTE_STEP_MARKER, + REMOTE_STEP_NOTIFICATION_CURSOR_TABLE, REMOTE_STEP_TABLE, RemoteStepJournal, WORKFLOW_REMOTE_STEP_MARKER, @@ -1720,6 +1734,179 @@ describe('remote step journal dispatch admission', () => { }); }); + it('persists a bounded notification recovery cursor across instances and wraps past the final active row', async () => { + const h = await fixture(); + const ids: string[] = []; + for (const suspendedAt of [500, 600, 700]) { + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': suspendedAt }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: `call-${suspendedAt}`, + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + ids.push(invocation.invocationId); + } + const expected = ids.sort(); + expect(await h.journal.nextNotificationRecoveryBatch(1)).toEqual([ + expected[0], + ]); + expect(await h.reopen().nextNotificationRecoveryBatch(1)).toEqual([ + expected[1], + ]); + expect(await h.reopen().nextNotificationRecoveryBatch(1)).toEqual([ + expected[2], + ]); + expect(await h.reopen().nextNotificationRecoveryBatch(1)).toEqual([ + expected[0], + ]); + h.sqlite + .prepare( + `UPDATE ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} SET version = 2`, + ) + .run(); + await expect( + h.reopen().nextNotificationRecoveryBatch(1), + ).rejects.toMatchObject({ status: 503 }); + }); + + it('routes recovery with the stored principal and continues past a failed item', async () => { + const h = await fixture(); + const ids: string[] = []; + for (const suspendedAt of [500, 600]) { + h.setSummary({ + runId: ADDRESS.runId, + status: 'suspended', + suspended: [['durable-agentic-execution']], + suspendedAt: { 'durable-agentic-execution': suspendedAt }, + suspendPayload: { + 'durable-agentic-execution': { + type: 'suspension', + toolCallId: `call-${suspendedAt}`, + toolName: 'remote_lookup', + toolCallSuspended: { + remoteStep: REMOTE_STEP_MARKER, + request: { account: 'acme' }, + }, + }, + }, + }); + const { invocation } = await h.retainResult(); + await h.journal.prepareNotificationOperation( + invocation.invocationId, + 'result-notification', + { + threadId: ADDRESS.threadId, + source: 'remote-step', + kind: 'result', + summary: 'Remote work completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }, + ); + await h.journal.admitNotificationEffect(invocation.invocationId); + ids.push(invocation.invocationId); + } + const sorted = ids.sort(); + const firstId = sorted[0]; + const secondId = sorted[1]; + if (!firstId || !secondId) throw new Error('recovery items are missing'); + const secret = 'test-deployment-identity-secret-0001'; + const sent: string[] = []; + let failFirst = true; + const namespace: ThreadNamespaceLike = { + idFromName: (name) => name, + get: (threadId) => ({ + fetch: (async ( + url: Request | string, + requestInit?: ThreadRequestInit, + ) => { + const request = + typeof url === 'string' ? new Request(url, requestInit) : url; + if ( + threadId !== ADDRESS.threadId || + !(await credentialsMatch( + request.headers.get(DEPLOYMENT_IDENTITY_HEADER) ?? '', + secret, + )) || + JSON.stringify( + decodeExecutionPrincipal( + request.headers.get(EXECUTION_PRINCIPAL_HEADER) ?? '', + ), + ) !== JSON.stringify(PRINCIPAL) + ) + return new Response(null, { status: 403 }); + const body = (await request.json()) as { invocationId: string }; + sent.push(body.invocationId); + if (body.invocationId === firstId && failFirst) + return new Response(null, { status: 503 }); + const operation = await h.journal.readNotificationOperation( + body.invocationId, + ); + if (!operation) return new Response(null, { status: 404 }); + if (operation.state === 'effect-admitted') { + const saved = JSON.parse(operation.requestJson) as Omit< + NotificationOperationInput, + 'createdAt' + > & { + createdAt: string; + }; + const receipt = await h.journal + .notificationStorage() + .createNotificationOnce(operation.operationKey, { + ...saved, + createdAt: new Date(saved.createdAt), + }); + await h.journal.recordNotificationReceipt( + body.invocationId, + receipt, + ); + } + return Response.json({ state: 'receipt-recorded' }); + }) as ReturnType['get']>['fetch'], + }), + }; + const tick = createRemoteStepNotificationRecoveryTick({ + journal: h.journal, + namespace, + deploymentIdentitySecret: secret, + limit: 2, + }); + await expect(tick()).rejects.toMatchObject({ status: 503 }); + expect(sent).toEqual(sorted); + expect((await h.journal.readNotificationOperation(secondId))?.state).toBe( + 'receipt-recorded', + ); + failFirst = false; + expect(await tick()).toEqual({ + scanned: 2, + recovered: 1, + unresolved: 1, + }); + expect(sent).toEqual([...sorted, firstId]); + }); + it('advances a limit-one remote-step inventory cursor past active rows', async () => { const h = await fixture(); const first = await h.retainResult(); diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index a40ee9a3..f2d8bca8 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -34,6 +34,15 @@ import { validateTablePrefix } from './table-prefix.js'; export const REMOTE_STEP_TABLE = 'flowsafe_remote_step_invocations'; export const REMOTE_STEP_SCHEMA_TABLE = 'flowsafe_remote_step_schema'; +export const REMOTE_STEP_NOTIFICATION_CURSOR_TABLE = + 'flowsafe_remote_step_notification_cursor'; +const REMOTE_STEP_NOTIFICATION_CURSOR_SQL = `CREATE TABLE ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} ( + deployment_id TEXT NOT NULL, + table_prefix TEXT NOT NULL, + version INTEGER NOT NULL, + after_invocation_id TEXT NOT NULL, + PRIMARY KEY (deployment_id, table_prefix) +)`; const REMOTE_STEP_RECOVERY_INDEX = 'flowsafe_remote_step_recovery'; const REMOTE_STEP_RECOVERY_INDEX_SQL = `CREATE INDEX ${REMOTE_STEP_RECOVERY_INDEX} ON ${REMOTE_STEP_TABLE} (invocation_id) WHERE status IN ('result-retained', 'resuming')`; const REMOTE_STEP_AGENT_INDEX = 'flowsafe_remote_step_agent_suspension'; @@ -936,6 +945,14 @@ export class RemoteStepJournal { version INTEGER NOT NULL )`) .run(); + await this.#db + .prepare( + REMOTE_STEP_NOTIFICATION_CURSOR_SQL.replace( + 'CREATE TABLE ', + 'CREATE TABLE IF NOT EXISTS ', + ), + ) + .run(); const readSchema = async () => { const schema = await this.#db .prepare(`PRAGMA table_xinfo(${REMOTE_STEP_TABLE})`) @@ -2673,6 +2690,98 @@ export class RemoteStepJournal { return outcome; } + async nextNotificationRecoveryBatch(limit: number): Promise { + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) + throw new RemoteStepRefusal( + 400, + 'notification recovery limit is malformed', + ); + await this.initialize(); + const definition = record( + await this.#db + .prepare('SELECT sql FROM sqlite_schema WHERE type = ? AND name = ?') + .bind('table', REMOTE_STEP_NOTIFICATION_CURSOR_TABLE) + .all(), + )?.results; + if ( + !Array.isArray(definition) || + definition.length !== 1 || + record(definition[0])?.sql !== REMOTE_STEP_NOTIFICATION_CURSOR_SQL + ) + throw new RemoteStepRefusal( + 503, + 'notification recovery cursor is unreadable', + ); + await this.#db + .prepare(`INSERT OR IGNORE INTO ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} + (deployment_id, table_prefix, version, after_invocation_id) + VALUES (?, ?, 1, '')`) + .bind(this.#deploymentId, this.#tablePrefix) + .run(); + const cursorRows = record( + await this.#db + .prepare(`SELECT version, after_invocation_id + FROM ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} + WHERE deployment_id = ? AND table_prefix = ?`) + .bind(this.#deploymentId, this.#tablePrefix) + .all(), + )?.results; + const cursor = + Array.isArray(cursorRows) && cursorRows.length === 1 + ? record(cursorRows[0]) + : undefined; + const after = cursor?.after_invocation_id; + if ( + cursor?.version !== 1 || + typeof after !== 'string' || + (after !== '' && !isPathSafeId(after)) + ) + throw new RemoteStepRefusal( + 503, + 'notification recovery cursor is unreadable', + ); + const scan = async (afterInvocationId: string) => { + const values = record( + await this.#db + .prepare(`SELECT invocation_id FROM ${REMOTE_STEP_TABLE} + INDEXED BY ${REMOTE_STEP_OPERATION_INDEX} + WHERE invocation_id > ? AND deployment_id = ? AND table_prefix = ? + AND target_kind = 'agent' + AND status IN ('result-retained', 'resuming') + AND operation_state IN ('prepared', 'effect-admitted', + 'receipt-recorded', 'resume-admitted', 'unresolved') + ORDER BY invocation_id LIMIT ?`) + .bind(afterInvocationId, this.#deploymentId, this.#tablePrefix, limit) + .all(), + )?.results; + if ( + !Array.isArray(values) || + values.some((value) => !isPathSafeId(record(value)?.invocation_id)) + ) + throw new RemoteStepRefusal( + 503, + 'notification recovery scan is unreadable', + ); + return values.map((value) => record(value)?.invocation_id as string); + }; + let ids = await scan(after); + if (ids.length === 0 && after !== '') ids = await scan(''); + const next = ids.at(-1) ?? ''; + const advanced = record( + await this.#db + .prepare(`UPDATE ${REMOTE_STEP_NOTIFICATION_CURSOR_TABLE} + SET after_invocation_id = ? + WHERE deployment_id = ? AND table_prefix = ? AND version = 1 + AND after_invocation_id = ? + RETURNING after_invocation_id`) + .bind(next, this.#deploymentId, this.#tablePrefix, after) + .all(), + )?.results; + if (!Array.isArray(advanced) || advanced.length !== 1) + throw new RemoteStepRefusal(409, 'notification recovery cursor changed'); + return ids; + } + async recoverableResults( limit: number, afterInvocationId = '', diff --git a/packages/flowsafe/src/host-kit/flowsafe-worker.ts b/packages/flowsafe/src/host-kit/flowsafe-worker.ts index 75b1dd87..599e7c19 100644 --- a/packages/flowsafe/src/host-kit/flowsafe-worker.ts +++ b/packages/flowsafe/src/host-kit/flowsafe-worker.ts @@ -129,6 +129,39 @@ export interface FlowsafeWorkerContext { /** Fixed instance name used by every deployment maintenance namespace. */ export const MAINTENANCE_INSTANCE_NAME = 'deployment-maintenance'; +export async function ensureRemoteStepNotificationRecoveryDuty( + env: FlowsafeWorkerEnv, +): Promise { + const id = env.MAINTENANCE.idFromName(MAINTENANCE_INSTANCE_NAME); + const response = await env.MAINTENANCE.get(id).fetch( + 'http://maintenance/ensure', + { + method: 'POST', + headers: deploymentIdentityHeaders(env.DEPLOYMENT_IDENTITY_SECRET), + }, + ); + if (!response.ok) + throw new Error('remote step notification recovery duty is unavailable'); + let value: unknown; + try { + value = await response.json(); + } catch { + throw new Error('remote step notification recovery duty is unreadable'); + } + const health = + value !== null && typeof value === 'object' + ? (value as Record) + : undefined; + const next = validTime(health?.nextRemoteStepRecoveryAt); + const alarmAt = validTime(health?.alarmAt); + if ( + next === undefined || + alarmAt === undefined || + alarmAt > Math.max(Date.now(), next) + ) + throw new Error('remote step notification recovery duty is not armed'); +} + export interface MaintenanceStubLike { fetch( url: string, @@ -147,20 +180,24 @@ export interface MaintenanceHealth { nextPurgeAt: number; nextTickAt?: number; nextNotificationAt?: number; + nextRemoteStepRecoveryAt?: number; lastSweepAt?: number; lastPurgeAt?: number; lastTickAt?: number; lastNotificationAt?: number; + lastRemoteStepRecoveryAt?: number; lastDeadlineAt?: number; lastSweepAttemptAt?: number; lastPurgeAttemptAt?: number; lastTickAttemptAt?: number; lastNotificationAttemptAt?: number; + lastRemoteStepRecoveryAttemptAt?: number; lastDeadlineAttemptAt?: number; lastSweepError?: string; lastPurgeError?: string; lastTickError?: string; lastNotificationError?: string; + lastRemoteStepRecoveryError?: string; lastDeadlineError?: string; } @@ -369,13 +406,15 @@ export interface FlowsafeWorkerConfig * Recurrence intervals for the singleton maintenance Durable Object. Every * duty receives its own alarm invocation; `tickIntervalMs` is required when * `scheduleTick` is configured and omitted otherwise, and - * `notificationIntervalMs` likewise with `notificationTick`. + * `notificationIntervalMs` likewise with `notificationTick`, and + * `remoteStepRecoveryIntervalMs` with `remoteStepRecoveryTick`. */ maintenance: { sweepIntervalMs: number; purgeIntervalMs: number; tickIntervalMs?: number; notificationIntervalMs?: number; + remoteStepRecoveryIntervalMs?: number; /** Deadline duty cadence. Defaults to sweepIntervalMs. */ deadlineIntervalMs?: number; /** Maximum runs per deadline pass. Default 100. */ @@ -551,6 +590,7 @@ export interface FlowsafeWorkerConfig * notification duty. */ notificationTick?: (env: Env) => () => Promise; + remoteStepRecoveryTick?: (env: Env) => () => Promise; /** * Extra purge duties. The * returned fields fold into the ONE combined `{type:'maintenance'}` log @@ -763,7 +803,8 @@ export type MaintenanceDuty = | 'sweep' | 'purge' | 'tick' - | 'notification'; + | 'notification' + | 'remoteStepRecovery'; /** * The cursor seam a duty resumes from and advances. Each field is optional @@ -814,6 +855,11 @@ const INJECTED_TICK_DUTIES = { interval: 'notificationIntervalMs', logType: 'notification-dispatch', }, + remoteStepRecovery: { + builder: 'remoteStepRecoveryTick', + interval: 'remoteStepRecoveryIntervalMs', + logType: 'remote-step-notification-recovery', + }, } as const; type InjectedTickDuty = keyof typeof INJECTED_TICK_DUTIES; @@ -1940,6 +1986,7 @@ const DUTY_ORDER = [ 'purge', 'tick', 'notification', + 'remoteStepRecovery', ] as const; export type MaintenanceDurableObjectConstructor = @@ -1963,6 +2010,7 @@ export function createFlowsafeMaintenanceDurableObject< purge: config.maintenance.purgeIntervalMs, tick: config.maintenance.tickIntervalMs, notification: config.maintenance.notificationIntervalMs, + remoteStepRecovery: config.maintenance.remoteStepRecoveryIntervalMs, }; for (const duty of DUTY_ORDER) { const interval = intervals[duty]; @@ -2346,6 +2394,12 @@ const DUTY_HEALTH_FIELDS = { attempt: 'lastNotificationAttemptAt', error: 'lastNotificationError', }, + remoteStepRecovery: { + next: 'nextRemoteStepRecoveryAt', + last: 'lastRemoteStepRecoveryAt', + attempt: 'lastRemoteStepRecoveryAttemptAt', + error: 'lastRemoteStepRecoveryError', + }, } as const satisfies Record< MaintenanceDuty, { diff --git a/packages/flowsafe/src/host-kit/index.ts b/packages/flowsafe/src/host-kit/index.ts index 354cfdfc..abd366d8 100644 --- a/packages/flowsafe/src/host-kit/index.ts +++ b/packages/flowsafe/src/host-kit/index.ts @@ -112,6 +112,7 @@ export { createFlowsafeMaintenanceDurableObject, createFlowsafeRunnerLifecycle, createFlowsafeWorker, + ensureRemoteStepNotificationRecoveryDuty, MAINTENANCE_INSTANCE_NAME, } from './flowsafe-worker.js'; // hostAuditSink stays module-internal for the same reason as diff --git a/packages/flowsafe/src/host-kit/maintenance-do.test.ts b/packages/flowsafe/src/host-kit/maintenance-do.test.ts index 3f7fe9ce..a573a33e 100644 --- a/packages/flowsafe/src/host-kit/maintenance-do.test.ts +++ b/packages/flowsafe/src/host-kit/maintenance-do.test.ts @@ -12,6 +12,7 @@ import { } from '../do-runner/start-reservation-contract.js'; import { createFlowsafeMaintenanceDurableObject, + ensureRemoteStepNotificationRecoveryDuty, type FlowsafeWorkerConfig, type FlowsafeWorkerEnv, MAINTENANCE_INSTANCE_NAME, @@ -1166,6 +1167,68 @@ describe('alarm-driven deployment maintenance', () => { ); }); + it('wakes the optional remote-step recovery duty again after restart without traffic', async () => { + vi.useFakeTimers(); + vi.setSystemTime(NOW); + vi.spyOn(console, 'log').mockImplementation(() => {}); + const calls: number[] = []; + const { instance, internalRequest, reconstruct } = harness({ + config: { + maintenance: { + sweepIntervalMs: 15 * 60_000, + purgeIntervalMs: 60 * 60_000, + remoteStepRecoveryIntervalMs: 1_000, + }, + remoteStepRecoveryTick: () => async () => { + calls.push(Date.now()); + return { scanned: 0, recovered: 0, unresolved: 0 }; + }, + }, + }); + await healthOf(instance, internalRequest('/ensure', 'POST')); + for (let index = 0; index < 4; index += 1) await instance.alarm(); + expect(calls).toEqual([NOW]); + const restarted = reconstruct(); + await restarted.alarm(); + expect(calls).toEqual([NOW]); + vi.advanceTimersByTime(1_000); + await restarted.alarm(); + expect(calls).toEqual([NOW, NOW + 1_000]); + }); + + it('arms and verifies the configured remote-step recovery duty before admission', async () => { + vi.useFakeTimers(); + vi.setSystemTime(NOW); + const active = harness({ + config: { + maintenance: { + sweepIntervalMs: 15 * 60_000, + purgeIntervalMs: 60 * 60_000, + remoteStepRecoveryIntervalMs: 1_000, + }, + remoteStepRecoveryTick: () => async () => ({ scanned: 0 }), + }, + }); + const linked = ( + fixture: ReturnType, + ): FlowsafeWorkerEnv => ({ + ...fixture.env, + MAINTENANCE: { + idFromName: (name) => name, + get: () => ({ + fetch: (url, init) => fixture.instance.fetch(new Request(url, init)), + }), + }, + }); + await expect( + ensureRemoteStepNotificationRecoveryDuty(linked(active)), + ).resolves.toBeUndefined(); + expect(active.storage.alarmAt).toBe(NOW); + await expect( + ensureRemoteStepNotificationRecoveryDuty(linked(harness())), + ).rejects.toThrow('not armed'); + }); + it('rejects non-singleton instances before touching maintenance state', async () => { vi.useFakeTimers(); vi.setSystemTime(NOW); From 4b516b889e288921c876be2e595237ab67ab787a Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 00:36:23 +0400 Subject: [PATCH 48/73] Keep notification delivery types in D1 signal layer Move shared type declarations below notification dispatch so the remote-step journal no longer reaches host-kit through a type-only import. --- .../src/signals/notification-dispatch.ts | 75 +++---------------- .../flowsafe/src/signals/notifications-d1.ts | 71 ++++++++++++++++-- 2 files changed, 76 insertions(+), 70 deletions(-) diff --git a/packages/flowsafe/src/signals/notification-dispatch.ts b/packages/flowsafe/src/signals/notification-dispatch.ts index 13939c8a..672f20ef 100644 --- a/packages/flowsafe/src/signals/notification-dispatch.ts +++ b/packages/flowsafe/src/signals/notification-dispatch.ts @@ -18,71 +18,18 @@ import { positiveSafeInteger, } from '../numeric-config.js'; import { jsonOrNull } from './d1-shared.js'; +import type { + NotificationDeliveryFailure, + NotificationDeliveryObservation, + NotificationDeliveryStorage, +} from './notifications-d1.js'; -/** Scalar values preserve the attempted observation across asynchronous writes. */ -export type NotificationDeliveryObservation = Readonly<{ - id: string; - threadId: string; - source: string; - kind: string; - priority: NotificationRecord['priority']; - status: NotificationRecord['status']; - summary: string; - payload: string | null; - resourceId: string | null; - agentId: string | null; - sourceId: string | null; - dedupeKey: string | null; - coalesceKey: string | null; - coalescedCount: number; - attributes: string | null; - createdAt: string; - updatedAt: string; - deliverAt: string | null; - summaryAt: string | null; - deliveryReason: string | null; - deliveryAttempts: number; - lastDeliveryAttemptAt: string | null; - lastDeliveryError: string | null; - deliveredSignalId: string | null; - summarySignalId: string | null; - deliveredAt: string | null; - seenAt: string | null; - dismissedAt: string | null; - archivedAt: string | null; - discardedAt: string | null; - metadata: string | null; -}>; - -export type NotificationDeliveryFailure = - | Readonly<{ - type: 'retry'; - updatedAt: string; - deliveryAttempts: number; - lastDeliveryAttemptAt: string; - lastDeliveryError: string; - deliverAt?: string; - summaryAt?: string; - }> - | Readonly<{ - type: 'discard'; - updatedAt: string; - deliveryAttempts: number; - lastDeliveryAttemptAt: string; - lastDeliveryError: string; - }> - | Readonly<{ type: 'exhausted'; updatedAt: string }>; - -export type NotificationDeliveryUpdateResult = - | { applied: true; record: NotificationRecord } - | { applied: false }; - -export interface NotificationDeliveryStorage extends NotificationsStorage { - updateNotificationDeliveryIfUnchanged(input: { - expected: NotificationDeliveryObservation; - failure: NotificationDeliveryFailure; - }): Promise; -} +export type { + NotificationDeliveryFailure, + NotificationDeliveryObservation, + NotificationDeliveryStorage, + NotificationDeliveryUpdateResult, +} from './notifications-d1.js'; /** Maximum route-valid ids in one trusted thread-DO dispatch request. */ export const MAX_NOTIFICATION_DISPATCH_IDS = 100; diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index d37b8438..ab09b9ef 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -41,12 +41,71 @@ import { type SignalDatabase, type SignalStatement, } from './d1-shared.js'; -import type { - NotificationDeliveryFailure, - NotificationDeliveryObservation, - NotificationDeliveryStorage, - NotificationDeliveryUpdateResult, -} from './notification-dispatch.js'; + +/** Scalar values preserve the attempted observation across asynchronous writes. */ +export type NotificationDeliveryObservation = Readonly<{ + id: string; + threadId: string; + source: string; + kind: string; + priority: NotificationRecord['priority']; + status: NotificationRecord['status']; + summary: string; + payload: string | null; + resourceId: string | null; + agentId: string | null; + sourceId: string | null; + dedupeKey: string | null; + coalesceKey: string | null; + coalescedCount: number; + attributes: string | null; + createdAt: string; + updatedAt: string; + deliverAt: string | null; + summaryAt: string | null; + deliveryReason: string | null; + deliveryAttempts: number; + lastDeliveryAttemptAt: string | null; + lastDeliveryError: string | null; + deliveredSignalId: string | null; + summarySignalId: string | null; + deliveredAt: string | null; + seenAt: string | null; + dismissedAt: string | null; + archivedAt: string | null; + discardedAt: string | null; + metadata: string | null; +}>; + +export type NotificationDeliveryFailure = + | Readonly<{ + type: 'retry'; + updatedAt: string; + deliveryAttempts: number; + lastDeliveryAttemptAt: string; + lastDeliveryError: string; + deliverAt?: string; + summaryAt?: string; + }> + | Readonly<{ + type: 'discard'; + updatedAt: string; + deliveryAttempts: number; + lastDeliveryAttemptAt: string; + lastDeliveryError: string; + }> + | Readonly<{ type: 'exhausted'; updatedAt: string }>; + +export type NotificationDeliveryUpdateResult = + | { applied: true; record: NotificationRecord } + | { applied: false }; + +export interface NotificationDeliveryStorage extends NotificationsStorage { + updateNotificationDeliveryIfUnchanged(input: { + expected: NotificationDeliveryObservation; + failure: NotificationDeliveryFailure; + }): Promise; +} /** The raw row shape `mastra_notifications` stores and `rowToRecord` reads. */ interface NotificationRow { From c3293a77508427d6eca926cd90a366f20dc6b39c Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 00:53:23 +0400 Subject: [PATCH 49/73] Assert notification receipt digests in workerd fidelity The workerd fixture now checks the persisted request digest and replay response alongside the receipt identity, matching the v5 notification contract. --- .../src/cloudflare-fidelity.workerd.test.ts | 33 ++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts index 9cc15365..0fe07422 100644 --- a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts +++ b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts @@ -21,6 +21,12 @@ interface TestBindings { } const db = (): D1Database => (env as unknown as TestBindings).DB; +const notificationRequestDigests = { + 'operation-1': + '69d0b9cae1fecec25e3fdad599bc1d4b441761843e4c399563c43b4f8e27699c', + 'operation-2': + '96bdec0f9c3160a3bdd190c24543168f4a4bcc5b7041507c996b042fc593f1dc', +} as const; function receiptActor(name: string) { const namespace = (env as unknown as TestBindings).SIGNAL_PROVIDER_HOST; @@ -129,11 +135,13 @@ describe('FlowSafe Cloudflare fidelity', () => { operationKey: 'operation-1', notificationId: 'notification-1', threadId: 'thread', + requestDigest: notificationRequestDigests['operation-1'], }, { operationKey: 'operation-1', notificationId: 'notification-1', threadId: 'thread', + requestDigest: notificationRequestDigests['operation-1'], }, ]); const receipts = await db() @@ -196,16 +204,32 @@ describe('FlowSafe Cloudflare fidelity', () => { 200, 409, ]); const receipts = await db() - .prepare(`SELECT operation_key, notification_id + .prepare(`SELECT operation_key, notification_id, request_digest FROM flowsafe_notification_operation_receipts WHERE operation_key IN ('operation-1', 'operation-2')`) - .all<{ operation_key: string; notification_id: string }>(); + .all<{ + operation_key: string; + notification_id: string; + request_digest: string; + }>(); const inbox = await db() .prepare('SELECT id FROM mastra_notifications') .all<{ id: string }>(); expect([ - [{ operation_key: 'operation-1', notification_id: 'notification-1' }], - [{ operation_key: 'operation-2', notification_id: 'notification-2' }], + [ + { + operation_key: 'operation-1', + notification_id: 'notification-1', + request_digest: notificationRequestDigests['operation-1'], + }, + ], + [ + { + operation_key: 'operation-2', + notification_id: 'notification-2', + request_digest: notificationRequestDigests['operation-2'], + }, + ], ]).toContainEqual(receipts.results); const winner = receipts.results[0]; if (!winner) throw new Error('receipt is missing'); @@ -231,6 +255,7 @@ describe('FlowSafe Cloudflare fidelity', () => { operationKey: winner.operation_key, notificationId: winner.notification_id, threadId: 'thread', + requestDigest: winner.request_digest, }); }, 30_000); }); From ad5dbec31299ccbf412c3fadfd3fe1ae0f910580 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 01:07:18 +0400 Subject: [PATCH 50/73] Keep internal plans outside public guide index The documentation reachability check now treats docs/plans as internal design records while still reporting unlinked public guides. --- scripts/docs-check.mjs | 1 + scripts/docs-check.test.mjs | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/scripts/docs-check.mjs b/scripts/docs-check.mjs index 22177488..7170bba0 100644 --- a/scripts/docs-check.mjs +++ b/scripts/docs-check.mjs @@ -861,6 +861,7 @@ function checkOrphanedGuides(root, markdownFiles) { const relativePath = toPosix(relative(root, file)); return ( relativePath.startsWith('docs/') && + !relativePath.startsWith('docs/plans/') && basename(file).toLowerCase() !== 'claude.md' && !reachable.has(file) ); diff --git a/scripts/docs-check.test.mjs b/scripts/docs-check.test.mjs index a7968832..25a253a2 100644 --- a/scripts/docs-check.test.mjs +++ b/scripts/docs-check.test.mjs @@ -389,6 +389,31 @@ test('reachability follows absolute repository links into the docs tree', () => ); }); +test('design plans do not require a public documentation index link', () => { + const root = fixture({ + 'docs/README.md': '# Documentation\n', + 'docs/plans/internal-recovery.md': '# Recovery design plan\n', + 'docs/unlinked.md': '# Unlinked guide\n', + }); + + const result = checkRepository({ + root, + markdownFiles: markdownFiles(root, [ + 'docs/README.md', + 'docs/plans/internal-recovery.md', + 'docs/unlinked.md', + ]), + packageChecks: false, + }); + + assert.deepEqual( + result.errors.map((error) => `${error.file}: ${error.message}`), + [ + 'docs/unlinked.md: guide is not reachable from a public documentation index', + ], + ); +}); + test('reachability skips ignored schemes, off-repository URLs, and missing blob targets', () => { const root = fixture({ 'docs/README.md': `# Documentation From 3e030ce5db33a805a4004989beff0fe7856ef555 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 01:07:46 +0400 Subject: [PATCH 51/73] Prepare private notification recovery staging topology Add an isolated default-off subject with dedicated D1 and thread DO, plus an authenticated loopback service-binding probe. Keep real effect proof gated on a guarded suspension producer. --- .../notification-staging-probe.worker.test.ts | 32 +++ .../notification-staging-probe.worker.ts | 41 ++++ .../notification-staging-probe.wrangler.jsonc | 16 ++ .../deploy/notification-staging.worker.ts | 207 ++++++++++++++++++ .../notification-staging.wrangler.jsonc | 43 ++++ packages/flowsafe/deploy/tsconfig.json | 2 + 6 files changed, 341 insertions(+) create mode 100644 packages/flowsafe/deploy/notification-staging-probe.worker.test.ts create mode 100644 packages/flowsafe/deploy/notification-staging-probe.worker.ts create mode 100644 packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc create mode 100644 packages/flowsafe/deploy/notification-staging.worker.ts create mode 100644 packages/flowsafe/deploy/notification-staging.wrangler.jsonc diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts new file mode 100644 index 00000000..8ff1c642 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts @@ -0,0 +1,32 @@ +import type { Fetcher } from '@cloudflare/workers-types'; +import { expect, it, vi } from 'vitest'; +import handler from './notification-staging-probe.worker.js'; + +it('forwards a staging request only after the local caller supplies the probe credential', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 204 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + }; + const request = (credential?: string) => + new Request('http://127.0.0.1/ensure', { + method: 'POST', + ...(credential + ? { headers: { authorization: `Bearer ${credential}` } } + : {}), + }); + + expect((await handler.fetch(request(), env)).status).toBe(403); + expect((await handler.fetch(request('wrong'), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + + expect((await handler.fetch(request(secret), env)).status).toBe(204); + expect(fetch).toHaveBeenCalledOnce(); + const forwarded = fetch.mock.calls[0]; + expect(forwarded?.[0]).toBe('http://subject/__staging/ensure'); + expect(forwarded?.[1]).toMatchObject({ method: 'POST' }); + expect(forwarded?.[1]?.headers).toMatchObject({ + 'x-flowsafe-staging-probe': secret, + }); +}); diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.ts new file mode 100644 index 00000000..b25897f5 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.ts @@ -0,0 +1,41 @@ +import type { Fetcher } from '@cloudflare/workers-types'; +import { + assertDeploymentIdentitySecret, + credentialsMatch, +} from '../src/do-runner/deployment-identity.js'; + +interface Env { + SUBJECT: Fetcher; + STAGING_PROBE_SECRET: string; +} + +export default { + async fetch(request: Request, env: Env): Promise { + const url = new URL(request.url); + if (url.hostname !== '127.0.0.1' && url.hostname !== 'localhost') + return new Response(null, { status: 404 }); + if (request.method !== 'POST' || url.pathname !== '/ensure') + return new Response(null, { status: 404 }); + try { + assertDeploymentIdentitySecret( + env.STAGING_PROBE_SECRET, + 'staging probe credential', + ); + } catch { + return new Response(null, { status: 503 }); + } + const authorization = request.headers.get('authorization'); + if ( + !authorization?.startsWith('Bearer ') || + !(await credentialsMatch( + authorization.slice('Bearer '.length), + env.STAGING_PROBE_SECRET, + )) + ) + return new Response(null, { status: 403 }); + return env.SUBJECT.fetch('http://subject/__staging/ensure', { + method: 'POST', + headers: { 'x-flowsafe-staging-probe': env.STAGING_PROBE_SECRET }, + }) as Promise; + }, +}; diff --git a/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc new file mode 100644 index 00000000..7f01e4f1 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc @@ -0,0 +1,16 @@ +{ + "$schema": "https://unpkg.com/wrangler/config-schema.json", + "name": "anchorage-flowsafe-remote-step-staging-probe-local", + "main": "./notification-staging-probe.worker.ts", + "compatibility_date": "2026-10-14", + "compatibility_flags": ["nodejs_compat"], + "workers_dev": false, + "dev": { "ip": "127.0.0.1" }, + "services": [ + { + "binding": "SUBJECT", + "service": "anchorage-flowsafe-remote-step-staging", + "remote": true + } + ] +} diff --git a/packages/flowsafe/deploy/notification-staging.worker.ts b/packages/flowsafe/deploy/notification-staging.worker.ts new file mode 100644 index 00000000..b87c8d74 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging.worker.ts @@ -0,0 +1,207 @@ +import type { + Request as CfRequest, + D1Database, + DurableObjectNamespace, + DurableObjectState, + ExecutionContext, +} from '@cloudflare/workers-types'; +import { + type AgentThreadStateStorage, + createRemoteStepNotificationRecoveryTick, + createThreadAgentHost, + type ThreadAgentHost, +} from '../src/agent-host/index.js'; +import { credentialsMatch } from '../src/do-runner/deployment-identity.js'; +import { + assertDeploymentIdentitySecret, + createD1Storage, + createHostPubSub, + DurableObjectRunner, + doErrorResponse, + ensureDeploymentIdentityBindings, + executionFenceFor, + type InitResult, + init, + type RunnerRuntime, + startIdempotencyFor, + ThreadDurableObject, + type ThreadScope, +} from '../src/do-runner/index.js'; +import { RemoteStepJournal } from '../src/do-runner/remote-step-journal.js'; +import { + approvalStoreFactoryFor, + createFlowsafeMaintenanceDurableObject, + createFlowsafeWorker, + ensureRemoteStepNotificationRecoveryDuty, + type FlowsafeWorkerConfig, + staticTokenVerifier, +} from '../src/host-kit/index.js'; + +interface Env { + DB: D1Database; + DEPLOYMENT_TENANT: string; + DEPLOYMENT_IDENTITY_SECRET: string; + MAINTENANCE_ADMIN_SECRET: string; + STAGING_PROBE_SECRET: string; + NOTIFICATION_PROOF_ENABLED?: string; + RUNNER: DurableObjectNamespace; + THREAD: DurableObjectNamespace; + MAINTENANCE: DurableObjectNamespace; +} + +const OPERATION_ID = 'staging-notification-enqueue-v1'; +const AGENT_ID = 'staging-notification-agent'; + +export class NotificationStagingRunner extends DurableObjectRunner { + protected build(env: Env): RunnerRuntime { + return init(env as never).runtime; + } + + protected runOwnership(env: Env) { + return approvalStoreFactoryFor(env.DB).resources(); + } +} + +function journalFor(env: Env, runtime: ReturnType['runtime']) { + return new RemoteStepJournal({ + db: env.DB as unknown as never, + fence: executionFenceFor(env.DB as unknown as never), + runtime, + deploymentId: env.DEPLOYMENT_TENANT, + tablePrefix: '', + allowedToolIds: ['staging_notification'], + verifyCompletion: async () => { + throw new Error('staging notification completion is not configured'); + }, + }); +} + +export class NotificationStagingThread extends ThreadDurableObject { + #host?: ThreadAgentHost; + + constructor(state: DurableObjectState, env: Env) { + super(state as never, env); + } + + protected build(env: Env): InitResult { + const storage = createD1Storage({ binding: env.DB as unknown as never }); + const app = init( + { storage }, + { + executionFence: executionFenceFor(env.DB as unknown as never), + startIdempotency: startIdempotencyFor(env.DB as unknown as never), + pubsub: createHostPubSub(), + }, + ); + const journal = journalFor(env, app.runtime); + this.#host = createThreadAgentHost({ + buildModules: () => [], + storage: () => storage, + stateStorage: () => { + if (!this.state?.storage) + throw new Error('thread storage is unavailable'); + return this.state.storage as AgentThreadStateStorage; + }, + resourceAccess: () => approvalStoreFactoryFor(env.DB).resources(), + approvalService: () => { + throw new Error('staging agent continuation is unavailable'); + }, + ...(env.NOTIFICATION_PROOF_ENABLED === 'true' + ? { + remoteSteps: journal, + remoteStepNotification: { + journal, + agentId: AGENT_ID, + operationId: OPERATION_ID, + source: 'flowsafe-staging', + kind: 'host-owned-notification', + summary: 'Staging notification recovery', + ensureScheduledRecovery: () => + ensureRemoteStepNotificationRecoveryDuty(env), + }, + } + : {}), + }); + return app; + } + + protected async route( + request: Request, + scope: ThreadScope, + ): Promise { + return ( + (await this.#host?.route(request, scope)) ?? + new Response(null, { status: 404 }) + ); + } + + protected async onAlarm( + _env: Env, + threadId: string, + app: InitResult, + deploymentTag?: string, + ): Promise { + if (!this.#host) throw new Error('thread host is unavailable'); + await this.#host.recoverOwnership({ threadId, init: app, deploymentTag }); + } +} + +const config = { + workflows: [], + systemPrincipalId: 'staging-notification-worker', + buildVerifier: () => staticTokenVerifier(new Map()), + maintenance: { + sweepIntervalMs: 60_000, + purgeIntervalMs: 60_000, + remoteStepRecoveryIntervalMs: 60_000, + }, + remoteStepRecoveryTick: (env: Env) => async () => { + if (env.NOTIFICATION_PROOF_ENABLED !== 'true') + return { scanned: 0, recovered: 0, unresolved: 0 }; + const journal = journalFor(env, init(env as never).runtime); + await journal.initialize(); + return createRemoteStepNotificationRecoveryTick({ + journal, + namespace: env.THREAD, + deploymentIdentitySecret: env.DEPLOYMENT_IDENTITY_SECRET, + })(); + }, + preRoutes: async (request: Request, env: Env) => { + const url = new URL(request.url); + if (!url.pathname.startsWith('/__staging/')) return null; + if (env.NOTIFICATION_PROOF_ENABLED !== 'true') + return new Response(null, { status: 404 }); + assertDeploymentIdentitySecret( + env.STAGING_PROBE_SECRET, + 'staging probe credential', + ); + const supplied = request.headers.get('x-flowsafe-staging-probe'); + if ( + !supplied || + !(await credentialsMatch(supplied, env.STAGING_PROBE_SECRET)) + ) + return new Response(null, { status: 403 }); + if (request.method === 'POST' && url.pathname === '/__staging/ensure') { + await ensureRemoteStepNotificationRecoveryDuty(env); + return Response.json({ armed: true }); + } + return new Response(null, { status: 404 }); + }, +} satisfies FlowsafeWorkerConfig; + +export class NotificationStagingMaintenance extends createFlowsafeMaintenanceDurableObject( + config, +) {} + +const worker = createFlowsafeWorker(config); + +export default { + async fetch(request: CfRequest, env: Env, ctx: ExecutionContext) { + try { + await ensureDeploymentIdentityBindings(env); + return await worker.fetch(request as unknown as Request, env, ctx); + } catch (error) { + return doErrorResponse(error); + } + }, +}; diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc new file mode 100644 index 00000000..5c3c2ca0 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -0,0 +1,43 @@ +{ + "$schema": "https://unpkg.com/wrangler/config-schema.json", + "name": "anchorage-flowsafe-remote-step-staging", + "main": "./notification-staging.worker.ts", + "compatibility_date": "2026-10-14", + "compatibility_flags": ["nodejs_compat"], + "workers_dev": false, + "secrets": { + "required": [ + "DEPLOYMENT_IDENTITY_SECRET", + "MAINTENANCE_ADMIN_SECRET", + "STAGING_PROBE_SECRET" + ] + }, + "durable_objects": { + "bindings": [ + { "name": "RUNNER", "class_name": "NotificationStagingRunner" }, + { "name": "THREAD", "class_name": "NotificationStagingThread" }, + { "name": "MAINTENANCE", "class_name": "NotificationStagingMaintenance" } + ] + }, + "migrations": [ + { + "tag": "notification-staging-v1", + "new_sqlite_classes": [ + "NotificationStagingRunner", + "NotificationStagingThread", + "NotificationStagingMaintenance" + ] + } + ], + "d1_databases": [ + { + "binding": "DB", + "database_name": "anchorage-flowsafe-remote-step-staging-db", + "database_id": "b863a219-2096-4e13-92fe-4db592b67944" + } + ], + "vars": { + "DEPLOYMENT_TENANT": "flowsafe-notification-staging", + "NOTIFICATION_PROOF_ENABLED": "false" + } +} diff --git a/packages/flowsafe/deploy/tsconfig.json b/packages/flowsafe/deploy/tsconfig.json index 4263bb90..aa6aaf52 100644 --- a/packages/flowsafe/deploy/tsconfig.json +++ b/packages/flowsafe/deploy/tsconfig.json @@ -17,6 +17,8 @@ }, "include": [ "./worker.ts", + "./notification-staging.worker.ts", + "./notification-staging-probe.worker.ts", "../src/**/*.ts", "../test-support/maintenance-alarm-harness-worker.ts" ], From edf09f7f52fa5b2a84050a567402d56346af879f Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 01:13:55 +0400 Subject: [PATCH 52/73] Use a valid FlowSafe staging deployment tag Keep the isolated Worker variable aligned with the D1 sentinel identity validator before provisioning or deployment. --- packages/flowsafe/deploy/notification-staging.wrangler.jsonc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc index 5c3c2ca0..781899e4 100644 --- a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -37,7 +37,7 @@ } ], "vars": { - "DEPLOYMENT_TENANT": "flowsafe-notification-staging", + "DEPLOYMENT_TENANT": "flowsafenotificationstaging", "NOTIFICATION_PROOF_ENABLED": "false" } } From bebe8559077131ad1743c0c041216453665e51b4 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 01:59:41 +0400 Subject: [PATCH 53/73] Add guarded notification producer to private staging Use a fixed system-owned agent suspension and no-op dispatch to exercise the selected notification path. Keep the feature off; local workerd proof covers one receipt and continuation. --- .../deploy/notification-staging-native.mjs | 201 +++++++++++++ .../notification-staging-probe.worker.test.ts | 28 ++ .../notification-staging-probe.worker.ts | 15 +- .../notification-staging-probe.wrangler.jsonc | 2 +- .../deploy/notification-staging.worker.ts | 274 +++++++++++++++++- .../notification-staging.wrangler.jsonc | 5 +- 6 files changed, 508 insertions(+), 17 deletions(-) create mode 100644 packages/flowsafe/deploy/notification-staging-native.mjs diff --git a/packages/flowsafe/deploy/notification-staging-native.mjs b/packages/flowsafe/deploy/notification-staging-native.mjs new file mode 100644 index 00000000..c7b6e79e --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-native.mjs @@ -0,0 +1,201 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + EXECUTION_FENCE_DDL, + EXECUTION_FENCE_ROW_ID, + EXECUTION_FENCE_TABLE, +} from '#deployment-identity-protocol'; +import { + createWorkerdServerLifecycle, + parsePort, +} from '../../../scripts/workerd-server-lifecycle.mjs'; + +const flowsafe = dirname(dirname(fileURLToPath(import.meta.url))); +const wrangler = join(flowsafe, '../../node_modules/.bin/wrangler'); +const config = join(flowsafe, 'deploy/notification-staging.wrangler.jsonc'); +const port = parsePort( + process.env.NOTIFICATION_STAGING_NATIVE_PORT ?? 8897, + 'NOTIFICATION_STAGING_NATIVE_PORT', +); +const tag = 'flowsafenotificationstaging'; +const database = 'anchorage-flowsafe-remote-step-staging-db'; +const secret = () => randomBytes(32).toString('hex'); +const deploymentSecret = secret(); +const maintenanceSecret = secret(); +const probeSecret = secret(); +const lifecycle = createWorkerdServerLifecycle({ port }); +let stateDir; +let serverOutput = ''; + +function workerdDiagnostics() { + return serverOutput + .replaceAll(deploymentSecret, '[redacted]') + .replaceAll(maintenanceSecret, '[redacted]') + .replaceAll(probeSecret, '[redacted]'); +} + +function expect(condition, message, detail) { + if (!condition) + throw new Error( + `${message}${detail === undefined ? '' : `: ${JSON.stringify(detail)}`}`, + ); +} + +function localD1(command) { + const result = spawnSync( + wrangler, + [ + 'd1', + 'execute', + database, + '--local', + '--yes', + '--json', + '--config', + config, + '--persist-to', + stateDir, + '--command', + command, + ], + { cwd: flowsafe, encoding: 'utf8', timeout: 30_000 }, + ); + expect( + result.status === 0, + 'local D1 query failed', + result.stderr?.slice(-800), + ); + return JSON.parse(result.stdout); +} + +async function post(path, authorization, body) { + const response = await fetch(`http://127.0.0.1:${port}${path}`, { + method: 'POST', + headers: { + ...(authorization ? { 'x-flowsafe-staging-probe': authorization } : {}), + ...(body ? { 'content-type': 'application/json' } : {}), + }, + ...(body ? { body: JSON.stringify(body) } : {}), + signal: AbortSignal.timeout(90_000), + }); + const text = await response.text(); + let value; + try { + value = JSON.parse(text); + } catch { + value = text.slice(0, 1000); + } + return { status: response.status, value }; +} + +async function run() { + await lifecycle.preflight(); + stateDir = mkdtempSync(join(tmpdir(), 'flowsafe-notification-native-')); + localD1( + 'CREATE TABLE IF NOT EXISTS flowsafe_deployment (id INTEGER PRIMARY KEY CHECK (id = 1), tenant_tag TEXT NOT NULL, provisioned_at TEXT NOT NULL); ' + + `INSERT OR IGNORE INTO flowsafe_deployment (id, tenant_tag, provisioned_at) VALUES (1, '${tag}', datetime('now')); ` + + `${EXECUTION_FENCE_DDL}; ` + + `INSERT OR IGNORE INTO ${EXECUTION_FENCE_TABLE} (id, state, proof_key, proof_run_id, updated_at) ` + + `VALUES ('${EXECUTION_FENCE_ROW_ID}', 'open', NULL, NULL, ${Date.now()});`, + ); + await lifecycle.start('notification-staging-native', () => { + const child = spawn( + wrangler, + [ + 'dev', + '--config', + config, + '--local', + '--port', + String(port), + '--ip', + '127.0.0.1', + '--inspector-port', + '0', + '--persist-to', + stateDir, + '--var', + `DEPLOYMENT_TENANT:${tag}`, + '--var', + `DEPLOYMENT_IDENTITY_SECRET:${deploymentSecret}`, + '--var', + `MAINTENANCE_ADMIN_SECRET:${maintenanceSecret}`, + '--var', + `STAGING_PROBE_SECRET:${probeSecret}`, + '--var', + 'NOTIFICATION_PROOF_ENABLED:true', + ], + { + cwd: flowsafe, + detached: true, + stdio: ['ignore', 'pipe', 'pipe'], + env: { ...process.env, WRANGLER_SEND_METRICS: 'false', NO_COLOR: '1' }, + }, + ); + const capture = (chunk) => { + serverOutput = `${serverOutput}${chunk.toString()}`.slice(-30000); + }; + child.stdout.on('data', capture); + child.stderr.on('data', capture); + return { child, generation: 'notification-staging-native' }; + }); + const refused = await post('/__staging/start'); + expect( + refused.status === 403, + 'private start admitted an unauthenticated call', + refused, + ); + const started = await post('/__staging/start', probeSecret); + expect(started.status === 200, 'guarded start/register failed', { + response: started, + workerd: workerdDiagnostics(), + }); + expect( + started.value.status === 'suspended', + 'guarded agent did not suspend', + started, + ); + expect( + typeof started.value.invocationId === 'string', + 'journal registration did not return an invocation', + started, + ); + const completed = await post('/__staging/complete', probeSecret, { + invocationId: started.value.invocationId, + }); + expect(completed.status === 200, 'selected notification completion failed', { + response: completed, + workerd: workerdDiagnostics(), + }); + const rows = localD1( + `SELECT COUNT(*) AS count FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification'`, + ); + expect( + rows?.[0]?.results?.[0]?.count === 1, + 'selected notification was not retained exactly once', + rows, + ); + console.log( + JSON.stringify({ + verdict: 'native_notification_staging_passed', + runId: started.value.runId, + invocationId: started.value.invocationId, + completionStatus: completed.value.status, + notificationCount: 1, + }), + ); +} + +try { + await run(); +} catch (error) { + console.error(workerdDiagnostics()); + throw error; +} finally { + await lifecycle.stop(); + if (stateDir) rmSync(stateDir, { recursive: true, force: true }); +} diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts index 8ff1c642..28bec20e 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts @@ -30,3 +30,31 @@ it('forwards a staging request only after the local caller supplies the probe cr 'x-flowsafe-staging-probe': secret, }); }); + +it('keeps start and completion on the authenticated private service binding', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 204 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + }; + for (const path of ['/start', '/complete']) { + const body = + path === '/complete' ? '{"invocationId":"invocation-1"}' : undefined; + const request = (credential?: string) => + new Request(`http://127.0.0.1${path}`, { + method: 'POST', + ...(credential + ? { headers: { authorization: `Bearer ${credential}` } } + : {}), + ...(body ? { body } : {}), + }); + expect((await handler.fetch(request(), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + expect((await handler.fetch(request(secret), env)).status).toBe(204); + const forwarded = fetch.mock.calls.at(-1); + expect(forwarded?.[0]).toBe(`http://subject/__staging${path}`); + if (body) expect(forwarded?.[1]?.body).toBe(body); + fetch.mockClear(); + } +}); diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.ts index b25897f5..2a915478 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.worker.ts +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.ts @@ -14,7 +14,10 @@ export default { const url = new URL(request.url); if (url.hostname !== '127.0.0.1' && url.hostname !== 'localhost') return new Response(null, { status: 404 }); - if (request.method !== 'POST' || url.pathname !== '/ensure') + if ( + request.method !== 'POST' || + !['/ensure', '/start', '/complete'].includes(url.pathname) + ) return new Response(null, { status: 404 }); try { assertDeploymentIdentitySecret( @@ -33,9 +36,15 @@ export default { )) ) return new Response(null, { status: 403 }); - return env.SUBJECT.fetch('http://subject/__staging/ensure', { + return env.SUBJECT.fetch(`http://subject/__staging${url.pathname}`, { method: 'POST', - headers: { 'x-flowsafe-staging-probe': env.STAGING_PROBE_SECRET }, + headers: { + 'x-flowsafe-staging-probe': env.STAGING_PROBE_SECRET, + ...(url.pathname === '/complete' + ? { 'content-type': 'application/json' } + : {}), + }, + ...(url.pathname === '/complete' ? { body: await request.text() } : {}), }) as Promise; }, }; diff --git a/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc index 7f01e4f1..a121acfe 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging-probe.wrangler.jsonc @@ -2,7 +2,7 @@ "$schema": "https://unpkg.com/wrangler/config-schema.json", "name": "anchorage-flowsafe-remote-step-staging-probe-local", "main": "./notification-staging-probe.worker.ts", - "compatibility_date": "2026-10-14", + "compatibility_date": "2026-09-10", "compatibility_flags": ["nodejs_compat"], "workers_dev": false, "dev": { "ip": "127.0.0.1" }, diff --git a/packages/flowsafe/deploy/notification-staging.worker.ts b/packages/flowsafe/deploy/notification-staging.worker.ts index b87c8d74..3367fcca 100644 --- a/packages/flowsafe/deploy/notification-staging.worker.ts +++ b/packages/flowsafe/deploy/notification-staging.worker.ts @@ -5,12 +5,26 @@ import type { DurableObjectState, ExecutionContext, } from '@cloudflare/workers-types'; +import type { Agent } from '@mastra/core/agent'; +import type { MastraModelConfig } from '@mastra/core/llm'; +import { Mastra } from '@mastra/core/mastra'; +import { createTool } from '@mastra/core/tools'; +import { createGuardedAgent } from '@proofoftech/breakwater/agent'; +import { AuditLogger } from '@proofoftech/breakwater/audit'; +import { z } from 'zod'; import { + type AgentModule, type AgentThreadStateStorage, + createAgentThreadTopology, createRemoteStepNotificationRecoveryTick, createThreadAgentHost, type ThreadAgentHost, } from '../src/agent-host/index.js'; +import { createFlowsafeDurableAgent } from '../src/agent-runner/index.js'; +import { + ApprovalService, + createPrincipalActorContext, +} from '../src/approval-api/index.js'; import { credentialsMatch } from '../src/do-runner/deployment-identity.js'; import { assertDeploymentIdentitySecret, @@ -27,7 +41,10 @@ import { ThreadDurableObject, type ThreadScope, } from '../src/do-runner/index.js'; -import { RemoteStepJournal } from '../src/do-runner/remote-step-journal.js'; +import { + REMOTE_STEP_MARKER, + RemoteStepJournal, +} from '../src/do-runner/remote-step-journal.js'; import { approvalStoreFactoryFor, createFlowsafeMaintenanceDurableObject, @@ -51,6 +68,95 @@ interface Env { const OPERATION_ID = 'staging-notification-enqueue-v1'; const AGENT_ID = 'staging-notification-agent'; +const TOOL_ID = 'staging_notification'; +const SYSTEM_ID = 'staging-notification-worker'; +const RESULT_JSON = '{"completed":true}'; +const EVIDENCE_JSON = '{"operation":"no-side-effect-staging-dispatch-v1"}'; + +function stagingAgent(): AgentModule { + const audit = new AuditLogger({ + sink: (event) => console.log(JSON.stringify(event)), + }); + const tool = createTool({ + id: TOOL_ID, + description: 'Suspend for the host-owned staging notification', + inputSchema: z.object({}), + outputSchema: z.object({ completed: z.boolean() }), + resumeSchema: z.object({ completed: z.boolean() }), + execute: async (_input, context) => { + if (context.agent?.resumeData) return { completed: true }; + if (!context.agent?.suspend) + throw new Error('staging tool suspension is unavailable'); + await context.agent.suspend({ + remoteStep: REMOTE_STEP_MARKER, + request: { notificationOperationId: OPERATION_ID }, + }); + throw new Error('staging tool suspension returned'); + }, + }); + const model: MastraModelConfig = { + specificationVersion: 'v2', + provider: 'flowsafe-staging', + modelId: 'fixed-notification-tool', + supportedUrls: {}, + doGenerate: async () => ({ + content: [ + { + type: 'tool-call', + toolCallId: `call_${crypto.randomUUID()}`, + toolName: TOOL_ID, + input: '{}', + }, + ], + finishReason: 'tool-calls', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + warnings: [], + }), + doStream: async () => ({ + stream: new ReadableStream({ + start(controller) { + controller.enqueue({ type: 'stream-start', warnings: [] }); + controller.enqueue({ + type: 'tool-call', + toolCallId: `call_${crypto.randomUUID()}`, + toolName: TOOL_ID, + input: '{}', + }); + controller.enqueue({ + type: 'finish', + finishReason: 'tool-calls', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + }); + controller.close(); + }, + }), + }), + }; + return { + meta: { + id: AGENT_ID, + title: 'Staging notification agent', + description: 'Produces a selected host-owned notification suspension', + allowedRoles: ['operator'], + allowedAutomation: [ + { kind: 'system', entryPaths: ['notification.dispatch'] }, + ], + }, + agent: createGuardedAgent({ + id: AGENT_ID, + name: 'Staging notification agent', + instructions: 'Call staging_notification once.', + model, + tools: { [TOOL_ID]: tool }, + allowedRoles: ['operator'], + allowedPrincipalKinds: ['system'], + policies: [], + audit, + maxSteps: 1, + toolChoice: 'required', + }), + }; +} export class NotificationStagingRunner extends DurableObjectRunner { protected build(env: Env): RunnerRuntime { @@ -69,13 +175,47 @@ function journalFor(env: Env, runtime: ReturnType['runtime']) { runtime, deploymentId: env.DEPLOYMENT_TENANT, tablePrefix: '', - allowedToolIds: ['staging_notification'], - verifyCompletion: async () => { - throw new Error('staging notification completion is not configured'); + allowedToolIds: [TOOL_ID], + verifyCompletion: async ({ invocation, resultJson, evidenceJson }) => { + if ( + invocation.request === null || + typeof invocation.request !== 'object' || + (invocation.request as { notificationOperationId?: unknown }) + .notificationOperationId !== OPERATION_ID || + resultJson !== RESULT_JSON || + evidenceJson !== EVIDENCE_JSON + ) + throw new Error( + 'staging no-op completion does not match the operation', + ); }, }); } +function stagingRuntime(env: Env): { + runtime: ReturnType['runtime']; + workflowId: string; +} { + const storage = createD1Storage({ binding: env.DB as unknown as never }); + const app = init( + { storage }, + { + executionFence: executionFenceFor(env.DB as unknown as never), + startIdempotency: startIdempotencyFor(env.DB as unknown as never), + }, + ); + const agents = { + [AGENT_ID]: stagingAgent().agent as unknown as Agent, + } as Record; + const mastra = new Mastra({ storage, agents }); + const durable = createFlowsafeDurableAgent({ + agent: mastra.getAgentById(AGENT_ID), + runtime: app.runtime, + threadRuntime: mastra.agentThreadStreamRuntime, + }); + return { runtime: app.runtime, workflowId: durable.getWorkflow().id }; +} + export class NotificationStagingThread extends ThreadDurableObject { #host?: ThreadAgentHost; @@ -91,11 +231,15 @@ export class NotificationStagingThread extends ThreadDurableObject { executionFence: executionFenceFor(env.DB as unknown as never), startIdempotency: startIdempotencyFor(env.DB as unknown as never), pubsub: createHostPubSub(), + requestContextForRun: (workflowId, runId, leg) => { + if (!this.#host) throw new Error('thread agent host is unavailable'); + return this.#host.requestContextForRun()(workflowId, runId, leg); + }, }, ); const journal = journalFor(env, app.runtime); this.#host = createThreadAgentHost({ - buildModules: () => [], + buildModules: () => [stagingAgent()], storage: () => storage, stateStorage: () => { if (!this.state?.storage) @@ -103,9 +247,12 @@ export class NotificationStagingThread extends ThreadDurableObject { return this.state.storage as AgentThreadStateStorage; }, resourceAccess: () => approvalStoreFactoryFor(env.DB).resources(), - approvalService: () => { - throw new Error('staging agent continuation is unavailable'); - }, + approvalService: () => + new ApprovalService({ + store: approvalStoreFactoryFor(env.DB).store(), + executionFence: executionFenceFor(env.DB as unknown as never), + workflowTablePrefix: '', + }), ...(env.NOTIFICATION_PROOF_ENABLED === 'true' ? { remoteSteps: journal, @@ -148,7 +295,7 @@ export class NotificationStagingThread extends ThreadDurableObject { const config = { workflows: [], - systemPrincipalId: 'staging-notification-worker', + systemPrincipalId: SYSTEM_ID, buildVerifier: () => staticTokenVerifier(new Map()), maintenance: { sweepIntervalMs: 60_000, @@ -158,7 +305,7 @@ const config = { remoteStepRecoveryTick: (env: Env) => async () => { if (env.NOTIFICATION_PROOF_ENABLED !== 'true') return { scanned: 0, recovered: 0, unresolved: 0 }; - const journal = journalFor(env, init(env as never).runtime); + const journal = journalFor(env, stagingRuntime(env).runtime); await journal.initialize(); return createRemoteStepNotificationRecoveryTick({ journal, @@ -166,7 +313,7 @@ const config = { deploymentIdentitySecret: env.DEPLOYMENT_IDENTITY_SECRET, })(); }, - preRoutes: async (request: Request, env: Env) => { + preRoutes: async (request: Request, env: Env, _ctx, kit) => { const url = new URL(request.url); if (!url.pathname.startsWith('/__staging/')) return null; if (env.NOTIFICATION_PROOF_ENABLED !== 'true') @@ -185,7 +332,110 @@ const config = { await ensureRemoteStepNotificationRecoveryDuty(env); return Response.json({ armed: true }); } - return new Response(null, { status: 404 }); + if ( + request.method !== 'POST' || + (url.pathname !== '/__staging/start' && + url.pathname !== '/__staging/complete') + ) + return new Response(null, { status: 404 }); + const { runtime, workflowId } = stagingRuntime(env); + const journal = journalFor(env, runtime); + await journal.initialize(); + const storeFactory = approvalStoreFactoryFor(env.DB); + const context = createPrincipalActorContext({ + principal: { + kind: 'system', + id: env.DEPLOYMENT_TENANT, + purpose: 'private-notification-staging-proof', + }, + storeFactory, + deploymentTag: env.DEPLOYMENT_TENANT, + buildService: (store) => + new ApprovalService({ + store, + executionFence: executionFenceFor(env.DB as unknown as never), + workflowTablePrefix: '', + }), + }); + const topology = createAgentThreadTopology( + env.THREAD, + env.DEPLOYMENT_IDENTITY_SECRET, + { + startIdempotency: startIdempotencyFor(env.DB as unknown as never), + executionFence: executionFenceFor(env.DB as unknown as never), + remoteStepDispatch: journal, + remoteSteps: journal, + }, + ); + if (url.pathname === '/__staging/start') { + const threadId = context.newThreadId(); + const resourceId = context.resourceIdFromKey(threadId); + await context.claimResource('thread', threadId); + await context.claimResource('resource', resourceId); + const started = await topology.start(context, { + agentId: AGENT_ID, + prompt: 'Produce the host-owned staging notification.', + entryPath: 'notification.dispatch', + threadId, + resourceId, + threaded: true, + }); + if (started.summary.status !== 'suspended') + throw new Error('staging agent did not suspend'); + const invocation = await journal.register( + { + deploymentId: env.DEPLOYMENT_TENANT, + tablePrefix: '', + workflowId, + agentId: started.agentId, + threadId: started.threadId, + resourceId: started.resourceId, + runId: started.runId, + }, + Date.now() + 120_000, + context.principal, + ); + return Response.json({ + invocationId: invocation.invocationId, + runId: started.runId, + threadId: started.threadId, + status: started.summary.status, + }); + } + const body = (await request.json()) as { invocationId?: unknown }; + if (typeof body.invocationId !== 'string') + return new Response(null, { status: 400 }); + const stored = await kit.contextForStoredInvocation( + journal, + body.invocationId, + ); + if ( + stored?.principal.kind !== 'system' || + stored.principal.id !== context.principal.id + ) + return new Response(null, { status: 404 }); + const ticket = await topology.issueRemoteStepDispatch( + stored, + body.invocationId, + 30_000, + ); + const consumed = await topology.consumeRemoteStepDispatch( + stored, + ticket.permit, + ); + if (consumed.kind !== 'accepted') + throw new Error('staging no-op dispatch was already consumed'); + await journal.recordResult( + body.invocationId, + consumed.rawPermit, + RESULT_JSON, + EVIDENCE_JSON, + ); + const resumed = await topology.resumeRemoteStep(stored, body.invocationId); + return Response.json({ + invocationId: body.invocationId, + status: resumed.summary.status, + }); }, } satisfies FlowsafeWorkerConfig; diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc index 781899e4..90e6bf28 100644 --- a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -2,8 +2,11 @@ "$schema": "https://unpkg.com/wrangler/config-schema.json", "name": "anchorage-flowsafe-remote-step-staging", "main": "./notification-staging.worker.ts", - "compatibility_date": "2026-10-14", + "compatibility_date": "2026-09-10", "compatibility_flags": ["nodejs_compat"], + "alias": { + "@proofoftech/breakwater/agent": "../../breakwater/src/agent/index.ts" + }, "workers_dev": false, "secrets": { "required": [ From cd344eebd25af5169e4bf3c6e8a9c14cabbef0ba Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 02:18:12 +0400 Subject: [PATCH 54/73] Proxy private staging fence administration locally Require the local staging credential before forwarding bounded fence and maintenance requests with a distinct admin secret over the private Service binding. --- .../notification-staging-probe.worker.test.ts | 87 +++++++++++++++++++ .../notification-staging-probe.worker.ts | 69 ++++++++++++--- 2 files changed, 142 insertions(+), 14 deletions(-) diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts index 28bec20e..6c7ce1c8 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts @@ -58,3 +58,90 @@ it('keeps start and completion on the authenticated private service binding', as fetch.mockClear(); } }); + +it('authenticates local admin requests and supplies only the proxy-owned admin credential', async () => { + const probeSecret = 'staging-local-probe-secret-32-bytes-minimum'; + const adminSecret = 'staging-admin-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 200 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: probeSecret, + MAINTENANCE_ADMIN_SECRET: adminSecret, + }; + for (const [method, path, body] of [ + ['GET', '/admin/execution-fence', undefined], + ['POST', '/admin/execution-fence', '{"expected":"open","next":"draining"}'], + ['GET', '/admin/maintenance-status', undefined], + ] as const) { + const request = (credential: string) => + new Request(`http://127.0.0.1${path}`, { + method, + headers: { authorization: `Bearer ${credential}` }, + ...(body ? { body } : {}), + }); + expect((await handler.fetch(request('wrong'), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + expect((await handler.fetch(request(probeSecret), env)).status).toBe(200); + expect(fetch).toHaveBeenCalledOnce(); + expect(fetch.mock.calls[0]?.[0]).toBe(`http://subject${path}`); + expect(fetch.mock.calls[0]?.[1]).toMatchObject({ + method, + headers: { authorization: `Bearer ${adminSecret}` }, + }); + expect(fetch.mock.calls[0]?.[1]?.body).toBe(body); + fetch.mockClear(); + } +}); + +it('rejects unlisted admin operations, query strings, and oversized writes', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 200 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + MAINTENANCE_ADMIN_SECRET: 'staging-admin-secret-32-bytes-minimum', + }; + const headers = { authorization: `Bearer ${secret}` }; + for (const request of [ + new Request('http://127.0.0.1/admin/inventory', { headers }), + new Request('http://127.0.0.1/admin/execution-fence', { + method: 'DELETE', + headers, + }), + new Request('http://127.0.0.1/admin/execution-fence?state=open', { + headers, + }), + ]) { + expect((await handler.fetch(request, env)).status).toBe(404); + } + expect( + ( + await handler.fetch( + new Request('http://127.0.0.1/admin/execution-fence', { + method: 'POST', + headers, + body: 'x'.repeat(4_097), + }), + env, + ) + ).status, + ).toBe(413); + expect(fetch).not.toHaveBeenCalled(); +}); + +it('fails closed when the admin credential is missing or reused as the caller credential', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => new Response(null, { status: 200 })); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + MAINTENANCE_ADMIN_SECRET: undefined as string | undefined, + }; + const request = new Request('http://127.0.0.1/admin/execution-fence', { + headers: { authorization: `Bearer ${secret}` }, + }); + expect((await handler.fetch(request, env)).status).toBe(503); + env.MAINTENANCE_ADMIN_SECRET = secret; + expect((await handler.fetch(request, env)).status).toBe(503); + expect(fetch).not.toHaveBeenCalled(); +}); diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.ts index 2a915478..0d602514 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.worker.ts +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.ts @@ -3,10 +3,14 @@ import { assertDeploymentIdentitySecret, credentialsMatch, } from '../src/do-runner/deployment-identity.js'; +import { readBoundedBody } from '../src/http-body.js'; + +const MAX_FORWARD_BODY_BYTES = 4_096; interface Env { SUBJECT: Fetcher; STAGING_PROBE_SECRET: string; + MAINTENANCE_ADMIN_SECRET?: string; } export default { @@ -14,11 +18,17 @@ export default { const url = new URL(request.url); if (url.hostname !== '127.0.0.1' && url.hostname !== 'localhost') return new Response(null, { status: 404 }); - if ( - request.method !== 'POST' || - !['/ensure', '/start', '/complete'].includes(url.pathname) - ) - return new Response(null, { status: 404 }); + if (url.search || url.hash) return new Response(null, { status: 404 }); + const staging = + request.method === 'POST' && + ['/ensure', '/start', '/complete'].includes(url.pathname); + const admin = + (request.method === 'GET' && + ['/admin/execution-fence', '/admin/maintenance-status'].includes( + url.pathname, + )) || + (request.method === 'POST' && url.pathname === '/admin/execution-fence'); + if (!staging && !admin) return new Response(null, { status: 404 }); try { assertDeploymentIdentitySecret( env.STAGING_PROBE_SECRET, @@ -36,15 +46,46 @@ export default { )) ) return new Response(null, { status: 403 }); - return env.SUBJECT.fetch(`http://subject/__staging${url.pathname}`, { - method: 'POST', - headers: { - 'x-flowsafe-staging-probe': env.STAGING_PROBE_SECRET, - ...(url.pathname === '/complete' - ? { 'content-type': 'application/json' } - : {}), + if (admin) { + try { + assertDeploymentIdentitySecret( + env.MAINTENANCE_ADMIN_SECRET, + 'staging maintenance admin credential', + ); + } catch { + return new Response(null, { status: 503 }); + } + if ( + await credentialsMatch( + env.STAGING_PROBE_SECRET, + env.MAINTENANCE_ADMIN_SECRET as string, + ) + ) + return new Response(null, { status: 503 }); + } + const hasBody = + (staging && url.pathname === '/complete') || + (admin && request.method === 'POST'); + const raw = await readBoundedBody( + request, + hasBody ? MAX_FORWARD_BODY_BYTES : 0, + ); + if (!raw.ok) + return new Response(null, { + status: raw.reason === 'payload-too-large' ? 413 : 400, + }); + return env.SUBJECT.fetch( + `http://subject${admin ? '' : '/__staging'}${url.pathname}`, + { + method: request.method, + headers: { + ...(admin + ? { authorization: `Bearer ${env.MAINTENANCE_ADMIN_SECRET}` } + : { 'x-flowsafe-staging-probe': env.STAGING_PROBE_SECRET }), + ...(hasBody ? { 'content-type': 'application/json' } : {}), + }, + ...(hasBody ? { body: raw.text } : {}), }, - ...(url.pathname === '/complete' ? { body: await request.text() } : {}), - }) as Promise; + ) as Promise; }, }; From 3a1a4ce7035464a6766b6aac4cfa28da423d1b72 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 02:37:17 +0400 Subject: [PATCH 55/73] Add locked notification staging control and watchdog Require v5 empty-journal preflight and exact D1 effect receipts. Relock a controller-owned fence on failure and keep live execution closed until the owning deployment lock is available. --- packages/flowsafe/deploy/README.md | 17 + .../deploy/notification-staging-control.mjs | 269 ++++++++++++ .../notification-staging-control.test.mjs | 227 ++++++++++ .../deploy/notification-staging-live.mjs | 391 ++++++++++++++++++ .../deploy/notification-staging-live.test.mjs | 134 ++++++ 5 files changed, 1038 insertions(+) create mode 100644 packages/flowsafe/deploy/notification-staging-control.mjs create mode 100644 packages/flowsafe/deploy/notification-staging-control.test.mjs create mode 100644 packages/flowsafe/deploy/notification-staging-live.mjs create mode 100644 packages/flowsafe/deploy/notification-staging-live.test.mjs diff --git a/packages/flowsafe/deploy/README.md b/packages/flowsafe/deploy/README.md index 2aa8c65a..499dc0e4 100644 --- a/packages/flowsafe/deploy/README.md +++ b/packages/flowsafe/deploy/README.md @@ -215,3 +215,20 @@ pnpm --filter @proofoftech/flowsafe spike:verify `spike:verify` adds the workerd boundary: suspend, kill, restart on persisted state, resume from an approved record, and refuse a mismatched deployment sentinel. See the repository [deployment reference](../../../docs/deployment-reference.md), [operations runbook](../../../docs/operations-runbook.md), and [security threat model](../../../docs/security-threat-model.md) before exposing traffic. + +## Run the isolated notification staging control + +Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The script refuses to unlock unless the fence is `migration-locked` at epoch and revision zero, the v5 journal is empty, and the recovery alarm is armed. Its detached watchdog relocks an open fence after controller death or its 15-minute deadline. Both processes request the tracked off deployment only while the current Worker version matches the enabled version supplied to the controller. + +The Wrangler version read and deploy are separate operations. Hold the owning deployment lock across the proof and off deployment, including the watchdog's recovery window. Do not run this control while another actor can deploy the subject Worker; the script cannot enforce that lock remotely. + +Set these environment variables before running `node deploy/notification-staging-live.mjs --run` from `packages/flowsafe`: + +| Variable | Value | +| --- | --- | +| `FLOWSAFE_STAGING_PROBE_PORT` | Port of the local Service-binding probe | +| `FLOWSAFE_STAGING_PROBE_SECRET_FILE` | Absolute path to the probe credential in a `0600` file | +| `FLOWSAFE_CF_TOKEN_FILE` | Absolute path to the Cloudflare API token in a `0600` file | +| `FLOWSAFE_STAGING_ENABLED_VERSION` | Exact 100% enabled subject Worker version ID | + +The controller prints the invocation ID only after D1 confirms one settled invocation, its notification, and its operation receipt. It writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the controller exits without a confirmed off deployment, inspect that private directory's `watchdog.log` and verify the fence and current Worker version through the authenticated proxy and Wrangler before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. diff --git a/packages/flowsafe/deploy/notification-staging-control.mjs b/packages/flowsafe/deploy/notification-staging-control.mjs new file mode 100644 index 00000000..ba4bb711 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-control.mjs @@ -0,0 +1,269 @@ +import { readFileSync, statSync } from 'node:fs'; + +const initialFence = { + state: 'migration-locked', + mutationEpoch: 0, + requireMutationEpoch: false, + transitionRevision: 0, +}; + +function assert(condition, message) { + if (!condition) throw new Error(message); +} + +function exactFence(value, expected) { + assert(value && typeof value === 'object', 'fence reading is missing'); + for (const [key, expectedValue] of Object.entries(expected)) + assert(value[key] === expectedValue, `fence ${key} differs`); + assert(value.proofKey === undefined, 'fence has a proof key'); + assert(value.proofRunId === undefined, 'fence has a proof run'); +} + +function nextFence(value, state) { + assert(Number.isSafeInteger(value.mutationEpoch), 'invalid fence epoch'); + assert( + Number.isSafeInteger(value.transitionRevision), + 'invalid fence revision', + ); + assert( + value.mutationEpoch < Number.MAX_SAFE_INTEGER, + 'fence epoch exhausted', + ); + assert( + value.transitionRevision < Number.MAX_SAFE_INTEGER, + 'fence revision exhausted', + ); + return { + state, + mutationEpoch: value.mutationEpoch + 1, + requireMutationEpoch: true, + transitionRevision: value.transitionRevision + 1, + }; +} + +export function readSecretFile(path) { + assert( + typeof path === 'string' && path.startsWith('/'), + 'absolute secret path required', + ); + const info = statSync(path); + assert( + info.isFile() && (info.mode & 0o077) === 0, + 'secret file must be private', + ); + const secret = readFileSync(path, 'utf8').trim(); + assert(/^[\x21-\x7e]{32,256}$/.test(secret), 'invalid probe secret'); + return secret; +} + +export function localProbe({ port, secret, fetcher = fetch }) { + assert( + Number.isInteger(port) && port > 0 && port < 65536, + 'invalid local port', + ); + return async (method, path, body) => { + const response = await fetcher(`http://127.0.0.1:${port}${path}`, { + method, + headers: { + authorization: `Bearer ${secret}`, + ...(body === undefined ? {} : { 'content-type': 'application/json' }), + }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + signal: AbortSignal.timeout(90_000), + }); + assert( + response.status === 200, + `local probe ${path} returned ${response.status}`, + ); + return response.json(); + }; +} + +export function trustedD1Checks(query) { + let baseline; + const counts = async () => { + const rows = await query( + "SELECT (SELECT COUNT(*) FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification') AS notifications, (SELECT COUNT(*) FROM flowsafe_notification_operation_receipts) AS receipts", + ); + assert( + rows.length === 1 && + Number.isSafeInteger(rows[0].notifications) && + Number.isSafeInteger(rows[0].receipts), + 'notification baseline is unreadable', + ); + return rows[0]; + }; + return { + preflightD1: async () => { + const schema = await query( + "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'flowsafe_remote_step_invocations'", + ); + assert( + schema.length === 1 && + typeof schema[0].sql === 'string' && + schema[0].sql.includes('write_generation INTEGER NOT NULL') && + schema[0].sql.includes('operation_state TEXT'), + 'remote step v5 schema is missing', + ); + const rows = await query( + 'SELECT COUNT(*) AS count FROM flowsafe_remote_step_invocations', + ); + assert( + rows.length === 1 && rows[0].count === 0, + 'remote step journal is not empty', + ); + baseline = await counts(); + }, + verifyD1: async (invocationId) => { + assert( + /^(?!\.{1,2}$)[A-Za-z0-9._~-]{1,200}$/.test(invocationId), + 'invalid invocation ID', + ); + const rows = await query( + `SELECT i.status, i.settlement_outcome, i.operation_state, i.thread_id, i.notification_id, ` + + `r.operation_key AS receipt_key, r.request_digest AS receipt_digest, ` + + `r.thread_id AS receipt_thread_id, r.notification_id AS receipt_notification_id, ` + + `n.id AS stored_notification_id, ` + + `n.source, n.kind FROM flowsafe_remote_step_invocations i ` + + `LEFT JOIN flowsafe_notification_operation_receipts r ON r.operation_key = i.operation_key ` + + `AND r.thread_id = i.thread_id AND r.notification_id = i.notification_id ` + + `AND r.request_digest = i.notification_request_digest ` + + `LEFT JOIN mastra_notifications n ON n.id = i.notification_id AND n.thread_id = i.thread_id ` + + `WHERE i.invocation_id = '${invocationId}'`, + ); + assert(rows.length === 1, 'expected exactly one staging invocation'); + const row = rows[0]; + assert( + row.status === 'settled' && + row.settlement_outcome === 'applied' && + row.operation_state === 'resume-admitted' && + typeof row.notification_id === 'string' && + row.notification_id === row.stored_notification_id && + typeof row.receipt_key === 'string' && + typeof row.receipt_thread_id === 'string' && + row.receipt_thread_id === row.thread_id && + row.receipt_notification_id === row.notification_id && + /^[a-f0-9]{64}$/.test(row.receipt_digest) && + row.source === 'flowsafe-staging' && + row.kind === 'host-owned-notification', + 'staging notification or receipt is unconfirmed', + ); + const after = await counts(); + assert( + baseline && + after.notifications === baseline.notifications + 1 && + after.receipts === baseline.receipts + 1, + 'staging notification or receipt count differs', + ); + }, + }; +} + +export async function runNotificationStagingControl({ + probe, + preflightD1, + verifyD1, + requestOffDeploy, + onBeforeUnlock = async () => {}, +}) { + assert( + typeof requestOffDeploy === 'function', + 'off deploy watchdog is required', + ); + let initial; + let unlockAttempted = false; + let outcome; + let failure; + try { + initial = await probe('GET', '/admin/execution-fence'); + exactFence(initial, initialFence); + const maintenance = await probe('GET', '/admin/maintenance-status'); + assert( + Number.isSafeInteger(maintenance.alarmAt) && + Number.isSafeInteger(maintenance.nextRemoteStepRecoveryAt) && + maintenance.alarmAt <= + Math.max(Date.now(), maintenance.nextRemoteStepRecoveryAt) && + maintenance.lastRemoteStepRecoveryError === undefined, + 'notification recovery duty is not armed', + ); + await preflightD1(); + await onBeforeUnlock(); + unlockAttempted = true; + const opened = await probe('POST', '/admin/execution-fence', { + expected: 'migration-locked', + next: 'open', + expectedMutationEpoch: initial.mutationEpoch, + expectedRevision: initial.transitionRevision, + advanceMutationEpoch: true, + }); + exactFence(opened, nextFence(initial, 'open')); + const started = await probe('POST', '/start'); + assert( + started?.status === 'suspended' && + typeof started.invocationId === 'string' && + started.invocationId.length > 0, + 'staging start did not suspend', + ); + const completed = await probe('POST', '/complete', { + invocationId: started.invocationId, + }); + assert( + completed?.invocationId === started.invocationId && + completed.status === 'success', + 'staging completion did not settle', + ); + await verifyD1(started.invocationId); + outcome = { + invocationId: started.invocationId, + verdict: 'notification_staging_passed', + }; + } catch (error) { + failure = error; + } finally { + if (unlockAttempted) { + try { + const current = await probe('GET', '/admin/execution-fence'); + if (current.state === 'open') { + assert( + current.requireMutationEpoch === true, + 'open fence has wrong epoch policy', + ); + assert( + current.mutationEpoch === initial.mutationEpoch + 1, + 'open fence epoch differs', + ); + assert( + current.transitionRevision === initial.transitionRevision + 1, + 'open fence revision differs', + ); + const locked = await probe('POST', '/admin/execution-fence', { + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: current.mutationEpoch, + expectedRevision: current.transitionRevision, + advanceMutationEpoch: true, + }); + exactFence(locked, nextFence(current, 'migration-locked')); + } else { + exactFence(current, initialFence); + assert(!outcome, 'successful proof lost its open fence'); + } + } catch (error) { + failure = new AggregateError( + failure ? [failure, error] : [error], + 'staging fence relock is unconfirmed', + ); + } + } + try { + await requestOffDeploy(); + } catch (error) { + failure = new AggregateError( + failure ? [failure, error] : [error], + 'staging off deploy is unconfirmed', + ); + } + } + if (failure) throw failure; + return outcome; +} diff --git a/packages/flowsafe/deploy/notification-staging-control.test.mjs b/packages/flowsafe/deploy/notification-staging-control.test.mjs new file mode 100644 index 00000000..dae426c5 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-control.test.mjs @@ -0,0 +1,227 @@ +import assert from 'node:assert/strict'; +import { chmodSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { DatabaseSync } from 'node:sqlite'; +import { test } from 'node:test'; +import { + localProbe, + readSecretFile, + runNotificationStagingControl, + trustedD1Checks, +} from './notification-staging-control.mjs'; + +function fixture(failAt, initialState = 'migration-locked') { + const calls = []; + let fence = { + state: initialState, + mutationEpoch: 0, + requireMutationEpoch: false, + transitionRevision: 0, + }; + const probe = async (method, path, body) => { + calls.push({ method, path, body }); + if (method === 'GET' && path === '/admin/execution-fence') + return { ...fence }; + if (method === 'GET' && path === '/admin/maintenance-status') { + return { alarmAt: 1, nextRemoteStepRecoveryAt: 1 }; + } + if (path === '/admin/execution-fence') { + assert.equal(body.expected, fence.state); + assert.equal(body.expectedMutationEpoch, fence.mutationEpoch); + assert.equal(body.expectedRevision, fence.transitionRevision); + fence = { + state: body.next, + mutationEpoch: fence.mutationEpoch + 1, + requireMutationEpoch: true, + transitionRevision: fence.transitionRevision + 1, + }; + if (failAt === 'unlock-response' && body.next === 'open') + throw new Error('response lost after accepted unlock'); + return { ...fence }; + } + if (path === '/start') { + if (failAt === 'start') throw new Error('start failed'); + return { status: 'suspended', invocationId: 'invocation-1' }; + } + if (path === '/complete') { + if (failAt === 'complete') throw new Error('complete failed'); + return { status: 'success', invocationId: body.invocationId }; + } + throw new Error('unexpected request'); + }; + let offRequests = 0; + const options = { + probe, + preflightD1: async () => { + if (failAt === 'preflight') throw new Error('preflight failed'); + }, + verifyD1: async () => { + if (failAt === 'verify') throw new Error('verification failed'); + }, + requestOffDeploy: async () => { + offRequests++; + }, + }; + return { + options, + calls, + fence: () => fence, + offRequests: () => offRequests, + }; +} + +test('preflight mismatch and read-only failure never unlock', async () => { + for (const scenario of [fixture(null, 'open'), fixture('preflight')]) { + await assert.rejects(runNotificationStagingControl(scenario.options)); + assert.equal( + scenario.calls.filter((call) => call.method === 'POST').length, + 0, + ); + assert.equal(scenario.offRequests(), 1); + } +}); + +test('each post-unlock failure requests a fresh CAS relock and off deploy', async () => { + for (const step of ['unlock-response', 'start', 'complete', 'verify']) { + const scenario = fixture(step); + await assert.rejects(runNotificationStagingControl(scenario.options)); + assert.deepEqual(scenario.fence(), { + state: 'migration-locked', + mutationEpoch: 2, + requireMutationEpoch: true, + transitionRevision: 2, + }); + assert.equal(scenario.offRequests(), 1); + const writes = scenario.calls.filter( + (call) => + call.method === 'POST' && call.path === '/admin/execution-fence', + ); + assert.deepEqual( + writes.map((call) => call.body.next), + ['open', 'migration-locked'], + ); + assert.deepEqual( + [writes[0].body.expectedRevision, writes[1].body.expectedRevision], + [0, 1], + ); + } +}); + +test('a successful proof verifies before relock and requests off deploy', async () => { + const scenario = fixture(); + assert.deepEqual(await runNotificationStagingControl(scenario.options), { + invocationId: 'invocation-1', + verdict: 'notification_staging_passed', + }); + assert.equal(scenario.fence().state, 'migration-locked'); + assert.equal(scenario.offRequests(), 1); +}); + +test('a conflicting fence is not overwritten while off deploy is still requested', async () => { + const scenario = fixture(); + scenario.options.verifyD1 = async () => { + throw new Error('verification failed'); + }; + const original = scenario.options.probe; + scenario.options.probe = async (method, path, body) => { + if ( + method === 'GET' && + path === '/admin/execution-fence' && + scenario.calls.some((call) => call.path === '/start') + ) { + return { + state: 'draining', + mutationEpoch: 1, + requireMutationEpoch: true, + transitionRevision: 2, + }; + } + return original(method, path, body); + }; + await assert.rejects( + runNotificationStagingControl(scenario.options), + /relock is unconfirmed/, + ); + assert.equal(scenario.fence().state, 'open'); + assert.equal(scenario.offRequests(), 1); +}); + +test('probe reads a private secret file and calls only loopback', async () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-probe-secret-test-')); + try { + const path = join(directory, 'secret'); + const secret = 'a'.repeat(32); + writeFileSync(path, secret, { mode: 0o600 }); + const requests = []; + const probe = localProbe({ + port: 8897, + secret: readSecretFile(path), + fetcher: async (url, init) => { + requests.push({ url, init }); + return Response.json({ state: 'migration-locked' }); + }, + }); + assert.deepEqual(await probe('GET', '/admin/execution-fence'), { + state: 'migration-locked', + }); + assert.equal( + requests[0].url, + 'http://127.0.0.1:8897/admin/execution-fence', + ); + assert.equal(requests[0].init.headers.authorization, `Bearer ${secret}`); + chmodSync(path, 0o644); + assert.throws(() => readSecretFile(path), /private/); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + +test('D1 proof requires the receipt tuple and one new notification and receipt', async () => { + const db = new DatabaseSync(':memory:'); + try { + db.exec(`CREATE TABLE flowsafe_remote_step_invocations ( + invocation_id TEXT PRIMARY KEY, status TEXT, settlement_outcome TEXT, + operation_state TEXT, thread_id TEXT, notification_id TEXT, + operation_key TEXT, notification_request_digest TEXT, + write_generation INTEGER NOT NULL + ); + CREATE TABLE flowsafe_notification_operation_receipts ( + operation_key TEXT PRIMARY KEY, request_digest TEXT, + thread_id TEXT, notification_id TEXT + ); + CREATE TABLE mastra_notifications ( + id TEXT, thread_id TEXT, source TEXT, kind TEXT + );`); + const checks = trustedD1Checks((sql) => db.prepare(sql).all()); + await checks.preflightD1(); + db.exec(`INSERT INTO flowsafe_remote_step_invocations VALUES + ('invocation-1', 'settled', 'applied', 'resume-admitted', + 'thread-1', 'notification-1', 'operation-1', '${'a'.repeat(64)}', 1); + INSERT INTO mastra_notifications VALUES + ('notification-1', 'thread-1', 'flowsafe-staging', 'host-owned-notification'); + INSERT INTO flowsafe_notification_operation_receipts VALUES + ('operation-1', '${'a'.repeat(64)}', 'other-thread', 'notification-1');`); + await assert.rejects(checks.verifyD1('invocation-1'), /unconfirmed/); + db.exec( + "UPDATE flowsafe_notification_operation_receipts SET thread_id = 'thread-1', notification_id = 'other-notification'", + ); + await assert.rejects(checks.verifyD1('invocation-1'), /unconfirmed/); + db.exec( + "UPDATE flowsafe_notification_operation_receipts SET notification_id = 'notification-1'", + ); + await checks.verifyD1('invocation-1'); + db.exec( + `UPDATE flowsafe_notification_operation_receipts SET request_digest = '${'b'.repeat(64)}'`, + ); + await assert.rejects(checks.verifyD1('invocation-1'), /unconfirmed/); + db.exec( + `UPDATE flowsafe_notification_operation_receipts SET request_digest = '${'a'.repeat(64)}'`, + ); + db.exec(`INSERT INTO mastra_notifications VALUES + ('notification-2', 'thread-2', 'flowsafe-staging', 'host-owned-notification');`); + await assert.rejects(checks.verifyD1('invocation-1'), /count differs/); + } finally { + db.close(); + } +}); diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs new file mode 100644 index 00000000..1cdb7ece --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -0,0 +1,391 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { + appendFileSync, + closeSync, + existsSync, + mkdirSync, + openSync, + readFileSync, + statSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + localProbe, + readSecretFile, + runNotificationStagingControl, + trustedD1Checks, +} from './notification-staging-control.mjs'; + +const here = dirname(fileURLToPath(import.meta.url)); +const flowsafe = dirname(here); +const wrangler = join(flowsafe, '../../node_modules/.bin/wrangler'); +const config = join(here, 'notification-staging.wrangler.jsonc'); +const database = 'anchorage-flowsafe-remote-step-staging-db'; +const versionPattern = + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; + +function assert(condition, message) { + if (!condition) throw new Error(message); +} + +function privateDiagnostic(error, secrets = []) { + const parts = []; + const visit = (value) => { + if (!(value instanceof Error)) return; + parts.push(value.message); + if (value instanceof AggregateError) value.errors.forEach(visit); + if (value.cause) visit(value.cause); + }; + visit(error); + return parts + .join(' | ') + .replaceAll(secrets[0] ?? '\u0000', '[redacted]') + .replaceAll(secrets[1] ?? '\u0000', '[redacted]') + .slice(0, 4_096); +} + +function privateFile(path) { + assert( + typeof path === 'string' && path.startsWith('/'), + 'absolute private path required', + ); + const info = statSync(path); + assert( + info.isFile() && (info.mode & 0o077) === 0, + 'private file permissions required', + ); + return readFileSync(path, 'utf8').trim(); +} + +export function createWranglerControl({ command = spawnSync, token }) { + function run(args, timeout = 30_000) { + const result = command(wrangler, args, { + cwd: flowsafe, + encoding: 'utf8', + timeout, + env: { + ...process.env, + CLOUDFLARE_API_TOKEN: token, + NO_COLOR: '1', + WRANGLER_SEND_METRICS: 'false', + }, + }); + assert(result.status === 0, `Wrangler ${args[0]} ${args[1]} failed`); + return result.stdout; + } + function json(args) { + const value = JSON.parse(run(args)); + assert(value !== null, 'Wrangler returned null'); + return value; + } + function currentVersion() { + const deployments = json([ + 'deployments', + 'list', + '--config', + config, + '--json', + ]); + assert( + Array.isArray(deployments) && deployments.length > 0, + 'deployment list is empty', + ); + const ordered = deployments + .map((deployment) => { + const createdAt = Date.parse(deployment.created_on); + assert( + Number.isFinite(createdAt), + 'deployment timestamp is unreadable', + ); + return { deployment, createdAt }; + }) + .sort((a, b) => b.createdAt - a.createdAt); + assert( + ordered.length === 1 || ordered[0].createdAt > ordered[1].createdAt, + 'latest deployment is ambiguous', + ); + const latest = ordered[0].deployment; + assert( + latest.versions?.length === 1 && latest.versions[0].percentage === 100, + 'deployment is not a single full version', + ); + const id = latest.versions[0].version_id; + assert(versionPattern.test(id), 'invalid deployment version'); + return id; + } + function proofEnabled(version) { + assert(versionPattern.test(version), 'invalid version'); + const detail = json([ + 'versions', + 'view', + version, + '--config', + config, + '--json', + ]); + const bindings = detail.resources?.bindings; + assert(Array.isArray(bindings), 'version bindings are unreadable'); + const flag = bindings.filter( + (binding) => binding.name === 'NOTIFICATION_PROOF_ENABLED', + ); + assert( + flag.length === 1 && flag[0].type === 'plain_text', + 'proof flag binding differs', + ); + return flag[0].text === 'true'; + } + return { + currentVersion, + proofEnabled, + query(sql) { + const output = json([ + 'd1', + 'execute', + database, + '--remote', + '--yes', + '--json', + '--config', + config, + '--command', + sql, + ]); + assert( + Array.isArray(output) && + output.length === 1 && + output[0].success === true && + Array.isArray(output[0].results), + 'D1 read failed', + ); + return output[0].results; + }, + deployOffIfCurrent(expectedVersion, outdir) { + assert(versionPattern.test(expectedVersion), 'invalid expected version'); + assert( + currentVersion() === expectedVersion, + 'enabled deployment changed', + ); + assert( + proofEnabled(expectedVersion), + 'expected deployment is not enabled', + ); + const tracked = JSON.parse(readFileSync(config, 'utf8')); + assert( + tracked.vars?.NOTIFICATION_PROOF_ENABLED === 'false' && + tracked.workers_dev === false, + 'tracked off config differs', + ); + run(['deploy', '--config', config, '--outdir', outdir], 180_000); + const next = currentVersion(); + assert( + next !== expectedVersion && proofEnabled(next) === false, + 'off deploy is unconfirmed', + ); + return next; + }, + }; +} + +export async function relockIfOpen(probe, expected) { + assert( + expected?.mutationEpoch === 1 && expected?.transitionRevision === 1, + 'watchdog owned fence version is missing', + ); + const current = await probe('GET', '/admin/execution-fence'); + if (current.state === 'migration-locked') return current; + assert(current.state === 'open', 'watchdog found an unexpected fence state'); + assert( + current.requireMutationEpoch === true && + current.mutationEpoch === expected.mutationEpoch && + current.transitionRevision === expected.transitionRevision, + 'watchdog fence metadata differs', + ); + const locked = await probe('POST', '/admin/execution-fence', { + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: current.mutationEpoch, + expectedRevision: current.transitionRevision, + advanceMutationEpoch: true, + }); + assert( + locked.state === 'migration-locked' && + locked.mutationEpoch === current.mutationEpoch + 1 && + locked.transitionRevision === current.transitionRevision + 1, + 'watchdog relock is unconfirmed', + ); + return locked; +} + +function settingsFromEnv(env) { + const port = Number(env.FLOWSAFE_STAGING_PROBE_PORT); + const expectedVersion = env.FLOWSAFE_STAGING_ENABLED_VERSION; + assert( + Number.isInteger(port) && port > 0 && port < 65536, + 'invalid local probe port', + ); + assert( + versionPattern.test(expectedVersion), + 'expected enabled version required', + ); + const probeSecretPath = env.FLOWSAFE_STAGING_PROBE_SECRET_FILE; + const cfTokenPath = env.FLOWSAFE_CF_TOKEN_FILE; + const secret = readSecretFile(probeSecretPath); + const token = privateFile(cfTokenPath); + assert(token.length > 0, 'Cloudflare token is empty'); + return { port, expectedVersion, probeSecretPath, cfTokenPath, secret, token }; +} + +async function watchdog(stateFile) { + const state = JSON.parse(privateFile(stateFile)); + const { + port, + expectedVersion, + probeSecretPath, + cfTokenPath, + parentPid, + deadlineAt, + outdir, + } = state; + const probe = localProbe({ port, secret: readSecretFile(probeSecretPath) }); + const cli = createWranglerControl({ token: privateFile(cfTokenPath) }); + assert( + cli.currentVersion() === expectedVersion && + cli.proofEnabled(expectedVersion), + 'watchdog enabled version differs', + ); + writeFileSync(`${stateFile}.ready`, '', { mode: 0o600, flag: 'wx' }); + while (Date.now() < deadlineAt && !existsSync(`${stateFile}.done`)) { + try { + process.kill(parentPid, 0); + } catch { + break; + } + await new Promise((resolve) => setTimeout(resolve, 1_000)); + } + if (existsSync(`${stateFile}.done`)) return; + let failure; + if (existsSync(`${stateFile}.unlock-attempted`)) { + try { + const owned = JSON.parse(privateFile(`${stateFile}.unlock-attempted`)); + await relockIfOpen(probe, owned); + } catch (error) { + failure = error; + } + } + try { + if (cli.currentVersion() === expectedVersion) + cli.deployOffIfCurrent(expectedVersion, outdir); + } catch (error) { + failure = new AggregateError( + failure ? [failure, error] : [error], + 'watchdog off deploy is unconfirmed', + ); + } + if (failure) throw failure; +} + +async function controller() { + const settings = settingsFromEnv(process.env); + const cli = createWranglerControl({ token: settings.token }); + assert( + cli.currentVersion() === settings.expectedVersion && + cli.proofEnabled(settings.expectedVersion), + 'enabled version preflight differs', + ); + const directory = join( + tmpdir(), + `flowsafe-notification-control-${process.pid}`, + ); + mkdirSync(directory, { mode: 0o700 }); + const stateFile = join(directory, 'watchdog.json'); + const outdir = join(directory, 'off-deploy'); + const watcherLog = openSync(join(directory, 'watchdog.log'), 'wx', 0o600); + writeFileSync( + stateFile, + JSON.stringify({ + port: settings.port, + expectedVersion: settings.expectedVersion, + probeSecretPath: settings.probeSecretPath, + cfTokenPath: settings.cfTokenPath, + parentPid: process.pid, + deadlineAt: Date.now() + 900_000, + outdir, + }), + { mode: 0o600, flag: 'wx' }, + ); + const watcher = spawn( + process.execPath, + [fileURLToPath(import.meta.url), '--watchdog', stateFile], + { + detached: true, + stdio: ['ignore', watcherLog, watcherLog], + env: process.env, + }, + ); + closeSync(watcherLog); + watcher.unref(); + assert(Number.isInteger(watcher.pid), 'watchdog did not start'); + const readyDeadline = Date.now() + 20_000; + while (!existsSync(`${stateFile}.ready`) && Date.now() < readyDeadline) { + await new Promise((resolve) => setTimeout(resolve, 100)); + } + assert(existsSync(`${stateFile}.ready`), 'watchdog readiness is unconfirmed'); + const probe = localProbe({ port: settings.port, secret: settings.secret }); + let offConfirmed = false; + const requestOffDeploy = async () => { + if (offConfirmed) return; + cli.deployOffIfCurrent(settings.expectedVersion, outdir); + offConfirmed = true; + }; + try { + const result = await runNotificationStagingControl({ + probe, + ...trustedD1Checks((sql) => cli.query(sql)), + requestOffDeploy, + onBeforeUnlock: async () => { + writeFileSync( + `${stateFile}.unlock-attempted`, + JSON.stringify({ + mutationEpoch: 1, + transitionRevision: 1, + }), + { + mode: 0o600, + flag: 'wx', + }, + ); + }, + }); + assert(offConfirmed, 'off deploy was not confirmed'); + writeFileSync(`${stateFile}.done`, '', { mode: 0o600, flag: 'wx' }); + return result; + } catch (error) { + appendFileSync( + join(directory, 'watchdog.log'), + `controller: ${privateDiagnostic(error, [settings.secret, settings.token])}\n`, + ); + throw new Error('staging controller failed; inspect private watchdog log'); + } +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const action = process.argv[2]; + const execute = + action === '--watchdog' + ? watchdog(process.argv[3]) + : action === '--run' && process.argv.length === 3 + ? controller() + : Promise.reject(new Error('usage: --run')); + execute.then( + (result) => { + if (result) process.stdout.write(`${JSON.stringify(result)}\n`); + }, + (error) => { + process.stderr.write(`${privateDiagnostic(error)}\n`); + process.exitCode = 1; + }, + ); +} diff --git a/packages/flowsafe/deploy/notification-staging-live.test.mjs b/packages/flowsafe/deploy/notification-staging-live.test.mjs new file mode 100644 index 00000000..1043f246 --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-live.test.mjs @@ -0,0 +1,134 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { + createWranglerControl, + relockIfOpen, +} from './notification-staging-live.mjs'; + +const enabled = 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa'; +const disabled = 'bbbbbbbb-bbbb-4bbb-bbbb-bbbbbbbbbbbb'; + +function fakeCli({ enabledFlag = true } = {}) { + const calls = []; + let version = enabled; + const command = (_executable, args) => { + calls.push(args); + if (args[0] === 'deployments') { + return { + status: 0, + stdout: JSON.stringify([ + { + created_on: '2026-10-15T00:00:00.000Z', + versions: [{ version_id: version, percentage: 100 }], + }, + ]), + }; + } + if (args[0] === 'versions') { + return { + status: 0, + stdout: JSON.stringify({ + resources: { + bindings: [ + { + name: 'NOTIFICATION_PROOF_ENABLED', + type: 'plain_text', + text: version === enabled && enabledFlag ? 'true' : 'false', + }, + ], + }, + }), + }; + } + if (args[0] === 'deploy') { + version = disabled; + return { status: 0, stdout: 'deployed' }; + } + if (args[0] === 'd1') { + return { + status: 0, + stdout: JSON.stringify([{ success: true, results: [{ count: 0 }] }]), + }; + } + throw new Error('unexpected CLI'); + }; + const cli = createWranglerControl({ command, token: 'fake-token' }); + return { + cli, + calls, + change() { + version = disabled; + }, + }; +} + +test('version-checked off deploy uses the tracked false config', () => { + const fixture = fakeCli(); + assert.equal(fixture.cli.currentVersion(), enabled); + assert.equal(fixture.cli.proofEnabled(enabled), true); + assert.equal( + fixture.cli.deployOffIfCurrent(enabled, '/tmp/staging-off-test'), + disabled, + ); + assert.equal(fixture.cli.proofEnabled(disabled), false); + assert.equal(fixture.calls.filter((args) => args[0] === 'deploy').length, 1); +}); + +test('changed version or disabled flag refuses before deploy', () => { + const changed = fakeCli(); + changed.change(); + assert.throws( + () => changed.cli.deployOffIfCurrent(enabled, '/tmp/staging-off-test'), + /changed/, + ); + assert.equal(changed.calls.filter((args) => args[0] === 'deploy').length, 0); + const disabledFlag = fakeCli({ enabledFlag: false }); + assert.throws( + () => disabledFlag.cli.deployOffIfCurrent(enabled, '/tmp/staging-off-test'), + /not enabled/, + ); + assert.equal( + disabledFlag.calls.filter((args) => args[0] === 'deploy').length, + 0, + ); +}); + +test('watchdog relock uses a fresh epoch and revision', async () => { + const calls = []; + let observedRevision = 1; + const probe = async (method, path, body) => { + calls.push({ method, path, body }); + if (method === 'GET') + return { + state: 'open', + mutationEpoch: 1, + transitionRevision: observedRevision, + requireMutationEpoch: true, + }; + return { + state: 'migration-locked', + mutationEpoch: 2, + transitionRevision: 2, + requireMutationEpoch: true, + }; + }; + assert.equal( + (await relockIfOpen(probe, { mutationEpoch: 1, transitionRevision: 1 })) + .state, + 'migration-locked', + ); + assert.deepEqual(calls[1].body, { + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: 1, + expectedRevision: 1, + advanceMutationEpoch: true, + }); + calls.length = 0; + observedRevision = 2; + await assert.rejects( + relockIfOpen(probe, { mutationEpoch: 1, transitionRevision: 1 }), + /metadata differs/, + ); + assert.equal(calls.length, 1); +}); From b76f7ef2708c4b0a02c3b90944bf2ed268eb9981 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 02:45:19 +0400 Subject: [PATCH 56/73] Retain private Wrangler failure diagnostics Keep bounded exit, signal, process error and stderr detail in the controller cause so a failed staging rollback can be diagnosed without printing credentials. --- .../deploy/notification-staging-live.mjs | 14 +++++++++++++- .../deploy/notification-staging-live.test.mjs | 16 ++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs index 1cdb7ece..2b790122 100644 --- a/packages/flowsafe/deploy/notification-staging-live.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -73,7 +73,19 @@ export function createWranglerControl({ command = spawnSync, token }) { WRANGLER_SEND_METRICS: 'false', }, }); - assert(result.status === 0, `Wrangler ${args[0]} ${args[1]} failed`); + if (result.status !== 0) { + const details = [ + `exit=${result.status ?? 'none'}`, + `signal=${result.signal ?? 'none'}`, + result.error?.message, + result.stderr?.slice(-2_048), + ] + .filter(Boolean) + .join(' | '); + throw new Error(`Wrangler ${args[0]} ${args[1]} failed`, { + cause: new Error(details), + }); + } return result.stdout; } function json(args) { diff --git a/packages/flowsafe/deploy/notification-staging-live.test.mjs b/packages/flowsafe/deploy/notification-staging-live.test.mjs index 1043f246..89e411ff 100644 --- a/packages/flowsafe/deploy/notification-staging-live.test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.test.mjs @@ -93,6 +93,22 @@ test('changed version or disabled flag refuses before deploy', () => { ); }); +test('Wrangler failure preserves exit and stderr in its diagnostic cause', () => { + const cli = createWranglerControl({ + token: 'fake-token', + command: () => ({ status: 1, signal: null, stderr: 'deployment rejected' }), + }); + assert.throws( + () => cli.currentVersion(), + (error) => { + assert.match(error.message, /Wrangler deployments list failed/); + assert.match(error.cause?.message, /exit=1/); + assert.match(error.cause?.message, /deployment rejected/); + return true; + }, + ); +}); + test('watchdog relock uses a fresh epoch and revision', async () => { const calls = []; let observedRevision = 1; From 6131bb6b81133e419c38bc743ad038da75af73b4 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 02:48:38 +0400 Subject: [PATCH 57/73] Exercise staging watchdog after controller death Run the local controller and detached watchdog with a loopback probe and fake Wrangler, then kill the controller and verify owned fence relock and off deployment. --- .../deploy/notification-staging-live.test.mjs | 187 ++++++++++++++++++ 1 file changed, 187 insertions(+) diff --git a/packages/flowsafe/deploy/notification-staging-live.test.mjs b/packages/flowsafe/deploy/notification-staging-live.test.mjs index 89e411ff..1226f395 100644 --- a/packages/flowsafe/deploy/notification-staging-live.test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.test.mjs @@ -1,5 +1,17 @@ import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; +import { + existsSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { createServer } from 'node:http'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; import { test } from 'node:test'; +import { fileURLToPath } from 'node:url'; import { createWranglerControl, relockIfOpen, @@ -148,3 +160,178 @@ test('watchdog relock uses a fresh epoch and revision', async () => { ); assert.equal(calls.length, 1); }); + +test('controller waits for watchdog readiness and watchdog recovers after parent death', async () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-live-watchdog-test-')); + const secret = 's'.repeat(32); + const tokenFile = join(directory, 'cf-token'); + const secretFile = join(directory, 'probe-secret'); + const preloadFile = join(directory, 'fake-wrangler.mjs'); + const offFile = join(directory, 'off-deploy'); + const offConfirmedFile = join(directory, 'off-confirmed'); + const watcherPidFile = join(directory, 'watchdog-pid'); + writeFileSync(tokenFile, 'fake-token', { mode: 0o600 }); + writeFileSync(secretFile, secret, { mode: 0o600 }); + writeFileSync( + preloadFile, + `import childProcess from 'node:child_process'; +import { syncBuiltinESMExports } from 'node:module'; +import { existsSync, writeFileSync } from 'node:fs'; +const originalSpawn = childProcess.spawn; +childProcess.spawn = (...args) => { + const child = originalSpawn(...args); + if (args[1]?.includes('--watchdog')) + writeFileSync(process.env.TEST_WATCHDOG_PID_FILE, String(child.pid)); + return child; +}; +childProcess.spawnSync = (executable, args) => { + if (!executable.endsWith('/node_modules/.bin/wrangler')) + throw new Error('unexpected synchronous command'); + const off = existsSync(process.env.TEST_OFF_FILE); + const command = args[0]; + if (off && command === 'deployments') + writeFileSync(process.env.TEST_OFF_CONFIRMED_FILE, 'checked'); + const response = command === 'deployments' ? [{ + created_on: '2026-10-15T00:00:00.000Z', + versions: [{ version_id: off ? '${disabled}' : '${enabled}', percentage: 100 }], + }] : command === 'versions' ? { + resources: { bindings: [{ + name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', + text: off ? 'false' : 'true', + }] }, + } : command === 'd1' ? [{ success: true, results: + args.at(-1).includes('sqlite_master') + ? [{ sql: 'write_generation INTEGER NOT NULL, operation_state TEXT' }] + : args.at(-1).includes('COUNT(*) AS count') + ? [{ count: 0 }] + : [{ notifications: 0, receipts: 0 }], + }] : null; + if (command === 'deploy') { + writeFileSync(process.env.TEST_OFF_FILE, 'deployed'); + return { status: 0, stdout: 'deployed' }; + } + if (response === null) throw new Error('unexpected Wrangler command'); + return { status: 0, stdout: JSON.stringify(response) }; +}; +syncBuiltinESMExports(); +`, + ); + + const requests = []; + let fence = { + state: 'migration-locked', + mutationEpoch: 0, + requireMutationEpoch: false, + transitionRevision: 0, + }; + let startReceived = false; + const server = createServer(async (request, response) => { + assert.equal(request.headers.authorization, `Bearer ${secret}`); + const path = request.url; + const method = request.method; + requests.push(`${method} ${path}`); + if (method === 'POST' && path === '/start') { + startReceived = true; + return; + } + let value; + if (path === '/admin/maintenance-status') { + value = { alarmAt: 1, nextRemoteStepRecoveryAt: 1 }; + } else if (path === '/admin/execution-fence' && method === 'POST') { + const chunks = []; + for await (const chunk of request) chunks.push(chunk); + const body = JSON.parse(Buffer.concat(chunks).toString('utf8')); + assert.equal(body.expected, fence.state); + assert.equal(body.expectedMutationEpoch, fence.mutationEpoch); + assert.equal(body.expectedRevision, fence.transitionRevision); + assert.equal(body.advanceMutationEpoch, true); + fence = { + state: body.next, + mutationEpoch: fence.mutationEpoch + 1, + requireMutationEpoch: true, + transitionRevision: fence.transitionRevision + 1, + }; + value = fence; + } else { + assert.equal(path, '/admin/execution-fence'); + value = fence; + } + response.writeHead(200, { 'content-type': 'application/json' }); + response.end(JSON.stringify(value)); + }); + let controller; + let watcherPid; + try { + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const port = server.address().port; + controller = spawn( + process.execPath, + [ + fileURLToPath( + new URL('./notification-staging-live.mjs', import.meta.url), + ), + '--run', + ], + { + env: { + ...process.env, + NODE_OPTIONS: `--import=${preloadFile}`, + TMPDIR: directory, + FLOWSAFE_STAGING_PROBE_PORT: String(port), + FLOWSAFE_STAGING_ENABLED_VERSION: enabled, + FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, + FLOWSAFE_CF_TOKEN_FILE: tokenFile, + TEST_OFF_FILE: offFile, + TEST_OFF_CONFIRMED_FILE: offConfirmedFile, + TEST_WATCHDOG_PID_FILE: watcherPidFile, + }, + stdio: 'ignore', + }, + ); + const stateFile = join( + directory, + `flowsafe-notification-control-${controller.pid}`, + 'watchdog.json', + ); + const deadline = Date.now() + 8_000; + const waitFor = async (condition) => { + while (!condition()) { + assert(Date.now() < deadline, 'watchdog process timed out'); + await new Promise((resolve) => setTimeout(resolve, 20)); + } + }; + await waitFor(() => existsSync(`${stateFile}.ready`)); + await waitFor(() => existsSync(watcherPidFile)); + watcherPid = Number(readFileSync(watcherPidFile, 'utf8')); + await waitFor(() => startReceived); + assert(existsSync(`${stateFile}.unlock-attempted`)); + assert.equal(fence.state, 'open'); + assert.equal(existsSync(offFile), false); + controller.kill('SIGKILL'); + await waitFor(() => existsSync(offConfirmedFile)); + assert.equal(readFileSync(offFile, 'utf8'), 'deployed'); + assert.deepEqual(fence, { + state: 'migration-locked', + mutationEpoch: 2, + requireMutationEpoch: true, + transitionRevision: 2, + }); + assert.deepEqual( + requests.filter((request) => request === 'POST /admin/execution-fence'), + ['POST /admin/execution-fence', 'POST /admin/execution-fence'], + ); + assert.equal(existsSync(`${stateFile}.done`), false); + } finally { + controller?.kill('SIGKILL'); + if (Number.isInteger(watcherPid)) { + try { + process.kill(watcherPid, 'SIGKILL'); + } catch { + // The watchdog may have exited after the off deployment. + } + } + server.closeAllConnections(); + await new Promise((resolve) => server.close(resolve)); + rmSync(directory, { recursive: true, force: true }); + } +}); From eba22e1f5a5a3e35a4158fa84a60565883f6b7bf Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 03:00:01 +0400 Subject: [PATCH 58/73] Hold staging deploy lock through watchdog recovery Share an exclusive Worker lock across controller and watchdog. Seal the fence against delayed unlock, converge lost relock responses, and let the watchdog alone confirm the off deployment. --- packages/flowsafe/deploy/README.md | 23 +- .../deploy/notification-staging-live.mjs | 190 +++++++++----- .../deploy/notification-staging-live.test.mjs | 234 +++++++++++++++--- 3 files changed, 346 insertions(+), 101 deletions(-) diff --git a/packages/flowsafe/deploy/README.md b/packages/flowsafe/deploy/README.md index 499dc0e4..3c6658bf 100644 --- a/packages/flowsafe/deploy/README.md +++ b/packages/flowsafe/deploy/README.md @@ -218,9 +218,25 @@ See the repository [deployment reference](../../../docs/deployment-reference.md) ## Run the isolated notification staging control -Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The script refuses to unlock unless the fence is `migration-locked` at epoch and revision zero, the v5 journal is empty, and the recovery alarm is armed. Its detached watchdog relocks an open fence after controller death or its 15-minute deadline. Both processes request the tracked off deployment only while the current Worker version matches the enabled version supplied to the controller. +Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The script refuses to unlock unless the fence is `migration-locked` at epoch and revision zero, the v5 journal is empty, and the recovery alarm is armed. It acquires a Linux `flock` keyed by Cloudflare account and Worker before reading the enabled version. Its detached watchdog inherits the same locked open file descriptor, seals or relocks the fence after controller death or its 15-minute deadline, and performs the tracked off deployment on controller request or recovery. Sealing advances the initially locked fence's revision, so a delayed controller unlock cannot succeed after watchdog shutdown starts. The controller never deploys off itself. -The Wrangler version read and deploy are separate operations. Hold the owning deployment lock across the proof and off deployment, including the watchdog's recovery window. Do not run this control while another actor can deploy the subject Worker; the script cannot enforce that lock remotely. +Use the same lock for **every** command that can upload, deploy, promote, or change traffic for `anchorage-flowsafe-remote-step-staging`. From `packages/flowsafe`, define the wrapper below and use it for the initial enabled deployment and any other same-Worker deployment command: + +```bash +export FLOWSAFE_CF_ACCOUNT_ID=replace-with-32-lowercase-hex-account-id +flowsafe_locked_deploy() { + [[ "$FLOWSAFE_CF_ACCOUNT_ID" =~ ^[0-9a-f]{32}$ ]] || return 2 + local lock_dir="/tmp/flowsafe-deploy-locks-$(id -u)" + install -d -m 700 "$lock_dir" || return + (umask 077; flock -n -E 75 "$lock_dir/${FLOWSAFE_CF_ACCOUNT_ID}-anchorage-flowsafe-remote-step-staging.lock" "$@") +} +flowsafe_locked_deploy pnpm exec wrangler deploy \ + --config deploy/notification-staging.wrangler.jsonc \ + --var NOTIFICATION_PROOF_ENABLED:true \ + --outdir /tmp/flowsafe-notification-enable +``` + +The wrapper exits `75` if another local deployment owns the lock. Run the control on the same host and Unix user as the wrapper; the lock directory is fixed under `/tmp`, regardless of `TMPDIR`. Set `FLOWSAFE_STAGING_ENABLED_VERSION` from the exact enabled deployment ID before starting the control. The lock serializes cooperating local processes only. A deployment from another host, the Cloudflare dashboard, or an actor that skips this wrapper can still race the version check and deploy. Confirm exclusive deployment control separately before treating the live proof as safe. Set these environment variables before running `node deploy/notification-staging-live.mjs --run` from `packages/flowsafe`: @@ -229,6 +245,7 @@ Set these environment variables before running `node deploy/notification-staging | `FLOWSAFE_STAGING_PROBE_PORT` | Port of the local Service-binding probe | | `FLOWSAFE_STAGING_PROBE_SECRET_FILE` | Absolute path to the probe credential in a `0600` file | | `FLOWSAFE_CF_TOKEN_FILE` | Absolute path to the Cloudflare API token in a `0600` file | +| `FLOWSAFE_CF_ACCOUNT_ID` | Cloudflare account ID as 32 lowercase hexadecimal characters; the same value used by the deployment wrapper | | `FLOWSAFE_STAGING_ENABLED_VERSION` | Exact 100% enabled subject Worker version ID | -The controller prints the invocation ID only after D1 confirms one settled invocation, its notification, and its operation receipt. It writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the controller exits without a confirmed off deployment, inspect that private directory's `watchdog.log` and verify the fence and current Worker version through the authenticated proxy and Wrangler before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. +The controller prints the invocation ID only after D1 confirms one settled invocation, its notification, its operation receipt, and the watchdog's confirmed off deployment. It writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the controller exits without a confirmed off deployment, inspect that private directory's `watchdog.log` and verify the fence and current Worker version through the authenticated proxy and Wrangler before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs index 2b790122..bca938d9 100644 --- a/packages/flowsafe/deploy/notification-staging-live.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -2,7 +2,9 @@ import { spawn, spawnSync } from 'node:child_process'; import { appendFileSync, closeSync, + constants, existsSync, + fstatSync, mkdirSync, openSync, readFileSync, @@ -26,6 +28,9 @@ const config = join(here, 'notification-staging.wrangler.jsonc'); const database = 'anchorage-flowsafe-remote-step-staging-db'; const versionPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; +const accountPattern = /^[0-9a-f]{32}$/; +const workerName = JSON.parse(readFileSync(config, 'utf8')).name; +assert(/^[a-z0-9-]+$/.test(workerName), 'invalid staging Worker name'); function assert(condition, message) { if (!condition) throw new Error(message); @@ -201,34 +206,43 @@ export function createWranglerControl({ command = spawnSync, token }) { }; } -export async function relockIfOpen(probe, expected) { - assert( - expected?.mutationEpoch === 1 && expected?.transitionRevision === 1, - 'watchdog owned fence version is missing', - ); +export async function sealFence(probe) { + for (let attempt = 0; attempt < 3; attempt++) { + const current = await probe('GET', '/admin/execution-fence'); + assert( + current.state === 'migration-locked' || current.state === 'open', + 'watchdog found an unexpected fence state', + ); + const initial = + current.state === 'migration-locked' && + current.mutationEpoch === 0 && + current.transitionRevision === 0 && + current.requireMutationEpoch === false; + const open = + current.state === 'open' && + current.mutationEpoch === 1 && + current.transitionRevision === 1 && + current.requireMutationEpoch === true; + if (current.state === 'migration-locked' && !initial) { + confirmedFence(current); + return current; + } + assert(initial || open, 'watchdog fence metadata differs'); + try { + await probe('POST', '/admin/execution-fence', { + expected: current.state, + next: 'migration-locked', + expectedMutationEpoch: current.mutationEpoch, + expectedRevision: current.transitionRevision, + advanceMutationEpoch: true, + }); + } catch { + // The authoritative read resolves a lost response or a competing open CAS. + } + } const current = await probe('GET', '/admin/execution-fence'); - if (current.state === 'migration-locked') return current; - assert(current.state === 'open', 'watchdog found an unexpected fence state'); - assert( - current.requireMutationEpoch === true && - current.mutationEpoch === expected.mutationEpoch && - current.transitionRevision === expected.transitionRevision, - 'watchdog fence metadata differs', - ); - const locked = await probe('POST', '/admin/execution-fence', { - expected: 'open', - next: 'migration-locked', - expectedMutationEpoch: current.mutationEpoch, - expectedRevision: current.transitionRevision, - advanceMutationEpoch: true, - }); - assert( - locked.state === 'migration-locked' && - locked.mutationEpoch === current.mutationEpoch + 1 && - locked.transitionRevision === current.transitionRevision + 1, - 'watchdog relock is unconfirmed', - ); - return locked; + confirmedFence(current); + return current; } function settingsFromEnv(env) { @@ -247,10 +261,66 @@ function settingsFromEnv(env) { const secret = readSecretFile(probeSecretPath); const token = privateFile(cfTokenPath); assert(token.length > 0, 'Cloudflare token is empty'); - return { port, expectedVersion, probeSecretPath, cfTokenPath, secret, token }; + const accountId = env.FLOWSAFE_CF_ACCOUNT_ID; + assert(accountPattern.test(accountId), 'Cloudflare account ID is required'); + return { + port, + expectedVersion, + probeSecretPath, + cfTokenPath, + secret, + token, + accountId, + }; +} + +function acquireDeploymentLock(accountId) { + const directory = join('/tmp', `flowsafe-deploy-locks-${process.getuid()}`); + mkdirSync(directory, { mode: 0o700, recursive: true }); + const directoryInfo = statSync(directory); + assert( + directoryInfo.isDirectory() && + directoryInfo.uid === process.getuid() && + (directoryInfo.mode & 0o077) === 0, + 'deployment lock directory is not private', + ); + const path = join(directory, `${accountId}-${workerName}.lock`); + const fd = openSync( + path, + constants.O_CREAT | constants.O_RDWR | constants.O_NOFOLLOW, + 0o600, + ); + const info = fstatSync(fd); + assert( + info.isFile() && info.uid === process.getuid() && (info.mode & 0o077) === 0, + 'deployment lock file is not private', + ); + const result = spawnSync('flock', ['--nonblock', '--exclusive', '3'], { + stdio: ['ignore', 'pipe', 'pipe', fd], + }); + assert(result.status === 0, 'deployment lock is unavailable'); + return fd; +} + +function confirmedFence(current) { + assert( + current.state === 'migration-locked', + 'staging fence relock is unconfirmed', + ); + const sealed = + current.mutationEpoch === 1 && + current.transitionRevision === 1 && + current.requireMutationEpoch === true; + const relocked = + current.mutationEpoch === 2 && + current.transitionRevision === 2 && + current.requireMutationEpoch === true; + assert(sealed || relocked, 'staging fence metadata differs'); } async function watchdog(stateFile) { + const lockFd = 3; + assert(fstatSync(lockFd).isFile(), 'inherited deployment lock is missing'); const state = JSON.parse(privateFile(stateFile)); const { port, @@ -269,7 +339,7 @@ async function watchdog(stateFile) { 'watchdog enabled version differs', ); writeFileSync(`${stateFile}.ready`, '', { mode: 0o600, flag: 'wx' }); - while (Date.now() < deadlineAt && !existsSync(`${stateFile}.done`)) { + while (Date.now() < deadlineAt && !existsSync(`${stateFile}.off-requested`)) { try { process.kill(parentPid, 0); } catch { @@ -277,30 +347,15 @@ async function watchdog(stateFile) { } await new Promise((resolve) => setTimeout(resolve, 1_000)); } - if (existsSync(`${stateFile}.done`)) return; - let failure; - if (existsSync(`${stateFile}.unlock-attempted`)) { - try { - const owned = JSON.parse(privateFile(`${stateFile}.unlock-attempted`)); - await relockIfOpen(probe, owned); - } catch (error) { - failure = error; - } - } - try { - if (cli.currentVersion() === expectedVersion) - cli.deployOffIfCurrent(expectedVersion, outdir); - } catch (error) { - failure = new AggregateError( - failure ? [failure, error] : [error], - 'watchdog off deploy is unconfirmed', - ); - } - if (failure) throw failure; + await sealFence(probe); + cli.deployOffIfCurrent(expectedVersion, outdir); + confirmedFence(await probe('GET', '/admin/execution-fence')); + writeFileSync(`${stateFile}.off-confirmed`, '', { mode: 0o600, flag: 'wx' }); } async function controller() { const settings = settingsFromEnv(process.env); + const lockFd = acquireDeploymentLock(settings.accountId); const cli = createWranglerControl({ token: settings.token }); assert( cli.currentVersion() === settings.expectedVersion && @@ -333,7 +388,7 @@ async function controller() { [fileURLToPath(import.meta.url), '--watchdog', stateFile], { detached: true, - stdio: ['ignore', watcherLog, watcherLog], + stdio: ['ignore', watcherLog, watcherLog, lockFd], env: process.env, }, ); @@ -344,12 +399,28 @@ async function controller() { while (!existsSync(`${stateFile}.ready`) && Date.now() < readyDeadline) { await new Promise((resolve) => setTimeout(resolve, 100)); } - assert(existsSync(`${stateFile}.ready`), 'watchdog readiness is unconfirmed'); + assert( + existsSync(`${stateFile}.ready`) && + watcher.exitCode === null && + watcher.signalCode === null, + 'watchdog readiness is unconfirmed', + ); + process.kill(watcher.pid, 0); const probe = localProbe({ port: settings.port, secret: settings.secret }); let offConfirmed = false; const requestOffDeploy = async () => { if (offConfirmed) return; - cli.deployOffIfCurrent(settings.expectedVersion, outdir); + writeFileSync(`${stateFile}.off-requested`, '', { + mode: 0o600, + flag: 'wx', + }); + while (!existsSync(`${stateFile}.off-confirmed`)) { + assert( + watcher.exitCode === null && watcher.signalCode === null, + 'watchdog exited before off confirmation', + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + } offConfirmed = true; }; try { @@ -358,21 +429,14 @@ async function controller() { ...trustedD1Checks((sql) => cli.query(sql)), requestOffDeploy, onBeforeUnlock: async () => { - writeFileSync( - `${stateFile}.unlock-attempted`, - JSON.stringify({ - mutationEpoch: 1, - transitionRevision: 1, - }), - { - mode: 0o600, - flag: 'wx', - }, + assert( + watcher.exitCode === null && watcher.signalCode === null, + 'watchdog exited before fence unlock', ); + process.kill(watcher.pid, 0); }, }); assert(offConfirmed, 'off deploy was not confirmed'); - writeFileSync(`${stateFile}.done`, '', { mode: 0o600, flag: 'wx' }); return result; } catch (error) { appendFileSync( diff --git a/packages/flowsafe/deploy/notification-staging-live.test.mjs b/packages/flowsafe/deploy/notification-staging-live.test.mjs index 1226f395..236701fb 100644 --- a/packages/flowsafe/deploy/notification-staging-live.test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.test.mjs @@ -1,7 +1,8 @@ import assert from 'node:assert/strict'; -import { spawn } from 'node:child_process'; +import { spawn, spawnSync } from 'node:child_process'; import { existsSync, + mkdirSync, mkdtempSync, readFileSync, rmSync, @@ -14,7 +15,7 @@ import { test } from 'node:test'; import { fileURLToPath } from 'node:url'; import { createWranglerControl, - relockIfOpen, + sealFence, } from './notification-staging-live.mjs'; const enabled = 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa'; @@ -121,47 +122,95 @@ test('Wrangler failure preserves exit and stderr in its diagnostic cause', () => ); }); -test('watchdog relock uses a fresh epoch and revision', async () => { +test('watchdog seals the initial fence against a delayed controller open', async () => { const calls = []; - let observedRevision = 1; + let fence = { + state: 'migration-locked', + mutationEpoch: 0, + transitionRevision: 0, + requireMutationEpoch: false, + }; const probe = async (method, path, body) => { calls.push({ method, path, body }); - if (method === 'GET') - return { + if (method === 'GET') return fence; + assert.equal(body.expected, 'migration-locked'); + assert.equal(body.next, 'migration-locked'); + assert.equal(body.expectedMutationEpoch, 0); + assert.equal(body.expectedRevision, 0); + fence = { + state: 'migration-locked', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + return fence; + }; + assert.deepEqual(await sealFence(probe), fence); + assert.equal(calls.filter(({ method }) => method === 'POST').length, 1); + assert.equal(fence.mutationEpoch, 1); + assert.equal(fence.transitionRevision, 1); +}); + +test('watchdog relocks when controller open wins the deadline CAS', async () => { + let fence = { + state: 'migration-locked', + mutationEpoch: 0, + transitionRevision: 0, + requireMutationEpoch: false, + }; + const probe = async (method, _path, body) => { + if (method === 'GET') return fence; + if (body.expected === 'migration-locked') { + fence = { state: 'open', mutationEpoch: 1, - transitionRevision: observedRevision, + transitionRevision: 1, requireMutationEpoch: true, }; - return { + throw new Error('controller open won'); + } + assert.deepEqual(body, { + expected: 'open', + next: 'migration-locked', + expectedMutationEpoch: 1, + expectedRevision: 1, + advanceMutationEpoch: true, + }); + fence = { state: 'migration-locked', mutationEpoch: 2, transitionRevision: 2, requireMutationEpoch: true, }; + return fence; }; - assert.equal( - (await relockIfOpen(probe, { mutationEpoch: 1, transitionRevision: 1 })) - .state, - 'migration-locked', - ); - assert.deepEqual(calls[1].body, { - expected: 'open', - next: 'migration-locked', - expectedMutationEpoch: 1, - expectedRevision: 1, - advanceMutationEpoch: true, - }); - calls.length = 0; - observedRevision = 2; - await assert.rejects( - relockIfOpen(probe, { mutationEpoch: 1, transitionRevision: 1 }), - /metadata differs/, - ); - assert.equal(calls.length, 1); + assert.deepEqual(await sealFence(probe), fence); + assert.equal(fence.state, 'migration-locked'); + assert.equal(fence.transitionRevision, 2); +}); + +test('watchdog resolves a lost relock response from the authoritative fence', async () => { + let fence = { + state: 'open', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + const probe = async (method, _path, body) => { + if (method === 'GET') return fence; + assert.equal(body.expected, 'open'); + fence = { + state: 'migration-locked', + mutationEpoch: 2, + transitionRevision: 2, + requireMutationEpoch: true, + }; + throw new Error('response lost after commit'); + }; + assert.deepEqual(await sealFence(probe), fence); }); -test('controller waits for watchdog readiness and watchdog recovers after parent death', async () => { +async function exerciseWatchdog(outcome) { const directory = mkdtempSync(join(tmpdir(), 'flowsafe-live-watchdog-test-')); const secret = 's'.repeat(32); const tokenFile = join(directory, 'cf-token'); @@ -170,6 +219,15 @@ test('controller waits for watchdog readiness and watchdog recovers after parent const offFile = join(directory, 'off-deploy'); const offConfirmedFile = join(directory, 'off-confirmed'); const watcherPidFile = join(directory, 'watchdog-pid'); + const deployStartedFile = join(directory, 'deploy-started'); + const releaseDeployFile = join(directory, 'release-deploy'); + const accountId = + outcome === 'parent death' ? 'c'.repeat(32) : 'd'.repeat(32); + const lockFile = join( + '/tmp', + `flowsafe-deploy-locks-${process.getuid()}`, + `${accountId}-anchorage-flowsafe-remote-step-staging.lock`, + ); writeFileSync(tokenFile, 'fake-token', { mode: 0o600 }); writeFileSync(secretFile, secret, { mode: 0o600 }); writeFileSync( @@ -178,13 +236,21 @@ test('controller waits for watchdog readiness and watchdog recovers after parent import { syncBuiltinESMExports } from 'node:module'; import { existsSync, writeFileSync } from 'node:fs'; const originalSpawn = childProcess.spawn; +const originalSpawnSync = childProcess.spawnSync; +const originalNow = Date.now; +Date.now = () => originalNow() + + (process.env.TEST_CLOCK_JUMP === 'true' && + !process.argv.includes('--watchdog') && + existsSync(process.env.TEST_DEPLOY_STARTED_FILE) ? 300_000 : 0); +let counts = 0; childProcess.spawn = (...args) => { const child = originalSpawn(...args); if (args[1]?.includes('--watchdog')) writeFileSync(process.env.TEST_WATCHDOG_PID_FILE, String(child.pid)); return child; }; -childProcess.spawnSync = (executable, args) => { +childProcess.spawnSync = (executable, args, options) => { + if (executable === 'flock') return originalSpawnSync(executable, args, options); if (!executable.endsWith('/node_modules/.bin/wrangler')) throw new Error('unexpected synchronous command'); const off = existsSync(process.env.TEST_OFF_FILE); @@ -204,9 +270,20 @@ childProcess.spawnSync = (executable, args) => { ? [{ sql: 'write_generation INTEGER NOT NULL, operation_state TEXT' }] : args.at(-1).includes('COUNT(*) AS count') ? [{ count: 0 }] - : [{ notifications: 0, receipts: 0 }], + : args.at(-1).includes('FROM flowsafe_remote_step_invocations i') + ? [{ status: 'settled', settlement_outcome: 'applied', + operation_state: 'resume-admitted', thread_id: 'thread', + notification_id: 'notification', receipt_key: 'key', + receipt_digest: 'a'.repeat(64), receipt_thread_id: 'thread', + receipt_notification_id: 'notification', + stored_notification_id: 'notification', source: 'flowsafe-staging', + kind: 'host-owned-notification' }] + : [{ notifications: counts, receipts: counts++ }], }] : null; if (command === 'deploy') { + writeFileSync(process.env.TEST_DEPLOY_STARTED_FILE, 'started'); + while (!existsSync(process.env.TEST_RELEASE_DEPLOY_FILE)) + originalSpawnSync('sleep', ['0.02']); writeFileSync(process.env.TEST_OFF_FILE, 'deployed'); return { status: 0, stdout: 'deployed' }; } @@ -232,10 +309,17 @@ syncBuiltinESMExports(); requests.push(`${method} ${path}`); if (method === 'POST' && path === '/start') { startReceived = true; + if (outcome === 'parent death') return; + response.writeHead(200, { 'content-type': 'application/json' }); + response.end( + JSON.stringify({ status: 'suspended', invocationId: 'invocation' }), + ); return; } let value; - if (path === '/admin/maintenance-status') { + if (path === '/complete') { + value = { invocationId: 'invocation', status: 'success' }; + } else if (path === '/admin/maintenance-status') { value = { alarmAt: 1, nextRemoteStepRecoveryAt: 1 }; } else if (path === '/admin/execution-fence' && method === 'POST') { const chunks = []; @@ -281,9 +365,13 @@ syncBuiltinESMExports(); FLOWSAFE_STAGING_ENABLED_VERSION: enabled, FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, FLOWSAFE_CF_TOKEN_FILE: tokenFile, + FLOWSAFE_CF_ACCOUNT_ID: accountId, TEST_OFF_FILE: offFile, TEST_OFF_CONFIRMED_FILE: offConfirmedFile, TEST_WATCHDOG_PID_FILE: watcherPidFile, + TEST_DEPLOY_STARTED_FILE: deployStartedFile, + TEST_RELEASE_DEPLOY_FILE: releaseDeployFile, + TEST_CLOCK_JUMP: outcome === 'controller request' ? 'true' : 'false', }, stdio: 'ignore', }, @@ -304,10 +392,17 @@ syncBuiltinESMExports(); await waitFor(() => existsSync(watcherPidFile)); watcherPid = Number(readFileSync(watcherPidFile, 'utf8')); await waitFor(() => startReceived); - assert(existsSync(`${stateFile}.unlock-attempted`)); - assert.equal(fence.state, 'open'); assert.equal(existsSync(offFile), false); - controller.kill('SIGKILL'); + if (outcome === 'parent death') { + assert.equal(fence.state, 'open'); + controller.kill('SIGKILL'); + } + await waitFor(() => existsSync(deployStartedFile)); + assert.equal( + spawnSync('flock', ['--nonblock', lockFile, 'true']).status, + 1, + ); + writeFileSync(releaseDeployFile, 'release'); await waitFor(() => existsSync(offConfirmedFile)); assert.equal(readFileSync(offFile, 'utf8'), 'deployed'); assert.deepEqual(fence, { @@ -320,7 +415,13 @@ syncBuiltinESMExports(); requests.filter((request) => request === 'POST /admin/execution-fence'), ['POST /admin/execution-fence', 'POST /admin/execution-fence'], ); - assert.equal(existsSync(`${stateFile}.done`), false); + if (outcome === 'controller request') { + await waitFor(() => controller.exitCode !== null); + assert.equal(controller.exitCode, 0); + } + await waitFor( + () => spawnSync('flock', ['--nonblock', lockFile, 'true']).status === 0, + ); } finally { controller?.kill('SIGKILL'); if (Number.isInteger(watcherPid)) { @@ -334,4 +435,67 @@ syncBuiltinESMExports(); await new Promise((resolve) => server.close(resolve)); rmSync(directory, { recursive: true, force: true }); } +} + +test('watchdog owns off deployment after parent death', () => + exerciseWatchdog('parent death')); + +test('watchdog owns off deployment after controller request', () => + exerciseWatchdog('controller request')); + +test('controller refuses a contended deployment lock before Wrangler preflight', async () => { + const directory = mkdtempSync( + join(tmpdir(), 'flowsafe-lock-contention-test-'), + ); + const accountId = 'e'.repeat(32); + const lockDirectory = join( + '/tmp', + `flowsafe-deploy-locks-${process.getuid()}`, + ); + const lockFile = join( + lockDirectory, + `${accountId}-anchorage-flowsafe-remote-step-staging.lock`, + ); + mkdirSync(lockDirectory, { recursive: true, mode: 0o700 }); + writeFileSync(lockFile, '', { mode: 0o600 }); + const secretFile = join(directory, 'probe-secret'); + const tokenFile = join(directory, 'cf-token'); + writeFileSync(secretFile, 's'.repeat(32), { mode: 0o600 }); + writeFileSync(tokenFile, 'fake-token', { mode: 0o600 }); + const holder = spawn('flock', ['--exclusive', lockFile, 'sleep', '5'], { + stdio: 'ignore', + }); + try { + const deadline = Date.now() + 3_000; + while (spawnSync('flock', ['--nonblock', lockFile, 'true']).status !== 1) { + assert(Date.now() < deadline, 'lock holder did not acquire lock'); + await new Promise((resolve) => setTimeout(resolve, 20)); + } + const result = spawnSync( + process.execPath, + [ + fileURLToPath( + new URL('./notification-staging-live.mjs', import.meta.url), + ), + '--run', + ], + { + encoding: 'utf8', + env: { + ...process.env, + FLOWSAFE_STAGING_PROBE_PORT: '12345', + FLOWSAFE_STAGING_ENABLED_VERSION: enabled, + FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, + FLOWSAFE_CF_TOKEN_FILE: tokenFile, + FLOWSAFE_CF_ACCOUNT_ID: accountId, + }, + }, + ); + assert.equal(result.status, 1); + assert.match(result.stderr, /deployment lock is unavailable/); + assert.doesNotMatch(result.stderr, /Wrangler/); + } finally { + holder.kill('SIGKILL'); + rmSync(directory, { recursive: true, force: true }); + } }); From 3f0388078184af8ee81181427f552ee8ae0b8248 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 03:11:15 +0400 Subject: [PATCH 59/73] Keep Node staging tests out of Vitest discovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Name the local node:test files outside Vitest’s test glob so the package suite does not collect files without Vitest suites. --- ...ontrol.test.mjs => notification-staging-control.node-test.mjs} | 0 ...ging-live.test.mjs => notification-staging-live.node-test.mjs} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename packages/flowsafe/deploy/{notification-staging-control.test.mjs => notification-staging-control.node-test.mjs} (100%) rename packages/flowsafe/deploy/{notification-staging-live.test.mjs => notification-staging-live.node-test.mjs} (100%) diff --git a/packages/flowsafe/deploy/notification-staging-control.test.mjs b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs similarity index 100% rename from packages/flowsafe/deploy/notification-staging-control.test.mjs rename to packages/flowsafe/deploy/notification-staging-control.node-test.mjs diff --git a/packages/flowsafe/deploy/notification-staging-live.test.mjs b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs similarity index 100% rename from packages/flowsafe/deploy/notification-staging-live.test.mjs rename to packages/flowsafe/deploy/notification-staging-live.node-test.mjs From ced8376eb1b056ac98c18cb9c50472847edadf9a Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 04:09:10 +0400 Subject: [PATCH 60/73] Document referenced FlowSafe public types Export the type-only references through their owning subpaths so TypeDoc can resolve the public signatures without changing runtime exports. --- packages/flowsafe/src/agent-host/index.ts | 3 +++ .../flowsafe/src/agent-host/workflow-remote-step-route.ts | 2 +- packages/flowsafe/src/do-runner/index.ts | 2 ++ packages/flowsafe/src/host-kit/index.ts | 2 ++ packages/flowsafe/src/host-kit/state-egress.ts | 4 ++-- packages/flowsafe/src/signals/index.ts | 1 + 6 files changed, 11 insertions(+), 3 deletions(-) diff --git a/packages/flowsafe/src/agent-host/index.ts b/packages/flowsafe/src/agent-host/index.ts index 847d51b9..a04faed2 100644 --- a/packages/flowsafe/src/agent-host/index.ts +++ b/packages/flowsafe/src/agent-host/index.ts @@ -13,7 +13,9 @@ export { export type { RecordedRemoteStepResult, RemoteStepInvocation, + RemoteStepNotificationOperation, RemoteStepPermit, + WorkflowRemoteStepAddress, WorkflowRemoteStepInvocation, } from '../do-runner/remote-step-journal.js'; export { @@ -82,6 +84,7 @@ export type { TrustedAgentExecution, } from './types.js'; export { isPermissionIdentifier } from './types.js'; +export type { WorkflowDispatchPort } from './workflow-remote-step-route.js'; export { createWorkflowRemoteStepPreRoute, WORKFLOW_REMOTE_STEP_ROUTE, diff --git a/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts b/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts index a07a7877..e1a7d48a 100644 --- a/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts +++ b/packages/flowsafe/src/agent-host/workflow-remote-step-route.ts @@ -19,7 +19,7 @@ import type { export const WORKFLOW_REMOTE_STEP_ROUTE = '/_flowsafe/workflow-remote-step'; -type WorkflowDispatchPort = Pick< +export type WorkflowDispatchPort = Pick< AgentThreadDispatchTopology, | 'registerWorkflowRemoteStep' | 'issueWorkflowRemoteStepDispatch' diff --git a/packages/flowsafe/src/do-runner/index.ts b/packages/flowsafe/src/do-runner/index.ts index 49beab91..f5d2a449 100644 --- a/packages/flowsafe/src/do-runner/index.ts +++ b/packages/flowsafe/src/do-runner/index.ts @@ -228,7 +228,9 @@ export { type RemoteStepAddress, type RemoteStepInvocation, RemoteStepJournal, + type RemoteStepNotificationOperation, type RemoteStepPermit, + type WorkflowRemoteStepAddress, } from './remote-step-journal.js'; export type { RunEconomicOperation, diff --git a/packages/flowsafe/src/host-kit/index.ts b/packages/flowsafe/src/host-kit/index.ts index abd366d8..9a0f4fb5 100644 --- a/packages/flowsafe/src/host-kit/index.ts +++ b/packages/flowsafe/src/host-kit/index.ts @@ -179,6 +179,8 @@ export type { export { createRunRouter } from './run-router.js'; export type { StateEgressBinding, + StateEgressCommonContext, + StateEgressCommonEnv, StateEgressContext, StateEgressEnv, } from './state-egress.js'; diff --git a/packages/flowsafe/src/host-kit/state-egress.ts b/packages/flowsafe/src/host-kit/state-egress.ts index 846ddcea..45907be0 100644 --- a/packages/flowsafe/src/host-kit/state-egress.ts +++ b/packages/flowsafe/src/host-kit/state-egress.ts @@ -50,7 +50,7 @@ export interface StateEgressBinding { fetch(request: Request): Promise; } -interface StateEgressCommonContext { +export interface StateEgressCommonContext { readonly credential: string; readonly tenantTag: string; readonly environment: string; @@ -75,7 +75,7 @@ export type StateEgressContext = StateEgressCommonContext & }> ); -interface StateEgressCommonEnv { +export interface StateEgressCommonEnv { readonly OUTBOUND_PROXY: StateEgressBinding; readonly OUTBOUND_PROXY_CREDENTIAL: string; readonly OUTBOUND_TENANT_ID: string; diff --git a/packages/flowsafe/src/signals/index.ts b/packages/flowsafe/src/signals/index.ts index 11d9e3ca..ea8bdafc 100644 --- a/packages/flowsafe/src/signals/index.ts +++ b/packages/flowsafe/src/signals/index.ts @@ -28,6 +28,7 @@ export { type NotificationDispatchTickOptions, type NotificationDispatchTickResult, } from './notification-dispatch.js'; +export type { NotificationOperationInput } from './notifications-d1.js'; export { D1NotificationsStorage } from './notifications-d1.js'; // The P6 ingestion trust boundary router (CI-M-004-004). export { From 572ff9e82bed7939605f17eb4b884aab1132e014 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 06:26:45 +0400 Subject: [PATCH 61/73] Reuse validated workflow dispatch row Avoid a second D1 getter read while retaining the stored principal, deployment, resource and run-access checks on the same invocation. --- .../src/agent-host/thread-topology.test.ts | 18 ++++++++++++++++++ .../flowsafe/src/agent-host/thread-topology.ts | 14 +++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/packages/flowsafe/src/agent-host/thread-topology.test.ts b/packages/flowsafe/src/agent-host/thread-topology.test.ts index 977183d8..bc3b5603 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.test.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.test.ts @@ -195,9 +195,14 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', invocation.invocationId, 60_000, ); + const workflowReadsBeforeGet = vi.mocked(reader.getWorkflow).mock.calls + .length; expect( await topology.getWorkflowRemoteStep(owner, invocation.invocationId), ).toEqual(dispatched); + expect(vi.mocked(reader.getWorkflow).mock.calls.length).toBe( + workflowReadsBeforeGet + 1, + ); expect( await topology.consumeWorkflowRemoteStepDispatch(owner, permit), ).toEqual({ kind: 'accepted', rawPermit: 'winner-secret' }); @@ -236,6 +241,9 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', 60_000, ), ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.getWorkflowRemoteStep(otherPrincipal, invocation.invocationId), + ).rejects.toMatchObject({ status: 404 }); await expect( topology.consumeWorkflowRemoteStepDispatch(otherPrincipal, permit), ).rejects.toMatchObject({ status: 404 }); @@ -264,6 +272,9 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', 60_000, ), ).rejects.toMatchObject({ status: 404 }); + await expect( + topology.getWorkflowRemoteStep(deniedRun, invocation.invocationId), + ).rejects.toMatchObject({ status: 404 }); vi.mocked(reader.getWorkflow) .mockResolvedValueOnce(dispatched) .mockResolvedValueOnce({ @@ -288,6 +299,13 @@ it('keeps workflow remote-step dispatch within the stored owner and deployment', 60_000, ), ).rejects.toMatchObject({ status: 404 }); + vi.mocked(reader.getWorkflow).mockResolvedValueOnce({ + ...dispatched, + deploymentId: 'another-deployment', + }); + await expect( + topology.getWorkflowRemoteStep(owner, invocation.invocationId), + ).rejects.toMatchObject({ status: 404 }); expect(reader.issuePermit).toHaveBeenCalledTimes(2); expect(reader.consumePermit).toHaveBeenCalledTimes(2); expect(reader.recordResult).toHaveBeenCalledTimes(1); diff --git a/packages/flowsafe/src/agent-host/thread-topology.ts b/packages/flowsafe/src/agent-host/thread-topology.ts index 98f74538..8d47a7dc 100644 --- a/packages/flowsafe/src/agent-host/thread-topology.ts +++ b/packages/flowsafe/src/agent-host/thread-topology.ts @@ -408,12 +408,14 @@ export function createAgentThreadTopology( context: ActorContext, invocationId: string, access: 'read' | 'write', + loadedInvocation?: WorkflowRemoteStepInvocation, ) => { const reader = options.workflowRemoteStepDispatch; if (!reader) throw new RunRouteError(503, 'remote steps are unavailable'); if (!isPathSafeId(invocationId)) throw new RunRouteError(404, 'invocation not found'); - const invocation = await reader.getWorkflow(invocationId); + const invocation = + loadedInvocation ?? (await reader.getWorkflow(invocationId)); if ( invocation?.targetKind !== 'workflow' || invocation.invocationId !== invocationId || @@ -617,8 +619,14 @@ export function createAgentThreadTopology( throw new RunRouteError(404, 'invocation not found'); const invocation = await reader.getWorkflow(invocationId); if (!invocation) return undefined; - return (await workflowDispatchInvocation(context, invocationId, 'read')) - .invocation; + return ( + await workflowDispatchInvocation( + context, + invocationId, + 'read', + invocation, + ) + ).invocation; }, issueWorkflowRemoteStepDispatch: async ( context, From cada4fbed4736dd57b6c2c909d2e004c374b0ecf Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 06:43:16 +0400 Subject: [PATCH 62/73] Reuse validated remote-step journal rows Share agent/workflow lifecycle checks and avoid duplicate notification reads while preserving fresh post-write CAS and result checks. --- .../src/do-runner/remote-step-journal.ts | 190 ++++++++---------- 1 file changed, 87 insertions(+), 103 deletions(-) diff --git a/packages/flowsafe/src/do-runner/remote-step-journal.ts b/packages/flowsafe/src/do-runner/remote-step-journal.ts index f2d8bca8..249c9483 100644 --- a/packages/flowsafe/src/do-runner/remote-step-journal.ts +++ b/packages/flowsafe/src/do-runner/remote-step-journal.ts @@ -411,6 +411,63 @@ function rowResult(value: unknown): JournalRow | undefined { : undefined; } +function validLifecycle(row: JournalRow): boolean { + return ( + Number.isSafeInteger(row.resume_count) && + (row.resume_count === -1 || row.resume_count >= 1) && + Number.isSafeInteger(row.permit_generation) && + row.permit_generation >= 0 && + [ + 'pending', + 'dispatched', + 'result-retained', + 'resuming', + 'settled', + 'failed', + ].includes(row.status) && + (row.settlement_outcome === null || + [ + 'applied', + 'terminal-not-success', + 'deadline-expired', + 'advanced-without-proof', + ].includes(row.settlement_outcome)) && + (row.status !== 'resuming' || + (isPathSafeId(row.resume_lease_owner) && + positiveTime(row.resume_lease_expires_at) && + row.result_json !== null)) && + (row.status === 'resuming' || + (row.resume_lease_owner === null && + row.resume_lease_expires_at === null)) && + (row.status !== 'settled' || + (row.settlement_outcome === 'applied' && + row.result_json !== null && + row.result_digest !== null && + positiveTime(row.executor_quiescent_at))) && + (row.settlement_outcome === null || + ((row.status === 'settled' || row.status === 'failed') && + positiveTime(row.settled_at))) + ); +} + +function validPayloadBounds(row: JournalRow): boolean { + return ( + typeof row.step_path === 'string' && + row.step_path.length <= 1_024 && + typeof row.request_json === 'string' && + new TextEncoder().encode(row.request_json).length <= + MAX_REMOTE_STEP_REQUEST_BYTES + ); +} + +function validStepPath(value: unknown): value is string[] { + return ( + Array.isArray(value) && + value.length > 0 && + value.every((part) => typeof part === 'string' && part !== '') + ); +} + function decode(row: JournalRow): RemoteStepInvocation { if ( row.target_kind !== 'agent' || @@ -419,12 +476,8 @@ function decode(row: JournalRow): RemoteStepInvocation { !Number.isSafeInteger(row.write_revision) || row.write_revision < 0 || !positiveTime(row.write_generation) || - typeof row.step_path !== 'string' || - row.step_path.length > 1_024 || - typeof row.request_json !== 'string' || + !validPayloadBounds(row) || typeof row.principal_json !== 'string' || - new TextEncoder().encode(row.request_json).length > - MAX_REMOTE_STEP_REQUEST_BYTES || !isPathSafeId(row.deployment_id) || !isPathSafeId(row.workflow_id) || !isPathSafeId(row.agent_id) || @@ -453,46 +506,11 @@ function decode(row: JournalRow): RemoteStepInvocation { throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); } if ( - !Array.isArray(stepPath) || - stepPath.length === 0 || - stepPath.some((part) => typeof part !== 'string' || part === '') || + !validStepPath(stepPath) || !isPathSafeId(row.invocation_id) || !positiveTime(row.suspended_at) || !positiveTime(row.deadline_at) || - !Number.isSafeInteger(row.resume_count) || - (row.resume_count !== -1 && row.resume_count < 1) || - !Number.isSafeInteger(row.permit_generation) || - row.permit_generation < 0 || - ![ - 'pending', - 'dispatched', - 'result-retained', - 'resuming', - 'settled', - 'failed', - ].includes(row.status) || - (row.settlement_outcome !== null && - ![ - 'applied', - 'terminal-not-success', - 'deadline-expired', - 'advanced-without-proof', - ].includes(row.settlement_outcome)) || - (row.status === 'resuming' && - (!isPathSafeId(row.resume_lease_owner) || - !positiveTime(row.resume_lease_expires_at) || - row.result_json === null)) || - (row.status !== 'resuming' && - (row.resume_lease_owner !== null || - row.resume_lease_expires_at !== null)) || - (row.status === 'settled' && row.settlement_outcome !== 'applied') || - (row.status === 'settled' && - (row.result_json === null || - row.result_digest === null || - !positiveTime(row.executor_quiescent_at))) || - (row.settlement_outcome !== null && - ((row.status !== 'settled' && row.status !== 'failed') || - !positiveTime(row.settled_at))) + !validLifecycle(row) ) throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); return { @@ -557,30 +575,8 @@ function decodeWorkflow(row: JournalRow): WorkflowRemoteStepInvocation { !positiveTime(row.write_generation) || !positiveTime(row.suspended_at) || !positiveTime(row.deadline_at) || - !Number.isSafeInteger(row.resume_count) || - (row.resume_count !== -1 && row.resume_count < 1) || - !Number.isSafeInteger(row.permit_generation) || - row.permit_generation < 0 || - ![ - 'pending', - 'dispatched', - 'result-retained', - 'resuming', - 'settled', - 'failed', - ].includes(row.status) || - (row.status === 'resuming' && - (!isPathSafeId(row.resume_lease_owner) || - !positiveTime(row.resume_lease_expires_at) || - row.result_json === null)) || - (row.status !== 'resuming' && - (row.resume_lease_owner !== null || - row.resume_lease_expires_at !== null)) || - typeof row.step_path !== 'string' || - row.step_path.length > 1_024 || - typeof row.request_json !== 'string' || - new TextEncoder().encode(row.request_json).length > - MAX_REMOTE_STEP_REQUEST_BYTES + !validLifecycle(row) || + !validPayloadBounds(row) ) throw new RemoteStepRefusal(503, 'remote step journal is unreadable'); try { @@ -591,27 +587,7 @@ function decodeWorkflow(row: JournalRow): WorkflowRemoteStepInvocation { JSON.parse(row.principal_json), 'stored remote step principal', ); - if ( - !Array.isArray(stepPath) || - stepPath.length === 0 || - stepPath.some((part) => typeof part !== 'string' || part === '') || - (row.settlement_outcome !== null && - ![ - 'applied', - 'terminal-not-success', - 'deadline-expired', - 'advanced-without-proof', - ].includes(row.settlement_outcome)) || - (row.status === 'settled' && row.settlement_outcome !== 'applied') || - (row.status === 'settled' && - (row.result_json === null || - row.result_digest === null || - !positiveTime(row.executor_quiescent_at))) || - (row.settlement_outcome !== null && - ((row.status !== 'settled' && row.status !== 'failed') || - !positiveTime(row.settled_at))) - ) - throw new Error('invalid workflow row'); + if (!validStepPath(stepPath)) throw new Error('invalid workflow row'); return { targetKind: 'workflow', invocationId: row.invocation_id, @@ -2129,16 +2105,15 @@ export class RemoteStepJournal { async readRetainedResult( invocationId: string, ): Promise { - if (!isPathSafeId(invocationId)) - throw new RemoteStepRefusal(400, 'invocation id is malformed'); - const row = rowResult( - await this.#db - .prepare(`SELECT * FROM ${REMOTE_STEP_TABLE} WHERE invocation_id = ?`) - .bind(invocationId) - .all(), - ); + const row = await this.#getRow(invocationId); if (!row) return undefined; decodeAny(row); + return this.#retainedResultFromRow(row); + } + + async #retainedResultFromRow( + row: JournalRow, + ): Promise { const resultJson = row.result_json; const resultDigest = row.result_digest; const quiescenceDigest = row.quiescence_digest; @@ -2163,7 +2138,7 @@ export class RemoteStepJournal { if ((await sha256Hex(resultJson)) !== resultDigest) throw new RemoteStepRefusal(503, 'remote step result is unreadable'); return { - invocationId, + invocationId: row.invocation_id, resultJson, resultDigest, quiescenceDigest, @@ -2177,6 +2152,12 @@ export class RemoteStepJournal { const row = await this.#getRow(invocationId); if (!row) return undefined; decodeAny(row); + return this.#notificationOperationFromRow(row); + } + + async #notificationOperationFromRow( + row: JournalRow, + ): Promise { const operation = decodeNotificationOperation(row); if ( operation && @@ -2196,7 +2177,10 @@ export class RemoteStepJournal { const stored = await this.#getRow(invocationId); const invocation = stored?.target_kind === 'agent' ? decode(stored) : undefined; - const retained = await this.readRetainedResult(invocationId); + if (stored && !invocation) decodeAny(stored); + const retained = stored + ? await this.#retainedResultFromRow(stored) + : undefined; if ( !stored || !invocation || @@ -2229,7 +2213,7 @@ export class RemoteStepJournal { 400, 'remote step operation exceeds its limit', ); - const existing = await this.readNotificationOperation(invocationId); + const existing = await this.#notificationOperationFromRow(stored); if (existing) { if ( existing.operationKey !== operationKey || @@ -2289,14 +2273,14 @@ export class RemoteStepJournal { const stored = await this.#getRow(invocationId); if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); const invocation = decode(stored); - const operation = await this.readNotificationOperation(invocationId); + const operation = await this.#notificationOperationFromRow(stored); if ( !operation || !['prepared', 'effect-admitted'].includes(operation.state) ) throw new RemoteStepRefusal(409, 'remote step operation is unavailable'); if (operation.state === 'effect-admitted') return operation; - const retained = await this.readRetainedResult(invocationId); + const retained = await this.#retainedResultFromRow(stored); if (!retained || invocation.status !== 'result-retained') throw new RemoteStepRefusal(409, 'remote step result is unavailable'); const { suspension, snapshot } = await this.#stableSnapshot(invocation); @@ -2389,7 +2373,7 @@ export class RemoteStepJournal { const stored = await this.#getRow(invocationId); if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); decode(stored); - const operation = await this.readNotificationOperation(invocationId); + const operation = await this.#notificationOperationFromRow(stored); if ( operation?.state !== 'effect-admitted' || receipt.operationKey !== operation.operationKey || @@ -2444,7 +2428,7 @@ export class RemoteStepJournal { const stored = await this.#getRow(invocationId); if (!stored) throw new RemoteStepRefusal(404, 'invocation not found'); const invocation = decode(stored); - const operation = await this.readNotificationOperation(invocationId); + const operation = await this.#notificationOperationFromRow(stored); if ( operation?.state !== 'receipt-recorded' || invocation.status !== 'result-retained' @@ -2453,7 +2437,7 @@ export class RemoteStepJournal { 409, 'remote step continuation is unavailable', ); - const retained = await this.readRetainedResult(invocationId); + const retained = await this.#retainedResultFromRow(stored); if (!retained) throw new RemoteStepRefusal(409, 'remote step result is unavailable'); const { suspension, snapshot } = await this.#stableSnapshot(invocation); From 297cce41c37468c3970728f33c1f5da46558516e Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 08:08:45 +0400 Subject: [PATCH 63/73] Fence notification receipt cap to one D1 scope Register one receipt-writing prefix and limit per deployment, preserving ordinary prefixed storage and refusing ambiguous retained histories. --- docs/deployment-reference.md | 9 + packages/flowsafe/src/do-runner/inventory.ts | 9 + .../src/signals/notifications-d1.test.ts | 155 ++++++++++++++++++ .../flowsafe/src/signals/notifications-d1.ts | 101 ++++++++++++ 4 files changed, 274 insertions(+) diff --git a/docs/deployment-reference.md b/docs/deployment-reference.md index 05e399ac..09c1f786 100644 --- a/docs/deployment-reference.md +++ b/docs/deployment-reference.md @@ -305,6 +305,15 @@ Decommission the whole physical resource set after revoking traffic and credenti When `init()` or `createD1Storage()` receives `tablePrefix`, pass the same value as `storageTablePrefix` to `createFlowsafeWorker()`. The shared rule accepts an empty prefix or requires an ASCII letter or underscore first, followed by ASCII letters, numbers, or underscores. The prefix can contain at most 39 characters so every final Mastra table identifier stays within its 63-character limit. Maintenance applies it to workflow snapshots, threads/messages, background tasks and their snapshots, notifications, thread state, and schedule-trigger history. All six exported low-level purge functions validate the same contract before preparing D1 statements. Approval, resource-owner, deployment-sentinel, subscription, and other fixed-schema tables remain unprefixed. A valid but mismatched value targets a different table family; Flowsafe does not auto-discover it. +Host-owned notification effects register one receipt-writing table prefix and one receipt limit in the deployment's D1 database. The default limit is 100,000 retained receipts; a deployment can set a lower limit. Existing receipts remain available for replay at the limit. Ordinary notification storage can still use other prefixes, but those prefixes cannot write operation receipts in the same deployment. + +To upgrade a deployment with retained operation receipts: + +1. Lock its execution fence and wait for earlier receipt writers to finish. Keep every Worker and Durable Object that can write the D1 database under exclusive deployment control. +2. Inspect the D1 tables ending in `flowsafe_notification_operation_receipts`. If more than one prefix has retained rows, leave the fence locked and resolve the histories before enabling effects. Do not delete receipts to meet the limit. +3. Deploy the version that registers receipt scope to every writer of that D1 database. Set one prefix and one limit across those writers. Check the active versions before unlocking. An older writer can bypass the registration, so do not run mixed versions with the fence open. +4. Unlock the fence. The first new receipt operation registers the prefix and limit; later writes with a different value fail closed. Check the `flowsafe_notification_receipt_scope` row after the first operation. Fence the deployment again before any rollback to an older writer. + Idle-thread retention does not release thread or resource ownership. A thread can still have an agent binding, schedule, subscription, goal, or other standing wake source after its idle memory rows expire. An explicit host teardown must remove every authoritative standing record before it calls `ActorContext.releaseResource()` for the thread and resource claims. ## Egress and process execution diff --git a/packages/flowsafe/src/do-runner/inventory.ts b/packages/flowsafe/src/do-runner/inventory.ts index bd16bf71..e9488a93 100644 --- a/packages/flowsafe/src/do-runner/inventory.ts +++ b/packages/flowsafe/src/do-runner/inventory.ts @@ -353,6 +353,15 @@ export const FLOWSAFE_TABLES: readonly FlowsafeTableEntry[] = [ 'a receipt records an effect already committed with its inbox row; replay must retain it after the inbox row is settled or removed, so it is historical evidence rather than active work', }, }, + { + table: 'flowsafe_notification_receipt_scope', + purpose: + 'the deployment receipt prefix and retained receipt admission limit', + accounting: { + excluded: + 'the singleton configures receipt admission and contains no active notification effect', + }, + }, { table: RESOURCE_OWNER_TABLE, purpose: diff --git a/packages/flowsafe/src/signals/notifications-d1.test.ts b/packages/flowsafe/src/signals/notifications-d1.test.ts index 9416f671..1db3161e 100644 --- a/packages/flowsafe/src/signals/notifications-d1.test.ts +++ b/packages/flowsafe/src/signals/notifications-d1.test.ts @@ -39,6 +39,161 @@ function sharedStores(): [D1NotificationsStorage, D1NotificationsStorage] { } describe('host-owned notification operation receipt', () => { + it('admits only one receipt prefix in a D1 deployment', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, 'left_', 1); + const right = new D1NotificationsStorage(db, 'right_', 1); + await Promise.all([left.init(), right.init()]); + const input = { + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const outcomes = await Promise.allSettled([ + left.createNotificationOnce('operation-left', { + ...input, + id: 'notification-left', + }), + right.createNotificationOnce('operation-right', { + ...input, + id: 'notification-right', + }), + ]); + expect( + outcomes.filter((outcome) => outcome.status === 'fulfilled'), + ).toHaveLength(1); + expect( + outcomes.filter((outcome) => outcome.status === 'rejected'), + ).toHaveLength(1); + }); + + it('retains a prefix and limit across fresh storage instances', async () => { + const db = database(); + const input = { + id: 'notification-1', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + const first = new D1NotificationsStorage(db, 'agent_', 1); + const receipt = await first.createNotificationOnce('operation-1', input); + const fresh = new D1NotificationsStorage(db, 'agent_', 1); + expect(await fresh.createNotificationOnce('operation-1', input)).toEqual( + receipt, + ); + await expect( + new D1NotificationsStorage(db, 'agent_', 2).createNotificationOnce( + 'operation-2', + { ...input, id: 'notification-2' }, + ), + ).rejects.toThrow( + 'notification receipt scope differs from this deployment', + ); + expect( + await fresh.getNotification({ threadId: 'thread', id: 'notification-2' }), + ).toBeNull(); + }); + + it('refuses ambiguous nonempty legacy receipt histories', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, 'left_'); + const right = new D1NotificationsStorage(db, 'right_'); + await Promise.all([left.init(), right.init()]); + for (const prefix of ['left_', 'right_']) { + await db + .prepare(`INSERT INTO ${prefix}flowsafe_notification_operation_receipts + (operation_key, request_digest, thread_id, notification_id) + VALUES (?, ?, ?, ?)`) + .bind( + `legacy-${prefix}`, + 'a'.repeat(64), + 'thread', + `notification-${prefix}`, + ) + .run(); + } + await expect( + left.createNotificationOnce('operation-new', { + id: 'notification-new', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }), + ).rejects.toThrow('notification receipt history spans another prefix'); + const receipt = await db + .prepare( + 'SELECT operation_key FROM left_flowsafe_notification_operation_receipts WHERE operation_key = ?', + ) + .bind('operation-new') + .first(); + expect(receipt).toBeNull(); + }); + + it('adopts the single nonempty legacy prefix', async () => { + const db = database(); + const left = new D1NotificationsStorage(db, 'left_'); + const right = new D1NotificationsStorage(db, 'right_'); + await Promise.all([left.init(), right.init()]); + await db + .prepare(`INSERT INTO left_flowsafe_notification_operation_receipts + (operation_key, request_digest, thread_id, notification_id) + VALUES ('legacy', ?, 'thread', 'notification-legacy')`) + .bind('a'.repeat(64)) + .run(); + const input = { + id: 'notification-new', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }; + await expect( + right.createNotificationOnce('operation-new', input), + ).rejects.toThrow('notification receipt history spans another prefix'); + await expect( + left.createNotificationOnce('operation-new', input), + ).resolves.toMatchObject({ + operationKey: 'operation-new', + notificationId: 'notification-new', + }); + }); + + it('refuses an unknown receipt scope version', async () => { + const db = database(); + await db + .prepare(`CREATE TABLE flowsafe_notification_receipt_scope ( + id INTEGER PRIMARY KEY, + version INTEGER NOT NULL, + table_prefix TEXT NOT NULL, + receipt_limit INTEGER NOT NULL + )`) + .run(); + await db + .prepare(`INSERT INTO flowsafe_notification_receipt_scope + (id, version, table_prefix, receipt_limit) VALUES (1, 2, '', 100000)`) + .run(); + const notifications = new D1NotificationsStorage(db); + await expect( + notifications.createNotificationOnce('operation-new', { + id: 'notification-new', + threadId: 'thread', + source: 'remote-step', + kind: 'result', + summary: 'Completed', + createdAt: new Date('2026-10-13T00:00:00.000Z'), + }), + ).rejects.toThrow( + 'notification receipt scope differs from this deployment', + ); + }); + it('refuses a new operation at the retained receipt cap while replay still succeeds', async () => { const db = database(); const notifications = new D1NotificationsStorage(db, '', 1); diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index ab09b9ef..369efe63 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -396,6 +396,8 @@ export const NOTIFICATION_OPERATION_PROJECTION_VERSION = 'notification-record-values-v1'; export const NOTIFICATION_OPERATION_RECEIPTS_TABLE = 'flowsafe_notification_operation_receipts'; +export const NOTIFICATION_RECEIPT_SCOPE_TABLE = + 'flowsafe_notification_receipt_scope'; export const DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT = 100_000; export type NotificationOperationInput = CreateNotificationInput & { @@ -470,10 +472,12 @@ export class D1NotificationsStorage readonly #sequenceTable: string; readonly #operationReceiptTable: string; readonly #operationReceiptLimit: number; + readonly #prefix: string; readonly #ordinalIndex: string; readonly #legacyReplaceTrigger: string; readonly #legacyInsertTrigger: string; #ready?: Promise; + #receiptScopeReady?: Promise; constructor( db: SignalDatabase, @@ -483,6 +487,7 @@ export class D1NotificationsStorage super(); const prefix = validateTablePrefix(tablePrefix) ?? ''; this.#db = db; + this.#prefix = prefix; this.#table = `${prefix}mastra_notifications`; this.#sequenceTable = `${prefix}${NOTIFICATION_SEQUENCE_TABLE}`; this.#operationReceiptTable = `${prefix}${NOTIFICATION_OPERATION_RECEIPTS_TABLE}`; @@ -490,6 +495,12 @@ export class D1NotificationsStorage operationReceiptLimit, 'operationReceiptLimit', ); + if ( + this.#operationReceiptLimit > DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT + ) + throw new RangeError( + 'operationReceiptLimit exceeds the deployment maximum', + ); this.#ordinalIndex = `idx_${this.#table}_insertion_ordinal`; this.#legacyReplaceTrigger = `trg_${this.#table}_preserve_insertion_ordinal`; this.#legacyInsertTrigger = `trg_${this.#table}_allocate_insertion_ordinal`; @@ -697,6 +708,95 @@ export class D1NotificationsStorage await this.#ensureSchema(); } + #ensureReceiptScope(): Promise { + if (!this.#receiptScopeReady) { + this.#receiptScopeReady = this.#registerReceiptScope().catch( + (error: unknown) => { + this.#receiptScopeReady = undefined; + throw error; + }, + ); + } + return this.#receiptScopeReady; + } + + async #registerReceiptScope(): Promise { + const legacyTables = await this.#db + .prepare(`SELECT name FROM sqlite_master + WHERE type = 'table' AND substr(name, -length(?)) = ?`) + .bind( + NOTIFICATION_OPERATION_RECEIPTS_TABLE, + NOTIFICATION_OPERATION_RECEIPTS_TABLE, + ) + .all<{ name: string }>(); + for (const { name } of legacyTables.results) { + const prefix = name.slice( + 0, + -NOTIFICATION_OPERATION_RECEIPTS_TABLE.length, + ); + if (validateTablePrefix(prefix) !== prefix) + throw new Error('notification receipt history has an invalid prefix'); + const retained = await this.#db + .prepare(`SELECT 1 AS retained FROM "${name}" LIMIT 1`) + .first<{ retained: number }>(); + if (retained && prefix !== this.#prefix) + throw new Error('notification receipt history spans another prefix'); + } + await this.#db + .prepare(`CREATE TABLE IF NOT EXISTS ${NOTIFICATION_RECEIPT_SCOPE_TABLE} ( + id INTEGER PRIMARY KEY CHECK (id = 1), + version INTEGER NOT NULL CHECK (version = 1), + table_prefix TEXT NOT NULL, + receipt_limit INTEGER NOT NULL CHECK (receipt_limit BETWEEN 0 AND ${DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT}) + )`) + .run(); + const columns = await this.#db + .prepare(`PRAGMA table_info(${NOTIFICATION_RECEIPT_SCOPE_TABLE})`) + .all<{ name: string; type: string; notnull: number; pk: number }>(); + if ( + JSON.stringify( + columns.results.map(({ name, type, notnull, pk }) => [ + name, + type, + notnull, + pk, + ]), + ) !== + JSON.stringify([ + ['id', 'INTEGER', 0, 1], + ['version', 'INTEGER', 1, 0], + ['table_prefix', 'TEXT', 1, 0], + ['receipt_limit', 'INTEGER', 1, 0], + ]) + ) + throw new Error('notification receipt scope schema is unknown'); + await this.#db + .prepare(`INSERT INTO ${NOTIFICATION_RECEIPT_SCOPE_TABLE} + (id, version, table_prefix, receipt_limit) + VALUES (1, 1, ?, ?) ON CONFLICT(id) DO NOTHING`) + .bind(this.#prefix, this.#operationReceiptLimit) + .run(); + const scopes = await this.#db + .prepare(`SELECT id, version, table_prefix, receipt_limit + FROM ${NOTIFICATION_RECEIPT_SCOPE_TABLE}`) + .all<{ + id: number; + version: number; + table_prefix: string; + receipt_limit: number; + }>(); + if ( + scopes.results.length !== 1 || + scopes.results[0]?.id !== 1 || + scopes.results[0]?.version !== 1 || + scopes.results[0]?.table_prefix !== this.#prefix || + scopes.results[0]?.receipt_limit !== this.#operationReceiptLimit + ) + throw new Error( + 'notification receipt scope differs from this deployment', + ); + } + async createNotificationOnce( operationKey: string, input: NotificationOperationInput, @@ -716,6 +816,7 @@ export class D1NotificationsStorage throw new TypeError('notification operation identity is malformed'); } await this.#ensureSchema(); + await this.#ensureReceiptScope(); const batch = this.#db.batch?.bind(this.#db); if (!batch) throw new Error('notification operation requires database.batch()'); From bf26b96413e9bb2a246637e8291b703a7431e3a7 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 08:13:36 +0400 Subject: [PATCH 64/73] Prove cross-prefix receipt exclusion in native D1 Race two Durable Objects at scope registration and assert one winner with no losing receipt or notification. --- .../src/cloudflare-fidelity.workerd.test.ts | 97 ++++++++++++++++++- .../src/signals/notifications-d1.test.ts | 29 ++++++ .../test-support/cloudflare-test-worker.ts | 27 ++++-- 3 files changed, 145 insertions(+), 8 deletions(-) diff --git a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts index 0fe07422..c9110211 100644 --- a/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts +++ b/packages/flowsafe/src/cloudflare-fidelity.workerd.test.ts @@ -38,7 +38,12 @@ function createReceipt( operationKey: string, notificationId: string, receiptLimit: number, - barrier?: { id: string; participant: string }, + barrier?: { + id: string; + participant: string; + phase?: 'receipt-read' | 'scope-census'; + }, + tablePrefix = '', ) { return actor.fetch( new Request('http://localhost/receipt', { @@ -47,8 +52,10 @@ function createReceipt( operationKey, notificationId, receiptLimit, + tablePrefix, barrierId: barrier?.id, participant: barrier?.participant, + barrierPhase: barrier?.phase, }), }), ); @@ -258,4 +265,92 @@ describe('FlowSafe Cloudflare fidelity', () => { requestDigest: winner.request_digest, }); }, 30_000); + + it('admits one receipt prefix across distinct durable objects', async () => { + await Promise.all([ + new D1NotificationsStorage(db(), 'left_').init(), + new D1NotificationsStorage(db(), 'right_').init(), + ]); + await db() + .prepare(`CREATE TABLE notification_test_barrier ( + barrier_id TEXT NOT NULL, + participant TEXT NOT NULL, + PRIMARY KEY (barrier_id, participant) + )`) + .run(); + const responses = await Promise.all([ + createReceipt( + receiptActor('prefix-left'), + 'operation-1', + 'notification-1', + 1, + { id: 'prefix-claim', participant: 'left', phase: 'scope-census' }, + 'left_', + ), + createReceipt( + receiptActor('prefix-right'), + 'operation-2', + 'notification-2', + 1, + { id: 'prefix-claim', participant: 'right', phase: 'scope-census' }, + 'right_', + ), + ]); + const ready = await db() + .prepare(`SELECT participant FROM notification_test_barrier + WHERE barrier_id = 'prefix-claim' ORDER BY participant`) + .all<{ participant: string }>(); + expect(ready.results).toEqual([ + { participant: 'left' }, + { participant: 'right' }, + ]); + expect(responses.map((response) => response.status).sort()).toEqual([ + 200, 409, + ]); + const loser = responses.find((response) => response.status === 409); + expect(await loser?.json()).toEqual({ + error: 'notification receipt scope differs from this deployment', + }); + const [leftReceipts, rightReceipts, leftInbox, rightInbox] = + await Promise.all([ + db() + .prepare( + 'SELECT operation_key FROM left_flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string }>(), + db() + .prepare( + 'SELECT operation_key FROM right_flowsafe_notification_operation_receipts', + ) + .all<{ operation_key: string }>(), + db() + .prepare('SELECT id FROM left_mastra_notifications') + .all<{ id: string }>(), + db() + .prepare('SELECT id FROM right_mastra_notifications') + .all<{ id: string }>(), + ]); + const expected = + responses[0]?.status === 200 + ? { + receipts: [[{ operation_key: 'operation-1' }], []], + inbox: [[{ id: 'notification-1' }], []], + prefix: 'left_', + } + : { + receipts: [[], [{ operation_key: 'operation-2' }]], + inbox: [[], [{ id: 'notification-2' }]], + prefix: 'right_', + }; + expect([leftReceipts.results, rightReceipts.results]).toEqual( + expected.receipts, + ); + expect([leftInbox.results, rightInbox.results]).toEqual(expected.inbox); + const scope = await db() + .prepare( + 'SELECT table_prefix, receipt_limit FROM flowsafe_notification_receipt_scope', + ) + .first<{ table_prefix: string; receipt_limit: number }>(); + expect(scope).toEqual({ table_prefix: expected.prefix, receipt_limit: 1 }); + }); }); diff --git a/packages/flowsafe/src/signals/notifications-d1.test.ts b/packages/flowsafe/src/signals/notifications-d1.test.ts index 1db3161e..3f909926 100644 --- a/packages/flowsafe/src/signals/notifications-d1.test.ts +++ b/packages/flowsafe/src/signals/notifications-d1.test.ts @@ -67,6 +67,35 @@ describe('host-owned notification operation receipt', () => { expect( outcomes.filter((outcome) => outcome.status === 'rejected'), ).toHaveLength(1); + const rejected = outcomes.find((outcome) => outcome.status === 'rejected'); + expect(rejected).toMatchObject({ + reason: { + message: 'notification receipt scope differs from this deployment', + }, + }); + const loserPrefix = outcomes[0]?.status === 'rejected' ? 'left_' : 'right_'; + const loserOperation = + outcomes[0]?.status === 'rejected' ? 'operation-left' : 'operation-right'; + const loserNotification = + outcomes[0]?.status === 'rejected' + ? 'notification-left' + : 'notification-right'; + expect( + await db + .prepare( + `SELECT operation_key FROM ${loserPrefix}flowsafe_notification_operation_receipts WHERE operation_key = ?`, + ) + .bind(loserOperation) + .first(), + ).toBeNull(); + expect( + await db + .prepare( + `SELECT id FROM ${loserPrefix}mastra_notifications WHERE id = ?`, + ) + .bind(loserNotification) + .first(), + ).toBeNull(); }); it('retains a prefix and limit across fresh storage instances', async () => { diff --git a/packages/flowsafe/test-support/cloudflare-test-worker.ts b/packages/flowsafe/test-support/cloudflare-test-worker.ts index 23171b69..ce97d6ca 100644 --- a/packages/flowsafe/test-support/cloudflare-test-worker.ts +++ b/packages/flowsafe/test-support/cloudflare-test-worker.ts @@ -32,8 +32,9 @@ function receiptBarrierDatabase( hub: DurableObjectNamespace, barrierId: string, participant: string, + phase: 'receipt-read' | 'scope-census', ): SignalDatabase { - let observedMissing = false; + let barrierEntered = false; function intercept(statement: SignalStatement): SignalStatement { return { bind(...values) { @@ -41,14 +42,19 @@ function receiptBarrierDatabase( }, async first() { const row = await statement.first(); - if (row === null && !observedMissing) { - observedMissing = true; + if (phase === 'receipt-read' && row === null && !barrierEntered) { + barrierEntered = true; await waitForReceiptReaders(db, hub, barrierId, participant); } return row; }, - all() { - return statement.all(); + async all() { + const rows = await statement.all(); + if (phase === 'scope-census' && !barrierEntered) { + barrierEntered = true; + await waitForReceiptReaders(db, hub, barrierId, participant); + } + return rows; }, run() { return statement.run(); @@ -58,7 +64,11 @@ function receiptBarrierDatabase( return { prepare(query) { const statement = db.prepare(query); - return query.includes('SELECT request_digest, thread_id, notification_id') + return ( + phase === 'scope-census' + ? query.includes('SELECT name FROM sqlite_master') + : query.includes('SELECT request_digest, thread_id, notification_id') + ) ? intercept(statement) : statement; }, @@ -128,8 +138,10 @@ export class DemoSignalProviderHost extends TestDurableObject { operationKey: string; notificationId: string; receiptLimit: number; + tablePrefix?: string; barrierId?: string; participant?: string; + barrierPhase?: 'receipt-read' | 'scope-census'; }; const database = input.barrierId && input.participant @@ -138,11 +150,12 @@ export class DemoSignalProviderHost extends TestDurableObject { this.bindings.HUB, input.barrierId, input.participant, + input.barrierPhase ?? 'receipt-read', ) : this.bindings.DB; const notifications = new D1NotificationsStorage( database, - '', + input.tablePrefix ?? '', input.receiptLimit, ); try { From 2f9b137b062c5067fa353ccfb880934c97a63ef0 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 08:36:03 +0400 Subject: [PATCH 65/73] Create receipt scope table during notification initialization Keep the inventory census exact while registering the prefix and limit only on first receipt admission. --- .../flowsafe/src/signals/notifications-d1.ts | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/packages/flowsafe/src/signals/notifications-d1.ts b/packages/flowsafe/src/signals/notifications-d1.ts index 369efe63..0fc1def5 100644 --- a/packages/flowsafe/src/signals/notifications-d1.ts +++ b/packages/flowsafe/src/signals/notifications-d1.ts @@ -574,6 +574,14 @@ export class D1NotificationsStorage )`, ) .run(); + await this.#db + .prepare(`CREATE TABLE IF NOT EXISTS ${NOTIFICATION_RECEIPT_SCOPE_TABLE} ( + id INTEGER PRIMARY KEY CHECK (id = 1), + version INTEGER NOT NULL CHECK (version = 1), + table_prefix TEXT NOT NULL, + receipt_limit INTEGER NOT NULL CHECK (receipt_limit BETWEEN 0 AND ${DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT}) + )`) + .run(); } async #migrateOrdinalSchema(retryDuplicateColumn: boolean): Promise { @@ -742,14 +750,6 @@ export class D1NotificationsStorage if (retained && prefix !== this.#prefix) throw new Error('notification receipt history spans another prefix'); } - await this.#db - .prepare(`CREATE TABLE IF NOT EXISTS ${NOTIFICATION_RECEIPT_SCOPE_TABLE} ( - id INTEGER PRIMARY KEY CHECK (id = 1), - version INTEGER NOT NULL CHECK (version = 1), - table_prefix TEXT NOT NULL, - receipt_limit INTEGER NOT NULL CHECK (receipt_limit BETWEEN 0 AND ${DEFAULT_NOTIFICATION_OPERATION_RECEIPT_LIMIT}) - )`) - .run(); const columns = await this.#db .prepare(`PRAGMA table_info(${NOTIFICATION_RECEIPT_SCOPE_TABLE})`) .all<{ name: string; type: string; notnull: number; pk: number }>(); From 2634b779ccc7fa6c415cf4ab6fa878d850d28214 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 20:26:02 +0400 Subject: [PATCH 66/73] Handle absent FlowSafe staging receipt table at preflight Treat a confirmed missing receipt table as an empty initial state while requiring the real table and receipt after the selected effect. --- .../deploy/notification-staging-control.mjs | 22 ++++++++++--- ...notification-staging-control.node-test.mjs | 32 +++++++++++++++++++ .../notification-staging-live.node-test.mjs | 23 +++++++++---- 3 files changed, 66 insertions(+), 11 deletions(-) diff --git a/packages/flowsafe/deploy/notification-staging-control.mjs b/packages/flowsafe/deploy/notification-staging-control.mjs index ba4bb711..c330fb3e 100644 --- a/packages/flowsafe/deploy/notification-staging-control.mjs +++ b/packages/flowsafe/deploy/notification-staging-control.mjs @@ -81,17 +81,19 @@ export function localProbe({ port, secret, fetcher = fetch }) { export function trustedD1Checks(query) { let baseline; - const counts = async () => { + const counts = async (receiptTableExists = true) => { const rows = await query( - "SELECT (SELECT COUNT(*) FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification') AS notifications, (SELECT COUNT(*) FROM flowsafe_notification_operation_receipts) AS receipts", + receiptTableExists + ? "SELECT (SELECT COUNT(*) FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification') AS notifications, (SELECT COUNT(*) FROM flowsafe_notification_operation_receipts) AS receipts" + : "SELECT COUNT(*) AS notifications FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification'", ); assert( rows.length === 1 && Number.isSafeInteger(rows[0].notifications) && - Number.isSafeInteger(rows[0].receipts), + (receiptTableExists ? Number.isSafeInteger(rows[0].receipts) : true), 'notification baseline is unreadable', ); - return rows[0]; + return receiptTableExists ? rows[0] : { ...rows[0], receipts: 0 }; }; return { preflightD1: async () => { @@ -112,7 +114,17 @@ export function trustedD1Checks(query) { rows.length === 1 && rows[0].count === 0, 'remote step journal is not empty', ); - baseline = await counts(); + const receiptSchema = await query( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'flowsafe_notification_operation_receipts'", + ); + assert( + receiptSchema.length === 0 || + (receiptSchema.length === 1 && + receiptSchema[0].name === + 'flowsafe_notification_operation_receipts'), + 'receipt schema is unreadable', + ); + baseline = await counts(receiptSchema.length === 1); }, verifyD1: async (invocationId) => { assert( diff --git a/packages/flowsafe/deploy/notification-staging-control.node-test.mjs b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs index dae426c5..739ba22b 100644 --- a/packages/flowsafe/deploy/notification-staging-control.node-test.mjs +++ b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs @@ -225,3 +225,35 @@ test('D1 proof requires the receipt tuple and one new notification and receipt', db.close(); } }); + +test('D1 preflight accepts an absent receipt table but proof requires its effect', async () => { + const db = new DatabaseSync(':memory:'); + try { + db.exec(`CREATE TABLE flowsafe_remote_step_invocations ( + invocation_id TEXT PRIMARY KEY, status TEXT, settlement_outcome TEXT, + operation_state TEXT, thread_id TEXT, notification_id TEXT, + operation_key TEXT, notification_request_digest TEXT, + write_generation INTEGER NOT NULL + ); + CREATE TABLE mastra_notifications ( + id TEXT, thread_id TEXT, source TEXT, kind TEXT + );`); + const checks = trustedD1Checks((sql) => db.prepare(sql).all()); + await checks.preflightD1(); + db.exec(`INSERT INTO flowsafe_remote_step_invocations VALUES + ('invocation-1', 'settled', 'applied', 'resume-admitted', + 'thread-1', 'notification-1', 'operation-1', '${'a'.repeat(64)}', 1); + INSERT INTO mastra_notifications VALUES + ('notification-1', 'thread-1', 'flowsafe-staging', 'host-owned-notification');`); + await assert.rejects(checks.verifyD1('invocation-1'), /no such table/); + db.exec(`CREATE TABLE flowsafe_notification_operation_receipts ( + operation_key TEXT PRIMARY KEY, request_digest TEXT, + thread_id TEXT, notification_id TEXT + ); + INSERT INTO flowsafe_notification_operation_receipts VALUES + ('operation-1', '${'a'.repeat(64)}', 'thread-1', 'notification-1');`); + await checks.verifyD1('invocation-1'); + } finally { + db.close(); + } +}); diff --git a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs index 236701fb..19f7ebec 100644 --- a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs @@ -221,6 +221,7 @@ async function exerciseWatchdog(outcome) { const watcherPidFile = join(directory, 'watchdog-pid'); const deployStartedFile = join(directory, 'deploy-started'); const releaseDeployFile = join(directory, 'release-deploy'); + const effectStartedFile = join(directory, 'effect-started'); const accountId = outcome === 'parent death' ? 'c'.repeat(32) : 'd'.repeat(32); const lockFile = join( @@ -242,7 +243,6 @@ Date.now = () => originalNow() + (process.env.TEST_CLOCK_JUMP === 'true' && !process.argv.includes('--watchdog') && existsSync(process.env.TEST_DEPLOY_STARTED_FILE) ? 300_000 : 0); -let counts = 0; childProcess.spawn = (...args) => { const child = originalSpawn(...args); if (args[1]?.includes('--watchdog')) @@ -257,6 +257,11 @@ childProcess.spawnSync = (executable, args, options) => { const command = args[0]; if (off && command === 'deployments') writeFileSync(process.env.TEST_OFF_CONFIRMED_FILE, 'checked'); + const sql = args.at(-1); + const effectStarted = existsSync(process.env.TEST_EFFECT_STARTED_FILE); + if (command === 'd1' && !effectStarted && + sql.includes('FROM flowsafe_notification_operation_receipts')) + return { status: 1, stderr: 'no such table: flowsafe_notification_operation_receipts' }; const response = command === 'deployments' ? [{ created_on: '2026-10-15T00:00:00.000Z', versions: [{ version_id: off ? '${disabled}' : '${enabled}', percentage: 100 }], @@ -266,11 +271,13 @@ childProcess.spawnSync = (executable, args, options) => { text: off ? 'false' : 'true', }] }, } : command === 'd1' ? [{ success: true, results: - args.at(-1).includes('sqlite_master') - ? [{ sql: 'write_generation INTEGER NOT NULL, operation_state TEXT' }] - : args.at(-1).includes('COUNT(*) AS count') + sql.includes("name = 'flowsafe_notification_operation_receipts'") + ? effectStarted ? [{ name: 'flowsafe_notification_operation_receipts' }] : [] + : sql.includes('sqlite_master') + ? [{ sql: 'write_generation INTEGER NOT NULL, operation_state TEXT' }] + : sql.includes('COUNT(*) AS count') ? [{ count: 0 }] - : args.at(-1).includes('FROM flowsafe_remote_step_invocations i') + : sql.includes('FROM flowsafe_remote_step_invocations i') ? [{ status: 'settled', settlement_outcome: 'applied', operation_state: 'resume-admitted', thread_id: 'thread', notification_id: 'notification', receipt_key: 'key', @@ -278,7 +285,9 @@ childProcess.spawnSync = (executable, args, options) => { receipt_notification_id: 'notification', stored_notification_id: 'notification', source: 'flowsafe-staging', kind: 'host-owned-notification' }] - : [{ notifications: counts, receipts: counts++ }], + : effectStarted + ? [{ notifications: 1, receipts: 1 }] + : [{ notifications: 0 }], }] : null; if (command === 'deploy') { writeFileSync(process.env.TEST_DEPLOY_STARTED_FILE, 'started'); @@ -309,6 +318,7 @@ syncBuiltinESMExports(); requests.push(`${method} ${path}`); if (method === 'POST' && path === '/start') { startReceived = true; + writeFileSync(effectStartedFile, 'started'); if (outcome === 'parent death') return; response.writeHead(200, { 'content-type': 'application/json' }); response.end( @@ -371,6 +381,7 @@ syncBuiltinESMExports(); TEST_WATCHDOG_PID_FILE: watcherPidFile, TEST_DEPLOY_STARTED_FILE: deployStartedFile, TEST_RELEASE_DEPLOY_FILE: releaseDeployFile, + TEST_EFFECT_STARTED_FILE: effectStartedFile, TEST_CLOCK_JUMP: outcome === 'controller request' ? 'true' : 'false', }, stdio: 'ignore', From ba74213c31ecc7f82cc92d9f5982103791c34a13 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 22:16:44 +0400 Subject: [PATCH 67/73] Keep REST SDK out of FlowSafe staging startup Alias the unused Cloudflare REST constructor to a fail-closed staging stub so binding-backed D1 proof can start within the Worker budget. --- packages/flowsafe/deploy/notification-staging-no-rest.ts | 7 +++++++ .../flowsafe/deploy/notification-staging.wrangler.jsonc | 3 ++- 2 files changed, 9 insertions(+), 1 deletion(-) create mode 100644 packages/flowsafe/deploy/notification-staging-no-rest.ts diff --git a/packages/flowsafe/deploy/notification-staging-no-rest.ts b/packages/flowsafe/deploy/notification-staging-no-rest.ts new file mode 100644 index 00000000..697fcf6a --- /dev/null +++ b/packages/flowsafe/deploy/notification-staging-no-rest.ts @@ -0,0 +1,7 @@ +export default class UnavailableStagingRestClient { + constructor() { + throw new Error( + 'Cloudflare REST client is unavailable in notification staging', + ); + } +} diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc index 90e6bf28..06e591a2 100644 --- a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -5,7 +5,8 @@ "compatibility_date": "2026-09-10", "compatibility_flags": ["nodejs_compat"], "alias": { - "@proofoftech/breakwater/agent": "../../breakwater/src/agent/index.ts" + "@proofoftech/breakwater/agent": "../../breakwater/src/agent/index.ts", + "cloudflare": "./notification-staging-no-rest.ts" }, "workers_dev": false, "secrets": { From dbfd57cf74ae6d8367071172d69539eae5b53446 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 22:34:25 +0400 Subject: [PATCH 68/73] Recover empty FlowSafe staging proof after sealed fence Accept lazily absent notification tables before the selected effect, and allow one exact monotonic retry from a verified off 1/1 seal with watchdog rollback. --- packages/flowsafe/deploy/README.md | 5 +- .../deploy/notification-staging-control.mjs | 83 ++++++++++---- ...notification-staging-control.node-test.mjs | 61 ++++++++-- .../deploy/notification-staging-live.mjs | 107 ++++++++++++------ .../notification-staging-live.node-test.mjs | 102 ++++++++++++++--- 5 files changed, 273 insertions(+), 85 deletions(-) diff --git a/packages/flowsafe/deploy/README.md b/packages/flowsafe/deploy/README.md index 3c6658bf..f059139d 100644 --- a/packages/flowsafe/deploy/README.md +++ b/packages/flowsafe/deploy/README.md @@ -218,7 +218,9 @@ See the repository [deployment reference](../../../docs/deployment-reference.md) ## Run the isolated notification staging control -Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The script refuses to unlock unless the fence is `migration-locked` at epoch and revision zero, the v5 journal is empty, and the recovery alarm is armed. It acquires a Linux `flock` keyed by Cloudflare account and Worker before reading the enabled version. Its detached watchdog inherits the same locked open file descriptor, seals or relocks the fence after controller death or its 15-minute deadline, and performs the tracked off deployment on controller request or recovery. Sealing advances the initially locked fence's revision, so a delayed controller unlock cannot succeed after watchdog shutdown starts. The controller never deploys off itself. +Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The initial run requires a `migration-locked` fence at epoch and revision zero, an empty v5 journal, and an armed recovery alarm. It acquires a Linux `flock` keyed by Cloudflare account and Worker before reading the enabled version. Its detached watchdog inherits the same locked open file descriptor, seals or relocks the fence after controller death or its 15-minute deadline, and performs the tracked off deployment on controller request or recovery. Sealing advances the locked fence's revision, so a delayed controller unlock cannot succeed after watchdog shutdown starts. The controller never deploys off itself. + +A failed preflight may leave the fence sealed at `migration-locked`, epoch and revision `1`, with `requireMutationEpoch=true`, even though no effect ran. An operator may retry this exact state once: verify the prior run's tracked off Worker version, confirm the remote-step journal is empty and the notification and receipt tables are absent, deploy a fresh enabled version under the same lock, and set `FLOWSAFE_STAGING_RETRY_OFF_VERSION` to that immediately prior 100% off deployment ID. The controller checks deployment history, the prior version's off flag, the exact sealed fence and the D1 preflight before unlocking. It persists the selected baseline in private watchdog state before readiness. The retry opens at epoch and revision `2`; the watchdog seals a still-locked fence at `2` or relocks an open fence at `3`. A different fence or a second sealed retry requires operator review; do not update the D1 fence row directly. Use the same lock for **every** command that can upload, deploy, promote, or change traffic for `anchorage-flowsafe-remote-step-staging`. From `packages/flowsafe`, define the wrapper below and use it for the initial enabled deployment and any other same-Worker deployment command: @@ -247,5 +249,6 @@ Set these environment variables before running `node deploy/notification-staging | `FLOWSAFE_CF_TOKEN_FILE` | Absolute path to the Cloudflare API token in a `0600` file | | `FLOWSAFE_CF_ACCOUNT_ID` | Cloudflare account ID as 32 lowercase hexadecimal characters; the same value used by the deployment wrapper | | `FLOWSAFE_STAGING_ENABLED_VERSION` | Exact 100% enabled subject Worker version ID | +| `FLOWSAFE_STAGING_RETRY_OFF_VERSION` | Retry only: immediately prior 100% off Worker version ID; omit for an initial run | The controller prints the invocation ID only after D1 confirms one settled invocation, its notification, its operation receipt, and the watchdog's confirmed off deployment. It writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the controller exits without a confirmed off deployment, inspect that private directory's `watchdog.log` and verify the fence and current Worker version through the authenticated proxy and Wrangler before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. diff --git a/packages/flowsafe/deploy/notification-staging-control.mjs b/packages/flowsafe/deploy/notification-staging-control.mjs index c330fb3e..1b2118eb 100644 --- a/packages/flowsafe/deploy/notification-staging-control.mjs +++ b/packages/flowsafe/deploy/notification-staging-control.mjs @@ -6,6 +6,12 @@ const initialFence = { requireMutationEpoch: false, transitionRevision: 0, }; +export const sealedRetryFence = { + state: 'migration-locked', + mutationEpoch: 1, + requireMutationEpoch: true, + transitionRevision: 1, +}; function assert(condition, message) { if (!condition) throw new Error(message); @@ -81,22 +87,33 @@ export function localProbe({ port, secret, fetcher = fetch }) { export function trustedD1Checks(query) { let baseline; - const counts = async (receiptTableExists = true) => { - const rows = await query( - receiptTableExists - ? "SELECT (SELECT COUNT(*) FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification') AS notifications, (SELECT COUNT(*) FROM flowsafe_notification_operation_receipts) AS receipts" - : "SELECT COUNT(*) AS notifications FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification'", - ); + const count = async (sql) => { + const rows = await query(sql); assert( - rows.length === 1 && - Number.isSafeInteger(rows[0].notifications) && - (receiptTableExists ? Number.isSafeInteger(rows[0].receipts) : true), + rows.length === 1 && Number.isSafeInteger(rows[0].count), 'notification baseline is unreadable', ); - return receiptTableExists ? rows[0] : { ...rows[0], receipts: 0 }; + return rows[0].count; + }; + const counts = async ({ + notificationTableExists = true, + receiptTableExists = true, + } = {}) => { + return { + notifications: notificationTableExists + ? await count( + "SELECT COUNT(*) AS count FROM mastra_notifications WHERE source = 'flowsafe-staging' AND kind = 'host-owned-notification'", + ) + : 0, + receipts: receiptTableExists + ? await count( + 'SELECT COUNT(*) AS count FROM flowsafe_notification_operation_receipts', + ) + : 0, + }; }; return { - preflightD1: async () => { + preflightD1: async ({ requireAbsentEffectTables = false } = {}) => { const schema = await query( "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'flowsafe_remote_step_invocations'", ); @@ -114,17 +131,29 @@ export function trustedD1Checks(query) { rows.length === 1 && rows[0].count === 0, 'remote step journal is not empty', ); - const receiptSchema = await query( - "SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'flowsafe_notification_operation_receipts'", + const notificationSchemas = await query( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')", ); + const present = new Set(notificationSchemas.map((row) => row.name)); assert( - receiptSchema.length === 0 || - (receiptSchema.length === 1 && - receiptSchema[0].name === - 'flowsafe_notification_operation_receipts'), - 'receipt schema is unreadable', + notificationSchemas.length === present.size && + [...present].every( + (name) => + name === 'mastra_notifications' || + name === 'flowsafe_notification_operation_receipts', + ), + 'notification schema is unreadable', ); - baseline = await counts(receiptSchema.length === 1); + assert( + !requireAbsentEffectTables || present.size === 0, + 'retry effect tables are present', + ); + baseline = await counts({ + notificationTableExists: present.has('mastra_notifications'), + receiptTableExists: present.has( + 'flowsafe_notification_operation_receipts', + ), + }); }, verifyD1: async (invocationId) => { assert( @@ -177,18 +206,26 @@ export async function runNotificationStagingControl({ verifyD1, requestOffDeploy, onBeforeUnlock = async () => {}, + expectedInitialFence = initialFence, }) { assert( typeof requestOffDeploy === 'function', 'off deploy watchdog is required', ); + assert( + expectedInitialFence === initialFence || + Object.entries(sealedRetryFence).every( + ([key, value]) => expectedInitialFence?.[key] === value, + ), + 'unsupported staging fence baseline', + ); let initial; let unlockAttempted = false; let outcome; let failure; try { initial = await probe('GET', '/admin/execution-fence'); - exactFence(initial, initialFence); + exactFence(initial, expectedInitialFence); const maintenance = await probe('GET', '/admin/maintenance-status'); assert( Number.isSafeInteger(maintenance.alarmAt) && @@ -198,7 +235,9 @@ export async function runNotificationStagingControl({ maintenance.lastRemoteStepRecoveryError === undefined, 'notification recovery duty is not armed', ); - await preflightD1(); + await preflightD1({ + requireAbsentEffectTables: expectedInitialFence.mutationEpoch === 1, + }); await onBeforeUnlock(); unlockAttempted = true; const opened = await probe('POST', '/admin/execution-fence', { @@ -257,7 +296,7 @@ export async function runNotificationStagingControl({ }); exactFence(locked, nextFence(current, 'migration-locked')); } else { - exactFence(current, initialFence); + exactFence(current, expectedInitialFence); assert(!outcome, 'successful proof lost its open fence'); } } catch (error) { diff --git a/packages/flowsafe/deploy/notification-staging-control.node-test.mjs b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs index 739ba22b..78bb8677 100644 --- a/packages/flowsafe/deploy/notification-staging-control.node-test.mjs +++ b/packages/flowsafe/deploy/notification-staging-control.node-test.mjs @@ -11,13 +11,13 @@ import { trustedD1Checks, } from './notification-staging-control.mjs'; -function fixture(failAt, initialState = 'migration-locked') { +function fixture(failAt, initialState = 'migration-locked', initialEpoch = 0) { const calls = []; let fence = { state: initialState, - mutationEpoch: 0, - requireMutationEpoch: false, - transitionRevision: 0, + mutationEpoch: initialEpoch, + requireMutationEpoch: initialEpoch !== 0, + transitionRevision: initialEpoch, }; const probe = async (method, path, body) => { calls.push({ method, path, body }); @@ -118,6 +118,38 @@ test('a successful proof verifies before relock and requests off deploy', async assert.equal(scenario.offRequests(), 1); }); +test('explicit sealed retry advances the fence monotonically', async () => { + const scenario = fixture(null, 'migration-locked', 1); + const expectedInitialFence = { + state: 'migration-locked', + mutationEpoch: 1, + requireMutationEpoch: true, + transitionRevision: 1, + }; + assert.deepEqual( + await runNotificationStagingControl({ + ...scenario.options, + expectedInitialFence, + }), + { invocationId: 'invocation-1', verdict: 'notification_staging_passed' }, + ); + assert.deepEqual(scenario.fence(), { + state: 'migration-locked', + mutationEpoch: 3, + requireMutationEpoch: true, + transitionRevision: 3, + }); + assert.deepEqual( + scenario.calls + .filter( + (call) => + call.method === 'POST' && call.path === '/admin/execution-fence', + ) + .map((call) => call.body.expectedRevision), + [1, 2], + ); +}); + test('a conflicting fence is not overwritten while off deploy is still requested', async () => { const scenario = fixture(); scenario.options.verifyD1 = async () => { @@ -194,6 +226,10 @@ test('D1 proof requires the receipt tuple and one new notification and receipt', id TEXT, thread_id TEXT, source TEXT, kind TEXT );`); const checks = trustedD1Checks((sql) => db.prepare(sql).all()); + await assert.rejects( + checks.preflightD1({ requireAbsentEffectTables: true }), + /retry effect tables are present/, + ); await checks.preflightD1(); db.exec(`INSERT INTO flowsafe_remote_step_invocations VALUES ('invocation-1', 'settled', 'applied', 'resume-admitted', @@ -226,7 +262,7 @@ test('D1 proof requires the receipt tuple and one new notification and receipt', } }); -test('D1 preflight accepts an absent receipt table but proof requires its effect', async () => { +test('D1 preflight accepts absent notification tables but proof requires the effect', async () => { const db = new DatabaseSync(':memory:'); try { db.exec(`CREATE TABLE flowsafe_remote_step_invocations ( @@ -234,19 +270,20 @@ test('D1 preflight accepts an absent receipt table but proof requires its effect operation_state TEXT, thread_id TEXT, notification_id TEXT, operation_key TEXT, notification_request_digest TEXT, write_generation INTEGER NOT NULL - ); - CREATE TABLE mastra_notifications ( - id TEXT, thread_id TEXT, source TEXT, kind TEXT );`); const checks = trustedD1Checks((sql) => db.prepare(sql).all()); + await checks.preflightD1({ requireAbsentEffectTables: true }); await checks.preflightD1(); db.exec(`INSERT INTO flowsafe_remote_step_invocations VALUES ('invocation-1', 'settled', 'applied', 'resume-admitted', - 'thread-1', 'notification-1', 'operation-1', '${'a'.repeat(64)}', 1); - INSERT INTO mastra_notifications VALUES - ('notification-1', 'thread-1', 'flowsafe-staging', 'host-owned-notification');`); + 'thread-1', 'notification-1', 'operation-1', '${'a'.repeat(64)}', 1);`); await assert.rejects(checks.verifyD1('invocation-1'), /no such table/); - db.exec(`CREATE TABLE flowsafe_notification_operation_receipts ( + db.exec(`CREATE TABLE mastra_notifications ( + id TEXT, thread_id TEXT, source TEXT, kind TEXT + ); + INSERT INTO mastra_notifications VALUES + ('notification-1', 'thread-1', 'flowsafe-staging', 'host-owned-notification'); + CREATE TABLE flowsafe_notification_operation_receipts ( operation_key TEXT PRIMARY KEY, request_digest TEXT, thread_id TEXT, notification_id TEXT ); diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs index bca938d9..221f9a01 100644 --- a/packages/flowsafe/deploy/notification-staging-live.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -18,6 +18,7 @@ import { localProbe, readSecretFile, runNotificationStagingControl, + sealedRetryFence, trustedD1Checks, } from './notification-staging-control.mjs'; @@ -98,19 +99,10 @@ export function createWranglerControl({ command = spawnSync, token }) { assert(value !== null, 'Wrangler returned null'); return value; } - function currentVersion() { - const deployments = json([ - 'deployments', - 'list', - '--config', - config, - '--json', - ]); - assert( - Array.isArray(deployments) && deployments.length > 0, - 'deployment list is empty', - ); - const ordered = deployments + function deployments() { + const list = json(['deployments', 'list', '--config', config, '--json']); + assert(Array.isArray(list) && list.length > 0, 'deployment list is empty'); + const ordered = list .map((deployment) => { const createdAt = Date.parse(deployment.created_on); assert( @@ -124,15 +116,21 @@ export function createWranglerControl({ command = spawnSync, token }) { ordered.length === 1 || ordered[0].createdAt > ordered[1].createdAt, 'latest deployment is ambiguous', ); - const latest = ordered[0].deployment; + return ordered.map(({ deployment }) => deployment); + } + function fullVersion(deployment) { assert( - latest.versions?.length === 1 && latest.versions[0].percentage === 100, + deployment.versions?.length === 1 && + deployment.versions[0].percentage === 100, 'deployment is not a single full version', ); - const id = latest.versions[0].version_id; + const id = deployment.versions[0].version_id; assert(versionPattern.test(id), 'invalid deployment version'); return id; } + function currentVersion() { + return fullVersion(deployments()[0]); + } function proofEnabled(version) { assert(versionPattern.test(version), 'invalid version'); const detail = json([ @@ -157,6 +155,16 @@ export function createWranglerControl({ command = spawnSync, token }) { return { currentVersion, proofEnabled, + priorOffVersion(expectedEnabled) { + const ordered = deployments(); + assert( + ordered.length > 1 && fullVersion(ordered[0]) === expectedEnabled, + 'retry deployment history differs', + ); + const prior = fullVersion(ordered[1]); + assert(!proofEnabled(prior), 'prior deployment is not off'); + return prior; + }, query(sql) { const output = json([ 'd1', @@ -206,7 +214,11 @@ export function createWranglerControl({ command = spawnSync, token }) { }; } -export async function sealFence(probe) { +export async function sealFence(probe, baselineEpoch = 0) { + assert( + baselineEpoch === 0 || baselineEpoch === 1, + 'invalid watchdog baseline', + ); for (let attempt = 0; attempt < 3; attempt++) { const current = await probe('GET', '/admin/execution-fence'); assert( @@ -215,16 +227,16 @@ export async function sealFence(probe) { ); const initial = current.state === 'migration-locked' && - current.mutationEpoch === 0 && - current.transitionRevision === 0 && - current.requireMutationEpoch === false; + current.mutationEpoch === baselineEpoch && + current.transitionRevision === baselineEpoch && + current.requireMutationEpoch === (baselineEpoch !== 0); const open = current.state === 'open' && - current.mutationEpoch === 1 && - current.transitionRevision === 1 && + current.mutationEpoch === baselineEpoch + 1 && + current.transitionRevision === baselineEpoch + 1 && current.requireMutationEpoch === true; if (current.state === 'migration-locked' && !initial) { - confirmedFence(current); + confirmedFence(current, baselineEpoch); return current; } assert(initial || open, 'watchdog fence metadata differs'); @@ -241,7 +253,7 @@ export async function sealFence(probe) { } } const current = await probe('GET', '/admin/execution-fence'); - confirmedFence(current); + confirmedFence(current, baselineEpoch); return current; } @@ -263,6 +275,11 @@ function settingsFromEnv(env) { assert(token.length > 0, 'Cloudflare token is empty'); const accountId = env.FLOWSAFE_CF_ACCOUNT_ID; assert(accountPattern.test(accountId), 'Cloudflare account ID is required'); + const priorOffVersion = env.FLOWSAFE_STAGING_RETRY_OFF_VERSION; + assert( + priorOffVersion === undefined || versionPattern.test(priorOffVersion), + 'invalid prior off version', + ); return { port, expectedVersion, @@ -271,6 +288,7 @@ function settingsFromEnv(env) { secret, token, accountId, + priorOffVersion, }; } @@ -302,18 +320,18 @@ function acquireDeploymentLock(accountId) { return fd; } -function confirmedFence(current) { +function confirmedFence(current, baselineEpoch = 0) { assert( current.state === 'migration-locked', 'staging fence relock is unconfirmed', ); const sealed = - current.mutationEpoch === 1 && - current.transitionRevision === 1 && + current.mutationEpoch === baselineEpoch + 1 && + current.transitionRevision === baselineEpoch + 1 && current.requireMutationEpoch === true; const relocked = - current.mutationEpoch === 2 && - current.transitionRevision === 2 && + current.mutationEpoch === baselineEpoch + 2 && + current.transitionRevision === baselineEpoch + 2 && current.requireMutationEpoch === true; assert(sealed || relocked, 'staging fence metadata differs'); } @@ -330,7 +348,12 @@ async function watchdog(stateFile) { parentPid, deadlineAt, outdir, + baselineEpoch, } = state; + assert( + baselineEpoch === 0 || baselineEpoch === 1, + 'invalid watchdog baseline', + ); const probe = localProbe({ port, secret: readSecretFile(probeSecretPath) }); const cli = createWranglerControl({ token: privateFile(cfTokenPath) }); assert( @@ -347,9 +370,9 @@ async function watchdog(stateFile) { } await new Promise((resolve) => setTimeout(resolve, 1_000)); } - await sealFence(probe); + await sealFence(probe, baselineEpoch); cli.deployOffIfCurrent(expectedVersion, outdir); - confirmedFence(await probe('GET', '/admin/execution-fence')); + confirmedFence(await probe('GET', '/admin/execution-fence'), baselineEpoch); writeFileSync(`${stateFile}.off-confirmed`, '', { mode: 0o600, flag: 'wx' }); } @@ -362,6 +385,25 @@ async function controller() { cli.proofEnabled(settings.expectedVersion), 'enabled version preflight differs', ); + const probe = localProbe({ port: settings.port, secret: settings.secret }); + const baselineEpoch = settings.priorOffVersion === undefined ? 0 : 1; + if (baselineEpoch === 1) { + assert( + settings.priorOffVersion !== settings.expectedVersion && + cli.priorOffVersion(settings.expectedVersion) === + settings.priorOffVersion, + 'retry prior off version is unconfirmed', + ); + const fence = await probe('GET', '/admin/execution-fence'); + assert( + Object.entries(sealedRetryFence).every( + ([key, value]) => fence[key] === value, + ) && + fence.proofKey === undefined && + fence.proofRunId === undefined, + 'retry fence is not the exact sealed baseline', + ); + } const directory = join( tmpdir(), `flowsafe-notification-control-${process.pid}`, @@ -380,6 +422,7 @@ async function controller() { parentPid: process.pid, deadlineAt: Date.now() + 900_000, outdir, + baselineEpoch, }), { mode: 0o600, flag: 'wx' }, ); @@ -406,7 +449,6 @@ async function controller() { 'watchdog readiness is unconfirmed', ); process.kill(watcher.pid, 0); - const probe = localProbe({ port: settings.port, secret: settings.secret }); let offConfirmed = false; const requestOffDeploy = async () => { if (offConfirmed) return; @@ -427,6 +469,7 @@ async function controller() { const result = await runNotificationStagingControl({ probe, ...trustedD1Checks((sql) => cli.query(sql)), + expectedInitialFence: baselineEpoch === 1 ? sealedRetryFence : undefined, requestOffDeploy, onBeforeUnlock: async () => { assert( diff --git a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs index 19f7ebec..dab3e287 100644 --- a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs @@ -21,7 +21,7 @@ import { const enabled = 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa'; const disabled = 'bbbbbbbb-bbbb-4bbb-bbbb-bbbbbbbbbbbb'; -function fakeCli({ enabledFlag = true } = {}) { +function fakeCli({ enabledFlag = true, priorVersion } = {}) { const calls = []; let version = enabled; const command = (_executable, args) => { @@ -34,6 +34,14 @@ function fakeCli({ enabledFlag = true } = {}) { created_on: '2026-10-15T00:00:00.000Z', versions: [{ version_id: version, percentage: 100 }], }, + ...(priorVersion + ? [ + { + created_on: '2026-10-14T23:59:00.000Z', + versions: [{ version_id: priorVersion, percentage: 100 }], + }, + ] + : []), ]), }; } @@ -46,7 +54,7 @@ function fakeCli({ enabledFlag = true } = {}) { { name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', - text: version === enabled && enabledFlag ? 'true' : 'false', + text: args[2] === enabled && enabledFlag ? 'true' : 'false', }, ], }, @@ -106,6 +114,15 @@ test('changed version or disabled flag refuses before deploy', () => { ); }); +test('retry evidence requires the immediately prior full off deployment', () => { + const prior = fakeCli({ priorVersion: disabled }); + assert.equal(prior.cli.priorOffVersion(enabled), disabled); + const absent = fakeCli(); + assert.throws(() => absent.cli.priorOffVersion(enabled), /history differs/); + const enabledPrior = fakeCli({ priorVersion: enabled }); + assert.throws(() => enabledPrior.cli.priorOffVersion(enabled), /not off/); +}); + test('Wrangler failure preserves exit and stderr in its diagnostic cause', () => { const cli = createWranglerControl({ token: 'fake-token', @@ -151,6 +168,33 @@ test('watchdog seals the initial fence against a delayed controller open', async assert.equal(fence.transitionRevision, 1); }); +test('watchdog seals the explicit retry baseline against a delayed controller open', async () => { + let fence = { + state: 'migration-locked', + mutationEpoch: 1, + transitionRevision: 1, + requireMutationEpoch: true, + }; + const probe = async (method, _path, body) => { + if (method === 'GET') return fence; + assert.deepEqual(body, { + expected: 'migration-locked', + next: 'migration-locked', + expectedMutationEpoch: 1, + expectedRevision: 1, + advanceMutationEpoch: true, + }); + fence = { + state: 'migration-locked', + mutationEpoch: 2, + transitionRevision: 2, + requireMutationEpoch: true, + }; + return fence; + }; + assert.deepEqual(await sealFence(probe, 1), fence); +}); + test('watchdog relocks when controller open wins the deadline CAS', async () => { let fence = { state: 'migration-locked', @@ -223,7 +267,11 @@ async function exerciseWatchdog(outcome) { const releaseDeployFile = join(directory, 'release-deploy'); const effectStartedFile = join(directory, 'effect-started'); const accountId = - outcome === 'parent death' ? 'c'.repeat(32) : 'd'.repeat(32); + outcome === 'parent death' + ? 'c'.repeat(32) + : outcome === 'sealed retry' + ? 'e'.repeat(32) + : 'd'.repeat(32); const lockFile = join( '/tmp', `flowsafe-deploy-locks-${process.getuid()}`, @@ -260,23 +308,31 @@ childProcess.spawnSync = (executable, args, options) => { const sql = args.at(-1); const effectStarted = existsSync(process.env.TEST_EFFECT_STARTED_FILE); if (command === 'd1' && !effectStarted && - sql.includes('FROM flowsafe_notification_operation_receipts')) - return { status: 1, stderr: 'no such table: flowsafe_notification_operation_receipts' }; + (sql.includes('FROM flowsafe_notification_operation_receipts') || + sql.includes('FROM mastra_notifications'))) + return { status: 1, stderr: 'notification table is absent' }; const response = command === 'deployments' ? [{ created_on: '2026-10-15T00:00:00.000Z', versions: [{ version_id: off ? '${disabled}' : '${enabled}', percentage: 100 }], - }] : command === 'versions' ? { + }, ...(process.env.TEST_RETRY === 'true' ? [{ + created_on: '2026-10-14T23:59:00.000Z', + versions: [{ version_id: '${disabled}', percentage: 100 }], + }] : [])] : command === 'versions' ? { resources: { bindings: [{ name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', - text: off ? 'false' : 'true', + text: off || args[2] === '${disabled}' ? 'false' : 'true', }] }, } : command === 'd1' ? [{ success: true, results: - sql.includes("name = 'flowsafe_notification_operation_receipts'") - ? effectStarted ? [{ name: 'flowsafe_notification_operation_receipts' }] : [] + sql.includes("name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')") + ? effectStarted ? [{ name: 'mastra_notifications' }, + { name: 'flowsafe_notification_operation_receipts' }] : [] : sql.includes('sqlite_master') ? [{ sql: 'write_generation INTEGER NOT NULL, operation_state TEXT' }] - : sql.includes('COUNT(*) AS count') + : sql.includes('COUNT(*) AS count FROM flowsafe_remote_step_invocations') ? [{ count: 0 }] + : sql.includes('COUNT(*) AS count FROM mastra_notifications') || + sql.includes('COUNT(*) AS count FROM flowsafe_notification_operation_receipts') + ? [{ count: effectStarted ? 1 : 0 }] : sql.includes('FROM flowsafe_remote_step_invocations i') ? [{ status: 'settled', settlement_outcome: 'applied', operation_state: 'resume-admitted', thread_id: 'thread', @@ -285,9 +341,7 @@ childProcess.spawnSync = (executable, args, options) => { receipt_notification_id: 'notification', stored_notification_id: 'notification', source: 'flowsafe-staging', kind: 'host-owned-notification' }] - : effectStarted - ? [{ notifications: 1, receipts: 1 }] - : [{ notifications: 0 }], + : [], }] : null; if (command === 'deploy') { writeFileSync(process.env.TEST_DEPLOY_STARTED_FILE, 'started'); @@ -306,9 +360,9 @@ syncBuiltinESMExports(); const requests = []; let fence = { state: 'migration-locked', - mutationEpoch: 0, - requireMutationEpoch: false, - transitionRevision: 0, + mutationEpoch: outcome === 'sealed retry' ? 1 : 0, + requireMutationEpoch: outcome === 'sealed retry', + transitionRevision: outcome === 'sealed retry' ? 1 : 0, }; let startReceived = false; const server = createServer(async (request, response) => { @@ -373,6 +427,9 @@ syncBuiltinESMExports(); TMPDIR: directory, FLOWSAFE_STAGING_PROBE_PORT: String(port), FLOWSAFE_STAGING_ENABLED_VERSION: enabled, + ...(outcome === 'sealed retry' + ? { FLOWSAFE_STAGING_RETRY_OFF_VERSION: disabled } + : {}), FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, FLOWSAFE_CF_TOKEN_FILE: tokenFile, FLOWSAFE_CF_ACCOUNT_ID: accountId, @@ -383,6 +440,7 @@ syncBuiltinESMExports(); TEST_RELEASE_DEPLOY_FILE: releaseDeployFile, TEST_EFFECT_STARTED_FILE: effectStartedFile, TEST_CLOCK_JUMP: outcome === 'controller request' ? 'true' : 'false', + TEST_RETRY: outcome === 'sealed retry' ? 'true' : 'false', }, stdio: 'ignore', }, @@ -400,6 +458,11 @@ syncBuiltinESMExports(); } }; await waitFor(() => existsSync(`${stateFile}.ready`)); + if (outcome === 'sealed retry') + assert.equal( + JSON.parse(readFileSync(stateFile, 'utf8')).baselineEpoch, + 1, + ); await waitFor(() => existsSync(watcherPidFile)); watcherPid = Number(readFileSync(watcherPidFile, 'utf8')); await waitFor(() => startReceived); @@ -418,9 +481,9 @@ syncBuiltinESMExports(); assert.equal(readFileSync(offFile, 'utf8'), 'deployed'); assert.deepEqual(fence, { state: 'migration-locked', - mutationEpoch: 2, + mutationEpoch: outcome === 'sealed retry' ? 3 : 2, requireMutationEpoch: true, - transitionRevision: 2, + transitionRevision: outcome === 'sealed retry' ? 3 : 2, }); assert.deepEqual( requests.filter((request) => request === 'POST /admin/execution-fence'), @@ -454,6 +517,9 @@ test('watchdog owns off deployment after parent death', () => test('watchdog owns off deployment after controller request', () => exerciseWatchdog('controller request')); +test('sealed retry persists watchdog baseline before unlocking and relocks at three', () => + exerciseWatchdog('sealed retry')); + test('controller refuses a contended deployment lock before Wrangler preflight', async () => { const directory = mkdtempSync( join(tmpdir(), 'flowsafe-lock-contention-test-'), From 922a2580b2ede829cfe7805a96a081459d50a2f8 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 15 Oct 2026 23:06:04 +0400 Subject: [PATCH 69/73] Bind fresh FlowSafe staging proof to one isolated resource Validate the enabled Worker, D1, tenant and authenticated probe before opening the fence. Prepare a private off bundle so watchdog rollback survives checkout changes; retain the old sealed resource. --- packages/flowsafe/deploy/README.md | 14 + .../deploy/notification-staging-live.mjs | 386 +++++++++++++----- .../notification-staging-live.node-test.mjs | 330 ++++++++++++++- .../notification-staging-probe.worker.test.ts | 27 ++ .../notification-staging-probe.worker.ts | 5 +- .../deploy/notification-staging.worker.ts | 13 + .../notification-staging.wrangler.jsonc | 1 + 7 files changed, 655 insertions(+), 121 deletions(-) diff --git a/packages/flowsafe/deploy/README.md b/packages/flowsafe/deploy/README.md index f059139d..74152c4c 100644 --- a/packages/flowsafe/deploy/README.md +++ b/packages/flowsafe/deploy/README.md @@ -240,6 +240,18 @@ flowsafe_locked_deploy pnpm exec wrangler deploy \ The wrapper exits `75` if another local deployment owns the lock. Run the control on the same host and Unix user as the wrapper; the lock directory is fixed under `/tmp`, regardless of `TMPDIR`. Set `FLOWSAFE_STAGING_ENABLED_VERSION` from the exact enabled deployment ID before starting the control. The lock serializes cooperating local processes only. A deployment from another host, the Cloudflare dashboard, or an actor that skips this wrapper can still race the version check and deploy. Confirm exclusive deployment control separately before treating the live proof as safe. +For a fresh isolated proof, create a new Worker and D1 database. Copy `notification-staging.wrangler.jsonc` to an absolute `0600` path outside the repository. Change its Worker name, D1 name and ID, and deployment tenant to unique values. Resolve `main` and each `alias` target to their absolute paths in this checkout. Keep every other field unchanged, including the off flag, disabled public route, version metadata binding, and Durable Object migration. Do not reuse a sealed D1 database for an initial run. + +Copy `notification-staging-probe.wrangler.jsonc` to a second absolute `0600` path. Set its `services[0].service` to the new Worker name and resolve its `main` path; keep its other fields unchanged. Set `FLOWSAFE_STAGING_CONFIG_FILE` and `FLOWSAFE_STAGING_PROBE_CONFIG_FILE` to the private paths, and set `FLOWSAFE_STAGING_PROBE_PORT` to the local port. Start the probe before enabling the subject Worker: + +```bash +pnpm exec wrangler dev \ + --config "$FLOWSAFE_STAGING_PROBE_CONFIG_FILE" \ + --port "$FLOWSAFE_STAGING_PROBE_PORT" +``` + +Use the new Worker name and subject config path in the deployment wrapper. The controller checks the enabled version's D1 and tenant bindings and the probe subject's authenticated version identity before unlocking. It compiles a private off bundle before arming the watchdog; rollback uses that bundle. + Set these environment variables before running `node deploy/notification-staging-live.mjs --run` from `packages/flowsafe`: | Variable | Value | @@ -250,5 +262,7 @@ Set these environment variables before running `node deploy/notification-staging | `FLOWSAFE_CF_ACCOUNT_ID` | Cloudflare account ID as 32 lowercase hexadecimal characters; the same value used by the deployment wrapper | | `FLOWSAFE_STAGING_ENABLED_VERSION` | Exact 100% enabled subject Worker version ID | | `FLOWSAFE_STAGING_RETRY_OFF_VERSION` | Retry only: immediately prior 100% off Worker version ID; omit for an initial run | +| `FLOWSAFE_STAGING_CONFIG_FILE` | Fresh isolated proof only: absolute path to the private `0600` off config; omit to use the tracked staging Worker | +| `FLOWSAFE_STAGING_PROBE_CONFIG_FILE` | Fresh isolated proof only: absolute path to the private `0600` local probe config targeting the selected Worker | The controller prints the invocation ID only after D1 confirms one settled invocation, its notification, its operation receipt, and the watchdog's confirmed off deployment. It writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the controller exits without a confirmed off deployment, inspect that private directory's `watchdog.log` and verify the fence and current Worker version through the authenticated proxy and Wrangler before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs index 221f9a01..56fb1138 100644 --- a/packages/flowsafe/deploy/notification-staging-live.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -12,8 +12,9 @@ import { writeFileSync, } from 'node:fs'; import { tmpdir } from 'node:os'; -import { dirname, join } from 'node:path'; +import { dirname, join, resolve as resolvePath } from 'node:path'; import { fileURLToPath } from 'node:url'; +import { isDeepStrictEqual } from 'node:util'; import { localProbe, readSecretFile, @@ -25,13 +26,19 @@ import { const here = dirname(fileURLToPath(import.meta.url)); const flowsafe = dirname(here); const wrangler = join(flowsafe, '../../node_modules/.bin/wrangler'); -const config = join(here, 'notification-staging.wrangler.jsonc'); -const database = 'anchorage-flowsafe-remote-step-staging-db'; +const trackedConfigPath = join(here, 'notification-staging.wrangler.jsonc'); +const trackedConfig = JSON.parse(readFileSync(trackedConfigPath, 'utf8')); +const trackedProbeConfigPath = join( + here, + 'notification-staging-probe.wrangler.jsonc', +); +const trackedProbeConfig = JSON.parse( + readFileSync(trackedProbeConfigPath, 'utf8'), +); const versionPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; const accountPattern = /^[0-9a-f]{32}$/; -const workerName = JSON.parse(readFileSync(config, 'utf8')).name; -assert(/^[a-z0-9-]+$/.test(workerName), 'invalid staging Worker name'); +const workerNamePattern = /^[a-z0-9-]+$/; function assert(condition, message) { if (!condition) throw new Error(message); @@ -66,8 +73,86 @@ function privateFile(path) { return readFileSync(path, 'utf8').trim(); } -export function createWranglerControl({ command = spawnSync, token }) { +function stagingConfig(configPath) { + const source = + configPath === trackedConfigPath + ? readFileSync(configPath, 'utf8') + : privateFile(configPath); + const selected = JSON.parse(source); + const expected = structuredClone(trackedConfig); + if (configPath !== trackedConfigPath) { + assert( + workerNamePattern.test(selected.name) && + selected.name !== trackedConfig.name && + /^[a-z0-9-]+$/.test(selected.d1_databases?.[0]?.database_name) && + selected.d1_databases[0].database_name !== + trackedConfig.d1_databases[0].database_name && + versionPattern.test(selected.d1_databases[0].database_id) && + selected.d1_databases[0].database_id !== + trackedConfig.d1_databases[0].database_id && + /^[a-z0-9]{1,32}$/.test(selected.vars?.DEPLOYMENT_TENANT) && + selected.vars.DEPLOYMENT_TENANT !== + trackedConfig.vars.DEPLOYMENT_TENANT, + 'private staging config resource identity differs', + ); + expected.name = selected.name; + expected.main = resolvePath(here, trackedConfig.main); + expected.alias = Object.fromEntries( + Object.entries(trackedConfig.alias).map(([name, path]) => [ + name, + resolvePath(here, path), + ]), + ); + expected.d1_databases[0].database_name = + selected.d1_databases[0].database_name; + expected.d1_databases[0].database_id = selected.d1_databases[0].database_id; + expected.vars.DEPLOYMENT_TENANT = selected.vars.DEPLOYMENT_TENANT; + } + assert( + isDeepStrictEqual(selected, expected), + 'private staging config differs from the off template', + ); + return { + source, + name: selected.name, + database: selected.d1_databases[0].database_name, + databaseId: selected.d1_databases[0].database_id, + tenant: selected.vars.DEPLOYMENT_TENANT, + }; +} + +function verifyProbeConfig(configPath, workerName, fresh) { + assert( + fresh === (configPath !== trackedProbeConfigPath), + 'private probe config is required for the selected Worker', + ); + const selected = JSON.parse( + configPath === trackedProbeConfigPath + ? readFileSync(configPath, 'utf8') + : privateFile(configPath), + ); + const expected = structuredClone(trackedProbeConfig); + if (fresh) { + expected.main = resolvePath(here, trackedProbeConfig.main); + expected.services[0].service = workerName; + } + assert( + isDeepStrictEqual(selected, expected), + 'private probe config differs from the selected Worker', + ); +} + +export function createWranglerControl({ + command = spawnSync, + token, + configPath = trackedConfigPath, +}) { + const selected = stagingConfig(configPath); function run(args, timeout = 30_000) { + assert( + stagingConfig(configPath).source === selected.source, + 'staging config changed since preflight', + ); const result = command(wrangler, args, { cwd: flowsafe, encoding: 'utf8', @@ -100,7 +185,13 @@ export function createWranglerControl({ command = spawnSync, token }) { return value; } function deployments() { - const list = json(['deployments', 'list', '--config', config, '--json']); + const list = json([ + 'deployments', + 'list', + '--config', + configPath, + '--json', + ]); assert(Array.isArray(list) && list.length > 0, 'deployment list is empty'); const ordered = list .map((deployment) => { @@ -138,7 +229,7 @@ export function createWranglerControl({ command = spawnSync, token }) { 'view', version, '--config', - config, + configPath, '--json', ]); const bindings = detail.resources?.bindings; @@ -146,10 +237,26 @@ export function createWranglerControl({ command = spawnSync, token }) { const flag = bindings.filter( (binding) => binding.name === 'NOTIFICATION_PROOF_ENABLED', ); + const d1 = bindings.filter((binding) => binding.name === 'DB'); + const tenant = bindings.filter( + (binding) => binding.name === 'DEPLOYMENT_TENANT', + ); assert( flag.length === 1 && flag[0].type === 'plain_text', 'proof flag binding differs', ); + assert( + d1.length === 1 && + d1[0].type === 'd1' && + d1[0].database_id === selected.databaseId, + 'enabled D1 binding differs', + ); + assert( + tenant.length === 1 && + tenant[0].type === 'plain_text' && + tenant[0].text === selected.tenant, + 'enabled deployment tenant differs', + ); return flag[0].text === 'true'; } return { @@ -169,12 +276,12 @@ export function createWranglerControl({ command = spawnSync, token }) { const output = json([ 'd1', 'execute', - database, + selected.database, '--remote', '--yes', '--json', '--config', - config, + configPath, '--command', sql, ]); @@ -187,8 +294,37 @@ export function createWranglerControl({ command = spawnSync, token }) { ); return output[0].results; }, - deployOffIfCurrent(expectedVersion, outdir) { + prepareOffBundle(directory) { + assert(directory.startsWith('/'), 'absolute off bundle path required'); + mkdirSync(directory, { mode: 0o700 }); + run( + ['deploy', '--dry-run', '--config', configPath, '--outdir', directory], + 180_000, + ); + const bundlePath = join(directory, 'notification-staging.worker.js'); + assert(statSync(bundlePath).isFile(), 'off bundle is missing'); + run( + [ + 'deploy', + bundlePath, + '--dry-run', + '--config', + configPath, + '--outdir', + join(directory, 'bundle-rehearsal'), + ], + 180_000, + ); + return bundlePath; + }, + deployOffIfCurrent(expectedVersion, outdir, bundlePath) { assert(versionPattern.test(expectedVersion), 'invalid expected version'); + assert( + typeof bundlePath === 'string' && + bundlePath.startsWith('/') && + statSync(bundlePath).isFile(), + 'pinned off bundle is required', + ); assert( currentVersion() === expectedVersion, 'enabled deployment changed', @@ -197,13 +333,10 @@ export function createWranglerControl({ command = spawnSync, token }) { proofEnabled(expectedVersion), 'expected deployment is not enabled', ); - const tracked = JSON.parse(readFileSync(config, 'utf8')); - assert( - tracked.vars?.NOTIFICATION_PROOF_ENABLED === 'false' && - tracked.workers_dev === false, - 'tracked off config differs', + run( + ['deploy', bundlePath, '--config', configPath, '--outdir', outdir], + 180_000, ); - run(['deploy', '--config', config, '--outdir', outdir], 180_000); const next = currentVersion(); assert( next !== expectedVersion && proofEnabled(next) === false, @@ -280,6 +413,13 @@ function settingsFromEnv(env) { priorOffVersion === undefined || versionPattern.test(priorOffVersion), 'invalid prior off version', ); + const configPath = env.FLOWSAFE_STAGING_CONFIG_FILE ?? trackedConfigPath; + const selected = stagingConfig(configPath); + verifyProbeConfig( + env.FLOWSAFE_STAGING_PROBE_CONFIG_FILE ?? trackedProbeConfigPath, + selected.name, + configPath !== trackedConfigPath, + ); return { port, expectedVersion, @@ -289,10 +429,12 @@ function settingsFromEnv(env) { token, accountId, priorOffVersion, + configPath, + selected, }; } -function acquireDeploymentLock(accountId) { +function acquireDeploymentLock(accountId, workerName) { const directory = join('/tmp', `flowsafe-deploy-locks-${process.getuid()}`); mkdirSync(directory, { mode: 0o700, recursive: true }); const directoryInfo = statSync(directory); @@ -349,13 +491,30 @@ async function watchdog(stateFile) { deadlineAt, outdir, baselineEpoch, + configPath, + workerName, + bundlePath, } = state; assert( baselineEpoch === 0 || baselineEpoch === 1, 'invalid watchdog baseline', ); const probe = localProbe({ port, secret: readSecretFile(probeSecretPath) }); - const cli = createWranglerControl({ token: privateFile(cfTokenPath) }); + const selected = stagingConfig(configPath); + assert(selected.name === workerName, 'watchdog Worker identity differs'); + assert( + bundlePath === + join( + dirname(stateFile), + 'pinned-off-bundle', + 'notification-staging.worker.js', + ) && statSync(bundlePath).isFile(), + 'watchdog pinned off bundle is missing', + ); + const cli = createWranglerControl({ + token: privateFile(cfTokenPath), + configPath, + }); assert( cli.currentVersion() === expectedVersion && cli.proofEnabled(expectedVersion), @@ -371,101 +530,123 @@ async function watchdog(stateFile) { await new Promise((resolve) => setTimeout(resolve, 1_000)); } await sealFence(probe, baselineEpoch); - cli.deployOffIfCurrent(expectedVersion, outdir); + cli.deployOffIfCurrent(expectedVersion, outdir, bundlePath); confirmedFence(await probe('GET', '/admin/execution-fence'), baselineEpoch); writeFileSync(`${stateFile}.off-confirmed`, '', { mode: 0o600, flag: 'wx' }); } async function controller() { const settings = settingsFromEnv(process.env); - const lockFd = acquireDeploymentLock(settings.accountId); - const cli = createWranglerControl({ token: settings.token }); - assert( - cli.currentVersion() === settings.expectedVersion && - cli.proofEnabled(settings.expectedVersion), - 'enabled version preflight differs', + const lockFd = acquireDeploymentLock( + settings.accountId, + settings.selected.name, ); - const probe = localProbe({ port: settings.port, secret: settings.secret }); - const baselineEpoch = settings.priorOffVersion === undefined ? 0 : 1; - if (baselineEpoch === 1) { - assert( - settings.priorOffVersion !== settings.expectedVersion && - cli.priorOffVersion(settings.expectedVersion) === - settings.priorOffVersion, - 'retry prior off version is unconfirmed', - ); - const fence = await probe('GET', '/admin/execution-fence'); - assert( - Object.entries(sealedRetryFence).every( - ([key, value]) => fence[key] === value, - ) && - fence.proofKey === undefined && - fence.proofRunId === undefined, - 'retry fence is not the exact sealed baseline', - ); - } const directory = join( tmpdir(), `flowsafe-notification-control-${process.pid}`, ); mkdirSync(directory, { mode: 0o700 }); - const stateFile = join(directory, 'watchdog.json'); - const outdir = join(directory, 'off-deploy'); - const watcherLog = openSync(join(directory, 'watchdog.log'), 'wx', 0o600); - writeFileSync( - stateFile, - JSON.stringify({ - port: settings.port, - expectedVersion: settings.expectedVersion, - probeSecretPath: settings.probeSecretPath, - cfTokenPath: settings.cfTokenPath, - parentPid: process.pid, - deadlineAt: Date.now() + 900_000, - outdir, - baselineEpoch, - }), - { mode: 0o600, flag: 'wx' }, - ); - const watcher = spawn( - process.execPath, - [fileURLToPath(import.meta.url), '--watchdog', stateFile], - { - detached: true, - stdio: ['ignore', watcherLog, watcherLog, lockFd], - env: process.env, - }, - ); - closeSync(watcherLog); - watcher.unref(); - assert(Number.isInteger(watcher.pid), 'watchdog did not start'); - const readyDeadline = Date.now() + 20_000; - while (!existsSync(`${stateFile}.ready`) && Date.now() < readyDeadline) { - await new Promise((resolve) => setTimeout(resolve, 100)); - } - assert( - existsSync(`${stateFile}.ready`) && - watcher.exitCode === null && - watcher.signalCode === null, - 'watchdog readiness is unconfirmed', - ); - process.kill(watcher.pid, 0); - let offConfirmed = false; - const requestOffDeploy = async () => { - if (offConfirmed) return; - writeFileSync(`${stateFile}.off-requested`, '', { + const configPath = + settings.configPath === trackedConfigPath + ? trackedConfigPath + : join(directory, 'off.wrangler.jsonc'); + if (configPath !== trackedConfigPath) + writeFileSync(configPath, settings.selected.source, { mode: 0o600, flag: 'wx', }); - while (!existsSync(`${stateFile}.off-confirmed`)) { + const stateFile = join(directory, 'watchdog.json'); + const outdir = join(directory, 'off-deploy'); + let stage = 'enabled-version-preflight'; + try { + const cli = createWranglerControl({ token: settings.token, configPath }); + assert( + cli.currentVersion() === settings.expectedVersion && + cli.proofEnabled(settings.expectedVersion), + 'enabled version preflight differs', + ); + const probe = localProbe({ port: settings.port, secret: settings.secret }); + const baselineEpoch = settings.priorOffVersion === undefined ? 0 : 1; + if (baselineEpoch === 1) { assert( - watcher.exitCode === null && watcher.signalCode === null, - 'watchdog exited before off confirmation', + settings.priorOffVersion !== settings.expectedVersion && + cli.priorOffVersion(settings.expectedVersion) === + settings.priorOffVersion, + 'retry prior off version is unconfirmed', ); + const fence = await probe('GET', '/admin/execution-fence'); + assert( + Object.entries(sealedRetryFence).every( + ([key, value]) => fence[key] === value, + ) && + fence.proofKey === undefined && + fence.proofRunId === undefined, + 'retry fence is not the exact sealed baseline', + ); + } + stage = 'off-bundle-preparation'; + const bundlePath = cli.prepareOffBundle( + join(directory, 'pinned-off-bundle'), + ); + stage = 'watchdog-start'; + const watcherLog = openSync(join(directory, 'watchdog.log'), 'wx', 0o600); + writeFileSync( + stateFile, + JSON.stringify({ + port: settings.port, + expectedVersion: settings.expectedVersion, + probeSecretPath: settings.probeSecretPath, + cfTokenPath: settings.cfTokenPath, + parentPid: process.pid, + deadlineAt: Date.now() + 900_000, + outdir, + baselineEpoch, + configPath, + workerName: settings.selected.name, + bundlePath, + }), + { mode: 0o600, flag: 'wx' }, + ); + const watcher = spawn( + process.execPath, + [fileURLToPath(import.meta.url), '--watchdog', stateFile], + { + detached: true, + stdio: ['ignore', watcherLog, watcherLog, lockFd], + env: process.env, + }, + ); + closeSync(watcherLog); + watcher.unref(); + assert(Number.isInteger(watcher.pid), 'watchdog did not start'); + const readyDeadline = Date.now() + 20_000; + while (!existsSync(`${stateFile}.ready`) && Date.now() < readyDeadline) { await new Promise((resolve) => setTimeout(resolve, 100)); } - offConfirmed = true; - }; - try { + assert( + existsSync(`${stateFile}.ready`) && + watcher.exitCode === null && + watcher.signalCode === null, + 'watchdog readiness is unconfirmed', + ); + process.kill(watcher.pid, 0); + let offConfirmed = false; + const requestOffDeploy = async () => { + if (offConfirmed) return; + writeFileSync(`${stateFile}.off-requested`, '', { + mode: 0o600, + flag: 'wx', + }); + while (!existsSync(`${stateFile}.off-confirmed`)) { + assert( + watcher.exitCode === null && watcher.signalCode === null, + 'watchdog exited before off confirmation', + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + } + offConfirmed = true; + }; + stage = 'proof'; const result = await runNotificationStagingControl({ probe, ...trustedD1Checks((sql) => cli.query(sql)), @@ -477,6 +658,12 @@ async function controller() { 'watchdog exited before fence unlock', ); process.kill(watcher.pid, 0); + const identity = await probe('GET', '/identity'); + assert( + identity.versionId === settings.expectedVersion && + identity.deploymentTenant === settings.selected.tenant, + 'authenticated probe subject differs from enabled Worker', + ); }, }); assert(offConfirmed, 'off deploy was not confirmed'); @@ -484,9 +671,12 @@ async function controller() { } catch (error) { appendFileSync( join(directory, 'watchdog.log'), - `controller: ${privateDiagnostic(error, [settings.secret, settings.token])}\n`, + `controller ${stage}: ${privateDiagnostic(error, [settings.secret, settings.token])}\n`, + { mode: 0o600 }, + ); + throw new Error( + `staging controller failed at ${stage}; inspect private watchdog log`, ); - throw new Error('staging controller failed; inspect private watchdog log'); } } @@ -503,7 +693,7 @@ if (process.argv[1] === fileURLToPath(import.meta.url)) { if (result) process.stdout.write(`${JSON.stringify(result)}\n`); }, (error) => { - process.stderr.write(`${privateDiagnostic(error)}\n`); + process.stderr.write(`${error.message}\n`); process.exitCode = 1; }, ); diff --git a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs index dab3e287..0fd44fc7 100644 --- a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs @@ -10,8 +10,8 @@ import { } from 'node:fs'; import { createServer } from 'node:http'; import { tmpdir } from 'node:os'; -import { join } from 'node:path'; -import { test } from 'node:test'; +import { dirname, join } from 'node:path'; +import { after, test } from 'node:test'; import { fileURLToPath } from 'node:url'; import { createWranglerControl, @@ -20,6 +20,55 @@ import { const enabled = 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa'; const disabled = 'bbbbbbbb-bbbb-4bbb-bbbb-bbbbbbbbbbbb'; +const bundleDirectory = mkdtempSync( + join(tmpdir(), 'flowsafe-test-off-bundle-'), +); +const bundlePath = join(bundleDirectory, 'notification-staging.worker.js'); +writeFileSync(bundlePath, 'export default {};', { mode: 0o600 }); +after(() => rmSync(bundleDirectory, { recursive: true, force: true })); + +function freshConfig() { + const source = JSON.parse( + readFileSync( + new URL('./notification-staging.wrangler.jsonc', import.meta.url), + 'utf8', + ), + ); + source.name = 'anchorage-flowsafe-remote-step-staging-fresh'; + source.main = fileURLToPath( + new URL('./notification-staging.worker.ts', import.meta.url), + ); + source.alias = Object.fromEntries( + Object.entries(source.alias).map(([key, value]) => [ + key, + fileURLToPath(new URL(value, import.meta.url)), + ]), + ); + source.d1_databases[0].database_name = 'anchorage-flowsafe-fresh-db'; + source.d1_databases[0].database_id = 'cccccccc-cccc-4ccc-cccc-cccccccccccc'; + source.vars.DEPLOYMENT_TENANT = 'flowsafenotificationfresh'; + return source; +} + +function versionBindings(source, proofEnabled) { + return [ + { + name: 'NOTIFICATION_PROOF_ENABLED', + type: 'plain_text', + text: String(proofEnabled), + }, + { + name: 'DEPLOYMENT_TENANT', + type: 'plain_text', + text: source.vars.DEPLOYMENT_TENANT, + }, + { + name: 'DB', + type: 'd1', + database_id: source.d1_databases[0].database_id, + }, + ]; +} function fakeCli({ enabledFlag = true, priorVersion } = {}) { const calls = []; @@ -46,17 +95,20 @@ function fakeCli({ enabledFlag = true, priorVersion } = {}) { }; } if (args[0] === 'versions') { + const source = JSON.parse( + readFileSync( + new URL('./notification-staging.wrangler.jsonc', import.meta.url), + 'utf8', + ), + ); return { status: 0, stdout: JSON.stringify({ resources: { - bindings: [ - { - name: 'NOTIFICATION_PROOF_ENABLED', - type: 'plain_text', - text: args[2] === enabled && enabledFlag ? 'true' : 'false', - }, - ], + bindings: versionBindings( + source, + args[2] === enabled && enabledFlag, + ), }, }), }; @@ -88,7 +140,11 @@ test('version-checked off deploy uses the tracked false config', () => { assert.equal(fixture.cli.currentVersion(), enabled); assert.equal(fixture.cli.proofEnabled(enabled), true); assert.equal( - fixture.cli.deployOffIfCurrent(enabled, '/tmp/staging-off-test'), + fixture.cli.deployOffIfCurrent( + enabled, + '/tmp/staging-off-test', + bundlePath, + ), disabled, ); assert.equal(fixture.cli.proofEnabled(disabled), false); @@ -99,13 +155,23 @@ test('changed version or disabled flag refuses before deploy', () => { const changed = fakeCli(); changed.change(); assert.throws( - () => changed.cli.deployOffIfCurrent(enabled, '/tmp/staging-off-test'), + () => + changed.cli.deployOffIfCurrent( + enabled, + '/tmp/staging-off-test', + bundlePath, + ), /changed/, ); assert.equal(changed.calls.filter((args) => args[0] === 'deploy').length, 0); const disabledFlag = fakeCli({ enabledFlag: false }); assert.throws( - () => disabledFlag.cli.deployOffIfCurrent(enabled, '/tmp/staging-off-test'), + () => + disabledFlag.cli.deployOffIfCurrent( + enabled, + '/tmp/staging-off-test', + bundlePath, + ), /not enabled/, ); assert.equal( @@ -123,6 +189,137 @@ test('retry evidence requires the immediately prior full off deployment', () => assert.throws(() => enabledPrior.cli.priorOffVersion(enabled), /not off/); }); +test('enabled version requires exact D1 and tenant bindings', () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-d1-mismatch-test-')); + try { + const source = freshConfig(); + const path = join(directory, 'private.wrangler.jsonc'); + writeFileSync(path, JSON.stringify(source), { mode: 0o600 }); + let boundDatabaseId = 'dddddddd-dddd-4ddd-dddd-dddddddddddd'; + let boundTenant = source.vars.DEPLOYMENT_TENANT; + const cli = createWranglerControl({ + configPath: path, + token: 'fake-token', + command: (_executable, args) => { + assert.equal(args[0], 'versions'); + return { + status: 0, + stdout: JSON.stringify({ + resources: { + bindings: [ + { + name: 'NOTIFICATION_PROOF_ENABLED', + type: 'plain_text', + text: 'true', + }, + { + name: 'DEPLOYMENT_TENANT', + type: 'plain_text', + text: boundTenant, + }, + { + name: 'DB', + type: 'd1', + database_id: boundDatabaseId, + }, + ], + }, + }), + }; + }, + }); + assert.throws( + () => cli.proofEnabled(enabled), + /enabled D1 binding differs/, + ); + boundDatabaseId = source.d1_databases[0].database_id; + boundTenant = 'wrongtenant'; + assert.throws( + () => cli.proofEnabled(enabled), + /enabled deployment tenant differs/, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + +test('private staging config scopes version, D1, and off deployment to one fresh resource', () => { + const directory = mkdtempSync(join(tmpdir(), 'flowsafe-fresh-config-test-')); + try { + const source = freshConfig(); + const path = join(directory, 'private.wrangler.jsonc'); + writeFileSync(path, JSON.stringify(source), { mode: 0o600 }); + const calls = []; + let currentVersion = enabled; + const command = (_executable, args) => { + calls.push(args); + if (args[0] === 'deployments') + return { + status: 0, + stdout: JSON.stringify([ + { + created_on: '2026-10-15T00:00:00.000Z', + versions: [{ version_id: currentVersion, percentage: 100 }], + }, + ]), + }; + if (args[0] === 'versions') + return { + status: 0, + stdout: JSON.stringify({ + resources: { + bindings: versionBindings(source, args[2] === enabled), + }, + }), + }; + if (args[0] === 'd1') + return { + status: 0, + stdout: JSON.stringify([{ success: true, results: [] }]), + }; + if (args[0] === 'deploy') { + currentVersion = disabled; + return { status: 0, stdout: 'deployed' }; + } + throw new Error('unexpected CLI'); + }; + const cli = createWranglerControl({ + command, + token: 'fake-token', + configPath: path, + }); + assert.equal(cli.currentVersion(), enabled); + assert.deepEqual(cli.query('SELECT 1'), []); + assert.equal( + cli.deployOffIfCurrent(enabled, join(directory, 'off'), bundlePath), + disabled, + ); + assert.equal(calls.filter((args) => args[0] === 'deploy').length, 1); + for (const args of calls) { + assert.equal(args[args.indexOf('--config') + 1], path); + if (args[0] === 'd1') + assert.equal(args[2], source.d1_databases[0].database_name); + } + source.routes = [{ pattern: 'example.com/*', zone_name: 'example.com' }]; + writeFileSync(path, JSON.stringify(source), { mode: 0o600 }); + assert.throws( + () => cli.query('SELECT 1'), + /private staging config differs/, + ); + assert.throws( + () => + createWranglerControl({ + command, + token: 'fake-token', + configPath: path, + }), + /private staging config differs/, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + test('Wrangler failure preserves exit and stderr in its diagnostic cause', () => { const cli = createWranglerControl({ token: 'fake-token', @@ -271,19 +468,47 @@ async function exerciseWatchdog(outcome) { ? 'c'.repeat(32) : outcome === 'sealed retry' ? 'e'.repeat(32) - : 'd'.repeat(32); + : outcome === 'fresh config' + ? 'f'.repeat(32) + : outcome === 'mismatched probe' + ? '1'.repeat(32) + : 'd'.repeat(32); + const workerName = + outcome === 'fresh config' + ? 'anchorage-flowsafe-remote-step-staging-fresh' + : 'anchorage-flowsafe-remote-step-staging'; const lockFile = join( '/tmp', `flowsafe-deploy-locks-${process.getuid()}`, - `${accountId}-anchorage-flowsafe-remote-step-staging.lock`, + `${accountId}-${workerName}.lock`, ); writeFileSync(tokenFile, 'fake-token', { mode: 0o600 }); writeFileSync(secretFile, secret, { mode: 0o600 }); + const privateConfigPath = join(directory, 'fresh.wrangler.jsonc'); + const privateProbeConfigPath = join(directory, 'fresh-probe.wrangler.jsonc'); + if (outcome === 'fresh config') { + writeFileSync(privateConfigPath, JSON.stringify(freshConfig()), { + mode: 0o600, + }); + const probeConfig = JSON.parse( + readFileSync( + new URL('./notification-staging-probe.wrangler.jsonc', import.meta.url), + 'utf8', + ), + ); + probeConfig.main = fileURLToPath( + new URL('./notification-staging-probe.worker.ts', import.meta.url), + ); + probeConfig.services[0].service = workerName; + writeFileSync(privateProbeConfigPath, JSON.stringify(probeConfig), { + mode: 0o600, + }); + } writeFileSync( preloadFile, `import childProcess from 'node:child_process'; import { syncBuiltinESMExports } from 'node:module'; -import { existsSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, writeFileSync } from 'node:fs'; const originalSpawn = childProcess.spawn; const originalSpawnSync = childProcess.spawnSync; const originalNow = Date.now; @@ -301,6 +526,13 @@ childProcess.spawnSync = (executable, args, options) => { if (executable === 'flock') return originalSpawnSync(executable, args, options); if (!executable.endsWith('/node_modules/.bin/wrangler')) throw new Error('unexpected synchronous command'); + if (process.env.TEST_FRESH_CONFIG === 'true') { + const configPath = args[args.indexOf('--config') + 1]; + if (!configPath.endsWith('/off.wrangler.jsonc')) + throw new Error('fresh config snapshot was not used'); + if (args[0] === 'd1' && args[2] !== 'anchorage-flowsafe-fresh-db') + throw new Error('fresh D1 was not selected'); + } const off = existsSync(process.env.TEST_OFF_FILE); const command = args[0]; if (off && command === 'deployments') @@ -321,6 +553,15 @@ childProcess.spawnSync = (executable, args, options) => { resources: { bindings: [{ name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', text: off || args[2] === '${disabled}' ? 'false' : 'true', + }, { + name: 'DEPLOYMENT_TENANT', type: 'plain_text', + text: process.env.TEST_FRESH_CONFIG === 'true' + ? 'flowsafenotificationfresh' : 'flowsafenotificationstaging', + }, { + name: 'DB', type: 'd1', + database_id: process.env.TEST_FRESH_CONFIG === 'true' + ? 'cccccccc-cccc-4ccc-cccc-cccccccccccc' + : 'b863a219-2096-4e13-92fe-4db592b67944', }] }, } : command === 'd1' ? [{ success: true, results: sql.includes("name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')") @@ -344,6 +585,15 @@ childProcess.spawnSync = (executable, args, options) => { : [], }] : null; if (command === 'deploy') { + if (args.includes('--dry-run')) { + const outdir = args[args.indexOf('--outdir') + 1]; + mkdirSync(outdir, { recursive: true, mode: 0o700 }); + writeFileSync(outdir + '/notification-staging.worker.js', 'export default {};'); + return { status: 0, stdout: 'bundled' }; + } + if (!args[1]?.endsWith('/pinned-off-bundle/notification-staging.worker.js') || + !existsSync(args[1])) + throw new Error('pinned off bundle was not used'); writeFileSync(process.env.TEST_DEPLOY_STARTED_FILE, 'started'); while (!existsSync(process.env.TEST_RELEASE_DEPLOY_FILE)) originalSpawnSync('sleep', ['0.02']); @@ -373,7 +623,7 @@ syncBuiltinESMExports(); if (method === 'POST' && path === '/start') { startReceived = true; writeFileSync(effectStartedFile, 'started'); - if (outcome === 'parent death') return; + if (outcome === 'parent death' || outcome === 'fresh config') return; response.writeHead(200, { 'content-type': 'application/json' }); response.end( JSON.stringify({ status: 'suspended', invocationId: 'invocation' }), @@ -383,6 +633,14 @@ syncBuiltinESMExports(); let value; if (path === '/complete') { value = { invocationId: 'invocation', status: 'success' }; + } else if (path === '/identity') { + value = { + versionId: outcome === 'mismatched probe' ? disabled : enabled, + deploymentTenant: + outcome === 'fresh config' + ? 'flowsafenotificationfresh' + : 'flowsafenotificationstaging', + }; } else if (path === '/admin/maintenance-status') { value = { alarmAt: 1, nextRemoteStepRecoveryAt: 1 }; } else if (path === '/admin/execution-fence' && method === 'POST') { @@ -433,6 +691,12 @@ syncBuiltinESMExports(); FLOWSAFE_STAGING_PROBE_SECRET_FILE: secretFile, FLOWSAFE_CF_TOKEN_FILE: tokenFile, FLOWSAFE_CF_ACCOUNT_ID: accountId, + ...(outcome === 'fresh config' + ? { + FLOWSAFE_STAGING_CONFIG_FILE: privateConfigPath, + FLOWSAFE_STAGING_PROBE_CONFIG_FILE: privateProbeConfigPath, + } + : {}), TEST_OFF_FILE: offFile, TEST_OFF_CONFIRMED_FILE: offConfirmedFile, TEST_WATCHDOG_PID_FILE: watcherPidFile, @@ -441,6 +705,7 @@ syncBuiltinESMExports(); TEST_EFFECT_STARTED_FILE: effectStartedFile, TEST_CLOCK_JUMP: outcome === 'controller request' ? 'true' : 'false', TEST_RETRY: outcome === 'sealed retry' ? 'true' : 'false', + TEST_FRESH_CONFIG: outcome === 'fresh config' ? 'true' : 'false', }, stdio: 'ignore', }, @@ -458,6 +723,18 @@ syncBuiltinESMExports(); } }; await waitFor(() => existsSync(`${stateFile}.ready`)); + if (outcome === 'fresh config') { + const state = JSON.parse(readFileSync(stateFile, 'utf8')); + assert.equal(state.workerName, workerName); + assert.equal( + state.configPath, + join(dirname(stateFile), 'off.wrangler.jsonc'), + ); + assert.equal( + readFileSync(state.configPath, 'utf8'), + readFileSync(privateConfigPath, 'utf8'), + ); + } if (outcome === 'sealed retry') assert.equal( JSON.parse(readFileSync(stateFile, 'utf8')).baselineEpoch, @@ -465,9 +742,9 @@ syncBuiltinESMExports(); ); await waitFor(() => existsSync(watcherPidFile)); watcherPid = Number(readFileSync(watcherPidFile, 'utf8')); - await waitFor(() => startReceived); + if (outcome !== 'mismatched probe') await waitFor(() => startReceived); assert.equal(existsSync(offFile), false); - if (outcome === 'parent death') { + if (outcome === 'parent death' || outcome === 'fresh config') { assert.equal(fence.state, 'open'); controller.kill('SIGKILL'); } @@ -481,14 +758,19 @@ syncBuiltinESMExports(); assert.equal(readFileSync(offFile, 'utf8'), 'deployed'); assert.deepEqual(fence, { state: 'migration-locked', - mutationEpoch: outcome === 'sealed retry' ? 3 : 2, + mutationEpoch: + outcome === 'sealed retry' ? 3 : outcome === 'mismatched probe' ? 1 : 2, requireMutationEpoch: true, - transitionRevision: outcome === 'sealed retry' ? 3 : 2, + transitionRevision: + outcome === 'sealed retry' ? 3 : outcome === 'mismatched probe' ? 1 : 2, }); assert.deepEqual( requests.filter((request) => request === 'POST /admin/execution-fence'), - ['POST /admin/execution-fence', 'POST /admin/execution-fence'], + outcome === 'mismatched probe' + ? ['POST /admin/execution-fence'] + : ['POST /admin/execution-fence', 'POST /admin/execution-fence'], ); + if (outcome === 'mismatched probe') assert.equal(startReceived, false); if (outcome === 'controller request') { await waitFor(() => controller.exitCode !== null); assert.equal(controller.exitCode, 0); @@ -514,6 +796,12 @@ syncBuiltinESMExports(); test('watchdog owns off deployment after parent death', () => exerciseWatchdog('parent death')); +test('fresh Worker watchdog uses the private off config snapshot after parent death', () => + exerciseWatchdog('fresh config')); + +test('mismatched authenticated probe subject prevents fence unlock and effect', () => + exerciseWatchdog('mismatched probe')); + test('watchdog owns off deployment after controller request', () => exerciseWatchdog('controller request')); diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts index 6c7ce1c8..2deb0079 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.test.ts @@ -31,6 +31,33 @@ it('forwards a staging request only after the local caller supplies the probe cr }); }); +it('returns the authenticated subject version through the private binding', async () => { + const secret = 'staging-local-probe-secret-32-bytes-minimum'; + const fetch = vi.fn(async () => + Response.json({ + versionId: 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa', + deploymentTenant: 'flowsafenotificationfresh', + }), + ); + const env = { + SUBJECT: { fetch } as unknown as Fetcher, + STAGING_PROBE_SECRET: secret, + }; + const request = (credential: string) => + new Request('http://127.0.0.1/identity', { + headers: { authorization: `Bearer ${credential}` }, + }); + expect((await handler.fetch(request('wrong'), env)).status).toBe(403); + expect(fetch).not.toHaveBeenCalled(); + const response = await handler.fetch(request(secret), env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ + versionId: 'aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa', + deploymentTenant: 'flowsafenotificationfresh', + }); + expect(fetch.mock.calls[0]?.[0]).toBe('http://subject/__staging/identity'); +}); + it('keeps start and completion on the authenticated private service binding', async () => { const secret = 'staging-local-probe-secret-32-bytes-minimum'; const fetch = vi.fn(async () => new Response(null, { status: 204 })); diff --git a/packages/flowsafe/deploy/notification-staging-probe.worker.ts b/packages/flowsafe/deploy/notification-staging-probe.worker.ts index 0d602514..1a7934e3 100644 --- a/packages/flowsafe/deploy/notification-staging-probe.worker.ts +++ b/packages/flowsafe/deploy/notification-staging-probe.worker.ts @@ -20,8 +20,9 @@ export default { return new Response(null, { status: 404 }); if (url.search || url.hash) return new Response(null, { status: 404 }); const staging = - request.method === 'POST' && - ['/ensure', '/start', '/complete'].includes(url.pathname); + (request.method === 'POST' && + ['/ensure', '/start', '/complete'].includes(url.pathname)) || + (request.method === 'GET' && url.pathname === '/identity'); const admin = (request.method === 'GET' && ['/admin/execution-fence', '/admin/maintenance-status'].includes( diff --git a/packages/flowsafe/deploy/notification-staging.worker.ts b/packages/flowsafe/deploy/notification-staging.worker.ts index 3367fcca..86e44b0b 100644 --- a/packages/flowsafe/deploy/notification-staging.worker.ts +++ b/packages/flowsafe/deploy/notification-staging.worker.ts @@ -61,6 +61,7 @@ interface Env { MAINTENANCE_ADMIN_SECRET: string; STAGING_PROBE_SECRET: string; NOTIFICATION_PROOF_ENABLED?: string; + CF_VERSION_METADATA: { id: string }; RUNNER: DurableObjectNamespace; THREAD: DurableObjectNamespace; MAINTENANCE: DurableObjectNamespace; @@ -328,6 +329,18 @@ const config = { !(await credentialsMatch(supplied, env.STAGING_PROBE_SECRET)) ) return new Response(null, { status: 403 }); + if (request.method === 'GET' && url.pathname === '/__staging/identity') { + if ( + !/^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/.test( + env.CF_VERSION_METADATA?.id, + ) + ) + return new Response(null, { status: 503 }); + return Response.json({ + versionId: env.CF_VERSION_METADATA.id, + deploymentTenant: env.DEPLOYMENT_TENANT, + }); + } if (request.method === 'POST' && url.pathname === '/__staging/ensure') { await ensureRemoteStepNotificationRecoveryDuty(env); return Response.json({ armed: true }); diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc index 06e591a2..f2c42ec8 100644 --- a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -9,6 +9,7 @@ "cloudflare": "./notification-staging-no-rest.ts" }, "workers_dev": false, + "version_metadata": { "binding": "CF_VERSION_METADATA" }, "secrets": { "required": [ "DEPLOYMENT_IDENTITY_SECRET", From f1b7668198005c81951d00559491ac83199b1b07 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:16:24 +0400 Subject: [PATCH 70/73] Bind staging execution to its trusted mutation epoch Match the live fence transition in the native proof and pass the version-pinned epoch to start and completion. Attempt exact-version off recovery when the probe fails and confirm the locked fence through D1. --- packages/flowsafe/deploy/README.md | 7 +- .../deploy/notification-staging-live.mjs | 66 ++++++++-- .../notification-staging-live.node-test.mjs | 121 ++++++++++++++++-- .../deploy/notification-staging-native.mjs | 26 +++- .../deploy/notification-staging.worker.ts | 5 + .../notification-staging.wrangler.jsonc | 1 + 6 files changed, 207 insertions(+), 19 deletions(-) diff --git a/packages/flowsafe/deploy/README.md b/packages/flowsafe/deploy/README.md index 74152c4c..e6820090 100644 --- a/packages/flowsafe/deploy/README.md +++ b/packages/flowsafe/deploy/README.md @@ -220,7 +220,7 @@ See the repository [deployment reference](../../../docs/deployment-reference.md) Use `notification-staging-live.mjs` only with the isolated notification staging Worker and its loopback Service-binding probe. Keep the probe bound to `127.0.0.1`. The initial run requires a `migration-locked` fence at epoch and revision zero, an empty v5 journal, and an armed recovery alarm. It acquires a Linux `flock` keyed by Cloudflare account and Worker before reading the enabled version. Its detached watchdog inherits the same locked open file descriptor, seals or relocks the fence after controller death or its 15-minute deadline, and performs the tracked off deployment on controller request or recovery. Sealing advances the locked fence's revision, so a delayed controller unlock cannot succeed after watchdog shutdown starts. The controller never deploys off itself. -A failed preflight may leave the fence sealed at `migration-locked`, epoch and revision `1`, with `requireMutationEpoch=true`, even though no effect ran. An operator may retry this exact state once: verify the prior run's tracked off Worker version, confirm the remote-step journal is empty and the notification and receipt tables are absent, deploy a fresh enabled version under the same lock, and set `FLOWSAFE_STAGING_RETRY_OFF_VERSION` to that immediately prior 100% off deployment ID. The controller checks deployment history, the prior version's off flag, the exact sealed fence and the D1 preflight before unlocking. It persists the selected baseline in private watchdog state before readiness. The retry opens at epoch and revision `2`; the watchdog seals a still-locked fence at `2` or relocks an open fence at `3`. A different fence or a second sealed retry requires operator review; do not update the D1 fence row directly. +A failed preflight may leave the fence sealed at `migration-locked`, epoch and revision `1`, with `requireMutationEpoch=true`, even though no effect ran. An operator may retry this exact state once: verify the prior run's tracked off Worker version, confirm the remote-step journal is empty and the notification and receipt tables are absent, deploy a fresh enabled version under the same lock with `--var STAGING_MUTATION_EPOCH:2`, and set `FLOWSAFE_STAGING_RETRY_OFF_VERSION` to that immediately prior 100% off deployment ID. The retry opens at epoch and revision `2`; the watchdog seals a still-locked fence at `2` or relocks an open fence at `3`. A different fence or a second sealed retry requires operator review; do not update the D1 fence row directly. Use the same lock for **every** command that can upload, deploy, promote, or change traffic for `anchorage-flowsafe-remote-step-staging`. From `packages/flowsafe`, define the wrapper below and use it for the initial enabled deployment and any other same-Worker deployment command: @@ -235,11 +235,14 @@ flowsafe_locked_deploy() { flowsafe_locked_deploy pnpm exec wrangler deploy \ --config deploy/notification-staging.wrangler.jsonc \ --var NOTIFICATION_PROOF_ENABLED:true \ + --var STAGING_MUTATION_EPOCH:1 \ --outdir /tmp/flowsafe-notification-enable ``` The wrapper exits `75` if another local deployment owns the lock. Run the control on the same host and Unix user as the wrapper; the lock directory is fixed under `/tmp`, regardless of `TMPDIR`. Set `FLOWSAFE_STAGING_ENABLED_VERSION` from the exact enabled deployment ID before starting the control. The lock serializes cooperating local processes only. A deployment from another host, the Cloudflare dashboard, or an actor that skips this wrapper can still race the version check and deploy. Confirm exclusive deployment control separately before treating the live proof as safe. +Pin `STAGING_MUTATION_EPOCH` in the enabled version to the epoch its controller will open: `1` for an initial run or `2` for the permitted retry. Keep the off config at `0`. The epoch is deployment configuration; never derive it from a request or adopt the current D1 epoch. + For a fresh isolated proof, create a new Worker and D1 database. Copy `notification-staging.wrangler.jsonc` to an absolute `0600` path outside the repository. Change its Worker name, D1 name and ID, and deployment tenant to unique values. Resolve `main` and each `alias` target to their absolute paths in this checkout. Keep every other field unchanged, including the off flag, disabled public route, version metadata binding, and Durable Object migration. Do not reuse a sealed D1 database for an initial run. Copy `notification-staging-probe.wrangler.jsonc` to a second absolute `0600` path. Set its `services[0].service` to the new Worker name and resolve its `main` path; keep its other fields unchanged. Set `FLOWSAFE_STAGING_CONFIG_FILE` and `FLOWSAFE_STAGING_PROBE_CONFIG_FILE` to the private paths, and set `FLOWSAFE_STAGING_PROBE_PORT` to the local port. Start the probe before enabling the subject Worker: @@ -265,4 +268,4 @@ Set these environment variables before running `node deploy/notification-staging | `FLOWSAFE_STAGING_CONFIG_FILE` | Fresh isolated proof only: absolute path to the private `0600` off config; omit to use the tracked staging Worker | | `FLOWSAFE_STAGING_PROBE_CONFIG_FILE` | Fresh isolated proof only: absolute path to the private `0600` local probe config targeting the selected Worker | -The controller prints the invocation ID only after D1 confirms one settled invocation, its notification, its operation receipt, and the watchdog's confirmed off deployment. It writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the controller exits without a confirmed off deployment, inspect that private directory's `watchdog.log` and verify the fence and current Worker version through the authenticated proxy and Wrangler before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. +The controller writes watchdog state and its error log under `/tmp/flowsafe-notification-control-/`. If the probe becomes unavailable, the watchdog still attempts the version-checked off deployment and reads the fence through the D1 API. An open or unreadable fence leaves completion unconfirmed. Inspect that private directory's `watchdog.log` and verify the fence and current Worker version before another attempt. An unconfirmed relock or changed deployment requires operator review; do not directly update the D1 fence row. diff --git a/packages/flowsafe/deploy/notification-staging-live.mjs b/packages/flowsafe/deploy/notification-staging-live.mjs index 56fb1138..60897ba9 100644 --- a/packages/flowsafe/deploy/notification-staging-live.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.mjs @@ -15,6 +15,11 @@ import { tmpdir } from 'node:os'; import { dirname, join, resolve as resolvePath } from 'node:path'; import { fileURLToPath } from 'node:url'; import { isDeepStrictEqual } from 'node:util'; +import { + decodeExecutionFenceMutationMetadata, + EXECUTION_FENCE_ROW_ID, + EXECUTION_FENCE_TABLE, +} from '#deployment-identity-protocol'; import { localProbe, readSecretFile, @@ -222,7 +227,7 @@ export function createWranglerControl({ function currentVersion() { return fullVersion(deployments()[0]); } - function proofEnabled(version) { + function proofEnabled(version, expectedMutationEpoch) { assert(versionPattern.test(version), 'invalid version'); const detail = json([ 'versions', @@ -257,6 +262,21 @@ export function createWranglerControl({ tenant[0].text === selected.tenant, 'enabled deployment tenant differs', ); + if (expectedMutationEpoch !== undefined) { + assert( + expectedMutationEpoch === 1 || expectedMutationEpoch === 2, + 'invalid enabled epoch', + ); + const epoch = bindings.filter( + (binding) => binding.name === 'STAGING_MUTATION_EPOCH', + ); + assert( + epoch.length === 1 && + epoch[0].type === 'plain_text' && + epoch[0].text === String(expectedMutationEpoch), + 'enabled epoch binding differs', + ); + } return flag[0].text === 'true'; } return { @@ -499,7 +519,9 @@ async function watchdog(stateFile) { baselineEpoch === 0 || baselineEpoch === 1, 'invalid watchdog baseline', ); - const probe = localProbe({ port, secret: readSecretFile(probeSecretPath) }); + const secret = readSecretFile(probeSecretPath); + const token = privateFile(cfTokenPath); + const probe = localProbe({ port, secret }); const selected = stagingConfig(configPath); assert(selected.name === workerName, 'watchdog Worker identity differs'); assert( @@ -512,12 +534,12 @@ async function watchdog(stateFile) { 'watchdog pinned off bundle is missing', ); const cli = createWranglerControl({ - token: privateFile(cfTokenPath), + token, configPath, }); assert( cli.currentVersion() === expectedVersion && - cli.proofEnabled(expectedVersion), + cli.proofEnabled(expectedVersion, baselineEpoch + 1), 'watchdog enabled version differs', ); writeFileSync(`${stateFile}.ready`, '', { mode: 0o600, flag: 'wx' }); @@ -529,9 +551,37 @@ async function watchdog(stateFile) { } await new Promise((resolve) => setTimeout(resolve, 1_000)); } - await sealFence(probe, baselineEpoch); + let sealFailure; + try { + await sealFence(probe, baselineEpoch); + } catch (error) { + sealFailure = error; + } cli.deployOffIfCurrent(expectedVersion, outdir, bundlePath); - confirmedFence(await probe('GET', '/admin/execution-fence'), baselineEpoch); + try { + const rows = cli.query(`SELECT * FROM ${EXECUTION_FENCE_TABLE}`); + assert( + rows.length === 1 && rows[0].id === EXECUTION_FENCE_ROW_ID, + 'authoritative fence row differs', + ); + const row = rows[0]; + const metadata = decodeExecutionFenceMutationMetadata(row); + assert( + row.proof_key === null && + row.proof_run_id === null && + metadata.proofTablePrefix === null && + metadata.proofWorkflowId === null && + metadata.proofStartToken === null, + 'authoritative fence proof differs', + ); + confirmedFence({ state: row.state, ...metadata }, baselineEpoch); + } catch (error) { + const failure = new AggregateError( + sealFailure ? [sealFailure, error] : [error], + 'staging fence relock is unconfirmed', + ); + throw new Error(privateDiagnostic(failure, [secret, token])); + } writeFileSync(`${stateFile}.off-confirmed`, '', { mode: 0o600, flag: 'wx' }); } @@ -560,13 +610,13 @@ async function controller() { let stage = 'enabled-version-preflight'; try { const cli = createWranglerControl({ token: settings.token, configPath }); + const baselineEpoch = settings.priorOffVersion === undefined ? 0 : 1; assert( cli.currentVersion() === settings.expectedVersion && - cli.proofEnabled(settings.expectedVersion), + cli.proofEnabled(settings.expectedVersion, baselineEpoch + 1), 'enabled version preflight differs', ); const probe = localProbe({ port: settings.port, secret: settings.secret }); - const baselineEpoch = settings.priorOffVersion === undefined ? 0 : 1; if (baselineEpoch === 1) { assert( settings.priorOffVersion !== settings.expectedVersion && diff --git a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs index 0fd44fc7..dcc0400a 100644 --- a/packages/flowsafe/deploy/notification-staging-live.node-test.mjs +++ b/packages/flowsafe/deploy/notification-staging-live.node-test.mjs @@ -50,13 +50,26 @@ function freshConfig() { return source; } -function versionBindings(source, proofEnabled) { +function versionBindings( + source, + proofEnabled, + mutationEpoch = proofEnabled ? 1 : 0, +) { return [ { name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', text: String(proofEnabled), }, + ...(mutationEpoch === null + ? [] + : [ + { + name: 'STAGING_MUTATION_EPOCH', + type: 'plain_text', + text: String(mutationEpoch), + }, + ]), { name: 'DEPLOYMENT_TENANT', type: 'plain_text', @@ -70,7 +83,7 @@ function versionBindings(source, proofEnabled) { ]; } -function fakeCli({ enabledFlag = true, priorVersion } = {}) { +function fakeCli({ enabledFlag = true, priorVersion, mutationEpoch } = {}) { const calls = []; let version = enabled; const command = (_executable, args) => { @@ -108,6 +121,7 @@ function fakeCli({ enabledFlag = true, priorVersion } = {}) { bindings: versionBindings( source, args[2] === enabled && enabledFlag, + mutationEpoch, ), }, }), @@ -189,6 +203,24 @@ test('retry evidence requires the immediately prior full off deployment', () => assert.throws(() => enabledPrior.cli.priorOffVersion(enabled), /not off/); }); +test('enabled version must pin the epoch the controller will open', () => { + for (const epoch of [null, '01', '0', '2']) { + const fixture = fakeCli({ mutationEpoch: epoch }); + assert.throws( + () => fixture.cli.proofEnabled(enabled, 1), + /epoch binding differs/, + ); + } + assert.equal( + fakeCli({ mutationEpoch: 1 }).cli.proofEnabled(enabled, 1), + true, + ); + assert.equal( + fakeCli({ mutationEpoch: 2 }).cli.proofEnabled(enabled, 2), + true, + ); +}); + test('enabled version requires exact D1 and tenant bindings', () => { const directory = mkdtempSync(join(tmpdir(), 'flowsafe-d1-mismatch-test-')); try { @@ -463,6 +495,10 @@ async function exerciseWatchdog(outcome) { const deployStartedFile = join(directory, 'deploy-started'); const releaseDeployFile = join(directory, 'release-deploy'); const effectStartedFile = join(directory, 'effect-started'); + const fenceFile = join(directory, 'fence.json'); + const deadProbe = + outcome === 'dead probe after relock' || + outcome === 'dead probe while open'; const accountId = outcome === 'parent death' ? 'c'.repeat(32) @@ -508,7 +544,7 @@ async function exerciseWatchdog(outcome) { preloadFile, `import childProcess from 'node:child_process'; import { syncBuiltinESMExports } from 'node:module'; -import { existsSync, mkdirSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; const originalSpawn = childProcess.spawn; const originalSpawnSync = childProcess.spawnSync; const originalNow = Date.now; @@ -553,6 +589,9 @@ childProcess.spawnSync = (executable, args, options) => { resources: { bindings: [{ name: 'NOTIFICATION_PROOF_ENABLED', type: 'plain_text', text: off || args[2] === '${disabled}' ? 'false' : 'true', + }, { + name: 'STAGING_MUTATION_EPOCH', type: 'plain_text', + text: off || args[2] === '${disabled}' ? '0' : process.env.TEST_RETRY === 'true' ? '2' : '1', }, { name: 'DEPLOYMENT_TENANT', type: 'plain_text', text: process.env.TEST_FRESH_CONFIG === 'true' @@ -564,7 +603,9 @@ childProcess.spawnSync = (executable, args, options) => { : 'b863a219-2096-4e13-92fe-4db592b67944', }] }, } : command === 'd1' ? [{ success: true, results: - sql.includes("name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')") + sql.includes('FROM flowsafe_execution_fence') + ? [JSON.parse(readFileSync(process.env.TEST_FENCE_FILE, 'utf8'))] + : sql.includes("name IN ('mastra_notifications', 'flowsafe_notification_operation_receipts')") ? effectStarted ? [{ name: 'mastra_notifications' }, { name: 'flowsafe_notification_operation_receipts' }] : [] : sql.includes('sqlite_master') @@ -615,14 +656,44 @@ syncBuiltinESMExports(); transitionRevision: outcome === 'sealed retry' ? 1 : 0, }; let startReceived = false; + let probeUnavailable = false; + const recordFence = () => + writeFileSync( + fenceFile, + JSON.stringify({ + id: 'deployment', + state: fence.state, + proof_key: null, + proof_run_id: null, + updated_at: 1, + last_transition_request: + fence.mutationEpoch === 0 ? null : 'test-transition', + transition_revision: fence.transitionRevision, + mutation_epoch: fence.mutationEpoch, + require_mutation_epoch: Number(fence.requireMutationEpoch), + proof_table_prefix: null, + proof_workflow_id: null, + proof_start_token: null, + }), + ); + recordFence(); const server = createServer(async (request, response) => { assert.equal(request.headers.authorization, `Bearer ${secret}`); const path = request.url; const method = request.method; requests.push(`${method} ${path}`); + if (probeUnavailable) { + response.destroy(); + return; + } if (method === 'POST' && path === '/start') { startReceived = true; writeFileSync(effectStartedFile, 'started'); + if (outcome === 'dead probe while open') { + probeUnavailable = true; + response.destroy(); + return; + } if (outcome === 'parent death' || outcome === 'fresh config') return; response.writeHead(200, { 'content-type': 'application/json' }); response.end( @@ -657,6 +728,12 @@ syncBuiltinESMExports(); requireMutationEpoch: true, transitionRevision: fence.transitionRevision + 1, }; + recordFence(); + if ( + outcome === 'dead probe after relock' && + fence.state === 'migration-locked' + ) + probeUnavailable = true; value = fence; } else { assert.equal(path, '/admin/execution-fence'); @@ -703,6 +780,7 @@ syncBuiltinESMExports(); TEST_DEPLOY_STARTED_FILE: deployStartedFile, TEST_RELEASE_DEPLOY_FILE: releaseDeployFile, TEST_EFFECT_STARTED_FILE: effectStartedFile, + TEST_FENCE_FILE: fenceFile, TEST_CLOCK_JUMP: outcome === 'controller request' ? 'true' : 'false', TEST_RETRY: outcome === 'sealed retry' ? 'true' : 'false', TEST_FRESH_CONFIG: outcome === 'fresh config' ? 'true' : 'false', @@ -757,16 +835,26 @@ syncBuiltinESMExports(); await waitFor(() => existsSync(offConfirmedFile)); assert.equal(readFileSync(offFile, 'utf8'), 'deployed'); assert.deepEqual(fence, { - state: 'migration-locked', + state: outcome === 'dead probe while open' ? 'open' : 'migration-locked', mutationEpoch: - outcome === 'sealed retry' ? 3 : outcome === 'mismatched probe' ? 1 : 2, + outcome === 'sealed retry' + ? 3 + : outcome === 'mismatched probe' || + outcome === 'dead probe while open' + ? 1 + : 2, requireMutationEpoch: true, transitionRevision: - outcome === 'sealed retry' ? 3 : outcome === 'mismatched probe' ? 1 : 2, + outcome === 'sealed retry' + ? 3 + : outcome === 'mismatched probe' || + outcome === 'dead probe while open' + ? 1 + : 2, }); assert.deepEqual( requests.filter((request) => request === 'POST /admin/execution-fence'), - outcome === 'mismatched probe' + outcome === 'mismatched probe' || outcome === 'dead probe while open' ? ['POST /admin/execution-fence'] : ['POST /admin/execution-fence', 'POST /admin/execution-fence'], ); @@ -778,6 +866,17 @@ syncBuiltinESMExports(); await waitFor( () => spawnSync('flock', ['--nonblock', lockFile, 'true']).status === 0, ); + if (deadProbe) { + await waitFor(() => controller.exitCode !== null); + assert.equal( + controller.exitCode, + outcome === 'dead probe after relock' ? 0 : 1, + ); + assert.equal( + existsSync(`${stateFile}.off-confirmed`), + outcome === 'dead probe after relock', + ); + } } finally { controller?.kill('SIGKILL'); if (Number.isInteger(watcherPid)) { @@ -808,6 +907,12 @@ test('watchdog owns off deployment after controller request', () => test('sealed retry persists watchdog baseline before unlocking and relocks at three', () => exerciseWatchdog('sealed retry')); +test('watchdog deploys off through probe failure after the controller relocks', () => + exerciseWatchdog('dead probe after relock')); + +test('watchdog deploys off but refuses confirmation when the dead probe leaves an open fence', () => + exerciseWatchdog('dead probe while open')); + test('controller refuses a contended deployment lock before Wrangler preflight', async () => { const directory = mkdtempSync( join(tmpdir(), 'flowsafe-lock-contention-test-'), diff --git a/packages/flowsafe/deploy/notification-staging-native.mjs b/packages/flowsafe/deploy/notification-staging-native.mjs index c7b6e79e..182a741a 100644 --- a/packages/flowsafe/deploy/notification-staging-native.mjs +++ b/packages/flowsafe/deploy/notification-staging-native.mjs @@ -100,7 +100,7 @@ async function run() { `INSERT OR IGNORE INTO flowsafe_deployment (id, tenant_tag, provisioned_at) VALUES (1, '${tag}', datetime('now')); ` + `${EXECUTION_FENCE_DDL}; ` + `INSERT OR IGNORE INTO ${EXECUTION_FENCE_TABLE} (id, state, proof_key, proof_run_id, updated_at) ` + - `VALUES ('${EXECUTION_FENCE_ROW_ID}', 'open', NULL, NULL, ${Date.now()});`, + `VALUES ('${EXECUTION_FENCE_ROW_ID}', 'migration-locked', NULL, NULL, ${Date.now()});`, ); await lifecycle.start('notification-staging-native', () => { const child = spawn( @@ -128,6 +128,8 @@ async function run() { `STAGING_PROBE_SECRET:${probeSecret}`, '--var', 'NOTIFICATION_PROOF_ENABLED:true', + '--var', + 'STAGING_MUTATION_EPOCH:1', ], { cwd: flowsafe, @@ -149,6 +151,28 @@ async function run() { 'private start admitted an unauthenticated call', refused, ); + const opened = await fetch(`http://127.0.0.1:${port}/admin/execution-fence`, { + method: 'POST', + headers: { + authorization: `Bearer ${maintenanceSecret}`, + 'content-type': 'application/json', + }, + body: JSON.stringify({ + expected: 'migration-locked', + next: 'open', + expectedMutationEpoch: 0, + expectedRevision: 0, + advanceMutationEpoch: true, + }), + signal: AbortSignal.timeout(90_000), + }); + expect(opened.status === 200, 'native execution fence did not open'); + const fence = await opened.json(); + expect( + fence.mutationEpoch === 1 && fence.requireMutationEpoch === true, + 'native execution fence did not require the enabled epoch', + fence, + ); const started = await post('/__staging/start', probeSecret); expect(started.status === 200, 'guarded start/register failed', { response: started, diff --git a/packages/flowsafe/deploy/notification-staging.worker.ts b/packages/flowsafe/deploy/notification-staging.worker.ts index 86e44b0b..206bbd2b 100644 --- a/packages/flowsafe/deploy/notification-staging.worker.ts +++ b/packages/flowsafe/deploy/notification-staging.worker.ts @@ -36,6 +36,7 @@ import { executionFenceFor, type InitResult, init, + mutationEpochFromHeader, type RunnerRuntime, startIdempotencyFor, ThreadDurableObject, @@ -60,6 +61,7 @@ interface Env { DEPLOYMENT_IDENTITY_SECRET: string; MAINTENANCE_ADMIN_SECRET: string; STAGING_PROBE_SECRET: string; + STAGING_MUTATION_EPOCH: string; NOTIFICATION_PROOF_ENABLED?: string; CF_VERSION_METADATA: { id: string }; RUNNER: DurableObjectNamespace; @@ -296,6 +298,8 @@ export class NotificationStagingThread extends ThreadDurableObject { const config = { workflows: [], + mutationEpoch: (env: Env) => + mutationEpochFromHeader(env.STAGING_MUTATION_EPOCH), systemPrincipalId: SYSTEM_ID, buildVerifier: () => staticTokenVerifier(new Map()), maintenance: { @@ -363,6 +367,7 @@ const config = { }, storeFactory, deploymentTag: env.DEPLOYMENT_TENANT, + mutationEpoch: mutationEpochFromHeader(env.STAGING_MUTATION_EPOCH), buildService: (store) => new ApprovalService({ store, diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc index f2c42ec8..eb2846a4 100644 --- a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -43,6 +43,7 @@ ], "vars": { "DEPLOYMENT_TENANT": "flowsafenotificationstaging", + "STAGING_MUTATION_EPOCH": "0", "NOTIFICATION_PROOF_ENABLED": "false" } } From c2cf93411583255d2f9831c06bd86fd6bf6ff456 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:27:03 +0400 Subject: [PATCH 71/73] Minify the isolated notification staging Worker Reduce bundled startup parsing work while retaining Wrangler name preservation. Native epoch-fenced notification proof passes with the minified build. --- packages/flowsafe/deploy/notification-staging.wrangler.jsonc | 1 + 1 file changed, 1 insertion(+) diff --git a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc index eb2846a4..4996282c 100644 --- a/packages/flowsafe/deploy/notification-staging.wrangler.jsonc +++ b/packages/flowsafe/deploy/notification-staging.wrangler.jsonc @@ -2,6 +2,7 @@ "$schema": "https://unpkg.com/wrangler/config-schema.json", "name": "anchorage-flowsafe-remote-step-staging", "main": "./notification-staging.worker.ts", + "minify": true, "compatibility_date": "2026-09-10", "compatibility_flags": ["nodejs_compat"], "alias": { From e8e68de7e1e479e8fe68c5e01192a7e0a812c298 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:45:00 +0400 Subject: [PATCH 72/73] Verify remote-step APIs from the packed FlowSafe consumer --- .../flowsafe/scripts/agent-host-pack-test.mjs | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/packages/flowsafe/scripts/agent-host-pack-test.mjs b/packages/flowsafe/scripts/agent-host-pack-test.mjs index 7008df81..c4d7dd7c 100644 --- a/packages/flowsafe/scripts/agent-host-pack-test.mjs +++ b/packages/flowsafe/scripts/agent-host-pack-test.mjs @@ -717,7 +717,7 @@ import { type SignalDatabase, type SignalRouterOptions, } from '@proofoftech/flowsafe/signals'; import { - type RunnerRuntime, type RunExecutionIdentity, type StartRunOptions, + type RunnerRuntime, type RunExecutionIdentity, type StartRunOptions, type RemoteStepJournal, type ThreadScope, type StartReservationReading, type PersistedStartResult, type RunSummary, } from '@proofoftech/flowsafe/do-runner'; @@ -729,6 +729,9 @@ import { import { type AgentStartAuthority, type FlowsafeDurableAgent, } from '@proofoftech/flowsafe/agent-runner'; +import { + createWorkflowRemoteStepPreRoute, type AgentThreadDispatchTopology, +} from '@proofoftech/flowsafe/agent-host'; // @ts-expect-error internal context capture is not a root export import type { captureActorContext as RootCapture } from '@proofoftech/flowsafe'; // @ts-expect-error internal context capture is not an approval export @@ -839,6 +842,9 @@ const workerConfig: FlowsafeWorkerConfig = { createFlowsafeWorker(workerConfig); createFlowsafeWorker({ ...workerConfig, mutationEpoch: 0 }); createFlowsafeWorker({ ...workerConfig, beforeStart: legacyWorkerHook }); +declare const remoteStepJournal: RemoteStepJournal; +declare const remoteStepDispatch: AgentThreadDispatchTopology; +createFlowsafeWorker({ ...workerConfig, preRoutes: createWorkflowRemoteStepPreRoute(remoteStepJournal, remoteStepDispatch) }); void [contextInput, doContextInput, legacyDoInput, legacyRouterHook, routerPolicyResult, workerPolicyResult]; const onPrepared = (execution: RunExecutionIdentity): void => { void execution.startToken; }; const legacyOptions: StartRunOptions = { runId: 'legacy' }; @@ -1369,6 +1375,18 @@ const repair = await capability.withInitialAdmission({ execution: repairExecutio }, () => workflow.createRun({ runId: repairExecution.runId })); const repairRequest = { expected: repair.witness.row, execution: repairExecution, attemptToken: 'repair-correlation', nowMs: 1700000000123 }; const observer = doRunner.init({ storage }, { executionFence: new doRunner.ExecutionFenceStore(binding), startIdempotency: 'none' }).runtime; +const remoteStepBinding = sqliteUnitDatabase(openSqlite()); +const remoteStepJournal = new doRunner.RemoteStepJournal({ + db: remoteStepBinding, fence: new doRunner.ExecutionFenceStore(remoteStepBinding), runtime: observer, + deploymentId: 'packed-remote-step', tablePrefix: '', allowedToolIds: ['remote-tool'], + verifyCompletion: async () => { throw new Error('no remote completion is configured'); }, +}); +await remoteStepJournal.initialize(); +assert.equal(await remoteStepJournal.getWorkflow('absent-invocation'), undefined); +const privateRemoteRoute = host.createWorkflowRemoteStepPreRoute(remoteStepJournal, {}); +const privateRemoteMethod = await privateRemoteRoute(new Request('https://packed.test/_flowsafe/workflow-remote-step/ticket'), {}, {}, {}); +assert.equal(privateRemoteMethod.status, 405); +assert.deepEqual(await privateRemoteMethod.json(), { error: 'method not allowed' }); observer.register(workflow); const initial = await observer.authoritativeStartState(workflow.id, repairExecution.runId); assert.equal(initial.kind, 'initial'); From 68d2b354c3b8e956f88abab5a50ffc00cbd71e94 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 16 Oct 2026 05:48:43 +0000 Subject: [PATCH 73/73] release: version packages --- .changeset/flowsafe-remote-step-broker.md | 5 ----- packages/agent-starter/CHANGELOG.md | 7 +++++++ packages/agent-starter/package.json | 2 +- packages/flowsafe/CHANGELOG.md | 6 ++++++ packages/flowsafe/package.json | 2 +- packages/showcase/CHANGELOG.md | 7 +++++++ packages/showcase/package.json | 2 +- 7 files changed, 23 insertions(+), 8 deletions(-) delete mode 100644 .changeset/flowsafe-remote-step-broker.md diff --git a/.changeset/flowsafe-remote-step-broker.md b/.changeset/flowsafe-remote-step-broker.md deleted file mode 100644 index ecfbe192..00000000 --- a/.changeset/flowsafe-remote-step-broker.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@proofoftech/flowsafe': minor ---- - -Add a D1-backed remote-step journal for agent and workflow suspensions, one-time host dispatch permits, retained results, and a host context derived from the stored invocation. Versioned private state egress carries internal dispatch metadata without changing the legacy route. A host-owned notification operation stores its effect and replay receipt in one D1 transaction with a configurable retained-receipt cap. General Mastra continuation remains fail-closed without generation-fenced snapshot writes; hosts must supply trusted dispatch transport and downstream effect receipts before enabling remote execution. diff --git a/packages/agent-starter/CHANGELOG.md b/packages/agent-starter/CHANGELOG.md index dfa67860..7c483f58 100644 --- a/packages/agent-starter/CHANGELOG.md +++ b/packages/agent-starter/CHANGELOG.md @@ -1,5 +1,12 @@ # anchorage-agent-starter +## 0.0.27 + +### Patch Changes + +- Updated dependencies [ee4003a] + - @proofoftech/flowsafe@0.29.0 + ## 0.0.26 ### Patch Changes diff --git a/packages/agent-starter/package.json b/packages/agent-starter/package.json index 00625d3d..ae58c9b6 100644 --- a/packages/agent-starter/package.json +++ b/packages/agent-starter/package.json @@ -1,6 +1,6 @@ { "name": "anchorage-agent-starter", - "version": "0.0.26", + "version": "0.0.27", "private": true, "description": "Production-shaped Cloudflare Workers starter for durable, approval-gated Anchorage agents", "type": "module", diff --git a/packages/flowsafe/CHANGELOG.md b/packages/flowsafe/CHANGELOG.md index 33aaf677..097bc5d0 100644 --- a/packages/flowsafe/CHANGELOG.md +++ b/packages/flowsafe/CHANGELOG.md @@ -1,5 +1,11 @@ # @proofoftech/flowsafe +## 0.29.0 + +### Minor Changes + +- ee4003a: Add a D1-backed remote-step journal for agent and workflow suspensions, one-time host dispatch permits, retained results, and a host context derived from the stored invocation. Versioned private state egress carries internal dispatch metadata without changing the legacy route. A host-owned notification operation stores its effect and replay receipt in one D1 transaction with a configurable retained-receipt cap. General Mastra continuation remains fail-closed without generation-fenced snapshot writes; hosts must supply trusted dispatch transport and downstream effect receipts before enabling remote execution. + ## 0.28.0 ### Minor Changes diff --git a/packages/flowsafe/package.json b/packages/flowsafe/package.json index 5373f10d..eea6389c 100644 --- a/packages/flowsafe/package.json +++ b/packages/flowsafe/package.json @@ -1,6 +1,6 @@ { "name": "@proofoftech/flowsafe", - "version": "0.28.0", + "version": "0.29.0", "publishConfig": { "access": "public" }, diff --git a/packages/showcase/CHANGELOG.md b/packages/showcase/CHANGELOG.md index 74610893..dbac3fb1 100644 --- a/packages/showcase/CHANGELOG.md +++ b/packages/showcase/CHANGELOG.md @@ -1,5 +1,12 @@ # showcase +## 0.0.33 + +### Patch Changes + +- Updated dependencies [ee4003a] + - @proofoftech/flowsafe@0.29.0 + ## 0.0.32 ### Patch Changes diff --git a/packages/showcase/package.json b/packages/showcase/package.json index b53cc076..9902f688 100644 --- a/packages/showcase/package.json +++ b/packages/showcase/package.json @@ -1,6 +1,6 @@ { "name": "showcase", - "version": "0.0.32", + "version": "0.0.33", "private": true, "description": "Anchorage showcase — six runnable workflows and seven guardrail scenarios behind one React frontend", "type": "module",