Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
617a6a2
Plan durable FlowSafe remote-step broker
gcharang Oct 11, 2026
02d5975
Validate persisted tool resume data before runtime continuation
gcharang Oct 11, 2026
0615cd8
Accept scalar tool suspension payloads on resume
gcharang Oct 11, 2026
fa06037
Journal remote-step suspensions and gated dispatch permits
gcharang Oct 11, 2026
667c07c
Bind remote permits to current run authority
gcharang Oct 11, 2026
13f426d
Align direct proof bounds with remote-step inventory
gcharang Oct 11, 2026
2afc8ff
Bind remote results to one consumed dispatch permit
gcharang Oct 11, 2026
d109072
Retain remote results until exact continuation proof
gcharang Oct 11, 2026
ebe133e
Align FlowSafe inventory spike with remote-step work
gcharang Oct 11, 2026
3c286b8
Add optional FlowSafe remote-step host continuation
gcharang Oct 11, 2026
f7db3e6
Account for remote-step inventory in Fleet scenario
gcharang Oct 11, 2026
8888fa9
Plan authoritative resource binding for remote steps
gcharang Oct 11, 2026
9740480
Preserve remote resume principal kind and timeout authority
gcharang Oct 11, 2026
6943aa6
Export remote-step host types for API docs
gcharang Oct 11, 2026
82fed07
Prove retained remote result survives a fresh host
gcharang Oct 11, 2026
469f766
feat(flowsafe): version remote step journal schema
gcharang Oct 11, 2026
940c0ce
fix(flowsafe): validate journal schema before migration
gcharang Oct 11, 2026
17896ec
feat(flowsafe): bind remote journal to start resource
gcharang Oct 11, 2026
3c207d4
fix(flowsafe): bind host resume to start resource
gcharang Oct 11, 2026
4096fbd
feat(flowsafe): settle retained remote results durably
gcharang Oct 11, 2026
ff64a6a
docs: plan fenced remote resume recovery
gcharang Oct 11, 2026
d8c54b6
fix(flowsafe): bind remote resume to topology resource
gcharang Oct 12, 2026
61b157b
test(flowsafe): exercise remote step journal across workerd restarts
gcharang Oct 12, 2026
839f462
test(flowsafe): derive spike status from persisted snapshot
gcharang Oct 12, 2026
66e239e
feat(flowsafe): record atomic notification effects
gcharang Oct 13, 2026
ca2937c
feat(flowsafe): expose host-owned dispatch permit port
gcharang Oct 13, 2026
c3b86e3
feat(flowsafe): derive host context from stored invocation
gcharang Oct 13, 2026
9126395
test(flowsafe): verify remote journal replay after restart
gcharang Oct 13, 2026
30c3e43
Merge origin/dev into FlowSafe remote-step plan
gcharang Oct 13, 2026
f1fd6d9
feat(flowsafe): cap retained notification receipts
gcharang Oct 13, 2026
61d4150
test(flowsafe): prove local authenticated dispatch admission
gcharang Oct 13, 2026
ee4003a
chore(flowsafe): record remote-step broker release
gcharang Oct 13, 2026
06ef249
docs(flowsafe): plan workflow-target remote-step journal
gcharang Oct 13, 2026
937c12a
feat(flowsafe): support private state-egress identity
gcharang Oct 13, 2026
ed9bfad
Add versioned workflow remote-step journal
gcharang Oct 13, 2026
e176092
Describe workflow journal and private state egress in release note
gcharang Oct 13, 2026
592d9a6
Expose host-owned workflow remote-step port
gcharang Oct 13, 2026
c538793
Persist verified workflow start principal after admission
gcharang Oct 13, 2026
25a5277
Read workflow start principal through private DO authority
gcharang Oct 13, 2026
c5d9464
Add authenticated private workflow remote-step route
gcharang Oct 13, 2026
785ef55
Prove notification receipt contention in native D1
gcharang Oct 13, 2026
9723703
Prove private workflow route across workerd restart
gcharang Oct 13, 2026
9a455ff
Plan restricted notification recovery authority
gcharang Oct 13, 2026
648dee9
Fence restricted notification recovery in FlowSafe journal
gcharang Oct 13, 2026
c0ede41
Verify notification effects before FlowSafe resume admission
gcharang Oct 14, 2026
a3af945
Run selected notification effects under FlowSafe host authority
gcharang Oct 14, 2026
ddcf497
Recover admitted notification effects through trusted host
gcharang Oct 14, 2026
cf1f2c8
Schedule bounded notification effect recovery
gcharang Oct 14, 2026
4b516b8
Keep notification delivery types in D1 signal layer
gcharang Oct 14, 2026
c3293a7
Assert notification receipt digests in workerd fidelity
gcharang Oct 14, 2026
ad5dbec
Keep internal plans outside public guide index
gcharang Oct 14, 2026
3e030ce
Prepare private notification recovery staging topology
gcharang Oct 14, 2026
edf09f7
Use a valid FlowSafe staging deployment tag
gcharang Oct 14, 2026
bebe855
Add guarded notification producer to private staging
gcharang Oct 14, 2026
cd344ee
Proxy private staging fence administration locally
gcharang Oct 14, 2026
3a1a4ce
Add locked notification staging control and watchdog
gcharang Oct 14, 2026
b76f7ef
Retain private Wrangler failure diagnostics
gcharang Oct 14, 2026
6131bb6
Exercise staging watchdog after controller death
gcharang Oct 14, 2026
eba22e1
Hold staging deploy lock through watchdog recovery
gcharang Oct 14, 2026
3f03880
Keep Node staging tests out of Vitest discovery
gcharang Oct 14, 2026
ced8376
Document referenced FlowSafe public types
gcharang Oct 15, 2026
572ff9e
Reuse validated workflow dispatch row
gcharang Oct 15, 2026
cada4fb
Reuse validated remote-step journal rows
gcharang Oct 15, 2026
297cce4
Fence notification receipt cap to one D1 scope
gcharang Oct 15, 2026
bf26b96
Prove cross-prefix receipt exclusion in native D1
gcharang Oct 15, 2026
2f9b137
Create receipt scope table during notification initialization
gcharang Oct 15, 2026
2634b77
Handle absent FlowSafe staging receipt table at preflight
gcharang Oct 15, 2026
ba74213
Keep REST SDK out of FlowSafe staging startup
gcharang Oct 15, 2026
dbfd57c
Recover empty FlowSafe staging proof after sealed fence
gcharang Oct 15, 2026
922a258
Bind fresh FlowSafe staging proof to one isolated resource
gcharang Oct 15, 2026
f1b7668
Bind staging execution to its trusted mutation epoch
gcharang Oct 16, 2026
c2cf934
Minify the isolated notification staging Worker
gcharang Oct 16, 2026
e8e68de
Verify remote-step APIs from the packed FlowSafe consumer
gcharang Oct 16, 2026
c4ef38b
Merge pull request #123 from ProofOfTechOrg/plan/flowsafe-remote-step…
gcharang Oct 16, 2026
68d2b35
release: version packages
github-actions[bot] Oct 16, 2026
91db19e
Merge pull request #128 from ProofOfTechOrg/changeset-release/dev
gcharang Oct 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/deployment-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,15 @@ Decommission the whole physical resource set after revoking traffic and credenti

When `init()` or `createD1Storage()` receives `tablePrefix`, pass the same value as `storageTablePrefix` to `createFlowsafeWorker()`. The shared rule accepts an empty prefix or requires an ASCII letter or underscore first, followed by ASCII letters, numbers, or underscores. The prefix can contain at most 39 characters so every final Mastra table identifier stays within its 63-character limit. Maintenance applies it to workflow snapshots, threads/messages, background tasks and their snapshots, notifications, thread state, and schedule-trigger history. All six exported low-level purge functions validate the same contract before preparing D1 statements. Approval, resource-owner, deployment-sentinel, subscription, and other fixed-schema tables remain unprefixed. A valid but mismatched value targets a different table family; Flowsafe does not auto-discover it.

Host-owned notification effects register one receipt-writing table prefix and one receipt limit in the deployment's D1 database. The default limit is 100,000 retained receipts; a deployment can set a lower limit. Existing receipts remain available for replay at the limit. Ordinary notification storage can still use other prefixes, but those prefixes cannot write operation receipts in the same deployment.

To upgrade a deployment with retained operation receipts:

1. Lock its execution fence and wait for earlier receipt writers to finish. Keep every Worker and Durable Object that can write the D1 database under exclusive deployment control.
2. Inspect the D1 tables ending in `flowsafe_notification_operation_receipts`. If more than one prefix has retained rows, leave the fence locked and resolve the histories before enabling effects. Do not delete receipts to meet the limit.
3. Deploy the version that registers receipt scope to every writer of that D1 database. Set one prefix and one limit across those writers. Check the active versions before unlocking. An older writer can bypass the registration, so do not run mixed versions with the fence open.
4. Unlock the fence. The first new receipt operation registers the prefix and limit; later writes with a different value fail closed. Check the `flowsafe_notification_receipt_scope` row after the first operation. Fence the deployment again before any rollback to an older writer.

Idle-thread retention does not release thread or resource ownership. A thread can still have an agent binding, schedule, subscription, goal, or other standing wake source after its idle memory rows expire. An explicit host teardown must remove every authoritative standing record before it calls `ActorContext.releaseResource()` for the thread and resource claims.

## Egress and process execution
Expand Down
333 changes: 333 additions & 0 deletions docs/plans/2026-10-12-remote-step-durable-resume-recovery.md

Large diffs are not rendered by default.

81 changes: 81 additions & 0 deletions docs/plans/2026-10-13-workflow-remote-step-journal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Workflow-target remote-step journal

> Content type: implementation plan. Audience: FlowSafe maintainers. Status: planned; customer dispatch remains disabled.

## Baseline and environment

- Branch `plan/flowsafe-remote-step-broker` at `ee4003a42882c0d1282004d5fed0274a5b4e2939` (2026-10-13). Reconfirm the appendix anchors against the checked-out commit before editing. The concurrent state-egress checkpoint owns `src/host-kit/state-egress.ts` and `flowsafe-worker.ts`; this plan does not edit them.
- Node and pnpm come from the root manifest. Work in the linked worktree with the shared package store. Capture checks in `/tmp` with a final `EXIT_STATUS` line. D1 `batch()` is transactional according to [Cloudflare's API documentation](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch).
- This is a default-off journal contract. No workflow host route, ASP permit authority, customer dispatch, or automatic `run.resume` is enabled by this checkpoint.

## Decision

Extend the existing `RemoteStepJournal` D1 authority to distinguish `agent` and `workflow` target rows. Preserve every v3 agent row and its permit/result state. A workflow row uses the trusted start provenance `{kind:'workflow', id: workflowId}` and has no agent, thread, or resource ID. Its suspension carries a versioned marker and exact JSON request bytes; the journal stores a SHA-256 digest of those bytes. Registration and permit consumption compare the same authoritative snapshot, principal, deployment, start token, fence generation, suspension identity, request bytes, and deadline as the agent path. The first retained result remains immutable under the consumed permit digest.

The workflow suspension wire shape is `suspendPayload[stepPath.join('.')].remoteStep = { marker: 'flowsafe.remote-step.workflow/v1', request: <JSON value> }`, alongside `type: 'suspension'`. The trusted workflow host must emit this shape; user code cannot obtain authority merely by constructing the payload. The marker is a versioned parser selector, not proof of ownership. Confirm it against the ASP v2 author before wiring a caller.

Reject synthetic agent/thread/resource IDs: they would make a workflow row appear to be an agent row and let agent-host routing or cleanup misinterpret its authority. Reject a second workflow permit table: two authorities would need a new cross-table uniqueness and replay protocol. Reject a preflight-only public API: it could be mistaken for dispatch authority while storing no permit or result. Keep general Mastra `run.resume` and automatic recovery fail-closed: the pinned `@mastra/core@1.73.0` does not fence all snapshot write paths by remote attempt generation (see [durable-resume plan](2026-10-12-remote-step-durable-resume-recovery.md)).

## Reuse and package decision

Reuse `RemoteStepJournal`, `ExecutionFenceStore`, `assertExecutionPrincipal`, `isPathSafeId`, authoritative runtime status/start state, D1 conditional updates, and Web Crypto SHA-256 in `src/do-runner/remote-step-journal.ts`. The existing direct dependencies are `@mastra/core@1.73.0`, `@mastra/cloudflare-d1@1.4.0`, and `jose@6.2.8`; none provides this host-specific journal schema and CAS contract. A new package adds no useful boundary. A separate custom store duplicates the existing permit protocol. The narrow custom change is a discriminated target row and suspension selector in the existing journal. No dependency changes.

## Checkpoints

1. **Schema v4.** Rebuild the journal table transactionally because SQLite cannot remove `NOT NULL` from agent identity columns in place. Add `target_kind`, `suspension_marker`, `request_digest`, and `write_revision`; make agent/thread/resource nullable. Copy v3 rows as `agent`, with their original marker, `NULL` request digest, and revision zero; preserve their status, permit, result, lease, and settlement fields. The agent branch continues to compare the exact stored `request_json`; it does not require a digest on migrated rows. A workflow row requires a digest. Add separate partial unique indexes for agent and workflow suspension identities, plus the recovery index. Validate exact schema/index/trigger definitions and refuse unknown versions. A v4 update increments `write_revision` in the same statement as its state transition. A D1 trigger rejects updates that do not advance it; v3 code omits that column, so an old in-flight isolate cannot mutate a row after migration. `target_kind NOT NULL` rejects its inserts. Test a v3 initialized journal instance attempting both operations after another instance migrates. A v4 binary reads migrated agent rows unchanged. Keep rollback to a v4-aware binary; an old v3 binary fails closed.
2. **Workflow storage port.** Add a workflow-specific registration method, a workflow getter, and typed workflow invocation. Registration accepts a trusted workflow address and original principal, verifies start provenance and exact suspension, then performs a snapshot-guarded insert/update. Existing agent-host `get` and `register` remain agent-only. Shared permit, consume, result and retained-result methods dispatch by target kind inside the same journal authority. No caller is wired yet.
3. **Proof.** Use real SQLite-backed D1 semantics for old-row preservation, conflicting registration, restart/reopen, permit replay, wrong request/principal/start token/fence/deadline, and result conflict. Add a workerd fresh-isolate scenario before claiming a live persistence proof. The default-off package may merge without a customer route; it may not be described as a working WFP dispatch path until the authenticated host route and workerd proof land.

## Verification

After each code checkpoint, derive scoped checks from changed paths. The journal schema is a shared contract, so the owning package's full test suite applies once under the quality-gate exception. Candidate commands:

```bash
pnpm --filter @proofoftech/flowsafe exec vitest run src/do-runner/remote-step-journal.test.ts
pnpm exec biome check packages/flowsafe/src/do-runner/remote-step-journal.ts packages/flowsafe/src/do-runner/remote-step-journal.test.ts
pnpm --filter @proofoftech/flowsafe typecheck:test
pnpm --filter @proofoftech/flowsafe build
pnpm --filter @proofoftech/flowsafe test
```

Each command needs its own `/tmp` log and actual exit status. Done for this checkpoint means schema upgrade preserves v3 rows, workflow register/permit/result behavior passes, agent-host behavior passes, and no public workflow dispatch or resume route is enabled. The top-level quality-gate owner runs T2 clean-code, QA, architecture, and security lanes, then commits. Live enablement additionally requires a real D1/workerd restart and replay proof, authenticated host binding, and ASP grant verification.

## Appendix: current-state anchors

At the baseline SHA, `src/do-runner/remote-step-journal.ts` declares an agent-only address and row:

```ts
export interface RemoteStepAddress {
deploymentId: string;
tablePrefix: string;
workflowId: string;
agentId: string;
threadId: string;
resourceId: string;
runId: string;
}
```

The `RemoteStepJournal.initialize` table has `agent_id TEXT NOT NULL`, `thread_id TEXT NOT NULL`, and `resource_id TEXT NOT NULL`, with a unique key containing those columns. Its `readSchema` accepts version 3 or 2 and exact column/index definitions. Before → after: v3 agent-only fields and uniqueness → v4 discriminated target with nullable workflow agent fields and partial target-specific unique indexes. A simple nullable-column alteration is insufficient because the existing unique key cannot represent workflow identity.

`RemoteStepJournal.#execution` currently refuses workflow starts:

```ts
if (
state?.storage !== 'd1' ||
target?.kind !== 'agent' ||
!isPathSafeId(target.resourceId)
)
throw new RemoteStepRefusal(
503,
'remote step run identity is unreadable',
);
```

Before → after: the agent branch retains its exact checks; the workflow branch requires `target.kind === 'workflow'`, `target.id === address.workflowId`, no agent-start provenance, matching original principal, deployment/table prefix/run ID/start token, and the current execution fence. It never fills agent fields from workflow IDs.

`selectedSuspension` currently matches `toolCallSuspended.remoteStep === REMOTE_STEP_MARKER` and reads `tool.request`. Before → after: keep that agent selector and add a separate workflow selector for the versioned `remoteStep.marker` and `remoteStep.request`, with one selected path and exact request bytes/digest.

`RemoteStepJournal.issuePermit` and `consumePermit` compare `suspension.requestJson` against `JSON.stringify(invocation.request)` and conditionally update the journal with `request_json = ?`, snapshot equality, and `executionFenceAdmissionSql`. Before → after: retain those CAS predicates and add the target kind and marker. The workflow branch also compares its stored request digest; legacy agent rows continue to use exact request bytes without a digest. Every v4 state update advances `write_revision` under a D1 trigger, which rejects a previously initialized v3 isolate's updates. `recordResult` keeps `permit_digest` and immutable result comparison. No code path retries `run.resume` from a retained result.

`src/do-runner/execution-admission.ts` defines `StartIdentity.target` as a union with `{kind:'workflow',id}`. `src/do-runner/runtime.ts` already checks that a recovered workflow target ID equals its execution workflow ID. The new journal branch uses that trusted provenance rather than creating another identity format.
1 change: 1 addition & 0 deletions docs/proposals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Documents in this directory are design records. Read each document's status befo

- [Breakwater improvement roadmap](breakwater-improvement-roadmap.md): mixed shipped and planned work
- [Durable result gate](durable-result-gate.md): proposal; not implemented
- [Remote step broker](flowsafe-remote-step-broker.md): implementation plan; not implemented
- [Mature package adoption plan](mature-package-adoption-plan.md): implemented and verified
- [Model gateway policy](model-gateway-policy.md): partially implemented; provider, budget, model-egress, structured-output, and scoring gates remain planned

Expand Down
Loading
Loading