Skip to content

feat: add external storage runner - #1696

Merged
jmaeagle99 merged 15 commits into
mainfrom
extstore-foundation
Oct 23, 2026
Merged

jmaeagle99 merged 15 commits into
mainfrom
extstore-foundation

Conversation

@jmaeagle99

@jmaeagle99 jmaeagle99 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

Adds the external storage runner: the passes that move payloads to and from a storage driver, and the reference detection they depend on. ExternalStorageRunner offloads eligible payloads from a message on the way out and resolves references back to their payloads on the way in.

Why?

External storage (keeping large payloads out of history by offloading them to a user-supplied driver) has to apply the same offload and retrieval steps at every worker and client path that sends or receives payloads: measure against the threshold, group by driver, and swap references for payloads. This runner is that logic in one place, so those paths call a single component instead of each reimplementing it.

Out of scope

Deliberately not in this change, so the runner can land first:

  • Fields are visited one at a time. Per-field concurrency is a later change.
  • Reference encoding and decoding are not supported and will be added in a later change.
  • No ambient namespace on the select/store contexts, and no Nexus storage target kind.
  • Nothing builds the concurrency limiter. The runner takes limiters from the caller, and there is no warning when a driver completes an operation without taking a permit.
  • No metrics.
  • Payloads inside system Nexus envelopes are not offloaded.
  • Per-command targets, and the replay and failure-reporting rules, are left to whatever calls the runner.

Checklist

  1. How was this tested: unit tests covering both directions over real messages and payload slices, plus cargo +nightly fmt --check, cargo lint, clippy, and cargo test with and without experimental.

  2. Any docs updates needed? No

@jmaeagle99 jmaeagle99 added the skip-changelog PR changes do not require changelog updates label Oct 21, 2026
@jmaeagle99
jmaeagle99 force-pushed the extstore-foundation branch 7 times, most recently from 0ce814d to 4d36a8a Compare October 22, 2026 19:56
@jmaeagle99
jmaeagle99 marked this pull request as ready for review October 22, 2026 20:24
@jmaeagle99
jmaeagle99 requested a review from a team as a code owner October 22, 2026 20:24
Comment thread crates/common/src/external_storage_references.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
Comment thread crates/common/src/external_storage_runner.rs Outdated
@jmaeagle99
jmaeagle99 merged commit 42e9ada into main Oct 23, 2026
44 of 45 checks passed
@jmaeagle99
jmaeagle99 deleted the extstore-foundation branch October 23, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog PR changes do not require changelog updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants