Skip to content

feat(rust): codeupipe-compile, a pipeline written as plain Rust - #26

Merged
JoshuaWink merged 2 commits into
mainfrom
feat/rust-compile
Oct 29, 2026
Merged

JoshuaWink merged 2 commits into
mainfrom
feat/rust-compile

Conversation

@JoshuaWink

Copy link
Copy Markdown
Member

What

packages/codeupipe-compile-rs: codeupipe-compile reads a Rust pipeline and its steps with syn, and writes them as one module of plain Rust with no codeupipe:

  • a struct with one Option field for each key that the steps use;
  • one function for each step, whose body is the step's own code, with each key as a field and each setting as its literal;
  • pub fn run(input: &XData) -> Result<XData, StepError>.

The same rule as the other compilers: code that compiles gives the same output as the pipeline. A pipeline for which the compiler cannot keep that rule is refused, with the place and the fix. The refusals cover a Hook, a nested pipeline, a setting that is not a literal, a key that is not text in the code, a key with two types, a read with no type, and another method of the Payload. The compiler is itself a codeupipe pipeline of six Filters.

Two commits

  1. The compiler, in the struct form that the measurements chose (see docs/rust-plan.md, "Phase 3").
  2. No copy of the caller's text. When a key is text, the steps work on a struct that borrows the caller's text (a Cow for each text field). A step that writes text owns it, and run copies into the output only the text that no step wrote. A String setting is made once (a OnceLock), not at each run. The signature of run does not change. This was written by hand and measured before it was built.

How fast

cargo run --release --example bench, Apple M1, best of 9 × 20,000:

Pipeline Pipeline::run Compiled
pricing: 3 steps of arithmetic 0.237 µs 0.0017 µs 140x
signup: 4 steps of text 0.50 µs 0.086 µs 5.9x (3.1x before commit 2)

--keep, which computes only the keys that the caller keeps, waits for a real pipeline that needs it: #25 has the measurements and the rules.

How it is checked

  • tests/same_output.rs: the expected modules are compiled into the test and give the output and the errors of their pipelines on many inputs, including the caller's own value of a key that a Valve may write.
  • tests/allocations.rs: a run allocates only for the text that its steps make, and for the caller's text that it gives back.
  • tests/compile.rs: the compiler writes those modules, and refuses each shape with its place.
  • CI: a new rust-compile job runs fmt, clippy, the tests, the bench, and Rust 1.75. The rust-lint job also lints the compiler's code.

Local: 19 tests pass on stable and on 1.75, clippy and fmt are clean, and the linter with --strict has no findings. The runtime's own pricing example is refused at audit_tap.rs:9:33, because its Tap prints payload.to_dict().

Not yet

A rust:static conformance target, nested pipelines, from_map/to_map, and --keep (#25).

🤖 Generated with Claude Code

JoshuaWink and others added 2 commits October 28, 2026 22:53
packages/codeupipe-compile-rs: codeupipe-compile reads a Rust pipeline
and its steps with syn, and writes them as a module of plain Rust: a
struct with one Option field for each key that the steps use, one
function for each step, and a run function. The body of each step is
its own code, with each key of the Payload as a field and each setting
as its literal. It does not run the code.

The rule: code that compiles gives the same output as the pipeline. A
pipeline for which the compiler cannot keep that rule is refused, with
the place and the fix: a Hook, a nested pipeline, a setting that is not
a literal, a key that is not text in the code, a key with two types, a
read with no type, another method of the Payload, and others.

The compiler is a codeupipe pipeline of five Filters. tests/same_output.rs
compiles the expected modules into the test and compares them with the
pipelines on many inputs and errors. On an M1, pricing (3 steps of
arithmetic) runs 150x faster than Pipeline::run, signup (4 steps of
text) 3.1x.

A Cargo workspace of its own, as the linter. A new CI job runs fmt,
clippy, the tests, the bench and Rust 1.75; the linter job also checks
its code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a key is text, the compiled steps work on a second struct that
borrows the caller's text (SignupWork, a Cow for each text field). A step
that writes text owns it, and run copies into the output only the
caller's text that no step wrote. A String setting is made once, by a
function of its own with a OnceLock, and not at each run. The signature
of run does not change. A pipeline with no text key is compiled as
before.

The compiler gets a sixth Filter for this, BorrowTextFilter, after the
types of the keys are known.

Signup: 0.164 to 0.086 µs, 5.9x faster than Pipeline::run (it was 3.1x).
A new greet fixture gives back the caller's note when its Valve does not
run, and tests/allocations.rs checks that a run allocates only for the
text that its steps make and the caller's text that it gives back.

--keep, to compute only the keys that the caller keeps, waits in #25.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JoshuaWink
JoshuaWink merged commit fe1d29e into main Oct 29, 2026
13 checks passed
@JoshuaWink
JoshuaWink deleted the feat/rust-compile branch October 29, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant