A Rust runtime, written to the contract

.github/workflows/ci.yml

@@ -61,11 +61,44 @@ jobs:
         run: node packages/codeupipe-ts/examples/pricing/main.ts
       - name: The linter on the TypeScript code of this repository
         run: npm run lint -w codeupipe-lint
+      - name: Install the Python package, for the judge
+        # conformance/run.py is a codeupipe pipeline (conformance/judging/).
+        run: python -m pip install -e packages/codeupipe
       - name: Conformance
         run: python conformance/run.py typescript:run typescript:static
       - name: The packages as npm makes them
         run: npm pack --workspaces --dry-run
 
+  rust:
+    name: Rust packages
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+      # The runner image has rustup and a stable toolchain. Nothing is
+      # downloaded from crates.io: the crates have no dependencies.
+      - name: Format
+        run: cargo fmt --all --check
+      - name: Clippy
+        run: cargo clippy --offline --all-targets -- -D warnings
+      - name: Tests
+        run: cargo test --offline
+      - name: Examples
+        run: |
+          cargo run --offline --example pricing -p codeupipe
+          cargo run --offline --example typed_pricing -p codeupipe
+      - name: Install the Python package, for the judge
+        # conformance/run.py is a codeupipe pipeline (conformance/judging/).
+        run: python -m pip install -e packages/codeupipe
+      - name: Conformance
+        run: python conformance/run.py rust:run
+      - name: The oldest Rust that the crate supports (rust-version)
+        run: |
+          rustup toolchain install 1.75 --profile minimal
+          cargo +1.75 test --offline -p codeupipe
+
   export:
     name: Export against dynamic run
     runs-on: ubuntu-latest
@@ -104,11 +137,12 @@ jobs:
           /tmp/installed/bin/python -m codeupipe_lint.invariants --rules
           /tmp/installed/bin/python -m codeupipe_lint.anonymous_steps --rules
 
-  speed:
-    name: Speed check (does not block)
+  benchmark:
+    name: Benchmark (numbers only, never fails the run)
     runs-on: ubuntu-latest
-    # The baseline was recorded on a different machine. This job shows how
-    # far a CI machine is from it; it does not fail the run until that is known.
+    # A shared CI machine is too noisy to judge speed, so nothing here is
+    # compared with a baseline. The job measures and keeps the numbers: in
+    # the job summary and as an artifact.
     continue-on-error: true
     steps:
       - uses: actions/checkout@v5
@@ -119,5 +153,13 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           python -m pip install -e packages/codeupipe -e packages/codeupipe-lint -r requirements-dev.txt
-      - name: Compare with the recorded baseline
-        run: python conformance/bench.py --check
+      - name: Measure
+        run: |
+          set -o pipefail
+          python conformance/bench.py | tee -a "$GITHUB_STEP_SUMMARY"
+      - name: Keep the numbers
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark
+          path: conformance/results/bench-latest.*

.gitignore

@@ -9,3 +9,5 @@ conformance/results/
 .venv-*/
 scorecard/results/
 node_modules/
+
+/target/

Cargo.toml

@@ -0,0 +1,5 @@
+# The Cargo workspace of the Rust packages and the Rust conformance runner.
+# It is not a package.
+[workspace]
+resolver = "2"
+members = ["packages/codeupipe-rs", "conformance/runners/rust"]

README.md

@@ -28,7 +28,7 @@ If you know Unix, these are close to things you know.
 | Hook | A git hook: code that runs before and after a step |
 | State | `$?` and `PIPESTATUS` |
 
-This repository has five packages.
+This repository has six packages.
 
 | Package | Contents | Reference |
 |---|---|---|
@@ -37,6 +37,7 @@ This repository has five packages.
 | `codeupipe` for TypeScript | The eight primitives. No dependencies | [packages/codeupipe-ts/README.md](packages/codeupipe-ts/README.md) |
 | `codeupipe-lint` for TypeScript | A linter for pipeline code in TypeScript | [packages/codeupipe-lint-ts/README.md](packages/codeupipe-lint-ts/README.md) |
 | `codeupipe-compile` for TypeScript | Writes a pipeline as plain TypeScript with no import of codeupipe | [packages/codeupipe-compile-ts/README.md](packages/codeupipe-compile-ts/README.md) |
+| `codeupipe` for Rust | The primitives, 0.1.0. No dependencies. No compiler or linter yet | [packages/codeupipe-rs/README.md](packages/codeupipe-rs/README.md) |
 
 ## Install
 
@@ -86,6 +87,11 @@ compatible. They are not on npm yet:
 [packages/codeupipe-ts/README.md](packages/codeupipe-ts/README.md) says how to
 install them from this repository.
 
+The Rust crate is new too, version `0.1.0` on the same terms. It is not on
+crates.io yet:
+[packages/codeupipe-rs/README.md](packages/codeupipe-rs/README.md) says how to
+use it from this repository.
+
 ## Example
 
 [examples/pricing/](examples/pricing/) has one class in each file. The
@@ -141,7 +147,7 @@ class PricingPipeline(Pipeline):
         self.add_filter(DiscountFilter(0.1))
         # MarkBulkFilter runs only when the predicate is true.
         self.add_valve(AtLeastPredicate("qty", 10), MarkBulkFilter())
-        # A Tap failure is recorded and does not stop the run.
+        # A Tap only reads. If it fails, State records the error and the run goes on.
         self.add_tap(AuditTap())
 ```
 
@@ -187,7 +193,7 @@ An editable install needs pip 21.3 or later.
 | `python -m codeupipe_lint.anonymous_steps` | Runs the anonymous-step linter. Paths come from `cup.config.json` |
 | `python -m codeupipe_lint.invariants` | Runs the invariant linter. Paths come from `cup.config.json` |
 | `python -m codeupipe.compile examples.pricing.pricing_pipeline:PricingPipeline --out build/pricing` | Compiles the example to `build/pricing/src/steps/` |
-| `python conformance/bench.py --check` | Compares benchmark numbers with the recorded baseline. Fails outside 25% |
+| `python conformance/bench.py` | Measures every target on the benchmark scenarios. It reports numbers and judges nothing |
 | `python -m scorecard.differential` | Compares the compiled code with the dynamic run on random pipelines |
 
 ## Layout
@@ -199,6 +205,7 @@ An editable install needs pip 21.3 or later.
 | `packages/codeupipe-ts/` | The TypeScript runtime, with its tests and its example |
 | `packages/codeupipe-lint-ts/` | The TypeScript linter, with its tests |
 | `packages/codeupipe-compile-ts/` | The TypeScript compiler, with its tests |
+| `packages/codeupipe-rs/` | The Rust runtime, with its tests and its example |
 | `conformance/` | The contract ([API.md](conformance/API.md)), the cases, the runners, the benchmark |
 | `tests/` | The tests for the two Python packages |
 | `examples/pricing/` | The example, with its test in `tests/examples/` |
@@ -207,14 +214,17 @@ An editable install needs pip 21.3 or later.
 
 The root `pyproject.toml` is not a package. It holds the test settings. The
 root `package.json` is not a package: it makes the three TypeScript packages
-one npm workspace.
+one npm workspace. The root `Cargo.toml` is not a package either: it makes
+the Rust runtime and its conformance runner one Cargo workspace.
 
 ## Other languages
 
 The contract and the conformance cases do not depend on a language. Python
 is the first runtime and TypeScript is the second
 ([packages/codeupipe-ts/](packages/codeupipe-ts/)), with its own linter and
-its own compiler. A runtime in another language must implement
+its own compiler. Rust is the third
+([packages/codeupipe-rs/](packages/codeupipe-rs/)); its plan is
+[docs/rust-plan.md](docs/rust-plan.md). A runtime in another language must implement
 the contract and add a conformance runner.
 
 ## License

conformance/API.md

@@ -14,10 +14,10 @@ parameters, their defaults, and how they fail.
 | State | records what happened |
 | Pipeline | runs the steps in order |
 
-Python and TypeScript implement it. Each runtime has the same contract,
+Python, TypeScript and Rust implement it. Each runtime has the same contract,
 with identifier casing in the style of its language, and the same
 conformance cases judge it. The tables below show the Python names. The
-TypeScript names and differences are in "Runtimes".
+TypeScript and Rust names and differences are in "Runtimes".
 
 ## Principles
 
@@ -71,6 +71,24 @@ Rules:
 - A Tap, a Hook or a Predicate that wants its own changed version, or that
   keeps the Payload for later, takes a `copy()`.
 
+### A type of your own
+
+The Payload is the default, and the form that this contract, the
+conformance cases and the compilers use: a map that any developer can read,
+which shows that anything can go there. A pipeline can carry a type of your
+own instead, such as the class or struct that your steps would use anyway.
+Its Filters, Predicates, Taps and Hooks then take that type.
+
+- **The Pipeline asks one thing of it: `copy()`**, for the copy that `run`
+  works on. Every rule above stays the same.
+- **Lineage is kept only by a type that has `_stamp(label)`** (`stamp` in
+  Rust). For any other type, `observe(lineage=True)` does nothing: the run
+  goes on, and `state.executed` still lists the steps.
+- **The compilers read Payload operations only.** A step that uses a type of
+  your own stays dynamic; the TypeScript compiler refuses it.
+
+How each runtime spells it is in "Runtimes".
+
 ## Filter
 
 `call(payload) -> payload`. A failure raises.
@@ -217,15 +235,16 @@ map and fails the comparison.
 
 ## Runtimes
 
-| | Python | TypeScript |
-|---|---|---|
-| Package | `packages/codeupipe` | `packages/codeupipe-ts` |
-| Names | snake_case | camelCase: `addFilter`, `addValve`, `toDict`, `withTrace`, `traceId`, `onError` |
-| Run | `run` is sync. `run_async` is opt-in | One `run`. It returns a Promise. A step can be sync or async |
-| Absent | `None` | `null` |
-| Label of a step | `type(step).__name__` | `step.constructor.name` |
-| Compile | `codeupipe.compile` | `packages/codeupipe-compile-ts` |
-| Linter | `packages/codeupipe-lint` | `packages/codeupipe-lint-ts` |
+| | Python | TypeScript | Rust |
+|---|---|---|---|
+| Package | `packages/codeupipe` | `packages/codeupipe-ts` | `packages/codeupipe-rs` (crate `codeupipe`) |
+| Names | snake_case | camelCase: `addFilter`, `addValve`, `toDict`, `withTrace`, `traceId`, `onError` | snake_case, as Python |
+| Run | `run` is sync. `run_async` is opt-in | One `run`. It returns a Promise. A step can be sync or async | `run` is sync, and `run_with_state` for the caller's State. No async run yet |
+| Absent | `None` | `null` | `None` from `get`; `Value::Null` is a JSON null |
+| Label of a step | `type(step).__name__` | `step.constructor.name` | The name of the step's type, from `std::any::type_name` |
+| Compile | `codeupipe.compile` | `packages/codeupipe-compile-ts` | None yet |
+| Linter | `packages/codeupipe-lint` | `packages/codeupipe-lint-ts` | None yet. The compiler refuses several of the shapes that the linters find |
+| A type of your own | Any object with `copy()` | `class Order implements PipelineData<Order>`, then `new Pipeline<Order>()`, `Filter<Order>` | `impl PipelineData for Order {}`, then `Pipeline<Order>`, `Filter<Order>` |
 
 Notes on TypeScript:
 
@@ -240,3 +259,23 @@ Notes on TypeScript:
 - **State and concurrent runs.** Two runs of one pipeline at the same time
   each write to their own State. `state` is the State of the run that
   started last.
+
+Notes on Rust:
+
+- **Refused when added, at compile time.** A closure is not a step, the
+  arguments of `add_valve` are in one order only, and a Tap, a Hook or a
+  Predicate gets `&Payload`, so it cannot write. Each is a type error.
+- **A Filter changes the Payload in place.** `call(&self, &mut Payload)`
+  returns `Result<(), StepError>`. A failure is an `Err`; a panic is not a
+  failure of the step and is not caught.
+- **Values.** A value is a `Value`: null, a bool, a number (`f64`), a string,
+  a list or a map that keeps the order of its keys, as JSON has.
+- **A State for each run.** `run(&self, …)` takes the pipeline by shared
+  reference, so a step can hold the pipeline that it is in (an `Rc`) and run
+  it again. The run with the caller's State is
+  `run_with_state(&input, &mut state)`. `state()` gives an `Rc<State>`: the
+  State of the run that started last, once it has ended, as a snapshot.md](../docs/rust-plan.md) has the
+  plan.
+- **A type of your own** is a type parameter of each primitive, with
+  `Payload` as its default, so a step for one type in a pipeline of another
+  is a compile error. TypeScript does the same with its type check.

conformance/BENCH.md

@@ -5,25 +5,28 @@ A snapshot of `python conformance/bench.py`. Scenarios:
 Every target's output is checked against the others before its time counts.
 Apple Silicon, one run.
 
-These numbers are a snapshot for reading. The numbers that the speed check
-compares are in `bench/baseline.json` (`python conformance/bench.py --check`,
-[README.md](README.md)).
+These numbers are a snapshot for reading. Nothing fails because of them: CI
+measures again on every run and keeps the numbers, and does not compare them
+([README.md](README.md)).
 
 `python:run` is the dynamic run. The three `static-*` targets are the
 compiles: `static-observe` records State and calls Hooks as `run` does; the
 two fast modes do neither, and refuse the `hook` scenario rather than
 silently dropping the Hook.
 
+`rust:run` is the dynamic run of the Rust crate, built with `--release`. It
+records State and calls Hooks as `python:run` does.
+
 µs per run (best of 5 × 20,000, after 20,000 warmup runs); lower is better.
 
-| scenario | steps | python:run | python:static-steps | python:static-inline | python:static-observe |
-|---|---|---|---|---|---|
-| `micro-pass` | 10 | 4.92 | 0.838 | 0.580 | 1.95 |
-| `valve-gated` | 10 | 5.61 | 1.15 | 0.900 | 2.30 |
-| `taps` | 13 | 5.21 | 0.887 | 0.629 | 2.07 |
-| `hook` | 10 | 5.96 | — | — | 3.03 |
-| `heavy-allocation` | 50 | 17.14 | 2.80 | 1.49 | 7.22 |
+| scenario | steps | python:run | python:static-steps | python:static-inline | python:static-observe | rust:run |
+|---|---|---|---|---|---|---|
+| `micro-pass` | 10 | 4.94 | 0.824 | 0.606 | 1.92 | 0.246 |
+| `valve-gated` | 10 | 5.81 | 1.11 | 0.801 | 2.25 | 0.301 |
+| `taps` | 13 | 5.34 | 0.899 | 0.602 | 2.03 | 0.256 |
+| `hook` | 10 | 5.88 | — | — | 2.97 | 0.247 |
+| `heavy-allocation` | 50 | 16.05 | 2.84 | 1.49 | 7.01 | 4.24 |
 
 Not measured (the target refuses the scenario):
-- python:static-steps hook: UnsupportedFilterShape: <pipeline>.call at line 0:0 — cannot lower Hook. Hint: Hooks do not run in the fast mode. Compile with observe=True (top-level Hooks only), or remove them.
-- python:static-inline hook: UnsupportedFilterShape: <pipeline>.call at line 0:0 — cannot lower Hook. Hint: Hooks do not run in the fast mode. Compile with observe=True (top-level Hooks only), or remove them.
+- python:static-steps hook: UnsupportedFilterShape: <pipeline>.call at line 0:0 — cannot lower Hook. Hint: Hooks do not run in the fast static export. Export with observe=True (top-level Hooks only), or remove them.
+- python:static-inline hook: UnsupportedFilterShape: <pipeline>.call at line 0:0 — cannot lower Hook. Hint: Hooks do not run in the fast static export. Export with observe=True (top-level Hooks only), or remove them.