Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,47 @@ jobs:
rustup toolchain install 1.75 --profile minimal
cargo +1.75 test --locked

go:
name: Go package
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v6
with:
python-version: "3.12"
# The package and the runner have no dependencies, and no go.sum to
# cache. Only staticcheck, a tool, is downloaded.
- uses: actions/setup-go@v7
with:
go-version: stable
cache: false
- name: Format
run: test -z "$(gofmt -l packages/codeupipe-go conformance/runners/go)"
- name: Vet
run: |
(cd packages/codeupipe-go && go vet ./...)
(cd conformance/runners/go && go vet ./...)
- name: Staticcheck
run: |
(cd packages/codeupipe-go && go run honnef.co/go/tools/cmd/staticcheck@2026.2.2 ./...)
(cd conformance/runners/go && go run honnef.co/go/tools/cmd/staticcheck@2026.2.2 ./...)
- name: Tests
working-directory: packages/codeupipe-go
run: go test -race ./...
- name: Example
working-directory: packages/codeupipe-go
run: go run ./examples/pricing
- name: Install the Python package, for the judge
# conformance/run.py is a codeupipe pipeline (conformance/judging/).
run: python -m pip install -e packages/codeupipe
- name: Conformance
run: python conformance/run.py go:run
- name: The oldest Go that the package supports (go.mod)
working-directory: packages/codeupipe-go
env:
GOTOOLCHAIN: go1.22.12
run: go test -race ./...

export:
name: Export against dynamic run
runs-on: ubuntu-latest
Expand Down
20 changes: 15 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ If you know Unix, these are close to things you know.
| Hook | A git hook: code that runs before and after a step |
| State | `$?` and `PIPESTATUS` |

This repository has seven packages.
This repository has nine packages.

| Package | Contents | Reference |
|---|---|---|
Expand All @@ -40,6 +40,7 @@ This repository has seven packages.
| `codeupipe` for Rust | The primitives, 0.1.0. No dependencies | [packages/codeupipe-rs/README.md](packages/codeupipe-rs/README.md) |
| `codeupipe-lint` for Rust | A linter for pipeline code in Rust, for the rules that the compiler does not check | [packages/codeupipe-lint-rs/README.md](packages/codeupipe-lint-rs/README.md) |
| `codeupipe-compile` for Rust | Writes a pipeline as plain Rust: a struct and functions, with no codeupipe | [packages/codeupipe-compile-rs/README.md](packages/codeupipe-compile-rs/README.md) |
| `codeupipe` for Go | The eight primitives, 0.1.0. No dependencies | [packages/codeupipe-go/README.md](packages/codeupipe-go/README.md) |

## Install

Expand Down Expand Up @@ -94,6 +95,11 @@ crates.io yet:
[packages/codeupipe-rs/README.md](packages/codeupipe-rs/README.md) says how to
use it from this repository.

The Go package is new too, version `0.1.0` on the same terms. No version is
tagged yet:
[packages/codeupipe-go/README.md](packages/codeupipe-go/README.md) says how to
use it from the `main` branch.

## Example

[examples/pricing/](examples/pricing/) has one class in each file. The
Expand Down Expand Up @@ -210,6 +216,7 @@ An editable install needs pip 21.3 or later.
| `packages/codeupipe-rs/` | The Rust runtime, with its tests and its examples |
| `packages/codeupipe-lint-rs/` | The Rust linter, with its tests. A Cargo workspace of its own |
| `packages/codeupipe-compile-rs/` | The Rust compiler, with its tests. A Cargo workspace of its own |
| `packages/codeupipe-go/` | The Go runtime, with its tests and its example. A Go module of its own |
| `conformance/` | The contract ([API.md](conformance/API.md)), the cases, the runners, the benchmark |
| `tests/` | The tests for the two Python packages |
| `examples/pricing/` | The example, with its test in `tests/examples/` |
Expand All @@ -221,17 +228,20 @@ root `package.json` is not a package: it makes the three TypeScript packages
one npm workspace. The root `Cargo.toml` is not a package either: it makes
the Rust runtime and its conformance runner one Cargo workspace, which
builds with no download. The Rust linter needs `syn`, so it is a workspace
of its own.
of its own. The Go runtime and the Go conformance runner are two Go modules;
the runner's `go.mod` points at the runtime with a `replace`.

## Other languages

The contract and the conformance cases do not depend on a language. Python
is the first runtime and TypeScript is the second
([packages/codeupipe-ts/](packages/codeupipe-ts/)), with its own linter and
its own compiler. Rust is the third
([packages/codeupipe-rs/](packages/codeupipe-rs/)), with its own linter; its plan is
[docs/rust-plan.md](docs/rust-plan.md). A runtime in another language must implement
the contract and add a conformance runner.
([packages/codeupipe-rs/](packages/codeupipe-rs/)), with its own linter and
its own compiler; its plan is [docs/rust-plan.md](docs/rust-plan.md). Go is
the fourth ([packages/codeupipe-go/](packages/codeupipe-go/)); its plan is
[docs/go-plan.md](docs/go-plan.md). A runtime in another language must
implement the contract and add a conformance runner.

## License

Expand Down
52 changes: 37 additions & 15 deletions conformance/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ parameters, their defaults, and how they fail.
| State | records what happened |
| Pipeline | runs the steps in order |

Python, TypeScript and Rust implement it. Each runtime has the same contract,
with identifier casing in the style of its language, and the same
Python, TypeScript, Rust and Go implement it. Each runtime has the same
contract, with identifier casing in the style of its language, and the same
conformance cases judge it. The tables below show the Python names. The
TypeScript and Rust names and differences are in "Runtimes".
TypeScript, Rust and Go names and differences are in "Runtimes".

## Principles

Expand Down Expand Up @@ -235,16 +235,16 @@ map and fails the comparison.

## Runtimes

| | Python | TypeScript | Rust |
|---|---|---|---|
| Package | `packages/codeupipe` | `packages/codeupipe-ts` | `packages/codeupipe-rs` (crate `codeupipe`) |
| Names | snake_case | camelCase: `addFilter`, `addValve`, `toDict`, `withTrace`, `traceId`, `onError` | snake_case, as Python |
| Run | `run` is sync. `run_async` is opt-in | One `run`. It returns a Promise. A step can be sync or async | `run` is sync, and `run_with_state` for the caller's State. No async run yet |
| Absent | `None` | `null` | `None` from `get`; `Value::Null` is a JSON null |
| Label of a step | `type(step).__name__` | `step.constructor.name` | The name of the step's type, from `std::any::type_name` |
| Compile | `codeupipe.compile` | `packages/codeupipe-compile-ts` | `packages/codeupipe-compile-rs`: a struct with one field for each key. No conformance target yet |
| Linter | `packages/codeupipe-lint` | `packages/codeupipe-lint-ts` | None yet. The compiler refuses several of the shapes that the linters find |
| A type of your own | Any object with `copy()` | `class Order implements PipelineData<Order>`, then `new Pipeline<Order>()`, `Filter<Order>` | `impl PipelineData for Order {}`, then `Pipeline<Order>`, `Filter<Order>` |
| | Python | TypeScript | Rust | Go |
|---|---|---|---|---|
| Package | `packages/codeupipe` | `packages/codeupipe-ts` | `packages/codeupipe-rs` (crate `codeupipe`) | `packages/codeupipe-go` (package `codeupipe`) |
| Names | snake_case | camelCase: `addFilter`, `addValve`, `toDict`, `withTrace`, `traceId`, `onError` | snake_case, as Python | PascalCase: `AddFilter`, `AddValve`, `ToDict`, `WithTrace`, `TraceID`, `OnError` |
| Run | `run` is sync. `run_async` is opt-in | One `run`. It returns a Promise. A step can be sync or async | `run` is sync, and `run_with_state` for the caller's State. No async run yet | `Run(ctx, p)` is sync, and `RunWithState(ctx, p, state)` for the caller's State. Go needs no async run |
| Absent | `None` | `null` | `None` from `get`; `Value::Null` is a JSON null | `nil` from `Get`, for an absent key and for a JSON null; `Has` tells them apart |
| Label of a step | `type(step).__name__` | `step.constructor.name` | The name of the step's type, from `std::any::type_name` | The name of the step's type, from `reflect` |
| Compile | `codeupipe.compile` | `packages/codeupipe-compile-ts` | `packages/codeupipe-compile-rs`: a struct with one field for each key. No conformance target yet | None yet |
| Linter | `packages/codeupipe-lint` | `packages/codeupipe-lint-ts` | `packages/codeupipe-lint-rs` | None yet |
| A type of your own | Any object with `copy()` | `class Order implements PipelineData<Order>`, then `new Pipeline<Order>()`, `Filter<Order>` | `impl PipelineData for Order {}`, then `Pipeline<Order>`, `Filter<Order>` | A type with `Copy() *Order`, then `NewPipelineOf[*Order]()`, `FilterOf[*Order]` |

Notes on TypeScript:

Expand Down Expand Up @@ -274,8 +274,30 @@ Notes on Rust:
reference, so a step can hold the pipeline that it is in (an `Rc`) and run
it again. The run with the caller's State is
`run_with_state(&input, &mut state)`. `state()` gives an `Rc<State>`: the
State of the run that started last, once it has ended, as a snapshot.md](../docs/rust-plan.md) has the
plan.
State of the run that started last, once it has ended, as a snapshot.
[docs/rust-plan.md](../docs/rust-plan.md) has the plan.
- **A type of your own** is a type parameter of each primitive, with
`Payload` as its default, so a step for one type in a pipeline of another
is a compile error. TypeScript does the same with its type check.

Notes on Go:

- **`ctx` comes first.** Each step method and `Run` take a
`context.Context`, as Go code that can wait does. The Pipeline gives it to
each step and does nothing else with it: a cancelled `ctx` does not stop a
run between steps. So the contract does not change.
- **Refused when added, at compile time.** A function is not a step, and the
arguments of `AddValve` are in one order only: each is a type error. A nil
step compiles, and is refused with a panic when it is added. Go has no
pointer that only reads, so a Tap, a Hook or a Predicate that writes is the
linter's rule (`CUP-I004`), as in Python and TypeScript.
- **Values.** A value is an `any` in the shapes that `encoding/json` gives:
`nil`, `bool`, `float64`, `string`, `[]any`, `map[string]any`. A number of
another Go type is kept as a `float64`. The keys have no order.
- **A State for each run.** A Pipeline can be run from many goroutines at
the same time, and each run writes into a State of its own. `State()` is
the State of the run that started last, once it has ended, as in Rust.
- **A type of your own.** Go has no default type parameter, so each
primitive has a generic form (`FilterOf[P]`) and a short name for the
Payload (`Filter` is `FilterOf[*Payload]`, the same type).
[docs/go-plan.md](../docs/go-plan.md) has the plan.
18 changes: 10 additions & 8 deletions conformance/BENCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,20 @@ two fast modes do neither, and refuse the `hook` scenario rather than
silently dropping the Hook.

`rust:run` is the dynamic run of the Rust crate, built with `--release`. It
records State and calls Hooks as `python:run` does.
records State and calls Hooks as `python:run` does. `go:run` is the dynamic
run of the Go package, which does the same.

µs per run (best of 5 × 20,000, after 20,000 warmup runs); lower is better.

| scenario | steps | python:run | python:static-steps | python:static-inline | python:static-observe | rust:run |
|---|---|---|---|---|---|---|
| `micro-pass` | 10 | 4.94 | 0.824 | 0.606 | 1.92 | 0.246 |
| `valve-gated` | 10 | 5.81 | 1.11 | 0.801 | 2.25 | 0.301 |
| `taps` | 13 | 5.34 | 0.899 | 0.602 | 2.03 | 0.256 |
| `hook` | 10 | 5.88 | — | — | 2.97 | 0.247 |
| `heavy-allocation` | 50 | 16.05 | 2.84 | 1.49 | 7.01 | 4.24 |
| scenario | steps | python:run | python:static-steps | python:static-inline | python:static-observe | rust:run | go:run |
|---|---|---|---|---|---|---|---|
| `micro-pass` | 10 | 4.79 | 0.824 | 0.627 | 1.93 | 0.276 | 0.546 |
| `valve-gated` | 10 | 5.14 | 1.11 | 0.860 | 2.26 | 0.326 | 0.612 |
| `taps` | 13 | 5.03 | 0.895 | 0.626 | 2.03 | 0.293 | 0.567 |
| `hook` | 10 | 5.41 | — | — | 2.98 | 0.286 | 0.626 |
| `heavy-allocation` | 50 | 15.03 | 2.87 | 1.51 | 7.21 | 4.40 | 2.15 |

Not measured (the target refuses the scenario):
- python:static-steps hook: UnsupportedFilterShape: <pipeline>.call at line 0:0 — cannot lower Hook. Hint: Hooks do not run in the fast static export. Export with observe=True (top-level Hooks only), or remove them.
- python:static-inline hook: UnsupportedFilterShape: <pipeline>.call at line 0:0 — cannot lower Hook. Hint: Hooks do not run in the fast static export. Export with observe=True (top-level Hooks only), or remove them.

68 changes: 35 additions & 33 deletions conformance/MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ if any target has a ❌.

**Every target passes every case, and nothing is unsupported.** The targets
are the Python dynamic run, async run and three compiled forms, the
TypeScript run and its compiled form, and the Rust run.
TypeScript run and its compiled form, the Rust run, and the Go run.

The fast modes (`python:static-steps`, `python:static-inline`,
`typescript:static`) expose no State or Hooks, so they pass those cases as
Expand All @@ -16,38 +16,38 @@ the case that asks for `state` on the Pipeline object. A compiled form is a
function and has no such object. See
[docs/compile.md](../docs/compile.md).

| case | python:run | python:run_async | python:static-steps | python:static-inline | python:static-observe | typescript:run | typescript:static | rust:run |
|---|---|---|---|---|---|---|---|---|
| `hook/lifecycle-on-success` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `hook/on-error-when-a-filter-fails` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `hook/not-notified-of-a-tap-failure` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `hook/on-error-names-a-failing-valve` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `payload/insert-adds-a-key` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/insert-overwrites` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/missing-key-reads-as-null` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/nested-input-is-never-mutated` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/json-value-types-round-trip` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/remove-drops-a-key` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/remove-of-a-missing-key-is-a-no-op` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/merge-right-side-wins` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/has-reports-presence-even-for-null` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/mutable-escape-hatch-sets-in-place` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `pipeline/empty-returns-input` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `pipeline/steps-run-in-order` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `pipeline/filter-error-halts-the-run` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `pipeline/nested-pipeline-is-a-step` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `state/the-caller-gives-the-state-of-a-run` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `state/a-run-that-starts-inside-another-run-has-its-own-state` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `state/with-no-state-from-the-caller-the-pipeline-has-the-state-of-the-run-that-started-last` | ✅ | ✅ | ✅* | ✅* | ✅* | ✅ | ✅* | ✅ |
| `tap/observes-without-changing-output` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `tap/failure-never-halts-the-pipeline` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `tap/failure-does-not-stop-later-taps` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `tap/sees-payload-after-a-skipped-valve` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `valve/runs-when-predicate-true` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `valve/skips-when-predicate-false` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `valve/missing-key-is-falsy` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `valve/sees-upstream-changes` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| `valve/independent-gates` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ |
| case | python:run | python:run_async | python:static-steps | python:static-inline | python:static-observe | typescript:run | typescript:static | rust:run | go:run |
|---|---|---|---|---|---|---|---|---|---|
| `hook/lifecycle-on-success` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `hook/on-error-when-a-filter-fails` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `hook/not-notified-of-a-tap-failure` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `hook/on-error-names-a-failing-valve` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `payload/insert-adds-a-key` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/insert-overwrites` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/missing-key-reads-as-null` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/nested-input-is-never-mutated` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/json-value-types-round-trip` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/remove-drops-a-key` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/remove-of-a-missing-key-is-a-no-op` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/merge-right-side-wins` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/has-reports-presence-even-for-null` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `payload/mutable-escape-hatch-sets-in-place` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `pipeline/empty-returns-input` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `pipeline/steps-run-in-order` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `pipeline/filter-error-halts-the-run` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `pipeline/nested-pipeline-is-a-step` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `state/the-caller-gives-the-state-of-a-run` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `state/a-run-that-starts-inside-another-run-has-its-own-state` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `state/with-no-state-from-the-caller-the-pipeline-has-the-state-of-the-run-that-started-last` | ✅ | ✅ | ✅* | ✅* | ✅* | ✅ | ✅* | ✅ | ✅ |
| `tap/observes-without-changing-output` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `tap/failure-never-halts-the-pipeline` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `tap/failure-does-not-stop-later-taps` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `tap/sees-payload-after-a-skipped-valve` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `valve/runs-when-predicate-true` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `valve/skips-when-predicate-false` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `valve/missing-key-is-falsy` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `valve/sees-upstream-changes` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |
| `valve/independent-gates` | ✅ | ✅ | ✅* | ✅* | ✅ | ✅ | ✅* | ✅ | ✅ |

- **python:run**: 30 pass, 0 pass*, 0 fail, 0 unsupported
- **python:run_async**: 30 pass, 0 pass*, 0 fail, 0 unsupported
Expand All @@ -57,5 +57,7 @@ function and has no such object. See
- **typescript:run**: 30 pass, 0 pass*, 0 fail, 0 unsupported
- **typescript:static**: 11 pass, 19 pass*, 0 fail, 0 unsupported
- **rust:run**: 30 pass, 0 pass*, 0 fail, 0 unsupported
- **go:run**: 30 pass, 0 pass*, 0 fail, 0 unsupported

✅* passed everything the target can observe; it does not expose State or Hooks (the fast mode). ➖ the target cannot express the case.

Loading
Loading