Repository navigation
refactor(conformance): the judge is a codeupipe pipeline - #15
Merged
Merged
Conversation
The judge was eight plain functions in run.py. It is now a pipeline in conformance/judging: read the cases, choose the targets, run each target, compare, count, format, and write the results. Each step is a primitive in its own file and has tests that start no process. run.py keeps its command line and the function names that the tests and bench.py use. Each one calls a step of the pipeline, so there is one judge. The new judge was run beside the old one. Their reports are the same byte for byte on all 7 targets, and they gave the same verdict for each of 6550 results, 2700 of them wrong on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(conformance): the Python runner is codeupipe pipelines The runner was 15 plain functions in runner.py. It is now two pipelines in conformance/runners/python/case_runner: one plays a case on a target and one times a scenario. They share the steps that read the target, write the Predicates, build the pipeline and compile it. Each step is a primitive in its own file with tests of its own. runner.py is the command line. The new runner was run beside the old one on each Python target, with all 30 cases and 7 more that were made to fail. The reports are the same byte for byte. In the timing mode the outputs and the refusals are the same, and the times are the same within the noise of a run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(scorecard): use the steps of the new runner The differential test tool used two functions of the old runner file. It now uses the same steps that the runner uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
JoshuaWink
added a commit
that referenced
this pull request
Oct 28, 2026
conformance/run.py is a codeupipe pipeline since #15, so it imports codeupipe. The TypeScript and Rust jobs did not install the package, and their conformance step failed with ModuleNotFoundError (on #15 as well; CI does not run on pushes to beta). Checked locally with nothing but packages/codeupipe on the path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JoshuaWink
added a commit
that referenced
this pull request
Oct 28, 2026
* feat(rust): plan the Rust runtime and build its core crate
docs/rust-plan.md maps the Rust runtime against the 2.0 contract: the
packages/ layout, decisions D1-D10, and four phases. Phase 1 is
packages/codeupipe-rs (crate codeupipe, 0.1.0, no dependencies), built
test-first from conformance/API.md with codeupipe-ts as the reference.
A Filter takes &mut Payload (D2) because TS changes the Payload in place
and on_error must see what the failing Filter left. Taps, Predicates and
Hooks get &Payload, so "only a Filter writes" is a compile error.
46 tests, clippy -D warnings and fmt clean. The pricing example prints
the same result as the TS one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(examples): the pricing AuditTap reads instead of failing
The pricing example's AuditTap failed on every run, in Python, TS and
Rust, to show that a Tap failure is recorded and the run goes on. Now it
prints what it reads and State has no errors. The rule is unchanged and
the conformance cases and core tests still check it.
The compile-ts strict type check had no console, which is why the TS
Tap threw instead of logging. The check now declares console, which
every JavaScript host has, and nothing else of a browser or of Node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): conformance runner, rust:run passes all 27 cases
conformance/runners/rust runs the cases against packages/codeupipe-rs,
with its own small JSON reader and writer (no download, so it builds
offline) and one file per step type, named as in the TS runner. run.py
gets the rust:run target and leaves it out when there is no cargo.
add_filter and add_valve take the step's type, which gives its label, so
the runner hands each Filter over by its own type through a small generic
trait and not as a Box<dyn Filter>.
Checked that the runner catches a broken runtime: with Tap failures not
recorded, 3 cases fail. API.md gets a Rust column, the README a Rust
package row, MATRIX.md all eight targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: measure speed without judging it, and add a Rust job
The speed job compared each number with a baseline. A shared CI machine
is too noisy for that, so the benchmark job now only measures every
target, Rust included, and keeps the numbers in the job summary and as an
artifact. It never fails the run. The perf pytest test is removed;
bench.py --check stays for a machine with fixed resources.
bench.py measures rust:run. Its baseline is kept under its own key,
"rust", since it does not depend on the Python version, and --record
replaces only the targets it measured.
The new rust job runs fmt, clippy, the tests, the example and rust:run
conformance, and builds with Rust 1.75, the crate's rust-version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* perf(rust): cut per-run allocation in the core
A run spent its time in allocation, not in dispatch. Three changes, with
no change to behaviour or API:
- Map::insert looks the key up first, so a key that is there makes no
new String.
- State keeps its storage between runs and writes the labels over.
- A key is an Arc<str>, made once and shared by the index and by every
copy. A map of 8 keys or fewer is searched in order and has no index;
a larger one gets an index with room to grow. The hash stays SipHash,
so crafted keys from untrusted JSON cannot slow a map down.
10 filters: 1.10 -> 0.25 us. 50 new keys: 7.26 -> 4.24 us. The dynamic run
is now faster than compiled Python in 4 of 5 scenarios, so the plan puts
the Rust compiler (phase 3) on hold. Guard tests first: a shorter run
after a longer one, maps on both sides of the index size, Send + Sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): a payload of your own, with Payload as the default
Payload stays the default: a map that any developer can read, which
shows that anything can go there. A type of your own can take its place:
each primitive takes the data type as a parameter that defaults to
Payload (Filter<P = Payload>, Pipeline<P = Payload>), and the type opts
in with impl PipelineData for Order {}, which asks for Clone and an
optional stamp for lineage. Code written for Payload does not change.
A step for one type in a pipeline of another is a compile error, checked
by a doc test. examples/typed_pricing is the pricing example with an
Order struct and gives the same result. Typed, the pricing run takes
0.024 us against 0.18 us, and 50 writes 0.46 us against 4.3 us.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: a type of your own in Python and TypeScript, as in Rust
The contract gets "A type of your own": Payload stays the default, the
form that the contract, the conformance cases and the compilers use. A
pipeline can carry another type instead; the Pipeline asks only for
copy(). Lineage is kept only by a type that has _stamp(label) (stamp in
Rust); for any other type it does nothing and the run goes on.
Python took such an object already, by duck typing, but crashed with
lineage on. It now calls _stamp only when the object has it. TypeScript
gets the type parameter that Rust has: Filter, Predicate, Tap, Hook and
Pipeline default to Payload, and PipelineData<P> asks for copy() and an
optional _stamp. Mixing types does not type-check (@ts-expect-error).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(compile): follow the Payload parameter under any name
A Filter or Tap whose parameter was not called payload (call(self,
order)) compiled to a function of data whose body still used order, so
the compiled step raised NameError: code that compiled and did not give
the output of the dynamic run. The rewrite knew only the name payload.
The rewrite now takes the parameter's own name. A Valve's predicate keeps
its name too, where before it was renamed to payload, which merged it
with any unrelated local called payload. A refusal names the parameter as
the step wrote it (order.price, not payload.price), which also covers a
step over a type of your own.
Tests first: the three output modes against the dynamic run, with
parameters called order, p and data and a local called payload, and the
refusal of order.price. 538 tests, conformance and 2,000 random pipelines
pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): the State for each run is not in Rust yet
The contract on beta now says that a State belongs to one run: run(p,
state), a run inside a run, and state on the Pipeline as the State of the
run that started last. rust:run reports the three state/ cases as
unsupported, not as passed. API.md says so in its notes on Rust, and the
plan lists what the change needs in Rust: run(&self), a run with the
caller's State, state() behind a RefCell, shared errors, and the cost to
the State storage that is kept between runs today.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: install codeupipe for the judge in the TypeScript and Rust jobs
conformance/run.py is a codeupipe pipeline since #15, so it imports
codeupipe. The TypeScript and Rust jobs did not install the package, and
their conformance step failed with ModuleNotFoundError (on #15 as well;
CI does not run on pushes to beta). Checked locally with nothing but
packages/codeupipe on the path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): a State for each run; rust:run passes all 30 cases
The contract says that a State belongs to one run, not to the Pipeline.
In Rust:
- run(&self) in place of run(&mut self), so a step can hold the pipeline
it is in (an Rc) and run it again while it runs.
- run_with_state(&input, &mut state) for the caller's State, and
State::new().
- state() gives an Rc<State>: the State of the run that started last,
once it has ended. Each run takes a number when it starts; its State
is kept only if no later run has started. A snapshot never changes
under a reader, and holding one never blocks a run.
- A State's errors are Arc<dyn Error> (SharedError), so a State can be
cloned: with the caller's State, the Pipeline keeps a copy.
Two States take turns, so a run still makes no new label String; the Rc
of a run costs about 0.03 us (10 filters: 0.25 -> 0.28 us). The runner
plays run_again as the TypeScript runner does and reports the caller's
State, pipeline_state and inner_states. Tests first (state_per_run.rs);
breaking the "started last" rule fails one case and one unit test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The judge of the conformance suite is now a codeupipe pipeline. It was eight plain functions in
conformance/run.py.This PR is on top of #14. Merge #14 first.
Why
The test tools were the last large part of the repository that was plain code, so the repository's own rule (CUP-I008) needed a comment on each of their functions. This is the first of them written the way the product is written: as primitives.
The pipeline (
conformance/judging/)LoadCasesFilterNoTargetsNamedPredicate+DefaultTargetsFilterRunTargetsFilterJudgeResultsFilterSummarizeFilterFormatMatrixFilterWriteResultsTaplatest.jsonandlatest.mdTwo edges stay plain, each with its reason:
ProcessLauncher(starts a runner) andNodeProbe(asks which Node the machine has).run.pykeeps its command line and the names that the tests andbench.pyuse (load_cases,run_target,judge,default_targets,TARGETS). Each one calls a step of the pipeline, so there is one judge and not two.How it was checked against the old judge
A fault in the judge can make a broken runtime look correct. So the new judge ran beside the old one:
latest.jsonandlatest.mdare the same byte for byte, on all 7 targets.truein the place of1), and both judges failed each of them with the same problems.The old judge is not in this PR. It was used for this comparison only.
What it gains
allow-plaincomments are gone, and 2 are added for the edges. The pinned number in the repository self-check goes from 81 to 75.tests/conformance/test_judge_pipeline.py, 14 tests). Before, the judge was tested only through a full run of a target.Checks
bench.pyuses the same launcher and the same comparison as before.Not in this change
conformance/runners/python/runner.py) andscorecard/are still plain code with comments.🤖 Generated with Claude Code