Repository navigation
codeupipe 2.0 beta: merge beta into main - #21
Merged
Merged
Conversation
State and Hooks reported each step by the full path of its class (module.QualName). They now report the class name: DiscountFilter. The second use of a class in one pipeline is DiscountFilter#2. A Valve has the label of its Filter. Removed, because they existed only for the full path: - AnonymousStep, and the refusal of classes from a function or from the entry script. A step without the method of its role is refused with a TypeError when it is added. - Pipeline.identities (now Pipeline.labels) and Pipeline.step_sites. - Lint rule CUP-A006 (entry-script step). BREAKING CHANGE: the strings in state.executed, state.skipped, state.errors, state.timings and in Hook calls are class names, not module paths. AnonymousStep, Pipeline.identities and Pipeline.step_sites are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
packages/codeupipe-ts has the eight primitives in TypeScript, with no dependencies. The 27 conformance cases judge it as the target typescript:run, with the same judge as the Python targets. - One run, which returns a Promise. A step can be sync or async. - Names are the names of the contract in camelCase. Absent is null. - A step is reported by its label, the name of its class. A function is refused as a step, because every JavaScript function has a call method. - Each run writes to its own State. - No export command and no linter in this package. conformance/run.py runs the TypeScript target when the machine has Node 22.18 or later, which runs the .ts runner as it is. A target that is named and cannot run is now a failure. CI has a job for the package: type check, tests, build, example and conformance. The contract has a "Runtimes" section with the differences between the two runtimes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test makes a browser bundle of the example with esbuild and runs it: plain, minified, and minified with class names kept. The result is the same in all three. The labels change only in the minified bundle that does not keep names, as the README says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
packages/codeupipe-lint-ts reads TypeScript source and reports: - a step with no name: a function, an object literal, a class that is declared in a function, a plain nested Pipeline (CUP-A001, A002, A003, A005, A008, A009); - a Filter or a Predicate that reads State (CUP-I003); - a Tap, a Predicate or a Hook that writes to the Payload (CUP-I004); - more than one primitive in a file, a file name that is not the class name in kebab-case, a class name with no role (CUP-I005, I006, I007). The linter is a codeupipe pipeline. It uses the typescript package to parse. A finding is turned off with a comment marker, --exclude, --disable, or the lint section of cup.config.json. CUP-I001 is not needed in TypeScript, because the data of a Payload is in a private field. CUP-I002 is not built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
packages/codeupipe-compile-ts reads the source of a pipeline class and writes one module with no import of codeupipe: a function for each step and an async run(input). The data is a Map, each operation of the Payload is an operation of the Map, and each setting of an instance is a constant. A pipeline for which the compiled code cannot give the same output is refused, with the place named and the fix: Hooks, a constructor that does more than add steps, settings that are not literals, a step that keeps state, a Payload that is copied or given to other code, a Tap or a Predicate that writes. Nothing is written for a refused pipeline. The conformance suite has the target typescript:static. The runner writes each case as a pipeline class, compiles it, imports the file and runs it. 11 cases pass, and the 16 cases that also check State or Hooks pass on what plain code can show, as for the Python fast export. The compiler is a codeupipe pipeline. The Taps of the TypeScript runner now record to a module, so that compiled code can do the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One name for one thing. The module was "lower", the documents said "export", and the config key was "compile". All three are now "compile", as in the TypeScript package. - python -m codeupipe.lower is python -m codeupipe.compile. - docs/static-export.md is docs/compile.md. - The documents say "compile", "compiled code", "fast mode" and "observable mode". - tests/lower is tests/compile. Not changed: the --lowering option and the lowering key of the config file (steps or inline), and the names inside the module, such as build_export_pipeline_v2. BREAKING CHANGE: an import of codeupipe.lower, and the command python -m codeupipe.lower, must use codeupipe.compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The step labels and the rename of codeupipe.lower are not compatible with 1.0.0. They go out as a beta of version 2, so that 1.0.0 stays the version that pip installs. - Both Python packages are 2.0.0b1, and codeupipe-lint needs that version of codeupipe. - The README says which version is stable and which is the beta, how to install each one, and what changed. - The release workflow needs no change: the tag v2.0.0b1 publishes the beta, and PyPI keeps a beta apart from the stable version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three TypeScript packages have a pre-release version, so that the version says that they are a beta. They stay private and are not on npm. The README points to the beta branch for the install of version 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The packages are no longer private. Each one publishes with the npm tag "beta", so a person gets it with npm install codeupipe@beta. - The three packages are one npm workspace, with one lock file at the root of the repository. codeupipe-lint and codeupipe-compile need the exact version of codeupipe, where they had a link to its directory. - The built core package has no rule for the Node version: its JavaScript runs in a browser and in older Node. The two tools need Node 20. - CI has one job for the TypeScript packages, and it also makes the packages as npm does. Nothing is published by this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A version below 1.0 says that the packages are not stable yet. They publish with the normal npm tag, so npm install codeupipe gives them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The packages are private again, so that a publish cannot happen by accident. The README says how to make them from this repository and install the files that npm pack writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Report a step by a label, the name of its class
A TypeScript runtime, written to the contract
A linter for TypeScript pipeline code
A compiler that writes a pipeline as plain TypeScript
Rename the Python module codeupipe.lower to codeupipe.compile
Version 2.0.0b1 for Python, a beta
TypeScript packages: version 0.1.0, one npm workspace, install from this repository
* fix: a State belongs to one run, not to the Pipeline Two runs of one Pipeline that were in progress at the same time wrote into one State, so their steps mixed. Each run now writes into a State of its own, and the caller can give it: run(payload, state). Python: the State is a local of the run. The Pipeline asks the Predicate of a Valve itself and no longer reads a flag from the Valve, which each run in progress shared. TypeScript: run takes an optional State. The contract has the rule, and three conformance cases: the caller gives the State, and a run that starts inside another run. Closes #11 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(conformance): the compiled forms run the cases with a second run The reference step no longer holds a Pipeline. The runner gives it the way to run the pipeline again: a method of the Pipeline, or for a compiled form a second call of its function. Both compilers lower the step, so no case is unsupported on any target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…P-I008) (#10) * feat(lint-ts): CUP-I008, a warning for logic that is not in a primitive A project that uses codeupipe writes its logic as Filters, Predicates, Taps, Hooks and Pipelines. Logic in a plain function or a plain class works as well, so nothing showed the mistake. The linter now reports a function, a class or the code at the top of a file that has logic (if, ? :, switch, a loop, try) and is not a primitive. - It is a warning: it is in the report, it is counted apart, and it does not fail the run. --strict makes a warning fail the run. - A comment with a reason allows plain code: // cup: allow-plain <why>. - Test files are not checked. The code of the framework itself is marked. The TypeScript linter and the TypeScript compiler of this repository have 40 warnings: their helpers are plain functions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lint-ts): CUP-I009, a comment that allows nothing A cup: allow-... comment that allowed no finding is a warning. It stays when the code that it was for is gone or has moved, or when its kind has a spelling error. Rules that are switched off are dropped last, so that a comment for such a rule is not reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lint): CUP-I008 and CUP-I009 in the Python linter CUP-I008 is logic that is not in a primitive. CUP-I009 is a comment that allows nothing. Both are warnings: the linter now has warnings, and --strict makes one fail the run. Rules that are switched off are dropped last, so that a comment for such a rule is not reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The 120 CUP-I008 warnings are resolved. 52 in the code itself: the helper functions of the compiler Filters and of three visitors are now methods of their class, the export pipeline is a named Pipeline, and three small decisions are tables. 68 with a comment that gives the reason, on code that is plain by design: a primitive itself, a visitor of a syntax tree, the entry of a program, a test harness. 13 more comments are on the conformance judge and speed check, which the repository self-check also covers. CI runs the invariant linter with --strict. The self-check test pins the number of comments, so a new one is a change that someone sees. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(conformance): the judge is a codeupipe pipeline The judge was eight plain functions in run.py. It is now a pipeline in conformance/judging: read the cases, choose the targets, run each target, compare, count, format, and write the results. Each step is a primitive in its own file and has tests that start no process. run.py keeps its command line and the function names that the tests and bench.py use. Each one calls a step of the pipeline, so there is one judge. The new judge was run beside the old one. Their reports are the same byte for byte on all 7 targets, and they gave the same verdict for each of 6550 results, 2700 of them wrong on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(conformance): the Python runner is codeupipe pipelines (#16) * refactor(conformance): the Python runner is codeupipe pipelines The runner was 15 plain functions in runner.py. It is now two pipelines in conformance/runners/python/case_runner: one plays a case on a target and one times a scenario. They share the steps that read the target, write the Predicates, build the pipeline and compile it. Each step is a primitive in its own file with tests of its own. runner.py is the command line. The new runner was run beside the old one on each Python target, with all 30 cases and 7 more that were made to fail. The reports are the same byte for byte. In the timing mode the outputs and the refusals are the same, and the times are the same within the noise of a run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(scorecard): use the steps of the new runner The differential test tool used two functions of the old runner file. It now uses the same steps that the runner uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): plan the Rust runtime and build its core crate
docs/rust-plan.md maps the Rust runtime against the 2.0 contract: the
packages/ layout, decisions D1-D10, and four phases. Phase 1 is
packages/codeupipe-rs (crate codeupipe, 0.1.0, no dependencies), built
test-first from conformance/API.md with codeupipe-ts as the reference.
A Filter takes &mut Payload (D2) because TS changes the Payload in place
and on_error must see what the failing Filter left. Taps, Predicates and
Hooks get &Payload, so "only a Filter writes" is a compile error.
46 tests, clippy -D warnings and fmt clean. The pricing example prints
the same result as the TS one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(examples): the pricing AuditTap reads instead of failing
The pricing example's AuditTap failed on every run, in Python, TS and
Rust, to show that a Tap failure is recorded and the run goes on. Now it
prints what it reads and State has no errors. The rule is unchanged and
the conformance cases and core tests still check it.
The compile-ts strict type check had no console, which is why the TS
Tap threw instead of logging. The check now declares console, which
every JavaScript host has, and nothing else of a browser or of Node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): conformance runner, rust:run passes all 27 cases
conformance/runners/rust runs the cases against packages/codeupipe-rs,
with its own small JSON reader and writer (no download, so it builds
offline) and one file per step type, named as in the TS runner. run.py
gets the rust:run target and leaves it out when there is no cargo.
add_filter and add_valve take the step's type, which gives its label, so
the runner hands each Filter over by its own type through a small generic
trait and not as a Box<dyn Filter>.
Checked that the runner catches a broken runtime: with Tap failures not
recorded, 3 cases fail. API.md gets a Rust column, the README a Rust
package row, MATRIX.md all eight targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: measure speed without judging it, and add a Rust job
The speed job compared each number with a baseline. A shared CI machine
is too noisy for that, so the benchmark job now only measures every
target, Rust included, and keeps the numbers in the job summary and as an
artifact. It never fails the run. The perf pytest test is removed;
bench.py --check stays for a machine with fixed resources.
bench.py measures rust:run. Its baseline is kept under its own key,
"rust", since it does not depend on the Python version, and --record
replaces only the targets it measured.
The new rust job runs fmt, clippy, the tests, the example and rust:run
conformance, and builds with Rust 1.75, the crate's rust-version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* perf(rust): cut per-run allocation in the core
A run spent its time in allocation, not in dispatch. Three changes, with
no change to behaviour or API:
- Map::insert looks the key up first, so a key that is there makes no
new String.
- State keeps its storage between runs and writes the labels over.
- A key is an Arc<str>, made once and shared by the index and by every
copy. A map of 8 keys or fewer is searched in order and has no index;
a larger one gets an index with room to grow. The hash stays SipHash,
so crafted keys from untrusted JSON cannot slow a map down.
10 filters: 1.10 -> 0.25 us. 50 new keys: 7.26 -> 4.24 us. The dynamic run
is now faster than compiled Python in 4 of 5 scenarios, so the plan puts
the Rust compiler (phase 3) on hold. Guard tests first: a shorter run
after a longer one, maps on both sides of the index size, Send + Sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): a payload of your own, with Payload as the default
Payload stays the default: a map that any developer can read, which
shows that anything can go there. A type of your own can take its place:
each primitive takes the data type as a parameter that defaults to
Payload (Filter<P = Payload>, Pipeline<P = Payload>), and the type opts
in with impl PipelineData for Order {}, which asks for Clone and an
optional stamp for lineage. Code written for Payload does not change.
A step for one type in a pipeline of another is a compile error, checked
by a doc test. examples/typed_pricing is the pricing example with an
Order struct and gives the same result. Typed, the pricing run takes
0.024 us against 0.18 us, and 50 writes 0.46 us against 4.3 us.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: a type of your own in Python and TypeScript, as in Rust
The contract gets "A type of your own": Payload stays the default, the
form that the contract, the conformance cases and the compilers use. A
pipeline can carry another type instead; the Pipeline asks only for
copy(). Lineage is kept only by a type that has _stamp(label) (stamp in
Rust); for any other type it does nothing and the run goes on.
Python took such an object already, by duck typing, but crashed with
lineage on. It now calls _stamp only when the object has it. TypeScript
gets the type parameter that Rust has: Filter, Predicate, Tap, Hook and
Pipeline default to Payload, and PipelineData<P> asks for copy() and an
optional _stamp. Mixing types does not type-check (@ts-expect-error).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(compile): follow the Payload parameter under any name
A Filter or Tap whose parameter was not called payload (call(self,
order)) compiled to a function of data whose body still used order, so
the compiled step raised NameError: code that compiled and did not give
the output of the dynamic run. The rewrite knew only the name payload.
The rewrite now takes the parameter's own name. A Valve's predicate keeps
its name too, where before it was renamed to payload, which merged it
with any unrelated local called payload. A refusal names the parameter as
the step wrote it (order.price, not payload.price), which also covers a
step over a type of your own.
Tests first: the three output modes against the dynamic run, with
parameters called order, p and data and a local called payload, and the
refusal of order.price. 538 tests, conformance and 2,000 random pipelines
pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): the State for each run is not in Rust yet
The contract on beta now says that a State belongs to one run: run(p,
state), a run inside a run, and state on the Pipeline as the State of the
run that started last. rust:run reports the three state/ cases as
unsupported, not as passed. API.md says so in its notes on Rust, and the
plan lists what the change needs in Rust: run(&self), a run with the
caller's State, state() behind a RefCell, shared errors, and the cost to
the State storage that is kept between runs today.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: install codeupipe for the judge in the TypeScript and Rust jobs
conformance/run.py is a codeupipe pipeline since #15, so it imports
codeupipe. The TypeScript and Rust jobs did not install the package, and
their conformance step failed with ModuleNotFoundError (on #15 as well;
CI does not run on pushes to beta). Checked locally with nothing but
packages/codeupipe on the path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): a State for each run; rust:run passes all 30 cases
The contract says that a State belongs to one run, not to the Pipeline.
In Rust:
- run(&self) in place of run(&mut self), so a step can hold the pipeline
it is in (an Rc) and run it again while it runs.
- run_with_state(&input, &mut state) for the caller's State, and
State::new().
- state() gives an Rc<State>: the State of the run that started last,
once it has ended. Each run takes a number when it starts; its State
is kept only if no later run has started. A snapshot never changes
under a reader, and holding one never blocks a run.
- A State's errors are Arc<dyn Error> (SharedError), so a State can be
cloned: with the caller's State, the Pipeline keeps a copy.
Two States take turns, so a run still makes no new label String; the Rc
of a run costs about 0.03 us (10 filters: 0.25 -> 0.28 us). The runner
plays run_again as the TypeScript runner does and reports the caller's
State, pipeline_state and inner_states. Tests first (state_per_run.rs);
breaking the "started last" rule fails one case and one unit test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
packages/codeupipe-lint-rs: codeupipe-lint, a codeupipe pipeline of 11 Filters that carries a LintRun of its own in place of a Payload. It reads the source with syn and checks the rules that the Rust compiler does not: CUP-I000, A003, A005, I003, I005, I006, I007, and the warnings I008 and I009. The ids, the report, the exit codes, the markers and cup.config.json are those of the Python and TypeScript linters. It is a Cargo workspace of its own, so the runtime still builds offline. A new CI job runs fmt, clippy, the tests, the linter with --strict on all the Rust code of the repository, and Rust 1.75. The Rust code passes it: step types declared in test functions moved to the top of their files, and cup: allow-plain markers with a reason on the runtime's own types, the conformance runner and the linter's helpers. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the 2.0 work from
betatomain, the default branch, so that a visitor sees 2.0 first.beta: the eight primitives in Python 2.0.0b1, the TypeScript packages 0.1.0, the Rust runtime and its linter 0.1.0, the conformance suite and the API contract.mainhas no commit thatbetadoes not have, so the merge has no conflict.main(@main,branch = "main"), not at thebetabranch.The version marks the beta, not the branch:
pip install codeupipestill gives 1.0.0, and the README says that 2.0.0b1 is a beta.🤖 Generated with Claude Code