Skip to content

Annotate jawohl itself: no surface crate, and Stream crosses as a handle - #12

Open
zmaril wants to merge 5 commits into
mainfrom
demo/jedem-bindings
Open

Annotate jawohl itself: no surface crate, and Stream crosses as a handle#12
zmaril wants to merge 5 commits into
mainfrom
demo/jedem-bindings

Conversation

@zmaril

@zmaril zmaril commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Replaces the hand-written surface crate with annotations on jawohl's own
definitions, and upgrades the bindings from a batch shim to the real
incremental parser.

What changed

bindings/surface is deleted. It was a crate whose whole job was to restate
jawohl's API: two enums mirrored variant for variant with From impls in both
directions, plus eight functions wrapping Stream. Nothing connected it to the
thing it described, so nothing could catch it drifting.

The annotations now sit on the definitions:

  • Syntax and Validation derive jedem::Enum where they are declared
  • complete_json and get_closing_string_for_partial_json carry #[jedem::export]
  • the Stream impl carries it once, for the whole block

There is no hand-written code under bindings/ any more — only this README and
the two host tests.

Stream is a handle now

Python and JavaScript get the incremental parser itself. The old shim re-parsed
from byte zero on every call, which is quadratic over a document — the exact
thing the library exists to avoid.

s = jawohl.Stream.from_json_schema(schema)
for chunk in response:
    s.push(chunk)
    if s.is_irrecoverable():
        break                      # stop paying for a generation that cannot succeed

The host tests are rewritten around it, and now assert something the batch API
could not express: cancellation lands on the chunk that decided the schema was
unsatisfiable, not at end of document.

What is still not across

The two Rust move-builders cross unchanged too — with_number_profile and
with_max_depth take self and return Self, which names the same object, so
jedem mutates in place and hands the same handle back. Stream().with_max_depth(64)
chains in Python and JavaScript exactly as in Rust, with no annotation on either
method.

Four methods carry #[jedem(skip)], each with its reason in place:

method why
snapshot, changes need unionsValue is recursive, Event has struct variants
error, lowering_report need records, and return a reference

changes is the one to want back: it is how events would be consumed instead of
polling status.

Two consequences worth stating plainly

jawohl now depends on jedem. Nothing survives to runtime — the descriptor is
&'static data, and the generator is reached only from the #[cfg(test)] drift
guard — but it is a real dependency on an unpublished crate. cargo package
cannot resolve it, so jawohl 0.2.0 cannot ship until jedem 0.1 is on
crates.io
. The packaging job is marked continue-on-error with that written
beside it, so the failure stays visible without turning unrelated PRs red.

The MSRV rises from 1.70 to 1.78. jedem uses
#[diagnostic::on_unimplemented], stabilised in 1.78, and its proc-macro chain
floors at 1.71. Declared rather than patched around, and verified against a real
1.78.0 toolchain. jedem now declares the same floor in its own manifest, so the
next crate to depend on it does not have to discover this the way jawohl did.

The dependency currently points at a jedem branch (feat/handles), because
the handle and bindings: support it needs is in jedem PRs #11 and #12. Those
merge first; then the branch key comes off.

The drift guard moved into jawohl's own test suite, since the surface is now
jawohl's annotations. cargo test checks the committed bindings, JEDEM_WRITE=1 cargo test rewrites them.

Verified

  • cargo test --all-targets, cargo test --doc, clippy and fmt all clean
  • both host processes pass end to end against freshly generated bindings

Base automatically changed from chore/0.2.0-release-prep to main August 19, 2026 15:32
zmaril and others added 2 commits August 19, 2026 17:32
What jedem looks like applied to jawohl. bindings/surface/src/lib.rs is the only
hand-written file; the Python extension module and the Node addon are both
generated from it, and both are exercised by a real host process.

    python:  jawohl.is_irrecoverable(schema, '{"role":"sup')   -> True
    node:    jawohl.isIrrecoverable(schema, '{"role":"sup')    -> true

Each language gets what it would have written: snake_case in Python and
camelCase in JS from the same Rust names, Option<T> arriving as None and null,
a Result raising a ValueError and throwing an Error, and a synchronous function
staying synchronous rather than returning a promise.

It is a separate workspace on purpose. The published jawohl crate gains no
dependency from any of this -- still regex-automata and nothing else -- and
bindings/** is excluded from the package.

The honest limit, stated in the README rather than glossed: this is the
BATCH-SHAPED SUBSET. jedem v1 exposes functions over plain values, and jawohl's
real API is a stateful Stream, which is a handle. So every function here takes
everything received so far and answers one question about it, which means every
call re-parses from the beginning. For a tool call of a few hundred bytes that
costs nothing, and the most valuable question -- can this generation still
succeed? -- answers fine that way. For a long document it is quadratic. The
incremental Stream API reaches these languages when jedem grows handles, which
is exactly the argument for jedem steps 3 through 5.

A drift guard in the surface crate regenerates in memory and diffs against the
committed bindings, so a surface change nobody regenerated fails the build; it
also asserts jedem::Target::ALL is fully covered, so a new backend cannot be
added without a guard. Verified by mutation: renaming a function fails the test
with the regeneration command.

The README's "Not yet" section is replaced. It promised JavaScript and Python
wrappers in 2023 and never delivered, so this says plainly what exists, where,
and what does not.

One test caught me: in Python r'...\\' is TWO backslashes -- a complete escape,
not a dangling one -- so the completion correctly kept it. The test now asserts
both cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HsDxLrGdx6nPaXkVEWkNvS
Rewrites jawohl's bindings on the five jedem changes that landed, which existed
because of what this PR cost the first time.

The binding crates are generated in full now, not just their source: manifests,
module shims and node's build.rs all come from jedem. The two run scripts are
gone in favour of `cargo jedem run --python test.py`. The sixteen-line generator
main() is a ten-line file that is mostly the macro invocation. Hand-written
scaffolding drops from 323 lines to 26 across three files, and the surface --
the only thing anyone actually writes -- is the rest.

Three things got better rather than merely shorter.

Statuses and validations cross as REAL ENUMS. Python receives Syntax.Complete
and Validation.IrrecoverablyInvalid as members of a class; TypeScript receives
"Complete" and "IrrecoverablyInvalid" as a string-literal union. Both reject a
value that is not a variant. Previously a describe() helper flattened them to
strings and Python got 'irrecoverably_invalid' with nothing checking it. The
enums are mirrors of jawohl's own, because #[derive(jedem::Enum)] has to be
applied where a type is defined and jawohl takes no dependency on jedem; the
cost is one From impl each.

The six .map_err(|e| e.to_string()) calls are gone. Functions that can fail two
ways return Box<dyn Error> directly, which always worked -- jedem never
inspected the error type.

The marker `pub struct Jawohl;` is gone; the surface is a module.

Two rough edges found by using it, both recorded rather than papered over.

jedem keys a generated manifest's dependency on the RUST CRATE name, so a
package named jawohl-surface produced a manifest cargo could not resolve. The
package is now jawohl_surface so the two spellings match. jedem's own demo never
hit this because `hello` has no hyphen; worth fixing there.

Generated code must not be formatted. rustfmt rewraps method chains exceeding
its chain_width, which jedem cannot predict, so `cargo fmt --all` made the
committed bindings differ from a fresh generation and the drift guard failed.
Formatting generated code is the wrong operation on it -- you regenerate it. A
rustfmt.toml records the rule, and only the hand-written crate is formatted.

Also fixed: test.mjs still required ./.nodeimport/jawohl.node, a stale addon
from before `cargo jedem run` existed, which silently shadowed the real one and
produced confusing failures against an old surface.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HsDxLrGdx6nPaXkVEWkNvS
@zmaril
zmaril force-pushed the demo/jedem-bindings branch 2 times, most recently from 8a4aec8 to 87345a0 Compare August 19, 2026 19:15
@zmaril zmaril changed the title jawohl through jedem: Python and Node from one annotated surface Annotate jawohl itself: no surface crate, and Stream crosses as a handle Aug 19, 2026
The bindings were described by a separate crate that restated jawohl's
API: two enums mirrored variant for variant with `From` impls both ways,
and eight functions wrapping `Stream`. Every one of them was a place the
description could drift from the thing described, and the compiler could
not catch it because nothing connected them.

Now the annotations live on the definitions. `Syntax` and `Validation`
derive `jedem::Enum` where they are declared, `complete_json` and
`get_closing_string_for_partial_json` carry `#[jedem::export]`, and the
`Stream` impl carries it once for the whole block. `bindings/surface` is
gone -- there is no hand-written code under `bindings/` at all.

`Stream` crosses as a handle, so Python and JavaScript now get the
incremental parser rather than the batch shim. That shim re-parsed from
byte zero on every call, which is quadratic over a document; the point
of the library is that it does not have to be. The host tests are
rewritten around it: push chunk by chunk, watch a value settle, and
cancel on the chunk that decided the schema was unsatisfiable rather
than at end of document.

Six methods are marked `#[jedem(skip)]`, each with its reason in place.
`with_number_profile` and `with_max_depth` consume `self`, which cannot
mean anything once another language owns the handle. `snapshot`,
`changes`, `error` and `lowering_report` return unions or records, which
jedem cannot lower yet -- `changes` is the one to want back, since it is
how events would be consumed instead of polling `status`.

Two consequences worth being plain about:

jawohl now depends on jedem. Nothing of it survives to runtime -- the
descriptor is `&'static` data and the generator is reached only from the
`#[cfg(test)]` drift guard -- but it is a real dependency on a crate
that is not published, so `cargo package` cannot resolve it and jawohl
0.2.0 cannot ship until jedem 0.1 is on crates.io. The packaging job is
marked `continue-on-error` with that written next to it, so the failure
stays visible without turning unrelated pull requests red.

The drift guard moved into jawohl's own test suite, because the surface
is now jawohl's annotations. `cargo test` checks the committed bindings;
`JEDEM_WRITE=1 cargo test` rewrites them.
@zmaril
zmaril force-pushed the demo/jedem-bindings branch from 87345a0 to f82452a Compare August 19, 2026 19:17
jedem uses `#[diagnostic::on_unimplemented]`, stabilised in 1.78, and
its proc-macro chain (proc-macro2, quote, syn, unicode-ident) floors at
1.71. `rust-version` is a promise to consumers, so it moves rather than
the dependency being pinned or patched around; 1.78 is over two years
old, which is a reasonable floor to ask for.

Verified by building against a real 1.78.0 toolchain, not inferred.

jedem now declares the same MSRV in its own manifest, so the next crate
to depend on it reads the floor instead of discovering it here.

The MSRV job's comment no longer singles out serde_json: dev-dependencies
are excluded on principle, not because of one crate.
@zmaril
zmaril force-pushed the demo/jedem-bindings branch from bb504db to 8bdb7ed Compare August 19, 2026 19:24
jedem lowers Rust's move-builder now, so `with_number_profile` and
`with_max_depth` needed nothing done to them: the `#[jedem(skip)]`
markers come off and the methods stay exactly as they were. Python and
JavaScript get real chaining -- `Stream().with_max_depth(64)` -- and the
returned object is the same handle, not a copy.

`NumberProfile` derives `jedem::Enum`, which is what the builder taking
it needs. The compile error that asked for it named the missing derive
outright, which is what `#[diagnostic::on_unimplemented]` is there for.

Four skips remain, all of them return types with no lowering yet:
`snapshot` and `changes` need unions, `error` and `lowering_report` need
records.

Both host tests now drive the builders, including the case that only
makes sense with one: `with_number_profile(Exact)` defers a numeric
bound that `PlainDecimal` decides immediately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant