Synsema

Artifact for the NeurIPS 2026 paper:

Synsema: Syntax-Guided Learning of Semantically Valid Programs
Levin Winter, Cong Li, Hao Sun, and Zhendong Su.

Generating inputs for systems with strict syntactic and semantic constraints like compilers is hard: hand-written generators demand deep domain expertise, while learned models must master syntax and semantics at the same time. Synsema combines both worlds by reformulating the task as syntax-guided reinforcement learning: a grammar guarantees syntactic validity, so the agent only has to learn the language's semantics from a drastically reduced sampling space. We instantiate it on Rust and Java and apply it to compiler testing against rustc and javac.

Repository layout

• src/ — Synsema agent and environments (Rust, libtorch via tch).
• grammars/{rust,java}/ — Yacc grammars and substitutions used by the agent.
• targets/rust/ — Build script for the instrumented rustc, with the revision pinned in REVISION and coverage patches in patches/.
• figures/figures.py — DuckDB + matplotlib script that regenerates the paper's plots and tables from a directory of run logs.

Setup

1. libtorch

The tch crate needs libtorch 2.9 at build and run time. Download the matching CPU or CUDA build from https://pytorch.org/get-started/locally/, unpack it, and set the LIBTORCH environment variable to the absolute path of the unpacked directory. Either set it in your shell, or create a .cargo/config.toml with:

[env]
LIBTORCH = "/absolute/path/to/libtorch"

2. Build the agent

cargo build --release

This compiles the synsema binary and the bundled C shims (libcoverage, libreprl), and downloads JaCoCo for Java coverage.

3. Build the target compilers

For Rust, build the instrumented rustc (revision pinned in targets/rust/REVISION):

(cd targets/rust && bash build.sh)
# instrumented compiler ends up under targets/rust/rust/build/<host-triple>/stage1/bin/rustc

For Java, install a JDK (i.e. sdk install java) and build the javac-diagnostics-wrapper.

git clone https://github.com/eisopux/javac-diagnostics-wrapper \
    targets/javac/javac-diagnostics-wrapper
(cd targets/javac/javac-diagnostics-wrapper && ./gradlew assemble)
# produces build/libs/javac-diagnostics-wrapper-all.jar

Coverage uses JaCoCo, which build.rs downloads automatically so no patching of the JDK is needed. At run time, point --compiler-path at the JDK's java binary (not javac), and run synsema from the repo root so the relative wrapper jar path resolves.

Reproducing the paper

The evaluation has three experiments, each repeated three times per configuration (weights are randomly initialized each run and no fixed seed). Run them all, then point figures/figures.py at the shared root directory.

Run the experiments

The first positional argument selects the agent (top-down, bottom-up, chart, baseline). --profile picks the target language. --data-dir is where logs and checkpoints are written. The layout below is what figures.py expects.

# main comparison: 4 agents × 2 languages × 3 repetitions (24 runs of 5 days each)
./target/release/synsema <agent> \
    --profile <rust|java> \
    --compiler-path <path-to-rustc-or-java> \
    --data-dir <root>/<rust|java>/<agent>/<n>

# ablations on Rust + chart agent (3 configs × 3 repetitions, n in {1,2,3})
./target/release/synsema chart --profile rust ... --data-dir <root>/rust/no-span-loc/<n> \
    --disable-span-localization
./target/release/synsema chart --profile rust ... --data-dir <root>/rust/prog-decay/<n> \
    --success-reward-mode program-decay
./target/release/synsema chart --profile rust ... --data-dir <root>/rust/disable-all/<n> \
    --disable-span-localization --success-reward-mode program-decay

The unmodified chart runs from the main comparison double as the "default" ablation baseline, so they don't need to be re-run.

run_experiments.sh enumerates all 33 commands. It runs them back-to-back by default. Override RUN to dispatch each command through your job scheduler of choice. To smoke-test the matrix instead by running one repetition per config, each capped at 60 seconds, run SANITY=1 bash run_experiments.sh.

Hyperparameters default to the values reported in the paper (GPT-2 124M: 12 layers/heads, 768-dim embeddings). See synsema --help and the appendix for the full list.

Regenerate tables and figures

The full set of run logs used in the paper is available on Zenodo: https://doi.org/10.5281/zenodo.20339950. Download and unpack experiments.tar.zst. Use the absolute path of the directory containing rust/ and java/ as <root> below.

Install the Python dependencies, then run the script:

python3 -m pip install ".[plots]"
(cd figures && python3 figures.py <root>)

<root> is the shared parent of the run directories above. The script reads actor.csv and program.csv from each run and writes:

Paper artifact	Produced by
Table 1 (effectiveness, success rate, steps/token)	stdout of figures.py
Figure 2 — edge coverage on rustc	figures/edge_coverage_rust.pdf
Figure 3 — edge coverage on javac	figures/edge_coverage_java.pdf
Figure 4 — reward ablation	figures/ablation.pdf
Final ablation numbers	stdout of figures.py

Additional analyses

analysis/README.md describes how to collect data and reproduce the Fuzz4All comparison, program NLL, and semantic-feature results from the data files on Zenodo.

Citation

If you use Synsema, please cite our NeurIPS 2026 paper.

@inproceedings{winter2026synsema,
  title = {Synsema: Syntax-Guided Learning of Semantically Valid Programs},
  author = {Levin Winter and Cong Li and Hao Sun and Zhendong Su},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year = {2026},
  url = {https://openreview.net/forum?id=HR79HDRXwx}
}