Skip to content

dev: add fuzzing profiles - #328

Merged
DecimalTurn merged 34 commits into
latestfrom
dev-fuzz5
Oct 31, 2026
Merged

DecimalTurn merged 34 commits into
latestfrom
dev-fuzz5

Conversation

@DecimalTurn

@DecimalTurn DecimalTurn commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Summary

Add composable fuzz profiles for patch() and use structure-first generation to cover AOT layouts that the general TOML generator rarely produces. The new profiles exposed nested AOT formatting and mixed-array append cases, which this PR also fixes.

Changes

  • Refactor the fuzz runner into a shared lifecycle for source generation, mutation, formatting and comparison. Keep the existing numbered variants and their seed behavior.
  • Add the aot, aot-format, aot-general and general-format profiles, plus a structure-first AOT generator and runner.
  • Add seed inspection and failure distillation tools. Generated regression tests now resolve imports from the output file, preserve nested Date values, replay key deletion, and snapshot recorded mutation plans.
  • Fix patching of mixed AOT arrays when appending primitive values, and preserve nested table and AOT section formatting when rebuilding entries unless the caller explicitly selects inline-table formatting.
  • Add regression coverage for nested AOT ownership, including the seed 112 case and document the fuzzing profiles and workflow.

Copilot AI balanced review requested due to automatic review settings October 18, 2026 01:05
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Benchmark comparison

Each workload measured this branch against latest on the same runner.
Ratios close to 1.0 are within run to run noise. They indicate improvements (higher is better) compared to baseline.

Parse Benchmark Results

Time per iteration. Lower is better. Change is throughput vs. reference: + is faster, - is slower.

📖 Parse, smol-toml spec example (toml-spec-example.toml)

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 (Date) 7.93 µs/iter 1x
🥈 smol-toml@1.9.1 (Temporal) 13.00 µs/iter 1.64x
🥉 @iarna/toml@3.0.0 27.85 µs/iter 3.51x
4 @decimalturn/toml-patch@3.3.1 (baseline) 45.05 µs/iter 5.68x
5 @decimalturn/toml-patch (Date, current) 45.12 µs/iter 5.69x 0%
6 @decimalturn/toml-patch (Temporal, current) 61.83 µs/iter 7.80x

📖 Parse, smol-toml 5MB mixed (5mb-mixed.toml)

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 (Date) 72.35 ms/iter 1x
🥈 smol-toml@1.9.1 (Temporal) 264.40 ms/iter 3.65x
🥉 @decimalturn/toml-patch@3.3.1 (baseline) 438.40 ms/iter 6.06x
4 @decimalturn/toml-patch (Date, current) 440.45 ms/iter 6.09x 0%
5 @decimalturn/toml-patch (Temporal, current) 709.54 ms/iter 9.81x
- @iarna/toml@3.0.0 DNF DNF

📖 Parse, iarna spec example v0.4.0 (0A-spec-01-example-v0.4.0.toml)

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 (Date) 48.00 µs/iter 1x
🥈 smol-toml@1.9.1 (Temporal) 61.43 µs/iter 1.28x
🥉 @iarna/toml@3.0.0 175.53 µs/iter 3.66x
4 @decimalturn/toml-patch@3.3.1 (baseline) 256.57 µs/iter 5.35x
5 @decimalturn/toml-patch (Date, current) 265.28 µs/iter 5.53x -3%
6 @decimalturn/toml-patch (Temporal, current) 292.98 µs/iter 6.10x

Stringify Benchmark Results

Time per iteration. Lower is better. Change is throughput vs. reference: + is faster, - is slower.

✍️ Stringify, smol-toml spec example (toml-spec-example.toml)

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 7.57 µs/iter 1x
🥈 @iarna/toml@3.0.0 29.36 µs/iter 3.88x
🥉 @decimalturn/toml-patch@3.3.1 (baseline) 133.12 µs/iter 17.59x
4 @decimalturn/toml-patch (current) 137.41 µs/iter 18.15x -3%

✍️ Stringify, smol-toml 5MB mixed (5mb-mixed.toml)

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 98.30 ms/iter 1x
🥈 @iarna/toml@3.0.0 302.52 ms/iter 3.08x
🥉 @decimalturn/toml-patch (current) 904.23 ms/iter 9.20x +1%
4 @decimalturn/toml-patch@3.3.1 (baseline) 910.97 ms/iter 9.27x

✍️ Stringify, iarna spec example v0.4.0 (0A-spec-01-example-v0.4.0.toml)

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 32.21 µs/iter 1x
🥈 @iarna/toml@3.0.0 130.04 µs/iter 4.04x
🥉 @decimalturn/toml-patch (current) 605.72 µs/iter 18.81x 0%
4 @decimalturn/toml-patch@3.3.1 (baseline) 607.93 µs/iter 18.88x

🧩 Patch benchmarks

Full and lite patching are compared with smol-toml. The smol-toml method parses the document, updates the object, and stringifies it. Timings include each implementation’s complete operation; fixture creation and correctness checks are outside the timed section.

Current and base-branch full and lite patch builds are compared in the same run.

Edit existing scalar values

Change several existing root and nested scalar values.

🧩 Edit existing scalar values, representative-document

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 6.92 µs/iter 1x
🥈 @decimalturn/toml-patch (lite, current) 104.74 µs/iter 15.13x +24%
🥉 @decimalturn/toml-patch@3.3.1 (lite, baseline) 130.04 µs/iter 18.79x
4 @decimalturn/toml-patch@3.3.1 (full, baseline) 359.89 µs/iter 51.99x
5 @decimalturn/toml-patch (full, current) 366.63 µs/iter 52.97x -2%

Edit an unrelated scalar with many top-level AOT scopes

Change one root value while each named AOT owns a nested subtable.

🧩 Edit an unrelated scalar with many top-level AOT scopes, aot-scopes-64

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 166.43 µs/iter 1x
🥈 @decimalturn/toml-patch (lite, current) 1.65 ms/iter 9.92x +147%
🥉 @decimalturn/toml-patch@3.3.1 (lite, baseline) 4.07 ms/iter 24.48x
4 @decimalturn/toml-patch@3.3.1 (full, baseline) 4.25 ms/iter 25.56x
5 @decimalturn/toml-patch (full, current) 4.68 ms/iter 28.11x -9%

🧩 Edit an unrelated scalar with many top-level AOT scopes, aot-scopes-256

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 676.95 µs/iter 1x
🥈 @decimalturn/toml-patch (lite, current) 6.55 ms/iter 9.67x +115%
🥉 @decimalturn/toml-patch@3.3.1 (lite, baseline) 14.08 ms/iter 20.80x
4 @decimalturn/toml-patch@3.3.1 (full, baseline) 21.16 ms/iter 31.26x
5 @decimalturn/toml-patch (full, current) 24.97 ms/iter 36.89x -15%

🧩 Edit an unrelated scalar with many top-level AOT scopes, aot-scopes-1024

Library Performance Slowdown Change
🥇 smol-toml@1.9.1 2.88 ms/iter 1x
🥈 @decimalturn/toml-patch@3.3.1 (lite, baseline) 28.00 ms/iter 9.73x
🥉 @decimalturn/toml-patch (lite, current) 29.15 ms/iter 10.13x -4%
4 @decimalturn/toml-patch@3.3.1 (full, baseline) 141.82 ms/iter 49.27x
5 @decimalturn/toml-patch (full, current) 152.73 ms/iter 53.06x -7%

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Format inconsistencies and scaffold-generation defects prevent reliable reproduction of fuzz failures.

5 open issues
What changed in this PR

Refactors patch fuzzing into composable profiles and adds focused arrays-of-tables coverage.

Changes:

  • Extracts configurable generation, mutation, formatting and comparison strategies.
  • Adds numbered runners, profile inspection and regression scaffolding.
  • Adds deterministic tests and updates documentation and CI.
File Description
src/​__tests__/​inspect-fuzz-seed.ts Adds profile inspection.
src/​__tests__/​fuzz5.test.ts Tests structural seeds.
src/​__tests__/​fuzz1.test.ts Tests baseline seeds.
src/​__tests__/​fuzz-profiles.ts Defines composable profiles.
src/​__tests__/​fuzz-patch5.ts Adds structural generation and mutations.
src/​__tests__/​fuzz-patch4.ts Migrates date-focused configuration.
src/​__tests__/​fuzz-patch3.ts Migrates layout configuration.
src/​__tests__/​fuzz-patch2.ts Migrates dotted-key configuration.
src/​__tests__/​fuzz-patch1.ts Adds numbered baseline entry point.
src/​__tests__/​fuzz-patch.ts Extracts shared runner and metadata.
src/​__tests__/​fuzz-composable.test.ts Checks composition and determinism.
scripts/​fuzz-run5.ts Adds structural seed sweeps.
scripts/​fuzz-run1.ts Adds numbered baseline sweeps.
scripts/​fuzz-run.ts Removes unnumbered runner.
scripts/​distill-profile.ts Generates regression scaffolds.
package.json Updates fuzz commands.
docs/​PLAN-Composable-Fuzz-Harness.md Records architecture and migration plan.
docs/​Fuzz-Testing.md Documents variants and profiles.
.github/​workflows/​test-and-build.yml Uses numbered baseline command.

Review effort: Balanced

Comment thread scripts/distill-profile.ts Resolved
Comment thread scripts/distill-profile.ts Resolved
Comment thread scripts/distill-profile.ts Resolved
Comment thread src/__tests__/fuzz-patch.ts Resolved
Comment thread src/__tests__/fuzz-patch5.ts Resolved
@DecimalTurn
DecimalTurn marked this pull request as ready for review October 29, 2026 19:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/patch.ts Resolved
Comment thread src/__tests__/fuzz-patch.ts Resolved
Comment thread src/__tests__/inspect-fuzz-seed.ts Resolved
Comment thread src/patch.ts Resolved
Comment thread src/patch.ts Resolved
Comment thread src/patch.ts Resolved
Comment thread src/patch.ts Resolved
@DecimalTurn
DecimalTurn merged commit 3039eff into latest Oct 31, 2026
43 of 44 checks passed
@DecimalTurn
DecimalTurn deleted the dev-fuzz5 branch October 31, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants