Repository navigation
dev: add fuzzing profiles - #328
Conversation
Benchmark comparisonEach workload measured this branch against Parse Benchmark ResultsTime per iteration. Lower is better. Change is throughput vs. reference: + is faster, - is slower. 📖 Parse, smol-toml spec example (toml-spec-example.toml)
📖 Parse, smol-toml 5MB mixed (5mb-mixed.toml)
📖 Parse, iarna spec example v0.4.0 (0A-spec-01-example-v0.4.0.toml)
Stringify Benchmark ResultsTime per iteration. Lower is better. Change is throughput vs. reference: + is faster, - is slower. ✍️ Stringify, smol-toml spec example (toml-spec-example.toml)
✍️ Stringify, smol-toml 5MB mixed (5mb-mixed.toml)
✍️ Stringify, iarna spec example v0.4.0 (0A-spec-01-example-v0.4.0.toml)
🧩 Patch benchmarksFull and lite patching are compared with smol-toml. The smol-toml method parses the document, updates the object, and stringifies it. Timings include each implementation’s complete operation; fixture creation and correctness checks are outside the timed section. Current and base-branch full and lite patch builds are compared in the same run. Edit existing scalar valuesChange several existing root and nested scalar values. 🧩 Edit existing scalar values, representative-document
Edit an unrelated scalar with many top-level AOT scopesChange one root value while each named AOT owns a nested subtable. 🧩 Edit an unrelated scalar with many top-level AOT scopes, aot-scopes-64
🧩 Edit an unrelated scalar with many top-level AOT scopes, aot-scopes-256
🧩 Edit an unrelated scalar with many top-level AOT scopes, aot-scopes-1024
|
There was a problem hiding this comment.
🟡 Changes recommended
Format inconsistencies and scaffold-generation defects prevent reliable reproduction of fuzz failures.
5 open issues
Resolve generated imports relative to the output directory · New Preserve Date values when serializing generated mutations · New Generate property deletion for delete-key mutations · New Deep-clone mutation plan payloads when recording · New Disable randomized format generation for the AOT runner · New
What changed in this PR
Refactors patch fuzzing into composable profiles and adds focused arrays-of-tables coverage.
Changes:
- Extracts configurable generation, mutation, formatting and comparison strategies.
- Adds numbered runners, profile inspection and regression scaffolding.
- Adds deterministic tests and updates documentation and CI.
| File | Description |
|---|---|
src/__tests__/inspect-fuzz-seed.ts |
Adds profile inspection. |
src/__tests__/fuzz5.test.ts |
Tests structural seeds. |
src/__tests__/fuzz1.test.ts |
Tests baseline seeds. |
src/__tests__/fuzz-profiles.ts |
Defines composable profiles. |
src/__tests__/fuzz-patch5.ts |
Adds structural generation and mutations. |
src/__tests__/fuzz-patch4.ts |
Migrates date-focused configuration. |
src/__tests__/fuzz-patch3.ts |
Migrates layout configuration. |
src/__tests__/fuzz-patch2.ts |
Migrates dotted-key configuration. |
src/__tests__/fuzz-patch1.ts |
Adds numbered baseline entry point. |
src/__tests__/fuzz-patch.ts |
Extracts shared runner and metadata. |
src/__tests__/fuzz-composable.test.ts |
Checks composition and determinism. |
scripts/fuzz-run5.ts |
Adds structural seed sweeps. |
scripts/fuzz-run1.ts |
Adds numbered baseline sweeps. |
scripts/fuzz-run.ts |
Removes unnumbered runner. |
scripts/distill-profile.ts |
Generates regression scaffolds. |
package.json |
Updates fuzz commands. |
docs/PLAN-Composable-Fuzz-Harness.md |
Records architecture and migration plan. |
docs/Fuzz-Testing.md |
Documents variants and profiles. |
.github/workflows/test-and-build.yml |
Uses numbered baseline command. |
Review effort: Balanced
…ecture and migration strategy
b3a5164 to
c6bbcbf
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unrelated edits can trigger quadratic document scans, while array reconstruction can delete existing comments or apply incorrect formatting.
7 open issues
Avoid quadratic scans of unrelated AOT scopes · New Include source and patched TOML in successful results · New Dispatch variant 5 to fuzzOne5 with its default mutation count · New Propagate inlineTableStart through TomlDocument.patch · New Preserve comments when rebuilding existing AOT entries · New Scope style precedents to the owning AOT entry · New Cache converted CST data across Add operations · New
5 resolved since last review
Review effort: Balanced


Summary
Add composable fuzz profiles for
patch()and use structure-first generation to cover AOT layouts that the general TOML generator rarely produces. The new profiles exposed nested AOT formatting and mixed-array append cases, which this PR also fixes.Changes
aot,aot-format,aot-generalandgeneral-formatprofiles, plus a structure-first AOT generator and runner.Datevalues, replay key deletion, and snapshot recorded mutation plans.