test: Debian-oracle scoreboard in CI with per-tool floors (+ sed -f regression it caught)

[chaliy]

Requested by Михайло · project thread

What changed

• New scoreboard. just debian-oracle (scripts/debian-oracle/run.sh) runs the pseudo-linus test bench against bashkit. The bench has about 9,600 command cases across 45 tools, each recorded on real Debian: stdout, stderr, exit status and the final file tree.
• How the bench is run.
  • It is cloned at a pinned commit into target/debian-oracle/ at run time, never vendored. Parts of its corpus derive from GPL test suites (GNU grep and sed).
  • A small adapter (scripts/debian-oracle/adapter/) loads each fixture into the in-memory filesystem and scores the result with the bench's own comparator.
• Floors. scripts/debian-oracle/floors.tsv records each tool's minimum lenient pass count. A run fails when a tool drops more than one case below its floor; the one case of slack covers wall-clock timeouts. After a fix, --update-floors locks the gain in.
• New workflow .github/workflows/debian-oracle.yml.
  • Runs on PRs touching builtins, the interpreter, the parser, the VFS or jaq-json, plus weekly and on demand.
  • Puts the score table in the job summary and uploads the failing cases as an artifact.
  • Takes about 3 minutes to build and 1 minute to run.
• sed fix. A sed -f script file ending in i\ + newline inserts an empty line again. fix(builtins): GNU parity for sort, xargs, nl, fold, od, du and 20 more coreutils (166 BashBox cases) #2585 made a bare i\ add nothing, and trimming the file's trailing newline turned the escaped newline into exactly that. The scoreboard caught it on its first run (sed/misc-insert-nl), which is the kind of regression it is meant to stop.
• Docs: a new "Debian-Oracle Scoreboard" section in knowledge/operations/testing.md.

Why

The awk, jq, sed and grep work (#2583 to #2609) was driven by this bench. Without a gate, those gains can slip back, as the sed case above already did once between merges.

Before / After

Before: nothing in CI compares bashkit with real Debian output.

After, current main:

| awk  | 260 | 93.1% | 94.2% (245) |
| grep | 185 | 96.8% | 96.8% (179) |
| jq   | 194 | 79.4% | 88.7% (172) |
| sed  | 172 | 96.5% | 99.4% (171) |
| **Total** | 9579 | 27.8% | 32.9% |

On those four tools, lenient scores were 63.8% to 99.4% when this work started; the table shows where they are now. The sed regression:

$ printf '/foo/i\\\n' > in1; printf 'bar\nfoo\n' | sed -f in1
GNU sed / after: bar, (empty line), foo
before:          bar, foo

Two consecutive local runs against the committed floors both passed. Only coreutils moved, by one case, which is within the tolerance.

Risk

• Low. This adds a new CI job plus a two-line sed fix with a test.
• The job depends on GitHub fetching the pinned pseudo-linus commit, and builds with --ignore-rust-version because the bench declares Rust 1.98 while we pin 1.95; it builds clean on 1.95.

Checklist

• Tests added or updated (sed -f regression test; the scoreboard runs green against its floors)
• Backward compatibility considered

---

Generated by Claude Code

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	bashkit	333dfbd	Commit Preview URL Branch Preview URL	Oct 07 2026, 11:59 PM