Skip to content

feat(awk): rewrite awk to match gawk 5.2 - #2583

Merged
chaliy merged 3 commits into
mainfrom
claude/project-thread-1mku65
Oct 7, 2026
Merged

chaliy merged 3 commits into
mainfrom
claude/project-thread-1mku65

Conversation

@chaliy

@chaliy chaliy commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Михайло · project thread

What changed

awk (and gawk/mawk/nawk) is rewritten to behave like gawk 5.2, which is Debian's awk.

  • Language. All of POSIX awk is supported. The gawk extensions agents use are supported too: gensub, asort/asorti, patsplit, strftime/mktime/systime, BEGINFILE/ENDFILE, switch, arrays of arrays, PROCINFO["sorted_in"], FIELDWIDTHS/FPAT, IGNORECASE, regex RS with RT, typeof and the bit functions.
  • Commands work. system(), print | cmd and cmd | getline run in the sandbox shell through an execution-plan driver, the same mechanism make uses. Output stays in order.
  • gawk output. Number printing, printf (%c, %i, *, %n$, inf/nan), rand()'s sequence, regex matching (POSIX leftmost-longest) and substr/getline/typeof edge cases all follow gawk.
  • gawk diagnostics. Syntax errors print awk: cmd. line:N: with a caret and exit 1. Runtime errors print located fatal: messages and exit 2. Usage goes to stderr with exit 1.
  • Options. -f (repeatable), -e/--source, --, -F t, -v with escapes, --csv, and gawk's no-op flags are accepted.
  • Limits. Every existing resource cap is kept with the same messages. split() and record splitting now stop at the memory cap instead of first building a huge vector.

Why

pseudo-linus runs a Debian-oracle bench, and on it bashkit's awk matched only 51.5% of 260 cases. It was the weakest common text tool there. The old design could not reach gawk: it was a sync evaluator that hopped threads for I/O, arrays were stored as "name[key]" variables, function locals were broken, and commands were unsupported.

Before / After

Debian-oracle differential run (gawk 5.2.1 goldens, 260 awk cases):

before: awk | 260 | 50.8% strict | 51.5% lenient
after:  awk | 260 | 92.7% strict | 93.8% lenient
  • All remaining failures but one invoke mawk by name and expect its own dialect (-W options, messages, substr, division by zero; L-AWK-004).
  • The one gawk case left is unsorted for (k in a) order, which follows gawk's hash tables (L-AWK-001). That is a follow-up.
$ awk 'BEGIN { print "before"; r = system("echo inside; exit 3"); print "after", r }'
before:  before / after          (command silently not run, empty status)
after:   before / inside / after 3

$ printf 'b\na\n' | awk '{ print | "sort" } END { close("sort"); print "done" }'
before:  awk: pipe output redirection (|) is not supported   (exit 2)
after:   a / b / done

$ awk 'BEGIN { print ( }'
after:   awk: cmd. line:1: BEGIN { print ( }
         awk: cmd. line:1:                 ^ syntax error      (exit 1)
  • Tests: bashkit lib 2837 and integration 1346 pass. awk spec 144/144, including new cases for system, pipes, cmd | getline, syntax errors and substr.
  • Stack: the depth-64 recursion test needed about 3 MiB of debug-build stack in the first cut. Splitting each expression and statement kind into its own boxed future brought it under 512 KiB.

Risk

  • Medium. This is a full rewrite of a heavily used builtin. It is covered by the existing unit, spec and security suites plus the oracle run.
  • Intentional output changes toward gawk. A missing input file now exits 2. --csv no longer sets OFS to ,. substr(s, 0, n) keeps n characters. Redirected getline leaves NR alone. -0 prints 0. Non-ASCII identifiers are now a syntax error.
  • Commands through Builtin::execute. Direct calls have no shell, so commands report 127 with a notice (L-AWK-003).

Checklist

  • Tests added or updated (unit tests per module, spec cases, updated security/regression tests where gawk differs)
  • Backward compatibility considered (gawk fidelity wins; caps and their messages unchanged; knowledge, limitations L-AWK-001..004 and TM-DOS-128 updated)

https://claude.ai/code/session_01VscpEzxtqpCKx5eRMpywNj

New lexer, parser, async evaluator and I/O layer targeting gawk, Debian's
awk. Commands (system, print | cmd, cmd | getline) run in the sandbox
shell through an execution-plan driver. ERE translation adds POSIX
leftmost-longest matching. Number output, printf, rand, substr, getline
and diagnostics follow gawk. All resource caps are kept; split and record
splitting now stop at the memory cap before building their vectors.

Debian-oracle awk score: 51.5% -> 93.8% of 260 cases.

Claude-Session: https://claude.ai/code/session_01VscpEzxtqpCKx5eRMpywNj
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 8bbe6de Commit Preview URL

Branch Preview URL
Oct 07 2026, 06:55 PM

chaliy added 2 commits October 7, 2026 18:32
…-1mku65

# Conflicts:
#	knowledge/foundations/builtins.md
#	knowledge/log.md
@chaliy
chaliy merged commit eee7287 into main Oct 7, 2026
47 checks passed
@chaliy
chaliy deleted the claude/project-thread-1mku65 branch October 7, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant