Skip to content

fix(jq): bound stream event traversal and allocation - #2621

Merged
chaliy merged 2 commits into
mainfrom
codex/jq-stream-budget
Oct 8, 2026
Merged

chaliy merged 2 commits into
mainfrom
codex/jq-stream-budget

Conversation

@chaliy

@chaliy chaliy commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

What changed

jq --stream now produces one path event at a time with shared ancestor key strings. Traversal state, event arrays, and --slurp retention are checked against the live-memory meter before allocation. Every traversal step consumes shared work and checks cancellation/deadlines, including with a non-emitting filter.

Why

Compact JSON could multiply long ancestor keys into a retained list of events before jq installed its memory meter. A script such as jq --stream empty /input.json therefore bypassed the intermediate-memory limit. This fixes Codex Security finding commit:1e898b2a69508191a49db08b6c85349e.

Before / After

Executed safely through Bash::exec / VFS and exec_with_options:

  • A 65,536-byte ancestor key with two null leaves and a 32,768-byte memory limit: before exit 0, after exit 5, jq: error: value size limit (32768 bytes) exceeded. A subsequent shell request succeeds.
  • A 1,024-byte key with 2,000 null leaves under a 65,536-byte limit and --stream --slurp empty: before exit 0, after exit 5 with the configured value-size limit.
  • Incremental consumption of those 2,000 leaves under the same small limit succeeds, counting 2,002 events (leaves plus closing events).
  • Non-emitting traversal exhausts a deliberately small shared work budget. Event ordering, filenames, slurp, and partial inputs consumption retain jq behavior.

These are bounded regressions; no host-OOM test was attempted.

Risk

Medium: input conversion and slurp now occur inside metered evaluation. Inputs exceeding the configured live-memory allowance fail cleanly. Input-depth restrictions and existing jq output/recursion protections remain enforced.

Validation

  • Rebased jq unit suite: 342 passed.
  • Rebased execution-budget regressions: 6 passed (including all 4 stream regressions).
  • All-feature clippy with warnings denied; formatting; capability/OKF/doc/workflow/changelog checks; cargo vet; 150 repository-script tests.
  • CLI smoke plus 100 generated inputs in each of 3 streaming modes match installed jq.
  • Parallel Criterion benchmark completed; just bench: 96 cases, zero errors, 100% Bash output match.
  • Broad local run: 3,670 unit tests passed. Integration execution then aborted in threat_model_tests::nesting_depth_security::misconfig_huge_ast_depth_still_safe on macOS. The same exact test also aborts on unmodified current main f286bf332c6eadcda068c645077432d7da7f729d with the same feature set. It parses nested Bash if statements and never invokes jq. No test was disabled and no stack limit was raised. Full CI is required before merge.

Checklist

  • Tests added or updated
  • Backward compatibility considered

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit a6cd92d Commit Preview URL

Branch Preview URL
Oct 08 2026, 01:38 AM

@chaliy
chaliy merged commit c3c52c1 into main Oct 8, 2026
49 checks passed
@chaliy
chaliy deleted the codex/jq-stream-budget branch October 8, 2026 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant