ReadEnough reproducibility package

This anonymous package contains the code needed to build the benchmark inputs, record complete long-context trajectories, replay the stopping policies, and regenerate the reported tables. It intentionally contains no author metadata, credentials, local paths, benchmark plaintext, model caches, checkpoints, or paid inference outputs.

Method frozen by this package

The shared ReadEnough controller uses:

• 24,000-character document chunks;
• 6,000-character rolling-tail notes;
• confidence threshold theta = 0.995;
• stability tolerance epsilon = 0.05;
• a trailing window of W = 3 complete answer states;
• numeric verbalized-confidence baselines stop at 99.5 on the recorded 0--100 scale;
• no question-only probe; MCQ decisions use the contextual posterior directly.

For MCQ, confidence is the largest exact A–D posterior probability and change is natural-log Jensen–Shannon divergence. For open answers, confidence is the geometric mean probability of the generated draft tokens and change is 1 - token_F1 between normalized drafts. For open answers, all three drafts must be valid and individually exceed theta; the mean of the two adjacent changes must be at most epsilon. Empty, abstaining, or low-confidence drafts reset eligibility until three fresh qualifying states have been observed. For MCQ, the current posterior must clear theta and the same W3 stability test is applied.

All reported comparisons should be made by recording every sample to full context once (rule: none) and replaying the policies offline. This makes the comparisons paired and prevents a truncated run from being treated as a full trajectory.

Execution flow and gold isolation

The prepared JSONL keeps reference answers because they are required for offline scoring, but the reader model and stopping controller never receive them. For every chunk, run_fold.py exposes only:

1. the question (and all answer choices for an MCQ), the previous notes, and the current context chunk to the notes updater;
2. the question and current notes to the measured answer probe and optional verbalized/END baselines; and
3. the resulting confidence and stability measurements to the stopping rule.

The runner does not put answer, answers, evidence_char_starts, needle_position_chars, or any correctness result into a model prompt or a stopping decision. Evidence and answer fields are used only after inference by the offline scorer, evidence-timing analysis, and (for BrowseComp-Plus) the separate semantic judge. The answer may naturally occur inside the context; locating it by reading the chunks is the benchmark task, not gold leakage.

The common reproduction workflow is therefore:

prepare benchmark JSONL
    -> record complete trajectories with --rule none
    -> add deterministic evidence labels when the benchmark requires them
    -> replay policies and score the saved trajectories offline

Use a fresh output path for each benchmark/model/configuration. Never combine different models, dataset builds, chunk sizes, or online-stopped trajectories in one JSONL. The commands in the benchmark sections below follow this exact workflow and can be copied from the repository root unless that section says to change directories first.

Each successful row stores a deterministic execution fingerprint covering the model, endpoint, optional provider pin, prompts/controller protocol, chunk and notes limits, answer-token limit, stopping constants, and recorded comparator signals. Resumption requires an exact fingerprint match and rejects older or incompatible rows. The sole cross-input exception is the documented pair of disjoint, explicitly tagged RULER-HotpotQA cohorts, which must still have the same execution fingerprint.

Baseline scope

Headline comparisons use exactly four baselines: full reading, random stop (uniform over chunk endpoints, averaged over 200 draws), verbalized gate (numeric confidence at 99.5), and the separately prompted END gate. The fixed-at-25% and oracle-stop rows are evidence-timing references only. Confidence-only is an ablation only; it is not a headline baseline.

Included experiments

Experiment	Input size	Code
LongBench-v2	all 503 MCQs	repository root
S-NIAH	50 each at 8K/16K/32K/64K/128K	repository root
Five-model S-NIAH replication	250 per model	experiments/sniah_five_model/
Qwen3-14B S-NIAH ablations	250 matched questions	experiments/sniah_ablations/
RULER-HotpotQA	100 questions at five lengths (500 trajectories)	experiments/ruler_hotpotqa/
BrowseComp-Plus	all 830 questions, prepared as one build	experiments/browsecomp_plus/
Kimi LongBench-v2 folding ablations	frozen 80-ID cohort	experiments/longbench_kimi_ablations/

The LongBench ablation scripts reproduce the exact ordered 80-ID cohort frozen in matched80_ids.json. The preparer materializes those IDs directly from the official 503-row export and fails closed if any ID is absent.

BrowseComp-Plus construction

BrowseComp-Plus is prepared as one deterministic 830-question build. Every question receives at least ten documents and retains every mandatory evidence/gold document. A question with more than ten mandatory documents is kept with all of them; no question is split into a separate cohort.

Installation

Use Python 3.10 or newer. The recorded environment used Python 3.13.

python -m venv .venv

PowerShell:

.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r .\requirements.txt

Linux/macOS:

source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For local GPU serving, install a vLLM build compatible with the machine's CUDA stack separately. Hosted runs use the OpenAI-compatible client in requirements.txt.

Validate before inference

python validate_release.py

The validator checks syntax, required files, empty credential fields, the controller invariants, excluded result/data/cache directories, and common absolute-path or secret patterns. Run the included semantic regression tests as well:

python -m unittest -v test_release_semantics.py

Authentication and endpoints

Never put a key in YAML. For OpenRouter:

$env:OPENROUTER_API_KEY = "YOUR_KEY"

export OPENROUTER_API_KEY="YOUR_KEY"

For a local OpenAI-compatible endpoint, set server.base_url, server.model, and server.api_key in a copied config, or use the supported environment overrides MODEL; the endpoint remains explicit in the selected YAML config.

The server must expose generated-token log probabilities. MCQ additionally requires constrained A–D generation with a measured score for every choice. The run aborts if an endpoint omits required probabilities; it never inserts a missing-label fallback.

LongBench-v2: full 503

Prepare once:

python prepare_data.py longbench_v2 --config configs/longbench_full.yaml

Qwen3.5-397B-A17B full trajectories:

python -u run_fold.py \
  --data data/longbench_v2_all.jsonl \
  --out results/longbench_qwen35_full.jsonl \
  --config configs/longbench_full.yaml \
  --workers 8 --rule none --record-verbalized --record-gate

Kimi K2.5 uses the same input/configuration and overrides only the model/output:

MODEL=moonshotai/kimi-k2.5 python -u run_fold.py \
  --data data/longbench_v2_all.jsonl \
  --out results/longbench_kimi_full.jsonl \
  --config configs/longbench_full.yaml \
  --workers 8 --rule none --record-verbalized --record-gate

Rerun the identical command after a transport failure. Successful IDs are resumed rather than repeated. Use a small paid smoke test first by adding --limit 1 --workers 1 and a separate output path.

Replay the frozen controller on all complete trajectories:

python -u analyze.py policies \
  --traj results/longbench_qwen35_full.jsonl \
  --config configs/longbench_full.yaml --verbalized-at 99.5

Set MODEL=moonshotai/kimi-k2.5 when replaying the Kimi trajectories. The stopping parameters remain the same frozen theta=0.995 configuration. The fixed headline controller is never changed by analysis. The paper's same-question exploratory upper-bound sweep is reproduced separately with:

python -u analyze_threshold_sweep.py \
  --traj results/longbench_qwen35_full.jsonl \
  --config configs/longbench_full.yaml \
  --out results/longbench_qwen35_theta_sweep

Appendix timing uses the stated mean-time-per-chunk estimator and reports the total time over the complete benchmark, rather than a per-sample latency:

python -u analyze_timing.py \
  --traj results/longbench_qwen35_full.jsonl \
  --config configs/longbench_full.yaml \
  --out results/longbench_qwen35_timing

S-NIAH

Prepare the pinned 250-row, five-length dataset:

python prepare_data.py sniah --config configs/sniah_full.yaml

Record and label full trajectories:

python -u run_fold.py \
  --data data/sniah.jsonl --out results/sniah_qwen35_full.jsonl \
  --config configs/sniah_full.yaml \
  --workers 8 --rule none --record-verbalized --record-gate
python add_oracle_labels.py results/sniah_qwen35_full.jsonl data/sniah.jsonl
python analyze.py policies --traj results/sniah_qwen35_full.jsonl \
  --config configs/sniah_full.yaml
python analyze.py evidence --traj results/sniah_qwen35_full.jsonl \
  --config configs/sniah_full.yaml

The five-model replication reuses the canonical runtime at the repository root:

cd experiments/sniah_five_model
python -u run_five_models.py --model all --workers 8 --prepare
python -u analyze_five_model_table.py
python -u analyze_parameter_selection.py

--prepare creates the shared pinned 250-question input before running every model. The runner performs the required API preflight for each model, records complete trajectories, resumes failed samples on rerun, and adds the deterministic needle-position labels used by the two analyzers.

Run the Qwen3-14B chunk/notes ablations from experiments/sniah_ablations/:

python -u run_sniah_ablations.py \
  --arms base24_notes6 chunk12 chunk48 notes3 notes12 \
  --workers 8 --prepare
python -u analyze_sniah_ablations.py

This runs the five unique full-trajectory configurations over the same 250 questions. Rerun the identical command to resume transport failures before starting the analyzer.

RULER-HotpotQA

This adapter pins NVIDIA/RULER and builds official qa_2 data. From experiments/ruler_hotpotqa/, generate the official pool and select each disjoint 5x50 cohort without consulting model outputs:

python -m pip install -r ../../requirements.txt
python -m pip install -r requirements_data_generation.txt
python -u generate_official_ruler_data.py \
  --repo-dir official_ruler \
  --output-root official_ruler_qwen3_14b \
  --tokenizer-type hf --tokenizer-path Qwen/Qwen3-14B \
  --lengths 8192 16384 32768 65536 131072 --num-samples 500
python -u prepare_ruler_hotpotqa.py --ruler-root official_ruler_qwen3_14b \
  --lengths 8192 16384 32768 65536 131072 \
  --samples-per-length 50 --seed 42 --selection-offset 0 \
  --out data/ruler_hotpotqa_first.jsonl
python -u prepare_ruler_hotpotqa.py --ruler-root official_ruler_qwen3_14b \
  --lengths 8192 16384 32768 65536 131072 \
  --samples-per-length 50 --seed 42 --selection-offset 50 \
  --out data/ruler_hotpotqa_second.jsonl

Run each model with the canonical root trajectory recorder. Append both cohorts to the same output; successful IDs are resumed automatically:

MODEL=qwen/qwen3-14b python -u ../../run_fold.py \
  --data data/ruler_hotpotqa_first.jsonl \
  --config configs/config_openrouter.yaml \
  --out results/qwen14_full.jsonl --workers 8 \
  --rule none --record-verbalized --record-gate
MODEL=qwen/qwen3-14b python -u ../../run_fold.py \
  --data data/ruler_hotpotqa_second.jsonl \
  --config configs/config_openrouter.yaml \
  --out results/qwen14_full.jsonl --workers 8 \
  --rule none --record-verbalized --record-gate

The other reported model overrides are moonshotai/kimi-k2.5 and qwen/qwen3.5-397b-a17b.

Analyze complete trajectories with the shared defaults and mean adjacent change:

python -u analyze_ruler_hotpotqa.py \
  --traj results/qwen14_full.jsonl \
  --data data/ruler_hotpotqa_first.jsonl \
         data/ruler_hotpotqa_second.jsonl \
  --out-dir results/analysis \
  --theta 0.995 --eps 0.05 --window 3

The RULER threshold and located-answer-proxy tables are separate analyses:

python -u analyze_threshold_sweep.py --traj results/qwen14_full.jsonl \
  --data data/ruler_hotpotqa_first.jsonl data/ruler_hotpotqa_second.jsonl \
  --out results/theta_sweep
python -u analyze_evidence_proxy.py --traj results/qwen14_full.jsonl \
  --data data/ruler_hotpotqa_first.jsonl data/ruler_hotpotqa_second.jsonl \
  --out results/evidence_proxy

BrowseComp-Plus

From experiments/browsecomp_plus/, build the complete benchmark once:

python -u prepare_browsecomp_all830.py

Plan before spending, then run each model:

python -u run_browsecomp_all830.py --arm kimi --plan-only
python -u run_browsecomp_all830.py --arm qwen35 --plan-only
python -u run_browsecomp_all830.py --arm qwen14 --plan-only
python -u run_browsecomp_all830.py --arm kimi \
  --workers 8 --confirm-paid-run
python -u run_browsecomp_all830.py --arm qwen35 \
  --workers 8 --confirm-paid-run
python -u run_browsecomp_all830.py --arm qwen14 \
  --workers 8 --confirm-paid-run
python -u analyze_browsecomp_all830.py --arm kimi --judge-model qwen/qwen3-32b
python -u analyze_browsecomp_all830.py --arm qwen35 --judge-model qwen/qwen3-32b
python -u analyze_browsecomp_all830.py --arm qwen14 --judge-model qwen/qwen3-32b

The judge model is explicit so endpoint incompatibility cannot silently change the evaluation protocol. Use the same validated judge for all three arms. The judge runs at temperature zero. Its cache is keyed by a content hash but stores judge outputs, so it remains under ignored results/ and must never be committed. Decrypted benchmark text must never be committed.

LongBench-v2 Kimi ablations

The ablation code is included because the ablation tables are reported results and must be reproducible. The directory reuses the canonical root runner. From experiments/longbench_kimi_ablations/:

python -u make_matched_80.py --source ../../data/longbench_v2_all.jsonl
python -u run_ablations.py --source-root ../.. --workers 8 --arm all
python -u analyze_ablations.py

These scripts intentionally retain the exact 80-ID execution protocol from the reported ablation. The IDs and their order are frozen in matched80_ids.json; make_matched_80.py materializes only those IDs and never reads model outputs. Do not replace this cohort or relabel a different subset as the reported experiment. Every headline fixed-policy experiment and ablation uses the shared frozen theta=0.995. Rows explicitly labelled ``best theta'' are exploratory same-question upper bounds produced by the sweep scripts and do not replace the fixed controller. By default every arm, including the 24K/6K baseline, is rerun. Reuse an external compatible baseline only with the explicit --reuse-existing-base PATH option; the runner validates its IDs, model, chunking, and recorded comparator signals before accepting it.

Outputs and accounting

Trajectory JSONL is append-only and resumable. Each step stores the notes, answer state, confidence, divergence inputs, split token counters, and latency. Logical policy cost charges only the fold/probe/gate calls consumed through the policy's stopping step. Physical recording cost includes every signal requested during full-trajectory collection.

Expected generated files are ignored by version control:

• data/: locally prepared benchmark text;
• results/: trajectories, tables, and judge cache;
• checkpoints/: per-chunk recovery state;
• hf_cache/, vendor/, and local virtual environments.

Source map

• common.py: API transport, retry policy, scoring, normalization, and I/O.
• draft_cleaner.py: shared open-draft normalization and conservative abstention detection.
• run_fold.py: folding, rolling-tail notes, probes, and online controller.
• analyze.py: offline policy replay and evidence-relative analysis.
• benchmark_data/: LongBench-v2, S-NIAH, and benchmark builders.
• configs/: frozen full-trajectory profiles.
• experiments/: benchmark-specific adapters and ablations. These import the canonical root runtime; byte-identical copies are intentionally excluded.

Reproducibility rules

1. Do not tune on the reported test trajectories unless the row is explicitly labelled post-hoc.
2. Do not compare policies from different sample-ID sets.
3. Do not treat a truncated trajectory as full context.
4. Do not replace missing logits or token log probabilities with constants.
5. Do not charge one policy for another policy's probe calls.
6. Record the model identifier, endpoint, build ID, and configuration with every result.