Skip to content

Latest commit

 

History

History

experiments

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Reproducing the experiments

This directory is the code behind the paper's results: a harness that runs Claude Code, Codex and the Copilot CLI on SWE-Bench and Web-Bench, the launchers, and the aggregation scripts. It is not part of the plugin; the method is described in the main README.

Path Contents
te_harness/ runs an agent CLI in each SWE-Bench task's Docker image (on the host for Web-Bench) and grades it with the benchmark's own grader
te_bench/ the experiment arms, the benchmark adapters, and the evaluated task sets (workloads.py)
scripts/ launch/ (the launchers), run_experiment.py (one arm on one task stream), aggregate.py and compare_arms.py
workloads/ the evaluated task ids: swebench.json (277 tasks), webbench.json (102), and the security sets
tests/ offline tests: no Docker, data or API keys

Requirements

  • Linux x86_64 with Docker usable without sudo, and about 200 GB of disk for the SWE-Bench images.
  • Python 3.10 or newer, git, and Node.js with npm (for Web-Bench).
  • An Anthropic API key and an OpenAI API key. Claude Code and the Copilot CLI bill the first, Codex the second.
  • The agent CLIs on the host, at the versions the paper used. The harness mounts them into each task container.
curl -fsSL https://claude.ai/install.sh | bash -s 2.1.170          # Claude Code
npm install -g @openai/codex@0.153.1                                 # Codex
curl -fsSL https://gh.io/copilot-install | VERSION=1.0.84-7 bash     # Copilot CLI

Setup

1. Install and check the code. Every command after cd experiments runs from this directory.

git clone https://github.com/microsoft/TokenEconomizer.git && cd TokenEconomizer
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[test,baselines]'        # baselines: aider, for the RepoMap arms
cd experiments
python -m pytest tests -q
bash scripts/launch/smoke_test.sh         # every arm wired end to end, with a stub agent

2. SWE-Bench. Pick a data directory outside the repository, and use its canonical path: Web-Bench fails every test when reached through a symlink.

export DATA="$(readlink -f ~/te_data)"
git clone https://github.com/SWE-bench/SWE-bench.git "$DATA/SWE-bench"
pip install -e "$DATA/SWE-bench"          # the official grader, into this venv
python scripts/export_swebench_tasks.py --out-dir "$DATA/swebench"
docker login                              # anonymous pulls are rate-limited
xargs -a "$DATA/swebench/image_tags.txt" -P 4 -n 1 docker pull -q

The export writes the task list, the grading file and the list of the 277 task images.

3. Web-Bench. The five evaluated projects, built, with the Playwright browser they pin:

git clone https://github.com/bytedance/web-bench.git "$DATA/web-bench"
( cd "$DATA/web-bench" && npm install -g @microsoft/rush && rush install \
  && rush build --to @web-bench/eval --to @web-bench/calculator --to @web-bench/expressjs \
                --to @web-bench/react --to @web-bench/zustand --to @web-bench/table \
  && cd projects/calculator && npx playwright install chromium chromium-headless-shell )
export TOKENECONOMIZER_WEBBENCH_PROJECTS="$DATA/web-bench/projects"
export TOKENECONOMIZER_WEBBENCH_TASKS="$DATA/webbench_tasks.json"
python scripts/webbench_tasks_from_yaml.py --all    # zustand ships only tasks.yml
python scripts/export_webbench_tasks.py

4. Keys and paths. cp .env.example .env, then fill it in with absolute paths (the file does not expand variables):

Variable Value
ANTHROPIC_API_KEY, OPENAI_API_KEY your keys
COPILOT_PROVIDER_API_KEY your Anthropic key again: the Copilot CLI bills Anthropic through its own provider settings, already set in the file
DATA the data directory
TOKENECONOMIZER_RESULTS where results go, e.g. $DATA/results
TOKENECONOMIZER_SWEBENCH_TASKS $DATA/swebench/swebench_tasks.json
TE_SWEBENCH_REPO $DATA/SWE-bench
TOKENECONOMIZER_WEBBENCH_PROJECTS, TOKENECONOMIZER_WEBBENCH_TASKS as exported in step 3

5. One paid task. Before a full run, check that one task produces a graded row:

AGENT=claude MODEL=claude-sonnet-5 DOMAIN=swebench ARMS=baseline REPOS=sympy/sympy SEEDS=0 \
  LIMIT=1 DETACH=0 RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh

Its row.json, under $TOKENECONOMIZER_RESULTS/claude_swebench_sonnet5/baseline/sympy_s0/, should show num_turns above zero, token counts, and a resolved value.

Running the paper's experiments

main_sweep.sh runs Claude Code and the Copilot CLI, codex_ab.sh runs Codex. Each command below is one lane, one agent and model on one benchmark: every arm it names, on every repository or project, with seeds 0, 1 and 2. A lane runs in the background and prints where its log is and how to stop it, so the commands in a loop run in parallel. Run a command again to resume a lane: cells already finished are skipped. DRYRUN=1 prints a lane's plan without running it, and DETACH=0 keeps it in the foreground.

The arms, by the names the paper uses:

Arm In the paper
baseline, baseline-medium, baseline-low default execution at high, medium and low effort
rtk, repomap the fixed strategies RTK and RepoMap
trimech (codex-trimech on Codex) TokenEconomizer
trimech-rtk, trimech-repomap (codex-trimech-…) TokenEconomizer+RTK, TokenEconomizer+RepoMap
inject-only, untool-only, spiral-only micro-skills, tool pruning and the failure-loop guard alone
trimech-web TokenEconomizer on Web-Bench, whose streams are short

SWE-Bench: Table 1, Figures 1(b) and 5, and the ablation in Table 2(a).

SWE="baseline baseline-medium baseline-low rtk repomap trimech trimech-rtk trimech-repomap"
for MODEL in claude-sonnet-5 claude-opus-4-8; do
  AGENT=claude MODEL=$MODEL DOMAIN=swebench ARMS="$SWE inject-only untool-only spiral-only" \
    RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
done
AGENT=copilot MODEL=claude-sonnet-5 DOMAIN=swebench ARMS="$SWE" \
  RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
for MODEL in gpt-5.6-terra gpt-5.6-sol; do
  MODEL=$MODEL DOMAIN=swebench \
    ARMS="baseline baseline-medium baseline-low rtk repomap codex-trimech codex-trimech-rtk codex-trimech-repomap" \
    RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
done

Web-Bench: Table 2(b). The same agent–model settings as on SWE-Bench.

for MODEL in claude-sonnet-5 claude-opus-4-8; do
  AGENT=claude MODEL=$MODEL DOMAIN=webbench ARMS="baseline trimech-web" \
    RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
done
AGENT=copilot MODEL=claude-sonnet-5 DOMAIN=webbench ARMS="baseline trimech-web" \
  RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
for MODEL in gpt-5.6-terra gpt-5.6-sol; do
  MODEL=$MODEL DOMAIN=webbench ARMS="baseline codex-trimech" \
    RUNNER=te_harness.webbench:make_runner bash scripts/launch/codex_ab.sh
done

Security benchmarks. The CredData and ReposVul task sets (workloads/creddata.json, workloads/reposvul.json) and their adapters are included, but this harness does not run or grade them.

Every arm, the baseline included, runs behind an answer-key guard that blocks the benchmarks' public answers on GitHub. The rtk binary is downloaded into $DATA/scaffolds.

Results

Each lane writes one results tree, $TOKENECONOMIZER_RESULTS/<agent>_<benchmark>_<model>/ (for example claude_swebench_sonnet5), with one directory per arm and cell (<arm>/<repository>_s<seed>/) and a run directory per task holding row.json, transcript.jsonl and run_config.json. At the end of a lane, aggregate.py writes index/all_rows.csv, one row per task run, and index/summary_by_cell.csv, one row per arm, repository and seed; run python scripts/aggregate.py <tree> to refresh it. A task run again keeps both runs, and the index takes the newest.

python scripts/compare_arms.py --baseline <tree>/baseline/<cell> --arm <tree>/<arm>/<cell>

compare_arms.py pairs one cell's tasks by instance and reports cost and resolved tasks against the baseline. Cost is in cache-aware effective tokens and includes the arm's own model calls. Web-Bench rows carry resolved_p2, the Pass@2 the paper reports. Never pool two trees: the agent and the model are both part of the cost basis.

Troubleshooting

Symptom Cause
a launch prints RUNNER=stub -- SYNTHETIC run RUNNER was not set; the stub runs no agent
ModuleNotFoundError at grading the SWE-Bench grader is not installed in the venv the launcher uses (step 2)
every Web-Bench test fails and the app never renders a non-canonical data path; readlink -f it (step 2)
Executable doesn't exist at …/chromium_headless_shell-… Playwright was installed outside a project, so its browser build differs from the one the projects pin (step 3)
Codex gets 401 with a valid key a key issued for a regional endpoint; set OPENAI_BASE_URL in .env
a lane needs stopping kill -TERM -$(cat <log>.pid), as printed at launch; the leading - stops the whole process group