This directory is the code behind the paper's results: a harness that runs Claude Code, Codex and the Copilot CLI on SWE-Bench and Web-Bench, the launchers, and the aggregation scripts. It is not part of the plugin; the method is described in the main README.
| Path | Contents |
|---|---|
te_harness/ |
runs an agent CLI in each SWE-Bench task's Docker image (on the host for Web-Bench) and grades it with the benchmark's own grader |
te_bench/ |
the experiment arms, the benchmark adapters, and the evaluated task sets (workloads.py) |
scripts/ |
launch/ (the launchers), run_experiment.py (one arm on one task stream), aggregate.py and compare_arms.py |
workloads/ |
the evaluated task ids: swebench.json (277 tasks), webbench.json (102), and the security sets |
tests/ |
offline tests: no Docker, data or API keys |
- Linux x86_64 with Docker usable without
sudo, and about 200 GB of disk for the SWE-Bench images. - Python 3.10 or newer, git, and Node.js with npm (for Web-Bench).
- An Anthropic API key and an OpenAI API key. Claude Code and the Copilot CLI bill the first, Codex the second.
- The agent CLIs on the host, at the versions the paper used. The harness mounts them into each task container.
curl -fsSL https://claude.ai/install.sh | bash -s 2.1.170 # Claude Code
npm install -g @openai/codex@0.153.1 # Codex
curl -fsSL https://gh.io/copilot-install | VERSION=1.0.84-7 bash # Copilot CLI1. Install and check the code. Every command after cd experiments runs from this directory.
git clone https://github.com/microsoft/TokenEconomizer.git && cd TokenEconomizer
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[test,baselines]' # baselines: aider, for the RepoMap arms
cd experiments
python -m pytest tests -q
bash scripts/launch/smoke_test.sh # every arm wired end to end, with a stub agent2. SWE-Bench. Pick a data directory outside the repository, and use its canonical path: Web-Bench fails every test when reached through a symlink.
export DATA="$(readlink -f ~/te_data)"
git clone https://github.com/SWE-bench/SWE-bench.git "$DATA/SWE-bench"
pip install -e "$DATA/SWE-bench" # the official grader, into this venv
python scripts/export_swebench_tasks.py --out-dir "$DATA/swebench"
docker login # anonymous pulls are rate-limited
xargs -a "$DATA/swebench/image_tags.txt" -P 4 -n 1 docker pull -qThe export writes the task list, the grading file and the list of the 277 task images.
3. Web-Bench. The five evaluated projects, built, with the Playwright browser they pin:
git clone https://github.com/bytedance/web-bench.git "$DATA/web-bench"
( cd "$DATA/web-bench" && npm install -g @microsoft/rush && rush install \
&& rush build --to @web-bench/eval --to @web-bench/calculator --to @web-bench/expressjs \
--to @web-bench/react --to @web-bench/zustand --to @web-bench/table \
&& cd projects/calculator && npx playwright install chromium chromium-headless-shell )
export TOKENECONOMIZER_WEBBENCH_PROJECTS="$DATA/web-bench/projects"
export TOKENECONOMIZER_WEBBENCH_TASKS="$DATA/webbench_tasks.json"
python scripts/webbench_tasks_from_yaml.py --all # zustand ships only tasks.yml
python scripts/export_webbench_tasks.py4. Keys and paths. cp .env.example .env, then fill it in with absolute paths (the file
does not expand variables):
| Variable | Value |
|---|---|
ANTHROPIC_API_KEY, OPENAI_API_KEY |
your keys |
COPILOT_PROVIDER_API_KEY |
your Anthropic key again: the Copilot CLI bills Anthropic through its own provider settings, already set in the file |
DATA |
the data directory |
TOKENECONOMIZER_RESULTS |
where results go, e.g. $DATA/results |
TOKENECONOMIZER_SWEBENCH_TASKS |
$DATA/swebench/swebench_tasks.json |
TE_SWEBENCH_REPO |
$DATA/SWE-bench |
TOKENECONOMIZER_WEBBENCH_PROJECTS, TOKENECONOMIZER_WEBBENCH_TASKS |
as exported in step 3 |
5. One paid task. Before a full run, check that one task produces a graded row:
AGENT=claude MODEL=claude-sonnet-5 DOMAIN=swebench ARMS=baseline REPOS=sympy/sympy SEEDS=0 \
LIMIT=1 DETACH=0 RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.shIts row.json, under $TOKENECONOMIZER_RESULTS/claude_swebench_sonnet5/baseline/sympy_s0/,
should show num_turns above zero, token counts, and a resolved value.
main_sweep.sh runs Claude Code and the Copilot CLI, codex_ab.sh runs Codex. Each command
below is one lane, one agent and model on one benchmark: every arm it names, on every
repository or project, with seeds 0, 1 and 2. A lane runs in the background and prints where
its log is and how to stop it, so the commands in a loop run in parallel. Run a command again
to resume a lane: cells already finished are skipped. DRYRUN=1 prints a lane's plan without
running it, and DETACH=0 keeps it in the foreground.
The arms, by the names the paper uses:
| Arm | In the paper |
|---|---|
baseline, baseline-medium, baseline-low |
default execution at high, medium and low effort |
rtk, repomap |
the fixed strategies RTK and RepoMap |
trimech (codex-trimech on Codex) |
TokenEconomizer |
trimech-rtk, trimech-repomap (codex-trimech-…) |
TokenEconomizer+RTK, TokenEconomizer+RepoMap |
inject-only, untool-only, spiral-only |
micro-skills, tool pruning and the failure-loop guard alone |
trimech-web |
TokenEconomizer on Web-Bench, whose streams are short |
SWE-Bench: Table 1, Figures 1(b) and 5, and the ablation in Table 2(a).
SWE="baseline baseline-medium baseline-low rtk repomap trimech trimech-rtk trimech-repomap"
for MODEL in claude-sonnet-5 claude-opus-4-8; do
AGENT=claude MODEL=$MODEL DOMAIN=swebench ARMS="$SWE inject-only untool-only spiral-only" \
RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
done
AGENT=copilot MODEL=claude-sonnet-5 DOMAIN=swebench ARMS="$SWE" \
RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
for MODEL in gpt-5.6-terra gpt-5.6-sol; do
MODEL=$MODEL DOMAIN=swebench \
ARMS="baseline baseline-medium baseline-low rtk repomap codex-trimech codex-trimech-rtk codex-trimech-repomap" \
RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
doneWeb-Bench: Table 2(b). The same agent–model settings as on SWE-Bench.
for MODEL in claude-sonnet-5 claude-opus-4-8; do
AGENT=claude MODEL=$MODEL DOMAIN=webbench ARMS="baseline trimech-web" \
RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
done
AGENT=copilot MODEL=claude-sonnet-5 DOMAIN=webbench ARMS="baseline trimech-web" \
RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
for MODEL in gpt-5.6-terra gpt-5.6-sol; do
MODEL=$MODEL DOMAIN=webbench ARMS="baseline codex-trimech" \
RUNNER=te_harness.webbench:make_runner bash scripts/launch/codex_ab.sh
doneSecurity benchmarks. The CredData and ReposVul task sets (workloads/creddata.json,
workloads/reposvul.json) and their adapters are included, but this harness does not run or
grade them.
Every arm, the baseline included, runs behind an answer-key guard that blocks the benchmarks'
public answers on GitHub. The rtk binary is downloaded into $DATA/scaffolds.
Each lane writes one results tree, $TOKENECONOMIZER_RESULTS/<agent>_<benchmark>_<model>/
(for example claude_swebench_sonnet5), with one directory per arm and cell
(<arm>/<repository>_s<seed>/) and a run directory per task holding row.json,
transcript.jsonl and run_config.json. At the end of a lane, aggregate.py writes
index/all_rows.csv, one row per task run, and index/summary_by_cell.csv, one row per arm,
repository and seed; run python scripts/aggregate.py <tree> to refresh it. A task run again
keeps both runs, and the index takes the newest.
python scripts/compare_arms.py --baseline <tree>/baseline/<cell> --arm <tree>/<arm>/<cell>compare_arms.py pairs one cell's tasks by instance and reports cost and resolved tasks
against the baseline. Cost is in cache-aware effective tokens and includes the arm's own model
calls. Web-Bench rows carry resolved_p2, the Pass@2 the paper reports. Never pool two trees:
the agent and the model are both part of the cost basis.
| Symptom | Cause |
|---|---|
a launch prints RUNNER=stub -- SYNTHETIC run |
RUNNER was not set; the stub runs no agent |
ModuleNotFoundError at grading |
the SWE-Bench grader is not installed in the venv the launcher uses (step 2) |
| every Web-Bench test fails and the app never renders | a non-canonical data path; readlink -f it (step 2) |
Executable doesn't exist at …/chromium_headless_shell-… |
Playwright was installed outside a project, so its browser build differs from the one the projects pin (step 3) |
| Codex gets 401 with a valid key | a key issued for a regional endpoint; set OPENAI_BASE_URL in .env |
| a lane needs stopping | kill -TERM -$(cat <log>.pid), as printed at launch; the leading - stops the whole process group |