Winnow

Code for the paper Tool-Creating LLM Agents Gain Little from Keeping Their Tools, by Marek Šuppa and Jaroslav Kopčan, COLM 2026.

Tool-creating LLM agents are often reported to gain from keeping a library of the tools they write. Winnow separates the parts of that pipeline: writing a tool for the current task, keeping it, and retrieving it for later tasks. It runs each part as its own condition on BigCodeBench-Hard and tau2-airline, so that the value of keeping tools can be measured on its own.

Repository layout

src/winnow/   conditions, tool memory and retrieval, Inspect AI tasks and runner,
              log readers and report builders, CLI
scripts/      experiment runners, summaries, statistics, paper tables and figures
docs/results/ aggregated results of the runs reported in the paper
docker/       BigCodeBench sandbox image (adds libsndfile)
tests/        unit tests

Conditions

Condition (--condition)	Paper name	Creates tools	Keeps tools	Retrieves
direct	Direct
cot_extended	CoT-Extended
structured_cot	Structured-CoT
create_discard_full	Create+Discard	✓
create_invoke_once	Create+Invoke_once	✓
create_use	Create+Reuse	✓	✓	✓
create_admit	Create+Admit	✓	✓ (admission-gated)	✓
trajectory_retrieval	Trajectory Retrieval		✓ (trajectories)	✓
create_oracle_use	Create+Oracle	✓	✓	✓ (oracle)
direct_active	Direct-Active

Setup

Requires Python 3.13, uv and Docker (the BigCodeBench-Hard sandbox).

uv sync --dev
uv run pytest
cp .env.example .env    # API keys, see the comments in the file

The paper's runs used Inspect AI 0.3.200 and reached the models through an OpenAI-compatible endpoint. To do the same, set WINNOW_MODEL_BASE_URL and WINNOW_MODEL_API_KEY and add these options to every run command:

--model-base-url "$WINNOW_MODEL_BASE_URL" --model-arg-env api_key=WINNOW_MODEL_API_KEY

The models in the paper are anthropic.claude-sonnet-4-5-20250929-v1:0, gpt-5-mini, gemini-2.5-pro and o4-mini-global on that endpoint, plus GPT-5.4 and Opus 4.5 (through OpenRouter, --model openrouter/...).

Running the experiments

winnow inspect preview prints a run plan without calling a model; winnow inspect run runs it. winnow inspect status <log-dir> reports the progress of a running job.

Experiment 1: BigCodeBench-Hard (148 tasks). One run per model and condition:

for c in direct cot_extended structured_cot create_discard_full \
         create_invoke_once create_use trajectory_retrieval create_admit; do
  uv run winnow inspect run --benchmark bigcodebench-hard --condition $c \
      --model openai/$MODEL --log-dir runs/exp1/$MODEL/$c
done

Experiment 2: tau2-airline (50 tasks), with temperature 0 and at most 30 messages per conversation:

uv run python scripts/run_tau2_airline.py --model openai/$MODEL \
    --conditions direct create_discard_full create_use create_admit \
    --message-limit 30 --log-dir runs/tau2/$MODEL
# Wrapper-matched controls (Sonnet): Direct and Structured-CoT through the
# same wrapper as the tool-creating conditions, and Direct-Active
uv run python scripts/run_tau2_airline.py --model openai/$MODEL --force-wrapper \
    --conditions direct structured_cot direct_active \
    --message-limit 30 --log-dir runs/tau2-wrapper/$MODEL

Retrieval variants (BigCodeBench-Hard and tau2-airline):

--condition create_oracle_use                                    # oracle retrieval
--condition create_use --retrieval-mode embedding                # text-embedding-3-small
--condition create_use --retrieval-mode embedding \
    --retrieval-min-score 0.55                                   # selective retrieval

Dependency-aware retrieval with a warm-started library. Export the tool library of a finished Create+Reuse run, then run the frozen-library conditions:

uv run python scripts/export_tool_library.py --log-dir runs/exp1/$MODEL/create_use \
    --model openai/$MODEL
uv run python scripts/run_dependency_experiments.py --model openai/$MODEL

Library management and admission (BigCodeBench-Hard, code "Experiment 3"):

uv run python scripts/run_exp3_admission_batch.py --model openai/$MODEL \
    --distances 0.50 0.90 --log-root runs/exp3-admission
uv run winnow inspect run --benchmark bigcodebench-hard --condition create_use \
    --library-management fifo --library-capacity 10 \
    --model openai/$MODEL --log-dir runs/exp3-management/cap10

From logs to the paper

The scripts read Inspect log directories (logs.json and .eval bundles); see --help for their arguments. compute_bootstrap_cis.py and generate_paper_figures.py take no arguments: the per-condition solve counts they use are written in the scripts. calibrate_retrieval_threshold.py and compare_retrievers.py read run directories named in the scripts.

Paper element	Script
Exp. 1 tables and macros	export_exp1_paper_tables.py, summarize_exp1_matrix.py, sync_paper_artifacts.py
Confidence intervals and paired tests	compute_bootstrap_cis.py
Concordance across conditions	compute_concordance.py
Compute usage	summarize_compute_usage.py, summarize_inspect_run.py
tau2-airline tables, per-task results, paired counts	summarize_tau2_airline.py, compute_tau2_error_analysis.py
tau2-airline engagement metrics	analyze_tau2_agent_traces.py
Library management and admission tables	export_exp3_paper_artifacts.py, summarize_admission_sweep.py, summarize_management_sweep.py
Condition flips	analyze_condition_flips.py, analyze_cross_model_traces.py
Retrieval precision	analyze_retrieval_precision.py
Retrieval magnets	analyze_create_use_retrieval.py
Tool taxonomy	analyze_tool_taxonomy.py
Retrieval threshold calibration	calibrate_retrieval_threshold.py, compare_retrievers.py
Figures	generate_paper_figures.py

Paper outputs are written to paper/generated/ and paper/figures/. docs/results/ holds the aggregated numbers of the reported runs: the BigCodeBench-Hard concordance, three-run replication and oracle runs, the embedding and dependency-aware retrieval runs, the tau2-airline runs (cross-model, clean Sonnet reruns, wrapper-matched controls and Direct-Active), and the bootstrap intervals.

Citation

@inproceedings{suppa2026toolcreating,
  author    = {Marek {\v{S}}uppa and Jaroslav Kop{\v{c}}an},
  title     = {Tool-Creating {LLM} Agents Gain Little from Keeping Their Tools},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}