Skip to content

0.1.2 (13/15): publish benchmarks and action measurements - #14

Merged
trevoedwards merged 2 commits into
mainfrom
pr/0.1.2-13-benchmarks
Nov 1, 2026
Merged

trevoedwards merged 2 commits into
mainfrom
pr/0.1.2-13-benchmarks

Conversation

@trevoedwards

Copy link
Copy Markdown
Collaborator

Part 13 of the 0.1.2 stack. Depends on part 12; base: pr/0.1.2-12-script-fixes. Merge the preceding PR first. Existing topic commits and AI co-author trailers are preserved; already-merged CI fixes are carried through the stack.

Adds a benchmarks page linked from the README and changelog, with the 31-task foreground-agent evaluation and installed-agent action measurements.

  • Compares the measured 0.1.1 and 0.1.2 runs: both passed 31/31, with tool-call, screenshot, input-token, time and cost totals plus per-task results.
  • Documents the test Mac, model, GUI-only tooling, external result checks and single-run/noise limits. It does not attribute the gains to one change in isolation.
  • Includes median action times and Accessibility-walk counts from the 30-round measurement harness, with reproduction commands.
  • Removes two accidentally included, truncated raw debug JSON lines from the original page; the tables and measured values are unchanged.

This is documentation only. The published numbers refer to the recorded foreground evaluation at the script-fix stage, before the later background-mode addition.

PR prepared with Codex. Original implementation co-author trailers are retained.

@trevoedwards
trevoedwards added this pull request to stack #17 November 1, 2026 03:53
@trevoedwards
trevoedwards force-pushed the pr/0.1.2-13-benchmarks branch 3 times, most recently from bfc9ac6 to 2f544a3 Compare November 1, 2026 03:54
Base automatically changed from pr/0.1.2-12-script-fixes to main November 1, 2026 03:54
trevoedwards and others added 2 commits October 31, 2026 20:54
….1, and action timings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the published result tables and measurement descriptions; drop two partial raw JSON lines.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@trevoedwards
trevoedwards force-pushed the pr/0.1.2-13-benchmarks branch from 2f544a3 to b84952d Compare November 1, 2026 03:55
@trevoedwards
trevoedwards merged commit 2040141 into main Nov 1, 2026
1 check passed
@trevoedwards
trevoedwards deleted the pr/0.1.2-13-benchmarks branch November 1, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant