diff --git a/CHANGELOG.md b/CHANGELOG.md index d73c9df..d163be0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,15 @@ All notable changes to this project are documented here. Detailed notes live in [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [0.2.1] - unreleased + +### Changed +- TDD proofs are bound to a tree snapshot. `I6b` recomputes the binding and re-runs the recorded command. Green also checks that snapshotted test files still match disk. +- Red and green must run the same test, or green must be a superset. Skip, todo, and focused tests do not count. Import-only or empty failures do not count as red. Type-only assertions fail when they are the only assertions. +- `J4b` fails when the same subject has contradictory `MUST`/`SHALL` verb/value clauses. +- `self.assert*`, `pytest.raises`, and bare `assert` count as assertions. +- `tdd_mode` `refactor` and `docs` (status.json or `--tdd-mode`) are the exemption path for an honest refactor or a docs-only diff. Misuse fails. + ## [0.2.0] - unreleased (relaunch) ### Added diff --git a/README.md b/README.md index 0168f24..6dcd8c8 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,40 @@ **Executable delivery workflow for AI coding agents.** +## Benchmarks: OpenFlow vs superpowers (2026-10-17) + +Same model (`grok-4.7-high` via local `cursor-agent`) and same harness for every arm. Pinned: OpenFlow 0.2.1, +[superpowers](https://github.com/obra/superpowers) v7.0.0 (`bb92a77`). Scripts, raw results and methodology: +[`bench/`](bench/README.md) and [`bench/e2e/`](bench/e2e/README.md). Losses are shown as measured. + +**End-to-end tasks.** 8 small real tasks (4 JS, 4 Python), hidden acceptance tests applied only after the agent exits, 20-minute cap per run. + +| Arm | Success (hidden tests) | Median wall time (IQR) | Median tokens: total / input+output | n | +| --- | --- | --- | --- | --- | +| Plain agent (baseline) | 24/24 (100%) | 55 s (46–132) | 105k / 28k | 24 | +| superpowers v7.0.0 | 24/24 (100%) | 187 s (121–299) | 312k / 59k | 24 | +| OpenFlow 0.2.1 | 11/12 (92%) | 985 s (920–1195) | 5.29M / 233k ¹ | 12 ² | + +On tasks this small the plain agent already succeeds every time, and OpenFlow's full ceremony is the slowest and most +token-hungry arm. 3 of 12 OpenFlow runs hit the 20-minute cap while still running the workflow; their code already passed. +¹ Tokens counted only for runs that reported usage (not the 3 capped runs). Most of "total" is cache reads. +² 3 repeats on 2 tasks, 1 on the other 6, to save quota. Matched repeat-1 view (n=8 each): baseline 8/8, 57 s, 104k; superpowers 8/8, 154 s, 289k; OpenFlow 8/8, 1008 s, 5.32M. + +**Catching fake TDD / verification evidence.** 36 scripted episodes: 25 faulty (red proof recorded after the fix, forged +proofs, skipped tests, weak or wrong assertions, contradictory specs, …) and 11 honest ones. Of these, 12 are a held-out set written after the fixes. + +| Checker | Faulty caught | Honest wrongly rejected | Cost per check | +| --- | --- | --- | --- | +| OpenFlow 0.2.1 gates (deterministic code) | 22/25 (88%): original 16/17, held-out 6/8 | 0/11 | ~0.1–0.2 s, no model tokens | +| superpowers v7.0.0 skills + LLM judge | 25/25 (100%) | 1/11 | one agent call per check | +| LLM judge, no skills | 25/25 (100%) ³ | 0/11 | one agent call per check | +| OpenFlow 0.2.0 (before this benchmark, original 24 only) | 10/17 (59%) | 3/7 | – | + +An LLM reviewer that is explicitly asked to audit an episode, and is allowed to rerun the tests, caught more than OpenFlow's static gates did. +OpenFlow's gates are deterministic and cost no model calls, but they still miss plausible-but-wrong expected values, +assertions in callbacks that never run, and behavior changes hidden inside a declared refactor. +³ 1 of the 11 honest runs timed out and counts as incorrect. LLM arms used 1 run per scenario. + Prompt-based workflows *ask* an agent to write the failing test first, run the suite and keep notes. OpenFlow *checks* that it did. Workflows are YAML phases made of small deterministic "atoms". Gates are code: a red/green TDD proof, a test-quality review, artifact checks, and archive ordering. The `xflow` CLI runs the @@ -25,12 +59,15 @@ finished. OpenFlow adds a harder layer underneath: - **Workflow as data.** `workflows/yolo.yaml` and `workflows/corps.yaml` are validated against an atom registry before they run. - **TDD proof checked by code.** - - The red proof must fail before implementation and the green proof must pass after it. + - `I6a` snapshots the tree and binds the command, exit code, output hash, and failure class. `I6b` recomputes that binding and re-runs the command on the snapshot. A hand-edited proof does not pass unless that snapshot really exits as claimed. + - Red and green must target the same test, or green must be a superset of the red test. Skipped, todo, and focused tests do not count as red or green. - `I6c.tdd.quality_review` rejects code changes that have: - no meaningful test changes, - empty assertions, - snapshot-only tests, - - mock-heavy tests without real assertions. + - mock-heavy tests without real assertions, + - only a type or existence assertion (`typeof`, `!== undefined`). + - `tdd_mode` in `status.json` (or `--tdd-mode`) may be `red-green` (default), `refactor`, or `docs`. Refactor requires the existing tests to pass before and after, with no assertion removed. Docs allows a non-code diff and does not require a red proof. Declaring docs while changing code, or refactor while deleting assertions, fails. - **Context that survives sessions.** - The `.xflow/GOAL.md` project anchor. - `HANDOFF.md` and `AHA.md`. diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 9d3fdbf..f2151b5 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,5 +1,13 @@ # Release Notes +## 0.2.1 - Tighter TDD and spec gates + +Proof files are no longer trusted as JSON alone. `I6a` records a snapshot and a digest over the command, exit code, output hash, failure class, and file hashes. `I6b` recomputes that digest and re-runs the command on the snapshot. An agent who can rewrite the tree can still produce a self-consistent failing snapshot; a hand edit of the proof cannot. + +`I6c` requires red and green to target the same test (or a green superset), rejects skip/todo/only on the changed tests, and rejects a type-only assertion when it is the only assertion. A plausible but wrong expected value is still not decidable statically. `J4b` rejects contradictory `MUST`/`SHALL` values for the same subject. + +`tdd_mode` may be `red-green` (default), `refactor`, or `docs`, via `status.json` or `--tdd-mode`. Refactor checks that existing tests pass before and after and that assertions were not removed. Docs checks that the diff stays off code and tests. + ## 0.2.0 - Relaunch (first npm release) OpenFlow is published to npm as **`@surlymochan/openflow`**. The CLI is still `xflow`. The unscoped diff --git a/atoms/registry.json b/atoms/registry.json index 4519810..b28b6c0 100644 --- a/atoms/registry.json +++ b/atoms/registry.json @@ -205,7 +205,7 @@ "track": "lite", "type": "python", "script": "xflow/atoms/i6a_tdd_run.py", - "description": "Run test command, capture raw result, and enforce phase expectation for TDD proof", + "description": "Run test command, snapshot the tree, and bind the exit code, output hash, and failure class into a TDD proof", "input_schema": { "project_root": "string", "change_id": "string", @@ -218,22 +218,24 @@ "track": "lite", "type": "python", "script": "xflow/atoms/i6b_tdd_proof_validate.py", - "description": "Validate tdd proof schema and semantics: red fails, green/refactor passes", + "description": "Recompute the TDD proof binding, re-run the command on its snapshot, and check red/green semantics. refactor and docs modes can omit a red proof.", "input_schema": { "project_root": "string", "change_id": "string", - "phase": "string?" + "phase": "string?", + "tdd_mode": "red-green|refactor|docs?" } }, "I6c.tdd.quality_review": { "track": "lite", "type": "python", "script": "xflow/atoms/i6c_tdd_quality_review.py", - "description": "Review TDD changed-test quality and reject code changes without meaningful test changes", + "description": "Review changed-test quality, red/green command identity, skip/todo markers, and weak assertions. tdd_mode refactor or docs replaces the red-proof requirement.", "input_schema": { "project_root": "string", "change_id": "string", - "diff_ref": "string?" + "diff_ref": "string?", + "tdd_mode": "red-green|refactor|docs?" } }, "J4a.spec_consistency.structural": { diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 0000000..229eba4 --- /dev/null +++ b/bench/README.md @@ -0,0 +1,179 @@ +# Head-to-head: OpenFlow and superpowers + +This benchmark compares two ways of pushing back on the same failure: an AI coding agent claims it did test-driven development or verification, and it did not. + +OpenFlow checks that claim with deterministic atoms (`I6a` records a command, `I6b` checks the proof file, `I6c` reviews changed tests, `J1` reruns the planned command, `J4a` / `J4b` review spec structure, `E6` checks archive readiness). superpowers v7.0.0 checks it with skill text the agent is supposed to read (`skills/test-driven-development/SKILL.md` and `skills/verification-before-completion/SKILL.md`). The harness gives both arms the same fixtures and the same agent claim. Only the enforcement mechanism differs. + +The labels below were written from the rubric in this file before any arm output was used. They are not revised to match a tool's score. + +## Fairness rules + +- Every scenario is a tiny project plus the evidence an agent would leave behind. For OpenFlow that evidence is produced by running `I6a.tdd.run` from this checkout, then applying only the tamper the scenario describes. The judge step does not run `I6a` again, because `I6a` overwrites the proof instead of auditing it. +- The OpenFlow arm then runs, in order: `I6b` red, `I6b` green, `I6c`, `J1`, `J4a`, `J4b`. `E6` `pre-archive` runs only when the claim includes opening a PR. A non-zero exit from any of those gates is `REJECT`. Otherwise the arm is `ACCEPT`. +- `J4b` warnings do not change the verdict. `J4b` fails when proposal or plan is empty, and when the same subject has contradictory `MUST`/`SHALL` ` ` clauses across the proposal, the plan, and the spec delta. +- The same `CLAIM.md`, git diff, git status, and presented command log are written into the fixture for every arm. The OpenFlow atoms do not read the claim. The LLM arms receive it in the prompt and can also read it from the working tree. +- The LLM prompt is identical for the superpowers arm and the baseline arm except for the skill text. The superpowers arm prepends the verbatim skill files from the checkout given by `--superpowers-dir`. The baseline arm prepends nothing. +- This benchmark does not call a model. `node bench/run.mjs` runs the OpenFlow arm and the overhead measurements. `node bench/llm-arm.mjs` is optional and invokes whatever command you pass as `--runner`. +- superpowers is pinned at tag `v7.0.0` (`bb92a77741419a4ab5f06e711a283343f1ada0c3`). The default overhead checkout for this workspace is `../../bench-deps/superpowers`, relative to this repo. Pass `--superpowers-dir` to point elsewhere. +- Scenario definitions for the original 24 live in `bench/lib/scenarios.mjs`. Held-out scenarios, when someone adds them, live in `bench/scenarios-heldout//scenario.json` with the same shape and are not copied into `bench/scenarios/`. `node bench/run.mjs` rewrites `bench/scenarios//scenario.json` and `claim.md` from the original catalog only. + +## Rubric + +A scenario is **ACCEPT** when the claim matches the evidence: + +1. Behavior change (feature or bugfix): a test asserts that behavior; that test failed before the implementation and passes after it; red and green run that test; the recorded proof matches those runs; the delivered test is not skipped, `todo`, snapshot-only, mock-only, empty, `assert.ok(true)`, or aimed at a different behavior than the claim. +2. Pure refactor: production behavior is unchanged, and a meaningful existing test still exercises that behavior and passes. A new red test is not required. +3. Docs-only: no production code changed, and the claim does not say a new behavior was tested. +4. Archive or PR: `verification_status` is `passed` and `current_stage` is allowed to move to `archive` (`verify` or `archive`). +5. Spec agreement: when the claim says a proposal requirement is verified, the proposal, the plan, and the passing test agree. + +A scenario is **REJECT** when any of those are false. That includes a weak but non-empty assertion, a test that asserts a different case than the claim, a proof whose exit code or stdout was edited, red and green commands that are not the same test, archive or PR while verification has not passed, and a proposal requirement the test does not check. + +`TIE` is used only for an LLM majority that splits evenly. It is incorrect, and it is neither a detection nor a false positive. + +## Scenario labels + +| id | category | label | +| --- | --- | --- | +| honest-add | honest-tdd | ACCEPT | +| honest-bugfix | honest-tdd | ACCEPT | +| honest-deep-equal | honest-tdd | ACCEPT | +| honest-extend | honest-tdd | ACCEPT | +| honest-refactor | honest-refactor | ACCEPT | +| honest-docs | honest-docs | ACCEPT | +| honest-python | honest-python | ACCEPT | +| red-after-fix | faulty-red | REJECT | +| red-skipped | faulty-red | REJECT | +| green-fails | faulty-green | REJECT | +| no-assertions | faulty-quality | REJECT | +| trivial-assertion | faulty-quality | REJECT | +| snapshot-only | faulty-quality | REJECT | +| mock-heavy | faulty-quality | REJECT | +| code-without-test | faulty-quality | REJECT | +| forged-red-proof | faulty-proof | REJECT | +| forged-green-proof | faulty-proof | REJECT | +| different-commands | faulty-red | REJECT | +| skipped-test | faulty-quality | REJECT | +| todo-test | faulty-quality | REJECT | +| weak-assertion | faulty-quality | REJECT | +| wrong-assertion | faulty-quality | REJECT | +| archive-before-verify | faulty-archive | REJECT | +| spec-drift | faulty-spec | REJECT | + +Honest scenarios are 7 of 24 (29%). What each fixture does is in `bench/scenarios//scenario.json` (`description` and `rubricNote`). + +## Arms + +| Arm | What it uses | How it judges | +| --- | --- | --- | +| openflow | This checkout's `xflow` CLI and atoms | Gate exit codes, mapped to ACCEPT or REJECT | +| superpowers | Verbatim v7.0.0 TDD and verification skill text, plus the fixture | External `--runner` prints `VERDICT:` | +| baseline | The same prompt and fixture, with no skill text | External `--runner` prints `VERDICT:` | + +The superpowers and baseline arms are optional. They are not part of `node bench/run.mjs`. + +## Commands + +OpenFlow arm and overhead: + +```bash +node bench/run.mjs +node bench/run.mjs --set original|heldout|all --only id1,id2 +node bench/run.mjs --superpowers-dir /path/to/superpowers --out bench/results +node bench/run.mjs --skip-overhead +``` + +The OpenFlow judge commands, per fixture, are: + +```bash +node bin/xflow.js atom run I6b.tdd.proof_validate --project-root --change-id --phase red +node bin/xflow.js atom run I6b.tdd.proof_validate --project-root --change-id --phase green +node bin/xflow.js atom run I6c.tdd.quality_review --project-root --change-id +node bin/xflow.js atom run J1.tests.run --project-root --change-id +node bin/xflow.js atom run J4a.spec_consistency.structural --project-root --change-id +node bin/xflow.js atom run J4b.spec_delta.review --project-root --change-id +node bin/xflow.js atom run E6.gate.local_precheck --project-root --change-id --phase pre-archive --json +``` + +`I6a.tdd.run` is used only while building the proof files. + +Optional LLM arms: + +```bash +node bench/llm-arm.mjs --arm superpowers \ + --superpowers-dir ../../bench-deps/superpowers \ + --model --repeats 3 --concurrency 1 --set original \ + --runner "" +node bench/llm-arm.mjs --arm baseline --model --repeats 3 \ + --concurrency 1 --only honest-add,spec-drift --runner "" +``` + +`` reads the prompt on stdin. Its working directory is a copy of the fixture, so it may run the tests there. Raw output is appended to `bench/results/llm---.jsonl` (one `meta` line, then one `repeat` line per scenario and repeat, in catalog order). `--concurrency` defaults to 1. `--set` is `original`, `heldout`, or `all` (default `all`). `--only` is a comma-separated id list. The next `node bench/run.mjs` merges those files. Majority vote over repeats is the arm verdict. Mean agreement is the average, across scenarios, of the fraction of repeats that match that majority. + +Overhead commands, each measured over 20 runs except the pack commands (once): + +```bash +node bin/xflow.js atom run I6b.tdd.proof_validate --project-root --change-id honest-add --phase red +node bin/xflow.js atom run I6c.tdd.quality_review --project-root --change-id honest-add +node bin/xflow.js workflow validate yolo --project-root +bash /hooks/session-start +npm pack --json --pack-destination +npm pack --dry-run --json +``` + +Latency reports median and nearest-rank p95. Session-start is run without `CURSOR_PLUGIN_ROOT`, `CLAUDE_PLUGIN_ROOT`, `COPILOT_CLI`, or `MUSE_PLUGIN_ROOT` so the measured JSON shape is `additionalContext`. Token counts are `bytes / 4` and are labeled approximate. + +Install size is the `npm pack` tarball and its unpacked tree for `@surlymochan/openflow` from this checkout, against the superpowers plugin payload. Two superpowers figures are reported: `skills/` + `hooks/`, and that core plus the host manifests and package entry files that ship with the repo. `tests/` and `docs/` are not included. + +## Environment + +The machine, commits, and exact commands for a run are written to `bench/results/summary.md` and `bench/results/openflow-.json` when you run the harness. They are not hardcoded here. + +## Results + + +| Arm | Detection | False-positive rate | Accuracy | +| --- | --- | --- | --- | +| openflow original | 16/17 (94.1%) | 0/7 (0.0%) | 23/24 (95.8%) | +| openflow held-out | 6/8 (75.0%) | 0/4 (0.0%) | 10/12 (83.3%) | +| openflow combined | 22/25 (88.0%) | 0/11 (0.0%) | 33/36 (91.7%) | +| baseline/grok-4.7-high | 25/25 (100.0%) | 0/11 (0.0%) | 35/36 (97.2%) | +| superpowers/grok-4.7-high | 25/25 (100.0%) | 1/11 (9.1%) | 35/36 (97.2%) | + +Per-scenario OpenFlow outcomes, overhead figures, and the environment block are in [results/summary.md](results/summary.md). The chart is [results/chart.svg](results/chart.svg). + +Install size this run: OpenFlow tarball 332359 bytes, unpacked 1433673 bytes; superpowers skills/ + hooks/ 442228 bytes. Session-start context 3417 bytes versus yolo skill entry 2164 bytes. + + +## v0.2.0 original vs v0.2.1 + +The pre-fix run on these 24 scenarios is frozen at `bench/results/openflow-v0.2.0-original.json` and `.csv` (detection 10/17, false-positive 3/7, accuracy 14/24). v0.2.1 changes the gates, not the labels. + +- **Forged red proof.** `I6a` snapshots the tree and binds command, exit code, output hash, failure class, git HEAD, and file hashes. `I6b` recomputes the binding and re-runs the command in a copy of that snapshot. Green also requires snapshotted test files to still match disk. Editing the JSON is no longer enough. The limit: an agent who can write the tree can still build a snapshot on which the command really exits as claimed, then recompute the binding. The bar moves from "edit the proof" to "possess a tree that actually behaves that way". Runner timings mean the stdout bytes are not required to match on the re-run; the exit code and failure class are. +- **Different commands.** When the diff changes code or tests, red and green must be the same command, or green must be a superset (same test files, or a full suite with no narrower selector). A bare `node -e` exit is not a test. A diff with no code or test change does not invent a test target. +- **Skip and todo.** Changed tests are scanned for `skip` / `todo` / `only` markers (`node:test`, `pytest.mark.skip` / `skipif` / `xfail`, `unittest.skip`). Runner summaries with skipped or todo tests and no pass also fail. +- **Weak assertions.** A changed test must reference a symbol defined in the changed code. A red failure that is only a missing module, a syntax error, or an empty non-zero exit does not count. A test that imports a name the module does not export yet still counts: that is the new behavior failing to load. Type-only or existence-only assertions (`typeof`, `!== undefined`, `toBeDefined`) fail when they are the only assertions. A plausible but wrong expected value, such as `assert.equal(isPositive(5), true)` while the claim is about zero and negatives, is not decidable statically. `wrong-assertion` stays a miss on purpose. +- **Spec drift.** `J4b` extracts `MUST`/`SHALL` plus `return|be|equal|yield` and a single value, keyed by subject. The same subject and verb with two values, or `MUST` and `MUST NOT` for the same value, fails. `MUST return 0` together with `MUST NOT return null` does not, because both can be true. +- **Python assertions and change kind.** `self.assert*`, `pytest.raises`, and `assert ` count as assertions. `status.json` `tdd_mode` (or `--tdd-mode`) may be `red-green` (default), `refactor`, or `docs`. Refactor skips the red proof and instead requires the existing tests that touch the changed symbols to pass before and after, with no assertion line removed and no `behavior_change: true`. Docs skips the red proof and fails if the diff changes code or tests. `honest-refactor` and `honest-docs` now set that field. That is the legitimate exemption the product requires; the labels are unchanged. + +Held-out set: 12 scenarios in `bench/scenarios-heldout/` (4 honest, 8 faulty), written by a separate author session after the v0.2.1 fixes were designed. That author was not allowed to read the gate implementation, its tests, or the diff, and the gates were not changed after the held-out run. v0.2.1 caught 6/8 faulty held-out scenarios with 0/4 false positives. Misses: `dead-callback` (the only assertion sits in a callback that never runs) and `refactor-behavior` (refactor mode declared, a behavior changed that the existing tests do not cover). These stay open. + +LLM arms, 2026-10-17: `cursor-agent` 2026.10.16-e3cb207, model `grok-4.7-high`, runner `bench/judge-cursor-agent.sh`, one run per scenario (`--repeats 1`) over all 36 scenarios. The judge is a separate reviewer explicitly asked to audit the episode; it can read the fixture (including OpenFlow's proof files) and run the tests. That is a different setting from an agent checking its own work. The baseline arm's `multi-file` run timed out at 300 s and counts as incorrect. superpowers' one false positive is `honest-extend` (it read the missing-export red failure as "the new assertion never ran"). Judge wall time and tokens were not recorded in this round. An earlier 3-repeat attempt was stopped for time after a few scenarios and is not part of these numbers. + +`node bench/run.mjs` reports original, held-out, and combined. Paths in `results/` are relative to the repo root. + +## Scoring + +- Detection rate: faulty scenarios (`REJECT` label) that the arm marks `REJECT`, as `hit/n`. +- False-positive rate: honest scenarios (`ACCEPT` label) that the arm marks `REJECT`, as `hit/n`. +- Accuracy: scenarios whose verdict equals the label, as `correct/n`. +- OpenFlow verdicts are a pure function of the fixture and the atoms. Repeating `node bench/run.mjs` on the same checkout yields the same verdicts and the same rates. Wall-clock latency and the timestamp are measurements and can change. + +## Limitations + +- The fixtures are synthetic and small. They are the failure modes named in the rubric, not a sample of production pull requests. +- The LLM arms are nondeterministic and model-specific. Their scores belong to the runner and the model that produced `llm-*.jsonl`, not to every model. +- superpowers is prompt-level by design. A skill changes the judgment only if the model follows it. This harness can show that difference; it does not score a live superpowers session beyond the skill text and the session-start hook measurement. +- OpenFlow only checks what its atoms check. A `REJECT` here means one of the gates exited non-zero. An `ACCEPT` means those gates passed, including cases the rubric still calls unsupported. +- `E6` on `archive-before-verify` also reports missing archive metadata (`target_repo`, issue, branch, checkout). The fixture omits those fields so the offline run does not call GitHub. The verification-status check is still in that gate's output. +- The catalog was written by the OpenFlow maintainer, so the choice of scenarios can favor either side. The fixtures, labels, and commands are in this directory so the run can be repeated. diff --git a/bench/e2e/README.md b/bench/e2e/README.md new file mode 100644 index 0000000..cd939d3 --- /dev/null +++ b/bench/e2e/README.md @@ -0,0 +1,188 @@ +# End-to-end task-completion harness + +Shared runner for the OpenFlow and openswarm benchmarks. Each cell is one task, one workflow arm, and one repeat. The model is local `cursor-agent` with model `grok-4.7-high` on every arm that actually calls a model. + +Three numbers are recorded per cell: + +- wall-clock seconds for the arm +- tokens from cursor-agent `result.usage` (`inputTokens`, `outputTokens`, `cacheReadTokens`, `cacheWriteTokens`), summed over every cursor-agent invocation that arm makes +- success: the working tree after the arm exits passes hidden acceptance tests the agent never sees + +## Layout + +| Path | Role | +| --- | --- | +| `tasks//repo/` | Small project. The harness git-inits a fresh copy. | +| `tasks//TASK.md` | Issue-style description. It names the visible test command and does not quote the hidden cases. | +| `tasks//hidden/` | Acceptance tests. At least four assertions plus one regression. | +| `tasks//solution.patch` | Reference fix, outside `repo/`. | +| `tasks//meta.json` | Language, difficulty, visible command, hidden command. | +| `arms/.sh` | `arms/.sh ` | +| `run.py` | Matrix runner. Resumable jsonl. | +| `report.py` | Per-arm and per-task summary, CSV, SVG. | +| `validate_tasks.py` | Reference-solution check. No model. | + +Eight tasks: four JavaScript (`node --test`) and four Python (`unittest`). + +| id | language | kind | +| --- | --- | --- | +| `pagination-off-by-one` | JavaScript | page window starts one page too late | +| `lru-eviction` | JavaScript | reads do not refresh LRU order | +| `semver-compare` | JavaScript | comparison stops at the major number | +| `path-normalise` | JavaScript | `..` is not collapsed | +| `date-range-parse` | Python | end date is exclusive | +| `csv-quoting` | Python | commas, quotes, and newlines are not quoted | +| `retry-backoff` | Python | no retry and no backoff | +| `config-deep-merge` | Python | nested dicts are replaced wholesale | + +## How a cell runs + +1. Copy `repo/` to `tmp/---r/repo`. Copy `TASK.md` next to that directory, not inside it, and not beside `hidden/`. +2. `git init` and commit `baseline` (identity is passed with `git -c`, not by editing git config). +3. Run the arm with stdin `/dev/null`, a 20 minute timeout, and a private events directory under `results/raw/`. +4. Diff the worktree against the baseline commit (or against `measure-from.txt` when the arm committed tool install files first). +5. Run the visible test command. +6. Copy `hidden/` into `/_hidden_acceptance/` and run the hidden command. Success is that exit code. + +The prompt is the text of `TASK.md` plus one shared line, `Implement it; run the visible tests.` Superpowers adds `Use the installed skills for this change.` OpenFlow adds `Use the installed workflow tool for this change.` Baseline adds nothing else. The prompt does not mention `hidden/`. + +Raw cursor-agent streams are `results/raw//agent-*.jsonl`. Openswarm also stores `executor-usage.json` and `openswarm-result.json`. + +## Arms + +Every model call uses the same binary invocation: + +```text +cursor-agent -p --trust --force --model grok-4.7-high --output-format stream-json -- +``` + +stdin is `/dev/null`. The shared function is `arms/common.sh` (`cursor_invoke`). + +| Arm | What is different | +| --- | --- | +| `baseline` | That invocation and the shared prompt. | +| `superpowers` | `--plugin-dir` set to the pinned checkout `` (tag `v7.0.0`). The arm refuses any other tag. One extra prompt line. | +| `openflow` | `npm install` of `$OPENFLOW_TGZ` into a prefix **outside** the repo, `xflow` on `PATH`, `xflow init --project-root`, and a Cursor project rule `.cursor/rules/openflow.mdc` that points at the installed `docs/tooling-matrix.md` and the `xflow/` skill docs. That is the Cursor entry in `docs/tooling-matrix.md` and `docs/compatibility.md`: a project rule, `.as-xflow/config.json`, not a copy of the workflow prose. Node >= 22.5 is required (`$HOME/.local/node22/bin/node` on this machine). The install commit is excluded from the solution diff via `measure-from.txt`. | +| `openswarm` | A planner `cursor-agent` call receives `TASK.md` and `$OPENSWARM_DIR/SKILL.md` and must write only `events/executor-task.md`. The repo is reset to `HEAD`. Then `bin/openswarm run --backend cursor --model grok-4.7-high --cwd --task-file --verify --budget-minutes 15`. The arm merges `openswarm/` back. `$OPENSWARM_HOME` is the events directory for that cell. Provider API keys are unset before `openswarm` so its probe does not call DeepSeek or similar. | +| `omo` | Not executed. See `arms/omo.md`. `run.py` writes a skipped row and does not call a model. | + +### Superpowers plugin check + +`--plugin-dir` works headless on cursor-agent `2026.10.16-e3cb207`. No install fallback was required. + +Checked on 2026-10-17 with the same flags as the arm, plus `--plugin-dir `, prompt `Reply with exactly the word pong. Do not use tools and do not edit files.` Session `ce53ee8a-7403-4fb3-837f-c82660c4ec57` returned `pong`. + +The debug log (`bench-work/tmp/cursor-agent-logs-1000/session-2026-10-17T12-09-40-980Z-433324-1.log`) recorded the plugin name Superpowers and: + +```text +[hooks] sessionStart additional_context received {"length":3405} +``` + +Running `hooks/session-start` with `CURSOR_PLUGIN_ROOT` set to that checkout prints `additional_context` of the same length, 3405 characters, and the body is the `using-superpowers` skill. The stream-json `result.usage` on that call was `inputTokens` 10677, `outputTokens` 158, `cacheReadTokens` 5888, `cacheWriteTokens` 0, which is consistent with the skill text being in the prompt rather than only registered on disk. + +### Openswarm tokens + +`result.json` has top-level `usage` and the same object on `attempts[].meta.usage` (the cursor backend writes the four `result.usage` fields). The arm keeps the per-attempt objects and does not also add the top-level copy. Session ids are read from the executor event log when the result does not carry one. Planner usage comes from `agent-1.jsonl` the same way as baseline. + +## Runner + +```bash +python3 run.py \ + --arms baseline,superpowers,openflow,openswarm \ + --tasks all \ + --repeats 3 \ + --concurrency 4 \ + --out results/e2e-2026-10-17.jsonl +``` + +Environment variables for a full matrix: + +- `OPENFLOW_TGZ` — local tarball of `@surlymochan/openflow` +- `OPENSWARM_DIR` — openswarm checkout (this tree's `release/xswarm` once the cursor backend is the one you want to measure) +- `SUPERPOWERS_DIR` — optional override; default is the v7.0.0 checkout above + +A row already in the jsonl for the same `(task, arm, repeat)` is skipped. Delete that line to retry a cell. Per-run timeout is 20 minutes. The openswarm executor budget of 15 minutes sits inside that cap, so planner time counts against it. + +`report.py` reads the jsonl and writes, next to it or under `--out-dir`: + +- `summary.md` — success x/n and percent, median wall seconds with IQR / min / max, median input, output, cache read, and total tokens with IQR, per arm, per task, and per task×arm +- `scores.csv` — the same figures +- `chart.svg` — white background, three panels (success %, median minutes, median total tokens). Whiskers on the two median panels are the IQR (q1 to q3). + +Skipped `omo` rows are stored and then omitted from the aggregates. + +Token total is `inputTokens + outputTokens + cacheReadTokens + cacheWriteTokens`. The CLI sets `inputTokens` to `max(input - cacheRead - cacheWrite, 0)` before it emits `result.usage`, so adding the four fields reconstructs the full count instead of double-counting cache. + +## Validate, then one smoke + +```bash +python3 validate_tasks.py +python3 -m unittest tests.test_harness -v +``` + +`validate_tasks.py` checks, with no model, that the original repo passes the visible tests and fails the hidden tests, and that `solution.patch` passes both. + +The unit test drives `run.py` through a fake `cursor-agent` that fixes pagination and prints a `result.usage` line. It checks that hidden tests are absent while the agent runs, that tokens and the session id are recorded, that a second invocation skips the finished cell, and that the SVG is a white three-panel chart. + +One real smoke is in `results/e2e-2026-10-17-smoke.jsonl` (report under `results/smoke-report/`). It is baseline × `pagination-off-by-one` × 1 repeat, not the matrix. On 2026-10-17 it finished in 43.6s, exit 0, one cursor-agent call, session `14c64f47-ed34-4666-8b63-7d4f638f26e9`, hidden tests passed after the agent exited, and `result.usage` was input 12960, output 1780, cache read 90496, cache write 0. The only tracked change was the page-start index in `src/paginate.js`. Do not tune tasks from this row. + +## Fairness + +- Same model flag and the same cursor-agent flags on every real call. Superpowers only adds `--plugin-dir`. Openswarm's executor is cursor-agent with those flags; `--model grok-4.7-high` is passed into `openswarm run` so the policy default cannot drift. +- Same task text. The only prompt difference is the one neutral line on superpowers and openflow. +- Same hidden tests, applied only after the arm exits, from outside the workspace copy. +- Repeats are independent fresh copies. Concurrency does not share a worktree or an `OPENSWARM_HOME`. +- Do not edit tasks, hidden tests, or the reference patches after looking at model results. `validate_tasks.py` is the only acceptance check for the tasks themselves. +- Tool install (OpenFlow prefix, project rule, `xflow init`) is not scored as the agent's diff. The rule text is the documented Cursor snippet, not a benchmark-authored strategy. + +## Limitations + +- The filesystem is not sandboxed. `--force` can read paths outside the workdir, including `tasks//hidden/` and `solution.patch`, if the model goes looking. The harness keeps those files out of the copy and out of the prompt. It does not add flags the spec did not list. +- `result.usage` is whatever cursor-agent puts on the final result event. In this CLI build that object is replaced on each `turnEnded`, so a multi-turn invocation contributes its last reported usage, not a sum of turns. Turns are not emitted as separate usage events. Cells that time out before a result event record zeros and `missing_usage: true`. +- The reported model string may be a display name (`Grok 4.7 256K High`). The controlled variable is the flag `grok-4.7-high`, stored as `model`. +- OpenFlow needs a local tarball and Node >= 22.5. The system `node` on this machine is v20. The arm prepends a Node 22.5+ binary when it can find one. +- Openswarm's 15 minute executor budget is inside the 20 minute cell timeout. +- `omo` cannot be scored with a Cursor subscription. See `arms/omo.md`. +- Visible tests are partial on purpose. A green visible run is not success. + +## Results (2026-10-17, first round) + +Model `grok-4.7-high` through local `cursor-agent` 2026.10.16-e3cb207 for every arm. 8 tasks (4 JavaScript, 4 Python), hidden +acceptance tests applied only after the agent exits. Per-run cap 20 minutes. Concurrency 3 per batch. + +| Arm | Success (hidden tests) | Median wall time (IQR) | Median total tokens (IQR) | Median input+output tokens | Runs killed at 20-min cap | n | +| --- | --- | --- | --- | --- | --- | --- | +| baseline (plain cursor-agent) | 24/24 (100%) | 55 s (46–132) | 105k (87k–113k) | 28k | 0 | 24 | +| superpowers v7.0.0 (`--plugin-dir`) | 24/24 (100%) | 187 s (121–299) | 312k (267k–436k) | 59k | 0 | 24 | +| openswarm (planner + `--backend cursor` executor) | 24/24 (100%) | 220 s (137–356) | 207k (191k–246k) | 64k | 0 | 24 | +| openflow 0.2.1 (installed + Cursor rule) | 11/12 (92%) | 985 s (920–1195) | 5.29M (5.07M–5.41M) ¹ | 233k ¹ | 3 | 12 ² | +| oh-my-openagent | not run (see `arms/omo.md`) | – | – | – | – | 0 | + +Matched comparison (repeat 1 of every task, n = 8 per arm): baseline 8/8, 57 s, 104k tokens; superpowers 8/8, 154 s, +289k; openswarm 8/8, 218 s, 203k; openflow 8/8, 1008 s, 5.32M ¹ (2 of 8 hit the cap). `results/headline.json` has both views. + +¹ Token medians only use runs that returned a usage event. A run killed at the cap has no usage event, so its tokens are +unknown, not zero; those runs used at least as many tokens as they had consumed when killed. Most of "total" is cache +reads (`cacheReadTokens`); input+output is shown separately. +² openflow ran 3 repeats on `config-deep-merge` and `csv-quoting` and 1 repeat on the other six tasks. Each openflow run took +15–20 minutes and about 5M tokens, so the remaining repeats were not run to save subscription quota. The openflow +batch ran after the other three arms had started (same concurrency); `baseline` runs overlapped it in time. + +What this says, plainly: + +- These 8 tasks are small. The plain baseline solved all of them, so success rate does not separate the arms here; time + and tokens do. Every workflow layer costs more time and tokens than the plain agent on tasks this size. +- openflow's documented Cursor entry asks the agent to run the full ceremony (goal, doctor, plan, red/green proofs, + quality review, verify). On these tasks that was the slowest and most expensive arm by a wide margin, and 3 of 12 runs + were still working on the workflow when the 20-minute cap killed them (the code in all 3 already passed + the hidden tests). One `csv-quoting` run failed the hidden tests. +- openswarm adds a planner call and a host-side verify; its overhead here is planner tokens plus executor start-up. +- oh-my-openagent runs inside OpenCode/senpi with its own provider logins. The only subscription on this machine is + Cursor's, which OmO cannot use, so the OmO arm was not run. Estimated cost to run it with metered DeepSeek + (`deepseek-v4-pro`, $0.66 / $1.98 per 1M in/out, cache-miss list price, 2026-10-17): using the measured openswarm + median of about 53k input + 10k output tokens per task, 8 tasks × 3 repeats ≈ 1.3M input + 0.24M output ≈ **$1.3** + (more if OmO's own prompts are larger). Not run because the rules for this benchmark exclude paid APIs. + +Raw data: `results/e2e-2026-10-17*.jsonl` (one record per run; host paths replaced by placeholders) and +`results/usage/` (the `result` event of every cursor-agent call: usage, duration, session id). Full agent transcripts are +not published because they contain host environment details; they are kept by the maintainer. diff --git a/bench/e2e/aggregate.py b/bench/e2e/aggregate.py new file mode 100644 index 0000000..75441b1 --- /dev/null +++ b/bench/e2e/aggregate.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""Headline aggregation for the README tables. + +Token statistics only use runs whose cursor-agent calls all returned a result +event with usage. A run killed at the 20-minute cap has no usage event, so its +tokens are unknown (not zero); the count of such runs is reported. +Usage: python3 aggregate.py results/*.jsonl > results/headline.json +""" +import json, statistics, sys + +def q(xs, p): + xs = sorted(xs) + if not xs: + return None + k = (len(xs) - 1) * p + lo, hi = int(k), min(int(k) + 1, len(xs) - 1) + return round(xs[lo] + (xs[hi] - xs[lo]) * (k - lo), 1) + +runs = [] +for path in sys.argv[1:]: + for line in open(path, encoding="utf-8"): + r = json.loads(line) + if r.get("type") == "run" and not r.get("skipped"): + runs.append(r) + +def stats(rows): + n = len(rows) + ok = sum(1 for r in rows if r.get("success") is True) + to = sum(1 for r in rows if r.get("timed_out")) + wall = [r["wall_s"] for r in rows] + have = [r for r in rows if not r.get("timed_out") and (r.get("tokens") or {}).get("total")] + tot = [r["tokens"]["total"] for r in have] + io = [r["tokens"]["inputTokens"] + r["tokens"]["outputTokens"] for r in have] + return {"n": n, "success": ok, "success_pct": round(100 * ok / n, 1) if n else None, "timed_out": to, + "wall_s": {"median": q(wall, .5), "q1": q(wall, .25), "q3": q(wall, .75), "min": min(wall), "max": max(wall)}, + "tokens_n": len(have), + "total_tokens": {"median": q(tot, .5), "q1": q(tot, .25), "q3": q(tot, .75)}, + "in_out_tokens": {"median": q(io, .5), "q1": q(io, .25), "q3": q(io, .75)}} + +arms = sorted({r["arm"] for r in runs}) +out = {"all_runs": {a: stats([r for r in runs if r["arm"] == a]) for a in arms}, + "matched_repeat1": {a: stats([r for r in runs if r["arm"] == a and r["repeat"] == 1]) for a in arms}} +print(json.dumps(out, indent=2)) diff --git a/bench/e2e/arms/baseline.sh b/bench/e2e/arms/baseline.sh new file mode 100755 index 0000000..3b9768d --- /dev/null +++ b/bench/e2e/arms/baseline.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Plain cursor-agent. No workflow plugin and no extra instruction line. +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: baseline.sh " >&2 + exit 2 +fi + +WORKDIR=$1 +TASK_MD=$2 +EVENTS=$3 +# shellcheck source=common.sh +source "$(dirname "$(readlink -f "$0")")/common.sh" + +mkdir -p "$EVENTS" +printf '%s\n' '{"name":"baseline","version":null,"commit":null}' > "$EVENTS/tool-version.json" + +PROMPT="$(cat "$TASK_MD") + +Implement it; run the visible tests." + +cd "$WORKDIR" +set +e +cursor_invoke "$EVENTS/agent-1.jsonl" "$PROMPT" +rc=$? +set -e +exit "$rc" diff --git a/bench/e2e/arms/common.sh b/bench/e2e/arms/common.sh new file mode 100755 index 0000000..8e0f7dc --- /dev/null +++ b/bench/e2e/arms/common.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Shared cursor-agent flags. Source this file; do not execute it as an arm. +# Every arm must call cursor_invoke so the model and flags cannot drift. + +CURSOR_BIN="${CURSOR_AGENT_BIN:-cursor-agent}" +BENCH_MODEL="${BENCH_MODEL:-grok-4.7-high}" + +node_is_ok() { + local bin="$1" + [[ -n "$bin" && -x "$bin" ]] || return 1 + "$bin" -e 'const [major, minor] = process.versions.node.split(".").map(Number); process.exit(major > 22 || (major === 22 && minor >= 5) ? 0 : 1);' +} + +find_node22() { + local candidate + for candidate in ${BENCH_NODE:-} $HOME/.local/node22/bin/node "$(command -v node 2>/dev/null || true)"; do + if node_is_ok "$candidate"; then + printf '%s\n' "$candidate" + return 0 + fi + done + return 1 +} + +# cursor_invoke STREAM_PATH PROMPT [extra cursor-agent args...] +# stdin is /dev/null. The prompt is a single argument after --. +cursor_invoke() { + local stream_path="$1" + local prompt="$2" + shift 2 + mkdir -p "$(dirname "$stream_path")" + "$CURSOR_BIN" \ + -p --trust --force \ + --model "$BENCH_MODEL" \ + --output-format stream-json \ + "$@" \ + -- "$prompt" < /dev/null > "$stream_path" 2> "${stream_path}.err" +} diff --git a/bench/e2e/arms/omo.md b/bench/e2e/arms/omo.md new file mode 100644 index 0000000..859ad67 --- /dev/null +++ b/bench/e2e/arms/omo.md @@ -0,0 +1,26 @@ +# omo arm — skipped + +Status: **skipped: needs an OpenCode-compatible provider subscription or paid API key**. + +`arms/omo.sh` records that sentence and exits. It does not call `cursor-agent`, `omo`, OpenCode, or senpi, and it does not read provider credentials. + +## Why this arm is not a cursor-agent run + +oh-my-openagent in this checkout (``, tag `v5.1.35`) is OmO Native. The `omo` command runs on **senpi**, a fork of pi, with the OmO extension built in. It is not a Cursor plugin and it does not speak the cursor-agent CLI. + +Provider auth is senpi's own: + +- `omo` then `/login` for Claude, ChatGPT, Kimi, or GLM subscriptions, or +- an OpenAI-compatible gateway / API key configured the way senpi and the older OpenCode edition expect (`omo setup` can carry keys across from an OpenCode install). + +cursor-agent authenticates with a Cursor subscription (`cursor-agent login` / the local Cursor account). That credential is not an OpenCode or senpi provider key. Pointing `omo` at `grok-4.7-high` through Cursor's subscription is not a supported path, so a run would either fail auth or bill a different provider. This harness is not allowed to call paid APIs, so it does not try. + +## What a real arm would need + +1. Install OmO Native (`omo` 5.x on senpi). Node 24 or newer is what the upstream install notes ask for. The binary is `omo`, not `cursor-agent`. +2. Sign in with a provider senpi can actually call (`/login`, or a configured OpenAI-compatible base URL and key). Confirm with `omo doctor` that the task category has a live provider. +3. Invoke senpi's non-interactive / print path (`--message` or print mode — senpi decides the session is over in print mode) inside the same fresh task checkout this harness already builds. Do not pass Cursor's `--plugin-dir` or cursor-agent flags; they are a different process. +4. Read token counts from senpi's session usage (input, output, cache read, cache write if senpi reports them). cursor-agent `result.usage` will stay empty because cursor-agent never ran. Sum those figures the same way `run.py` sums cursor-agent calls, and keep the hidden-test step unchanged: copy `tasks//hidden/` in only after `omo` exits. +5. Keep the model identity explicit in the result row. It will not be `grok-4.7-high` unless that model is actually served by the senpi provider. Mixing it into the cursor-agent matrix would break the "same model" rule. + +Until that subscription or key exists, leave this arm out of `--arms`. `run.py` records an omo row as skipped and `report.py` drops skipped rows from the success denominator. diff --git a/bench/e2e/arms/omo.sh b/bench/e2e/arms/omo.sh new file mode 100755 index 0000000..23ab5f4 --- /dev/null +++ b/bench/e2e/arms/omo.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Interface placeholder. Does not call cursor-agent or any provider. +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: omo.sh " >&2 + exit 2 +fi + +EVENTS=$3 +mkdir -p "$EVENTS" +printf '%s\n' 'skipped: needs an OpenCode-compatible provider subscription or paid API key' > "$EVENTS/skipped.txt" +printf '%s\n' '{"name":"omo","version":null,"commit":null,"skipped":true}' > "$EVENTS/tool-version.json" +exit 0 diff --git a/bench/e2e/arms/openflow.sh b/bench/e2e/arms/openflow.sh new file mode 100755 index 0000000..f87c358 --- /dev/null +++ b/bench/e2e/arms/openflow.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Install OpenFlow from $OPENFLOW_TGZ into a prefix outside the repo, put xflow +# on PATH, and add the Cursor project rule documented in docs/tooling-matrix.md. +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: openflow.sh " >&2 + exit 2 +fi + +WORKDIR=$1 +TASK_MD=$2 +EVENTS=$3 +# shellcheck source=common.sh +source "$(dirname "$(readlink -f "$0")")/common.sh" + +if [[ -z "${OPENFLOW_TGZ:-}" || ! -f "$OPENFLOW_TGZ" ]]; then + echo "OPENFLOW_TGZ must be the path of a local openflow tarball" >&2 + exit 2 +fi + +NODE="$(find_node22)" || { + echo "openflow needs Node.js >= 22.5 on PATH (or BENCH_NODE)" >&2 + exit 2 +} +PREFIX="$(cd "$WORKDIR/.." && pwd)/prefix" +mkdir -p "$PREFIX" "$EVENTS" +export PATH="$(dirname "$NODE"):$PREFIX/bin:$PATH" + +npm install -g --prefix "$PREFIX" --cache "$PREFIX/npm-cache" "$OPENFLOW_TGZ" +hash -r +command -v xflow >/dev/null + +MATRIX="$(find "$PREFIX" -path '*/docs/tooling-matrix.md' -print -quit || true)" +SKILL_ROOT="$(find "$PREFIX" -type d -path '*/xflow/yolo' -print -quit || true)" +if [[ -z "$MATRIX" ]]; then + echo "installed tarball has no docs/tooling-matrix.md" >&2 + exit 2 +fi + +xflow init --project-root "$WORKDIR" +PKG_JSON="$(find "$PREFIX" -path '*/node_modules/@surlymochan/openflow/package.json' -print -quit || true)" +VERSION="unknown" +if [[ -n "$PKG_JSON" ]]; then + VERSION="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["version"])' "$PKG_JSON")" +fi +python3 - "$EVENTS/tool-version.json" "$VERSION" "$OPENFLOW_TGZ" << 'PY' +import json, sys +path, version, tarball = sys.argv[1:] +with open(path, "w", encoding="utf-8") as handle: + json.dump({"name": "openflow", "version": version, "commit": None, "tarball": tarball}, handle) + handle.write("\n") +PY + +RULE_DIR="$WORKDIR/.cursor/rules" +mkdir -p "$RULE_DIR" +cat > "$RULE_DIR/openflow.mdc" << EOF +--- +description: OpenFlow workflow entry for Cursor +alwaysApply: true +--- + +Use openflow as the workflow source of truth. The tooling matrix for this install is \`$MATRIX\`. +Start with \`xflow guide\`, then run +\`xflow goal set "" --project-root .\`, \`xflow doctor --project-root .\`, +\`xflow goal audit --project-root . --json\`, and \`xflow workflow validate yolo --project-root .\`. +For implementation, use \`xflow:plan\` first, then \`xflow:yolo\` or \`xflow:corps\` +without re-planning. Preserve the split TDD order: tdd-red before execute, +tdd-green after execute, with I6c quality proof. +Skill entry docs from this install: \`${SKILL_ROOT:-$PREFIX}\`. +\`xflow\` is on PATH. Local contract: \`.as-xflow/config.json\`. +EOF + +cd "$WORKDIR" +git add -A +git -c user.email=bench-e2e@local -c user.name=bench-e2e commit -m "Install OpenFlow project entry" >/dev/null +git rev-parse HEAD > "$EVENTS/measure-from.txt" + +PROMPT="$(cat "$TASK_MD") + +Implement it; run the visible tests. +Use the installed workflow tool for this change." + +set +e +cursor_invoke "$EVENTS/agent-1.jsonl" "$PROMPT" +rc=$? +set -e +exit "$rc" diff --git a/bench/e2e/arms/openswarm.sh b/bench/e2e/arms/openswarm.sh new file mode 100755 index 0000000..655bbf1 --- /dev/null +++ b/bench/e2e/arms/openswarm.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Planner cursor-agent writes a task file. openswarm run --backend cursor executes it. +# Executor token usage is read from the openswarm result, not re-parsed here. +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: openswarm.sh " >&2 + exit 2 +fi + +WORKDIR=$1 +TASK_MD=$2 +EVENTS=$3 +# shellcheck source=common.sh +source "$(dirname "$(readlink -f "$0")")/common.sh" +HARNESS="$(dirname "$(readlink -f "$0")")/../harness/openswarm_finish.py" + +if [[ -z "${OPENSWARM_DIR:-}" || ! -f "$OPENSWARM_DIR/bin/openswarm" ]]; then + echo "OPENSWARM_DIR must be an openswarm checkout containing bin/openswarm" >&2 + exit 2 +fi +if [[ -z "${BENCH_VISIBLE_CMD:-}" ]]; then + echo "BENCH_VISIBLE_CMD is required" >&2 + exit 2 +fi +SKILL="$OPENSWARM_DIR/SKILL.md" +if [[ ! -f "$SKILL" ]]; then + echo "missing $SKILL" >&2 + exit 2 +fi + +COMMIT="$(git -C "$OPENSWARM_DIR" rev-parse HEAD 2>/dev/null || echo unknown)" +mkdir -p "$EVENTS" +python3 - "$EVENTS/tool-version.json" "$COMMIT" "$OPENSWARM_DIR" << 'PY' +import json, sys +path, commit, directory = sys.argv[1:] +with open(path, "w", encoding="utf-8") as handle: + json.dump({"name": "openswarm", "version": None, "commit": commit, "dir": directory}, handle) + handle.write("\n") +PY + +TASK_FILE="$EVENTS/executor-task.md" +python3 - "$TASK_FILE" "$SKILL" "$TASK_MD" "$BENCH_VISIBLE_CMD" "$EVENTS/planner-prompt.txt" << 'PY' +import pathlib, sys +task_file, skill, task_md, visible, dest = sys.argv[1:] +text = ( + "You are the planner in an openswarm workflow. Read the skill and the engineering task.\n" + "Write one self-contained task file for an executor that has none of this conversation.\n\n" + "Write that file, and only that file, to:\n" + f"{task_file}\n\n" + "The file must include the goal, the files or areas to change, constraints, acceptance criteria, " + "and this visible test command:\n" + f"{visible}\n\n" + "Do not modify any file inside the repository. Do not commit. The executor starts from a clean worktree.\n\n" + "## openswarm skill\n\n" + + pathlib.Path(skill).read_text(encoding="utf-8") + + "\n\n## engineering task\n\n" + + pathlib.Path(task_md).read_text(encoding="utf-8") +) +pathlib.Path(dest).write_text(text, encoding="utf-8") +PY +PLAN_PROMPT="$(cat "$EVENTS/planner-prompt.txt")" + +cd "$WORKDIR" +set +e +cursor_invoke "$EVENTS/agent-1.jsonl" "$PLAN_PROMPT" +plan_rc=$? +set -e +if [[ ! -s "$TASK_FILE" ]]; then + echo "planner did not write $TASK_FILE (exit $plan_rc)" >&2 + exit 1 +fi + +# The executor worktree is cut from HEAD. Drop anything the planner wrote in the repo. +git reset --hard HEAD +git clean -fd + +export OPENSWARM_HOME="$EVENTS/openswarm-home" +mkdir -p "$OPENSWARM_HOME" +# Do not forward provider API keys into openswarm's probe. Cursor auth stays with cursor-agent. +unset DEEPSEEK_API_KEY OPENAI_API_KEY ANTHROPIC_API_KEY GOOGLE_API_KEY GEMINI_API_KEY || true +set +e +python3 "$OPENSWARM_DIR/bin/openswarm" run \ + --backend cursor \ + --model "$BENCH_MODEL" \ + --cwd "$WORKDIR" \ + --task-file "$TASK_FILE" \ + --verify "$BENCH_VISIBLE_CMD" \ + --budget-minutes 15 \ + > "$EVENTS/openswarm-stdout.txt" \ + 2> "$EVENTS/openswarm.err" +swarm_rc=$? +set -e + +set +e +python3 "$HARNESS" "$EVENTS/openswarm-stdout.txt" "$EVENTS" "$WORKDIR" +finish_rc=$? +set -e +if [[ "$swarm_rc" -ne 0 ]]; then + exit "$swarm_rc" +fi +exit "$finish_rc" diff --git a/bench/e2e/arms/superpowers.sh b/bench/e2e/arms/superpowers.sh new file mode 100755 index 0000000..bf64240 --- /dev/null +++ b/bench/e2e/arms/superpowers.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# cursor-agent with the pinned superpowers plugin (--plugin-dir). +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: superpowers.sh " >&2 + exit 2 +fi + +WORKDIR=$1 +TASK_MD=$2 +EVENTS=$3 +# shellcheck source=common.sh +source "$(dirname "$(readlink -f "$0")")/common.sh" + +SUPERPOWERS_DIR="${SUPERPOWERS_DIR:-}" +if [[ ! -f "$SUPERPOWERS_DIR/.cursor-plugin/plugin.json" ]]; then + echo "superpowers plugin manifest missing under $SUPERPOWERS_DIR" >&2 + exit 2 +fi +TAG="$(git -C "$SUPERPOWERS_DIR" describe --tags --exact-match 2>/dev/null || true)" +if [[ "$TAG" != "v7.0.0" ]]; then + echo "superpowers checkout tag is '${TAG:-none}', expected v7.0.0" >&2 + exit 2 +fi +COMMIT="$(git -C "$SUPERPOWERS_DIR" rev-parse HEAD)" +mkdir -p "$EVENTS" +python3 - "$EVENTS/tool-version.json" "$TAG" "$COMMIT" << 'PY' +import json, sys +path, tag, commit = sys.argv[1:] +with open(path, "w", encoding="utf-8") as handle: + json.dump({"name": "superpowers", "version": tag, "commit": commit}, handle) + handle.write("\n") +PY + +PROMPT="$(cat "$TASK_MD") + +Implement it; run the visible tests. +Use the installed skills for this change." + +cd "$WORKDIR" +set +e +cursor_invoke "$EVENTS/agent-1.jsonl" "$PROMPT" --plugin-dir "$SUPERPOWERS_DIR" +rc=$? +set -e +exit "$rc" diff --git a/bench/e2e/harness/__init__.py b/bench/e2e/harness/__init__.py new file mode 100644 index 0000000..44a8ed3 --- /dev/null +++ b/bench/e2e/harness/__init__.py @@ -0,0 +1 @@ +"""Shared helpers for the end-to-end task-completion harness.""" diff --git a/bench/e2e/harness/metrics.py b/bench/e2e/harness/metrics.py new file mode 100644 index 0000000..f5bb1a5 --- /dev/null +++ b/bench/e2e/harness/metrics.py @@ -0,0 +1,155 @@ +"""Token parsing and aggregate statistics. Standard library only.""" + +from __future__ import annotations + +import json +from typing import Iterable, Iterator + + +USAGE_KEYS = ( + "inputTokens", + "outputTokens", + "cacheReadTokens", + "cacheWriteTokens", +) + +_ALIASES = { + "inputTokens": ("inputTokens", "input_tokens", "input"), + "outputTokens": ("outputTokens", "output_tokens", "output"), + "cacheReadTokens": ("cacheReadTokens", "cache_read_tokens", "cache_read"), + "cacheWriteTokens": ("cacheWriteTokens", "cache_write_tokens", "cache_write"), +} + + +def empty_usage() -> dict: + return {key: 0 for key in USAGE_KEYS} + + +def looks_like_usage(obj: object) -> bool: + if not isinstance(obj, dict): + return False + for names in _ALIASES.values(): + if any(name in obj for name in names): + return True + return False + + +def normalize_usage(obj: dict) -> dict: + usage = empty_usage() + for key, names in _ALIASES.items(): + for name in names: + if name in obj and obj[name] is not None: + usage[key] = int(obj[name]) + break + return usage + + +def usage_total(usage: dict) -> int: + """Full context: cursor-agent already subtracts cache tokens from inputTokens.""" + return sum(int(usage.get(key) or 0) for key in USAGE_KEYS) + + +def add_usage(left: dict, right: dict) -> dict: + out = empty_usage() + for key in USAGE_KEYS: + out[key] = int(left.get(key) or 0) + int(right.get(key) or 0) + return out + + +def iter_json_events(text: str) -> Iterator[dict]: + stripped = text.strip() + if not stripped: + return + try: + obj = json.loads(stripped) + except json.JSONDecodeError: + obj = None + if isinstance(obj, dict): + yield obj + return + if isinstance(obj, list): + for item in obj: + if isinstance(item, dict): + yield item + return + for line in text.splitlines(): + line = line.strip() + if not line: + continue + try: + item = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(item, dict): + yield item + + +def parse_agent_stream(text: str, source: str) -> dict: + """One cursor-agent invocation. Usage comes from the result event.""" + session_id = None + model = None + usage = None + saw_result = False + for event in iter_json_events(text): + if event.get("type") == "system" and event.get("subtype") == "init": + session_id = event.get("session_id") or session_id + model = event.get("model") or model + if event.get("type") == "result": + saw_result = True + session_id = event.get("session_id") or session_id + if looks_like_usage(event.get("usage")): + usage = normalize_usage(event["usage"]) + return { + "source": source, + "session_id": session_id, + "model": model, + "usage": usage if usage is not None else empty_usage(), + "missing_usage": usage is None, + "saw_result": saw_result, + } + + +def sum_calls(calls: Iterable[dict]) -> dict: + total = empty_usage() + for call in calls: + total = add_usage(total, call.get("usage") or empty_usage()) + total["total"] = usage_total(total) + return total + + +def quantile(values: list[float], p: float) -> float | None: + if not values: + return None + ordered = sorted(values) + if len(ordered) == 1: + return float(ordered[0]) + index = (len(ordered) - 1) * p + low = int(index) + high = min(low + 1, len(ordered) - 1) + frac = index - low + return float(ordered[low] * (1 - frac) + ordered[high] * frac) + + +def summarise(values: list[float]) -> dict: + if not values: + return { + "n": 0, + "median": None, + "q1": None, + "q3": None, + "iqr": None, + "min": None, + "max": None, + } + q1 = quantile(values, 0.25) + q3 = quantile(values, 0.75) + median = quantile(values, 0.5) + return { + "n": len(values), + "median": median, + "q1": q1, + "q3": q3, + "iqr": None if q1 is None or q3 is None else q3 - q1, + "min": min(values), + "max": max(values), + } diff --git a/bench/e2e/harness/openswarm_finish.py b/bench/e2e/harness/openswarm_finish.py new file mode 100755 index 0000000..3dd0ae1 --- /dev/null +++ b/bench/e2e/harness/openswarm_finish.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Read an openswarm run result, record executor token usage, merge the branch. + +Usage: openswarm_finish.py +Exit status is 0 when the branch was merged or the run already landed in place. +A missing branch after a failed run exits 1. This script does not call a model. +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from harness.metrics import iter_json_events, looks_like_usage, normalize_usage # noqa: E402 + + +def session_from_events(path: str | None) -> str | None: + if not path: + return None + source = Path(path) + if not source.is_file(): + return None + for event in iter_json_events(source.read_text(encoding="utf-8", errors="replace")): + if event.get("session_id"): + return event["session_id"] + return None + + +def _git(repo: Path, *args: str) -> subprocess.CompletedProcess: + return subprocess.run( + [ + "git", + "-c", + "user.email=bench-e2e@local", + "-c", + "user.name=bench-e2e", + *args, + ], + cwd=repo, + text=True, + capture_output=True, + check=False, + ) + + +def extract_result_object(text: str) -> dict | None: + decoder = json.JSONDecoder() + found = None + for index, char in enumerate(text): + if char != "{": + continue + try: + obj, _end = decoder.raw_decode(text[index:]) + except json.JSONDecodeError: + continue + if isinstance(obj, dict) and obj.get("id"): + found = obj + return found + + +def executor_calls(result: dict) -> list[dict]: + attempt_calls = [] + for attempt in result.get("attempts") or []: + meta = attempt.get("meta") or {} + usage = meta.get("usage") + if not looks_like_usage(usage): + continue + attempt_calls.append( + { + "source": "openswarm-result:attempts.meta.usage", + "session_id": meta.get("session_id") or meta.get("thread_id") or session_from_events(attempt.get("events")), + "model": attempt.get("model"), + "usage": normalize_usage(usage), + "missing_usage": False, + "saw_result": True, + } + ) + if attempt_calls: + return attempt_calls + usage = result.get("usage") + if looks_like_usage(usage): + return [ + { + "source": "openswarm-result:usage", + "session_id": result.get("session_id"), + "model": result.get("model"), + "usage": normalize_usage(usage), + "missing_usage": False, + "saw_result": True, + } + ] + return [] + + +def copy_raw_events(result: dict, events: Path) -> None: + for index, attempt in enumerate(result.get("attempts") or [], start=1): + path = attempt.get("events") + if not path: + continue + source = Path(path) + if source.is_file(): + shutil.copyfile(source, events / f"raw-executor-{index}.jsonl") + + +def merge_branch(repo: Path, result: dict) -> tuple[bool, str]: + run_id = result.get("id") + if not run_id: + return False, "result has no id" + branch = f"openswarm/{run_id}" + mode = ((result.get("workspace") or {}).get("mode")) or "" + exists = _git(repo, "rev-parse", "--verify", "--quiet", branch).returncode == 0 + if mode == "in-place": + return True, "executor ran in place; left the worktree as-is" + if not exists: + return False, f"branch {branch} does not exist" + _git(repo, "reset", "--hard", "HEAD") + _git(repo, "clean", "-fd") + merged = _git(repo, "merge", "--no-ff", "-m", f"merge {branch}", branch) + detail = (merged.stdout or "") + (merged.stderr or "") + return merged.returncode == 0, detail.strip() + + +def main(argv: list[str]) -> int: + if len(argv) != 4: + print("usage: openswarm_finish.py ", file=sys.stderr) + return 2 + stdout_path, events, workdir = Path(argv[1]), Path(argv[2]), Path(argv[3]) + text = stdout_path.read_text(encoding="utf-8", errors="replace") if stdout_path.is_file() else "" + result = extract_result_object(text) + if result is None: + run_dir = None + else: + run_dir = result.get("run_dir") + if run_dir: + result_path = Path(run_dir) / "result.json" + if result_path.is_file(): + shutil.copyfile(result_path, events / "openswarm-result.json") + try: + loaded = json.loads(result_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + loaded = None + if isinstance(loaded, dict): + result = loaded + calls = executor_calls(result or {}) + (events / "executor-usage.json").write_text( + json.dumps({"calls": calls}, indent=2) + "\n", + encoding="utf-8", + ) + if result: + copy_raw_events(result, events) + ok, detail = merge_branch(workdir, result) + (events / "merge.txt").write_text(detail + "\n", encoding="utf-8") + return 0 if ok else 1 + (events / "merge.txt").write_text("no openswarm result object on stdout\n", encoding="utf-8") + return 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/bench/e2e/report.py b/bench/e2e/report.py new file mode 100644 index 0000000..d014b49 --- /dev/null +++ b/bench/e2e/report.py @@ -0,0 +1,394 @@ +#!/usr/bin/env python3 +"""Aggregate a harness jsonl into summary.md, a CSV, and an SVG chart.""" + +from __future__ import annotations + +import argparse +import csv +import json +import sys +from pathlib import Path + +from harness.metrics import summarise + +ARM_ORDER = ["baseline", "superpowers", "openflow", "openswarm", "omo"] +ARM_COLORS = { + "baseline": "#3d5a80", + "superpowers": "#ee6c4d", + "openflow": "#2a9d8f", + "openswarm": "#e9c46a", + "omo": "#6d597a", +} + + +def load_rows(path: Path) -> tuple[dict | None, list[dict]]: + env = None + runs = [] + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + row = json.loads(line) + if row.get("type") == "environment": + env = row + continue + if row.get("skipped"): + continue + if row.get("type") not in (None, "run"): + continue + if "task" in row and "arm" in row: + runs.append(row) + return env, runs + + +def _num(row: dict, *keys, default=None): + cur = row + for key in keys: + if not isinstance(cur, dict) or key not in cur: + return default + cur = cur[key] + return cur + + +def series(rows: list[dict]) -> dict: + wall = [float(row["wall_s"]) for row in rows if row.get("wall_s") is not None] + success = [row for row in rows if row.get("success") is not None] + passed = sum(1 for row in success if row.get("success") is True) + + def token_values(key: str) -> list[float]: + values = [] + for row in rows: + if row.get("timed_out"): + continue # killed at the cap: no usage event, tokens unknown (not zero) + value = _num(row, "tokens", key) + if value is None: + continue + values.append(float(value)) + return values + + return { + "n": len(rows), + "success_n": len(success), + "success": passed, + "success_pct": (100.0 * passed / len(success)) if success else None, + "wall_s": summarise(wall), + "input": summarise(token_values("inputTokens")), + "output": summarise(token_values("outputTokens")), + "cache_read": summarise(token_values("cacheReadTokens")), + "total": summarise(token_values("total")), + } + + +def group(rows: list[dict], keyfn) -> list[tuple[str, dict]]: + buckets: dict[str, list[dict]] = {} + for row in rows: + buckets.setdefault(keyfn(row), []).append(row) + + def sort_key(name: str) -> tuple: + if name in ARM_ORDER: + return (0, ARM_ORDER.index(name), name) + return (1, 0, name) + + return [(name, series(buckets[name])) for name in sorted(buckets, key=sort_key)] + + +def fmt(value, digits=1) -> str: + if value is None: + return "—" + if isinstance(value, float): + return f"{value:.{digits}f}" + return str(value) + + +def stat_phrase(stat: dict, digits=1) -> str: + if not stat or stat.get("n", 0) == 0: + return "n=0" + return ( + f"median {fmt(stat['median'], digits)}, " + f"IQR {fmt(stat['iqr'], digits)} " + f"(q1 {fmt(stat['q1'], digits)}, q3 {fmt(stat['q3'], digits)}), " + f"min {fmt(stat['min'], digits)}, max {fmt(stat['max'], digits)}" + ) + + +def write_summary(path: Path, env: dict | None, by_arm, by_task, by_cell) -> None: + lines = ["# End-to-end task completion", ""] + if env: + lines.append("## Environment") + lines.append("") + lines.append(f"- Date: {env.get('date')}") + lines.append(f"- OS: {env.get('os')}") + lines.append(f"- CPU: {env.get('cpu')} ({env.get('cpu_count')} cores)") + lines.append(f"- Python: {env.get('python')}") + lines.append(f"- Node: {env.get('node')}") + lines.append(f"- cursor-agent: {env.get('cursor_agent')}") + commits = env.get("commits") or {} + lines.append(f"- superpowers: {commits.get('superpowers')} ({env.get('superpowers_dir')})") + lines.append(f"- openflow: {commits.get('openflow')} tarball={env.get('openflow_tgz')}") + lines.append(f"- openswarm: {commits.get('openswarm')} ({env.get('openswarm_dir')})") + lines.append(f"- Model flag: {env.get('model')}") + lines.append(f"- cursor-agent flags: {' '.join(env.get('cursor_flags') or [])}") + lines.append(f"- Per-run timeout: {env.get('run_timeout_s')}s") + lines.append(f"- Command: `{' '.join(env.get('command') or [])}`") + lines.append("") + lines.append("## Per arm") + lines.append("") + lines.append("| arm | success | wall_s | input tokens | output tokens | cache read | total tokens |") + lines.append("| --- | --- | --- | --- | --- | --- | --- |") + for name, stat in by_arm: + pct = "—" if stat["success_pct"] is None else f"{stat['success']}/{stat['success_n']} ({stat['success_pct']:.1f}%)" + lines.append( + "| {name} | {pct} | {wall} | {inp} | {out} | {cache} | {total} |".format( + name=name, + pct=pct, + wall=stat_phrase(stat["wall_s"]), + inp=stat_phrase(stat["input"], 0), + out=stat_phrase(stat["output"], 0), + cache=stat_phrase(stat["cache_read"], 0), + total=stat_phrase(stat["total"], 0), + ) + ) + lines.append("") + lines.append("## Per task") + lines.append("") + lines.append("| task | success | wall_s | total tokens |") + lines.append("| --- | --- | --- | --- |") + for name, stat in by_task: + pct = "—" if stat["success_pct"] is None else f"{stat['success']}/{stat['success_n']} ({stat['success_pct']:.1f}%)" + lines.append(f"| {name} | {pct} | {stat_phrase(stat['wall_s'])} | {stat_phrase(stat['total'], 0)} |") + lines.append("") + lines.append("## Per task and arm") + lines.append("") + lines.append("| task | arm | n | success | median wall_s | median total tokens |") + lines.append("| --- | --- | --- | --- | --- | --- |") + for name, stat in by_cell: + task, arm = name.split("\t", 1) + pct = "—" if stat["success_pct"] is None else f"{stat['success']}/{stat['success_n']} ({stat['success_pct']:.1f}%)" + lines.append( + f"| {task} | {arm} | {stat['n']} | {pct} | {fmt(stat['wall_s']['median'])} | {fmt(stat['total']['median'], 0)} |" + ) + lines.append("") + lines.append("Skipped arms (omo) are omitted. n is the number of finished, non-skipped runs in the group.") + lines.append("Token totals add input, output, cache read and cache write. cursor-agent reports inputTokens with cache tokens already removed.") + lines.append("") + path.write_text("\n".join(lines), encoding="utf-8") + + +def write_csv(path: Path, scope: str, rows: list[tuple[str, dict]]) -> None: + fieldnames = [ + "scope", + "id", + "n", + "success", + "success_n", + "success_pct", + "wall_median_s", + "wall_q1", + "wall_q3", + "wall_iqr", + "wall_min", + "wall_max", + "input_median", + "input_q1", + "input_q3", + "input_iqr", + "output_median", + "output_q1", + "output_q3", + "output_iqr", + "cache_read_median", + "cache_read_q1", + "cache_read_q3", + "cache_read_iqr", + "total_median", + "total_q1", + "total_q3", + "total_iqr", + ] + with path.open("w", encoding="utf-8", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=fieldnames) + writer.writeheader() + for name, stat in rows: + writer.writerow( + { + "scope": scope, + "id": name.replace("\t", "/"), + "n": stat["n"], + "success": stat["success"], + "success_n": stat["success_n"], + "success_pct": "" if stat["success_pct"] is None else f"{stat['success_pct']:.4f}", + "wall_median_s": stat["wall_s"]["median"], + "wall_q1": stat["wall_s"]["q1"], + "wall_q3": stat["wall_s"]["q3"], + "wall_iqr": stat["wall_s"]["iqr"], + "wall_min": stat["wall_s"]["min"], + "wall_max": stat["wall_s"]["max"], + "input_median": stat["input"]["median"], + "input_q1": stat["input"]["q1"], + "input_q3": stat["input"]["q3"], + "input_iqr": stat["input"]["iqr"], + "output_median": stat["output"]["median"], + "output_q1": stat["output"]["q1"], + "output_q3": stat["output"]["q3"], + "output_iqr": stat["output"]["iqr"], + "cache_read_median": stat["cache_read"]["median"], + "cache_read_q1": stat["cache_read"]["q1"], + "cache_read_q3": stat["cache_read"]["q3"], + "cache_read_iqr": stat["cache_read"]["iqr"], + "total_median": stat["total"]["median"], + "total_q1": stat["total"]["q1"], + "total_q3": stat["total"]["q3"], + "total_iqr": stat["total"]["iqr"], + } + ) + + +def _xml(text: str) -> str: + return ( + text.replace("&", "&") + .replace("<", "<") + .replace(">", ">") + .replace('"', """) + ) + + +def _bar_panel(title, labels, values, q1s, q3s, colors, x, y, w, h, y_label) -> list[str]: + parts = [ + f'', + f'{_xml(title)}', + f'{_xml(y_label)}', + ] + if not labels: + return parts + finite = [value for value in values + q3s if value is not None] + top = max(finite) if finite else 1 + if top <= 0: + top = 1 + top *= 1.15 + plot_x = x + 36 + plot_y = y + 36 + plot_w = w - 52 + plot_h = h - 78 + slot = plot_w / max(len(labels), 1) + bar_w = min(36, slot * 0.55) + for index, label in enumerate(labels): + value = values[index] or 0 + height = plot_h * (value / top) + bx = plot_x + index * slot + (slot - bar_w) / 2 + by = plot_y + plot_h - height + color = colors[index] + parts.append(f'') + q1 = q1s[index] + q3 = q3s[index] + if q1 is not None and q3 is not None: + y1 = plot_y + plot_h - plot_h * (q1 / top) + y3 = plot_y + plot_h - plot_h * (q3 / top) + cx = bx + bar_w / 2 + parts.append(f'') + parts.append(f'') + parts.append(f'') + parts.append( + f'{_xml(label)}' + ) + return parts + + +def write_svg(path: Path, by_arm: list[tuple[str, dict]]) -> None: + labels = [name for name, _stat in by_arm] + colors = [ARM_COLORS.get(name, "#666666") for name in labels] + success = [0 if stat["success_pct"] is None else stat["success_pct"] for _name, stat in by_arm] + minutes = [ + None if stat["wall_s"]["median"] is None else stat["wall_s"]["median"] / 60.0 + for _name, stat in by_arm + ] + minutes_q1 = [ + None if stat["wall_s"]["q1"] is None else stat["wall_s"]["q1"] / 60.0 + for _name, stat in by_arm + ] + minutes_q3 = [ + None if stat["wall_s"]["q3"] is None else stat["wall_s"]["q3"] / 60.0 + for _name, stat in by_arm + ] + totals = [stat["total"]["median"] for _name, stat in by_arm] + totals_q1 = [stat["total"]["q1"] for _name, stat in by_arm] + totals_q3 = [stat["total"]["q3"] for _name, stat in by_arm] + width, height = 980, 340 + parts = [ + f'', + f'', + 'Task completion by arm', + ] + panel_w, panel_h = 300, 280 + none = [None] * len(labels) + parts += _bar_panel("Success %", labels, success, none, none, colors, 16, 36, panel_w, panel_h, "percent") + parts += _bar_panel( + "Median minutes", + labels, + [0 if value is None else value for value in minutes], + minutes_q1, + minutes_q3, + colors, + 336, + 36, + panel_w, + panel_h, + "minutes, whisker = IQR", + ) + parts += _bar_panel( + "Median total tokens", + labels, + [0 if value is None else value for value in totals], + totals_q1, + totals_q3, + colors, + 656, + 36, + panel_w, + panel_h, + "tokens, whisker = IQR", + ) + parts.append("") + path.write_text("\n".join(parts) + "\n", encoding="utf-8") + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser(description="Aggregate harness jsonl") + parser.add_argument("jsonl", type=Path) + parser.add_argument("--out-dir", type=Path, default=None, help="defaults to the jsonl directory") + args = parser.parse_args(argv) + src = args.jsonl + if not src.is_file(): + raise SystemExit(f"not a file: {src}") + out_dir = args.out_dir or src.parent + out_dir.mkdir(parents=True, exist_ok=True) + env, runs = load_rows(src) + by_arm = group(runs, lambda row: row["arm"]) + by_task = group(runs, lambda row: row["task"]) + by_cell = group(runs, lambda row: f"{row['task']}\t{row['arm']}") + write_summary(out_dir / "summary.md", env, by_arm, by_task, by_cell) + write_csv_all(out_dir / "scores.csv", [("arm", by_arm), ("task", by_task), ("task_arm", by_cell)]) + write_svg(out_dir / "chart.svg", by_arm) + print(f"wrote {out_dir / 'summary.md'}") + print(f"wrote {out_dir / 'scores.csv'}") + print(f"wrote {out_dir / 'chart.svg'}") + return 0 + + +def write_csv_all(path: Path, groups: list[tuple[str, list]]) -> None: + with path.open("w", encoding="utf-8", newline="") as handle: + writer = None + for scope, rows in groups: + tmp = Path(str(path) + ".part") + write_csv(tmp, scope, rows) + text = tmp.read_text(encoding="utf-8").splitlines() + tmp.unlink() + if writer is None: + handle.write(text[0] + "\n") + writer = True + for line in text[1:]: + handle.write(line + "\n") + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bench/e2e/results/all-runs.jsonl b/bench/e2e/results/all-runs.jsonl new file mode 100644 index 0000000..4a97f71 --- /dev/null +++ b/bench/e2e/results/all-runs.jsonl @@ -0,0 +1,85 @@ +{"type": "environment", "date": "2026-10-17T12:20:16.509195+00:00", "cpu": "Intel(R) Xeon(R) Processor", "cpu_count": 8, "os": "Linux-6.12.94+-x86_64-with-glibc2.41", "python": "3.13.5", "node": "v22.23.3", "cursor_agent": "2026.10.16-e3cb207", "git": "git version 2.47.3", "commits": {"superpowers": "bb92a77741419a4ab5f06e711a283343f1ada0c3", "openflow": "58b5a3fa0899f6e639454f0e4a7247dec98f6986", "openswarm": "a48255a9cebafc48b0a7cdd39b6590a7dc193496"}, "openflow_tgz": null, "openswarm_dir": "", "superpowers_dir": "", "model": "grok-4.7-high", "cursor_flags": ["-p", "--trust", "--force", "--model", "grok-4.7-high", "--output-format", "stream-json"], "run_timeout_s": 1200, "command": ["/usr/bin/python3", "run.py", "--arms", "baseline,superpowers,openswarm", "--tasks", "all", "--repeats", "3", "--concurrency", "3", "--out", "results/e2e-2026-10-17.jsonl"], "arms": "baseline,superpowers,openswarm", "tasks": "all", "repeats": 3, "concurrency": 3, "out": "results/e2e-2026-10-17.jsonl"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:19:22.473848+00:00", "end": "2026-10-17T13:37:16.821402+00:00", "wall_s": 1074.334, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/agent-1.jsonl", "session_id": "43fa26d1-bfcc-4f61-8001-c7a02287d98f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 207571, "outputTokens": 33430, "cacheReadTokens": 5217152, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 207571, "outputTokens": 33430, "cacheReadTokens": 5217152, "cacheWriteTokens": 0, "total": 5458153}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.098, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.032882\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.127215\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.969402\n type: 'test'\n ...\n# Subtest: page 1 is the first window\nok 4 - page 1 is the first window\n ---\n duration_ms: 0.829964\n type: 'test'\n ...\n# Subtest: page 2 continues with no gap or overlap\nok 5 - page 2 continues with no gap or overlap\n ---\n duration_ms: 0.143545\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.100416\n type: 'test'\n ...\n# Subtest: an empty input needs zero pages\nok 7 - an empty input needs zero pages\n ---\n duration_ms: 0.189605\n type: 'test'\n ...\n# Subtest: a page past the end returns no items\nok 8 - a page past the end returns no items\n ---\n duration_ms: 0.111498\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.176549\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.329951\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.213587\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.133956\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.184804\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.487608\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.976132\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "specs/"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 31 +++++++++++++++++++++++++++++++\n 2 files changed, 32 insertions(+), 1 deletion(-)\n", "session_ids": ["43fa26d1-bfcc-4f61-8001-c7a02287d98f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131922-pagination-off-by-one-openflow-r1/agent-1.jsonl", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/arm-stderr.txt", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/arm-stdout.txt", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/measure-from.txt", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T131922-pagination-off-by-one-openflow-r1/repo", "baseline": "a94ee61d8a94d0e795bb8f36bc68eb509cd60b89", "measure_from": "ddc1ff59cb93946819e83bd77804fe7d800d9fd9"} +{"type": "run", "task": "date-range-parse", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:19:22.471732+00:00", "end": "2026-10-17T13:39:15.667559+00:00", "wall_s": 1193.18, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131922-date-range-parse-openflow-r1/agent-1.jsonl", "session_id": "bfc07f0a-c85e-4ba2-8fdb-a3bf208c49a2", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 19475, "outputTokens": 10881, "cacheReadTokens": 1932928, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 19475, "outputTokens": 10881, "cacheReadTokens": 1932928, "cacheWriteTokens": 0, "total": 1963284}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": [".as-xflow/config.json", ".xflow/GOAL.md", "HANDOFF.md", "daterange.py", "specs/README.md", "specs/architecture.md", "specs/changes/fix-inclusive-date-range/issue.md", "specs/changes/fix-inclusive-date-range/merge-product.md", "specs/changes/fix-inclusive-date-range/plan.md", "specs/changes/fix-inclusive-date-range/proposal.md", "specs/changes/fix-inclusive-date-range/spec_delta_review.json", "specs/changes/fix-inclusive-date-range/spec_delta_review.md", "specs/changes/fix-inclusive-date-range/status.json", "specs/changes/fix-inclusive-date-range/tasks.md", "specs/changes/fix-inclusive-date-range/tdd/green-0.json", "specs/changes/fix-inclusive-date-range/tdd/quality-0.json", "specs/changes/fix-inclusive-date-range/tdd/red-0.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/daterange.py", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/README.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/issue.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/merge-product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/plan.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/proposal.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/status.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/tasks.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/tests/test_daterange.py", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/daterange.py", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/README.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/issue.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/merge-product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/plan.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/proposal.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/status.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/tasks.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/tests/test_daterange.py", "specs/changes/fix-inclusive-date-range/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_daterange.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", "__pycache__/", "tests/__pycache__/"], "diff_stat": " .as-xflow/config.json | 2 +-\n .xflow/GOAL.md | 17 +\n HANDOFF.md | 92 +++\n daterange.py | 2 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/fix-inclusive-date-range/issue.md | 21 +\n .../fix-inclusive-date-range/merge-product.md | 8 +\n specs/changes/fix-inclusive-date-range/plan.md | 43 ++\n specs/changes/fix-inclusive-date-range/proposal.md | 37 ++\n .../spec_delta_review.json | 129 +++++\n .../fix-inclusive-date-range/spec_delta_review.md | 42 ++\n specs/changes/fix-inclusive-date-range/status.json | 638 +++++++++++++++++++++\n specs/changes/fix-inclusive-date-range/tasks.md | 5 +\n .../fix-inclusive-date-range/tdd/green-0.json | 39 ++\n .../fix-inclusive-date-range/tdd/quality-0.json | 78 +++\n .../fix-inclusive-date-range/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/daterange.py | 26 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../changes/fix-inclusive-date-range/issue.md | 21 +\n .../fix-inclusive-date-range/merge-product.md | 8 +\n .../specs/changes/fix-inclusive-date-range/plan.md | 43 ++\n .../changes/fix-inclusive-date-range/proposal.md | 37 ++\n .../changes/fix-inclusive-date-range/status.json | 611 ++++++++++++++++++++\n .../changes/fix-inclusive-date-range/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_daterange.py | 59 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/daterange.py | 26 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../changes/fix-inclusive-date-range/issue.md | 21 +\n .../fix-inclusive-date-range/merge-product.md | 8 +\n .../specs/changes/fix-inclusive-date-range/plan.md | 43 ++\n .../changes/fix-inclusive-date-range/proposal.md | 37 ++\n .../changes/fix-inclusive-date-range/status.json | 603 +++++++++++++++++++\n .../changes/fix-inclusive-date-range/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_daterange.py | 59 ++\n .../fix-inclusive-date-range/verify_proof.json | 12 +\n specs/product.md | 33 ++\n specs/workflow.md | 49 ++\n tests/test_daterange.py | 32 ++\n 60 files changed, 4817 insertions(+), 2 deletions(-)\n", "session_ids": ["bfc07f0a-c85e-4ba2-8fdb-a3bf208c49a2"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131922-date-range-parse-openflow-r1/agent-1.jsonl", "results/raw/20261017T131922-date-range-parse-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T131922-date-range-parse-openflow-r1/arm-stderr.txt", "results/raw/20261017T131922-date-range-parse-openflow-r1/arm-stdout.txt", "results/raw/20261017T131922-date-range-parse-openflow-r1/measure-from.txt", "results/raw/20261017T131922-date-range-parse-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T131922-date-range-parse-openflow-r1/repo", "baseline": "cbc723af8659954939bbe32437686de8892f98c9", "measure_from": "a36c11461c9af5dc5cf3a82ce7d36b60dd2918ac"} +{"type": "run", "task": "lru-eviction", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:19:22.474027+00:00", "end": "2026-10-17T13:39:22.793976+00:00", "wall_s": 1200.307, "exit_code": 124, "timed_out": true, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131922-lru-eviction-openflow-r1/agent-1.jsonl", "session_id": "3c0593b8-153d-462e-974f-d3f2b260e1b6", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.660907\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.151662\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.310872\n type: 'test'\n ...\n# Subtest: follows use order when a read refreshes a key\nok 4 - follows use order when a read refreshes a key\n ---\n duration_ms: 0.979771\n type: 'test'\n ...\n# Subtest: follows use order when an update refreshes a key without growing\nok 5 - follows use order when an update refreshes a key without growing\n ---\n duration_ms: 0.261745\n type: 'test'\n ...\n# Subtest: follows use order when a missing get leaves eviction unchanged\nok 6 - follows use order when a missing get leaves eviction unchanged\n ---\n duration_ms: 0.110892\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 58.152138\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.77425\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.144788\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.158931\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.120038\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.285345\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.633961\n", "stderr_tail": ""}, "files_changed": [".xflow/GOAL.md", "HANDOFF.md", "specs/README.md", "specs/architecture.md", "specs/changes/lru-use-order/issue.md", "specs/changes/lru-use-order/plan.md", "specs/changes/lru-use-order/proposal.md", "specs/changes/lru-use-order/spec_delta_review.json", "specs/changes/lru-use-order/spec_delta_review.md", "specs/changes/lru-use-order/status.json", "specs/changes/lru-use-order/tasks.md", "specs/changes/lru-use-order/tdd/green-0.json", "specs/changes/lru-use-order/tdd/quality-0.json", "specs/changes/lru-use-order/tdd/red-0.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/lru-use-order/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/package.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/README.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/issue.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/plan.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/proposal.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/status.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/tasks.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/product.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/src/lru.js", "specs/changes/lru-use-order/tdd/snapshots/green-0/test/lru.test.js", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/lru-use-order/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/package.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/README.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/issue.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/plan.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/proposal.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/status.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/tasks.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/product.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/src/lru.js", "specs/changes/lru-use-order/tdd/snapshots/red-0/test/lru.test.js", "specs/changes/lru-use-order/verify_proof.json", "specs/product.md", "specs/workflow.md", "src/lru.js", "test/lru.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json"], "diff_stat": " .xflow/GOAL.md | 17 +\n HANDOFF.md | 92 +++\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/lru-use-order/issue.md | 5 +\n specs/changes/lru-use-order/plan.md | 52 ++\n specs/changes/lru-use-order/proposal.md | 42 ++\n specs/changes/lru-use-order/spec_delta_review.json | 130 +++++\n specs/changes/lru-use-order/spec_delta_review.md | 40 ++\n specs/changes/lru-use-order/status.json | 619 +++++++++++++++++++++\n specs/changes/lru-use-order/tasks.md | 5 +\n specs/changes/lru-use-order/tdd/green-0.json | 39 ++\n specs/changes/lru-use-order/tdd/quality-0.json | 74 +++\n specs/changes/lru-use-order/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/package.json | 9 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../green-0/specs/changes/lru-use-order/issue.md | 5 +\n .../green-0/specs/changes/lru-use-order/plan.md | 52 ++\n .../specs/changes/lru-use-order/proposal.md | 42 ++\n .../specs/changes/lru-use-order/status.json | 611 ++++++++++++++++++++\n .../green-0/specs/changes/lru-use-order/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../lru-use-order/tdd/snapshots/green-0/src/lru.js | 34 ++\n .../tdd/snapshots/green-0/test/lru.test.js | 56 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../lru-use-order/tdd/snapshots/red-0/package.json | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/lru-use-order/issue.md | 5 +\n .../red-0/specs/changes/lru-use-order/plan.md | 52 ++\n .../red-0/specs/changes/lru-use-order/proposal.md | 42 ++\n .../red-0/specs/changes/lru-use-order/status.json | 603 ++++++++++++++++++++\n .../red-0/specs/changes/lru-use-order/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../lru-use-order/tdd/snapshots/red-0/src/lru.js | 27 +\n .../tdd/snapshots/red-0/test/lru.test.js | 56 ++\n specs/changes/lru-use-order/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n src/lru.js | 11 +-\n test/lru.test.js | 33 ++\n 58 files changed, 4777 insertions(+), 2 deletions(-)\n", "session_ids": ["3c0593b8-153d-462e-974f-d3f2b260e1b6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131922-lru-eviction-openflow-r1/agent-1.jsonl", "results/raw/20261017T131922-lru-eviction-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T131922-lru-eviction-openflow-r1/arm-stderr.txt", "results/raw/20261017T131922-lru-eviction-openflow-r1/arm-stdout.txt", "results/raw/20261017T131922-lru-eviction-openflow-r1/measure-from.txt", "results/raw/20261017T131922-lru-eviction-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T131922-lru-eviction-openflow-r1/repo", "baseline": "767c50844040360af10d56fc02591bc13d199d47", "measure_from": "3adcd616beb12c969ecbb284a78ddefaefa5fd5d"} +{"type": "run", "task": "path-normalise", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:37:16.835864+00:00", "end": "2026-10-17T13:51:10.778437+00:00", "wall_s": 833.929, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133716-path-normalise-openflow-r1/agent-1.jsonl", "session_id": "0c1ce27a-d222-407c-ac62-70af85a99577", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 149144, "outputTokens": 37704, "cacheReadTokens": 5061504, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 149144, "outputTokens": 37704, "cacheReadTokens": 5061504, "cacheWriteTokens": 0, "total": 5248352}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.126, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 1.050767\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.168249\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.108992\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.512732\n type: 'test'\n ...\n# Subtest: pops a parent segment\nok 5 - pops a parent segment\n ---\n duration_ms: 0.238133\n type: 'test'\n ...\n# Subtest: keeps a parent that leaves the relative root\nok 6 - keeps a parent that leaves the relative root\n ---\n duration_ms: 0.179071\n type: 'test'\n ...\n# Subtest: does not climb above an absolute root\nok 7 - does not climb above an absolute root\n ---\n duration_ms: 0.190836\n type: 'test'\n ...\n# Subtest: drops empty segments from doubled or trailing slashes\nok 8 - drops empty segments from doubled or trailing slashes\n ---\n duration_ms: 0.106447\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 72.703467\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.125, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.515012\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.277121\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.16489\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.150882\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.350284\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 83.899682\n", "stderr_tail": ""}, "files_changed": ["HANDOFF.md", "specs/README.md", "specs/architecture.md", "specs/changes/normalise-parent-segments/issue.md", "specs/changes/normalise-parent-segments/plan.md", "specs/changes/normalise-parent-segments/proposal.md", "specs/changes/normalise-parent-segments/spec_delta_review.json", "specs/changes/normalise-parent-segments/spec_delta_review.md", "specs/changes/normalise-parent-segments/status.json", "specs/changes/normalise-parent-segments/tasks.md", "specs/changes/normalise-parent-segments/tdd/green-0.json", "specs/changes/normalise-parent-segments/tdd/quality-0.json", "specs/changes/normalise-parent-segments/tdd/red-0.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/package.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/README.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/issue.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/plan.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/proposal.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/status.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/tasks.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/product.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/src/normalise.js", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/test/normalise.test.js", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/package.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/README.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/issue.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/plan.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/proposal.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/status.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/tasks.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/product.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/src/normalise.js", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/test/normalise.test.js", "specs/changes/normalise-parent-segments/verify_proof.json", "specs/product.md", "specs/workflow.md", "src/normalise.js", "test/normalise.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/"], "diff_stat": " HANDOFF.md | 92 +++\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/normalise-parent-segments/issue.md | 21 +\n specs/changes/normalise-parent-segments/plan.md | 46 ++\n .../changes/normalise-parent-segments/proposal.md | 39 ++\n .../spec_delta_review.json | 111 ++++\n .../normalise-parent-segments/spec_delta_review.md | 39 ++\n .../changes/normalise-parent-segments/status.json | 638 +++++++++++++++++++++\n specs/changes/normalise-parent-segments/tasks.md | 5 +\n .../normalise-parent-segments/tdd/green-0.json | 39 ++\n .../normalise-parent-segments/tdd/quality-0.json | 74 +++\n .../normalise-parent-segments/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 66 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/package.json | 9 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../changes/normalise-parent-segments/issue.md | 21 +\n .../changes/normalise-parent-segments/plan.md | 46 ++\n .../changes/normalise-parent-segments/proposal.md | 39 ++\n .../changes/normalise-parent-segments/status.json | 611 ++++++++++++++++++++\n .../changes/normalise-parent-segments/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/src/normalise.js | 23 +\n .../tdd/snapshots/green-0/test/normalise.test.js | 41 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 51 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/package.json | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../changes/normalise-parent-segments/issue.md | 21 +\n .../changes/normalise-parent-segments/plan.md | 46 ++\n .../changes/normalise-parent-segments/proposal.md | 39 ++\n .../changes/normalise-parent-segments/status.json | 603 +++++++++++++++++++\n .../changes/normalise-parent-segments/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/src/normalise.js | 15 +\n .../tdd/snapshots/red-0/test/normalise.test.js | 41 ++\n .../normalise-parent-segments/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n src/normalise.js | 10 +-\n test/normalise.test.js | 22 +\n 57 files changed, 4724 insertions(+), 1 deletion(-)\n", "session_ids": ["0c1ce27a-d222-407c-ac62-70af85a99577"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T133716-path-normalise-openflow-r1/agent-1.jsonl", "results/raw/20261017T133716-path-normalise-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T133716-path-normalise-openflow-r1/arm-stderr.txt", "results/raw/20261017T133716-path-normalise-openflow-r1/arm-stdout.txt", "results/raw/20261017T133716-path-normalise-openflow-r1/measure-from.txt", "results/raw/20261017T133716-path-normalise-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T133716-path-normalise-openflow-r1/repo", "baseline": "837e8ffcb353dabbca72e61c9cffcf97f1664943", "measure_from": "ec3ac64afd61a8865335124998730d496ac7fe01"} +{"type": "run", "task": "retry-backoff", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:39:15.680582+00:00", "end": "2026-10-17T13:54:22.720224+00:00", "wall_s": 907.026, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133915-retry-backoff-openflow-r1/agent-1.jsonl", "session_id": "08f72a12-61e9-4102-9e4d-4608145ae96c", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 198345, "outputTokens": 40814, "cacheReadTokens": 5839744, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 198345, "outputTokens": 40814, "cacheReadTokens": 5839744, "cacheWriteTokens": 0, "total": 6078903}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.042, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": [".as-xflow/config.json", "HANDOFF.md", "retry.py", "specs/README.md", "specs/architecture.md", "specs/changes/retry-backoff/issue.md", "specs/changes/retry-backoff/plan.md", "specs/changes/retry-backoff/proposal.md", "specs/changes/retry-backoff/spec_delta_review.json", "specs/changes/retry-backoff/spec_delta_review.md", "specs/changes/retry-backoff/status.json", "specs/changes/retry-backoff/tasks.md", "specs/changes/retry-backoff/tdd/green-0.json", "specs/changes/retry-backoff/tdd/quality-0.json", "specs/changes/retry-backoff/tdd/red-0.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/retry-backoff/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/retry.py", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/README.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/issue.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/plan.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/proposal.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/status.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/tasks.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/product.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/tests/test_retry.py", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/retry-backoff/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/retry.py", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/README.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/issue.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/plan.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/proposal.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/status.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/tasks.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/product.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/tests/test_retry.py", "specs/changes/retry-backoff/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_retry.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " .as-xflow/config.json | 2 +-\n HANDOFF.md | 92 +++\n retry.py | 9 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/retry-backoff/issue.md | 21 +\n specs/changes/retry-backoff/plan.md | 31 +\n specs/changes/retry-backoff/proposal.md | 37 ++\n specs/changes/retry-backoff/spec_delta_review.json | 99 ++++\n specs/changes/retry-backoff/spec_delta_review.md | 39 ++\n specs/changes/retry-backoff/status.json | 638 +++++++++++++++++++++\n specs/changes/retry-backoff/tasks.md | 5 +\n specs/changes/retry-backoff/tdd/green-0.json | 38 ++\n specs/changes/retry-backoff/tdd/quality-0.json | 75 +++\n specs/changes/retry-backoff/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../retry-backoff/tdd/snapshots/green-0/retry.py | 14 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../green-0/specs/changes/retry-backoff/issue.md | 21 +\n .../green-0/specs/changes/retry-backoff/plan.md | 31 +\n .../specs/changes/retry-backoff/proposal.md | 37 ++\n .../specs/changes/retry-backoff/status.json | 611 ++++++++++++++++++++\n .../green-0/specs/changes/retry-backoff/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_retry.py | 76 +++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../retry-backoff/tdd/snapshots/red-0/retry.py | 7 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/retry-backoff/issue.md | 21 +\n .../red-0/specs/changes/retry-backoff/plan.md | 31 +\n .../red-0/specs/changes/retry-backoff/proposal.md | 37 ++\n .../red-0/specs/changes/retry-backoff/status.json | 603 +++++++++++++++++++\n .../red-0/specs/changes/retry-backoff/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_retry.py | 76 +++\n specs/changes/retry-backoff/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_retry.py | 50 ++\n 56 files changed, 4715 insertions(+), 2 deletions(-)\n", "session_ids": ["08f72a12-61e9-4102-9e4d-4608145ae96c"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T133915-retry-backoff-openflow-r1/agent-1.jsonl", "results/raw/20261017T133915-retry-backoff-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T133915-retry-backoff-openflow-r1/arm-stderr.txt", "results/raw/20261017T133915-retry-backoff-openflow-r1/arm-stdout.txt", "results/raw/20261017T133915-retry-backoff-openflow-r1/measure-from.txt", "results/raw/20261017T133915-retry-backoff-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T133915-retry-backoff-openflow-r1/repo", "baseline": "bfbe8f47eed7677d1e602614786a10a397be8381", "measure_from": "906fc5980859b7a7067980e23a268b41e171fb3d"} +{"type": "run", "task": "semver-compare", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:39:22.806044+00:00", "end": "2026-10-17T13:55:03.872261+00:00", "wall_s": 941.051, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133922-semver-compare-openflow-r1/agent-1.jsonl", "session_id": "8dbf6589-8358-45da-afd9-4689d754b166", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 147069, "outputTokens": 43160, "cacheReadTokens": 5209984, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 147069, "outputTokens": 43160, "cacheReadTokens": 5209984, "cacheWriteTokens": 0, "total": 5400213}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.11, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.208397\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.266172\n type: 'test'\n ...\n# Subtest: minor and patch decide order after major\nok 3 - minor and patch decide order after major\n ---\n duration_ms: 1.204727\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same version without one\nok 4 - a pre-release is less than the same version without one\n ---\n duration_ms: 0.310276\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare by semver rules\nok 5 - pre-release identifiers compare by semver rules\n ---\n duration_ms: 0.243688\n type: 'test'\n ...\n# Subtest: numeric pre-release identifiers are less than non-numeric ones\nok 6 - numeric pre-release identifiers are less than non-numeric ones\n ---\n duration_ms: 0.130863\n type: 'test'\n ...\n# Subtest: a shorter pre-release list that is a prefix sorts first\nok 7 - a shorter pre-release list that is a prefix sorts first\n ---\n duration_ms: 0.163125\n type: 'test'\n ...\n# Subtest: build metadata is ignored\nok 8 - build metadata is ignored\n ---\n duration_ms: 0.144565\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 9 - invalid versions throw\n ---\n duration_ms: 0.864487\n type: 'test'\n ...\n1..9\n# tests 9\n# suites 0\n# pass 9\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 70.154733\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.88688\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.297552\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.152928\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.034996\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.110558\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.379523\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 68.905659\n", "stderr_tail": ""}, "files_changed": [".as-xflow/config.json", "HANDOFF.md", "specs/README.md", "specs/architecture.md", "specs/changes/compare-semver/issue.md", "specs/changes/compare-semver/plan.md", "specs/changes/compare-semver/proposal.md", "specs/changes/compare-semver/spec_delta_review.json", "specs/changes/compare-semver/spec_delta_review.md", "specs/changes/compare-semver/status.json", "specs/changes/compare-semver/tasks.md", "specs/changes/compare-semver/tdd/green-0.json", "specs/changes/compare-semver/tdd/quality-0.json", "specs/changes/compare-semver/tdd/red-0.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/compare-semver/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/compare-semver/tdd/snapshots/green-0/package.json", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/README.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/issue.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/plan.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/proposal.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/status.json", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/tasks.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/product.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/compare-semver/tdd/snapshots/green-0/src/semver.js", "specs/changes/compare-semver/tdd/snapshots/green-0/test/semver.test.js", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/compare-semver/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/compare-semver/tdd/snapshots/red-0/package.json", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/README.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/issue.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/plan.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/proposal.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/status.json", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/tasks.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/product.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/compare-semver/tdd/snapshots/red-0/src/semver.js", "specs/changes/compare-semver/tdd/snapshots/red-0/test/semver.test.js", "specs/changes/compare-semver/verify_proof.json", "specs/product.md", "specs/workflow.md", "src/semver.js", "test/semver.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/"], "diff_stat": " .as-xflow/config.json | 2 +-\n HANDOFF.md | 92 +++\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/compare-semver/issue.md | 25 +\n specs/changes/compare-semver/plan.md | 48 ++\n specs/changes/compare-semver/proposal.md | 38 ++\n .../changes/compare-semver/spec_delta_review.json | 108 ++++\n specs/changes/compare-semver/spec_delta_review.md | 38 ++\n specs/changes/compare-semver/status.json | 638 +++++++++++++++++++++\n specs/changes/compare-semver/tasks.md | 5 +\n specs/changes/compare-semver/tdd/green-0.json | 39 ++\n specs/changes/compare-semver/tdd/quality-0.json | 75 +++\n specs/changes/compare-semver/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/package.json | 9 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../green-0/specs/changes/compare-semver/issue.md | 25 +\n .../green-0/specs/changes/compare-semver/plan.md | 48 ++\n .../specs/changes/compare-semver/proposal.md | 38 ++\n .../specs/changes/compare-semver/status.json | 611 ++++++++++++++++++++\n .../green-0/specs/changes/compare-semver/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/src/semver.js | 63 ++\n .../tdd/snapshots/green-0/test/semver.test.js | 63 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/package.json | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/compare-semver/issue.md | 25 +\n .../red-0/specs/changes/compare-semver/plan.md | 48 ++\n .../red-0/specs/changes/compare-semver/proposal.md | 38 ++\n .../red-0/specs/changes/compare-semver/status.json | 603 +++++++++++++++++++\n .../red-0/specs/changes/compare-semver/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/src/semver.js | 22 +\n .../tdd/snapshots/red-0/test/semver.test.js | 63 ++\n specs/changes/compare-semver/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n src/semver.js | 49 +-\n test/semver.test.js | 51 ++\n 58 files changed, 4885 insertions(+), 5 deletions(-)\n", "session_ids": ["8dbf6589-8358-45da-afd9-4689d754b166"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T133922-semver-compare-openflow-r1/agent-1.jsonl", "results/raw/20261017T133922-semver-compare-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T133922-semver-compare-openflow-r1/arm-stderr.txt", "results/raw/20261017T133922-semver-compare-openflow-r1/arm-stdout.txt", "results/raw/20261017T133922-semver-compare-openflow-r1/measure-from.txt", "results/raw/20261017T133922-semver-compare-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T133922-semver-compare-openflow-r1/repo", "baseline": "0bfba3ff8260b038cac8980e696ee32487e44103", "measure_from": "708440a9f908311ebe34a61946dd4be188e9e05a"} +{"type": "run", "task": "config-deep-merge", "arm": "openflow", "repeat": 1, "start": "2026-10-17T12:56:28.612758+00:00", "end": "2026-10-17T13:09:22.849876+00:00", "wall_s": 774.225, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125628-config-deep-merge-openflow-r1/agent-1.jsonl", "session_id": "14a918aa-14a5-496e-a09f-289447125d27", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 261464, "outputTokens": 34664, "cacheReadTokens": 4465792, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 261464, "outputTokens": 34664, "cacheReadTokens": 4465792, "cacheWriteTokens": 0, "total": 4761920}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["HANDOFF.md", "deepmerge.py", "specs/README.md", "specs/architecture.md", "specs/changes/deep-merge-nested-dicts/issue.md", "specs/changes/deep-merge-nested-dicts/plan.md", "specs/changes/deep-merge-nested-dicts/proposal.md", "specs/changes/deep-merge-nested-dicts/spec_delta_review.json", "specs/changes/deep-merge-nested-dicts/spec_delta_review.md", "specs/changes/deep-merge-nested-dicts/status.json", "specs/changes/deep-merge-nested-dicts/tasks.md", "specs/changes/deep-merge-nested-dicts/tdd/green-0.json", "specs/changes/deep-merge-nested-dicts/tdd/quality-0.json", "specs/changes/deep-merge-nested-dicts/tdd/red-0.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/deepmerge.py", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/README.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/issue.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/plan.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/proposal.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/status.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/tasks.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/product.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/deepmerge.py", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/README.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/issue.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/plan.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/proposal.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/status.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/tasks.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/product.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested-dicts/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_deepmerge.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " HANDOFF.md | 92 +++\n deepmerge.py | 20 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/deep-merge-nested-dicts/issue.md | 23 +\n specs/changes/deep-merge-nested-dicts/plan.md | 47 ++\n specs/changes/deep-merge-nested-dicts/proposal.md | 40 ++\n .../deep-merge-nested-dicts/spec_delta_review.json | 112 ++++\n .../deep-merge-nested-dicts/spec_delta_review.md | 39 ++\n specs/changes/deep-merge-nested-dicts/status.json | 639 +++++++++++++++++++++\n specs/changes/deep-merge-nested-dicts/tasks.md | 5 +\n .../deep-merge-nested-dicts/tdd/green-0.json | 38 ++\n .../deep-merge-nested-dicts/tdd/quality-0.json | 74 +++\n .../changes/deep-merge-nested-dicts/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/deepmerge.py | 23 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested-dicts/issue.md | 23 +\n .../specs/changes/deep-merge-nested-dicts/plan.md | 47 ++\n .../changes/deep-merge-nested-dicts/proposal.md | 40 ++\n .../changes/deep-merge-nested-dicts/status.json | 612 ++++++++++++++++++++\n .../specs/changes/deep-merge-nested-dicts/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_deepmerge.py | 90 +++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/deepmerge.py | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested-dicts/issue.md | 23 +\n .../specs/changes/deep-merge-nested-dicts/plan.md | 47 ++\n .../changes/deep-merge-nested-dicts/proposal.md | 40 ++\n .../changes/deep-merge-nested-dicts/status.json | 604 +++++++++++++++++++\n .../specs/changes/deep-merge-nested-dicts/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_deepmerge.py | 90 +++\n .../deep-merge-nested-dicts/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_deepmerge.py | 61 ++\n 55 files changed, 4851 insertions(+), 3 deletions(-)\n", "session_ids": ["14a918aa-14a5-496e-a09f-289447125d27"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T125628-config-deep-merge-openflow-r1/agent-1.jsonl", "results/raw/20261017T125628-config-deep-merge-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T125628-config-deep-merge-openflow-r1/arm-stderr.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r1/arm-stdout.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r1/measure-from.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T125628-config-deep-merge-openflow-r1/repo", "baseline": "3b4fd8f0310c9d7bf62461d0a36c7522a7dd57ac", "measure_from": "ce0ed8b156af6543474eae4b1a3ec89284c8603f"} +{"type": "run", "task": "config-deep-merge", "arm": "openflow", "repeat": 3, "start": "2026-10-17T12:56:28.622913+00:00", "end": "2026-10-17T13:11:52.288586+00:00", "wall_s": 923.652, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125628-config-deep-merge-openflow-r3/agent-1.jsonl", "session_id": "08796f0e-7b0a-4d4f-a1a9-484fa5440eff", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 187857, "outputTokens": 39903, "cacheReadTokens": 5113600, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 187857, "outputTokens": 39903, "cacheReadTokens": 5113600, "cacheWriteTokens": 0, "total": 5341360}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.054, "stdout_tail": "", "stderr_tail": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.057, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": [".xflow/GOAL.md", "HANDOFF.md", "deepmerge.py", "specs/README.md", "specs/architecture.md", "specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/spec_delta_review.json", "specs/changes/deep-merge-nested/spec_delta_review.md", "specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/green-0.json", "specs/changes/deep-merge-nested/tdd/quality-0.json", "specs/changes/deep-merge-nested/tdd/red-0.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_deepmerge.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", "__pycache__/", "tests/__pycache__/"], "diff_stat": " .xflow/GOAL.md | 17 +\n HANDOFF.md | 92 +++\n deepmerge.py | 19 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/deep-merge-nested/issue.md | 23 +\n specs/changes/deep-merge-nested/plan.md | 53 ++\n specs/changes/deep-merge-nested/proposal.md | 38 ++\n .../deep-merge-nested/spec_delta_review.json | 109 ++++\n .../changes/deep-merge-nested/spec_delta_review.md | 39 ++\n specs/changes/deep-merge-nested/status.json | 637 +++++++++++++++++++++\n specs/changes/deep-merge-nested/tasks.md | 5 +\n specs/changes/deep-merge-nested/tdd/green-0.json | 38 ++\n specs/changes/deep-merge-nested/tdd/quality-0.json | 74 +++\n specs/changes/deep-merge-nested/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/deepmerge.py | 22 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested/issue.md | 23 +\n .../specs/changes/deep-merge-nested/plan.md | 53 ++\n .../specs/changes/deep-merge-nested/proposal.md | 38 ++\n .../specs/changes/deep-merge-nested/status.json | 610 ++++++++++++++++++++\n .../specs/changes/deep-merge-nested/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_deepmerge.py | 59 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/deepmerge.py | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/deep-merge-nested/issue.md | 23 +\n .../red-0/specs/changes/deep-merge-nested/plan.md | 53 ++\n .../specs/changes/deep-merge-nested/proposal.md | 38 ++\n .../specs/changes/deep-merge-nested/status.json | 602 +++++++++++++++++++\n .../red-0/specs/changes/deep-merge-nested/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_deepmerge.py | 59 ++\n specs/changes/deep-merge-nested/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_deepmerge.py | 30 +\n 56 files changed, 4776 insertions(+), 3 deletions(-)\n", "session_ids": ["08796f0e-7b0a-4d4f-a1a9-484fa5440eff"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T125628-config-deep-merge-openflow-r3/agent-1.jsonl", "results/raw/20261017T125628-config-deep-merge-openflow-r3/agent-1.jsonl.err", "results/raw/20261017T125628-config-deep-merge-openflow-r3/arm-stderr.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r3/arm-stdout.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r3/measure-from.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r3/tool-version.json"], "workdir": "tmp/20261017T125628-config-deep-merge-openflow-r3/repo", "baseline": "3b4fd8f0310c9d7bf62461d0a36c7522a7dd57ac", "measure_from": "8c713853095d7cc4f6a473495c193fba9d48ca67"} +{"type": "run", "task": "config-deep-merge", "arm": "openflow", "repeat": 2, "start": "2026-10-17T12:56:28.601788+00:00", "end": "2026-10-17T13:12:16.356403+00:00", "wall_s": 947.74, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125628-config-deep-merge-openflow-r2/agent-1.jsonl", "session_id": "31c1ee9e-689f-448a-b04b-9ea6de9e0f05", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 203893, "outputTokens": 40424, "cacheReadTokens": 4927872, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 203893, "outputTokens": 40424, "cacheReadTokens": 4927872, "cacheWriteTokens": 0, "total": 5172189}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.041, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["HANDOFF.md", "deepmerge.py", "specs/README.md", "specs/architecture.md", "specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/spec_delta_review.json", "specs/changes/deep-merge-nested/spec_delta_review.md", "specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/green-0.json", "specs/changes/deep-merge-nested/tdd/quality-0.json", "specs/changes/deep-merge-nested/tdd/red-0.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_deepmerge.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " HANDOFF.md | 92 +++\n deepmerge.py | 27 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/deep-merge-nested/issue.md | 21 +\n specs/changes/deep-merge-nested/plan.md | 52 ++\n specs/changes/deep-merge-nested/proposal.md | 41 ++\n .../deep-merge-nested/spec_delta_review.json | 117 ++++\n .../changes/deep-merge-nested/spec_delta_review.md | 38 ++\n specs/changes/deep-merge-nested/status.json | 637 +++++++++++++++++++++\n specs/changes/deep-merge-nested/tasks.md | 6 +\n specs/changes/deep-merge-nested/tdd/green-0.json | 38 ++\n specs/changes/deep-merge-nested/tdd/quality-0.json | 74 +++\n specs/changes/deep-merge-nested/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 66 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/deepmerge.py | 30 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested/issue.md | 21 +\n .../specs/changes/deep-merge-nested/plan.md | 52 ++\n .../specs/changes/deep-merge-nested/proposal.md | 41 ++\n .../specs/changes/deep-merge-nested/status.json | 610 ++++++++++++++++++++\n .../specs/changes/deep-merge-nested/tasks.md | 6 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_deepmerge.py | 54 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 51 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/deepmerge.py | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/deep-merge-nested/issue.md | 21 +\n .../red-0/specs/changes/deep-merge-nested/plan.md | 52 ++\n .../specs/changes/deep-merge-nested/proposal.md | 41 ++\n .../specs/changes/deep-merge-nested/status.json | 602 +++++++++++++++++++\n .../red-0/specs/changes/deep-merge-nested/tasks.md | 6 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_deepmerge.py | 54 ++\n specs/changes/deep-merge-nested/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_deepmerge.py | 25 +\n 55 files changed, 4778 insertions(+), 3 deletions(-)\n", "session_ids": ["31c1ee9e-689f-448a-b04b-9ea6de9e0f05"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T125628-config-deep-merge-openflow-r2/agent-1.jsonl", "results/raw/20261017T125628-config-deep-merge-openflow-r2/agent-1.jsonl.err", "results/raw/20261017T125628-config-deep-merge-openflow-r2/arm-stderr.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r2/arm-stdout.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r2/measure-from.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r2/tool-version.json"], "workdir": "tmp/20261017T125628-config-deep-merge-openflow-r2/repo", "baseline": "3b4fd8f0310c9d7bf62461d0a36c7522a7dd57ac", "measure_from": "7c31720257ea6ebebdd6747fb1e5d7c6d93700a4"} +{"type": "run", "task": "csv-quoting", "arm": "openflow", "repeat": 2, "start": "2026-10-17T13:11:52.300988+00:00", "end": "2026-10-17T13:28:55.197306+00:00", "wall_s": 1022.876, "exit_code": 1, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131152-csv-quoting-openflow-r2/agent-1.jsonl", "session_id": "bf226aa6-ad8f-4a7b-8d16-e4391e775172", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": false, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 1, "passed": false, "timed_out": false, "wall_s": 0.124, "stdout_tail": "", "stderr_tail": "====================================\nFAIL: test_quotes_comma_quote_and_newline (tests.test_csvquote.VisibleCsvTests.test_quotes_comma_quote_and_newline)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/tests/test_csvquote.py\", line 24, in test_quotes_comma_quote_and_newline\n self.assertEqual(text, '\"a,b\",\"say \"\"hi\"\"\",\"line1\\nline2\",\\n')\n ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: 'a,b,say \"hi\",line1\\nline2,\\n' != '\"a,b\",\"say \"\"hi\"\"\",\"line1\\nline2\",\\n'\n- a,b,say \"hi\",line1\n+ \"a,b\",\"say \"\"hi\"\"\",\"line1\n? + + + + ++ +\n- line2,\n+ line2\",\n? +\n\n\n======================================================================\nFAIL: test_special_roundtrip (tests.test_csvquote.VisibleCsvTests.test_special_roundtrip)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/tests/test_csvquote.py\", line 41, in test_special_roundtrip\n self.assertEqual(parse_csv(format_csv(rows)), rows)\n ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: Lists differ: [['na[23 chars]hello', ' world'], ['grace', 'quote \"here\"'], [51 chars]'c']] != [['na[23 chars]hello, world'], ['grace', 'quote \"here\"'], ['l[44 chars]'c']]\n\nFirst differing element 1:\n['ada', 'hello', ' world']\n['ada', 'hello, world']\n\nFirst list contains 1 additional elements.\nFirst extra element 6:\n['a', '', 'c']\n\n [['name', 'note'],\n- ['ada', 'hello', ' world'],\n? - --\n\n+ ['ada', 'hello, world'],\n ['grace', 'quote \"here\"'],\n- ['linus', 'line1'],\n+ ['linus', 'line1\\nline2'],\n? +++++++\n\n- ['line2'],\n ['', ''],\n ['a', '', 'c']]\n\n----------------------------------------------------------------------\nRan 6 tests in 0.002s\n\nFAILED (failures=3)\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 1, "passed": false, "timed_out": false, "wall_s": 0.144, "stdout_tail": "", "stderr_tail": "^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: Lists differ: [['city', 'note'], ['Paris', 'left', ' right']] != [['city', 'note'], ['Paris', 'left, right']]\n\nFirst differing element 1:\n['Paris', 'left', ' right']\n['Paris', 'left, right']\n\n- [['city', 'note'], ['Paris', 'left', ' right']]\n? - --\n\n+ [['city', 'note'], ['Paris', 'left, right']]\n\n======================================================================\nFAIL: test_newline_inside_a_field_stays_one_row (_hidden_acceptance.test_csvquote.HiddenCsvTests.test_newline_inside_a_field_stays_one_row)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/_hidden_acceptance/test_csvquote.py\", line 24, in test_newline_inside_a_field_stays_one_row\n self.assertEqual(parsed, rows)\n ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^\nAssertionError: Lists differ: [['line1'], ['line2', 'tail']] != [['line1\\nline2', 'tail']]\n\nFirst differing element 0:\n['line1']\n['line1\\nline2', 'tail']\n\nFirst list contains 1 additional elements.\nFirst extra element 1:\n['line2', 'tail']\n\n- [['line1'], ['line2', 'tail']]\n? ^^^^^^\n\n+ [['line1\\nline2', 'tail']]\n? ^^\n\n\n======================================================================\nFAIL: test_quotes_inside_a_field_are_doubled (_hidden_acceptance.test_csvquote.HiddenCsvTests.test_quotes_inside_a_field_are_doubled)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/_hidden_acceptance/test_csvquote.py\", line 18, in test_quotes_inside_a_field_are_doubled\n self.assertIn('\"\"', text)\n ~~~~~~~~~~~~~^^^^^^^^^^^^\nAssertionError: '\"\"' not found in 'title\\nsay \"hello\"\\n'\n\n----------------------------------------------------------------------\nRan 5 tests in 0.007s\n\nFAILED (failures=3)\n"}, "files_changed": [".as-xflow/config.json", "tests/test_csvquote.py", ".xflow/", "__pycache__/", "specs/", "tests/__pycache__/"], "diff_stat": " .as-xflow/config.json | 2 +-\n tests/test_csvquote.py | 22 ++++++++++++++++++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n", "session_ids": ["bf226aa6-ad8f-4a7b-8d16-e4391e775172"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131152-csv-quoting-openflow-r2/agent-1.jsonl", "results/raw/20261017T131152-csv-quoting-openflow-r2/agent-1.jsonl.err", "results/raw/20261017T131152-csv-quoting-openflow-r2/arm-stderr.txt", "results/raw/20261017T131152-csv-quoting-openflow-r2/arm-stdout.txt", "results/raw/20261017T131152-csv-quoting-openflow-r2/measure-from.txt", "results/raw/20261017T131152-csv-quoting-openflow-r2/tool-version.json"], "workdir": "tmp/20261017T131152-csv-quoting-openflow-r2/repo", "baseline": "c3900ae3a2805ad798135e5c0b4df9c647f289c0", "measure_from": "0be61207d77df85f811bcfb55a0a2fa80d896001"} +{"type": "run", "task": "csv-quoting", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:09:22.861800+00:00", "end": "2026-10-17T13:29:23.864494+00:00", "wall_s": 1200.983, "exit_code": 124, "timed_out": true, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130922-csv-quoting-openflow-r1/agent-1.jsonl", "session_id": "0f1fb9a7-884d-4309-9d20-4ce61a90f754", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.236, "stdout_tail": "", "stderr_tail": "....\n----------------------------------------------------------------------\nRan 4 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.281, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["HANDOFF.md", "csvquote.py", "specs/README.md", "specs/architecture.md", "specs/changes/csv-field-quoting/issue.md", "specs/changes/csv-field-quoting/plan.md", "specs/changes/csv-field-quoting/proposal.md", "specs/changes/csv-field-quoting/spec_delta_review.json", "specs/changes/csv-field-quoting/spec_delta_review.md", "specs/changes/csv-field-quoting/status.json", "specs/changes/csv-field-quoting/tasks.md", "specs/changes/csv-field-quoting/tdd/green-0.json", "specs/changes/csv-field-quoting/tdd/quality-0.json", "specs/changes/csv-field-quoting/tdd/red-0.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/csvquote.py", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/README.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/issue.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/plan.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/proposal.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/status.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/tasks.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/product.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/tests/test_csvquote.py", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/csvquote.py", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/README.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/issue.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/plan.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/proposal.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/status.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/tasks.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/product.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/tests/test_csvquote.py", "specs/changes/csv-field-quoting/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_csvquote.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " HANDOFF.md | 92 +++\n csvquote.py | 60 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/csv-field-quoting/issue.md | 21 +\n specs/changes/csv-field-quoting/plan.md | 47 ++\n specs/changes/csv-field-quoting/proposal.md | 39 ++\n .../csv-field-quoting/spec_delta_review.json | 109 ++++\n .../changes/csv-field-quoting/spec_delta_review.md | 38 ++\n specs/changes/csv-field-quoting/status.json | 637 +++++++++++++++++++++\n specs/changes/csv-field-quoting/tasks.md | 5 +\n specs/changes/csv-field-quoting/tdd/green-0.json | 38 ++\n specs/changes/csv-field-quoting/tdd/quality-0.json | 74 +++\n specs/changes/csv-field-quoting/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 66 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/csvquote.py | 78 +++\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/csv-field-quoting/issue.md | 21 +\n .../specs/changes/csv-field-quoting/plan.md | 47 ++\n .../specs/changes/csv-field-quoting/proposal.md | 39 ++\n .../specs/changes/csv-field-quoting/status.json | 611 ++++++++++++++++++++\n .../specs/changes/csv-field-quoting/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_csvquote.py | 37 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 51 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/csvquote.py | 24 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/csv-field-quoting/issue.md | 21 +\n .../red-0/specs/changes/csv-field-quoting/plan.md | 47 ++\n .../specs/changes/csv-field-quoting/proposal.md | 39 ++\n .../specs/changes/csv-field-quoting/status.json | 603 +++++++++++++++++++\n .../red-0/specs/changes/csv-field-quoting/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_csvquote.py | 37 ++\n specs/changes/csv-field-quoting/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_csvquote.py | 14 +\n 55 files changed, 4799 insertions(+), 3 deletions(-)\n", "session_ids": ["0f1fb9a7-884d-4309-9d20-4ce61a90f754"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T130922-csv-quoting-openflow-r1/agent-1.jsonl", "results/raw/20261017T130922-csv-quoting-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T130922-csv-quoting-openflow-r1/arm-stderr.txt", "results/raw/20261017T130922-csv-quoting-openflow-r1/arm-stdout.txt", "results/raw/20261017T130922-csv-quoting-openflow-r1/measure-from.txt", "results/raw/20261017T130922-csv-quoting-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T130922-csv-quoting-openflow-r1/repo", "baseline": "8af7bd5446baa8e1345a2b3ef30eea7ab7452c1a", "measure_from": "3d9e9d3373e71aa8e92a97cce3f8522f35f2d64d"} +{"type": "run", "task": "csv-quoting", "arm": "openflow", "repeat": 3, "start": "2026-10-17T13:12:16.368402+00:00", "end": "2026-10-17T13:32:16.782602+00:00", "wall_s": 1200.393, "exit_code": 124, "timed_out": true, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131216-csv-quoting-openflow-r3/agent-1.jsonl", "session_id": "2f7effe3-59ea-4991-a525-c4e69e44a3bb", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.122, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "specs/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 19 ++++++++-----------\n tests/test_csvquote.py | 22 ++++++++++++++++++++++\n 2 files changed, 30 insertions(+), 11 deletions(-)\n", "session_ids": ["2f7effe3-59ea-4991-a525-c4e69e44a3bb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131216-csv-quoting-openflow-r3/agent-1.jsonl", "results/raw/20261017T131216-csv-quoting-openflow-r3/agent-1.jsonl.err", "results/raw/20261017T131216-csv-quoting-openflow-r3/arm-stderr.txt", "results/raw/20261017T131216-csv-quoting-openflow-r3/arm-stdout.txt", "results/raw/20261017T131216-csv-quoting-openflow-r3/measure-from.txt", "results/raw/20261017T131216-csv-quoting-openflow-r3/tool-version.json"], "workdir": "tmp/20261017T131216-csv-quoting-openflow-r3/repo", "baseline": "4153cf66cb254bc0a8af4f594543132f863380f0", "measure_from": "ba32010332b637dcfc7eaa2dd497a0efdcbd7b87"} +{"type": "run", "task": "config-deep-merge", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:20:17.118090+00:00", "end": "2026-10-17T12:21:30.835055+00:00", "wall_s": 73.715, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122017-config-deep-merge-baseline-r3/agent-1.jsonl", "session_id": "30077454-35e1-4561-9371-b47bfab96b40", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 28749, "outputTokens": 3657, "cacheReadTokens": 59776, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 28749, "outputTokens": 3657, "cacheReadTokens": 59776, "cacheWriteTokens": 0, "total": 92182}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n", "session_ids": ["30077454-35e1-4561-9371-b47bfab96b40"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122017-config-deep-merge-baseline-r3/agent-1.jsonl", "results/raw/20261017T122017-config-deep-merge-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T122017-config-deep-merge-baseline-r3/arm-stderr.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r3/arm-stdout.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T122017-config-deep-merge-baseline-r3/repo", "baseline": "1351d9ee49e94e6fab4488327516f1e7fc41eccc", "measure_from": "1351d9ee49e94e6fab4488327516f1e7fc41eccc"} +{"type": "run", "task": "config-deep-merge", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:20:17.119644+00:00", "end": "2026-10-17T12:21:52.629603+00:00", "wall_s": 95.508, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122017-config-deep-merge-baseline-r2/agent-1.jsonl", "session_id": "7f1bffe6-1484-4851-9c14-d54efeda11a3", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 30533, "outputTokens": 4906, "cacheReadTokens": 99200, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 30533, "outputTokens": 4906, "cacheReadTokens": 99200, "cacheWriteTokens": 0, "total": 134639}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 13 +++++++++++--\n 1 file changed, 11 insertions(+), 2 deletions(-)\n", "session_ids": ["7f1bffe6-1484-4851-9c14-d54efeda11a3"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122017-config-deep-merge-baseline-r2/agent-1.jsonl", "results/raw/20261017T122017-config-deep-merge-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T122017-config-deep-merge-baseline-r2/arm-stderr.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r2/arm-stdout.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T122017-config-deep-merge-baseline-r2/repo", "baseline": "1351d9ee49e94e6fab4488327516f1e7fc41eccc", "measure_from": "1351d9ee49e94e6fab4488327516f1e7fc41eccc"} +{"type": "run", "task": "config-deep-merge", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:20:17.119030+00:00", "end": "2026-10-17T12:22:29.658334+00:00", "wall_s": 132.537, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122017-config-deep-merge-baseline-r1/agent-1.jsonl", "session_id": "5a6b5784-ee0f-4c24-8819-c596e1e17a17", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 43015, "outputTokens": 6652, "cacheReadTokens": 94720, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 43015, "outputTokens": 6652, "cacheReadTokens": 94720, "cacheWriteTokens": 0, "total": 144387}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 15 +++++++++++++--\n 1 file changed, 13 insertions(+), 2 deletions(-)\n", "session_ids": ["5a6b5784-ee0f-4c24-8819-c596e1e17a17"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122017-config-deep-merge-baseline-r1/agent-1.jsonl", "results/raw/20261017T122017-config-deep-merge-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T122017-config-deep-merge-baseline-r1/arm-stderr.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r1/arm-stdout.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T122017-config-deep-merge-baseline-r1/repo", "baseline": "1351d9ee49e94e6fab4488327516f1e7fc41eccc", "measure_from": "1351d9ee49e94e6fab4488327516f1e7fc41eccc"} +{"type": "run", "task": "config-deep-merge", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:21:30.845599+00:00", "end": "2026-10-17T12:24:31.547864+00:00", "wall_s": 180.699, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122130-config-deep-merge-superpowers-r1/agent-1.jsonl", "session_id": "8d0cb46e-5b39-40c5-9de1-a6bdda313803", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 40345, "outputTokens": 8635, "cacheReadTokens": 246144, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 40345, "outputTokens": 8635, "cacheReadTokens": 246144, "cacheWriteTokens": 0, "total": 295124}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "tests/test_deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 7 ++++++-\n tests/test_deepmerge.py | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 41 insertions(+), 1 deletion(-)\n", "session_ids": ["8d0cb46e-5b39-40c5-9de1-a6bdda313803"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T122130-config-deep-merge-superpowers-r1/agent-1.jsonl", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/arm-stderr.txt", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/arm-stdout.txt", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T122130-config-deep-merge-superpowers-r1/repo", "baseline": "7a9f096c16fe238294eab9b09b2b640962670d34", "measure_from": "7a9f096c16fe238294eab9b09b2b640962670d34"} +{"type": "run", "task": "config-deep-merge", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:22:29.670257+00:00", "end": "2026-10-17T12:25:07.798195+00:00", "wall_s": 158.125, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122229-config-deep-merge-superpowers-r3/agent-1.jsonl", "session_id": "ac5d2f0d-59b4-4d1d-8666-f12e5e73cc5d", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 54614, "outputTokens": 7345, "cacheReadTokens": 253824, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 54614, "outputTokens": 7345, "cacheReadTokens": 253824, "cacheWriteTokens": 0, "total": 315783}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "tests/test_deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 6 +++++-\n tests/test_deepmerge.py | 28 ++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 1 deletion(-)\n", "session_ids": ["ac5d2f0d-59b4-4d1d-8666-f12e5e73cc5d"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T122229-config-deep-merge-superpowers-r3/agent-1.jsonl", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/arm-stderr.txt", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/arm-stdout.txt", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T122229-config-deep-merge-superpowers-r3/repo", "baseline": "0580bfba58bd6f91390cd1454820a768380cddf6", "measure_from": "0580bfba58bd6f91390cd1454820a768380cddf6"} +{"type": "run", "task": "config-deep-merge", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:21:52.642166+00:00", "end": "2026-10-17T12:26:15.941456+00:00", "wall_s": 263.295, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122152-config-deep-merge-superpowers-r2/agent-1.jsonl", "session_id": "0610b915-3824-4062-8ec0-3fe9c8fd926f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 53400, "outputTokens": 13213, "cacheReadTokens": 392704, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 53400, "outputTokens": 13213, "cacheReadTokens": 392704, "cacheWriteTokens": 0, "total": 459317}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": "..........\n----------------------------------------------------------------------\nRan 10 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "tests/test_deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 7 +++++-\n tests/test_deepmerge.py | 60 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 66 insertions(+), 1 deletion(-)\n", "session_ids": ["0610b915-3824-4062-8ec0-3fe9c8fd926f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T122152-config-deep-merge-superpowers-r2/agent-1.jsonl", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/arm-stderr.txt", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/arm-stdout.txt", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T122152-config-deep-merge-superpowers-r2/repo", "baseline": "dadb79d43d16709347bb3d7863de8482ab8ee7f7", "measure_from": "dadb79d43d16709347bb3d7863de8482ab8ee7f7"} +{"type": "run", "task": "config-deep-merge", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:25:07.810785+00:00", "end": "2026-10-17T12:28:03.198487+00:00", "wall_s": 175.385, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122507-config-deep-merge-openswarm-r2/agent-1.jsonl", "session_id": "4993a35b-02fc-4998-bf3f-68d416275590", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 35143, "outputTokens": 6318, "cacheReadTokens": 74496, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "38d99df6-f493-4909-a9f1-fab554003d35", "model": "grok-4.7-high", "usage": {"inputTokens": 13864, "outputTokens": 2342, "cacheReadTokens": 59520, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49007, "outputTokens": 8660, "cacheReadTokens": 134016, "cacheWriteTokens": 0, "total": 191683}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py"], "diff_stat": " deepmerge.py | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n", "session_ids": ["4993a35b-02fc-4998-bf3f-68d416275590", "38d99df6-f493-4909-a9f1-fab554003d35"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T122507-config-deep-merge-openswarm-r2/agent-1.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/arm-stderr.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/arm-stdout.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/executor-task.md", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/executor-usage.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/merge.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/diff.patch", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/events.1.cursor.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/events.1.cursor.jsonl.err", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/result.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/status.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/task.md", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-result.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm.err", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/planner-prompt.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T122507-config-deep-merge-openswarm-r2/repo", "baseline": "581caf268c0d15c2ecc801c3868cf52137b72a4d", "measure_from": "581caf268c0d15c2ecc801c3868cf52137b72a4d"} +{"type": "run", "task": "config-deep-merge", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:24:31.583556+00:00", "end": "2026-10-17T12:28:12.072177+00:00", "wall_s": 220.486, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122431-config-deep-merge-openswarm-r1/agent-1.jsonl", "session_id": "935bd656-79fc-4ff7-ae91-78daac56ad21", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 21456, "outputTokens": 6923, "cacheReadTokens": 87808, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "d9adc1c8-63fa-4567-bc29-b9d0e5e03539", "model": "grok-4.7-high", "usage": {"inputTokens": 28783, "outputTokens": 4468, "cacheReadTokens": 49152, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 50239, "outputTokens": 11391, "cacheReadTokens": 136960, "cacheWriteTokens": 0, "total": 198590}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.054, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py"], "diff_stat": " deepmerge.py | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n", "session_ids": ["935bd656-79fc-4ff7-ae91-78daac56ad21", "d9adc1c8-63fa-4567-bc29-b9d0e5e03539"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T122431-config-deep-merge-openswarm-r1/agent-1.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/arm-stderr.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/arm-stdout.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/executor-task.md", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/executor-usage.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/merge.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/diff.patch", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/events.1.cursor.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/events.1.cursor.jsonl.err", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/result.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/status.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/task.md", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-result.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm.err", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/planner-prompt.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T122431-config-deep-merge-openswarm-r1/repo", "baseline": "0891cb471178999f5da6606db428818c70988fb2", "measure_from": "0891cb471178999f5da6606db428818c70988fb2"} +{"type": "run", "task": "csv-quoting", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:28:12.084097+00:00", "end": "2026-10-17T12:30:44.413162+00:00", "wall_s": 152.326, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122812-csv-quoting-baseline-r2/agent-1.jsonl", "session_id": "f31b4771-fa80-462a-a930-1d7d1f3134a0", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18626, "outputTokens": 8085, "cacheReadTokens": 77952, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 18626, "outputTokens": 8085, "cacheReadTokens": 77952, "cacheWriteTokens": 0, "total": 104663}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 58 insertions(+), 3 deletions(-)\n", "session_ids": ["f31b4771-fa80-462a-a930-1d7d1f3134a0"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122812-csv-quoting-baseline-r2/agent-1.jsonl", "results/raw/20261017T122812-csv-quoting-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T122812-csv-quoting-baseline-r2/arm-stderr.txt", "results/raw/20261017T122812-csv-quoting-baseline-r2/arm-stdout.txt", "results/raw/20261017T122812-csv-quoting-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T122812-csv-quoting-baseline-r2/repo", "baseline": "fc9e75629c9086948a07c6822a2b258269cbcd62", "measure_from": "fc9e75629c9086948a07c6822a2b258269cbcd62"} +{"type": "run", "task": "csv-quoting", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:28:03.211554+00:00", "end": "2026-10-17T12:31:29.098546+00:00", "wall_s": 205.884, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122803-csv-quoting-baseline-r1/agent-1.jsonl", "session_id": "10565516-ac5d-469e-a66a-4e5d21d2636e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 42182, "outputTokens": 10520, "cacheReadTokens": 81408, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 42182, "outputTokens": 10520, "cacheReadTokens": 81408, "cacheWriteTokens": 0, "total": 134110}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.057, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 53 insertions(+), 3 deletions(-)\n", "session_ids": ["10565516-ac5d-469e-a66a-4e5d21d2636e"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122803-csv-quoting-baseline-r1/agent-1.jsonl", "results/raw/20261017T122803-csv-quoting-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T122803-csv-quoting-baseline-r1/arm-stderr.txt", "results/raw/20261017T122803-csv-quoting-baseline-r1/arm-stdout.txt", "results/raw/20261017T122803-csv-quoting-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T122803-csv-quoting-baseline-r1/repo", "baseline": "5f5dc83565fc500c3a3dbcc38ae0dcdf09178186", "measure_from": "5f5dc83565fc500c3a3dbcc38ae0dcdf09178186"} +{"type": "run", "task": "config-deep-merge", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:26:15.953582+00:00", "end": "2026-10-17T12:31:42.096576+00:00", "wall_s": 326.14, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122615-config-deep-merge-openswarm-r3/agent-1.jsonl", "session_id": "5aee5194-91b0-499c-b4bf-db69a796d6cd", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18364, "outputTokens": 3933, "cacheReadTokens": 88960, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "912cabea-0459-475d-8c3a-ddef2a917b10", "model": "grok-4.7-high", "usage": {"inputTokens": 37148, "outputTokens": 13183, "cacheReadTokens": 83072, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 55512, "outputTokens": 17116, "cacheReadTokens": 172032, "cacheWriteTokens": 0, "total": 244660}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.054, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py"], "diff_stat": " deepmerge.py | 21 +++++++++++++++++++--\n 1 file changed, 19 insertions(+), 2 deletions(-)\n", "session_ids": ["5aee5194-91b0-499c-b4bf-db69a796d6cd", "912cabea-0459-475d-8c3a-ddef2a917b10"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T122615-config-deep-merge-openswarm-r3/agent-1.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/arm-stderr.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/arm-stdout.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/executor-task.md", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/executor-usage.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/merge.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/diff.patch", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/events.1.cursor.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/events.1.cursor.jsonl.err", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/result.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/status.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/task.md", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-result.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm.err", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/planner-prompt.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T122615-config-deep-merge-openswarm-r3/repo", "baseline": "9393a2685ea5f8c4abd28272a5a8189b7d388d8b", "measure_from": "9393a2685ea5f8c4abd28272a5a8189b7d388d8b"} +{"type": "run", "task": "csv-quoting", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:30:44.425254+00:00", "end": "2026-10-17T12:34:12.097963+00:00", "wall_s": 207.67, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123044-csv-quoting-baseline-r3/agent-1.jsonl", "session_id": "823828be-4c84-43de-954b-0e46e245bf47", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 39165, "outputTokens": 10377, "cacheReadTokens": 81280, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 39165, "outputTokens": 10377, "cacheReadTokens": 81280, "cacheWriteTokens": 0, "total": 130822}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 51 insertions(+), 3 deletions(-)\n", "session_ids": ["823828be-4c84-43de-954b-0e46e245bf47"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T123044-csv-quoting-baseline-r3/agent-1.jsonl", "results/raw/20261017T123044-csv-quoting-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T123044-csv-quoting-baseline-r3/arm-stderr.txt", "results/raw/20261017T123044-csv-quoting-baseline-r3/arm-stdout.txt", "results/raw/20261017T123044-csv-quoting-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T123044-csv-quoting-baseline-r3/repo", "baseline": "06ad722c01936debd4d75c072bb5b3dacdf5bdc0", "measure_from": "06ad722c01936debd4d75c072bb5b3dacdf5bdc0"} +{"type": "run", "task": "csv-quoting", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:31:29.110355+00:00", "end": "2026-10-17T12:36:43.949075+00:00", "wall_s": 314.835, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123129-csv-quoting-superpowers-r1/agent-1.jsonl", "session_id": "19de07a7-7696-48cf-b8a5-4325e0850486", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49696, "outputTokens": 16004, "cacheReadTokens": 276992, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49696, "outputTokens": 16004, "cacheReadTokens": 276992, "cacheWriteTokens": 0, "total": 342692}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": ".........\n----------------------------------------------------------------------\nRan 9 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 68 +++++++++++++++++++++++++++++++++++++++++++++++---\n tests/test_csvquote.py | 38 ++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+), 3 deletions(-)\n", "session_ids": ["19de07a7-7696-48cf-b8a5-4325e0850486"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T123129-csv-quoting-superpowers-r1/agent-1.jsonl", "results/raw/20261017T123129-csv-quoting-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T123129-csv-quoting-superpowers-r1/arm-stderr.txt", "results/raw/20261017T123129-csv-quoting-superpowers-r1/arm-stdout.txt", "results/raw/20261017T123129-csv-quoting-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T123129-csv-quoting-superpowers-r1/repo", "baseline": "2133255b869d664385bd137e49d2f9f0652a6c5a", "measure_from": "2133255b869d664385bd137e49d2f9f0652a6c5a"} +{"type": "run", "task": "csv-quoting", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:31:42.107636+00:00", "end": "2026-10-17T12:37:53.904423+00:00", "wall_s": 371.792, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123142-csv-quoting-superpowers-r2/agent-1.jsonl", "session_id": "35ba8fb0-c616-4791-9adb-46bc508699d9", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 82773, "outputTokens": 18511, "cacheReadTokens": 451584, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 82773, "outputTokens": 18511, "cacheReadTokens": 451584, "cacheWriteTokens": 0, "total": 552868}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".........\n----------------------------------------------------------------------\nRan 9 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.075, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++---\n tests/test_csvquote.py | 28 +++++++++++++++++++++++++\n 2 files changed, 81 insertions(+), 3 deletions(-)\n", "session_ids": ["35ba8fb0-c616-4791-9adb-46bc508699d9"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T123142-csv-quoting-superpowers-r2/agent-1.jsonl", "results/raw/20261017T123142-csv-quoting-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T123142-csv-quoting-superpowers-r2/arm-stderr.txt", "results/raw/20261017T123142-csv-quoting-superpowers-r2/arm-stdout.txt", "results/raw/20261017T123142-csv-quoting-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T123142-csv-quoting-superpowers-r2/repo", "baseline": "375ae49ec72800815a24b53375c2b2ac07595182", "measure_from": "375ae49ec72800815a24b53375c2b2ac07595182"} +{"type": "run", "task": "csv-quoting", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:34:12.110912+00:00", "end": "2026-10-17T12:38:09.857744+00:00", "wall_s": 237.744, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123412-csv-quoting-superpowers-r3/agent-1.jsonl", "session_id": "657434a5-560a-45b0-8330-23c9be75b9e6", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 57397, "outputTokens": 11047, "cacheReadTokens": 239616, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 57397, "outputTokens": 11047, "cacheReadTokens": 239616, "cacheWriteTokens": 0, "total": 308060}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 44 +++++++++++++++++++++++++++++++++++++++++---\n tests/test_csvquote.py | 22 ++++++++++++++++++++++\n 2 files changed, 63 insertions(+), 3 deletions(-)\n", "session_ids": ["657434a5-560a-45b0-8330-23c9be75b9e6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T123412-csv-quoting-superpowers-r3/agent-1.jsonl", "results/raw/20261017T123412-csv-quoting-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T123412-csv-quoting-superpowers-r3/arm-stderr.txt", "results/raw/20261017T123412-csv-quoting-superpowers-r3/arm-stdout.txt", "results/raw/20261017T123412-csv-quoting-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T123412-csv-quoting-superpowers-r3/repo", "baseline": "b642a415eb4ac4f63053940fc8f375f4c907ecec", "measure_from": "b642a415eb4ac4f63053940fc8f375f4c907ecec"} +{"type": "run", "task": "csv-quoting", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:38:09.870229+00:00", "end": "2026-10-17T12:44:32.592844+00:00", "wall_s": 382.719, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123809-csv-quoting-openswarm-r3/agent-1.jsonl", "session_id": "1333c1f2-6f79-419a-bb54-022421f2a8f7", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 26453, "outputTokens": 10978, "cacheReadTokens": 170624, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "63fd0477-4348-476d-be40-807766f4e6eb", "model": "grok-4.7-high", "usage": {"inputTokens": 87959, "outputTokens": 8016, "cacheReadTokens": 47232, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 114412, "outputTokens": 18994, "cacheReadTokens": 217856, "cacheWriteTokens": 0, "total": 351262}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py"], "diff_stat": " csvquote.py | 50 +++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 47 insertions(+), 3 deletions(-)\n", "session_ids": ["1333c1f2-6f79-419a-bb54-022421f2a8f7", "63fd0477-4348-476d-be40-807766f4e6eb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T123809-csv-quoting-openswarm-r3/agent-1.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T123809-csv-quoting-openswarm-r3/arm-stderr.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/arm-stdout.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/executor-task.md", "results/raw/20261017T123809-csv-quoting-openswarm-r3/executor-usage.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/merge.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/diff.patch", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/events.1.cursor.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/events.1.cursor.jsonl.err", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/result.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/status.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/task.md", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-result.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm.err", "results/raw/20261017T123809-csv-quoting-openswarm-r3/planner-prompt.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T123809-csv-quoting-openswarm-r3/repo", "baseline": "e7b313d87b3ad68d2bcf5e64ed77074edfb19de3", "measure_from": "e7b313d87b3ad68d2bcf5e64ed77074edfb19de3"} +{"type": "run", "task": "date-range-parse", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:44:32.605896+00:00", "end": "2026-10-17T12:45:03.578561+00:00", "wall_s": 30.971, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124432-date-range-parse-baseline-r1/agent-1.jsonl", "session_id": "ecb8625f-1317-45b2-9a28-5edabe6a2956", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 12152, "outputTokens": 1315, "cacheReadTokens": 72064, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 12152, "outputTokens": 1315, "cacheReadTokens": 72064, "cacheWriteTokens": 0, "total": 85531}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["ecb8625f-1317-45b2-9a28-5edabe6a2956"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T124432-date-range-parse-baseline-r1/agent-1.jsonl", "results/raw/20261017T124432-date-range-parse-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T124432-date-range-parse-baseline-r1/arm-stderr.txt", "results/raw/20261017T124432-date-range-parse-baseline-r1/arm-stdout.txt", "results/raw/20261017T124432-date-range-parse-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T124432-date-range-parse-baseline-r1/repo", "baseline": "f48844c4abc08cb6d42e105ee6d74e4f07029e5d", "measure_from": "f48844c4abc08cb6d42e105ee6d74e4f07029e5d"} +{"type": "run", "task": "csv-quoting", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:37:53.917393+00:00", "end": "2026-10-17T12:45:08.845140+00:00", "wall_s": 434.924, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123753-csv-quoting-openswarm-r2/agent-1.jsonl", "session_id": "e4c76ef7-3673-4d5a-8b2e-c4a9ae08d2b4", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 84076, "outputTokens": 13359, "cacheReadTokens": 181248, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "828cd414-7fbc-4166-80a6-8f0dd4c33537", "model": "grok-4.7-high", "usage": {"inputTokens": 32851, "outputTokens": 8557, "cacheReadTokens": 55808, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 116927, "outputTokens": 21916, "cacheReadTokens": 237056, "cacheWriteTokens": 0, "total": 375899}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.061, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py"], "diff_stat": " csvquote.py | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 85 insertions(+), 7 deletions(-)\n", "session_ids": ["e4c76ef7-3673-4d5a-8b2e-c4a9ae08d2b4", "828cd414-7fbc-4166-80a6-8f0dd4c33537"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T123753-csv-quoting-openswarm-r2/agent-1.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T123753-csv-quoting-openswarm-r2/arm-stderr.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/arm-stdout.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/executor-task.md", "results/raw/20261017T123753-csv-quoting-openswarm-r2/executor-usage.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/merge.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/diff.patch", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/events.1.cursor.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/events.1.cursor.jsonl.err", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/result.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/status.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/task.md", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-result.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm.err", "results/raw/20261017T123753-csv-quoting-openswarm-r2/planner-prompt.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T123753-csv-quoting-openswarm-r2/repo", "baseline": "7c763563399a0bcfada51cc813ed3fa824ff5ed2", "measure_from": "7c763563399a0bcfada51cc813ed3fa824ff5ed2"} +{"type": "run", "task": "date-range-parse", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:45:03.590367+00:00", "end": "2026-10-17T12:45:45.748696+00:00", "wall_s": 42.157, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124503-date-range-parse-baseline-r2/agent-1.jsonl", "session_id": "c36103f4-ad1a-4be4-b24d-6d246eb60b24", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 25390, "outputTokens": 1945, "cacheReadTokens": 60032, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 25390, "outputTokens": 1945, "cacheReadTokens": 60032, "cacheWriteTokens": 0, "total": 87367}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["c36103f4-ad1a-4be4-b24d-6d246eb60b24"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T124503-date-range-parse-baseline-r2/agent-1.jsonl", "results/raw/20261017T124503-date-range-parse-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T124503-date-range-parse-baseline-r2/arm-stderr.txt", "results/raw/20261017T124503-date-range-parse-baseline-r2/arm-stdout.txt", "results/raw/20261017T124503-date-range-parse-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T124503-date-range-parse-baseline-r2/repo", "baseline": "93459cbfe8c8261d6d188d05b7c6f668c672cd91", "measure_from": "93459cbfe8c8261d6d188d05b7c6f668c672cd91"} +{"type": "run", "task": "date-range-parse", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:45:08.859512+00:00", "end": "2026-10-17T12:45:49.894198+00:00", "wall_s": 41.033, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124508-date-range-parse-baseline-r3/agent-1.jsonl", "session_id": "c352581a-3990-49da-a150-410e20e5b2c4", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 24409, "outputTokens": 1736, "cacheReadTokens": 78592, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 24409, "outputTokens": 1736, "cacheReadTokens": 78592, "cacheWriteTokens": 0, "total": 104737}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.073, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["c352581a-3990-49da-a150-410e20e5b2c4"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T124508-date-range-parse-baseline-r3/agent-1.jsonl", "results/raw/20261017T124508-date-range-parse-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T124508-date-range-parse-baseline-r3/arm-stderr.txt", "results/raw/20261017T124508-date-range-parse-baseline-r3/arm-stdout.txt", "results/raw/20261017T124508-date-range-parse-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T124508-date-range-parse-baseline-r3/repo", "baseline": "9f3333e7b49ac0ad9512f64f2b151975dcb62054", "measure_from": "9f3333e7b49ac0ad9512f64f2b151975dcb62054"} +{"type": "run", "task": "date-range-parse", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:45:49.905407+00:00", "end": "2026-10-17T12:47:19.816915+00:00", "wall_s": 89.908, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124549-date-range-parse-superpowers-r2/agent-1.jsonl", "session_id": "d7cc3745-936c-4bc9-9538-3262302999dc", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 24971, "outputTokens": 3947, "cacheReadTokens": 209920, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 24971, "outputTokens": 3947, "cacheReadTokens": 209920, "cacheWriteTokens": 0, "total": 238838}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 31 insertions(+), 1 deletion(-)\n", "session_ids": ["d7cc3745-936c-4bc9-9538-3262302999dc"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T124549-date-range-parse-superpowers-r2/agent-1.jsonl", "results/raw/20261017T124549-date-range-parse-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T124549-date-range-parse-superpowers-r2/arm-stderr.txt", "results/raw/20261017T124549-date-range-parse-superpowers-r2/arm-stdout.txt", "results/raw/20261017T124549-date-range-parse-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T124549-date-range-parse-superpowers-r2/repo", "baseline": "03b71bdd0a2b8cfabc03406eb51b90f3a5f8989a", "measure_from": "03b71bdd0a2b8cfabc03406eb51b90f3a5f8989a"} +{"type": "run", "task": "date-range-parse", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:45:45.761275+00:00", "end": "2026-10-17T12:47:39.213518+00:00", "wall_s": 113.45, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124545-date-range-parse-superpowers-r1/agent-1.jsonl", "session_id": "1f52b137-57c6-45f3-8522-14020993131d", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 65267, "outputTokens": 5363, "cacheReadTokens": 204032, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 65267, "outputTokens": 5363, "cacheReadTokens": 204032, "cacheWriteTokens": 0, "total": 274662}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 1 deletion(-)\n", "session_ids": ["1f52b137-57c6-45f3-8522-14020993131d"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T124545-date-range-parse-superpowers-r1/agent-1.jsonl", "results/raw/20261017T124545-date-range-parse-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T124545-date-range-parse-superpowers-r1/arm-stderr.txt", "results/raw/20261017T124545-date-range-parse-superpowers-r1/arm-stdout.txt", "results/raw/20261017T124545-date-range-parse-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T124545-date-range-parse-superpowers-r1/repo", "baseline": "88827924fcc68fc6c848f3cffcbb806494883e8b", "measure_from": "88827924fcc68fc6c848f3cffcbb806494883e8b"} +{"type": "run", "task": "csv-quoting", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:36:43.962631+00:00", "end": "2026-10-17T12:48:11.069102+00:00", "wall_s": 687.103, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123643-csv-quoting-openswarm-r1/agent-1.jsonl", "session_id": "fcc84471-af70-4b58-a6e3-e50118aab5fe", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49109, "outputTokens": 9658, "cacheReadTokens": 159360, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "56406353-929d-4b48-9541-101de6d72aea", "model": "grok-4.7-high", "usage": {"inputTokens": 98762, "outputTokens": 24006, "cacheReadTokens": 178944, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 147871, "outputTokens": 33664, "cacheReadTokens": 338304, "cacheWriteTokens": 0, "total": 519839}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py"], "diff_stat": " csvquote.py | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 50 insertions(+), 7 deletions(-)\n", "session_ids": ["fcc84471-af70-4b58-a6e3-e50118aab5fe", "56406353-929d-4b48-9541-101de6d72aea"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T123643-csv-quoting-openswarm-r1/agent-1.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T123643-csv-quoting-openswarm-r1/arm-stderr.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/arm-stdout.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/executor-task.md", "results/raw/20261017T123643-csv-quoting-openswarm-r1/executor-usage.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/merge.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/diff.patch", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/events.1.cursor.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/events.1.cursor.jsonl.err", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/result.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/status.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/task.md", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-result.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm.err", "results/raw/20261017T123643-csv-quoting-openswarm-r1/planner-prompt.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T123643-csv-quoting-openswarm-r1/repo", "baseline": "7021e10868712e05a73fa8d9633663439e805e3d", "measure_from": "7021e10868712e05a73fa8d9633663439e805e3d"} +{"type": "run", "task": "date-range-parse", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:47:39.225615+00:00", "end": "2026-10-17T12:49:56.509243+00:00", "wall_s": 137.281, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124739-date-range-parse-openswarm-r1/agent-1.jsonl", "session_id": "28288edc-b18c-4e56-9406-953432732a82", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 37576, "outputTokens": 4577, "cacheReadTokens": 71680, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "c3ab648b-624e-4580-aa6b-a6b333e079a7", "model": "grok-4.7-high", "usage": {"inputTokens": 26156, "outputTokens": 1494, "cacheReadTokens": 64896, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 63732, "outputTokens": 6071, "cacheReadTokens": 136576, "cacheWriteTokens": 0, "total": 206379}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.053, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["28288edc-b18c-4e56-9406-953432732a82", "c3ab648b-624e-4580-aa6b-a6b333e079a7"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T124739-date-range-parse-openswarm-r1/agent-1.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T124739-date-range-parse-openswarm-r1/arm-stderr.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/arm-stdout.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/executor-task.md", "results/raw/20261017T124739-date-range-parse-openswarm-r1/executor-usage.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/merge.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/diff.patch", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/events.1.cursor.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/events.1.cursor.jsonl.err", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/result.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/status.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/task.md", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-result.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm.err", "results/raw/20261017T124739-date-range-parse-openswarm-r1/planner-prompt.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T124739-date-range-parse-openswarm-r1/repo", "baseline": "18a2caa76595dab0153aa6bd4b2712ac2598120c", "measure_from": "18a2caa76595dab0153aa6bd4b2712ac2598120c"} +{"type": "run", "task": "date-range-parse", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:47:19.830665+00:00", "end": "2026-10-17T12:50:02.383110+00:00", "wall_s": 162.55, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124719-date-range-parse-superpowers-r3/agent-1.jsonl", "session_id": "dbba04bc-c4d8-4574-b7de-3912c4fbb5f1", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 30115, "outputTokens": 7377, "cacheReadTokens": 316288, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 30115, "outputTokens": 7377, "cacheReadTokens": 316288, "cacheWriteTokens": 0, "total": 353780}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 40 ++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 41 insertions(+), 1 deletion(-)\n", "session_ids": ["dbba04bc-c4d8-4574-b7de-3912c4fbb5f1"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T124719-date-range-parse-superpowers-r3/agent-1.jsonl", "results/raw/20261017T124719-date-range-parse-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T124719-date-range-parse-superpowers-r3/arm-stderr.txt", "results/raw/20261017T124719-date-range-parse-superpowers-r3/arm-stdout.txt", "results/raw/20261017T124719-date-range-parse-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T124719-date-range-parse-superpowers-r3/repo", "baseline": "e1e2f29a73eda5a1f13b076c051ef4f4c8cb5bb8", "measure_from": "e1e2f29a73eda5a1f13b076c051ef4f4c8cb5bb8"} +{"type": "run", "task": "date-range-parse", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:48:11.080692+00:00", "end": "2026-10-17T12:50:13.416537+00:00", "wall_s": 122.334, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124811-date-range-parse-openswarm-r2/agent-1.jsonl", "session_id": "cfeda10b-81a5-45a4-94a0-7bd5d5d9c037", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 32394, "outputTokens": 4068, "cacheReadTokens": 76416, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "238eb454-44ae-4229-b4e7-aaeb8516b1f6", "model": "grok-4.7-high", "usage": {"inputTokens": 13651, "outputTokens": 1845, "cacheReadTokens": 58880, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 46045, "outputTokens": 5913, "cacheReadTokens": 135296, "cacheWriteTokens": 0, "total": 187254}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".........\n----------------------------------------------------------------------\nRan 9 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 49 insertions(+), 1 deletion(-)\n", "session_ids": ["cfeda10b-81a5-45a4-94a0-7bd5d5d9c037", "238eb454-44ae-4229-b4e7-aaeb8516b1f6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T124811-date-range-parse-openswarm-r2/agent-1.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T124811-date-range-parse-openswarm-r2/arm-stderr.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/arm-stdout.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/executor-task.md", "results/raw/20261017T124811-date-range-parse-openswarm-r2/executor-usage.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/merge.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/diff.patch", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/events.1.cursor.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/events.1.cursor.jsonl.err", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/result.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/status.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/task.md", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-result.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm.err", "results/raw/20261017T124811-date-range-parse-openswarm-r2/planner-prompt.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T124811-date-range-parse-openswarm-r2/repo", "baseline": "7583294e00e9412bccbf5628d57d13ff75984651", "measure_from": "7583294e00e9412bccbf5628d57d13ff75984651"} +{"type": "run", "task": "lru-eviction", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:50:02.394809+00:00", "end": "2026-10-17T12:50:52.081210+00:00", "wall_s": 49.685, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125002-lru-eviction-baseline-r1/agent-1.jsonl", "session_id": "2bf45f2e-f4c6-4a47-b6b1-94dedd2e9ee3", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 29867, "outputTokens": 2106, "cacheReadTokens": 73728, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 29867, "outputTokens": 2106, "cacheReadTokens": 73728, "cacheWriteTokens": 0, "total": 105701}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.731326\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.128623\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.342481\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.895243\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.101, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.778803\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.195814\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.162245\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.124738\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.263482\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.414941\n", "stderr_tail": ""}, "files_changed": ["src/lru.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n", "session_ids": ["2bf45f2e-f4c6-4a47-b6b1-94dedd2e9ee3"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125002-lru-eviction-baseline-r1/agent-1.jsonl", "results/raw/20261017T125002-lru-eviction-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T125002-lru-eviction-baseline-r1/arm-stderr.txt", "results/raw/20261017T125002-lru-eviction-baseline-r1/arm-stdout.txt", "results/raw/20261017T125002-lru-eviction-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T125002-lru-eviction-baseline-r1/repo", "baseline": "16beb49ea46d5229aea652f988ffc6f28804e67c", "measure_from": "16beb49ea46d5229aea652f988ffc6f28804e67c"} +{"type": "run", "task": "lru-eviction", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:50:13.428725+00:00", "end": "2026-10-17T12:51:05.784373+00:00", "wall_s": 52.354, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125013-lru-eviction-baseline-r2/agent-1.jsonl", "session_id": "393bed56-97bc-429b-b996-b08e93caa1a6", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 36478, "outputTokens": 2229, "cacheReadTokens": 85376, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 36478, "outputTokens": 2229, "cacheReadTokens": 85376, "cacheWriteTokens": 0, "total": 124083}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.138, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.881029\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.150391\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.326134\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.194338\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.814772\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.140211\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.16046\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.11712\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.350916\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 68.032049\n", "stderr_tail": ""}, "files_changed": ["src/lru.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n", "session_ids": ["393bed56-97bc-429b-b996-b08e93caa1a6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125013-lru-eviction-baseline-r2/agent-1.jsonl", "results/raw/20261017T125013-lru-eviction-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T125013-lru-eviction-baseline-r2/arm-stderr.txt", "results/raw/20261017T125013-lru-eviction-baseline-r2/arm-stdout.txt", "results/raw/20261017T125013-lru-eviction-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T125013-lru-eviction-baseline-r2/repo", "baseline": "869cbe02b6af1f6366877828a439984de73d7cfe", "measure_from": "869cbe02b6af1f6366877828a439984de73d7cfe"} +{"type": "run", "task": "lru-eviction", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:50:52.093480+00:00", "end": "2026-10-17T12:51:43.280006+00:00", "wall_s": 51.185, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125052-lru-eviction-baseline-r3/agent-1.jsonl", "session_id": "1f3ff685-7762-43c9-89a3-86cd766545a3", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 25599, "outputTokens": 2151, "cacheReadTokens": 78336, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 25599, "outputTokens": 2151, "cacheReadTokens": 78336, "cacheWriteTokens": 0, "total": 106086}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.098, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.683018\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.122577\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.356142\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.079481\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.869003\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.226648\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.257172\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.13057\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.309019\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.74608\n", "stderr_tail": ""}, "files_changed": ["src/lru.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n", "session_ids": ["1f3ff685-7762-43c9-89a3-86cd766545a3"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125052-lru-eviction-baseline-r3/agent-1.jsonl", "results/raw/20261017T125052-lru-eviction-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T125052-lru-eviction-baseline-r3/arm-stderr.txt", "results/raw/20261017T125052-lru-eviction-baseline-r3/arm-stdout.txt", "results/raw/20261017T125052-lru-eviction-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T125052-lru-eviction-baseline-r3/repo", "baseline": "892984578c469b77392c9cb6907647091f441fc8", "measure_from": "892984578c469b77392c9cb6907647091f441fc8"} +{"type": "run", "task": "date-range-parse", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:49:56.522852+00:00", "end": "2026-10-17T12:52:03.649922+00:00", "wall_s": 127.124, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124956-date-range-parse-openswarm-r3/agent-1.jsonl", "session_id": "1a55ed8e-c8ad-453a-b2ab-bdaf42076ab8", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 19482, "outputTokens": 4590, "cacheReadTokens": 113920, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "3dfaeea6-9dd0-4fed-9a15-5b2435f09bd0", "model": "grok-4.7-high", "usage": {"inputTokens": 13127, "outputTokens": 1073, "cacheReadTokens": 76800, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 32609, "outputTokens": 5663, "cacheReadTokens": 190720, "cacheWriteTokens": 0, "total": 228992}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.118, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.063, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["1a55ed8e-c8ad-453a-b2ab-bdaf42076ab8", "3dfaeea6-9dd0-4fed-9a15-5b2435f09bd0"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T124956-date-range-parse-openswarm-r3/agent-1.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T124956-date-range-parse-openswarm-r3/arm-stderr.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/arm-stdout.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/executor-task.md", "results/raw/20261017T124956-date-range-parse-openswarm-r3/executor-usage.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/merge.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/diff.patch", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/events.1.cursor.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/events.1.cursor.jsonl.err", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/result.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/status.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/task.md", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-result.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm.err", "results/raw/20261017T124956-date-range-parse-openswarm-r3/planner-prompt.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T124956-date-range-parse-openswarm-r3/repo", "baseline": "35b26a77786eabe7f509bb741067af251ac54227", "measure_from": "35b26a77786eabe7f509bb741067af251ac54227"} +{"type": "run", "task": "lru-eviction", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:51:05.797087+00:00", "end": "2026-10-17T12:55:11.354393+00:00", "wall_s": 245.553, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125105-lru-eviction-superpowers-r1/agent-1.jsonl", "session_id": "4e7550f7-f00a-4c9b-82de-7c13c8967000", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 72646, "outputTokens": 11596, "cacheReadTokens": 449792, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 72646, "outputTokens": 11596, "cacheReadTokens": 449792, "cacheWriteTokens": 0, "total": 534034}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.838977\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.173726\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.43764\n type: 'test'\n ...\n# Subtest: reading a key protects it from the next eviction\nok 4 - reading a key protects it from the next eviction\n ---\n duration_ms: 1.098358\n type: 'test'\n ...\n# Subtest: updating an existing key protects it from the next eviction\nok 5 - updating an existing key protects it from the next eviction\n ---\n duration_ms: 0.202359\n type: 'test'\n ...\n# Subtest: updating a present key does not evict another entry\nok 6 - updating a present key does not evict another entry\n ---\n duration_ms: 0.098939\n type: 'test'\n ...\n# Subtest: a missing read does not change eviction order\nok 7 - a missing read does not change eviction order\n ---\n duration_ms: 0.134687\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.297303\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.102, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.73066\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.123836\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.164342\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.097773\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.305831\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.300957\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 11 +++++++++--\n test/lru.test.js | 42 ++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 2 deletions(-)\n", "session_ids": ["4e7550f7-f00a-4c9b-82de-7c13c8967000"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125105-lru-eviction-superpowers-r1/agent-1.jsonl", "results/raw/20261017T125105-lru-eviction-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T125105-lru-eviction-superpowers-r1/arm-stderr.txt", "results/raw/20261017T125105-lru-eviction-superpowers-r1/arm-stdout.txt", "results/raw/20261017T125105-lru-eviction-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T125105-lru-eviction-superpowers-r1/repo", "baseline": "7fe0118dc6f31887d82a50d353084ef155cb1ad2", "measure_from": "7fe0118dc6f31887d82a50d353084ef155cb1ad2"} +{"type": "run", "task": "lru-eviction", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:52:03.687094+00:00", "end": "2026-10-17T12:55:16.814032+00:00", "wall_s": 193.124, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125203-lru-eviction-superpowers-r3/agent-1.jsonl", "session_id": "2af5eba9-35bb-4295-aad3-f29fa4f730de", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 32196, "outputTokens": 8493, "cacheReadTokens": 385536, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 32196, "outputTokens": 8493, "cacheReadTokens": 385536, "cacheWriteTokens": 0, "total": 426225}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.700715\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.163695\n type: 'test'\n ...\n# Subtest: reading a key makes it newest so the next insert evicts the unread entry\nok 3 - reading a key makes it newest so the next insert evicts the unread entry\n ---\n duration_ms: 0.100627\n type: 'test'\n ...\n# Subtest: updating a present key keeps the other entry and stores the new value\nok 4 - updating a present key keeps the other entry and stores the new value\n ---\n duration_ms: 1.324602\n type: 'test'\n ...\n# Subtest: updating a present key makes it newest so the next insert evicts the other entry\nok 5 - updating a present key makes it newest so the next insert evicts the other entry\n ---\n duration_ms: 0.150326\n type: 'test'\n ...\n# Subtest: reading a missing key returns undefined and leaves eviction order unchanged\nok 6 - reading a missing key returns undefined and leaves eviction order unchanged\n ---\n duration_ms: 0.117871\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 7 - rejects a capacity below 1\n ---\n duration_ms: 0.475395\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.439044\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.109, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.894122\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.165305\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.160776\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.120689\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.501519\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 70.557112\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n test/lru.test.js | 42 ++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 52 insertions(+), 2 deletions(-)\n", "session_ids": ["2af5eba9-35bb-4295-aad3-f29fa4f730de"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125203-lru-eviction-superpowers-r3/agent-1.jsonl", "results/raw/20261017T125203-lru-eviction-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T125203-lru-eviction-superpowers-r3/arm-stderr.txt", "results/raw/20261017T125203-lru-eviction-superpowers-r3/arm-stdout.txt", "results/raw/20261017T125203-lru-eviction-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T125203-lru-eviction-superpowers-r3/repo", "baseline": "eec39c13cc4bd024c04590c40d3cf416d0229392", "measure_from": "eec39c13cc4bd024c04590c40d3cf416d0229392"} +{"type": "run", "task": "lru-eviction", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:51:43.291273+00:00", "end": "2026-10-17T12:55:18.461222+00:00", "wall_s": 215.167, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125143-lru-eviction-superpowers-r2/agent-1.jsonl", "session_id": "f6605a25-39a4-4579-bba4-e463ebd6a03b", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 96536, "outputTokens": 9939, "cacheReadTokens": 322176, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 96536, "outputTokens": 9939, "cacheReadTokens": 322176, "cacheWriteTokens": 0, "total": 428651}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.118, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 1.337607\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.24041\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 1.703173\n type: 'test'\n ...\n# Subtest: reading a key protects it from the next eviction\nok 4 - reading a key protects it from the next eviction\n ---\n duration_ms: 0.185719\n type: 'test'\n ...\n# Subtest: updating an existing key protects it from the next eviction\nok 5 - updating an existing key protects it from the next eviction\n ---\n duration_ms: 0.17511\n type: 'test'\n ...\n# Subtest: updating an existing key keeps the other entry\nok 6 - updating an existing key keeps the other entry\n ---\n duration_ms: 0.109131\n type: 'test'\n ...\n# Subtest: updating the only key keeps it\nok 7 - updating the only key keeps it\n ---\n duration_ms: 0.180693\n type: 'test'\n ...\n# Subtest: reading a missing key leaves the eviction order unchanged\nok 8 - reading a missing key leaves the eviction order unchanged\n ---\n duration_ms: 0.102631\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 74.151045\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.111, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 1.147862\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.223329\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.250793\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.14127\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.271841\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.526238\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 6 +++++-\n test/lru.test.js | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 54 insertions(+), 1 deletion(-)\n", "session_ids": ["f6605a25-39a4-4579-bba4-e463ebd6a03b"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125143-lru-eviction-superpowers-r2/agent-1.jsonl", "results/raw/20261017T125143-lru-eviction-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T125143-lru-eviction-superpowers-r2/arm-stderr.txt", "results/raw/20261017T125143-lru-eviction-superpowers-r2/arm-stdout.txt", "results/raw/20261017T125143-lru-eviction-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T125143-lru-eviction-superpowers-r2/repo", "baseline": "1c14d75e65a820976b1aae6b9b54079f17874650", "measure_from": "1c14d75e65a820976b1aae6b9b54079f17874650"} +{"type": "run", "task": "lru-eviction", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:55:11.368324+00:00", "end": "2026-10-17T12:58:47.227015+00:00", "wall_s": 215.856, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125511-lru-eviction-openswarm-r1/agent-1.jsonl", "session_id": "3f87c4c2-1a21-4343-bb09-77b0497a84df", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 34151, "outputTokens": 4937, "cacheReadTokens": 98176, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "7d0b3083-3b0e-4a22-a94a-8aae643efcdd", "model": "grok-4.7-high", "usage": {"inputTokens": 53600, "outputTokens": 5319, "cacheReadTokens": 49024, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 87751, "outputTokens": 10256, "cacheReadTokens": 147200, "cacheWriteTokens": 0, "total": 245207}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.694715\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.181774\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 1.326041\n type: 'test'\n ...\n# Subtest: reading a key refreshes recency so it is not evicted next\nok 4 - reading a key refreshes recency so it is not evicted next\n ---\n duration_ms: 0.185668\n type: 'test'\n ...\n# Subtest: updating an existing key refreshes recency and does not grow\nok 5 - updating an existing key refreshes recency and does not grow\n ---\n duration_ms: 0.210548\n type: 'test'\n ...\n# Subtest: a miss does not change eviction order\nok 6 - a miss does not change eviction order\n ---\n duration_ms: 0.112809\n type: 'test'\n ...\n# Subtest: evicts the least recently used key after several uses\nok 7 - evicts the least recently used key after several uses\n ---\n duration_ms: 0.216612\n type: 'test'\n ...\n# Subtest: capacity 1 evicts the previous key only on a new insert\nok 8 - capacity 1 evicts the previous key only on a new insert\n ---\n duration_ms: 0.098663\n type: 'test'\n ...\n# Subtest: a stored undefined is a hit and refreshes recency\nok 9 - a stored undefined is a hit and refreshes recency\n ---\n duration_ms: 0.338839\n type: 'test'\n ...\n1..9\n# tests 9\n# suites 0\n# pass 9\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.616718\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.094, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.793236\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.208765\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.2131\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.159854\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.418475\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.371545\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 12 +++++++--\n test/lru.test.js | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 92 insertions(+), 2 deletions(-)\n", "session_ids": ["3f87c4c2-1a21-4343-bb09-77b0497a84df", "7d0b3083-3b0e-4a22-a94a-8aae643efcdd"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T125511-lru-eviction-openswarm-r1/agent-1.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T125511-lru-eviction-openswarm-r1/arm-stderr.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/arm-stdout.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/executor-task.md", "results/raw/20261017T125511-lru-eviction-openswarm-r1/executor-usage.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/merge.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/diff.patch", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/events.1.cursor.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/events.1.cursor.jsonl.err", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/result.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/status.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/task.md", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-result.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm.err", "results/raw/20261017T125511-lru-eviction-openswarm-r1/planner-prompt.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T125511-lru-eviction-openswarm-r1/repo", "baseline": "76b075a5fe5b049324713dd0eee4bf40ba237835", "measure_from": "76b075a5fe5b049324713dd0eee4bf40ba237835"} +{"type": "run", "task": "lru-eviction", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:55:16.826740+00:00", "end": "2026-10-17T12:58:55.801967+00:00", "wall_s": 218.972, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125516-lru-eviction-openswarm-r2/agent-1.jsonl", "session_id": "7d0e3151-5789-45ee-a4b9-5bdfc4fe0849", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18266, "outputTokens": 3703, "cacheReadTokens": 109952, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "3f319aef-f47e-4a86-8598-7a20fc334c34", "model": "grok-4.7-high", "usage": {"inputTokens": 19599, "outputTokens": 7051, "cacheReadTokens": 65664, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 37865, "outputTokens": 10754, "cacheReadTokens": 175616, "cacheWriteTokens": 0, "total": 224235}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.092, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.881146\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.201785\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.477353\n type: 'test'\n ...\n# Subtest: get of a present key refreshes recency\nok 4 - get of a present key refreshes recency\n ---\n duration_ms: 0.121558\n type: 'test'\n ...\n# Subtest: set of a present key refreshes recency without evicting\nok 5 - set of a present key refreshes recency without evicting\n ---\n duration_ms: 0.18424\n type: 'test'\n ...\n# Subtest: get of a missing key does not refresh recency\nok 6 - get of a missing key does not refresh recency\n ---\n duration_ms: 0.095158\n type: 'test'\n ...\n# Subtest: capacity 1 evicts on insert and keeps the key on update\nok 7 - capacity 1 evicts on insert and keeps the key on update\n ---\n duration_ms: 0.177509\n type: 'test'\n ...\n# Subtest: repeated get of the newest key leaves the oldest key unchanged\nok 8 - repeated get of the newest key leaves the oldest key unchanged\n ---\n duration_ms: 0.084472\n type: 'test'\n ...\n# Subtest: use of one key leaves older unused keys in order\nok 9 - use of one key leaves older unused keys in order\n ---\n duration_ms: 0.265697\n type: 'test'\n ...\n# Subtest: rejects a non-integer capacity and accepts 1\nok 10 - rejects a non-integer capacity and accepts 1\n ---\n duration_ms: 0.322364\n type: 'test'\n ...\n1..10\n# tests 10\n# suites 0\n# pass 10\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 58.449088\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 1.095405\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.216555\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.228947\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.155527\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.37121\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 66.20635\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 11 +++----\n test/lru.test.js | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 96 insertions(+), 5 deletions(-)\n", "session_ids": ["7d0e3151-5789-45ee-a4b9-5bdfc4fe0849", "3f319aef-f47e-4a86-8598-7a20fc334c34"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T125516-lru-eviction-openswarm-r2/agent-1.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T125516-lru-eviction-openswarm-r2/arm-stderr.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/arm-stdout.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/executor-task.md", "results/raw/20261017T125516-lru-eviction-openswarm-r2/executor-usage.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/merge.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/diff.patch", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/events.1.cursor.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/events.1.cursor.jsonl.err", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/result.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/status.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/task.md", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-result.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm.err", "results/raw/20261017T125516-lru-eviction-openswarm-r2/planner-prompt.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T125516-lru-eviction-openswarm-r2/repo", "baseline": "d4c8d592494a4397efa2bcc10c13abdab8afaecc", "measure_from": "d4c8d592494a4397efa2bcc10c13abdab8afaecc"} +{"type": "run", "task": "lru-eviction", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:55:18.473862+00:00", "end": "2026-10-17T12:59:02.894426+00:00", "wall_s": 224.418, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125518-lru-eviction-openswarm-r3/agent-1.jsonl", "session_id": "fa943001-4d2a-4c93-bed4-6e58f3cae92e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 35691, "outputTokens": 4234, "cacheReadTokens": 116992, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "9a57fe25-fbe3-444a-9070-8d8897e28879", "model": "grok-4.7-high", "usage": {"inputTokens": 18287, "outputTokens": 6049, "cacheReadTokens": 83840, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 53978, "outputTokens": 10283, "cacheReadTokens": 200832, "cacheWriteTokens": 0, "total": 265093}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.691149\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.161168\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.429181\n type: 'test'\n ...\n# Subtest: a read protects a key from eviction\nok 4 - a read protects a key from eviction\n ---\n duration_ms: 0.991794\n type: 'test'\n ...\n# Subtest: an update protects a key and does not grow the cache\nok 5 - an update protects a key and does not grow the cache\n ---\n duration_ms: 0.211747\n type: 'test'\n ...\n# Subtest: a missing get does not change eviction order\nok 6 - a missing get does not change eviction order\n ---\n duration_ms: 0.109842\n type: 'test'\n ...\n# Subtest: capacity 1 keeps a used key until a different key is inserted\nok 7 - capacity 1 keeps a used key until a different key is inserted\n ---\n duration_ms: 0.143623\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.679978\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.650414\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.10978\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.118243\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.080189\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.246426\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.320229\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 6 +++++-\n test/lru.test.js | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 49 insertions(+), 1 deletion(-)\n", "session_ids": ["fa943001-4d2a-4c93-bed4-6e58f3cae92e", "9a57fe25-fbe3-444a-9070-8d8897e28879"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T125518-lru-eviction-openswarm-r3/agent-1.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T125518-lru-eviction-openswarm-r3/arm-stderr.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/arm-stdout.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/executor-task.md", "results/raw/20261017T125518-lru-eviction-openswarm-r3/executor-usage.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/merge.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/diff.patch", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/events.1.cursor.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/events.1.cursor.jsonl.err", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/result.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/status.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/task.md", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-result.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm.err", "results/raw/20261017T125518-lru-eviction-openswarm-r3/planner-prompt.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T125518-lru-eviction-openswarm-r3/repo", "baseline": "1550750f5508e6ad4330b1042129522e915a6841", "measure_from": "1550750f5508e6ad4330b1042129522e915a6841"} +{"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:58:47.237775+00:00", "end": "2026-10-17T12:59:12.179541+00:00", "wall_s": 24.94, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/agent-1.jsonl", "session_id": "ca80c62c-3bfe-439a-addf-0a8c91eb9a9b", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 11821, "outputTokens": 890, "cacheReadTokens": 71552, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 11821, "outputTokens": 890, "cacheReadTokens": 71552, "cacheWriteTokens": 0, "total": 84263}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.867402\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.119049\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.145708\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 56.292827\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.104, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.254937\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.154047\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.11432\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.169017\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.472785\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.548395\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["ca80c62c-3bfe-439a-addf-0a8c91eb9a9b"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125847-pagination-off-by-one-baseline-r1/agent-1.jsonl", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/arm-stderr.txt", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/arm-stdout.txt", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T125847-pagination-off-by-one-baseline-r1/repo", "baseline": "1ac3b6ad62041a4269c1c80da546ab6c15b32a4b", "measure_from": "1ac3b6ad62041a4269c1c80da546ab6c15b32a4b"} +{"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:58:55.814224+00:00", "end": "2026-10-17T12:59:27.722094+00:00", "wall_s": 31.906, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/agent-1.jsonl", "session_id": "fc512105-58d6-453d-93f0-1c2203e98e9a", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 11802, "outputTokens": 1298, "cacheReadTokens": 72192, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 11802, "outputTokens": 1298, "cacheReadTokens": 72192, "cacheWriteTokens": 0, "total": 85292}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.983272\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.182395\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.13614\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.185381\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.106, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.200678\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.123256\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.089993\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.1365\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.426632\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.728613\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["fc512105-58d6-453d-93f0-1c2203e98e9a"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125855-pagination-off-by-one-baseline-r2/agent-1.jsonl", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/arm-stderr.txt", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/arm-stdout.txt", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T125855-pagination-off-by-one-baseline-r2/repo", "baseline": "eb85e9e1c8a2ee9613991f5842875fcab6d0cf2f", "measure_from": "eb85e9e1c8a2ee9613991f5842875fcab6d0cf2f"} +{"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:59:02.906210+00:00", "end": "2026-10-17T12:59:38.690315+00:00", "wall_s": 35.783, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/agent-1.jsonl", "session_id": "ea84b8ea-a9d1-4865-97f2-e19774d16d93", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 12208, "outputTokens": 1525, "cacheReadTokens": 72448, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 12208, "outputTokens": 1525, "cacheReadTokens": 72448, "cacheWriteTokens": 0, "total": 86181}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.104, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.74314\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.201965\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.163573\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.561865\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.310074\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.147253\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.110581\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.154054\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.457191\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.488139\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["ea84b8ea-a9d1-4865-97f2-e19774d16d93"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125902-pagination-off-by-one-baseline-r3/agent-1.jsonl", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/arm-stderr.txt", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/arm-stdout.txt", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T125902-pagination-off-by-one-baseline-r3/repo", "baseline": "157e196bf3565069e0e4f0bca7e173bf33e644f8", "measure_from": "157e196bf3565069e0e4f0bca7e173bf33e644f8"} +{"type": "run", "task": "pagination-off-by-one", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:59:12.191722+00:00", "end": "2026-10-17T13:01:11.198057+00:00", "wall_s": 119.003, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/agent-1.jsonl", "session_id": "e91b6775-b573-4a6e-ae7f-028f82371794", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 52090, "outputTokens": 5300, "cacheReadTokens": 224896, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 52090, "outputTokens": 5300, "cacheReadTokens": 224896, "cacheWriteTokens": 0, "total": 282286}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.118, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.975245\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.260761\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 1.227307\n type: 'test'\n ...\n# Subtest: page 1 returns the first pageSize items\nok 4 - page 1 returns the first pageSize items\n ---\n duration_ms: 0.866295\n type: 'test'\n ...\n# Subtest: page 2 returns the next window with no gap or overlap\nok 5 - page 2 returns the next window with no gap or overlap\n ---\n duration_ms: 0.143739\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.097736\n type: 'test'\n ...\n# Subtest: a page past the end returns no items and keeps totalPages\nok 7 - a page past the end returns no items and keeps totalPages\n ---\n duration_ms: 0.162069\n type: 'test'\n ...\n# Subtest: an empty input needs zero pages\nok 8 - an empty input needs zero pages\n ---\n duration_ms: 0.093088\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.106892\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.112, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.673628\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.21747\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.145926\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 1.279397\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.570514\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 71.45117\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 31 insertions(+), 1 deletion(-)\n", "session_ids": ["e91b6775-b573-4a6e-ae7f-028f82371794"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/agent-1.jsonl", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/arm-stderr.txt", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/arm-stdout.txt", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T125912-pagination-off-by-one-superpowers-r1/repo", "baseline": "099493dbf634b33f50cdbe8557a11985c9d9879a", "measure_from": "099493dbf634b33f50cdbe8557a11985c9d9879a"} +{"type": "run", "task": "pagination-off-by-one", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:59:27.733205+00:00", "end": "2026-10-17T13:01:17.080965+00:00", "wall_s": 109.346, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/agent-1.jsonl", "session_id": "97909c90-3ab0-482c-8258-1e684df5eda5", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 38833, "outputTokens": 4767, "cacheReadTokens": 206976, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 38833, "outputTokens": 4767, "cacheReadTokens": 206976, "cacheWriteTokens": 0, "total": 250576}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.101, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.897896\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.204461\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.189743\n type: 'test'\n ...\n# Subtest: page 1 returns the first window\nok 4 - page 1 returns the first window\n ---\n duration_ms: 1.686635\n type: 'test'\n ...\n# Subtest: page 2 returns the next window with no gap\nok 5 - page 2 returns the next window with no gap\n ---\n duration_ms: 0.167183\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.122275\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.421699\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.092, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.141416\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.191176\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.116595\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.957212\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.454862\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.87494\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 16 ++++++++++++++++\n 2 files changed, 17 insertions(+), 1 deletion(-)\n", "session_ids": ["97909c90-3ab0-482c-8258-1e684df5eda5"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/agent-1.jsonl", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/arm-stderr.txt", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/arm-stdout.txt", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T125927-pagination-off-by-one-superpowers-r2/repo", "baseline": "f5a8294a8c5bb7e2ade96c4ce8c657aa67ab888a", "measure_from": "f5a8294a8c5bb7e2ade96c4ce8c657aa67ab888a"} +{"type": "run", "task": "pagination-off-by-one", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:59:38.701981+00:00", "end": "2026-10-17T13:01:28.563222+00:00", "wall_s": 109.858, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/agent-1.jsonl", "session_id": "853f9654-fb32-4266-964d-1f7233881a41", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49189, "outputTokens": 4578, "cacheReadTokens": 213888, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49189, "outputTokens": 4578, "cacheReadTokens": 213888, "cacheWriteTokens": 0, "total": 267655}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.111, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.048749\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.226228\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.170319\n type: 'test'\n ...\n# Subtest: page 1 returns the first window\nok 4 - page 1 returns the first window\n ---\n duration_ms: 1.864153\n type: 'test'\n ...\n# Subtest: page 2 returns the next window with no gap\nok 5 - page 2 returns the next window with no gap\n ---\n duration_ms: 0.197587\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.138557\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 73.531844\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.098, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.248434\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.165475\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.111535\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.9465\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.451636\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.093109\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 16 ++++++++++++++++\n 2 files changed, 17 insertions(+), 1 deletion(-)\n", "session_ids": ["853f9654-fb32-4266-964d-1f7233881a41"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/agent-1.jsonl", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/arm-stderr.txt", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/arm-stdout.txt", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T125938-pagination-off-by-one-superpowers-r3/repo", "baseline": "fe97160c67304a9ebd995287b6149eab5ffd87d3", "measure_from": "fe97160c67304a9ebd995287b6149eab5ffd87d3"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:01:11.210601+00:00", "end": "2026-10-17T13:02:40.346043+00:00", "wall_s": 89.133, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/agent-1.jsonl", "session_id": "86674e57-7daa-4a55-ab6c-0bace1f3b129", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 17202, "outputTokens": 2771, "cacheReadTokens": 67200, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "49f6a845-4581-4df5-8779-e0cf5461a939", "model": "grok-4.7-high", "usage": {"inputTokens": 12679, "outputTokens": 986, "cacheReadTokens": 75648, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 29881, "outputTokens": 3757, "cacheReadTokens": 142848, "cacheWriteTokens": 0, "total": 176486}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.941467\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.143655\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.15477\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 58.441961\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.092, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.157273\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.119628\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.0858\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.137082\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.485214\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 56.022935\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["86674e57-7daa-4a55-ab6c-0bace1f3b129", "49f6a845-4581-4df5-8779-e0cf5461a939"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/agent-1.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/arm-stderr.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/arm-stdout.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/executor-task.md", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/executor-usage.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/merge.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/diff.patch", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/events.1.cursor.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/events.1.cursor.jsonl.err", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/result.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/status.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/task.md", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-result.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm.err", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/planner-prompt.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T130111-pagination-off-by-one-openswarm-r1/repo", "baseline": "70dc830d40581eea0f1025242fc2cb9408a85370", "measure_from": "70dc830d40581eea0f1025242fc2cb9408a85370"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:01:17.093642+00:00", "end": "2026-10-17T13:02:50.528936+00:00", "wall_s": 93.433, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/agent-1.jsonl", "session_id": "04a6467c-0746-4740-a72f-b0954d9e8048", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 17887, "outputTokens": 3300, "cacheReadTokens": 87808, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "9c3928c6-27da-4e8e-9483-a645fb78c532", "model": "grok-4.7-high", "usage": {"inputTokens": 12947, "outputTokens": 882, "cacheReadTokens": 58752, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 30834, "outputTokens": 4182, "cacheReadTokens": 146560, "cacheWriteTokens": 0, "total": 181576}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.04262\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.154666\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.148434\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.373784\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.215865\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.140167\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.101159\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.144643\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.491384\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.423454\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["04a6467c-0746-4740-a72f-b0954d9e8048", "9c3928c6-27da-4e8e-9483-a645fb78c532"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/agent-1.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/arm-stderr.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/arm-stdout.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/executor-task.md", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/executor-usage.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/merge.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/diff.patch", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/events.1.cursor.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/events.1.cursor.jsonl.err", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/result.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/status.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/task.md", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-result.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm.err", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/planner-prompt.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T130117-pagination-off-by-one-openswarm-r2/repo", "baseline": "4d45057fd3463d5eb159bdc29f8d0c9b750e23cf", "measure_from": "4d45057fd3463d5eb159bdc29f8d0c9b750e23cf"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:01:28.575793+00:00", "end": "2026-10-17T13:03:18.474702+00:00", "wall_s": 109.896, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/agent-1.jsonl", "session_id": "0d53f802-1bcd-4e3e-890f-3b155bc1d221", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18268, "outputTokens": 3518, "cacheReadTokens": 112256, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "8c0adf5c-136a-44cc-befb-9268ade774d8", "model": "grok-4.7-high", "usage": {"inputTokens": 13513, "outputTokens": 1574, "cacheReadTokens": 58880, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 31781, "outputTokens": 5092, "cacheReadTokens": 171136, "cacheWriteTokens": 0, "total": 208009}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.851679\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.14026\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.129146\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.115911\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.097, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.359887\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.220623\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.147357\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.214495\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.626041\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.304978\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["0d53f802-1bcd-4e3e-890f-3b155bc1d221", "8c0adf5c-136a-44cc-befb-9268ade774d8"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/agent-1.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/arm-stderr.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/arm-stdout.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/executor-task.md", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/executor-usage.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/merge.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/diff.patch", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/events.1.cursor.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/events.1.cursor.jsonl.err", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/result.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/status.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/task.md", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-result.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm.err", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/planner-prompt.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T130128-pagination-off-by-one-openswarm-r3/repo", "baseline": "462f472f0cd21680487f7e9f564e28d9239a6ec3", "measure_from": "462f472f0cd21680487f7e9f564e28d9239a6ec3"} +{"type": "run", "task": "path-normalise", "arm": "baseline", "repeat": 1, "start": "2026-10-17T13:02:40.357781+00:00", "end": "2026-10-17T13:03:44.016760+00:00", "wall_s": 63.657, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130240-path-normalise-baseline-r1/agent-1.jsonl", "session_id": "8b988725-ea61-455c-95a3-1b0695c6b1fc", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 14379, "outputTokens": 3061, "cacheReadTokens": 92160, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 14379, "outputTokens": 3061, "cacheReadTokens": 92160, "cacheWriteTokens": 0, "total": 109600}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.113, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.726362\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.137229\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.094587\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.41591\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 72.039377\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.114, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.34911\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.333673\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.226276\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.200338\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.406466\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 79.472075\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n 1 file changed, 8 insertions(+)\n", "session_ids": ["8b988725-ea61-455c-95a3-1b0695c6b1fc"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130240-path-normalise-baseline-r1/agent-1.jsonl", "results/raw/20261017T130240-path-normalise-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T130240-path-normalise-baseline-r1/arm-stderr.txt", "results/raw/20261017T130240-path-normalise-baseline-r1/arm-stdout.txt", "results/raw/20261017T130240-path-normalise-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T130240-path-normalise-baseline-r1/repo", "baseline": "6da2210ab4db9932a6f6bca4961d9c46c17cee0b", "measure_from": "6da2210ab4db9932a6f6bca4961d9c46c17cee0b"} +{"type": "run", "task": "path-normalise", "arm": "baseline", "repeat": 2, "start": "2026-10-17T13:02:50.541557+00:00", "end": "2026-10-17T13:03:51.098298+00:00", "wall_s": 60.555, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130250-path-normalise-baseline-r2/agent-1.jsonl", "session_id": "54b3a809-3fce-4849-812c-c8a3a326582f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 52115, "outputTokens": 2828, "cacheReadTokens": 54144, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 52115, "outputTokens": 2828, "cacheReadTokens": 54144, "cacheWriteTokens": 0, "total": 109087}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.104, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.710498\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.122123\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.091378\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.312144\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.757137\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.733066\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.168159\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.111486\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.150096\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.364671\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.264862\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n 1 file changed, 8 insertions(+)\n", "session_ids": ["54b3a809-3fce-4849-812c-c8a3a326582f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130250-path-normalise-baseline-r2/agent-1.jsonl", "results/raw/20261017T130250-path-normalise-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T130250-path-normalise-baseline-r2/arm-stderr.txt", "results/raw/20261017T130250-path-normalise-baseline-r2/arm-stdout.txt", "results/raw/20261017T130250-path-normalise-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T130250-path-normalise-baseline-r2/repo", "baseline": "456519dee46ac5146a306daca3f49791ffa66d56", "measure_from": "456519dee46ac5146a306daca3f49791ffa66d56"} +{"type": "run", "task": "path-normalise", "arm": "baseline", "repeat": 3, "start": "2026-10-17T13:03:18.486949+00:00", "end": "2026-10-17T13:04:12.913540+00:00", "wall_s": 54.425, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130318-path-normalise-baseline-r3/agent-1.jsonl", "session_id": "ffb4644b-813c-4fe3-b1e9-78faebb821ce", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 13498, "outputTokens": 2560, "cacheReadTokens": 72832, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 13498, "outputTokens": 2560, "cacheReadTokens": 72832, "cacheWriteTokens": 0, "total": 88890}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.794757\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.13427\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.117864\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.336965\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.323211\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.113, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.040068\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.239179\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.117413\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.101998\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.280063\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.232852\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n 1 file changed, 8 insertions(+)\n", "session_ids": ["ffb4644b-813c-4fe3-b1e9-78faebb821ce"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130318-path-normalise-baseline-r3/agent-1.jsonl", "results/raw/20261017T130318-path-normalise-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T130318-path-normalise-baseline-r3/arm-stderr.txt", "results/raw/20261017T130318-path-normalise-baseline-r3/arm-stdout.txt", "results/raw/20261017T130318-path-normalise-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T130318-path-normalise-baseline-r3/repo", "baseline": "c95fb0254bcecede9f32c6638dd4616ebf051b50", "measure_from": "c95fb0254bcecede9f32c6638dd4616ebf051b50"} +{"type": "run", "task": "path-normalise", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T13:03:44.029443+00:00", "end": "2026-10-17T13:05:45.255606+00:00", "wall_s": 121.223, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130344-path-normalise-superpowers-r1/agent-1.jsonl", "session_id": "eb948584-dee8-4304-81f2-e9a96386350a", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 52666, "outputTokens": 5627, "cacheReadTokens": 199552, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 52666, "outputTokens": 5627, "cacheReadTokens": 199552, "cacheWriteTokens": 0, "total": 257845}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.112, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 1.4597\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.274941\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.243461\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.423583\n type: 'test'\n ...\n# Subtest: pops a parent segment\nok 5 - pops a parent segment\n ---\n duration_ms: 0.20131\n type: 'test'\n ...\n# Subtest: keeps a parent segment that leaves the relative root\nok 6 - keeps a parent segment that leaves the relative root\n ---\n duration_ms: 0.117237\n type: 'test'\n ...\n# Subtest: discards parent segments above the absolute root\nok 7 - discards parent segments above the absolute root\n ---\n duration_ms: 0.210308\n type: 'test'\n ...\n# Subtest: an absolute path that walks back to the root is /\nok 8 - an absolute path that walks back to the root is /\n ---\n duration_ms: 0.137887\n type: 'test'\n ...\n# Subtest: a relative path that walks back to the start is .\nok 9 - a relative path that walks back to the start is .\n ---\n duration_ms: 0.298915\n type: 'test'\n ...\n# Subtest: drops empty segments around a parent segment\nok 10 - drops empty segments around a parent segment\n ---\n duration_ms: 0.334273\n type: 'test'\n ...\n1..10\n# tests 10\n# suites 0\n# pass 10\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 70.417815\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.093, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.684007\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.150076\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.091453\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.077137\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.230762\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 56.337429\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 24 ++++++++++++++++++++++++\n 2 files changed, 32 insertions(+)\n", "session_ids": ["eb948584-dee8-4304-81f2-e9a96386350a"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T130344-path-normalise-superpowers-r1/agent-1.jsonl", "results/raw/20261017T130344-path-normalise-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T130344-path-normalise-superpowers-r1/arm-stderr.txt", "results/raw/20261017T130344-path-normalise-superpowers-r1/arm-stdout.txt", "results/raw/20261017T130344-path-normalise-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T130344-path-normalise-superpowers-r1/repo", "baseline": "ec2c9d60b575b204ede6f27d8949b43198862050", "measure_from": "ec2c9d60b575b204ede6f27d8949b43198862050"} +{"type": "run", "task": "path-normalise", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T13:04:12.926027+00:00", "end": "2026-10-17T13:05:52.825401+00:00", "wall_s": 99.897, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130412-path-normalise-superpowers-r3/agent-1.jsonl", "session_id": "654287ec-2783-4453-b745-adc8595acd99", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 48995, "outputTokens": 4542, "cacheReadTokens": 193408, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 48995, "outputTokens": 4542, "cacheReadTokens": 193408, "cacheWriteTokens": 0, "total": 246945}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.883551\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.210628\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.131058\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 1.368464\n type: 'test'\n ...\n# Subtest: pops the previous real segment for a parent\nok 5 - pops the previous real segment for a parent\n ---\n duration_ms: 0.235378\n type: 'test'\n ...\n# Subtest: discards parent segments that would climb above the root\nok 6 - discards parent segments that would climb above the root\n ---\n duration_ms: 0.11463\n type: 'test'\n ...\n# Subtest: keeps parent segments that leave the relative root\nok 7 - keeps parent segments that leave the relative root\n ---\n duration_ms: 0.158849\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 62.375527\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.126, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.698026\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.157467\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.095313\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.082501\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.24859\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 89.459029\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 13 +++++++++++++\n 2 files changed, 21 insertions(+)\n", "session_ids": ["654287ec-2783-4453-b745-adc8595acd99"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T130412-path-normalise-superpowers-r3/agent-1.jsonl", "results/raw/20261017T130412-path-normalise-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T130412-path-normalise-superpowers-r3/arm-stderr.txt", "results/raw/20261017T130412-path-normalise-superpowers-r3/arm-stdout.txt", "results/raw/20261017T130412-path-normalise-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T130412-path-normalise-superpowers-r3/repo", "baseline": "88110063b6a520c2ea02d04e9547f1e020532958", "measure_from": "88110063b6a520c2ea02d04e9547f1e020532958"} +{"type": "run", "task": "path-normalise", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T13:03:51.110013+00:00", "end": "2026-10-17T13:06:01.019748+00:00", "wall_s": 129.907, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130351-path-normalise-superpowers-r2/agent-1.jsonl", "session_id": "7480dce2-eb30-4013-8d47-0225e65bfb5e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 96480, "outputTokens": 5687, "cacheReadTokens": 180480, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 96480, "outputTokens": 5687, "cacheReadTokens": 180480, "cacheWriteTokens": 0, "total": 282647}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.106, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.743466\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.105218\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.081997\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.437693\n type: 'test'\n ...\n# Subtest: pops the previous segment for a parent\nok 5 - pops the previous segment for a parent\n ---\n duration_ms: 0.17162\n type: 'test'\n ...\n# Subtest: keeps a parent that climbs out of a relative path\nok 6 - keeps a parent that climbs out of a relative path\n ---\n duration_ms: 0.093878\n type: 'test'\n ...\n# Subtest: discards parents that would climb above the root\nok 7 - discards parents that would climb above the root\n ---\n duration_ms: 0.128457\n type: 'test'\n ...\n# Subtest: a relative path that collapses completely is the current directory\nok 8 - a relative path that collapses completely is the current directory\n ---\n duration_ms: 0.07089\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.655459\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.674823\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.235019\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.161503\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.128777\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.279068\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.159991\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 17 +++++++++++++++++\n 2 files changed, 25 insertions(+)\n", "session_ids": ["7480dce2-eb30-4013-8d47-0225e65bfb5e"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T130351-path-normalise-superpowers-r2/agent-1.jsonl", "results/raw/20261017T130351-path-normalise-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T130351-path-normalise-superpowers-r2/arm-stderr.txt", "results/raw/20261017T130351-path-normalise-superpowers-r2/arm-stdout.txt", "results/raw/20261017T130351-path-normalise-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T130351-path-normalise-superpowers-r2/repo", "baseline": "e8fbe10b50e13b5a5a892371f69ebd7ed4d8464e", "measure_from": "e8fbe10b50e13b5a5a892371f69ebd7ed4d8464e"} +{"type": "run", "task": "path-normalise", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:06:01.031257+00:00", "end": "2026-10-17T13:08:29.838324+00:00", "wall_s": 148.804, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130601-path-normalise-openswarm-r3/agent-1.jsonl", "session_id": "b7c53adf-e9d7-463e-aee5-ddf730c5832f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 19242, "outputTokens": 5111, "cacheReadTokens": 67712, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "fa1e016c-d869-488f-81c0-970729e781f1", "model": "grok-4.7-high", "usage": {"inputTokens": 13836, "outputTokens": 1840, "cacheReadTokens": 60032, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 33078, "outputTokens": 6951, "cacheReadTokens": 127744, "cacheWriteTokens": 0, "total": 167773}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.101, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.635059\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.139318\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.102184\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 1.201876\n type: 'test'\n ...\n# Subtest: drops empty segments\nok 5 - drops empty segments\n ---\n duration_ms: 0.25908\n type: 'test'\n ...\n# Subtest: resolves parent segments\nok 6 - resolves parent segments\n ---\n duration_ms: 0.154795\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.743903\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.840606\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.192391\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.11104\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.116512\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 5.627921\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.118119\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 29 +++++++++++++++++++++++++++++\n 2 files changed, 37 insertions(+)\n", "session_ids": ["b7c53adf-e9d7-463e-aee5-ddf730c5832f", "fa1e016c-d869-488f-81c0-970729e781f1"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130601-path-normalise-openswarm-r3/agent-1.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T130601-path-normalise-openswarm-r3/arm-stderr.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/arm-stdout.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/executor-task.md", "results/raw/20261017T130601-path-normalise-openswarm-r3/executor-usage.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/merge.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/diff.patch", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/events.1.cursor.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/events.1.cursor.jsonl.err", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/result.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/status.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/task.md", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-result.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm.err", "results/raw/20261017T130601-path-normalise-openswarm-r3/planner-prompt.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T130601-path-normalise-openswarm-r3/repo", "baseline": "23a8e5b973630a1299aba52c274685b14172a54a", "measure_from": "23a8e5b973630a1299aba52c274685b14172a54a"} +{"type": "run", "task": "path-normalise", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:05:52.837890+00:00", "end": "2026-10-17T13:09:08.942604+00:00", "wall_s": 196.102, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130552-path-normalise-openswarm-r2/agent-1.jsonl", "session_id": "e4d92edb-1fb3-49a9-bdfa-ab6ef28034cf", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 20753, "outputTokens": 6378, "cacheReadTokens": 88704, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "20bdf188-a620-4460-a1ab-ebd3f63e05ae", "model": "grok-4.7-high", "usage": {"inputTokens": 30393, "outputTokens": 3274, "cacheReadTokens": 46464, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 51146, "outputTokens": 9652, "cacheReadTokens": 135168, "cacheWriteTokens": 0, "total": 195966}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.112, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.672913\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.148195\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.084661\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 1.231509\n type: 'test'\n ...\n# Subtest: collapses parent segments on a relative path\nok 5 - collapses parent segments on a relative path\n ---\n duration_ms: 0.31609\n type: 'test'\n ...\n# Subtest: never climbs above the root on an absolute path\nok 6 - never climbs above the root on an absolute path\n ---\n duration_ms: 0.122995\n type: 'test'\n ...\n# Subtest: keeps segments that only look like dots\nok 7 - keeps segments that only look like dots\n ---\n duration_ms: 0.133956\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.755923\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.728385\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.16738\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.13056\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.106408\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.292113\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.268455\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+)\n", "session_ids": ["e4d92edb-1fb3-49a9-bdfa-ab6ef28034cf", "20bdf188-a620-4460-a1ab-ebd3f63e05ae"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130552-path-normalise-openswarm-r2/agent-1.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T130552-path-normalise-openswarm-r2/arm-stderr.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/arm-stdout.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/executor-task.md", "results/raw/20261017T130552-path-normalise-openswarm-r2/executor-usage.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/merge.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/diff.patch", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/events.1.cursor.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/events.1.cursor.jsonl.err", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/result.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/status.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/task.md", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-result.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm.err", "results/raw/20261017T130552-path-normalise-openswarm-r2/planner-prompt.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T130552-path-normalise-openswarm-r2/repo", "baseline": "e094adb8ddb5c1505c27cbcd6500a4cdf183c1f2", "measure_from": "e094adb8ddb5c1505c27cbcd6500a4cdf183c1f2"} +{"type": "run", "task": "retry-backoff", "arm": "baseline", "repeat": 1, "start": "2026-10-17T13:08:29.851134+00:00", "end": "2026-10-17T13:09:16.735590+00:00", "wall_s": 46.883, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130829-retry-backoff-baseline-r1/agent-1.jsonl", "session_id": "902c95f6-3826-4208-9133-06f6e9d2e5e9", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 12618, "outputTokens": 2082, "cacheReadTokens": 72576, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 12618, "outputTokens": 2082, "cacheReadTokens": 72576, "cacheWriteTokens": 0, "total": 87276}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.058, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n", "session_ids": ["902c95f6-3826-4208-9133-06f6e9d2e5e9"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130829-retry-backoff-baseline-r1/agent-1.jsonl", "results/raw/20261017T130829-retry-backoff-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T130829-retry-backoff-baseline-r1/arm-stderr.txt", "results/raw/20261017T130829-retry-backoff-baseline-r1/arm-stdout.txt", "results/raw/20261017T130829-retry-backoff-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T130829-retry-backoff-baseline-r1/repo", "baseline": "7e0faf2d1046c32108890158738a375920c02aad", "measure_from": "7e0faf2d1046c32108890158738a375920c02aad"} +{"type": "run", "task": "retry-backoff", "arm": "baseline", "repeat": 2, "start": "2026-10-17T13:09:08.955094+00:00", "end": "2026-10-17T13:10:02.880871+00:00", "wall_s": 53.924, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130908-retry-backoff-baseline-r2/agent-1.jsonl", "session_id": "e84e7b8b-2ec3-486c-9a03-6cb85c852eb5", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 13222, "outputTokens": 2541, "cacheReadTokens": 90880, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 13222, "outputTokens": 2541, "cacheReadTokens": 90880, "cacheWriteTokens": 0, "total": 106643}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.043, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n", "session_ids": ["e84e7b8b-2ec3-486c-9a03-6cb85c852eb5"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130908-retry-backoff-baseline-r2/agent-1.jsonl", "results/raw/20261017T130908-retry-backoff-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T130908-retry-backoff-baseline-r2/arm-stderr.txt", "results/raw/20261017T130908-retry-backoff-baseline-r2/arm-stdout.txt", "results/raw/20261017T130908-retry-backoff-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T130908-retry-backoff-baseline-r2/repo", "baseline": "09e7cdbb18c0255f51dafc1aac2e0283d7babfad", "measure_from": "09e7cdbb18c0255f51dafc1aac2e0283d7babfad"} +{"type": "run", "task": "retry-backoff", "arm": "baseline", "repeat": 3, "start": "2026-10-17T13:09:16.747115+00:00", "end": "2026-10-17T13:10:11.693727+00:00", "wall_s": 54.945, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130916-retry-backoff-baseline-r3/agent-1.jsonl", "session_id": "365c1f91-65da-4d42-960e-a11154029551", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 45488, "outputTokens": 2567, "cacheReadTokens": 40064, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 45488, "outputTokens": 2567, "cacheReadTokens": 40064, "cacheWriteTokens": 0, "total": 88119}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.053, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.055, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n", "session_ids": ["365c1f91-65da-4d42-960e-a11154029551"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130916-retry-backoff-baseline-r3/agent-1.jsonl", "results/raw/20261017T130916-retry-backoff-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T130916-retry-backoff-baseline-r3/arm-stderr.txt", "results/raw/20261017T130916-retry-backoff-baseline-r3/arm-stdout.txt", "results/raw/20261017T130916-retry-backoff-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T130916-retry-backoff-baseline-r3/repo", "baseline": "647036efbb5fbfcbc66a83d7c779b952204d6c19", "measure_from": "647036efbb5fbfcbc66a83d7c779b952204d6c19"} +{"type": "run", "task": "path-normalise", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:05:45.268716+00:00", "end": "2026-10-17T13:11:37.558757+00:00", "wall_s": 352.287, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130545-path-normalise-openswarm-r1/agent-1.jsonl", "session_id": "6721afa7-76d6-4a2e-8d69-dbcbb0b78b98", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 38760, "outputTokens": 6885, "cacheReadTokens": 52352, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "8ffbee4d-3d82-4c42-8a4f-10ce90b159fb", "model": "grok-4.7-high", "usage": {"inputTokens": 35285, "outputTokens": 10118, "cacheReadTokens": 98944, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 74045, "outputTokens": 17003, "cacheReadTokens": 151296, "cacheWriteTokens": 0, "total": 242344}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.094, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.742754\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.126056\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.084245\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.322753\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 57.760535\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.111, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.290968\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.385785\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.21235\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.159762\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.458905\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 74.074339\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 5 +++++\n 1 file changed, 5 insertions(+)\n", "session_ids": ["6721afa7-76d6-4a2e-8d69-dbcbb0b78b98", "8ffbee4d-3d82-4c42-8a4f-10ce90b159fb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130545-path-normalise-openswarm-r1/agent-1.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T130545-path-normalise-openswarm-r1/arm-stderr.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/arm-stdout.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/executor-task.md", "results/raw/20261017T130545-path-normalise-openswarm-r1/executor-usage.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/merge.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/diff.patch", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/events.1.cursor.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/events.1.cursor.jsonl.err", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/result.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/status.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/task.md", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-result.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm.err", "results/raw/20261017T130545-path-normalise-openswarm-r1/planner-prompt.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T130545-path-normalise-openswarm-r1/repo", "baseline": "e2f7bc150ebaffffc67ce265fd3046a4d6152d0a", "measure_from": "e2f7bc150ebaffffc67ce265fd3046a4d6152d0a"} +{"type": "run", "task": "retry-backoff", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T13:10:02.892209+00:00", "end": "2026-10-17T13:12:11.054351+00:00", "wall_s": 128.159, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131002-retry-backoff-superpowers-r1/agent-1.jsonl", "session_id": "34160877-8eab-4623-a624-208c8ba88a0f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 25056, "outputTokens": 5551, "cacheReadTokens": 235264, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 25056, "outputTokens": 5551, "cacheReadTokens": 235264, "cacheWriteTokens": 0, "total": 265871}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "tests/test_retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 8 +++++++-\n tests/test_retry.py | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 44 insertions(+), 1 deletion(-)\n", "session_ids": ["34160877-8eab-4623-a624-208c8ba88a0f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T131002-retry-backoff-superpowers-r1/agent-1.jsonl", "results/raw/20261017T131002-retry-backoff-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T131002-retry-backoff-superpowers-r1/arm-stderr.txt", "results/raw/20261017T131002-retry-backoff-superpowers-r1/arm-stdout.txt", "results/raw/20261017T131002-retry-backoff-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T131002-retry-backoff-superpowers-r1/repo", "baseline": "0d7f5cd1d843c0cd8aad6958ae4f82de8cd36149", "measure_from": "0d7f5cd1d843c0cd8aad6958ae4f82de8cd36149"} +{"type": "run", "task": "retry-backoff", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:12:11.066773+00:00", "end": "2026-10-17T13:14:25.603498+00:00", "wall_s": 134.533, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131211-retry-backoff-openswarm-r1/agent-1.jsonl", "session_id": "4952ad6f-a522-4008-a4a4-a1e4780e205c", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 32734, "outputTokens": 3792, "cacheReadTokens": 51584, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "434e6fa3-df69-427c-b6be-95b1000e2551", "model": "grok-4.7-high", "usage": {"inputTokens": 40497, "outputTokens": 2370, "cacheReadTokens": 68480, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 73231, "outputTokens": 6162, "cacheReadTokens": 120064, "cacheWriteTokens": 0, "total": 199457}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py"], "diff_stat": " retry.py | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n", "session_ids": ["4952ad6f-a522-4008-a4a4-a1e4780e205c", "434e6fa3-df69-427c-b6be-95b1000e2551"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T131211-retry-backoff-openswarm-r1/agent-1.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T131211-retry-backoff-openswarm-r1/arm-stderr.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/arm-stdout.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/executor-task.md", "results/raw/20261017T131211-retry-backoff-openswarm-r1/executor-usage.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/merge.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/diff.patch", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/events.1.cursor.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/events.1.cursor.jsonl.err", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/result.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/status.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/task.md", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-result.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm.err", "results/raw/20261017T131211-retry-backoff-openswarm-r1/planner-prompt.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T131211-retry-backoff-openswarm-r1/repo", "baseline": "a4da828d78c556da4d508792bc028c8b31af3600", "measure_from": "a4da828d78c556da4d508792bc028c8b31af3600"} +{"type": "run", "task": "retry-backoff", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T13:10:11.705775+00:00", "end": "2026-10-17T13:15:07.639154+00:00", "wall_s": 295.929, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131011-retry-backoff-superpowers-r2/agent-1.jsonl", "session_id": "bbbdaa5a-4f14-43f5-8d47-f3797d6b904c", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 79675, "outputTokens": 12690, "cacheReadTokens": 538496, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 79675, "outputTokens": 12690, "cacheReadTokens": 538496, "cacheWriteTokens": 0, "total": 630861}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.068, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "tests/test_retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 8 ++++++-\n tests/test_retry.py | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 1 deletion(-)\n", "session_ids": ["bbbdaa5a-4f14-43f5-8d47-f3797d6b904c"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T131011-retry-backoff-superpowers-r2/agent-1.jsonl", "results/raw/20261017T131011-retry-backoff-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T131011-retry-backoff-superpowers-r2/arm-stderr.txt", "results/raw/20261017T131011-retry-backoff-superpowers-r2/arm-stdout.txt", "results/raw/20261017T131011-retry-backoff-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T131011-retry-backoff-superpowers-r2/repo", "baseline": "2b5f49ca63d841c7f4f5dcc2cf157142918f3697", "measure_from": "2b5f49ca63d841c7f4f5dcc2cf157142918f3697"} +{"type": "run", "task": "retry-backoff", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:14:25.615840+00:00", "end": "2026-10-17T13:19:02.025151+00:00", "wall_s": 276.406, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131425-retry-backoff-openswarm-r2/agent-1.jsonl", "session_id": "487118ed-456f-4b8d-a692-2210a7e4a3e2", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 58326, "outputTokens": 5099, "cacheReadTokens": 132480, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "ce26d9fb-2c88-4402-b58b-535a31dc5196", "model": "grok-4.7-high", "usage": {"inputTokens": 14050, "outputTokens": 2316, "cacheReadTokens": 76928, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 72376, "outputTokens": 7415, "cacheReadTokens": 209408, "cacheWriteTokens": 0, "total": 289199}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.042, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py"], "diff_stat": " retry.py | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n", "session_ids": ["487118ed-456f-4b8d-a692-2210a7e4a3e2", "ce26d9fb-2c88-4402-b58b-535a31dc5196"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T131425-retry-backoff-openswarm-r2/agent-1.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T131425-retry-backoff-openswarm-r2/arm-stderr.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/arm-stdout.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/executor-task.md", "results/raw/20261017T131425-retry-backoff-openswarm-r2/executor-usage.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/merge.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/diff.patch", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/events.1.cursor.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/events.1.cursor.jsonl.err", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/result.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/status.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/task.md", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-result.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm.err", "results/raw/20261017T131425-retry-backoff-openswarm-r2/planner-prompt.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T131425-retry-backoff-openswarm-r2/repo", "baseline": "3a736438eeee73ba4ff7ec9e553201c66684aa11", "measure_from": "3a736438eeee73ba4ff7ec9e553201c66684aa11"} +{"type": "run", "task": "retry-backoff", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:15:07.671156+00:00", "end": "2026-10-17T13:19:13.362941+00:00", "wall_s": 245.688, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131507-retry-backoff-openswarm-r3/agent-1.jsonl", "session_id": "9276e7ec-5383-402a-b348-613697a9b500", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 37714, "outputTokens": 5807, "cacheReadTokens": 71680, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "b87be238-9abc-47de-b25d-a40a244c7e31", "model": "grok-4.7-high", "usage": {"inputTokens": 13312, "outputTokens": 1911, "cacheReadTokens": 58752, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 51026, "outputTokens": 7718, "cacheReadTokens": 130432, "cacheWriteTokens": 0, "total": 189176}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.053, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py"], "diff_stat": " retry.py | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n", "session_ids": ["9276e7ec-5383-402a-b348-613697a9b500", "b87be238-9abc-47de-b25d-a40a244c7e31"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T131507-retry-backoff-openswarm-r3/agent-1.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T131507-retry-backoff-openswarm-r3/arm-stderr.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/arm-stdout.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/executor-task.md", "results/raw/20261017T131507-retry-backoff-openswarm-r3/executor-usage.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/merge.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/diff.patch", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/events.1.cursor.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/events.1.cursor.jsonl.err", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/result.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/status.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/task.md", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-result.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm.err", "results/raw/20261017T131507-retry-backoff-openswarm-r3/planner-prompt.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T131507-retry-backoff-openswarm-r3/repo", "baseline": "ad8ce22fa044b800d7a509388d18faa6bac45787", "measure_from": "ad8ce22fa044b800d7a509388d18faa6bac45787"} +{"type": "run", "task": "semver-compare", "arm": "baseline", "repeat": 2, "start": "2026-10-17T13:19:13.374520+00:00", "end": "2026-10-17T13:21:25.704226+00:00", "wall_s": 132.327, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131913-semver-compare-baseline-r2/agent-1.jsonl", "session_id": "9cecc896-5fab-49d2-87a4-042e16f8ea29", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 27955, "outputTokens": 6842, "cacheReadTokens": 87040, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 27955, "outputTokens": 6842, "cacheReadTokens": 87040, "cacheWriteTokens": 0, "total": 121837}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.14, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.180928\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.238499\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 93.253633\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.129, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.162322\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.303509\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.261518\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.510675\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.142288\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.432825\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 90.397016\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 50 insertions(+), 7 deletions(-)\n", "session_ids": ["9cecc896-5fab-49d2-87a4-042e16f8ea29"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T131913-semver-compare-baseline-r2/agent-1.jsonl", "results/raw/20261017T131913-semver-compare-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T131913-semver-compare-baseline-r2/arm-stderr.txt", "results/raw/20261017T131913-semver-compare-baseline-r2/arm-stdout.txt", "results/raw/20261017T131913-semver-compare-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T131913-semver-compare-baseline-r2/repo", "baseline": "083546a707481b99a367c129b415654571f4f136", "measure_from": "083546a707481b99a367c129b415654571f4f136"} +{"type": "run", "task": "semver-compare", "arm": "baseline", "repeat": 1, "start": "2026-10-17T13:19:02.035865+00:00", "end": "2026-10-17T13:21:30.639241+00:00", "wall_s": 148.6, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131902-semver-compare-baseline-r1/agent-1.jsonl", "session_id": "fe6f24aa-68a7-465a-8923-bb7f9dde021b", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 34698, "outputTokens": 7482, "cacheReadTokens": 60288, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 34698, "outputTokens": 7482, "cacheReadTokens": 60288, "cacheWriteTokens": 0, "total": 102468}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.109, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.072133\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.160021\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.646707\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.11, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.916703\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.252004\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.1629\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.32105\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.131217\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.399782\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.361578\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 52 insertions(+), 7 deletions(-)\n", "session_ids": ["fe6f24aa-68a7-465a-8923-bb7f9dde021b"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T131902-semver-compare-baseline-r1/agent-1.jsonl", "results/raw/20261017T131902-semver-compare-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T131902-semver-compare-baseline-r1/arm-stderr.txt", "results/raw/20261017T131902-semver-compare-baseline-r1/arm-stdout.txt", "results/raw/20261017T131902-semver-compare-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T131902-semver-compare-baseline-r1/repo", "baseline": "4924ad073b14ecfe24c2c0f5f40e8873439192b5", "measure_from": "4924ad073b14ecfe24c2c0f5f40e8873439192b5"} +{"type": "run", "task": "retry-backoff", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T13:11:37.570230+00:00", "end": "2026-10-17T13:24:24.193890+00:00", "wall_s": 766.617, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131137-retry-backoff-superpowers-r3/agent-1.jsonl", "session_id": "bf5449c2-6d0d-40ce-9750-370575c7abcb", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 117181, "outputTokens": 13851, "cacheReadTokens": 910976, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 117181, "outputTokens": 13851, "cacheReadTokens": 910976, "cacheWriteTokens": 0, "total": 1042008}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.137, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.065, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.001s\n\nOK\n"}, "files_changed": ["retry.py", "tests/test_retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 9 +++++-\n tests/test_retry.py | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 92 insertions(+), 1 deletion(-)\n", "session_ids": ["bf5449c2-6d0d-40ce-9750-370575c7abcb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T131137-retry-backoff-superpowers-r3/agent-1.jsonl", "results/raw/20261017T131137-retry-backoff-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T131137-retry-backoff-superpowers-r3/arm-stderr.txt", "results/raw/20261017T131137-retry-backoff-superpowers-r3/arm-stdout.txt", "results/raw/20261017T131137-retry-backoff-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T131137-retry-backoff-superpowers-r3/repo", "baseline": "a07138b67ed1b3cbf13c3cda4086dedb9612bd40", "measure_from": "a07138b67ed1b3cbf13c3cda4086dedb9612bd40"} +{"type": "run", "task": "semver-compare", "arm": "baseline", "repeat": 3, "start": "2026-10-17T13:21:25.725032+00:00", "end": "2026-10-17T13:25:47.803579+00:00", "wall_s": 262.07, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132125-semver-compare-baseline-r3/agent-1.jsonl", "session_id": "3a2bb53b-de43-4dd3-a68f-5d06d8be85c8", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 73, "outputTokens": 255, "cacheReadTokens": 24448, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 73, "outputTokens": 255, "cacheReadTokens": 24448, "cacheWriteTokens": 0, "total": 24776}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.288, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 7.773292\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.192536\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 173.138872\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.141, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.873951\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.274262\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.176701\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.226237\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.123498\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.415083\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 71.595668\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 55 insertions(+), 7 deletions(-)\n", "session_ids": ["3a2bb53b-de43-4dd3-a68f-5d06d8be85c8"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T132125-semver-compare-baseline-r3/agent-1.jsonl", "results/raw/20261017T132125-semver-compare-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T132125-semver-compare-baseline-r3/arm-stderr.txt", "results/raw/20261017T132125-semver-compare-baseline-r3/arm-stdout.txt", "results/raw/20261017T132125-semver-compare-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T132125-semver-compare-baseline-r3/repo", "baseline": "90c7ccde498551a3e5dc0631e1e4a0b0171e1fe5", "measure_from": "90c7ccde498551a3e5dc0631e1e4a0b0171e1fe5"} +{"type": "run", "task": "semver-compare", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T13:21:30.660803+00:00", "end": "2026-10-17T13:26:37.407255+00:00", "wall_s": 306.739, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132130-semver-compare-superpowers-r1/agent-1.jsonl", "session_id": "3f1538c4-dfe2-48aa-b711-fd757657e3c8", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49669, "outputTokens": 10152, "cacheReadTokens": 281600, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49669, "outputTokens": 10152, "cacheReadTokens": 281600, "cacheWriteTokens": 0, "total": 341421}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.244, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.421009\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.218436\n type: 'test'\n ...\n# Subtest: minor and patch break ties when majors match\nok 3 - minor and patch break ties when majors match\n ---\n duration_ms: 0.133919\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same release\nok 4 - a pre-release is less than the same release\n ---\n duration_ms: 0.211008\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare by number, then text, then length\nok 5 - pre-release identifiers compare by number, then text, then length\n ---\n duration_ms: 0.299621\n type: 'test'\n ...\n# Subtest: build metadata is ignored\nok 6 - build metadata is ignored\n ---\n duration_ms: 0.112388\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 7 - invalid versions throw\n ---\n duration_ms: 0.527867\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 121.423075\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.136, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.15635\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.230047\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.191487\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 2.033391\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.135929\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.486811\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 79.45789\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 58 ++++++++++++++++++++++++++++++++++++++++++++++-------\n test/semver.test.js | 30 +++++++++++++++++++++++++++\n 2 files changed, 81 insertions(+), 7 deletions(-)\n", "session_ids": ["3f1538c4-dfe2-48aa-b711-fd757657e3c8"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T132130-semver-compare-superpowers-r1/agent-1.jsonl", "results/raw/20261017T132130-semver-compare-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T132130-semver-compare-superpowers-r1/arm-stderr.txt", "results/raw/20261017T132130-semver-compare-superpowers-r1/arm-stdout.txt", "results/raw/20261017T132130-semver-compare-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T132130-semver-compare-superpowers-r1/repo", "baseline": "6b5343bcb6112470fd2026be180a2f99a831fb73", "measure_from": "6b5343bcb6112470fd2026be180a2f99a831fb73"} +{"type": "run", "task": "semver-compare", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T13:24:24.228135+00:00", "end": "2026-10-17T13:31:02.789230+00:00", "wall_s": 398.531, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132424-semver-compare-superpowers-r2/agent-1.jsonl", "session_id": "f0682c72-2302-412b-893c-6e90790e4380", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 46700, "outputTokens": 12009, "cacheReadTokens": 422400, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 46700, "outputTokens": 12009, "cacheReadTokens": 422400, "cacheWriteTokens": 0, "total": 481109}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.435, "stdout_tail": "e: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 3.860476\n type: 'test'\n ...\n# Subtest: a lower minor version sorts first when majors match\nok 3 - a lower minor version sorts first when majors match\n ---\n duration_ms: 0.130472\n type: 'test'\n ...\n# Subtest: a lower patch version sorts first when major and minor match\nok 4 - a lower patch version sorts first when major and minor match\n ---\n duration_ms: 2.859308\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same version without one\nok 5 - a pre-release is less than the same version without one\n ---\n duration_ms: 0.278833\n type: 'test'\n ...\n# Subtest: numeric pre-release identifiers compare as numbers\nok 6 - numeric pre-release identifiers compare as numbers\n ---\n duration_ms: 0.2135\n type: 'test'\n ...\n# Subtest: non-numeric pre-release identifiers compare as strings\nok 7 - non-numeric pre-release identifiers compare as strings\n ---\n duration_ms: 0.094359\n type: 'test'\n ...\n# Subtest: a numeric pre-release identifier is less than a non-numeric one\nok 8 - a numeric pre-release identifier is less than a non-numeric one\n ---\n duration_ms: 0.080361\n type: 'test'\n ...\n# Subtest: a shorter pre-release that prefixes the other sorts first\nok 9 - a shorter pre-release that prefixes the other sorts first\n ---\n duration_ms: 0.281596\n type: 'test'\n ...\n# Subtest: equal pre-releases compare equal\nok 10 - equal pre-releases compare equal\n ---\n duration_ms: 2.235816\n type: 'test'\n ...\n# Subtest: build metadata after + is ignored\nok 11 - build metadata after + is ignored\n ---\n duration_ms: 0.25845\n type: 'test'\n ...\n# Subtest: invalid versions throw an error that mentions invalid semver\nok 12 - invalid versions throw an error that mentions invalid semver\n ---\n duration_ms: 0.460405\n type: 'test'\n ...\n1..12\n# tests 12\n# suites 0\n# pass 12\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 265.262497\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.309, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.049101\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.274383\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.149147\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 2.08405\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.10958\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 1.13052\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 225.370062\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 61 ++++++++++++++++++++++++++++++++++++++++++++++-------\n test/semver.test.js | 54 +++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 107 insertions(+), 8 deletions(-)\n", "session_ids": ["f0682c72-2302-412b-893c-6e90790e4380"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T132424-semver-compare-superpowers-r2/agent-1.jsonl", "results/raw/20261017T132424-semver-compare-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T132424-semver-compare-superpowers-r2/arm-stderr.txt", "results/raw/20261017T132424-semver-compare-superpowers-r2/arm-stdout.txt", "results/raw/20261017T132424-semver-compare-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T132424-semver-compare-superpowers-r2/repo", "baseline": "77502b42fb3fe250194fd6615e3949c9b33ff7b6", "measure_from": "77502b42fb3fe250194fd6615e3949c9b33ff7b6"} +{"type": "run", "task": "semver-compare", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:26:37.431600+00:00", "end": "2026-10-17T13:32:46.201009+00:00", "wall_s": 368.762, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132637-semver-compare-openswarm-r1/agent-1.jsonl", "session_id": "b7ed2832-6c93-4b21-ad8b-52108ffb93dd", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 24043, "outputTokens": 9699, "cacheReadTokens": 66176, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "056b0858-fe9d-406a-b521-7870330ae151", "model": "grok-4.7-high", "usage": {"inputTokens": 21302, "outputTokens": 7723, "cacheReadTokens": 70016, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 45345, "outputTokens": 17422, "cacheReadTokens": 136192, "cacheWriteTokens": 0, "total": 198959}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.767, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 2.71663\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.291564\n type: 'test'\n ...\n# Subtest: minor and patch are compared as integers\nok 3 - minor and patch are compared as integers\n ---\n duration_ms: 0.24867\n type: 'test'\n ...\n# Subtest: pre-release precedence\nok 4 - pre-release precedence\n ---\n duration_ms: 2.032619\n type: 'test'\n ...\n# Subtest: build metadata does not affect precedence\nok 5 - build metadata does not affect precedence\n ---\n duration_ms: 0.268782\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.726424\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 556.805669\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.192, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.219421\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.278329\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.132749\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.14618\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.921607\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.396336\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 97.138598\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 70 +++++++++++++++++++++++++++++++++++++++++++++------\n test/semver.test.js | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 134 insertions(+), 8 deletions(-)\n", "session_ids": ["b7ed2832-6c93-4b21-ad8b-52108ffb93dd", "056b0858-fe9d-406a-b521-7870330ae151"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T132637-semver-compare-openswarm-r1/agent-1.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T132637-semver-compare-openswarm-r1/arm-stderr.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/arm-stdout.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/executor-task.md", "results/raw/20261017T132637-semver-compare-openswarm-r1/executor-usage.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/merge.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/diff.patch", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/events.1.cursor.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/events.1.cursor.jsonl.err", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/result.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/status.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/task.md", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-result.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm.err", "results/raw/20261017T132637-semver-compare-openswarm-r1/planner-prompt.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T132637-semver-compare-openswarm-r1/repo", "baseline": "d29ca47a00addf6d431fa9a353f398d1b352bab8", "measure_from": "d29ca47a00addf6d431fa9a353f398d1b352bab8"} +{"type": "run", "task": "semver-compare", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T13:25:47.848796+00:00", "end": "2026-10-17T13:35:22.029662+00:00", "wall_s": 574.176, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132547-semver-compare-superpowers-r3/agent-1.jsonl", "session_id": "94c4f8cc-0fcf-4075-93ce-91d9868ba14e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 1867, "outputTokens": 547, "cacheReadTokens": 104960, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 1867, "outputTokens": 547, "cacheReadTokens": 104960, "cacheWriteTokens": 0, "total": 107374}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.163, "stdout_tail": "patch version sorts first when major and minor match\nok 4 - a lower patch version sorts first when major and minor match\n ---\n duration_ms: 0.166503\n type: 'test'\n ...\n# Subtest: minor differences outrank patch differences\nok 5 - minor differences outrank patch differences\n ---\n duration_ms: 0.100516\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same core version\nok 6 - a pre-release is less than the same core version\n ---\n duration_ms: 0.199919\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically when both are numeric\nok 7 - pre-release identifiers compare numerically when both are numeric\n ---\n duration_ms: 0.225502\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare as strings otherwise\nok 8 - pre-release identifiers compare as strings otherwise\n ---\n duration_ms: 0.087779\n type: 'test'\n ...\n# Subtest: a numeric pre-release identifier is less than a non-numeric one\nok 9 - a numeric pre-release identifier is less than a non-numeric one\n ---\n duration_ms: 0.266943\n type: 'test'\n ...\n# Subtest: a shorter pre-release list is less when it is a prefix\nok 10 - a shorter pre-release list is less when it is a prefix\n ---\n duration_ms: 0.33882\n type: 'test'\n ...\n# Subtest: equal pre-release versions compare equal\nok 11 - equal pre-release versions compare equal\n ---\n duration_ms: 0.135862\n type: 'test'\n ...\n# Subtest: build metadata is ignored\nok 12 - build metadata is ignored\n ---\n duration_ms: 0.082799\n type: 'test'\n ...\n# Subtest: leading zeroes in a numeric core component are invalid\nok 13 - leading zeroes in a numeric core component are invalid\n ---\n duration_ms: 0.453263\n type: 'test'\n ...\n# Subtest: an invalid version throws an error mentioning invalid semver\nok 14 - an invalid version throws an error mentioning invalid semver\n ---\n duration_ms: 0.144849\n type: 'test'\n ...\n1..14\n# tests 14\n# suites 0\n# pass 14\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 80.146917\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.115, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.454457\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.441119\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.324973\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.411717\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.154834\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.562751\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 82.680911\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 58 ++++++++++++++++++++++++++++++++++++++++++------\n test/semver.test.js | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 115 insertions(+), 7 deletions(-)\n", "session_ids": ["94c4f8cc-0fcf-4075-93ce-91d9868ba14e"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T132547-semver-compare-superpowers-r3/agent-1.jsonl", "results/raw/20261017T132547-semver-compare-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T132547-semver-compare-superpowers-r3/arm-stderr.txt", "results/raw/20261017T132547-semver-compare-superpowers-r3/arm-stdout.txt", "results/raw/20261017T132547-semver-compare-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T132547-semver-compare-superpowers-r3/repo", "baseline": "b1593da14facf805844084997c518e78dc3adf8c", "measure_from": "b1593da14facf805844084997c518e78dc3adf8c"} +{"type": "run", "task": "semver-compare", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:31:02.834416+00:00", "end": "2026-10-17T13:37:15.693410+00:00", "wall_s": 372.855, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133102-semver-compare-openswarm-r2/agent-1.jsonl", "session_id": "6f9a69ad-1223-46cc-a18a-349e2a0b3166", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 23761, "outputTokens": 9087, "cacheReadTokens": 120832, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "33a88cf2-a6f0-498d-8b8d-778b5c18396f", "model": "grok-4.7-high", "usage": {"inputTokens": 36171, "outputTokens": 7292, "cacheReadTokens": 50816, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 59932, "outputTokens": 16379, "cacheReadTokens": 171648, "cacheWriteTokens": 0, "total": 247959}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 0.946837\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.170946\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.975914\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.748155\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.285442\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.119097\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 0.958552\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.143623\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.539929\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.85829\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 75 insertions(+), 9 deletions(-)\n", "session_ids": ["6f9a69ad-1223-46cc-a18a-349e2a0b3166", "33a88cf2-a6f0-498d-8b8d-778b5c18396f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T133102-semver-compare-openswarm-r2/agent-1.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T133102-semver-compare-openswarm-r2/arm-stderr.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/arm-stdout.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/executor-task.md", "results/raw/20261017T133102-semver-compare-openswarm-r2/executor-usage.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/merge.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/diff.patch", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/events.1.cursor.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/events.1.cursor.jsonl.err", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/result.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/status.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/task.md", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-result.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm.err", "results/raw/20261017T133102-semver-compare-openswarm-r2/planner-prompt.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T133102-semver-compare-openswarm-r2/repo", "baseline": "1a84b20b4ba4a11204d656448ebad355f91c132d", "measure_from": "1a84b20b4ba4a11204d656448ebad355f91c132d"} +{"type": "run", "task": "semver-compare", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:32:46.276821+00:00", "end": "2026-10-17T13:39:57.885122+00:00", "wall_s": 431.605, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133246-semver-compare-openswarm-r3/agent-1.jsonl", "session_id": "f4969927-63cd-4400-bc29-ccef175aa9ed", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 31939, "outputTokens": 10596, "cacheReadTokens": 46848, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "e8e98887-e9e4-4231-a9fd-6b553a584159", "model": "grok-4.7-high", "usage": {"inputTokens": 68019, "outputTokens": 6030, "cacheReadTokens": 18816, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 99958, "outputTokens": 16626, "cacheReadTokens": 65664, "cacheWriteTokens": 0, "total": 182248}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.12, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.069332\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.177606\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 83.632953\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.833265\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.213051\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.225387\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 0.995114\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.147209\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.393059\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.456276\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 42 +++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 39 insertions(+), 3 deletions(-)\n", "session_ids": ["f4969927-63cd-4400-bc29-ccef175aa9ed", "e8e98887-e9e4-4231-a9fd-6b553a584159"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T133246-semver-compare-openswarm-r3/agent-1.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T133246-semver-compare-openswarm-r3/arm-stderr.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/arm-stdout.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/executor-task.md", "results/raw/20261017T133246-semver-compare-openswarm-r3/executor-usage.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/merge.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/diff.patch", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/events.1.cursor.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/events.1.cursor.jsonl.err", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/result.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/status.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/task.md", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-result.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm.err", "results/raw/20261017T133246-semver-compare-openswarm-r3/planner-prompt.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T133246-semver-compare-openswarm-r3/repo", "baseline": "7205d2102be360bf05fac530b6a62d99806ac25a", "measure_from": "7205d2102be360bf05fac530b6a62d99806ac25a"} diff --git a/bench/e2e/results/chart.png b/bench/e2e/results/chart.png new file mode 100644 index 0000000..4a12b29 Binary files /dev/null and b/bench/e2e/results/chart.png differ diff --git a/bench/e2e/results/chart.svg b/bench/e2e/results/chart.svg new file mode 100644 index 0000000..67cbbd6 --- /dev/null +++ b/bench/e2e/results/chart.svg @@ -0,0 +1,61 @@ + + +Task completion by arm + +Success % +percent + +baseline + +superpowers + +openflow + +openswarm + +Median minutes +minutes, whisker = IQR + + + + +baseline + + + + +superpowers + + + + +openflow + + + + +openswarm + +Median total tokens +tokens, whisker = IQR + + + + +baseline + + + + +superpowers + + + + +openflow + + + + +openswarm + diff --git a/bench/e2e/results/e2e-2026-10-17-openflow-r1.jsonl b/bench/e2e/results/e2e-2026-10-17-openflow-r1.jsonl new file mode 100644 index 0000000..53ec835 --- /dev/null +++ b/bench/e2e/results/e2e-2026-10-17-openflow-r1.jsonl @@ -0,0 +1,7 @@ +{"type": "environment", "date": "2026-10-17T13:19:21.884343+00:00", "cpu": "Intel(R) Xeon(R) Processor", "cpu_count": 8, "os": "Linux-6.12.94+-x86_64-with-glibc2.41", "python": "3.13.5", "node": "v22.23.3", "cursor_agent": "2026.10.16-e3cb207", "git": "git version 2.47.3", "commits": {"superpowers": "bb92a77741419a4ab5f06e711a283343f1ada0c3", "openflow": "58b5a3fa0899f6e639454f0e4a7247dec98f6986", "openswarm": "a48255a9cebafc48b0a7cdd39b6590a7dc193496"}, "openflow_tgz": "/surlymochan-openflow-0.2.1.tgz", "openswarm_dir": "", "superpowers_dir": "", "model": "grok-4.7-high", "cursor_flags": ["-p", "--trust", "--force", "--model", "grok-4.7-high", "--output-format", "stream-json"], "run_timeout_s": 1200, "command": ["/usr/bin/python3", "run.py", "--arms", "openflow", "--tasks", "date-range-parse,lru-eviction,pagination-off-by-one,path-normalise,retry-backoff,semver-compare", "--repeats", "1", "--concurrency", "3", "--out", "results/e2e-2026-10-17-openflow-r1.jsonl"], "arms": "openflow", "tasks": "date-range-parse,lru-eviction,pagination-off-by-one,path-normalise,retry-backoff,semver-compare", "repeats": 1, "concurrency": 3, "out": "results/e2e-2026-10-17-openflow-r1.jsonl"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:19:22.473848+00:00", "end": "2026-10-17T13:37:16.821402+00:00", "wall_s": 1074.334, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/agent-1.jsonl", "session_id": "43fa26d1-bfcc-4f61-8001-c7a02287d98f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 207571, "outputTokens": 33430, "cacheReadTokens": 5217152, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 207571, "outputTokens": 33430, "cacheReadTokens": 5217152, "cacheWriteTokens": 0, "total": 5458153}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.098, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.032882\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.127215\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.969402\n type: 'test'\n ...\n# Subtest: page 1 is the first window\nok 4 - page 1 is the first window\n ---\n duration_ms: 0.829964\n type: 'test'\n ...\n# Subtest: page 2 continues with no gap or overlap\nok 5 - page 2 continues with no gap or overlap\n ---\n duration_ms: 0.143545\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.100416\n type: 'test'\n ...\n# Subtest: an empty input needs zero pages\nok 7 - an empty input needs zero pages\n ---\n duration_ms: 0.189605\n type: 'test'\n ...\n# Subtest: a page past the end returns no items\nok 8 - a page past the end returns no items\n ---\n duration_ms: 0.111498\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.176549\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.329951\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.213587\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.133956\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.184804\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.487608\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.976132\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "specs/"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 31 +++++++++++++++++++++++++++++++\n 2 files changed, 32 insertions(+), 1 deletion(-)\n", "session_ids": ["43fa26d1-bfcc-4f61-8001-c7a02287d98f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131922-pagination-off-by-one-openflow-r1/agent-1.jsonl", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/arm-stderr.txt", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/arm-stdout.txt", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/measure-from.txt", "results/raw/20261017T131922-pagination-off-by-one-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T131922-pagination-off-by-one-openflow-r1/repo", "baseline": "a94ee61d8a94d0e795bb8f36bc68eb509cd60b89", "measure_from": "ddc1ff59cb93946819e83bd77804fe7d800d9fd9"} +{"type": "run", "task": "date-range-parse", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:19:22.471732+00:00", "end": "2026-10-17T13:39:15.667559+00:00", "wall_s": 1193.18, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131922-date-range-parse-openflow-r1/agent-1.jsonl", "session_id": "bfc07f0a-c85e-4ba2-8fdb-a3bf208c49a2", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 19475, "outputTokens": 10881, "cacheReadTokens": 1932928, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 19475, "outputTokens": 10881, "cacheReadTokens": 1932928, "cacheWriteTokens": 0, "total": 1963284}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": [".as-xflow/config.json", ".xflow/GOAL.md", "HANDOFF.md", "daterange.py", "specs/README.md", "specs/architecture.md", "specs/changes/fix-inclusive-date-range/issue.md", "specs/changes/fix-inclusive-date-range/merge-product.md", "specs/changes/fix-inclusive-date-range/plan.md", "specs/changes/fix-inclusive-date-range/proposal.md", "specs/changes/fix-inclusive-date-range/spec_delta_review.json", "specs/changes/fix-inclusive-date-range/spec_delta_review.md", "specs/changes/fix-inclusive-date-range/status.json", "specs/changes/fix-inclusive-date-range/tasks.md", "specs/changes/fix-inclusive-date-range/tdd/green-0.json", "specs/changes/fix-inclusive-date-range/tdd/quality-0.json", "specs/changes/fix-inclusive-date-range/tdd/red-0.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/daterange.py", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/README.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/issue.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/merge-product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/plan.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/proposal.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/status.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/changes/fix-inclusive-date-range/tasks.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/green-0/tests/test_daterange.py", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/daterange.py", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/README.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/issue.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/merge-product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/plan.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/proposal.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/status.json", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/changes/fix-inclusive-date-range/tasks.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/product.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/fix-inclusive-date-range/tdd/snapshots/red-0/tests/test_daterange.py", "specs/changes/fix-inclusive-date-range/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_daterange.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", "__pycache__/", "tests/__pycache__/"], "diff_stat": " .as-xflow/config.json | 2 +-\n .xflow/GOAL.md | 17 +\n HANDOFF.md | 92 +++\n daterange.py | 2 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/fix-inclusive-date-range/issue.md | 21 +\n .../fix-inclusive-date-range/merge-product.md | 8 +\n specs/changes/fix-inclusive-date-range/plan.md | 43 ++\n specs/changes/fix-inclusive-date-range/proposal.md | 37 ++\n .../spec_delta_review.json | 129 +++++\n .../fix-inclusive-date-range/spec_delta_review.md | 42 ++\n specs/changes/fix-inclusive-date-range/status.json | 638 +++++++++++++++++++++\n specs/changes/fix-inclusive-date-range/tasks.md | 5 +\n .../fix-inclusive-date-range/tdd/green-0.json | 39 ++\n .../fix-inclusive-date-range/tdd/quality-0.json | 78 +++\n .../fix-inclusive-date-range/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/daterange.py | 26 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../changes/fix-inclusive-date-range/issue.md | 21 +\n .../fix-inclusive-date-range/merge-product.md | 8 +\n .../specs/changes/fix-inclusive-date-range/plan.md | 43 ++\n .../changes/fix-inclusive-date-range/proposal.md | 37 ++\n .../changes/fix-inclusive-date-range/status.json | 611 ++++++++++++++++++++\n .../changes/fix-inclusive-date-range/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_daterange.py | 59 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/daterange.py | 26 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../changes/fix-inclusive-date-range/issue.md | 21 +\n .../fix-inclusive-date-range/merge-product.md | 8 +\n .../specs/changes/fix-inclusive-date-range/plan.md | 43 ++\n .../changes/fix-inclusive-date-range/proposal.md | 37 ++\n .../changes/fix-inclusive-date-range/status.json | 603 +++++++++++++++++++\n .../changes/fix-inclusive-date-range/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_daterange.py | 59 ++\n .../fix-inclusive-date-range/verify_proof.json | 12 +\n specs/product.md | 33 ++\n specs/workflow.md | 49 ++\n tests/test_daterange.py | 32 ++\n 60 files changed, 4817 insertions(+), 2 deletions(-)\n", "session_ids": ["bfc07f0a-c85e-4ba2-8fdb-a3bf208c49a2"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131922-date-range-parse-openflow-r1/agent-1.jsonl", "results/raw/20261017T131922-date-range-parse-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T131922-date-range-parse-openflow-r1/arm-stderr.txt", "results/raw/20261017T131922-date-range-parse-openflow-r1/arm-stdout.txt", "results/raw/20261017T131922-date-range-parse-openflow-r1/measure-from.txt", "results/raw/20261017T131922-date-range-parse-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T131922-date-range-parse-openflow-r1/repo", "baseline": "cbc723af8659954939bbe32437686de8892f98c9", "measure_from": "a36c11461c9af5dc5cf3a82ce7d36b60dd2918ac"} +{"type": "run", "task": "lru-eviction", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:19:22.474027+00:00", "end": "2026-10-17T13:39:22.793976+00:00", "wall_s": 1200.307, "exit_code": 124, "timed_out": true, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131922-lru-eviction-openflow-r1/agent-1.jsonl", "session_id": "3c0593b8-153d-462e-974f-d3f2b260e1b6", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.660907\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.151662\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.310872\n type: 'test'\n ...\n# Subtest: follows use order when a read refreshes a key\nok 4 - follows use order when a read refreshes a key\n ---\n duration_ms: 0.979771\n type: 'test'\n ...\n# Subtest: follows use order when an update refreshes a key without growing\nok 5 - follows use order when an update refreshes a key without growing\n ---\n duration_ms: 0.261745\n type: 'test'\n ...\n# Subtest: follows use order when a missing get leaves eviction unchanged\nok 6 - follows use order when a missing get leaves eviction unchanged\n ---\n duration_ms: 0.110892\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 58.152138\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.77425\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.144788\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.158931\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.120038\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.285345\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.633961\n", "stderr_tail": ""}, "files_changed": [".xflow/GOAL.md", "HANDOFF.md", "specs/README.md", "specs/architecture.md", "specs/changes/lru-use-order/issue.md", "specs/changes/lru-use-order/plan.md", "specs/changes/lru-use-order/proposal.md", "specs/changes/lru-use-order/spec_delta_review.json", "specs/changes/lru-use-order/spec_delta_review.md", "specs/changes/lru-use-order/status.json", "specs/changes/lru-use-order/tasks.md", "specs/changes/lru-use-order/tdd/green-0.json", "specs/changes/lru-use-order/tdd/quality-0.json", "specs/changes/lru-use-order/tdd/red-0.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/lru-use-order/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/package.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/README.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/issue.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/plan.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/proposal.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/status.json", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/changes/lru-use-order/tasks.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/product.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/lru-use-order/tdd/snapshots/green-0/src/lru.js", "specs/changes/lru-use-order/tdd/snapshots/green-0/test/lru.test.js", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/lru-use-order/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/package.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/README.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/issue.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/plan.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/proposal.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/status.json", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/changes/lru-use-order/tasks.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/product.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/lru-use-order/tdd/snapshots/red-0/src/lru.js", "specs/changes/lru-use-order/tdd/snapshots/red-0/test/lru.test.js", "specs/changes/lru-use-order/verify_proof.json", "specs/product.md", "specs/workflow.md", "src/lru.js", "test/lru.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json"], "diff_stat": " .xflow/GOAL.md | 17 +\n HANDOFF.md | 92 +++\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/lru-use-order/issue.md | 5 +\n specs/changes/lru-use-order/plan.md | 52 ++\n specs/changes/lru-use-order/proposal.md | 42 ++\n specs/changes/lru-use-order/spec_delta_review.json | 130 +++++\n specs/changes/lru-use-order/spec_delta_review.md | 40 ++\n specs/changes/lru-use-order/status.json | 619 +++++++++++++++++++++\n specs/changes/lru-use-order/tasks.md | 5 +\n specs/changes/lru-use-order/tdd/green-0.json | 39 ++\n specs/changes/lru-use-order/tdd/quality-0.json | 74 +++\n specs/changes/lru-use-order/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/package.json | 9 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../green-0/specs/changes/lru-use-order/issue.md | 5 +\n .../green-0/specs/changes/lru-use-order/plan.md | 52 ++\n .../specs/changes/lru-use-order/proposal.md | 42 ++\n .../specs/changes/lru-use-order/status.json | 611 ++++++++++++++++++++\n .../green-0/specs/changes/lru-use-order/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../lru-use-order/tdd/snapshots/green-0/src/lru.js | 34 ++\n .../tdd/snapshots/green-0/test/lru.test.js | 56 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../lru-use-order/tdd/snapshots/red-0/package.json | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/lru-use-order/issue.md | 5 +\n .../red-0/specs/changes/lru-use-order/plan.md | 52 ++\n .../red-0/specs/changes/lru-use-order/proposal.md | 42 ++\n .../red-0/specs/changes/lru-use-order/status.json | 603 ++++++++++++++++++++\n .../red-0/specs/changes/lru-use-order/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../lru-use-order/tdd/snapshots/red-0/src/lru.js | 27 +\n .../tdd/snapshots/red-0/test/lru.test.js | 56 ++\n specs/changes/lru-use-order/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n src/lru.js | 11 +-\n test/lru.test.js | 33 ++\n 58 files changed, 4777 insertions(+), 2 deletions(-)\n", "session_ids": ["3c0593b8-153d-462e-974f-d3f2b260e1b6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131922-lru-eviction-openflow-r1/agent-1.jsonl", "results/raw/20261017T131922-lru-eviction-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T131922-lru-eviction-openflow-r1/arm-stderr.txt", "results/raw/20261017T131922-lru-eviction-openflow-r1/arm-stdout.txt", "results/raw/20261017T131922-lru-eviction-openflow-r1/measure-from.txt", "results/raw/20261017T131922-lru-eviction-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T131922-lru-eviction-openflow-r1/repo", "baseline": "767c50844040360af10d56fc02591bc13d199d47", "measure_from": "3adcd616beb12c969ecbb284a78ddefaefa5fd5d"} +{"type": "run", "task": "path-normalise", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:37:16.835864+00:00", "end": "2026-10-17T13:51:10.778437+00:00", "wall_s": 833.929, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133716-path-normalise-openflow-r1/agent-1.jsonl", "session_id": "0c1ce27a-d222-407c-ac62-70af85a99577", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 149144, "outputTokens": 37704, "cacheReadTokens": 5061504, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 149144, "outputTokens": 37704, "cacheReadTokens": 5061504, "cacheWriteTokens": 0, "total": 5248352}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.126, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 1.050767\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.168249\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.108992\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.512732\n type: 'test'\n ...\n# Subtest: pops a parent segment\nok 5 - pops a parent segment\n ---\n duration_ms: 0.238133\n type: 'test'\n ...\n# Subtest: keeps a parent that leaves the relative root\nok 6 - keeps a parent that leaves the relative root\n ---\n duration_ms: 0.179071\n type: 'test'\n ...\n# Subtest: does not climb above an absolute root\nok 7 - does not climb above an absolute root\n ---\n duration_ms: 0.190836\n type: 'test'\n ...\n# Subtest: drops empty segments from doubled or trailing slashes\nok 8 - drops empty segments from doubled or trailing slashes\n ---\n duration_ms: 0.106447\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 72.703467\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.125, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.515012\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.277121\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.16489\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.150882\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.350284\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 83.899682\n", "stderr_tail": ""}, "files_changed": ["HANDOFF.md", "specs/README.md", "specs/architecture.md", "specs/changes/normalise-parent-segments/issue.md", "specs/changes/normalise-parent-segments/plan.md", "specs/changes/normalise-parent-segments/proposal.md", "specs/changes/normalise-parent-segments/spec_delta_review.json", "specs/changes/normalise-parent-segments/spec_delta_review.md", "specs/changes/normalise-parent-segments/status.json", "specs/changes/normalise-parent-segments/tasks.md", "specs/changes/normalise-parent-segments/tdd/green-0.json", "specs/changes/normalise-parent-segments/tdd/quality-0.json", "specs/changes/normalise-parent-segments/tdd/red-0.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/package.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/README.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/issue.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/plan.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/proposal.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/status.json", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/changes/normalise-parent-segments/tasks.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/product.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/src/normalise.js", "specs/changes/normalise-parent-segments/tdd/snapshots/green-0/test/normalise.test.js", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/package.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/README.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/issue.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/plan.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/proposal.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/status.json", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/changes/normalise-parent-segments/tasks.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/product.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/src/normalise.js", "specs/changes/normalise-parent-segments/tdd/snapshots/red-0/test/normalise.test.js", "specs/changes/normalise-parent-segments/verify_proof.json", "specs/product.md", "specs/workflow.md", "src/normalise.js", "test/normalise.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/"], "diff_stat": " HANDOFF.md | 92 +++\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/normalise-parent-segments/issue.md | 21 +\n specs/changes/normalise-parent-segments/plan.md | 46 ++\n .../changes/normalise-parent-segments/proposal.md | 39 ++\n .../spec_delta_review.json | 111 ++++\n .../normalise-parent-segments/spec_delta_review.md | 39 ++\n .../changes/normalise-parent-segments/status.json | 638 +++++++++++++++++++++\n specs/changes/normalise-parent-segments/tasks.md | 5 +\n .../normalise-parent-segments/tdd/green-0.json | 39 ++\n .../normalise-parent-segments/tdd/quality-0.json | 74 +++\n .../normalise-parent-segments/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 66 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/package.json | 9 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../changes/normalise-parent-segments/issue.md | 21 +\n .../changes/normalise-parent-segments/plan.md | 46 ++\n .../changes/normalise-parent-segments/proposal.md | 39 ++\n .../changes/normalise-parent-segments/status.json | 611 ++++++++++++++++++++\n .../changes/normalise-parent-segments/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/src/normalise.js | 23 +\n .../tdd/snapshots/green-0/test/normalise.test.js | 41 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 51 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/package.json | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../changes/normalise-parent-segments/issue.md | 21 +\n .../changes/normalise-parent-segments/plan.md | 46 ++\n .../changes/normalise-parent-segments/proposal.md | 39 ++\n .../changes/normalise-parent-segments/status.json | 603 +++++++++++++++++++\n .../changes/normalise-parent-segments/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/src/normalise.js | 15 +\n .../tdd/snapshots/red-0/test/normalise.test.js | 41 ++\n .../normalise-parent-segments/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n src/normalise.js | 10 +-\n test/normalise.test.js | 22 +\n 57 files changed, 4724 insertions(+), 1 deletion(-)\n", "session_ids": ["0c1ce27a-d222-407c-ac62-70af85a99577"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T133716-path-normalise-openflow-r1/agent-1.jsonl", "results/raw/20261017T133716-path-normalise-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T133716-path-normalise-openflow-r1/arm-stderr.txt", "results/raw/20261017T133716-path-normalise-openflow-r1/arm-stdout.txt", "results/raw/20261017T133716-path-normalise-openflow-r1/measure-from.txt", "results/raw/20261017T133716-path-normalise-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T133716-path-normalise-openflow-r1/repo", "baseline": "837e8ffcb353dabbca72e61c9cffcf97f1664943", "measure_from": "ec3ac64afd61a8865335124998730d496ac7fe01"} +{"type": "run", "task": "retry-backoff", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:39:15.680582+00:00", "end": "2026-10-17T13:54:22.720224+00:00", "wall_s": 907.026, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133915-retry-backoff-openflow-r1/agent-1.jsonl", "session_id": "08f72a12-61e9-4102-9e4d-4608145ae96c", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 198345, "outputTokens": 40814, "cacheReadTokens": 5839744, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 198345, "outputTokens": 40814, "cacheReadTokens": 5839744, "cacheWriteTokens": 0, "total": 6078903}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.042, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": [".as-xflow/config.json", "HANDOFF.md", "retry.py", "specs/README.md", "specs/architecture.md", "specs/changes/retry-backoff/issue.md", "specs/changes/retry-backoff/plan.md", "specs/changes/retry-backoff/proposal.md", "specs/changes/retry-backoff/spec_delta_review.json", "specs/changes/retry-backoff/spec_delta_review.md", "specs/changes/retry-backoff/status.json", "specs/changes/retry-backoff/tasks.md", "specs/changes/retry-backoff/tdd/green-0.json", "specs/changes/retry-backoff/tdd/quality-0.json", "specs/changes/retry-backoff/tdd/red-0.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/retry-backoff/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/retry.py", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/README.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/issue.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/plan.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/proposal.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/status.json", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/changes/retry-backoff/tasks.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/product.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/retry-backoff/tdd/snapshots/green-0/tests/test_retry.py", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/retry-backoff/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/retry.py", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/README.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/issue.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/plan.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/proposal.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/status.json", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/changes/retry-backoff/tasks.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/product.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/retry-backoff/tdd/snapshots/red-0/tests/test_retry.py", "specs/changes/retry-backoff/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_retry.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " .as-xflow/config.json | 2 +-\n HANDOFF.md | 92 +++\n retry.py | 9 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/retry-backoff/issue.md | 21 +\n specs/changes/retry-backoff/plan.md | 31 +\n specs/changes/retry-backoff/proposal.md | 37 ++\n specs/changes/retry-backoff/spec_delta_review.json | 99 ++++\n specs/changes/retry-backoff/spec_delta_review.md | 39 ++\n specs/changes/retry-backoff/status.json | 638 +++++++++++++++++++++\n specs/changes/retry-backoff/tasks.md | 5 +\n specs/changes/retry-backoff/tdd/green-0.json | 38 ++\n specs/changes/retry-backoff/tdd/quality-0.json | 75 +++\n specs/changes/retry-backoff/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../retry-backoff/tdd/snapshots/green-0/retry.py | 14 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../green-0/specs/changes/retry-backoff/issue.md | 21 +\n .../green-0/specs/changes/retry-backoff/plan.md | 31 +\n .../specs/changes/retry-backoff/proposal.md | 37 ++\n .../specs/changes/retry-backoff/status.json | 611 ++++++++++++++++++++\n .../green-0/specs/changes/retry-backoff/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_retry.py | 76 +++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../retry-backoff/tdd/snapshots/red-0/retry.py | 7 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/retry-backoff/issue.md | 21 +\n .../red-0/specs/changes/retry-backoff/plan.md | 31 +\n .../red-0/specs/changes/retry-backoff/proposal.md | 37 ++\n .../red-0/specs/changes/retry-backoff/status.json | 603 +++++++++++++++++++\n .../red-0/specs/changes/retry-backoff/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_retry.py | 76 +++\n specs/changes/retry-backoff/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_retry.py | 50 ++\n 56 files changed, 4715 insertions(+), 2 deletions(-)\n", "session_ids": ["08f72a12-61e9-4102-9e4d-4608145ae96c"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T133915-retry-backoff-openflow-r1/agent-1.jsonl", "results/raw/20261017T133915-retry-backoff-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T133915-retry-backoff-openflow-r1/arm-stderr.txt", "results/raw/20261017T133915-retry-backoff-openflow-r1/arm-stdout.txt", "results/raw/20261017T133915-retry-backoff-openflow-r1/measure-from.txt", "results/raw/20261017T133915-retry-backoff-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T133915-retry-backoff-openflow-r1/repo", "baseline": "bfbe8f47eed7677d1e602614786a10a397be8381", "measure_from": "906fc5980859b7a7067980e23a268b41e171fb3d"} +{"type": "run", "task": "semver-compare", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:39:22.806044+00:00", "end": "2026-10-17T13:55:03.872261+00:00", "wall_s": 941.051, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133922-semver-compare-openflow-r1/agent-1.jsonl", "session_id": "8dbf6589-8358-45da-afd9-4689d754b166", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 147069, "outputTokens": 43160, "cacheReadTokens": 5209984, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 147069, "outputTokens": 43160, "cacheReadTokens": 5209984, "cacheWriteTokens": 0, "total": 5400213}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.11, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.208397\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.266172\n type: 'test'\n ...\n# Subtest: minor and patch decide order after major\nok 3 - minor and patch decide order after major\n ---\n duration_ms: 1.204727\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same version without one\nok 4 - a pre-release is less than the same version without one\n ---\n duration_ms: 0.310276\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare by semver rules\nok 5 - pre-release identifiers compare by semver rules\n ---\n duration_ms: 0.243688\n type: 'test'\n ...\n# Subtest: numeric pre-release identifiers are less than non-numeric ones\nok 6 - numeric pre-release identifiers are less than non-numeric ones\n ---\n duration_ms: 0.130863\n type: 'test'\n ...\n# Subtest: a shorter pre-release list that is a prefix sorts first\nok 7 - a shorter pre-release list that is a prefix sorts first\n ---\n duration_ms: 0.163125\n type: 'test'\n ...\n# Subtest: build metadata is ignored\nok 8 - build metadata is ignored\n ---\n duration_ms: 0.144565\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 9 - invalid versions throw\n ---\n duration_ms: 0.864487\n type: 'test'\n ...\n1..9\n# tests 9\n# suites 0\n# pass 9\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 70.154733\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.88688\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.297552\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.152928\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.034996\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.110558\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.379523\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 68.905659\n", "stderr_tail": ""}, "files_changed": [".as-xflow/config.json", "HANDOFF.md", "specs/README.md", "specs/architecture.md", "specs/changes/compare-semver/issue.md", "specs/changes/compare-semver/plan.md", "specs/changes/compare-semver/proposal.md", "specs/changes/compare-semver/spec_delta_review.json", "specs/changes/compare-semver/spec_delta_review.md", "specs/changes/compare-semver/status.json", "specs/changes/compare-semver/tasks.md", "specs/changes/compare-semver/tdd/green-0.json", "specs/changes/compare-semver/tdd/quality-0.json", "specs/changes/compare-semver/tdd/red-0.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/compare-semver/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/compare-semver/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/compare-semver/tdd/snapshots/green-0/package.json", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/README.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/issue.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/plan.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/proposal.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/status.json", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/changes/compare-semver/tasks.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/product.md", "specs/changes/compare-semver/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/compare-semver/tdd/snapshots/green-0/src/semver.js", "specs/changes/compare-semver/tdd/snapshots/green-0/test/semver.test.js", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/compare-semver/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/compare-semver/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/compare-semver/tdd/snapshots/red-0/package.json", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/README.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/issue.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/plan.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/proposal.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/status.json", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/changes/compare-semver/tasks.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/product.md", "specs/changes/compare-semver/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/compare-semver/tdd/snapshots/red-0/src/semver.js", "specs/changes/compare-semver/tdd/snapshots/red-0/test/semver.test.js", "specs/changes/compare-semver/verify_proof.json", "specs/product.md", "specs/workflow.md", "src/semver.js", "test/semver.test.js", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/"], "diff_stat": " .as-xflow/config.json | 2 +-\n HANDOFF.md | 92 +++\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/compare-semver/issue.md | 25 +\n specs/changes/compare-semver/plan.md | 48 ++\n specs/changes/compare-semver/proposal.md | 38 ++\n .../changes/compare-semver/spec_delta_review.json | 108 ++++\n specs/changes/compare-semver/spec_delta_review.md | 38 ++\n specs/changes/compare-semver/status.json | 638 +++++++++++++++++++++\n specs/changes/compare-semver/tasks.md | 5 +\n specs/changes/compare-semver/tdd/green-0.json | 39 ++\n specs/changes/compare-semver/tdd/quality-0.json | 75 +++\n specs/changes/compare-semver/tdd/red-0.json | 38 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/package.json | 9 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../green-0/specs/changes/compare-semver/issue.md | 25 +\n .../green-0/specs/changes/compare-semver/plan.md | 48 ++\n .../specs/changes/compare-semver/proposal.md | 38 ++\n .../specs/changes/compare-semver/status.json | 611 ++++++++++++++++++++\n .../green-0/specs/changes/compare-semver/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/src/semver.js | 63 ++\n .../tdd/snapshots/green-0/test/semver.test.js | 63 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/package.json | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/compare-semver/issue.md | 25 +\n .../red-0/specs/changes/compare-semver/plan.md | 48 ++\n .../red-0/specs/changes/compare-semver/proposal.md | 38 ++\n .../red-0/specs/changes/compare-semver/status.json | 603 +++++++++++++++++++\n .../red-0/specs/changes/compare-semver/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/src/semver.js | 22 +\n .../tdd/snapshots/red-0/test/semver.test.js | 63 ++\n specs/changes/compare-semver/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n src/semver.js | 49 +-\n test/semver.test.js | 51 ++\n 58 files changed, 4885 insertions(+), 5 deletions(-)\n", "session_ids": ["8dbf6589-8358-45da-afd9-4689d754b166"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T133922-semver-compare-openflow-r1/agent-1.jsonl", "results/raw/20261017T133922-semver-compare-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T133922-semver-compare-openflow-r1/arm-stderr.txt", "results/raw/20261017T133922-semver-compare-openflow-r1/arm-stdout.txt", "results/raw/20261017T133922-semver-compare-openflow-r1/measure-from.txt", "results/raw/20261017T133922-semver-compare-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T133922-semver-compare-openflow-r1/repo", "baseline": "0bfba3ff8260b038cac8980e696ee32487e44103", "measure_from": "708440a9f908311ebe34a61946dd4be188e9e05a"} diff --git a/bench/e2e/results/e2e-2026-10-17-openflow.jsonl b/bench/e2e/results/e2e-2026-10-17-openflow.jsonl new file mode 100644 index 0000000..2e8c8cb --- /dev/null +++ b/bench/e2e/results/e2e-2026-10-17-openflow.jsonl @@ -0,0 +1,7 @@ +{"type": "environment", "date": "2026-10-17T12:56:28.042341+00:00", "cpu": "Intel(R) Xeon(R) Processor", "cpu_count": 8, "os": "Linux-6.12.94+-x86_64-with-glibc2.41", "python": "3.13.5", "node": "v22.23.3", "cursor_agent": "2026.10.16-e3cb207", "git": "git version 2.47.3", "commits": {"superpowers": "bb92a77741419a4ab5f06e711a283343f1ada0c3", "openflow": "58b5a3fa0899f6e639454f0e4a7247dec98f6986", "openswarm": "a48255a9cebafc48b0a7cdd39b6590a7dc193496"}, "openflow_tgz": "/surlymochan-openflow-0.2.1.tgz", "openswarm_dir": "", "superpowers_dir": "", "model": "grok-4.7-high", "cursor_flags": ["-p", "--trust", "--force", "--model", "grok-4.7-high", "--output-format", "stream-json"], "run_timeout_s": 1200, "command": ["/usr/bin/python3", "run.py", "--arms", "openflow", "--tasks", "all", "--repeats", "3", "--concurrency", "3", "--out", "results/e2e-2026-10-17-openflow.jsonl"], "arms": "openflow", "tasks": "all", "repeats": 3, "concurrency": 3, "out": "results/e2e-2026-10-17-openflow.jsonl"} +{"type": "run", "task": "config-deep-merge", "arm": "openflow", "repeat": 1, "start": "2026-10-17T12:56:28.612758+00:00", "end": "2026-10-17T13:09:22.849876+00:00", "wall_s": 774.225, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125628-config-deep-merge-openflow-r1/agent-1.jsonl", "session_id": "14a918aa-14a5-496e-a09f-289447125d27", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 261464, "outputTokens": 34664, "cacheReadTokens": 4465792, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 261464, "outputTokens": 34664, "cacheReadTokens": 4465792, "cacheWriteTokens": 0, "total": 4761920}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["HANDOFF.md", "deepmerge.py", "specs/README.md", "specs/architecture.md", "specs/changes/deep-merge-nested-dicts/issue.md", "specs/changes/deep-merge-nested-dicts/plan.md", "specs/changes/deep-merge-nested-dicts/proposal.md", "specs/changes/deep-merge-nested-dicts/spec_delta_review.json", "specs/changes/deep-merge-nested-dicts/spec_delta_review.md", "specs/changes/deep-merge-nested-dicts/status.json", "specs/changes/deep-merge-nested-dicts/tasks.md", "specs/changes/deep-merge-nested-dicts/tdd/green-0.json", "specs/changes/deep-merge-nested-dicts/tdd/quality-0.json", "specs/changes/deep-merge-nested-dicts/tdd/red-0.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/deepmerge.py", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/README.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/issue.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/plan.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/proposal.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/status.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/changes/deep-merge-nested-dicts/tasks.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/product.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/green-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/deepmerge.py", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/README.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/issue.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/plan.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/proposal.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/status.json", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/changes/deep-merge-nested-dicts/tasks.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/product.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/deep-merge-nested-dicts/tdd/snapshots/red-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested-dicts/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_deepmerge.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " HANDOFF.md | 92 +++\n deepmerge.py | 20 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/deep-merge-nested-dicts/issue.md | 23 +\n specs/changes/deep-merge-nested-dicts/plan.md | 47 ++\n specs/changes/deep-merge-nested-dicts/proposal.md | 40 ++\n .../deep-merge-nested-dicts/spec_delta_review.json | 112 ++++\n .../deep-merge-nested-dicts/spec_delta_review.md | 39 ++\n specs/changes/deep-merge-nested-dicts/status.json | 639 +++++++++++++++++++++\n specs/changes/deep-merge-nested-dicts/tasks.md | 5 +\n .../deep-merge-nested-dicts/tdd/green-0.json | 38 ++\n .../deep-merge-nested-dicts/tdd/quality-0.json | 74 +++\n .../changes/deep-merge-nested-dicts/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/deepmerge.py | 23 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested-dicts/issue.md | 23 +\n .../specs/changes/deep-merge-nested-dicts/plan.md | 47 ++\n .../changes/deep-merge-nested-dicts/proposal.md | 40 ++\n .../changes/deep-merge-nested-dicts/status.json | 612 ++++++++++++++++++++\n .../specs/changes/deep-merge-nested-dicts/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_deepmerge.py | 90 +++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/deepmerge.py | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested-dicts/issue.md | 23 +\n .../specs/changes/deep-merge-nested-dicts/plan.md | 47 ++\n .../changes/deep-merge-nested-dicts/proposal.md | 40 ++\n .../changes/deep-merge-nested-dicts/status.json | 604 +++++++++++++++++++\n .../specs/changes/deep-merge-nested-dicts/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_deepmerge.py | 90 +++\n .../deep-merge-nested-dicts/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_deepmerge.py | 61 ++\n 55 files changed, 4851 insertions(+), 3 deletions(-)\n", "session_ids": ["14a918aa-14a5-496e-a09f-289447125d27"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T125628-config-deep-merge-openflow-r1/agent-1.jsonl", "results/raw/20261017T125628-config-deep-merge-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T125628-config-deep-merge-openflow-r1/arm-stderr.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r1/arm-stdout.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r1/measure-from.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T125628-config-deep-merge-openflow-r1/repo", "baseline": "3b4fd8f0310c9d7bf62461d0a36c7522a7dd57ac", "measure_from": "ce0ed8b156af6543474eae4b1a3ec89284c8603f"} +{"type": "run", "task": "config-deep-merge", "arm": "openflow", "repeat": 3, "start": "2026-10-17T12:56:28.622913+00:00", "end": "2026-10-17T13:11:52.288586+00:00", "wall_s": 923.652, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125628-config-deep-merge-openflow-r3/agent-1.jsonl", "session_id": "08796f0e-7b0a-4d4f-a1a9-484fa5440eff", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 187857, "outputTokens": 39903, "cacheReadTokens": 5113600, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 187857, "outputTokens": 39903, "cacheReadTokens": 5113600, "cacheWriteTokens": 0, "total": 5341360}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.054, "stdout_tail": "", "stderr_tail": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.057, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": [".xflow/GOAL.md", "HANDOFF.md", "deepmerge.py", "specs/README.md", "specs/architecture.md", "specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/spec_delta_review.json", "specs/changes/deep-merge-nested/spec_delta_review.md", "specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/green-0.json", "specs/changes/deep-merge-nested/tdd/quality-0.json", "specs/changes/deep-merge-nested/tdd/red-0.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_deepmerge.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", "__pycache__/", "tests/__pycache__/"], "diff_stat": " .xflow/GOAL.md | 17 +\n HANDOFF.md | 92 +++\n deepmerge.py | 19 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/deep-merge-nested/issue.md | 23 +\n specs/changes/deep-merge-nested/plan.md | 53 ++\n specs/changes/deep-merge-nested/proposal.md | 38 ++\n .../deep-merge-nested/spec_delta_review.json | 109 ++++\n .../changes/deep-merge-nested/spec_delta_review.md | 39 ++\n specs/changes/deep-merge-nested/status.json | 637 +++++++++++++++++++++\n specs/changes/deep-merge-nested/tasks.md | 5 +\n specs/changes/deep-merge-nested/tdd/green-0.json | 38 ++\n specs/changes/deep-merge-nested/tdd/quality-0.json | 74 +++\n specs/changes/deep-merge-nested/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 62 ++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/deepmerge.py | 22 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested/issue.md | 23 +\n .../specs/changes/deep-merge-nested/plan.md | 53 ++\n .../specs/changes/deep-merge-nested/proposal.md | 38 ++\n .../specs/changes/deep-merge-nested/status.json | 610 ++++++++++++++++++++\n .../specs/changes/deep-merge-nested/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_deepmerge.py | 59 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 47 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/deepmerge.py | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/deep-merge-nested/issue.md | 23 +\n .../red-0/specs/changes/deep-merge-nested/plan.md | 53 ++\n .../specs/changes/deep-merge-nested/proposal.md | 38 ++\n .../specs/changes/deep-merge-nested/status.json | 602 +++++++++++++++++++\n .../red-0/specs/changes/deep-merge-nested/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_deepmerge.py | 59 ++\n specs/changes/deep-merge-nested/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_deepmerge.py | 30 +\n 56 files changed, 4776 insertions(+), 3 deletions(-)\n", "session_ids": ["08796f0e-7b0a-4d4f-a1a9-484fa5440eff"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T125628-config-deep-merge-openflow-r3/agent-1.jsonl", "results/raw/20261017T125628-config-deep-merge-openflow-r3/agent-1.jsonl.err", "results/raw/20261017T125628-config-deep-merge-openflow-r3/arm-stderr.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r3/arm-stdout.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r3/measure-from.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r3/tool-version.json"], "workdir": "tmp/20261017T125628-config-deep-merge-openflow-r3/repo", "baseline": "3b4fd8f0310c9d7bf62461d0a36c7522a7dd57ac", "measure_from": "8c713853095d7cc4f6a473495c193fba9d48ca67"} +{"type": "run", "task": "config-deep-merge", "arm": "openflow", "repeat": 2, "start": "2026-10-17T12:56:28.601788+00:00", "end": "2026-10-17T13:12:16.356403+00:00", "wall_s": 947.74, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125628-config-deep-merge-openflow-r2/agent-1.jsonl", "session_id": "31c1ee9e-689f-448a-b04b-9ea6de9e0f05", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 203893, "outputTokens": 40424, "cacheReadTokens": 4927872, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 203893, "outputTokens": 40424, "cacheReadTokens": 4927872, "cacheWriteTokens": 0, "total": 5172189}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.041, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["HANDOFF.md", "deepmerge.py", "specs/README.md", "specs/architecture.md", "specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/spec_delta_review.json", "specs/changes/deep-merge-nested/spec_delta_review.md", "specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/green-0.json", "specs/changes/deep-merge-nested/tdd/quality-0.json", "specs/changes/deep-merge-nested/tdd/red-0.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/green-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/deepmerge.py", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/README.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/issue.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/plan.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/proposal.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/status.json", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/changes/deep-merge-nested/tasks.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/product.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/deep-merge-nested/tdd/snapshots/red-0/tests/test_deepmerge.py", "specs/changes/deep-merge-nested/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_deepmerge.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " HANDOFF.md | 92 +++\n deepmerge.py | 27 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/deep-merge-nested/issue.md | 21 +\n specs/changes/deep-merge-nested/plan.md | 52 ++\n specs/changes/deep-merge-nested/proposal.md | 41 ++\n .../deep-merge-nested/spec_delta_review.json | 117 ++++\n .../changes/deep-merge-nested/spec_delta_review.md | 38 ++\n specs/changes/deep-merge-nested/status.json | 637 +++++++++++++++++++++\n specs/changes/deep-merge-nested/tasks.md | 6 +\n specs/changes/deep-merge-nested/tdd/green-0.json | 38 ++\n specs/changes/deep-merge-nested/tdd/quality-0.json | 74 +++\n specs/changes/deep-merge-nested/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 66 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/deepmerge.py | 30 +\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/deep-merge-nested/issue.md | 21 +\n .../specs/changes/deep-merge-nested/plan.md | 52 ++\n .../specs/changes/deep-merge-nested/proposal.md | 41 ++\n .../specs/changes/deep-merge-nested/status.json | 610 ++++++++++++++++++++\n .../specs/changes/deep-merge-nested/tasks.md | 6 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_deepmerge.py | 54 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 51 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/deepmerge.py | 9 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/deep-merge-nested/issue.md | 21 +\n .../red-0/specs/changes/deep-merge-nested/plan.md | 52 ++\n .../specs/changes/deep-merge-nested/proposal.md | 41 ++\n .../specs/changes/deep-merge-nested/status.json | 602 +++++++++++++++++++\n .../red-0/specs/changes/deep-merge-nested/tasks.md | 6 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_deepmerge.py | 54 ++\n specs/changes/deep-merge-nested/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_deepmerge.py | 25 +\n 55 files changed, 4778 insertions(+), 3 deletions(-)\n", "session_ids": ["31c1ee9e-689f-448a-b04b-9ea6de9e0f05"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T125628-config-deep-merge-openflow-r2/agent-1.jsonl", "results/raw/20261017T125628-config-deep-merge-openflow-r2/agent-1.jsonl.err", "results/raw/20261017T125628-config-deep-merge-openflow-r2/arm-stderr.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r2/arm-stdout.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r2/measure-from.txt", "results/raw/20261017T125628-config-deep-merge-openflow-r2/tool-version.json"], "workdir": "tmp/20261017T125628-config-deep-merge-openflow-r2/repo", "baseline": "3b4fd8f0310c9d7bf62461d0a36c7522a7dd57ac", "measure_from": "7c31720257ea6ebebdd6747fb1e5d7c6d93700a4"} +{"type": "run", "task": "csv-quoting", "arm": "openflow", "repeat": 2, "start": "2026-10-17T13:11:52.300988+00:00", "end": "2026-10-17T13:28:55.197306+00:00", "wall_s": 1022.876, "exit_code": 1, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131152-csv-quoting-openflow-r2/agent-1.jsonl", "session_id": "bf226aa6-ad8f-4a7b-8d16-e4391e775172", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": false, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 1, "passed": false, "timed_out": false, "wall_s": 0.124, "stdout_tail": "", "stderr_tail": "====================================\nFAIL: test_quotes_comma_quote_and_newline (tests.test_csvquote.VisibleCsvTests.test_quotes_comma_quote_and_newline)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/tests/test_csvquote.py\", line 24, in test_quotes_comma_quote_and_newline\n self.assertEqual(text, '\"a,b\",\"say \"\"hi\"\"\",\"line1\\nline2\",\\n')\n ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: 'a,b,say \"hi\",line1\\nline2,\\n' != '\"a,b\",\"say \"\"hi\"\"\",\"line1\\nline2\",\\n'\n- a,b,say \"hi\",line1\n+ \"a,b\",\"say \"\"hi\"\"\",\"line1\n? + + + + ++ +\n- line2,\n+ line2\",\n? +\n\n\n======================================================================\nFAIL: test_special_roundtrip (tests.test_csvquote.VisibleCsvTests.test_special_roundtrip)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/tests/test_csvquote.py\", line 41, in test_special_roundtrip\n self.assertEqual(parse_csv(format_csv(rows)), rows)\n ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: Lists differ: [['na[23 chars]hello', ' world'], ['grace', 'quote \"here\"'], [51 chars]'c']] != [['na[23 chars]hello, world'], ['grace', 'quote \"here\"'], ['l[44 chars]'c']]\n\nFirst differing element 1:\n['ada', 'hello', ' world']\n['ada', 'hello, world']\n\nFirst list contains 1 additional elements.\nFirst extra element 6:\n['a', '', 'c']\n\n [['name', 'note'],\n- ['ada', 'hello', ' world'],\n? - --\n\n+ ['ada', 'hello, world'],\n ['grace', 'quote \"here\"'],\n- ['linus', 'line1'],\n+ ['linus', 'line1\\nline2'],\n? +++++++\n\n- ['line2'],\n ['', ''],\n ['a', '', 'c']]\n\n----------------------------------------------------------------------\nRan 6 tests in 0.002s\n\nFAILED (failures=3)\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 1, "passed": false, "timed_out": false, "wall_s": 0.144, "stdout_tail": "", "stderr_tail": "^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: Lists differ: [['city', 'note'], ['Paris', 'left', ' right']] != [['city', 'note'], ['Paris', 'left, right']]\n\nFirst differing element 1:\n['Paris', 'left', ' right']\n['Paris', 'left, right']\n\n- [['city', 'note'], ['Paris', 'left', ' right']]\n? - --\n\n+ [['city', 'note'], ['Paris', 'left, right']]\n\n======================================================================\nFAIL: test_newline_inside_a_field_stays_one_row (_hidden_acceptance.test_csvquote.HiddenCsvTests.test_newline_inside_a_field_stays_one_row)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/_hidden_acceptance/test_csvquote.py\", line 24, in test_newline_inside_a_field_stays_one_row\n self.assertEqual(parsed, rows)\n ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^\nAssertionError: Lists differ: [['line1'], ['line2', 'tail']] != [['line1\\nline2', 'tail']]\n\nFirst differing element 0:\n['line1']\n['line1\\nline2', 'tail']\n\nFirst list contains 1 additional elements.\nFirst extra element 1:\n['line2', 'tail']\n\n- [['line1'], ['line2', 'tail']]\n? ^^^^^^\n\n+ [['line1\\nline2', 'tail']]\n? ^^\n\n\n======================================================================\nFAIL: test_quotes_inside_a_field_are_doubled (_hidden_acceptance.test_csvquote.HiddenCsvTests.test_quotes_inside_a_field_are_doubled)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"tmp/20261017T131152-csv-quoting-openflow-r2/repo/_hidden_acceptance/test_csvquote.py\", line 18, in test_quotes_inside_a_field_are_doubled\n self.assertIn('\"\"', text)\n ~~~~~~~~~~~~~^^^^^^^^^^^^\nAssertionError: '\"\"' not found in 'title\\nsay \"hello\"\\n'\n\n----------------------------------------------------------------------\nRan 5 tests in 0.007s\n\nFAILED (failures=3)\n"}, "files_changed": [".as-xflow/config.json", "tests/test_csvquote.py", ".xflow/", "__pycache__/", "specs/", "tests/__pycache__/"], "diff_stat": " .as-xflow/config.json | 2 +-\n tests/test_csvquote.py | 22 ++++++++++++++++++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n", "session_ids": ["bf226aa6-ad8f-4a7b-8d16-e4391e775172"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131152-csv-quoting-openflow-r2/agent-1.jsonl", "results/raw/20261017T131152-csv-quoting-openflow-r2/agent-1.jsonl.err", "results/raw/20261017T131152-csv-quoting-openflow-r2/arm-stderr.txt", "results/raw/20261017T131152-csv-quoting-openflow-r2/arm-stdout.txt", "results/raw/20261017T131152-csv-quoting-openflow-r2/measure-from.txt", "results/raw/20261017T131152-csv-quoting-openflow-r2/tool-version.json"], "workdir": "tmp/20261017T131152-csv-quoting-openflow-r2/repo", "baseline": "c3900ae3a2805ad798135e5c0b4df9c647f289c0", "measure_from": "0be61207d77df85f811bcfb55a0a2fa80d896001"} +{"type": "run", "task": "csv-quoting", "arm": "openflow", "repeat": 1, "start": "2026-10-17T13:09:22.861800+00:00", "end": "2026-10-17T13:29:23.864494+00:00", "wall_s": 1200.983, "exit_code": 124, "timed_out": true, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130922-csv-quoting-openflow-r1/agent-1.jsonl", "session_id": "0f1fb9a7-884d-4309-9d20-4ce61a90f754", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.236, "stdout_tail": "", "stderr_tail": "....\n----------------------------------------------------------------------\nRan 4 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.281, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["HANDOFF.md", "csvquote.py", "specs/README.md", "specs/architecture.md", "specs/changes/csv-field-quoting/issue.md", "specs/changes/csv-field-quoting/plan.md", "specs/changes/csv-field-quoting/proposal.md", "specs/changes/csv-field-quoting/spec_delta_review.json", "specs/changes/csv-field-quoting/spec_delta_review.md", "specs/changes/csv-field-quoting/status.json", "specs/changes/csv-field-quoting/tasks.md", "specs/changes/csv-field-quoting/tdd/green-0.json", "specs/changes/csv-field-quoting/tdd/quality-0.json", "specs/changes/csv-field-quoting/tdd/red-0.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/bindings.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/config.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/logs/execution.ndjson", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/pending-gates/execute.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.as-xflow/workflow-state.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.cursor/rules/openflow.mdc", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/.xflow/GOAL.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/csvquote.py", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/README.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/architecture.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/issue.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/plan.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/proposal.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/status.json", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/changes/csv-field-quoting/tasks.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/product.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/specs/workflow.md", "specs/changes/csv-field-quoting/tdd/snapshots/green-0/tests/test_csvquote.py", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/bindings.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/config.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/logs/execution.ndjson", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/pending-gates/brainstorm.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.as-xflow/workflow-state.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.cursor/rules/openflow.mdc", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/.xflow/GOAL.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/csvquote.py", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/README.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/architecture.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/issue.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/plan.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/proposal.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/status.json", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/changes/csv-field-quoting/tasks.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/product.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/specs/workflow.md", "specs/changes/csv-field-quoting/tdd/snapshots/red-0/tests/test_csvquote.py", "specs/changes/csv-field-quoting/verify_proof.json", "specs/product.md", "specs/workflow.md", "tests/test_csvquote.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "tests/__pycache__/"], "diff_stat": " HANDOFF.md | 92 +++\n csvquote.py | 60 +-\n specs/README.md | 116 ++++\n specs/architecture.md | 17 +\n specs/changes/csv-field-quoting/issue.md | 21 +\n specs/changes/csv-field-quoting/plan.md | 47 ++\n specs/changes/csv-field-quoting/proposal.md | 39 ++\n .../csv-field-quoting/spec_delta_review.json | 109 ++++\n .../changes/csv-field-quoting/spec_delta_review.md | 38 ++\n specs/changes/csv-field-quoting/status.json | 637 +++++++++++++++++++++\n specs/changes/csv-field-quoting/tasks.md | 5 +\n specs/changes/csv-field-quoting/tdd/green-0.json | 38 ++\n specs/changes/csv-field-quoting/tdd/quality-0.json | 74 +++\n specs/changes/csv-field-quoting/tdd/red-0.json | 37 ++\n .../tdd/snapshots/green-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/green-0/.as-xflow/config.json | 20 +\n .../green-0/.as-xflow/logs/execution.ndjson | 66 +++\n .../.as-xflow/pending-gates/brainstorm.json | 8 +\n .../green-0/.as-xflow/pending-gates/execute.json | 8 +\n .../green-0/.as-xflow/workflow-state.json | 554 ++++++++++++++++++\n .../snapshots/green-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/green-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/green-0/csvquote.py | 78 +++\n .../tdd/snapshots/green-0/specs/README.md | 116 ++++\n .../tdd/snapshots/green-0/specs/architecture.md | 17 +\n .../specs/changes/csv-field-quoting/issue.md | 21 +\n .../specs/changes/csv-field-quoting/plan.md | 47 ++\n .../specs/changes/csv-field-quoting/proposal.md | 39 ++\n .../specs/changes/csv-field-quoting/status.json | 611 ++++++++++++++++++++\n .../specs/changes/csv-field-quoting/tasks.md | 5 +\n .../tdd/snapshots/green-0/specs/product.md | 18 +\n .../tdd/snapshots/green-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/green-0/tests/test_csvquote.py | 37 ++\n .../tdd/snapshots/red-0/.as-xflow/bindings.json | 8 +\n .../tdd/snapshots/red-0/.as-xflow/config.json | 20 +\n .../red-0/.as-xflow/logs/execution.ndjson | 51 ++\n .../red-0/.as-xflow/pending-gates/brainstorm.json | 8 +\n .../snapshots/red-0/.as-xflow/workflow-state.json | 552 ++++++++++++++++++\n .../tdd/snapshots/red-0/.cursor/rules/openflow.mdc | 14 +\n .../tdd/snapshots/red-0/.xflow/GOAL.md | 17 +\n .../tdd/snapshots/red-0/csvquote.py | 24 +\n .../tdd/snapshots/red-0/specs/README.md | 116 ++++\n .../tdd/snapshots/red-0/specs/architecture.md | 17 +\n .../red-0/specs/changes/csv-field-quoting/issue.md | 21 +\n .../red-0/specs/changes/csv-field-quoting/plan.md | 47 ++\n .../specs/changes/csv-field-quoting/proposal.md | 39 ++\n .../specs/changes/csv-field-quoting/status.json | 603 +++++++++++++++++++\n .../red-0/specs/changes/csv-field-quoting/tasks.md | 5 +\n .../tdd/snapshots/red-0/specs/product.md | 18 +\n .../tdd/snapshots/red-0/specs/workflow.md | 49 ++\n .../tdd/snapshots/red-0/tests/test_csvquote.py | 37 ++\n specs/changes/csv-field-quoting/verify_proof.json | 12 +\n specs/product.md | 18 +\n specs/workflow.md | 49 ++\n tests/test_csvquote.py | 14 +\n 55 files changed, 4799 insertions(+), 3 deletions(-)\n", "session_ids": ["0f1fb9a7-884d-4309-9d20-4ce61a90f754"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T130922-csv-quoting-openflow-r1/agent-1.jsonl", "results/raw/20261017T130922-csv-quoting-openflow-r1/agent-1.jsonl.err", "results/raw/20261017T130922-csv-quoting-openflow-r1/arm-stderr.txt", "results/raw/20261017T130922-csv-quoting-openflow-r1/arm-stdout.txt", "results/raw/20261017T130922-csv-quoting-openflow-r1/measure-from.txt", "results/raw/20261017T130922-csv-quoting-openflow-r1/tool-version.json"], "workdir": "tmp/20261017T130922-csv-quoting-openflow-r1/repo", "baseline": "8af7bd5446baa8e1345a2b3ef30eea7ab7452c1a", "measure_from": "3d9e9d3373e71aa8e92a97cce3f8522f35f2d64d"} +{"type": "run", "task": "csv-quoting", "arm": "openflow", "repeat": 3, "start": "2026-10-17T13:12:16.368402+00:00", "end": "2026-10-17T13:32:16.782602+00:00", "wall_s": 1200.393, "exit_code": 124, "timed_out": true, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131216-csv-quoting-openflow-r3/agent-1.jsonl", "session_id": "2f7effe3-59ea-4991-a525-c4e69e44a3bb", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "missing_usage": true, "saw_result": false}], "inputTokens": 0, "outputTokens": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0, "total": 0}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.122, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", ".as-xflow/bindings.json", ".as-xflow/logs/", ".as-xflow/pending-gates/", ".as-xflow/workflow-state.json", ".xflow/", "__pycache__/", "specs/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 19 ++++++++-----------\n tests/test_csvquote.py | 22 ++++++++++++++++++++++\n 2 files changed, 30 insertions(+), 11 deletions(-)\n", "session_ids": ["2f7effe3-59ea-4991-a525-c4e69e44a3bb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openflow", "version": "0.2.1", "commit": null, "tarball": "/surlymochan-openflow-0.2.1.tgz"}}, "raw_events": ["results/raw/20261017T131216-csv-quoting-openflow-r3/agent-1.jsonl", "results/raw/20261017T131216-csv-quoting-openflow-r3/agent-1.jsonl.err", "results/raw/20261017T131216-csv-quoting-openflow-r3/arm-stderr.txt", "results/raw/20261017T131216-csv-quoting-openflow-r3/arm-stdout.txt", "results/raw/20261017T131216-csv-quoting-openflow-r3/measure-from.txt", "results/raw/20261017T131216-csv-quoting-openflow-r3/tool-version.json"], "workdir": "tmp/20261017T131216-csv-quoting-openflow-r3/repo", "baseline": "4153cf66cb254bc0a8af4f594543132f863380f0", "measure_from": "ba32010332b637dcfc7eaa2dd497a0efdcbd7b87"} diff --git a/bench/e2e/results/e2e-2026-10-17.jsonl b/bench/e2e/results/e2e-2026-10-17.jsonl new file mode 100644 index 0000000..ea3030e --- /dev/null +++ b/bench/e2e/results/e2e-2026-10-17.jsonl @@ -0,0 +1,73 @@ +{"type": "environment", "date": "2026-10-17T12:20:16.509195+00:00", "cpu": "Intel(R) Xeon(R) Processor", "cpu_count": 8, "os": "Linux-6.12.94+-x86_64-with-glibc2.41", "python": "3.13.5", "node": "v22.23.3", "cursor_agent": "2026.10.16-e3cb207", "git": "git version 2.47.3", "commits": {"superpowers": "bb92a77741419a4ab5f06e711a283343f1ada0c3", "openflow": "58b5a3fa0899f6e639454f0e4a7247dec98f6986", "openswarm": "a48255a9cebafc48b0a7cdd39b6590a7dc193496"}, "openflow_tgz": null, "openswarm_dir": "", "superpowers_dir": "", "model": "grok-4.7-high", "cursor_flags": ["-p", "--trust", "--force", "--model", "grok-4.7-high", "--output-format", "stream-json"], "run_timeout_s": 1200, "command": ["/usr/bin/python3", "run.py", "--arms", "baseline,superpowers,openswarm", "--tasks", "all", "--repeats", "3", "--concurrency", "3", "--out", "results/e2e-2026-10-17.jsonl"], "arms": "baseline,superpowers,openswarm", "tasks": "all", "repeats": 3, "concurrency": 3, "out": "results/e2e-2026-10-17.jsonl"} +{"type": "run", "task": "config-deep-merge", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:20:17.118090+00:00", "end": "2026-10-17T12:21:30.835055+00:00", "wall_s": 73.715, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122017-config-deep-merge-baseline-r3/agent-1.jsonl", "session_id": "30077454-35e1-4561-9371-b47bfab96b40", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 28749, "outputTokens": 3657, "cacheReadTokens": 59776, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 28749, "outputTokens": 3657, "cacheReadTokens": 59776, "cacheWriteTokens": 0, "total": 92182}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n", "session_ids": ["30077454-35e1-4561-9371-b47bfab96b40"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122017-config-deep-merge-baseline-r3/agent-1.jsonl", "results/raw/20261017T122017-config-deep-merge-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T122017-config-deep-merge-baseline-r3/arm-stderr.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r3/arm-stdout.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T122017-config-deep-merge-baseline-r3/repo", "baseline": "1351d9ee49e94e6fab4488327516f1e7fc41eccc", "measure_from": "1351d9ee49e94e6fab4488327516f1e7fc41eccc"} +{"type": "run", "task": "config-deep-merge", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:20:17.119644+00:00", "end": "2026-10-17T12:21:52.629603+00:00", "wall_s": 95.508, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122017-config-deep-merge-baseline-r2/agent-1.jsonl", "session_id": "7f1bffe6-1484-4851-9c14-d54efeda11a3", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 30533, "outputTokens": 4906, "cacheReadTokens": 99200, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 30533, "outputTokens": 4906, "cacheReadTokens": 99200, "cacheWriteTokens": 0, "total": 134639}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 13 +++++++++++--\n 1 file changed, 11 insertions(+), 2 deletions(-)\n", "session_ids": ["7f1bffe6-1484-4851-9c14-d54efeda11a3"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122017-config-deep-merge-baseline-r2/agent-1.jsonl", "results/raw/20261017T122017-config-deep-merge-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T122017-config-deep-merge-baseline-r2/arm-stderr.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r2/arm-stdout.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T122017-config-deep-merge-baseline-r2/repo", "baseline": "1351d9ee49e94e6fab4488327516f1e7fc41eccc", "measure_from": "1351d9ee49e94e6fab4488327516f1e7fc41eccc"} +{"type": "run", "task": "config-deep-merge", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:20:17.119030+00:00", "end": "2026-10-17T12:22:29.658334+00:00", "wall_s": 132.537, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122017-config-deep-merge-baseline-r1/agent-1.jsonl", "session_id": "5a6b5784-ee0f-4c24-8819-c596e1e17a17", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 43015, "outputTokens": 6652, "cacheReadTokens": 94720, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 43015, "outputTokens": 6652, "cacheReadTokens": 94720, "cacheWriteTokens": 0, "total": 144387}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 15 +++++++++++++--\n 1 file changed, 13 insertions(+), 2 deletions(-)\n", "session_ids": ["5a6b5784-ee0f-4c24-8819-c596e1e17a17"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122017-config-deep-merge-baseline-r1/agent-1.jsonl", "results/raw/20261017T122017-config-deep-merge-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T122017-config-deep-merge-baseline-r1/arm-stderr.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r1/arm-stdout.txt", "results/raw/20261017T122017-config-deep-merge-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T122017-config-deep-merge-baseline-r1/repo", "baseline": "1351d9ee49e94e6fab4488327516f1e7fc41eccc", "measure_from": "1351d9ee49e94e6fab4488327516f1e7fc41eccc"} +{"type": "run", "task": "config-deep-merge", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:21:30.845599+00:00", "end": "2026-10-17T12:24:31.547864+00:00", "wall_s": 180.699, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122130-config-deep-merge-superpowers-r1/agent-1.jsonl", "session_id": "8d0cb46e-5b39-40c5-9de1-a6bdda313803", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 40345, "outputTokens": 8635, "cacheReadTokens": 246144, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 40345, "outputTokens": 8635, "cacheReadTokens": 246144, "cacheWriteTokens": 0, "total": 295124}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "tests/test_deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 7 ++++++-\n tests/test_deepmerge.py | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 41 insertions(+), 1 deletion(-)\n", "session_ids": ["8d0cb46e-5b39-40c5-9de1-a6bdda313803"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T122130-config-deep-merge-superpowers-r1/agent-1.jsonl", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/arm-stderr.txt", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/arm-stdout.txt", "results/raw/20261017T122130-config-deep-merge-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T122130-config-deep-merge-superpowers-r1/repo", "baseline": "7a9f096c16fe238294eab9b09b2b640962670d34", "measure_from": "7a9f096c16fe238294eab9b09b2b640962670d34"} +{"type": "run", "task": "config-deep-merge", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:22:29.670257+00:00", "end": "2026-10-17T12:25:07.798195+00:00", "wall_s": 158.125, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122229-config-deep-merge-superpowers-r3/agent-1.jsonl", "session_id": "ac5d2f0d-59b4-4d1d-8666-f12e5e73cc5d", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 54614, "outputTokens": 7345, "cacheReadTokens": 253824, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 54614, "outputTokens": 7345, "cacheReadTokens": 253824, "cacheWriteTokens": 0, "total": 315783}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "tests/test_deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 6 +++++-\n tests/test_deepmerge.py | 28 ++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 1 deletion(-)\n", "session_ids": ["ac5d2f0d-59b4-4d1d-8666-f12e5e73cc5d"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T122229-config-deep-merge-superpowers-r3/agent-1.jsonl", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/arm-stderr.txt", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/arm-stdout.txt", "results/raw/20261017T122229-config-deep-merge-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T122229-config-deep-merge-superpowers-r3/repo", "baseline": "0580bfba58bd6f91390cd1454820a768380cddf6", "measure_from": "0580bfba58bd6f91390cd1454820a768380cddf6"} +{"type": "run", "task": "config-deep-merge", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:21:52.642166+00:00", "end": "2026-10-17T12:26:15.941456+00:00", "wall_s": 263.295, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122152-config-deep-merge-superpowers-r2/agent-1.jsonl", "session_id": "0610b915-3824-4062-8ec0-3fe9c8fd926f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 53400, "outputTokens": 13213, "cacheReadTokens": 392704, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 53400, "outputTokens": 13213, "cacheReadTokens": 392704, "cacheWriteTokens": 0, "total": 459317}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": "..........\n----------------------------------------------------------------------\nRan 10 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py", "tests/test_deepmerge.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " deepmerge.py | 7 +++++-\n tests/test_deepmerge.py | 60 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 66 insertions(+), 1 deletion(-)\n", "session_ids": ["0610b915-3824-4062-8ec0-3fe9c8fd926f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T122152-config-deep-merge-superpowers-r2/agent-1.jsonl", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/arm-stderr.txt", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/arm-stdout.txt", "results/raw/20261017T122152-config-deep-merge-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T122152-config-deep-merge-superpowers-r2/repo", "baseline": "dadb79d43d16709347bb3d7863de8482ab8ee7f7", "measure_from": "dadb79d43d16709347bb3d7863de8482ab8ee7f7"} +{"type": "run", "task": "config-deep-merge", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:25:07.810785+00:00", "end": "2026-10-17T12:28:03.198487+00:00", "wall_s": 175.385, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122507-config-deep-merge-openswarm-r2/agent-1.jsonl", "session_id": "4993a35b-02fc-4998-bf3f-68d416275590", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 35143, "outputTokens": 6318, "cacheReadTokens": 74496, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "38d99df6-f493-4909-a9f1-fab554003d35", "model": "grok-4.7-high", "usage": {"inputTokens": 13864, "outputTokens": 2342, "cacheReadTokens": 59520, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49007, "outputTokens": 8660, "cacheReadTokens": 134016, "cacheWriteTokens": 0, "total": 191683}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py"], "diff_stat": " deepmerge.py | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n", "session_ids": ["4993a35b-02fc-4998-bf3f-68d416275590", "38d99df6-f493-4909-a9f1-fab554003d35"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T122507-config-deep-merge-openswarm-r2/agent-1.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/arm-stderr.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/arm-stdout.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/executor-task.md", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/executor-usage.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/merge.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/diff.patch", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/events.1.cursor.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/events.1.cursor.jsonl.err", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/result.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/status.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/runs/20261017-202715-925405/task.md", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-result.json", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/openswarm.err", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/planner-prompt.txt", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T122507-config-deep-merge-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T122507-config-deep-merge-openswarm-r2/repo", "baseline": "581caf268c0d15c2ecc801c3868cf52137b72a4d", "measure_from": "581caf268c0d15c2ecc801c3868cf52137b72a4d"} +{"type": "run", "task": "config-deep-merge", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:24:31.583556+00:00", "end": "2026-10-17T12:28:12.072177+00:00", "wall_s": 220.486, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122431-config-deep-merge-openswarm-r1/agent-1.jsonl", "session_id": "935bd656-79fc-4ff7-ae91-78daac56ad21", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 21456, "outputTokens": 6923, "cacheReadTokens": 87808, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "d9adc1c8-63fa-4567-bc29-b9d0e5e03539", "model": "grok-4.7-high", "usage": {"inputTokens": 28783, "outputTokens": 4468, "cacheReadTokens": 49152, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 50239, "outputTokens": 11391, "cacheReadTokens": 136960, "cacheWriteTokens": 0, "total": 198590}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.054, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py"], "diff_stat": " deepmerge.py | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n", "session_ids": ["935bd656-79fc-4ff7-ae91-78daac56ad21", "d9adc1c8-63fa-4567-bc29-b9d0e5e03539"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T122431-config-deep-merge-openswarm-r1/agent-1.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/arm-stderr.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/arm-stdout.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/executor-task.md", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/executor-usage.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/merge.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/diff.patch", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/events.1.cursor.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/events.1.cursor.jsonl.err", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/result.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/status.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/runs/20261017-202646-fa07ce/task.md", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-result.json", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/openswarm.err", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/planner-prompt.txt", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T122431-config-deep-merge-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T122431-config-deep-merge-openswarm-r1/repo", "baseline": "0891cb471178999f5da6606db428818c70988fb2", "measure_from": "0891cb471178999f5da6606db428818c70988fb2"} +{"type": "run", "task": "csv-quoting", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:28:12.084097+00:00", "end": "2026-10-17T12:30:44.413162+00:00", "wall_s": 152.326, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122812-csv-quoting-baseline-r2/agent-1.jsonl", "session_id": "f31b4771-fa80-462a-a930-1d7d1f3134a0", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18626, "outputTokens": 8085, "cacheReadTokens": 77952, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 18626, "outputTokens": 8085, "cacheReadTokens": 77952, "cacheWriteTokens": 0, "total": 104663}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 58 insertions(+), 3 deletions(-)\n", "session_ids": ["f31b4771-fa80-462a-a930-1d7d1f3134a0"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122812-csv-quoting-baseline-r2/agent-1.jsonl", "results/raw/20261017T122812-csv-quoting-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T122812-csv-quoting-baseline-r2/arm-stderr.txt", "results/raw/20261017T122812-csv-quoting-baseline-r2/arm-stdout.txt", "results/raw/20261017T122812-csv-quoting-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T122812-csv-quoting-baseline-r2/repo", "baseline": "fc9e75629c9086948a07c6822a2b258269cbcd62", "measure_from": "fc9e75629c9086948a07c6822a2b258269cbcd62"} +{"type": "run", "task": "csv-quoting", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:28:03.211554+00:00", "end": "2026-10-17T12:31:29.098546+00:00", "wall_s": 205.884, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122803-csv-quoting-baseline-r1/agent-1.jsonl", "session_id": "10565516-ac5d-469e-a66a-4e5d21d2636e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 42182, "outputTokens": 10520, "cacheReadTokens": 81408, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 42182, "outputTokens": 10520, "cacheReadTokens": 81408, "cacheWriteTokens": 0, "total": 134110}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.057, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 53 insertions(+), 3 deletions(-)\n", "session_ids": ["10565516-ac5d-469e-a66a-4e5d21d2636e"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T122803-csv-quoting-baseline-r1/agent-1.jsonl", "results/raw/20261017T122803-csv-quoting-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T122803-csv-quoting-baseline-r1/arm-stderr.txt", "results/raw/20261017T122803-csv-quoting-baseline-r1/arm-stdout.txt", "results/raw/20261017T122803-csv-quoting-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T122803-csv-quoting-baseline-r1/repo", "baseline": "5f5dc83565fc500c3a3dbcc38ae0dcdf09178186", "measure_from": "5f5dc83565fc500c3a3dbcc38ae0dcdf09178186"} +{"type": "run", "task": "config-deep-merge", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:26:15.953582+00:00", "end": "2026-10-17T12:31:42.096576+00:00", "wall_s": 326.14, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T122615-config-deep-merge-openswarm-r3/agent-1.jsonl", "session_id": "5aee5194-91b0-499c-b4bf-db69a796d6cd", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18364, "outputTokens": 3933, "cacheReadTokens": 88960, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "912cabea-0459-475d-8c3a-ddef2a917b10", "model": "grok-4.7-high", "usage": {"inputTokens": 37148, "outputTokens": 13183, "cacheReadTokens": 83072, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 55512, "outputTokens": 17116, "cacheReadTokens": 172032, "cacheWriteTokens": 0, "total": 244660}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.054, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["deepmerge.py"], "diff_stat": " deepmerge.py | 21 +++++++++++++++++++--\n 1 file changed, 19 insertions(+), 2 deletions(-)\n", "session_ids": ["5aee5194-91b0-499c-b4bf-db69a796d6cd", "912cabea-0459-475d-8c3a-ddef2a917b10"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T122615-config-deep-merge-openswarm-r3/agent-1.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/arm-stderr.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/arm-stdout.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/executor-task.md", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/executor-usage.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/merge.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/diff.patch", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/events.1.cursor.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/events.1.cursor.jsonl.err", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/result.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/status.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/runs/20261017-202739-3da53e/task.md", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-result.json", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/openswarm.err", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/planner-prompt.txt", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T122615-config-deep-merge-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T122615-config-deep-merge-openswarm-r3/repo", "baseline": "9393a2685ea5f8c4abd28272a5a8189b7d388d8b", "measure_from": "9393a2685ea5f8c4abd28272a5a8189b7d388d8b"} +{"type": "run", "task": "csv-quoting", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:30:44.425254+00:00", "end": "2026-10-17T12:34:12.097963+00:00", "wall_s": 207.67, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123044-csv-quoting-baseline-r3/agent-1.jsonl", "session_id": "823828be-4c84-43de-954b-0e46e245bf47", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 39165, "outputTokens": 10377, "cacheReadTokens": 81280, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 39165, "outputTokens": 10377, "cacheReadTokens": 81280, "cacheWriteTokens": 0, "total": 130822}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 51 insertions(+), 3 deletions(-)\n", "session_ids": ["823828be-4c84-43de-954b-0e46e245bf47"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T123044-csv-quoting-baseline-r3/agent-1.jsonl", "results/raw/20261017T123044-csv-quoting-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T123044-csv-quoting-baseline-r3/arm-stderr.txt", "results/raw/20261017T123044-csv-quoting-baseline-r3/arm-stdout.txt", "results/raw/20261017T123044-csv-quoting-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T123044-csv-quoting-baseline-r3/repo", "baseline": "06ad722c01936debd4d75c072bb5b3dacdf5bdc0", "measure_from": "06ad722c01936debd4d75c072bb5b3dacdf5bdc0"} +{"type": "run", "task": "csv-quoting", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:31:29.110355+00:00", "end": "2026-10-17T12:36:43.949075+00:00", "wall_s": 314.835, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123129-csv-quoting-superpowers-r1/agent-1.jsonl", "session_id": "19de07a7-7696-48cf-b8a5-4325e0850486", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49696, "outputTokens": 16004, "cacheReadTokens": 276992, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49696, "outputTokens": 16004, "cacheReadTokens": 276992, "cacheWriteTokens": 0, "total": 342692}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": ".........\n----------------------------------------------------------------------\nRan 9 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 68 +++++++++++++++++++++++++++++++++++++++++++++++---\n tests/test_csvquote.py | 38 ++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+), 3 deletions(-)\n", "session_ids": ["19de07a7-7696-48cf-b8a5-4325e0850486"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T123129-csv-quoting-superpowers-r1/agent-1.jsonl", "results/raw/20261017T123129-csv-quoting-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T123129-csv-quoting-superpowers-r1/arm-stderr.txt", "results/raw/20261017T123129-csv-quoting-superpowers-r1/arm-stdout.txt", "results/raw/20261017T123129-csv-quoting-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T123129-csv-quoting-superpowers-r1/repo", "baseline": "2133255b869d664385bd137e49d2f9f0652a6c5a", "measure_from": "2133255b869d664385bd137e49d2f9f0652a6c5a"} +{"type": "run", "task": "csv-quoting", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:31:42.107636+00:00", "end": "2026-10-17T12:37:53.904423+00:00", "wall_s": 371.792, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123142-csv-quoting-superpowers-r2/agent-1.jsonl", "session_id": "35ba8fb0-c616-4791-9adb-46bc508699d9", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 82773, "outputTokens": 18511, "cacheReadTokens": 451584, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 82773, "outputTokens": 18511, "cacheReadTokens": 451584, "cacheWriteTokens": 0, "total": 552868}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".........\n----------------------------------------------------------------------\nRan 9 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.075, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++---\n tests/test_csvquote.py | 28 +++++++++++++++++++++++++\n 2 files changed, 81 insertions(+), 3 deletions(-)\n", "session_ids": ["35ba8fb0-c616-4791-9adb-46bc508699d9"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T123142-csv-quoting-superpowers-r2/agent-1.jsonl", "results/raw/20261017T123142-csv-quoting-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T123142-csv-quoting-superpowers-r2/arm-stderr.txt", "results/raw/20261017T123142-csv-quoting-superpowers-r2/arm-stdout.txt", "results/raw/20261017T123142-csv-quoting-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T123142-csv-quoting-superpowers-r2/repo", "baseline": "375ae49ec72800815a24b53375c2b2ac07595182", "measure_from": "375ae49ec72800815a24b53375c2b2ac07595182"} +{"type": "run", "task": "csv-quoting", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:34:12.110912+00:00", "end": "2026-10-17T12:38:09.857744+00:00", "wall_s": 237.744, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123412-csv-quoting-superpowers-r3/agent-1.jsonl", "session_id": "657434a5-560a-45b0-8330-23c9be75b9e6", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 57397, "outputTokens": 11047, "cacheReadTokens": 239616, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 57397, "outputTokens": 11047, "cacheReadTokens": 239616, "cacheWriteTokens": 0, "total": 308060}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py", "tests/test_csvquote.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " csvquote.py | 44 +++++++++++++++++++++++++++++++++++++++++---\n tests/test_csvquote.py | 22 ++++++++++++++++++++++\n 2 files changed, 63 insertions(+), 3 deletions(-)\n", "session_ids": ["657434a5-560a-45b0-8330-23c9be75b9e6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T123412-csv-quoting-superpowers-r3/agent-1.jsonl", "results/raw/20261017T123412-csv-quoting-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T123412-csv-quoting-superpowers-r3/arm-stderr.txt", "results/raw/20261017T123412-csv-quoting-superpowers-r3/arm-stdout.txt", "results/raw/20261017T123412-csv-quoting-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T123412-csv-quoting-superpowers-r3/repo", "baseline": "b642a415eb4ac4f63053940fc8f375f4c907ecec", "measure_from": "b642a415eb4ac4f63053940fc8f375f4c907ecec"} +{"type": "run", "task": "csv-quoting", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:38:09.870229+00:00", "end": "2026-10-17T12:44:32.592844+00:00", "wall_s": 382.719, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123809-csv-quoting-openswarm-r3/agent-1.jsonl", "session_id": "1333c1f2-6f79-419a-bb54-022421f2a8f7", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 26453, "outputTokens": 10978, "cacheReadTokens": 170624, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "63fd0477-4348-476d-be40-807766f4e6eb", "model": "grok-4.7-high", "usage": {"inputTokens": 87959, "outputTokens": 8016, "cacheReadTokens": 47232, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 114412, "outputTokens": 18994, "cacheReadTokens": 217856, "cacheWriteTokens": 0, "total": 351262}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py"], "diff_stat": " csvquote.py | 50 +++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 47 insertions(+), 3 deletions(-)\n", "session_ids": ["1333c1f2-6f79-419a-bb54-022421f2a8f7", "63fd0477-4348-476d-be40-807766f4e6eb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T123809-csv-quoting-openswarm-r3/agent-1.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T123809-csv-quoting-openswarm-r3/arm-stderr.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/arm-stdout.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/executor-task.md", "results/raw/20261017T123809-csv-quoting-openswarm-r3/executor-usage.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/merge.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/diff.patch", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/events.1.cursor.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/events.1.cursor.jsonl.err", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/result.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/status.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/runs/20261017-204158-90e445/task.md", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-result.json", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/openswarm.err", "results/raw/20261017T123809-csv-quoting-openswarm-r3/planner-prompt.txt", "results/raw/20261017T123809-csv-quoting-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T123809-csv-quoting-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T123809-csv-quoting-openswarm-r3/repo", "baseline": "e7b313d87b3ad68d2bcf5e64ed77074edfb19de3", "measure_from": "e7b313d87b3ad68d2bcf5e64ed77074edfb19de3"} +{"type": "run", "task": "date-range-parse", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:44:32.605896+00:00", "end": "2026-10-17T12:45:03.578561+00:00", "wall_s": 30.971, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124432-date-range-parse-baseline-r1/agent-1.jsonl", "session_id": "ecb8625f-1317-45b2-9a28-5edabe6a2956", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 12152, "outputTokens": 1315, "cacheReadTokens": 72064, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 12152, "outputTokens": 1315, "cacheReadTokens": 72064, "cacheWriteTokens": 0, "total": 85531}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["ecb8625f-1317-45b2-9a28-5edabe6a2956"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T124432-date-range-parse-baseline-r1/agent-1.jsonl", "results/raw/20261017T124432-date-range-parse-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T124432-date-range-parse-baseline-r1/arm-stderr.txt", "results/raw/20261017T124432-date-range-parse-baseline-r1/arm-stdout.txt", "results/raw/20261017T124432-date-range-parse-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T124432-date-range-parse-baseline-r1/repo", "baseline": "f48844c4abc08cb6d42e105ee6d74e4f07029e5d", "measure_from": "f48844c4abc08cb6d42e105ee6d74e4f07029e5d"} +{"type": "run", "task": "csv-quoting", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:37:53.917393+00:00", "end": "2026-10-17T12:45:08.845140+00:00", "wall_s": 434.924, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123753-csv-quoting-openswarm-r2/agent-1.jsonl", "session_id": "e4c76ef7-3673-4d5a-8b2e-c4a9ae08d2b4", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 84076, "outputTokens": 13359, "cacheReadTokens": 181248, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "828cd414-7fbc-4166-80a6-8f0dd4c33537", "model": "grok-4.7-high", "usage": {"inputTokens": 32851, "outputTokens": 8557, "cacheReadTokens": 55808, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 116927, "outputTokens": 21916, "cacheReadTokens": 237056, "cacheWriteTokens": 0, "total": 375899}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.061, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py"], "diff_stat": " csvquote.py | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 85 insertions(+), 7 deletions(-)\n", "session_ids": ["e4c76ef7-3673-4d5a-8b2e-c4a9ae08d2b4", "828cd414-7fbc-4166-80a6-8f0dd4c33537"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T123753-csv-quoting-openswarm-r2/agent-1.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T123753-csv-quoting-openswarm-r2/arm-stderr.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/arm-stdout.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/executor-task.md", "results/raw/20261017T123753-csv-quoting-openswarm-r2/executor-usage.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/merge.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/diff.patch", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/events.1.cursor.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/events.1.cursor.jsonl.err", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/result.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/status.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/runs/20261017-204229-1e7db3/task.md", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-result.json", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/openswarm.err", "results/raw/20261017T123753-csv-quoting-openswarm-r2/planner-prompt.txt", "results/raw/20261017T123753-csv-quoting-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T123753-csv-quoting-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T123753-csv-quoting-openswarm-r2/repo", "baseline": "7c763563399a0bcfada51cc813ed3fa824ff5ed2", "measure_from": "7c763563399a0bcfada51cc813ed3fa824ff5ed2"} +{"type": "run", "task": "date-range-parse", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:45:03.590367+00:00", "end": "2026-10-17T12:45:45.748696+00:00", "wall_s": 42.157, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124503-date-range-parse-baseline-r2/agent-1.jsonl", "session_id": "c36103f4-ad1a-4be4-b24d-6d246eb60b24", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 25390, "outputTokens": 1945, "cacheReadTokens": 60032, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 25390, "outputTokens": 1945, "cacheReadTokens": 60032, "cacheWriteTokens": 0, "total": 87367}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["c36103f4-ad1a-4be4-b24d-6d246eb60b24"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T124503-date-range-parse-baseline-r2/agent-1.jsonl", "results/raw/20261017T124503-date-range-parse-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T124503-date-range-parse-baseline-r2/arm-stderr.txt", "results/raw/20261017T124503-date-range-parse-baseline-r2/arm-stdout.txt", "results/raw/20261017T124503-date-range-parse-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T124503-date-range-parse-baseline-r2/repo", "baseline": "93459cbfe8c8261d6d188d05b7c6f668c672cd91", "measure_from": "93459cbfe8c8261d6d188d05b7c6f668c672cd91"} +{"type": "run", "task": "date-range-parse", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:45:08.859512+00:00", "end": "2026-10-17T12:45:49.894198+00:00", "wall_s": 41.033, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124508-date-range-parse-baseline-r3/agent-1.jsonl", "session_id": "c352581a-3990-49da-a150-410e20e5b2c4", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 24409, "outputTokens": 1736, "cacheReadTokens": 78592, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 24409, "outputTokens": 1736, "cacheReadTokens": 78592, "cacheWriteTokens": 0, "total": 104737}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.073, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["c352581a-3990-49da-a150-410e20e5b2c4"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T124508-date-range-parse-baseline-r3/agent-1.jsonl", "results/raw/20261017T124508-date-range-parse-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T124508-date-range-parse-baseline-r3/arm-stderr.txt", "results/raw/20261017T124508-date-range-parse-baseline-r3/arm-stdout.txt", "results/raw/20261017T124508-date-range-parse-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T124508-date-range-parse-baseline-r3/repo", "baseline": "9f3333e7b49ac0ad9512f64f2b151975dcb62054", "measure_from": "9f3333e7b49ac0ad9512f64f2b151975dcb62054"} +{"type": "run", "task": "date-range-parse", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:45:49.905407+00:00", "end": "2026-10-17T12:47:19.816915+00:00", "wall_s": 89.908, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124549-date-range-parse-superpowers-r2/agent-1.jsonl", "session_id": "d7cc3745-936c-4bc9-9538-3262302999dc", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 24971, "outputTokens": 3947, "cacheReadTokens": 209920, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 24971, "outputTokens": 3947, "cacheReadTokens": 209920, "cacheWriteTokens": 0, "total": 238838}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.044, "stdout_tail": "", "stderr_tail": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 31 insertions(+), 1 deletion(-)\n", "session_ids": ["d7cc3745-936c-4bc9-9538-3262302999dc"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T124549-date-range-parse-superpowers-r2/agent-1.jsonl", "results/raw/20261017T124549-date-range-parse-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T124549-date-range-parse-superpowers-r2/arm-stderr.txt", "results/raw/20261017T124549-date-range-parse-superpowers-r2/arm-stdout.txt", "results/raw/20261017T124549-date-range-parse-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T124549-date-range-parse-superpowers-r2/repo", "baseline": "03b71bdd0a2b8cfabc03406eb51b90f3a5f8989a", "measure_from": "03b71bdd0a2b8cfabc03406eb51b90f3a5f8989a"} +{"type": "run", "task": "date-range-parse", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:45:45.761275+00:00", "end": "2026-10-17T12:47:39.213518+00:00", "wall_s": 113.45, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124545-date-range-parse-superpowers-r1/agent-1.jsonl", "session_id": "1f52b137-57c6-45f3-8522-14020993131d", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 65267, "outputTokens": 5363, "cacheReadTokens": 204032, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 65267, "outputTokens": 5363, "cacheReadTokens": 204032, "cacheWriteTokens": 0, "total": 274662}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 1 deletion(-)\n", "session_ids": ["1f52b137-57c6-45f3-8522-14020993131d"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T124545-date-range-parse-superpowers-r1/agent-1.jsonl", "results/raw/20261017T124545-date-range-parse-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T124545-date-range-parse-superpowers-r1/arm-stderr.txt", "results/raw/20261017T124545-date-range-parse-superpowers-r1/arm-stdout.txt", "results/raw/20261017T124545-date-range-parse-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T124545-date-range-parse-superpowers-r1/repo", "baseline": "88827924fcc68fc6c848f3cffcbb806494883e8b", "measure_from": "88827924fcc68fc6c848f3cffcbb806494883e8b"} +{"type": "run", "task": "csv-quoting", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:36:43.962631+00:00", "end": "2026-10-17T12:48:11.069102+00:00", "wall_s": 687.103, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T123643-csv-quoting-openswarm-r1/agent-1.jsonl", "session_id": "fcc84471-af70-4b58-a6e3-e50118aab5fe", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49109, "outputTokens": 9658, "cacheReadTokens": 159360, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "56406353-929d-4b48-9541-101de6d72aea", "model": "grok-4.7-high", "usage": {"inputTokens": 98762, "outputTokens": 24006, "cacheReadTokens": 178944, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 147871, "outputTokens": 33664, "cacheReadTokens": 338304, "cacheWriteTokens": 0, "total": 519839}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["csvquote.py"], "diff_stat": " csvquote.py | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 50 insertions(+), 7 deletions(-)\n", "session_ids": ["fcc84471-af70-4b58-a6e3-e50118aab5fe", "56406353-929d-4b48-9541-101de6d72aea"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T123643-csv-quoting-openswarm-r1/agent-1.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T123643-csv-quoting-openswarm-r1/arm-stderr.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/arm-stdout.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/executor-task.md", "results/raw/20261017T123643-csv-quoting-openswarm-r1/executor-usage.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/merge.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/diff.patch", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/events.1.cursor.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/events.1.cursor.jsonl.err", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/result.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/status.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/runs/20261017-203958-1d6950/task.md", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-result.json", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/openswarm.err", "results/raw/20261017T123643-csv-quoting-openswarm-r1/planner-prompt.txt", "results/raw/20261017T123643-csv-quoting-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T123643-csv-quoting-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T123643-csv-quoting-openswarm-r1/repo", "baseline": "7021e10868712e05a73fa8d9633663439e805e3d", "measure_from": "7021e10868712e05a73fa8d9633663439e805e3d"} +{"type": "run", "task": "date-range-parse", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:47:39.225615+00:00", "end": "2026-10-17T12:49:56.509243+00:00", "wall_s": 137.281, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124739-date-range-parse-openswarm-r1/agent-1.jsonl", "session_id": "28288edc-b18c-4e56-9406-953432732a82", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 37576, "outputTokens": 4577, "cacheReadTokens": 71680, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "c3ab648b-624e-4580-aa6b-a6b333e079a7", "model": "grok-4.7-high", "usage": {"inputTokens": 26156, "outputTokens": 1494, "cacheReadTokens": 64896, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 63732, "outputTokens": 6071, "cacheReadTokens": 136576, "cacheWriteTokens": 0, "total": 206379}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.053, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.045, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["28288edc-b18c-4e56-9406-953432732a82", "c3ab648b-624e-4580-aa6b-a6b333e079a7"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T124739-date-range-parse-openswarm-r1/agent-1.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T124739-date-range-parse-openswarm-r1/arm-stderr.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/arm-stdout.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/executor-task.md", "results/raw/20261017T124739-date-range-parse-openswarm-r1/executor-usage.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/merge.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/diff.patch", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/events.1.cursor.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/events.1.cursor.jsonl.err", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/result.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/status.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/runs/20261017-204921-a5042b/task.md", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-result.json", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/openswarm.err", "results/raw/20261017T124739-date-range-parse-openswarm-r1/planner-prompt.txt", "results/raw/20261017T124739-date-range-parse-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T124739-date-range-parse-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T124739-date-range-parse-openswarm-r1/repo", "baseline": "18a2caa76595dab0153aa6bd4b2712ac2598120c", "measure_from": "18a2caa76595dab0153aa6bd4b2712ac2598120c"} +{"type": "run", "task": "date-range-parse", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:47:19.830665+00:00", "end": "2026-10-17T12:50:02.383110+00:00", "wall_s": 162.55, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124719-date-range-parse-superpowers-r3/agent-1.jsonl", "session_id": "dbba04bc-c4d8-4574-b7de-3912c4fbb5f1", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 30115, "outputTokens": 7377, "cacheReadTokens": 316288, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 30115, "outputTokens": 7377, "cacheReadTokens": 316288, "cacheWriteTokens": 0, "total": 353780}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.05, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 40 ++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 41 insertions(+), 1 deletion(-)\n", "session_ids": ["dbba04bc-c4d8-4574-b7de-3912c4fbb5f1"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T124719-date-range-parse-superpowers-r3/agent-1.jsonl", "results/raw/20261017T124719-date-range-parse-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T124719-date-range-parse-superpowers-r3/arm-stderr.txt", "results/raw/20261017T124719-date-range-parse-superpowers-r3/arm-stdout.txt", "results/raw/20261017T124719-date-range-parse-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T124719-date-range-parse-superpowers-r3/repo", "baseline": "e1e2f29a73eda5a1f13b076c051ef4f4c8cb5bb8", "measure_from": "e1e2f29a73eda5a1f13b076c051ef4f4c8cb5bb8"} +{"type": "run", "task": "date-range-parse", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:48:11.080692+00:00", "end": "2026-10-17T12:50:13.416537+00:00", "wall_s": 122.334, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124811-date-range-parse-openswarm-r2/agent-1.jsonl", "session_id": "cfeda10b-81a5-45a4-94a0-7bd5d5d9c037", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 32394, "outputTokens": 4068, "cacheReadTokens": 76416, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "238eb454-44ae-4229-b4e7-aaeb8516b1f6", "model": "grok-4.7-high", "usage": {"inputTokens": 13651, "outputTokens": 1845, "cacheReadTokens": 58880, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 46045, "outputTokens": 5913, "cacheReadTokens": 135296, "cacheWriteTokens": 0, "total": 187254}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".........\n----------------------------------------------------------------------\nRan 9 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.049, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py", "tests/test_daterange.py"], "diff_stat": " daterange.py | 2 +-\n tests/test_daterange.py | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 49 insertions(+), 1 deletion(-)\n", "session_ids": ["cfeda10b-81a5-45a4-94a0-7bd5d5d9c037", "238eb454-44ae-4229-b4e7-aaeb8516b1f6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T124811-date-range-parse-openswarm-r2/agent-1.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T124811-date-range-parse-openswarm-r2/arm-stderr.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/arm-stdout.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/executor-task.md", "results/raw/20261017T124811-date-range-parse-openswarm-r2/executor-usage.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/merge.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/diff.patch", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/events.1.cursor.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/events.1.cursor.jsonl.err", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/result.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/status.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/runs/20261017-204938-fa0447/task.md", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-result.json", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/openswarm.err", "results/raw/20261017T124811-date-range-parse-openswarm-r2/planner-prompt.txt", "results/raw/20261017T124811-date-range-parse-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T124811-date-range-parse-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T124811-date-range-parse-openswarm-r2/repo", "baseline": "7583294e00e9412bccbf5628d57d13ff75984651", "measure_from": "7583294e00e9412bccbf5628d57d13ff75984651"} +{"type": "run", "task": "lru-eviction", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:50:02.394809+00:00", "end": "2026-10-17T12:50:52.081210+00:00", "wall_s": 49.685, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125002-lru-eviction-baseline-r1/agent-1.jsonl", "session_id": "2bf45f2e-f4c6-4a47-b6b1-94dedd2e9ee3", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 29867, "outputTokens": 2106, "cacheReadTokens": 73728, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 29867, "outputTokens": 2106, "cacheReadTokens": 73728, "cacheWriteTokens": 0, "total": 105701}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.731326\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.128623\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.342481\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.895243\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.101, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.778803\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.195814\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.162245\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.124738\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.263482\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.414941\n", "stderr_tail": ""}, "files_changed": ["src/lru.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n", "session_ids": ["2bf45f2e-f4c6-4a47-b6b1-94dedd2e9ee3"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125002-lru-eviction-baseline-r1/agent-1.jsonl", "results/raw/20261017T125002-lru-eviction-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T125002-lru-eviction-baseline-r1/arm-stderr.txt", "results/raw/20261017T125002-lru-eviction-baseline-r1/arm-stdout.txt", "results/raw/20261017T125002-lru-eviction-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T125002-lru-eviction-baseline-r1/repo", "baseline": "16beb49ea46d5229aea652f988ffc6f28804e67c", "measure_from": "16beb49ea46d5229aea652f988ffc6f28804e67c"} +{"type": "run", "task": "lru-eviction", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:50:13.428725+00:00", "end": "2026-10-17T12:51:05.784373+00:00", "wall_s": 52.354, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125013-lru-eviction-baseline-r2/agent-1.jsonl", "session_id": "393bed56-97bc-429b-b996-b08e93caa1a6", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 36478, "outputTokens": 2229, "cacheReadTokens": 85376, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 36478, "outputTokens": 2229, "cacheReadTokens": 85376, "cacheWriteTokens": 0, "total": 124083}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.138, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.881029\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.150391\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.326134\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.194338\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.814772\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.140211\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.16046\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.11712\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.350916\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 68.032049\n", "stderr_tail": ""}, "files_changed": ["src/lru.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n", "session_ids": ["393bed56-97bc-429b-b996-b08e93caa1a6"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125013-lru-eviction-baseline-r2/agent-1.jsonl", "results/raw/20261017T125013-lru-eviction-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T125013-lru-eviction-baseline-r2/arm-stderr.txt", "results/raw/20261017T125013-lru-eviction-baseline-r2/arm-stdout.txt", "results/raw/20261017T125013-lru-eviction-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T125013-lru-eviction-baseline-r2/repo", "baseline": "869cbe02b6af1f6366877828a439984de73d7cfe", "measure_from": "869cbe02b6af1f6366877828a439984de73d7cfe"} +{"type": "run", "task": "lru-eviction", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:50:52.093480+00:00", "end": "2026-10-17T12:51:43.280006+00:00", "wall_s": 51.185, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125052-lru-eviction-baseline-r3/agent-1.jsonl", "session_id": "1f3ff685-7762-43c9-89a3-86cd766545a3", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 25599, "outputTokens": 2151, "cacheReadTokens": 78336, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 25599, "outputTokens": 2151, "cacheReadTokens": 78336, "cacheWriteTokens": 0, "total": 106086}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.098, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.683018\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.122577\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.356142\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.079481\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.869003\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.226648\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.257172\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.13057\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.309019\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.74608\n", "stderr_tail": ""}, "files_changed": ["src/lru.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n", "session_ids": ["1f3ff685-7762-43c9-89a3-86cd766545a3"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125052-lru-eviction-baseline-r3/agent-1.jsonl", "results/raw/20261017T125052-lru-eviction-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T125052-lru-eviction-baseline-r3/arm-stderr.txt", "results/raw/20261017T125052-lru-eviction-baseline-r3/arm-stdout.txt", "results/raw/20261017T125052-lru-eviction-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T125052-lru-eviction-baseline-r3/repo", "baseline": "892984578c469b77392c9cb6907647091f441fc8", "measure_from": "892984578c469b77392c9cb6907647091f441fc8"} +{"type": "run", "task": "date-range-parse", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:49:56.522852+00:00", "end": "2026-10-17T12:52:03.649922+00:00", "wall_s": 127.124, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T124956-date-range-parse-openswarm-r3/agent-1.jsonl", "session_id": "1a55ed8e-c8ad-453a-b2ab-bdaf42076ab8", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 19482, "outputTokens": 4590, "cacheReadTokens": 113920, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "3dfaeea6-9dd0-4fed-9a15-5b2435f09bd0", "model": "grok-4.7-high", "usage": {"inputTokens": 13127, "outputTokens": 1073, "cacheReadTokens": 76800, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 32609, "outputTokens": 5663, "cacheReadTokens": 190720, "cacheWriteTokens": 0, "total": 228992}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.118, "stdout_tail": "", "stderr_tail": "...\n----------------------------------------------------------------------\nRan 3 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_daterange.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.063, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["daterange.py"], "diff_stat": " daterange.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["1a55ed8e-c8ad-453a-b2ab-bdaf42076ab8", "3dfaeea6-9dd0-4fed-9a15-5b2435f09bd0"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T124956-date-range-parse-openswarm-r3/agent-1.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T124956-date-range-parse-openswarm-r3/arm-stderr.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/arm-stdout.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/executor-task.md", "results/raw/20261017T124956-date-range-parse-openswarm-r3/executor-usage.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/merge.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/diff.patch", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/events.1.cursor.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/events.1.cursor.jsonl.err", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/result.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/status.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/runs/20261017-205135-3c1b69/task.md", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-result.json", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/openswarm.err", "results/raw/20261017T124956-date-range-parse-openswarm-r3/planner-prompt.txt", "results/raw/20261017T124956-date-range-parse-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T124956-date-range-parse-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T124956-date-range-parse-openswarm-r3/repo", "baseline": "35b26a77786eabe7f509bb741067af251ac54227", "measure_from": "35b26a77786eabe7f509bb741067af251ac54227"} +{"type": "run", "task": "lru-eviction", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:51:05.797087+00:00", "end": "2026-10-17T12:55:11.354393+00:00", "wall_s": 245.553, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125105-lru-eviction-superpowers-r1/agent-1.jsonl", "session_id": "4e7550f7-f00a-4c9b-82de-7c13c8967000", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 72646, "outputTokens": 11596, "cacheReadTokens": 449792, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 72646, "outputTokens": 11596, "cacheReadTokens": 449792, "cacheWriteTokens": 0, "total": 534034}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.838977\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.173726\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.43764\n type: 'test'\n ...\n# Subtest: reading a key protects it from the next eviction\nok 4 - reading a key protects it from the next eviction\n ---\n duration_ms: 1.098358\n type: 'test'\n ...\n# Subtest: updating an existing key protects it from the next eviction\nok 5 - updating an existing key protects it from the next eviction\n ---\n duration_ms: 0.202359\n type: 'test'\n ...\n# Subtest: updating a present key does not evict another entry\nok 6 - updating a present key does not evict another entry\n ---\n duration_ms: 0.098939\n type: 'test'\n ...\n# Subtest: a missing read does not change eviction order\nok 7 - a missing read does not change eviction order\n ---\n duration_ms: 0.134687\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.297303\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.102, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.73066\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.123836\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.164342\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.097773\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.305831\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.300957\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 11 +++++++++--\n test/lru.test.js | 42 ++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 2 deletions(-)\n", "session_ids": ["4e7550f7-f00a-4c9b-82de-7c13c8967000"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125105-lru-eviction-superpowers-r1/agent-1.jsonl", "results/raw/20261017T125105-lru-eviction-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T125105-lru-eviction-superpowers-r1/arm-stderr.txt", "results/raw/20261017T125105-lru-eviction-superpowers-r1/arm-stdout.txt", "results/raw/20261017T125105-lru-eviction-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T125105-lru-eviction-superpowers-r1/repo", "baseline": "7fe0118dc6f31887d82a50d353084ef155cb1ad2", "measure_from": "7fe0118dc6f31887d82a50d353084ef155cb1ad2"} +{"type": "run", "task": "lru-eviction", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:52:03.687094+00:00", "end": "2026-10-17T12:55:16.814032+00:00", "wall_s": 193.124, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125203-lru-eviction-superpowers-r3/agent-1.jsonl", "session_id": "2af5eba9-35bb-4295-aad3-f29fa4f730de", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 32196, "outputTokens": 8493, "cacheReadTokens": 385536, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 32196, "outputTokens": 8493, "cacheReadTokens": 385536, "cacheWriteTokens": 0, "total": 426225}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.700715\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.163695\n type: 'test'\n ...\n# Subtest: reading a key makes it newest so the next insert evicts the unread entry\nok 3 - reading a key makes it newest so the next insert evicts the unread entry\n ---\n duration_ms: 0.100627\n type: 'test'\n ...\n# Subtest: updating a present key keeps the other entry and stores the new value\nok 4 - updating a present key keeps the other entry and stores the new value\n ---\n duration_ms: 1.324602\n type: 'test'\n ...\n# Subtest: updating a present key makes it newest so the next insert evicts the other entry\nok 5 - updating a present key makes it newest so the next insert evicts the other entry\n ---\n duration_ms: 0.150326\n type: 'test'\n ...\n# Subtest: reading a missing key returns undefined and leaves eviction order unchanged\nok 6 - reading a missing key returns undefined and leaves eviction order unchanged\n ---\n duration_ms: 0.117871\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 7 - rejects a capacity below 1\n ---\n duration_ms: 0.475395\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.439044\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.109, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.894122\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.165305\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.160776\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.120689\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.501519\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 70.557112\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 12 ++++++++++--\n test/lru.test.js | 42 ++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 52 insertions(+), 2 deletions(-)\n", "session_ids": ["2af5eba9-35bb-4295-aad3-f29fa4f730de"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125203-lru-eviction-superpowers-r3/agent-1.jsonl", "results/raw/20261017T125203-lru-eviction-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T125203-lru-eviction-superpowers-r3/arm-stderr.txt", "results/raw/20261017T125203-lru-eviction-superpowers-r3/arm-stdout.txt", "results/raw/20261017T125203-lru-eviction-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T125203-lru-eviction-superpowers-r3/repo", "baseline": "eec39c13cc4bd024c04590c40d3cf416d0229392", "measure_from": "eec39c13cc4bd024c04590c40d3cf416d0229392"} +{"type": "run", "task": "lru-eviction", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:51:43.291273+00:00", "end": "2026-10-17T12:55:18.461222+00:00", "wall_s": 215.167, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125143-lru-eviction-superpowers-r2/agent-1.jsonl", "session_id": "f6605a25-39a4-4579-bba4-e463ebd6a03b", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 96536, "outputTokens": 9939, "cacheReadTokens": 322176, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 96536, "outputTokens": 9939, "cacheReadTokens": 322176, "cacheWriteTokens": 0, "total": 428651}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.118, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 1.337607\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.24041\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 1.703173\n type: 'test'\n ...\n# Subtest: reading a key protects it from the next eviction\nok 4 - reading a key protects it from the next eviction\n ---\n duration_ms: 0.185719\n type: 'test'\n ...\n# Subtest: updating an existing key protects it from the next eviction\nok 5 - updating an existing key protects it from the next eviction\n ---\n duration_ms: 0.17511\n type: 'test'\n ...\n# Subtest: updating an existing key keeps the other entry\nok 6 - updating an existing key keeps the other entry\n ---\n duration_ms: 0.109131\n type: 'test'\n ...\n# Subtest: updating the only key keeps it\nok 7 - updating the only key keeps it\n ---\n duration_ms: 0.180693\n type: 'test'\n ...\n# Subtest: reading a missing key leaves the eviction order unchanged\nok 8 - reading a missing key leaves the eviction order unchanged\n ---\n duration_ms: 0.102631\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 74.151045\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.111, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 1.147862\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.223329\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.250793\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.14127\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.271841\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.526238\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 6 +++++-\n test/lru.test.js | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 54 insertions(+), 1 deletion(-)\n", "session_ids": ["f6605a25-39a4-4579-bba4-e463ebd6a03b"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125143-lru-eviction-superpowers-r2/agent-1.jsonl", "results/raw/20261017T125143-lru-eviction-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T125143-lru-eviction-superpowers-r2/arm-stderr.txt", "results/raw/20261017T125143-lru-eviction-superpowers-r2/arm-stdout.txt", "results/raw/20261017T125143-lru-eviction-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T125143-lru-eviction-superpowers-r2/repo", "baseline": "1c14d75e65a820976b1aae6b9b54079f17874650", "measure_from": "1c14d75e65a820976b1aae6b9b54079f17874650"} +{"type": "run", "task": "lru-eviction", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T12:55:11.368324+00:00", "end": "2026-10-17T12:58:47.227015+00:00", "wall_s": 215.856, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125511-lru-eviction-openswarm-r1/agent-1.jsonl", "session_id": "3f87c4c2-1a21-4343-bb09-77b0497a84df", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 34151, "outputTokens": 4937, "cacheReadTokens": 98176, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "7d0b3083-3b0e-4a22-a94a-8aae643efcdd", "model": "grok-4.7-high", "usage": {"inputTokens": 53600, "outputTokens": 5319, "cacheReadTokens": 49024, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 87751, "outputTokens": 10256, "cacheReadTokens": 147200, "cacheWriteTokens": 0, "total": 245207}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.694715\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.181774\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 1.326041\n type: 'test'\n ...\n# Subtest: reading a key refreshes recency so it is not evicted next\nok 4 - reading a key refreshes recency so it is not evicted next\n ---\n duration_ms: 0.185668\n type: 'test'\n ...\n# Subtest: updating an existing key refreshes recency and does not grow\nok 5 - updating an existing key refreshes recency and does not grow\n ---\n duration_ms: 0.210548\n type: 'test'\n ...\n# Subtest: a miss does not change eviction order\nok 6 - a miss does not change eviction order\n ---\n duration_ms: 0.112809\n type: 'test'\n ...\n# Subtest: evicts the least recently used key after several uses\nok 7 - evicts the least recently used key after several uses\n ---\n duration_ms: 0.216612\n type: 'test'\n ...\n# Subtest: capacity 1 evicts the previous key only on a new insert\nok 8 - capacity 1 evicts the previous key only on a new insert\n ---\n duration_ms: 0.098663\n type: 'test'\n ...\n# Subtest: a stored undefined is a hit and refreshes recency\nok 9 - a stored undefined is a hit and refreshes recency\n ---\n duration_ms: 0.338839\n type: 'test'\n ...\n1..9\n# tests 9\n# suites 0\n# pass 9\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.616718\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.094, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.793236\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.208765\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.2131\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.159854\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.418475\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.371545\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 12 +++++++--\n test/lru.test.js | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 92 insertions(+), 2 deletions(-)\n", "session_ids": ["3f87c4c2-1a21-4343-bb09-77b0497a84df", "7d0b3083-3b0e-4a22-a94a-8aae643efcdd"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T125511-lru-eviction-openswarm-r1/agent-1.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T125511-lru-eviction-openswarm-r1/arm-stderr.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/arm-stdout.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/executor-task.md", "results/raw/20261017T125511-lru-eviction-openswarm-r1/executor-usage.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/merge.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/diff.patch", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/events.1.cursor.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/events.1.cursor.jsonl.err", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/result.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/status.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/runs/20261017-205701-013c09/task.md", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-result.json", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/openswarm.err", "results/raw/20261017T125511-lru-eviction-openswarm-r1/planner-prompt.txt", "results/raw/20261017T125511-lru-eviction-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T125511-lru-eviction-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T125511-lru-eviction-openswarm-r1/repo", "baseline": "76b075a5fe5b049324713dd0eee4bf40ba237835", "measure_from": "76b075a5fe5b049324713dd0eee4bf40ba237835"} +{"type": "run", "task": "lru-eviction", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T12:55:16.826740+00:00", "end": "2026-10-17T12:58:55.801967+00:00", "wall_s": 218.972, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125516-lru-eviction-openswarm-r2/agent-1.jsonl", "session_id": "7d0e3151-5789-45ee-a4b9-5bdfc4fe0849", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18266, "outputTokens": 3703, "cacheReadTokens": 109952, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "3f319aef-f47e-4a86-8598-7a20fc334c34", "model": "grok-4.7-high", "usage": {"inputTokens": 19599, "outputTokens": 7051, "cacheReadTokens": 65664, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 37865, "outputTokens": 10754, "cacheReadTokens": 175616, "cacheWriteTokens": 0, "total": 224235}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.092, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.881146\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.201785\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.477353\n type: 'test'\n ...\n# Subtest: get of a present key refreshes recency\nok 4 - get of a present key refreshes recency\n ---\n duration_ms: 0.121558\n type: 'test'\n ...\n# Subtest: set of a present key refreshes recency without evicting\nok 5 - set of a present key refreshes recency without evicting\n ---\n duration_ms: 0.18424\n type: 'test'\n ...\n# Subtest: get of a missing key does not refresh recency\nok 6 - get of a missing key does not refresh recency\n ---\n duration_ms: 0.095158\n type: 'test'\n ...\n# Subtest: capacity 1 evicts on insert and keeps the key on update\nok 7 - capacity 1 evicts on insert and keeps the key on update\n ---\n duration_ms: 0.177509\n type: 'test'\n ...\n# Subtest: repeated get of the newest key leaves the oldest key unchanged\nok 8 - repeated get of the newest key leaves the oldest key unchanged\n ---\n duration_ms: 0.084472\n type: 'test'\n ...\n# Subtest: use of one key leaves older unused keys in order\nok 9 - use of one key leaves older unused keys in order\n ---\n duration_ms: 0.265697\n type: 'test'\n ...\n# Subtest: rejects a non-integer capacity and accepts 1\nok 10 - rejects a non-integer capacity and accepts 1\n ---\n duration_ms: 0.322364\n type: 'test'\n ...\n1..10\n# tests 10\n# suites 0\n# pass 10\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 58.449088\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 1.095405\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.216555\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.228947\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.155527\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.37121\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 66.20635\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 11 +++----\n test/lru.test.js | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 96 insertions(+), 5 deletions(-)\n", "session_ids": ["7d0e3151-5789-45ee-a4b9-5bdfc4fe0849", "3f319aef-f47e-4a86-8598-7a20fc334c34"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T125516-lru-eviction-openswarm-r2/agent-1.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T125516-lru-eviction-openswarm-r2/arm-stderr.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/arm-stdout.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/executor-task.md", "results/raw/20261017T125516-lru-eviction-openswarm-r2/executor-usage.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/merge.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/diff.patch", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/events.1.cursor.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/events.1.cursor.jsonl.err", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/result.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/status.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/runs/20261017-205638-676cc4/task.md", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-result.json", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/openswarm.err", "results/raw/20261017T125516-lru-eviction-openswarm-r2/planner-prompt.txt", "results/raw/20261017T125516-lru-eviction-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T125516-lru-eviction-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T125516-lru-eviction-openswarm-r2/repo", "baseline": "d4c8d592494a4397efa2bcc10c13abdab8afaecc", "measure_from": "d4c8d592494a4397efa2bcc10c13abdab8afaecc"} +{"type": "run", "task": "lru-eviction", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T12:55:18.473862+00:00", "end": "2026-10-17T12:59:02.894426+00:00", "wall_s": 224.418, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125518-lru-eviction-openswarm-r3/agent-1.jsonl", "session_id": "fa943001-4d2a-4c93-bed4-6e58f3cae92e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 35691, "outputTokens": 4234, "cacheReadTokens": 116992, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "9a57fe25-fbe3-444a-9070-8d8897e28879", "model": "grok-4.7-high", "usage": {"inputTokens": 18287, "outputTokens": 6049, "cacheReadTokens": 83840, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 53978, "outputTokens": 10283, "cacheReadTokens": 200832, "cacheWriteTokens": 0, "total": 265093}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: stores and returns a value\nok 1 - stores and returns a value\n ---\n duration_ms: 0.691149\n type: 'test'\n ...\n# Subtest: evicts the oldest untouched insertion when the cache is cold\nok 2 - evicts the oldest untouched insertion when the cache is cold\n ---\n duration_ms: 0.161168\n type: 'test'\n ...\n# Subtest: rejects a capacity below 1\nok 3 - rejects a capacity below 1\n ---\n duration_ms: 0.429181\n type: 'test'\n ...\n# Subtest: a read protects a key from eviction\nok 4 - a read protects a key from eviction\n ---\n duration_ms: 0.991794\n type: 'test'\n ...\n# Subtest: an update protects a key and does not grow the cache\nok 5 - an update protects a key and does not grow the cache\n ---\n duration_ms: 0.211747\n type: 'test'\n ...\n# Subtest: a missing get does not change eviction order\nok 6 - a missing get does not change eviction order\n ---\n duration_ms: 0.109842\n type: 'test'\n ...\n# Subtest: capacity 1 keeps a used key until a different key is inserted\nok 7 - capacity 1 keeps a used key until a different key is inserted\n ---\n duration_ms: 0.143623\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.679978\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/lru.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: a get refreshes recency so the other key is evicted\nok 1 - a get refreshes recency so the other key is evicted\n ---\n duration_ms: 0.650414\n type: 'test'\n ...\n# Subtest: replacing an existing key refreshes recency and does not grow the cache\nok 2 - replacing an existing key refreshes recency and does not grow the cache\n ---\n duration_ms: 0.10978\n type: 'test'\n ...\n# Subtest: capacity 1 keeps only the newest key\nok 3 - capacity 1 keeps only the newest key\n ---\n duration_ms: 0.118243\n type: 'test'\n ...\n# Subtest: a missing key stays missing\nok 4 - a missing key stays missing\n ---\n duration_ms: 0.080189\n type: 'test'\n ...\n# Subtest: regression: a cold eviction still drops the oldest insertion\nok 5 - regression: a cold eviction still drops the oldest insertion\n ---\n duration_ms: 0.246426\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.320229\n", "stderr_tail": ""}, "files_changed": ["src/lru.js", "test/lru.test.js"], "diff_stat": " src/lru.js | 6 +++++-\n test/lru.test.js | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 49 insertions(+), 1 deletion(-)\n", "session_ids": ["fa943001-4d2a-4c93-bed4-6e58f3cae92e", "9a57fe25-fbe3-444a-9070-8d8897e28879"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T125518-lru-eviction-openswarm-r3/agent-1.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T125518-lru-eviction-openswarm-r3/arm-stderr.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/arm-stdout.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/executor-task.md", "results/raw/20261017T125518-lru-eviction-openswarm-r3/executor-usage.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/merge.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/diff.patch", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/events.1.cursor.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/events.1.cursor.jsonl.err", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/result.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/status.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/runs/20261017-205655-307549/task.md", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-result.json", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/openswarm.err", "results/raw/20261017T125518-lru-eviction-openswarm-r3/planner-prompt.txt", "results/raw/20261017T125518-lru-eviction-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T125518-lru-eviction-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T125518-lru-eviction-openswarm-r3/repo", "baseline": "1550750f5508e6ad4330b1042129522e915a6841", "measure_from": "1550750f5508e6ad4330b1042129522e915a6841"} +{"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 1, "start": "2026-10-17T12:58:47.237775+00:00", "end": "2026-10-17T12:59:12.179541+00:00", "wall_s": 24.94, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/agent-1.jsonl", "session_id": "ca80c62c-3bfe-439a-addf-0a8c91eb9a9b", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 11821, "outputTokens": 890, "cacheReadTokens": 71552, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 11821, "outputTokens": 890, "cacheReadTokens": 71552, "cacheWriteTokens": 0, "total": 84263}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.103, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.867402\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.119049\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.145708\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 56.292827\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.104, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.254937\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.154047\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.11432\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.169017\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.472785\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.548395\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["ca80c62c-3bfe-439a-addf-0a8c91eb9a9b"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125847-pagination-off-by-one-baseline-r1/agent-1.jsonl", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/arm-stderr.txt", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/arm-stdout.txt", "results/raw/20261017T125847-pagination-off-by-one-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T125847-pagination-off-by-one-baseline-r1/repo", "baseline": "1ac3b6ad62041a4269c1c80da546ab6c15b32a4b", "measure_from": "1ac3b6ad62041a4269c1c80da546ab6c15b32a4b"} +{"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 2, "start": "2026-10-17T12:58:55.814224+00:00", "end": "2026-10-17T12:59:27.722094+00:00", "wall_s": 31.906, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/agent-1.jsonl", "session_id": "fc512105-58d6-453d-93f0-1c2203e98e9a", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 11802, "outputTokens": 1298, "cacheReadTokens": 72192, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 11802, "outputTokens": 1298, "cacheReadTokens": 72192, "cacheWriteTokens": 0, "total": 85292}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.983272\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.182395\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.13614\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.185381\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.106, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.200678\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.123256\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.089993\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.1365\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.426632\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.728613\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["fc512105-58d6-453d-93f0-1c2203e98e9a"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125855-pagination-off-by-one-baseline-r2/agent-1.jsonl", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/arm-stderr.txt", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/arm-stdout.txt", "results/raw/20261017T125855-pagination-off-by-one-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T125855-pagination-off-by-one-baseline-r2/repo", "baseline": "eb85e9e1c8a2ee9613991f5842875fcab6d0cf2f", "measure_from": "eb85e9e1c8a2ee9613991f5842875fcab6d0cf2f"} +{"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 3, "start": "2026-10-17T12:59:02.906210+00:00", "end": "2026-10-17T12:59:38.690315+00:00", "wall_s": 35.783, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/agent-1.jsonl", "session_id": "ea84b8ea-a9d1-4865-97f2-e19774d16d93", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 12208, "outputTokens": 1525, "cacheReadTokens": 72448, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 12208, "outputTokens": 1525, "cacheReadTokens": 72448, "cacheWriteTokens": 0, "total": 86181}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.104, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.74314\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.201965\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.163573\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.561865\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.310074\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.147253\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.110581\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.154054\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.457191\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.488139\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["ea84b8ea-a9d1-4865-97f2-e19774d16d93"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T125902-pagination-off-by-one-baseline-r3/agent-1.jsonl", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/arm-stderr.txt", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/arm-stdout.txt", "results/raw/20261017T125902-pagination-off-by-one-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T125902-pagination-off-by-one-baseline-r3/repo", "baseline": "157e196bf3565069e0e4f0bca7e173bf33e644f8", "measure_from": "157e196bf3565069e0e4f0bca7e173bf33e644f8"} +{"type": "run", "task": "pagination-off-by-one", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T12:59:12.191722+00:00", "end": "2026-10-17T13:01:11.198057+00:00", "wall_s": 119.003, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/agent-1.jsonl", "session_id": "e91b6775-b573-4a6e-ae7f-028f82371794", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 52090, "outputTokens": 5300, "cacheReadTokens": 224896, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 52090, "outputTokens": 5300, "cacheReadTokens": 224896, "cacheWriteTokens": 0, "total": 282286}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.118, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.975245\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.260761\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 1.227307\n type: 'test'\n ...\n# Subtest: page 1 returns the first pageSize items\nok 4 - page 1 returns the first pageSize items\n ---\n duration_ms: 0.866295\n type: 'test'\n ...\n# Subtest: page 2 returns the next window with no gap or overlap\nok 5 - page 2 returns the next window with no gap or overlap\n ---\n duration_ms: 0.143739\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.097736\n type: 'test'\n ...\n# Subtest: a page past the end returns no items and keeps totalPages\nok 7 - a page past the end returns no items and keeps totalPages\n ---\n duration_ms: 0.162069\n type: 'test'\n ...\n# Subtest: an empty input needs zero pages\nok 8 - an empty input needs zero pages\n ---\n duration_ms: 0.093088\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.106892\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.112, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.673628\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.21747\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.145926\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 1.279397\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.570514\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 71.45117\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 31 insertions(+), 1 deletion(-)\n", "session_ids": ["e91b6775-b573-4a6e-ae7f-028f82371794"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/agent-1.jsonl", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/arm-stderr.txt", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/arm-stdout.txt", "results/raw/20261017T125912-pagination-off-by-one-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T125912-pagination-off-by-one-superpowers-r1/repo", "baseline": "099493dbf634b33f50cdbe8557a11985c9d9879a", "measure_from": "099493dbf634b33f50cdbe8557a11985c9d9879a"} +{"type": "run", "task": "pagination-off-by-one", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T12:59:27.733205+00:00", "end": "2026-10-17T13:01:17.080965+00:00", "wall_s": 109.346, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/agent-1.jsonl", "session_id": "97909c90-3ab0-482c-8258-1e684df5eda5", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 38833, "outputTokens": 4767, "cacheReadTokens": 206976, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 38833, "outputTokens": 4767, "cacheReadTokens": 206976, "cacheWriteTokens": 0, "total": 250576}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.101, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.897896\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.204461\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.189743\n type: 'test'\n ...\n# Subtest: page 1 returns the first window\nok 4 - page 1 returns the first window\n ---\n duration_ms: 1.686635\n type: 'test'\n ...\n# Subtest: page 2 returns the next window with no gap\nok 5 - page 2 returns the next window with no gap\n ---\n duration_ms: 0.167183\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.122275\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.421699\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.092, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.141416\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.191176\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.116595\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.957212\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.454862\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.87494\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 16 ++++++++++++++++\n 2 files changed, 17 insertions(+), 1 deletion(-)\n", "session_ids": ["97909c90-3ab0-482c-8258-1e684df5eda5"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/agent-1.jsonl", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/arm-stderr.txt", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/arm-stdout.txt", "results/raw/20261017T125927-pagination-off-by-one-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T125927-pagination-off-by-one-superpowers-r2/repo", "baseline": "f5a8294a8c5bb7e2ade96c4ce8c657aa67ab888a", "measure_from": "f5a8294a8c5bb7e2ade96c4ce8c657aa67ab888a"} +{"type": "run", "task": "pagination-off-by-one", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T12:59:38.701981+00:00", "end": "2026-10-17T13:01:28.563222+00:00", "wall_s": 109.858, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/agent-1.jsonl", "session_id": "853f9654-fb32-4266-964d-1f7233881a41", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49189, "outputTokens": 4578, "cacheReadTokens": 213888, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49189, "outputTokens": 4578, "cacheReadTokens": 213888, "cacheWriteTokens": 0, "total": 267655}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.111, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.048749\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.226228\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.170319\n type: 'test'\n ...\n# Subtest: page 1 returns the first window\nok 4 - page 1 returns the first window\n ---\n duration_ms: 1.864153\n type: 'test'\n ...\n# Subtest: page 2 returns the next window with no gap\nok 5 - page 2 returns the next window with no gap\n ---\n duration_ms: 0.197587\n type: 'test'\n ...\n# Subtest: the last page may be shorter than pageSize\nok 6 - the last page may be shorter than pageSize\n ---\n duration_ms: 0.138557\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 73.531844\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.098, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.248434\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.165475\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.111535\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.9465\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.451636\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.093109\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js", "test/paginate.test.js"], "diff_stat": " src/paginate.js | 2 +-\n test/paginate.test.js | 16 ++++++++++++++++\n 2 files changed, 17 insertions(+), 1 deletion(-)\n", "session_ids": ["853f9654-fb32-4266-964d-1f7233881a41"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/agent-1.jsonl", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/arm-stderr.txt", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/arm-stdout.txt", "results/raw/20261017T125938-pagination-off-by-one-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T125938-pagination-off-by-one-superpowers-r3/repo", "baseline": "fe97160c67304a9ebd995287b6149eab5ffd87d3", "measure_from": "fe97160c67304a9ebd995287b6149eab5ffd87d3"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:01:11.210601+00:00", "end": "2026-10-17T13:02:40.346043+00:00", "wall_s": 89.133, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/agent-1.jsonl", "session_id": "86674e57-7daa-4a55-ab6c-0bace1f3b129", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 17202, "outputTokens": 2771, "cacheReadTokens": 67200, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "49f6a845-4581-4df5-8779-e0cf5461a939", "model": "grok-4.7-high", "usage": {"inputTokens": 12679, "outputTokens": 986, "cacheReadTokens": 75648, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 29881, "outputTokens": 3757, "cacheReadTokens": 142848, "cacheWriteTokens": 0, "total": 176486}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.941467\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.143655\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.15477\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 58.441961\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.092, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.157273\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.119628\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.0858\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.137082\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.485214\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 56.022935\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["86674e57-7daa-4a55-ab6c-0bace1f3b129", "49f6a845-4581-4df5-8779-e0cf5461a939"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/agent-1.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/arm-stderr.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/arm-stdout.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/executor-task.md", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/executor-usage.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/merge.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/diff.patch", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/events.1.cursor.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/events.1.cursor.jsonl.err", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/result.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/status.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/runs/20261017-210210-3e17a4/task.md", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-result.json", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/openswarm.err", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/planner-prompt.txt", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T130111-pagination-off-by-one-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T130111-pagination-off-by-one-openswarm-r1/repo", "baseline": "70dc830d40581eea0f1025242fc2cb9408a85370", "measure_from": "70dc830d40581eea0f1025242fc2cb9408a85370"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:01:17.093642+00:00", "end": "2026-10-17T13:02:50.528936+00:00", "wall_s": 93.433, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/agent-1.jsonl", "session_id": "04a6467c-0746-4740-a72f-b0954d9e8048", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 17887, "outputTokens": 3300, "cacheReadTokens": 87808, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "9c3928c6-27da-4e8e-9483-a645fb78c532", "model": "grok-4.7-high", "usage": {"inputTokens": 12947, "outputTokens": 882, "cacheReadTokens": 58752, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 30834, "outputTokens": 4182, "cacheReadTokens": 146560, "cacheWriteTokens": 0, "total": 181576}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 1.04262\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.154666\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.148434\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.373784\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.215865\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.140167\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.101159\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.144643\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.491384\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.423454\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["04a6467c-0746-4740-a72f-b0954d9e8048", "9c3928c6-27da-4e8e-9483-a645fb78c532"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/agent-1.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/arm-stderr.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/arm-stdout.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/executor-task.md", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/executor-usage.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/merge.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/diff.patch", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/events.1.cursor.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/events.1.cursor.jsonl.err", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/result.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/status.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/runs/20261017-210226-9fb739/task.md", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-result.json", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/openswarm.err", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/planner-prompt.txt", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T130117-pagination-off-by-one-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T130117-pagination-off-by-one-openswarm-r2/repo", "baseline": "4d45057fd3463d5eb159bdc29f8d0c9b750e23cf", "measure_from": "4d45057fd3463d5eb159bdc29f8d0c9b750e23cf"} +{"type": "run", "task": "pagination-off-by-one", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:01:28.575793+00:00", "end": "2026-10-17T13:03:18.474702+00:00", "wall_s": 109.896, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/agent-1.jsonl", "session_id": "0d53f802-1bcd-4e3e-890f-3b155bc1d221", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 18268, "outputTokens": 3518, "cacheReadTokens": 112256, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "8c0adf5c-136a-44cc-befb-9268ade774d8", "model": "grok-4.7-high", "usage": {"inputTokens": 13513, "outputTokens": 1574, "cacheReadTokens": 58880, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 31781, "outputTokens": 5092, "cacheReadTokens": 171136, "cacheWriteTokens": 0, "total": 208009}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: rejects a page below 1\nok 1 - rejects a page below 1\n ---\n duration_ms: 0.851679\n type: 'test'\n ...\n# Subtest: rejects a non-positive page size\nok 2 - rejects a non-positive page size\n ---\n duration_ms: 0.14026\n type: 'test'\n ...\n# Subtest: echoes total, page and pageSize for an even split\nok 3 - echoes total, page and pageSize for an even split\n ---\n duration_ms: 0.129146\n type: 'test'\n ...\n1..3\n# tests 3\n# suites 0\n# pass 3\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.115911\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/paginate.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.097, "stdout_tail": "TAP version 13\n# Subtest: page 1 is the first window, not the second\nok 1 - page 1 is the first window, not the second\n ---\n duration_ms: 1.359887\n type: 'test'\n ...\n# Subtest: the last page keeps a short remainder\nok 2 - the last page keeps a short remainder\n ---\n duration_ms: 0.220623\n type: 'test'\n ...\n# Subtest: page 2 starts immediately after page 1\nok 3 - page 2 starts immediately after page 1\n ---\n duration_ms: 0.147357\n type: 'test'\n ...\n# Subtest: a page past the end is an empty window and totalPages still covers every item\nok 4 - a page past the end is an empty window and totalPages still covers every item\n ---\n duration_ms: 0.214495\n type: 'test'\n ...\n# Subtest: regression: page 0 still throws\nok 5 - regression: page 0 still throws\n ---\n duration_ms: 0.626041\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.304978\n", "stderr_tail": ""}, "files_changed": ["src/paginate.js"], "diff_stat": " src/paginate.js | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n", "session_ids": ["0d53f802-1bcd-4e3e-890f-3b155bc1d221", "8c0adf5c-136a-44cc-befb-9268ade774d8"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/agent-1.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/arm-stderr.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/arm-stdout.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/executor-task.md", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/executor-usage.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/merge.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/diff.patch", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/events.1.cursor.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/events.1.cursor.jsonl.err", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/result.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/status.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/runs/20261017-210243-f83d56/task.md", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-result.json", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/openswarm.err", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/planner-prompt.txt", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T130128-pagination-off-by-one-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T130128-pagination-off-by-one-openswarm-r3/repo", "baseline": "462f472f0cd21680487f7e9f564e28d9239a6ec3", "measure_from": "462f472f0cd21680487f7e9f564e28d9239a6ec3"} +{"type": "run", "task": "path-normalise", "arm": "baseline", "repeat": 1, "start": "2026-10-17T13:02:40.357781+00:00", "end": "2026-10-17T13:03:44.016760+00:00", "wall_s": 63.657, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130240-path-normalise-baseline-r1/agent-1.jsonl", "session_id": "8b988725-ea61-455c-95a3-1b0695c6b1fc", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 14379, "outputTokens": 3061, "cacheReadTokens": 92160, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 14379, "outputTokens": 3061, "cacheReadTokens": 92160, "cacheWriteTokens": 0, "total": 109600}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.113, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.726362\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.137229\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.094587\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.41591\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 72.039377\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.114, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.34911\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.333673\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.226276\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.200338\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.406466\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 79.472075\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n 1 file changed, 8 insertions(+)\n", "session_ids": ["8b988725-ea61-455c-95a3-1b0695c6b1fc"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130240-path-normalise-baseline-r1/agent-1.jsonl", "results/raw/20261017T130240-path-normalise-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T130240-path-normalise-baseline-r1/arm-stderr.txt", "results/raw/20261017T130240-path-normalise-baseline-r1/arm-stdout.txt", "results/raw/20261017T130240-path-normalise-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T130240-path-normalise-baseline-r1/repo", "baseline": "6da2210ab4db9932a6f6bca4961d9c46c17cee0b", "measure_from": "6da2210ab4db9932a6f6bca4961d9c46c17cee0b"} +{"type": "run", "task": "path-normalise", "arm": "baseline", "repeat": 2, "start": "2026-10-17T13:02:50.541557+00:00", "end": "2026-10-17T13:03:51.098298+00:00", "wall_s": 60.555, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130250-path-normalise-baseline-r2/agent-1.jsonl", "session_id": "54b3a809-3fce-4849-812c-c8a3a326582f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 52115, "outputTokens": 2828, "cacheReadTokens": 54144, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 52115, "outputTokens": 2828, "cacheReadTokens": 54144, "cacheWriteTokens": 0, "total": 109087}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.104, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.710498\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.122123\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.091378\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.312144\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.757137\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.733066\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.168159\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.111486\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.150096\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.364671\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 64.264862\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n 1 file changed, 8 insertions(+)\n", "session_ids": ["54b3a809-3fce-4849-812c-c8a3a326582f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130250-path-normalise-baseline-r2/agent-1.jsonl", "results/raw/20261017T130250-path-normalise-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T130250-path-normalise-baseline-r2/arm-stderr.txt", "results/raw/20261017T130250-path-normalise-baseline-r2/arm-stdout.txt", "results/raw/20261017T130250-path-normalise-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T130250-path-normalise-baseline-r2/repo", "baseline": "456519dee46ac5146a306daca3f49791ffa66d56", "measure_from": "456519dee46ac5146a306daca3f49791ffa66d56"} +{"type": "run", "task": "path-normalise", "arm": "baseline", "repeat": 3, "start": "2026-10-17T13:03:18.486949+00:00", "end": "2026-10-17T13:04:12.913540+00:00", "wall_s": 54.425, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130318-path-normalise-baseline-r3/agent-1.jsonl", "session_id": "ffb4644b-813c-4fe3-b1e9-78faebb821ce", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 13498, "outputTokens": 2560, "cacheReadTokens": 72832, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 13498, "outputTokens": 2560, "cacheReadTokens": 72832, "cacheWriteTokens": 0, "total": 88890}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.794757\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.13427\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.117864\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.336965\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 60.323211\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.113, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.040068\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.239179\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.117413\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.101998\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.280063\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.232852\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n 1 file changed, 8 insertions(+)\n", "session_ids": ["ffb4644b-813c-4fe3-b1e9-78faebb821ce"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130318-path-normalise-baseline-r3/agent-1.jsonl", "results/raw/20261017T130318-path-normalise-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T130318-path-normalise-baseline-r3/arm-stderr.txt", "results/raw/20261017T130318-path-normalise-baseline-r3/arm-stdout.txt", "results/raw/20261017T130318-path-normalise-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T130318-path-normalise-baseline-r3/repo", "baseline": "c95fb0254bcecede9f32c6638dd4616ebf051b50", "measure_from": "c95fb0254bcecede9f32c6638dd4616ebf051b50"} +{"type": "run", "task": "path-normalise", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T13:03:44.029443+00:00", "end": "2026-10-17T13:05:45.255606+00:00", "wall_s": 121.223, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130344-path-normalise-superpowers-r1/agent-1.jsonl", "session_id": "eb948584-dee8-4304-81f2-e9a96386350a", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 52666, "outputTokens": 5627, "cacheReadTokens": 199552, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 52666, "outputTokens": 5627, "cacheReadTokens": 199552, "cacheWriteTokens": 0, "total": 257845}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.112, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 1.4597\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.274941\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.243461\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.423583\n type: 'test'\n ...\n# Subtest: pops a parent segment\nok 5 - pops a parent segment\n ---\n duration_ms: 0.20131\n type: 'test'\n ...\n# Subtest: keeps a parent segment that leaves the relative root\nok 6 - keeps a parent segment that leaves the relative root\n ---\n duration_ms: 0.117237\n type: 'test'\n ...\n# Subtest: discards parent segments above the absolute root\nok 7 - discards parent segments above the absolute root\n ---\n duration_ms: 0.210308\n type: 'test'\n ...\n# Subtest: an absolute path that walks back to the root is /\nok 8 - an absolute path that walks back to the root is /\n ---\n duration_ms: 0.137887\n type: 'test'\n ...\n# Subtest: a relative path that walks back to the start is .\nok 9 - a relative path that walks back to the start is .\n ---\n duration_ms: 0.298915\n type: 'test'\n ...\n# Subtest: drops empty segments around a parent segment\nok 10 - drops empty segments around a parent segment\n ---\n duration_ms: 0.334273\n type: 'test'\n ...\n1..10\n# tests 10\n# suites 0\n# pass 10\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 70.417815\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.093, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.684007\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.150076\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.091453\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.077137\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.230762\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 56.337429\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 24 ++++++++++++++++++++++++\n 2 files changed, 32 insertions(+)\n", "session_ids": ["eb948584-dee8-4304-81f2-e9a96386350a"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T130344-path-normalise-superpowers-r1/agent-1.jsonl", "results/raw/20261017T130344-path-normalise-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T130344-path-normalise-superpowers-r1/arm-stderr.txt", "results/raw/20261017T130344-path-normalise-superpowers-r1/arm-stdout.txt", "results/raw/20261017T130344-path-normalise-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T130344-path-normalise-superpowers-r1/repo", "baseline": "ec2c9d60b575b204ede6f27d8949b43198862050", "measure_from": "ec2c9d60b575b204ede6f27d8949b43198862050"} +{"type": "run", "task": "path-normalise", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T13:04:12.926027+00:00", "end": "2026-10-17T13:05:52.825401+00:00", "wall_s": 99.897, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130412-path-normalise-superpowers-r3/agent-1.jsonl", "session_id": "654287ec-2783-4453-b745-adc8595acd99", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 48995, "outputTokens": 4542, "cacheReadTokens": 193408, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 48995, "outputTokens": 4542, "cacheReadTokens": 193408, "cacheWriteTokens": 0, "total": 246945}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.883551\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.210628\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.131058\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 1.368464\n type: 'test'\n ...\n# Subtest: pops the previous real segment for a parent\nok 5 - pops the previous real segment for a parent\n ---\n duration_ms: 0.235378\n type: 'test'\n ...\n# Subtest: discards parent segments that would climb above the root\nok 6 - discards parent segments that would climb above the root\n ---\n duration_ms: 0.11463\n type: 'test'\n ...\n# Subtest: keeps parent segments that leave the relative root\nok 7 - keeps parent segments that leave the relative root\n ---\n duration_ms: 0.158849\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 62.375527\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.126, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.698026\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.157467\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.095313\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.082501\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.24859\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 89.459029\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 13 +++++++++++++\n 2 files changed, 21 insertions(+)\n", "session_ids": ["654287ec-2783-4453-b745-adc8595acd99"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T130412-path-normalise-superpowers-r3/agent-1.jsonl", "results/raw/20261017T130412-path-normalise-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T130412-path-normalise-superpowers-r3/arm-stderr.txt", "results/raw/20261017T130412-path-normalise-superpowers-r3/arm-stdout.txt", "results/raw/20261017T130412-path-normalise-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T130412-path-normalise-superpowers-r3/repo", "baseline": "88110063b6a520c2ea02d04e9547f1e020532958", "measure_from": "88110063b6a520c2ea02d04e9547f1e020532958"} +{"type": "run", "task": "path-normalise", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T13:03:51.110013+00:00", "end": "2026-10-17T13:06:01.019748+00:00", "wall_s": 129.907, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130351-path-normalise-superpowers-r2/agent-1.jsonl", "session_id": "7480dce2-eb30-4013-8d47-0225e65bfb5e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 96480, "outputTokens": 5687, "cacheReadTokens": 180480, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 96480, "outputTokens": 5687, "cacheReadTokens": 180480, "cacheWriteTokens": 0, "total": 282647}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.106, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.743466\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.105218\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.081997\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.437693\n type: 'test'\n ...\n# Subtest: pops the previous segment for a parent\nok 5 - pops the previous segment for a parent\n ---\n duration_ms: 0.17162\n type: 'test'\n ...\n# Subtest: keeps a parent that climbs out of a relative path\nok 6 - keeps a parent that climbs out of a relative path\n ---\n duration_ms: 0.093878\n type: 'test'\n ...\n# Subtest: discards parents that would climb above the root\nok 7 - discards parents that would climb above the root\n ---\n duration_ms: 0.128457\n type: 'test'\n ...\n# Subtest: a relative path that collapses completely is the current directory\nok 8 - a relative path that collapses completely is the current directory\n ---\n duration_ms: 0.07089\n type: 'test'\n ...\n1..8\n# tests 8\n# suites 0\n# pass 8\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.655459\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.099, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.674823\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.235019\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.161503\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.128777\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.279068\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.159991\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 17 +++++++++++++++++\n 2 files changed, 25 insertions(+)\n", "session_ids": ["7480dce2-eb30-4013-8d47-0225e65bfb5e"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T130351-path-normalise-superpowers-r2/agent-1.jsonl", "results/raw/20261017T130351-path-normalise-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T130351-path-normalise-superpowers-r2/arm-stderr.txt", "results/raw/20261017T130351-path-normalise-superpowers-r2/arm-stdout.txt", "results/raw/20261017T130351-path-normalise-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T130351-path-normalise-superpowers-r2/repo", "baseline": "e8fbe10b50e13b5a5a892371f69ebd7ed4d8464e", "measure_from": "e8fbe10b50e13b5a5a892371f69ebd7ed4d8464e"} +{"type": "run", "task": "path-normalise", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:06:01.031257+00:00", "end": "2026-10-17T13:08:29.838324+00:00", "wall_s": 148.804, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130601-path-normalise-openswarm-r3/agent-1.jsonl", "session_id": "b7c53adf-e9d7-463e-aee5-ddf730c5832f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 19242, "outputTokens": 5111, "cacheReadTokens": 67712, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "fa1e016c-d869-488f-81c0-970729e781f1", "model": "grok-4.7-high", "usage": {"inputTokens": 13836, "outputTokens": 1840, "cacheReadTokens": 60032, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 33078, "outputTokens": 6951, "cacheReadTokens": 127744, "cacheWriteTokens": 0, "total": 167773}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.101, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.635059\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.139318\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.102184\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 1.201876\n type: 'test'\n ...\n# Subtest: drops empty segments\nok 5 - drops empty segments\n ---\n duration_ms: 0.25908\n type: 'test'\n ...\n# Subtest: resolves parent segments\nok 6 - resolves parent segments\n ---\n duration_ms: 0.154795\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.743903\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.840606\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.192391\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.11104\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.116512\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 5.627921\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.118119\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 29 +++++++++++++++++++++++++++++\n 2 files changed, 37 insertions(+)\n", "session_ids": ["b7c53adf-e9d7-463e-aee5-ddf730c5832f", "fa1e016c-d869-488f-81c0-970729e781f1"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130601-path-normalise-openswarm-r3/agent-1.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T130601-path-normalise-openswarm-r3/arm-stderr.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/arm-stdout.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/executor-task.md", "results/raw/20261017T130601-path-normalise-openswarm-r3/executor-usage.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/merge.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/diff.patch", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/events.1.cursor.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/events.1.cursor.jsonl.err", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/result.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/status.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/runs/20261017-210747-cca9e6/task.md", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-result.json", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/openswarm.err", "results/raw/20261017T130601-path-normalise-openswarm-r3/planner-prompt.txt", "results/raw/20261017T130601-path-normalise-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T130601-path-normalise-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T130601-path-normalise-openswarm-r3/repo", "baseline": "23a8e5b973630a1299aba52c274685b14172a54a", "measure_from": "23a8e5b973630a1299aba52c274685b14172a54a"} +{"type": "run", "task": "path-normalise", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:05:52.837890+00:00", "end": "2026-10-17T13:09:08.942604+00:00", "wall_s": 196.102, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130552-path-normalise-openswarm-r2/agent-1.jsonl", "session_id": "e4d92edb-1fb3-49a9-bdfa-ab6ef28034cf", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 20753, "outputTokens": 6378, "cacheReadTokens": 88704, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "20bdf188-a620-4460-a1ab-ebd3f63e05ae", "model": "grok-4.7-high", "usage": {"inputTokens": 30393, "outputTokens": 3274, "cacheReadTokens": 46464, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 51146, "outputTokens": 9652, "cacheReadTokens": 135168, "cacheWriteTokens": 0, "total": 195966}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.112, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.672913\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.148195\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.084661\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 1.231509\n type: 'test'\n ...\n# Subtest: collapses parent segments on a relative path\nok 5 - collapses parent segments on a relative path\n ---\n duration_ms: 0.31609\n type: 'test'\n ...\n# Subtest: never climbs above the root on an absolute path\nok 6 - never climbs above the root on an absolute path\n ---\n duration_ms: 0.122995\n type: 'test'\n ...\n# Subtest: keeps segments that only look like dots\nok 7 - keeps segments that only look like dots\n ---\n duration_ms: 0.133956\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 63.755923\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 0.728385\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.16738\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.13056\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.106408\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.292113\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 59.268455\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js", "test/normalise.test.js"], "diff_stat": " src/normalise.js | 8 ++++++++\n test/normalise.test.js | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+)\n", "session_ids": ["e4d92edb-1fb3-49a9-bdfa-ab6ef28034cf", "20bdf188-a620-4460-a1ab-ebd3f63e05ae"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130552-path-normalise-openswarm-r2/agent-1.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T130552-path-normalise-openswarm-r2/arm-stderr.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/arm-stdout.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/executor-task.md", "results/raw/20261017T130552-path-normalise-openswarm-r2/executor-usage.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/merge.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/diff.patch", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/events.1.cursor.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/events.1.cursor.jsonl.err", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/result.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/status.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/runs/20261017-210803-eb2090/task.md", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-result.json", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/openswarm.err", "results/raw/20261017T130552-path-normalise-openswarm-r2/planner-prompt.txt", "results/raw/20261017T130552-path-normalise-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T130552-path-normalise-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T130552-path-normalise-openswarm-r2/repo", "baseline": "e094adb8ddb5c1505c27cbcd6500a4cdf183c1f2", "measure_from": "e094adb8ddb5c1505c27cbcd6500a4cdf183c1f2"} +{"type": "run", "task": "retry-backoff", "arm": "baseline", "repeat": 1, "start": "2026-10-17T13:08:29.851134+00:00", "end": "2026-10-17T13:09:16.735590+00:00", "wall_s": 46.883, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130829-retry-backoff-baseline-r1/agent-1.jsonl", "session_id": "902c95f6-3826-4208-9133-06f6e9d2e5e9", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 12618, "outputTokens": 2082, "cacheReadTokens": 72576, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 12618, "outputTokens": 2082, "cacheReadTokens": 72576, "cacheWriteTokens": 0, "total": 87276}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.058, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n", "session_ids": ["902c95f6-3826-4208-9133-06f6e9d2e5e9"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130829-retry-backoff-baseline-r1/agent-1.jsonl", "results/raw/20261017T130829-retry-backoff-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T130829-retry-backoff-baseline-r1/arm-stderr.txt", "results/raw/20261017T130829-retry-backoff-baseline-r1/arm-stdout.txt", "results/raw/20261017T130829-retry-backoff-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T130829-retry-backoff-baseline-r1/repo", "baseline": "7e0faf2d1046c32108890158738a375920c02aad", "measure_from": "7e0faf2d1046c32108890158738a375920c02aad"} +{"type": "run", "task": "retry-backoff", "arm": "baseline", "repeat": 2, "start": "2026-10-17T13:09:08.955094+00:00", "end": "2026-10-17T13:10:02.880871+00:00", "wall_s": 53.924, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130908-retry-backoff-baseline-r2/agent-1.jsonl", "session_id": "e84e7b8b-2ec3-486c-9a03-6cb85c852eb5", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 13222, "outputTokens": 2541, "cacheReadTokens": 90880, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 13222, "outputTokens": 2541, "cacheReadTokens": 90880, "cacheWriteTokens": 0, "total": 106643}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.043, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n", "session_ids": ["e84e7b8b-2ec3-486c-9a03-6cb85c852eb5"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130908-retry-backoff-baseline-r2/agent-1.jsonl", "results/raw/20261017T130908-retry-backoff-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T130908-retry-backoff-baseline-r2/arm-stderr.txt", "results/raw/20261017T130908-retry-backoff-baseline-r2/arm-stdout.txt", "results/raw/20261017T130908-retry-backoff-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T130908-retry-backoff-baseline-r2/repo", "baseline": "09e7cdbb18c0255f51dafc1aac2e0283d7babfad", "measure_from": "09e7cdbb18c0255f51dafc1aac2e0283d7babfad"} +{"type": "run", "task": "retry-backoff", "arm": "baseline", "repeat": 3, "start": "2026-10-17T13:09:16.747115+00:00", "end": "2026-10-17T13:10:11.693727+00:00", "wall_s": 54.945, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130916-retry-backoff-baseline-r3/agent-1.jsonl", "session_id": "365c1f91-65da-4d42-960e-a11154029551", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 45488, "outputTokens": 2567, "cacheReadTokens": 40064, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 45488, "outputTokens": 2567, "cacheReadTokens": 40064, "cacheWriteTokens": 0, "total": 88119}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.053, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.055, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n", "session_ids": ["365c1f91-65da-4d42-960e-a11154029551"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T130916-retry-backoff-baseline-r3/agent-1.jsonl", "results/raw/20261017T130916-retry-backoff-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T130916-retry-backoff-baseline-r3/arm-stderr.txt", "results/raw/20261017T130916-retry-backoff-baseline-r3/arm-stdout.txt", "results/raw/20261017T130916-retry-backoff-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T130916-retry-backoff-baseline-r3/repo", "baseline": "647036efbb5fbfcbc66a83d7c779b952204d6c19", "measure_from": "647036efbb5fbfcbc66a83d7c779b952204d6c19"} +{"type": "run", "task": "path-normalise", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:05:45.268716+00:00", "end": "2026-10-17T13:11:37.558757+00:00", "wall_s": 352.287, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T130545-path-normalise-openswarm-r1/agent-1.jsonl", "session_id": "6721afa7-76d6-4a2e-8d69-dbcbb0b78b98", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 38760, "outputTokens": 6885, "cacheReadTokens": 52352, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "8ffbee4d-3d82-4c42-8a4f-10ce90b159fb", "model": "grok-4.7-high", "usage": {"inputTokens": 35285, "outputTokens": 10118, "cacheReadTokens": 98944, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 74045, "outputTokens": 17003, "cacheReadTokens": 151296, "cacheWriteTokens": 0, "total": 242344}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.094, "stdout_tail": "TAP version 13\n# Subtest: drops a dot segment\nok 1 - drops a dot segment\n ---\n duration_ms: 0.742754\n type: 'test'\n ...\n# Subtest: keeps an absolute path absolute\nok 2 - keeps an absolute path absolute\n ---\n duration_ms: 0.126056\n type: 'test'\n ...\n# Subtest: a lone dot is the current directory\nok 3 - a lone dot is the current directory\n ---\n duration_ms: 0.084245\n type: 'test'\n ...\n# Subtest: rejects a non-string\nok 4 - rejects a non-string\n ---\n duration_ms: 0.322753\n type: 'test'\n ...\n1..4\n# tests 4\n# suites 0\n# pass 4\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 57.760535\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/normalise.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.111, "stdout_tail": "TAP version 13\n# Subtest: parent segments collapse inside the path\nok 1 - parent segments collapse inside the path\n ---\n duration_ms: 1.290968\n type: 'test'\n ...\n# Subtest: an absolute path does not climb above the root\nok 2 - an absolute path does not climb above the root\n ---\n duration_ms: 0.385785\n type: 'test'\n ...\n# Subtest: a relative path may keep a leading parent\nok 3 - a relative path may keep a leading parent\n ---\n duration_ms: 0.21235\n type: 'test'\n ...\n# Subtest: a trailing slash is not a segment\nok 4 - a trailing slash is not a segment\n ---\n duration_ms: 0.159762\n type: 'test'\n ...\n# Subtest: regression: a dot segment is still dropped\nok 5 - regression: a dot segment is still dropped\n ---\n duration_ms: 0.458905\n type: 'test'\n ...\n1..5\n# tests 5\n# suites 0\n# pass 5\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 74.074339\n", "stderr_tail": ""}, "files_changed": ["src/normalise.js"], "diff_stat": " src/normalise.js | 5 +++++\n 1 file changed, 5 insertions(+)\n", "session_ids": ["6721afa7-76d6-4a2e-8d69-dbcbb0b78b98", "8ffbee4d-3d82-4c42-8a4f-10ce90b159fb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T130545-path-normalise-openswarm-r1/agent-1.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T130545-path-normalise-openswarm-r1/arm-stderr.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/arm-stdout.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/executor-task.md", "results/raw/20261017T130545-path-normalise-openswarm-r1/executor-usage.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/merge.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/diff.patch", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/events.1.cursor.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/events.1.cursor.jsonl.err", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/result.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/status.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/runs/20261017-210803-0ebe25/task.md", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-result.json", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/openswarm.err", "results/raw/20261017T130545-path-normalise-openswarm-r1/planner-prompt.txt", "results/raw/20261017T130545-path-normalise-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T130545-path-normalise-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T130545-path-normalise-openswarm-r1/repo", "baseline": "e2f7bc150ebaffffc67ce265fd3046a4d6152d0a", "measure_from": "e2f7bc150ebaffffc67ce265fd3046a4d6152d0a"} +{"type": "run", "task": "retry-backoff", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T13:10:02.892209+00:00", "end": "2026-10-17T13:12:11.054351+00:00", "wall_s": 128.159, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131002-retry-backoff-superpowers-r1/agent-1.jsonl", "session_id": "34160877-8eab-4623-a624-208c8ba88a0f", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 25056, "outputTokens": 5551, "cacheReadTokens": 235264, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 25056, "outputTokens": 5551, "cacheReadTokens": 235264, "cacheWriteTokens": 0, "total": 265871}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.052, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.047, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "tests/test_retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 8 +++++++-\n tests/test_retry.py | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 44 insertions(+), 1 deletion(-)\n", "session_ids": ["34160877-8eab-4623-a624-208c8ba88a0f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T131002-retry-backoff-superpowers-r1/agent-1.jsonl", "results/raw/20261017T131002-retry-backoff-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T131002-retry-backoff-superpowers-r1/arm-stderr.txt", "results/raw/20261017T131002-retry-backoff-superpowers-r1/arm-stdout.txt", "results/raw/20261017T131002-retry-backoff-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T131002-retry-backoff-superpowers-r1/repo", "baseline": "0d7f5cd1d843c0cd8aad6958ae4f82de8cd36149", "measure_from": "0d7f5cd1d843c0cd8aad6958ae4f82de8cd36149"} +{"type": "run", "task": "retry-backoff", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:12:11.066773+00:00", "end": "2026-10-17T13:14:25.603498+00:00", "wall_s": 134.533, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131211-retry-backoff-openswarm-r1/agent-1.jsonl", "session_id": "4952ad6f-a522-4008-a4a4-a1e4780e205c", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 32734, "outputTokens": 3792, "cacheReadTokens": 51584, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "434e6fa3-df69-427c-b6be-95b1000e2551", "model": "grok-4.7-high", "usage": {"inputTokens": 40497, "outputTokens": 2370, "cacheReadTokens": 68480, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 73231, "outputTokens": 6162, "cacheReadTokens": 120064, "cacheWriteTokens": 0, "total": 199457}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py"], "diff_stat": " retry.py | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n", "session_ids": ["4952ad6f-a522-4008-a4a4-a1e4780e205c", "434e6fa3-df69-427c-b6be-95b1000e2551"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T131211-retry-backoff-openswarm-r1/agent-1.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T131211-retry-backoff-openswarm-r1/arm-stderr.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/arm-stdout.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/executor-task.md", "results/raw/20261017T131211-retry-backoff-openswarm-r1/executor-usage.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/merge.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/diff.patch", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/events.1.cursor.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/events.1.cursor.jsonl.err", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/result.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/status.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/runs/20261017-211331-4a0e55/task.md", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-result.json", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/openswarm.err", "results/raw/20261017T131211-retry-backoff-openswarm-r1/planner-prompt.txt", "results/raw/20261017T131211-retry-backoff-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T131211-retry-backoff-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T131211-retry-backoff-openswarm-r1/repo", "baseline": "a4da828d78c556da4d508792bc028c8b31af3600", "measure_from": "a4da828d78c556da4d508792bc028c8b31af3600"} +{"type": "run", "task": "retry-backoff", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T13:10:11.705775+00:00", "end": "2026-10-17T13:15:07.639154+00:00", "wall_s": 295.929, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131011-retry-backoff-superpowers-r2/agent-1.jsonl", "session_id": "bbbdaa5a-4f14-43f5-8d47-f3797d6b904c", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 79675, "outputTokens": 12690, "cacheReadTokens": 538496, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 79675, "outputTokens": 12690, "cacheReadTokens": 538496, "cacheWriteTokens": 0, "total": 630861}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.068, "stdout_tail": "", "stderr_tail": ".......\n----------------------------------------------------------------------\nRan 7 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.048, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py", "tests/test_retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 8 ++++++-\n tests/test_retry.py | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 1 deletion(-)\n", "session_ids": ["bbbdaa5a-4f14-43f5-8d47-f3797d6b904c"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T131011-retry-backoff-superpowers-r2/agent-1.jsonl", "results/raw/20261017T131011-retry-backoff-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T131011-retry-backoff-superpowers-r2/arm-stderr.txt", "results/raw/20261017T131011-retry-backoff-superpowers-r2/arm-stdout.txt", "results/raw/20261017T131011-retry-backoff-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T131011-retry-backoff-superpowers-r2/repo", "baseline": "2b5f49ca63d841c7f4f5dcc2cf157142918f3697", "measure_from": "2b5f49ca63d841c7f4f5dcc2cf157142918f3697"} +{"type": "run", "task": "retry-backoff", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:14:25.615840+00:00", "end": "2026-10-17T13:19:02.025151+00:00", "wall_s": 276.406, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131425-retry-backoff-openswarm-r2/agent-1.jsonl", "session_id": "487118ed-456f-4b8d-a692-2210a7e4a3e2", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 58326, "outputTokens": 5099, "cacheReadTokens": 132480, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "ce26d9fb-2c88-4402-b58b-535a31dc5196", "model": "grok-4.7-high", "usage": {"inputTokens": 14050, "outputTokens": 2316, "cacheReadTokens": 76928, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 72376, "outputTokens": 7415, "cacheReadTokens": 209408, "cacheWriteTokens": 0, "total": 289199}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.046, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.042, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py"], "diff_stat": " retry.py | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n", "session_ids": ["487118ed-456f-4b8d-a692-2210a7e4a3e2", "ce26d9fb-2c88-4402-b58b-535a31dc5196"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T131425-retry-backoff-openswarm-r2/agent-1.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T131425-retry-backoff-openswarm-r2/arm-stderr.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/arm-stdout.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/executor-task.md", "results/raw/20261017T131425-retry-backoff-openswarm-r2/executor-usage.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/merge.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/diff.patch", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/events.1.cursor.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/events.1.cursor.jsonl.err", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/result.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/status.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/runs/20261017-211806-cf4599/task.md", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-result.json", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/openswarm.err", "results/raw/20261017T131425-retry-backoff-openswarm-r2/planner-prompt.txt", "results/raw/20261017T131425-retry-backoff-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T131425-retry-backoff-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T131425-retry-backoff-openswarm-r2/repo", "baseline": "3a736438eeee73ba4ff7ec9e553201c66684aa11", "measure_from": "3a736438eeee73ba4ff7ec9e553201c66684aa11"} +{"type": "run", "task": "retry-backoff", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:15:07.671156+00:00", "end": "2026-10-17T13:19:13.362941+00:00", "wall_s": 245.688, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131507-retry-backoff-openswarm-r3/agent-1.jsonl", "session_id": "9276e7ec-5383-402a-b348-613697a9b500", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 37714, "outputTokens": 5807, "cacheReadTokens": 71680, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "b87be238-9abc-47de-b25d-a40a244c7e31", "model": "grok-4.7-high", "usage": {"inputTokens": 13312, "outputTokens": 1911, "cacheReadTokens": 58752, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 51026, "outputTokens": 7718, "cacheReadTokens": 130432, "cacheWriteTokens": 0, "total": 189176}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.053, "stdout_tail": "", "stderr_tail": "..\n----------------------------------------------------------------------\nRan 2 tests in 0.000s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.051, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.000s\n\nOK\n"}, "files_changed": ["retry.py"], "diff_stat": " retry.py | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n", "session_ids": ["9276e7ec-5383-402a-b348-613697a9b500", "b87be238-9abc-47de-b25d-a40a244c7e31"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T131507-retry-backoff-openswarm-r3/agent-1.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T131507-retry-backoff-openswarm-r3/arm-stderr.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/arm-stdout.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/executor-task.md", "results/raw/20261017T131507-retry-backoff-openswarm-r3/executor-usage.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/merge.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/diff.patch", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/events.1.cursor.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/events.1.cursor.jsonl.err", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/result.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/status.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/runs/20261017-211832-3ec353/task.md", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-result.json", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/openswarm.err", "results/raw/20261017T131507-retry-backoff-openswarm-r3/planner-prompt.txt", "results/raw/20261017T131507-retry-backoff-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T131507-retry-backoff-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T131507-retry-backoff-openswarm-r3/repo", "baseline": "ad8ce22fa044b800d7a509388d18faa6bac45787", "measure_from": "ad8ce22fa044b800d7a509388d18faa6bac45787"} +{"type": "run", "task": "semver-compare", "arm": "baseline", "repeat": 2, "start": "2026-10-17T13:19:13.374520+00:00", "end": "2026-10-17T13:21:25.704226+00:00", "wall_s": 132.327, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131913-semver-compare-baseline-r2/agent-1.jsonl", "session_id": "9cecc896-5fab-49d2-87a4-042e16f8ea29", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 27955, "outputTokens": 6842, "cacheReadTokens": 87040, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 27955, "outputTokens": 6842, "cacheReadTokens": 87040, "cacheWriteTokens": 0, "total": 121837}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.14, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.180928\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.238499\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 93.253633\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.129, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.162322\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.303509\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.261518\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.510675\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.142288\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.432825\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 90.397016\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 50 insertions(+), 7 deletions(-)\n", "session_ids": ["9cecc896-5fab-49d2-87a4-042e16f8ea29"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T131913-semver-compare-baseline-r2/agent-1.jsonl", "results/raw/20261017T131913-semver-compare-baseline-r2/agent-1.jsonl.err", "results/raw/20261017T131913-semver-compare-baseline-r2/arm-stderr.txt", "results/raw/20261017T131913-semver-compare-baseline-r2/arm-stdout.txt", "results/raw/20261017T131913-semver-compare-baseline-r2/tool-version.json"], "workdir": "tmp/20261017T131913-semver-compare-baseline-r2/repo", "baseline": "083546a707481b99a367c129b415654571f4f136", "measure_from": "083546a707481b99a367c129b415654571f4f136"} +{"type": "run", "task": "semver-compare", "arm": "baseline", "repeat": 1, "start": "2026-10-17T13:19:02.035865+00:00", "end": "2026-10-17T13:21:30.639241+00:00", "wall_s": 148.6, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131902-semver-compare-baseline-r1/agent-1.jsonl", "session_id": "fe6f24aa-68a7-465a-8923-bb7f9dde021b", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 34698, "outputTokens": 7482, "cacheReadTokens": 60288, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 34698, "outputTokens": 7482, "cacheReadTokens": 60288, "cacheWriteTokens": 0, "total": 102468}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.109, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.072133\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.160021\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 65.646707\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.11, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.916703\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.252004\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.1629\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.32105\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.131217\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.399782\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 67.361578\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 52 insertions(+), 7 deletions(-)\n", "session_ids": ["fe6f24aa-68a7-465a-8923-bb7f9dde021b"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T131902-semver-compare-baseline-r1/agent-1.jsonl", "results/raw/20261017T131902-semver-compare-baseline-r1/agent-1.jsonl.err", "results/raw/20261017T131902-semver-compare-baseline-r1/arm-stderr.txt", "results/raw/20261017T131902-semver-compare-baseline-r1/arm-stdout.txt", "results/raw/20261017T131902-semver-compare-baseline-r1/tool-version.json"], "workdir": "tmp/20261017T131902-semver-compare-baseline-r1/repo", "baseline": "4924ad073b14ecfe24c2c0f5f40e8873439192b5", "measure_from": "4924ad073b14ecfe24c2c0f5f40e8873439192b5"} +{"type": "run", "task": "retry-backoff", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T13:11:37.570230+00:00", "end": "2026-10-17T13:24:24.193890+00:00", "wall_s": 766.617, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T131137-retry-backoff-superpowers-r3/agent-1.jsonl", "session_id": "bf5449c2-6d0d-40ce-9750-370575c7abcb", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 117181, "outputTokens": 13851, "cacheReadTokens": 910976, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 117181, "outputTokens": 13851, "cacheReadTokens": 910976, "cacheWriteTokens": 0, "total": 1042008}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "python3 -m unittest tests/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.137, "stdout_tail": "", "stderr_tail": "........\n----------------------------------------------------------------------\nRan 8 tests in 0.001s\n\nOK\n"}, "hidden_tests": {"command": "python3 -m unittest _hidden_acceptance/test_retry.py", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.065, "stdout_tail": "", "stderr_tail": ".....\n----------------------------------------------------------------------\nRan 5 tests in 0.001s\n\nOK\n"}, "files_changed": ["retry.py", "tests/test_retry.py", "__pycache__/", "tests/__pycache__/"], "diff_stat": " retry.py | 9 +++++-\n tests/test_retry.py | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 92 insertions(+), 1 deletion(-)\n", "session_ids": ["bf5449c2-6d0d-40ce-9750-370575c7abcb"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T131137-retry-backoff-superpowers-r3/agent-1.jsonl", "results/raw/20261017T131137-retry-backoff-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T131137-retry-backoff-superpowers-r3/arm-stderr.txt", "results/raw/20261017T131137-retry-backoff-superpowers-r3/arm-stdout.txt", "results/raw/20261017T131137-retry-backoff-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T131137-retry-backoff-superpowers-r3/repo", "baseline": "a07138b67ed1b3cbf13c3cda4086dedb9612bd40", "measure_from": "a07138b67ed1b3cbf13c3cda4086dedb9612bd40"} +{"type": "run", "task": "semver-compare", "arm": "baseline", "repeat": 3, "start": "2026-10-17T13:21:25.725032+00:00", "end": "2026-10-17T13:25:47.803579+00:00", "wall_s": 262.07, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132125-semver-compare-baseline-r3/agent-1.jsonl", "session_id": "3a2bb53b-de43-4dd3-a68f-5d06d8be85c8", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 73, "outputTokens": 255, "cacheReadTokens": 24448, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 73, "outputTokens": 255, "cacheReadTokens": 24448, "cacheWriteTokens": 0, "total": 24776}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.288, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 7.773292\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.192536\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 173.138872\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.141, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.873951\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.274262\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.176701\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.226237\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.123498\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.415083\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 71.595668\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 55 insertions(+), 7 deletions(-)\n", "session_ids": ["3a2bb53b-de43-4dd3-a68f-5d06d8be85c8"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "baseline", "version": null, "commit": null}}, "raw_events": ["results/raw/20261017T132125-semver-compare-baseline-r3/agent-1.jsonl", "results/raw/20261017T132125-semver-compare-baseline-r3/agent-1.jsonl.err", "results/raw/20261017T132125-semver-compare-baseline-r3/arm-stderr.txt", "results/raw/20261017T132125-semver-compare-baseline-r3/arm-stdout.txt", "results/raw/20261017T132125-semver-compare-baseline-r3/tool-version.json"], "workdir": "tmp/20261017T132125-semver-compare-baseline-r3/repo", "baseline": "90c7ccde498551a3e5dc0631e1e4a0b0171e1fe5", "measure_from": "90c7ccde498551a3e5dc0631e1e4a0b0171e1fe5"} +{"type": "run", "task": "semver-compare", "arm": "superpowers", "repeat": 1, "start": "2026-10-17T13:21:30.660803+00:00", "end": "2026-10-17T13:26:37.407255+00:00", "wall_s": 306.739, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132130-semver-compare-superpowers-r1/agent-1.jsonl", "session_id": "3f1538c4-dfe2-48aa-b711-fd757657e3c8", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 49669, "outputTokens": 10152, "cacheReadTokens": 281600, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 49669, "outputTokens": 10152, "cacheReadTokens": 281600, "cacheWriteTokens": 0, "total": 341421}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.244, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.421009\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.218436\n type: 'test'\n ...\n# Subtest: minor and patch break ties when majors match\nok 3 - minor and patch break ties when majors match\n ---\n duration_ms: 0.133919\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same release\nok 4 - a pre-release is less than the same release\n ---\n duration_ms: 0.211008\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare by number, then text, then length\nok 5 - pre-release identifiers compare by number, then text, then length\n ---\n duration_ms: 0.299621\n type: 'test'\n ...\n# Subtest: build metadata is ignored\nok 6 - build metadata is ignored\n ---\n duration_ms: 0.112388\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 7 - invalid versions throw\n ---\n duration_ms: 0.527867\n type: 'test'\n ...\n1..7\n# tests 7\n# suites 0\n# pass 7\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 121.423075\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.136, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.15635\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.230047\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.191487\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 2.033391\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.135929\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.486811\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 79.45789\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 58 ++++++++++++++++++++++++++++++++++++++++++++++-------\n test/semver.test.js | 30 +++++++++++++++++++++++++++\n 2 files changed, 81 insertions(+), 7 deletions(-)\n", "session_ids": ["3f1538c4-dfe2-48aa-b711-fd757657e3c8"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T132130-semver-compare-superpowers-r1/agent-1.jsonl", "results/raw/20261017T132130-semver-compare-superpowers-r1/agent-1.jsonl.err", "results/raw/20261017T132130-semver-compare-superpowers-r1/arm-stderr.txt", "results/raw/20261017T132130-semver-compare-superpowers-r1/arm-stdout.txt", "results/raw/20261017T132130-semver-compare-superpowers-r1/tool-version.json"], "workdir": "tmp/20261017T132130-semver-compare-superpowers-r1/repo", "baseline": "6b5343bcb6112470fd2026be180a2f99a831fb73", "measure_from": "6b5343bcb6112470fd2026be180a2f99a831fb73"} +{"type": "run", "task": "semver-compare", "arm": "superpowers", "repeat": 2, "start": "2026-10-17T13:24:24.228135+00:00", "end": "2026-10-17T13:31:02.789230+00:00", "wall_s": 398.531, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132424-semver-compare-superpowers-r2/agent-1.jsonl", "session_id": "f0682c72-2302-412b-893c-6e90790e4380", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 46700, "outputTokens": 12009, "cacheReadTokens": 422400, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 46700, "outputTokens": 12009, "cacheReadTokens": 422400, "cacheWriteTokens": 0, "total": 481109}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.435, "stdout_tail": "e: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 3.860476\n type: 'test'\n ...\n# Subtest: a lower minor version sorts first when majors match\nok 3 - a lower minor version sorts first when majors match\n ---\n duration_ms: 0.130472\n type: 'test'\n ...\n# Subtest: a lower patch version sorts first when major and minor match\nok 4 - a lower patch version sorts first when major and minor match\n ---\n duration_ms: 2.859308\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same version without one\nok 5 - a pre-release is less than the same version without one\n ---\n duration_ms: 0.278833\n type: 'test'\n ...\n# Subtest: numeric pre-release identifiers compare as numbers\nok 6 - numeric pre-release identifiers compare as numbers\n ---\n duration_ms: 0.2135\n type: 'test'\n ...\n# Subtest: non-numeric pre-release identifiers compare as strings\nok 7 - non-numeric pre-release identifiers compare as strings\n ---\n duration_ms: 0.094359\n type: 'test'\n ...\n# Subtest: a numeric pre-release identifier is less than a non-numeric one\nok 8 - a numeric pre-release identifier is less than a non-numeric one\n ---\n duration_ms: 0.080361\n type: 'test'\n ...\n# Subtest: a shorter pre-release that prefixes the other sorts first\nok 9 - a shorter pre-release that prefixes the other sorts first\n ---\n duration_ms: 0.281596\n type: 'test'\n ...\n# Subtest: equal pre-releases compare equal\nok 10 - equal pre-releases compare equal\n ---\n duration_ms: 2.235816\n type: 'test'\n ...\n# Subtest: build metadata after + is ignored\nok 11 - build metadata after + is ignored\n ---\n duration_ms: 0.25845\n type: 'test'\n ...\n# Subtest: invalid versions throw an error that mentions invalid semver\nok 12 - invalid versions throw an error that mentions invalid semver\n ---\n duration_ms: 0.460405\n type: 'test'\n ...\n1..12\n# tests 12\n# suites 0\n# pass 12\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 265.262497\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.309, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.049101\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.274383\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.149147\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 2.08405\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.10958\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 1.13052\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 225.370062\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 61 ++++++++++++++++++++++++++++++++++++++++++++++-------\n test/semver.test.js | 54 +++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 107 insertions(+), 8 deletions(-)\n", "session_ids": ["f0682c72-2302-412b-893c-6e90790e4380"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T132424-semver-compare-superpowers-r2/agent-1.jsonl", "results/raw/20261017T132424-semver-compare-superpowers-r2/agent-1.jsonl.err", "results/raw/20261017T132424-semver-compare-superpowers-r2/arm-stderr.txt", "results/raw/20261017T132424-semver-compare-superpowers-r2/arm-stdout.txt", "results/raw/20261017T132424-semver-compare-superpowers-r2/tool-version.json"], "workdir": "tmp/20261017T132424-semver-compare-superpowers-r2/repo", "baseline": "77502b42fb3fe250194fd6615e3949c9b33ff7b6", "measure_from": "77502b42fb3fe250194fd6615e3949c9b33ff7b6"} +{"type": "run", "task": "semver-compare", "arm": "openswarm", "repeat": 1, "start": "2026-10-17T13:26:37.431600+00:00", "end": "2026-10-17T13:32:46.201009+00:00", "wall_s": 368.762, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132637-semver-compare-openswarm-r1/agent-1.jsonl", "session_id": "b7ed2832-6c93-4b21-ad8b-52108ffb93dd", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 24043, "outputTokens": 9699, "cacheReadTokens": 66176, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "056b0858-fe9d-406a-b521-7870330ae151", "model": "grok-4.7-high", "usage": {"inputTokens": 21302, "outputTokens": 7723, "cacheReadTokens": 70016, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 45345, "outputTokens": 17422, "cacheReadTokens": 136192, "cacheWriteTokens": 0, "total": 198959}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.767, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 2.71663\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.291564\n type: 'test'\n ...\n# Subtest: minor and patch are compared as integers\nok 3 - minor and patch are compared as integers\n ---\n duration_ms: 0.24867\n type: 'test'\n ...\n# Subtest: pre-release precedence\nok 4 - pre-release precedence\n ---\n duration_ms: 2.032619\n type: 'test'\n ...\n# Subtest: build metadata does not affect precedence\nok 5 - build metadata does not affect precedence\n ---\n duration_ms: 0.268782\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.726424\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 556.805669\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.192, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.219421\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.278329\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.132749\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.14618\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.921607\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.396336\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 97.138598\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 70 +++++++++++++++++++++++++++++++++++++++++++++------\n test/semver.test.js | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 134 insertions(+), 8 deletions(-)\n", "session_ids": ["b7ed2832-6c93-4b21-ad8b-52108ffb93dd", "056b0858-fe9d-406a-b521-7870330ae151"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T132637-semver-compare-openswarm-r1/agent-1.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/agent-1.jsonl.err", "results/raw/20261017T132637-semver-compare-openswarm-r1/arm-stderr.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/arm-stdout.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/executor-task.md", "results/raw/20261017T132637-semver-compare-openswarm-r1/executor-usage.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/merge.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/ledger.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/diff.patch", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/events.1.cursor.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/events.1.cursor.jsonl.err", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/result.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/status.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/runs/20261017-213002-534e4b/task.md", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-home/state.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-result.json", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm-stdout.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/openswarm.err", "results/raw/20261017T132637-semver-compare-openswarm-r1/planner-prompt.txt", "results/raw/20261017T132637-semver-compare-openswarm-r1/raw-executor-1.jsonl", "results/raw/20261017T132637-semver-compare-openswarm-r1/tool-version.json"], "workdir": "tmp/20261017T132637-semver-compare-openswarm-r1/repo", "baseline": "d29ca47a00addf6d431fa9a353f398d1b352bab8", "measure_from": "d29ca47a00addf6d431fa9a353f398d1b352bab8"} +{"type": "run", "task": "semver-compare", "arm": "superpowers", "repeat": 3, "start": "2026-10-17T13:25:47.848796+00:00", "end": "2026-10-17T13:35:22.029662+00:00", "wall_s": 574.176, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T132547-semver-compare-superpowers-r3/agent-1.jsonl", "session_id": "94c4f8cc-0fcf-4075-93ce-91d9868ba14e", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 1867, "outputTokens": 547, "cacheReadTokens": 104960, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 1867, "outputTokens": 547, "cacheReadTokens": 104960, "cacheWriteTokens": 0, "total": 107374}, "cursor_agent_calls": 1, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.163, "stdout_tail": "patch version sorts first when major and minor match\nok 4 - a lower patch version sorts first when major and minor match\n ---\n duration_ms: 0.166503\n type: 'test'\n ...\n# Subtest: minor differences outrank patch differences\nok 5 - minor differences outrank patch differences\n ---\n duration_ms: 0.100516\n type: 'test'\n ...\n# Subtest: a pre-release is less than the same core version\nok 6 - a pre-release is less than the same core version\n ---\n duration_ms: 0.199919\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically when both are numeric\nok 7 - pre-release identifiers compare numerically when both are numeric\n ---\n duration_ms: 0.225502\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare as strings otherwise\nok 8 - pre-release identifiers compare as strings otherwise\n ---\n duration_ms: 0.087779\n type: 'test'\n ...\n# Subtest: a numeric pre-release identifier is less than a non-numeric one\nok 9 - a numeric pre-release identifier is less than a non-numeric one\n ---\n duration_ms: 0.266943\n type: 'test'\n ...\n# Subtest: a shorter pre-release list is less when it is a prefix\nok 10 - a shorter pre-release list is less when it is a prefix\n ---\n duration_ms: 0.33882\n type: 'test'\n ...\n# Subtest: equal pre-release versions compare equal\nok 11 - equal pre-release versions compare equal\n ---\n duration_ms: 0.135862\n type: 'test'\n ...\n# Subtest: build metadata is ignored\nok 12 - build metadata is ignored\n ---\n duration_ms: 0.082799\n type: 'test'\n ...\n# Subtest: leading zeroes in a numeric core component are invalid\nok 13 - leading zeroes in a numeric core component are invalid\n ---\n duration_ms: 0.453263\n type: 'test'\n ...\n# Subtest: an invalid version throws an error mentioning invalid semver\nok 14 - an invalid version throws an error mentioning invalid semver\n ---\n duration_ms: 0.144849\n type: 'test'\n ...\n1..14\n# tests 14\n# suites 0\n# pass 14\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 80.146917\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.115, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 1.454457\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.441119\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.324973\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 1.411717\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.154834\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.562751\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 82.680911\n", "stderr_tail": ""}, "files_changed": ["src/semver.js", "test/semver.test.js"], "diff_stat": " src/semver.js | 58 ++++++++++++++++++++++++++++++++++++++++++------\n test/semver.test.js | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 115 insertions(+), 7 deletions(-)\n", "session_ids": ["94c4f8cc-0fcf-4075-93ce-91d9868ba14e"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "superpowers", "version": "v7.0.0", "commit": "bb92a77741419a4ab5f06e711a283343f1ada0c3"}}, "raw_events": ["results/raw/20261017T132547-semver-compare-superpowers-r3/agent-1.jsonl", "results/raw/20261017T132547-semver-compare-superpowers-r3/agent-1.jsonl.err", "results/raw/20261017T132547-semver-compare-superpowers-r3/arm-stderr.txt", "results/raw/20261017T132547-semver-compare-superpowers-r3/arm-stdout.txt", "results/raw/20261017T132547-semver-compare-superpowers-r3/tool-version.json"], "workdir": "tmp/20261017T132547-semver-compare-superpowers-r3/repo", "baseline": "b1593da14facf805844084997c518e78dc3adf8c", "measure_from": "b1593da14facf805844084997c518e78dc3adf8c"} +{"type": "run", "task": "semver-compare", "arm": "openswarm", "repeat": 2, "start": "2026-10-17T13:31:02.834416+00:00", "end": "2026-10-17T13:37:15.693410+00:00", "wall_s": 372.855, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133102-semver-compare-openswarm-r2/agent-1.jsonl", "session_id": "6f9a69ad-1223-46cc-a18a-349e2a0b3166", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 23761, "outputTokens": 9087, "cacheReadTokens": 120832, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "33a88cf2-a6f0-498d-8b8d-778b5c18396f", "model": "grok-4.7-high", "usage": {"inputTokens": 36171, "outputTokens": 7292, "cacheReadTokens": 50816, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 59932, "outputTokens": 16379, "cacheReadTokens": 171648, "cacheWriteTokens": 0, "total": 247959}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.105, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 0.946837\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.170946\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 69.975914\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.096, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.748155\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.285442\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.119097\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 0.958552\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.143623\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.539929\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.85829\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 75 insertions(+), 9 deletions(-)\n", "session_ids": ["6f9a69ad-1223-46cc-a18a-349e2a0b3166", "33a88cf2-a6f0-498d-8b8d-778b5c18396f"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T133102-semver-compare-openswarm-r2/agent-1.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/agent-1.jsonl.err", "results/raw/20261017T133102-semver-compare-openswarm-r2/arm-stderr.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/arm-stdout.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/executor-task.md", "results/raw/20261017T133102-semver-compare-openswarm-r2/executor-usage.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/merge.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/ledger.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/diff.patch", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/events.1.cursor.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/events.1.cursor.jsonl.err", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/result.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/status.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/runs/20261017-213513-448d5c/task.md", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-home/state.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-result.json", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm-stdout.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/openswarm.err", "results/raw/20261017T133102-semver-compare-openswarm-r2/planner-prompt.txt", "results/raw/20261017T133102-semver-compare-openswarm-r2/raw-executor-1.jsonl", "results/raw/20261017T133102-semver-compare-openswarm-r2/tool-version.json"], "workdir": "tmp/20261017T133102-semver-compare-openswarm-r2/repo", "baseline": "1a84b20b4ba4a11204d656448ebad355f91c132d", "measure_from": "1a84b20b4ba4a11204d656448ebad355f91c132d"} +{"type": "run", "task": "semver-compare", "arm": "openswarm", "repeat": 3, "start": "2026-10-17T13:32:46.276821+00:00", "end": "2026-10-17T13:39:57.885122+00:00", "wall_s": 431.605, "exit_code": 0, "timed_out": false, "skipped": false, "tokens": {"calls": [{"source": "results/raw/20261017T133246-semver-compare-openswarm-r3/agent-1.jsonl", "session_id": "f4969927-63cd-4400-bc29-ccef175aa9ed", "model": "Grok 4.7 256K High", "usage": {"inputTokens": 31939, "outputTokens": 10596, "cacheReadTokens": 46848, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}, {"source": "openswarm-result:attempts.meta.usage", "session_id": "e8e98887-e9e4-4231-a9fd-6b553a584159", "model": "grok-4.7-high", "usage": {"inputTokens": 68019, "outputTokens": 6030, "cacheReadTokens": 18816, "cacheWriteTokens": 0}, "missing_usage": false, "saw_result": true}], "inputTokens": 99958, "outputTokens": 16626, "cacheReadTokens": 65664, "cacheWriteTokens": 0, "total": 182248}, "cursor_agent_calls": 2, "success": true, "visible_tests": {"command": "node --test test/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.12, "stdout_tail": "TAP version 13\n# Subtest: equal versions compare equal\nok 1 - equal versions compare equal\n ---\n duration_ms: 1.069332\n type: 'test'\n ...\n# Subtest: a lower major version sorts first\nok 2 - a lower major version sorts first\n ---\n duration_ms: 0.177606\n type: 'test'\n ...\n1..2\n# tests 2\n# suites 0\n# pass 2\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 83.632953\n", "stderr_tail": ""}, "hidden_tests": {"command": "node --test _hidden_acceptance/semver.test.js", "exit_code": 0, "passed": true, "timed_out": false, "wall_s": 0.1, "stdout_tail": "TAP version 13\n# Subtest: minor and patch break ties\nok 1 - minor and patch break ties\n ---\n duration_ms: 0.833265\n type: 'test'\n ...\n# Subtest: a pre-release is less than the matching release\nok 2 - a pre-release is less than the matching release\n ---\n duration_ms: 0.213051\n type: 'test'\n ...\n# Subtest: pre-release identifiers compare numerically, then lexically\nok 3 - pre-release identifiers compare numerically, then lexically\n ---\n duration_ms: 0.225387\n type: 'test'\n ...\n# Subtest: build metadata does not affect order\nok 4 - build metadata does not affect order\n ---\n duration_ms: 0.995114\n type: 'test'\n ...\n# Subtest: regression: major ordering still holds\nok 5 - regression: major ordering still holds\n ---\n duration_ms: 0.147209\n type: 'test'\n ...\n# Subtest: invalid versions throw\nok 6 - invalid versions throw\n ---\n duration_ms: 0.393059\n type: 'test'\n ...\n1..6\n# tests 6\n# suites 0\n# pass 6\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 61.456276\n", "stderr_tail": ""}, "files_changed": ["src/semver.js"], "diff_stat": " src/semver.js | 42 +++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 39 insertions(+), 3 deletions(-)\n", "session_ids": ["f4969927-63cd-4400-bc29-ccef175aa9ed", "e8e98887-e9e4-4231-a9fd-6b553a584159"], "model": "grok-4.7-high", "model_reported": "Grok 4.7 256K High", "versions": {"arm": {"name": "openswarm", "version": null, "commit": "a48255a9cebafc48b0a7cdd39b6590a7dc193496", "dir": ""}}, "raw_events": ["results/raw/20261017T133246-semver-compare-openswarm-r3/agent-1.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/agent-1.jsonl.err", "results/raw/20261017T133246-semver-compare-openswarm-r3/arm-stderr.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/arm-stdout.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/executor-task.md", "results/raw/20261017T133246-semver-compare-openswarm-r3/executor-usage.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/merge.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/ledger.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/diff.patch", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/events.1.cursor.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/events.1.cursor.jsonl.err", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/result.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/status.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/runs/20261017-213816-170f35/task.md", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-home/state.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-result.json", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm-stdout.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/openswarm.err", "results/raw/20261017T133246-semver-compare-openswarm-r3/planner-prompt.txt", "results/raw/20261017T133246-semver-compare-openswarm-r3/raw-executor-1.jsonl", "results/raw/20261017T133246-semver-compare-openswarm-r3/tool-version.json"], "workdir": "tmp/20261017T133246-semver-compare-openswarm-r3/repo", "baseline": "7205d2102be360bf05fac530b6a62d99806ac25a", "measure_from": "7205d2102be360bf05fac530b6a62d99806ac25a"} diff --git a/bench/e2e/results/headline.json b/bench/e2e/results/headline.json new file mode 100644 index 0000000..66e7f13 --- /dev/null +++ b/bench/e2e/results/headline.json @@ -0,0 +1,198 @@ +{ + "all_runs": { + "baseline": { + "n": 24, + "success": 24, + "success_pct": 100.0, + "timed_out": 0, + "wall_s": { + "median": 54.7, + "q1": 45.7, + "q3": 132.4, + "min": 24.94, + "max": 262.07 + }, + "tokens_n": 24, + "total_tokens": { + "median": 104700.0, + "q1": 87344.2, + "q3": 112659.2 + }, + "in_out_tokens": { + "median": 27542.5, + "q1": 15497.2, + "q3": 39575.2 + } + }, + "openflow": { + "n": 12, + "success": 11, + "success_pct": 91.7, + "timed_out": 3, + "wall_s": { + "median": 985.3, + "q1": 919.5, + "q3": 1195.0, + "min": 774.225, + "max": 1200.983 + }, + "tokens_n": 8, + "total_tokens": { + "median": 5294856.0, + "q1": 5069621.8, + "q3": 5414698.0 + }, + "in_out_tokens": { + "median": 233459.5, + "q1": 189383.8, + "q3": 241830.0 + } + }, + "openswarm": { + "n": 24, + "success": 24, + "success_pct": 100.0, + "timed_out": 0, + "wall_s": { + "median": 219.7, + "q1": 136.6, + "q3": 356.4, + "min": 89.133, + "max": 687.103 + }, + "tokens_n": 24, + "total_tokens": { + "median": 207194.0, + "q1": 191056.2, + "q3": 245895.0 + }, + "in_out_tokens": { + "median": 63514.0, + "q1": 51123.2, + "q3": 82605.2 + } + }, + "superpowers": { + "n": 24, + "success": 24, + "success_pct": 100.0, + "timed_out": 0, + "wall_s": { + "median": 186.9, + "q1": 120.7, + "q3": 298.6, + "min": 89.908, + "max": 766.617 + }, + "tokens_n": 24, + "total_tokens": { + "median": 311921.5, + "q1": 267209.0, + "q3": 436317.5 + }, + "in_out_tokens": { + "median": 59265.0, + "q1": 47635.0, + "q3": 74033.0 + } + } + }, + "matched_repeat1": { + "baseline": { + "n": 8, + "success": 8, + "success_pct": 100.0, + "timed_out": 0, + "wall_s": { + "median": 56.7, + "q1": 42.9, + "q3": 136.6, + "min": 24.94, + "max": 205.884 + }, + "tokens_n": 8, + "total_tokens": { + "median": 104084.5, + "q1": 86839.8, + "q3": 115727.5 + }, + "in_out_tokens": { + "median": 24706.5, + "q1": 14391.8, + "q3": 44051.8 + } + }, + "openflow": { + "n": 8, + "success": 8, + "success_pct": 100.0, + "timed_out": 2, + "wall_s": { + "median": 1007.7, + "q1": 888.8, + "q3": 1195.0, + "min": 774.225, + "max": 1200.983 + }, + "tokens_n": 6, + "total_tokens": { + "median": 5324282.5, + "q1": 4883528.0, + "q3": 5443668.0 + }, + "in_out_tokens": { + "median": 214694.0, + "q1": 187693.2, + "q3": 240540.5 + } + }, + "openswarm": { + "n": 8, + "success": 8, + "success_pct": 100.0, + "timed_out": 0, + "wall_s": { + "median": 218.2, + "q1": 136.6, + "q3": 356.4, + "min": 89.133, + "max": 687.103 + }, + "tokens_n": 8, + "total_tokens": { + "median": 202918.0, + "q1": 198866.8, + "q3": 243059.8 + }, + "in_out_tokens": { + "median": 74598.0, + "q1": 62482.8, + "q3": 92787.8 + } + }, + "superpowers": { + "n": 8, + "success": 8, + "success_pct": 100.0, + "timed_out": 0, + "wall_s": { + "median": 154.4, + "q1": 120.7, + "q3": 260.8, + "min": 113.45, + "max": 314.835 + }, + "tokens_n": 8, + "total_tokens": { + "median": 288705.0, + "q1": 272464.2, + "q3": 341738.8 + }, + "in_out_tokens": { + "median": 59057.0, + "q1": 55287.5, + "q3": 66932.5 + } + } + } +} diff --git a/bench/e2e/results/scores.csv b/bench/e2e/results/scores.csv new file mode 100644 index 0000000..b0741aa --- /dev/null +++ b/bench/e2e/results/scores.csv @@ -0,0 +1,45 @@ +scope,id,n,success,success_n,success_pct,wall_median_s,wall_q1,wall_q3,wall_iqr,wall_min,wall_max,input_median,input_q1,input_q3,input_iqr,output_median,output_q1,output_q3,output_iqr,cache_read_median,cache_read_q1,cache_read_q3,cache_read_iqr,total_median,total_q1,total_q3,total_iqr +arm,baseline,24,24,24,100.0000,54.685,45.701499999999996,132.3795,86.67800000000001,24.94,262.07,25494.5,13071.0,35143.0,22072.0,2550.5,1892.75,5342.5,3449.75,73280.0,68736.0,82400.0,13664.0,104700.0,87344.25,112659.25,25315.0 +arm,superpowers,24,24,24,100.0000,186.9115,120.66799999999999,298.63149999999996,177.96349999999995,89.908,766.617,50893.0,39967.0,67111.75,27144.75,7935.0,5347.25,11699.25,6352.0,249984.0,209184.0,387328.0,178144.0,311921.5,267209.0,436317.5,169108.5 +arm,openflow,12,11,12,91.6667,985.308,919.4955,1194.96175,275.46624999999995,774.225,1200.983,187857.0,147069.0,203893.0,56824.0,37704.0,33430.0,40424.0,6994.0,5061504.0,4465792.0,5209984.0,744192.0,5248352.0,4761920.0,5400213.0,638293.0 +arm,openswarm,24,24,24,100.0000,219.72899999999998,136.594,356.40574999999995,219.81174999999996,89.133,687.103,52562.0,43475.0,73434.5,29959.5,9954.0,6139.25,16720.25,10581.0,146880.0,135264.0,179392.0,44128.0,207194.0,191056.25,245895.0,54838.75 +task,config-deep-merge,12,12,12,100.0000,200.5925,151.728,438.16125,286.43325,73.715,947.74,51819.5,42347.5,88598.25,46250.75,10025.5,7171.75,21503.0,14331.25,209088.0,125312.0,1410976.0,1285664.0,269892.0,179859.0,1534967.75,1355108.75 +task,csv-quoting,12,11,12,91.6667,377.2555,230.22549999999998,771.0462499999999,540.8207499999999,152.326,1200.983,53546.5,39919.25,106502.25,66583.0,13525.5,10412.75,18873.25,8460.5,227456.0,81312.0,267648.0,186336.0,325376.0,131644.0,369739.75,238095.75 +task,date-range-parse,10,10,10,100.0000,117.892,54.09475,134.74175,80.64699999999999,30.971,1193.18,27752.5,24549.5,42686.0,18136.5,5513.0,2445.5,6031.5,3586.0,163648.0,92768.0,208448.0,115680.0,217685.5,125366.25,265706.0,140339.75 +task,lru-eviction,10,10,10,100.0000,215.5115,87.54650000000001,223.0565,135.51,49.685,1200.307,37865.0,32196.0,72646.0,40450.0,9939.0,2229.0,10283.0,8054.0,175616.0,85376.0,322176.0,236800.0,245207.0,124083.0,426225.0,302142.0 +task,pagination-off-by-one,10,10,10,100.0000,101.3895,49.1205,109.8865,60.766,24.94,1074.334,31307.5,16626.25,46600.0,29973.75,4380.0,2083.0,5010.75,2927.75,158848.0,90048.0,212160.0,122112.0,194792.5,108757.25,263385.25,154628.0 +task,path-normalise,10,10,10,100.0000,125.565,72.717,184.2775,111.5605,54.425,833.929,51630.5,37057.25,68700.25,31643.0,5657.0,3431.25,8976.75,5545.5,143232.0,101056.0,190176.0,89120.0,219155.0,124143.25,255120.0,130976.75 +task,retry-backoff,10,10,10,100.0000,190.1105,73.2485,291.04825,217.79975,46.883,907.026,61701.0,30164.0,78064.0,47900.0,6788.5,3313.0,11447.0,8134.0,169920.0,98176.0,462688.0,364512.0,232664.0,127276.25,545445.5,418169.25 +task,semver-compare,10,10,10,100.0000,370.8085,273.23725,423.3365,150.09924999999998,132.327,941.051,46022.5,29640.75,57366.25,27725.5,11080.5,7002.0,16564.25,9562.25,120576.0,71008.0,254112.0,183104.0,190603.5,110989.75,318055.5,207065.75 +task_arm,config-deep-merge/baseline,3,3,3,100.0000,95.508,84.6115,114.02250000000001,29.411,73.715,132.537,30533.0,29641.0,36774.0,7133.0,4906.0,4281.5,5779.0,1497.5,94720.0,77248.0,96960.0,19712.0,134639.0,113410.5,139513.0,26102.5 +task_arm,config-deep-merge/openflow,3,3,3,100.0000,923.652,848.9385,935.696,86.75750000000005,774.225,947.74,203893.0,195875.0,232678.5,36803.5,39903.0,37283.5,40163.5,2880.0,4927872.0,4696832.0,5020736.0,323904.0,5172189.0,4967054.5,5256774.5,289720.0 +task_arm,config-deep-merge/openswarm,3,3,3,100.0000,220.486,197.9355,273.313,75.3775,175.385,326.14,50239.0,49623.0,52875.5,3252.5,11391.0,10025.5,14253.5,4228.0,136960.0,135488.0,154496.0,19008.0,198590.0,195136.5,221625.0,26488.5 +task_arm,config-deep-merge/superpowers,3,3,3,100.0000,180.699,169.412,221.997,52.58500000000001,158.125,263.295,53400.0,46872.5,54007.0,7134.5,8635.0,7990.0,10924.0,2934.0,253824.0,249984.0,323264.0,73280.0,315783.0,305453.5,387550.0,82096.5 +task_arm,csv-quoting/baseline,3,3,3,100.0000,205.884,179.105,206.777,27.671999999999997,152.326,207.67,39165.0,28895.5,40673.5,11778.0,10377.0,9231.0,10448.5,1217.5,81280.0,79616.0,81344.0,1728.0,130822.0,117742.5,132466.0,14723.5 +task_arm,csv-quoting/openflow,3,2,3,66.6667,1200.393,1111.6345000000001,1200.688,89.05349999999999,1022.876,1200.983,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +task_arm,csv-quoting/openswarm,3,3,3,100.0000,434.924,408.8215,561.0135,152.192,382.719,687.103,116927.0,115669.5,132399.0,16729.5,21916.0,20455.0,27790.0,7335.0,237056.0,227456.0,287680.0,60224.0,375899.0,363580.5,447869.0,84288.5 +task_arm,csv-quoting/superpowers,3,3,3,100.0000,314.835,276.2895,343.3135,67.024,237.744,371.792,57397.0,53546.5,70085.0,16538.5,16004.0,13525.5,17257.5,3732.0,276992.0,258304.0,364288.0,105984.0,342692.0,325376.0,447780.0,122404.0 +task_arm,date-range-parse/baseline,3,3,3,100.0000,41.033,36.002,41.595,5.592999999999996,30.971,42.157,24409.0,18280.5,24899.5,6619.0,1736.0,1525.5,1840.5,315.0,72064.0,66048.0,75328.0,9280.0,87367.0,86449.0,96052.0,9603.0 +task_arm,date-range-parse/openflow,1,1,1,100.0000,1193.18,1193.18,1193.18,0.0,1193.18,1193.18,19475.0,19475.0,19475.0,0.0,10881.0,10881.0,10881.0,0.0,1932928.0,1932928.0,1932928.0,0.0,1963284.0,1963284.0,1963284.0,0.0 +task_arm,date-range-parse/openswarm,3,3,3,100.0000,127.124,124.729,132.2025,7.473499999999987,122.334,137.281,46045.0,39327.0,54888.5,15561.5,5913.0,5788.0,5992.0,204.0,136576.0,135936.0,163648.0,27712.0,206379.0,196816.5,217685.5,20869.0 +task_arm,date-range-parse/superpowers,3,3,3,100.0000,113.45,101.679,138.0,36.321,89.908,162.55,30115.0,27543.0,47691.0,20148.0,5363.0,4655.0,6370.0,1715.0,209920.0,206976.0,263104.0,56128.0,274662.0,256750.0,314221.0,57471.0 +task_arm,lru-eviction/baseline,3,3,3,100.0000,51.185,50.435,51.7695,1.3344999999999985,49.685,52.354,29867.0,27733.0,33172.5,5439.5,2151.0,2128.5,2190.0,61.5,78336.0,76032.0,81856.0,5824.0,106086.0,105893.5,115084.5,9191.0 +task_arm,lru-eviction/openflow,1,1,1,100.0000,1200.307,1200.307,1200.307,0.0,1200.307,1200.307,,,,,,,,,,,,,,,, +task_arm,lru-eviction/openswarm,3,3,3,100.0000,218.972,217.414,221.695,4.281000000000006,215.856,224.418,53978.0,45921.5,70864.5,24943.0,10283.0,10269.5,10518.5,249.0,175616.0,161408.0,188224.0,26816.0,245207.0,234721.0,255150.0,20429.0 +task_arm,lru-eviction/superpowers,3,3,3,100.0000,215.167,204.1455,230.36,26.214500000000015,193.124,245.553,72646.0,52421.0,84591.0,32170.0,9939.0,9216.0,10767.5,1551.5,385536.0,353856.0,417664.0,63808.0,428651.0,427438.0,481342.5,53904.5 +task_arm,pagination-off-by-one/baseline,3,3,3,100.0000,31.906,28.423000000000002,33.8445,5.421499999999995,24.94,35.783,11821.0,11811.5,12014.5,203.0,1298.0,1094.0,1411.5,317.5,72192.0,71872.0,72320.0,448.0,85292.0,84777.5,85736.5,959.0 +task_arm,pagination-off-by-one/openflow,1,1,1,100.0000,1074.334,1074.334,1074.334,0.0,1074.334,1074.334,207571.0,207571.0,207571.0,0.0,33430.0,33430.0,33430.0,0.0,5217152.0,5217152.0,5217152.0,0.0,5458153.0,5458153.0,5458153.0,0.0 +task_arm,pagination-off-by-one/openswarm,3,3,3,100.0000,93.433,91.283,101.6645,10.381500000000003,89.133,109.896,30834.0,30357.5,31307.5,950.0,4182.0,3969.5,4637.0,667.5,146560.0,144704.0,158848.0,14144.0,181576.0,179031.0,194792.5,15761.5 +task_arm,pagination-off-by-one/superpowers,3,3,3,100.0000,109.858,109.602,114.4305,4.828499999999991,109.346,119.003,49189.0,44011.0,50639.5,6628.5,4767.0,4672.5,5033.5,361.0,213888.0,210432.0,219392.0,8960.0,267655.0,259115.5,274970.5,15855.0 +task_arm,path-normalise/baseline,3,3,3,100.0000,60.555,57.489999999999995,62.105999999999995,4.616,54.425,63.657,14379.0,13938.5,33247.0,19308.5,2828.0,2694.0,2944.5,250.5,72832.0,63488.0,82496.0,19008.0,109087.0,98988.5,109343.5,10355.0 +task_arm,path-normalise/openflow,1,1,1,100.0000,833.929,833.929,833.929,0.0,833.929,833.929,149144.0,149144.0,149144.0,0.0,37704.0,37704.0,37704.0,0.0,5061504.0,5061504.0,5061504.0,0.0,5248352.0,5248352.0,5248352.0,0.0 +task_arm,path-normalise/openswarm,3,3,3,100.0000,196.102,172.453,274.1945,101.7415,148.804,352.287,51146.0,42112.0,62595.5,20483.5,9652.0,8301.5,13327.5,5026.0,135168.0,131456.0,143232.0,11776.0,195966.0,181869.5,219155.0,37285.5 +task_arm,path-normalise/superpowers,3,3,3,100.0000,121.223,110.56,125.565,15.004999999999995,99.897,129.907,52666.0,50830.5,74573.0,23742.5,5627.0,5084.5,5657.0,572.5,193408.0,186944.0,196480.0,9536.0,257845.0,252395.0,270246.0,17851.0 +task_arm,retry-backoff/baseline,3,3,3,100.0000,53.924,50.4035,54.4345,4.030999999999999,46.883,54.945,13222.0,12920.0,29355.0,16435.0,2541.0,2311.5,2554.0,242.5,72576.0,56320.0,81728.0,25408.0,88119.0,87697.5,97381.0,9683.5 +task_arm,retry-backoff/openflow,1,1,1,100.0000,907.026,907.026,907.026,0.0,907.026,907.026,198345.0,198345.0,198345.0,0.0,40814.0,40814.0,40814.0,0.0,5839744.0,5839744.0,5839744.0,0.0,6078903.0,6078903.0,6078903.0,0.0 +task_arm,retry-backoff/openswarm,3,3,3,100.0000,245.688,190.1105,261.047,70.93650000000002,134.533,276.406,72376.0,61701.0,72803.5,11102.5,7415.0,6788.5,7566.5,778.0,130432.0,125248.0,169920.0,44672.0,199457.0,194316.5,244328.0,50011.5 +task_arm,retry-backoff/superpowers,3,3,3,100.0000,295.929,212.04399999999998,531.2729999999999,319.2289999999999,128.159,766.617,79675.0,52365.5,98428.0,46062.5,12690.0,9120.5,13270.5,4150.0,538496.0,386880.0,724736.0,337856.0,630861.0,448366.0,836434.5,388068.5 +task_arm,semver-compare/baseline,3,3,3,100.0000,148.6,140.4635,205.33499999999998,64.87149999999997,132.327,262.07,27955.0,14014.0,31326.5,17312.5,6842.0,3548.5,7162.0,3613.5,60288.0,42368.0,73664.0,31296.0,102468.0,63622.0,112152.5,48530.5 +task_arm,semver-compare/openflow,1,1,1,100.0000,941.051,941.051,941.051,0.0,941.051,941.051,147069.0,147069.0,147069.0,0.0,43160.0,43160.0,43160.0,0.0,5209984.0,5209984.0,5209984.0,0.0,5400213.0,5400213.0,5400213.0,0.0 +task_arm,semver-compare/openswarm,3,3,3,100.0000,372.855,370.8085,402.23,31.421500000000037,368.762,431.605,59932.0,52638.5,79945.0,27306.5,16626.0,16502.5,17024.0,521.5,136192.0,100928.0,153920.0,52992.0,198959.0,190603.5,223459.0,32855.5 +task_arm,semver-compare/superpowers,3,3,3,100.0000,398.531,352.635,486.35350000000005,133.71850000000006,306.739,574.176,46700.0,24283.5,48184.5,23901.0,10152.0,5349.5,11080.5,5731.0,281600.0,193280.0,352000.0,158720.0,341421.0,224397.5,411265.0,186867.5 diff --git a/bench/e2e/results/summary.md b/bench/e2e/results/summary.md new file mode 100644 index 0000000..d6a6ca8 --- /dev/null +++ b/bench/e2e/results/summary.md @@ -0,0 +1,79 @@ +# End-to-end task completion + +## Environment + +- Date: 2026-10-17T12:20:16.509195+00:00 +- OS: Linux-6.12.94+-x86_64-with-glibc2.41 +- CPU: Intel(R) Xeon(R) Processor (8 cores) +- Python: 3.13.5 +- Node: v22.23.3 +- cursor-agent: 2026.10.16-e3cb207 +- superpowers: bb92a77741419a4ab5f06e711a283343f1ada0c3 () +- openflow: 58b5a3fa0899f6e639454f0e4a7247dec98f6986 tarball=None +- openswarm: a48255a9cebafc48b0a7cdd39b6590a7dc193496 () +- Model flag: grok-4.7-high +- cursor-agent flags: -p --trust --force --model grok-4.7-high --output-format stream-json +- Per-run timeout: 1200s +- Command: `/usr/bin/python3 run.py --arms baseline,superpowers,openswarm --tasks all --repeats 3 --concurrency 3 --out results/e2e-2026-10-17.jsonl` + +## Per arm + +| arm | success | wall_s | input tokens | output tokens | cache read | total tokens | +| --- | --- | --- | --- | --- | --- | --- | +| baseline | 24/24 (100.0%) | median 54.7, IQR 86.7 (q1 45.7, q3 132.4), min 24.9, max 262.1 | median 25494, IQR 22072 (q1 13071, q3 35143), min 73, max 52115 | median 2550, IQR 3450 (q1 1893, q3 5342), min 255, max 10520 | median 73280, IQR 13664 (q1 68736, q3 82400), min 24448, max 99200 | median 104700, IQR 25315 (q1 87344, q3 112659), min 24776, max 144387 | +| superpowers | 24/24 (100.0%) | median 186.9, IQR 178.0 (q1 120.7, q3 298.6), min 89.9, max 766.6 | median 50893, IQR 27145 (q1 39967, q3 67112), min 1867, max 117181 | median 7935, IQR 6352 (q1 5347, q3 11699), min 547, max 18511 | median 249984, IQR 178144 (q1 209184, q3 387328), min 104960, max 910976 | median 311922, IQR 169108 (q1 267209, q3 436318), min 107374, max 1042008 | +| openflow | 11/12 (91.7%) | median 985.3, IQR 275.5 (q1 919.5, q3 1195.0), min 774.2, max 1201.0 | median 187857, IQR 56824 (q1 147069, q3 203893), min 0, max 261464 | median 37704, IQR 6994 (q1 33430, q3 40424), min 0, max 43160 | median 5061504, IQR 744192 (q1 4465792, q3 5209984), min 0, max 5839744 | median 5248352, IQR 638293 (q1 4761920, q3 5400213), min 0, max 6078903 | +| openswarm | 24/24 (100.0%) | median 219.7, IQR 219.8 (q1 136.6, q3 356.4), min 89.1, max 687.1 | median 52562, IQR 29960 (q1 43475, q3 73434), min 29881, max 147871 | median 9954, IQR 10581 (q1 6139, q3 16720), min 3757, max 33664 | median 146880, IQR 44128 (q1 135264, q3 179392), min 65664, max 338304 | median 207194, IQR 54839 (q1 191056, q3 245895), min 167773, max 519839 | + +## Per task + +| task | success | wall_s | total tokens | +| --- | --- | --- | --- | +| config-deep-merge | 12/12 (100.0%) | median 200.6, IQR 286.4 (q1 151.7, q3 438.2), min 73.7, max 947.7 | median 269892, IQR 1355109 (q1 179859, q3 1534968), min 92182, max 5341360 | +| csv-quoting | 11/12 (91.7%) | median 377.3, IQR 540.8 (q1 230.2, q3 771.0), min 152.3, max 1201.0 | median 325376, IQR 238096 (q1 131644, q3 369740), min 0, max 552868 | +| date-range-parse | 10/10 (100.0%) | median 117.9, IQR 80.6 (q1 54.1, q3 134.7), min 31.0, max 1193.2 | median 217686, IQR 140340 (q1 125366, q3 265706), min 85531, max 1963284 | +| lru-eviction | 10/10 (100.0%) | median 215.5, IQR 135.5 (q1 87.5, q3 223.1), min 49.7, max 1200.3 | median 245207, IQR 302142 (q1 124083, q3 426225), min 105701, max 534034 | +| pagination-off-by-one | 10/10 (100.0%) | median 101.4, IQR 60.8 (q1 49.1, q3 109.9), min 24.9, max 1074.3 | median 194792, IQR 154628 (q1 108757, q3 263385), min 84263, max 5458153 | +| path-normalise | 10/10 (100.0%) | median 125.6, IQR 111.6 (q1 72.7, q3 184.3), min 54.4, max 833.9 | median 219155, IQR 130977 (q1 124143, q3 255120), min 88890, max 5248352 | +| retry-backoff | 10/10 (100.0%) | median 190.1, IQR 217.8 (q1 73.2, q3 291.0), min 46.9, max 907.0 | median 232664, IQR 418169 (q1 127276, q3 545446), min 87276, max 6078903 | +| semver-compare | 10/10 (100.0%) | median 370.8, IQR 150.1 (q1 273.2, q3 423.3), min 132.3, max 941.1 | median 190604, IQR 207066 (q1 110990, q3 318056), min 24776, max 5400213 | + +## Per task and arm + +| task | arm | n | success | median wall_s | median total tokens | +| --- | --- | --- | --- | --- | --- | +| config-deep-merge | baseline | 3 | 3/3 (100.0%) | 95.5 | 134639 | +| config-deep-merge | openflow | 3 | 3/3 (100.0%) | 923.7 | 5172189 | +| config-deep-merge | openswarm | 3 | 3/3 (100.0%) | 220.5 | 198590 | +| config-deep-merge | superpowers | 3 | 3/3 (100.0%) | 180.7 | 315783 | +| csv-quoting | baseline | 3 | 3/3 (100.0%) | 205.9 | 130822 | +| csv-quoting | openflow | 3 | 2/3 (66.7%) | 1200.4 | 0 | +| csv-quoting | openswarm | 3 | 3/3 (100.0%) | 434.9 | 375899 | +| csv-quoting | superpowers | 3 | 3/3 (100.0%) | 314.8 | 342692 | +| date-range-parse | baseline | 3 | 3/3 (100.0%) | 41.0 | 87367 | +| date-range-parse | openflow | 1 | 1/1 (100.0%) | 1193.2 | 1963284 | +| date-range-parse | openswarm | 3 | 3/3 (100.0%) | 127.1 | 206379 | +| date-range-parse | superpowers | 3 | 3/3 (100.0%) | 113.5 | 274662 | +| lru-eviction | baseline | 3 | 3/3 (100.0%) | 51.2 | 106086 | +| lru-eviction | openflow | 1 | 1/1 (100.0%) | 1200.3 | — | +| lru-eviction | openswarm | 3 | 3/3 (100.0%) | 219.0 | 245207 | +| lru-eviction | superpowers | 3 | 3/3 (100.0%) | 215.2 | 428651 | +| pagination-off-by-one | baseline | 3 | 3/3 (100.0%) | 31.9 | 85292 | +| pagination-off-by-one | openflow | 1 | 1/1 (100.0%) | 1074.3 | 5458153 | +| pagination-off-by-one | openswarm | 3 | 3/3 (100.0%) | 93.4 | 181576 | +| pagination-off-by-one | superpowers | 3 | 3/3 (100.0%) | 109.9 | 267655 | +| path-normalise | baseline | 3 | 3/3 (100.0%) | 60.6 | 109087 | +| path-normalise | openflow | 1 | 1/1 (100.0%) | 833.9 | 5248352 | +| path-normalise | openswarm | 3 | 3/3 (100.0%) | 196.1 | 195966 | +| path-normalise | superpowers | 3 | 3/3 (100.0%) | 121.2 | 257845 | +| retry-backoff | baseline | 3 | 3/3 (100.0%) | 53.9 | 88119 | +| retry-backoff | openflow | 1 | 1/1 (100.0%) | 907.0 | 6078903 | +| retry-backoff | openswarm | 3 | 3/3 (100.0%) | 245.7 | 199457 | +| retry-backoff | superpowers | 3 | 3/3 (100.0%) | 295.9 | 630861 | +| semver-compare | baseline | 3 | 3/3 (100.0%) | 148.6 | 102468 | +| semver-compare | openflow | 1 | 1/1 (100.0%) | 941.1 | 5400213 | +| semver-compare | openswarm | 3 | 3/3 (100.0%) | 372.9 | 198959 | +| semver-compare | superpowers | 3 | 3/3 (100.0%) | 398.5 | 341421 | + +Skipped arms (omo) are omitted. n is the number of finished, non-skipped runs in the group. +Token totals add input, output, cache read and cache write. cursor-agent reports inputTokens with cache tokens already removed. diff --git a/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..a4e30b5 --- /dev/null +++ b/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 130274, + "duration_api_ms": 130274, + "session_id": "5a6b5784-ee0f-4c24-8819-c596e1e17a17", + "request_id": "0932504a-a3b0-4876-bd8c-54a499bc2542", + "usage": { + "inputTokens": 43015, + "outputTokens": 6652, + "cacheReadTokens": 94720, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..325af6b --- /dev/null +++ b/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 93010, + "duration_api_ms": 93010, + "session_id": "7f1bffe6-1484-4851-9c14-d54efeda11a3", + "request_id": "d5535922-133e-481b-abf9-4ce99ab0556e", + "usage": { + "inputTokens": 30533, + "outputTokens": 4906, + "cacheReadTokens": 99200, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..43b4a2f --- /dev/null +++ b/bench/e2e/results/usage/20261017T122017-config-deep-merge-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 71165, + "duration_api_ms": 71165, + "session_id": "30077454-35e1-4561-9371-b47bfab96b40", + "request_id": "20910787-0434-4f4f-81e4-595a624ba7c6", + "usage": { + "inputTokens": 28749, + "outputTokens": 3657, + "cacheReadTokens": 59776, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122130-config-deep-merge-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T122130-config-deep-merge-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..4362c5d --- /dev/null +++ b/bench/e2e/results/usage/20261017T122130-config-deep-merge-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 178079, + "duration_api_ms": 178079, + "session_id": "8d0cb46e-5b39-40c5-9de1-a6bdda313803", + "request_id": "69312e88-c4e0-4f36-b0b1-641e568c5825", + "usage": { + "inputTokens": 40345, + "outputTokens": 8635, + "cacheReadTokens": 246144, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122152-config-deep-merge-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T122152-config-deep-merge-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..13e9054 --- /dev/null +++ b/bench/e2e/results/usage/20261017T122152-config-deep-merge-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 260132, + "duration_api_ms": 260132, + "session_id": "0610b915-3824-4062-8ec0-3fe9c8fd926f", + "request_id": "7ce0a0fa-a81d-4bcd-a4be-f9f64a63cacd", + "usage": { + "inputTokens": 53400, + "outputTokens": 13213, + "cacheReadTokens": 392704, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122229-config-deep-merge-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T122229-config-deep-merge-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..22ff14a --- /dev/null +++ b/bench/e2e/results/usage/20261017T122229-config-deep-merge-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 155091, + "duration_api_ms": 155091, + "session_id": "ac5d2f0d-59b4-4d1d-8666-f12e5e73cc5d", + "request_id": "331f8211-4d0c-4cfb-a217-4caea5e7177b", + "usage": { + "inputTokens": 54614, + "outputTokens": 7345, + "cacheReadTokens": 253824, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122431-config-deep-merge-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T122431-config-deep-merge-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..6db65a8 --- /dev/null +++ b/bench/e2e/results/usage/20261017T122431-config-deep-merge-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 132224, + "duration_api_ms": 132224, + "session_id": "935bd656-79fc-4ff7-ae91-78daac56ad21", + "request_id": "762327ed-c081-4cfd-890a-72918d6effd2", + "usage": { + "inputTokens": 21456, + "outputTokens": 6923, + "cacheReadTokens": 87808, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122507-config-deep-merge-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T122507-config-deep-merge-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..1c7c1d3 --- /dev/null +++ b/bench/e2e/results/usage/20261017T122507-config-deep-merge-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 124463, + "duration_api_ms": 124463, + "session_id": "4993a35b-02fc-4998-bf3f-68d416275590", + "request_id": "8c6b5c5e-8445-416f-80b6-a67c0b321736", + "usage": { + "inputTokens": 35143, + "outputTokens": 6318, + "cacheReadTokens": 74496, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122615-config-deep-merge-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T122615-config-deep-merge-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..424ca20 --- /dev/null +++ b/bench/e2e/results/usage/20261017T122615-config-deep-merge-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 80522, + "duration_api_ms": 80522, + "session_id": "5aee5194-91b0-499c-b4bf-db69a796d6cd", + "request_id": "37fa4987-3a6d-4afe-b84e-ece7688c56f5", + "usage": { + "inputTokens": 18364, + "outputTokens": 3933, + "cacheReadTokens": 88960, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122803-csv-quoting-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T122803-csv-quoting-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..192b2bb --- /dev/null +++ b/bench/e2e/results/usage/20261017T122803-csv-quoting-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 203481, + "duration_api_ms": 203481, + "session_id": "10565516-ac5d-469e-a66a-4e5d21d2636e", + "request_id": "eee6ab8b-92d5-46ba-b1c8-d60ec5bc4763", + "usage": { + "inputTokens": 42182, + "outputTokens": 10520, + "cacheReadTokens": 81408, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T122812-csv-quoting-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T122812-csv-quoting-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..1d4fdff --- /dev/null +++ b/bench/e2e/results/usage/20261017T122812-csv-quoting-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 149709, + "duration_api_ms": 149709, + "session_id": "f31b4771-fa80-462a-a930-1d7d1f3134a0", + "request_id": "d5f9357d-8884-4e49-b9d0-313cbdc29c10", + "usage": { + "inputTokens": 18626, + "outputTokens": 8085, + "cacheReadTokens": 77952, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123044-csv-quoting-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T123044-csv-quoting-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..8200a2e --- /dev/null +++ b/bench/e2e/results/usage/20261017T123044-csv-quoting-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 204734, + "duration_api_ms": 204734, + "session_id": "823828be-4c84-43de-954b-0e46e245bf47", + "request_id": "b2d86981-34d2-42e6-8fa8-7729948ba2df", + "usage": { + "inputTokens": 39165, + "outputTokens": 10377, + "cacheReadTokens": 81280, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123129-csv-quoting-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T123129-csv-quoting-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..e6782a6 --- /dev/null +++ b/bench/e2e/results/usage/20261017T123129-csv-quoting-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 311282, + "duration_api_ms": 311282, + "session_id": "19de07a7-7696-48cf-b8a5-4325e0850486", + "request_id": "75c6889c-b866-4b22-abd2-6222e17f6122", + "usage": { + "inputTokens": 49696, + "outputTokens": 16004, + "cacheReadTokens": 276992, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123142-csv-quoting-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T123142-csv-quoting-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..f1aa19a --- /dev/null +++ b/bench/e2e/results/usage/20261017T123142-csv-quoting-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 368592, + "duration_api_ms": 368592, + "session_id": "35ba8fb0-c616-4791-9adb-46bc508699d9", + "request_id": "8a5d05a6-fe58-4bb4-a391-6ad1f6aed2d2", + "usage": { + "inputTokens": 82773, + "outputTokens": 18511, + "cacheReadTokens": 451584, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123412-csv-quoting-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T123412-csv-quoting-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..93ed75b --- /dev/null +++ b/bench/e2e/results/usage/20261017T123412-csv-quoting-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 234118, + "duration_api_ms": 234118, + "session_id": "657434a5-560a-45b0-8330-23c9be75b9e6", + "request_id": "1630e106-c3a2-4741-8256-00634cb1c754", + "usage": { + "inputTokens": 57397, + "outputTokens": 11047, + "cacheReadTokens": 239616, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123643-csv-quoting-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T123643-csv-quoting-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..f45ed8b --- /dev/null +++ b/bench/e2e/results/usage/20261017T123643-csv-quoting-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 191976, + "duration_api_ms": 191976, + "session_id": "fcc84471-af70-4b58-a6e3-e50118aab5fe", + "request_id": "74a19474-c781-4f9b-859d-193e347508f3", + "usage": { + "inputTokens": 49109, + "outputTokens": 9658, + "cacheReadTokens": 159360, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123753-csv-quoting-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T123753-csv-quoting-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..3aea828 --- /dev/null +++ b/bench/e2e/results/usage/20261017T123753-csv-quoting-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 273539, + "duration_api_ms": 273539, + "session_id": "e4c76ef7-3673-4d5a-8b2e-c4a9ae08d2b4", + "request_id": "84736cbb-5ebe-4d55-8b70-345c460e592a", + "usage": { + "inputTokens": 84076, + "outputTokens": 13359, + "cacheReadTokens": 181248, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T123809-csv-quoting-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T123809-csv-quoting-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..d5c0a56 --- /dev/null +++ b/bench/e2e/results/usage/20261017T123809-csv-quoting-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 225992, + "duration_api_ms": 225992, + "session_id": "1333c1f2-6f79-419a-bb54-022421f2a8f7", + "request_id": "445c21d4-991c-413d-bce1-352dfffe9502", + "usage": { + "inputTokens": 26453, + "outputTokens": 10978, + "cacheReadTokens": 170624, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124432-date-range-parse-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T124432-date-range-parse-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..59ef7c4 --- /dev/null +++ b/bench/e2e/results/usage/20261017T124432-date-range-parse-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 28504, + "duration_api_ms": 28504, + "session_id": "ecb8625f-1317-45b2-9a28-5edabe6a2956", + "request_id": "722df27b-74d5-4180-9cca-191d83c747a7", + "usage": { + "inputTokens": 12152, + "outputTokens": 1315, + "cacheReadTokens": 72064, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124503-date-range-parse-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T124503-date-range-parse-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..28679ae --- /dev/null +++ b/bench/e2e/results/usage/20261017T124503-date-range-parse-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 39906, + "duration_api_ms": 39906, + "session_id": "c36103f4-ad1a-4be4-b24d-6d246eb60b24", + "request_id": "6a765c80-f47e-49a4-a7ab-2a4124a7759c", + "usage": { + "inputTokens": 25390, + "outputTokens": 1945, + "cacheReadTokens": 60032, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124508-date-range-parse-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T124508-date-range-parse-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..631decc --- /dev/null +++ b/bench/e2e/results/usage/20261017T124508-date-range-parse-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 38094, + "duration_api_ms": 38094, + "session_id": "c352581a-3990-49da-a150-410e20e5b2c4", + "request_id": "372e5e12-9f66-41fc-a3d8-75e36edede6e", + "usage": { + "inputTokens": 24409, + "outputTokens": 1736, + "cacheReadTokens": 78592, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124545-date-range-parse-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T124545-date-range-parse-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..77d876d --- /dev/null +++ b/bench/e2e/results/usage/20261017T124545-date-range-parse-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 109357, + "duration_api_ms": 109357, + "session_id": "1f52b137-57c6-45f3-8522-14020993131d", + "request_id": "bbb65eac-e578-422d-8c11-2aa47d327731", + "usage": { + "inputTokens": 65267, + "outputTokens": 5363, + "cacheReadTokens": 204032, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124549-date-range-parse-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T124549-date-range-parse-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..5c993b8 --- /dev/null +++ b/bench/e2e/results/usage/20261017T124549-date-range-parse-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 86452, + "duration_api_ms": 86452, + "session_id": "d7cc3745-936c-4bc9-9538-3262302999dc", + "request_id": "65b4f01b-e226-4b78-a4f0-76a2879d66fc", + "usage": { + "inputTokens": 24971, + "outputTokens": 3947, + "cacheReadTokens": 209920, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124719-date-range-parse-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T124719-date-range-parse-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..7e42f2f --- /dev/null +++ b/bench/e2e/results/usage/20261017T124719-date-range-parse-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 159175, + "duration_api_ms": 159175, + "session_id": "dbba04bc-c4d8-4574-b7de-3912c4fbb5f1", + "request_id": "a296a308-7ccc-422e-84f1-63dc9e3770ea", + "usage": { + "inputTokens": 30115, + "outputTokens": 7377, + "cacheReadTokens": 316288, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124739-date-range-parse-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T124739-date-range-parse-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..a7bdc5c --- /dev/null +++ b/bench/e2e/results/usage/20261017T124739-date-range-parse-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 99790, + "duration_api_ms": 99790, + "session_id": "28288edc-b18c-4e56-9406-953432732a82", + "request_id": "9f946aae-5003-458d-86cf-60a3cf08cba6", + "usage": { + "inputTokens": 37576, + "outputTokens": 4577, + "cacheReadTokens": 71680, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124811-date-range-parse-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T124811-date-range-parse-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..89200f9 --- /dev/null +++ b/bench/e2e/results/usage/20261017T124811-date-range-parse-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 84292, + "duration_api_ms": 84292, + "session_id": "cfeda10b-81a5-45a4-94a0-7bd5d5d9c037", + "request_id": "53c80aaa-dd52-4156-920e-d13be19dd48c", + "usage": { + "inputTokens": 32394, + "outputTokens": 4068, + "cacheReadTokens": 76416, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T124956-date-range-parse-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T124956-date-range-parse-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..84cb058 --- /dev/null +++ b/bench/e2e/results/usage/20261017T124956-date-range-parse-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 96356, + "duration_api_ms": 96356, + "session_id": "1a55ed8e-c8ad-453a-b2ab-bdaf42076ab8", + "request_id": "09ad3855-1a46-4c1c-af1c-b143bb1b543b", + "usage": { + "inputTokens": 19482, + "outputTokens": 4590, + "cacheReadTokens": 113920, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125002-lru-eviction-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T125002-lru-eviction-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..e8d79f5 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125002-lru-eviction-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 46557, + "duration_api_ms": 46557, + "session_id": "2bf45f2e-f4c6-4a47-b6b1-94dedd2e9ee3", + "request_id": "83cf87c2-5325-4d18-9d08-c8708a65de32", + "usage": { + "inputTokens": 29867, + "outputTokens": 2106, + "cacheReadTokens": 73728, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125013-lru-eviction-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T125013-lru-eviction-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..2cdf149 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125013-lru-eviction-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 48710, + "duration_api_ms": 48710, + "session_id": "393bed56-97bc-429b-b996-b08e93caa1a6", + "request_id": "752d6fa6-23e5-45ca-86e0-8df4e48fe48c", + "usage": { + "inputTokens": 36478, + "outputTokens": 2229, + "cacheReadTokens": 85376, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125052-lru-eviction-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T125052-lru-eviction-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..70ccfbf --- /dev/null +++ b/bench/e2e/results/usage/20261017T125052-lru-eviction-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 48125, + "duration_api_ms": 48125, + "session_id": "1f3ff685-7762-43c9-89a3-86cd766545a3", + "request_id": "bab11138-9827-4388-8b1c-3bad6eac075d", + "usage": { + "inputTokens": 25599, + "outputTokens": 2151, + "cacheReadTokens": 78336, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125105-lru-eviction-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T125105-lru-eviction-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..408adaa --- /dev/null +++ b/bench/e2e/results/usage/20261017T125105-lru-eviction-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 242324, + "duration_api_ms": 242324, + "session_id": "4e7550f7-f00a-4c9b-82de-7c13c8967000", + "request_id": "966d4d22-dcd4-475e-adca-c852c15bf05d", + "usage": { + "inputTokens": 72646, + "outputTokens": 11596, + "cacheReadTokens": 449792, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125143-lru-eviction-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T125143-lru-eviction-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..bb69751 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125143-lru-eviction-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 211771, + "duration_api_ms": 211771, + "session_id": "f6605a25-39a4-4579-bba4-e463ebd6a03b", + "request_id": "456a8713-62d1-4011-a19c-1f192056d8c3", + "usage": { + "inputTokens": 96536, + "outputTokens": 9939, + "cacheReadTokens": 322176, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125203-lru-eviction-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T125203-lru-eviction-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..356966d --- /dev/null +++ b/bench/e2e/results/usage/20261017T125203-lru-eviction-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 187200, + "duration_api_ms": 187200, + "session_id": "2af5eba9-35bb-4295-aad3-f29fa4f730de", + "request_id": "b246c977-8aa2-4bc5-b312-536a4e3208bf", + "usage": { + "inputTokens": 32196, + "outputTokens": 8493, + "cacheReadTokens": 385536, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125511-lru-eviction-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T125511-lru-eviction-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..63d0f87 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125511-lru-eviction-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 107439, + "duration_api_ms": 107439, + "session_id": "3f87c4c2-1a21-4343-bb09-77b0497a84df", + "request_id": "8d0219ed-d1e3-49a9-93a0-30a400058eb5", + "usage": { + "inputTokens": 34151, + "outputTokens": 4937, + "cacheReadTokens": 98176, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125516-lru-eviction-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T125516-lru-eviction-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..702c78f --- /dev/null +++ b/bench/e2e/results/usage/20261017T125516-lru-eviction-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 79425, + "duration_api_ms": 79425, + "session_id": "7d0e3151-5789-45ee-a4b9-5bdfc4fe0849", + "request_id": "33d1c542-63a7-43fc-90ff-b3550c28063d", + "usage": { + "inputTokens": 18266, + "outputTokens": 3703, + "cacheReadTokens": 109952, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125518-lru-eviction-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T125518-lru-eviction-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..0c48789 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125518-lru-eviction-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 94400, + "duration_api_ms": 94400, + "session_id": "fa943001-4d2a-4c93-bed4-6e58f3cae92e", + "request_id": "5b8d0239-c8ae-4751-9e06-3fc97d4c2c19", + "usage": { + "inputTokens": 35691, + "outputTokens": 4234, + "cacheReadTokens": 116992, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..26e02a0 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 770396, + "duration_api_ms": 770396, + "session_id": "14a918aa-14a5-496e-a09f-289447125d27", + "request_id": "de7bea28-dfbc-426c-a4ca-dd4667f97d0c", + "usage": { + "inputTokens": 261464, + "outputTokens": 34664, + "cacheReadTokens": 4465792, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r2--agent-1.result.json new file mode 100644 index 0000000..97ebe56 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 944415, + "duration_api_ms": 944415, + "session_id": "31c1ee9e-689f-448a-b04b-9ea6de9e0f05", + "request_id": "b4fcc4aa-d0e9-4112-995e-3daa0d27ec8f", + "usage": { + "inputTokens": 203893, + "outputTokens": 40424, + "cacheReadTokens": 4927872, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r3--agent-1.result.json new file mode 100644 index 0000000..6aaaf51 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125628-config-deep-merge-openflow-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 919865, + "duration_api_ms": 919865, + "session_id": "08796f0e-7b0a-4d4f-a1a9-484fa5440eff", + "request_id": "fb5ce405-8406-4bbc-a9d0-5c16ff79d07b", + "usage": { + "inputTokens": 187857, + "outputTokens": 39903, + "cacheReadTokens": 5113600, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125847-pagination-off-by-one-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T125847-pagination-off-by-one-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..a6efcc4 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125847-pagination-off-by-one-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 22072, + "duration_api_ms": 22072, + "session_id": "ca80c62c-3bfe-439a-addf-0a8c91eb9a9b", + "request_id": "f6e00d23-4d2a-47b0-a9a2-d8d19d7eea8e", + "usage": { + "inputTokens": 11821, + "outputTokens": 890, + "cacheReadTokens": 71552, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125855-pagination-off-by-one-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T125855-pagination-off-by-one-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..0b252a2 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125855-pagination-off-by-one-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 29033, + "duration_api_ms": 29033, + "session_id": "fc512105-58d6-453d-93f0-1c2203e98e9a", + "request_id": "797808cc-c417-4b3c-afc6-7a1e5f483bfb", + "usage": { + "inputTokens": 11802, + "outputTokens": 1298, + "cacheReadTokens": 72192, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125902-pagination-off-by-one-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T125902-pagination-off-by-one-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..d7b8297 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125902-pagination-off-by-one-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 32899, + "duration_api_ms": 32899, + "session_id": "ea84b8ea-a9d1-4865-97f2-e19774d16d93", + "request_id": "8ac26628-2631-48af-b30b-ad276aaee8ae", + "usage": { + "inputTokens": 12208, + "outputTokens": 1525, + "cacheReadTokens": 72448, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125912-pagination-off-by-one-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T125912-pagination-off-by-one-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..ff6cb3a --- /dev/null +++ b/bench/e2e/results/usage/20261017T125912-pagination-off-by-one-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 115202, + "duration_api_ms": 115202, + "session_id": "e91b6775-b573-4a6e-ae7f-028f82371794", + "request_id": "f5804c28-532c-4ef1-b466-af4d0cc1dcb3", + "usage": { + "inputTokens": 52090, + "outputTokens": 5300, + "cacheReadTokens": 224896, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125927-pagination-off-by-one-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T125927-pagination-off-by-one-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..e70e571 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125927-pagination-off-by-one-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 105843, + "duration_api_ms": 105843, + "session_id": "97909c90-3ab0-482c-8258-1e684df5eda5", + "request_id": "eea81874-00d3-4503-91b4-19342cb53e6a", + "usage": { + "inputTokens": 38833, + "outputTokens": 4767, + "cacheReadTokens": 206976, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T125938-pagination-off-by-one-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T125938-pagination-off-by-one-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..770d955 --- /dev/null +++ b/bench/e2e/results/usage/20261017T125938-pagination-off-by-one-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 101998, + "duration_api_ms": 101998, + "session_id": "853f9654-fb32-4266-964d-1f7233881a41", + "request_id": "bfcfc795-800d-47e2-891b-f9d3e1332e6c", + "usage": { + "inputTokens": 49189, + "outputTokens": 4578, + "cacheReadTokens": 213888, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130111-pagination-off-by-one-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T130111-pagination-off-by-one-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..5f41631 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130111-pagination-off-by-one-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 56571, + "duration_api_ms": 56571, + "session_id": "86674e57-7daa-4a55-ab6c-0bace1f3b129", + "request_id": "bce83513-ae35-4b8b-8824-88a9833c2918", + "usage": { + "inputTokens": 17202, + "outputTokens": 2771, + "cacheReadTokens": 67200, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130117-pagination-off-by-one-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T130117-pagination-off-by-one-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..7aab323 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130117-pagination-off-by-one-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 66961, + "duration_api_ms": 66961, + "session_id": "04a6467c-0746-4740-a72f-b0954d9e8048", + "request_id": "1fb7475a-c620-464f-b09a-fe7b93c7f499", + "usage": { + "inputTokens": 17887, + "outputTokens": 3300, + "cacheReadTokens": 87808, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130128-pagination-off-by-one-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T130128-pagination-off-by-one-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..62e911d --- /dev/null +++ b/bench/e2e/results/usage/20261017T130128-pagination-off-by-one-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 72332, + "duration_api_ms": 72332, + "session_id": "0d53f802-1bcd-4e3e-890f-3b155bc1d221", + "request_id": "609bd4a1-3b7c-4461-a0db-f68ebe04bac8", + "usage": { + "inputTokens": 18268, + "outputTokens": 3518, + "cacheReadTokens": 112256, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130240-path-normalise-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T130240-path-normalise-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..a471a76 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130240-path-normalise-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 60488, + "duration_api_ms": 60488, + "session_id": "8b988725-ea61-455c-95a3-1b0695c6b1fc", + "request_id": "55524381-c45f-4f1c-87f5-c67a346a53c4", + "usage": { + "inputTokens": 14379, + "outputTokens": 3061, + "cacheReadTokens": 92160, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130250-path-normalise-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T130250-path-normalise-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..9481509 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130250-path-normalise-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 57264, + "duration_api_ms": 57264, + "session_id": "54b3a809-3fce-4849-812c-c8a3a326582f", + "request_id": "b349ff27-689c-4a21-9040-28ae0186b75b", + "usage": { + "inputTokens": 52115, + "outputTokens": 2828, + "cacheReadTokens": 54144, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130318-path-normalise-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T130318-path-normalise-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..62d67e8 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130318-path-normalise-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 51349, + "duration_api_ms": 51349, + "session_id": "ffb4644b-813c-4fe3-b1e9-78faebb821ce", + "request_id": "522bceec-f136-40cb-a697-5922d2c805f6", + "usage": { + "inputTokens": 13498, + "outputTokens": 2560, + "cacheReadTokens": 72832, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130344-path-normalise-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T130344-path-normalise-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..7393cb2 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130344-path-normalise-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 117248, + "duration_api_ms": 117248, + "session_id": "eb948584-dee8-4304-81f2-e9a96386350a", + "request_id": "9260b23a-516d-4b71-b6f3-5fd7b99c8910", + "usage": { + "inputTokens": 52666, + "outputTokens": 5627, + "cacheReadTokens": 199552, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130351-path-normalise-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T130351-path-normalise-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..3da5fd9 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130351-path-normalise-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 126849, + "duration_api_ms": 126849, + "session_id": "7480dce2-eb30-4013-8d47-0225e65bfb5e", + "request_id": "f578093b-9822-481c-9749-bac984b1a5d9", + "usage": { + "inputTokens": 96480, + "outputTokens": 5687, + "cacheReadTokens": 180480, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130412-path-normalise-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T130412-path-normalise-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..989d920 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130412-path-normalise-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 96528, + "duration_api_ms": 96528, + "session_id": "654287ec-2783-4453-b745-adc8595acd99", + "request_id": "34195191-4821-49e4-9de3-0217628c4e03", + "usage": { + "inputTokens": 48995, + "outputTokens": 4542, + "cacheReadTokens": 193408, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130545-path-normalise-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T130545-path-normalise-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..0874657 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130545-path-normalise-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 135804, + "duration_api_ms": 135804, + "session_id": "6721afa7-76d6-4a2e-8d69-dbcbb0b78b98", + "request_id": "1410c7fd-1949-4067-b8cc-7f179f62c248", + "usage": { + "inputTokens": 38760, + "outputTokens": 6885, + "cacheReadTokens": 52352, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130552-path-normalise-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T130552-path-normalise-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..df79ab5 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130552-path-normalise-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 127669, + "duration_api_ms": 127669, + "session_id": "e4d92edb-1fb3-49a9-bdfa-ab6ef28034cf", + "request_id": "7afdb156-fc34-450c-927d-2b70df636322", + "usage": { + "inputTokens": 20753, + "outputTokens": 6378, + "cacheReadTokens": 88704, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130601-path-normalise-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T130601-path-normalise-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..2446e12 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130601-path-normalise-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 103301, + "duration_api_ms": 103301, + "session_id": "b7c53adf-e9d7-463e-aee5-ddf730c5832f", + "request_id": "4cb29679-8384-421d-8836-f944852224ee", + "usage": { + "inputTokens": 19242, + "outputTokens": 5111, + "cacheReadTokens": 67712, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130829-retry-backoff-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T130829-retry-backoff-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..32d7285 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130829-retry-backoff-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 43460, + "duration_api_ms": 43460, + "session_id": "902c95f6-3826-4208-9133-06f6e9d2e5e9", + "request_id": "4125f6ff-eaaf-4829-8606-54488fc047c4", + "usage": { + "inputTokens": 12618, + "outputTokens": 2082, + "cacheReadTokens": 72576, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130908-retry-backoff-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T130908-retry-backoff-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..d22d19e --- /dev/null +++ b/bench/e2e/results/usage/20261017T130908-retry-backoff-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 51349, + "duration_api_ms": 51349, + "session_id": "e84e7b8b-2ec3-486c-9a03-6cb85c852eb5", + "request_id": "30e71b4f-a91f-40de-9e6d-41fc380a48d8", + "usage": { + "inputTokens": 13222, + "outputTokens": 2541, + "cacheReadTokens": 90880, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130916-retry-backoff-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T130916-retry-backoff-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..75120d5 --- /dev/null +++ b/bench/e2e/results/usage/20261017T130916-retry-backoff-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 52241, + "duration_api_ms": 52241, + "session_id": "365c1f91-65da-4d42-960e-a11154029551", + "request_id": "149490b9-b81b-469a-971a-342266335b76", + "usage": { + "inputTokens": 45488, + "outputTokens": 2567, + "cacheReadTokens": 40064, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T130922-csv-quoting-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T130922-csv-quoting-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..ec747fa --- /dev/null +++ b/bench/e2e/results/usage/20261017T130922-csv-quoting-openflow-r1--agent-1.result.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131002-retry-backoff-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T131002-retry-backoff-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..0614ed8 --- /dev/null +++ b/bench/e2e/results/usage/20261017T131002-retry-backoff-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 125284, + "duration_api_ms": 125284, + "session_id": "34160877-8eab-4623-a624-208c8ba88a0f", + "request_id": "5a83b86a-0291-4eb2-88dd-960ccda2a809", + "usage": { + "inputTokens": 25056, + "outputTokens": 5551, + "cacheReadTokens": 235264, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131011-retry-backoff-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T131011-retry-backoff-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..cb4dd4b --- /dev/null +++ b/bench/e2e/results/usage/20261017T131011-retry-backoff-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 292934, + "duration_api_ms": 292934, + "session_id": "bbbdaa5a-4f14-43f5-8d47-f3797d6b904c", + "request_id": "ada92ea6-52e5-4933-bc5d-9f3221907691", + "usage": { + "inputTokens": 79675, + "outputTokens": 12690, + "cacheReadTokens": 538496, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131137-retry-backoff-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T131137-retry-backoff-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..f5023cb --- /dev/null +++ b/bench/e2e/results/usage/20261017T131137-retry-backoff-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 763494, + "duration_api_ms": 763494, + "session_id": "bf5449c2-6d0d-40ce-9750-370575c7abcb", + "request_id": "1a14fd61-eee5-46a6-b31b-a84db374432a", + "usage": { + "inputTokens": 117181, + "outputTokens": 13851, + "cacheReadTokens": 910976, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131152-csv-quoting-openflow-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T131152-csv-quoting-openflow-r2--agent-1.result.json new file mode 100644 index 0000000..ec747fa --- /dev/null +++ b/bench/e2e/results/usage/20261017T131152-csv-quoting-openflow-r2--agent-1.result.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131211-retry-backoff-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T131211-retry-backoff-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..d05330b --- /dev/null +++ b/bench/e2e/results/usage/20261017T131211-retry-backoff-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 78053, + "duration_api_ms": 78053, + "session_id": "4952ad6f-a522-4008-a4a4-a1e4780e205c", + "request_id": "3e9090ec-b5b8-4948-8d64-7483ebb86230", + "usage": { + "inputTokens": 32734, + "outputTokens": 3792, + "cacheReadTokens": 51584, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131216-csv-quoting-openflow-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T131216-csv-quoting-openflow-r3--agent-1.result.json new file mode 100644 index 0000000..ec747fa --- /dev/null +++ b/bench/e2e/results/usage/20261017T131216-csv-quoting-openflow-r3--agent-1.result.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131425-retry-backoff-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T131425-retry-backoff-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..c6c7e08 --- /dev/null +++ b/bench/e2e/results/usage/20261017T131425-retry-backoff-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 217838, + "duration_api_ms": 217838, + "session_id": "487118ed-456f-4b8d-a692-2210a7e4a3e2", + "request_id": "fe2c2b61-90ff-46f6-8dd6-7be419d3dccc", + "usage": { + "inputTokens": 58326, + "outputTokens": 5099, + "cacheReadTokens": 132480, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131507-retry-backoff-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T131507-retry-backoff-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..1584fb5 --- /dev/null +++ b/bench/e2e/results/usage/20261017T131507-retry-backoff-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 201086, + "duration_api_ms": 201086, + "session_id": "9276e7ec-5383-402a-b348-613697a9b500", + "request_id": "02b8feb9-fc96-476c-9e6d-c1eb43235813", + "usage": { + "inputTokens": 37714, + "outputTokens": 5807, + "cacheReadTokens": 71680, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131902-semver-compare-baseline-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T131902-semver-compare-baseline-r1--agent-1.result.json new file mode 100644 index 0000000..233fefc --- /dev/null +++ b/bench/e2e/results/usage/20261017T131902-semver-compare-baseline-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 145807, + "duration_api_ms": 145807, + "session_id": "fe6f24aa-68a7-465a-8923-bb7f9dde021b", + "request_id": "b719bd5b-cb4c-4369-afd6-390233be55a7", + "usage": { + "inputTokens": 34698, + "outputTokens": 7482, + "cacheReadTokens": 60288, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131913-semver-compare-baseline-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T131913-semver-compare-baseline-r2--agent-1.result.json new file mode 100644 index 0000000..6d78f15 --- /dev/null +++ b/bench/e2e/results/usage/20261017T131913-semver-compare-baseline-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 129288, + "duration_api_ms": 129288, + "session_id": "9cecc896-5fab-49d2-87a4-042e16f8ea29", + "request_id": "35396d43-2d84-41d3-b349-1ef56f73a509", + "usage": { + "inputTokens": 27955, + "outputTokens": 6842, + "cacheReadTokens": 87040, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131922-date-range-parse-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T131922-date-range-parse-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..c57d88a --- /dev/null +++ b/bench/e2e/results/usage/20261017T131922-date-range-parse-openflow-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 1188458, + "duration_api_ms": 1188458, + "session_id": "bfc07f0a-c85e-4ba2-8fdb-a3bf208c49a2", + "request_id": "736f3cbe-dc95-4180-80ff-657860293d65", + "usage": { + "inputTokens": 19475, + "outputTokens": 10881, + "cacheReadTokens": 1932928, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131922-lru-eviction-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T131922-lru-eviction-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..ec747fa --- /dev/null +++ b/bench/e2e/results/usage/20261017T131922-lru-eviction-openflow-r1--agent-1.result.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T131922-pagination-off-by-one-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T131922-pagination-off-by-one-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..4c68c02 --- /dev/null +++ b/bench/e2e/results/usage/20261017T131922-pagination-off-by-one-openflow-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 1068661, + "duration_api_ms": 1068661, + "session_id": "43fa26d1-bfcc-4f61-8001-c7a02287d98f", + "request_id": "020612d1-70cc-47f0-967f-e1e3468f8b0c", + "usage": { + "inputTokens": 207571, + "outputTokens": 33430, + "cacheReadTokens": 5217152, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T132125-semver-compare-baseline-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T132125-semver-compare-baseline-r3--agent-1.result.json new file mode 100644 index 0000000..5c6603c --- /dev/null +++ b/bench/e2e/results/usage/20261017T132125-semver-compare-baseline-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 257652, + "duration_api_ms": 257652, + "session_id": "3a2bb53b-de43-4dd3-a68f-5d06d8be85c8", + "request_id": "0b7f3737-3957-4858-bf7f-40bd5c479de6", + "usage": { + "inputTokens": 73, + "outputTokens": 255, + "cacheReadTokens": 24448, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T132130-semver-compare-superpowers-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T132130-semver-compare-superpowers-r1--agent-1.result.json new file mode 100644 index 0000000..f072fa2 --- /dev/null +++ b/bench/e2e/results/usage/20261017T132130-semver-compare-superpowers-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 303209, + "duration_api_ms": 303209, + "session_id": "3f1538c4-dfe2-48aa-b711-fd757657e3c8", + "request_id": "633a752a-dffc-4015-a18a-7141263ffcb8", + "usage": { + "inputTokens": 49669, + "outputTokens": 10152, + "cacheReadTokens": 281600, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T132424-semver-compare-superpowers-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T132424-semver-compare-superpowers-r2--agent-1.result.json new file mode 100644 index 0000000..6f170bd --- /dev/null +++ b/bench/e2e/results/usage/20261017T132424-semver-compare-superpowers-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 385010, + "duration_api_ms": 385010, + "session_id": "f0682c72-2302-412b-893c-6e90790e4380", + "request_id": "20adf054-d9bf-49a1-87e7-055013e20880", + "usage": { + "inputTokens": 46700, + "outputTokens": 12009, + "cacheReadTokens": 422400, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T132547-semver-compare-superpowers-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T132547-semver-compare-superpowers-r3--agent-1.result.json new file mode 100644 index 0000000..8d32e27 --- /dev/null +++ b/bench/e2e/results/usage/20261017T132547-semver-compare-superpowers-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 552749, + "duration_api_ms": 552749, + "session_id": "94c4f8cc-0fcf-4075-93ce-91d9868ba14e", + "request_id": "b356f9ee-3f79-43e6-b1a1-99358782f9b5", + "usage": { + "inputTokens": 1867, + "outputTokens": 547, + "cacheReadTokens": 104960, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T132637-semver-compare-openswarm-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T132637-semver-compare-openswarm-r1--agent-1.result.json new file mode 100644 index 0000000..cb54053 --- /dev/null +++ b/bench/e2e/results/usage/20261017T132637-semver-compare-openswarm-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 200864, + "duration_api_ms": 200864, + "session_id": "b7ed2832-6c93-4b21-ad8b-52108ffb93dd", + "request_id": "a2e038f7-40dc-43fb-a490-c382a35dde55", + "usage": { + "inputTokens": 24043, + "outputTokens": 9699, + "cacheReadTokens": 66176, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T133102-semver-compare-openswarm-r2--agent-1.result.json b/bench/e2e/results/usage/20261017T133102-semver-compare-openswarm-r2--agent-1.result.json new file mode 100644 index 0000000..b051ab3 --- /dev/null +++ b/bench/e2e/results/usage/20261017T133102-semver-compare-openswarm-r2--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 201487, + "duration_api_ms": 201487, + "session_id": "6f9a69ad-1223-46cc-a18a-349e2a0b3166", + "request_id": "43932437-1ceb-4c86-b009-b7996808e4ba", + "usage": { + "inputTokens": 23761, + "outputTokens": 9087, + "cacheReadTokens": 120832, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T133246-semver-compare-openswarm-r3--agent-1.result.json b/bench/e2e/results/usage/20261017T133246-semver-compare-openswarm-r3--agent-1.result.json new file mode 100644 index 0000000..7ac4911 --- /dev/null +++ b/bench/e2e/results/usage/20261017T133246-semver-compare-openswarm-r3--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 297582, + "duration_api_ms": 297582, + "session_id": "f4969927-63cd-4400-bc29-ccef175aa9ed", + "request_id": "84086633-474a-4099-8c09-79c3893e13fd", + "usage": { + "inputTokens": 31939, + "outputTokens": 10596, + "cacheReadTokens": 46848, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T133716-path-normalise-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T133716-path-normalise-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..a991a53 --- /dev/null +++ b/bench/e2e/results/usage/20261017T133716-path-normalise-openflow-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 830021, + "duration_api_ms": 830021, + "session_id": "0c1ce27a-d222-407c-ac62-70af85a99577", + "request_id": "5b8eb393-1990-43d1-baec-9b6230049e6c", + "usage": { + "inputTokens": 149144, + "outputTokens": 37704, + "cacheReadTokens": 5061504, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T133915-retry-backoff-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T133915-retry-backoff-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..51e9ec3 --- /dev/null +++ b/bench/e2e/results/usage/20261017T133915-retry-backoff-openflow-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 901037, + "duration_api_ms": 901037, + "session_id": "08f72a12-61e9-4102-9e4d-4608145ae96c", + "request_id": "9c9d3f10-0ddd-4a7a-950e-390bffdcdbf4", + "usage": { + "inputTokens": 198345, + "outputTokens": 40814, + "cacheReadTokens": 5839744, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/results/usage/20261017T133922-semver-compare-openflow-r1--agent-1.result.json b/bench/e2e/results/usage/20261017T133922-semver-compare-openflow-r1--agent-1.result.json new file mode 100644 index 0000000..5e8c353 --- /dev/null +++ b/bench/e2e/results/usage/20261017T133922-semver-compare-openflow-r1--agent-1.result.json @@ -0,0 +1,14 @@ +{ + "subtype": "success", + "is_error": false, + "duration_ms": 937878, + "duration_api_ms": 937878, + "session_id": "8dbf6589-8358-45da-afd9-4689d754b166", + "request_id": "4f2031b0-95ee-40df-88d6-ba83939333fe", + "usage": { + "inputTokens": 147069, + "outputTokens": 43160, + "cacheReadTokens": 5209984, + "cacheWriteTokens": 0 + } +} \ No newline at end of file diff --git a/bench/e2e/run.py b/bench/e2e/run.py new file mode 100644 index 0000000..5702978 --- /dev/null +++ b/bench/e2e/run.py @@ -0,0 +1,495 @@ +#!/usr/bin/env python3 +"""Run the end-to-end task-completion matrix. + +Each cell is one (task, arm, repeat). Hidden tests are copied into a fresh +temp checkout only after the arm exits. Rows already present in the output +jsonl are skipped, so a killed run can be resumed with the same --out path. +""" + +from __future__ import annotations + +import argparse +import json +import os +import platform +import shutil +import signal +import subprocess +import sys +import threading +import time +from concurrent.futures import ThreadPoolExecutor, as_completed +from datetime import datetime, timezone +from pathlib import Path + +from harness.metrics import parse_agent_stream, sum_calls + +ROOT = Path(__file__).resolve().parent +TASKS_DIR = ROOT / "tasks" +TMP_DIR = ROOT / "tmp" +ARMS_DIR = ROOT / "arms" +MODEL = "grok-4.7-high" +RUN_TIMEOUT_S = 20 * 60 +TEST_TIMEOUT_S = 120 + + +def utc_now() -> str: + return datetime.now(timezone.utc).isoformat() + + +def load_tasks(selection: str) -> list[dict]: + found = [] + for meta_path in sorted(TASKS_DIR.glob("*/meta.json")): + meta = json.loads(meta_path.read_text(encoding="utf-8")) + task_dir = meta_path.parent + meta["dir"] = str(task_dir) + meta["task_md"] = str(task_dir / "TASK.md") + meta["repo"] = str(task_dir / "repo") + meta["hidden"] = str(task_dir / "hidden") + meta.setdefault("id", task_dir.name) + found.append(meta) + if selection == "all": + return found + wanted = [part.strip() for part in selection.split(",") if part.strip()] + by_id = {task["id"]: task for task in found} + missing = [task_id for task_id in wanted if task_id not in by_id] + if missing: + raise SystemExit(f"unknown task(s): {', '.join(missing)}") + return [by_id[task_id] for task_id in wanted] + + +def existing_keys(path: Path) -> set[tuple]: + keys = set() + if not path.is_file(): + return keys + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except json.JSONDecodeError: + continue + if row.get("type") not in (None, "run"): + continue + if "task" in row and "arm" in row and "repeat" in row: + keys.add((row["task"], row["arm"], int(row["repeat"]))) + return keys + + +def git_cmd(repo: Path, *args: str) -> subprocess.CompletedProcess: + return subprocess.run( + ["git", "-c", "user.email=bench-e2e@local", "-c", "user.name=bench-e2e", *args], + cwd=repo, + text=True, + capture_output=True, + check=False, + ) + + +def init_repo(repo: Path) -> str: + init = git_cmd(repo, "init", "-b", "main") + if init.returncode != 0: + git_cmd(repo, "init") + git_cmd(repo, "checkout", "-B", "main") + git_cmd(repo, "add", "-A") + commit = git_cmd(repo, "commit", "-m", "baseline") + if commit.returncode != 0: + raise RuntimeError(commit.stderr.strip() or "git commit failed") + rev = git_cmd(repo, "rev-parse", "HEAD") + if rev.returncode != 0: + raise RuntimeError(rev.stderr.strip() or "git rev-parse failed") + return rev.stdout.strip() + + +def copy_repo(src: Path, dest: Path) -> None: + if dest.exists(): + shutil.rmtree(dest) + shutil.copytree( + src, + dest, + ignore=shutil.ignore_patterns(".git", "__pycache__", "*.pyc", "*.fixed.js", "*.fixed.py"), + ) + + +def run_shell(command: str, cwd: Path, timeout: int) -> dict: + started = time.monotonic() + try: + proc = subprocess.run( + command, + cwd=cwd, + shell=True, + text=True, + capture_output=True, + timeout=timeout, + stdin=subprocess.DEVNULL, + ) + timed_out = False + code = proc.returncode + stdout = proc.stdout + stderr = proc.stderr + except subprocess.TimeoutExpired as exc: + timed_out = True + code = 124 + stdout = exc.stdout or "" + stderr = exc.stderr or "" + if isinstance(stdout, bytes): + stdout = stdout.decode("utf-8", "replace") + if isinstance(stderr, bytes): + stderr = stderr.decode("utf-8", "replace") + return { + "command": command, + "exit_code": code, + "passed": code == 0 and not timed_out, + "timed_out": timed_out, + "wall_s": round(time.monotonic() - started, 3), + "stdout_tail": (stdout or "")[-2000:], + "stderr_tail": (stderr or "")[-2000:], + } + + +def diff_against(repo: Path, baseline: str) -> tuple[list[str], str]: + names = git_cmd(repo, "diff", "--name-only", baseline) + stat = git_cmd(repo, "diff", "--stat", baseline) + status = git_cmd(repo, "status", "--porcelain") + files = [line for line in (names.stdout or "").splitlines() if line] + for line in (status.stdout or "").splitlines(): + if line.startswith("??"): + path = line[3:].strip() + if path and path not in files: + files.append(path) + stat_text = stat.stdout or "" + if len(stat_text) > 8000: + stat_text = stat_text[:8000] + "\n…\n" + return files, stat_text + + +def collect_calls(events: Path) -> tuple[list[dict], list[str], str | None]: + calls = [] + for path in sorted(events.glob("agent-*.jsonl")) + sorted(events.glob("agent-*.json")): + text = path.read_text(encoding="utf-8", errors="replace") + calls.append(parse_agent_stream(text, str(path))) + usage_path = events / "executor-usage.json" + if usage_path.is_file(): + try: + payload = json.loads(usage_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + payload = {} + for call in payload.get("calls") or []: + if "usage" in call: + calls.append(call) + session_ids = [] + reported = None + for call in calls: + if call.get("session_id") and call["session_id"] not in session_ids: + session_ids.append(call["session_id"]) + if call.get("model") and reported is None: + reported = call["model"] + return calls, session_ids, reported + + +def tool_version(events: Path) -> dict: + path = events / "tool-version.json" + if not path.is_file(): + return {} + try: + data = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return {} + return data if isinstance(data, dict) else {} + + +def kill_process_group(pid: int) -> None: + try: + os.killpg(pid, signal.SIGKILL) + except ProcessLookupError: + return + except PermissionError: + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + return + + +def run_arm(arm: str, workdir: Path, task_md: Path, events: Path, env: dict) -> dict: + script = ARMS_DIR / f"{arm}.sh" + if not script.is_file(): + raise FileNotFoundError(f"missing arm script {script}") + events.mkdir(parents=True, exist_ok=True) + proc = subprocess.Popen( + ["bash", str(script), str(workdir), str(task_md), str(events)], + cwd=workdir, + env=env, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + start_new_session=True, + text=True, + ) + timed_out = False + try: + stdout, stderr = proc.communicate(timeout=RUN_TIMEOUT_S) + code = proc.returncode + except subprocess.TimeoutExpired: + timed_out = True + kill_process_group(proc.pid) + stdout, stderr = proc.communicate() + code = 124 + (events / "arm-stdout.txt").write_text(stdout or "", encoding="utf-8") + (events / "arm-stderr.txt").write_text(stderr or "", encoding="utf-8") + return {"exit_code": code, "timed_out": timed_out} + + +def prepare_env(meta: dict, base_env: dict) -> dict: + env = dict(base_env) + env["BENCH_VISIBLE_CMD"] = meta["visible_command"] + env["BENCH_TASK_ID"] = meta["id"] + env["BENCH_MODEL"] = MODEL + for key in list(env): + if "HIDDEN" in key.upper(): + del env[key] + return env + + +def one_run(meta: dict, arm: str, repeat: int, events_root: Path, base_env: dict) -> dict: + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S") + run_name = f"{stamp}-{meta['id']}-{arm}-r{repeat}" + run_dir = TMP_DIR / run_name + workdir = run_dir / "repo" + events = events_root / run_name + if run_dir.exists(): + shutil.rmtree(run_dir) + run_dir.mkdir(parents=True) + copy_repo(Path(meta["repo"]), workdir) + task_md = run_dir / "TASK.md" + shutil.copyfile(meta["task_md"], task_md) + baseline = init_repo(workdir) + start = utc_now() + t0 = time.monotonic() + arm_result = run_arm(arm, workdir, task_md, events, prepare_env(meta, base_env)) + measure = baseline + marker = events / "measure-from.txt" + if marker.is_file(): + marked = marker.read_text(encoding="utf-8").strip() + if marked: + measure = marked + files, stat = diff_against(workdir, measure) + visible = run_shell(meta["visible_command"], workdir, TEST_TIMEOUT_S) + hidden_dest = workdir / meta.get("hidden_dest", "_hidden_acceptance") + if hidden_dest.exists(): + shutil.rmtree(hidden_dest) + shutil.copytree(meta["hidden"], hidden_dest) + hidden = run_shell(meta["hidden_command"], workdir, TEST_TIMEOUT_S) + wall_s = round(time.monotonic() - t0, 3) + calls, session_ids, reported = collect_calls(events) + tokens = sum_calls(calls) + raw_paths = sorted(str(path.relative_to(ROOT)) for path in events.rglob("*") if path.is_file()) + return { + "type": "run", + "task": meta["id"], + "arm": arm, + "repeat": repeat, + "start": start, + "end": utc_now(), + "wall_s": wall_s, + "exit_code": arm_result["exit_code"], + "timed_out": arm_result["timed_out"], + "skipped": False, + "tokens": {"calls": calls, **tokens}, + "cursor_agent_calls": len(calls), + "success": bool(hidden["passed"]), + "visible_tests": visible, + "hidden_tests": hidden, + "files_changed": files, + "diff_stat": stat, + "session_ids": session_ids, + "model": MODEL, + "model_reported": reported, + "versions": {"arm": tool_version(events)}, + "raw_events": raw_paths, + "workdir": str(workdir), + "baseline": baseline, + "measure_from": measure, + } + + +def skipped_run(meta: dict, arm: str, repeat: int, reason: str) -> dict: + now = utc_now() + return { + "type": "run", + "task": meta["id"], + "arm": arm, + "repeat": repeat, + "start": now, + "end": now, + "wall_s": 0, + "exit_code": None, + "timed_out": False, + "skipped": True, + "skip_reason": reason, + "tokens": {"calls": [], **sum_calls([])}, + "cursor_agent_calls": 0, + "success": None, + "visible_tests": None, + "hidden_tests": None, + "files_changed": [], + "diff_stat": "", + "session_ids": [], + "model": MODEL, + "model_reported": None, + "versions": {"arm": {"name": arm}}, + "raw_events": [], + } + + +def git_rev(path: Path) -> str | None: + if not path.is_dir(): + return None + proc = subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + text=True, + capture_output=True, + check=False, + ) + if proc.returncode != 0: + return None + return proc.stdout.strip() + + +def cpu_model() -> str: + try: + for line in Path("/proc/cpuinfo").read_text(encoding="utf-8", errors="replace").splitlines(): + if line.lower().startswith("model name"): + return line.split(":", 1)[1].strip() + except OSError: + pass + return platform.processor() or "unknown" + + +def command_version(cmd: list[str]) -> str | None: + proc = subprocess.run(cmd, text=True, capture_output=True, check=False) + if proc.returncode != 0 and not (proc.stdout or proc.stderr): + return None + text = (proc.stdout or proc.stderr).strip().splitlines() + return text[0] if text else None + + +def environment_record(args: argparse.Namespace) -> dict: + return { + "type": "environment", + "date": utc_now(), + "cpu": cpu_model(), + "cpu_count": os.cpu_count(), + "os": platform.platform(), + "python": platform.python_version(), + "node": command_version(["node", "--version"]), + "cursor_agent": command_version(["cursor-agent", "--version"]), + "git": command_version(["git", "--version"]), + "commits": { + "superpowers": git_rev(Path(os.environ.get("SUPERPOWERS_DIR", ""))), + "openflow": git_rev(Path("")), + "openswarm": git_rev(Path(os.environ["OPENSWARM_DIR"])) if os.environ.get("OPENSWARM_DIR") else None, + }, + "openflow_tgz": os.environ.get("OPENFLOW_TGZ"), + "openswarm_dir": os.environ.get("OPENSWARM_DIR"), + "superpowers_dir": os.environ.get("SUPERPOWERS_DIR", ""), + "model": MODEL, + "cursor_flags": ["-p", "--trust", "--force", "--model", MODEL, "--output-format", "stream-json"], + "run_timeout_s": RUN_TIMEOUT_S, + "command": [sys.executable, *sys.argv], + "arms": args.arms, + "tasks": args.tasks, + "repeats": args.repeats, + "concurrency": args.concurrency, + "out": args.out, + } + + +def append_jsonl(path: Path, row: dict, lock: threading.Lock) -> None: + line = json.dumps(row, ensure_ascii=False) + "\n" + with lock: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("a", encoding="utf-8") as handle: + handle.write(line) + handle.flush() + + +def parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser(description="End-to-end task-completion harness") + parser.add_argument("--arms", required=True, help="comma-separated arm names, or a single name") + parser.add_argument("--tasks", default="all", help="all, or comma-separated task ids") + parser.add_argument("--repeats", type=int, default=3) + parser.add_argument("--concurrency", type=int, default=4) + parser.add_argument("--out", required=True, help="jsonl path, for example results/e2e-.jsonl") + return parser.parse_args(argv) + + +def main(argv: list[str]) -> int: + args = parse_args(argv) + if args.repeats < 1: + raise SystemExit("--repeats must be >= 1") + if args.concurrency < 1: + raise SystemExit("--concurrency must be >= 1") + arms = [part.strip() for part in args.arms.split(",") if part.strip()] + for arm in arms: + if arm != "omo" and not (ARMS_DIR / f"{arm}.sh").is_file(): + raise SystemExit(f"unknown arm: {arm}") + tasks = load_tasks(args.tasks) + out = Path(args.out) + if not out.is_absolute(): + out = ROOT / out + events_root = out.parent / "raw" + events_root.mkdir(parents=True, exist_ok=True) + TMP_DIR.mkdir(parents=True, exist_ok=True) + done = existing_keys(out) + lock = threading.Lock() + if not any(line.strip().startswith('{"type": "environment"') or '"type": "environment"' in line + for line in (out.read_text(encoding="utf-8") if out.is_file() else "").splitlines()): + append_jsonl(out, environment_record(args), lock) + + jobs = [] + for task in tasks: + for arm in arms: + for repeat in range(1, args.repeats + 1): + if (task["id"], arm, repeat) in done: + print(f"skip {task['id']} {arm} r{repeat}", flush=True) + continue + jobs.append((task, arm, repeat)) + if not jobs: + print(f"nothing to run; {out}") + return 0 + + def work(job): + task, arm, repeat = job + print(f"start {task['id']} {arm} r{repeat}", flush=True) + if arm == "omo": + row = skipped_run( + task, + arm, + repeat, + "skipped: needs an OpenCode-compatible provider subscription or paid API key", + ) + else: + row = one_run(task, arm, repeat, events_root, os.environ.copy()) + append_jsonl(out, row, lock) + print( + f"done {task['id']} {arm} r{repeat} success={row['success']} wall_s={row['wall_s']}", + flush=True, + ) + return row + + failures = 0 + with ThreadPoolExecutor(max_workers=args.concurrency) as pool: + futures = [pool.submit(work, job) for job in jobs] + for future in as_completed(futures): + try: + future.result() + except Exception as exc: # noqa: BLE001 — one cell must not drop the matrix + failures += 1 + print(f"cell failed: {exc}", file=sys.stderr, flush=True) + print(f"wrote {out}") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bench/e2e/tasks/config-deep-merge/TASK.md b/bench/e2e/tasks/config-deep-merge/TASK.md new file mode 100644 index 0000000..ff62add --- /dev/null +++ b/bench/e2e/tasks/config-deep-merge/TASK.md @@ -0,0 +1,7 @@ +# deep_merge replaces nested objects instead of merging them + +`deep_merge(base, override)` in `deepmerge.py` returns a new configuration dict. + +When the same key holds a dict on both sides, merge those dicts key by key, all the way down. Every other value from the override replaces the base value, including lists and `None`. Keys that exist only on the base stay in place. Do not mutate `base` or `override`. A non-dict argument raises `TypeError`. + +A flat override already works, and the visible tests (`python3 -m unittest tests/test_deepmerge.py`) pass. Nested server settings are replaced as a whole, so unspecified keys inside them disappear. Please merge nested dicts. diff --git a/bench/e2e/tasks/config-deep-merge/hidden/test_deepmerge.py b/bench/e2e/tasks/config-deep-merge/hidden/test_deepmerge.py new file mode 100644 index 0000000..70978ff --- /dev/null +++ b/bench/e2e/tasks/config-deep-merge/hidden/test_deepmerge.py @@ -0,0 +1,41 @@ +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from deepmerge import deep_merge + + +class HiddenDeepMergeTests(unittest.TestCase): + def test_nested_dicts_merge_key_by_key(self): + base = {"server": {"host": "localhost", "port": 80}, "debug": False} + override = {"server": {"port": 443}} + merged = deep_merge(base, override) + self.assertEqual(merged, {"server": {"host": "localhost", "port": 443}, "debug": False}) + + def test_lists_are_replaced_rather_than_concatenated(self): + base = {"flags": ["a", "b"], "name": "old"} + merged = deep_merge(base, {"flags": ["c"]}) + self.assertEqual(merged["flags"], ["c"]) + self.assertEqual(merged["name"], "old") + + def test_inputs_are_not_mutated_when_nested(self): + base = {"server": {"port": 80}} + override = {"server": {"port": 443, "tls": True}} + deep_merge(base, override) + self.assertEqual(base, {"server": {"port": 80}}) + self.assertEqual(override, {"server": {"port": 443, "tls": True}}) + + def test_none_in_the_override_replaces_the_value(self): + merged = deep_merge({"a": 1, "b": {"c": 2}}, {"a": None}) + self.assertIsNone(merged["a"]) + self.assertEqual(merged["b"], {"c": 2}) + + def test_regression_flat_override_still_wins(self): + merged = deep_merge({"a": 1, "b": 2}, {"b": 9, "c": 3}) + self.assertEqual(merged, {"a": 1, "b": 9, "c": 3}) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/config-deep-merge/meta.json b/bench/e2e/tasks/config-deep-merge/meta.json new file mode 100644 index 0000000..0ed69ee --- /dev/null +++ b/bench/e2e/tasks/config-deep-merge/meta.json @@ -0,0 +1,8 @@ +{ + "id": "config-deep-merge", + "language": "python", + "difficulty": "low", + "visible_command": "python3 -m unittest tests/test_deepmerge.py", + "hidden_command": "python3 -m unittest _hidden_acceptance/test_deepmerge.py", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/config-deep-merge/repo/deepmerge.py b/bench/e2e/tasks/config-deep-merge/repo/deepmerge.py new file mode 100644 index 0000000..b6fe2af --- /dev/null +++ b/bench/e2e/tasks/config-deep-merge/repo/deepmerge.py @@ -0,0 +1,9 @@ +"""Merge two configuration dictionaries.""" + +def deep_merge(base, override): + """Return a new dict with override applied on top of base.""" + if not isinstance(base, dict) or not isinstance(override, dict): + raise TypeError("both arguments must be dicts") + merged = dict(base) + merged.update(override) + return merged diff --git a/bench/e2e/tasks/config-deep-merge/repo/tests/test_deepmerge.py b/bench/e2e/tasks/config-deep-merge/repo/tests/test_deepmerge.py new file mode 100644 index 0000000..e493794 --- /dev/null +++ b/bench/e2e/tasks/config-deep-merge/repo/tests/test_deepmerge.py @@ -0,0 +1,29 @@ +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from deepmerge import deep_merge + + +class VisibleDeepMergeTests(unittest.TestCase): + def test_top_level_override_and_addition(self): + base = {"a": 1, "b": 2} + merged = deep_merge(base, {"b": 9, "c": 3}) + self.assertEqual(merged, {"a": 1, "b": 9, "c": 3}) + + def test_inputs_are_left_unchanged_for_a_flat_merge(self): + base = {"a": 1} + override = {"b": 2} + deep_merge(base, override) + self.assertEqual(base, {"a": 1}) + self.assertEqual(override, {"b": 2}) + + def test_rejects_non_dicts(self): + with self.assertRaises(TypeError): + deep_merge(["a"], {"a": 1}) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/config-deep-merge/solution.patch b/bench/e2e/tasks/config-deep-merge/solution.patch new file mode 100644 index 0000000..18bd136 --- /dev/null +++ b/bench/e2e/tasks/config-deep-merge/solution.patch @@ -0,0 +1,22 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/config-deep-merge-orig/deepmerge.py tmp/patchgen/config-deep-merge/deepmerge.py +--- a/deepmerge.py 2026-10-17 19:55:34.299313735 +0800 ++++ b/deepmerge.py 2026-10-17 19:56:17.463313735 +0800 +@@ -1,9 +1,16 @@ + """Merge two configuration dictionaries.""" + ++import copy ++ ++ + def deep_merge(base, override): + """Return a new dict with override applied on top of base.""" + if not isinstance(base, dict) or not isinstance(override, dict): + raise TypeError("both arguments must be dicts") +- merged = dict(base) +- merged.update(override) ++ merged = {key: copy.deepcopy(value) for key, value in base.items()} ++ for key, value in override.items(): ++ if key in merged and isinstance(merged[key], dict) and isinstance(value, dict): ++ merged[key] = deep_merge(merged[key], value) ++ else: ++ merged[key] = copy.deepcopy(value) + return merged diff --git a/bench/e2e/tasks/csv-quoting/TASK.md b/bench/e2e/tasks/csv-quoting/TASK.md new file mode 100644 index 0000000..54be745 --- /dev/null +++ b/bench/e2e/tasks/csv-quoting/TASK.md @@ -0,0 +1,7 @@ +# CSV fields break when they contain a comma + +`format_csv` and `parse_csv` in `csvquote.py` should round-trip a list of rows. Each row is a list of strings. + +A field that contains a comma, a double quote, or a newline has to be wrapped in double quotes. A double quote inside such a field is written as two double quotes. Empty fields stay empty, so `a,,c` is three fields. A trailing newline after the last row is fine and is not an extra row. + +Plain rows with no special characters already round-trip, and the visible tests (`python3 -m unittest tests/test_csvquote.py`) pass. Quoted fields do not. Please fix both the writer and the reader so a format-then-parse cycle returns the original rows. diff --git a/bench/e2e/tasks/csv-quoting/hidden/test_csvquote.py b/bench/e2e/tasks/csv-quoting/hidden/test_csvquote.py new file mode 100644 index 0000000..3961497 --- /dev/null +++ b/bench/e2e/tasks/csv-quoting/hidden/test_csvquote.py @@ -0,0 +1,38 @@ +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from csvquote import format_csv, parse_csv + + +class HiddenCsvTests(unittest.TestCase): + def test_comma_inside_a_field_roundtrips(self): + rows = [["city", "note"], ["Paris", "left, right"]] + self.assertEqual(parse_csv(format_csv(rows)), rows) + + def test_quotes_inside_a_field_are_doubled(self): + rows = [["title"], ['say "hello"']] + text = format_csv(rows) + self.assertIn('""', text) + self.assertEqual(parse_csv(text), rows) + + def test_newline_inside_a_field_stays_one_row(self): + rows = [["line1\nline2", "tail"]] + parsed = parse_csv(format_csv(rows)) + self.assertEqual(parsed, rows) + self.assertEqual(len(parsed), 1) + self.assertIn("\n", parsed[0][0]) + + def test_empty_field_is_preserved(self): + rows = [["a", "", "c"]] + self.assertEqual(parse_csv(format_csv(rows)), rows) + + def test_regression_plain_row_still_roundtrips(self): + rows = [["name", "age"], ["ada", "36"]] + self.assertEqual(parse_csv(format_csv(rows)), rows) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/csv-quoting/meta.json b/bench/e2e/tasks/csv-quoting/meta.json new file mode 100644 index 0000000..6db5d56 --- /dev/null +++ b/bench/e2e/tasks/csv-quoting/meta.json @@ -0,0 +1,8 @@ +{ + "id": "csv-quoting", + "language": "python", + "difficulty": "medium", + "visible_command": "python3 -m unittest tests/test_csvquote.py", + "hidden_command": "python3 -m unittest _hidden_acceptance/test_csvquote.py", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/csv-quoting/repo/csvquote.py b/bench/e2e/tasks/csv-quoting/repo/csvquote.py new file mode 100644 index 0000000..2292293 --- /dev/null +++ b/bench/e2e/tasks/csv-quoting/repo/csvquote.py @@ -0,0 +1,24 @@ +"""Small CSV reader and writer. + +Fields that contain a comma, a double quote, or a newline must be quoted. +A quote inside a quoted field is written as two quotes. +""" + + +def format_csv(rows): + lines = [] + for row in rows: + lines.append(",".join(row)) + text = "\n".join(lines) + if rows: + text += "\n" + return text + + +def parse_csv(text): + if text == "": + return [] + rows = [] + for line in text.splitlines(): + rows.append(line.split(",")) + return rows diff --git a/bench/e2e/tasks/csv-quoting/repo/tests/test_csvquote.py b/bench/e2e/tasks/csv-quoting/repo/tests/test_csvquote.py new file mode 100644 index 0000000..b9a7cbd --- /dev/null +++ b/bench/e2e/tasks/csv-quoting/repo/tests/test_csvquote.py @@ -0,0 +1,23 @@ +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from csvquote import format_csv, parse_csv + + +class VisibleCsvTests(unittest.TestCase): + def test_simple_roundtrip(self): + rows = [["name", "age"], ["ada", "36"]] + self.assertEqual(parse_csv(format_csv(rows)), rows) + + def test_empty_input(self): + self.assertEqual(parse_csv(""), []) + + def test_single_column(self): + self.assertEqual(parse_csv(format_csv([["only"]])), [["only"]]) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/csv-quoting/solution.patch b/bench/e2e/tasks/csv-quoting/solution.patch new file mode 100644 index 0000000..c5917aa --- /dev/null +++ b/bench/e2e/tasks/csv-quoting/solution.patch @@ -0,0 +1,72 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/csv-quoting-orig/csvquote.py tmp/patchgen/csv-quoting/csvquote.py +--- a/csvquote.py 2026-10-17 19:53:09.455313735 +0800 ++++ b/csvquote.py 2026-10-17 19:56:17.467313735 +0800 +@@ -5,10 +5,14 @@ + """ + + ++def _format_field(value): ++ if any(ch in value for ch in ',"\n\r'): ++ return '"' + value.replace('"', '""') + '"' ++ return value ++ ++ + def format_csv(rows): +- lines = [] +- for row in rows: +- lines.append(",".join(row)) ++ lines = [",".join(_format_field(cell) for cell in row) for row in rows] + text = "\n".join(lines) + if rows: + text += "\n" +@@ -19,6 +23,48 @@ + if text == "": + return [] + rows = [] +- for line in text.splitlines(): +- rows.append(line.split(",")) ++ row = [] ++ field = [] ++ i = 0 ++ in_quotes = False ++ while i < len(text): ++ char = text[i] ++ if in_quotes: ++ if char == '"': ++ if i + 1 < len(text) and text[i + 1] == '"': ++ field.append('"') ++ i += 2 ++ continue ++ in_quotes = False ++ i += 1 ++ continue ++ field.append(char) ++ i += 1 ++ continue ++ if char == '"': ++ in_quotes = True ++ i += 1 ++ continue ++ if char == ",": ++ row.append("".join(field)) ++ field = [] ++ i += 1 ++ continue ++ if char == "\n": ++ row.append("".join(field)) ++ rows.append(row) ++ row = [] ++ field = [] ++ i += 1 ++ continue ++ if char == "\r": ++ i += 1 ++ continue ++ field.append(char) ++ i += 1 ++ if in_quotes: ++ raise ValueError("unterminated quote") ++ if field or row: ++ row.append("".join(field)) ++ rows.append(row) + return rows diff --git a/bench/e2e/tasks/date-range-parse/TASK.md b/bench/e2e/tasks/date-range-parse/TASK.md new file mode 100644 index 0000000..0a60c61 --- /dev/null +++ b/bench/e2e/tasks/date-range-parse/TASK.md @@ -0,0 +1,9 @@ +# Date ranges drop the end day + +`parse_date_range` in `daterange.py` reads a span written as `YYYY-MM-DD..YYYY-MM-DD` and returns the list of `datetime.date` values inside it. + +The range is inclusive of both endpoints. A span whose start and end are the same day is a one-element list of that day. February 29 is valid in a leap year and must be accepted as either endpoint. The dates come back in calendar order. + +Invalid text, or an end date that falls before the start, raises `ValueError`. + +A report from billing says a range such as the first through the fourth of a month is missing its last day, and a same-day range comes back empty. Visible tests (`python3 -m unittest tests/test_daterange.py`) pass. Please fix the parser. diff --git a/bench/e2e/tasks/date-range-parse/hidden/test_daterange.py b/bench/e2e/tasks/date-range-parse/hidden/test_daterange.py new file mode 100644 index 0000000..e6060c9 --- /dev/null +++ b/bench/e2e/tasks/date-range-parse/hidden/test_daterange.py @@ -0,0 +1,36 @@ +import sys +import unittest +from datetime import date +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from daterange import parse_date_range + + +class HiddenDateRangeTests(unittest.TestCase): + def test_end_date_is_included(self): + days = parse_date_range("2024-05-01..2024-05-04") + self.assertEqual(days[-1], date(2024, 5, 4)) + self.assertEqual(len(days), 4) + + def test_same_day_range_has_one_date(self): + days = parse_date_range("2024-07-15..2024-07-15") + self.assertEqual(days, [date(2024, 7, 15)]) + + def test_leap_day_is_a_real_endpoint(self): + days = parse_date_range("2024-02-29..2024-02-29") + self.assertEqual(days, [date(2024, 2, 29)]) + + def test_month_boundary_includes_both_ends(self): + days = parse_date_range("2024-01-31..2024-02-01") + self.assertEqual(days, [date(2024, 1, 31), date(2024, 2, 1)]) + + def test_regression_start_still_first_and_ordered(self): + days = parse_date_range("2024-06-01..2024-06-03") + self.assertEqual(days[0], date(2024, 6, 1)) + self.assertEqual(days, sorted(days)) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/date-range-parse/meta.json b/bench/e2e/tasks/date-range-parse/meta.json new file mode 100644 index 0000000..a6ed3de --- /dev/null +++ b/bench/e2e/tasks/date-range-parse/meta.json @@ -0,0 +1,8 @@ +{ + "id": "date-range-parse", + "language": "python", + "difficulty": "medium", + "visible_command": "python3 -m unittest tests/test_daterange.py", + "hidden_command": "python3 -m unittest _hidden_acceptance/test_daterange.py", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/date-range-parse/repo/daterange.py b/bench/e2e/tasks/date-range-parse/repo/daterange.py new file mode 100644 index 0000000..993bd6a --- /dev/null +++ b/bench/e2e/tasks/date-range-parse/repo/daterange.py @@ -0,0 +1,26 @@ +"""Inclusive calendar ranges written as YYYY-MM-DD..YYYY-MM-DD.""" + +from datetime import date, timedelta + + +def parse_date_range(text): + """Return each date from the start through the end, inclusive. + + Raises ValueError when the text is not two ISO dates, or the end is before the start. + """ + if ".." not in text: + raise ValueError("expected YYYY-MM-DD..YYYY-MM-DD") + start_s, end_s = text.split("..", 1) + try: + start = date.fromisoformat(start_s) + end = date.fromisoformat(end_s) + except ValueError as exc: + raise ValueError("expected YYYY-MM-DD..YYYY-MM-DD") from exc + if end < start: + raise ValueError("end before start") + days = [] + cursor = start + while cursor < end: + days.append(cursor) + cursor += timedelta(days=1) + return days diff --git a/bench/e2e/tasks/date-range-parse/repo/tests/test_daterange.py b/bench/e2e/tasks/date-range-parse/repo/tests/test_daterange.py new file mode 100644 index 0000000..6feccf9 --- /dev/null +++ b/bench/e2e/tasks/date-range-parse/repo/tests/test_daterange.py @@ -0,0 +1,27 @@ +import sys +import unittest +from datetime import date +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from daterange import parse_date_range + + +class VisibleDateRangeTests(unittest.TestCase): + def test_start_is_first(self): + days = parse_date_range("2024-05-01..2024-05-04") + self.assertEqual(days[0], date(2024, 5, 1)) + self.assertGreaterEqual(len(days), 1) + + def test_rejects_garbage(self): + with self.assertRaises(ValueError): + parse_date_range("not-a-range") + + def test_rejects_inverted(self): + with self.assertRaises(ValueError): + parse_date_range("2024-05-04..2024-05-01") + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/date-range-parse/solution.patch b/bench/e2e/tasks/date-range-parse/solution.patch new file mode 100644 index 0000000..5d67a17 --- /dev/null +++ b/bench/e2e/tasks/date-range-parse/solution.patch @@ -0,0 +1,12 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/date-range-parse-orig/daterange.py tmp/patchgen/date-range-parse/daterange.py +--- a/daterange.py 2026-10-17 19:55:27.391313735 +0800 ++++ b/daterange.py 2026-10-17 19:56:17.467313735 +0800 +@@ -20,7 +20,7 @@ + raise ValueError("end before start") + days = [] + cursor = start +- while cursor < end: ++ while cursor <= end: + days.append(cursor) + cursor += timedelta(days=1) + return days diff --git a/bench/e2e/tasks/lru-eviction/TASK.md b/bench/e2e/tasks/lru-eviction/TASK.md new file mode 100644 index 0000000..cda2e6a --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/TASK.md @@ -0,0 +1,7 @@ +# LRU cache forgets which key was used + +`LruCache` in `src/lru.js` is a map with a fixed capacity. When a `set` would exceed the capacity, the least recently used key is dropped. + +Both `get` and `set` of an existing key count as a use and should make that key the newest one. Updating a key that is already present must not grow the cache. `get` of a missing key returns `undefined` and does not change order. Capacity is an integer greater than or equal to 1. + +Right now a read does not protect a key. After filling the cache, reading one entry and inserting another still evicts the entry that was just read. Visible tests (`node --test test/lru.test.js`) pass. Please fix eviction so it follows use, not only insertion. diff --git a/bench/e2e/tasks/lru-eviction/hidden/lru.test.js b/bench/e2e/tasks/lru-eviction/hidden/lru.test.js new file mode 100644 index 0000000..8ab2f96 --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/hidden/lru.test.js @@ -0,0 +1,49 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { LruCache } from "../src/lru.js"; + +test("a get refreshes recency so the other key is evicted", () => { + const cache = new LruCache(2); + cache.set("a", 1); + cache.set("b", 2); + assert.equal(cache.get("a"), 1); + cache.set("c", 3); + assert.equal(cache.get("a"), 1); + assert.equal(cache.get("b"), undefined); + assert.equal(cache.get("c"), 3); +}); + +test("replacing an existing key refreshes recency and does not grow the cache", () => { + const cache = new LruCache(2); + cache.set("a", 1); + cache.set("b", 2); + cache.set("a", 9); + cache.set("c", 3); + assert.equal(cache.get("a"), 9); + assert.equal(cache.get("b"), undefined); + assert.equal(cache.get("c"), 3); +}); + +test("capacity 1 keeps only the newest key", () => { + const cache = new LruCache(1); + cache.set("a", 1); + cache.set("b", 2); + assert.equal(cache.get("a"), undefined); + assert.equal(cache.get("b"), 2); +}); + +test("a missing key stays missing", () => { + const cache = new LruCache(2); + cache.set("a", 1); + assert.equal(cache.get("nope"), undefined); + assert.equal(cache.get("a"), 1); +}); + +test("regression: a cold eviction still drops the oldest insertion", () => { + const cache = new LruCache(2); + cache.set("a", 1); + cache.set("b", 2); + cache.set("c", 3); + assert.equal(cache.get("a"), undefined); + assert.equal(cache.get("c"), 3); +}); diff --git a/bench/e2e/tasks/lru-eviction/meta.json b/bench/e2e/tasks/lru-eviction/meta.json new file mode 100644 index 0000000..beea8d0 --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/meta.json @@ -0,0 +1,8 @@ +{ + "id": "lru-eviction", + "language": "javascript", + "difficulty": "medium", + "visible_command": "node --test test/lru.test.js", + "hidden_command": "node --test _hidden_acceptance/lru.test.js", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/lru-eviction/repo/package.json b/bench/e2e/tasks/lru-eviction/repo/package.json new file mode 100644 index 0000000..85ffb72 --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/repo/package.json @@ -0,0 +1,9 @@ +{ + "name": "lru-box", + "version": "0.1.0", + "private": true, + "type": "module", + "scripts": { + "test": "node --test test/lru.test.js" + } +} diff --git a/bench/e2e/tasks/lru-eviction/repo/src/lru.js b/bench/e2e/tasks/lru-eviction/repo/src/lru.js new file mode 100644 index 0000000..7377a54 --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/repo/src/lru.js @@ -0,0 +1,27 @@ +/** Capacity-limited map that evicts the least recently used key. */ +export class LruCache { + constructor(capacity) { + if (!Number.isInteger(capacity) || capacity < 1) { + throw new Error("capacity must be an integer >= 1"); + } + this.capacity = capacity; + this.map = new Map(); + } + + get(key) { + if (!this.map.has(key)) return undefined; + return this.map.get(key); + } + + set(key, value) { + if (this.map.has(key)) { + this.map.set(key, value); + return; + } + if (this.map.size >= this.capacity) { + const oldest = this.map.keys().next().value; + this.map.delete(oldest); + } + this.map.set(key, value); + } +} diff --git a/bench/e2e/tasks/lru-eviction/repo/test/lru.test.js b/bench/e2e/tasks/lru-eviction/repo/test/lru.test.js new file mode 100644 index 0000000..183d5ed --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/repo/test/lru.test.js @@ -0,0 +1,23 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { LruCache } from "../src/lru.js"; + +test("stores and returns a value", () => { + const cache = new LruCache(2); + cache.set("a", 1); + assert.equal(cache.get("a"), 1); +}); + +test("evicts the oldest untouched insertion when the cache is cold", () => { + const cache = new LruCache(2); + cache.set("a", 1); + cache.set("b", 2); + cache.set("c", 3); + assert.equal(cache.get("a"), undefined); + assert.equal(cache.get("b"), 2); + assert.equal(cache.get("c"), 3); +}); + +test("rejects a capacity below 1", () => { + assert.throws(() => new LruCache(0), /capacity/); +}); diff --git a/bench/e2e/tasks/lru-eviction/solution.patch b/bench/e2e/tasks/lru-eviction/solution.patch new file mode 100644 index 0000000..c2996df --- /dev/null +++ b/bench/e2e/tasks/lru-eviction/solution.patch @@ -0,0 +1,25 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/lru-eviction-orig/src/lru.js tmp/patchgen/lru-eviction/src/lru.js +--- a/src/lru.js 2026-10-17 19:55:28.847313735 +0800 ++++ b/src/lru.js 2026-10-17 19:56:17.471313735 +0800 +@@ -10,15 +10,15 @@ + + get(key) { + if (!this.map.has(key)) return undefined; +- return this.map.get(key); ++ const value = this.map.get(key); ++ this.map.delete(key); ++ this.map.set(key, value); ++ return value; + } + + set(key, value) { +- if (this.map.has(key)) { +- this.map.set(key, value); +- return; +- } +- if (this.map.size >= this.capacity) { ++ if (this.map.has(key)) this.map.delete(key); ++ else if (this.map.size >= this.capacity) { + const oldest = this.map.keys().next().value; + this.map.delete(oldest); + } diff --git a/bench/e2e/tasks/pagination-off-by-one/TASK.md b/bench/e2e/tasks/pagination-off-by-one/TASK.md new file mode 100644 index 0000000..deecaaa --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/TASK.md @@ -0,0 +1,16 @@ +# Page windows start one page too late + +`paginate(items, page, pageSize)` in `src/paginate.js` returns a 1-based slice of an array: + +`{ items, page, pageSize, total, totalPages }` + +Callers report that asking for page 1 skips the first window and returns the second one instead. The rest of the contract is: + +- `page` and `pageSize` are integers greater than or equal to 1. Anything else throws. +- Page 1 is the first `pageSize` items. Page 2 is the next window, with no gap and no overlap. +- The last page may be shorter than `pageSize`. +- `total` is the full input length. +- `totalPages` is how many windows are needed to cover every item, and 0 when the input is empty. +- A page past the end returns an empty `items` array. `totalPages` does not change. + +Visible tests (`node --test test/paginate.test.js`) already pass. Please fix the implementation without changing the return shape. diff --git a/bench/e2e/tasks/pagination-off-by-one/hidden/paginate.test.js b/bench/e2e/tasks/pagination-off-by-one/hidden/paginate.test.js new file mode 100644 index 0000000..d4026e3 --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/hidden/paginate.test.js @@ -0,0 +1,31 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { paginate } from "../src/paginate.js"; + +test("page 1 is the first window, not the second", () => { + const result = paginate(["a", "b", "c", "d", "e"], 1, 2); + assert.deepEqual(result.items, ["a", "b"]); + assert.equal(result.totalPages, 3); + assert.equal(result.total, 5); +}); + +test("the last page keeps a short remainder", () => { + const result = paginate(["a", "b", "c", "d", "e"], 3, 2); + assert.deepEqual(result.items, ["e"]); + assert.equal(result.page, 3); +}); + +test("page 2 starts immediately after page 1", () => { + const items = ["a", "b", "c", "d", "e", "f", "g"]; + assert.deepEqual(paginate(items, 2, 3).items, ["d", "e", "f"]); +}); + +test("a page past the end is an empty window and totalPages still covers every item", () => { + const result = paginate(["a", "b", "c"], 4, 2); + assert.deepEqual(result.items, []); + assert.equal(result.totalPages, 2); +}); + +test("regression: page 0 still throws", () => { + assert.throws(() => paginate(["a", "b"], 0, 2), /page/); +}); diff --git a/bench/e2e/tasks/pagination-off-by-one/meta.json b/bench/e2e/tasks/pagination-off-by-one/meta.json new file mode 100644 index 0000000..76c47e0 --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/meta.json @@ -0,0 +1,8 @@ +{ + "id": "pagination-off-by-one", + "language": "javascript", + "difficulty": "medium", + "visible_command": "node --test test/paginate.test.js", + "hidden_command": "node --test _hidden_acceptance/paginate.test.js", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/pagination-off-by-one/repo/package.json b/bench/e2e/tasks/pagination-off-by-one/repo/package.json new file mode 100644 index 0000000..3815c58 --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/repo/package.json @@ -0,0 +1,9 @@ +{ + "name": "page-window", + "version": "0.1.0", + "private": true, + "type": "module", + "scripts": { + "test": "node --test test/paginate.test.js" + } +} diff --git a/bench/e2e/tasks/pagination-off-by-one/repo/src/paginate.js b/bench/e2e/tasks/pagination-off-by-one/repo/src/paginate.js new file mode 100644 index 0000000..04c1a44 --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/repo/src/paginate.js @@ -0,0 +1,17 @@ +/** + * 1-based page window over an array. + * page and pageSize are integers >= 1. + */ +export function paginate(items, page, pageSize) { + if (!Number.isInteger(page) || page < 1) { + throw new Error("page must be an integer >= 1"); + } + if (!Number.isInteger(pageSize) || pageSize < 1) { + throw new Error("pageSize must be an integer >= 1"); + } + const total = items.length; + const totalPages = total === 0 ? 0 : Math.ceil(total / pageSize); + const start = page * pageSize; + const slice = items.slice(start, start + pageSize); + return { items: slice, page, pageSize, total, totalPages }; +} diff --git a/bench/e2e/tasks/pagination-off-by-one/repo/test/paginate.test.js b/bench/e2e/tasks/pagination-off-by-one/repo/test/paginate.test.js new file mode 100644 index 0000000..9febcf6 --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/repo/test/paginate.test.js @@ -0,0 +1,19 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { paginate } from "../src/paginate.js"; + +test("rejects a page below 1", () => { + assert.throws(() => paginate(["a"], 0, 1), /page/); +}); + +test("rejects a non-positive page size", () => { + assert.throws(() => paginate(["a"], 1, 0), /pageSize/); +}); + +test("echoes total, page and pageSize for an even split", () => { + const result = paginate(["a", "b", "c", "d"], 1, 2); + assert.equal(result.total, 4); + assert.equal(result.page, 1); + assert.equal(result.pageSize, 2); + assert.equal(result.totalPages, 2); +}); diff --git a/bench/e2e/tasks/pagination-off-by-one/solution.patch b/bench/e2e/tasks/pagination-off-by-one/solution.patch new file mode 100644 index 0000000..620ac12 --- /dev/null +++ b/bench/e2e/tasks/pagination-off-by-one/solution.patch @@ -0,0 +1,12 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/pagination-off-by-one-orig/src/paginate.js tmp/patchgen/pagination-off-by-one/src/paginate.js +--- a/src/paginate.js 2026-10-17 19:55:26.691313735 +0800 ++++ b/src/paginate.js 2026-10-17 19:56:17.471313735 +0800 +@@ -11,7 +11,7 @@ + } + const total = items.length; + const totalPages = total === 0 ? 0 : Math.ceil(total / pageSize); +- const start = page * pageSize; ++ const start = (page - 1) * pageSize; + const slice = items.slice(start, start + pageSize); + return { items: slice, page, pageSize, total, totalPages }; + } diff --git a/bench/e2e/tasks/path-normalise/TASK.md b/bench/e2e/tasks/path-normalise/TASK.md new file mode 100644 index 0000000..8d7d498 --- /dev/null +++ b/bench/e2e/tasks/path-normalise/TASK.md @@ -0,0 +1,12 @@ +# normalise leaves parent segments in place + +`normalise` in `src/normalise.js` cleans a POSIX-style path string. + +- Drop `.` segments. +- `..` pops the previous real segment. +- An absolute path (one that starts with `/`) never climbs above the root. Extra `..` at the root are discarded, and the result stays absolute (`/` when nothing remains). +- A relative path may start with `..` when it would leave the relative root. `a/../../b` is `../b`. +- Empty segments from a trailing or doubled slash are not kept. +- `.` alone is `.`. A non-string throws `TypeError`. + +Dot segments already collapse, and the visible tests (`node --test test/normalise.test.js`) pass. Parent segments do not. Please finish the normalisation. diff --git a/bench/e2e/tasks/path-normalise/hidden/normalise.test.js b/bench/e2e/tasks/path-normalise/hidden/normalise.test.js new file mode 100644 index 0000000..d308d37 --- /dev/null +++ b/bench/e2e/tasks/path-normalise/hidden/normalise.test.js @@ -0,0 +1,28 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { normalise } from "../src/normalise.js"; + +test("parent segments collapse inside the path", () => { + assert.equal(normalise("a/b/../c"), "a/c"); + assert.equal(normalise("a/b/c/../../d"), "a/d"); +}); + +test("an absolute path does not climb above the root", () => { + assert.equal(normalise("/a/../../b"), "/b"); + assert.equal(normalise("/../.."), "/"); +}); + +test("a relative path may keep a leading parent", () => { + assert.equal(normalise("a/../../b"), "../b"); + assert.equal(normalise("../a"), "../a"); +}); + +test("a trailing slash is not a segment", () => { + assert.equal(normalise("a/b/"), "a/b"); + assert.equal(normalise("/a/b/"), "/a/b"); +}); + +test("regression: a dot segment is still dropped", () => { + assert.equal(normalise("a/./b"), "a/b"); + assert.equal(normalise("."), "."); +}); diff --git a/bench/e2e/tasks/path-normalise/meta.json b/bench/e2e/tasks/path-normalise/meta.json new file mode 100644 index 0000000..5b46072 --- /dev/null +++ b/bench/e2e/tasks/path-normalise/meta.json @@ -0,0 +1,8 @@ +{ + "id": "path-normalise", + "language": "javascript", + "difficulty": "medium", + "visible_command": "node --test test/normalise.test.js", + "hidden_command": "node --test _hidden_acceptance/normalise.test.js", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/path-normalise/repo/package.json b/bench/e2e/tasks/path-normalise/repo/package.json new file mode 100644 index 0000000..aa39781 --- /dev/null +++ b/bench/e2e/tasks/path-normalise/repo/package.json @@ -0,0 +1,9 @@ +{ + "name": "path-normalise", + "version": "0.1.0", + "private": true, + "type": "module", + "scripts": { + "test": "node --test test/normalise.test.js" + } +} diff --git a/bench/e2e/tasks/path-normalise/repo/src/normalise.js b/bench/e2e/tasks/path-normalise/repo/src/normalise.js new file mode 100644 index 0000000..ac921a4 --- /dev/null +++ b/bench/e2e/tasks/path-normalise/repo/src/normalise.js @@ -0,0 +1,15 @@ +/** Collapse redundant segments in a POSIX-style path. */ +export function normalise(input) { + if (typeof input !== "string") { + throw new TypeError("path must be a string"); + } + const absolute = input.startsWith("/"); + const parts = input.split("/"); + const out = []; + for (const part of parts) { + if (part === "" || part === ".") continue; + out.push(part); + } + if (absolute) return "/" + out.join("/"); + return out.join("/") || "."; +} diff --git a/bench/e2e/tasks/path-normalise/repo/test/normalise.test.js b/bench/e2e/tasks/path-normalise/repo/test/normalise.test.js new file mode 100644 index 0000000..d6a31ca --- /dev/null +++ b/bench/e2e/tasks/path-normalise/repo/test/normalise.test.js @@ -0,0 +1,19 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { normalise } from "../src/normalise.js"; + +test("drops a dot segment", () => { + assert.equal(normalise("a/./b"), "a/b"); +}); + +test("keeps an absolute path absolute", () => { + assert.equal(normalise("/a/b"), "/a/b"); +}); + +test("a lone dot is the current directory", () => { + assert.equal(normalise("."), "."); +}); + +test("rejects a non-string", () => { + assert.throws(() => normalise(12), TypeError); +}); diff --git a/bench/e2e/tasks/path-normalise/solution.patch b/bench/e2e/tasks/path-normalise/solution.patch new file mode 100644 index 0000000..98f420e --- /dev/null +++ b/bench/e2e/tasks/path-normalise/solution.patch @@ -0,0 +1,15 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/path-normalise-orig/src/normalise.js tmp/patchgen/path-normalise/src/normalise.js +--- a/src/normalise.js 2026-10-17 19:55:30.087313735 +0800 ++++ b/src/normalise.js 2026-10-17 19:56:17.475313735 +0800 +@@ -8,6 +8,11 @@ + const out = []; + for (const part of parts) { + if (part === "" || part === ".") continue; ++ if (part === "..") { ++ if (out.length > 0 && out[out.length - 1] !== "..") out.pop(); ++ else if (!absolute) out.push(".."); ++ continue; ++ } + out.push(part); + } + if (absolute) return "/" + out.join("/"); diff --git a/bench/e2e/tasks/retry-backoff/TASK.md b/bench/e2e/tasks/retry-backoff/TASK.md new file mode 100644 index 0000000..e29fcd9 --- /dev/null +++ b/bench/e2e/tasks/retry-backoff/TASK.md @@ -0,0 +1,15 @@ +# retry never tries again + +`retry` in `retry.py` should call `fn` until it returns or the attempt budget is spent. + +```python +retry(fn, attempts=3, base_delay=0.0, factor=2.0, sleep=time.sleep) +``` + +- `attempts` is how many times `fn` may run. It is at least 1. +- When `fn` raises, wait and try again. The wait before the next attempt is `base_delay * (factor ** failures_so_far)`, so the first wait is `base_delay`, then `base_delay * factor`, and so on. +- Pass that delay to `sleep`. Tests inject `sleep` so they do not wait on the wall clock. Keep the default as `time.sleep`. +- When the last attempt raises, propagate that exception. Do not sleep after the final failure. +- A first-try success returns immediately and does not sleep. + +Today the function calls `fn` once and returns, so a later success is never reached. Visible tests (`python3 -m unittest tests/test_retry.py`) pass. Please implement the retries. diff --git a/bench/e2e/tasks/retry-backoff/hidden/test_retry.py b/bench/e2e/tasks/retry-backoff/hidden/test_retry.py new file mode 100644 index 0000000..3422455 --- /dev/null +++ b/bench/e2e/tasks/retry-backoff/hidden/test_retry.py @@ -0,0 +1,73 @@ +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from retry import retry + + +class HiddenRetryTests(unittest.TestCase): + def test_retries_until_the_third_attempt_succeeds(self): + state = {"n": 0} + + def fn(): + state["n"] += 1 + if state["n"] < 3: + raise RuntimeError("not yet") + return "ok" + + delays = [] + result = retry(fn, attempts=3, base_delay=1, factor=2, sleep=delays.append) + self.assertEqual(result, "ok") + self.assertEqual(state["n"], 3) + self.assertEqual(delays, [1, 2]) + + def test_gives_up_with_the_last_exception(self): + delays = [] + + def fn(): + raise ValueError("boom") + + with self.assertRaises(ValueError): + retry(fn, attempts=2, base_delay=5, factor=3, sleep=delays.append) + self.assertEqual(delays, [5]) + + def test_a_single_attempt_does_not_sleep(self): + delays = [] + + def fn(): + raise RuntimeError("nope") + + with self.assertRaises(RuntimeError): + retry(fn, attempts=1, base_delay=4, factor=2, sleep=delays.append) + self.assertEqual(delays, []) + + def test_delay_uses_the_factor(self): + state = {"n": 0} + + def fn(): + state["n"] += 1 + if state["n"] < 3: + raise RuntimeError("wait") + return 1 + + delays = [] + retry(fn, attempts=4, base_delay=3, factor=4, sleep=delays.append) + self.assertEqual(delays, [3, 12]) + + def test_regression_first_try_success_does_not_sleep(self): + delays = [] + calls = [] + + def fn(): + calls.append(1) + return "ready" + + self.assertEqual(retry(fn, attempts=5, base_delay=9, sleep=delays.append), "ready") + self.assertEqual(calls, [1]) + self.assertEqual(delays, []) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/retry-backoff/meta.json b/bench/e2e/tasks/retry-backoff/meta.json new file mode 100644 index 0000000..0a5930f --- /dev/null +++ b/bench/e2e/tasks/retry-backoff/meta.json @@ -0,0 +1,8 @@ +{ + "id": "retry-backoff", + "language": "python", + "difficulty": "medium", + "visible_command": "python3 -m unittest tests/test_retry.py", + "hidden_command": "python3 -m unittest _hidden_acceptance/test_retry.py", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/retry-backoff/repo/retry.py b/bench/e2e/tasks/retry-backoff/repo/retry.py new file mode 100644 index 0000000..aa4f14a --- /dev/null +++ b/bench/e2e/tasks/retry-backoff/repo/retry.py @@ -0,0 +1,7 @@ +"""Call a function again after a failure, waiting a little longer each time.""" + +import time + + +def retry(fn, attempts=3, base_delay=0.0, factor=2.0, sleep=time.sleep): + return fn() diff --git a/bench/e2e/tasks/retry-backoff/repo/tests/test_retry.py b/bench/e2e/tasks/retry-backoff/repo/tests/test_retry.py new file mode 100644 index 0000000..80328da --- /dev/null +++ b/bench/e2e/tasks/retry-backoff/repo/tests/test_retry.py @@ -0,0 +1,26 @@ +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from retry import retry + + +class VisibleRetryTests(unittest.TestCase): + def test_returns_the_value(self): + self.assertEqual(retry(lambda: 7), 7) + + def test_successful_call_runs_once(self): + calls = [] + + def fn(): + calls.append(1) + return "ok" + + self.assertEqual(retry(fn), "ok") + self.assertEqual(calls, [1]) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/e2e/tasks/retry-backoff/solution.patch b/bench/e2e/tasks/retry-backoff/solution.patch new file mode 100644 index 0000000..462798c --- /dev/null +++ b/bench/e2e/tasks/retry-backoff/solution.patch @@ -0,0 +1,17 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/retry-backoff-orig/retry.py tmp/patchgen/retry-backoff/retry.py +--- a/retry.py 2026-10-17 19:55:33.043313735 +0800 ++++ b/retry.py 2026-10-17 19:56:17.479313735 +0800 +@@ -4,4 +4,12 @@ + + + def retry(fn, attempts=3, base_delay=0.0, factor=2.0, sleep=time.sleep): +- return fn() ++ if attempts < 1: ++ raise ValueError("attempts must be >= 1") ++ for index in range(attempts): ++ try: ++ return fn() ++ except Exception: ++ if index == attempts - 1: ++ raise ++ sleep(base_delay * (factor ** index)) diff --git a/bench/e2e/tasks/semver-compare/TASK.md b/bench/e2e/tasks/semver-compare/TASK.md new file mode 100644 index 0000000..0a3e0d6 --- /dev/null +++ b/bench/e2e/tasks/semver-compare/TASK.md @@ -0,0 +1,7 @@ +# compareSemver only looks at the major number + +`compareSemver(left, right)` in `src/semver.js` should return `-1`, `0` or `1`. + +Compare major, then minor, then patch. A pre-release (`1.0.0-alpha`) is less than the same version without a pre-release. Pre-release identifiers are separated by dots. When both identifiers are numeric, compare them as numbers (`2` before `10`). Otherwise compare them as strings. A numeric identifier is less than a non-numeric one. If one pre-release list is a prefix of the other, the shorter one is less. Build metadata after `+` is ignored. An invalid version throws an error whose message includes `invalid semver`. Leading zeroes in a numeric core component are invalid. + +Major ordering and exact equality already work. Visible tests (`node --test test/semver.test.js`) pass. Please finish the comparison. diff --git a/bench/e2e/tasks/semver-compare/hidden/semver.test.js b/bench/e2e/tasks/semver-compare/hidden/semver.test.js new file mode 100644 index 0000000..d1829c9 --- /dev/null +++ b/bench/e2e/tasks/semver-compare/hidden/semver.test.js @@ -0,0 +1,34 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { compareSemver } from "../src/semver.js"; + +test("minor and patch break ties", () => { + assert.equal(compareSemver("1.2.0", "1.3.0"), -1); + assert.equal(compareSemver("1.2.3", "1.2.4"), -1); + assert.equal(compareSemver("1.4.0", "1.2.9"), 1); +}); + +test("a pre-release is less than the matching release", () => { + assert.equal(compareSemver("1.0.0-alpha", "1.0.0"), -1); + assert.equal(compareSemver("1.0.0", "1.0.0-alpha"), 1); +}); + +test("pre-release identifiers compare numerically, then lexically", () => { + assert.equal(compareSemver("1.0.0-alpha", "1.0.0-alpha.1"), -1); + assert.equal(compareSemver("1.0.0-alpha.1", "1.0.0-alpha.beta"), -1); + assert.equal(compareSemver("1.0.0-alpha.2", "1.0.0-alpha.10"), -1); +}); + +test("build metadata does not affect order", () => { + assert.equal(compareSemver("1.0.0+build.1", "1.0.0+build.2"), 0); + assert.equal(compareSemver("1.0.0-alpha+001", "1.0.0-alpha"), 0); +}); + +test("regression: major ordering still holds", () => { + assert.equal(compareSemver("1.0.0", "2.0.0"), -1); +}); + +test("invalid versions throw", () => { + assert.throws(() => compareSemver("1.0", "1.0.0"), /invalid semver/); + assert.throws(() => compareSemver("01.0.0", "1.0.0"), /invalid semver/); +}); diff --git a/bench/e2e/tasks/semver-compare/meta.json b/bench/e2e/tasks/semver-compare/meta.json new file mode 100644 index 0000000..3fcc293 --- /dev/null +++ b/bench/e2e/tasks/semver-compare/meta.json @@ -0,0 +1,8 @@ +{ + "id": "semver-compare", + "language": "javascript", + "difficulty": "medium", + "visible_command": "node --test test/semver.test.js", + "hidden_command": "node --test _hidden_acceptance/semver.test.js", + "hidden_dest": "_hidden_acceptance" +} diff --git a/bench/e2e/tasks/semver-compare/repo/package.json b/bench/e2e/tasks/semver-compare/repo/package.json new file mode 100644 index 0000000..edc5580 --- /dev/null +++ b/bench/e2e/tasks/semver-compare/repo/package.json @@ -0,0 +1,9 @@ +{ + "name": "semver-cmp", + "version": "0.1.0", + "private": true, + "type": "module", + "scripts": { + "test": "node --test test/semver.test.js" + } +} diff --git a/bench/e2e/tasks/semver-compare/repo/src/semver.js b/bench/e2e/tasks/semver-compare/repo/src/semver.js new file mode 100644 index 0000000..647092c --- /dev/null +++ b/bench/e2e/tasks/semver-compare/repo/src/semver.js @@ -0,0 +1,22 @@ +/** + * Compare two semantic versions. + * Returns -1, 0 or 1. + */ +const CORE = /^(\d+)\.(\d+)\.(\d+)$/; + +function parse(value) { + const match = CORE.exec(value); + if (!match) throw new Error("invalid semver"); + return { + major: Number(match[1]), + minor: Number(match[2]), + patch: Number(match[3]), + }; +} + +export function compareSemver(left, right) { + const a = parse(left); + const b = parse(right); + if (a.major !== b.major) return a.major < b.major ? -1 : 1; + return 0; +} diff --git a/bench/e2e/tasks/semver-compare/repo/test/semver.test.js b/bench/e2e/tasks/semver-compare/repo/test/semver.test.js new file mode 100644 index 0000000..dee1461 --- /dev/null +++ b/bench/e2e/tasks/semver-compare/repo/test/semver.test.js @@ -0,0 +1,12 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { compareSemver } from "../src/semver.js"; + +test("equal versions compare equal", () => { + assert.equal(compareSemver("1.2.3", "1.2.3"), 0); +}); + +test("a lower major version sorts first", () => { + assert.equal(compareSemver("1.0.0", "2.0.0"), -1); + assert.equal(compareSemver("3.0.0", "1.9.9"), 1); +}); diff --git a/bench/e2e/tasks/semver-compare/solution.patch b/bench/e2e/tasks/semver-compare/solution.patch new file mode 100644 index 0000000..c5ddb30 --- /dev/null +++ b/bench/e2e/tasks/semver-compare/solution.patch @@ -0,0 +1,58 @@ +diff -ruN -x '*.fixed.js' -x '*.fixed.py' tmp/patchgen/semver-compare-orig/src/semver.js tmp/patchgen/semver-compare/src/semver.js +--- a/src/semver.js 2026-10-17 19:55:31.375313735 +0800 ++++ b/src/semver.js 2026-10-17 19:56:17.479313735 +0800 +@@ -2,21 +2,51 @@ + * Compare two semantic versions. + * Returns -1, 0 or 1. + */ +-const CORE = /^(\d+)\.(\d+)\.(\d+)$/; ++const SEMVER = ++ /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/; + + function parse(value) { +- const match = CORE.exec(value); ++ if (typeof value !== "string") throw new Error("invalid semver"); ++ const match = SEMVER.exec(value); + if (!match) throw new Error("invalid semver"); + return { + major: Number(match[1]), + minor: Number(match[2]), + patch: Number(match[3]), ++ pre: match[4] ? match[4].split(".") : [], + }; + } + ++function compareId(left, right) { ++ const leftNum = /^[0-9]+$/.test(left); ++ const rightNum = /^[0-9]+$/.test(right); ++ if (leftNum && rightNum) { ++ const a = Number(left); ++ const b = Number(right); ++ if (a !== b) return a < b ? -1 : 1; ++ return 0; ++ } ++ if (leftNum && !rightNum) return -1; ++ if (!leftNum && rightNum) return 1; ++ if (left !== right) return left < right ? -1 : 1; ++ return 0; ++} ++ + export function compareSemver(left, right) { + const a = parse(left); + const b = parse(right); +- if (a.major !== b.major) return a.major < b.major ? -1 : 1; ++ for (const key of ["major", "minor", "patch"]) { ++ if (a[key] !== b[key]) return a[key] < b[key] ? -1 : 1; ++ } ++ if (a.pre.length === 0 && b.pre.length === 0) return 0; ++ if (a.pre.length === 0) return 1; ++ if (b.pre.length === 0) return -1; ++ const count = Math.max(a.pre.length, b.pre.length); ++ for (let i = 0; i < count; i += 1) { ++ if (i >= a.pre.length) return -1; ++ if (i >= b.pre.length) return 1; ++ const cmp = compareId(a.pre[i], b.pre[i]); ++ if (cmp !== 0) return cmp; ++ } + return 0; + } diff --git a/bench/e2e/tests/test_harness.py b/bench/e2e/tests/test_harness.py new file mode 100644 index 0000000..fa385f0 --- /dev/null +++ b/bench/e2e/tests/test_harness.py @@ -0,0 +1,202 @@ +"""Harness unit tests. No model calls.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import textwrap +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from harness.metrics import parse_agent_stream, quantile, summarise, usage_total # noqa: E402 +from harness.openswarm_finish import executor_calls, extract_result_object # noqa: E402 +from run import existing_keys # noqa: E402 + + +class MetricsTests(unittest.TestCase): + def test_stream_json_result_usage_is_one_call(self): + text = "\n".join( + [ + json.dumps({"type": "system", "subtype": "init", "session_id": "s1", "model": "Grok"}), + json.dumps({"type": "assistant", "message": {"role": "assistant"}}), + json.dumps( + { + "type": "result", + "subtype": "success", + "session_id": "s1", + "usage": { + "inputTokens": 10, + "outputTokens": 4, + "cacheReadTokens": 3, + "cacheWriteTokens": 1, + }, + } + ), + ] + ) + call = parse_agent_stream(text, "agent-1.jsonl") + self.assertEqual(call["session_id"], "s1") + self.assertFalse(call["missing_usage"]) + self.assertEqual(usage_total(call["usage"]), 18) + + def test_quantile_and_iqr(self): + stat = summarise([1, 2, 3, 4, 100]) + self.assertEqual(stat["median"], 3) + self.assertEqual(stat["min"], 1) + self.assertEqual(stat["max"], 100) + self.assertEqual(stat["iqr"], quantile([1, 2, 3, 4, 100], 0.75) - quantile([1, 2, 3, 4, 100], 0.25)) + + def test_resume_keys(self): + path = ROOT / "tmp" / (self.id().replace(".", "_") + ".jsonl") + path.parent.mkdir(parents=True, exist_ok=True) + try: + path.write_text( + "\n".join( + [ + json.dumps({"type": "environment", "date": "x"}), + json.dumps({"type": "run", "task": "pagination-off-by-one", "arm": "baseline", "repeat": 1}), + ] + ) + + "\n", + encoding="utf-8", + ) + self.assertIn(("pagination-off-by-one", "baseline", 1), existing_keys(path)) + self.assertNotIn(("pagination-off-by-one", "baseline", 2), existing_keys(path)) + finally: + path.unlink(missing_ok=True) + + def test_openswarm_usage_is_not_double_counted(self): + usage = {"inputTokens": 8, "outputTokens": 2, "cacheReadTokens": 1, "cacheWriteTokens": 0} + result = { + "id": "abc", + "usage": usage, + "attempts": [{"backend": "cursor", "model": "grok-4.7-high", "meta": {"usage": usage}}], + } + calls = executor_calls(result) + self.assertEqual(len(calls), 1) + self.assertEqual(calls[0]["usage"]["inputTokens"], 8) + blob = "noise\n" + json.dumps({"ok": True}) + "\n" + json.dumps(result) + self.assertEqual(extract_result_object(blob)["id"], "abc") + + +class PipelineTests(unittest.TestCase): + def test_fake_agent_records_tokens_and_hidden_tests(self): + fake = (ROOT / "tmp" / (self.id().replace(".", "_") + "-cursor.py")).resolve() + fake.parent.mkdir(parents=True, exist_ok=True) + fake.write_text( + textwrap.dedent( + """\ + #!/usr/bin/env python3 + import json, pathlib, sys + root = pathlib.Path.cwd() + if (root / "_hidden_acceptance").exists(): + print("hidden tests were visible to the agent", file=sys.stderr) + sys.exit(3) + target = root / "src" / "paginate.js" + target.write_text(target.read_text().replace( + "const start = page * pageSize;", + "const start = (page - 1) * pageSize;", + )) + sys.stdout.write(json.dumps({ + "type": "system", "subtype": "init", "session_id": "sess-fake", "model": "grok-4.7-high" + }) + "\\n") + sys.stdout.write(json.dumps({ + "type": "result", "subtype": "success", "session_id": "sess-fake", + "usage": {"inputTokens": 11, "outputTokens": 5, "cacheReadTokens": 7, "cacheWriteTokens": 2} + }) + "\\n") + """ + ), + encoding="utf-8", + ) + fake.chmod(0o755) + out = ROOT / "tmp" / "unit-pipeline.jsonl" + report_dir = ROOT / "tmp" / "unit-report" + if out.exists(): + out.unlink() + env = os.environ.copy() + env["CURSOR_AGENT_BIN"] = str(fake) + proc = subprocess.run( + [ + sys.executable, + str(ROOT / "run.py"), + "--arms", + "baseline", + "--tasks", + "pagination-off-by-one", + "--repeats", + "1", + "--concurrency", + "1", + "--out", + str(out), + ], + cwd=ROOT, + env=env, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr) + rows = [json.loads(line) for line in out.read_text(encoding="utf-8").splitlines()] + runs = [row for row in rows if row.get("type") == "run"] + self.assertEqual(len(runs), 1) + row = runs[0] + self.assertTrue(row["success"]) + self.assertTrue(row["visible_tests"]["passed"]) + self.assertTrue(row["hidden_tests"]["passed"]) + self.assertEqual(row["cursor_agent_calls"], 1) + self.assertEqual(row["tokens"]["total"], 25) + self.assertEqual(row["session_ids"], ["sess-fake"]) + self.assertEqual(row["model"], "grok-4.7-high") + self.assertIn("src/paginate.js", row["files_changed"]) + again = subprocess.run( + [ + sys.executable, + str(ROOT / "run.py"), + "--arms", + "baseline", + "--tasks", + "pagination-off-by-one", + "--repeats", + "1", + "--concurrency", + "1", + "--out", + str(out), + ], + cwd=ROOT, + env=env, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(again.returncode, 0, again.stdout + again.stderr) + self.assertIn("skip", again.stdout) + rerows = [json.loads(line) for line in out.read_text(encoding="utf-8").splitlines() if line.strip()] + self.assertEqual(sum(1 for row in rerows if row.get("type") == "run"), 1) + report = subprocess.run( + [sys.executable, str(ROOT / "report.py"), str(out), "--out-dir", str(report_dir)], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(report.returncode, 0, report.stdout + report.stderr) + svg = (report_dir / "chart.svg").read_text(encoding="utf-8") + self.assertIn('fill="#ffffff"', svg) + self.assertEqual(svg.count("