Broad-Side: lens queue via headless agent (@openrouter/agent) — sync-priced, tool-using variant

[TheAmericanMaker]

Roadmap item (see ROADMAP.md). Explores the middle tier between Broad-Side's batch API (50% pricing, no tools, single-turn) and the full interactive CodeCartographer pipeline (agent loop + tools, driven by a host).

OpenRouter's official scaffold for this is the create-headless-agent skill (OpenRouterTeam/skills): a queue worker that runs @openrouter/agent's callModel loop per prompt with local tools (grep/file/shell), retry-with-backoff, session persistence, output-schema validation, and stop conditions including maxCost(amount).

Open questions the design must answer before this is worth building:

• Cost: sync API = ~2x batch pricing per lens. Does tool-using capability (grep first, read only what matches) recover the difference by shrinking input tokens vs Broad-Side's stuff-everything-into-context approach? Only a real comparison run answers this.
• Overlap: the tool-using variant re-implements much of what the interactive pipeline already does. Its niche would be unattended multi-turn verification (e.g., 'confirm the finding by reading the cited line'), which batch cannot do.
• Hybrid (the likelier winner): keep batch API for the six lens sweeps, then spend one sync-priced verification pass on the top N synthesis findings — read the cited file:line, confirm or discard each lead, promote survivors to verified status. This attacks the biggest weakness (unverified leads) at a fraction of full-pipeline cost.

Related: #133 (truncation repair), #135 (triage lens), #141 (multi-model). Retry-safety note: batch requests are pure, so resubmission is always safe; a tool-using variant must adopt the headless-agent rule (retry only before tool calls, since replaying a mutating tool double-executes it).

[TheAmericanMaker]

Status, 2026-09-12 — deliberately still open, not started; recording why so it is not re-derived.

The design question this issue poses cannot be answered from the code: whether a tool-using sync-priced lens recovers its ~2× price by reading less needs a paid comparison run on a real repository, and nobody has funded one. What has changed since it was filed makes the hybrid the only variant worth that spend:

• The default model's measured scorecard (Broad-Side: multi-model batch endpoints behind the lens registry #141): good recall on local, pattern-matchable defects (6/6 confirmed), the same two false positives repeated, and 0/4 of the serious lifecycle/concurrency bugs found by other means. The weakness is not coverage but verification — about a third of what the batch pass raised did not survive a human reading the cited line. A verification pass over the top-N synthesis findings is exactly the shape that attacks this; a tool-using re-run of the six sweeps is not.
• The batch side has since grown the pieces a hybrid would sit on: triage (Broad-Side: triage lens — prioritized fix queue from defect + security findings #135) ranks the findings it would verify, lens_models / model (Broad-Side: multi-model batch endpoints behind the lens registry #141) let the sweep and the verification use different models, and the truncation retry now runs as one batch per model (Broad-Side: the truncation retry is serialized, so a chatty model turns a 5-minute collect into an hour #206), so the batch pass is no longer the wall-clock problem.
• Retry-safety still divides the two: batch requests are pure and re-submittable; a tool-using pass must retry only before its first tool call. A verification pass that only reads (grep, read the cited file:line) keeps the pure property and sidesteps that rule entirely, which is one more reason to start there.

Smallest viable form when someone funds it: after collect, take the top N triage items, one sync-priced call each with read-only tools, output schema {id, verdict: confirmed|discarded|unclear, evidence}, write verified.md next to triage.md, cap with maxCost. Measure precision lift against the #141 scorecard before generalizing. Until then this stays a roadmap entry.

[TheAmericanMaker]

The comparison run this issue was waiting on, 2026-09-13, on this repository at bcb34ba (39 TypeScript source files), with a scoped key. Total spend for the whole experiment: $0.51.

Setup

1. Batch sweep (the existing product): defect, security, api, architecture lenses + synthesis + triage on the default model. Lens batches $0.25 estimated; the defect lens alone was 14 slices at ~$0.19. 24 findings came back (12 medium, 12 low).
2. Ground truth: I read the cited code for the 12 mediums and recorded a verdict for each before running any verifier (reproduced below).
3. Verification pass (prototype, ~150 lines, not product code): for each of the same 12, one sync chat completion on google/gemini-3.7-flash with three read-only tools (read_file with line ranges, grep over tracked files, list_dir), at most 8 tool calls, reasoning: {effort: "low"}, then a schema-forced JSON verdict {verdict, confidence, evidence[], reasoning}.

Ground truth (recorded first)

#	finding (all medium)	truth
1	amendment.ts: pipeline outcome resolved outside the status lock	true, minor (check-then-lock; consequence-free except across a concurrent pipeline switch)
2	broadside.ts: dangling abort listener in sleepUnlessAborted	false — the timer path removes the listener; the abort path is once
3	broadside.ts: fetchBatch sets http_status on a primitive	false — the non-object guard runs first
4	dashboard.ts: renderUsageInsights reduce on undefined tuples	false — size === 0 returns first; typed Map
5	library.ts: .git check misses worktrees	false — pathExists is access(F_OK), true for a worktree's .git file
6	orchestrator-config.ts: PathLike cast	false — no caller passes anything but a string
7	status.ts: applyHandoff indexes a missing phase	false — the key came from iterating status.phases
8	yaml.ts: loadYamlFile cast without validation	false — call sites normalize/validate
9	agent-rewriter.ts: .at(-1) below ES2022	false — target ES2022, Node ≥ 20
10	dashboard-narrator.ts: msg.content may be a string	false — assistant messages carry block arrays
11	index.ts: --namespace as the last token is silently dropped	true, low
12	server.ts: stdin listeners leak on server.close()	false — once per process

Batch pass alone, top 12 by severity: 2 true / 10 false → 17% precision. (Consistent with the #141 scorecard: pattern-matchable "unsafe cast" observations dominate the top of the list.)

Verification pass, rubric v1 (confirmed / discarded / unclear)

Confirmed {1, 6, 8, 11}; discarded {2, 3, 4, 5, 7, 9, 12}; error on 10 (tool budget exhausted without a verdict). $0.132 for 12 ($0.011 each).

• Kept both true findings; discarded 7 of the 10 false ones.
• The two false ones it kept (refactor: extract framework into core/; add invariant tests + CI #6, chore: remove dev-only artifacts from repo #8) were "confirmed" as literal statements about the code — there is a cast — not as reachable defects. A rubric problem, not a capability problem.

Verification pass, rubric v2

Added a fourth verdict, not-a-defect (the claim is literally true but no caller, input, or state can reach the failure), the rule that confirmed requires naming the trigger found in the code, and a hard tool budget (a final schema-forced call with no tools). $0.129 for 12 ($0.011 each).

#	truth	v2 verdict
1	true	confirmed
2	false	discarded
3	false	discarded
4	false	discarded
5	false	discarded
6	false	not-a-defect
7	false	not-a-defect
8	false	not-a-defect
9	false	discarded
10	false	not-a-defect
11	true	confirmed
12	false	not-a-defect

12/12 agreement with the ground truth. Precision of the confirmed set 17% → 100%; recall of true findings 100%; zero errors. Cost of verifying the 12 was two thirds of the defect lens's batch cost, and every discarded item came with the reason (the guard, the line, the config value) — which is what a reader wants anyway.

What this settles

• The variant worth building is the verification pass, not a tool-using re-run of the sweeps: the batch pass's weakness is precision, and reading the cited line with tools fixes exactly that, at ~$0.01 per finding on the default model.
• The rubric matters more than the model: v1 and v2 used the same model and tools and differed by two false positives. not-a-defect as a distinct verdict is what separates "the code does what the claim says" from "and that is a bug".
• Read-only tools keep the batch retry-safety property (nothing to double-execute), so the concern in the issue about retries does not apply.
• Caveat: N=12 on one repository, ground truth is one reviewer's judgment (the same method as the Broad-Side: multi-model batch endpoints behind the lens registry #141 scorecard), and the sweep model and the verifier model are the same family — the lift comes from the tools and the rubric, not from a stronger model.

Next: implementing the smallest viable form as a product action (verify after collect: top-N findings by severity, read-only tools confined to the repository, verified.md/verified.json next to triage.md, max_cost as a running cap, low reasoning effort) — with the same fake-fetcher tests the rest of Broad-Side has, and this experiment as its live verification.

[TheAmericanMaker]

Follow-up experiment, 2026-09-14 — the rubric pushed upstream into the batch prompt (#341). Two runs per arm on this repository, hand-judged: findings halve (24, 26 → 10, 13), medium-severity precision roughly doubles (2/12, 3/13 → 1/4, 3/6), real defects per run unchanged on average (2, 3 → 1, 4). Either prompt misses a real one on some runs, so verify stays the precision mechanism and a second run is cheap insurance. Numbers and run ids in the PR.