Broad-Side: multi-model batch endpoints behind the lens registry

[TheAmericanMaker]

Roadmap item (see ROADMAP.md). BROADSIDE_MODEL is a single constant. OpenRouter exposes other batch-priced models (DeepSeek, Anthropic Claude batch, Gemini Pro batch). Support a model→pricing table, per-model prompt tweaks (stronger models for semantic-heavy lenses, cheapest for mechanical scans), and config-driven selection in broadside/config.yaml. Cost estimates already parameterized on per-token pricing — make them per-model.

User context: Gemini 3.7 Flash batch is cheap but weak; a stronger batch model at ~2-3× cost would raise precision on defect/security lenses.

[TheAmericanMaker]

Ran the experiment this issue implies, using a scoped key. Short version: three of the five things that block this feature are not model-quality problems, they are availability, quota and plumbing problems — and the one model-quality result went the opposite way from the issue's premise.

Also worth stating up front: three of this issue's four asks already ship.

ask	status
model → pricing table	done — live catalog, on-disk cache, built-in fallback
config-driven per-lens selection	done — lens_models, documented in SKILL.md and config.yaml
per-model cost estimates	done — per-lens pricing, mixedModels flag
per-model prompt tweaks	not done

So the capability is here. What follows is what happens when you actually try to use it.

1. /models over-reports batch availability, badly

Of five stronger-than-default models I probed, three have no batch endpoint at all:

openai/o3:batch                       REJECTED  Model 'openai/o3:batch' does not have a :batch endpoint.
anthropic/claude-sonnet-4.6:batch     REJECTED  Model 'anthropic/claude-sonnet-4.6:batch' does not have a :batch endpoint.
anthropic/claude-haiku-4.5:batch      REJECTED  Model 'anthropic/claude-haiku-4.5:batch' does not have a :batch endpoint.
deepseek/deepseek-v4-pro-0813:batch   ACCEPTED
google/gemini-3.8-flash:batch         ACCEPTED

All five are listed by codecarto_broadside action: models, with batch pricing, structured: yes. /models returns a :batch id variant for models whose Batch API rejects it, and nothing in the response distinguishes them — identical supported_parameters, identical pricing key shape. Our filter is id.endsWith(":batch"), which is the only thing the data supports.

The user-visible effect: the models action exists to help you pick a model, and an unknown fraction of what it offers cannot be submitted. Cheap to discover — a rejected batch costs nothing and is reported correctly as rejected with the message preserved — but only at submit time.

Worth considering: probe-on-first-use and cache the result per model, or mark the catalog advisory in the models output.

2. There is a 16-concurrent-job quota per account

HTTP 400: invalid batch inference job: job-submission-count for account …, in use: 16, quota: 16

Broad-Side submits one job per lens, so roughly three six-lens runs saturate it. This bit twice during the experiment, and the second time all 13 requests came back failed with the quota message buried in the batch's error object rather than as a clean submit-time refusal. Cost was correctly $0.

This matters directly for multi-model work: comparing N models across M lenses is N×M jobs, and the ceiling arrives fast. Something that reads the quota, or serializes lens submission when close to it, would help.

3. The truncation retry is still serialized

The same serialization #136 fixed for lens polling:

for (const stored of allLensResults) {
    if (!stored.truncated) continue;
    const { batchId } = await submitBatch([bumped], …);          // one slice
    const batch = await pollBatchUntilTerminal(batchId, …);        // …polled to terminal before the next
}

Lens batches poll concurrently. Retries do not. With a model that truncates heavily this dominates the wall clock — the arm below truncated 11 of 13 slices, so its collect is doing eleven sequential batch round trips. That is the difference between a five-minute collect and an hour-long one, and it is invisible until you use a chattier model, which is precisely what this issue is about.

4. Neither surface exposes model selection

core/ supports model and per-lens lens_models. The Pi command takes neither; handleBroadside's args are cwd, action, lenses, api_key, wait_seconds, include_synthesis, include_triage, retry_truncated, max_cost, force, include_benchmarks, incremental — no model. The MCP tool schema has no model property either.

So the only way to select a model is hand-editing config.yaml. That works and is documented, but it means the models action helps you choose a model you then cannot pass through the same interface. If per-lens model choice is the point of this issue, a model / lens_models parameter on both surfaces is the smallest useful change here.

5. The quality result went the other way

The premise was "a stronger batch model at ~2-3× cost would raise precision." The available stronger models are largely unsubmittable (above), so I tested the free-upgrade hypothesis instead: google/gemini-3.8-flash:batch is priced identically to the 3.7 default ($0.375/$1.875) with the same 65k output cap. Same commit (7a39dc5), same lens, same 13 slices.

arm	slices	parsed	unparseable	findings
gemini-3.7-flash:batch (default)	13	12	1	24
gemini-3.8-flash:batch	13	2	11	6

Same price, same cap, and the newer model truncated mid-JSON on 11 of 13 slices. Its retries are still running, so its final recall is not yet scored — but a newer model at the same price is clearly not a free upgrade, and output discipline against our schemas varies more between models than price does.

The default model's actual scorecard

Scored against bugs I verified and fixed on that same commit, so the labels are outcomes rather than opinions:

• 6 of 6 bugs it reported that I confirmed real and fixed (two MCP validation gaps, handleConfig cwd, estimateCost per-slice, both YAML parser defects)
• 2 of 2 false positives repeated — the high-rated catalog.entry! assertion and the safeRelativeHref backslash claim, both verified sound
• 0 of 4 of the serious bugs found by other means: the state.json lost update, wait_seconds scope, /codecarto-complete's unguarded await, and the stale-ctx process crash

The nearest miss is instructive. It flagged "Deterministic temp file and staging directory collisions under concurrent execution" at core/library.ts:389 — the right class as the state.json lost update, at a different and less harmful site, while the actual data-loss bug went unmentioned.

That is the shape of the tool working as designed: good recall on local, single-file, pattern-matchable defects; no purchase on lifecycle and concurrency bugs that need reasoning across a call graph. It also argues that #143's hybrid — batch sweeps then a verification pass over the leads — attacks the real weakness, since a third of what I chased here did not survive verification.

Suggested split

I would narrow this issue to the two things that are actually missing and actionable:

1. a model / lens_models parameter on the Pi command and the MCP tool
2. batch-endpoint availability handling — probe and cache, or mark the models output advisory

Plus one spin-off, which is a straight bug rather than a feature: parallelize the truncation retry, matching #136.

I would drop "per-model prompt tweaks" unless there is a specific model in mind — on this evidence, the variable that matters is whether a model can hold a JSON schema for 8k tokens, not prompt wording.

[TheAmericanMaker]

Addendum — the truncation had nothing to do with verbosity or model strength. It is reasoning tokens eating the output cap.

I dug into the raw batch response rather than trusting the byte counts. max_tokens was 6000 (the defect lens's maxTokens), and:

defect-core-6         finish_reason=length  completion_tokens=5984
                      completion_tokens_details: {reasoning_tokens: 5758}
                      content: 921 chars (~230 tokens)

defect-extensions-12  finish_reason=length  completion_tokens=5986
                      completion_tokens_details: {reasoning_tokens: 5758}
                      content: 845 chars (~211 tokens)

defect-extensions-9   finish_reason=stop    completion_tokens=904
                      completion_tokens_details: {reasoning_tokens: 0}
                      content: 3733 chars (~933 tokens)

gemini-3.8-flash spends 5,758 of its 6,000-token output budget thinking, leaving ~230 tokens for the JSON, which then truncates mid-structure. finish_reason: length on 11 of 13. The two that succeeded are exactly the two with reasoning_tokens: 0.

Three consequences, all of which reframe this issue:

1. The selection criterion is not price or strength — it is whether the model reasons. Nothing in /models exposes this. Not price, not context_length, not max_completion_tokens, not supported_parameters. Two models at identical prices with identical caps behave completely differently, and the only way to find out is to run one and read completion_tokens_details.

2. The cost model is wrong for these models. Reasoning tokens bill at the full output rate. We paid for ~6,000 output tokens per slice to receive ~230 tokens of usable JSON — a 26× waste factor — and estimateCost has no way to anticipate it, since it budgets output as a fraction of maxTokens on the assumption that output means JSON.

3. It explains why the retry barely helped. The doubled cap (6,000 → 12,000) scales the reasoning budget too, so 24 completed retry jobs moved gem-3.8 from 2 parsed slices to 3. The truncation-repair mechanism from #133 assumes truncation means "the JSON was too long", which is not the failure here.

There is a plausible fix, and it is small: OpenRouter accepts a reasoning parameter on the request. Setting it to disabled (or minimal effort) for lens requests would make reasoning models usable, stop paying output rates for discarded thinking, and make maxTokens mean what estimateCost already assumes it means. Broad-Side currently sends no reasoning field at all, so it inherits whatever each model defaults to — which is why the default model works and a same-priced sibling does not.

Filed separately as a bug, since it affects any reasoning-capable batch model rather than just multi-model support. It also means the per-lens lens_models feature that already ships is quietly unusable for a whole class of models until it is addressed.