Skip to content

fix(agent): read the turn's finish reason so a cut answer is not treated as complete - #2298

Open
koraysrn wants to merge 4 commits into
libredb:mainfrom
koraysrn:fix/agent-truncated-turn
Open

koraysrn wants to merge 4 commits into
libredb:mainfrom
koraysrn:fix/agent-truncated-turn

Conversation

@koraysrn

Copy link
Copy Markdown
Contributor

Description

The agent loop discarded the model's finish reason, so a turn the server stopped at its output limit
looked like a turn the model chose to end. That cut turn then reached the prompted readers and
dispatch, and an agent run holding no report ended model-stopped.

This reads the finish reason and treats a cut turn as cut: nothing from it is kept, an agent run is
asked once for a shorter move, and a second cut ends the run with its own reason.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • Documentation update

Related Issue

Closes #2291

Changes Made

1. fix(agent): read the finish reason of every model turn

  • ModelTurn carries finishReason: FinishReason | null, read from finish-step and finish, and
    null on the abort path, where no finish part ever arrives.
  • The SDK's own union is used rather than a parallel vocabulary, so a value it adds becomes a
    compiler question.

2. fix(agent): end a cut answer with its own reason, and ask once for a shorter move

  • New stop reason model-output-limit, a guidance id of the same name, a notice, and
    AGENT_MODEL_OUTPUT_LIMIT_ASKS = 1 counted off the ledger.
  • The branch sits after the abort handling and before the prompted readers and dispatch, so nothing
    from the cut turn is kept: the text is cleared, the calls are not run, and no assistant message is
    pushed.
  • An agent run with turns and time left is told once and takes another turn; otherwise the run ends
    failed / model-output-limit.
  • Planning runs are not told: the notice names compose_report, which that mode does not hold.
  • In the app, the timeline entry, the answer card and the rail get a line for the new ending, and the
    rail a headline for the notice. No layout or interaction changes.

3. docs(agent): say which limit stopped a cut answer

  • docs/AGENT.md states the bound beside the per-call ceiling; docs/AGENT_GUIDE.md states it where
    the ceilings are described.

Alternatives considered

  • Infer a cut from the text. Rejected: the provider states it, and inference misfires on every
    short answer.
  • Run the call the cut turn carried. Rejected: a call cut in half and one written just before the
    cut are indistinguishable from the client, and the one place a truncated call must never land is a
    statement.
  • Push the partial text back so the model can continue. Rejected: that spends the next turn on
    the same length that was already refused.
  • Cap the answer with max_tokens. Rejected on the ground agent: read each turn's token usage, bound the run by it, and show the figure #2290 was closed on: it changes every
    request, which invalidates the cells measured on the default shape, and it cuts reasoning turns.
  • Cover planning runs too. Deferred: it needs a second sentence and a re-measurement of the locked
    plan cells.
  • Count the ask per drive. Rejected: the model has already heard the sentence.

Evidence

  • The provider states the cut, and it is the fact the loop was throwing away. Against Ollama 0.40.15,
    a cut text turn ends finish_reason: "length" and nothing else in the body says the answer was
    stopped.
  • Live, through a local proxy that adds max_tokens so the application's own request is unchanged:
    the notice reached the model verbatim on the next call, the transcript grew by exactly one message,
    and a run cut twice ended failed / model-output-limit.
  • On main, the same capped scenario filed a truncated envelope as the run's closing-statement and
    never named the limit.
  • A cut that lands inside tool arguments reports a different reason on that endpoint and is already
    answered as an unreadable call; it is out of scope here, and both the code and docs/AGENT.md say
    so.

Full token counts, the second wire shape, the suite figures and the coverage check are in the first
comment.

What this does not do

Testing

  • I have tested this locally
  • I have added/updated tests
  • All existing tests pass

Tests cover the cut shape on each of the three wires, and the negative cases: a turn that ended by
itself is still model-stopped, a planning run is not told, both notice gates end the run instead,
and the surface that shows the sentence is pinned for a run that also carries a no-report verdict.

Test Environment

  • LibreDB Studio Version: 1.3.0, branch built on 6fd6d4c0a
  • Browser: not exercised; the UI change is text in the timeline, the answer card and the rail
    status line, covered by component tests
  • OS: Linux
  • Node.js/Bun Version: Bun 1.4.4, Node 24.22.1
  • Database Type: postgres through a stub provider on the ledger side, so the tool pipeline is
    real to the point of execution and no statement leaves the process; the model is a real Ollama
    0.40.15 over the custom provider kind

Checklist

  • My code follows the project's code style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • The required CI test job passes the 100% line-coverage gate: bun run test:coverage and
    bun run coverage:check ran here on this branch, 211355/211355 lines (100.00%)
  • Provider triad: N/A, no src/lib/db/providers/ change
  • Agent language pack: N/A, no statement-language row changed
  • Any dependent changes have been merged and published: N/A, nothing depends on this change

Additional Notes

Re-ask transcript shape validation

The re-ask transcript format was validated against a live Ollama endpoint. When a turn is cut off,
the partial assistant message is omitted and the notice is appended directly after the previous
message, which may result in consecutive user messages. A follow-up completion request succeeded
with this transcript structure.

Why the text is cleared in the branch rather than in conclude()

conclude() is shared across every terminal path, and moving the reset into it would also change
what a model-timeout run records. The reset is applied only where the turn is explicitly discarded.

The agent loop read only text deltas, tool calls, errors and aborts out of the
stream, so the provider's own finish reason was discarded before anything could
look at it. A turn a provider stopped at an output limit therefore reached the
loop looking exactly like a turn the model chose to end.

takeTurn now keeps the finishReason the SDK maps off the provider's field, read
from both finish-step and finish, and null on the abort path where no finish
part is ever emitted.

Refs libredb#2291.
…shorter move

A turn the endpoint stopped at its output limit was treated as a turn the model
had finished. Nothing from it is trustworthy: its text is a truncated sentence
and a call it carried may be truncated too, so the branch sits before the
prompted readers and before dispatch, drops the calls, and clears the run's own
text so no partial answer is filed as its closing statement.

An agent run is told once, in a sentence that names the length limit and asks
for less writing rather than less waiting, and a second cut ends the run as
failed/output-limit instead of model-stopped. Planning runs are not told: the
sentence names compose_report, which they do not hold, and their prose is what
the plan verifier already judges.

Measured against Ollama 0.40.15 with a num_predict cap: the final frame of a cut
text turn is finish_reason length. The shape a cut inside tool arguments takes
there is tool_calls with unparsable arguments, which reaches the loop as an
unreadable tool call and is answered by that path.

Refs libredb#2291.
docs/AGENT.md states the new bound beside the per-call ceiling, with the measured
finish reasons and the shape a cut inside tool arguments takes on the endpoint it
was measured on. docs/AGENT_GUIDE.md states it where the rails' ceilings are
described, because the reader's next question is why an answer stopped
mid-sentence and the two limits are repaired in opposite ways.

Refs libredb#2291.
The comment on ModelTurn.finishReason said the values other than length are "the
model's own decisions", which is false for content-filter and error, and it cited
Anthropic's max_tokens although this build has no anthropic provider kind. It now
states what each value is and that the two unhandled ones are follow-ups.

The notice asked for "one tool call, with its arguments and nothing else", which
reads as an instruction to write bare arguments on the prompted protocol. It now
says what its sibling says, and the docblock records why the shape of a call is
left to the protocol reminder.

The stop reason is renamed model-output-limit so it reads beside model-timeout
rather than turn-limit: one bounds turns, the other one answer.

Two tests pin the surface a real run reaches, because a run ended here also
carries a no-report verdict and the timeline prefers the shortfall. One more takes
the arm that withholds the notice when the run is inside its report reserve, and
the finishReason assignment is guarded against an adapter that omits the field.

Refs libredb#2291.
@koraysrn

Copy link
Copy Markdown
Contributor Author

Evidence and measurements for #2291

Posted as the first comment so the description stays short. Everything here was run on this branch.

1. The provider states the cut

Ollama 0.40.15, 127.0.0.1:11434, OpenAI-compatible endpoint, qwen3:8b with a Modelfile
num_predict of 64:

  • the body is ordinary deltas, and the final frame is
    {"choices":[{"delta":{},"finish_reason":"length"}]}, completion_tokens 64;
  • nothing else in the body says the answer was stopped.

2. The shape the issue did not predict

Same endpoint, qwen2.5:1.5b, num_predict varied:

num_predict What arrived
48 no tool_calls at all, finish_reason: "length"
56, 64, 72 a tool_calls frame whose arguments hold a single {, finish_reason: "tool_calls"
90 the call completes, finish_reason: "tool_calls"

A cut that lands inside tool arguments is therefore not a length on this endpoint. It reaches the
loop as unreadable arguments and is answered by AGENT_UNREADABLE_TOOL_CALL_ANSWERS. This branch does
not claim that shape.

3. Live end to end

qwen2.5:1.5b, prompted tool protocol, a local proxy in front of the endpoint that injects
max_tokens, so the application sends no different byte.

  • Cap on the first call, 48 tokens. finish_reason=length; the next call carried the notice
    verbatim; the transcript went from 5 messages to 6, so the cut turn's assistant message was not
    pushed; the run continued, made two tool calls and ended turn-limit at the 4-turn ceiling.
  • Cap on every call, 20 tokens. Two length frames, then failed / model-output-limit after 2
    turns, exactly one notice, no closing statement, empty result text.
  • Same scenario on main, cap on every call. 4 turns, no notice, and
    closing-statement: {"action": "run_read_query", "sql": "SELECT * FROM orders LIMIT 10"}.

Both live records carry verdict=["no-report"], which is why the surface tests use that shape.

4. Tests and gates

  • 9 loop tests over the real provider package and the wire fixtures: Chat Completions cut text, cut
    tool call, Responses response.incomplete with max_output_tokens, Gemini MAX_TOKENS.
  • Negative cases: a turn that ended by itself is still model-stopped, a planning run is not told,
    the no-turn gate and the report-reserve gate each end the run instead of spending the notice.
  • Surface tests: the timeline entry of a run that also missed its report shows the shortfall while
    describeRunFailure shows the cut sentence, and the answer card names the limit.
  • One digest row for the new sentence (af0b65edb1d75818) and one headline test.
  • The tests bite: changing "length" to "content-filter" in the branch turns 6 of the 9 red, and
    loosening the reserve gate to > 0 turns exactly one red.
  • bun run test: 2028 files, 75,795 tests, 75,790 pass, 5 skipped (all pre-existing). 31 Helm and
    operator files did not run on this machine because Helm is not installed.
  • bun run typecheck: clean. bun run lint: 0 errors, 432 pre-existing warnings. bun run knip:
    fails only on pre-existing vscode-extension findings, identical on main.
  • Coverage of every added line was checked on a local coverage run: the branch lines in
    investigation.ts, the guidanceDelivered row, the counter, the guard in takeTurn, and the
    sentence in timeline.ts are all hit.

5. Size

197 added source lines, of which 74 are comments, 11 blank and 112 code. The published library bundle
is unchanged at 0 bytes, because no module under src/exports/ reaches src/lib/agent.

6. Not claimed

@codecov

codecov Bot commented Nov 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] An agent turn that stops at an output limit is treated as complete (finishReason is never read)

1 participant