Skip to content

fix(llm-client): keep response text out of JSON error diagnostics - #1016

Merged
rwmjhb merged 1 commit into
masterfrom
fix/llm-json-error-redaction-1014
Nov 7, 2026
Merged

rwmjhb merged 1 commit into
masterfrom
fix/llm-json-error-redaction-1014

Conversation

@AliceLJY

@AliceLJY AliceLJY commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Closes #1014

What changed

When createLlmClient().completeJson() got a response that was not JSON, could not be parsed, or could not be repaired, the diagnostic sent to the log callback (wired to api.logger.debug) and returned by getLastError() copied response text. It got in two ways:

  1. explicit preview= / jsonPreview= fields, and
  2. raw JSON.parse error messages, which quote their input (V8: Unexpected token 'S', "{"note": SYNTHETIC_"... is not valid JSON).

All three transports (host callback, direct API, OAuth) had the same three error sites. In src/llm-client.ts, all nine now use one shared formatJsonFailure() formatter, backed by describeJsonParseError(), which keeps only the error class and a numeric position. The unused previewText() helper is removed. No opt-in raw preview is added: the client has no existing debug flag that fits.

Unchanged:

  • valid JSON responses
  • the heuristic-repair success log line
  • empty-response and request-failure diagnostics
  • the existing no JSON object found / JSON.parse failed message prefixes

dist/src/llm-client.js is rebuilt (it is the only dist file that changed), and there is a CHANGELOG entry.

Diagnostic fields kept

  • stage label: [label]
  • transport tag: host-transport / OAuth / none (direct API)
  • category=non-json | parse-failure | repair-failure
  • chars= response length
  • jsonChars= length of the extracted JSON candidate (parse and repair failures)
  • error= / repairError=: parse error class (e.g. SyntaxError), plus at position N when the engine message has a numeric position

Example (host transport):

memory-lancedb-pro: llm-client [extract-candidates] no JSON object found in host-transport response (category=non-json, chars=15)
memory-lancedb-pro: llm-client [extract-candidates] host-transport JSON.parse failed (category=parse-failure, error=SyntaxError, chars=13, jsonChars=13)
memory-lancedb-pro: llm-client [extract-candidates] host-transport JSON.parse failed; heuristic repair failed (category=repair-failure, error=SyntaxError, repairError=SyntaxError, chars=14, jsonChars=14)

Tests

test/llm-json-error-redaction.test.mjs is registered in npm test and in the llm-clients-and-auth CI group. It calls the real createLlmClient() through each transport:

  • host: a fake runtimeLlmComplete
  • direct API: a loopback HTTP server
  • OAuth: a temporary fake auth file plus an intercepted fetch that returns SSE

Each transport runs four cases:

  • non-JSON prose with a synthetic marker
  • invalid JSON with the marker
  • JSON whose heuristic repair still fails
  • a valid JSON control

The tests check that neither log lines nor getLastError() contain the marker, a preview= field, or the raw JSON.parse message. They also check that the label, category, length and error class are still there, and that the valid control parses with no diagnostic.

$ node --test test/llm-json-error-redaction.test.mjs
# tests 12
# pass 12
# fail 0

Against the previous src/llm-client.ts, 9 of 12 fail (all error cases) and the 3 valid controls pass.

$ npm run build        # tsc -p tsconfig.json, clean
$ node scripts/verify-ci-test-manifest.mjs
CI test manifest covers baseline exactly once (154 entries)
$ npm test             # exit 0; node:test totals across the suite: tests 1264, pass 1264, fail 0

Out of scope

  • OAuth non-2xx errors still include up to 500 chars of the HTTP error body. That is a provider error payload, not model output, so it is left as is.
  • index.ts has separate ingress/auto-capture debug lines with preview=. Those are not part of this LLM-client path and are not touched.

Generated by Claude Code

)

When completeJson() got a response that was not JSON, failed to parse, or
failed heuristic repair, the diagnostic sent to the log callback (wired to
api.logger.debug) and returned by getLastError() copied response text: via
preview/jsonPreview fields and via raw JSON.parse messages, which quote
their input. That text can be the conversation or memory being extracted.

All three transports (host, direct API, OAuth) now build these diagnostics
with one shared formatter that keeps the stage label, a category
(non-json / parse-failure / repair-failure), the response length (chars),
the extracted candidate length (jsonChars) and the parse error class plus a
numeric position when the engine reports one. No raw preview opt-in is
added. Valid JSON responses and the repair-success log line are unchanged.

test/llm-json-error-redaction.test.mjs drives the real createLlmClient()
through each transport (fake runtimeLlmComplete, loopback HTTP server,
fake OAuth file + intercepted SSE fetch) with a synthetic marker and fails
9/12 against the previous implementation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019whQZvXdU3jHYnAnTxvvPC

AliceLJY commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

CI status on head 1c7dd6b: all six test jobs pass (version-sync, cli-smoke, core-regression, storage-and-schema, llm-clients-and-auth, packaging-and-workflow). Two non-test jobs are red, and neither comes from this diff, which only touches src/llm-client.ts, its dist build, one new test, the test manifest/script and the CHANGELOG.

assign-prs (Auto Assign workflow): pozil/auto-assign-issue@v1 throws Couldn't find issue info in current context on pull_request events. It fails the same way on every same-repo PR where the job actually ran: 8 of the last 100 runs, the rest skipped or action_required, none successful (e.g. port/local-runtime-fixes, fix/hybrid-fusion-weighting). It only fires on opened, so a re-run would hit the same error. No fix exists in the repo yet. A possible fix is to swap the PR job to an action that reads PR context, or to gh pr edit ${{ github.event.pull_request.number }} --add-assignee rwmjhb. I left it out of this PR to keep the scope tight.

claude-review: the action stops at its first turn with is_error: true, total_cost_usd: 0 and empty modelUsage. It fails before any model call or review of the code. A single re-run failed identically. That points to the review workflow's setup (credentials or quota for its token), which can't be fixed from this PR.


Generated by Claude Code

@AliceLJY
AliceLJY marked this pull request as ready for review November 5, 2026 09:10
@AliceLJY
AliceLJY requested a review from rwmjhb November 5, 2026 09:10

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1c7dd6b through the orchestrator. Approved as an improvement over the existing raw-response previews and parser messages; full-suite verification passed. Automated build verification was skipped.

Please address the remaining bounded redaction gap: describeJsonParseError() searches the entire parser message with an unanchored position-number regex. On Node 24, malformed fenced output containing "position 9876543210" can have those response-derived digits reported as a parser offset in logs and getLastError(), including the repair-error path. This does not retain the whole response, but it means redaction is not complete. Omit offsets or extract only proven engine metadata outside quoted input.

Add numeric-marker regressions across the three transports, plus wrapped/fenced responses where raw and extracted JSON lengths differ; assert chars and jsonChars separately. Existing alphabetic-marker exclusions do not catch this numeric leak.

@rwmjhb
rwmjhb merged commit 1ded52f into master Nov 7, 2026
14 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redact response text from LLM JSON error diagnostics

3 participants