Skip to content

Run as a LiteLLM sidecar: reasoning, failover errors, usage, image - #1

Merged
seankoji merged 2 commits into
mainfrom
feat/litellm-sidecar
Sep 28, 2026
Merged

seankoji merged 2 commits into
mainfrom
feat/litellm-sidecar

Conversation

@seankoji

Copy link
Copy Markdown

Summary

  • Patches the upstream proxy for use behind LiteLLM on the NAS (nas-ai stack in seankoji-com/careynas.net). The full list is in the README's "Fork changes" table.
  • Correctness fixes found against the live gateway and the reverse-engineered wire protocol:
    • reasoning is now sent as reasoning_content instead of being mixed into content
    • an upstream error before any output returns 502, so the router can fail over; previously it was a 200 carrying an SSE error
    • late events after an error could write to an ended response and crash Node
    • usage was always 0
    • tool-call input was sent as a JSON string instead of an object
    • image parts had the wrong shape
    • the x-session-id and x-cli-environment headers were missing
    • the CLI version was pinned at 1.54.0 while npm is at 1.66.0; it now tracks npm
  • Container: Dockerfile running as the non-root node user with a healthcheck, logs to stdout only, graceful SIGTERM, and the Windows netstat/taskkill code removed.
  • CI: unit tests plus an image build that publishes ghcr.io/seankoji-com/proxy-commandcode:sha-<commit> on every branch push. It runs on GitHub-hosted runners because the org's self-hosted runner groups refuse public repos.

Test plan

  • npm test: 6/6 pass locally and in CI
  • Branch image sha-c922ce4 built, pulled on the NAS (amd64, user node)
  • Live on the NAS through the proxy:
    • sync, streaming and tool-call round trip all succeed
    • a bad key returns 401 and a model outside GOAT returns 403
    • 23 of 25 mapped model ids answer

🤖 Generated with Claude Code

seankoji and others added 2 commits September 28, 2026 13:05
…, usage, container image

Fixes found against the live gateway and the reverse-engineered wire
protocol: reasoning streamed as content, errors swallowed into a 200 (and
late events writing to an ended response), zero usage, tool-call input sent
as a JSON string, image parts in the wrong shape, missing session headers,
and a CLI version header pinned 12 releases stale. Adds a Dockerfile, unit
tests, and an image workflow that publishes sha-<commit> tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The org Universal ruleset requires the gatekeeper check and code-scanning
results on main. The caller distributor had not reached this fork yet, so
add the public-repo callers (gatekeeper, static analysis, agent readiness,
Dependabot auto-merge) from the hub and a CodeQL workflow for JavaScript and
Actions. Pin the image workflow's actions to commit SHAs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seankoji
seankoji merged commit c87863c into main Sep 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant