Skip to content

fix: echo back the model name the client sent - #367

Merged
Priyanshu-u07 merged 1 commit into
mainfrom
fix/echo-requested-model-name
Sep 30, 2026
Merged

Priyanshu-u07 merged 1 commit into
mainfrom
fix/echo-requested-model-name

Conversation

@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

Request {"model": "vllm-14b"} came back as "model": "Qwen/Qwen2.5-14B-Instruct-AWQ". Every response path that carries a model field now echoes the name the client used.

Non-streaming

One shared helper, echo_requested_model in worker_routing.py, alongside upstream_model which already owns the send-side naming. All four handlers call it: chat completions, embeddings, images and video. It is a no-op for engines whose response has no model field, so images and video are covered without assuming what a custom diffusion engine returns.

The client's name is still available at that point — the pipeline only ever substitutes the upstream id into the outgoing body, never onto ctx.model.

Streaming

This needed more than an overwrite. Upstream is read with aiter_raw, so chunks are arbitrary byte fragments: a single data: line, or the model name inside it, can arrive split across two chunks, and a chunk cannot be rewritten on its own.

process_stream now re-frames into whole lines when a rewrite is requested, holding any trailing fragment for the next chunk and flushing at the end. It also decodes incrementally, because a per-chunk decode(errors="ignore") throws away both halves of a multibyte character that straddles a boundary and deletes it from the response. A 3-byte Devanagari character split after its first byte disappeared entirely before that fix.

Only events actually being rewritten are re-serialised. Error payloads, [DONE], blank separators and non-JSON lines are returned byte for byte. With no rewrite requested the original yield chunk path runs unchanged, so other callers are unaffected, and the usage-parsing buffer is untouched.

Tests

25 tests. The ones that matter are the boundary cases: a line split across chunks, a split inside the model name, one byte at a time, a multibyte character split mid-sequence, and a final line with no trailing newline.

Each fix is mutation-verified — a naive byte replace instead of line framing passes 11 tests and fails exactly the 3 boundary ones; the old per-chunk decode fails the multibyte test; removing the helper call fails that path's echo test.

Closes #353

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07
Priyanshu-u07 merged commit 3ccafb2 into main Sep 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Response model name doesn't match the request

1 participant