feat(binding-llm): extract token usage across OpenAI/Anthropic dialects by jfallows · Pull Request #2600 · aklivity/zilla

jfallows · 2026-09-20T22:34:12Z

Description

Extends the llm binding's LlmDialect SPI and wire shape to normalize token-usage reporting across the OpenAI and Anthropic dialects, the same way it already normalizes request/response shape.

Wire shape. Adds a new LlmUsage struct — inputTokens, cacheWriteTokens, cacheReadTokens, outputTokens, reasoningTokens, totalTokens (each independently absent-capable via this repo's existing int32 = -1 sentinel convention), and nativeUsage (the recognized fields re-serialized, for reconciliation) — embedded in two new extension types, LlmEndEx and LlmAbortEx. The frame kind is the completeness signal: a clean response close carries LlmEndEx with the final tally; an abnormal mid-stream termination carries LlmAbortEx with whatever was accumulated so far — no separate completeness flag needed. Removes LlmDataEx.inputTokens/outputTokens, which were declared but never populated by any production code path.

Extraction. LlmDialect.supplyExtractor's response-direction branch — previously always identity() — now returns a per-dialect usage-extracting JsonTransform (LlmOpenaiUsageExtractTransform, LlmAnthropicUsageExtractTransform), a pure observer that forwards every JSON event unchanged while copying recognized usage fields into the existing per-stream JsonEnvelope side-channel (the same mechanism already used for model extraction). Anthropic reports usage across two documents (message_start for input/cache tokens, message_delta for output tokens); the envelope's repeatable-per-name semantics give latest-value-wins accumulation for free, with no extra bookkeeping.

Wiring. LlmClientFactory reads the accumulated fields back off the envelope at the two natural termination points — the (consolidated) doAppEnd and doAppAbort — and builds the corresponding extension. Along the way, folds the file's own doAppEnd/doAppEndNow split into a single doAppEnd, since that two-method naming pattern was unique to this file and read as the conditionality-in-a-name smell this repo's style guide calls out elsewhere (...IfNecessary).

No changes were needed in LlmServerFactory or LlmProxyFactory — both already relay every frame kind's extension bytes verbatim regardless of type.

Testing

New unit tests (LlmOpenaiUsageExtractTransformTest, LlmAnthropicUsageExtractTransformTest) driving each extractor through a real JsonPipeline in isolation, including Anthropic's two-document accumulation.
New paired k3po .rpt scenarios covering both streaming and non-streaming, both dialects — openai.streaming.usage, openai.usage, anthropic.streaming.usage, anthropic.usage, plus anthropic.streaming.abort for the partial-usage-on-abort case — each with a NetworkIT/ApplicationIT peer-to-peer self-consistency test plus a LlmClientIT test against a live engine. The openai.usage/openai.streaming.usage/anthropic.usage/anthropic.streaming.usage bodies are response bodies actually captured from the OpenAI and Anthropic APIs (gpt-4o-mini, claude-haiku-4-5-20251001), not hand-authored — see "Relationship to binding-llm/openai dialect: token usage extraction #2588 / binding-llm/anthropic dialect: token usage extraction #2589" below.
Extended LlmFunctions.java/LlmFunctionsTest.java with builder/matcher support for the new extension types.
Updated both dialects' response JSON schemas to declare the newly-recognized usage sub-fields.
Full ./mvnw clean install (checkstyle, license headers, unit tests, k3po ITs, JaCoCo coverage) is green for incubator/binding-llm and incubator/binding-llm.spec.

Fixes #2586
Fixes #2588
Fixes #2589

Relationship to #2588 / #2589

Both issues scope the same extraction work this PR implements for their respective dialect, for both streaming and non-streaming response modes, plus "fixture-based tests using captured real response bodies".

Extraction logic: done for both dialects, both modes.
Fixtures: the openai.usage, openai.streaming.usage, anthropic.usage, and anthropic.streaming.usage k3po scenarios now use response bodies genuinely captured from the live OpenAI and Anthropic APIs, not synthetic JSON — replacing the hand-authored versions from this PR's first iteration.
One residual gap, disclosed rather than hidden: Anthropic's cache_creation_input_tokens/cache_read_input_tokens fields could not be captured as real nonzero values — three attempts (pushing the cached context past both Sonnet's 1024-token and Haiku's 2048-token cache minimums, then adding the prompt-caching-2024-07-31 beta header) all returned genuine zeros, meaning prompt caching isn't triggering on the available account/API surface. Extraction correctness for those two fields is still verified, at the unit-test level (LlmAnthropicUsageExtractTransformTest) and via the existing anthropic.streaming.abort scenario's synthetic partial-usage values — just not via a live-captured nonzero body.
Running the real OpenAI streaming capture through the actual engine (not just the self-paired spec-level IT) surfaced a genuine bug this synthetic-only testing had missed: OpenAI's real streaming chunks send a literal "usage":null on every non-final chunk when stream_options.include_usage is set, but openai.response.schema.json only declared usage as type: "object" — real null failed schema validation and the stream was rejected. Fixed by widening the schema to type: ["object", "null"].

Dependency note

This branch is stacked on #2599 (feat(binding-llm): llm server/client/proxy), which introduces the llm binding itself and hasn't merged yet. Until #2599 merges, this PR's diff includes its commits too; once it merges into develop, this diff will reduce to just this PR's own commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FUH1iFFVB7P6bwhNB8Rhic

Generated by Claude Code

Scaffold incubator/binding-llm.spec per AGENTS.md conventions and define LlmBeginEx, LlmDataEx, and the LlmFlushEx union, modelled on binding-mcp.spec's idl. LlmBeginEx carries dialect only; model routing is deferred. LlmDataEx has no fields: content flows through the DATA frame's own payload octets and INIT/FIN through its existing flags, so nothing survives in the extension once block identity moves to the FLUSH plane. LlmFlushEx is a 7-case union covering message start, block start/end, finish, usage, keepalive, and an opaque native/raw case for re-encoding events a same-dialect route doesn't recognize. Fixes #2476 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AuZoMsETwEczJx3cbkb8EJ

@Incubating

Scaffolds incubator/binding-llm and incubator/binding-llm.conf, modelled on binding-mcp's SERVER/CLIENT BindingContext structure. LlmBindingInfo is annotated @Incubating so type: llm config loading is gated behind ZILLA_INCUBATOR_ENABLED via FeatureFilter, matching the AmqpBindingInfo/ PgsqlBindingInfo/RisingwaveBindingInfo precedent. Fixes #2477 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0142buJWS7C89AKr9uDJtSy4

…t-type Registers by content-type and hands back a per-stream LlmContentDecoder; stays in an internal, unexported package for now with no concrete implementation registered yet. Fixes #2478 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

Closes the non-streaming half of #2478's own scope: "Non-streaming application/json goes through the same abstraction as one event, rather than a special-cased branch." Only text/event-stream had an LlmContentDecoderSpi implementation; application/json requests (non-streaming dialect responses) had no decoder to dispatch to. LlmJsonContentDecoder treats the entire buffered document as a single event (one data + one flush call, no framing loop), mirroring LlmSseContentDecoder's structure and unit-test conventions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

…odecSpi LlmContentDecoderSpi and LlmContentEncoderSpi (the latter previously living downstream in #2571) let a content-type register a decoder with no matching encoder, or vice versa, since each was ServiceLoader-discovered and dispatched independently. LlmContentCodecSpi makes "this content-type is fully supported" one type-enforced fact: a single contentType() key with both supplyDecoder() and supplyEncoder(), one META-INF/services registration per content-type, one LlmContentCodecFactory dispatching both directions. Pulls the internal/encode/ base package and its text/event-stream implementation (LlmContentEncoder, LlmSseContentEncoder) forward from #2571 so the collapse can happen where decode already lives, rather than forking that package ahead of its own introduction there; #2571 will need to rebase on top of this and drop its now-duplicate copies. LlmSseContentDecoder, LlmSseContentEncoder, LlmJsonContentDecoder widen from package-private to public (unchanged otherwise) since their new LlmSseContentCodecSpi/LlmJsonContentCodecSpi providers construct them from the sibling internal.codec package. Adds LlmJsonContentEncoder (new): the application/json inverse of LlmJsonContentDecoder, copying content bytes through unchanged with no framing on flush. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

Defines the pluggable-dialect contract for binding-llm, exported from the start (unlike LlmContentDecoderSpi, which stays internal): LlmDialect exposes name()/detect()/contentType() plus supplyDecoder(Kind)/ supplyEncoder(Kind) returning common-json JsonTransform stages, and LlmDialectFactorySpi is the ServiceLoader-registered entry point. HttpHeaders is a minimal read-only accessor for detect(path, headers), since no HTTP header abstraction previously existed in this codebase and pulling in jakarta.ws.rs would add a dependency never otherwise used here. Kind is nested on LlmDialect, distinguishing request/response schemas. No concrete dialect implementations yet (OpenAI/Anthropic land later) -- module-info.java exports the dialect package and declares uses without a corresponding provides. Unit-tested via a stub LlmTestDialect/ LlmTestDialectFactorySpi registered under test-scope META-INF/services, mirroring this module's existing LlmContentDecoderSpi/ LlmTestContentDecoderFactorySpi pattern. Fixes #2480 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…er instance contentType() previously took no parameters, so a dialect could only report one fixed content-type for its lifetime -- insufficient for an API whose response framing (event-stream vs. a single JSON document) depends on a flag in the request body, since neither contentType() nor detect(String, HttpHeaders) offered any way to inspect it. Adds HttpRequestBody, a minimal read-only scalar-member accessor mirroring HttpHeaders, and changes contentType() to contentType(Kind, HttpHeaders, HttpRequestBody): Kind lets request and response resolve independently (a dialect's request body content-type can be fixed while its response varies), and the headers/body context lets that resolution depend on the actual request rather than being fixed at dialect-instance-creation time. Both parameters are nullable for callers without that context available. LlmTestDialect now resolves text/test-event-stream for a streaming response and application/test+json otherwise, exercising the new per-Kind, per-request resolution the stub previously couldn't express. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…velope, not whole-body buffering Replaces LlmDialect's HttpHeaders/HttpRequestBody/contentType() with the engine's existing ModelEnvelope/ModelTransform (runtime/engine/.../model/), reusing machinery this codebase already has instead of inventing LLM-specific buffering to resolve text/event-stream vs. application/json, or a model name, by peeking one field of an otherwise-unbuffered body. detect(ModelEnvelope) folds :path/:method into the same envelope as ordinary headers -- no separate path parameter. supplyDecoder/supplyEncoder now take (Kind, ModelEnvelope) and return ModelTransform: a per-field stage that can extract a field (e.g. a model name) into the envelope while the body still flows through unchanged, mirroring KafkaExtractTransform (runtime/binding-kafka/.../cache/) -- so a caller reads that signal back off the envelope as decoding proceeds rather than buffering the whole body first to inspect it. contentType() is removed entirely: nothing in this shape needs it once the streaming/non-streaming signal is just another envelope entry a caller reads after extraction. common-json/JsonTransform is no longer used anywhere in this module now that the dialect SPI itself doesn't need it, so the dependency comes back out of module-info.java and pom.xml along with it. LlmTestDialect/LlmTestDialectFactorySpi become LlmTestConditionalDialect/ LlmTestConditionalDialectFactorySpi, since what they now demonstrate is exactly this: request detection and model-name extraction conditional on envelope contents, not a fixed per-instance answer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…ken detail Extend LlmFlushEx's block-lifecycle skeleton (from #2476) with the two places OpenAI's parallel completions and per-token detail need to survive in the vocabulary, per the issue's scope: - choiceIndex (default 0) on messageStart, blockStart, blockEnd, finish, and native/raw: the (choice, block) compound index's outer half. Anthropic is always choice 0, so every existing dialect mapping is unaffected; OpenAI's n > 1 becomes one messageStart per parallel completion, distinguished by choiceIndex. usage stays choiceIndex-free since every dialect reports it aggregated across choices, never per choice. - logProbability (nullable) on LlmDataEx: the one per-delta detail the vocabulary carries directly, for dialects exposing per-token detail (OpenAI logprobs) without reopening the DATA/FLUSH split from #2476 or growing the vocabulary for the full log-probability structure — richer detail than one value per token stays behind LlmNativeFlushEx. Documents the three lossiness cases as doc comments alongside the fields they concern: choiceIndex and logProbability both drop out on any cross-dialect route to Anthropic (structurally exactly one choice, no per-token detail); message-start input token counts are resolved by decoupling inputTokens into its own deferred usage event rather than emitting a placeholder on messageStart and correcting it later, so a source that discloses tokens late (OpenAI) just emits usage late instead of needing a correction event. Fixes #2481 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYGZDrytLjmG2AQqUN1hsu

…lects Translates between each dialect's native streaming event sequence and the canonical vocabulary from #2557, in both directions: - LlmAnthropicEventMapper: holds input_tokens from message_start until the paired usage event at message_delta; tracks the currently open block's type to know when content_block_stop needs a canonical blockEnd (tool calls only) and to route content_block_delta payloads (text_delta vs input_json_delta) on encode. - LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into the canonical (Anthropic-shaped) block index via a per-stream map, and synthesizes blockEnd lazily -- deferred until the next tool call starts or the stream finishes, since OpenAI has no explicit block-close event. Unit-tested against both worked-example tables from the issue (message role/content/tool-call cardinality changes in each direction), plus the held-usage/already-consumed and lazy-blockEnd edge cases. Fixes #2482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw

LlmDialectResolver dispatches path/header detection across every LlmDialect registered via LlmDialectFactorySpi, without hardcoding any dialect's signals. A configured fixed dialect name bypasses detection entirely, including when it matches no registered dialect. When detection matches more than one dialect, or none, resolution is ambiguous and returns null so the caller rejects the request rather than guessing. LlmOptionsConfig adds the optional server-kind `dialect` option (schema, config, and adapter) used to pin a fixed dialect. Fixes #2483 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AtkteS7C66qiVLGeEDJ2FP

The sys: namespace's patch contract (Binding.system()/Exporter.system()) lets a component contribute a shared binding (e.g. http_client) but has no pre-seeded slot for a shared catalog, so a patch adding one has nothing to append into. Pre-seed catalogs: {} alongside the existing bindings: {} in the base sys namespace skeleton, the same "pre-seed the extension point" convention already used for the JSON-schema *-ext scaffolds, so any component can share a catalog-backed resource the way bindings are already shared. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

…ect schemas Adds LlmDialectFactorySpi.schema(Kind), letting a dialect contribute a URL to its own JSON schema for the request or response direction without knowing anything about catalogs or patches. LlmBinding.system() enumerates every registered dialect, reads each contributed schema, and generates a sys: namespace patch adding one shared inline catalog (llm_dialects) with a <dialect>.request/<dialect>.response subject per schema a dialect contributes -- built once, at engine startup, since the dialect set is ServiceLoader-discovered off the classpath and therefore fixed for the JVM's life, the same way sys: already shares a binding (e.g. http_client) across every binding that references it. LlmDataUrlStreamHandler decodes a base64 data: URL (RFC 2397) in memory, scoped to a single URL via the URL.of(URI, URLStreamHandler) factory -- no temporary file, no globally-registered protocol handler -- used to hand the generated patch to Binding.system()'s URL-returning contract without writing it to disk. LlmBeginEx gains contentType and model fields alongside dialect, for a server-kind stream factory to stamp once it has resolved the dialect and read the request's content-type. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

… LlmBeginEx.dialect Implements request/response framing decode-and-forward for the LLM server binding (#2484): LlmServerFactory drives the request body through the resolved dialect's ModelPipeline as bytes arrive off the wire, forwards transformed content to app0 incrementally rather than buffering the whole body, and threads the resolved dialect name onto LlmBeginEx so app0 can see which wire dialect produced the request. Flow control between the client, this binding, and app0 is enforced with dedicated decodeSlot/encodeSlot buffers on each side of the exchange (LlmServer for the network-facing leg, LlmStream for the app-facing leg), each granting credit strictly from its own local slot occupancy rather than copying a peer's sequence numbers across independent byte domains. LlmState tracks per-direction open/closing/closed transitions and end-of-stream deferral while a buffer is still draining. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

…fferPool handles DefaultBufferPool.buffer(slot) rewraps and returns a single shared mutable field per pool instance, so holding a buffer reference across a nested call that fetches a different slot from the same pool silently repoints it. LlmServer.decodeNetwork() directly, synchronously calls LlmStream's request-relay staging method as a plain nested Java method call, so a single pool instance backing both the network-decode slot and the app0 request-relay slot would alias between them. Give LlmServerFactory two BufferPool handles instead of one: decodePool (network decode) and encodePool (app0 relay, both directions), obtained via context.bufferPool() and .duplicate() -- matching McpServerFactory's decodePool/encodePool precedent. The two relay directions sharing encodePool (the reply-direction slot on LlmServer and the request-direction slot on LlmStream) never appear in the same call stack: cross-binding accept() is ring-buffer-mediated and dispatched on a later engine tick, not a nested call, so one encodePool instance can't alias between them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

…nline.conf LlmSystemNamespaceGenerator emits a real "type": "inline" catalog config into the generated sys: namespace patch, serviced at actual runtime by catalog-inline's CatalogFactorySpi via ServiceLoader -- not just exercised by this module's own tests. test scope kept it off the runtime classpath entirely; provided scope wouldn't fit either, since nothing in main source compiles against catalog-inline's Java API (the reference is a plain string), so there's nothing to satisfy at compile time. Match binding-asyncapi/binding-openapi's precedent for this same generated-"type: inline"-config pattern: catalog-inline at runtime scope, plus the companion catalog-inline.conf (config-schema side) at default scope alongside the existing model-json.conf dependency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

LlmBindingConfig.newModelConfig() constructs a real JsonModelConfig in main source, resolved to a working ModelHandler by context.supplyModel() via a ModelFactorySpi lookup at actual runtime -- not just exercised by this module's own tests. Matches binding-asyncapi/binding-mcp-openapi/ binding-openapi, each of which also builds JsonModelConfig directly in main source and declares model-json at runtime scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

Stale relative to the catalog-inline/model-json scope fixes on binding-llm (catalog-inline.conf and model-json.conf now roll up into this aggregate), plus a pre-existing gap for binding-http.spec's license entry. Regenerated via ./mvnw notice:generate -pl incubator -amd, never hand-edited, per AGENTS.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

Add LlmServerConfig/LlmServerConfigBuilder (host/port, nested builder) following the existing *OptionsConfigAdapter pattern used by binding-kafka's options.servers, wired into LlmOptionsConfig via a new `server` field so `llm client` bindings can configure their upstream endpoint. Config adapter unit tests cover parsing and serializing options.server. Fixes #2485 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FQ4XvYNCp1NzQC4Ne4eWbK

Exercise the inject() path on LlmServerConfigBuilder so the nested server builder reaches the module's required 100% instruction coverage, mirroring the existing shouldInjectBuilder test for LlmOptionsConfigBuilder. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FQ4XvYNCp1NzQC4Ne4eWbK

…→encoder chaining Adds LlmClientFactory (kind: client), the first real integration point for LlmDialect.supplyDecoder/supplyEncoder: it compares the inbound app-declared dialect (LlmBeginEx.dialect) against the binding's own configured dialect and only chains a JsonPipeline payload transform when they differ, forwarding framing-only re-encoded content when they match. - internal/encode: LlmContentEncoder/Spi/Factory + LlmSseContentEncoder, the SSE-framing encode counterpart to internal/decode's existing SSE decoder - LlmDialectResolver.dialectNamed(String): by-name lookup for resolving the inbound dialect when it differs from the client's configured one - Schema patch: kind: client options (dialect, server); also fixes kind: server to accept options.server (previously unreachable under its additionalProperties: false, despite LlmOptionsConfig already supporting it since PR 2570) - Spec scripts (same.dialect, cross.dialect, client.opaque.fallback, client.abort) plus a new LlmClientIT, written first per this repo's test-first discipline, confirmed failing before LlmClientFactory existed Fixes #2486 Real dialect implementations (openai, anthropic), the mock backend, and the full round-trip identity test are separate, sibling issues (#2487, #2490, Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qssroe9MdjvpvQrthi15ex

…aders, body) Rebased onto #2570's latest, which changed LlmDialect.contentType() to take (Kind, headers, body) parameters, resolved per request rather than fixed once per dialect instance. The client resolves content-type separately for each direction now (REQUEST for its outbound encoder, RESPONSE for its inbound decoder) rather than a single shared call, matching the interface's own point: request and response can have different native content-types. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qssroe9MdjvpvQrthi15ex

LlmClientFactory.newStream resolved the RESPONSE content decoder eagerly at BEGIN time with body=null, so LlmDialect.contentType(Kind.RESPONSE, ...) could never actually see request-body content when deciding between a streaming and non-streaming response, defeating the per-request capability added for that method. REQUEST-side encoder resolution is unaffected since it does not depend on body content. Accumulate the request body (post cross-dialect transform, pre-framing) into a buffer-pool-backed slot as DATA arrives, and resolve the decoder at onAppEnd, once the full request is available, relying on this binding's half-duplex transmission convention to guarantee no response bytes arrive before then. Add LlmJsonRequestBody, a pure-Java HttpRequestBody view driven by common-json's one-shot parser API, plus a unit test. Add a new test-conditional dialect and two client IT/spec scenarios proving the decoder now differs based on a stream field in the request body. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qssroe9MdjvpvQrthi15ex

…line SPI LlmDialect no longer exposes contentType(Kind, HttpHeaders, HttpRequestBody); dialects now detect via ModelEnvelope and supply ModelTransform-based decoders/encoders driven through the engine's ModelHandler/ModelPipeline SPI. Content-type is read directly off real wire headers instead of being computed per-dialect, so the request-body buffering added to defer response-decoder selection (LlmJsonRequestBody) is no longer needed and is removed. LlmClientFactory is rebuilt against the new SPI: dialect resolution via LlmBindingConfig.resolveDialect/dialectNamed, a per-stream ModelEnvelope, and a ModelPipeline (ModelTransform.NONE for same-dialect) run unconditionally in both directions so downstream code always sees validated, well-formed payloads. Test dialects are renamed (test-client, test-client-sse, test-client-sse-alt) to avoid colliding with the real upstream test fixtures, and gain request/response JSON schemas so the pipeline can actually validate their payloads instead of rejecting everything for lack of a schema. Adds the missing llm:flushEx()/llm:matchFlushEx() k3po functions that the client k3po scripts already relied on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qssroe9MdjvpvQrthi15ex

…sponse transforms Implements LlmDialect for OpenAI Chat Completions, registered via LlmDialectFactorySpi: detect() matches POST /v1/chat/completions and POST /v1/completions; contentType() is text/event-stream, the only LlmContentDecoderSpi/LlmContentEncoderSpi framing codec this module has today (a non-streaming application/json pair is a content-decoder- layer gap, not a dialect one, and is left as follow-up). supplyDecoder/supplyEncoder rename OpenAI-native request/response JSON members to the canonical vocabulary this dialect defines a synonym for -- max_tokens/maxOutputTokens, top_p/topP, n/choiceCount and friends on requests; index/choiceIndex, finish_reason/finishReason (remapping tool_calls/tool_call), logprobs/logProbability, and the usage token counts on responses -- via a depth-tracked JsonTransform that renames JSON events without DOM parsing. Everything without an established canonical synonym (id, model, messages, tools, the whole delta/tool_calls structure including streamed function.arguments fragments) forwards unchanged, at any depth, so decode -> encode round-trips with no loss. Tests cover dialect detection/registration and both Kinds of transform, including full round-trips through the canonical form. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

… rename model-json's ModelTransform integration was observation-only (ModelFieldBridge), discarding REPLACED/DECLINED answers instead of writing them. JsonModelFieldTransform drives a wired ModelTransform inline as JSON streams through, computing a JSON-pointer path per scalar field (with array-index segments) at any nesting depth, and writes FIELD/REPLACED/DECLINED answers straight to the destination -- including a REPLACED substitute redirecting a field to a sibling key of the same enclosing object. Adds the missing key-write for container-valued members entering a named object member, fixes a resumed key/value write re-offering the whole text instead of the remainder (TextSource now tracks its own consumed() progress), and reuses the already-decoded scalar text/key for an unchanged FIELD answer instead of a fresh per-field allocation. Also fixes JsonModelHandlerImpl.supplyEncoder silently dropping its transform parameter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

…e SPI Rebases LlmOpenAiDialect and its request/response transforms onto the redesigned LlmDialect SPI: detect(ModelEnvelope), no more contentType() (content-type is now resolved from real upstream Content-Type headers), supplyDecoder/supplyEncoder(Kind, ModelEnvelope) returning a ModelTransform. The request/response transforms are rewritten as plain ModelTransform implementations matching each field's own full path (e.g. $.choices[0].index, $.usage.prompt_tokens) rather than tracking JSON structural depth, since the model-json adapter now computes paths itself. This drops the old JsonEvent-token depth-tracking machinery and the JsonSource/JsonController wrappers (LlmOpenAiStructuredController is no longer needed); the renamed LlmOpenAiSubstitutedSource is now a plain ModelSource. The choices[]/usage direct-member path checks defer their substring() until after confirming the path is actually a direct member, so the many deeply nested per-chunk fields that share the prefix (delta.tool_calls[].index and the like) cost no allocation. The one dropped rename from the prior implementation is logprobs/logProbability: it names a container-valued field, and this dialect only renames scalar leaves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

…openai dialect LlmOpenAiRequestTransform renamed a fixed set of top-level fields but never observed model, so LlmServerFactory.doAppBegin's server.envelope.get("model", 0) read right after running the request through this transform came back empty and LlmBeginEx.model was never stamped for real dialect: openai traffic. Mirrors LlmTestPermissiveDialect's inline ModelExtractTransform (and KafkaExtractTransform's own pattern): on a FIELD event at $.model, copy the value into the envelope alongside the existing rename-or-forward decision, without touching the RENAMES table. LlmOpenAiResponseTransform needs no equivalent change -- nothing reads model back off a response. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

…requests CI's LlmServerIT.shouldRejectRequestWithUnresolvedDialect failed: detect() matched on :method/:path alone, so it ambiguously co-matched every existing k3po server fixture -- all of which hit the same /v1/chat/completions path with a test-specific content-type (application/vnd.zilla.test-permissive+json, application/vnd.zilla.test-strict+json) to select a *different* dialect unambiguously. Only one failure surfaced in CI because failsafe stops after the first failure, but the same ambiguity affected every other fixture in that class too -- confirmed by LlmServerIT going from 6 run/1 failed/1 skipped to 8/8 passing with this fix, and LlmClientIT unaffected at 4/4. detect() now also requires content-type: application/json, the only content-type a genuine OpenAI request ever carries, so a request to the same path with a different dialect's own content-type no longer collides. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

The llm.schema.patch.json allowed options.server on kind:server bindings, but LlmServerFactory never reads binding.options.server — only LlmClientFactory dials it as the upstream endpoint. Restrict the kind:server options schema to dialect (fixed-dialect mode), matching actual runtime usage and the milestone's example configs. Add LlmSchemaValidationTest exercising the full EngineConfigReader pipeline against real zilla.yaml text for both kind:server and kind:client, covering acceptance (bare server, fixed-dialect server, full client) and rejection (server option on kind:server, missing required client fields, malformed server pattern, unknown kind). Fixes #2488 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Aawd4ZFaUE3FgQnzkABj9F

Adds a nonstreaming openai-to-anthropic request scenario whose content field spans multiple network windows/transform() calls, the exact shape #2597 reported as silently forwarding the source dialect's JSON verbatim once a request no longer arrives in a single decode call. Confirms the JsonPipeline/JsonTransform rework (replacing the old ModelPipeline/ModelTransform request path #2597 was filed against) already fixes it: LlmClientIT#shouldTransformOpenaiToAnthropic100k passes end-to-end through the real engine, and a renamed openai-only field (presence_penalty -> presencePenalty) confirms the large content actually traverses decode+encode rather than passing through unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…tDecoderOutput SPI Since every implementer is owned by this binding, stop leaning on default interface methods to paper over gaps -- make LlmDialect.credentialsHeader()/ unauthorizedBody() and LlmContentDecoderOutput.available()/event() abstract so a future dialect or output implementer must declare its own instead of silently inheriting a value that may not be valid for it. This surfaced a real gap: LlmOpenaiDialect had never actually overridden credentialsHeader(), relying on the default happening to match OpenAI's real convention. Rename LlmDialect.supplyValidator to supplyExtractor: it never validated anything (schema validation is supplySchemaValidator's job, backed by common-json's JsonSchema), only extracts model without renaming, for kind:server's no-target-dialect case. Rename LlmContentEncoder.encodeEventName to encodeEvent for symmetry with encodeData/encodeFlush. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

Baselines for the decodeMessage/encodeMessage consolidation onto the streaming pipeline: tool-call-only responses, a multi-tool-call response, and the Anthropic->OpenAI direction of the existing 100k cross-dialect test, none of which had any regression coverage before. All pass against the current DOM-based implementation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…he streaming pipeline LlmDialect.decodeMessage/encodeMessage duplicated the same OpenAI<-> canonical<->Anthropic mapping rules the streaming decode/encode transforms already expressed, in a completely different (jakarta.json DOM) style, as a second, independently-maintained place a mapping fix had to land twice. Routes a non-streaming whole document through the same long-lived event pipeline streaming already uses: - LlmOpenaiDecodeTransform treats a "message" key as the non-streaming alias of "delta" (same relative shape), defaults a tool call's index from its array position when no explicit "index" key is present, queues text before tool calls regardless of native field order, and opens the canonical TEXT block lazily for a whole document (only once real text is seen, or not at all for a tool-call-only response) while keeping streaming's own eager open-at-message-start behavior unchanged. - LlmAnthropicDecodeTransform adds a nativeEvent-null branch walking a non-streaming document's own content[] array directly, reconstructing a tool_use block's "input" object into the same JSON-string shape streaming's partial_json already carries. - Both decode transforms now queue a canonical "end" action for a whole document, mirroring the real "[DONE]"/message_stop termination signal streaming relies on to tell the encode sink when to flush. - LlmOpenaiEncodeSink/LlmAnthropicEncodeSink read a new per-stream "streaming" envelope entry (written by LlmClientFactory from the real response content-type) once, at the first action, and either keep emitting one native chunk per action (streaming) or only accumulate fields and build the whole native document once at the final action (non-streaming). - LlmClientFactory.transformNativeEvent() now always drives the event pipeline; the old DOM-based branch, LlmDialect.decodeMessage/encodeMessage, and the now-unused LlmDialectJson helper are removed. Existing cross-dialect non-streaming fixtures needed no byte changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…ponse transform unit tests LlmOpenaiToAnthropicResponseTransformTest and LlmAnthropicToOpenaiResponseTransformTest only ever drove the shared pipeline with JsonEnvelope.NONE, which always reads back as streaming. Add shouldEncodeWholeDocumentWhenNonStreaming to each, building an isolated pipeline over an envelope that reads streaming=false, to exercise the encode sinks' accumulate-then-emit-once path that the streaming ITs and other unit tests never touch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…cribing code Strip the multi-paragraph class javadoc and inline "why" comments added while building the non-streaming decode/encode consolidation. The method/field names (ensureMessageStarted, flushTextIfPending, onToolCallEnd, wholeDocumentSteps, streaming/held*/doc*) already carry that meaning, so the prose was restating what the code already says. Behavior is unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…ment bound JsonSchemaImpl's schema-validator stage kept its per-document eval/failed state across nextDocument() calls, so a permissive schema's already-VALID verdict from a prior document leaked into the next one and reported Status.COMPLETED before any of the new document's own content was read. Reset that state on START_DOCUMENT instead. Separately, JsonTokenizer.onScalarStarved() decided whether to fragment a scalar spanning windows by comparing its own scanned bytes against the whole window length. That comparison never trips for a value that doesn't start at byte 0 of its window (preceded by other JSON in the same window), so the tokenizer kept rewinding and waiting for a window large enough to hold the value whole -- which never arrives for a value larger than any single window. Compare against the room actually left for the value in its window instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…er slots LlmSseContentDecoder buffered a whole SSE line before processing it, so a single data: field whose value exceeded one buffer slot could never complete a line and the client stalled indefinitely. Redesign it as a persistent byte-level state machine that streams a data: value to LlmContentDecoderOutput#data as bytes arrive, without waiting for the value's own terminating line break, so a single field's value may span any number of decode() calls of any total size. event:/id: fields stay whole-line-buffered since they are always short in practice. LlmContentDecoderOutput#data gains a last flag so the decoder can tell a downstream consumer exactly when a value's own line terminator was found, rather than always signalling completion at the end of a whole buffer. LlmClientFactory's same-dialect forward path (forwardResponseContent) now suspends and resumes cleanly against application backpressure instead of tearing down the connection when a single fragment's write can't fully drain in one pass, mirroring the existing eventPipeline suspend/resume pattern. LlmCanonicalEncodeSink-derived sinks (LlmOpenaiEncodeSink, LlmAnthropicEncodeSink) chunk a streamed text/argument value into fixed-size fragments across resumed calls instead of writing it as one atomic step, so cross-dialect re-encoding of an unbounded value no longer requires it to fit the sink's generator buffer in one shot. anthropic.response.schema.json now also models the streaming SSE envelope (index/delta/content_block/message) alongside the non-streaming message shape it already covered, so the response validator has an applicable, content-independent schema for a streamed text delta instead of falling back to whole-value reassembly. No change to buffer.slot.capacity/decodeMax anywhere in this change -- arbitrarily large field values now stream through a fixed-size buffer pool by design rather than needing a bigger one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…orting or corrupting SSE framing LlmServerIT#shouldForwardOpenaiStreaming100k/shouldForwardAnthropicStreaming100k hung, then aborted, then delivered corrupted content once flow control stopped aborting: a large response value spanning multiple app-level DATA frames could overrun the encode slot (raw byte credit was granted without accounting for the SSE framing overhead each encoded chunk adds), and each chunk of one logical data value was independently wrapped in its own "data: "/newline framing, splicing spurious bytes into the middle of the value. LlmServerFactory now reserves headroom in the reply window for framing overhead, tracks exactly how many raw bytes are still unflushed in the encode slot so app credit is only re-granted once their encoded form has actually drained, and forwards each response chunk with the flags that reflect its real position in the value instead of always claiming to be self-contained. LlmContentEncoder#encodeData now takes first/last markers so a value's "data: " prefix and terminating newline are written once across every fragment of that value, not once per fragment; LlmJsonContentEncoder, whose encoding has no per-value framing to duplicate, ignores them. No buffer.slot.capacity/decodeMax/encodeMax change anywhere. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…ed streamed values LlmClientIT#shouldTransformOpenaiToAnthropicStreaming100k crashed with "nextDocument() requires the prior transform() to have returned COMPLETED" when a single native SSE content value large enough to span multiple decode buffer slots was cross-dialect transformed while streaming: once the app-side reply window backed up mid-transform, LlmClientFactory$LlmHttpClient's resumeEventPipeline() advanced the event pipeline's document itself whenever the drained transform completed, but the SSE decoder then resumed past the value's trailing blank line in the very same decodeNet() call and invoked onEventFlush(), which advanced the same document a second time. eventPipelineCompleted now latches a COMPLETED transform() until whichever of onEventFlush()/resumeEventPipeline() first observes it calls nextDocument() and clears the flag, so the advance happens exactly once regardless of which path resolves the suspension. Added openai.streaming.transformed.100k (application) and anthropic.streaming.transformed.100k (network) k3po fixtures -- paired client/server scripts per the spec module's convention -- combining the existing streaming-transformed scenario with a ${core:randomBase64(100000)} content value, plus LlmClientIT#shouldTransformOpenaiToAnthropicStreaming100k and the matching ApplicationIT/NetworkIT self-consistency entries. No buffer.slot.capacity/decodeMax/encodeMax change anywhere. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…erhead Replace the fixed window-width margin subtracted from the reply window's maximum with Zilla's own per-frame padding field on the WINDOW: the reply window now grants the encode slot's full capacity, and each frame's own reserved credit is inflated by the SSE framing overhead instead of a single lump-sum deduction from the whole window. This scales correctly regardless of how many frames a value is split across, where the old fixed margin only happened to cover the frame counts exercised by the current tests. No buffer.slot.capacity/decodeMax/encodeMax change anywhere. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

… keys already do ProtobufJsonTest#shouldStreamJsonAcrossTinyWindows and ProtobufJsonChunkingTest#shouldMatchFieldNameKeyThatFragmentsAcrossJsonInputWindows started failing with REJECTED once common-json's JsonTokenizer began correctly fragmenting a scalar value that doesn't start at byte 0 of its input window (a separate common-json fix). ProtobufJsonParserImpl's own contract already requires one JSON leaf value to arrive whole -- messageStep()/mapStep() already decline an incomplete KEY_NAME fragment by leaving it unacted-on and re-pulling, relying on the parser's own accumulation to reassemble it -- but valueStep(), arrayStep(), and the map-entry value step never applied that same treatment to a VALUE_STRING/VALUE_NUMBER token, so a value delivered as a genuine fragment got dispatched as if it were the whole thing. Give scalar values the same decline as keys: when parser.deferredBytes() is still true, leave the value unacted-on and let the next pull() continue the reassembly, exactly mirroring the existing KEY_NAME handling. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…ntax fallback LlmOptionsConfigAdapter.defaultPort()'s http branch (PORT_HTTP) was never exercised by any existing test - every server-option test either supplies an explicit port or uses the https scheme, so only the https branch of the ternary was covered. Add shouldReadServerOptionWithDefaultHttpPort using "http://example.com" (no explicit port) to cover it. Also add shouldReadInvalidServerUriAsAbsent using a genuinely malformed URI (an unclosed IPv6-literal host) to exercise adaptServer()'s catch (URISyntaxException ex) block, which the existing shouldReadMalformedServerOptionAsAbsent test does not reach (its input parses successfully as a relative URI, only failing the subsequent host/scheme check). Together these restore binding-llm.conf's required 100% instruction coverage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…erver option llm(client)'s server option schema description/pattern changed from a host:port form to a full base-URL form, but examples/inspect.schema's golden schema.expected.json (which asserts the full merged JSON schema byte-for-byte via `zilla inspect schema`) was never updated to match, breaking the inspect.schema example's CI check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…ng to http LlmServerFactory (kind: server) and LlmClientFactory (kind: client) both sit directly upstream of a real `http` binding, which grants a non-zero `padding` in its WINDOW frames whenever it will re-frame the forwarded bytes (chunked Transfer-Encoding, used whenever no Content-Length is known upfront - true for every response llm(server) writes, and every request llm(client) writes, since neither declares a content-length). Per the established convention (see HttpServerFactory's own `replyPad` usage, and runtime/AGENTS.md's per-stream field table), the receiver's required padding must be added to `reserved` on every outbound DATA frame, and subtracted from the available window before sizing a frame's payload. Both classes ignored `window.padding()` entirely and sent `reserved` equal to the raw payload length, with no field even tracking it. Against http's own synthetic k3po test doubles (which always grant padding=0, since no existing binding-llm test simulates a real http peer's actual chunked- encoding requirement) this went unnoticed, but a real `http` binding's non-zero padding causes it to reject or truncate the frame, producing exactly the symptom seen in examples/llm.proxy's CI run: curl error 18 ("transfer closed with outstanding read data remaining") for a plain non-streaming request, and a hang on the request-encode side once padding is large enough to matter there too. Fix: add `replyPad`/`initialPad` fields to the two classes, capture `window.padding()` from the real WINDOW, and include it in both the available-window check and the `reserved` value of each outbound DATA frame. With padding=0 (every pre-existing test fixture) the arithmetic is unchanged, so no existing behavior is affected. Added regression coverage that simulates a real http peer's non-zero padding via `option zilla:padding` on the accept/connect side: - LlmServerIT#shouldForwardOpenaiNonstreamingWithReplyPadding (response leg) - LlmClientIT#shouldForwardOpenaiRequestWithReplyPadding (request leg) plus their NetworkIT/ApplicationIT peer self-consistency counterparts. Both new IT tests fail (truncated response / request-encode hang) against the pre-fix code and pass with it. Also added LlmProxyIT#shouldRouteOpenaiToAppZero, a small (non-10k) single-frame reply relay case through the proxy - the existing proxy relay tests only ever exercised multi-frame (10k+) replies, leaving the single-frame path unverified; it was ruled out as a source of this bug but is worth keeping as coverage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…client LlmHttpClient.encodeNet only ever set FLAG_INIT on the DATA frames it sends toward a real http:client binding, never FLAG_FIN. HttpClientFactory's chunked-body encoder (doEncodeHttp1Body) gates writing the chunk-size-hex prefix on FLAG_FIN and the trailing CRLF on FLAG_INIT, so every outbound request chunk was missing its size-hex line entirely, producing invalid HTTP/1.1 chunked encoding. This made any real downstream backend abort the connection before reading a complete request body, since llm never declares Content-Length on outbound requests (translated JSON body length is unknown up front), forcing chunked transfer-encoding. Since each physical write to encodeNet already becomes its own self-contained wire-level chunk (chunk-size scoped to just that write, independent of upstream logical message boundaries), and FLAG_INIT/FLAG_FIN here are consumed solely by the chunk-framing decision (request headers are written separately, and HTTP/2 ignores these flags entirely), always setting both bits is correct regardless of how encodeNet's own backpressure buffering may fragment a single JSON encode-output slice across multiple physical network writes. initialStarted becomes dead and is removed. Full binding-llm unit + k3po IT suite (136 unit tests, 57 ITs) passes unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…ect paths Each mock backend registered its POST handler on "/" under the assumption that south_llm_client_* always issues its outbound request there regardless of dialect. That assumption was stale: LlmClientFactory actually sends the dialect's own canonical path (LlmAnthropicDialect.MESSAGES_PATH = "/v1/messages", LlmOpenaiDialect.CHAT_COMPLETIONS_PATH = "/v1/chat/completions"), matching the real upstream APIs. Confirmed directly from a live frame-level trace (decoded via the engine's own ring-buffer event log) of the request zilla sends to south_tcp_client_anthropic: "POST /v1/messages HTTP/1.1 ...". With the request-side FLAG_FIN fix (096eb25) the request body now arrives at each mock intact, but Express 404s ("Cannot POST /v1/messages") since no route matched "/" -- masked before that fix because the malformed chunked body made the mock abort during body-parsing, before routing even ran. Fixed by registering each mock's route on the dialect's real path instead. Confirmed against the actual CI-built image (loaded locally, not rebuilt): all 8 examples/llm.proxy/etc/test/verify.sh assertions pass end-to-end, including cross-dialect translation, tool calls, credential pass-through, and model-based secondary routing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

…d message_start LlmAnthropicEventMapper.encodeMessageStart built the outbound Anthropic message_start event with only id/type/role/model, omitting content, stop_reason, and stop_sequence -- fields the real Anthropic API always includes and that the official anthropic-sdk-python's streaming accumulator depends on to initialize its per-message state. Without content: [], the SDK crashed on the first content_block_start/delta with 'NoneType' object has no attribute 'append', surfaced while exercising examples/llm.proxy's cross-dialect streaming translation through the real SDK instead of hand-built assertions. usage stays intentionally omitted here, per the existing comment on LlmUsageFlushEx: input token counts are decoupled from message_start so a dialect that only discloses them later (OpenAI) doesn't need a placeholder zero corrected afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5EdovtZRhxkqY5DF6MsoG

…eal SDKs Replace etc/test/verify.sh's hand-built curl+grep assertions with etc/test/verify.py, driven through the official openai and anthropic Python SDKs against the example's openai-facing and anthropic-facing frontends respectively. Proves the dialects are wire-compatible enough for an off-the-shelf client to succeed unmodified, not just compatible with our own request/response shapes -- and, in doing so, caught the message_start gap fixed in the preceding commit that curl+grep never would have (an official SDK enforces the real wire contract; a substring match on a curl response does not). Covers both dialects' default (cross-dialect translation) and secondary (intra-dialect, model-routed) legs, non-streaming and streaming, plus tool calls and credential pass-through, mirroring the prior script's scenario coverage. All four mock backends (mock-openai, mock-openai-secondary, mock-anthropic, mock-anthropic-secondary) gained SSE streaming support -- previously they only ever returned static non-streaming JSON regardless of the request's `stream` field -- since the SDKs' streaming iterators need a real event stream to consume through the proxy/translation layer. The verify compose service switches from node:20-alpine (curl) to python:3.13-alpine (openai + anthropic pip packages), since the checks no longer need Node or curl at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5EdovtZRhxkqY5DF6MsoG

The prior commit added content/stop_reason/stop_sequence to LlmAnthropicEventMapper's encoded message_start but left usage omitted when input tokens aren't known yet (still true for an OpenAI-sourced stream, which never discloses them before its terminal chunk). That omission is a different problem from the deferred-disclosure one LlmUsageFlushEx's own comment addresses: anthropic-sdk-python's streaming accumulator initializes its per-message snapshot from message_start and then patches usage.output_tokens on it in place when message_delta arrives -- with no usage object to patch, that crashed with 'NoneType' object has no attribute 'output_tokens' (confirmed streaming an openai-to-anthropic cross-dialect response through the real SDK via examples/llm.proxy). message_start now always carries a usage object; input_tokens is a provisional 0 for a source dialect that hasn't disclosed it yet, corrected by nothing further since Anthropic's own message_delta usage never carries input_tokens either -- only output_tokens, which encodeFinish already corrects from whatever the source discloses. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5EdovtZRhxkqY5DF6MsoG

Replace the curl commands with the openai/anthropic Python SDKs for all four demo scenarios, folding the separate SDK section into Try it so there's one walkthrough instead of two overlapping ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5EdovtZRhxkqY5DF6MsoG

…izfewx Resolves overlapping changes with PR #2598 (llm(proxy) routing): - Mock server route paths: kept this branch's fix (register on the dialect's real canonical path /v1/messages or /v1/chat/completions) layered on top of PR #2598's added SSE streaming support for each mock, since both changes touch the same route-registration line but are otherwise independent. - README.md: took PR #2598's version entirely (SDK-based walkthrough rewrite); this branch never touched the file. - LlmAnthropicEventMapper.java/Test.java and the openai.to.anthropic.streaming k3po fixture: kept deleted. These predate this branch's LlmFlushEx -> LlmDataEx wire-model rewrite and JsonPipeline consolidation, and the fixture still asserts the retired zilla:flush/matchFlushEx() shape. Ported the real bug PR #2598 fixed in the now-deleted mapper -- Anthropic's message_start event needs content/stop_reason/stop_sequence and a structural usage object, or anthropic-sdk-python's streaming accumulator crashes -- into this branch's replacement, LlmAnthropicEncodeSink.messageStartSteps(), and updated this branch's own equivalent fixture (application/anthropic.streaming.transformed) to match. Full binding-llm.spec + binding-llm verify suite green after the port. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

Extends LlmDialect.supplyExtractor's response direction with per-dialect usage extraction (LlmOpenaiUsageExtractTransform/LlmAnthropicUsageExtractTransform), observing native usage fields as they stream through and writing them into the per-stream JsonEnvelope. LlmClientFactory reads them back to populate a new LlmUsage struct (inputTokens, cacheWriteTokens, cacheReadTokens, outputTokens, reasoningTokens, totalTokens, nativeUsage), each field individually absent-capable via this repo's existing sentinel-default convention. Usage rides the END frame's LlmEndEx extension on a clean completion, and the ABORT frame's LlmAbortEx extension with whatever was accumulated so far on abnormal termination -- the frame kind itself is the completeness signal, so no separate flag is needed. Removes the dead, never-populated inputTokens/outputTokens fields from LlmDataEx. Folds doAppEnd/doAppEndNow into a single doAppEnd, matching this repo's naming convention elsewhere (the two-method split was unique to this file and read as conditionality-in-the-name). Fixes #2586. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUH1iFFVB7P6bwhNB8Rhic

…se flush Adds non-streaming .rpt scenarios (openai.usage, anthropic.usage) and their NetworkIT/ApplicationIT/LlmClientIT coverage, closing the test-coverage gap against #2588/#2589's stated scope (streaming and non-streaming, both dialects). This surfaced a real bug: LlmUsageExtractTransform only flushed captured fields into the envelope on JsonEvent.END_DOCUMENT, which the streaming event-pipeline's canonical sink reliably delivers, but the plain re-serializing sink used by the same-dialect passthrough response pipeline does not -- it reports Status.COMPLETED as soon as the top-level value's own closing token is written, without a distinct END_DOCUMENT event ever reaching the transform chain. Fixes it to also flush when the top-level container itself closes (depth returns to 0), which covers both sinks; a genuine END_DOCUMENT that follows is a harmless no-op once the chunk state is already cleared. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUH1iFFVB7P6bwhNB8Rhic

… real bodies Swaps the synthetic JSON/SSE payloads in the openai.usage, anthropic.usage, openai.streaming.usage, and anthropic.streaming.usage k3po scenarios for bodies actually captured from the OpenAI and Anthropic APIs (gpt-4o-mini and claude-haiku-4-5-20251001), closing the "fixture-based tests using captured real response bodies" scope item from #2588/#2589. Running the real OpenAI streaming capture through the engine (not just the self-paired spec-level IT) surfaced a genuine schema bug: OpenAI's actual streaming chunks send a literal "usage":null on every non-final chunk when stream_options.include_usage is set, but openai.response.schema.json only declared usage as type "object" -- real null failed validation and the stream was rejected/aborted. The hand-authored fixtures never caught this because they simply omitted the usage key on intermediate chunks instead of sending null. Widens the schema to type ["object", "null"]. Anthropic's cache_creation_input_tokens/cache_read_input_tokens fields remain uncaptured -- three attempts (raising the cached context past both Sonnet's 1024-token and Haiku's 2048-token cache minimums, then adding the prompt-caching-2024-07-31 beta header) all returned genuine zeros, meaning prompt caching isn't triggering on this account/API surface. Coverage for those two fields remains at the unit-test level (LlmAnthropicUsageExtractTransformTest) and the existing anthropic.streaming.abort scenario's synthetic partial-usage values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUH1iFFVB7P6bwhNB8Rhic

claude added 30 commits September 11, 2026 19:43

claude added 13 commits September 20, 2026 14:15

jfallows force-pushed the claude/nice-newton-b6gdzn branch from 4f99f54 to c755fc4 Compare September 20, 2026 22:35

claude added 3 commits September 20, 2026 22:41

jfallows force-pushed the claude/nice-newton-b6gdzn branch from 6740462 to 06d1c3f Compare September 21, 2026 01:20

jfallows force-pushed the claude/nice-newton-b6gdzn branch from f238ba9 to dcee052 Compare September 21, 2026 02:18

jfallows mentioned this pull request Sep 21, 2026

feat(binding-llm): export LlmContentCodecSpi for external content-type codecs #2601

Open

claude added 8 commits September 22, 2026 22:40

jfallows force-pushed the claude/nice-newton-b6gdzn branch from dcee052 to 75ad3fa Compare September 23, 2026 00:58

Navigation Menu

feat(binding-llm): extract token usage across OpenAI/Anthropic dialects - #2600

feat(binding-llm): extract token usage across OpenAI/Anthropic dialects#2600
jfallows wants to merge 235 commits into
developfrom
claude/nice-newton-b6gdzn

jfallows commented Sep 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jfallows commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Relationship to #2588 / #2589

Dependency note

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfallows commented Sep 20, 2026 •

edited

Loading