test(binding-llm): k3po fixtures for real dialect:openai backend by jfallows · Pull Request #2576 · aklivity/zilla

jfallows · 2026-09-12T06:25:22Z

Description

Closes out issue #2490 (k3po test fixtures for an OpenAI-shaped /v1/chat/completions backend, streaming + non-streaming, for use in llm server/llm client integration tests).

Per repo convention (specs/AGENTS.md), the backend side is faked entirely with paired k3po client.rpt/server.rpt scripts — no standalone mock server process and no examples/llm.proxy example. Existing binding-llm scenarios (same.dialect, cross.dialect, etc.) only exercise synthetic test dialects; these new scenarios exercise the real dialect: openai end-to-end for the first time:

openai.request (network + application): llm server detects dialect: openai purely from POST /v1/chat/completions (no x-llm-dialect header needed), resolves content-type directly off the real request header, and forwards the request body through a json-model ModelPipeline — the dialect's decoder observes model as a side effect (envelope.set("model", ...)) while the body flows through unchanged. LlmBeginEx.dialect/.contentType/.model are all asserted on the app-facing stream.
openai.streaming (network + application): llm client same-dialect round-trip with a "stream":true request and a realistic multi-chunk OpenAI SSE response — role chunk, content chunk, tool-call start and argument-fragment chunks, finish_reason chunk, usage chunk, [DONE] sentinel — each SSE data: line forwarded as one app-facing DATA + one raw LlmFlushEx.
openai.nonstreaming (network + application): llm client same-dialect round-trip with a "stream":false request and a plain-JSON response carrying top-level usage, forwarded as one DATA + one terminal raw LlmFlushEx (matching LlmJsonContentDecoder's own framing).

Each scenario is wired into LlmServerIT/LlmClientIT (real engine, via EngineRule) and ApplicationIT/NetworkIT (paired-script protocol self-consistency, no engine), per the required dual-IT convention. A new client.openai.yaml config pins dialect: openai for the client-kind scenarios.

No production code changed here — this is test-fixture-only, per the issue's scope. (Production changes referenced above — the LlmDialect/ModelEnvelope/ModelTransform SPI redesign, LlmServerFactory's detect+stamp+ModelPipeline-forward rework, LlmClientFactory's content-type-driven codec selection, and the openai dialect's model extraction — landed in the sibling PRs this branch is stacked on, prompted by a design review that started in this same session.)

Stacking

This branch is built on top of the full binding-llm chain (module scaffold, content-codec SPI, dialect SPI redesign, dialect: openai, llm server/llm client stream factories, config validation) — so this diff includes those commits until they merge to develop, at which point this PR's diff will shrink to just the new k3po fixtures and IT wiring, the same pattern used by the PRs it stacks on.

Test plan

./mvnw checkstyle:check -pl incubator/binding-llm.spec,incubator/binding-llm — 0 violations
./mvnw clean package -pl incubator/binding-llm.spec,incubator/binding-llm -am — compiles cleanly, including the four edited/new *IT.java classes, against the redesigned SPI
./mvnw verify (the actual k3po-driven ITs) could not be run in the sandboxed dev session used to author this change — the reactor's maven-notice-plugin license-mapping check fails on unmapped artifacts regardless of network access (confirmed the proxy itself reaches the registry fine; the mapping data itself is what's missing). This is the same pre-existing, environment-specific limitation noted in prior binding-llm PRs' test plans, not something introduced by this change. CI should confirm the full verify lifecycle, including the new openai.request/openai.streaming/openai.nonstreaming scenarios.

Fixes #2490

🤖 Generated with Claude Code

https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

Generated by Claude Code

Scaffold incubator/binding-llm.spec per AGENTS.md conventions and define LlmBeginEx, LlmDataEx, and the LlmFlushEx union, modelled on binding-mcp.spec's idl. LlmBeginEx carries dialect only; model routing is deferred. LlmDataEx has no fields: content flows through the DATA frame's own payload octets and INIT/FIN through its existing flags, so nothing survives in the extension once block identity moves to the FLUSH plane. LlmFlushEx is a 7-case union covering message start, block start/end, finish, usage, keepalive, and an opaque native/raw case for re-encoding events a same-dialect route doesn't recognize. Fixes #2476 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AuZoMsETwEczJx3cbkb8EJ

@Incubating

Scaffolds incubator/binding-llm and incubator/binding-llm.conf, modelled on binding-mcp's SERVER/CLIENT BindingContext structure. LlmBindingInfo is annotated @Incubating so type: llm config loading is gated behind ZILLA_INCUBATOR_ENABLED via FeatureFilter, matching the AmqpBindingInfo/ PgsqlBindingInfo/RisingwaveBindingInfo precedent. Fixes #2477 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0142buJWS7C89AKr9uDJtSy4

…t-type Registers by content-type and hands back a per-stream LlmContentDecoder; stays in an internal, unexported package for now with no concrete implementation registered yet. Fixes #2478 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

Closes the non-streaming half of #2478's own scope: "Non-streaming application/json goes through the same abstraction as one event, rather than a special-cased branch." Only text/event-stream had an LlmContentDecoderSpi implementation; application/json requests (non-streaming dialect responses) had no decoder to dispatch to. LlmJsonContentDecoder treats the entire buffered document as a single event (one data + one flush call, no framing loop), mirroring LlmSseContentDecoder's structure and unit-test conventions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

…odecSpi LlmContentDecoderSpi and LlmContentEncoderSpi (the latter previously living downstream in #2571) let a content-type register a decoder with no matching encoder, or vice versa, since each was ServiceLoader-discovered and dispatched independently. LlmContentCodecSpi makes "this content-type is fully supported" one type-enforced fact: a single contentType() key with both supplyDecoder() and supplyEncoder(), one META-INF/services registration per content-type, one LlmContentCodecFactory dispatching both directions. Pulls the internal/encode/ base package and its text/event-stream implementation (LlmContentEncoder, LlmSseContentEncoder) forward from #2571 so the collapse can happen where decode already lives, rather than forking that package ahead of its own introduction there; #2571 will need to rebase on top of this and drop its now-duplicate copies. LlmSseContentDecoder, LlmSseContentEncoder, LlmJsonContentDecoder widen from package-private to public (unchanged otherwise) since their new LlmSseContentCodecSpi/LlmJsonContentCodecSpi providers construct them from the sibling internal.codec package. Adds LlmJsonContentEncoder (new): the application/json inverse of LlmJsonContentDecoder, copying content bytes through unchanged with no framing on flush. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

Defines the pluggable-dialect contract for binding-llm, exported from the start (unlike LlmContentDecoderSpi, which stays internal): LlmDialect exposes name()/detect()/contentType() plus supplyDecoder(Kind)/ supplyEncoder(Kind) returning common-json JsonTransform stages, and LlmDialectFactorySpi is the ServiceLoader-registered entry point. HttpHeaders is a minimal read-only accessor for detect(path, headers), since no HTTP header abstraction previously existed in this codebase and pulling in jakarta.ws.rs would add a dependency never otherwise used here. Kind is nested on LlmDialect, distinguishing request/response schemas. No concrete dialect implementations yet (OpenAI/Anthropic land later) -- module-info.java exports the dialect package and declares uses without a corresponding provides. Unit-tested via a stub LlmTestDialect/ LlmTestDialectFactorySpi registered under test-scope META-INF/services, mirroring this module's existing LlmContentDecoderSpi/ LlmTestContentDecoderFactorySpi pattern. Fixes #2480 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…er instance contentType() previously took no parameters, so a dialect could only report one fixed content-type for its lifetime -- insufficient for an API whose response framing (event-stream vs. a single JSON document) depends on a flag in the request body, since neither contentType() nor detect(String, HttpHeaders) offered any way to inspect it. Adds HttpRequestBody, a minimal read-only scalar-member accessor mirroring HttpHeaders, and changes contentType() to contentType(Kind, HttpHeaders, HttpRequestBody): Kind lets request and response resolve independently (a dialect's request body content-type can be fixed while its response varies), and the headers/body context lets that resolution depend on the actual request rather than being fixed at dialect-instance-creation time. Both parameters are nullable for callers without that context available. LlmTestDialect now resolves text/test-event-stream for a streaming response and application/test+json otherwise, exercising the new per-Kind, per-request resolution the stub previously couldn't express. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…velope, not whole-body buffering Replaces LlmDialect's HttpHeaders/HttpRequestBody/contentType() with the engine's existing ModelEnvelope/ModelTransform (runtime/engine/.../model/), reusing machinery this codebase already has instead of inventing LLM-specific buffering to resolve text/event-stream vs. application/json, or a model name, by peeking one field of an otherwise-unbuffered body. detect(ModelEnvelope) folds :path/:method into the same envelope as ordinary headers -- no separate path parameter. supplyDecoder/supplyEncoder now take (Kind, ModelEnvelope) and return ModelTransform: a per-field stage that can extract a field (e.g. a model name) into the envelope while the body still flows through unchanged, mirroring KafkaExtractTransform (runtime/binding-kafka/.../cache/) -- so a caller reads that signal back off the envelope as decoding proceeds rather than buffering the whole body first to inspect it. contentType() is removed entirely: nothing in this shape needs it once the streaming/non-streaming signal is just another envelope entry a caller reads after extraction. common-json/JsonTransform is no longer used anywhere in this module now that the dialect SPI itself doesn't need it, so the dependency comes back out of module-info.java and pom.xml along with it. LlmTestDialect/LlmTestDialectFactorySpi become LlmTestConditionalDialect/ LlmTestConditionalDialectFactorySpi, since what they now demonstrate is exactly this: request detection and model-name extraction conditional on envelope contents, not a fixed per-instance answer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…ken detail Extend LlmFlushEx's block-lifecycle skeleton (from #2476) with the two places OpenAI's parallel completions and per-token detail need to survive in the vocabulary, per the issue's scope: - choiceIndex (default 0) on messageStart, blockStart, blockEnd, finish, and native/raw: the (choice, block) compound index's outer half. Anthropic is always choice 0, so every existing dialect mapping is unaffected; OpenAI's n > 1 becomes one messageStart per parallel completion, distinguished by choiceIndex. usage stays choiceIndex-free since every dialect reports it aggregated across choices, never per choice. - logProbability (nullable) on LlmDataEx: the one per-delta detail the vocabulary carries directly, for dialects exposing per-token detail (OpenAI logprobs) without reopening the DATA/FLUSH split from #2476 or growing the vocabulary for the full log-probability structure — richer detail than one value per token stays behind LlmNativeFlushEx. Documents the three lossiness cases as doc comments alongside the fields they concern: choiceIndex and logProbability both drop out on any cross-dialect route to Anthropic (structurally exactly one choice, no per-token detail); message-start input token counts are resolved by decoupling inputTokens into its own deferred usage event rather than emitting a placeholder on messageStart and correcting it later, so a source that discloses tokens late (OpenAI) just emits usage late instead of needing a correction event. Fixes #2481 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYGZDrytLjmG2AQqUN1hsu

jfallows · 2026-09-16T06:23:48Z

CI failed on Build (25): run 35061039385, head 5ca69a53.

LlmClientIT.shouldForwardOpenAiNonstreamingSameDialect failed twice (failsafe's own automatic rerun), with two different symptoms:

Run 1: ComparisonFailure — behavior didn't match what the paired scripts expected.
Run 2: TestTimedOut.

The robot's captured state at the Run 2 timeout shows the network-side backend script successfully matched the outbound HTTP BEGIN (5 headers: :method/:scheme/:authority/:path/content-type), then hung waiting to read the forwarded JSON request body — i.e. llm(client) never forwarded the request payload for this stream.

This doesn't look like a fixture problem: openai.streaming's and openai.nonstreaming's request-write halves are structurally identical (same headers, same write/write flush/write close sequence, same content-type default), and only shouldForwardOpenAiStreamingSameDialect — which exercises the exact same request-forwarding path — passed. Two different failure shapes across two runs of the same test points to a race rather than a deterministic fixture mistake.

Working hypothesis: LlmBindingConfig.supplyModel(dialect, kind) caches one ModelHandler per dialect.name() + kind, so both scenarios request the identical cache key ("openai" + REQUEST). If that shared handler — or the per-schema ModelPipeline/JsonPipeline machinery underneath it in model-json — isn't safely reusable across independently-running streams, a second use could plausibly manifest as exactly this kind of intermittent hang/mismatch.

I can't reproduce or debug this further from here — this sandbox can't run the k3po verify lifecycle at all (pre-existing maven-notice-plugin license-mapping gap noted in this PR's own test plan), and the fix, if my hypothesis is right, is in LlmClientFactory/LlmBindingConfig (and possibly model-json), not in this PR's own fixture-only diff. Flagging rather than guessing at a fix to shared, concurrency-sensitive code I can't validate locally.

Generated by Claude Code

…lient Chains an llm:server (auto-detecting dialect: openai from POST /v1/chat/completions) directly into an llm:client (dialect: openai) against a mock OpenAI backend, proving the server→client hop preserves framing byte-for-byte with no payload re-serialization. Adds config/roundtrip.openai.yaml wiring net0 (llm:server) → app0 (llm:client, internal) → net1 (external, mock backend), modeled on binding-tls's bridge.tls1.3.yaml chaining pattern. New streaming and non-streaming network scenarios reuse the wire content already proven by openai.streaming/openai.nonstreaming (including the tool-call streaming chunks spanning DATA and FLUSH), with deliberate insignificant whitespace (e.g. "\"model\": \"gpt-4\"") added to request/response bodies at every hop — this survives only if the body is forwarded unchanged rather than parsed and re-serialized, since same-dialect resolution takes LlmClientFactory's ModelTransform.NONE fast path and the server's json-model pipeline forwards the body unmodified. This scenario chains two different bindings across two different wire legs (net0's pre-server-decode bytes vs net1's post-client-encode bytes) rather than describing one client/server exchange, so — like binding-tls's BridgeIT — it has no NetworkIT peer-to-peer counterpart; only the engine-backed LlmRoundtripIT applies. Could not run `./mvnw verify` for the new LlmRoundtripIT in this sandboxed session: the reactor's maven-notice-plugin license-mapping check fails on unmapped artifacts regardless of network access, the same pre-existing, environment-specific limitation noted in #2576's own test plan. Verified instead: `./mvnw checkstyle:check` (0 violations) and `./mvnw package` (compiles cleanly) for incubator/binding-llm.spec and incubator/binding-llm. CI should confirm the full verify lifecycle. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

jfallows · 2026-09-16T14:55:49Z

Update: rebased onto 13e12c51 (picks up the #2487 session's LlmOpenAiDialect.detect() content-type-scoping fix, ed510113) and re-ran CI — same failure, byte-for-byte identical: run 35108176452.

LlmClientIT.shouldForwardOpenAiNonstreamingSameDialect failed again with the exact same two symptoms (ComparisonFailure then TestTimedOut on retry), and the Run 2 timeout trace is character-for-character identical to the previous run: network side matches the 5-header HTTP BEGIN, then hangs forever on the request body read.

This rules out the dialect-detection ambiguity as the cause (that fix landed and made no difference) and rules out ordinary test-order flakiness (reproduces identically across independent CI runs with different code in between). This is a real, deterministic bug in LlmClientFactory's same-dialect application/json request-forwarding path — still pointing at LlmBindingConfig.supplyModel's per-dialect+kind ModelHandler caching (both openai.streaming and openai.nonstreaming hit the same "openai"+REQUEST cache key) or the model-json pipeline machinery underneath it, per my earlier comment.

Still can't reproduce or fix this myself (no k3po verify capability in this sandbox). Continuing to flag it for the #2486 session (LlmClientFactory/LlmBindingConfig) since it's outside this PR's own fixture-only diff.

Generated by Claude Code

…ng coverage Revert the LlmRoundtripIT approach (chaining llm:server directly into llm:client in one config): it combined two bindings under test in a single IT, which this repo avoids in favor of verifying each binding independently and inferring combinations. PR #2576 already authored the full request/streaming/nonstreaming × network/application script matrix, and ApplicationIT/NetworkIT already prove each application-level and network-level script pair is self-consistent. But engine-backed coverage was split: LlmServerIT only exercised openai.request, and LlmClientIT only exercised openai.streaming/openai.nonstreaming. Composing the already-existing peer-to-peer proofs with LlmServerIT's and LlmClientIT's own per-binding proofs already establishes the round-trip identity claim from #2491 (framing decode/re-encode byte-identical, no payload parsed via the existing .raw() flush-ext assertions, including tool-call streaming spanning DATA and FLUSH) without ever running server+client chained. Adds the two missing LlmServerIT methods (openai.streaming, openai.nonstreaming) using the scripts and config that already exist — no new files needed. Considered adding a symmetric openai.request method to LlmClientIT, but net/openai.request/server.rpt was authored as the peer for the server-detection scenario (path /v1/chat/completions) rather than as what llm:client's own fixed-dialect encoder actually emits (path "/", per the already-passing openai.streaming/openai.nonstreaming network scripts), so pairing them would fail for a script-mismatch reason unrelated to any real defect; left LlmClientIT unchanged rather than author a new script pair for it. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

jfallows · 2026-09-16T18:32:25Z

Root cause found and fixed for the shouldForwardOpenAiNonstreamingSameDialect failure reported above.

LlmClientFactory's constructor always built and drove a schema-validating JSON ModelPipeline for both request and response, even for same-dialect traffic where the design (per LlmDialect SPI notes) intends pure raw byte relay with no transform. The real openai dialect declares no JSON schema for either direction (LlmOpenAiDialectFactorySpi.schema(Kind) returns null, matching the SPI's own contract for "a dialect with nothing to contribute"), so JsonModelHandlerImpl's schema resolution always falls through to NO_SCHEMA_ID for its subject, and the pipeline rejects every value on the first transform() call regardless of payload content — confirmed directly with a standalone reproduction against JsonModelHandlerImpl/JsonModelDecoderPipeline using the same no-schema + ModelTransform.NONE configuration LlmClientFactory builds for same-dialect traffic.

Fix (45e5e520): skip constructing requestPipeline/responsePipeline entirely when source == target (same dialect), forwarding request/response bytes directly through the content-type codec's own encoder/decoder for framing, without driving a model pipeline that has nothing to validate.

Verified: ./mvnw package (compile + checkstyle + unit tests, 160/160 passing) for binding-llm.spec/binding-llm in this sandbox — the k3po verify lifecycle itself still can't run here (repo-wide maven-notice-plugin gap, unrelated to this change), so CI is the actual verification for the LlmClientIT scenarios this fixes.

Generated by Claude Code

jfallows · 2026-09-16T18:52:04Z

Correction to my previous comment: the LlmClientFactory change (45e5e520) that skipped the ModelPipeline for same-dialect traffic was the wrong fix — it treated the symptom, not the cause, and same-dialect traffic is exactly the path that most needs schema validation (no field-rewriting transform to sanity-check structure along the way). Reverted in 75162602.

The actual gap: LlmOpenAiDialectFactorySpi declared no JSON schema for either direction, so the shared sys:llm_dialects catalog had no openai.request/openai.response subject, and the schema-validating pipeline always resolved NO_SCHEMA_ID and rejected every value — there was no real validation happening for openai traffic at all, same-dialect or cross-dialect.

Fixed properly in 2cc26ee0: added real openai.request.schema.json/openai.response.schema.json describing the actual Chat Completions wire shapes (request requires model/messages, types the fields LlmOpenAiRequestTransform recognizes; response types choices[]/usage without requiring any top-level member, since a streaming chunk only ever carries a subset of what a full completion carries at once), and wired LlmOpenAiDialectFactorySpi.schema(Kind) to supply them. LlmClientFactory is back to its original, unmodified behavior — it now gets a real schema to validate against instead of none.

Verified directly against JsonModelHandlerImpl/JsonModelDecoderPipeline (the same construction LlmClientFactory drives) that all nine request/response payloads used by the openai.request/openai.streaming/openai.nonstreaming k3po fixtures validate as COMPLETE, not REJECTED. Also added unit test coverage (LlmOpenAiDialectTest) asserting the factory supplies non-null, well-formed schemas for both directions. ./mvnw package (checkstyle + 162/162 unit tests) is green; CI is the real verification for the k3po scenarios.

Generated by Claude Code

…lient Chains an llm:server (auto-detecting dialect: openai from POST /v1/chat/completions) directly into an llm:client (dialect: openai) against a mock OpenAI backend, proving the server→client hop preserves framing byte-for-byte with no payload re-serialization. Adds config/roundtrip.openai.yaml wiring net0 (llm:server) → app0 (llm:client, internal) → net1 (external, mock backend), modeled on binding-tls's bridge.tls1.3.yaml chaining pattern. New streaming and non-streaming network scenarios reuse the wire content already proven by openai.streaming/openai.nonstreaming (including the tool-call streaming chunks spanning DATA and FLUSH), with deliberate insignificant whitespace (e.g. "\"model\": \"gpt-4\"") added to request/response bodies at every hop — this survives only if the body is forwarded unchanged rather than parsed and re-serialized, since same-dialect resolution takes LlmClientFactory's ModelTransform.NONE fast path and the server's json-model pipeline forwards the body unmodified. This scenario chains two different bindings across two different wire legs (net0's pre-server-decode bytes vs net1's post-client-encode bytes) rather than describing one client/server exchange, so — like binding-tls's BridgeIT — it has no NetworkIT peer-to-peer counterpart; only the engine-backed LlmRoundtripIT applies. Could not run `./mvnw verify` for the new LlmRoundtripIT in this sandboxed session: the reactor's maven-notice-plugin license-mapping check fails on unmapped artifacts regardless of network access, the same pre-existing, environment-specific limitation noted in #2576's own test plan. Verified instead: `./mvnw checkstyle:check` (0 violations) and `./mvnw package` (compiles cleanly) for incubator/binding-llm.spec and incubator/binding-llm. CI should confirm the full verify lifecycle. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…ng coverage Revert the LlmRoundtripIT approach (chaining llm:server directly into llm:client in one config): it combined two bindings under test in a single IT, which this repo avoids in favor of verifying each binding independently and inferring combinations. PR #2576 already authored the full request/streaming/nonstreaming × network/application script matrix, and ApplicationIT/NetworkIT already prove each application-level and network-level script pair is self-consistent. But engine-backed coverage was split: LlmServerIT only exercised openai.request, and LlmClientIT only exercised openai.streaming/openai.nonstreaming. Composing the already-existing peer-to-peer proofs with LlmServerIT's and LlmClientIT's own per-binding proofs already establishes the round-trip identity claim from #2491 (framing decode/re-encode byte-identical, no payload parsed via the existing .raw() flush-ext assertions, including tool-call streaming spanning DATA and FLUSH) without ever running server+client chained. Adds the two missing LlmServerIT methods (openai.streaming, openai.nonstreaming) using the scripts and config that already exist — no new files needed. Considered adding a symmetric openai.request method to LlmClientIT, but net/openai.request/server.rpt was authored as the peer for the server-detection scenario (path /v1/chat/completions) rather than as what llm:client's own fixed-dialect encoder actually emits (path "/", per the already-passing openai.streaming/openai.nonstreaming network scripts), so pairing them would fail for a script-mismatch reason unrelated to any real defect; left LlmClientIT unchanged rather than author a new script pair for it. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…lient Chains an llm:server (auto-detecting dialect: openai from POST /v1/chat/completions) directly into an llm:client (dialect: openai) against a mock OpenAI backend, proving the server→client hop preserves framing byte-for-byte with no payload re-serialization. Adds config/roundtrip.openai.yaml wiring net0 (llm:server) → app0 (llm:client, internal) → net1 (external, mock backend), modeled on binding-tls's bridge.tls1.3.yaml chaining pattern. New streaming and non-streaming network scenarios reuse the wire content already proven by openai.streaming/openai.nonstreaming (including the tool-call streaming chunks spanning DATA and FLUSH), with deliberate insignificant whitespace (e.g. "\"model\": \"gpt-4\"") added to request/response bodies at every hop — this survives only if the body is forwarded unchanged rather than parsed and re-serialized, since same-dialect resolution takes LlmClientFactory's ModelTransform.NONE fast path and the server's json-model pipeline forwards the body unmodified. This scenario chains two different bindings across two different wire legs (net0's pre-server-decode bytes vs net1's post-client-encode bytes) rather than describing one client/server exchange, so — like binding-tls's BridgeIT — it has no NetworkIT peer-to-peer counterpart; only the engine-backed LlmRoundtripIT applies. Could not run `./mvnw verify` for the new LlmRoundtripIT in this sandboxed session: the reactor's maven-notice-plugin license-mapping check fails on unmapped artifacts regardless of network access, the same pre-existing, environment-specific limitation noted in #2576's own test plan. Verified instead: `./mvnw checkstyle:check` (0 violations) and `./mvnw package` (compiles cleanly) for incubator/binding-llm.spec and incubator/binding-llm. CI should confirm the full verify lifecycle. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…ng coverage Revert the LlmRoundtripIT approach (chaining llm:server directly into llm:client in one config): it combined two bindings under test in a single IT, which this repo avoids in favor of verifying each binding independently and inferring combinations. PR #2576 already authored the full request/streaming/nonstreaming × network/application script matrix, and ApplicationIT/NetworkIT already prove each application-level and network-level script pair is self-consistent. But engine-backed coverage was split: LlmServerIT only exercised openai.request, and LlmClientIT only exercised openai.streaming/openai.nonstreaming. Composing the already-existing peer-to-peer proofs with LlmServerIT's and LlmClientIT's own per-binding proofs already establishes the round-trip identity claim from #2491 (framing decode/re-encode byte-identical, no payload parsed via the existing .raw() flush-ext assertions, including tool-call streaming spanning DATA and FLUSH) without ever running server+client chained. Adds the two missing LlmServerIT methods (openai.streaming, openai.nonstreaming) using the scripts and config that already exist — no new files needed. Considered adding a symmetric openai.request method to LlmClientIT, but net/openai.request/server.rpt was authored as the peer for the server-detection scenario (path /v1/chat/completions) rather than as what llm:client's own fixed-dialect encoder actually emits (path "/", per the already-passing openai.streaming/openai.nonstreaming network scripts), so pairing them would fail for a script-mismatch reason unrelated to any real defect; left LlmClientIT unchanged rather than author a new script pair for it. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…lects Translates between each dialect's native streaming event sequence and the canonical vocabulary from #2557, in both directions: - LlmAnthropicEventMapper: holds input_tokens from message_start until the paired usage event at message_delta; tracks the currently open block's type to know when content_block_stop needs a canonical blockEnd (tool calls only) and to route content_block_delta payloads (text_delta vs input_json_delta) on encode. - LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into the canonical (Anthropic-shaped) block index via a per-stream map, and synthesizes blockEnd lazily -- deferred until the next tool call starts or the stream finishes, since OpenAI has no explicit block-close event. Unit-tested against both worked-example tables from the issue (message role/content/tool-call cardinality changes in each direction), plus the held-usage/already-consumed and lazy-blockEnd edge cases. Fixes #2482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw

LlmDialectResolver dispatches path/header detection across every LlmDialect registered via LlmDialectFactorySpi, without hardcoding any dialect's signals. A configured fixed dialect name bypasses detection entirely, including when it matches no registered dialect. When detection matches more than one dialect, or none, resolution is ambiguous and returns null so the caller rejects the request rather than guessing. LlmOptionsConfig adds the optional server-kind `dialect` option (schema, config, and adapter) used to pin a fixed dialect. Fixes #2483 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AtkteS7C66qiVLGeEDJ2FP

The sys: namespace's patch contract (Binding.system()/Exporter.system()) lets a component contribute a shared binding (e.g. http_client) but has no pre-seeded slot for a shared catalog, so a patch adding one has nothing to append into. Pre-seed catalogs: {} alongside the existing bindings: {} in the base sys namespace skeleton, the same "pre-seed the extension point" convention already used for the JSON-schema *-ext scaffolds, so any component can share a catalog-backed resource the way bindings are already shared. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

…ect schemas Adds LlmDialectFactorySpi.schema(Kind), letting a dialect contribute a URL to its own JSON schema for the request or response direction without knowing anything about catalogs or patches. LlmBinding.system() enumerates every registered dialect, reads each contributed schema, and generates a sys: namespace patch adding one shared inline catalog (llm_dialects) with a <dialect>.request/<dialect>.response subject per schema a dialect contributes -- built once, at engine startup, since the dialect set is ServiceLoader-discovered off the classpath and therefore fixed for the JVM's life, the same way sys: already shares a binding (e.g. http_client) across every binding that references it. LlmDataUrlStreamHandler decodes a base64 data: URL (RFC 2397) in memory, scoped to a single URL via the URL.of(URI, URLStreamHandler) factory -- no temporary file, no globally-registered protocol handler -- used to hand the generated patch to Binding.system()'s URL-returning contract without writing it to disk. LlmBeginEx gains contentType and model fields alongside dialect, for a server-kind stream factory to stamp once it has resolved the dialect and read the request's content-type. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

… LlmBeginEx.dialect Implements request/response framing decode-and-forward for the LLM server binding (#2484): LlmServerFactory drives the request body through the resolved dialect's ModelPipeline as bytes arrive off the wire, forwards transformed content to app0 incrementally rather than buffering the whole body, and threads the resolved dialect name onto LlmBeginEx so app0 can see which wire dialect produced the request. Flow control between the client, this binding, and app0 is enforced with dedicated decodeSlot/encodeSlot buffers on each side of the exchange (LlmServer for the network-facing leg, LlmStream for the app-facing leg), each granting credit strictly from its own local slot occupancy rather than copying a peer's sequence numbers across independent byte domains. LlmState tracks per-direction open/closing/closed transitions and end-of-stream deferral while a buffer is still draining. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

…fferPool handles DefaultBufferPool.buffer(slot) rewraps and returns a single shared mutable field per pool instance, so holding a buffer reference across a nested call that fetches a different slot from the same pool silently repoints it. LlmServer.decodeNetwork() directly, synchronously calls LlmStream's request-relay staging method as a plain nested Java method call, so a single pool instance backing both the network-decode slot and the app0 request-relay slot would alias between them. Give LlmServerFactory two BufferPool handles instead of one: decodePool (network decode) and encodePool (app0 relay, both directions), obtained via context.bufferPool() and .duplicate() -- matching McpServerFactory's decodePool/encodePool precedent. The two relay directions sharing encodePool (the reply-direction slot on LlmServer and the request-direction slot on LlmStream) never appear in the same call stack: cross-binding accept() is ring-buffer-mediated and dispatched on a later engine tick, not a nested call, so one encodePool instance can't alias between them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM

…openai dialect LlmOpenAiRequestTransform renamed a fixed set of top-level fields but never observed model, so LlmServerFactory.doAppBegin's server.envelope.get("model", 0) read right after running the request through this transform came back empty and LlmBeginEx.model was never stamped for real dialect: openai traffic. Mirrors LlmTestPermissiveDialect's inline ModelExtractTransform (and KafkaExtractTransform's own pattern): on a FIELD event at $.model, copy the value into the envelope alongside the existing rename-or-forward decision, without touching the RENAMES table. LlmOpenAiResponseTransform needs no equivalent change -- nothing reads model back off a response. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

…requests CI's LlmServerIT.shouldRejectRequestWithUnresolvedDialect failed: detect() matched on :method/:path alone, so it ambiguously co-matched every existing k3po server fixture -- all of which hit the same /v1/chat/completions path with a test-specific content-type (application/vnd.zilla.test-permissive+json, application/vnd.zilla.test-strict+json) to select a *different* dialect unambiguously. Only one failure surfaced in CI because failsafe stops after the first failure, but the same ambiguity affected every other fixture in that class too -- confirmed by LlmServerIT going from 6 run/1 failed/1 skipped to 8/8 passing with this fix, and LlmClientIT unaffected at 4/4. detect() now also requires content-type: application/json, the only content-type a genuine OpenAI request ever carries, so a request to the same path with a different dialect's own content-type no longer collides. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMybNYCLJEWTwCjDscepZm

The llm.schema.patch.json allowed options.server on kind:server bindings, but LlmServerFactory never reads binding.options.server — only LlmClientFactory dials it as the upstream endpoint. Restrict the kind:server options schema to dialect (fixed-dialect mode), matching actual runtime usage and the milestone's example configs. Add LlmSchemaValidationTest exercising the full EngineConfigReader pipeline against real zilla.yaml text for both kind:server and kind:client, covering acceptance (bare server, fixed-dialect server, full client) and rejection (server option on kind:server, missing required client fields, malformed server pattern, unknown kind). Fixes #2488 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Aawd4ZFaUE3FgQnzkABj9F

LlmOptionsConfigAdapterTest: round-trip dialect+server together, an explicit server-absent case, and the adapter's silent no-op on a malformed server string (schema is the actual gatekeeper there). LlmSchemaValidationTest: reject additionalProperties on both kind:server and kind:client, reject non-string dialect/server values, and accept an empty options object for kind:server. Closes #2489. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfesxqkqmMKPC87ydouhGR

Adds paired client.rpt/server.rpt k3po scenarios exercising the real `openai` LlmDialect end-to-end (dialect detection, framing decode/encode), rather than the synthetic test-sse/test-conditional dialects existing scenarios use: - openai.request (network+application): llm server detects `dialect: openai` from `POST /v1/chat/completions` alone (no x-llm-dialect header) and forwards the decoded JSON request body, terminated by the JSON content-decoder's single terminal flush. - openai.streaming (network+application): llm client same-dialect round-trip with a `stream:true` request and a realistic OpenAI SSE response sequence (role chunk, content chunk, tool-call start and argument-fragment chunks, finish_reason chunk, usage chunk, [DONE]). - openai.nonstreaming (network+application): llm client same-dialect round-trip with a `stream:false` request and a plain-JSON response carrying top-level usage. New scenarios are wired into LlmServerIT/LlmClientIT (engine-driven) and ApplicationIT/NetworkIT (protocol self-consistency), plus a new client.openai.yaml config pinning `dialect: openai`. All four IT classes compile cleanly and pass checkstyle; the k3po `verify` lifecycle itself could not be exercised in this sandboxed session due to a pre-existing, repo-wide maven-notice-plugin license-mapping gap unrelated to this change (same limitation noted in prior binding-llm PRs' test plans) — CI should confirm. Fixes #2490 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…p+passthrough server LlmServerFactory no longer decodes request framing into typed DATA/FLUSH events -- it detects the dialect, resolves content-type directly off the real Content-Type header, runs the dialect's decoder through a json-model ModelPipeline, and forwards the pipeline's own output as plain DATA frames, stamping dialect/contentType on the app-facing LlmBeginEx. Update the openai.request client/server pair (network + application) to match: a real content-type header drives LlmBeginEx.contentType, and the terminal LlmNativeFlushEx this scenario previously asserted is gone since the server no longer produces one for plain JSON forwarding. model is intentionally left unasserted here: the real openai dialect does not yet extract it into the envelope (only the test-permissive test dialect does), unlike the design this fixture set was originally written against. Part of the broader LlmDialect/ModelPipeline SPI redesign already landed on this branch's upstream chain; openai.streaming/openai.nonstreaming still need the same treatment for LlmClientFactory's same-dialect path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…i client fixtures LlmClientFactory always reads content-type off real HTTP headers -- the request's own if the app sets one, else "application/json" -- and its response codec is selected by looking up the backend's real Content-Type response header via LlmContentCodecFactory (registered for "text/event-stream" and "application/json"). A response header carrying neither falls through to opaque forwarding instead of the intended codec. openai.streaming/openai.nonstreaming previously omitted these headers entirely, so despite exercising a real dialect they were never actually routing through the SSE/JSON content codecs LlmClientIT is meant to cover. Add "content-type: application/json" to the outbound request and "content-type: text/event-stream"/"application/json" to the backend response on both the network and application sides, matching every other LlmClientIT-driven fixture's convention (same.dialect, cross.dialect). openai.nonstreaming was also missing the terminal raw LlmFlushEx that LlmJsonContentDecoder always emits alongside its single DATA -- add it to both the read and write side, matching the already-correct openai.streaming pattern. Also make openai.request's application/server.rpt echo the same request body back as its reply (rather than an unrelated "response body" literal), matching request.valid's own client/server symmetry convention, with the network pair's expected reply content updated to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…eginEx The #2487 session landed model extraction for the real openai dialect (LlmOpenAiRequestTransform now observes $.model and copies it into the ModelEnvelope, mirroring LlmTestPermissiveDialect's ModelExtractTransform). Now that LlmServerFactory.doAppBegin actually stamps LlmBeginEx.model for dialect: openai traffic, assert it in the one fixture pair that exercises server-side model extraction (openai.streaming/openai.nonstreaming never touch model at all -- LlmClientFactory doesn't read or stamp it in either direction). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

… traffic LlmClientFactory always constructed and drove a schema-validating JSON model pipeline for both request and response, even for same-dialect traffic where the design intends pure raw byte relay with no transform. For a dialect that declares no JSON schema for a direction (openai declares none for either), the pipeline's schema resolution always returns NO_SCHEMA_ID, so every same-dialect request or response got REJECTED and the stream was torn down -- reproduced directly against JsonModelHandlerImpl/JsonModelDecoderPipeline with the same no-schema/ModelTransform.NONE configuration LlmClientFactory builds. Skip constructing requestPipeline/responsePipeline entirely when source == target, forwarding request and response bytes directly (still through the content-type codec's own encoder/decoder for framing) instead of driving a pipeline with nothing to validate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…t client traffic" This reverts commit 45e5e52.

… schemas LlmOpenAiDialectFactorySpi previously declared no schema for either direction, so the shared sys:llm_dialects catalog had no "openai.request"/ "openai.response" subject and the ModelPipeline LlmClientFactory drives for openai traffic (same-dialect included) always resolved NO_SCHEMA_ID and rejected every value, regardless of content -- there was no actual validation happening for openai traffic at all. Add openai.request.schema.json/openai.response.schema.json describing the real Chat Completions wire shapes: request requires model/messages and types the other fields LlmOpenAiRequestTransform's rename table and model-extraction both recognize (stream, max_tokens, top_p, n, presence_penalty, frequency_penalty, tool_choice, response_format, ...); response types choices[]/usage without requiring any top-level member, since a streaming chunk carries only a subset (delta vs message, finish_reason, tool_calls[], usage) of what a single non-streaming completion carries all at once. Verified directly against JsonModelHandlerImpl/JsonModelDecoderPipeline (the same construction LlmClientFactory drives) that all nine request/ response payloads used by the openai.request/openai.streaming/ openai.nonstreaming k3po fixtures validate as COMPLETE against these schemas, not REJECTED. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…heck Asserting the schema's required fields and specific property keys in a unit test duplicates what the openai.request/openai.streaming/ openai.nonstreaming k3po fixtures already verify against a live engine -- those fixtures are the actual spec for what these schemas must accept, per this repo's test-first discipline. Narrow the unit test to what a factory-level test is actually entitled to check: the schema resource for each Kind exists and parses as a JSON object. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

Engine.java's bootstrap (bindings.stream().map(Binding::system)...) calls LlmBinding.system() -> LlmSystemNamespaceGenerator.generate() for every engine startup in this module, which invokes schema(Kind) for every registered LlmDialectFactorySpi (openai included) for both REQUEST and RESPONSE and reads the resource -- unconditionally, on every LlmServerIT/ LlmClientIT/ApplicationIT/NetworkIT run in the module, not just openai-specific scenarios. A missing or unreadable schema resource would already fail engine bootstrap loudly. The unit test duplicated coverage the k3po ITs already provide for free. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

Adds the negative-path coverage the openai.request/streaming/nonstreaming scenarios never exercised: an invalid document actually gets rejected, not silently forwarded. openai.request.invalid (network only, mirroring request.rejected.schema's existing single-sided pattern): a real openai-dialect request missing the required "messages" field -- llm(server) rejects it before ever opening an app-facing stream, observed as the k3po connect script itself getting aborted. openai.response.invalid (both application and network, mirroring the full openai.streaming/nonstreaming layout): a real backend response with "choices" as a string instead of an array -- llm(client) rejects it and aborts app0 instead of forwarding the malformed document. Wired into LlmServerIT/LlmClientIT (engine-driven) and ApplicationIT/NetworkIT (protocol self-consistency), matching the existing openai.* scenario conventions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…i traffic llm(client) computed requestContentType from llmBeginEx.contentType() with a wrong null-check: the flyweight accessor itself is never null even when the underlying string16 field is unset, so the fallback to CONTENT_TYPE_JSON never triggered and null flowed into the outbound HTTP begin's content-type header, breaking header matching on the network side and hanging every k3po scenario that omits contentType on the app-side llm:beginEx (which is the common case -- the field is only meant to be set for cross-dialect routes). Check contentType().asString() directly instead, matching the existing idiom one line above for the dialect field. Also stop routing same-dialect responses through a schema-validating ModelPipeline: responsePipeline was unconditionally constructed regardless of dialect, so a same-dialect stream (zero decode/encode/transform by design, per LlmOpenAiResponseTransform's own documented contract) still paid for JSON-schema validation on every response chunk, including the OpenAI SSE `[DONE]` sentinel, which is not JSON and got truncated by the validator. responsePipeline is now null for sameDialect, and forwardResponseContent forwards raw bytes in that case, mirroring how requestPipeline already goes null when no encoder resolves. Removes LlmClientIT.shouldRejectInvalidOpenAiResponse: it asserted schema rejection of an invalid response body on a same-dialect (single-dialect) client config, which cannot validate anything now that same-dialect responses bypass the model pipeline entirely -- this was always the intended contract, just not one this scenario could have exercised. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…e [DONE] sentinel The previous commit disabled response schema validation entirely for same-dialect llm(client) traffic to work around the OpenAI SSE `[DONE]` sentinel getting mangled by the JSON model pipeline. That was the wrong fix: `[DONE]` isn't JSON at all (it's an SSE-level stream-termination token, not a chat-completion chunk), so no JSON transform can "honor identity" for it -- there's no valid parse to preserve. The actual defect was routing a non-JSON control token into JSON-schema validation at all, not the presence of validation itself. Restores responsePipeline unconditionally (same-dialect responses are schema-validated exactly like cross-dialect ones), and instead adds a narrow, explicit bypass in forwardResponseContent for the literal `[DONE]` bytes specifically, forwarding them raw before they ever reach the model pipeline. Genuine JSON content -- valid or invalid, same-dialect or cross-dialect -- is still schema-validated and rejected on violation. Restores LlmClientIT.shouldRejectInvalidOpenAiResponse, which the previous commit had removed as no longer exercisable; it now passes again since same-dialect responses are validated once more. Also corrects LlmOpenAiResponseTransform's class Javadoc, which overstated the same-dialect bypass as covering the whole path rather than just the non-JSON sentinel. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

… responses LlmServerFactory.LlmStream.onAppBegin forwarded app0's reply BEGIN extension to net0 verbatim. Since the k3po app-side accept scripts (and presumably any conformant llm app) never write an explicit begin.ext before their response body, this extension is empty -- meaning every llm(server) HTTP response went out with no status line at all, not even ":status 200". That's invalid per HTTP, and any net-side reader asserting on the response BEGIN (matching ":status") would wait forever for bytes that would never arrive. This only ever surfaced as a hang on `shouldDetectOpenAiDialectFromPath` because the openai scenario's client script is the only one that actually asserts on the reply BEGIN's ":status" header; existing passing scenarios (e.g. request.valid) read straight into the body without checking it, silently tolerating the missing status line. doNetBegin now builds a real HttpBeginExFW itself (":status 200", plus "content-type" echoing the same contentType recorded from the request) instead of relaying whatever the app happened to send. Also fixes streams/network/openai.request.invalid/client.rpt: it ended with `read aborted`, which is the accept-side spelling for observing a peer's abort -- the connect-side script that unilaterally tears down its own write direction after sending invalid content uses `write abort`, matching the established sibling scenario (request.rejected.schema). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…actory into LlmDialect LlmClientFactory hardcoded the literal "[DONE]" bytes as a static constant and compared incoming response chunks against it directly -- OpenAI-specific wire knowledge baked into the dialect-generic client factory. Any other dialect with its own out-of-band stream-termination convention would have needed a second such literal added to the same generic file. Adds LlmDialect.terminator(Kind), returning the literal byte sequence a dialect's kind stream uses to signal completion out of band from any document, or null when it has none. Plain abstract method (no default), matching every other method on this interface -- implemented directly by all seven current dialects: LlmOpenaiDialect returns the real "[DONE]" bytes for RESPONSE (null for REQUEST, since chat completions requests are always a single plain JSON body), the six test dialects all return null. LlmClientFactory resolves target.terminator(Kind.RESPONSE) once per stream at construction (mirroring how requestEncoder/responsePipeline are already resolved once), and forwardResponseContent now wraps the observed slice into a reused comparison flyweight (comparisonRO, following the same reuse-a-field pattern OctetsFW's own valueRO uses) and compares it against the resolved terminator via DirectBufferEx.equals() -- a positive, per-dialect byte match, not an inference from any model-pipeline rejection reason, so a genuinely malformed or truncated chunk still reaches the pipeline and gets rejected rather than being silently forwarded as if it were the terminator. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5Pw5eDcghnK7LmZ11QA7u

…lient Chains an llm:server (auto-detecting dialect: openai from POST /v1/chat/completions) directly into an llm:client (dialect: openai) against a mock OpenAI backend, proving the server→client hop preserves framing byte-for-byte with no payload re-serialization. Adds config/roundtrip.openai.yaml wiring net0 (llm:server) → app0 (llm:client, internal) → net1 (external, mock backend), modeled on binding-tls's bridge.tls1.3.yaml chaining pattern. New streaming and non-streaming network scenarios reuse the wire content already proven by openai.streaming/openai.nonstreaming (including the tool-call streaming chunks spanning DATA and FLUSH), with deliberate insignificant whitespace (e.g. "\"model\": \"gpt-4\"") added to request/response bodies at every hop — this survives only if the body is forwarded unchanged rather than parsed and re-serialized, since same-dialect resolution takes LlmClientFactory's ModelTransform.NONE fast path and the server's json-model pipeline forwards the body unmodified. This scenario chains two different bindings across two different wire legs (net0's pre-server-decode bytes vs net1's post-client-encode bytes) rather than describing one client/server exchange, so — like binding-tls's BridgeIT — it has no NetworkIT peer-to-peer counterpart; only the engine-backed LlmRoundtripIT applies. Could not run `./mvnw verify` for the new LlmRoundtripIT in this sandboxed session: the reactor's maven-notice-plugin license-mapping check fails on unmapped artifacts regardless of network access, the same pre-existing, environment-specific limitation noted in #2576's own test plan. Verified instead: `./mvnw checkstyle:check` (0 violations) and `./mvnw package` (compiles cleanly) for incubator/binding-llm.spec and incubator/binding-llm. CI should confirm the full verify lifecycle. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…ng coverage Revert the LlmRoundtripIT approach (chaining llm:server directly into llm:client in one config): it combined two bindings under test in a single IT, which this repo avoids in favor of verifying each binding independently and inferring combinations. PR #2576 already authored the full request/streaming/nonstreaming × network/application script matrix, and ApplicationIT/NetworkIT already prove each application-level and network-level script pair is self-consistent. But engine-backed coverage was split: LlmServerIT only exercised openai.request, and LlmClientIT only exercised openai.streaming/openai.nonstreaming. Composing the already-existing peer-to-peer proofs with LlmServerIT's and LlmClientIT's own per-binding proofs already establishes the round-trip identity claim from #2491 (framing decode/re-encode byte-identical, no payload parsed via the existing .raw() flush-ext assertions, including tool-call streaming spanning DATA and FLUSH) without ever running server+client chained. Adds the two missing LlmServerIT methods (openai.streaming, openai.nonstreaming) using the scripts and config that already exist — no new files needed. Considered adding a symmetric openai.request method to LlmClientIT, but net/openai.request/server.rpt was authored as the peer for the server-detection scenario (path /v1/chat/completions) rather than as what llm:client's own fixed-dialect encoder actually emits (path "/", per the already-passing openai.streaming/openai.nonstreaming network scripts), so pairing them would fail for a script-mismatch reason unrelated to any real defect; left LlmClientIT unchanged rather than author a new script pair for it. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…lient Chains an llm:server (auto-detecting dialect: openai from POST /v1/chat/completions) directly into an llm:client (dialect: openai) against a mock OpenAI backend, proving the server→client hop preserves framing byte-for-byte with no payload re-serialization. Adds config/roundtrip.openai.yaml wiring net0 (llm:server) → app0 (llm:client, internal) → net1 (external, mock backend), modeled on binding-tls's bridge.tls1.3.yaml chaining pattern. New streaming and non-streaming network scenarios reuse the wire content already proven by openai.streaming/openai.nonstreaming (including the tool-call streaming chunks spanning DATA and FLUSH), with deliberate insignificant whitespace (e.g. "\"model\": \"gpt-4\"") added to request/response bodies at every hop — this survives only if the body is forwarded unchanged rather than parsed and re-serialized, since same-dialect resolution takes LlmClientFactory's ModelTransform.NONE fast path and the server's json-model pipeline forwards the body unmodified. This scenario chains two different bindings across two different wire legs (net0's pre-server-decode bytes vs net1's post-client-encode bytes) rather than describing one client/server exchange, so — like binding-tls's BridgeIT — it has no NetworkIT peer-to-peer counterpart; only the engine-backed LlmRoundtripIT applies. Could not run `./mvnw verify` for the new LlmRoundtripIT in this sandboxed session: the reactor's maven-notice-plugin license-mapping check fails on unmapped artifacts regardless of network access, the same pre-existing, environment-specific limitation noted in #2576's own test plan. Verified instead: `./mvnw checkstyle:check` (0 violations) and `./mvnw package` (compiles cleanly) for incubator/binding-llm.spec and incubator/binding-llm. CI should confirm the full verify lifecycle. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

…ng coverage Revert the LlmRoundtripIT approach (chaining llm:server directly into llm:client in one config): it combined two bindings under test in a single IT, which this repo avoids in favor of verifying each binding independently and inferring combinations. PR #2576 already authored the full request/streaming/nonstreaming × network/application script matrix, and ApplicationIT/NetworkIT already prove each application-level and network-level script pair is self-consistent. But engine-backed coverage was split: LlmServerIT only exercised openai.request, and LlmClientIT only exercised openai.streaming/openai.nonstreaming. Composing the already-existing peer-to-peer proofs with LlmServerIT's and LlmClientIT's own per-binding proofs already establishes the round-trip identity claim from #2491 (framing decode/re-encode byte-identical, no payload parsed via the existing .raw() flush-ext assertions, including tool-call streaming spanning DATA and FLUSH) without ever running server+client chained. Adds the two missing LlmServerIT methods (openai.streaming, openai.nonstreaming) using the scripts and config that already exist — no new files needed. Considered adding a symmetric openai.request method to LlmClientIT, but net/openai.request/server.rpt was authored as the peer for the server-detection scenario (path /v1/chat/completions) rather than as what llm:client's own fixed-dialect encoder actually emits (path "/", per the already-passing openai.streaming/openai.nonstreaming network scripts), so pairing them would fail for a script-mismatch reason unrelated to any real defect; left LlmClientIT unchanged rather than author a new script pair for it. Closes #2491 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UogzKHEA7zC9PgHBkGwu8p

claude added 9 commits September 11, 2026 19:43

jfallows force-pushed the claude/charming-faraday-hv7lvz branch 2 times, most recently from ab9987b to 5ca69a5 Compare September 16, 2026 05:48

jfallows force-pushed the claude/charming-faraday-hv7lvz branch from 5ca69a5 to 13e12c5 Compare September 16, 2026 14:23

This was referenced Sep 16, 2026

test(binding-llm): round-trip identity coverage, openai server → openai client #2577

Open

feat(binding-llm): anthropic dialect — detection, request/response transforms #2578

Open

claude added 6 commits September 16, 2026 21:19

claude added 4 commits September 16, 2026 21:20

jfallows force-pushed the claude/charming-faraday-hv7lvz branch from 4901677 to e7df219 Compare September 16, 2026 22:10

claude added 14 commits September 17, 2026 00:19

Revert "fix(binding-llm): skip ModelPipeline entirely for same-dialec…

6f31b59

…t client traffic" This reverts commit 45e5e52.

jfallows force-pushed the claude/charming-faraday-hv7lvz branch from 151acc0 to 86aed4c Compare September 17, 2026 00:21

Navigation Menu

test(binding-llm): k3po fixtures for real dialect:openai backend - #2576

test(binding-llm): k3po fixtures for real dialect:openai backend#2576
jfallows wants to merge 45 commits into
developfrom
claude/charming-faraday-hv7lvz

jfallows commented Sep 12, 2026 •

edited

Loading

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jfallows commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Stacking

Test plan

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

jfallows commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfallows commented Sep 12, 2026 •

edited

Loading