llm-stream-test

Test the code that reads an LLM stream.

LLM APIs stream their answer as Server-Sent Events. The network delivers those bytes in chunks whose boundaries have nothing to do with the events: a chunk can end mid-line, mid-JSON, or even mid-character. Reader code written the obvious way — decode the chunk, split by newline, JSON.parse each line — silently drops events or throws, depending on how the bytes happen to arrive. It works on your machine and loses half the answer behind one proxy. On a recording of a real backend, 688 of the 1452 ways to cut the stream broke such a reader — including the chunks exactly as the network delivered them.

This package records a real response once, then replays it every way the network can cut it, and requires the same result every time. When a cut changes the result, the failure names the kind of cut, points at the byte, and hints at the usual cause.

Quick start

1. Record one real response. Copy the request from DevTools ("Copy as cURL") and paste it after the options — the leading curl may stay:

npm i -D llm-stream-test
npx llm-stream-test record --out fixtures/chat.json \
  curl 'https://api.example.com/chat' \
    -H 'authorization: Bearer …' \
    --data-raw '{"messages":[…]}'

The file stores the chunks exactly as the server sent them, with timings. Secret-looking names are redacted to [REDACTED] — in headers (authorization, x-api-key, cookie, …), query parameters, the request body (JSON fields and form fields: api_key, accessToken, secret, password, credential, …) and user:password@ in the URL. A bearer token or a JWT is redacted whatever name it goes under. So the file can be committed. A chunk that ends mid-character is stored base64. Commit it: tests run offline, in milliseconds, forever.

2. Test your reader. The code under test gets a ReadableStream<Uint8Array> — the type response.body gives it in the browser and in Node:

import { it } from "vitest";
import { everyCut, fixture } from "llm-stream-test";

it("reads the stream the same however the network cuts it", async () => {
  await everyCut(fixture("fixtures/chat.json"), (body) => readChat(body));
});

fixture takes a path or a URL. Jest runs the same test with const { everyCut, fixture } = require("llm-stream-test") — a CommonJS build is shipped. For code that reads through XMLHttpRequest — React Native and the like — hand it a fake XHR instead:

await everyCutXhr(fixture("fixtures/chat.json"), (xhr) => {
  xhr.open("POST", "https://api.example.com/chat");
  xhr.send();
  return readAnswerFromXhr(xhr); // the code under test: onprogress, responseText…
});

The fake fires events in the browser's order (text grows → readystatechange → progress; load/error/timeout at the end), decodes text across chunk borders the way React Native does, and an error thrown inside a handler counts as the result — a browser only logs it, and the answer quietly stays unfinished.

3. Read the report. Real output, a naive reader on a small recording:

everyCut: 169 of 183 cuts changed the result (stream of 161 bytes).

  ✗ cut inside a line           145 of 149 failed   usually: keep the unfinished line until the next chunk
  ✗ cut inside a character        4 of 4 failed   usually: a cut right before the letter fails too: fix that first
  ✓ cut right after a line        0 of 4 failed
  ✓ cut right after an event      0 of 3 failed
  ✗ every byte apart              1 of 1 failed
  ✗ random pieces                19 of 20 failed
  ✓ all in one chunk              0 of 1 failed
  ✓ as recorded                   0 of 1 failed

✗ split@1: cut inside a line, line 1
    d✂ata: {"type":"delta","te…
  expected  'Hello, naïve reader 👋'
  received  ', naïve reader 👋'

✗ split@74: cut inside "ï" (bytes C3 | AF), line 3
    …e":"delta","text":", na[ï]ve"}\n\ndata: {"type":"de…
  expected  'Hello, naïve reader 👋'
  received  threw SyntaxError: Unterminated string in JSON at position 29 (line 1 column 30)

✗ every-byte: every byte as its own chunk
  expected  'Hello, naïve reader 👋'
  received  ''

Rerun one cut: everyCut(recording, consume, { only: "split@1" })

One line per kind of cut, with the usual cause. One example per failing kind, with the cut marked ✂ and both outcomes around their first difference. And a ready line to rerun just that cut in a debugger.

What everyCut checks

• the chunks as recorded, all in one chunk, every byte apart;
• a single cut at every position of the stream, classified: inside a line, inside a character, between \r and \n, right after a line, right after an event (above 2000 positions, sampled evenly across the kinds);
• 20 random cuts with a fixed seed — the same ones every run;
• a recording of a network error must produce the same error on every cut.

The expected result comes from one event per chunk, cross-checked against all in one chunk. If the two disagree, neither is trusted: the failure shows both and the two usual causes, instead of quietly picking a side. And when a cut inside a line, a letter or \r\n fails, the same cut is retried where that line or letter starts — if it fails there too, the hint says "fix that first", because the split itself is not the problem.

If nothing happens for hangAfter ms (default 1000), the run stops and says what the code was doing: "the code has not read the stream", "the code has not called xhr.send()", "the stream had ended".

Replay instead of the real API

The same recording can answer requests, so the frontend runs from a file and spends no tokens. The bytes, the chunk boundaries, the status and the headers are the server's own, and nothing leaves the machine. replayFetch returns a function that works like fetch: put it where your code calls fetch.

In a test — the code's real path, offline:

globalThis.fetch = replayFetch(fixture("fixtures/chat.json"));

In your code, while you work on it — swap fetch for the replay where the request is made, and the UI gets the recorded answer at the server's pace:

import { replayFetch } from "llm-stream-test/replay"; // no Node inside
import type { Recording } from "llm-stream-test";
import chat from "./fixtures/chat.json";

// Once, at module level. Production keeps the real fetch.
const chatFetch =
  process.env.NODE_ENV === "production"
    ? fetch
    : replayFetch(chat as Recording, { timing: "recorded" });

// was: await fetch("/api/chat", { … })
const response = await chatFetch("/api/chat", { method: "POST", body, signal });

The whole app without a backend — for a demo, or an evening on the UI: replace the global fetch once, before the app starts, behind a switch. In Next.js, in instrumentation-client.ts:

// NEXT_PUBLIC_REPLAY=1 npm run dev
import { replayFetch } from "llm-stream-test/replay";
import type { Recording } from "llm-stream-test";
import first from "./fixtures/chat-1.json";
import second from "./fixtures/chat-2.json";

if (process.env.NEXT_PUBLIC_REPLAY === "1") {
  window.fetch = replayFetch([first, second] as Recording[], {
    timing: "recorded",
    passThrough: true, // other requests go to the real API
  });
}

In Vite, the same goes at the top of main.tsx, with import.meta.env.VITE_REPLAY. Keep the switch: without it, the recordings answer real users in production.

In the app, import recordings as JSON: fixture reads from disk, in Node only.

A recording matches by method and path; the origin and the query are ignored. A bare recording.exchanges[i].response answers every request, and an array of recordings serves several endpoints. Several recordings of one endpoint answer in turn, the first again after the last: record a few answers, and the chat goes on. Create the replay once — a new one starts from the first recording again. With timing: "recorded" the replay keeps the server's pace — and a recorded network error ends the stream the same way, so a demo can show how the UI survives a dropped connection. By default a request no recording matches fails loudly, so a test never reaches the network by accident.

API

everyCut(source, consume, options?)

source — a recording, or recording.exchanges[i].response for one exchange of a multi-exchange recording. consume — the code under test: receives the ReadableStream<Uint8Array>, returns a value (compared deep-equal across cuts); a thrown error is a result too. Resolves { checked } when every cut gives the same result, throws a report otherwise.

everyCutXhr(source, consume, options?)

The same for XHR readers. consume receives an object that behaves like XMLHttpRequest: call open() and send() as usual, set onprogress and the other handlers, read the growing responseText.

fixture(path)

Reads and validates a recording file made by record. Takes a path or a URL.

replayFetch(source, options?)

A drop-in fetch answered from recordings — see Replay instead of the real API. source is a recording, a bare response, or an array of either. In app code, import it from llm-stream-test/replay: the same function, without the Node-only parts of the package.

Option	Default	Meaning
timing	"instant"	"recorded" waits between chunks the way the server did, network errors included
passThrough	false	Send unmatched requests to the real fetch instead of failing

Options

Option	Default	Meaning
maxSplits	2000	Most single cuts to try; a shorter stream tries every position
randomRuns	20	Random cuts
seed	1	Seed of the random cuts, so runs repeat
only	—	Run just this cut, e.g. "split@57" — 57 is the byte offset
show	3	Failure examples in detail, one per kind (1 minimum)
hangAfter	1000	Ms without progress before the run counts as stuck; raise it if the code is legitimately slow

Requirements

Node ≥ 22. The shipped types assume TypeScript ≥ 5.7 and Node's globals (ReadableStream); everyCutXhr's signature references DOM's XMLHttpRequest. With skipLibCheck: true — the default in Vitest, Next.js and React Native configs — this just works; with skipLibCheck: false add the usual libs (dom or @types/node).

Big recordings

A 150 KB stream takes about 1.5 s to check on a laptop — over two thousand deliveries. Vitest's default test timeout is 5 s, and CI runners are slower: for a large recording, lower maxSplits or raise the test timeout.

CLI

Usage: record --out <file.json> [--timeout <ms>] [--notes <text>] <curl arguments>

Understood curl options: <url> or --url, -X/--request, -H/--header, -d/--data/--data-raw/--data-binary, -b/--cookie; --compressed, -L and -N change nothing and are accepted. -x/--proxy is ignored: the response is recorded straight from the server, so the file keeps the server's chunk boundaries, not the proxy's.

rechunk(size)

A dev-time helper for the other direction: cut a live stream into chunks of exactly size bytes — response.body.pipeThrough(rechunk(7)) — to watch a real UI deal with a hostile network.

Limits

• everyCut and fixture run in Node. Tests run in Vitest, or in Jest through require (a CommonJS build is shipped) — React Native's Jest tests run in Node too, which is the main target. The browser is a later step.
• Text streams (SSE and friends). Binary is not supported yet.
• A reader that is wrong the same way on every cut — \r\n never recognized, for one — passes everyCut. That is the job of a plain unit test with an expected answer.
• replayFetch works on the web only for now: in a browser app and in Node tests. Expo and React Native are the next goal.
• Reconnect checks (Last-Event-ID: continue after a drop, nothing lost or doubled) are planned.

Development

npm install
npm test        # vitest
npm run build   # tsc → dist/

MIT.