From 02fa557fe9463a5b622d11dd14e59ef6dc91860e Mon Sep 17 00:00:00 2001 From: liujiahua123123 Date: Wed, 28 Oct 2026 15:15:56 +0000 Subject: [PATCH] =?UTF-8?q?docs(realtime-stt):=20recommend=2020=E2=80=9340?= =?UTF-8?q?=20ms=20audio=20messages?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the 80–100 ms audio message recommendation. Any size from 20 to 100 ms works and gives the same results, but each result waits for the message that carries its audio (on average half a message), so smaller messages give lower latency; 20–40 ms is a good size for live audio. - WebSocket reference: the audio format paragraph says why, the audio message section recommends 20–40 ms and notes that clients which batch small writes should set TCP_NODELAY. - AsyncAPI: the message sequence and the audio message description. - Realtime overview: the stream-audio step, and the latency section says how much the message size adds. - Cookbooks: CHUNK_MS 100 -> 40 in the Python and Node file streamers and the voice agent, and the browser AudioWorklet posts 40 ms (sampleRate / 25). Every supported rate gives a whole number of samples per 40 ms message. Co-Authored-By: Claude Opus 5.5 --- api-reference/asyncapi-asr-realtime.yml | 8 +++++--- api-reference/endpoint/websocket/asr-realtime.mdx | 4 ++-- .../sdk-guide/cookbook/realtime-transcription.mdx | 12 ++++++------ .../sdk-guide/cookbook/realtime-voice-agent.mdx | 4 ++-- features/speech-to-text/realtime.mdx | 4 ++-- 5 files changed, 17 insertions(+), 15 deletions(-) diff --git a/api-reference/asyncapi-asr-realtime.yml b/api-reference/asyncapi-asr-realtime.yml index 0e21a0a..5822836 100644 --- a/api-reference/asyncapi-asr-realtime.yml +++ b/api-reference/asyncapi-asr-realtime.yml @@ -127,7 +127,7 @@ operations: **Message Sequence:** 1. Client sends `config` once, as the first message, within 10 seconds of connecting - 2. Client streams `audio` after `session_started`, 80–100 ms per message + 2. Client streams `audio` after `session_started`, 20–40 ms per message for live audio 3. Client can send `finalize`, `configure`, `context_add`, `keepalive` and `agent_speaking` at any time 4. Client sends `audio_done` when there is no more audio @@ -253,8 +253,10 @@ components: nothing in them is transcribed or counted as received audio. Any length; an odd trailing byte waits for the next message. description: | - Audio in the format from `config`. Send it after `session_started`; - 80–100 ms per message is a good size. A message can be at most 1 MiB. + Audio in the format from `config`. Send it after `session_started`. + Anything from 20 to 100 ms of audio per message works, and smaller + messages give lower latency; for live audio, 20–40 ms per message is a + good size. A message can be at most 1 MiB. examples: - payload: type: audio diff --git a/api-reference/endpoint/websocket/asr-realtime.mdx b/api-reference/endpoint/websocket/asr-realtime.mdx index 6f3d5fe..3616f5e 100644 --- a/api-reference/endpoint/websocket/asr-realtime.mdx +++ b/api-reference/endpoint/websocket/asr-realtime.mdx @@ -126,7 +126,7 @@ A value of the wrong type for `ignore_timestamps`, or a `diarize` value other th Send 16-bit little-endian PCM, mono, at one of the listed sample rates. Other encodings (such as μ-law, A-law or Opus) and stereo audio are `unsupported_audio`; decode phone audio in μ-law or A-law to 16-bit PCM first, and send it at 8000 Hz as is. Audio at 16000 Hz is used as is; every other rate is converted on the server, so you never need to resample. The conversion adds at most about 20 ms of delay. Every `*_ms` value is in milliseconds of your audio, whatever its rate. -An `audio` message can hold any number of bytes; an odd trailing byte waits for the next message. Send 80–100 ms of audio per message. A message can be at most 1 MiB, which holds 65.5 seconds of audio at 8 kHz or 10.9 seconds at 48 kHz. +An `audio` message can hold any number of bytes; an odd trailing byte waits for the next message. Anything from 20 to 100 ms of audio per message works and gives the same results, but each result waits for the message that carries its audio, so smaller messages give lower latency. For live audio, 20–40 ms per message is a good size. A message can be at most 1 MiB, which holds 65.5 seconds of audio at 8 kHz or 10.9 seconds at 48 kHz. #### Turn detection modes @@ -174,7 +174,7 @@ The default, the allowed range (inclusive) and the meaning depend on the mode: Audio in the format from `config`. `data` must be MessagePack `bin` holding the raw PCM bytes. Do not base64-encode the audio, and do not send it as a string or an array of numbers. If you do, you receive a recoverable `invalid_request`, and those messages are dropped: nothing in them is transcribed or counted as received audio. -Send audio after `session_started` (see [`config`](#config) for audio sent earlier). For live audio, send it as it is captured; 80–100 ms per message is a good size. You can send faster than real time, for example from a file: the server accepts it and reads it at a bounded rate, and results keep following the audio. +Send audio after `session_started` (see [`config`](#config) for audio sent earlier). For live audio, send it as it is captured, 20–40 ms per message (see [Audio format](#audio-format)). If your client batches small writes (Nagle's algorithm), turn that off with `TCP_NODELAY`; browsers and common WebSocket libraries already send each message right away. You can send faster than real time, for example from a file: the server accepts it and reads it at a bounded rate, and results keep following the audio. ### `finalize` diff --git a/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx b/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx index eb7c010..37a8d7b 100644 --- a/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx +++ b/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx @@ -31,7 +31,7 @@ You also need a 16-bit PCM mono WAV file, at 8000, 12000, 16000, 22050, 24000, 3 ## Recipe -The script connects to `wss://api.fish.audio/v1/asr/realtime`, sends a `config` message, waits for `session_started`, and then streams the file in 100 ms messages at real-time speed, as a live microphone would. It prints every message the server sends until the session ends. +The script connects to `wss://api.fish.audio/v1/asr/realtime`, sends a `config` message, waits for `session_started`, and then streams the file in 40 ms messages at real-time speed, as a live microphone would. It prints every message the server sends until the session ends. @@ -53,7 +53,7 @@ import msgpack import websockets URL = "wss://api.fish.audio/v1/asr/realtime" -CHUNK_MS = 100 # 80-100 ms of audio per message +CHUNK_MS = 40 # ms of audio per message; 20-40 ms suits live audio def pack(message): @@ -134,7 +134,7 @@ import { encode, decode } from "@msgpack/msgpack"; import WebSocket from "ws"; const URL = "wss://api.fish.audio/v1/asr/realtime"; -const CHUNK_MS = 100; // 80-100 ms of audio per message +const CHUNK_MS = 40; // ms of audio per message; 20-40 ms suits live audio // Read a 16-bit mono PCM WAV file: its sample rate and its samples. function readWav(path) { @@ -378,10 +378,10 @@ if __name__ == "__main__": decode, } from "https://cdn.jsdelivr.net/npm/@msgpack/msgpack@3.1.2/+esm"; - // Runs on the audio thread: float samples in, 16-bit PCM out, ~100 ms per message. + // Runs on the audio thread: float samples in, 16-bit PCM out, 40 ms per message. const WORKLET = ` class Pcm16 extends AudioWorkletProcessor { - constructor() { super(); this.size = Math.round(sampleRate / 10); this.buf = new Int16Array(this.size); this.n = 0; } + constructor() { super(); this.size = Math.round(sampleRate / 25); this.buf = new Int16Array(this.size); this.n = 0; } process([input]) { const ch = input[0] || []; for (let i = 0; i < ch.length; i++) { @@ -491,7 +491,7 @@ registerProcessor("pcm16", Pcm16);`; Install `aiohttp`, put both files in one folder, run `python relay.py`, and open `http://localhost:8080`. - The page captures audio at the device's own sample rate and sends that rate in `config`; the server converts it, so the page does not resample. The rate must be one of the supported rates. -- An `AudioWorklet` converts the samples to 16-bit PCM and posts about 100 ms per message. +- An `AudioWorklet` converts the samples to 16-bit PCM and posts 40 ms per message. - **Stop** sends `audio_done`, so the server commits the last turn before it closes the connection. - The relay forwards the API's close code to the page. In production, add your own user authentication, per-user session limits and input checks to the relay. diff --git a/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx b/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx index 7809082..867dce2 100644 --- a/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx +++ b/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx @@ -71,7 +71,7 @@ import msgpack import websockets URL = "wss://api.fish.audio/v1/asr/realtime" -CHUNK_MS = 100 +CHUNK_MS = 40 # ms of audio per message; 20-40 ms suits live audio TAGS = re.compile(r"<\|[^|]*\|>|\[[^\[\]]*\]") # speaker tags such as <|speaker:0|>, sound tags such as [laughter] T0 = time.monotonic() @@ -304,7 +304,7 @@ The server uses the conversation to recognize names and follow-ups: - **Keep echo cancellation on.** Without it, the agent's own voice reaches the microphone, opens turns and can be taken for the user. In a browser, request `echoCancellation: true` from `getUserMedia`. Some echo can remain even with echo cancellation. It is usually committed as a non-user turn, but it still opens turns (`speech_start`) while your agent speaks, which is why ducks need a cap. If the start of most replies is ducked, echo is leaking: check echo cancellation, or test with headphones. - **Turn automatic gain control off where you can** (`autoGainControl: false` in a browser), so quiet background sound is not amplified between phrases. - **Test with headphones first.** They keep the agent's voice out of the microphone, so you can check turn-taking before you tune echo cancellation. -- **Send audio as you capture it**, mono 16-bit PCM in 80–100 ms messages, at the device's own rate if it is one of the supported rates. The server converts the rate, so do not resample. Telephony audio at 8 kHz is supported; decode μ-law or A-law call audio to 16-bit PCM first. +- **Send audio as you capture it**, mono 16-bit PCM in 20–40 ms messages, at the device's own rate if it is one of the supported rates. The server converts the rate, so do not resample. Telephony audio at 8 kHz is supported; decode μ-law or A-law call audio to 16-bit PCM first. - **Watch your audio source.** The server closes a session after 30 seconds without any client message (`idle_timeout`, close 4408), but a WebSocket ping counts as one, and Python's `websockets` sends a ping every 20 seconds by default. With that default, a session stays open even when your audio stops, and it counts against your concurrency limit until you close it or the 60-minute session limit ends it. If your audio source can stall, watch it in your own code and send `audio_done` or close the connection when it stops, or pass `ping_interval=None` to `websockets.connect` to leave idle detection to the server. Node's `ws` sends no pings by default. - **Keep keys on your server.** In a browser, capture audio and relay it through your server, as in [Capture from a browser microphone](/developer-guide/sdk-guide/cookbook/realtime-transcription#capture-from-a-browser-microphone). Send `agent_speaking` on that same connection so it stays in order with the audio. diff --git a/features/speech-to-text/realtime.mdx b/features/speech-to-text/realtime.mdx index fdea6b0..c92c257 100644 --- a/features/speech-to-text/realtime.mdx +++ b/features/speech-to-text/realtime.mdx @@ -88,7 +88,7 @@ The SDKs do not wrap this endpoint; you connect with any WebSocket client and ex `session_started`. - Send `audio` messages with 16-bit mono PCM, 80–100 ms each, as you capture + Send `audio` messages with 16-bit mono PCM, 20–40 ms each, as you capture it. @@ -188,7 +188,7 @@ Measured on the production service with synthetic speech: - `eager_end_of_turn` arrives about 0.6 seconds after the speaker stops. - `committed` arrives about 1.1 seconds after the speaker stops. -Your network and how you send audio add to these times. Send audio as you capture it, in 80–100 ms messages. +Your network and how you send audio add to these times. Each result also waits for the message that carries its audio, on average half a message: about 50 ms with 100 ms messages and 20 ms with 40 ms messages. Send audio as you capture it, in 20–40 ms messages. ## Implementation details