From 54265e852507861e726310d79cd5c45d4197efd5 Mon Sep 17 00:00:00 2001 From: liujiahua123123 Date: Fri, 30 Oct 2026 12:48:06 +0000 Subject: [PATCH] docs(realtime-stt): document results.tag_audio_events New realtime option results.tag_audio_events (boolean, default true). false removes every sound tag from the text of a turn (draft, eager_end_of_turn, committed and its words) and collapses whitespace as with diarize false. - WebSocket reference: the config row; invalid_config for a non-boolean value; a configure change applies to turns that open after configured; a guarantee that no text of such a turn holds an opening tag bracket; the Sound tags section says tags can also come in full-width (U+FF3B/U+FF3D) or lenticular (U+3010/U+3011) brackets, widens the strip regular expression to match, and gives the bracket rules for tag_audio_events false (an unclosed bracket hides the rest of a draft but only itself in a final); the Speakers, Language and Degraded commits sections note how the option interacts with them. - AsyncAPI: the property in the config and configure schemas, the session_started config echo, and the committed text description. - Realtime overview and cookbooks: mention the option next to the sound tags; the cookbook strip regexes also match the full-width and lenticular brackets; the voice agent's answer rule notes the option. Co-Authored-By: Claude Opus 5.5 --- api-reference/asyncapi-asr-realtime.yml | 10 +++++++- .../endpoint/websocket/asr-realtime.mdx | 23 +++++++++++++------ .../cookbook/realtime-transcription.mdx | 7 ++++-- .../cookbook/realtime-voice-agent.mdx | 5 ++-- features/speech-to-text/realtime.mdx | 2 +- 5 files changed, 34 insertions(+), 13 deletions(-) diff --git a/api-reference/asyncapi-asr-realtime.yml b/api-reference/asyncapi-asr-realtime.yml index 5822836..e811e66 100644 --- a/api-reference/asyncapi-asr-realtime.yml +++ b/api-reference/asyncapi-asr-realtime.yml @@ -503,6 +503,7 @@ components: drafts: true ignore_timestamps: true diarize: true + tag_audio_events: true vad_events: false eot_events: false limits: @@ -780,7 +781,7 @@ components: type: integer text: type: string - description: The final transcript of the turn. Can contain speaker tags such as `<|speaker:0|>` and sound tags such as `[laughter]`. Can be empty. + description: The final transcript of the turn. Can contain speaker tags such as `<|speaker:0|>` (unless `results.diarize` is `false`) and sound tags such as `[laughter]` (unless `results.tag_audio_events` is `false`). Can be empty. words: type: array items: @@ -1175,6 +1176,10 @@ components: - "false" default: true description: Speaker tags in the text. `auto` is the same as `true`; `false` removes them. The strings `"true"` and `"false"` are accepted too. + tag_audio_events: + type: boolean + default: true + description: Sound tags such as `[laughter]` in the text; `false` removes them. Only `true` or `false`. vad_events: type: boolean default: false @@ -1229,6 +1234,9 @@ components: - "true" - "false" description: Speaker tags in the text. `auto` is the same as `true`; `false` removes them. The strings `"true"` and `"false"` are accepted too. + tag_audio_events: + type: boolean + description: Sound tags such as `[laughter]` in the text; `false` removes them. Only `true` or `false`. vad_events: type: boolean description: Send `vad` messages. diff --git a/api-reference/endpoint/websocket/asr-realtime.mdx b/api-reference/endpoint/websocket/asr-realtime.mdx index 3616f5e..6239003 100644 --- a/api-reference/endpoint/websocket/asr-realtime.mdx +++ b/api-reference/endpoint/websocket/asr-realtime.mdx @@ -114,13 +114,14 @@ Audio sent before `session_started` is not lost: once the session has started, t | `results.drafts` | No | `false` | `true` sends live [`draft`](#draft) messages. | | `results.ignore_timestamps` | No | `true` | `false` adds word timings (`words`) to `committed`. | | `results.diarize` | No | `true` | `true`, `false` or `"auto"` (same as `true`); the strings `"true"` and `"false"` are accepted too. `false` removes speaker tags from the text. See [Speakers](#speakers). | +| `results.tag_audio_events` | No | `true` | `false` removes sound tags such as `[laughter]` from the text. See [Sound tags](#sound-tags). | | `results.vad_events` | No | `false` | `true` sends [`vad`](#vad) messages. | | `results.eot_events` | No | `false` | `true` sends [`eot_prob`](#eot_prob) messages. | | `context.history` | No | `[]` | Earlier conversation, oldest first: up to 32 items of `{ "role": "user" \| "assistant", "text": str }`, each up to 2,000 characters. | | `context.terms` | No | `[]` | Up to 100 terms of up to 32 characters each: names, products and jargon to recognize. | | `client_reference_id` | No | | Your own ID for the session, up to 256 characters. It is recorded with the session and not sent back. | -A value of the wrong type for `ignore_timestamps`, or a `diarize` value other than those listed, is `invalid_config`. So is an unknown field, including a misspelled one; the error's `message` names the field. `invalid_config` closes the connection with 4400. Send only the fields listed here: not every other field is refused, so check `session_started.config`, which shows the settings in effect. +A value of the wrong type for `ignore_timestamps` or `tag_audio_events` (both take only `true` or `false`), or a `diarize` value other than those listed, is `invalid_config`. So is an unknown field, including a misspelled one; the error's `message` names the field. `invalid_config` closes the connection with 4400. Send only the fields listed here: not every other field is refused, so check `session_started.config`, which shows the settings in effect. #### Audio format @@ -201,7 +202,7 @@ Change settings during the session. `turn_detection`, `results` and `context` ta `audio` is fixed for the session: to change the audio format, open a new session. A `configure` with an unknown field, with an `audio` that differs from the session's, or with `context.history` gets `configured` with `ok: false`, and nothing in it is applied. An `audio` identical to the session's is accepted. As with `config`, send only the fields listed here; `configured.config` shows the settings in effect. -- A change to `ignore_timestamps` or `diarize` applies to turns that open after `configured`. A turn that is already open keeps the settings it started with, so its `eager_end_of_turn` and `committed` always agree. +- A change to `ignore_timestamps`, `diarize` or `tag_audio_events` applies to turns that open after `configured`. A turn that is already open keeps the settings it started with, so its `eager_end_of_turn` and `committed` always agree. - When `mode` changes to or from `auto`, the server first settles its turn decisions on the audio already sent, so `configured` can take a few seconds. - A turn's `user_turn` status follows the mode it opened in: a turn opened in `vad` or `manual` mode is always a user turn and gets no `user_turn_taken`. How an open turn ends follows the current mode. - Switching away from `auto` settles an open turn that has no `user_turn_taken` yet: it becomes a user turn (`user_turn_taken` is sent) if there was speech in about the last second; otherwise it is committed at once with `user_turn: false`. @@ -413,6 +414,7 @@ Sent after `audio_done`, just before the server closes the connection with 1000. - Every `draft` has `settled_text`, possibly empty, and the draft's `text` starts with it. - `language` and `language_code` are both present or both absent. - `committed.words` is present exactly when `ignore_timestamps` was `false` when the turn opened and the commit is not degraded. +- With `tag_audio_events` `false` when the turn opened, no text of the turn contains an opening tag bracket (`[`, U+FF3B or U+3010). ## Why a turn ended @@ -434,19 +436,26 @@ In every mode: ## Speakers -`text` can contain speaker tags `<|speaker:N|>` (N = 0, 1, ...). Numbers are stable within a session and start over in a new one. Strip the tags if you only need the words, or send `results.diarize: false` to get text without them. With `diarize: false`, whitespace in the text, line breaks included, is collapsed to single spaces and trimmed; sound tags stay (see [Sound tags](#sound-tags)). +`text` can contain speaker tags `<|speaker:N|>` (N = 0, 1, ...). Numbers are stable within a session and start over in a new one. Strip the tags if you only need the words, or send `results.diarize: false` to get text without them. With `diarize: false`, whitespace in the text, line breaks included, is collapsed to single spaces and trimmed; sound tags stay unless `tag_audio_events` is `false` (see [Sound tags](#sound-tags)). Whitespace right after a speaker tag is not significant and can vary, for example `<|speaker:0|>Can you` or `<|speaker:0|> Can you`. To get each speaker's text, split the text on the tags, then trim each segment. If you only need the words, remove the tags, then trim the text and collapse runs of whitespace into one space. ## Sound tags -`text` can contain sound tags in square brackets, such as `[laughter]`, `[laughs]`, `[sighs]` or `[pause]`, in every turn detection mode and whether `diarize` is on or off. The set is open: a tag can be any short description, sometimes several words long. A turn's text can be a tag alone. Language detection ignores tags. +`text` can contain sound tags in square brackets, such as `[laughter]`, `[laughs]`, `[sighs]` or `[pause]`, in every turn detection mode and whether `diarize` is on or off. The set is open: a tag can be any short description, sometimes several words long or in the language of the speech. A tag can also come in full-width square brackets (U+FF3B and U+FF3D) or lenticular brackets (U+3010 and U+3011) instead of `[` and `]`. A turn's text can be a tag alone. -Tags are sounds, not words. Strip them when you need only the words, for example for search, word counts, or deciding whether there is anything to answer: remove every match of the regular expression `\[[^\[\]]*\]`, then trim the text and collapse runs of whitespace into one space. If you keep them for a language model, tell it that they are sounds, and do not answer a turn that holds only tags. +Tags are sounds, not words. Strip them when you need only the words, for example for search, word counts, or deciding whether there is anything to answer: remove every match of the regular expression `\[[^\[\]]*\]|\uFF3B[^\uFF3B\uFF3D]*\uFF3D|\u3010[^\u3010\u3011]*\u3011`, then trim the text and collapse runs of whitespace into one space. If you keep them for a language model, tell it that they are sounds, and do not answer a turn that holds only tags. + +Or send `results.tag_audio_events: false` to get no sound tags at all. Every tag, in any of the three kinds of brackets and whatever is inside it, is then removed from every text of the turn (`draft`, `eager_end_of_turn` and `committed`), as are the `words` entries that belong to a tag, and whitespace is collapsed and trimmed as with `diarize: false`: + +- An opening bracket is closed only by its own kind: `[` by `]`, U+FF3B by U+FF3D, U+3010 by U+3011. A closing bracket ends the tag of the nearest opening bracket of its kind that is still open, and everything between them is removed with it, brackets of the other kinds included. A tag inside a tag of the same kind goes with the outer one. A closing bracket with no opening bracket of its kind open before it is ordinary text and stays. +- An opening bracket that nothing closes depends on the message. In a `draft` it is taken for the start of a tag that is still being written: it and the rest of that draft, speaker tags included, are not shown until a later draft closes it. In `eager_end_of_turn` and `committed`, the turn's final text, only that bracket is removed: the text after it is shown, and its `words` are kept. + +Speaker tags still depend on `diarize` alone. A turn that holds only tags still gets its `committed`, without a language, and with a `text` that is empty with `diarize: false`, or holds only the turn's speaker tags with `diarize` on. ## Language -`draft`, `eager_end_of_turn` and `committed` carry the language of their own `text`. Use `language_code` (ISO 639-1) in your code; `language` is its English name, for display, with the same names as [`POST /v1/asr`](/api-reference/endpoint/openapi-v1/speech-to-text). The language is detected from that message's text alone, ignoring speaker tags, sound tags and interjections such as "okay", "hmm" or "uh". It is one of: +`draft`, `eager_end_of_turn` and `committed` carry the language of their own `text`. Use `language_code` (ISO 639-1) in your code; `language` is its English name, for display, with the same names as [`POST /v1/asr`](/api-reference/endpoint/openapi-v1/speech-to-text). The language is detected from that message's text alone, ignoring speaker tags, sound tags in `[` and `]`, and interjections such as "okay", "hmm" or "uh". With `tag_audio_events: false`, the label can also reflect text that is not shown: the rest of a draft after an unclosed opening bracket, the inside of a tag that holds another tag, or a tag in full-width or lenticular brackets. It is one of: `ar`, `zh`, `cs`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `el`, `hi`, `hu`, `id`, `it`, `ja`, `ko`, `mk`, `ms`, `fa`, `pl`, `pt`, `ro`, `ru`, `es`, `sv`, `tl`, `th`, `tr`, `vi`. @@ -454,7 +463,7 @@ Both fields are omitted when the text is empty, too short to tell or ambiguous. ## Degraded commits -If the final pass for a turn fails, you receive `error` with `engine_error` and `recoverable: true`, and then a `committed` with `degraded: true`. Its `text` and language are those of the turn's last draft (empty and without a language when the turn had no draft, as with `results.drafts: false`), and it has no `words`. The session continues. +If the final pass for a turn fails, you receive `error` with `engine_error` and `recoverable: true`, and then a `committed` with `degraded: true`. Its `text` and language are those of the turn's last draft (empty and without a language when the turn had no draft, as with `results.drafts: false`), and it has no `words`. With `tag_audio_events: false`, an unclosed opening bracket in that draft is handled as in any `committed`: only the bracket is removed, so the text after it, and its language, can appear even if that draft showed none. The session continues. ## Errors and close codes diff --git a/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx b/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx index 37a8d7b..01a3f63 100644 --- a/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx +++ b/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx @@ -284,7 +284,7 @@ The clip ends mid-sentence, so the last turn is committed by `audio_done`. Draft - **`draft`** is the whole turn so far, and replaces the previous draft. Drafts come only with `results.drafts: true`; use them for live captions. - **`user_turn_taken`** says the server judged that the user is taking their turn, rather than noise or someone else nearby. - **`eager_end_of_turn`** comes with `turn_detection.eager: true` when the turn has probably ended. Its text equals the final transcript unless `turn_resumed` follows, so an agent can start its reply early. -- **`committed`** is the final transcript of the turn. `user_turn: false` marks a non-user turn, such as background noise or a short acknowledgement while your agent is speaking. Speaker tags such as `<|speaker:0|>` are stable within the session; strip them and trim the text, or send `results.diarize: false`. Text can also contain [sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags) such as `[laughter]`. `language_code` is omitted when the text is too short to tell, as for "Uh-huh." above. +- **`committed`** is the final transcript of the turn. `user_turn: false` marks a non-user turn, such as background noise or a short acknowledgement while your agent is speaking. Speaker tags such as `<|speaker:0|>` are stable within the session; strip them and trim the text, or send `results.diarize: false`. Text can also contain [sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags) such as `[laughter]`; strip them too, or send `results.tag_audio_events: false`. `language_code` is omitted when the text is too short to tell, as for "Uh-huh." above. - **`session_done`** follows `audio_done`, and the server then closes the connection with code 1000. Ignore message types and fields you do not know: new ones can be added. Every message is described in the [WebSocket reference](/api-reference/endpoint/websocket/asr-realtime#server-messages). @@ -403,7 +403,10 @@ registerProcessor("pcm16", Pcm16);`; // Speaker tags and sound tags out, whitespace collapsed. const strip = text => (text || "") - .replace(/<\|[^|]*\|>|\[[^\[\]]*\]/g, "") + .replace( + /<\|[^|]*\|>|\[[^\[\]]*\]|\uFF3B[^\uFF3B\uFF3D]*\uFF3D|\u3010[^\u3010\u3011]*\u3011/g, + "" + ) .replace(/\s+/g, " ") .trim(); let ws, diff --git a/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx b/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx index 867dce2..7545593 100644 --- a/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx +++ b/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx @@ -72,7 +72,8 @@ import websockets URL = "wss://api.fish.audio/v1/asr/realtime" CHUNK_MS = 40 # ms of audio per message; 20-40 ms suits live audio -TAGS = re.compile(r"<\|[^|]*\|>|\[[^\[\]]*\]") # speaker tags such as <|speaker:0|>, sound tags such as [laughter] +# Speaker tags such as <|speaker:0|>; sound tags such as [laughter], also in full-width brackets. +TAGS = re.compile(r"<\|[^|]*\|>|\[[^\[\]]*\]|\uFF3B[^\uFF3B\uFF3D]*\uFF3D|\u3010[^\u3010\u3011]*\u3011") T0 = time.monotonic() @@ -233,7 +234,7 @@ The recipe keeps a reply's full text in its history even when the user interrupt ## Decide when to answer -Answer a turn when `committed.user_turn` is `true` and its text, without speaker tags and [sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags), is not empty. Do not decide by `reason`: a turn can end for several reasons, and non-user turns end with `eot` too. If `user_turn` is ever missing, treat it as `true`. +Answer a turn when `committed.user_turn` is `true` and its text, without speaker tags and [sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags) (already removed with `results.diarize: false` and `results.tag_audio_events: false`), is not empty. Do not decide by `reason`: a turn can end for several reasons, and non-user turns end with `eot` too. If `user_turn` is ever missing, treat it as `true`. A turn with `user_turn: false` is not the user talking to your agent: usually background noise, a TV or another person nearby (`turn_kind: "other"`), or an acknowledgement while your agent was speaking (`turn_kind: "backchannel"`). Its text can be shown dimmed in a transcript, or left out. diff --git a/features/speech-to-text/realtime.mdx b/features/speech-to-text/realtime.mdx index c92c257..30ce596 100644 --- a/features/speech-to-text/realtime.mdx +++ b/features/speech-to-text/realtime.mdx @@ -198,7 +198,7 @@ Send 16-bit little-endian PCM, mono, at 8000, 12000, 16000, 22050, 24000, 32000, ### Speakers -Transcripts contain speaker tags such as `<|speaker:0|>`, stable within a session. Strip them, or send `results.diarize: false` for plain text. Whitespace right after a tag can vary, so trim the text after you strip the tags. Transcripts can also contain sound tags such as `[laughter]` or `[pause]`; see [Sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags). +Transcripts contain speaker tags such as `<|speaker:0|>`, stable within a session. Strip them, or send `results.diarize: false` for plain text. Whitespace right after a tag can vary, so trim the text after you strip the tags. Transcripts can also contain sound tags such as `[laughter]` or `[pause]`. Strip them too, or send `results.tag_audio_events: false` to remove them; see [Sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags). ### Keys and browsers