From ff98632363fcf6bc867f8b9a7cc49bf4576ae9d6 Mon Sep 17 00:00:00 2001 From: FORIFOR Date: Tue, 20 Oct 2026 02:50:59 +0900 Subject: [PATCH] cli: voice conversation batch against scripted characters, with line conditions (layer B, no carrier) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `phone test --level conversation --scenario ` runs the simulator's character (optionally adversarial) over the local μ-law line and reports evaluate()'s verdict, the character's truth and false completions - --mutation / --condition (clean, quiet, fast, noise-db, --noise-file) / --runs / --seed / --max-usd-total; summary.json with false completions, as-expected rate, reply latency p50/p95, cost; exit 1 on a false completion - turn-taking uses audible agent audio only; fatal runtime errors surface in the report - OpenAITTS: optional speed - simulator: 「お忙しいところ失礼いたします」 is an opening, not a goodbye - docs: real-call verification plan (layers A/B/C) and the first smoke findings Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0195nzJ5oDJAnjHtFzHM9sL3 --- docs/launch/real-call-plan.md | 186 ++++++++++++++++++ docs/quality/conversation-batch.md | 58 ++++++ packages/cli/src/audio-degrade.ts | 125 ++++++++++++ packages/cli/src/commands.ts | 3 +- packages/cli/src/conversation-batch.test.ts | 90 +++++++++ packages/cli/src/conversation-batch.ts | 102 ++++++++++ .../src/local-conversation-character.test.ts | 90 +++++++++ packages/cli/src/local-conversation.ts | 152 +++++++++++--- packages/cli/src/phone.ts | 85 +++++++- providers/openai/src/tts.ts | 6 + providers/simulator/src/character.ts | 3 +- providers/simulator/src/simulator.test.ts | 9 +- 12 files changed, 882 insertions(+), 27 deletions(-) create mode 100644 docs/launch/real-call-plan.md create mode 100644 docs/quality/conversation-batch.md create mode 100644 packages/cli/src/audio-degrade.ts create mode 100644 packages/cli/src/conversation-batch.test.ts create mode 100644 packages/cli/src/conversation-batch.ts create mode 100644 packages/cli/src/local-conversation-character.test.ts diff --git a/docs/launch/real-call-plan.md b/docs/launch/real-call-plan.md new file mode 100644 index 00000000..42a0c925 --- /dev/null +++ b/docs/launch/real-call-plan.md @@ -0,0 +1,186 @@ +# 実電話の検証計画（2026-10-19 作成） + +目的: 企業向けに「実電話 N 件、誤完了 0、成功率 XX%、p95 応答 X.X 秒、¥XX/通話」と、測った数字だけで言えるようにする。 + +方針: **電話なしで確かめられることは電話なしで全部済ませ、実電話は電話でしか確かめられないことに絞る。** 実電話は「直す作業」ではなく「数字を取る作業」にする。 + +前回の計画（[READINESS.md](../READINESS.md)「実電話 100 件の計画」、2026-09-13）は、100 件すべてを実電話で行う前提だった。今回はその区分を 3 つの層に振り分け直した。 + +## 現状（根拠） + +| 項目 | 状態 | 根拠 | +|--|--|--| +| 実電話の記録 | Twilio/GPT-Live で十数件。会話・同意・相手情報の取得まで成功した件は集計なし | [real-calls.md](real-calls.md) | +| 無音バグの修正（9/15） | 修正後の再発信は 2 件とも留守電で、未評価 | [real-calls.md](real-calls.md) | +| 取消の競合で `runtime_error`（9/20） | 修正済み。本物の OpenAI API では再現と修正を確認。実回線での再発信は BLOCKED | [realtime-recovery.md](../quality/realtime-recovery.md) | +| UI からの実発信（9/20） | 32 秒、7 発話で完了。ただし Twilio の price が null で、精算は BLOCKED | [user-ui-call.md](../quality/user-ui-call.md) | +| シミュレーター | 敵対的 1 万件で誤完了 0（14 種） | [ENTERPRISE_READINESS.md](../ENTERPRISE_READINESS.md) | +| Plivo / SIP / LiveKit / Gemini Live / 英語 | 実回線は未検証 | README | + +## 3 つの層 + +| 層 | 何を使うか | 費用 | 依頼者の go | +|--|--|--|--| +| A. 判定とガードレール | シミュレーター、ゲートウェイのテスト | ¥0 | 不要 | +| B. 音声の通し試験 | `phone test --level conversation`（電話会社なし、本物の音声 API） | 1 回 $0.05〜0.2 程度 | 不要（API 代のみ） | +| C. 実電話 | ゲートウェイのアプリから Twilio 直結で発信 | 1 件 3 分 ≈ ¥110 | **必要** | + +### 前回の 100 件の区分は、どの層で確かめるか + +| 区分 | A 判定 | B 音声 | C 実電話 | 実電話が必要な理由 | +|--|--|--|--|--| +| 成功 / 代案から成功 | 済 | ○ | ○ | 人の声での聞き取りと料金 | +| 部分成功 / 満席 / 仮押さえ / 拒否 | 済 | ○ | 少数 | 判定は A と B で十分 | +| 聞き返し / 転送 / 保留 | 済 | ○ | 少数 | 実際の無音・保留音での挙動 | +| 途中切断 | 済 | ○ | ○ | 電話会社の切断通知の扱い | +| 留守電 | 済 | ― | ○ | 携帯会社ごとに応答文と鳴り方が違う | +| 騒音 / 早口 / 小声 | ― | ○（合成） | ○ | 合成音は人の声の代わりにならない | +| 方言 | 済 | ○（合成） | ○ | 同上 | +| 営業: 受ける / 曖昧 / 断る / 連絡しないで / 取次ぎ | 一部済 | ○ | ○ | 名乗り・目的告知が実際に聞こえるか | +| 再発信停止・時間帯・上限・二重発信 | 済 | ― | ― | 電話をかける前に止まるので不要 | + +## 層 A — 判定とガードレール（¥0、すでにある） + +新しい作業は、下のコマンドを検証の起点として再実行し、結果を記録することだけ。 + +- `pnpm oathra eval --adversarial 10000` → `False Completion: 0 / 10000` +- `pnpm test`（`packages/evidence/src/appointment-fuzz.test.ts` の営業アポ 1 万件を含む） +- `pnpm test:gateway` に、電話をかける前に止める仕組みのテストがある: + - 連絡停止の番号は承認できない。発信直前にも確認し直す（`apps/gateway/test/gateway.node.mjs:49`, `:82`） + - 二重発信の防止（`gateway.node.mjs:39-41`） + - 日次上限（`gateway.node.mjs:55`） + - 発信時間帯（`schedules.node.mjs:222` ほか。営業電話の既定 9〜20 時を直接確かめるテストは未確認。無ければ追加する） + - 営業電話は連絡停止の番号・許可外の相手を拒否する（`agent-sales-inbound.node.mjs:90`） + +2026-10-19 時点: `pnpm test` 4310 件、`pnpm test:gateway` 637 件、すべて成功。 + +## 層 B — 音声の通し試験（電話会社なし） + +### 今の `--level conversation` でできること / できないこと + +`packages/cli/src/phone.ts:611` と `packages/cli/src/local-conversation.ts` を読んだ結果。 + +| 項目 | 今 | 根拠 | +|--|--|--| +| 本物の処理の流れ（依頼 → 契約 → ランタイム → 音声エンジン） | ○ | `phone.ts:617-628` | +| 電話と同じ音質（μ-law 8kHz） | ○ | `local-conversation.ts:88` | +| 相手が話している途中の割り込み 1 回 | ○ | `local-conversation.ts:21`, `:188-193` | +| 応答時間の計測（聞き取り・考える・話し始め） | ○ | `local-conversation.ts:195-199` | +| 依頼の種類 | **雑談の 1 種類だけ**（予約・営業アポは不可） | `phone.ts:617` | +| 相手の台本 | **固定の 5 行だけ**（CLI から変えられない） | `local-conversation.ts:17-23` | +| 騒音・早口・小声 | **なし** | ― | +| 完了判定（`evaluate()` の結果） | **出ない**（終わり方と応答時間のみ） | `local-conversation.ts:205` | +| 複数回の実行と集計 | **なし**（1 回ずつ） | ― | + +### B を使うために足す機能（実装が必要） + +1. **依頼を選べるようにする**: シナリオ（`restaurant-reservation`、営業アポ）を指定でき、その契約で `evaluate()` の判定を report に出す。 +2. **相手の台本をファイルで渡す**: 区分ごとの台本（上の表の区分）を `scenarios/` か `tests/` に置く。 +3. **音を悪くする**: 相手の音声に、騒音（テレビ・水音・街中）を指定の音量差で混ぜる。話す速さ（早口）と音量（小声）を変える。μ-law に落とす前に処理する。 +4. **まとめて回す**: 区分 × 回数を一度に実行し、誤完了・期待どおりの判定・応答時間（p50/p95）・費用を集計する。1 回ずつと合計の費用上限を守る。 + +方言は合成音声では再現が粗い。B では言い回しだけ確かめ、発音は C で確かめる。 + +### B の合格条件 + +- 予約・営業アポの各区分を 5 回ずつ（20 区分前後 × 5 = 約 100 回）。費用の目安は合計 $20 以内（実行前に試算し直す）。 +- 誤完了 0。 +- 期待どおりの判定 95% 以上。外れた場合は区分ごとに原因を記録する（聞き取り / 判定 / 会話の進め方）。 +- `runtime_error` 0。割り込みの後に古い応答の音声が流れない。 +- 騒音・早口・小声で、日時・人数・名前の誤認識の件数を記録する（ここは目標を置かず、実電話との比較に使う）。 + +## 層 C — 実電話（要 go） + +B が合格してから行う。CLAUDE.md の順で、`--level local` → `--level gateway` → PSTN（実際の電話網）。 + +### 原則 + +- 実在の店舗や企業にはかけない。**相手役は人間の協力者が自分の番号で受ける**。協力者には事前に、記録と目的を説明して同意を取る。 +- 失敗・つながらなかった・留守電も、すべて分母に残す。過去の記録は分母に足さない。 +- 音声と会話本文は公開しない。公開するのは集計表と、協力者が許可した書き起こしだけ。 +- シミュレーターと層 B の結果を、実電話の件数・遅延・料金として載せない。 + +### C0 — 準備（¥0） + +- **Twilio の料金データを取れるようにする。** 9/20 の通話の price を Twilio API で今取り直し、後から値が入るのか確かめる。入らない場合は、請求明細から集計する手順に切り替える。これが決まらないと、1 通話あたりの料金と精算を測れない。 +- `pnpm oathra phone doctor --to <協力者の番号>`、`phone test --level local`、`--level gateway` がすべて成功すること。 + +### C1 — 疎通確認 10 件（約 ¥1,100） + +ゲートウェイのアプリの画面から、協力者 1 人に 10 件。 + +| 確認すること | 合格条件 | +|--|--| +| 声が届くか（9/15 の無音バグ） | 10/10 で AI の声が相手に聞こえる | +| 冒頭の告知 | 10/10 で「この通話は記録されています。」が最初に流れる | +| 割り込み | 相手が割り込んでも `runtime_error` 0 | +| 完了判定 | 未確定の 4 件がすべて `incomplete` | +| 精算 | 10/10 で料金が確定し、確保していたクレジットが解放される | + +1 つでも不合格なら止め、原因を `oathra replay` で特定して直す。直した点は層 B で再確認してから C1 をやり直す。 + +### C2 — 電話でしか測れないもの 30 件（約 ¥3,300） + +| 区分 | 件数 | 相手役がやること | +|--|--:|--| +| 予約の成功（代案からの成功を含む） | 8 | 普通に受ける | +| 予約の未確定（満席・仮押さえ） | 4 | 断る / 「一応お取りしておきます」 | +| 留守電 | 4 | 実際の留守電につなぐ（携帯会社を 2 社以上） | +| 騒音・早口・小声 | 6 | テレビ音の中で / 早口 / 小声 | +| 方言 | 3 | 関西・東北など | +| 途中切断 | 2 | 会話の途中で切る | +| 営業（受ける / 断る / 連絡しないで） | 3 | 名乗り・目的告知が聞き取れるか相手役が記録 | + +30 件で誤完了 0 なら、言えるのは「誤完了率は約 10% 以下（95% 信頼区間の上限）」まで。応答時間、誤認識、料金の実測値はここで出る。 + +### C3 — 企業向けの数字を作る 100 件（任意、約 ¥11,000） + +「実電話 100 件で誤完了 0（約 3% 以下）」と言いたいときだけ行う。C2 の 30 件はこの 100 件には数えず、別に集計する。区分の配分は、C2 で問題が出た区分を厚くして決める。 + +### 1 件ごとに記録すること + +記録先は `docs/launch/real-calls.md`（番号・音声・本文は載せない）。 + +試行 ID / 日時 / 層と段階（C1〜C3） / 区分 / 相手役 / commit / キャリア・音声エンジン / Twilio の最終状態 / 通話秒数 / 最終判定 / 期待どおりか / 誤完了 / `runtime_error` / 最初の応答までの時間 / 日時・人数・名前の誤認識 / 人が後で直した点 / Twilio 料金 / モデル料金 / 備考 + +### KPI（C2・C3） + +| KPI | 定義 | 目標 | +|--|--|--| +| 誤完了 | 相手が確定していないのに `completed` になった件数 | **0**（1 件出たら即停止） | +| タスク成功率 | 期待どおりの結果になった割合 | 90% 以上 | +| 接続後の正常終了率 | 意図しない切断・`runtime_error` がない割合 | 95% 以上 | +| 人による修正率 | 人が後で結果を直す必要があった割合 | 5% 以下 | +| 最初の応答時間 | 相手が話し終えてから AI が話し始めるまで | p50 / p95 を実測し、層 B と比べる | +| 誤認識 | 日時・人数・名前の誤り | 件数を実測し、層 B と比べる | +| 料金 | Twilio + モデル（請求の実額） | 1 通話あたりの実測値 | + +### 中止する条件 + +- 誤完了が 1 件 → 即停止。原因を直し、層 B で再確認してから、その区分をやり直す。 +- `runtime_error` か無音が 2 件続く → 停止して原因を調べる。 +- 実費が予算上限の 80% に達する → 停止して依頼者に報告する。 +- 協力者が中止を求めたとき。 + +## 日程（目安） + +| 段階 | 内容 | 人手 | +|--|--|--| +| 1 | 層 A を再実行して記録。営業の時間帯テストが無ければ追加 | 実装者のみ | +| 2〜4 | 層 B の機能を足し（上の 4 点）、約 100 回を回して直す | 実装者のみ | +| 5 | C0・C1（10 件） | 協力者 1 人、約 40 分 | +| 6〜7 | C2（30 件） | 協力者 2〜3 人、各 1 時間 | +| 任意 | C3（100 件） | 協力者 2〜3 人、4〜5 日 | +| 最後 | 集計し、README・サイト・READINESS.md を更新（未達は未達のまま載せる） | 実装者のみ | + +## 決めてほしいこと + +1. 層 B の機能追加を進めてよいか（API 代は合計 $20 程度まで）。 +2. 実電話に進む go（C1、C2、C3 で別々に出す）。 +3. 実電話の予算上限（案: C1 + C2 で ¥5,000、C3 を行うなら追加で ¥12,000）。 +4. 相手役の協力者 2〜3 人と、その番号。方言を話せる人、携帯会社が違う人がいるか。 +5. 結果をどこまで公開するか（集計のみ / 書き起こしの一部も）。 + +## この計画で検証しないこと + +Plivo / SIP / LiveKit / Gemini Live、英語、同時に複数かける場合の負荷、実在の店舗・企業への発信、障害時の切り替え。これらは別の計画で扱う。 diff --git a/docs/quality/conversation-batch.md b/docs/quality/conversation-batch.md new file mode 100644 index 00000000..ffe1925f --- /dev/null +++ b/docs/quality/conversation-batch.md @@ -0,0 +1,58 @@ +# 層 B: 音声の通し試験（電話会社なし）の実装と初回試験（2026-10-20） + +計画: [real-call-plan.md](../launch/real-call-plan.md) の層 B。実電話ではない。相手の声は合成音声（OpenAI `tts-1`）で、回線はローカルの μ-law 8kHz ループ。 + +## 実装 + +| 変更 | ファイル | +|--|--| +| 相手役に台本キャラクター（シミュレーターの店員、敵対的な変異つき）を使えるようにした。AI の発話の文字起こしを聞いて返事する。通話後に `evaluate()` の判定、キャラクターが実際に約束した内容（`truth`）、誤完了かどうかを report に出す | `packages/cli/src/local-conversation.ts` | +| 相手側の音を悪くする: 小声（−18 dB）、騒音（指定の SNR、ピンクノイズまたは WAV）、早口（TTS の速度 1.5） | `packages/cli/src/audio-degrade.ts` | +| 変異 × 条件 × 回数をまとめて回し、誤完了・期待どおりの率・応答時間 p50/p95・費用を集計する。合計の費用上限を超える前に止まる | `packages/cli/src/conversation-batch.ts`, `packages/cli/src/phone.ts`, `packages/cli/src/commands.ts` | +| `OpenAITTS` に `speed` を追加（任意。未指定なら従来どおり） | `providers/openai/src/tts.ts` | +| 台本キャラクターが「お忙しいところ失礼いたします」を別れの挨拶と誤認して電話を切っていたのを修正 | `providers/simulator/src/character.ts` | + +使い方: + +```sh +pnpm oathra phone test --level conversation --engine gpt-live --scenario restaurant-reservation \ + --mutation none,tentative-hold,ask-back --condition clean,quiet,fast,noise-10db \ + --runs 5 --max-usd 0.3 --max-usd-total 20 +``` + +結果は `--out`（既定は `.oathra/local-conversations/batch-*`）に、1 回ごとの `report.json` と `call-mixed-8k.wav`、全体の `summary.json` として保存される。誤完了があるか、上限で途中停止すると exit 1。 + +## 検証 + +| 判定 | 内容 | 結果 | +|--|--|--| +| PASS | `pnpm build` | 成功 | +| PASS | `pnpm test` | 73 ファイル、4,321 件成功 / 1 skip（追加 11 件: 音の加工 5、集計 3、キャラクター相手の通し 2、別れの挨拶 1） | +| PASS | `pnpm test:gateway` | 637 件成功 | +| PASS | `pnpm lint:deps` | dependency direction ok | +| PASS | `pnpm oathra eval` | False Completion 0（7/8 完了。未完了は意図した `false-completion-trap`） | +| PASS | `pnpm oathra eval --adversarial 10000` | False Completion: 0 / 10000 | +| 観測 | 有料の試験 2 回（gpt-live、予約、通常と騒音 10 dB）。合計の推定 $0.078 | 下記 | + +## 初回試験の所見（2 回のみ。率としては扱わない） + +| 回 | 条件 | 相手が実際に言ったこと | AI 側の音声認識 | 判定 | +|--|--|--|--|--| +| 1 | clean | 「…田中様でご予約**承りました**」 | 「…田中様でご予約**のおりました**」 | incomplete（誤完了ではない） | +| 2 | noise-10db | 「…田中様でご予約**承りました**」 | 2 つの発話がつながり「…ご予約**でよろしいでしょうか?**」 | incomplete（誤完了ではない） | + +- 会話の流れ（日時・人数・名前の提示、満席からの代案、名前の確認）は 2 回とも成立した。 +- 2 回とも相手は予約を確定したのに、**音声認識が「承りました」を聞き違え、判定エンジンが完了にしなかった**。安全側の失敗（取りこぼし）で、誤完了ではない。 +- 前の試行では、「9月12日**いち**」のように余計な語が入っただけで、日付・時刻・人数がすべて未確認になった（同じ文から「いち」を除くと completed になることを、判定エンジン単体で確認した）。 +- 騒音 10 dB では、聞き取り確定までの時間が 7 秒を超えた発話があった。 +- 応答時間（相手が話し終えてから AI の声が出るまで）: p50 1,208 ms、p95 1,841 ms（n=13）。ローカル回線なので、実回線ではキャリアの遅延が加わる。 + +実電話（層 C）に進む前に、約 100 回の本番実行で「確定の言葉の聞き違い」による取りこぼしの率を測る。取りこぼしが多ければ、判定を緩めるのではなく、AI が確定内容を復唱して相手に確認させる進め方（聞き違いを会話で直す）を検討する。 + +## 制約 + +- 相手の声は合成音声で、人の声・電話回線の代わりにならない。 +- 台本キャラクターは決まった言い回しで返す。人間の店員の多様さはない。 +- 営業アポ（ゲートウェイの `SALES_CALL_POLICY` と `evaluateSales`）はまだ対象外。 +- 試験中、OpenAI の `gpt-4o-mini-tts` が 20〜50 秒応答しない時間帯があった（`tts-1` は 1〜2 秒）。そのため相手役の声は `tts-1` に固定した。 +- 証拠ファイルは `artifacts/quality/conversation-batch/smoke-20261020/`（git 管理外）。 diff --git a/packages/cli/src/audio-degrade.ts b/packages/cli/src/audio-degrade.ts new file mode 100644 index 00000000..aeda6089 --- /dev/null +++ b/packages/cli/src/audio-degrade.ts @@ -0,0 +1,125 @@ +/** + * Callee-side line conditions for the local conversation test: a quieter voice, steady background noise at a + * given signal-to-noise ratio, a faster speaker. Speech goes in and out as μ-law 8 kHz, the carrier's format, and + * the same seed always gives the same noise. Synthetic speech is not a person: these conditions compare engines and + * builds against each other, they never stand in for a real line. + */ +import { MULAW_8K, convert, toPcm16, type AudioSpec } from "@oathra/voice"; + +export type LineCondition = { id: string; gainDb: number; snrDb: number | null; fast: boolean }; + +/** The callee's TTS speaking rate under `fast` (1 = normal). */ +export const FAST_SPEECH_RATE = 1.5; + +/** `clean`, `quiet`, `fast`, `noise-db`, joined with `+` (`fast+noise-10db`). */ +export function parseCondition(id: string): LineCondition { + const condition: LineCondition = { id, gainDb: 0, snrDb: null, fast: false }; + for (const part of id.split("+")) { + const noise = /^noise-(-?\d{1,2})db$/.exec(part); + if (part === "clean") continue; + else if (part === "quiet") condition.gainDb = -18; + else if (part === "fast") condition.fast = true; + else if (noise) condition.snrDb = Number(noise[1]); + else throw new Error(`Unknown condition "${part}". Use clean, quiet, fast or noise-db, joined with + (e.g. fast+noise-10db)`); + } + return condition; +} + +const PCM_8K: AudioSpec = { format: "pcm_s16le", sampleRate: 8000, channels: 1 }; +const clip = (x: number) => Math.max(-32768, Math.min(32767, Math.round(x))); +const toMulaw = (pcm: Int16Array) => convert({ ...PCM_8K, data: new Uint8Array(pcm.buffer, pcm.byteOffset, pcm.byteLength) }, MULAW_8K).data; + +/** RMS over the 20 ms frames that carry speech, so pauses do not dilute the level. */ +export function voicedRms(pcm: Int16Array): number { + let sum = 0, n = 0; + for (let i = 0; i + 160 <= pcm.length; i += 160) { + let frame = 0; + for (let k = i; k < i + 160; k++) frame += pcm[k]! * pcm[k]!; + if (Math.sqrt(frame / 160) > 300) { sum += frame; n += 160; } + } + return n ? Math.sqrt(sum / n) : 0; +} + +/** Pink noise (Paul Kellet's economy filter) at RMS 1, seeded. */ +export function pinkNoise(samples: number, rng: () => number): Float32Array { + const out = new Float32Array(samples); + let b0 = 0, b1 = 0, b2 = 0, sq = 0; + for (let i = 0; i < samples; i++) { + const w = rng() * 2 - 1; + b0 = 0.99765 * b0 + w * 0.099046; b1 = 0.963 * b1 + w * 0.2965164; b2 = 0.57 * b2 + w * 1.0526913; + out[i] = b0 + b1 + b2 + w * 0.1848; + sq += out[i]! * out[i]!; + } + const rms = Math.sqrt(sq / samples) || 1; + for (let i = 0; i < samples; i++) out[i] = out[i]! / rms; + return out; +} + +/** A recorded noise bed at RMS 1: 16-bit PCM WAV, any rate, mixed to mono and resampled to 8 kHz. */ +export function noiseFromWav(bytes: Uint8Array): Float32Array { + const v = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + const tag = (o: number) => String.fromCharCode(...bytes.subarray(o, o + 4)); + if (bytes.length < 12 || tag(0) !== "RIFF" || tag(8) !== "WAVE") throw new Error("noise file is not a WAV file"); + let fmt: { channels: number; rate: number; bits: number; format: number } | undefined, data: Uint8Array | undefined; + for (let o = 12; o + 8 <= bytes.length; ) { + const size = v.getUint32(o + 4, true); + if (tag(o) === "fmt ") fmt = { format: v.getUint16(o + 8, true), channels: v.getUint16(o + 10, true), rate: v.getUint32(o + 12, true), bits: v.getUint16(o + 22, true) }; + else if (tag(o) === "data") data = bytes.subarray(o + 8, Math.min(bytes.length, o + 8 + size)); + o += 8 + size + (size % 2); + } + if (!fmt || !data || fmt.format !== 1 || fmt.bits !== 16) throw new Error("noise file must be 16-bit PCM WAV"); + const pcm = toPcm16(convert({ format: "pcm_s16le", sampleRate: fmt.rate, channels: fmt.channels, data: data.slice() }, PCM_8K)); + let sq = 0; + for (const x of pcm) sq += x * x; + const rms = Math.sqrt(sq / (pcm.length || 1)); + if (!pcm.length || rms < 1) throw new Error("noise file is silent"); + return Float32Array.from(pcm, (x) => x / rms); +} + +/** + * Applies one condition to the callee's side of the line. The noise level is set from the callee's first spoken + * line (its speech RMS divided by the SNR) and then held, the way a room's noise does not follow the voice; before + * that line it assumes a typical TTS level. Silence frames carry the noise too. + */ +export class LineConditioner { + private noiseAt = 0; + private noiseLevel: number; + private calibrated = false; + private readonly gain: number; + + constructor(readonly condition: LineCondition, private readonly noise: Float32Array | null) { + if (condition.snrDb !== null && (!noise || noise.length < 160)) throw new Error(`condition ${condition.id} needs a noise bed`); + this.gain = 10 ** (condition.gainDb / 20); + this.noiseLevel = this.levelFor(4000 * this.gain); + } + + private levelFor(speechRms: number): number { + return this.condition.snrDb === null ? 0 : speechRms / 10 ** (this.condition.snrDb / 20); + } + + private mix(pcm: Int16Array, speech: boolean): Int16Array { + const out = new Int16Array(pcm.length); + for (let i = 0; i < pcm.length; i++) { + const n = this.noise && this.noiseLevel ? this.noise[this.noiseAt++ % this.noise.length]! * this.noiseLevel : 0; + out[i] = clip((speech ? pcm[i]! * this.gain : 0) + n); + } + return out; + } + + /** One spoken line, as the agent's side hears it. */ + speech(mu: Uint8Array): Uint8Array { + const pcm = toPcm16({ ...MULAW_8K, data: mu }); + if (!this.calibrated) { + const rms = voicedRms(pcm); + if (rms > 0) { this.noiseLevel = this.levelFor(rms * this.gain); this.calibrated = true; } + } + if (this.gain === 1 && !this.noiseLevel) return mu; + return toMulaw(this.mix(pcm, true)); + } + + /** `bytes` of line silence: μ-law silence, or the noise bed alone. */ + silence(bytes: number): Uint8Array { + if (!this.noiseLevel) return new Uint8Array(bytes).fill(0xff); + return toMulaw(this.mix(new Int16Array(bytes), false)); + } +} diff --git a/packages/cli/src/commands.ts b/packages/cli/src/commands.ts index bb0ff077..3af85a5b 100644 --- a/packages/cli/src/commands.ts +++ b/packages/cli/src/commands.ts @@ -246,7 +246,7 @@ export async function cmdPhone(positional: string[], flags: Flags): Promise] which layer is broken: carrier, gateway, media, engine, latency, cost oathra phone test [--level …] local (telephony ¥0, API usage) · gateway ¥0 · pstn (paid) ${dim("--level conversation --engine character-tts --voice-preset character-female|character-male [--max-usd 0.05] [--out dir]: whole call, no carrier")} + ${dim("--level conversation --engine gpt-live --scenario [--mutation none,tentative-hold,…] [--condition clean,quiet,fast,noise-10db] [--runs 5] [--max-usd 0.3] [--max-usd-total 2] [--noise-file wav]: scripted callee, verdict per run")} oathra call --request-file --dry-run review a saved phone request without dialing oathra call --request-file --approve-request explicitly place the reviewed call oathra call --to place a call through your carrier ${dim("--scenario --engine gpt-live|gemini-live|character-tts|pipeline --provider ")} diff --git a/packages/cli/src/conversation-batch.test.ts b/packages/cli/src/conversation-batch.test.ts new file mode 100644 index 00000000..cf1b3429 --- /dev/null +++ b/packages/cli/src/conversation-batch.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, it } from "vitest"; +import { contractFromScenario, loadScenarioDir } from "@oathra/scenario"; +import { LineConditioner, noiseFromWav, parseCondition, pinkNoise, voicedRms } from "./audio-degrade.js"; +import { expectedComplete, percentile, planRuns, summarizeRuns, type BatchRun } from "./conversation-batch.js"; +import { wavBytes } from "./local-conversation.js"; +import { scenariosDir } from "./paths.js"; +import { MULAW_8K, convert, toPcm16 } from "@oathra/voice"; + +const rng = (seed: number) => () => { seed = (seed * 1103515245 + 12345) % 2 ** 31; return seed / 2 ** 31; }; +const speech = (ms: number, amp = 6000) => { + const pcm = new Int16Array(ms * 8); + for (let i = 0; i < pcm.length; i++) pcm[i] = Math.round(amp * Math.sin((2 * Math.PI * 300 * i) / 8000)); + return convert({ format: "pcm_s16le", sampleRate: 8000, channels: 1, data: new Uint8Array(pcm.buffer) }, MULAW_8K).data; +}; +const rmsOf = (mu: Uint8Array) => { const pcm = toPcm16({ ...MULAW_8K, data: mu }); let s = 0; for (const x of pcm) s += x * x; return Math.sqrt(s / pcm.length); }; + +describe("line conditions", () => { + it("parses conditions joined with +, and refuses unknown ones", () => { + expect(parseCondition("clean")).toEqual({ id: "clean", gainDb: 0, snrDb: null, fast: false }); + expect(parseCondition("fast+noise-10db+quiet")).toEqual({ id: "fast+noise-10db+quiet", gainDb: -18, snrDb: 10, fast: true }); + expect(() => parseCondition("loud")).toThrow(/Unknown condition/); + }); + + it("clean passes speech through untouched and keeps silence silent", () => { + const line = new LineConditioner(parseCondition("clean"), null), mu = speech(200); + expect(line.speech(mu)).toBe(mu); + expect([...line.silence(160)].every((b) => b === 0xff)).toBe(true); + }); + + it("quiet lowers the voice by 18 dB", () => { + const mu = speech(400), out = new LineConditioner(parseCondition("quiet"), null).speech(mu); + expect(20 * Math.log10(rmsOf(out) / rmsOf(mu))).toBeCloseTo(-18, 0); + }); + + it("noise is held at the requested SNR below the first spoken line, fills silence, and repeats for a seed", () => { + const make = () => new LineConditioner(parseCondition("noise-10db"), pinkNoise(8000 * 2, rng(3))); + const line = make(), mu = speech(500); + line.speech(mu); + const bed = line.silence(8000); + const snr = 20 * Math.log10(voicedRms(toPcm16({ ...MULAW_8K, data: mu })) / rmsOf(bed)); + expect(snr).toBeGreaterThan(8.5); + expect(snr).toBeLessThan(11.5); + const again = make(); + again.speech(mu); + expect(again.silence(8000)).toEqual(bed); + expect(() => new LineConditioner(parseCondition("noise-10db"), null)).toThrow(/noise bed/); + }); + + it("reads a WAV noise bed at any rate into 8 kHz at RMS 1", () => { + const pcm = new Int16Array(16000); + for (let i = 0; i < pcm.length; i++) pcm[i] = Math.round(2000 * Math.sin((2 * Math.PI * 500 * i) / 16000)); + const bed = noiseFromWav(wavBytes(pcm, 16000)); + expect(bed.length).toBe(8000); + expect(Math.sqrt(bed.reduce((s, x) => s + x * x, 0) / bed.length)).toBeCloseTo(1, 1); + expect(() => noiseFromWav(new Uint8Array(40))).toThrow(/WAV/); + }); +}); + +describe("conversation batch", () => { + it("plans repeats outermost with one seed per run, and refuses unknown mutations", () => { + const plan = planRuns({ mutations: ["none", "tentative-hold"], conditions: ["clean", "quiet"], runs: 2, seed: 10 }); + expect(plan).toHaveLength(8); + expect(plan.slice(0, 4).map((p) => `${p.mutation}/${p.condition}`)).toEqual(["none/clean", "none/quiet", "tentative-hold/clean", "tentative-hold/quiet"]); + expect(plan.map((p) => p.seed)).toEqual([10, 11, 12, 13, 14, 15, 16, 17]); + expect(() => planRuns({ mutations: ["polite"], conditions: ["clean"], runs: 1, seed: 1 })).toThrow(/Unknown mutation/); + expect(() => planRuns({ mutations: ["none"], conditions: ["clean"], runs: 0, seed: 1 })).toThrow(/--runs/); + }); + + it("expects completion only when the callee committed inside the contract's constraints", () => { + const contract = contractFromScenario(loadScenarioDir(scenariosDir()).find((s) => s.id === "restaurant-reservation")!); + expect(expectedComplete(contract, { confirmed: true, date: "2026-09-12", time: "19:30", partySize: 2 })).toBe(true); + expect(expectedComplete(contract, { confirmed: true, date: "2026-09-12", time: "18:30", partySize: 2 })).toBe(false); + expect(expectedComplete(contract, { confirmed: false })).toBe(false); + expect(expectedComplete(contract, null)).toBe(false); + }); + + it("summarizes false completions, expectations, errors and reply latency", () => { + const base: BatchRun = { mutation: "none", condition: "clean", seed: 1, index: 1, dir: "", status: "completed", complete: true, expectedComplete: true, asExpected: true, falseCompletion: false, disagreements: [], error: null, endReason: "agent_hangup", stoppedForBudget: false, replyAudioAfterMs: [800, 1200], estimatedUsd: 0.1 }; + const s = summarizeRuns([ + base, + { ...base, index: 2, mutation: "tentative-hold", complete: true, expectedComplete: false, asExpected: false, falseCompletion: true, replyAudioAfterMs: [1000] }, + { ...base, index: 3, condition: "quiet", complete: false, asExpected: false, error: "boom", endReason: null, replyAudioAfterMs: [3000] }, + ]); + expect(s).toMatchObject({ runs: 3, completed: 2, asExpected: 1, falseCompletions: 1, runtimeErrors: 1, estimatedUsd: 0.3, asExpectedRate: 0.333 }); + expect(s.replyAudioAfterMs).toEqual({ n: 4, p50: 1000, p95: 3000 }); + expect(s.byMutation["tentative-hold"]).toEqual({ runs: 1, completed: 1, asExpected: 0, falseCompletions: 1 }); + expect(s.byCondition.quiet).toEqual({ runs: 1, completed: 0, asExpected: 0, falseCompletions: 0 }); + expect(percentile([], 50)).toBeNull(); + }); +}); diff --git a/packages/cli/src/conversation-batch.ts b/packages/cli/src/conversation-batch.ts new file mode 100644 index 00000000..a449f837 --- /dev/null +++ b/packages/cli/src/conversation-batch.ts @@ -0,0 +1,102 @@ +/** + * Many local conversations against scripted characters: which runs to make (mutation × line condition × repeat), + * what each one should have ended as, and the totals. Pure; `phone test --level conversation --scenario` runs it. + */ +import { checkConstraints, type CallContract } from "@oathra/contract"; +import { MUTATIONS, type Mutation } from "@oathra/eval"; +import type { LocalConversationReport } from "./local-conversation.js"; + +export type BatchMutation = Mutation | "none"; +export type PlannedRun = { mutation: BatchMutation; condition: string; seed: number }; + +/** Repeats outermost, so a run stopped at the budget still covers every mutation and condition evenly. */ +export function planRuns(opts: { mutations: string[]; conditions: string[]; runs: number; seed: number }): PlannedRun[] { + if (!Number.isInteger(opts.runs) || opts.runs < 1 || opts.runs > 100) throw new Error("--runs must be an integer from 1 to 100"); + for (const m of opts.mutations) { + if (m !== "none" && !(MUTATIONS as readonly string[]).includes(m)) throw new Error(`Unknown mutation "${m}". Use none or one of: ${MUTATIONS.join(", ")}`); + } + if (!opts.mutations.length || !opts.conditions.length) throw new Error("need at least one mutation and one condition"); + const plan: PlannedRun[] = []; + for (let r = 0; r < opts.runs; r++) + for (const mutation of opts.mutations as BatchMutation[]) + for (const condition of opts.conditions) plan.push({ mutation, condition, seed: opts.seed + plan.length }); + return plan; +} + +/** The callee committed (or the serial matched) to terms inside the contract's constraints. */ +export function expectedComplete(contract: CallContract, truth: Record | null): boolean { + if (!truth || (truth.confirmed !== true && truth.matched !== true)) return false; + return checkConstraints(contract.constraints, truth).violations.length === 0; +} + +export type BatchRun = PlannedRun & { + index: number; + dir: string; + status: string | null; + complete: boolean; + expectedComplete: boolean; + asExpected: boolean; + falseCompletion: boolean; + disagreements: string[]; + error: string | null; + endReason: string | null; + stoppedForBudget: boolean; + replyAudioAfterMs: number[]; + estimatedUsd: number; +}; + +export function toBatchRun(planned: PlannedRun, index: number, dir: string, contract: CallContract, report: LocalConversationReport): BatchRun { + const complete = report.verdict?.complete ?? false; + const expected = expectedComplete(contract, report.truth); + return { + ...planned, index, dir, + status: report.verdict?.status ?? null, complete, expectedComplete: expected, + asExpected: report.error === null && !report.stoppedForBudget && complete === expected, + falseCompletion: report.falseCompletion, disagreements: report.disagreements, + error: report.error, endReason: report.endReason, stoppedForBudget: report.stoppedForBudget, + replyAudioAfterMs: report.replies.flatMap((r) => (r.replyAudioAfterMs === null ? [] : [r.replyAudioAfterMs])), + estimatedUsd: report.usage.estimatedUsd, + }; +} + +/** Nearest-rank percentile; null for no samples. */ +export function percentile(values: number[], p: number): number | null { + if (!values.length) return null; + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1))]!; +} + +type Group = { runs: number; completed: number; asExpected: number; falseCompletions: number }; + +export type BatchSummary = Group & { + asExpectedRate: number | null; + /** The run threw, or the call ended in `error`. */ + runtimeErrors: number; + stoppedForBudget: number; + /** Callee stops speaking → first audible agent reply, over every reply in every run. */ + replyAudioAfterMs: { n: number; p50: number | null; p95: number | null }; + estimatedUsd: number; + byMutation: Record; + byCondition: Record; +}; + +export function summarizeRuns(runs: BatchRun[]): BatchSummary { + const group = (rs: BatchRun[]): Group => ({ runs: rs.length, completed: rs.filter((r) => r.complete).length, asExpected: rs.filter((r) => r.asExpected).length, falseCompletions: rs.filter((r) => r.falseCompletion).length }); + const by = (key: (r: BatchRun) => string) => { + const out: Record = {}; + for (const k of new Set(runs.map(key))) out[k] = group(runs.filter((r) => key(r) === k)); + return out; + }; + const latency = runs.flatMap((r) => r.replyAudioAfterMs); + const all = group(runs); + return { + ...all, + asExpectedRate: runs.length ? +(all.asExpected / runs.length).toFixed(3) : null, + runtimeErrors: runs.filter((r) => r.error !== null || r.endReason === "error").length, + stoppedForBudget: runs.filter((r) => r.stoppedForBudget).length, + replyAudioAfterMs: { n: latency.length, p50: percentile(latency, 50), p95: percentile(latency, 95) }, + estimatedUsd: +runs.reduce((s, r) => s + r.estimatedUsd, 0).toFixed(4), + byMutation: by((r) => r.mutation), + byCondition: by((r) => r.condition), + }; +} diff --git a/packages/cli/src/local-conversation-character.test.ts b/packages/cli/src/local-conversation-character.test.ts new file mode 100644 index 00000000..56ecc1f4 --- /dev/null +++ b/packages/cli/src/local-conversation-character.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, it } from "vitest"; +import { defineCall } from "@oathra/contract"; +import type { BrainProvider } from "@oathra/core"; +import { AdversarialCharacter } from "@oathra/eval"; +import { contractFromScenario, loadScenarioDir } from "@oathra/scenario"; +import { createCharacter, mulberry32 } from "@oathra/simulator"; +import { pipelineEngine } from "@oathra/voice-pipeline"; +import { LineConditioner, parseCondition } from "./audio-degrade.js"; +import { toBatchRun } from "./conversation-batch.js"; +import { runLocalConversation } from "./local-conversation.js"; +import { scenariosDir } from "./paths.js"; + +// The real pipeline engine and runtime against the restaurant scenario's scripted character. Speech recognition, +// TTS and the brain are deterministic stand-ins: the callee's voice is a tone per character, recognized as the +// line it was made from, in the order the lines were synthesized. +const scenario = loadScenarioDir(scenariosDir()).find((s) => s.id === "restaurant-reservation")!; +const tone = (ms: number) => { const n = ms * 8, out = new Uint8Array(n); for (let i = 0; i < n; i++) out[i] = i % 16 < 8 ? 0x10 : 0x90; return out; }; +const isVoiced = (frame: Uint8Array) => frame.some((b) => b !== 0xff && b !== 0x7f); + +function calleeSide() { + const said: string[] = []; + let heard = 0; + const voice = { async synthesizeMulaw8k(text: string) { said.push(text); return tone(Math.min(2500, text.length * 25)); } }; + const stt = { + live() { + let listener: (e: unknown) => void = () => {}; + let ms = 0, start: number | null = null, lastVoice = 0; + return { + on(fn: (e: unknown) => void) { listener = fn; }, + async open() {}, + send(bytes: Uint8Array) { + ms += bytes.length / 8; + if (isVoiced(bytes)) { + if (start === null) { start = ms; listener({ type: "speech_started", t: start }); } + lastVoice = ms; + } else if (start !== null && ms - lastVoice >= 300) { + listener({ type: "final", text: said[heard] ?? "", startMs: start, endMs: lastVoice, confidence: 0.9, speechFinal: true }); + heard++; start = null; + } + }, + finalize() {}, close() {}, + }; + }, + }; + return { said, voice, stt }; +} +const agentTts = { async synthesizeMulaw8k(text: string) { return tone(Math.min(1500, text.length * 30)); }, async *synthesizeMulaw8kStream(text: string) { yield tone(Math.min(1500, text.length * 30)); } }; +const LINES = ["9月12日の19時以降で、2名で予約をお願いできますか。", "では19時半でお願いします。", "田中と申します。", "ありがとうございました。失礼します。"]; +function agentBrain(): BrainProvider { + let i = 0; + return { name: "fake", async respond() { const k = Math.min(i++, LINES.length - 1); return { text: LINES[k]!, action: k === LINES.length - 1 ? "hangup" as const : "continue" as const, usage: { inputTokens: 100, outputTokens: 10, costUsd: 0.001 } }; } }; +} + +async function run(mutation: "none" | "tentative-hold", condition = "clean") { + const side = calleeSide(), rng = mulberry32(7); + const base = createCharacter(scenario, rng); + const character = mutation === "none" ? base : new AdversarialCharacter(base, mutation, rng); + const brain = agentBrain(), contract = defineCall({ ...contractFromScenario(scenario), target: { phone: "+819000000000", name: scenario.callee.persona.name } }); + const report = await runLocalConversation({ + contract, brain, character, rng, callee: side.voice, maxMs: 60_000, + engine: pipelineEngine({ brain, stt: side.stt as never, tts: agentTts, acknowledgements: false }), + now: new Date(scenario.callee.knowledge.now as string), + line: new LineConditioner(parseCondition(condition), null), + }); + return { report, side, run: toBatchRun({ mutation, condition, seed: 7 }, 1, "", contract, report) }; +} + +describe("local conversation with a scripted character (no carrier)", () => { + it("books the table when the character commits, and reports the verdict and the character's truth", async () => { + const { report, side, run: r } = await run("none", "quiet"); + expect(report.error).toBeNull(); + expect(side.said[0]).toContain("お電話ありがとうございます"); + expect(report.truth).toMatchObject({ confirmed: true, time: "19:30", partySize: 2 }); + expect(report.verdict).toMatchObject({ status: "completed", complete: true, fields: { time: "19:30", partySize: 2, confirmed: true } }); + expect(report.falseCompletion).toBe(false); + expect(r).toMatchObject({ expectedComplete: true, asExpected: true }); + expect(report.replies.length).toBeGreaterThanOrEqual(3); + }, 90_000); + + it("a tentative hold never completes and is not a false completion", async () => { + const { report, side, run: r } = await run("tentative-hold"); + expect(report.error).toBeNull(); + expect(report.replies.length).toBeGreaterThanOrEqual(2); + expect(side.said.some((t) => /仮押さえ|一応|後ほど確認|確定ではございません|確定ではありません/.test(t))).toBe(true); + expect(report.truth?.confirmed).not.toBe(true); + expect(report.verdict?.complete).toBe(false); + expect(report.falseCompletion).toBe(false); + expect(r).toMatchObject({ expectedComplete: false, asExpected: true }); + }, 90_000); +}); diff --git a/packages/cli/src/local-conversation.ts b/packages/cli/src/local-conversation.ts index bbf323fd..a1914386 100644 --- a/packages/cli/src/local-conversation.ts +++ b/packages/cli/src/local-conversation.ts @@ -3,16 +3,27 @@ * path as a real call, with the telephone line replaced by a local loop and the callee played from a * short script. Nothing rings; the voice APIs are real and paid. Normal lines wait for the agent's reply to * finish playing; one line interrupts on purpose to check barge-in. + * + * With `character`, the callee is one of the simulator's scripted characters instead (optionally wrapped in an + * adversarial mutation): it hears the agent's transcript and answers turn by turn, and its `truth()` says what it + * actually committed to, so the report carries the verdict and whether it was a false completion. */ import type { PhoneRequest } from "@oathra/contract"; import type { BrainContext, BrainProvider, CallEvent } from "@oathra/core"; +import { detectFalseCompletion } from "@oathra/eval"; +import type { ResultStatus } from "@oathra/evidence"; import { PhoneTransport, type CarrierEvent, type CarrierMediaSession, type CarrierTransport } from "@oathra/phone"; -import { CallRuntime } from "@oathra/runtime"; +import { CallRuntime, type CallOutcome } from "@oathra/runtime"; +import { mulberry32, type CalleeCharacter } from "@oathra/simulator"; import { MULAW_8K, OutputQueue, toPcm16, type VoiceEngine } from "@oathra/voice"; +import type { LineConditioner } from "./audio-degrade.js"; export type CalleeLine = { text: string; interrupt?: boolean }; export type CalleeVoice = { synthesizeMulaw8k(text: string, opts: { language: "ja" | "en" }): Promise }; +/** What the callee says when the agent has been silent for 15 s (at most twice, then it hangs up). */ +const HELLO = { ja: "もしもし？", en: "Hello?" } as const; + /** A friend answering a character call. Line 4 cuts in while the agent is talking. */ export const DEFAULT_CALLEE_SCRIPT: CalleeLine[] = [ { text: "もしもし？" }, @@ -50,15 +61,30 @@ export type LocalConversationReport = { firstSystemPrompt: string | null; usage: { brainCostUsd: number; brainInputTokens: number; brainOutputTokens: number; ttsAudioSeconds: number; sttAudioSeconds: number; calleeTtsChars: number; estimatedUsd: number }; stoppedForBudget: boolean; + /** `evaluate()` at the end of the call, as the runtime reported it. Null when the run threw. */ + verdict: { status: ResultStatus; complete: boolean; fields: Record; missing: string[] } | null; + /** What a scripted character actually committed to; null for a plain script. */ + truth: Record | null; + falseCompletion: boolean; + disagreements: string[]; }; export type LocalConversationOptions = { - request: PhoneRequest; + /** Kept for the report; the call itself runs from `contract`. */ + request?: PhoneRequest; contract: import("@oathra/contract").CallContract; engine: VoiceEngine; brain: BrainProvider; callee: CalleeVoice; script?: CalleeLine[]; + /** A scripted character answers instead of `script`. */ + character?: CalleeCharacter; + /** Feeds the character's choices (default: a fixed seed). */ + rng?: () => number; + /** The call's reference time for dates and times ("tomorrow"), as in a scenario's `knowledge.now`. */ + now?: Date; + /** Quieter voice, background noise: applied to everything the callee's side sends. */ + line?: LineConditioner; /** Builds the prompt the brain sees, recorded once (the brain itself still builds its own). */ systemPrompt?: (ctx: BrainContext) => string; maxUsd?: number; @@ -72,6 +98,8 @@ const audible = (pcm: Int16Array) => { for (const x of pcm) if (x > 600 || x < - export async function runLocalConversation(opts: LocalConversationOptions): Promise { const script = opts.script ?? DEFAULT_CALLEE_SCRIPT; + const character = opts.character; + const language = opts.contract.language; const maxMs = opts.maxMs ?? 90_000; const maxUsd = opts.maxUsd ?? 0.05; const t0 = performance.now(); @@ -79,7 +107,9 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom // The line as the callee hears it: agent audio queued in order, like a carrier's playback buffer. const agentLine = new Int16Array(8 * maxMs), calleeLine = new Int16Array(8 * maxMs); - let playEnd = 0, sentMs = 0; + // playEnd counts every frame the line carries, silence included; voicedEnd only audible agent speech, which is + // what tells a listener that the agent has finished talking. + let playEnd = 0, voicedEnd = 0, sentMs = 0; const clears: number[] = []; const sends: Array<{ at: number; playAt: number }> = []; const events = new OutputQueue(); @@ -91,12 +121,12 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom const pcm = toPcm16(chunk), start = Math.max(now(), playEnd); for (let i = 0; i < pcm.length && start * 8 + i < agentLine.length; i++) agentLine[start * 8 + i] = pcm[i]!; playEnd = start + pcm.length / 8; sentMs += pcm.length / 8; - if (audible(pcm)) sends.push({ at: now(), playAt: start }); + if (audible(pcm)) { sends.push({ at: now(), playAt: start }); voicedEnd = playEnd; } }, clear() { const at = now(); for (let k = at * 8; k < Math.min(playEnd * 8, agentLine.length); k++) agentLine[k] = 0; - playEnd = at; clears.push(at); + playEnd = at; voicedEnd = Math.min(voicedEnd, at); clears.push(at); }, async hangup() { hungUp = true; }, now, @@ -116,6 +146,8 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom }, }; let calleeTtsChars = 0; + // A fatal runtime error ends the call as `error` without throwing; the report must still say why. + let fatal: string | null = null; const estimate = () => (now() / 60000) * LOCAL_CONVERSATION_RATES.sttPerMinute + (sentMs / 1000) * LOCAL_CONVERSATION_RATES.ttsPerSecond + brainCostUsd + calleeTtsChars * LOCAL_CONVERSATION_RATES.calleeTtsPerChar; const runtime = new CallRuntime({ @@ -124,8 +156,10 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom transport: new PhoneTransport(carrier, opts.engine, { transcriptNotice: true }), brain, callId: `local_${Date.now().toString(36)}`, + ...(opts.now ? { now: opts.now } : {}), onEvent: (e) => { const at = now(); + if (e.type === "error" && e.fatal && fatal === null) fatal = e.message; if (e.type === "transcript.final") transcript.push({ atMs: at, who: e.source === "callee" ? "callee" : "agent", text: e.text, ...(e.interrupted ? { interrupted: true } : {}) }); if (e.type === "agent.speech.ended" && e.interrupted) { const last = [...transcript].reverse().find((t) => t.who === "agent"); if (last) last.interrupted = true; } if (["transcript.final", "brain.response", "agent.speech.started", "call.ended"].includes(e.type)) log.push({ at, type: e.type, ...("source" in e ? { source: e.source } : {}), ...(e.type === "brain.response" ? { latencyMs: e.latencyMs } : {}) }); @@ -134,23 +168,45 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom openingTimeoutMs: 4000, }); - // Pre-synthesize the callee once (the same audio for every run of the script). + const synthesize = (text: string) => { calleeTtsChars += text.length; return opts.callee.synthesizeMulaw8k(text, { language }); }; + // Pre-synthesize what is known before the call (the script, or the character's greeting), so TTS time does not + // count as the callee's pause. const lines: Uint8Array[] = []; - for (const line of script) { lines.push(await opts.callee.synthesizeMulaw8k(line.text, { language: "ja" })); calleeTtsChars += line.text.length; } + if (!character) for (const line of script) lines.push(await synthesize(line.text)); + const greeting = character?.greeting(); + const greetingAudio = greeting ? await synthesize(greeting) : undefined; - const calleeTurns: Array<{ index: number; start: number; end: number }> = []; + const calleeTurns: Array<{ text: string; interrupt: boolean; start: number; end: number }> = []; // Set from the callee loop (a closure), so kept in a holder the compiler does not narrow to null. const cut: { start?: { calleeStartMs: number; agentWasSpeaking: boolean } } = {}; const silence = new Uint8Array(FRAME).fill(0xff); + const quiet = () => opts.line ? opts.line.silence(FRAME) : silence; const feed = async (mu: Uint8Array) => { for (let i = 0; i < mu.length && !hungUp; i += FRAME) { - const frame = mu.subarray(i, i + FRAME), pcm = toPcm16({ ...MULAW_8K, data: frame }), at = now(); - for (let k = 0; k < pcm.length && at * 8 + k < calleeLine.length; k++) calleeLine[at * 8 + k] = pcm[k]!; + const frame = mu.subarray(i, i + FRAME); + record(frame); events.push({ type: "audio", chunk: { ...MULAW_8K, data: frame } }); await sleep(20); } }; - const callee = (async () => { + const record = (frame: Uint8Array) => { + const pcm = toPcm16({ ...MULAW_8K, data: frame }), at = now(); + for (let k = 0; k < pcm.length && at * 8 + k < calleeLine.length; k++) calleeLine[at * 8 + k] = pcm[k]!; + }; + const idle = async () => { + const frame = quiet(); + if (opts.line) record(frame); + events.push({ type: "audio", chunk: { ...MULAW_8K, data: frame } }); + await sleep(20); + }; + const speak = async (text: string, mu: Uint8Array, interrupt = false) => { + const start = now(); + if (interrupt) cut.start = { calleeStartMs: start, agentWasSpeaking: playEnd > start }; + await feed(opts.line ? opts.line.speech(mu) : mu); + calleeTurns.push({ text, interrupt, start, end: now() }); + }; + + const scriptedCallee = async () => { let next = 0; const connectedAt = now(); while (!hungUp && now() < maxMs) { @@ -165,38 +221,88 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom else if (line?.interrupt) go = agentPlaying && !!agentSince && t - agentSince.playAt > 1200; else if (line) go = (replied && !agentPlaying && t - playEnd > 700) || t - lastEnd > 15_000; if (go && line) { - const start = now(); - if (line.interrupt) cut.start = { calleeStartMs: start, agentWasSpeaking: agentPlaying }; - await feed(lines[next]!); - calleeTurns.push({ index: next, start, end: now() }); + await speak(line.text, lines[next]!, !!line.interrupt); next++; continue; } if (!line && !agentPlaying && t - (calleeTurns.at(-1)?.end ?? 0) > 8000) break; - events.push({ type: "audio", chunk: { ...MULAW_8K, data: silence } }); - await sleep(20); + await idle(); } + }; + + // The character answers once the agent's reply has finished playing and its transcript is in. Its reply is + // synthesized while the line keeps running, so the agent's side never sees a gap in the audio. + const rng = opts.rng ?? mulberry32(1); + const slot: { ready?: { text: string; audio: Uint8Array | null; hangup: boolean } | undefined; waiting?: boolean } = {}; + let calleeError: string | null = null; + const prepare = (text: string, hangup: boolean) => { + slot.waiting = true; + (text.trim() ? synthesize(text) : Promise.resolve(null)).then( + (audio) => { slot.ready = { text, audio, hangup }; slot.waiting = false; }, + (e: Error) => { calleeError = `callee voice: ${e.message}`; slot.waiting = false; runtime.cancel(); }, + ); + }; + const characterCallee = async (c: CalleeCharacter) => { + const connectedAt = now(); + let turnIndex = 0, hellos = 0; + if (greeting && greetingAudio) slot.ready = { text: greeting, audio: greetingAudio, hangup: false }; + while (!hungUp && now() < maxMs && calleeError === null) { + if (estimate() > maxUsd) { stoppedForBudget = true; runtime.cancel(); break; } + const t = now(); + const ready = slot.ready; + if (ready && (calleeTurns.length || t - connectedAt > 800)) { + slot.ready = undefined; + if (ready.audio) await speak(ready.text, ready.audio); + if (ready.hangup) break; + continue; + } + if (!ready && !slot.waiting) { + const lastEnd = calleeTurns.at(-1)?.end ?? connectedAt; + const replied = sends.some((s) => s.at > lastEnd); + const heard = transcript.filter((x) => x.who === "agent" && x.atMs > lastEnd); + if (replied && t - voicedEnd > 700 && (heard.length || t - voicedEnd > 2500)) { + hellos = 0; + const reply = await c.respond({ + transcript: transcript.map((x, i) => ({ id: `turn-${i}`, source: x.who === "callee" ? "callee" : "caller", text: x.text, t: x.atMs })), + lastAgentText: heard.map((x) => x.text).join(" "), + language, turnIndex: turnIndex++, rng, + }); + prepare(reply.text, !!reply.hangup); + } else if (t - Math.max(lastEnd, voicedEnd) > 15_000) { + if (hellos++ >= 2) break; + prepare(HELLO[language], false); + } + } + await idle(); + } + }; + + const callee = (async () => { + await (character ? characterCallee(character) : scriptedCallee()); if (!hungUp) events.push({ type: "hangup", reason: "callee_done" }); })(); - let endReason: string | null = null, error: string | null = null; + let endReason: string | null = null, error: string | null = null, outcome: CallOutcome | null = null; const limit = setTimeout(() => runtime.cancel(), maxMs + 5000); - try { endReason = (await runtime.run()).endReason; } + try { outcome = await runtime.run(); endReason = outcome.endReason; } catch (e) { error = (e as Error).message; } finally { clearTimeout(limit); hungUp = true; await callee.catch(() => undefined); } + error ??= calleeError ?? fatal; + const truth = character ? (await character.truth?.()) ?? null : null; + const checked = outcome ? detectFalseCompletion(outcome, truth ?? undefined) : { falseCompletion: false, disagreements: [] }; - const interruptedTurn = calleeTurns.find((c) => script[c.index]?.interrupt); + const interruptedTurn = calleeTurns.find((c) => c.interrupt); let interruption: LocalConversationReport["interruption"] = null; if (cut.start && interruptedTurn) { const stopped = clears.find((at) => at >= interruptedTurn.start) ?? null; const nextAgentStart = log.find((l) => l.type === "agent.speech.started" && l.at > (stopped ?? interruptedTurn.start))?.at ?? Infinity; interruption = { ...cut.start, stoppedAfterMs: stopped === null ? null : stopped - interruptedTurn.start, staleAudioAfterStop: stopped !== null && sends.some((s) => s.at > stopped && s.at < nextAgentStart) }; } - const replies: ReplyTiming[] = calleeTurns.filter((c) => !script[c.index]?.interrupt).map((c) => { + const replies: ReplyTiming[] = calleeTurns.filter((c) => !c.interrupt).map((c) => { const stt = log.find((l) => l.type === "transcript.final" && l.source === "callee" && l.at >= c.start); const reply = log.find((l) => l.type === "brain.response" && l.at >= (stt?.at ?? c.end)); const audio = sends.find((s) => s.at >= c.end); - return { line: script[c.index]!.text, calleeEndMs: c.end, sttFinalAfterMs: stt ? stt.at - c.end : null, replyTextAfterMs: reply ? reply.at - c.end : null, brainLatencyMs: reply?.latencyMs ?? null, replyAudioAfterMs: audio ? audio.playAt - c.end : null }; + return { line: c.text, calleeEndMs: c.end, sttFinalAfterMs: stt ? stt.at - c.end : null, replyTextAfterMs: reply ? reply.at - c.end : null, brainLatencyMs: reply?.latencyMs ?? null, replyAudioAfterMs: audio ? audio.playAt - c.end : null }; }); const end = Math.min(agentLine.length, Math.ceil(Math.max(now(), playEnd) * 8)); const mixed = new Int16Array(end); @@ -204,6 +310,8 @@ export async function runLocalConversation(opts: LocalConversationOptions): Prom return { replies, interruption, transcript, endReason, error, durationMs: now(), agentAudio: agentLine.slice(0, end), mixedAudio: mixed, firstSystemPrompt, stoppedForBudget, + verdict: outcome ? { status: outcome.result.status, complete: outcome.result.complete, fields: outcome.result.fields, missing: outcome.result.missing } : null, + truth, falseCompletion: checked.falseCompletion, disagreements: checked.disagreements, usage: { brainCostUsd, brainInputTokens, brainOutputTokens, ttsAudioSeconds: +(sentMs / 1000).toFixed(1), sttAudioSeconds: +(now() / 1000).toFixed(1), calleeTtsChars, estimatedUsd: +estimate().toFixed(4) }, }; } diff --git a/packages/cli/src/phone.ts b/packages/cli/src/phone.ts index 251ed108..2d9f3f27 100644 --- a/packages/cli/src/phone.ts +++ b/packages/cli/src/phone.ts @@ -39,6 +39,8 @@ import { twilio } from "@oathra/phone-twilio"; import { defaultCallsDir, saveCall } from "@oathra/replay"; import { runCall } from "@oathra/runtime"; import { contractFromScenario, loadScenarioDir, loadScenarioFile, type Scenario } from "@oathra/scenario"; +import { AdversarialCharacter } from "@oathra/eval"; +import { createCharacter, mulberry32 } from "@oathra/simulator"; import { MULAW_8K, type VoiceEngine } from "@oathra/voice"; import { CHARACTER_TTS_STYLE, phoneRequestSystemPrompt, voiceSettingRecord } from "@oathra/voice-kit"; @@ -46,6 +48,9 @@ export { CHARACTER_TTS_STYLE, phoneRequestSystemPrompt }; import { pipelineEngine } from "@oathra/voice-pipeline"; import { liveModelOf, resolveBrain } from "./brains.js"; import { scenariosDir } from "./paths.js"; +import { FAST_SPEECH_RATE, LineConditioner, noiseFromWav, parseCondition, pinkNoise } from "./audio-degrade.js"; +import { planRuns, summarizeRuns, toBatchRun, type BatchRun } from "./conversation-batch.js"; +import type { LocalConversationReport } from "./local-conversation.js"; import { liveRenderer, resultBox } from "./render.js"; import { bad, bold, cyan, dim, green, ok, red, warn, yellow } from "./ui.js"; @@ -562,7 +567,7 @@ export async function phoneDoctor(flags: { provider?: string; to?: string; engin // phone test — Local (telephony ¥0, model API usage) / Gateway (¥0) / PSTN (paid) // --------------------------------------------------------------------------- -export async function phoneTest(flags: { level?: string; provider?: string; to?: string; engine?: string; scenario?: string; tts?: string; voicePreset?: string; out?: string; maxUsd?: string }): Promise { +export async function phoneTest(flags: { level?: string; provider?: string; to?: string; engine?: string; scenario?: string; tts?: string; voicePreset?: string; out?: string; maxUsd?: string } & ConversationBatchFlags): Promise { const config = loadPhoneConfig(); const level = flags.level ?? @@ -581,7 +586,7 @@ export async function phoneTest(flags: { level?: string; provider?: string; to?: if (flags.tts === "elevenlabs") await ensureEnvKeys(["ELEVENLABS_API_KEY", "ELEVENLABS_VOICE_ID"]); } if (level === "conversation") { - await localConversationTest(flags); + await (flags.scenario ? localConversationBatch(flags) : localConversationTest(flags)); return; } if (level === "local") { @@ -641,6 +646,82 @@ async function localConversationTest(flags: { engine?: string; voicePreset?: str console.log(`\n${dim("Saved:")} ${dir}\n${dim("The line is local: no transcript notice is played, and a real line adds carrier latency.")}`); } +type ConversationBatchFlags = { mutation?: string; condition?: string; runs?: string; seed?: string; maxUsdTotal?: string; noiseFile?: string }; + +/** + * `--level conversation --scenario `: the scenario's contract through the real runtime and voice engine, against + * the scenario's scripted character (optionally an adversarial mutation of it) speaking through TTS under a line + * condition. Each run records the verdict and the character's truth; the batch stops before a run that could pass + * --max-usd-total. Synthetic voices: compares builds and engines, never counts as a real call. + */ +async function localConversationBatch(flags: { engine?: string; scenario?: string; out?: string; maxUsd?: string } & ConversationBatchFlags): Promise { + if (!flags.engine) throw new Error("--level conversation needs an explicit --engine (e.g. gpt-live)"); + const spec = parseEngineSpec(flags.engine); + if (spec.id === "pipeline") throw new Error("--level conversation runs the phone-request engines: gpt-live, gemini-live or character-tts"); + const scenario = findScenario(flags.scenario!); + const conditions = (flags.condition ?? "clean").split(",").map((c) => c.trim()).filter(Boolean); + for (const c of conditions) parseCondition(c); + const seed = flags.seed === undefined ? 1 : Number(flags.seed); + if (!Number.isInteger(seed)) throw new Error("--seed must be an integer"); + const plan = planRuns({ mutations: (flags.mutation ?? "none").split(",").map((m) => m.trim()).filter(Boolean), conditions, runs: flags.runs === undefined ? 1 : Number(flags.runs), seed }); + const perRun = flags.maxUsd === undefined ? 0.3 : Number(flags.maxUsd); + if (!(perRun > 0 && perRun <= 1)) throw new Error("--max-usd must be between 0 and 1"); + const total = flags.maxUsdTotal === undefined ? 2 : Number(flags.maxUsdTotal); + if (!(total >= perRun && total <= 50)) throw new Error("--max-usd-total must be at least --max-usd and at most 50"); + const noiseBed = flags.noiseFile ? noiseFromWav(readFileSync(flags.noiseFile)) : null; + await ensureEnvKeys([...new Set([...ENGINE_CREDENTIALS[spec.id], "OPENAI_API_KEY"])]); + const now = typeof scenario.callee.knowledge.now === "string" ? new Date(scenario.callee.knowledge.now) : undefined; + let commit = "unknown"; + try { commit = (await import("node:child_process")).execSync("git rev-parse --short HEAD", { stdio: ["ignore", "pipe", "ignore"] }).toString().trim(); } catch { /* not a checkout */ } + const dir = resolve(flags.out ?? join(defaultCallsDir(), "..", "local-conversations", `batch-${new Date().toISOString().replace(/[:.]/g, "-")}`)); + mkdirSync(dir, { recursive: true }); + const { runLocalConversation, wavBytes } = await import("./local-conversation.js"); + const failedRun = (error: string): LocalConversationReport => ({ replies: [], interruption: null, transcript: [], endReason: null, error, durationMs: 0, agentAudio: new Int16Array(0), mixedAudio: new Int16Array(0), firstSystemPrompt: null, stoppedForBudget: false, verdict: null, truth: null, falseCompletion: false, disagreements: [], + usage: { brainCostUsd: 0, brainInputTokens: 0, brainOutputTokens: 0, ttsAudioSeconds: 0, sttAudioSeconds: 0, calleeTtsChars: 0, estimatedUsd: 0 } }); + + console.log(`\n${bold("Local conversations")} ${dim(`${scenario.id} · ${spec.id} · ${plan.length} runs`)} ${dim(`(no carrier · paid APIs · ≤ $${perRun}/run, stops before passing $${total})`)}\n`); + const runs: BatchRun[] = []; + let spent = 0; + for (const [index, planned] of plan.entries()) { + if (spent + perRun > total) { console.log(warn(` Stopped before run ${index + 1}/${plan.length}: it could pass the total $${total} (spent ≈ $${spent.toFixed(3)}).`)); break; } + const condition = parseCondition(planned.condition); + const rng = mulberry32(planned.seed); + const base = createCharacter(scenario, rng); + const character = planned.mutation === "none" ? base : new AdversarialCharacter(base, planned.mutation, rng); + const engine = buildEngine(spec, process.env); + const contract = defineCall({ ...contractFromScenario(scenario), target: { phone: "+819000000000", name: scenario.callee.persona.name } }); + const line = new LineConditioner(condition, condition.snrDb === null ? null : noiseBed ?? pinkNoise(8000 * 20, mulberry32(planned.seed + 0x5eed))); + let report: LocalConversationReport; + try { + report = await runLocalConversation({ + contract, engine, brain: engineBrain(spec, engine), character, rng, line, maxUsd: perRun, maxMs: 180_000, + // tts-1: a fixed, plain voice whose rate can be set; the callee's voice is the test's instrument, not its subject. + callee: new OpenAITTS({ model: "tts-1", timeoutMs: 30_000, ...(condition.fast ? { speed: FAST_SPEECH_RATE } : {}) }), + ...(now ? { now } : {}), + }); + } catch (e) { + // Before the call starts (the callee's greeting could not be synthesized): a failed run, not a failed batch. + report = failedRun((e as Error).message); + } + spent += report.usage.estimatedUsd; + const runDir = join(dir, `${String(index + 1).padStart(3, "0")}-${planned.mutation}-${planned.condition.replace(/\+/g, "_")}`); + mkdirSync(runDir, { recursive: true }); + writeFileSync(join(runDir, "call-mixed-8k.wav"), wavBytes(report.mixedAudio)); + const { agentAudio, mixedAudio, ...rest } = report; + writeFileSync(join(runDir, "report.json"), JSON.stringify({ createdAt: new Date().toISOString(), commit, engine: spec.id, scenario: scenario.id, ...planned, callee: "openai-tts:tts-1 (scripted character)", ...rest }, null, 2)); + const run = toBatchRun(planned, index + 1, runDir, contract, report); + runs.push(run); + const mark = run.falseCompletion ? red("✗ FALSE COMPLETION") : run.asExpected ? green("✓") : yellow("·"); + console.log(` ${mark} ${String(index + 1).padStart(3)} ${planned.mutation.padEnd(20)} ${planned.condition.padEnd(16)} ${run.status ?? "—"} ${dim(`(expected ${run.expectedComplete ? "completed" : "not completed"})`)}${run.error ? red(` ${run.error}`) : ""} ${dim(`$${run.estimatedUsd}`)}`); + } + const summary = summarizeRuns(runs); + writeFileSync(join(dir, "summary.json"), JSON.stringify({ createdAt: new Date().toISOString(), commit, engine: spec.id, scenario: scenario.id, planned: plan.length, maxUsdPerRun: perRun, maxUsdTotal: total, noise: flags.noiseFile ? "file" : "pink", summary, runs }, null, 2)); + console.log(`\n ${summary.falseCompletions ? red(`False Completion: ${summary.falseCompletions} / ${summary.runs}`) : green(`False Completion: 0 / ${summary.runs}`)}`); + console.log(` as expected ${summary.asExpected}/${summary.runs} runtime errors ${summary.runtimeErrors} reply audio p50 ${summary.replyAudioAfterMs.p50 ?? "—"} ms · p95 ${summary.replyAudioAfterMs.p95 ?? "—"} ms (n=${summary.replyAudioAfterMs.n}) estimated $${summary.estimatedUsd}`); + console.log(`\n${dim("Saved:")} ${dir}\n${dim("Synthetic callee voices on a local line: not a real call, and a real line adds carrier latency.")}`); + if (summary.falseCompletions || runs.length < plan.length) process.exitCode = 1; +} + /** Engine loopback: synthesized callee speech → engine → expect agent audio + transcripts. Costs a few yen of API, no telephony. */ async function localLoopback(spec: EngineSpec, voicePreset?: string): Promise { const engine = buildEngine(spec, process.env, undefined, voicePreset); diff --git a/providers/openai/src/tts.ts b/providers/openai/src/tts.ts index fffb1c6d..527495ce 100644 --- a/providers/openai/src/tts.ts +++ b/providers/openai/src/tts.ts @@ -9,6 +9,8 @@ export type OpenAITTSOptions = { voice?: string; /** Style hint passed as `instructions` (gpt-4o-mini-tts). */ instructions?: string; + /** Speaking rate, 0.25–4 (1 = normal). Absent: the API's default. */ + speed?: number; timeoutMs?: number; }; @@ -22,6 +24,7 @@ export class OpenAITTS implements TTSProvider { private readonly model: string; private readonly voice: string; private readonly instructions: string | undefined; + private readonly speed: number | undefined; private readonly timeoutMs: number; constructor(opts: OpenAITTSOptions = {}) { @@ -30,6 +33,7 @@ export class OpenAITTS implements TTSProvider { this.model = opts.model ?? "gpt-4o-mini-tts"; this.voice = opts.voice ?? "alloy"; this.instructions = opts.instructions; + this.speed = opts.speed; this.timeoutMs = opts.timeoutMs ?? 20_000; this.name = `openai-tts:${this.model}`; } @@ -45,6 +49,7 @@ export class OpenAITTS implements TTSProvider { }; const instructions = this.instructions ?? (opts.language === "ja" ? "自然で丁寧な日本語の電話応対。落ち着いた速さで話す。" : undefined); if (instructions && this.model.includes("tts") && !this.model.startsWith("tts-1")) body.instructions = instructions; + if (this.speed !== undefined) body.speed = this.speed; const res = await fetch(`${this.baseUrl}/audio/speech`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${this.apiKey}` }, @@ -99,6 +104,7 @@ export class OpenAITTS implements TTSProvider { const body: Record = { model: this.model, voice: opts.voice ?? this.voice, input: text, response_format: "pcm" }; const instructions = this.instructions ?? (opts.language === "ja" ? "自然で丁寧な日本語の電話応対。落ち着いた速さで話す。" : undefined); if (instructions && this.model.includes("tts") && !this.model.startsWith("tts-1")) body.instructions = instructions; + if (this.speed !== undefined) body.speed = this.speed; return fetch(`${this.baseUrl}/audio/speech`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${this.apiKey}` }, diff --git a/providers/simulator/src/character.ts b/providers/simulator/src/character.ts index f402cfdf..0b8ec35f 100644 --- a/providers/simulator/src/character.ts +++ b/providers/simulator/src/character.ts @@ -29,7 +29,8 @@ export type CharacterFactory = (scenario: Scenario, rng: () => number) => Callee // Shared phrase helpers ------------------------------------------------------- -export const GOODBYE_RE = /失礼(?:します|いたします)|ありがとうございました|では[、,]?また|さようなら|それでは失礼|goodbye|bye\b|thank you,? (?:that's all|goodbye)/i; +// "お忙しいところ失礼いたします" opens a call; it is not a goodbye. +export const GOODBYE_RE = /(? { expect(Date.now() - started).toBeLessThan(5000); }, 10000); }); + +describe("goodbye detection", () => { + it("an opening apology is not a goodbye; a closing 失礼いたします still is", () => { + for (const opening of ["お忙しいところ失礼いたします。9月12日の夜に二名で予約をお願いしたいのですが、ご都合いかがでしょうか。", "夜分遅くに失礼します。", "突然失礼いたします。", "お忙しいところ大変失礼いたします。"]) expect(GOODBYE_RE.test(opening)).toBe(false); + for (const closing of ["ありがとうございました。失礼いたします。", "では失礼します。", "お忙しいところ失礼いたしました。それでは失礼いたします。"]) expect(GOODBYE_RE.test(closing)).toBe(true); + }); +});