Repository navigation
Conversation
…conditions (layer B, no carrier) - `phone test --level conversation --scenario <id>` runs the simulator's character (optionally adversarial) over the local μ-law line and reports evaluate()'s verdict, the character's truth and false completions - --mutation / --condition (clean, quiet, fast, noise-<snr>db, --noise-file) / --runs / --seed / --max-usd-total; summary.json with false completions, as-expected rate, reply latency p50/p95, cost; exit 1 on a false completion - turn-taking uses audible agent audio only; fatal runtime errors surface in the report - OpenAITTS: optional speed - simulator: 「お忙しいところ失礼いたします」 is an opening, not a goodbye - docs: real-call verification plan (layers A/B/C) and the first smoke findings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0195nzJ5oDJAnjHtFzHM9sL3
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
実電話の前段(層 B)として、電話会社を使わずに本物の音声エンジンとシミュレーターの店員キャラクターを会話させ、誤完了と取りこぼしを測る試験を追加します。計画は
docs/launch/real-call-plan.md、実装と初回試験の記録はdocs/quality/conversation-batch.mdにあります。pnpm oathra phone test --level conversation --engine gpt-live --scenario restaurant-reservation \ --mutation none,tentative-hold,ask-back --condition clean,quiet,fast,noise-10db \ --runs 5 --max-usd 0.3 --max-usd-total 20変更
packages/cli/src/local-conversation.ts: 相手役に台本キャラクターを使えるようにしました(敵対的な変異にも対応)。report にevaluate()の判定、キャラクターの truth、誤完了の有無を出します。順番の判定は、実際に声が出ている部分だけで行います。実行時の致命的なエラーも report に出します。packages/cli/src/audio-degrade.ts(新規): 相手側の音の条件として、小声(−18 dB)、騒音(指定の SNR、ピンクノイズまたは WAV)、早口(話す速さ 1.5 倍)を追加しました。packages/cli/src/conversation-batch.ts(新規)、phone.ts、commands.ts: 変異 × 条件 × 回数をまとめて実行します。summary.jsonに集計を出し、合計の予算上限を超える前に止まります。誤完了が出たとき、または途中で止まったときは exit 1 で終わります。providers/openai/src/tts.ts: 任意のspeedを追加しました。指定しなければ従来どおりです。providers/simulator/src/character.ts: 「お忙しいところ失礼いたします」を別れの挨拶と誤認していたのを直しました。検証
pnpm buildpnpm testpnpm test:gatewaypnpm lint:depspnpm oathra evalfalse-completion-trap)pnpm oathra eval --adversarial 10000未検証
🤖 Generated with Claude Code
https://claude.ai/code/session_0195nzJ5oDJAnjHtFzHM9sL3