[longzhi]

Voice commands (#132, ADR-0017): hold K0, speak, and a resident session answers with a spoken reply in the voice and persona of the Character the buddy wears. Experimental: off unless the daemon runs with VIBEBUDDY_RESIDENT_SESSION=1. Contains #145 (microphone bring-up).

Summary

sequenceDiagram
    participant Box
    participant Daemon
    participant Service as vibebuddy-api (staging)
    participant Brain as Resident session (Codex or Claude Code)
    Box->>Daemon: VOICE START / DATA (24 kHz IMA ADPCM) / END
    Daemon->>Service: POST /v0/voice/transcribe (WAV)
    Service-->>Daemon: text (Groq, Workers AI fallback)
    Daemon->>Box: voice.heard
    Daemon->>Brain: turn (context + text)
    Brain->>Daemon: speak(text)
    Daemon->>Box: voice.reply
    Daemon->>Service: POST /v0/voice/speak (Character's voice)
    Service-->>Daemon: 24 kHz 48 kbps MP3 stream
    Daemon->>Box: voice.audio pieces, then voice.done

Loading

firmware-rs/
├── device/src/mic.rs          # ES8311 ADC on GPIO47, linear PSRAM capture; device.mic.record diagnostic
├── core/src/voice_command.rs  # push-to-talk, LISTENING / THINKING / reply overlay
├── core/src/spoken_reply.rs   # MP3 reply playback (vendored nanomp3), gaps skipped
├── core/src/font.rs           # 12 px CJK pixel font (OFL) for Chinese replies
└── device/src/main.rs         # RESET <reason> [PANIC] at boot
daemon/src/
├── voice.rs                   # turns: recognize → resident → synthesize, events paced into the link
├── resident.rs, app_server.rs # one Codex thread for good, own CODEX_HOME, only our tools
├── resident_claude.rs         # same on Claude Code, user's own Anthropic API key only
├── persona.rs                 # the Character's name, language, persona, form of address
├── speech.rs                  # voice service, or the user's own OpenAI-compatible recognizer
└── synthesis.rs               # voice service: ElevenLabs, Doubao (volc:), stock voice by gender
app/ + desktop/                # Conversation tab, brain picker, API key field (Mac and Linux)

• Box: the K0 long press is push-to-talk only while the computer offers voice (voice.available); otherwise it gives up the Pomodoro phase as before (ADR-0007 amendment). An Agent's news waits until the voice overlay closes.
• Resident session: one long-running session that only talks: no shell, no file edits, no plugins, skills or MCP servers, no shell snapshots. Tools are speak, list_activities, search_memories (Codex only) and web search. It reads the user's Codex memories read-only. Every command carries the local time, the time zone and a language reminder.
• Apps: a Conversation tab shows only when the resident session is on. Same features on Mac and Linux, zh-Hans strings included.
• Docs: ADR-0017, docs/resident-session.md, docs/protocol.md "Voice commands (experimental)", CONTEXT.md terms, docs/firmware-bringup.md (mic, reset reason).

Evidence

• cargo test --workspace --locked passes: daemon 317 tests, firmware core 143 tests, and the rest. cargo clippy adds no new warnings (the three it reports are on main too). tools/check-localization.py passes (309 keys). CI: rust and mac-app pass.
• On the box (30:ED, 2026-10-20), a Chinese question to Wanwan Xiaohe:
  • recognized in 0.4 to 0.5 s (Groq);
  • answered after a web search (11.9 s on Codex, 14.2 s on Claude just after a Character change);
  • spoken in her own Doubao voice (first audio 1.6 to 2.9 s);
  • played to the end.
• Before: voice.done was dropped with QueueFull behind a 46 KB reply, so the overlay hung until its timeout.
  After: a 47 KB reply ended cleanly; a_turns_events_wait_for_room_in_a_full_queue_instead_of_dropping covers it.
• Boot report checked by a reset over USB: RESET Some(CoreUsbUart), matching the ROM's rst:0x15.

Merge Danger

Door: two-way

Everything new is behind VIBEBUDDY_RESIDENT_SESSION=1. Firmware messages are new and additive. The voice service has no production route until accounts land (second-state/vibebuddy-cloud#2), so a release build has no recognizer or synthesis unless the user sets their own.

Blast Radius: contained

• The firmware changes the main stack layout (mic buffer on the heap) and the speaker's I2S setup (shares clocks with the mic receiver). Announcements and the boot report were rechecked on the box.
• The UART line parser now buffers UART bytes separately from USB.
• ADR number: claude/touchscreen (Touchscreen on boxes that have one: menu taps, petting, TOUCH status #149, Tap a task card to open that card's agent #150) also adds an ADR-0017. Whichever merges second renumbers; claude/accounts already uses 0017 for voice and 0018 for accounts.
• Test recordings committed by mistake in 2daaadb's parent are still in history: squash-merge.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.