Skip to content

feat(chat): render images in agent turns, paste-to-send from the input - #160

Merged
brettchien merged 1 commit into
mainfrom
orca/chat-image-content
Sep 13, 2026
Merged

brettchien merged 1 commit into
mainfrom
orca/chat-image-content

Conversation

@brettchien

Copy link
Copy Markdown
Contributor

Summary

  • Closes Chat panel: render images in agent turns + paste-to-send images from the input #158. The ACP wire already supports image ContentBlocks (crates/openab-gateway/src/adapters/acp_schema.rs:5311 in openabdev/openab) — the gap was the bridge and the console, not the protocol.
  • Bridge (crates/acp-tunnel, src-tauri/src/remote.rs): parse_inbound now classifies session/update content by its type (text → Inbound::AgentChunk, image → new Inbound::AgentImage), forwarded to the panel as its own agent-update (kind: "image"). Session::prompt takes a new images: &[PromptImage] param and appends image blocks after the text block; agent_prompt/OutMsg::Prompt thread it through.
  • Console render (console/src/chat.ts): ChatTurn gains an optional images field; appendImage mirrors appendChunk's open-turn rule. Images render as data: URI <img>s behind a mime allow-list (png/jpeg/gif/webp) + a base64-shape check, since agent-sourced content is less trusted than markdown text (same concern the issue flagged for a DOMPurify-safety pass).
  • Console input (console/src/chatPanel.ts): a paste handler on the textarea picks up clipboard image items, base64-encodes them client-side, and shows a removable preview strip above the input; submit() bundles pending images into the next agent_prompt call and allows an image-only send (empty text).
  • Size decision (issue's open question): 5MB raw per image, 4 images per message, enforced on paste; render side caps at ~8MB decoded/mimeType allow-list. No downscale in this slice — an oversized paste is rejected with a note rather than silently shrunk; that's a reasonable follow-up if it turns out to matter in practice.

Test plan

  • cargo test -p acp-tunnel — 38/38 pass (12 new: prompt image-block building, session/update image classification)
  • cargo check --workspace — clean (workspace excludes src-tauri, which needs GTK/webkit system libs this Linux container doesn't have — same reason desktop.yml's Linux compile-validation stage is disabled; src-tauri's edits are small/mechanical and covered by the macOS desktop.yml job on this PR)
  • npm run typecheck (console) — clean
  • npm test (console) — 117/117 pass (10 new: appendImage, isRenderableImage, turnHtml image rendering)
  • Manual verification against a live /acp gateway (paste an image in, confirm an agent-sent image renders) — not done in this session, no live gateway available here

🤖 Generated with Claude Code

Closes #158. The ACP wire already supports image ContentBlocks; the gap was
the bridge (only forwarded text chunks) and the console (no image render, no
paste handling). Bridge: acp-tunnel classifies session/update content by
type (text/image), remote.rs forwards image chunks as their own agent-update
and threads PromptImage attachments through agent_prompt/session.prompt.
Console: chat.ts renders images behind a mime allow-list + base64-shape
check (agent content is less trusted than markdown text), chatPanel.ts adds
a paste handler with a 5MB/image, 4-images/turn cap and a removable preview
strip above the input.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@brettchien
brettchien merged commit fb4d760 into main Sep 13, 2026
2 checks passed
@brettchien
brettchien deleted the orca/chat-image-content branch September 13, 2026 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat panel: render images in agent turns + paste-to-send images from the input

1 participant