diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/README.md b/docs/acceptance/browser-screenshot-viewer-20261022/README.md new file mode 100644 index 00000000..bbef6695 --- /dev/null +++ b/docs/acceptance/browser-screenshot-viewer-20261022/README.md @@ -0,0 +1,40 @@ +# Agent Browser screenshot viewer + +Clicking an Agent Browser screenshot now opens the existing original-image viewer: fit or 100% size, zoom, copy PNG, download the original, and Escape to return focus. An updated screenshot, browser session, or Run closes the previous preview. Verification or image-decoding failures do not enable a preview. + +## Real UI comparison + +Captured in Microsoft Edge on Windows. Both versions use the same production Go API (`cmd/workbenchjourney`) and the same screenshot produced by the native Chrome browser tool against its owned local test page. The offline provider supplies deterministic tool calls. The baseline is `fc0935d6fc03ca7abccbd148ce52d26c2bc154f8`; the updated view is the production Vite build from this change. + +The API keeps its startup bundle in memory, so the updated production bundle was served with Vite preview proxying the same API. The card comparisons are direct Playwright element screenshots at a 1280 × 900 viewport. The viewer screenshots use 1280 × 900 and 480 × 600 viewports. + +| Before: thumbnail only | After: open original | +| --- | --- | +| ![Before](before-card.png) | ![After](after-card.png) | + +![Original viewer](after-viewer.png) + +![Original viewer in the minimum desktop window](after-viewer-narrow.png) + +## Verification + +- Agent Browser component: 21 tests passed; existing image-viewer suite: 8 tests passed. Covers verified bytes, original dimensions, copy/download, focus restoration, screenshot/session/Run changes, rejected/undecodable images, and recovery after a failed screenshot download. +- TypeScript and production build passed. +- Actual browser clicks opened the screenshot, switched to 100%, downloaded the original, copied PNG to the clipboard, fitted the view, and closed with Escape. Clipboard image dimensions remained 1258 × 802; focus returned to the screenshot button. The 480 × 600 dialog stayed inside the viewport. +- Downloaded file SHA-256 matched the server screenshot: `1caabbcbf77b317544d9040675eb72da092e7163b769ec64f6e218622f2562b8` (36,058 bytes). + +The approach keeps UC's existing verified-screenshot model and reuses its image viewer. Browser inspection is an established workflow in [Codex](https://learn.chatgpt.com/docs/browser?surface=app) and [Claude Code Desktop](https://code.claude.com/docs/en/desktop); this change improves inspection of UC's captured pages. + +## Recovering a failed screenshot request + +A follow-up review reproduced a missing recovery action: if the first screenshot download failed, the card asked the user to refresh the status but provided no reload button. An unchanged screenshot identity also prevented subsequent status reads from retrying the image. The card now offers **Reload screenshot** after a failed read; it repeats the verified image GET for the same captured page. + +These additional screenshots use the production Go API, an actual native browser capture, and Edge at 1280 × 900. In each frontend, Playwright interrupted only the first screenshot request with `connectionreset`; no response bodies or DOM content were replaced. Before is commit `2358e46d4f80535858f10f0dba7a0976421992ad`; after is the production build containing the retry action. Both use the same browser session and image. + +| Before: no recovery action | After: explicit retry | +| --- | --- | +| ![Failed screenshot without retry](retry-before.png) | ![Failed screenshot with retry](retry-after.png) | + +![Same screenshot recovered after retry](retry-recovered.png) + +The retry returned a verified 1258 × 802 image, cleared the error and allowed the original viewer to open. No API mutation was issued during retry or preview. The browser was then stopped through the existing control. The new regression test failed on the previous implementation and passed with the fix. diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/after-card.png b/docs/acceptance/browser-screenshot-viewer-20261022/after-card.png new file mode 100644 index 00000000..6aff2da4 Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/after-card.png differ diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/after-viewer-narrow.png b/docs/acceptance/browser-screenshot-viewer-20261022/after-viewer-narrow.png new file mode 100644 index 00000000..cd327236 Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/after-viewer-narrow.png differ diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/after-viewer.png b/docs/acceptance/browser-screenshot-viewer-20261022/after-viewer.png new file mode 100644 index 00000000..3121ff9d Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/after-viewer.png differ diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/before-card.png b/docs/acceptance/browser-screenshot-viewer-20261022/before-card.png new file mode 100644 index 00000000..05908fc4 Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/before-card.png differ diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/retry-after.png b/docs/acceptance/browser-screenshot-viewer-20261022/retry-after.png new file mode 100644 index 00000000..a7a9c692 Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/retry-after.png differ diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/retry-before.png b/docs/acceptance/browser-screenshot-viewer-20261022/retry-before.png new file mode 100644 index 00000000..c8976510 Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/retry-before.png differ diff --git a/docs/acceptance/browser-screenshot-viewer-20261022/retry-recovered.png b/docs/acceptance/browser-screenshot-viewer-20261022/retry-recovered.png new file mode 100644 index 00000000..cfc46c20 Binary files /dev/null and b/docs/acceptance/browser-screenshot-viewer-20261022/retry-recovered.png differ diff --git a/web/src/v2/components/agent-browser.css b/web/src/v2/components/agent-browser.css index a5763fb3..307c779c 100644 --- a/web/src/v2/components/agent-browser.css +++ b/web/src/v2/components/agent-browser.css @@ -15,6 +15,9 @@ .v2-agent-browser-page a { display:block; max-width:100%; color:var(--v2-accent); text-decoration:underline; text-underline-offset:3px; } .v2-agent-browser-page a svg { display:inline; margin-left:5px; vertical-align:middle; } .v2-agent-browser img { display:block; width:100%; height:auto; max-height:320px; object-fit:contain; border:1px solid var(--v2-border); border-radius:8px; background:var(--prayu-surface-solid); } +.v2-agent-browser .v2-agent-browser-screenshot { position:relative; display:block; width:100%; min-width:0; padding:0; border:0; cursor:zoom-in; } +.v2-agent-browser-screenshot span { position:absolute; right:8px; top:8px; display:inline-flex; align-items:center; gap:5px; padding:4px 8px; border:1px solid var(--v2-border); border-radius:6px; background:var(--prayu-surface-solid); } +.v2-agent-browser .v2-agent-browser-screenshot:disabled { cursor:default; } .v2-agent-browser details { min-width:0; } .v2-agent-browser summary { cursor:pointer; color:var(--v2-muted); } .v2-agent-browser details p { margin-top:7px; } diff --git a/web/src/v2/components/agent-browser.test.tsx b/web/src/v2/components/agent-browser.test.tsx index 6929caf9..26bf82a1 100644 --- a/web/src/v2/components/agent-browser.test.tsx +++ b/web/src/v2/components/agent-browser.test.tsx @@ -1,7 +1,7 @@ import type { ReactElement } from "react"; import type { RenderOptions } from "@testing-library/react"; import { LocaleProvider } from "../../lib/locale"; -import { act, cleanup, fireEvent, render as renderComponent, screen, waitFor } from "@testing-library/react"; +import { act, cleanup, fireEvent, render as renderComponent, screen, waitFor, within } from "@testing-library/react"; import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { APIRequestError, type APIClient } from "../../api/client"; import { V2AgentBrowser } from "./agent-browser"; @@ -43,6 +43,100 @@ it("uses a read-only GET and renders only a verified screenshot", async () => { expect(screen.queryByRole("button", { name: /打开/ })).not.toBeInTheDocument(); }); +async function decodeScreenshot() { + const image = await screen.findByRole("img", { name: "Agent 浏览器页面：Report run-browser" }); + Object.defineProperties(image, { naturalWidth: { value: 1600 }, naturalHeight: { value: 900 } }); + fireEvent.load(image); + return screen.getByRole("button", { name: "查看浏览器截图原图" }); +} + +it("opens the decoded screenshot at its original dimensions and reuses verified bytes for copy and download", async () => { + const png = new Blob(["verified PNG"], { type: "image/png" }); + const items: Record>[] = []; + vi.stubGlobal("ClipboardItem", class { constructor(data: Record>) { items.push(data); } }); + const write = vi.fn(async () => {}); + vi.stubGlobal("navigator", Object.create(navigator, { clipboard: { value: { write } } })); + const client = { baseURL: "/api/v1", get: vi.fn(async () => status()), postControl: vi.fn(), + downloadVerifiedImage: vi.fn(async () => png) } as unknown as APIClient; + draw(client); + expect(await screen.findByRole("button", { name: "查看浏览器截图原图" })).toBeDisabled(); + const trigger = await decodeScreenshot(); + trigger.focus(); + fireEvent.click(trigger); + const dialog = screen.getByRole("dialog", { name: "图片预览 agent-browser-aaaaaaaaaaaa.png" }); + fireEvent.click(within(dialog).getByRole("button", { name: "100%" })); + expect(within(dialog).getByRole("img")).toHaveStyle({ width: "1600px", height: "900px" }); + const download = within(dialog).getByRole("link", { name: "下载原图 agent-browser-aaaaaaaaaaaa.png" }); + expect(download).toHaveAttribute("href", "blob:agent-browser"); + expect(download).toHaveAttribute("download", "agent-browser-aaaaaaaaaaaa.png"); + expect(write).not.toHaveBeenCalled(); + fireEvent.click(within(dialog).getByRole("button", { name: "复制图片" })); + expect(await screen.findByText("已复制图片")).toBeInTheDocument(); + expect(await items[0]["image/png"]).toBe(png); + expect(client.downloadVerifiedImage).toHaveBeenCalledTimes(1); + expect(client.postControl).not.toHaveBeenCalled(); + fireEvent.keyDown(dialog, { key: "Escape" }); + expect(screen.queryByRole("dialog")).not.toBeInTheDocument(); + expect(trigger).toHaveFocus(); +}); + +it.each(["screenshot", "session", "run"])("closes an original preview when the %s changes", async (change) => { + const client = { baseURL: "/api/v1", get: vi.fn(async () => status()), postControl: vi.fn(), + downloadVerifiedImage: vi.fn(async () => new Blob(["png"], { type: "image/png" })) } as unknown as APIClient; + const view = draw(client); + fireEvent.click(await decodeScreenshot()); + expect(screen.getByRole("dialog")).toBeInTheDocument(); + vi.mocked(client.downloadVerifiedImage).mockImplementationOnce(() => new Promise(() => {})); + if (change === "run") { + vi.mocked(client.get).mockResolvedValue(status("run-new", "agent-browser-new")); + view.rerender(); + } else { + const next = change === "session" ? status("run-browser", "agent-browser-new") + : { ...status(), screenshot: { ...status().screenshot, sha256: "b".repeat(64) } }; + await act(async () => { view.queries.setQueryData(agentBrowserQueryKey(client.baseURL, "run-browser"), next); }); + } + await waitFor(() => expect(screen.queryByRole("dialog")).not.toBeInTheDocument()); + expect(screen.queryByRole("img")).not.toBeInTheDocument(); + expect(URL.revokeObjectURL).toHaveBeenCalledWith("blob:agent-browser"); +}); + +it("does not offer an original preview when screenshot verification fails", async () => { + const client = { baseURL: "/api/v1", get: vi.fn(async () => status()), postControl: vi.fn(), + downloadVerifiedImage: vi.fn(async () => { throw new Error("hash mismatch"); }) } as unknown as APIClient; + draw(client); + expect(await screen.findByRole("alert")).toHaveTextContent("无法验证当前页面截图"); + expect(screen.queryByRole("button", { name: "查看浏览器截图原图" })).not.toBeInTheDocument(); + expect(screen.queryByRole("dialog")).not.toBeInTheDocument(); +}); + +it("can reload the same screenshot after a failed download without issuing browser actions", async () => { + const download = vi.fn() + .mockRejectedValueOnce(new TypeError("Failed to fetch")) + .mockResolvedValueOnce(new Blob(["verified PNG"], { type: "image/png" })); + const client = { baseURL: "/api/v1", get: vi.fn(async () => status()), postControl: vi.fn(), + downloadVerifiedImage: download } as unknown as APIClient; + draw(client); + expect(await screen.findByRole("alert")).toHaveTextContent("无法验证当前页面截图"); + fireEvent.click(screen.getByRole("button", { name: "重新读取截图" })); + const trigger = await decodeScreenshot(); + expect(trigger).toBeEnabled(); + expect(download).toHaveBeenCalledTimes(2); + expect(download.mock.calls[1].slice(0, 2)).toEqual(download.mock.calls[0].slice(0, 2)); + expect(screen.queryByRole("alert")).not.toBeInTheDocument(); + fireEvent.click(trigger); + expect(screen.getByRole("dialog")).toBeInTheDocument(); + expect(client.postControl).not.toHaveBeenCalled(); +}); + +it("reports an undecodable screenshot without enabling its original preview", async () => { + const client = { baseURL: "/api/v1", get: vi.fn(async () => status()), postControl: vi.fn(), + downloadVerifiedImage: vi.fn(async () => new Blob(["broken PNG"], { type: "image/png" })) } as unknown as APIClient; + draw(client); + fireEvent.error(await screen.findByRole("img", { name: "Agent 浏览器页面：Report run-browser" })); + expect(screen.getByRole("alert")).toHaveTextContent("无法显示当前页面截图"); + expect(screen.getByRole("button", { name: "查看浏览器截图原图" })).toBeDisabled(); +}); + it("closes the exact observed session once and distinguishes the Run", async () => { const get = vi.fn(async () => status()); const closed = { ...status(), state: "closed", screenshot: undefined, tree_reaped: true, profile_removed: true }; diff --git a/web/src/v2/components/agent-browser.tsx b/web/src/v2/components/agent-browser.tsx index e9d9b796..9d13dc81 100644 --- a/web/src/v2/components/agent-browser.tsx +++ b/web/src/v2/components/agent-browser.tsx @@ -1,9 +1,11 @@ -import { useEffect, useState } from "react"; +import { useEffect, useRef, useState } from "react"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; -import { ExternalLink, RefreshCw, X } from "lucide-react"; +import { ExternalLink, Maximize2, RefreshCw, X } from "lucide-react"; import { useLocale, type LocaleText } from "../../lib/locale"; import type { APIClient } from "../../api/client"; import { agentBrowserAPIAbsent, agentBrowserQueryKey, closeAgentBrowser, readAgentBrowser, type AgentBrowserStatus } from "../../api/agent-browser"; +import { V2ImageViewer } from "./image-viewer"; +import "./image-input.css"; import "./agent-browser.css"; export function V2AgentBrowser(props: { client: APIClient; runID: string; running: boolean; onDismiss?: () => void }) { @@ -55,7 +57,9 @@ function AgentBrowserCard({ client, runID, running, onDismiss }: { client: APICl const queries = useQueryClient(); const key = agentBrowserQueryKey(client.baseURL, runID); const [message, setMessage] = useState(""); - const [image, setImage] = useState<{ binding: string; url: string }>(); + const [image, setImage] = useState<{ binding: string; url: string; blob: Blob }>(); + const [failedImageBinding, setFailedImageBinding] = useState(""); + const [imageRetry, setImageRetry] = useState(0); const status = useQuery({ queryKey: key, queryFn: ({ signal }) => readAgentBrowser(client, runID, signal), retry: false, refetchOnMount: "always", refetchInterval: (query) => running && !query.state.error ? 3000 : false }); const current = status.data; @@ -75,14 +79,15 @@ function AgentBrowserCard({ client, runID, running, onDismiss }: { client: APICl ? `${runID}\0${current.session_id}\0${current.screenshot.locator}\0${current.screenshot.sha256}` : ""; useEffect(() => { setImage(undefined); + setFailedImageBinding(""); if (!current?.screenshot || !current.session_id || !imageBinding) return; const abort = new AbortController(); let objectURL = ""; void client.downloadVerifiedImage(screenshotPath(runID, current), current.screenshot, abort.signal).then((blob) => { if (abort.signal.aborted) return; - objectURL = URL.createObjectURL(blob); setImage({ binding: imageBinding, url: objectURL }); - }).catch(() => { if (!abort.signal.aborted) setMessage(t("无法验证当前页面截图，请重新读取状态。", "The current screenshot could not be verified. Refresh the browser status.")); }); + objectURL = URL.createObjectURL(blob); setImage({ binding: imageBinding, url: objectURL, blob }); + }).catch(() => { if (!abort.signal.aborted) setFailedImageBinding(imageBinding); }); return () => { abort.abort(); if (objectURL) URL.revokeObjectURL(objectURL); }; - }, [client, imageBinding, runID]); + }, [client, imageBinding, runID, imageRetry]); if (status.isError && agentBrowserAPIAbsent(status.error)) return null; if (status.isPending) return null; @@ -107,7 +112,13 @@ function AgentBrowserCard({ client, runID, running, onDismiss }: { client: APICl {current.title && {current.title}} {current.url && {current.url}} } - {image?.binding === imageBinding && {t(`Agent} + {image?.binding === imageBinding && } + {failedImageBinding && failedImageBinding === imageBinding &&

+ {t("无法验证当前页面截图。", "The current screenshot could not be verified.")}{" "} + +

} {(current.last_action || current.updated_at || current.failure_code) &&
{t("最近状态", "Latest status")} {current.last_action &&

{t("最近动作：", "Last action: ")}{actionText(current.last_action, t)}

}

{t("更新时间：", "Updated: ")}{new Date(current.updated_at).toLocaleTimeString(locale)}

@@ -118,3 +129,25 @@ function AgentBrowserCard({ client, runID, running, onDismiss }: { client: APICl ; } + +function BrowserScreenshot({ url, blob, name, alt }: { url: string; blob: Blob; name: string; alt: string }) { + const { t } = useLocale(); + const trigger = useRef(null); + const [dimensions, setDimensions] = useState<{ width: number; height: number }>(); + const [failed, setFailed] = useState(false); + const [viewing, setViewing] = useState(false); + return <> + + {failed &&

{t("无法显示当前页面截图，请让 Agent 重新截图。", "The current screenshot could not be displayed. Ask the agent to capture it again.")}

} + {viewing && dimensions && setViewing(false)} returnFocusRef={trigger} />} + ; +}