feat(website): live sites, browser/v1 sign-in and WARC replay (FD-4683) by williamloo-scale · Pull Request #447 · scaleapi/agentenv-framework

williamloo-scale · 2026-11-02T19:10:29Z

Stacked on #446.

Live and replayed websites.

Live sites. @website(domains=[...], live=True, allow_hosts=[...]) declares a real site: its server serves only health, the data plane and the card (which advertises live and allow_hosts). @session returns the cookies a browser adds to start signed in; self.setting(name) reads a credential the deploy passes.
browser/v1 (urn:agentenv:browser/v1): cookies/set and cookies/clear, operations of the same extension main's WebsiteBrowserEnvironment (feat(computer): WebsiteBrowserEnvironment, the base class for website browsers #449) declares with its build, clock and window; sign_in_browser picks a browser that offers cookies/set, so a website browser joins once it serves them (M5.11). The Playwright MCP browser serves them, writing its storage state and, through DevTools, any running context. Answers and errors never repeat a cookie's value.
sign_in_browser step. Calls each live site's session and hands the cookies to the browser; reset_env signs in again. A deploy with a live site is marked reproducible=False with its live_websites, checked against the computer's allowlist.
Website env settings. A website env's settings are its container's environment variables. A secret-looking setting must be a secret:/env: reference, resolved only into the compose file. The gateway's own variables, the role key among them, can't be set.
Replay. python -m agentenv_protocol.website.replay capture URL... writes WARC 1.1; ReplayWebsite serves a capture for any host, with @api routes taking precedence. Example in examples/website/replay/.

Before merging

The browser image's protocol floor rises to 0.1.405, the first release with website.browser, so it needs that release.
website.session is now the decorator, which shadows the module.
No run against a real browser or a real live site yet.

Test

Unit suite: 12,870 passed (one GitHub-clone test from main, which hangs on the local machine on main too, deselected).

🤖 Generated with Claude Code

Confidence Score: 4/5

Fix captured page links before merging; otherwise navigation and asset loads can leave the replay through gateway URLs.

Findings

Links leave the replay ▶
CSS assets are missing ▶
Capture expansion exhausts memory ▶

Fix with agent prompt

### Issue 1
packages/agentenv-protocol/src/agentenv_protocol/website/replay.py:485-486
`_answer` sends captured HTML unchanged. When a page opens at `/website/shop/`, a link to `/style.css` goes to the gateway root, while `https://shop.example/...` goes back to the real site. Navigation and asset loads therefore bypass the replay.

Rewrite same-site links and asset URLs for the address serving the replay. Rewriting `Location` alone does not cover page content.

### Issue 2
packages/agentenv-protocol/src/agentenv_protocol/website/replay.py:327-328
`_next` stops at each stylesheet, so images and fonts referenced by CSS are never saved. A page using `background-image: url('/image.png')` captures its stylesheet but gets a 404 for the image during replay, even with asset capture enabled.

Follow stylesheet imports and CSS asset URLs within the existing host and request limits.

### Issue 3
packages/agentenv-protocol/src/agentenv_protocol/website/replay.py:128-131
`Archive.read` expands the entire gzip file before checking its contents or size. The download limit only bounds the compressed capture. A highly compressed archive can pass that limit and exhaust the website container's memory during `data/add`.

Bound the expanded bytes while decompressing, and apply the same limit to inline and local-file captures.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds live website declarations, browser cookie sign-in, website settings, and WARC capture and replay.

Website environments declare live domains and pass referenced settings to their server.
A task can sign the browser into a live website and restore that session after resets.
Captured websites replay saved pages and let their API routes take precedence.

Diagram

sequenceDiagram
  participant Step as Sign-in step
  participant Site as Live website
  participant Browser as Browser recorder
  participant Chrome as Running browser
  Step->>Site: Request session cookies
  Site-->>Step: Cookies
  Step->>Browser: Clear site cookies
  Step->>Browser: Set session cookies
  Browser->>Browser: Save storage state
  Browser->>Chrome: Update open contexts
  Step->>Step: Save record without cookie values

_{Reviews (4) · Last reviewed commit: "Merge wloo/website-authoring-cli into wl..." · Reviewed by Greptile}

@website

…-4683) @website(domains=[...], live=True, allow_hosts=[...]) declares a real site: its server serves no pages, no API and no built-in session or /go route, only health, the data plane and the card, which advertises live and allow_hosts. @session returns the cookies a browser adds to start signed in; website/v1's session checks them, gives a cookie without a domain the site's first, and answers no_session (501) for a live site without one. self.setting(name) reads a credential its deploy passes. agentenv_protocol.website.browser defines urn:agentenv:browser/v1 with cookies/set and cookies/clear, the Cookie shape, and redacted() for what a record keeps of one; cookie errors name an index and field, never a value. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…FD-4683) python -m agentenv_protocol.website.replay capture URL... --out capture.warc.gz writes WARC 1.1 with the standard library and httpx: the start URLs, same-site links up to --depth, each page's assets, and redirects as answered. ReplayWebsite serves a capture for any host, matched by method, path and normalised query, 404 for a miss; @api routes win over the capture, the default store holds their state for the data plane, and data/add of a .warc(.gz) file part serves that capture until data/reset. The example replays a captured docs site with a synthetic sign-up. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…/set and cookies/clear (FD-4683) The recorder's card, the browser's child env, now serves browser/v1. Playwright MCP launches its browser at the agent's first tool call and starts each context from --storage-state, so cookies/set writes the cookies to that file (0600, merged by name, domain and path) and, through DevTools, into every context of a browser already running; cookies/clear removes those of some domains or all. Answers and errors never repeat a cookie's value. The recorder's protocol floor rises to 0.1.405, the first release with agentenv_protocol.website.browser. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…reproducible, website env settings (FD-4683) deploy_env marks a deploy with a live website child: context.metadata reproducible=False and live_websites (each site's domains and allow_hosts, checked against the run's computer allowlists, which can't change once a sandbox runs, so a host one leaves out is recorded and warned about), and the deploy record live. The sign_in_browser step calls each live site's website/v1 session and passes the cookies straight to the browser child's browser/v1 cookies/set; the context, logs and errors keep cookies by name, domain, path and expiry only. reset_env signs in again after a reset. A website env's settings are its container's environment variables: a secret is stored as a secret:/env: reference and resolved only into the compose file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ation by keyword (FD-4683) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ic site's card is its spec's; tests follow the session cookie and invoke's headers (FD-4683) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

…s sign_in_browser sets (FD-4683) main's WebsiteBrowserEnvironment (#449) declares urn:agentenv:browser/v1 with its build, clock and window; this branch's cookies/set and cookies/clear are operations of the same extension. A test keeps the two constants one URI. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

# Conflicts: # src/agent_env/env/envs/website.py # src/agent_env/providers/env_providers/env_gateway_provider.py

…s container, whatever its value holds (FD-4683) Each setting is written as one double-quoted YAML scalar, its quotes, backslashes and non-printable characters escaped and each $ doubled, so a newline can't add service keys like privileged or volumes and compose doesn't interpolate it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…and models it serves now (FD-4683) Whether /api/session and /go answer from the capture is decided per request against the archive being served, so a capture loaded or reset through the data plane is matched as the declared one. data/add and data/reset hand what isn't a capture to the base hooks, so a models replay loads and resets its database. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…t its hosts too (FD-4683) mark_live makes the network check of every live site recorded so far again on each deploy, with the env being deployed among the run's computers, so a website-first run records the allowlist that leaves a site's hosts out instead of unrestricted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

… by the site's own path (FD-4683) The gateway's X-Forwarded-Prefix leads the scope's path and raw_path, so the replay searched the capture by /website/<name>/... and answered 404. The replay and the captured built-in routes now match the path without the root_path, keeping its escaped segments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

compose_env_entry writes a setting as a quoted "NAME=VALUE" entry, which the secret-key mask didn't match, so a resolved token reached the deploy log. Every quoted setting line is now masked whatever its name, since a secret: or env: reference may resolve to a credential under any name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ugin's deployment sends (FD-4683) data/add of a capture took only its bytes or a file:// URI, so a load through a plugin's provider, which sends each file as a signed URL, failed after its data/reset. An https capture is now fetched, up to MAX_CAPTURE_URL_BYTES, and an error names its status, never the signed URL. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…s under it (FD-4683) A redirect to a captured host, and a root-relative one, are led by the request's root_path, so a replay opened at /website/<name>/ redirects to /website/<name>/new rather than the gateway's /new. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BrowserCookies holds a lock across each set or clear, its file write and DevTools calls together, so concurrent clears for different sites, or a set during a clear, never restore or erase what the other changed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

greptile-apps · 2026-11-03T02:52:26Z

+    def _answer(self, entry: Entry, head: bool, root: str = "") -> Response:
+        response = Response(b"" if head else entry.body, status_code=entry.status)


Links leave the replay

_answer sends captured HTML unchanged. When a page opens at /website/shop/, a link to /style.css goes to the gateway root, while https://shop.example/... goes back to the real site. Navigation and asset loads therefore bypass the replay.

Rewrite same-site links and asset URLs for the address serving the replay. Rewriting Location alone does not cover page content.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/agentenv-protocol/src/agentenv_protocol/website/replay.py Line: 485-486 Comment: **Links leave the replay** `_answer` sends captured HTML unchanged. When a page opens at `/website/shop/`, a link to `/style.css` goes to the gateway root, while `https://shop.example/...` goes back to the real site. Navigation and asset loads therefore bypass the replay. Rewrite same-site links and asset URLs for the address serving the replay. Rewriting `Location` alone does not cover page content. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-11-03T02:52:26Z

+    if is_asset or "html" not in response.headers.get("content-type", ""):
+        return


CSS assets are missing

_next stops at each stylesheet, so images and fonts referenced by CSS are never saved. A page using background-image: url('/image.png') captures its stylesheet but gets a 404 for the image during replay, even with asset capture enabled.

Follow stylesheet imports and CSS asset URLs within the existing host and request limits.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/agentenv-protocol/src/agentenv_protocol/website/replay.py Line: 327-328 Comment: **CSS assets are missing** `_next` stops at each stylesheet, so images and fonts referenced by CSS are never saved. A page using `background-image: url('/image.png')` captures its stylesheet but gets a 404 for the image during replay, even with asset capture enabled. Follow stylesheet imports and CSS asset URLs within the existing host and request limits. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-11-03T02:52:26Z

+        raw = source if isinstance(source, bytes) else Path(source).read_bytes()
+        if raw[:2] == b"\x1f\x8b":
+            raw = gzip.decompress(raw)
+        return cls(_entries(io.BytesIO(raw)), ignore_query)


Capture expansion exhausts memory

Archive.read expands the entire gzip file before checking its contents or size. The download limit only bounds the compressed capture. A highly compressed archive can pass that limit and exhaust the website container's memory during data/add.

Bound the expanded bytes while decompressing, and apply the same limit to inline and local-file captures.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/agentenv-protocol/src/agentenv_protocol/website/replay.py Line: 128-131 Comment: **Capture expansion exhausts memory** `Archive.read` expands the entire gzip file before checking its contents or size. The download limit only bounds the compressed capture. A highly compressed archive can pass that limit and exhaust the website container's memory during `data/add`. Bound the expanded bytes while decompressing, and apply the same limit to inline and local-file captures. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

williamloo-scale and others added 6 commits November 2, 2026 10:48

fix(website): the merged website env docstring, and @website's declar…

30e2ac2

…ation by keyword (FD-4683) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

fix(website): a card says signs_in only for a live site, so a synthet…

9f0d275

…ic site's card is its spec's; tests follow the session cookie and invoke's headers (FD-4683) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

williamloo-scale mentioned this pull request Nov 2, 2026

feat(computer): a computer reaches the run's websites by their names (FD-4683) #448

Open

williamloo-scale and others added 5 commits November 2, 2026 14:22

Merge wloo/website-authoring-cli into wloo/website-live-replay

987704f

Merge wloo/website-authoring-cli into wloo/website-live-replay

ce194e6

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

Merge wloo/website-authoring-cli into wloo/website-live-replay

c555d97

Merge wloo/website-authoring-cli into wloo/website-live-replay

877f06d

williamloo-scale changed the title ~~website: live sites, browser/v1 sign-in, WARC replay (FD-4683)~~ feat(website): live sites, browser/v1 sign-in and WARC replay (FD-4683) Nov 3, 2026

williamloo-scale marked this pull request as ready for review November 3, 2026 00:24

williamloo-scale requested a review from a team as a code owner November 3, 2026 00:24

This was referenced Nov 3, 2026

feat(website): a session cookie per site, /go?sid=, the clock's rate, conformance levels and models= (FD-4683) #444

Open

feat(website): the browser client, env website new and env website dev (FD-4683) #446

Open

greptile-apps Bot reviewed Nov 3, 2026

View reviewed changes

williamloo-scale and others added 6 commits November 2, 2026 16:37

Merge wloo/website-authoring-cli into wloo/website-live-replay

5db1e34

# Conflicts: # src/agent_env/env/envs/website.py # src/agent_env/providers/env_providers/env_gateway_provider.py

Merge wloo/website-authoring-cli into wloo/website-live-replay

1837172

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

Merge wloo/website-authoring-cli into wloo/website-live-replay

4a249ec

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

greptile-apps Bot reviewed Nov 3, 2026

View reviewed changes

Comment thread packages/agentenv-protocol/src/agentenv_protocol/website/replay.py Resolved

Comment thread src/agent_env/providers/env_providers/env_gateway_provider.py Resolved

Comment thread packages/agentenv-protocol/src/agentenv_protocol/website/replay.py Resolved

williamloo-scale and others added 4 commits November 2, 2026 17:52

Merge wloo/website-authoring-cli into wloo/website-live-replay

14571e2

greptile-apps Bot reviewed Nov 3, 2026

View reviewed changes

Comment thread packages/agentenv-protocol/src/agentenv_protocol/website/replay.py Resolved

Comment thread src/agent_env/env/envs/playwright_mcp/recorder.py Resolved

williamloo-scale and others added 3 commits November 2, 2026 18:28

Merge wloo/website-authoring-cli into wloo/website-live-replay

b592968

# Conflicts: # packages/agentenv-protocol/src/agentenv_protocol/website/environment.py

greptile-apps Bot reviewed Nov 3, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(website): live sites, browser/v1 sign-in and WARC replay (FD-4683) - #447

feat(website): live sites, browser/v1 sign-in and WARC replay (FD-4683)#447
williamloo-scale wants to merge 24 commits into
wloo/website-authoring-clifrom
wloo/website-live-replay

williamloo-scale commented Nov 2, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Nov 3, 2026

Uh oh!

greptile-apps Bot Nov 3, 2026

Uh oh!

greptile-apps Bot Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		def _answer(self, entry: Entry, head: bool, root: str = "") -> Response:
		response = Response(b"" if head else entry.body, status_code=entry.status)

		if is_asset or "html" not in response.headers.get("content-type", ""):
		return

Conversation

williamloo-scale commented Nov 2, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before merging

Test

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

williamloo-scale commented Nov 2, 2026 •

edited by greptile-apps Bot

Loading