feat(website): record a website env's traffic as PCAP, served through traffic/v1 (FD-4683)

packages/agentenv-protocol/README.md

@@ -198,6 +198,16 @@ with any other transport are kept beside the SDK's entry. Card URLs are paths, r
 address the card was fetched from. On the client side, `client.mcp_path(card)` returns the
 declared path, or `/mcp` for a card without one.
 
+A composed card, such as a gateway's, nests each backing environment's card under
+`children_environments`, its `url` and endpoints rooted at the route the composer serves it at.
+`client.children(card)` returns each as a `ChildEnv` (its `name`, `url` and `card`, with its
+`extensions`, `operations` and `serves(method)`, and `base_url(address)`, its data plane resolved
+against the composed card's address). Two children may share a name, as a website and an MCP
+server can, so `client.child_at(card, url)` finds one by its url or route; `client.child_named(card,
+name)` raises `AmbiguousChildError` for a shared name, and `client.find_child` returns the first.
+`client.serves(card, method)` says whether a card serves a data-plane method by its declared
+`operations`, None for a card that declares none.
+
 Extensions are invoked from what the card advertises. `client.find_extension_method(card, uri,
 method)` returns one advertised method, whose `endpoint` is the method's own or else the
 extension's, and `client.invoke_extension(base_url, card, uri, params, method=...)` calls it with
@@ -420,6 +430,33 @@ async with ToolSession(mcp_url=mcp_url, role="alice@example.com") as session:
 - **Tools:** `list_tools()` returns the tools the role may use as `EnvironmentTool`, the type a card's
   `capabilities.tools` holds. Each `inputSchema` is the JSON Schema a model API takes as the tool's parameters.
 
+## Website readiness
+
+`agentenv_protocol.website` is the one definition of whether a website env is healthy. A site's card advertises
+`urn:agentenv:website/v1`'s `ready` (`WEBSITE_READY_OP`), a GET answering a `WebsiteReadiness`: `ready`, a boolean,
+and `problems`, the names of what keeps it from being ready (a `WebsiteEnvironment`'s are its `@ready_check` methods
+not yet true). The model is strict about those two (`"yes"` is no `ready`) and ignores fields it doesn't know, so a
+reader keeps reading a newer site. Everything that judges a site reads it through these helpers: agent-env's deploy,
+`DeployedEnv.readiness()`, and the conformance checker's `health`, which fails an answer that isn't a
+`WebsiteReadiness`.
+
+```python
+from agentenv_protocol.website import env_readiness, website_readiness
+
+site = await website_readiness(base_url, card)   # WebsiteReadiness, or None for a card without ready
+env = await env_readiness(gateway_url, env_card)  # WebsiteReadiness of every site of the env
+print(env.ready, env.problems)                    # False ['shop: seeded']
+```
+
+- **One site:** `website_readiness(base_url, card, *, timeout=READY_TIMEOUT_S, headers=None)` asks the site as its card
+  advertises `ready`. It raises httpx's error when the site can't be reached, `ExtensionCallError` for a non-2xx
+  answer and `ValueError` (pydantic's `ValidationError`) for an answer that isn't a `WebsiteReadiness`.
+- **An env:** `env_readiness(base_url, card, ...)` asks, at once, the card itself when it advertises `ready` (a single
+  site's card), and every child under a composed card's `children_environments` that does, its endpoints rooted at
+  `base_url` as a gateway's are. A child's problems are prefixed with its name (`shop: seeded`); one that can't be
+  asked, or answers no `WebsiteReadiness`, is a problem itself. A card or child without `ready`, an MCP server's or an
+  older site's, isn't counted, so an env with no site serving it is ready with no problems.
+
 ## Computer drivers
 
 A computer driver is what runs inside a computer an agent uses through screenshots and input. Write

packages/agentenv-protocol/examples/website/notes/.gitignore

@@ -0,0 +1,2 @@
+web/node_modules/
+web/dist/

packages/agentenv-protocol/examples/website/notes/Dockerfile

@@ -0,0 +1,26 @@
+ARG NOTES=packages/agentenv-protocol/examples/website/notes
+
+FROM mirror.gcr.io/library/node:22-slim@sha256:4b150b48b4116e2a22f2c39cc27025fbf4f153e2078fd3b14d159fdb5c7279da AS web
+ARG NOTES
+WORKDIR /web
+COPY ${NOTES}/web/package.json ${NOTES}/web/package-lock.json ./
+RUN npm ci --no-audit --no-fund
+COPY ${NOTES}/web/ ./
+RUN npm run build
+
+FROM mirror.gcr.io/library/python:3.12-slim@sha256:423ed6ab25b1921a477529254bfeeabf5855151dc2c3141699a1bfc852199fbf
+ARG NOTES
+WORKDIR /app
+RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
+COPY packages/agentenv-protocol /tmp/agentenv-protocol
+RUN pip install --no-cache-dir /tmp/agentenv-protocol "uvicorn>=0.30" && rm -rf /tmp/agentenv-protocol
+COPY ${NOTES}/site.py ${NOTES}/website.toml /app/
+COPY ${NOTES}/data /app/data
+COPY --from=web /web/dist /app/web/dist
+ENV PORT=8000
+EXPOSE 8000
+RUN useradd --create-home --uid 10001 appuser
+USER appuser
+HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
+    CMD python -c "import os, urllib.request; urllib.request.urlopen('http://127.0.0.1:%s/healthz' % os.environ.get('PORT', '8000'), timeout=2)"
+CMD ["python", "site.py"]

packages/agentenv-protocol/examples/website/notes/Dockerfile.dockerignore

@@ -0,0 +1,6 @@
+**/node_modules
+**/dist
+**/__pycache__
+**/.pytest_cache
+**/.venv
+.git

packages/agentenv-protocol/examples/website/notes/README.md

@@ -0,0 +1,109 @@
+# Notes: a reference website/v1 site
+
+Notes is a small notes app built on `agentenv_protocol.website.WebsiteEnvironment`. It is the reference site for the
+`urn:agentenv:website/v1` server contract. Each user keeps their own notes and can create, edit, mark done and delete
+them. The page shows the universe's time.
+
+| File | What it is |
+| --- | --- |
+| `website.toml` | The site's spec: its name, domains, health path, source, seed, collections, personas, roles, routes, state read and clock (see `docs/website-spec.md`). |
+| `site.py` | The server: the `Notes` WebsiteEnvironment, declared from `website.toml`, its `@api` routes, and `serve()` on `PORT` (8000). |
+| `data/seed.json` | The seed data: two users (`ann`, `ben`), their notes, and `current_user`, the acting user. |
+| `web/` | A minimal Vite app (`index.html`, `src/main.js`, `src/style.css`, and `src/site-client.js`, a copy of the SDK's browser client). Its assets (`base: "./"`) and API calls are relative to the page, so it works at `/` and under the gateway's `/website/notes/` alike; opened at a path without its final slash, such as `/website/notes`, the page first reloads itself with it. |
+| `Dockerfile` | The server image: Node builds `web/`, then Python installs this repo's `packages/agentenv-protocol` with uvicorn. It installs `curl`, which the gateway's compose health check runs. |
+| `env.toml` | The website env as a bundle entry, pointing at `website.toml`. |
+
+## What it exercises
+
+- **Site declaration.** `@website(spec="website.toml")`. The spec gives the name, domains, static folder, health path
+  and seed, and the class is refused when its `@api` routes aren't the spec's. The card's website/v1 params are the
+  spec's `to_card_extension()`: the domains, the health path, every route by name, the session, the state read and
+  the clock.
+- **Routes.** `list_users`, `sign_in`, `list_notes` (with an optional `?done=`), `create_note` (201), `get_note`,
+  `edit_note`, `toggle_note` and `delete_note`. Errors are raised as `NotFound`, `BadRequest` and `Unauthorized`.
+- **Roles.** `all_notes` (`GET /api/admin/notes`) is declared with `roles=["admin"]`, and the spec's `[roles]` grants
+  it to `admin`. It answers only a caller that
+  sends `AgentEnv-Role: admin` with `AgentEnv-Role-Key` equal to the container's `WEBSITE_ROLE_KEY`; any other caller
+  gets 403. A deploy with `keyed_roles` gives the site its own role key as `WEBSITE_ROLE_KEY` and its gateway sends it
+  on only beside a role it verified with the same key, so a page cannot claim `admin` through it. Any other deploy sets
+  no key, and its gateway passes on whatever role a browser sends; with no `WEBSITE_ROLE_KEY` set, every caller is
+  `default` and the route is closed.
+- **Users.** `ApiContext.user` is the user in the signed session cookie, `agentenv_session_notes` (the site's own, so
+  it doesn't collide with another site's under the same gateway host; the card names it). With no cookie, it is the acting user (the
+  store's `current_user`), and `GET /api/session` signs that user in. `POST /api/sign-in` switches to another seeded
+  user. A user's notes are invisible to every other user, and another user's note answers 404.
+- **State.** The site writes no data hooks, so it uses the default store. `data/add` loads the seed (as a data part, or
+  as a file part by bytes or `file://`), `data/get` exports it, and `data/reset` empties it. `data/get` is how
+  agentenv reads what a task left the site with.
+- **Time.** Notes are stamped with `self.now()`, which follows the gateway's clock once `clock/v1`'s `sync_time` names
+  it. The frontend reads `X-Universe-Now` from every API response into `universeNow()` and shows it in the header.
+  It uses this helper rather than `clock_patch=True`, so the page works the same against a server written without the
+  SDK. The helper and the page's `api()`, `session()` and `signIn()` come from `src/site-client.js`, a copy of
+  `agentenv_protocol/website/client/site-client.js`, which a test keeps it the same as.
+- **Readiness.** `@ready_check seeded` holds `/healthz` at 503 until the store has users, for example after a
+  `data/reset` and before the next `data/add`.
+
+## Run it
+
+From this folder:
+
+```bash
+cd web && npm ci && npm run build && cd ..
+PYTHONPATH=<repo>/packages/agentenv-protocol/src python site.py
+```
+
+Open http://localhost:8000. To work on the frontend, run `npm run dev` in `web/` while `site.py` runs; Vite proxies
+`/api` to port 8000.
+
+Or run `agent-env env website dev` from this folder: it builds `web/` when it is out of date and rebuilds it as it
+changes, serves `site.py` with reload, syncs it to a local stand-in for the gateway's clock, loads `data/seed.json`
+through the data plane after every reload, and runs the conformance check once.
+
+## Test it
+
+```bash
+python -m pytest packages/agentenv-protocol/tests/test_website_notes_example.py
+```
+
+The test imports `site.py`, runs the conformance checker, and checks sign-in, per-user notes, the admin-only route,
+clock-synced timestamps, and the data plane's reset and seed load. It serves a stand-in page, so it needs no Vite
+build.
+
+## Build and verify the image
+
+The build context is the repo root, because the image installs the protocol package from this repo's source:
+
+```bash
+docker build -f packages/agentenv-protocol/examples/website/notes/Dockerfile -t notes-site .
+docker run -d --name notes-site -p 18080:8000 notes-site
+python -m agentenv_protocol.website.conformance http://localhost:18080 --clock-host host.docker.internal --strict
+curl -s http://localhost:18080/api/notes
+```
+
+Use `--clock-host` so the container can reach the checker's clock, and add
+`--spec packages/agentenv-protocol/examples/website/notes/website.toml` to check the server against its spec too.
+`--strict` fails on a recommended check's warning too (the card, `X-Universe-Now`, the clock, forwarded headers,
+cookies); without it only the contract's required checks, and the spec's when given, fail the run, and Notes passes
+every check either way. The clock check leaves the server following a
+clock that is gone once the check ends: later timestamps start from 2031 until the server is synced again, so restart
+the container before using it.
+
+## Register it
+
+```bash
+agent-env env website put --id notes-site \
+  --server-dockerfile packages/agentenv-protocol/examples/website/notes/Dockerfile \
+  --server-docker-context .
+```
+
+Run it from the repo root. `environment_name` comes from the card, which the spec names `notes`. The stored env
+has a server image and no frontend image. Pass `--skip-validation` to register without deploying.
+
+To record what the site sends and receives, deploy it with `--capture-traffic` (or `"capture_traffic": true` on a
+task's `deploy_env` step): tcpdump runs in the server container's network namespace and the gateway serves its pcap
+files as `traffic/v1`, which a `collect_traffic_capture` step with the env's `env_id` stores as file artifacts.
+
+`env.toml` describes the same env for a bundle. Its keys are `type`, `spec` and `backend_image`; the env takes its
+`environment_name`, health path and domains from the spec. A bundle builds `backend_image` with this
+folder as the build context, but this Dockerfile needs the repo root. Until the protocol package is installed from a
+published release instead, register the site with `put` as above.

packages/agentenv-protocol/examples/website/notes/data/seed.json

@@ -0,0 +1,15 @@
+{
+  "current_user": "ann",
+  "users": [
+    {"id": "ann", "name": "Ann Lee"},
+    {"id": "ben", "name": "Ben Ortiz"}
+  ],
+  "notes": [
+    {"id": 1, "owner": "ann", "title": "Buy milk", "body": "Oat, two cartons.", "done": false,
+     "created_at": "2026-01-05T09:00:00.000Z", "updated_at": "2026-01-05T09:00:00.000Z"},
+    {"id": 2, "owner": "ann", "title": "Book dentist", "body": "", "done": true,
+     "created_at": "2026-01-04T15:30:00.000Z", "updated_at": "2026-01-06T08:10:00.000Z"},
+    {"id": 3, "owner": "ben", "title": "Draft the quarterly report", "body": "Due Friday.", "done": false,
+     "created_at": "2026-01-03T11:00:00.000Z", "updated_at": "2026-01-03T11:00:00.000Z"}
+  ]
+}

packages/agentenv-protocol/examples/website/notes/env.toml

@@ -0,0 +1,3 @@
+type = "website"
+spec = "website.toml"
+backend_image = { dockerfile = "Dockerfile" }

packages/agentenv-protocol/examples/website/notes/site.py

@@ -0,0 +1,129 @@
+"""Notes: the reference site of ``urn:agentenv:website/v1``, a small notes app on WebsiteEnvironment. Each user keeps
+notes they create, edit, mark done and delete; a browser with no session is signed in as the acting user, the seed's
+``current_user``, and can sign in as another seeded user. The site keeps its state in the default store, which the data
+plane loads, exports and resets, and stamps each note with the universe's time. ``admin`` callers, as the gateway's
+``AgentEnv-Role`` names them, can list every user's notes.
+
+Run it with ``python site.py`` after building ``web/`` into ``web/dist``; it serves on ``PORT`` (8000)."""
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic import BaseModel
+
+from agentenv_protocol.website import (
+    ApiContext,
+    BadRequest,
+    NotFound,
+    Unauthorized,
+    WebsiteEnvironment,
+    api,
+    ready_check,
+    website,
+)
+from agentenv_protocol.website.clock import iso
+
+NOTES = "notes"
+USERS = "users"
+
+
+class NewNote(BaseModel):
+    title: str
+    body: str = ""
+
+
+class NoteEdit(BaseModel):
+    title: Optional[str] = None
+    body: Optional[str] = None
+    done: Optional[bool] = None
+
+
+class SignIn(BaseModel):
+    user: str
+
+
+@website(spec="website.toml")
+class Notes(WebsiteEnvironment):
+    @ready_check
+    def seeded(self) -> bool:
+        """The site is usable once its store knows who may sign in."""
+        return bool(self.store[USERS])
+
+    @api.get("/api/users", name="list_users")
+    def list_users(self) -> list:
+        """The users a browser can sign in as."""
+        return self.store[USERS]
+
+    @api.post("/api/sign-in", name="sign_in")
+    def sign_in(self, body: SignIn, ctx: ApiContext) -> dict:
+        """Sign the browser in as one of the site's users."""
+        if self.store.get(USERS, body.user) is None:
+            raise NotFound(f"no user {body.user!r}")
+        ctx.sign_in(body.user)
+        return {"user": ctx.user}
+
+    @api.get("/api/notes", name="list_notes")
+    def list_notes(self, ctx: ApiContext, done: Optional[bool] = None) -> list:
+        """The signed-in user's notes, newest first; with ``done``, only those done or not."""
+        match = {"owner": self._user(ctx)} if done is None else {"owner": self._user(ctx), "done": done}
+        return sorted(self.store.find(NOTES, **match), key=lambda note: note["created_at"], reverse=True)
+
+    @api.post("/api/notes", name="create_note", status=201)
+    def create_note(self, note: NewNote, ctx: ApiContext) -> dict:
+        """Add a note for the signed-in user, not done, created and updated now."""
+        now = iso(self.now())
+        return self.store.insert(NOTES, {"owner": self._user(ctx), "title": _title(note.title), "body": note.body,
+                                         "done": False, "created_at": now, "updated_at": now})
+
+    @api.get("/api/notes/{note_id:int}", name="get_note")
+    def get_note(self, note_id: int, ctx: ApiContext) -> dict:
+        return self._note(note_id, ctx)
+
+    @api.patch("/api/notes/{note_id:int}", name="edit_note")
+    def edit_note(self, note_id: int, edit: NoteEdit, ctx: ApiContext) -> dict:
+        """Change a note's title, body or done, and its ``updated_at``."""
+        note = self._note(note_id, ctx)
+        fields = edit.model_dump(exclude_none=True)
+        if "title" in fields:
+            fields["title"] = _title(fields["title"])
+        note.update(fields, updated_at=iso(self.now()))
+        return note
+
+    @api.post("/api/notes/{note_id:int}/toggle", name="toggle_note")
+    def toggle_note(self, note_id: int, ctx: ApiContext) -> dict:
+        """Mark a note done, or not done when it was."""
+        note = self._note(note_id, ctx)
+        note.update(done=not note["done"], updated_at=iso(self.now()))
+        return note
+
+    @api.delete("/api/notes/{note_id:int}", name="delete_note")
+    def delete_note(self, note_id: int, ctx: ApiContext) -> dict:
+        self.store.delete(NOTES, self._note(note_id, ctx)["id"])
+        return {"deleted": note_id}
+
+    @api.get("/api/admin/notes", name="all_notes", roles=["admin"])
+    def all_notes(self) -> list:
+        """Every user's notes, for an ``admin`` caller only."""
+        return self.store[NOTES]
+
+    def _user(self, ctx: ApiContext) -> str:
+        if ctx.user is None:
+            raise Unauthorized("sign in first")
+        return ctx.user
+
+    def _note(self, note_id: int, ctx: ApiContext) -> dict:
+        """The signed-in user's note ``note_id``; another user's is as missing as one that doesn't exist."""
+        note = self.store.get(NOTES, note_id)
+        if note is None or note["owner"] != self._user(ctx):
+            raise NotFound(f"no note {note_id}")
+        return note
+
+
+def _title(title: str) -> str:
+    if not title.strip():
+        raise BadRequest("a note needs a title", code="empty_title")
+    return title.strip()
+
+
+if __name__ == "__main__":
+    Notes().serve()

packages/agentenv-protocol/examples/website/notes/web/index.html

@@ -0,0 +1,37 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Notes</title>
+    <script>
+      if (!location.pathname.endsWith("/")) { const url = new URL(location.href); url.pathname += "/"; location.replace(url.href); }
+    </script>
+  </head>
+  <body>
+    <header class="bar">
+      <h1>Notes</h1>
+      <div class="who">
+        <label>Signed in as
+          <select id="user" aria-label="Signed-in user"></select>
+        </label>
+        <time id="universe-now" title="The universe's time, from X-Universe-Now"></time>
+      </div>
+    </header>
+    <main>
+      <form id="new-note" class="card">
+        <input id="new-title" name="title" placeholder="New note title" autocomplete="off" required />
+        <textarea id="new-body" name="body" placeholder="Details (optional)" rows="2"></textarea>
+        <button type="submit">Add note</button>
+      </form>
+      <nav class="filters" aria-label="Filter notes">
+        <button data-filter="all" class="active">All</button>
+        <button data-filter="open">Open</button>
+        <button data-filter="done">Done</button>
+      </nav>
+      <p id="error" role="alert" hidden></p>
+      <ul id="notes"></ul>
+    </main>
+    <script type="module" src="/src/main.js"></script>
+  </body>
+</html>