Skip to content

feat(website)!: WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) - #428

Open
williamloo-scale wants to merge 8 commits into
wloo/website-harness-gapsfrom
wloo/website-environment-sdk
Open

williamloo-scale wants to merge 8 commits into
wloo/website-harness-gapsfrom
wloo/website-environment-sdk

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #427.

agentenv_protocol.website: the easy path to a synthetic website that follows website/v1 (urn:agentenv:website/v1).

@environment_card(name="notes")
@website(domains=["notes.test"], static="web/dist", health="/healthz")
class Notes(WebsiteEnvironment):
    @get_data
    def export(self) -> list:
        return [DataPart(data={"notes": self.db.all_notes()})]

    @api.get("/api/notes")
    def list_notes(self, ctx: ApiContext) -> list:
        return self.db.notes(owner=ctx.user)
  • @api.get/post/put/patch/delete(path, name=, roles=, status=): path, query and body are bound from type hints, errors are exceptions mapped to statuses, and an OpenAPI 3.1 document is generated.
  • ApiContext: the signed-in user (else the acting user, so the agent starts signed in), the caller's role, and the public URL from X-Forwarded-Host/-Proto.
  • Roles: the gateway forwards whatever AgentEnv-Role a browser sends, so a site believes a non-default role only when the request also carries AgentEnv-Role-Key equal to the site's WEBSITE_ROLE_KEY (compared in constant time). Every other caller, and every caller of a site with no key set, is default, so roles= routes are closed until a key is set.
  • Serves the built pages with an index.html fallback, the health path with @ready_check, the data plane and a card declaring website/v1 with its routes and openapi/session operations. /agentenv… routes are matched before the site's own.
  • self.now() reads the universe clock (clock/v1 sync_time), read once per request from the gateway as MCP servers read it, so a frozen clock is exactly frozen. An unarmed clock is wall time, and an unreachable one answers 503, except on the data plane and the card. An empty env_get_time_url goes back to wall time. Every response carries X-Universe-Now; clock_patch=True patches the page's Date. sync_time also takes main's optional timezone (feat(sync_env_clock): an optional timezone reads virtual_time in the zone and tells every synced server (FD-5003) #442), which the site reads as self.timezone; now() stays UTC.
  • A site keeps its own data, usually in its schema of the env's database (database_url), and serves it through the data plane's hooks (@reset_data, @add_data, @get_data) as any AgentEnvEnvironment does. The SDK has no store of its own. A site's data is read and written only through the data plane, and the SDK serves no /go and no website/v1 state op.
  • Sessions: the signed agentenv_session cookie lasts its Max-Age (7 days), never an absolute Expires, so a browser on another clock judges it the same. A deploy gives each site its own WEBSITE_SESSION_SECRET, so sessions outlive a restart or a snapshot reload; a site run without one (local serve()) draws its own. website/v1's session {user?} answers a storage state, {cookies, local_storage}, each cookie with max_age.
  • Plain def @api handlers run in a worker thread, as FastAPI's do. A route whose type hints can't be resolved fails the app's build, and session cookies go on successful answers only. The static fallback never serves a dot-named path.
  • website_state_verifier, which feat(website): a website env is one server image, checked at its health path (FD-4683) #390 merged and which read a site's /go?sid=, is removed. Grading a website from data/get against the task's initial state moves to agentenvhub-sdk's cua_gym_verifier, a separate PR in scaleapi2.
  • python -m agentenv_protocol.website.conformance URL checks any server over HTTP: card, any host name, health, universe time, data plane, clock, forwarded headers, cookies. It sends nothing to a server other than the one under check, and its clock check syncs the server back to wall time at the end. A cookie with an absolute Expires and no Max-Age warns without failing the run.

Breaking

Known gaps

Each is fixed later in the stack, as noted.

Test

packages/agentenv-protocol/tests: 1,115 passed after the session edits (f1bae23); lint clean. Unit suite on CI. Before them: unit suite and protocol tests 12,659 passed.
packages/agentenv-protocol/tests: 946 passed; the website tests (59, with the Notes ones) pass after the role-key fix. Deployed locally behind the gateway, a spoofed AgentEnv-Role: admin now gets 403 on both /website/ and /svc/ (200 before).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

Update the dependent browser cookie reader before merging the new session shape.

Fix All in CursorFindings

  1. P1 Browser rejects session cookies ▶
Fix with agent prompt
### Issue 1
packages/agentenv-protocol/src/agentenv_protocol/website/session.py:51-52
`SessionSigner.cookie()` now returns `max_age`, but the browser reader in PR #447 uses `Cookie` with `extra="forbid"` and accepts only `expires`. Sending a cookie minted by `website/v1` to that PR's `cookies/set` fails with `invalid_params` instead of signing the browser in. The same reader remains unchanged through PR #469.

Update the browser cookie model and its Playwright/DevTools conversions alongside this change so they accept and apply the relative lifetime.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds WebsiteEnvironment, typed website routes, sessions, universe time, shared readiness checks, and an HTTP conformance checker.

  • Python sites can serve pages and APIs under website/v1.
  • Website requests can follow the gateway's virtual clock.
  • A command checks whether any server follows website/v1.
  • Deploys and environment records check each site's readiness answer.
  • Gateway website routes reserve the paths a site's card advertises.
  • Website and MCP children with the same name resolve to the right child.
  • Website state checks no longer use the website_state_verifier task step.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A["website/v1 session"] --> B["Cookie with max_age"]
  B --> C["browser/v1 cookies/set"]
  C --> D["Cookie model accepts expires only"]
  D --> E["invalid_params"]
Loading

Reviews (16) · Last reviewed commit: "fix(website): sessions last a Max-Age, s..." · Reviewed by Greptile

williamloo-scale added a commit that referenced this pull request Nov 3, 2026
Brings in #428's review fixes beside this branch's: the clock's sync
reads under the refresh lock and keeps the answered rate; conformance
checks X-Universe-Now last, on every answer, at the recommended level,
with the reserved-path probes counted; the handwritten test server
stamps its 404s and 405s like every other answer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tract, and a conformance check tests any server (FD-4683)

This PR's change, rebased as one commit onto #427.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/agent_env/env/env.py Resolved
@williamloo-scale
williamloo-scale force-pushed the wloo/website-harness-gaps branch from 7fc14cd to b18049d Compare November 3, 2026 21:50
@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 3579c2a to 76145d2 Compare November 3, 2026 21:50
Comment thread src/agent_env/env/gateway/gateway.py Resolved
williamloo-scale and others added 2 commits November 3, 2026 14:20
… keep older sites and shared names working (FD-4683)

- The gateway answers 503 on a site's route while its card can't be read, rather than forwarding with nothing reserved.
- A plugin's MultiEnv record looks each child up by its kind, so a website and an MCP server may share a name.
- The state verifier reads a site its definition names but its card doesn't list at /svc/<name>/go, deprecated.
- The clock script is put in streamed HTML holding only what comes before <head>.
- Conformance fails a website/v1 ready that answers an HTTP error, whatever its body says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ver HTTP client the test client uses (FD-4683)

After a sign-out the client's jar keeps the site's signed-out cookie under testserver.local, and setting the
forged one adds a second cookie of the same name. httpx2, which Starlette's TestClient uses when it is
installed (as on CodeBuild), sends the forged one first, so the site reads the valid signed-out cookie and
the test fails; httpx sends them the other way round. Clearing the jar first sends the forged cookie alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator

This is a huge red flag: it leaks CUA-Gym into agentenv-framework. /go?sid= is CUA-Gym's verifier endpoint, and this PR builds it into the SDK and the website/v1 contract rather than keeping it outside the framework:

  • The SDK: agentenv_protocol.website defines and exports STATE_PATH = "/go" (environment.py:51, __init__.py:15). Every store-backed SDK site serves its state there (environment.py:173) and advertises it as website/v1's params.state.path (environment.py:198).
  • Conformance: the checker reads state at params.state.path, else at /go (conformance.py:200-208), so it judges any server by CUA-Gym's convention.
  • Verifier and gateway: website_state_verifier falls back to params.state.path, else /go?sid= (website_state_verifier.py:37, :279-283). The gateway's guard is built around params.state.path (gateway.py:1556). The /go fallback predates this PR, but this PR extends it.

website/v1's state op should be the only state contract: an SDK site answers its state only through it. Please drop /go and params.state.path from the SDK, the conformance check and website/v1. If CUA-Gym sites still need to be read at /go, that compatibility belongs in a CUA-Gym adapter outside the framework, not in agentenv_protocol or the core verifier.

@earakely-scale

Copy link
Copy Markdown
Collaborator

Following up on my /go comment: please remove CUA-Gym's verifier model from the framework altogether. That model is a site serving {initial_state, current_state, state_diff} per sid, and a verifier reading it. This goes further than my earlier note, so drop the website/v1 state op too, since it's the same model under another name. agentenv already has the primitives for this:

What the verifier needs agentenv primitive
current state the data plane's data/get, which SDK sites already serve from the same store (_store_get)
initial state the universe the task loaded with data/add, or a snapshot_env taken before the agent starts
diff computed once, in the verifier, from those two, the same for every env instead of each site writing its own (that also removes the store-id diff problem)
sid not needed: each deploy gets its own site, and the SDK already ignores it
keeping state from the agent the data plane is framework-only and already refused on the browsing route

Concretely:

  • SDK: remove the /go route, STATE_PATH, params.state.path, the state op, and the store's site-side state and diff (Collections.state() / diff()). A site exposes its data through data/get only.
  • Verifier: grade by reading data/get against the initial state, keeping the check model (path, op, value, reduce, weight). It then works for any env, MCP servers included, not only websites.
  • Conformance and gateway: check data/get rather than /go or state. Drop the params.state.path reservation.
  • Main: website_state_verifier as feat(website): a website env is one server image, checked at its health path (FD-4683) #390 merged it reads /go?sid=. Replace it with the above rather than extending it. CUA-Gym images that serve only /go get an adapter outside the framework.

@williamloo-scale

williamloo-scale commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Agreed, and done in this stack rather than deferred. /go?sid= and the per-site {initial_state, current_state, state_diff} model were CUA-Gym's, and they're gone from the framework:

Each PR removes what it had added, so the stack never introduces any of this. Grading a website now belongs outside the framework: cua_gym_verifier in agentenvhub-sdk, in a separate scaleapi2 PR. It reads data/get, diffs it against the task's initial state (the universe it loaded or a snapshot_env), and keeps the check model: path, op, value, reduce and weight.

@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 1ffec4c to 05da818 Compare November 4, 2026 03:37
…op /go, the state op and website_state_verifier (FD-4683)

- The SDK serves no `/go`, no `params.state.path` and no `website/v1` `state` op; `STATE_PATH`, `WebsiteState`,
  `website_state`, `website_sites` and `website/state.py` are gone, and the default store keeps no initial state or diff.
- Conformance runs the data plane's round trip on every server, with no skip for a state that shows changes.
- The gateway reserves the card's `url` and every advertised endpoint, no longer a state path.
- `website_state_verifier` is removed from the framework; grading against a site's state belongs outside it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 05da818 to 33ad3b8 Compare November 4, 2026 03:56
… items in OpenAPI, and find the clock's sync_time by name (FD-4683)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale

Copy link
Copy Markdown
Contributor Author

@greptile-apps all three remaining review comments are fixed in d4efdbb, each with a test that fails without its fix:

  • Partial HTML downloads change (environment.py): _Universe no longer injects the clock script into a partial answer. A 206, or any answer with Content-Range, passes through unchanged, so its bytes still match the promised range.
  • Query lists lose item types (routes.py): a list[T] query parameter is documented as {"type": "array", "items": <T's schema>}, and it still binds as repeated values.
  • Working clocks can fail (conformance.py): _clock() looks up sync_time by name instead of taking the first operation the card lists.

The full unit suite passes on this commit (12,908) and lint is clean. All three threads are replied to and resolved.

@greptile-apps

greptile-apps Bot commented Nov 4, 2026

Copy link
Copy Markdown

Confirmed: all three findings are fixed in d4efdbbf and have regression coverage:

  • Partial HTML downloads: _Universe skips clock-script injection for status 206 and responses with Content-Range; the test verifies a range response remains unchanged.
  • Query-list item types: OpenAPI now includes the item schema for list query parameters, while binding still uses repeated query values; the test checks list[int] documentation and binding.
  • Clock operation selection: the checker finds sync_time by name; the test passes when another clock operation is listed first.

These comments can remain resolved. I verified the code and tests in the commit; I did not independently run the suite.

williamloo-scale added a commit that referenced this pull request Nov 4, 2026
…ses the conformance check (FD-4683)

This PR's change, rebased as one commit onto #428.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
earakely-scale and others added 2 commits November 4, 2026 09:43
… per request, and conformance stays on the server it checks (FD-4683)

- The SDK no longer keeps a store: `self.store`, `Collections`, `@website(data=...)` and the default data hooks are
  gone. A site serves its data through `@reset_data`, `@add_data` and `@get_data` as any AgentEnvEnvironment does,
  usually over its `database_url`; `acting_user()` is nobody unless a site says else.
- The clock is read once per request, as MCP servers read it: no extrapolation, rate inference or refresh lock, so a
  frozen gateway clock is exactly frozen and a slow one holds up only its own request. An unarmed clock (404) is wall
  time; an unreachable one answers 503, but the data plane and the card answer without the header. `sync_time` with
  an empty URL goes back to wall time.
- Conformance sends nothing off the server under check: a card's path or URL naming another scheme, host or port
  fails that check. The clock check ends by syncing the server back to wall time.
- Plain `def` `@api` handlers run in a worker thread; a route whose type hints can't be resolved fails the app's build;
  session cookies go on successful answers only.
- The static fallback serves no dot-named path and treats a path that can't be a file's as not found; a time parses
  with any fraction of a second on Python 3.10; the clock script goes after `<head>` (not `<header>`), else after the
  doctype and `<html>` tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title feat(website): WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) feat(website)!: WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) Nov 4, 2026
…session answers a storage state (FD-4932)

- The session cookie carries Max-Age, never an absolute Expires, so a browser on another clock judges it the same.
- website/v1's session answers {cookies, local_storage}, each cookie with max_age, in browser/v1's shape.
- The gateway provider gives each site its own WEBSITE_SESSION_SECRET at deploy, so its sessions outlive a restart
  or a snapshot reload, which keeps the compose file it recreates from.
- Conformance warns, without failing the run, on a cookie with an absolute Expires and no Max-Age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +51 to +52
return {"name": SESSION_COOKIE, "value": self.encode(user, ttl_s), "path": "/", "max_age": ttl_s,
"httpOnly": True, "sameSite": "Lax"}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Browser rejects session cookies

SessionSigner.cookie() now returns max_age, but the browser reader in PR #447 uses Cookie with extra="forbid" and accepts only expires. Sending a cookie minted by website/v1 to that PR's cookies/set fails with invalid_params instead of signing the browser in. The same reader remains unchanged through PR #469.

Update the browser cookie model and its Playwright/DevTools conversions alongside this change so they accept and apply the relative lifetime.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/agentenv-protocol/src/agentenv_protocol/website/session.py
Line: 51-52

Comment:
**Browser rejects session cookies**

`SessionSigner.cookie()` now returns `max_age`, but the browser reader in PR #447 uses `Cookie` with `extra="forbid"` and accepts only `expires`. Sending a cookie minted by `website/v1` to that PR's `cookies/set` fails with `invalid_params` instead of signing the browser in. The same reader remains unchanged through PR #469.

Update the browser cookie model and its Playwright/DevTools conversions alongside this change so they accept and apply the relative lifetime.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants