Seal per-scenario evidence so judged sub-goals can be decided off-sandbox

[KarthikAvinashFI]

What this does

Seals a per-scenario evidence bundle in the hosted guest, so a sub-goal that no code can settle can actually be decided later.

Today a judged sub-goal is never judged in a hosted run. scenario_source.py binds it to _judged_placeholder_check, a function that returns None, and it leaves the sandbox as held=None. Nothing downstream can decide it either, because the two things a judge needs never leave the guest: the final state of the world, which dies with the sandbox, and the claim itself.

Three changes, all guest-side.

A world snapshot on the receipt path. _capture_evidence runs at the one moment the graded world still exists, against the same read-only handle the checks use. It records every tool call with its arguments, results and refusal flag, plus real rows from the world afterwards.

The claim text, restored. folder.py writes no checks/<name>.py for a judged sub-goal, so the reader could only tell that a sub-goal was judged, never what it was meant to decide, and substituted a fixed marker. The text is in the bundle's own sub_goals.json and was simply not read. _with_claims restores it at suite level, leaving _load_one's contract untouched.

Shipped as an artifact, not inline. The wire receipt is digest-signed and a world snapshot does not belong inside it. OutboundAdapter.receipt uploads the bundle as an evidence artifact keyed by scenario_key, before pushing the receipt, matching the ordering every other referenced artifact already follows. The wire schema, its digest and the platform serializer are all unchanged.

Bounding

A seeded world is unbounded, so the snapshot is capped at 40 rows per table and 128 KB total. Every drop is counted rather than silent: each table reports rows_shown alongside rows_total, and tables that did not fit are listed in omitted_tables. A judge told "40 of 340 rows" can reason about the gap; one shown 40 rows and told nothing cannot.

Budget is spent where the run acted. Tables are ranked by how strongly the calls point at them, matching per underscore-separated word in both directions so book_ride counts as evidence about bookings. At equal relevance the smaller table wins, so one large log table cannot crowd out the table the scenario actually wrote to.

evidence is a reserved artifact kind, alongside transcript and tool_trace, for the same reason: if the budget drops it, judged sub-goals silently go unjudged again, which is the bug this is fixing. It is forbidden at metadata-only.

A world that cannot be read is not a failed scenario. The capture falls back to calls and claims alone and records why, with the DSN redacted through the existing _sanitize_cause.

Worth knowing

Judged sub-goals never had live world access, even in the local lane. run/__init__.py:199 builds the world evidence a judge receives as ", ".join(f"{name}: {len(rows)} rows"), table names and row counts and nothing else. So this is not a capability being moved off the world; it is a strict improvement on what any judge in either lane has ever been given.

Merge order

This ships the producer. The consumer side is a separate PR on future-agi, and that one must merge first: the platform validates artifact kind against a fixed set and will reject evidence until it is added there. A rejected upload is logged and non-fatal, so a run out of order degrades to today's behaviour rather than failing, but nothing is judged.

Testing

tests/harness/: 905 passed, 1 skipped, the whole harness suite.

Six new tests. Four on the capture itself: argument and claim carriage, truncation counting, relevance ranking against a large irrelevant table, and the unreadable-world path including DSN redaction. Two on the emitter seam: that evidence is sealed as an artifact and kept off the wire receipt, and that a receipt without evidence uploads nothing.

Not yet run end to end against a hosted job. That needs the platform side merged and the local stack restarted to pick it up.