Generalise the harness and close the defects that made runs look verified

[KarthikAvinashFI]

Draft, and an experiment. It is open for reading and for argument about the approach, not as a
merge candidate. Some of what it changes is a bet rather than a fix, and the sections below say
which is which.

1. Why

The harness could only test agents it had already been fitted to. Two separate problems sat behind
that, and this branch takes both.

It enumerated the agents it knew. The run stage branched on a fixed set of connectors, and the
build stage worked through two tools (write_env_file, run_env_command) that only ran container
commands. An agent reached some other way, or needing anything a docker command could not do,
could not be tested without a code change here first. Supporting a new kind of agent meant a
release of this repo.

Several stages reported a state they had not established. The worst of them: probe recorded
every tool that lives in the submitted agent's own runtime as passing, with the detail "executes
inside the submitted agent runtime", without executing it. In the hosted lane that runtime does not
exist yet when probe runs, so a world where nothing had been exercised scored perfectly and went
on to grade an agent against sub-goals it could not measure.

The concrete case this is for: an agent repository submitted by someone else, that nobody here has
read, which does not look like the two we built against.

2. What changed

The build stage gets a shell

src/fi/alk/harness/build.py:297

Before	After
builtins=("AskUserQuestion",)	("AskUserQuestion", "Read", "Glob", "Grep", "Write", "Edit", "Bash")
Everything the stage could do went through write_env_file and run_env_command	It reads the submitted repository, writes what the environment needs, runs it, reads the error and fixes it

The two world tools it replaces are removed from world/tools.py, and world/workspace.py, which
backed them, is deleted.

The deny-by-default permission model is not removed. UNWANTED, gate_hooks and
permission_gate all stay, and every other stage is bounded exactly as before. What changed is one
line: the hidden-tool list is now filtered against what the stage was granted, so a stage that asks
for Bash keeps it instead of being silently outranked by a blanket denial.

File	Change	Preserved
backends/claude.py:188	disallowed_tools becomes [name for name in UNWANTED if name not in set(allowed)]	The hook is still the enforcement, the callback still the backstop and the question route; ungated stages still bypass both
config.py:166	UNWANTED unchanged in content; its comment now states that a granted tool is kept	gate_hooks and permission_gate unchanged
config.py:127	read_only_session gains a docstring arguing why the stage that reads the customer's source keeps no shell, and an extra_builtins escape per source	Its tool set is unchanged: Read, Glob, Grep, AskUserQuestion

The reasoning for keeping the gate, and for the grant meaning the same thing hosted and locally, is
in config.py:215. A sandbox bounds the hosted lane, but the same stages run in-process on an
operator's machine where nothing bounds a shell, and a grant that means two different things stops
the local run being a rehearsal of the hosted one.

Transports become a declaration

New file src/fi/alk/harness/transports.py (293 lines).

A transport is resolved from a declaration (transport.json) rather than from a branch in the run
stage. Each transport carries its own claims predicate, so it recognises its own agent, and the
build stage either names one this repo implements or writes a runner and declares where it lives.
TransportUnresolved is raised before any world is leased rather than surfacing as a failed
scenario twenty minutes in.

File	Change	Preserved
transports.py	New. Transport, Evidence, TransportUnresolved, declaration loading, runner import by module:Attribute	n/a
hosted_scheduler.py	_register_builtin_transports registers the two this repo implements (LiveKit, chat), each with its own claims; _transport_requires reads what a runner owes	The existing LiveKit and chat call paths, now reached through the registry rather than a branch

Gates that prove rather than assume

File	Change	Preserved
world/probe.py:267	A tool living in the submitted runtime is appended to a new ProbeReport.unproven list instead of being recorded as a pass	Every probe that actually executes is scored exactly as before; unproven is kept out of the score in both directions
world/probe.py:446	New verify_runtime_tools(world, contract) -> RuntimeToolVerdict. Calls the agent's own tools against the built world once the runtime is up	A tool that refuses is working; only a crash or a server error counts against it
world/probe.py:426	New RuntimeToolVerdict with checked, broken, reason, tools, and ok = checked and not broken	n/a
hosted_scheduler.py:1532	_verify_world runs that gate once per world, keyed by the runtime object rather than its pool index, and demotes a world that fails it	Verification never takes the run down; an exception is caught and logged

RuntimeToolVerdict exists to keep three outcomes apart: tools were called and none was broken,
tools were called and these were broken, and there was no way to call anything. The third used to
read as the first. The seam is checked before the tool list for the same reason: a world with no
forward seam also cannot say which tools it holds, so asking what it declares answers "none" for
a world that was never able to answer, which turned the gate into a silent no-op on the whole
hosted lane.

All of _verify_world's outcomes log at WARNING, deliberately: the hosted guest emits WARNING and
above, so an INFO line there is indistinguishable from no line at all.

An unbuildable repository refuses, instead of crashing

File	Change	Preserved
build.py:90	EnvironmentNotBuildable(RuntimeError) carries problems as data, not only as a formatted message	require_buildable still raises on the same condition, with the same message text
build.py:108	record_refusal writes environment-refusal.json; refusal_at reads it back and distinguishes unreadable from absent	n/a
hosted_entrypoint.py	The refusal is reported with its own code, owned by the submitted agent, and is not retryable	Every other failure path keeps its existing code

The stage that decides this and the process that reports it upward are different processes, and
what crosses that boundary is an exit status. A non-zero exit reads as "the guest crashed", which
was classified as retryable and spent a second sandbox re-deriving the same refusal.

Receipts

File	Change	Preserved
hosted_scheduler.py	call_evidence_faults holds a runner to what its transport declared it owes; CallEvidenceMissing gets its own code and carries the outcome it rejected	A zero-turn outcome is deliberately left alone, so it still reports as evidence_missing/simulator rather than being relabelled as an unrenderable runner
hosted_scheduler.py	An empty "requires": [] is treated as a declaration that the runner owes nothing; only an absent key inherits	n/a

World kinds

File	Change	Preserved
world/kinds.py	ROW_STORES registers postgres, postgresql, mysql, mariadb and clickhouse against SqliteWorld	SqliteWorld itself is unchanged; it reads world.state(), so any row store is the same shape to look at
world/kinds.py:228	An unregistered store still gets a world, and now logs a WARNING naming the assumption and how to register the right kind	The existing fallback order (named kind, no-store, modality, sqlite)

Skills

Twenty files, +1437 / -245. The organising change is that a stage skill states how the stage
works for any agent, and everything that differs between kinds of agent moves into
references/*.md that the model selects after it has read the contract.

config.py:sub_skills builds the catalogue from each reference's frontmatter description, name
and description only. The body stays on disk until the model asks for it, so a stage carries an
index of everything it could do at a fraction of the cost of carrying all of it, and adding support
for a new kind of agent is a file in a directory rather than a release. _summarise reads the
frontmatter specifically, because falling back to the first line of prose summarises a skill as
--- and a catalogue that describes nothing cannot be chosen from.

File	Change
skills/harness.md	+48. Adds the division the rest depends on: the model decides and writes, the code executes, and no model is in the loop at call time. Also states that phases are checkpoints you may return to, and that memory is the files on disk rather than the conversation
skills/build-environment/SKILL.md	+80 / -11. New section on working out what kind of agent this is from evidence before choosing a reference, and when to ask instead of guessing. New credentials section: never print, echo, log or seed a credential value, because anything printed reaches the guest log, which is captured into artifacts that outlive the sandbox. The tool instructions change from write_env_file / run_env_command to Write, Edit, Bash, Read, Glob, Grep
build-environment/references/	Seven new files: _writing-a-runner.md, voice-livekit.md, voice-hosted-platform.md, voice-bland.md, voice-multi-actor.md, browser-and-computer-use.md, retrieval-and-assistants.md. Each opens with a selection check that restates the evidence justifying it, and says what it is not for
build-environment/scripts/	probe_voice_providers.py (101 lines) checks provider credentials and prints status codes only, never binding the response body, because a provider error can quote the credential it was sent. check_call_evidence.py (100 lines) checks a receipt against what the platform renders
skills/write-scenarios/SKILL.md	+31 / -94. Cut to the stage method; the framework and the per-modality method move to references
write-scenarios/references/	Six new files: _framework.md, _authoring-code.md, voice.md, chat.md, coding.md, cua.md
skills/understand-agent/SKILL.md	+28 / -4
skills/provision-environment/SKILL.md	Deleted, 136 lines. Nothing loaded it: build.open_stage has always used build-environment. ALK_HARNESS_PROVISION still selects the alternate opening line at build.py:320

Contract, bundling, seed

File	Change	Preserved
contract.py	http is no longer silently aliased to the fi.alk envelope, and there is now a way to declare neither	The existing envelope behaviour for agents that do use it
bundle_author_v2.py	The generate path no longer requires a file literally named agent.py	The documented bundle requirements, which a repo could previously satisfy in full and still fail on a filename
session.py:156	_elided shortens a long label from the middle, keeping the end that names the file	Values at or under 80 characters are returned unchanged

_elided is small and worth one line of why: every skill file under a stage shares a prefix well
past 77 characters, so truncating from the end recorded three reads of .../skills/write-scenar...
and could not say whether the model had opened the skill body or one of its references. Whether the
reference catalogue is used at all is the question this branch most needs answered, and the display
format was the only thing preventing it being answered from a log we already had.

3. Behaviour callouts

• The build stage runs an unfiltered shell, and the grant is the same hosted and locally. In
  the hosted lane a sandbox contains it. Run locally, nothing does. The skill tells the stage to
  keep its work inside the run's own directories, which is an instruction, not an enforcement.
• A world that cannot answer the agent's own tools no longer grades anyone. It is demoted. Runs
  that previously completed with a perfect probe score and meaningless sub-goal results will now
  either fail that gate or report unproven tools.
• ProbeReport.score changes meaning for any contract with runtime tools. Those tools used to
  count as passes; they now count as neither.
• An unbuildable repository produces a distinct, non-retryable outcome instead of a retryable
  guest_crashed/infrastructure classification.
• An unregistered data store now emits a WARNING naming the shape it is being inspected as.
  Behaviour is unchanged; the assumption is just no longer silent.

4. Tests

Nine new files, 109 test functions, plus additions to four existing files.

File	Tests	Locks
tests/harness/test_runtime_tool_gate.py	28	A world with no forward seam reports checked=False, not a pass; a refusing tool counts as working; a hosted-shaped world with neither forward nor runtime_tools does not read as clean
tests/harness/test_written_runner_loading.py	22	A runner declared as module:Attribute loads from the bundle and imports cleanly on its own
tests/harness/test_runner_conventions.py	18	What a check, a ready and a setup callback may return, and which return values are advisory versus broken; also that the runner reference invents no field and points at something that exists
tests/harness/test_environment_refusal.py	11	The refusal keeps its reasons as data, survives the process boundary, is owned by the submitted agent, is not retryable, and that an unreadable refusal is not read as no refusal
tests/harness/test_chat_envelope.py	11	The contract no longer aliases http to the fi.alk envelope
tests/harness/test_generated_entrypoint.py	10	Bundle generation does not depend on a file named agent.py
tests/harness/test_sqlite_to_postgres_schema.py	9	pk is not a boolean, and NOT NULL and DEFAULT survive translation
tests/harness/test_permission_gate.py	8	A stage cannot use a tool it was not granted; the callback alone cannot enforce that, so a hook does; a stage that asked for a shell keeps it; a harness tool from a server the stage does not hold is still refused
tests/harness/test_world_kind_selection.py	6	Row stores resolve to the same kind; an unregistered store is announced
tests/harness/test_suite_review.py	4	A crashed suite review is not a review that approved, and does not end the top-up loop unreported

Existing files: test_hosted_scheduler.py +119, test_harness.py +196 / -94, test_hosted_entrypoint.py +51 / -17, test_bundle_author_v2.py +5 / -3.

5. Commands

uv run pytest tests -q
uv run pytest tests/harness/test_permission_gate.py tests/harness/test_runtime_tool_gate.py -v

Terminal output

<image: full suite result on this branch>

6. Steps to test on local

1. Check out experiment/autonomous-harness and run uv sync.
2. Run uv run pytest tests -q. Record the pass and fail counts.
3. Check out feat/hosted-bundle-v2-production and run the same command. Confirm the failure set is
   the same, so nothing here is a regression against the base.
4. Back on the branch, run a hosted job against a voice agent repository per the existing runbook.
5. Confirm the guest log carries a verified N runtime tools or no runtime tools declared line at
   WARNING before the first graded call.
6. Point a job at a repository that ships no seam the environment could be built against. Confirm
   the job reports a refusal naming each tool and what the repository must expose, and that it does
   not consume a second sandbox attempt.

Video

<video: a hosted run reaching the runtime-tool gate, and a refusal on an unbuildable repository>

7. Scope in / scope out

In scope: the build stage's tool grant, transport resolution by declaration, the runtime-tool
gate and the probe reporting behind it, the refusal path, receipt requirements, row-store world
kinds, and the skills reorganisation.

Out of scope:

• A chat agent that runs its own tools against its own store still cannot be observed. The
  generated chat world stands up no tools endpoint to point it at, so no tool-based sub-goal can
  pass however the agent behaves. The bundle now says so rather than grading the agent for it. The
  fix is a design decision, not an implementation, so it is deliberately not attempted here.
• Sandbox-level confinement of the build stage's shell is not added. The grant is deliberately
  identical hosted and locally so that a local run is a rehearsal of the hosted one; making them
  differ would defeat that. Whether that trade is right is one of the things this PR is open for.
• ALK_HARNESS_PROVISION and build.py:320 are left in place. The skill file the legacy path
  once paired with was already unreferenced; removing the flag as well would widen the diff without
  changing behaviour.
• SqliteWorld is not renamed. The name is now inaccurate, since it describes the shape of the
  state rather than the engine, but renaming it touches the registry, the contract vocabulary and
  the skills, and it changes nothing at runtime.

8. Design choices

• Filter the hidden-tool list against the grant, rather than deleting the gate. The gate is what
  makes the grant mean something: a stage's tool list is a statement about what that stage is, and
  a blanket denial made that statement unenforceable in one direction while a blanket allow made it
  unenforceable in the other. Filtering keeps both true.
• A PreToolUse hook rather than the permission callback alone. An allowed_tools entry
  auto-approves before can_use_tool is consulted, so the callback never sees anything granted. The
  hook is consulted on every call, which is what makes deny-by-default true rather than intended.
• unproven as a third state, kept out of the score. Folding it into either passes or failures
  reintroduces the original defect in one direction or invents failures in the other.
• The verified-world map is keyed by the runtime object, not its pool index. An index is reused:
  reconcile replaces a demoted world at the same index, and that replacement is exactly when the
  check matters most, because the world it replaced may have been demoted by this gate.
• The refusal travels as a document, not an exit code. The deciding stage and the reporting
  process are different processes, and an exit status cannot carry a per-tool remedy.
• References are catalogued by frontmatter description, loaded on demand. A stage carries an
  index of everything it could do rather than the contents, and a new kind of agent is a file rather
  than a release.

9. What this does not claim

• Not tested against the dependency-deferral case. The failure this is partly aimed at is an
  authoring stage that saw its tools needed a package from requirements.txt, wrote that the
  runtime would have the full environment and that binding would happen there, and then failed a
  gate that asked for proof of binding and found none. Two things here bear on it, and neither has
  been run against it. The build stage can now install the package and execute the thing, rather
  than only writing a file and running a container command, so the deferral is no longer the only
  move available. And verify_runtime_tools makes the proof itself explicit and separately
  reportable, so "nothing was proven" is distinguishable from "nothing was broken". What is
  unchanged is that a run still stops when binding cannot be proved, and correctly so. Whether the
  extra capability is what the stage was actually short of is the untested part.
• The hosted evidence here predates the current base. Eight hosted runs were made during
  development: four on a voice agent, all completed, two fully clean, with real audio throughout and
  14 to 22 turn conversations carrying four recordings each; and four on a chat agent shipping no
  Compose file, no Dockerfile and no data store, ending in a graded pass covering contract, invented
  world, generated scenarios, live conversation, observed tool calls and graded sub-goals. Those
  runs were made before this branch was rebased onto its current base, so they are a record of what
  the approach did, not a verification of this exact tree.
• The build stage's shell has not been run adversarially. Nothing here tests what happens when
  the stage does something outside the run's directories on an operator's machine.