Derive a coverage grid from the agent's contract and sample it for any suite size

[KarthikAvinashFI]

Scenario generation answered "write N scenarios" by writing N, and nobody could say afterwards what was skipped, because nothing had enumerated what was possible. Measured across 718 scenarios this harness had generated: 5% carried any adversarial situation, none covered diagnosis, none covered multi-step navigation. The model wrote the easy cases because nothing required otherwise.

This changes the unit of work from "a scenario" to a coordinate in a grid derived from the agent's own contract, adds a planning stage above it, and makes the plan a ledger the run works from.

The three levels, named once

A scenario is one test: a folder, a setup, checks, a reference solution. A bucket is a kind of case holding several scenarios; its angle says what makes it worth testing. A theme groups buckets and is the unit a large plan is paged in and out by. "Scenario" and "situation" both name single instances, so neither works as the container, which is why a third word exists.

Planning, and why it is a separate stage

Asking for a thousand finished scenarios at once does not fit in a context. Writing them one at a time does fit and quietly converges: each is composed with the last few in view, so by fifty the suite has settled into one shape. Measured here, fifty scenarios came back with nine distinct people in them, forty-two of them American, living in two places, and every writer had been told to vary its work.

A thousand one-line intentions do fit. So plan is its own sub-skill: decide what every bucket is before any scenario is written, while the whole suite is still visible and still cheap to change.

The level is enforced rather than requested. An angle over 90 characters is refused as a script rather than an angle, because at the length the first attempt used, a plan for a thousand scenarios is 228KB and 57k tokens to emit in one response.

How a bucket's size is derived

want used to be guessed, and the guess was 1 everywhere, which is a list of scenarios with extra fields. It is now derived.

The planner reads the agent's data and rules and declares state axes: dimensions whose value changes what the agent should do. A level must exist in the data or be reachable by seeding, and must change the correct answer. Each bucket names the axes it crosses in live, and want is how many of their combinations survive masking. An axis nobody derived is refused; a want above one with neither axes nor a stated reason is refused.

On a real agent the planner derived seven axes unprompted, including a payment state with eight levels and an account status carrying two values that appear only in the rules rather than in the data.

The canvas is a ledger, not a document

Each bucket carries done, refused, attempts, state, claimed_by and notes. The loop claims a slice, dispatches one writer, folds the return, and re-ranks. Ranking weights outstanding work by how much of its theme is untouched, so a theme nobody has started outranks one nearly finished; and a slice never contains two buckets from one cell, because a writer handed a whole cell must invent that cell's variety alone.

done is counted from disk and never from the writer's report. The writer's own number is kept only to notice disagreement, which is itself a bug signal.

Writers can open buckets the plan never saw. The plan is written from outside the code; a writer is the first thing to look inside with the source open. It reports what it found, the stage records it with an id the canvas assigns, and it is dealt like any other. Ids are assigned centrally so simultaneous discoveries cannot collide.

The ceiling is measured rather than predicted. Three failed attempts marks a bucket blocked, and what remains when nothing is open is what the agent actually supports.

What a plan has to say about itself

Recording a canvas prints coverage against things outside the plan: which grid cells have nothing on them, how many of the agent's hard rules have a bucket testing them, how many precondition-gated tools are named, and how the scenarios split across happy, edge, adversarial and failing. A plan with nothing in one of those four is reported as such.

The rule line is the one worth reading. A rule with no bucket is something the agent is forbidden to get wrong that nobody is checking.

Defects found and fixed along the way

Each only reachable above the delegation threshold, which is why none had surfaced before:

• A fan-out could not save. Writers appended into a list the stage never read, so a run accepted fifty scenarios that passed every gate and wrote none.
• Writers ran in the background. AgentDefinition.background defaults to asynchronous, so the stage dispatched five writers, took its next turn, declared success and exited, killing all five. One scenario saved from a plan of fifty, reported as success.
• Reading the world rewrote it. restore reloads the snapshot into the store, truncating and reinserting every table, and three tools did it outside the lock. A scenario could be proved against a world a sibling had rewritten underneath it.
• Store engines leaked. One Postgres container per world, cleaned up by atexit, which does not run on SIGTERM. Now one engine per image with a database per world, released on signals too.
• Half the prompt was waste. The preamble was embedded twice and the planner carried the 44KB writing skill it was not using. Planning prompt went from 93KB to 44KB.
• Writers could drop siblings' work, and could not reach the tool the skill told them to call.
• Every session resolved relative paths from the wrong directory. cwd was the run directory's parent, but the environment bundle lives inside the run directory, so every path the skill names missed by one level. Writers could not read the agent under test, and instead read a neighbouring run's world, a neighbouring run's scenarios, and model-written scripts left in the shared directory that called accept_scenario directly, around the gates. One helper now resolves all three sessions from the run directory.
• A fan-out held its whole suite in memory until the final save. Saving rewrites the index and deletes folders it does not know about, so writers cannot each save; the stage saved once at the end. A fifty-scenario run therefore sat at forty-eight proved and nothing on disk. Each scenario is now journalled as it is proved, and a later run folds back whatever a killed one left. A torn final line costs one scenario.
• An interrupted restore left the world half loaded. The truncate committed and the inserts did not, so the next run died on a duplicate key before its first call. Restore is now one transaction.
• Slice writers forced thinking on regardless of the run's setting, which on one backend is the configuration that stalls a call at zero CPU on a read that never returns.
• Folding a writer's return could move a bucket's progress backwards. done was assigned from the current return's names alone, so a bucket filled over two rounds went three-then-two, burned an attempt per round, and could be marked blocked while genuinely being filled. Progress is now a ledger of credited scenario names on the bucket: rounds add up, and one name can never fill two buckets.
• The whole-plan refusals fired on every instalment. A canvas is recorded one theme at a time; a first theme covers few cells and may name no precondition-gated tool, so an honest large plan's first recording was refused and the refusal ordered the model to break the instalment discipline. Whole-plan checks now wait until the plan reaches its target - and the target is pinned to the count the stage was opened with, since every one of those checks guards on it.
• The write skill spoke to three roles at once. The same text serves the solo session, the orchestrator, and a dispatched slice writer, so a writer was reading instructions to delegate, claim slices, and save - with none of those tools. It now opens by having the model identify its role, and every orchestration section is marked as such.

Making the fan-out actually fan out

Declaring workers is not the same as using them. Measured on a 200-scenario run: the stage made 59 of the submissions itself and dispatched four writers, then spent its turns proving instead of dealing. Two changes close that.

A stage that has writers can no longer write. submit_scenario is withheld when workers are declared, exactly as generate_suite already is and for the same reason: offered both, the model does the work itself and the fan-out goes unused. Below the delegation threshold no workers exist, so a small suite keeps the tool and writes its own scenarios as before.

Writers run several at a time. The canvas was always built for it, and nothing used it: claim_slice marks its angles taken, so a second claim cannot return the first one's work and two writers can never be handed the same scenario. The skill previously said "one writer at a time"; it now claims per writer and dispatches them together.

A writer's turn budget follows the slice it can be handed. It was a flat number that predated the slice clamp, leaving 3.8 turns per scenario including the reading a writer does before writing anything, so slices came back part-filled and the next writer paid that reading cost again.

Surviving a long run

A suite of this size takes hours, and every one of these ended a run early before it was fixed.

• A provider's rate limit ended the run. A single 429 on the orchestrator killed a 500-scenario run in its fourteenth turn, still planning. A rate-limited turn is now waited out and asked again, keeping the session.
• A delegated writer never writes folders, because saving would delete its siblings' work. Three separate places asked "what exists?" by reading folders alone and got half an answer: folding credited nothing and marked live buckets blocked, which is what made a run give up at 61 of 200; the progress watchdog killed an attempt that was producing; and the quality report judged a stale sample. Anything asking what exists now asks both the folders and the journal.
• Progress could move backwards. done was assigned from the current return's names, so a bucket filled over two rounds went three-then-two and could be marked blocked while being filled. It is now a ledger of credited scenario names, persisted, so rounds add up and one name can never fill two buckets.

Verified

understand and build run unattended against a 20-tool voice agent: 12 collections seeded, 13 code-settled sub-goals, 22 probes at 1.00.

Planning on that agent: 63 cells, 168 buckets, 16 themes, 7 derived state axes, 500 scenarios planned.

Writing is not yet proven at scale. The largest completed suite is 50. Cost and throughput are recorded in the run notes rather than claimed here.

Coverage the plan cannot get from the grid

A cell is an operation on an object, so it says nothing about order. Two consecutive plans were told 0 of 10 tools with preconditions have a bucket that names them and neither acted on it, while fixing other reported faults in the same pass. Being asked for something before its precondition holds is exactly where this kind of agent fails, so a large plan that names none of those tools is now refused rather than warned about, beside the cell-coverage floor.

The bar is zero named, not a share. A share invites spreading tool names through the prose to clear the check without writing the ordering case.

Known weakness, not introduced here

prove.play_reference_step records a step as passing without running it when the tool has no endpoint bound, and says so in a warning. Where that happens the proof is partial. This is a build-lane fix and is deliberately not papered over in the scenario lane.

Supersedes #73 and #63.