Skip to content

Derive a coverage grid before writing scenarios, and sample it deliberately - #73

Draft
KarthikAvinashFI wants to merge 1 commit into
feat/hosted-bundle-v2-productionfrom
feat/scenario-grid-skill
Draft

Derive a coverage grid before writing scenarios, and sample it deliberately#73
KarthikAvinashFI wants to merge 1 commit into
feat/hosted-bundle-v2-productionfrom
feat/scenario-grid-skill

Conversation

@KarthikAvinashFI

Copy link
Copy Markdown
Contributor

What

Rewrites the write-scenarios skill so the scenarios stage derives a coverage grid from the agent's contract before writing anything, samples that grid deliberately, fans the writing out across the existing parallel writer sessions, and closes with a coverage report. Enables the parallel path in the hosted image and sizes its batches.

No code changes. The diff is one skill file and three ENV lines plus a revision bump in Dockerfile.hosted.

Why

Measured across 718 previously generated scenarios: 5 of the 12 canonical task operations were effectively untested (diagnose and navigate at 0%, compare at 1%), and only 5% carried any adversarial or safety overlay, with impersonation at zero. The suites collapse into create/execute because nothing tells the writing model what the space of situations is or which cells it must not skip.

The skill now instructs the model to:

  • derive task intent as a closed 12-operation set crossed with the contract's own domain objects, pruned, so intent coverage is exhaustive by construction rather than imagined
  • treat counterparty, disposition, channel, interaction and adversarial overlay as explicit axes, with one off-baseline axis per scenario so results stay attributable
  • distinguish world-backed overlays (impersonation, fraud: need setup and proof) from prompt-side ones (injection, pressure: live in the instruction)
  • mask incoherent cells, then sample with a hard-required checklist: emergency, injection, vulnerable caller, world-backed impersonation and fraud, a diagnose intent, a compare/explain intent, and an evidence-reached handoff must each exist before saving
  • run the suite as a workflow: slice writers along grid cells through generate_suite, review between batches, keep calling until the requested count is met (repeat calls merge with what is on disk and lose nothing)
  • encode each scenario's grid cell in its folder name so the saved index doubles as the machine-readable coverage record, and close the stage with the coverage summary

Very large asks are handled the way the grid defines: the number is the space to cover, the written suite is the sample over it, and the report states what the sample covers.

Not verified yet

This is instruction-only and the behaviour is model-driven, so it needs the verification runs before leaving draft:

  1. A ~40 ask on the default Gemini backend against an authored world: grid stated first, slices follow cells, hard-required cells present, names encode cells, all three gates pass, closing report emitted. Re-measure the operation/overlay distribution against the 5%/0% baseline.
  2. The same run with ALK_HARNESS=claude.
  3. Failure modes: an ask of 7 must not fan out; an ask above one batch must loop unprompted.
  4. One small hosted job to confirm the ENV lines reach the guest.

Steps to test

export HARNESS_PARALLEL_SCENARIOS=1 HARNESS_SUITE_BATCH=100 HARNESS_WRITERS_AT_ONCE=8
# run the scenarios stage against an existing authored world, ask for 40

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant