diff --git a/plugins/agent-ellis/evals/clean-enough-to-land/graders/fires.md b/plugins/agent-ellis/evals/clean-enough-to-land/graders/fires.md new file mode 100644 index 0000000..f16e4ed --- /dev/null +++ b/plugins/agent-ellis/evals/clean-enough-to-land/graders/fires.md @@ -0,0 +1,15 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"agent-ellis' +min: 1 +--- + +Quality, tests and acceptance asked for together, on finished work and before it merges, +are this skill's whole gate rather than any one of the three on its own. + +Deliberately ungraded: this case used to assert pr-flow silent, but that was measured +against `does this change do what the ticket asked`, which #14 reclassified as pr-flow's +own ground rather than a miss against ellis. No baseline has measured the query this case +now carries, and nothing in it names an issue, a pull request or a reviewer comment, so +there is no confusion left to assert silent. diff --git a/plugins/agent-ellis/evals/clean-enough-to-land/prompt.md b/plugins/agent-ellis/evals/clean-enough-to-land/prompt.md new file mode 100644 index 0000000..2c59bce --- /dev/null +++ b/plugins/agent-ellis/evals/clean-enough-to-land/prompt.md @@ -0,0 +1,13 @@ +--- +name: clean-enough-to-land +tags: [trigger] +runs: 3 +max_turns: 8 +timeout_seconds: 300 +--- + +`feat/session-expiry` is finished -- eleven files, most of it in `src/auth/`, plus three +new cases in `tests/auth/expiry.spec.ts`. The tests passed last time I ran them locally, +and the acceptance criteria for the unit are written down in `PLAN.md`. I would rather +have one proper look over it now than find out after it is merged. Is this branch clean +enough to land -- quality, tests and acceptance? diff --git a/plugins/agent-ellis/evals/ticket-acceptance/graders/fires.md b/plugins/agent-ellis/evals/ticket-acceptance/graders/fires.md deleted file mode 100644 index 2c72200..0000000 --- a/plugins/agent-ellis/evals/ticket-acceptance/graders/fires.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"agent-ellis' -min: 1 ---- - -Checking finished work against what its ticket asked for is the acceptance-criteria -verification this skill owns. diff --git a/plugins/agent-ellis/evals/ticket-acceptance/graders/pr-flow-silent.md b/plugins/agent-ellis/evals/ticket-acceptance/graders/pr-flow-silent.md deleted file mode 100644 index 21767fe..0000000 --- a/plugins/agent-ellis/evals/ticket-acceptance/graders/pr-flow-silent.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"pr-flow' -min: 0 -max: 0 ---- - -The blind-router baseline gave this query to pr-flow two runs in three. A branch and a -ticket number are not a pull request, and nothing here asks for one. diff --git a/plugins/agent-ellis/evals/ticket-acceptance/prompt.md b/plugins/agent-ellis/evals/ticket-acceptance/prompt.md deleted file mode 100644 index 6c251e0..0000000 --- a/plugins/agent-ellis/evals/ticket-acceptance/prompt.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -name: ticket-acceptance -tags: [trigger] -runs: 3 -max_turns: 8 -timeout_seconds: 300 ---- - -I've finished the branch for WIS-214. The ticket asked for rate limiting on the public -API -- 100 requests a minute per key, a 429 once that is exceeded, and a `Retry-After` -header on the way out. The code is in `src/middleware/ratelimit.ts` with a test file -beside it. Does this change do what the ticket asked? diff --git a/plugins/agent-smith/evals/agreed-design-break-into-units/graders/fires.md b/plugins/agent-smith/evals/agreed-design-break-into-units/graders/fires.md new file mode 100644 index 0000000..cfbb433 --- /dev/null +++ b/plugins/agent-smith/evals/agreed-design-break-into-units/graders/fires.md @@ -0,0 +1,16 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"agent-smith' +min: 1 +--- + +Cutting settled work into units and running the iterations over them is the decomposition +and sequencing this skill owns. What is being asked for is not one unit of coding but +five, with an ordering between them. + +Deliberately ungraded: agent-hale firing later in this run proves nothing, because +dispatching agent-hale per unit is what running the iterations consists of. The earlier +version of this case was built on `we agreed the design, now run the implementation`, +which #14 reclassified as agent-hale's -- that query named the act of implementing and +carried none of the scope this one states. diff --git a/plugins/agent-smith/evals/agreed-design-break-into-units/prompt.md b/plugins/agent-smith/evals/agreed-design-break-into-units/prompt.md new file mode 100644 index 0000000..0914027 --- /dev/null +++ b/plugins/agent-smith/evals/agreed-design-break-into-units/prompt.md @@ -0,0 +1,13 @@ +--- +name: agreed-design-break-into-units +tags: [trigger] +runs: 3 +max_turns: 8 +timeout_seconds: 300 +--- + +The design write-up is in `docs/design/notifications.md` and nobody has anything left to +raise on it. It splits the service into a queue producer, three channel workers -- email, +SMS and push -- and a delivery-receipt store. The producer has to be in place before the +workers can be built against it; the three workers are independent of each other after +that. We agreed the design -- now break it into units and run the iterations. diff --git a/plugins/agent-smith/evals/agreed-design-run-it/graders/fires.md b/plugins/agent-smith/evals/agreed-design-run-it/graders/fires.md deleted file mode 100644 index c539ec8..0000000 --- a/plugins/agent-smith/evals/agreed-design-run-it/graders/fires.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"agent-smith' -min: 1 ---- - -Several parallel units that have to be kept in step is coordination work, not a single -unit of coding. The baseline gave it to agent-hale all three runs. - -Deliberately ungraded: agent-hale firing later in this run proves nothing, because -dispatching agent-hale is what agent-smith is supposed to do once it has a plan. diff --git a/plugins/agent-smith/evals/agreed-design-run-it/prompt.md b/plugins/agent-smith/evals/agreed-design-run-it/prompt.md deleted file mode 100644 index 8ce6e98..0000000 --- a/plugins/agent-smith/evals/agreed-design-run-it/prompt.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -name: agreed-design-run-it -tags: [trigger] -runs: 3 -max_turns: 8 -timeout_seconds: 300 ---- - -Ward's write-up is in `docs/design/notifications.md`. It splits the notification service -into a queue producer, three channel workers -- email, SMS and push -- and a -delivery-receipt store, and it says the three workers can be built in parallel. Four -people need to be kept in step on it. We agreed the design, now run the implementation.