Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions apps/server/src/provider/agentContextBudget.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"claudeSystemAppendBytes": 5930,
"codexAdditionalContextBytes": 7324,
"claudeSystemAppendBytes": 6037,
"codexAdditionalContextBytes": 7431,
"mcpToolListJsonBytes": 52145
}
2 changes: 1 addition & 1 deletion evals/mcp-tool-use/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Cases are JSON lines in `cases/`, one file per area. The case schema in `run.ts`
- `experience`: the real mistake, in a sentence. `source`: where it happened.
- `setup`: the fake thread, its `world` (repo, branch, files, PRs, pages), `priorTurns`, and `mocks` for tool results. With `world.backgroundTasks`, `t3_background_wait` answers from T3's own handler and ThreadManagementService, so its result and errors are the checkout's; a prior wait with `"result": "default"` gets that result too.
- `setup.priorTurns[].toolCalls[].resultFile`: load a repository-relative file as the tool result instead of copying its contents into the case. An empty `prompt` resumes a top-level thread after its last prior tool result without adding a user message. Side-chat and subagent setups require a nonempty prompt.
- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).
- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. For `calls` and `callsMustNot`, `argsAnyOf` accepts any one of its partial argument objects, in addition to `args`. Only `calls` supports `success: true`, which requires a recorded successful result. `distinctCalls` supports only `tool` and `args`. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).

Check that the case builds with `--dry-run --case <id>`. Then run it on the text before your fix, where it must fail, and on the text after, where it must pass.

Expand Down
2 changes: 2 additions & 0 deletions evals/mcp-tool-use/cases/issues.ndjson
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"id":"work-on-issue-links-thread","area":"issues","kind":"bad","pattern":"Agent takes on a tracked issue without linking it to the thread","experience":"Agents linked pull requests but often left the issues they were working on unlinked.","privacy":"commit-safe","setup":{"role":"top","mocks":{"link_issue":{"projectId":"app","tracker":"github","host":"github.com","scope":"acme/app","key":"42","url":"https://github.com/acme/app/issues/42","alreadyLinked":false}},"world":{"repo":"acme/app","files":{"src/label.ts":"export const label = 'Setings';"},"issues":[{"url":"https://github.com/acme/app/issues/42","number":42,"title":"Typo in Settings label","body":"The Settings label says Setings.","state":"open"}],"pages":{"https://github.com/acme/app/issues/42":"Typo in Settings label\nThe Settings label says Setings."}},"context":"The current thread ID is issue-work-thread. Its project ID is app, configured for the GitHub repository acme/app."},"prompt":"Take on https://github.com/acme/app/issues/42. Investigate the Settings label typo, which says Setings, and propose the smallest fix.","expect":{"calls":[{"tool":"link_issue","argsAnyOf":[{"url":"https://github.com/acme/app/issues/42","threadId":null},{"url":"https://github.com/acme/app/issues/42","threadId":"issue-work-thread"},{"key":"42","url":null,"projectId":null,"threadId":null},{"key":"42","url":null,"projectId":null,"threadId":"issue-work-thread"},{"key":"42","url":null,"projectId":"app","threadId":null},{"key":"42","url":null,"projectId":"app","threadId":"issue-work-thread"},{"url":"https://github.com/acme/app/issues/42/","threadId":null},{"url":"https://github.com/acme/app/issues/42/","threadId":"issue-work-thread"}],"success":true}],"finalTextMust":["Setings","Settings"]},"source":{"pr":"https://github.com/pingdotgg/t3code/pull/38356"}}
{"id":"read-only-issue-stays-unlinked","area":"issues","kind":"control","pattern":"Read-only issue lookup leaves thread links alone","experience":"An issue used for reference does not belong in the thread's work links.","privacy":"commit-safe","setup":{"role":"top","mocks":{"read_issue":{"url":"https://github.com/acme/app/issues/42","title":"Typo in Settings label","body":"The Settings label says Setings."}},"world":{"repo":"acme/app","issues":[{"url":"https://github.com/acme/app/issues/42","number":42,"title":"Typo in Settings label","body":"The Settings label says Setings.","state":"open"}],"pages":{"https://github.com/acme/app/issues/42":"Typo in Settings label\nThe Settings label says Setings."}}},"prompt":"Summarize https://github.com/acme/app/issues/42 so I can decide whether to take it on.","expect":{"notCalls":["link_issue","unlink_issue","create_issue","comment_on_issue","set_issue_state"],"finalTextMust":["Setings","Settings"]},"source":{"pr":"https://github.com/pingdotgg/t3code/pull/38356"}}
49 changes: 49 additions & 0 deletions evals/mcp-tool-use/run.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,55 @@ it("accepts any tool listed as an alternative", () => {
assert.deepEqual(failed(expect, run("list_thread_pull_requests")), [
"calls link_pull_request|watch_pull_request",
]);
const issueCase = decodeDeviceCase(
NodeFS.readFileSync(new URL("./cases/issues.ndjson", import.meta.url), "utf8").split("\n")[0]!,
);
for (const [input, passes] of [
[{ url: "https://github.com/acme/app/issues/42" }, true],
[{ url: "https://github.com/acme/app/issues/42/" }, true],
[{ url: "https://github.com/acme/app/issues/42/", threadId: "issue-work-thread" }, true],
[{ url: "https://github.com/acme/app/issues/42/", threadId: "another-thread" }, false],
[{ key: "42", projectId: "app" }, true],
[{ key: "42" }, true],
[{ key: "42", threadId: "issue-work-thread" }, true],
[{ key: "42", threadId: "another-thread" }, false],
[{ key: "42", url: "https://github.com/acme/app/issues/43" }, false],
[{ url: "https://github.com/acme/app/issues/42", threadId: "issue-work-thread" }, true],
[{ key: "42", projectId: "app", threadId: "issue-work-thread" }, true],
[{ url: "https://github.com/acme/app/issues/42", threadId: "another-thread" }, false],
[{ key: "43", projectId: "app" }, false],
[{ key: "42", projectId: "other-project" }, false],
] as const) {
assert.equal(
failed(issueCase.expect, {
...run("link_issue"),
finalText: "Change Setings to Settings.",
calls: [{ step: 0, tool: "link_issue", input, failed: false }],
}).length === 0,
passes,
);
}
assert.isNotEmpty(
failed(issueCase.expect, {
...run("link_issue"),
finalText: "Change Setings to Settings.",
calls: [{ step: 0, tool: "link_issue", input: { key: "42" }, failed: true }],
}),
);
assert.isNotEmpty(
failed(issueCase.expect, {
...run("link_issue"),
finalText: "Linked the issue.",
calls: [{ step: 0, tool: "link_issue", input: { key: "42" }, failed: false }],
}),
);
assert.isNotEmpty(
failed(issueCase.expect, {
...run("link_issue"),
finalText: "Change Setings to Settings.",
calls: [{ step: 0, tool: "link_issue", input: { key: "42" } }],
}),
);
// A bound on alternatives counts their calls together.
const both: SampleRun = {
calls: ["preview_status", "preview_open"].map((tool) => ({ step: 0, tool, input: {} })),
Expand Down
27 changes: 22 additions & 5 deletions evals/mcp-tool-use/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ import { AttachmentToolkit } from "../../apps/server/src/mcp/toolkits/attachment
import { DeviceToolkit } from "../../apps/server/src/mcp/toolkits/device/tools.ts";
import { EnvironmentToolkit } from "../../apps/server/src/mcp/toolkits/environment/tools.ts";
import { HtmlToolkit } from "../../apps/server/src/mcp/toolkits/html/tools.ts";
import { IssuesToolkit } from "../../apps/server/src/mcp/toolkits/issues/tools.ts";
import { MessageToolkit } from "../../apps/server/src/mcp/toolkits/message/tools.ts";
import { OrchestratorToolkit } from "../../apps/server/src/mcp/toolkits/orchestrator/tools.ts";
import { PreviewToolkit } from "../../apps/server/src/mcp/toolkits/preview/tools.ts";
Expand Down Expand Up @@ -189,16 +190,24 @@ const McpToolUseCall = Schema.Struct({
tool: Schema.String,
args: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
});
type McpToolUseCall = typeof McpToolUseCall.Type;
const McpToolUseCallWithAlternatives = Schema.Struct({
...McpToolUseCall.fields,
argsAnyOf: Schema.optional(Schema.Array(Schema.Record(Schema.String, Schema.Unknown))),
});
const McpToolUseExpectedCall = Schema.Struct({
...McpToolUseCallWithAlternatives.fields,
success: Schema.optional(Schema.Literal(true)),
});
type McpToolUseCall = typeof McpToolUseExpectedCall.Type;

/** Grading rules. Tool names are bare: `delegate_task`, `Bash`, `Agent`. */
const McpToolUseExpect = Schema.Struct({
/** Each entry needs one matching call. */
calls: Schema.optional(Schema.Array(McpToolUseCall)),
calls: Schema.optional(Schema.Array(McpToolUseExpectedCall)),
/** Each entry matches a different call, and no call matches two entries; tool names are exact. */
distinctCalls: Schema.optional(Schema.Array(McpToolUseCall)),
/** No call may match any entry. */
callsMustNot: Schema.optional(Schema.Array(McpToolUseCall)),
callsMustNot: Schema.optional(Schema.Array(McpToolUseCallWithAlternatives)),
notCalls: Schema.optional(Schema.Array(Schema.String)),
/** First calls of these tools happen in this order. */
order: Schema.optional(Schema.Array(Schema.String)),
Expand Down Expand Up @@ -458,6 +467,7 @@ const t3SourceTools: ReadonlyArray<Tool.Any> = [
WorktreeToolkit,
PullRequestsToolkit,
PreviewToolkit,
IssuesToolkit,
DeviceToolkit,
HtmlToolkit,
MessageToolkit,
Expand Down Expand Up @@ -1860,8 +1870,15 @@ export const gradeSample = (
// `tool` may list alternatives, as in "link_pull_request|watch_pull_request"; counts add up.
const count = (tool: string) =>
calls.filter((call) => tool.split("|").includes(call.tool)).length;
const found = ({ tool, args = {} }: McpToolUseCall, among = calls) =>
among.some((call) => tool.split("|").includes(call.tool) && matches(args, call.input));
const found = ({ tool, args = {}, argsAnyOf, success }: McpToolUseCall, among = calls) =>
among.some(
(call) =>
tool.split("|").includes(call.tool) &&
(success === undefined || call.failed === false) &&
matches(args, call.input) &&
(argsAnyOf === undefined ||
argsAnyOf.some((alternative) => matches(alternative, call.input))),
);
const label = ({ tool, args }: McpToolUseCall) => (args ? `${tool} ${valueText(args)}` : tool);
for (const call of expect.calls ?? []) add(`calls ${label(call)}`, found(call));
if (expect.distinctCalls !== undefined) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ describe("buildRuntimeInstructions", () => {

assert.include(instructions, "Link each PR you create, take over, or push to");
assert.include(instructions, "every stack layer");
assert.include(instructions, "link_issue");
assert.include(instructions, "Skip PRs you only read or review");
assert.include(instructions, "If linking fails, say so.");
assert.notInclude(instructions, "git log -1");
Expand Down
1 change: 1 addition & 0 deletions packages/provider-core/src/server/runtimeInstructions.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
/** Session-invariant; harnesses with a native system-prompt append can carry it once per session. */
export const PULL_REQUEST_LINKING_INSTRUCTIONS = `<pull_request_linking>
Link each PR you create, take over, or push to: link_pull_request, or link_pull_requests for several, full URL, every stack layer, as your next call. Host tools don't link for you. If linking fails, say so. Skip PRs you only read or review and PRs named only as background. Call watch_pull_request when asked to monitor or babysit, or when work you were asked to do, such as a merge, still waits on checks or reviews, even after an earlier unwatch. Finish any needed branch update and push first, then watch and end your turn. Otherwise act now or hand back. Delegated children never watch, even if the brief says babysit. After PR work, a child's last line is: PR <full URL>, head <SHA>, checks and reviews <state>. Links and watches persist across pushes. For PR waits, no sleep, loops, gh --watch, Monitor or ScheduleWakeup. Before handback, call unwatch_pull_request on any PR you watch.
If available, call link_issue for each issue you take on or file, even via CLI; skip issues you only read.
</pull_request_linking>`;

/** Omit model and effort when the harness manages them dynamically. */
Expand Down
Loading