Summary
Five defects in the AgentBench evaluation harness at pin (R03/E1 audit).
1. DBBench grader runs eval() on the agent's final answer
The evaluated agent can execute arbitrary code in the grading process, including monkeypatching the grade checker.
2. DBBench treats empty/null/NaN/undefined/infinity answers as zero
A null answer collapses to a numeric 0, which matches any gold answer of 0. Wrong answers score correct.
3. One errored sample silently deletes the whole task from analysis
A single exception in the results file removes the entire task from the aggregate, inflating the reported score.
4. Evaluation control plane has no authentication
The shipped deployment procedure exposes the eval control plane without authentication.
5. OS verification scripts run inside the agent's container
The verification for OS-interaction tasks executes in the same container the agent controls, so the agent can tamper with the check.
All five reproduced with deterministic offline harnesses.
Summary
Five defects in the AgentBench evaluation harness at pin (R03/E1 audit).
1. DBBench grader runs eval() on the agent's final answer
The evaluated agent can execute arbitrary code in the grading process, including monkeypatching the grade checker.
2. DBBench treats empty/null/NaN/undefined/infinity answers as zero
A null answer collapses to a numeric 0, which matches any gold answer of 0. Wrong answers score correct.
3. One errored sample silently deletes the whole task from analysis
A single exception in the results file removes the entire task from the aggregate, inflating the reported score.
4. Evaluation control plane has no authentication
The shipped deployment procedure exposes the eval control plane without authentication.
5. OS verification scripts run inside the agent's container
The verification for OS-interaction tasks executes in the same container the agent controls, so the agent can tamper with the check.
All five reproduced with deterministic offline harnesses.