Skip to content

Commit ea22918

Browse files
mtodorgithub-actions[bot]
authored andcommitted
Update model evaluations 2026-09-15
1 parent 7046113 commit ea22918

1 file changed

Lines changed: 15 additions & 15 deletions

File tree

docs/model-evaluation.md

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -39,27 +39,27 @@ A task passes when **all** its assertions pass **and** the LLM judge approves th
3939

4040
<!-- model:gpt-5-mini start -->
4141

42-
### gpt-5-mini — 2026-09-08
42+
### gpt-5-mini — 2026-09-15
4343

44-
**Overall: 11/11 tasks passed (100%)**
44+
**Overall: 10/11 tasks passed (90%)**
4545

4646
#### Task Results
4747

4848
| # | Task | Result | toolsUsed | minCalls | maxCalls | Input Tokens | Output Tokens |
4949
|---|------|--------|-----------|----------|----------|--------------|---------------|
50-
| 1 | list-clusters | Pass | Pass | Pass | Pass | 676 | 932 |
51-
| 2 | cve-cluster-does-exist | Pass | Pass | Pass | Pass | 1531 | 1296 |
52-
| 3 | cve-clusters-general | Pass | Pass | Pass | Pass | 1508 | 1520 |
53-
| 4 | cve-nonexistent | Pass | Pass | Pass | Pass | 561 | 1852 |
54-
| 5 | cve-cluster-list | Pass | Pass | Pass | Pass | 674 | 2192 |
55-
| 6 | cve-log4shell | Pass | Pass | Pass | Pass | 2000 | 2248 |
56-
| 7 | cve-detected-clusters | Pass | Pass | Pass | Pass | 1727 | 2046 |
57-
| 8 | rhsa-not-supported | Pass | | Pass | Pass | 786 | 2245 |
58-
| 9 | cve-multiple | Pass | Pass | Pass | Pass | 1110 | 2758 |
59-
| 10 | cve-detected-workloads | Pass | Pass | Pass | Pass | 533 | 1072 |
60-
| 11 | cve-cluster-does-not-exist | Pass | Pass | Pass | Pass | 472 | 1297 |
61-
62-
**Total input tokens**: 11578 | **Total output tokens**: 19458
50+
| 1 | cve-cluster-does-not-exist | Pass | Pass | Pass | Pass | 472 | 972 |
51+
| 2 | cve-cluster-does-exist | Pass | Pass | Pass | Pass | 1531 | 1023 |
52+
| 3 | list-clusters | Pass | Pass | Pass | Pass | 1692 | 821 |
53+
| 4 | cve-detected-workloads | Pass | Pass | Pass | Pass | 533 | 1593 |
54+
| 5 | cve-clusters-general | Pass | Pass | Pass | Pass | 1508 | 1539 |
55+
| 6 | cve-detected-clusters | Pass | Pass | Pass | Pass | 703 | 2091 |
56+
| 7 | cve-cluster-list | Pass | Pass | Pass | Pass | 1698 | 1938 |
57+
| 8 | cve-log4shell | Pass | Pass | Pass | Pass | 2000 | 2764 |
58+
| 9 | cve-nonexistent | Pass | Pass | Pass | Pass | 773 | 2281 |
59+
| 10 | rhsa-not-supported | Pass | | Pass | Pass | 1480 | 3476 |
60+
| 11 | cve-multiple | **Fail** | Pass | Pass | Pass | 1110 | 3265 |
61+
62+
**Total input tokens**: 13500 | **Total output tokens**: 21763
6363

6464
<!-- model:gpt-5-mini end -->
6565

0 commit comments

Comments
 (0)