Repository navigation
feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track - #214
Conversation
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi eval native-import`
+ argument `log`
+ option `--answers-out`
+ option `--eval-out`
+ option `--help/-h`
+ added command `dokimi eval reference`
+ option `--answers`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--no-judge`
+ option `--output/-o`
+ option `--questions`
+ added command `dokimi questions import`
+ argument `path`
+ option `--format`
+ option `--grader`
+ option `--help/-h`
+ option `--output/-o`
+ option `--scorer`
+ option `--task`Updated on every push; the |
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #214 +/- ##
==========================================
- Coverage 92.94% 92.68% -0.26%
==========================================
Files 241 252 +11
Lines 30647 31899 +1252
Branches 5058 5321 +263
==========================================
+ Hits 28485 29567 +1082
- Misses 1460 1563 +103
- Partials 702 769 +67
🚀 New features to boost your workflow:
|
|
Ran the full test suite, ruff, mypy and bandit on the head plus the commits below. Re-ran each finding's input through the CLI. Open:
Pushed:
|
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 00b6731. The adapters and the check types are a sound design, and the verbatim baseline path works. Several graders are imported wrongly, though, and each one reports a wrong pass or drops the set. I'd fix the inline bugs before merge.
What I ran, on a Linux review host at 00b6731:
make ci-local: 26 passed, 3 failed, 7 skipped. 7,998 tests passed, changed-line coverage 91%. The three FAILs are the gitleaks install, the gitleaks scan andpip-audit, which fail on the lock files and the tool download, not on this PR's code. Running gitleaks directly over the PR's 10 commits finds no leaks.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the new importers and checks: 41 mutants of
reference_checks.py,reference_run.pyand the answerer, 18 of which pass both new test files; 45 mutants of the importers, 26 of which passtest_answer_imports.py.
I agree with @xaleryb that arms run ignores prompt_style: verbatim. New evidence: grep verbatim dokimi/eval/arm_runner.py returns nothing, and arm_runner.py:322 formats ANSWER_PROMPT_PLAIN, so docs/answer-formats.md:19 and :37 (arms run, and "a context arm puts the docs first and the prompt after them") describe behaviour that does not exist.
Findings that cannot be anchored inline:
- The PR is stacked on #210, which
AGENTS.mdforbids.git log origin/main..HEADlists 10 commits. Three are #210's (f4ce883,b9dd1a8,e4b4e80). They account for 76 of the 102 changed files and 4,854 of the 6,423 added lines. The description says "Review only the top commit". Rebase onto the base once #210 merges, or land #210 first. Thetask-format-detectfragment also announcestasks inspectagain, and theanswer-formatsfragment lists "task_inspect.v1 accepts thequestions importconsumer" as a change to a schema that is new in this diff. validate_gradingaccepts blocks that later abort the whole scoring run.reference_checks.py:316-330checks the type and the values only. A regex that does not compile,location: middle,mode: ALL,weight: "high"and a judge check with no criterion all pass it, then raise inside the thread pool (:163,:242,:269,:292,:297).eval referenceexits 2 with no output file, after judge calls for other questions were paid for, and the message names no question id. Validate regex compilation,location,mode, numericweightand judge criterion invalidate_grading.- A regex check has no backtracking guard.
reference_checks.py:267runsre.search(str(v), answer)with a pattern from an imported config.(a+)+$against"a" * 26 + "b"takes 3.05 s, and the cost doubles per added character. A promptfooregexassert with a nested quantifier hangseval referencein a worker thread. - The new tests let real regressions through. Mutation check: switching the with-docs verbatim prompt to the wrapped template, disabling the
--doc-source nonebranch (answers.py:83),>instead of>=on the threshold (reference_checks.py:108), dropping judgenegate, regexignore_case, makingchoicea subset test, one-wayfuzzy, and swapping numericbegin/endall pass the full new suites. On the importer side, the OpenAIincludesignore_casedefault,contains-allmode,icontains,starts-with, the Inspectmatchdefaults and the choice-target mapping do too. Add a test per mapping that asserts the check dict for one real row of each format, and a grading test at the exact threshold.
Checked, no issues
- Schemas:
questions.v2.jsononly adds optionalprompt_styleandgrading.task_inspect.v1is new and registered inSCHEMA_VERSIONS.eval referenceon an arms file keeps thewith_docsandwithout_docskeys and validates againsteval.v1. - Docs and CLI: the flags in
answer-formats.mdand the changelog match--helpforquestions import,eval reference,eval native-importandanswers generate --doc-source none.mkdocs build --strictpasses with the nav entry. - Path escape: an Inspect
json_datasetsymlink or../path and a promptfoofile://prompt outside the config directory are refused. Tree scans skip symlinked children. - Judge failures: a failing judge call gives a
Nonescore, which stays unscored. Negation of a 0.5 split stays not passed.
|
Merged main after #210 landed (e6d30d1): conflicts merged three-way against e4b4e80, the commit this branch was built on. The PR's own diff against main is the same 30 files, line for line; the consumers enum and the docs table now carry both On top, besides the inline fixes:
Full suite, ruff, mypy and bandit pass on Linux at e17a773. Still for the author: regex backtracking (Python |
|
A proposal for the regex backtracking point, the one open item here with a security angle (a set's Python If that works for you, I can push it here. |
|
Two commits on top of your latest push: 6598cd9: regex checks run with a timeout, as proposed above. The 224aa36: a CSV
The new Full suite, ruff, mypy, bandit and deptry pass on Linux at 224aa36. Not changed, both on the YAML path too: a bare |
|
Pushed b7d7bfd for the first "not changed" item above: a text assert ( |
xaleryb
left a comment
xaleryb
left a comment
There was a problem hiding this comment.
Approved at b7d7bfd: every review point is fixed at this head (a CSV header with a leading space is the one minor item left, named in the thread), the full suite, ruff, mypy, bandit and deptry pass on Linux, and CI is green.
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at b7d7bfd. All six of my earlier threads are fixed. The new checks and importers hold up, but two import paths still grade or crash wrongly, and the new tests let several mutants through. Details are inline.
What I ran, on a Linux review host at b7d7bfd:
make ci-local: 28 passed, 1 failed, 7 skipped. 8,423 tests, changed-line coverage 91%. The one FAIL istest_a_regex_that_backtracks_past_the_timeout_is_unscored[True], which asserts the timed-out check returns in under 2 s; it took 2.9 s with other jobs running and passes alone (40 passed for the file). See the nit inline.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the new checks, importers and answerer: 88 mutants, 10 of which pass the new test files.
Findings that cannot be anchored inline:
- Nothing pins that a question without
prompt_style: verbatimkeeps the expert wrapper, or thatanswers generate --doc-source noneworks. Mutation check:verbatim = Truefor every question inanswerer.py(around line 240) and removing thenonebranch inanswers.py(around line 83) both pass the 167 answerer, arm-runner and answers tests. The first would strip the wrapper from every existing question set. Add oneAnswerertest with a plain question that asserts the wrapped prompt, and oneanswers generate --doc-source nonetest. - Four Inspect and promptfoo mappings that the docs describe are not pinned. These mutants pass the suite:
multiple_correctnot passed through thejson_datasetpath,csv_datasetaccepted, Inspectincludesdropping punctuation handling, a negated judge assert (not-llm-rubric) let through as a judge check without negation, and adefaultTestthreshold no longer inherited. Thenot-llm-rubrictounsupportedmapping is also missing fromanswer-formats.md.
Still open from my earlier review: none.
Checked, no issues
- Earlier threads:
joke_fruits.jsonlwith a model-graded spec imports; a non-JSONjson_matchideal becomesunsupported; Inspectpattern()is refused; themultiple_correctprompt uses the multiple-answer template; afallback_noneanswer is unscored anddeltaisNone; malformedexpectations,filesandvarsend asErrorwith exit 2. - Regex timeout: it holds inside the worker pool (8 timing-out grades on 4 threads returned in 1.8 s) and
negateon a timed-out check stays unscored. - Docs and CLI: the flags, example commands and mapping rows in
answer-formats.mdmatch the code;mkdocs buildpasses with the nav entry;max_docs32 matches the file count;questions.v2andtask_inspect.v1are only widened. - Deps:
regexis a direct dependency inpyproject.tomlandrequirements.txt, and the lock already pins a satisfying version; deptry is clean.
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 56b1981, which fix the three nits from my review:
7c2abfe:dokimi/tests/test_reference_checks.py:100: the timing assert now allows 5 * REGEX_TIMEOUT.b966e5e:dokimi/docs/answer-formats.md:61: the checklist link now points to benchmarking-and-comparison.md.56b1981:dokimi/docs/answer-formats.md:60: withcontains, the answer may be any JSON value, or else it is the first JSON object or array anywhere in it.
make ci-local at 56b1981: 29 passed, 0 failed, 7 skipped.
Not approving yet. Still open: the two bugs and the five risks from my review.
…oo sets on the answers track questions import translates the answer-only formats that tasks inspect detects into questions.v2 records with a format-neutral grading block. eval reference scores answers or arms output against those checks: deterministic checks need no model, judge checks use the checklist judge in both polarities. eval native-import reads Inspect json logs and promptfoo outputs so native verdicts sit next to dokimi's. Inspect @task files are parsed with ast, never imported or run. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The skill an evals.json belongs to (skill_name, else the <skill>/evals/ directory) is each question's expected_skill, so the imported set also scores skill selection on a skill-catalog: arm. Without either, the import report notes it. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…o-judge arms.v1 requires judge_model and judge_provider strings next to evaluations, so eval reference --no-judge on an arms run file failed validation and wrote nothing. Stamp both with the scorer name, as each score block already does. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
An Inspect sample that errored has an empty completion and no score; it was stored as a plain empty answer, so eval reference graded it as wrong while Inspect left it unscored. The error now goes on the answer, which grade() already treats as unscored. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
A partial FieldSpec dropped the id and choices columns Inspect reads by default, a CSV choices string stayed one choice instead of read_choices' comma split, and a sample whose target matched no choice was written with an empty choice check that made eval reference refuse the whole set; it is now skipped with a lossy note. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
promptfoo reads a .txt prompt file as several prompts separated by lines that are ---; the importer sent the whole file as one prompt, so the questions joined both prompts and lost promptfoo's <test>.p<prompt> ids. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
ModelBasedClassify samples often carry only input (registry/data/test_modelgraded/joke_fruits.jsonl), and every such row was skipped, so the set imported as zero questions. Only the deterministic templates need an ideal now; a model-graded row with neither an ideal nor spec criteria gets an unsupported check, since the spec's own prompt is not read. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
detect() reports a second format within its margin in report.alternatives, and import_path took the first one anyway: joke_fruits.jsonl alone imported as Inspect samples, with match checks that had no values. The import now stops and names the candidates; --format picks one. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…eference With no retrieved docs the answerer writes "[FALLBACK: No relevant documentation found]" (doc_source fallback_none or none) as the context-arm answer, and eval reference graded it: a negated check passed it and the delta was corrupted. It now counts as unscored, like an errored answer. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…gex search Inspect's pattern scorer compares the extracted group with the target; the import emitted a regex check that dropped the target, so ANSWER: 5 passed for target 4, and --scorer pattern on a samples file had no pattern at all and matched every answer. pattern() now imports as unsupported, and --scorer pattern on a samples file is refused. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…etters
Only cot was read from multiple_choice(), so multiple_correct=True samples were asked with the single-answer template ('ANSWER: $LETTER') while the choice check needs the whole target set. They now get Inspect's multiple-answer template.
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
questions import catches ValueError only, so an inspect-task file that does not parse, skill-creator expectations: null, a promptfoo weight: null or vars: file://... gave a traceback; expectations or files given as a string were split into characters, and a truncated file:// JSON prompt became the question "None". Each is now a ValueError naming the file (null expectations read as none). Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
answer-formats.md says arms run answers an imported set, but the arm runner always formatted ANSWER_PROMPT_PLAIN or the context suffix around the question, so an exact-match or ANSWER: X grader saw a reply to a different prompt than answers generate sends. A verbatim question is now the whole prompt, after the context block on a context arm. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
… through 13 mutants from the review passed the suites: the with-docs verbatim prompt, the threshold at its exact value, judge negate, regex ignore_case, choice as a subset, one-way fuzzy, swapped numeric begin/end, and the OpenAI includes, promptfoo contains-all/icontains/starts-with, Inspect match default and choice-text mappings. One test per side now asserts each, and each mutant fails it. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…ring A __expected cell was split on its first colon only for a fixed list of types. javascript: became a literal equals check, not-contains: and grade: did the same, and contains-any / contains-all kept a comma list as one value. Cells now read as promptfoo's assertionFromString does: code is unsupported (not run), grade: is an llm-rubric judge, a not- prefix negates, and a *-any / *-all value splits on unescaped commas. Red test: the promptfoo CSV test failed on the previous code. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
With no judge model, or a failed judge call, a judge check has no verdict. A match check next to it still scored the record, so the record was counted as a pass with no judged check behind it. The summary now has judge_unscored: the number of counted records with a judged check that has no verdict. The verdict itself is unchanged. Red test: the tally test failed on the previous code (no such key). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ed, not a pass Every answer contains the empty string, so match [""] passed whatever the answer said. A CSV cell "contains-any:" or a YAML value '' now imports as unsupported and is not counted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… unscored Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…es and map it like a YAML assert The cell parser split on the first colon for a short list of types and fell back to equals for the rest. is-json, factuality, llm-rubric(0.5):, similar and fn: became literal equals checks, a *-any / *-all list kept the spaces after its commas and its empty fields, quoted fields were not read, and a blank cell was an equals check on whitespace. promptfoo_csv.py ports assertionFromString (src/csv.ts) and parseCommaSeparatedValues at promptfoo b4c2404a: the trimmed cell, its legacy forms, the [not-]type(threshold):value regex over promptfoo's full assertion type list, quoted list fields, equals otherwise. The assert it returns goes through the same _check as a YAML assert. A malformed quoted list field is an import error, as in promptfoo. Tests mirror promptfoo's test/csv.test.ts and pin, per cell, the promptfoo assert and that its CSV and YAML forms give the same check. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
… not graded _check turned a missing value into the string "None": a bare regex assert passed any answer containing None, and a bare contains failed every answer. YAML and CSV now both report it as unsupported. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
56b1981 to
804c83b
Compare
- A blank ideal or target imports as unsupported, not as a check that passes every answer. A promptfoo assert with a blank value does too. - eval reference refuses a match value that is blank once stripped, an empty regex pattern and a negative weight, before any run starts. Whitespace-only regex and unstripped match values stay valid. - A regex check stops at the first pattern that settles its mode, so a later pattern past the limit no longer unscores a decided check. - promptfoo threshold rejects booleans and non-finite numbers. - OpenAI json_match serialises a structured ideal as JSON, not repr. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
The five threads are fixed: the blank values, thresholds, weights and the timed-out regex are all handled. The blank-value guard has a gap that still passes every answer, so I would not merge yet. Three points are worth fixing before merge.
What I ran, on a Linux review host at 804c83b:
make ci-local: 29 passed, 0 failed, 7 skipped. Test suite and changed-line coverage pass; overall coverage 91%.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over
reference_checks.pyandreference_run.py: 24 mutants, 5 of which pass the three new test files. The five earlier mutants (anyforallin the pass decision, theweight > 0filter,unjudgedon any unscored check,judge_unscoredon the all-unscored path, a per-pattern regex timeout) now fail the suite.
Checked, no issues
- Threshold and weight at import: promptfoo
threshold: [1],true,.nan,.infandweight: [1]end asError:with exit 2. - Empty
ideal/target:""," ",[]andnullgiveunsupportedfor the OpenAImatch,includes,fuzzy_matchandjson_matchtemplates and for the Inspectmatch,includes,exactand model-graded scorers. validate_grading:weight: nullandtype: ["match"]return problem strings. Over 1,700 accepted check blocks,grade_checkraised on none.- Timed-out regex:
(a|aa)+$on 40aplus!next to a passingmatchstill gradespassed: true, but the check showspassed: nullandcheck_unscored: 1appears in the summary. The artifact validates againsteval.v1andarms.v1.
With the default ignore_punctuation, a match value of "..." or "!!" is "" once normalised, and every answer contains "". blank_value now applies the check's own punctuation normalisation, so validate_grading reports the value and eval reference refuses the block. The Inspect importer drops such a target for the match scorer, leaving the sample unsupported when none is left. OpenAI evals and promptfoo keep punctuation in every match they build, so a punctuation-only value there is a real check; tests pin both. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
float() accepted weight: -1, .nan and .inf, so the file imported and only eval reference refused it later. The weight now goes through the same finite-number check as the test threshold, and a negative weight is refused at import too. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…olds An all-mode regex whose first pattern matches must still try the next one, and validate_grading must refuse a nan or inf weight or threshold. Neither was covered: replacing the stop condition with "True in hits", or dropping math.isfinite from _is_number, passed the suite. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 1a9d30a. The three open threads are fixed: blank_value now applies the check's own normalisation, the promptfoo weight goes through the same finite-number helper as the threshold, and the all-mode and finite-number mutants fail the new tests. Two gaps of the same kind as the blank-value threads are left, one in the promptfoo importer and one in the Inspect importer. Both are inline.
What I ran, on a Linux review host at 1a9d30a:
make ci-local: 29 passed, 0 failed, 7 skipped. 8,621 tests, changed-line coverage check passes.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the delta: 20 mutants of
blank_value, the Inspectnormand the promptfoo_finite/weight check, 3 of which passtest_answer_imports.pyandtest_reference_checks.py(thekind == "match"guard on the punctuation strip,weight < 0to<= 0, and the Inspectexactcase below). Your two earlier mutants (if (True in hits):and nomath.isfinite) now fail 2 and 5 tests.
Checked, no issues
blank_valuevsmatch_text: over every code point plus punctuation and whitespace combinations, under all combinations ofignore_punctuation,stripandignore_case, the two agree on whether a value is found in every answer (locationany).- Importers:
...,!!,?,.and["!!", "??"]giveunsupportedfor Inspectmatch;["...", "x"]keepsx; OpenAImatch/includesand promptfoocontains/equals/starts-withkeep...as a check that only an answer containing it passes. - Weights: promptfoo
weightof-1,.nan,.inf,true,nulland[1]end asError:with exit 2.
| if base in _CONTAINS or base in ("equals", "starts-with", "regex"): | ||
| # No value to compare (`regex` would search for "None"), or a blank one, which | ||
| # eval reference refuses: left unsupported instead. | ||
| if value is None or any(not str(v).strip() for v in as_list(value)): |
There was a problem hiding this comment.
A promptfoo assert with a blank judge value, or an empty contains-any list, imports cleanly and then eval reference refuses the whole file.
The guard at :48 covers the text asserts only, and any([]) is False for an empty list. _other_check (:76) builds the judge check from str(value or ""), so a blank value leaves it with no criterion or reference.
{type: llm-rubric, value: ""}imports as{'type': 'judge', 'native': 'llm-rubric'};validate_gradingreturns['check 0: a judge check needs a criterion or a reference'].{type: factuality, value: " "}imports as{'type': 'judge', 'reference': ' ', ...}, same problem.{type: contains-any, value: []}imports as{'type': 'match', 'values': [], ...};validate_gradingreturns['check 0: match has no values'].
One such assert aborts eval reference for every other question in the file, which is the failure the blank-value threads were about. docs/answer-formats.md says "A promptfoo assert whose value is blank (whitespace included) imports as unsupported too", which does not hold for these.
Suggestion: in _text_check, also return None when not as_list(value); in _other_check, return None for a judged assert when not str(value or "").strip(). Add the three asserts above next to test_promptfoo_blank_values_are_unsupported_not_refused_later.
| def _checks(scorer: str, kwargs: Mapping[str, Any], targets: list[str], imp: Imported) -> list[dict[str, Any]]: | ||
| # A target the scorer normalises to nothing ("", or "..." for `match`) is in every answer, so it | ||
| # is dropped; with none left nothing is graded. | ||
| norm = {"ignore_punctuation": scorer == "match"} |
There was a problem hiding this comment.
An Inspect exact target such as ... imports as a check that no answer can pass.
norm strips punctuation only for match, so for exact the target ... is kept. The grader's exact squad-normalises it to "", which never equals a non-empty value, so {'type': 'exact', 'values': ['...']} validates and grades False for zzz, ... and ?. It fails safe, but it silently scores 0 and is not marked unsupported.
Suggestion: norm = {"ignore_punctuation": scorer in ("match", "exact")}, so the target is dropped and the sample becomes unsupported. Add a ... row with scorer="exact" to test_inspect_punctuation_only_target_is_unsupported_for_match_only.
vchamarthi
left a comment
vchamarthi
left a comment
There was a problem hiding this comment.
No blockers. 1 non-blocking comment.
No blockers; the punctuation-only match value and promptfoo weight/threshold fixes from earlier review are in and tested. One non-blocking question on JSON equality remains.
CI at send time: 0 failing, 0 pending.
| ok, parsed = parse_json(answer, contains=bool(check.get("contains"))) | ||
| if not ok: | ||
| return False, "not JSON" | ||
| return (not values or any(parsed == v for v in values)), "" |
There was a problem hiding this comment.
Question: Is the json check meant to treat true and 1 (and false and 0) as equal? parsed == v accepts both, which differs from promptfoo equals on an object value.
dokimi/dokimi/eval/reference_checks.py:309: return (not values or any(parsed == v for v in values)), "". In Python bool is a subclass of int, so {"ok": true} == {"ok": 1} and [true] == [1] are both True, and 1.0 == 1 is True. A target of {"ok": 1} therefore passes the answer {"ok": true}. promptfoo compares in JavaScript, where true and 1 are different values. If strictness matters here, compare after a recursive type-aware walk, or document that bool and number are interchangeable in json checks.
test_answers_command.py: keeps main's _stub_generate and write-error tests (intel#236) and this branch's --doc-source none test. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…ted (#266) Settles two review points of #214 (task-format importers). promptfoo: a judged assert (llm-rubric, factuality, g-eval, ...) whose value is missing or blank after strip() imported as a judge check with no criterion or reference, and a contains*/icontains* assert with an empty list imported as a match check with no values. validate_grading refuses both, so one such assert made `eval reference` refuse the whole file. _other_check now returns None for a blank judged value and _text_check returns None for an empty contains* list, so both import as `unsupported` with the "not translated" note, like a blank text value already did. Inspect: the grader's `exact` check SQuAD-normalises the target, so a target such as "..." or "the" becomes "" and equals no answer; it was kept as a check that always fails. _checks now drops such a target for the `exact` scorer, so a sample left with no target is `unsupported`. `includes` and the model-graded scorers are unchanged. docs/answer-formats.md states the new rules; changelog fragment (Fixed). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Stacked on #210 (
tasks inspect). Review only the top commit until #210 merges.Summary
This PR runs the answer-only formats that
tasks inspectdetects on the existing questions, answers and judge track. Each format gets an import adapter into one canonical record, so there is no parallel runner.What does not carry over is listed in
docs/answer-formats.md. The main points:multiple_choice/system_messageare not reproduced.fact, promptfoollm-rubric) all become the yes/no checklist judge.similar,costandlatencybecomeunsupported, which is reported and not counted.Evidence
Haiku 4.5 on Bedrock, at most 20 samples per set. Native tools were pinned: inspect-ai 0.3.283 and promptfoo 0.125.0. OpenAI evals graders were reproduced from upstream 8eac7a7d.
The deterministic graders agree on 100% of the same completions. Every disagreement is a judge verdict. A split verdict (0.5) fails here, which is stricter than Inspect
P.Tests:
tasks inspect born_first.jsonlgaveconsumers: [none-yet].[questions import].tests/test_answer_imports.py(20 tests) andtests/test_reference_checks.py(28 tests). Both are offline and use a fake judge.make ci-localpasses 28 checks and fails 1: pip-audit flags filelock and setuptools in uv.lock, which this PR does not change.Merge Danger
Door: two-way. The new commands and modules are additive. Two schemas are widened: questions.v2 gains
gradingandprompt_style, and task_inspect.v1 accepts thequestions importconsumer.Blast Radius: small. The answerer changes only for questions with
prompt_style: verbatim.answers generate --doc-source noneis a new opt-in flag.benchmarks/spec.v1.yamlmax_docsgoes from 31 to 32 for the new doc.Decisions for reviewers:
expected_skill?--doc-source noneacceptable as a flag?🤖 Generated with Claude Code