Repository navigation
Conversation
added 21 commits
October 8, 2026 17:10
The verifier crashed on every task with
"ModuleNotFoundError: No module named 'scoring'". It looked for
_evaluate_gates_and_scores on ai4sci_bench.core.task, which does not
define it, then fell back to a bare `import scoring` — a leftover from the
old adapter's vendored module that the native converter does not ship.
Fixing the import alone was not enough: the whole calling convention was
wrong, and each error would have surfaced only after the previous one was
fixed.
1. import from ai4sci_bench.core.scoring (the new shared core)
2. prompt_level is keyword-only, not the 5th positional argument
3. the function returns a 5-tuple, not a dict, so every result.get(...)
would have raised AttributeError
4. max_score is not part of that tuple; take it from scoring_max_score()
and normalize_task_score(), exactly as ai4sci_bench.benchflow does
5. prompt_level values are lowercase ("b1"); drop the .upper()
Reward normalization no longer applies score_divisor a second time.
normalize_task_score() already divides score AND maximum by it, so the
ratio is divisor-invariant; the previous code under-reported every task
with a divisor != 1.0 (ising, at 1.05) by that factor. The divisor is
still recorded in reward.json for provenance.
Fail-closed, per the repo's evaluator-failure policy: an evaluator fault
must never be recorded as a 0 score. scorer_internal_error lives in each
ScoreDetail.details, not at the top level, so it is now detected there and
reported with the offending scorer names. Exceptions escaping the scoring
call are treated as evaluator faults rather than being converted to a
0-score submission failure, and a non-positive max_score is an evaluator
config fault too. A genuine submission failure (missing or malformed
outputs) still scores low and exits 0.
evaluator_files.json ships core/scoring.py and pins the bundle to
apexin-ai/ASI-Bench f13175a (#33), the squash-merged commit that
introduces it; four other files' hashes move with it. All 11 copy entries
were hashed against that revision to confirm the manifest matches.
Verified on the host by materializing the bundle from the manifest and
driving score_entry.main() through a synthetic /verifier layout:
complete outputs -> 1.0; one output missing -> 0.6 (exit 0); divisor 1.05
-> unchanged 1.0; unregistered scorer, seed 42, and zero total weight ->
exit 1 with no reward and an asi_error.json; failed hard gate -> 0.0 with
exit 0. The old file reproduces the reported crash on the same input.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds native ASI-Bench seed31415 support to BenchFlow. The adapter converts pinned public ASI-Bench instances into BenchFlow task packages, runs the ASI scorer in the verifier, persists scored prediction artifacts, and provides ASI-style offline summarization.
This is a draft for collaborative validation. It is not ready to claim benchmark-wide scoring parity or complete coverage.
What is included
benchmarks/asi-bench/converter, pinned source/evaluator manifests, verifier template, task descriptor, CLI entrypoint, and summarizer.Current validation
insufficient-evidence.Remaining before ready for review
Scope is seed31415 local scoring only; seed42 and unsupported scorer/task classes remain excluded as documented.