benchmark()'s aa_state_path (resumable-evaluation) branch always crashes with RuntimeError after the AutoAttack run finishes

[shaurya416]

benchmark()'s aa_state_path (resumable-evaluation) branch always crashes with RuntimeError after the AutoAttack run finishes, because EvaluationState.robust_flags is a torch.bool tensor and .mean() cannot be called on it directly

robustbench/eval.py:117-120 (inside benchmark(), the aa_state_path is not None branch):

        else:
            aa_state = EvaluationState.from_disk(aa_state_path)
            assert aa_state.robust_flags is not None
            adv_accuracy = aa_state.robust_flags.mean().item()

aa_state_path lets a long AutoAttack evaluation be checkpointed and resumed (this is the pinned autoattack dependency's own EvaluationState/state_path mechanism, meant for exactly the long, interruption-prone runs where checkpointing matters most). run_standard_evaluation(..., state_path=aa_state_path) (the line just above this branch) already resumes and finishes the attacks, then writes the state to disk. This code path then reloads that state to compute the final accuracy.

EvaluationState.from_disk (in the pinned autoattack dependency, setup.py:20 pins it to commit a39220048b3c9f2cca9a4d3a54604793c68eca7e) always rebuilds robust_flags as a boolean tensor:

d["_robust_flags"] = torch.tensor(d["_robust_flags"], dtype=torch.bool)

In PyTorch, .mean() on a torch.bool tensor raises RuntimeError: mean(): could not infer output dtype. Input dtype must be either a floating point or complex dtype. Got: Bool, regardless of the tensor's contents or length. So robustbench/eval.py:120 crashes on every call that reaches it — a fully successful run, a partial one, any shape — not just on some inputs.

The same EvaluationState class already has the correct pattern one property below the one used here:

@property
def robust_accuracy(self) -> float:
    if self.robust_flags is None:
        raise ValueError("robust_flags is not set yet. Start the attack first.")
    if self.attacks_to_run - self.run_attacks:
        warnings.warn("You are checking `robust_accuracy` before all the attacks"
                      " have been run.")
    return self.robust_flags.float().mean().item()

robust_accuracy casts to float first (and warns if attacks are still pending) before averaging — robustbench/eval.py:120 does neither, and doesn't use this existing property at all.

Measured

Reproduced by fetching the pinned autoattack commit (a39220048b3c9f2cca9a4d3a54604793c68eca7e, confirmed via setup.py:20) and running EvaluationState.from_disk followed by the exact robustbench/eval.py:120 expression on real PyTorch 2.7.1, with no robustbench code imported:

Case	On-disk _robust_flags	aa_state.robust_flags.mean().item() (as eval.py:120 calls it)	aa_state.robust_flags.float().mean().item() (the existing robust_accuracy pattern)
Matches requested n_examples (10/10)	[1,1,1,0,1,0,1,1,0,1] (len 10)	RuntimeError: mean(): could not infer output dtype ... Got: Bool	0.7 (correct)
Empty	[]	same RuntimeError	nan
Any other length/content	any	same RuntimeError	works

Every case raises identically — the crash does not depend on the state file's length, content, or how it was produced. robustbench/eval.py:120 cannot ever return a value through this branch as written.

Consequence

Any benchmark() call that passes aa_state_path with an Linf or L2 threat model crashes after the entire (potentially GPU-hours-long) AutoAttack run has completed, at the very last step where the result is read back and reported. The resumable-evaluation feature this parameter exists for is unusable end to end: a user can never successfully get a result out of a checkpointed run through this code path, on any run, with any model. No adv_accuracy value, correct or wrong, is ever produced or written to the leaderboard JSON (update_json, reached only after this line) — the crash happens first, every time.

Suggested fix

Use the tensor's own .float() cast, or better, use the robust_accuracy property the same EvaluationState class already defines and exports for exactly this purpose:

adv_accuracy = aa_state.robust_accuracy

This also picks up the existing warning for the case where not all attacks have finished running yet.

Happy to open the PR.