MT-Bench-101 (2/2): Add MT-Bench-101 golden-context evaluation runner

[ErlisLushtaku]

Summary

This is the second PR in the MT-Bench-101 stack. It adds the specialized runner for --task mt-bench-101.

Changes

• Generate each evaluated response from the benchmark's golden dialogue history.
• Grade each model independently on a 1 to 10 scale with the task-specific prompts and [[N]] score format. The judge temperature defaults to 0.6, as specified in the paper.
• Use the lowest turn score as the dialogue score, then report absolute summaries and derive JudgeArena pairwise preferences from the two models' scores. swap_mode does not apply because the judge never sees both answers together.
• Save the raw judge annotations, benchmark report, and run metadata.

Reference behavior

The prompt composition, dialogue formatting, second-turn task rules, and minimum-score aggregation follow the OpenCompass fork and Bai et al. (2024). For mathematical and general reasoning, this runner passes the current turn's golden answer as the reference. The OpenCompass loader instead passes the full dialogue object, which appears to be a bug.

The pairwise preferences are JudgeArena output derived after single-answer grading; they are not part of the original MT-Bench-101 protocol.

Stacked on #119.

[kargibora]

If reasoning enabled

 <think>Perhaps [[2]]...</think>
   Rating: [[9]]

would return 2 although model answers 9. I think its better to get the last one

[kargibora]

I think the overall implementation is good and correct. However I recommend we use some structure we have defined prior in the latest main branch.

1) Parsed object

We have a ParsedPreference class, which is the object for parsing the judge annotation. This can be labels such as A/B/Tie, it can be scores A=10, B=5. Currently the preference column is a requirement (because we always had a pairwise pipeline, but design-wise this should not be required). So what we can do is

   @dataclass(slots=True)
   class ParsedPreference:
       preference: float | None = None # --> can be None
       label: str | None = None
       scores: dict[str, float] = field(default_factory=dict)
       details: dict[str, object] = field(default_factory=dict)

Then our parser returns a list of

   None # Parsing failed.
   ParsedPreference(scores={"score": 8.0}) # Successfully parsed a single-model rating.
   ParsedPreference(preference=0.5, scores={"A": 8.0, "B": 8.0}) # Successfully derived a comparison.

a general object. We can of course design this much better - but it is better to unify them AND change it afterwards.

Then what we need is to create a Parser object

   class MTBench101ScoreParser(JudgeParser):
       name = "mt-bench-101-score"

       def parse_result(
           self,
           judge_completion: str,
           *,
           top_logprobs: dict[str, float] | None = None,
       ) -> ParsedPreference | None:
           ...
           return ParsedPreference(scores={"score": score})

or even better, just a sample-wise parser which we expect the judge annotation has a single score. Then we can use it in sample wise pipeline as well (unless we expect very specific parsing strategy due to the prompt of MTBench101, but this parser can also inherit the ScoreParser etc)

After registering this, what we need is

   parsed = parser.parse_result(judge_completion)

   row["judge_completion"] = judge_completion
   row["score"] = None if parsed is None else parsed.scores["score"] 

Within runner then we can collect all the required information.

2) Using Metrics

Second is metrics are abstraction for getting feed with a dataframe object (battles) and calculating some result from it (also reporting it). Thus we can implement

   class MTBench101AbsoluteScoreMetric:
       def calculate(self, battles: pd.DataFrame) -> dict[str, object]:
           # Read existing dialogue scores and coverage counts.
           # Return absolute-score summaries for both models.
           ...

       @staticmethod
       def render(result: dict[str, object]) -> str:
           ...

instead of

  "model_A_scores": summarize_mt_bench_101_absolute_scores(scored_a),
   "model_B_scores": summarize_mt_bench_101_absolute_scores(scored_b),

as these are not different than any metric we want to put. (We dont need to render them if we just want to store them. I think its better to unify these so our metadata is clean).

We would only need to supply

   scoring:
     metrics:
       - metric: pairwise_win_rate
         breakdown_by: [task, ability, domain]
       - metric: mt_bench_101_absolute_score

to the YAML file. This might be too specific, but if we implement an modular parser this can also be used in others (if necessary of course. If this metric do fancy things like weighting w.r.t some specific categories we dont need to make it modular)

What do you think?