Official Task Variations (2/3): Make Scoring more Composable by kargibora · Pull Request #113 · OpenEuroLLM/JudgeArena

kargibora · 2026-09-03T09:35:08Z

Problem

Pairwise, Elo, and MT-Bench used separate scoring and reporting paths. Metric parameters were also split between task definitions and runner code. Adding a metric could therefore require changes in several pipelines.

Elo also handled fitting, bootstrap aggregation, calibration, and rendering inside its runner. This made the scoring boundary difficult to reuse.

Changes

Add configured, stateless metric classes with calculate(battles) and render(result) methods.
Add the shared metric functions available_metrics(), build_metric(), build_metrics(), calculate_metrics(), and render_metrics().
Add declarative MetricSpec and ScoringSpec task configuration.
Pass metric parameters through MetricSpec.parameters.
Use one scoring boundary: battle dataframe to metric calculation to plain result dictionary.
Use the same metric construction and execution path for pairwise, Elo, and MT-Bench.
Move report models into judgearena/reports.py.
Move PairScore temperature calibration into judgearena/benchmarks/elo/calibration.py.
Make Bradley-Terry own fitting, bootstrap ratings, human-reference ratings, uncertainty, and MAE.
Preserve aligned instruction fields, such as category and language, when pairwise runners build battle rows.
Build MT-Bench rows with explicit model names and direct or reversed orientation.

Parsed preferences and Soft Elo

Soft Elo uses the continuous preference returned by the configured parser. Verdict, token-logprob, and custom parser preferences are not replaced with PairScore parsing.

Temperature adjustment and calibration remain specific to PairScore. PairScore calibration reads the raw A/B scores stored in JudgeAnnotation.parsed.

The scoring path is:

JudgeParser
└── ParsedPreference
    └── canonical numeric pref
        └── battle dataframe
            └── configured metrics
                └── plain result dictionaries

Metrics consume the numeric pref column. Structured parser evidence remains on the judge annotation and can be added to battle rows later when a metric has a defined use for it.

Task configuration

Metrics and their parameters are selected in task YAML:

scoring:
  metrics:
    - metric: pairwise_win_rate
      group_by: [category]
    - metric: bradley_terry
      parameters:
        n_bootstraps: 0
        soft: false

Runners use the same execution path:

metrics = build_metrics(protocol.scoring.metrics)
results = calculate_metrics(battles, metrics)

Runtime-only values, such as a random number generator, remain explicit and are not serialized as metric configuration.

A new metric needs a configured metric class, its calculation and rendering methods, one internal registry entry, and a task YAML entry. It does not need a separate scoring implementation in each runner.

Tests

The tests cover shared metric construction, configuration validation, pairwise and MT-Bench battle rows, hard and soft Bradley-Terry behavior, PairScore calibration, generic continuous parser preferences, report rendering, and runtime dependencies.

uv run ruff check judgearena tests
uv run pytest -q tests

Result: 312 passed.

 │ Production Python      │ 1,167 │ 563     │ 1,730   │
 ├────────────────────────┼───────┼─────────┼─────────┤
 │ Tests                  │ 910   │ 200     │ 1,110   │
 ├────────────────────────┼───────┼─────────┼─────────┤
 │ YAML and documentation │ 30    │ 10      │ 40      │
 ├────────────────────────┼───────┼─────────┼─────────┤
 │ Total                  │ 2,107 │ 773     │ 2,880   │
 └────────────────────────┴───────┴─────────┴─────────┘

Estimated production changes

 ┌──────────────────────────────┬───────────────┐
 │ Type of change               │ Estimated LoC │
 ├──────────────────────────────┼───────────────┤
 │ New implementation           │ 420–470       │
 ├──────────────────────────────┼───────────────┤
 │ Moved or extracted code      │ 270–320       │
 ├──────────────────────────────┼───────────────┤
 │ Small refactors and wrappers │ 150–190       │
 ├──────────────────────────────┼───────────────┤
 │ Removed implementation       │ 20–40         │
 └──────────────────────────────┴───────────────┘

The new implementation is mainly the composable metric framework and length-controlled win-rate calculation.

Most of the movement comes from taking calibration, Bradley–Terry calculation, and report rendering out of the Elo runner. These were moved into dedicated metric, calibration, and report modules. The runner changes are therefore mostly extraction and integration rather than new calculation code.

kargibora · 2026-09-03T09:40:48Z

@@ -0,0 +1,143 @@
+"""PairScore temperature calibration against human arena preferences."""


This script should be self-contained. It was sitting on the runner.py which was making it harder to read; so its a simple refactor and not a new capability. It's only purpose is to use some rows to calculate a temperature.

kargibora · 2026-09-03T09:41:33Z

        prefs, our_model_is_position_a, opponent_models, strict=True
    ):
-        if _is_nan_pref(pref) or pref == 0.5:
+        if _is_nan_pref(pref):


Making nan pref's tie does not make sense. Tie is basically a indicator that both models perform same. ELO calculation for example makes these model ELO's closer to each other if they are tie. Thus it should be marked as NaN correctly.

kargibora · 2026-09-03T09:43:09Z

 logger = get_logger(__name__)


-class EloReport(Report):


Moved into report.py. Metric reporting is now contained within the metric, so they are re-usable. Report is just collection of key:values which we want to add to the result.

kargibora · 2026-09-03T09:44:17Z

    # rebuilt from the (revision-pinned) arena, not persisted per run.
    df_arena = arena_anchor_battles(df_arena_all)

-    df_results = pd.concat([df_llm_judge, df_arena], ignore_index=True)


No need for all the calibration code within the runner.

kargibora · 2026-09-03T09:47:49Z

-        battle_counts[row["model_b"]] = battle_counts.get(row["model_b"], 0) + 1
-
-    pref_col = "pref" if use_soft else "pref_hard"
-    bootstrap_ratings: list[dict[str, float]] = []


These are metrics. Not related to the runner.py. Thus they are moved into the classes which is going to use, in this case, BradleyTerryMetric which is basically ELO.

kargibora · 2026-09-03T09:50:14Z

    ]
-    elo_mean = (
-        float(np.mean(model_rating_values)) if model_rating_values else float("nan")
+    focal_is_a = pd.Series(our_model_is_position_a, dtype="bool")


Normalizes some column. I want to make the boundaries more clear so datasets are interchangable at the core (basically arena and pairwise pipelines almost same. The dataset includes a prompt and two responses, in arena it is between each model and in pairwise, it is between some baseline and other models. For now as we already normalized ELO datasets, this will allow us to use pairwise metrics easily (lenght contorlled etc).

kargibora · 2026-09-03T09:50:53Z

    )
-    elo_std = (
-        float(np.std(model_rating_values)) if model_rating_values else float("nan")
+    metric_results = calculate_metrics(


Pay-off. This will handle any metrics. No need to change anything in the pipeline when we add newer metric.

kargibora · 2026-09-03T09:51:20Z

+    baseline_model: str | None = None
+    soft: bool = True
+
+    def __post_init__(self) -> None:


This allow us to construct the class without setting some default values in the init.

kargibora · 2026-09-03T09:55:24Z

+    return bool(positive.max() <= negative.min() or negative.max() <= positive.min())
+
+
+def _fit_length_model(


Estimate the effect of the length differences in predicting the winrate.

kargibora · 2026-09-03T09:56:55Z

+        "judge_max_out_tokens",
+        *instructions_df.columns,
+    }
+    missing_groups = sorted(


Allow us to group_by orientation for example (shows normal and swapped winrates), or any category/lang exist in the task (allow us to also report winrates per lang easily).

kargibora · 2026-09-03T09:58:50Z

    )


-class Report(BaseModel, abc.ABC):


Moved into report.py

ErlisLushtaku · 2026-09-08T23:04:54Z

+    """One named calculation over a battle dataframe."""
+
+    metric: str = Field(min_length=1)
+    group_by: tuple[str, ...] = ()


group_by: [category, turn] normally means one grouping by the (category, turn) unique combination, however the executor calculates a separate breakdown for each field. Should we rename this to something like breakdown_by so isn't confused?

ErlisLushtaku

Just a small concern regarding the semantics of group_by.

kargibora changed the title ~~Refactor/composable pairwise metrics~~ Official Task Variation (2/n): Make Scoring more Composable Sep 3, 2026

kargibora commented Sep 3, 2026

View reviewed changes

Comment thread judgearena/utils/eval.py

)

class Report(BaseModel, abc.ABC):

kargibora Sep 3, 2026

Copy link
Copy Markdown

Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved into report.py

kargibora changed the title ~~Official Task Variation (2/n): Make Scoring more Composable~~ Official Task Variation (2/3): Make Scoring more Composable Sep 3, 2026

kargibora changed the title ~~Official Task Variation (2/3): Make Scoring more Composable~~ Official Task Variations (2/3): Make Scoring more Composable Sep 3, 2026

kargibora requested a review from ErlisLushtaku September 3, 2026 10:19

kargibora mentioned this pull request Sep 4, 2026

Official Task Variations (1/3): Refactor Prompt Parsing into Reusable Components #108

Open

kargibora force-pushed the refactor/composable-pairwise-metrics branch from fb02860 to cdee3d8 Compare September 4, 2026 09:48

kargibora mentioned this pull request Sep 4, 2026

Official Task Variations (3/3): Add Official AlpacaEval, Arena-Hard, and MT-Bench Tasks #115

Open

kargibora force-pushed the refactor/composable-pairwise-metrics branch from cdee3d8 to 6bb0a65 Compare September 4, 2026 11:42

ErlisLushtaku reviewed Sep 8, 2026

View reviewed changes

Comment thread judgearena/reports.py

ErlisLushtaku reviewed Sep 8, 2026

View reviewed changes

ErlisLushtaku approved these changes Sep 9, 2026

View reviewed changes

kargibora added 6 commits September 10, 2026 11:41

refactor: make pairwise metrics composable

b7bfa49

refactor: share metrics across battle pipelines

0d77f4a

fix: complete shared scoring boundaries

01b5f67

fix: scope Elo temperature calibration to PairScore

1402d75

fix: preserve pairwise metric contracts

a876f34

fix: preserve metric contracts and clarify breakdown configuration

162bf13

kargibora force-pushed the refactor/composable-pairwise-metrics branch from f4d9e77 to 162bf13 Compare September 10, 2026 09:55

		@@ -0,0 +1,143 @@
		"""PairScore temperature calibration against human arena preferences."""

		return bool(positive.max() <= negative.min() or negative.max() <= positive.min())


		def _fit_length_model(

Uh oh!

Conversation

kargibora commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Parsed preferences and Soft Elo

Task configuration

Tests

Estimated production changes

Uh oh!

kargibora Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kargibora Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ErlisLushtaku Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ErlisLushtaku left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kargibora commented Sep 3, 2026 •

edited

Loading

kargibora Sep 3, 2026 •

edited

Loading

kargibora Sep 3, 2026 •

edited

Loading

ErlisLushtaku Sep 8, 2026 •

edited

Loading