Proposal: a multilingual judge-ELO leaderboard

[kargibora]

This issue is related to the discussion we had, in which we want to create a (i) leaderboard that shows (ii) multilingaul performances of a pool of model in which (iii) can submit a new model quite easily.

Summary

Add a leaderboard to JudgeArena where anyone can submit a model with one command and get it ranked against a shared pool of models across many languages, judged by an LLM. The ranking uses soft-ELO calibrated against human preferences (can be changed easily with a flag), so a single judge instrument produces ratings that track human-ELO closely. The result will be published as a Hugging Face dataset and rendered by a modern Gradio Space.

Motivation

We already have the pieces — generation, LLM-as-judge, ELO estimation — but no standing, comparable ranking. Today comparing two models means running a bespoke ELO job and eyeballing the numbers; there's no frozen reference, no multilingual coverage, and no shared place to see where a new model lands. A leaderboard turns "run an eval" into "submit a model and see its rank," and makes results comparable across submissions and over time.

The idea

One command to submit a model. It is evaluated against a frozen pool of multilingual prompts (target: ≥15 languages, ~100 prompts each), battled against the existing models with an LLM judge (vLLM or OpenRouter), and the leaderboard updates with its rank, per-language breakdown, and diagnostics.

How it works

• Frozen panel. We curate a fixed set of judge-scored anchor battles (a battles.parquet + a panel.json describing it) from an arena dataset (e.g. LMArena/Comparia). This panel is the shared measuring stick — every submission is scored against the same instrument, so all submissions are comparable through the shared anchors (standard arena methodology; submissions don't battle each other by default).
• Quality gating. Per-language judge↔human agreement (Cohen's κ) decides which languages are trustworthy enough to include; low-κ languages are dropped or flagged.
• Publish + render. A publish step computes a self-contained bundle (rankings, per-language tables, calibration points, κ, win-rate and head-to-head matrices) and uploads it as a HF dataset; a thin Gradio Space renders the precomputed bundle (no eval stack imported into the Space). The rendering plots can be also a seperate package and we can push this to a seperate repo.

Scope (sub-projects)

1. Eval contract & panel — curation, frozen panel artifact, κ gating, soft-ELO calibration, ResultRecord.
2. CLIs — curate (build panel), submit (score a model, with --tag for repeated runs of the same model), board (render).
3. HF publish + Gradio Space — the bundle format, the dataset upload, and a thin, modern Space.
4. Insights & modern UI — human-ELO vs judge-ELO calibration plot with CI, per-language κ bar, win-rate-by-language heatmap, head-to-head model×model matrix, score distributions, examples viewer.

What's prototyped already

Most of is built on the feat/live-leaderboard (library + CLIs) and feat/leaderboard-hf-space (publish + Space + insights + head-to-head) branches, validated end-to-end with a small deepseek-v3.2 / OpenRouter smoke run. I will try to extend this to multilingual results as a test after getting cluster access

Open questions

• Panel source & size — which arena dataset(s), how many languages and prompts per language, and the roster selection rule (e.g. models with enough human annotations across enough languages).
• Comparability over time — how to handle panel versioning so older submissions stay comparable (panel hash / version gating).
• Direct submission-vs-submission — should we offer an opt-in compare flow that judges two submissions head-to-head (their completions already exist on the same frozen prompts; only the judging is missing), or rely solely on transitive ELO through the anchors?

Examples

(1) Submit a new model to arena

uv run judgearena-submit --panel-dir panels/deepseek-test-v1 --model OpenRouter/deepseek/deepseek-chat-v3.1 --tag "[test]"

Note: This does not submit a job from an external source. It should be done on the folder in which we keep the leaderboard (hugging-face space on later terms).

(2) Render it to the CLI

uv run judgearena-board --panel-dir panels/deepseek-test-v1 --format markdown
| Rank | Model | ELO | CI | n |
|---|---|---|---|---|
| 1 | gemini-2.5-pro | 1029.3 |  | 52 |
| 2 | gemini-2.5-flash | 1011.8 |  | 56 |
| 3 | OpenRouter/deepseek/deepseek-chat-v3.1 #[test] * | 993.3 | [983, 1002] | 100 |
| 4 | OpenRouter/deepseek/deepseek-v3.2 * | 992.6 | [983, 1002] | 100 |
| 5 | qwen3-235b-a22b-no-thinking | 989.5 |  | 46 |
| 6 | claude-opus-4-20250514 | 969.4 |  | 46 |

(3) Run the Leaderboard locally (before we push to hface)

 uv run judgearena-publish --panel-dir panels/deepseek-test-v1 --out /tmp/lb --dry-run
  uv run python space/app.py --local /tmp/lb

[geoalgo]

Thank you for opening the issue and apologies for not commenting earlier.

Panel source & size — which arena dataset(s), how many languages and prompts per language, and the roster selection rule (e.g. models with enough human annotations across enough languages).

We need to take the union of all datasets to have enough data for all languages. In addition I would suggest to have a cutoff date (like May 2026) so that we can evaluate easily fresh data (eg the data being populated in comparia which would not have been seen. That being said, we would also have lots of battles (for medium resource languages) which would be left out.

Comparability over time — how to handle panel versioning so older submissions stay comparable (panel hash / version gating).

We should version the metadata.json listing all the CLI and package version for reproducibility.

Direct submission-vs-submission — should we offer an opt-in compare flow that judges two submissions head-to-head (their completions already exist on the same frozen prompts; only the judging is missing), or rely solely on transitive ELO through the anchors?

I would first consider the simplest case to rely on transitive ELO.

Note: This does not submit a job from an external source. It should be done on the folder in which we keep the leaderboard (hugging-face space on later terms).

What do you mean here? I think having the leaderboard as another package would make sense to me.

[kargibora]

Hi David, no worries this was just a proposal before the leaderboard I have constructed:

Agreed on the cutoff date, versioning metadata.json with CLI/package versions, transitive ELO for now (no submission-vs-submission), and keeping the board as a separate rendering package. No disagreement there.

Here's what we actually constructed as a first panel:

• Source: lmarena-ai/arena-human-preference-140k → 116k single-turn battles, 53 distinct models after loading.
• Constraint we impose: every anchor model must have ≥10 human battles in every selected language (not just ≥10 battles in the language overall — every model covered in every language). This is what makes the per-language ratings trustworthy rather than dominated by whichever model happens to be frequent.
• That constraint yields a 31-model block across 18 languages. On top of it we sample 200 battles/language, balanced per model, and re-judge with gemma-4-31b calibrated (soft-ELO) against the human ELO.

So basically almost all models have similar amount of samples (~10 per each language) and thus all languages have around ~200 battles. This might be too low to test their ELO (the reason why their ELO MAE is large in the leaderboard). We can of course try to reduce the number of models and get more samples per each model (but less models means less available battles and less coverage unfortunutely).

Where the union proposal runs into trouble is the anchors. Union does not provide any more information (LMSYS140K is already very multilingual, LMSYS 55K and CompariA have different models). If we use existing completions from this Union our leaderboard may be disconnected (anchor models at least). So our option is either create a very mulitlingual proper dataset with low amount of existing completions and fill the rest (costly), or use local models to populate the leaderboard as anchor.

This is the reason why I am still a bit confused