Skip to content

Real SSL logs as replays, goal ledger, situation bench, and referee fixes - #143

Merged
isaac0804 merged 28 commits into
mainfrom
infra/real-match-eval
Oct 14, 2026
Merged

Real SSL logs as replays, goal ledger, situation bench, and referee fixes#143
isaac0804 merged 28 commits into
mainfrom
infra/real-match-eval

Conversation

@isaac0804

@isaac0804 isaac0804 commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Evaluation infrastructure from the 2026-10-12/13 sessions: official SSL game logs as replays, a goal ledger, the situation bench with bank v8, and three referee fixes the real logs exposed.

What's in it

Official SSL logs (utama_core/replay/ssl_log.py, tools/convert_ssl_log.py)

  • Converts official SSL game logs (tracker packets plus referee messages) into our columnar replays, or downloads and converts the published corpus.
  • Keeps teams, sides and the ball as played. Handles colour swaps, team-name swaps shorter than a minute, and referee clocks running through a taken free kick.
  • Vendors the SSL tracker protos (TrackerWrapperPacket) and their pb2 modules.

Goal ledger (utama_core/analysis/goal_ledger.py)

  • Attributes every goal to the situation that started its play (kickoff, free kick, high regain, open play ...).
  • --rates reports how often each situation happens per live minute. Real and sim rates are in scenario_bench/banks/situation_rates_*.json.

Situation bench and bank v8 (tools/scenario_bench.py, scenario_bench/)

  • Typed starts that each play against their own opponents, with the effect reported in goals per match.
  • Restart starts are taken where the replay first shows them and played in their own referee command. This fixes free kicks being replayed as ball placement.
  • apply_scenario starts each side's frame history from the scenario.
  • Bank v8 is harvested from tournament_20261011_230119; bank v7 is deleted.
  • bench_vs_standings checks whether the bench's effect predicts round-robin goal difference. Docs explain why the bench gets the level right but can't rank strategies.

Referee fixes (found by running our referee alongside 118 real matches), each with a regression test

  • The goalkeeper's own touch is not a Multiple Defenders foul.
  • The no-progress clock starts only once the ball is in play.
  • No-progress does not fire with the ball inside a defense area.
  • A ball in play stays in play after leaving a defense area.

Effect on evaluation

The referee and apply_scenario changes alter the round-robin fingerprint, so every cached match reruns. perf/planner-and-sim-speedup (separate PR) does the same, so a new fpp round-robin baseline will be run once both are merged.

Tests

On origin/main + this branch + the perf branch merged together: lint passes; pytest utama_core/tests/ --headless --level quick --ignore-glob "**/*grsim*" gives 12680 passed, 87 xfailed, 0 failed. That run excluded the perf branch's C++-only test, which is fixed in its own PR.

🤖 Generated with Claude Code

isaac0804 and others added 28 commits October 12, 2026 17:51
@isaac0804 @claude
Vendor the SSL tracker protos (TrackerWrapperPacket) and their pb2 mo…
13e5963 
…dules

The official game logs carry the autorefs' tracked frames (positions and velocities
in metres), which the log converter reads. Generated with the pinned protoc 3.20.2;
imports made package-relative like the other generated modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Bench: take a restart start where the replay first shows it
d5fa66d 
A replay frame carries the referee's decision on the frame before it, so the
frame nearest the match log's time for a free kick still showed ball
placement: every harvested free kick (bank_v7: 240 of 242) was stored, and
replayed, in BALL_PLACEMENT_*. `scenario_from_replay(showing=)` takes the
first frame from that time on that shows the command, and the harvester
uses it for every restart.

Kickoffs and penalties are now harvested where they are awarded
(-> PREPARE_KICKOFF_*/PREPARE_PENALTY_*), like free kicks, not at the
NORMAL_START after their preparation, and their perspective follows the
colour the command names (it was candidate_kicking whoever kicked).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
apply_scenario: play a start in its own referee command
b7ea54c 
A start's command was set over the bench runner's PREPARE_KICKOFF_YELLOW,
and the referee puts a STOP of its own before a restart set outside STOP,
held until every robot (the kicker too) is 0.5 m from the ball. NORMAL_START
was not set at all. So penalties and kickoffs were replayed late or as a
yellow kickoff, free kicks started in a STOP, and every open-play
NORMAL_START start (bank_v7: 496 of 860) was played inside the seeded
kickoff for up to 10 s.

A restart is now set from STOP, as the match awarded it, so its
preparation runs from the first tick; any other command, NORMAL_START
included, is set as it is. The kickoff anchor is a PREPARE_KICKOFF_YELLOW.
Bench results recorded before this commit are not comparable with later
ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Goal ledger: every goal attributed to the situation that started its …
604e56c 
…play

utama_core/analysis/goal_ledger.py ports the scratchpad ledger validated this
session. A goal goes to the latest restart going live or ball won in open play
less than 10 s before it, else "sustained possession". Per strategy, goals per
match = sum over situation types of situations per match x P(goal), for goals
scored and conceded. It reads only the .npz and .sparse_referee.pkl, so it also
runs on converted official logs:
- a free kick from the official game controller is itself live play (it goes
  straight to STOP), while ours waits for NORMAL_START;
- a goal is timed when play stopped for it;
- a lowered score withdraws a goal.

On tournament_20261011_230119 it reproduces the reference exactly: 1458/1458
goal attributions, all 648 table values. Over all five run dirs, goals match the
recorded scores in 299/299 matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Docs: goal ledger in signals.md, tools.md and the glossary
e1de28e 
Adds where goals come from (situation frequency x conversion), with split_shape
reference values from tournament_20261011_230119, the CLI to the tool index, and
"situation / goal ledger" to the AGENTS.md glossary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Docs: how a bench start is played, and what bank_v7 got wrong
77e5bda 
A restart start is the moment the referee awarded it and is played as that
restart; every robot starts at rest because rsim cannot place a moving
one. bank_v7's restart starts were taken one tick early, and bench results
recorded before b7ea54c are not comparable with later ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Convert official SSL game logs to columnar replays
6420538 
utama_core/replay/ssl_log.py reads a RoboCup SSL game log (.log.gz) and
writes it through ColumnarReplayWriter, so load_columnar_replay,
render_window, scenario_from_replay and the analyses read a real match
like one of ours. Yellow is the friendly team; side_is_right per tick
from blue_team_on_positive_half (yellow's goalkeeper for pre-2019 logs).
Robots and ball come from the autoref tracker with the most frames,
falling back to merged camera detections for logs without one. Each
robot id gets its own column, so substitutions lose no robot. Referee
messages are kept where something but the clocks changes, with
stage_time_left in seconds and designated_position in metres; a free
kick reads NORMAL_START once its ball has moved 0.05 m, as in our
referee. A <stem>.source.json sidecar records teams, score, tracker
source and field geometry.

Tests on synthetic logs pin the record reader, the referee dedupe, the
id and side mapping, units, and the free-kick boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
tools/convert_ssl_log.py: convert game logs, or download and convert …
e51c6ee 
…the published corpus

Converts given logs or directories, or with --corpus YEARS lists the
league's public share (WebDAV PROPFIND) for one division and, per log,
downloads it, converts it into OUT/<year>/ and deletes the download, so
at most --jobs raw logs are on disk. Existing replays are skipped, so an
interrupted run resumes; OUT/corpus.json records each log's outcome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
ssl_log: run referee clocks on from each message's arrival, through a…
f6d7216 
… taken free kick

A converted tick's referee was the last kept message with its clocks
run on from the first tick after it, and switching a taken free kick to
NORMAL_START rebuilt the message from scratch: the stage clock jumped
back by however long the free kick had waited (2-3 s in a 2025 match).
Clocks now run on from when the message arrived, and NORMAL_START is
applied to the state already run on to that tick.

The new test fails before (the clock reads 299.75 s again at the kick)
and passes after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
ssl_log: keep teams, sides and ball as played
7432823 
Found validating the converted 2022 Div B logs:
- Teams that swap colours at half time (the game controller moves names,
  scores and cards with them) are swapped back, so friendly stays one team
  and its score never falls.
- Where the referee's sides contradict the play (yellow's goalkeeper and
  yellow's robots both on the other half through 10 s of live play: one
  2022 match ran with them inverted throughout, cameras agreeing with the
  trackers), the sides are inverted.
- A ball beyond the field's walls is unseen: trackers' chip estimates put
  it up to hundreds of metres away for a few ticks (ball speeds to 838 m/s).

Each has a regression test that fails with its fix disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
ssl_log: team names swapped for under a minute are not a colour swap
3130260 
A 2018 Div A operator swapped the names for 3 s mid-half and back while
the robots kept their markers; the converter swapped the robots too. A
swap now counts only if it lasts 60 s. Regression test pins the boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
ssl_log: judge the referee's sides on taken free kicks too
368d7d9 
The game controller keeps DIRECT_FREE_* through the play after a free
kick, so raw NORMAL_START/FORCE_START left the second half of 2022
Ri-one vs UBC Thunderbots under the 10 s needed, and its inverted sides
stood. Live play is now what the converted replay calls live. The
regression test's only live play is a taken free kick.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Merge infra/bench-restart-fixes: restart starts played as they were
96f78f9 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Merge infra/goal-ledger: where goals come from
3d844b8 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
docs: real matches from official SSL game logs
99c1df9 
tools.md lists tools/convert_ssl_log.py; STRATEGY_DEVELOPMENT.md says what
a converted match holds and where it differs from a sim replay: perspective
and the corrections flagged in source.json, live share, robot columns,
has_ball, free kicks, and the pre-2020 vision fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Merge infra/ssl-log-converter: official SSL logs as replays
b9c9cdc 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
apply_scenario: start each side's frame history from the scenario
82d6eb6 
Velocities are a finite difference over the runner's frame history. After a
start was applied, that history still held the formation the bench runner was
built in, so the first tick read the teleport as a jump: a ball at 130 m/s,
robots faster. The ball filter then overshot by up to half a metre (a goal from
a ball the keeper held, or a ball out of play), and the referee charged
ball-speed and crashing fouls no robot made. In the open-play fidelity check
this turned a 7% chance the other side scored after a ball won in the
defensive third into 52%.

Each side now gets a new history and Game from the scenario, as StrategyRunner
does when the teams change ends. The regression test applies a start the way
the bench does, right after building the runner, and fails at 30 m/s without
the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
goal_ledger --rates: how often each situation happens per live minute
604c5d2 
Owner-neutral situation rates per team per live minute (force starts per
live minute), with live minutes per match: the weights the situation bench
turns its per-situation results into goals per match with. Committed for
the 118 published Division B matches and tournament_20261011_230119.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Situation bench: typed starts, their own opponents, effect in goals p…
cd6cfcd 
…er match

The harvester takes its starts from the goal ledger: every restart at the
first tick showing its command, and every ball won at the start of its
possession spell (live play only), each tagged with its situation type from
the candidate's view. The old open-play harvest (passes and losses from
turnover_breakdown) is gone, so there is one possession definition;
--open-play becomes --per-situation: up to N of each type (kickoffs N/2),
spread over source matches, seeded, near-duplicates skipped.

scenario_bench plays each start against its source match's config_b
(--opponent overrides; a start whose opponent is not on the branch is
skipped and counted), scores the first goal within 10 s, and reports per
type the paired change in P(for) and P(against), weighted by situation
rates (--weights, real Division B by default) into goals per match, with
a standard error. The ordinal outcome stays in each start's record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Bank v8 from tournament_20261011_230119; delete bank_v7
11071b8 
640 starts: 40 of each situation type, 20 of each kickoff type, from the
147 trustworthy matches, config_a's view. 293 are playable on this branch:
the rest come from matches against split_* variants that live only on
strategy/search-20261011.

    scenario_bench.py --harvest-from replays/tournament_20261011_230119 \
        --per-situation 40 --workers 15 --save-bank .../bank_v8.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
bench_vs_standings: does the bench's effect predict round-robin goal …
d0637ce 
…difference

Repurposed for the situation bench: for a baseline and several candidates,
each candidate's bench effect (change in goal difference per match, one or
more weights files) against its round-robin goal difference minus the
baseline's over the same opponents, with Pearson and Spearman. Starts whose
opponent is the candidate or the baseline are left out. The old standings
and per-scenario score comparison is retired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
docs: the situation bench, bank v8, situation rates, and how far to t…
586b659 
…rust it

AGENTS.md glossary: Start, Bank, Scenario bench and Situation updated,
Situation rates added. STRATEGY_DEVELOPMENT.md: the bench section rewritten
for situations, weights, per-start opponents, the 1.21x/1.58x over-count and
what the bench cannot see; bank_v7's caveats removed with it; the validation
against tournament_20261011_230119 (level right, ranking not: Spearman
-0.09 over six candidates, SE ~2.3 goals a match) and where its noise
comes from. tools.md, signals.md and the roadmap follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
docs: why the situation bench can't rank, beyond its noise
a582284 
The bench's estimate is biased per strategy, not only noisy. A 10 s horizon
misses goals from build-up, and those are 3-7% of the split_* family's goals
but 19-68% of the slower strategies' (counter_flow 68%). Split by the ledger,
the round-robin gap to split_shape is mostly conversion, which the bench does
measure (hence the right level). Between candidates, conversion, situation
frequency and build-up goals differ by similar amounts, and the bench sees
only the first. Per second of simulation it is also no more precise than
matches.

Also: the over-count need not be the same for every strategy, so it can
reorder close candidates (the docs said ranking was unaffected); playable
starts per type are 11-25, not 15-25; the glossary calls the bench a
diagnostic, not a screen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
fix(referee): goalkeeper's own touch is not a Multiple Defenders foul
1e96611 
DefenseAreaRule called any robot touching the ball inside its own box a
Multiple Defenders foul, so a keeper touch with a teammate also in the
box got a penalty. Real matches: 60 of 97 false penalty calls were this.

The keeper id is read from the referee data on the frame
(GameFrame.referee, team info goalkeeper). Only a non-keeper touch counts.
With no referee data attached, every touch still counts.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
fix(referee): no-progress clock starts once the ball is in play
e93c522 
NoProgressRule started counting at a kickoff/penalty NORMAL_START with the
ball unmoved and fired exactly 10 s later. The official clock only runs
once the ball is in play: 0.05 m of movement, or the 10 s auto-in-play of
a kickoff/penalty. In real matches 226 of 422 no-progress calls were this
false alarm.

After a NORMAL_START the clock now waits for one of those. A FORCE_START
puts the ball in play at once and is unchanged.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
fix(referee): no-progress does not fire with the ball inside a defens…
89a8b9c 
…e area

Officials never call no-progress with the ball inside either defense area
(0 of 506 real calls). Ours did, and pre-empted the keeper-held-ball rule
(attackers' free kick) with a neutral restart. While the ball is inside
either defense area the clock is now held and reset.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
fix(referee): a ball in play stays in play after leaving a defense area
d0183c8 
Under NORMAL_START (live play after a kickoff) the defense-area hold
cleared the in-play flag too, so a ball that had already moved went back to
waiting for the 10 s auto-in-play when it left a defense area: no-progress
took 20 s there instead of 10 s. Only a command change takes the ball out of
play now; the defense area just holds the clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 @claude
Merge infra/referee-fixes: three referee bugs found by the real-match…
75a1a81 
… shadow check

Goalkeeper's own touch is no longer a Multiple Defenders penalty; the
no-progress clock waits for the ball to be in play and is held inside the
defense areas. Changes the match fingerprint: cached matches rerun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECXiwf3VYxBTMmrLaexRQK
@isaac0804 isaac0804 added the release:minor Minor changes to main label Oct 14, 2026
@isaac0804 isaac0804 mentioned this pull request Oct 14, 2026
Merged
@isaac0804
isaac0804 merged commit 1849d12 into main Oct 14, 2026
4 of 5 checks passed
@isaac0804
isaac0804 deleted the infra/real-match-eval branch October 14, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

release:minor Minor changes to main

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@isaac0804