Real SSL logs as replays, goal ledger, situation bench, and referee fixes

AGENTS.md

@@ -31,23 +31,34 @@ One term per thing; reuse these instead of coining new ones.
   (`tools/evaluation/round_robin.py`); writes `replays/tournament_<id>/summary.json`. The ground truth for
   "which strategy is better". Slow in full; with `--reuse` only matches whose code changed rerun.
 - **Standings** — a round-robin's ranking. `round_robin.py` prints wins and draws;
-  `bench_vs_standings.py` uses points per match (3 a win, 1 a draw) and goal difference.
-- **Start** — one starting situation (kickoff, free kick, penalty, or an open-play moment)
-  that the scenario bench replays for 20 s. In code: `BenchScenario` (`scenario_bench/start.py`),
-  and "scenario" in flags and enums.
+  `metric_correlation.py` uses points per match (3 a win, 1 a draw) and goal difference.
+- **Start** — one situation (see below) the scenario bench replays for 10 s, tagged with its
+  type from the candidate's view. In code: `BenchScenario` (`scenario_bench/start.py`), and
+  "scenario" in flags and enums.
 - **Bank** — a versioned list of starts (`utama_core/scenario_bench/banks/bank_vN.json`),
-  harvested from one round-robin's replays plus a few hand-authored anchors.
+  harvested from one round-robin's replays, up to 40 of each situation type.
 - **Scenario bench** — `tools/scenario_bench.py`: a fast paired A/B screen of a candidate
-  against a baseline on a bank. A screen, not a ranking: confirm with a round-robin.
+  against a baseline on a bank, each start against the opponent it came from; it estimates the
+  change in goals per match as Σ situations per match (real or sim weights) × the change in
+  P(goal within 10 s). A diagnostic, not a ranking: it cannot see a candidate change how often
+  situations happen or score from build-up longer than 10 s, so rank with matches.
 - **Bench validators** — `tools/bench_vs_standings.py` and `tools/metric_correlation.py` check
-  how far the bench and cheap proxy metrics agree with round-robin standings.
+  how far the bench and cheap proxy metrics agree with round-robin results.
 - **Motion planning benchmark** — `tools/motion_planning_benchmark.py`: the planner alone on
   fixed scenarios. Unrelated to the scenario bench or any strategy.
 - **Pitch zones** — defensive / middle / attacking third, left wing / centre / right wing, the
   box, danger: defined once, with a diagram, in `docs/pitch_zones.md`. Use those names.
 - **Catch rate** — a diagnostic reported from the bank, never a gate (±4–8 pts between runs).
 - **Signal** — an interpretable measure of how a strategy plays (shots, danger conceded, real
   ball losses, stalls ...), for explaining results, never a target. All listed in `docs/signals.md`.
+- **Situation / goal ledger** — a situation starts when a restart goes live or a side wins the
+  ball in open play; the goal ledger (`utama_core/analysis/goal_ledger.py`) gives each goal to
+  the latest situation less than 10 s before it, so goals per match = Σ situations per match ×
+  P(goal). Situation types: own / opponent ball won or free kick in each third, own / opponent
+  kickoff or penalty, force start. See `docs/signals.md`.
+- **Situation rates** — how often each situation type happens per team per live minute, from
+  real Division B matches or our round-robin (`banks/situation_rates_{real,sim}.json`,
+  `goal_ledger --rates`): the scenario bench's weights.
 
 ## Repo map
 

docs/custom_referee.md

@@ -41,7 +41,7 @@ Built in `_build_active_rules` (`custom_referee.py`), in this priority order; de
 | `DefenseAreaRule` | live play | An extra defender (> `max_defenders`) touching the ball in its own area: penalty. An attacker touching the ball while at least partly in ours: non-stopping, 2 s re-raise (§8.4.2). |
 | `KeepOutRule` | `DIRECT_FREE_*`, `PREPARE_KICKOFF_*` | Defending team within 0.5m for 30 consecutive frames: non-stopping, 2 s re-raise, restarts the free kick's 10 s clock (§8.4.3). Not during bare `STOP` or penalties. |
 | `PenaltyTimeLimitRule` | `NORMAL_START`/`FORCE_START` after `PREPARE_PENALTY_*` | Penalty still in play 10 s after its normal start: goal kick for the defenders (§5.3.5). |
-| `NoProgressRule` | live play | Ball hasn't moved 0.05 m in 10 s: `STOP` then `FORCE_START` where it lies (§8.1). Last in the list, so a real foul on the same tick wins. |
+| `NoProgressRule` | live play | Ball hasn't moved 0.05 m in 10 s: `STOP` then `FORCE_START` where it lies (§8.1). The clock waits for the ball to be in play (a `NORMAL_START` restart: 0.05 m of movement or the 10 s auto-in-play) and is held while the ball is inside either defense area. Last in the list, so a real foul on the same tick wins. |
 | `PushingRule`, `KeeperHeldBallRule`, `ExcessiveDribblingRule`, `RobotStopSpeedRule`, `CrashingRule`, `DefenseAreaStoppageRule`, `BallPlacementInterferenceRule` | see each rule file | SSL §8.3/8.4 audit (`434ab29`). Rationale per rule: `custom_referee_design_decisions.md`. |
 
 ## State machine and auto-advance

docs/custom_referee_design_decisions.md

@@ -34,7 +34,7 @@ Checked against the [SSL rulebook](https://robocup-ssl.github.io/ssl-rules/sslru
 | Attacker Touched Ball In Opponent Defense Area (§8.4.2) | Touching the ball while partly or fully inside; non-stopping | Presence (centre inside) stopped play with a free kick | Fixed: needs a touch, counts partial overlap (`ROBOT_RADIUS`), non-stopping, 2 s re-raise |
 | Ball Speed (§8.4.2) | Over 6.5 m/s in 3D; non-stopping | Stopped play, free kick to the other team | Fixed: non-stopping. Gap: still measured on ground (x, y) speed, not 3D |
 | Goal validity (§7) | No goal if the scorer committed a non-stopping foul in the last 2 s, or the ball went above 0.15 m | Any ball in the goal scored | Fixed: foul in the last 2 s turns the goal into a goal kick. Gap: ball height is not checked |
-| No Progress In Game (§8.1) | 10 s without progress while both teams may play: stop, then force start | Missing | Fixed: `NoProgressRule`; STOP now continues to a queued FORCE_START once robots clear |
+| No Progress In Game (§8.1) | 10 s without progress while both teams may play: stop, then force start | Missing | Fixed: `NoProgressRule`; STOP now continues to a queued FORCE_START once robots clear. The clock starts once the ball is in play (after a kickoff/penalty `NORMAL_START` that means 0.05 m of movement or the 10 s auto-in-play; it was firing 10 s after an unmoved restart, 226 of 422 false alarms in real matches). Not counted while the ball is inside either defense area (0 of 506 real calls there) |
 | Penalty kick (§5.3.5) | Still in play after 10 s: stopped, no goal, goal kick for the defenders | The normal start turned into open play | Fixed: `PenaltyTimeLimitRule`. Not implemented: the keeper's 90° deflection and ball-moving-backwards endings |
 | Ball in play after a free kick (§5.4) | In play 10 s after the free kick command | `DIRECT_FREE_*` waited for the kicker forever | Fixed: `FORCE_START` after 10 s (clock restarted once, by the first Defender Too Close foul) |
 | Ball in play after a kick-off (§5.4) | In play 10 s after the kick-off | `NORMAL_START` → `FORCE_START` after 10 s if the ball hasn't moved | Already correct |
@@ -43,7 +43,7 @@ Checked against the [SSL rulebook](https://robocup-ssl.github.io/ssl-rules/sslru
 | Aimless kick (§6.2.3, Div B) | A ball touched by a robot that then crosses the halfway line and the opponent's goal line outside the goal, touching no other robot: free kick for the opponent from where it was kicked | Not implemented: a goal kick in the opponent's corner, so a blind clearance cost nothing (28% of goal kicks led to one at the other end within seconds, tournament_20261005_170958) | Fixed: `OutOfBoundsRule` keeps where the ball was last touched in this spell of play (`last_touch.touching_team`), and restarts from there when the touch was on the far side of the halfway line, strictly (a kick-off from the line itself is not aimless); the spot is kept 0.25 m inside the lines and 1 m off the defense areas like any free kick. `out_of_bounds.aimless_kick`, on in both profiles: we play Division B |
 | Free kick stages (§5.3.3) | One free-kick command; the kick puts the ball in play | `DIRECT_FREE_*` preparation, then `NORMAL_START` once the kicker is ready | Sim: kept. The two stages give the auto-advance a readiness check with no human referee |
 | Ball placement (§5.2) | The team places the ball | With a simulator controller, `StrategyRunner` teleports the ball to `designated_position` and force-starts | Sim: kept, so matches don't wait on physical placement. Real mode still places |
-| Multiple Defenders (§8.4.1) | A non-keeper entirely inside its own area touching the ball: penalty | More than `max_defenders` (1) with centres inside, one touching | Sim: kept. The rule doesn't know which robot is the keeper |
+| Multiple Defenders (§8.4.1) | A non-keeper entirely inside its own area touching the ball: penalty | More than `max_defenders` (1) with centres inside, one touching | Fixed: the goalkeeper's own touch is exempt. The keeper id comes from the referee data on the frame (`GameFrame.referee`, the team info's `goalkeeper`); with no referee data attached every touch counts. The occupancy proxy (`max_defenders`) is kept |
 | Double Touch (§8.2) | Kicker may not touch again before another robot; free kick from the ball position | Same | Already correct |
 | Crashing | §8.4.2 | Any contact with closing speeds within 0.3 m/s of each other was a "both at fault" crash, including robots at rest against each other (6694 fouls in 231 matches; 30 of 97 goals then voided by the goal-validity rule) | Fixed: a crash needs > 1.5 m/s projected on the line between the robots; the 0.3 m/s band then compares the robots' own speeds, as TIGERs AutoReferee's `BotCollisionDetector` does. Its 0.1 s brake lookahead is modelled too (each robot's speed less 0.4 m/s, floored at 0, where TIGERs reverses it): without it a round-robin had 226 crashes to TIGERs' 8 over the same frames |
 | Pushing, Keeper Held Ball, Excessive Dribbling, Robot Stop Speed, Too Close To Opponent Defense Area, Ball Placement Interference | §8.3–8.4 | Audited in `434ab29` | Already correct; not re-checked beyond thresholds (10 s, 1 m, 1.5 m/s, 0.2 m, 0.5 m, 2 s grace) |

docs/roadmap.md

@@ -35,7 +35,7 @@ resolved, replace it with a one-line pointer under "Done".
 
 1. **Current stall count (2026-10-01, `fpp`, `e83a7466`): 0/231 matches** in the full strict
    65s round-robin (`replays/tournament_20261001_094103/`), none flagged by the possession
-   backstop; 93% of restarts reach NORMAL_START. `utama_core/scenario_bench/banks/bank_v7.json` is
+   backstop; 93% of restarts reach NORMAL_START. `bank_v7` (deleted 2026-10-12, replaced by `bank_v8`) was
    harvested from it. Earlier stall families (overload-slot COMMITTED_FROZEN, DIRECT_FREE
    restart stalls, low_block's PassAndShoot holding the ball) are fixed; see `git log`.
    Confirm any stall fix against the full round-robin: small-subset re-runs have repeatedly

docs/signals.md

@@ -29,7 +29,9 @@ Every saved `tools/evaluation/round_robin.py` run prints its sections and writes
 round-robin (`tools/signal_report.py <run_dir>` regenerates them), with notes on what stands out.
 
 **Offline only** (not in a round-robin's output): `tools/metric_correlation.py <run_dir> ...`
-replays a run at 10 Hz for the signals marked *offline* below.
+replays a run at 10 Hz for the signals marked *offline* below, and
+`python -m utama_core.analysis.goal_ledger <run_dir>` writes the goal ledger
+([Where do goals come from?](#where-do-goals-come-from)).
 
 **Sides.** Per-match stats are config_a's view ("friendly"); a strategy's row reads its own
 side of every match. Signals marked *config_a* need the intentions log, which only config_a
@@ -98,6 +100,30 @@ Check these first: a broken strategy's other signals mean nothing.
 | Restart to first entry | `results[*].stats.restart_to_first_entry_s`, `n_restarts`, `n_restarts_with_entry` | seconds from a restart until the possessing side's ball reaches its attacking third; restarts with no entry are counted, not averaged | — | Slow or rarely entering: restarts are played backward or lost |
 | Restart outcomes | RESTARTS | see "Is something broken?" | | |
 
+## Where do goals come from?
+
+The goal ledger (`analysis/goal_ledger.py`, *offline*) gives every goal to the situation that
+started the play it ended, if that was less than 10 s before: a restart going live (kickoff,
+free kick, penalty, force start) or a side winning the ball in open play, each named from the
+strategy's side ("own" / "opponent") with the third in the owner's attack frame. Goals with no
+start that recent are "sustained possession". A strategy's goals per match is then exactly the
+sum over situations of situations per match × P(goal), for goals scored and conceded alike: a
+result splits into which situations a strategy creates or allows, and how well it converts or
+defends them. It reads only the replay and its referee file, so it runs on converted official
+logs too. The rules are in the module's docstring. `--rates PATH` writes how often each
+situation happens per team per live minute, whoever owns it: the weights of the
+[situation bench](STRATEGY_DEVELOPMENT.md#the-situation-bench), whose P(goal) also counts a goal
+that comes after a later situation started, unlike the table below.
+
+Reference: `split_shape` in `tournament_20261011_230119` (17 matches; 6.76 scored, 4.41
+conceded per match).
+
+| Signal | Where | Meaning | Reference | If it's off |
+|---|---|---|---|---|
+| Situations per match | `goal_ledger.json` `strategies[*].types[*].situations_per_match` | how often each situation happens in a strategy's matches | own free kicks in the attacking third 6.7, the opponent's 6.2; balls won in the attacking third: own 3.4, opponent's 3.6; in the defensive third ~19 each | Many "opponent ball won, attacking third": losing the ball near our own goal (build-up). Many opponent free kicks in the attacking third: fouls and clearances out near our goal |
+| P(goal) per situation | `p_scored`, `p_conceded` | share of a situation's occurrences followed by a goal before the next situation starts, within 10 s | scored: own penalty 88%, own ball won in the attacking third 47%, own free kick in the attacking third 28%; conceded: opponent penalty 100%, opponent free kick in the attacking third 20%, opponent ball won in the attacking third 20% | High P(conceded) from opponent free kicks: set-piece defence (wall, marking). High from balls the opponent wins high: nobody covers behind the press |
+| Goals per match by situation | `scored_per_match`, `conceded_per_match` | situations per match × P(goal); the column sums to goals per match | conceded 4.41 = opponent balls won 1.41 + free kicks 1.35 + penalties 0.76 + sustained possession 0.71 + kickoffs 0.18 | The largest row is where a fix pays most |
+
 ## Adding a signal
 
 Check this page first: it probably exists. A new signal should be interpretable on its own,

docs/tools.md

@@ -33,12 +33,19 @@ What these measure and how far to trust each: [STRATEGY_DEVELOPMENT.md](STRATEGY
 
 | Script | Purpose |
 |---|---|
-| `tools/scenario_bench.py` | Paired A/B of a candidate against a baseline on a bank of 20 s starts harvested from a round-robin; also builds new banks (`--harvest-from`), and plays one start with a timeline and picture (`--play`) |
+| `tools/scenario_bench.py` | Paired A/B of a candidate against a baseline on a bank of situation starts harvested from a round-robin (10 s each, against the opponent each came from): change in P(goal) per situation type, weighted by real or sim situation rates (`--weights`) into goals per match; also builds new banks (`--harvest-from`), and plays one start with a timeline and picture (`--play`) |
 | `tools/signal_report.py` | Figures of a round-robin's strategy signals for [signal_report.md](signal_report.md) |
-| `tools/bench_vs_standings.py` | How far scenario-bench scores rank strategies the way round-robin standings do (Spearman) |
+| `utama_core/analysis/goal_ledger.py` | `python -m utama_core.analysis.goal_ledger <run_dir or replays>`: every goal attributed to the situation that started its play (a restart, or the ball won in open play, by third); per strategy, situations per match × P(goal) = goals per match, scored and conceded. Reads only the replay and its referee file, so it runs on converted official logs too. `--rates` writes the situation rates the scenario bench weights by. See [signals.md](signals.md#where-do-goals-come-from) |
+| `tools/bench_vs_standings.py` | How well the scenario bench's effect on goal difference per match predicts a round-robin's, for several candidates against one baseline (Pearson, Spearman) |
 | `tools/metric_correlation.py` | Offline study of which cheap per-match metrics predict results; computes the *offline* signals in [signals.md](signals.md) |
 | `tools/check_strategy_branch.py` | CI check on `strategy/*` pull requests: the branch must not change the evaluation or its opponents |
 
+## Real matches
+
+| Script | Purpose |
+|---|---|
+| `tools/convert_ssl_log.py` | Official RoboCup SSL game logs to replays in our format, so the analyses run on real matches: given logs, or `--corpus 2022-2026` to download and convert every published Division B match, deleting each raw log once converted. What a converted match holds and its limits: [STRATEGY_DEVELOPMENT.md](STRATEGY_DEVELOPMENT.md#real-matches) |
+
 ## Debugging a match
 
 | Script | Purpose |