Skip to content

feat: add turn-indicator false-positive (spurious-transition) rate metric - #418

Draft
MasatoSaeki wants to merge 1 commit into
tier4:tier4-mainfrom
HansRobo:feat/turn_indicator_fp_metric
Draft

feat: add turn-indicator false-positive (spurious-transition) rate metric#418
MasatoSaeki wants to merge 1 commit into
tier4:tier4-mainfrom
HansRobo:feat/turn_indicator_fp_metric

Conversation

@MasatoSaeki

@MasatoSaeki MasatoSaeki commented Sep 11, 2026

Copy link
Copy Markdown

Summary

Adds a closed-loop turn-indicator false-positive (spurious-transition) rate metric, the complement of the existing turn_indicator_transition_accuracy (#403).

  • The existing metric answers: "when GT c?" (gated on a real GT transition,transition-based, not per-step).
  • This metric answers the mirror question: "when GT holds steady, does the model flip anyway?"

Both are computed in the same _score_turnpopulation of real-inference (scored) steps, and are mutually exclusive per step: transition_total + fp_total always equals the number of scored steps a segment.

Metric details

  • Scope: every scored step where GT did nscored step — regardless of which class(NONE/DISABLE/LEFT/RIGHT) it's holding, not restricted to None.
  • Event: a "false positive" is the model'g from its own previous scored prediction(turn_indicator_prev_scored_pred), not onstruction in this branch, so comparingagainst GT would just remeasure transittead of measuring spurious model behavior.
  • New _SegState fields: turn_indicator_fpl (accumulators) and turn_indicator_prev_scored_pred (baseline, lifecycle mirrors turn_indicator_prev_scored_gt: seeded in _seed_state, updated every scored step,port so the environment jump itself isnever counted as a spurious flip).
  • turn_indicator segment-row block extendtransition_total} to also carry {fp_count,fp_total}; the false_positive_rate ratien fp_total == 0) is derived at aggregatetime, same convention as transition_acc
  • Held/cached-plan steps (_hold_turn_indicator, no fresh inference) never touch the new counters, same as the existing ones.
  • Wired through closed_loop_eval.aggregate/format_summary_lines, closed_loop_score_keys.SCORE_EXTRACTORS, scenario_sim_metrics (zero-count block,at path), wandb_closed_loop (new tablecolumn + weighted-by-fp_total aggregatid_loop._write_groups_manifest.

Test plan

  • pytest scenario_generation/tests/test_turn_indicator_scoring.py -v — new tests cover: FP counted only whenGT is steady and the model's own predicsteady and prediction is stable;GT-transition steps never touch the FP pairs partition the scored steps,verified via transition_total + fp_totas don't touch the FP counters; unstickteleport re-seeds turn_indicator_prev_scored_pred.
  • pytest scenario_generation/tests/te— turn_indicator_block (4-arg), aggregatepooling of fp_count/fp_total/false_posi → None guard.
  • pytest scenario_generation/tests/te.py -v — fixture updated to the new blockshape.
  • pytest scenario_generation/tests/ - pre-existing failure(test_wandb_scenario_sim.py::test_serieonfirmed unrelated to this change(reproduces identically on the unmodifing order-dependent mock-state issue).

🤖 Generated with Claude Code

…tric

Mirrors the existing turn_indicator_transition_accuracy metric (which asks
"when GT changes, does the model follow?") with its complement: "when GT
holds steady, does the model flip anyway?" Both live in the same
_score_turn_indicator scoring function and partition the same population of
scored steps, so transition_total + fp_total always equals the number of
scored steps in a segment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant