Skip to content

Fix EMA updates across gradient accumulation and resume - #125

Open
zjn20030811 wants to merge 2 commits into
thu-ml:mainfrom
zjn20030811:fix/ema-optimizer-step-tracking
Open

Fix EMA updates across gradient accumulation and resume#125
zjn20030811 wants to merge 2 commits into
thu-ml:mainfrom
zjn20030811:fix/ema-optimizer-step-tracking

Conversation

@zjn20030811

Copy link
Copy Markdown

Summary

  • update the EMA model only after a synchronized optimizer step that was not skipped for overflow
  • persist the EMA optimization counter next to its averaged weights and restore it on resume
  • retain compatibility with older checkpoints by falling back to the checkpoint global_step when the sidecar is absent

Motivation

The training loop currently calls ema_model.step() after every microbatch. Accelerate suppresses optimizer.step() inside a gradient-accumulation window, but the standalone EMA update is not suppressed. With accumulation factor N, this advances the EMA warmup schedule N times per optimizer update, repeatedly averages unchanged parameters, and performs N full traversals of the 1B-parameter model.

Resume has a separate schedule discontinuity: only the averaged weights are loaded, while EMAModel.optimization_step starts again at zero. The next EMA update therefore uses zero decay and overwrites the loaded history with the online model.

This change keeps the existing global_step data-consumption semantics while tracking successful EMA updates independently. New checkpoints resume the exact schedule; old checkpoints continue to load with an explicit best-effort migration.

Validation

  • python -m pytest tests/test_ema_model.py -q — 6 passed with Accelerate 0.30.1
  • real CPU accumulation test confirms updates occur only on microbatches 4 and 8 for accumulation factor 4
  • overflow-skip regression leaves both EMA weights and counter unchanged
  • save/load continuation matches an uninterrupted EMA trajectory exactly
  • full Ruff check for tests, fatal-rule checks for changed source files, py_compile, and git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant