core: checkpoints from different TP/DP ranks overwrite each other #165

Problem

save_ckpt and save_lora_ckpt name files by pipeline stage only. With tp=2 both TP ranks write different shards to one file, so one shard is lost and a reload loads wrong weights; both ranks also delete old files (FileNotFoundError race). Without a postfix the directory name becomes checkpoint_None_<stage>.

Evidence and reproduction

opt_prime/opt_prime/opti_pri.py:1244, :1173, :465 (an or fallback that never triggers). Verified by reading; the shipped examples are PP-only, so this is latent.

Environment

optimus-rm @253be20; Python 3.10, torch 2.10.0+cu128, transformers 5.1.0; RNGD+A100 host (CPU / one A100 for the checks).

Scope

opti_pri.py checkpoint save/load naming.

Done when

Only DP rank 0 writes, TP shards get distinct names, the directory name has no None; a unit test covers the naming.

Owner

@gspark-etri