[leifdenby]

The continuation run intrigued-sow-455-from-step83299 crashed at its first validation (step 88,299, ~40 min in):

MLFlowLogger.after_save_checkpoint -> log_artifact(.../epoch=1-step=88299/epoch=1-step=88299.ckpt)
SSLEOFError: EOF occurred in violation of protocol  (7 retries over ~9 min)
mlflow.exceptions.MlflowException: ... Max retries exceeded

Cause: #6 turned on log_model="all", so every ~770 MB checkpoint is uploaded inside the training loop. mlflow.dmidev.org (or its proxy) cuts the connection on uploads that size (none of the run's checkpoint uploads succeeded; large PNGs were already getting 413s), and Lightning's MLFlowLogger lets the error stop training. My end-to-end test used a local file store, which has no size limit.

Fix

• use_mlflow_logger(upload_checkpoints=False): upload is opt-in again.
• ResilientMLFlowLogger (new, src/mlcast/loggers.py): a failed checkpoint upload is logged as a warning, and the checkpoints it covered are marked as handled so later saves don't retry them (each attempt blocks training for minutes of HTTP retries). The file is already on local disk.
• resume_training no longer forces upload on, and switches the logger of any config (including ones saved by Resume training from a checkpoint or an MLflow run #6-era runs with plain MLFlowLogger and log_model="all") to ResilientMLFlowLogger.
• Resuming still works without uploads via the local <experiment_id>/<run_id>/checkpoints fallback (same machine).

Testing

• New tests: upload opt-in, resilient logger warns once and doesn't retry on a failing log_artifact, old configs get the resilient logger with their upload setting kept. Full suite: 180 passed, 5 skipped.

To make cross-machine resume possible with this server, the upload size limit on mlflow.dmidev.org would need raising.

🤖 Generated with Claude Code