Repository navigation
Show training progress in the terminal and MLflow console log (upstream #32) - #4
Merged
leifdenby merged 4 commits intoOct 17, 2026
Merged
Conversation
…turing The console capture points fds 1/2 at pipes, so Lightning's default RichProgressBar sees no terminal and hides its live display, and a progress bar never ends its line, so nothing reached console.log. use_mlflow_logger now switches to TQDMProgressBar (unless a bar is configured) and adds LogTrainingProgressCallback, which logs a one-line progress summary every progress_log_interval_seconds (default 60). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit re-rendered it with a newer local Graphviz (12.2.1 vs CI's 2.43.0), which CI's --check flags as out of date. Nothing in this PR changes the base config the diagram is drawn from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rge/console-progress-into-dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the training-progress fix from upstream #32 (commit 20bb7fb there) into
dev/dmi-convgru-training, as a cherry-pick (#1 was squash-merged, so a merge of mlcast-community#32's branch conflicts with itself).Why
With
fiddler:use_mlflow_logger, training showed no progress bar in the terminal and nothing after "MLflow run started" in the run'slogs/console.log(seen onupbeat-croc-674on 27sj894):RichProgressBarsees no terminal and hides its live display.What
use_mlflow_loggerswitches the progress bar toTQDMProgressBar(unless one is already configured), which shows in the terminal again.LogTrainingProgressCallback, added byuse_mlflow_logger, logs one line every 60 s (progress_log_interval_seconds): step, epoch/batch,train_loss_step, steps/s, samples/s, elapsed and remaining time (frommax_time).Tests
console.log.uv run --extra cpu pytest tests: 146 passed, 5 skipped, 1 failed. The failure,tests/config/test_dmi.py::test_dmi_500m_10min_convgru_experiment_config, is already ondev/dmi-convgru-training: 214dbd5 changed the train split to start 2019-07-13 without updating the test. Fixed in the companion index-path PR.The running job (
upbeat-croc-674) is unaffected; this applies to runs started after it's merged and pulled.🤖 Generated with Claude Code