Debugging

Refs

• Analysing neural networks
• RETURNN Debugging doc
• Stas Bekman - ML debugging
  • Understanding Training Loss Patterns
• Stas Bekman - The Art of Debugging
• Albert Zeyer - debugging wiki
• Albert Zeyer - profiling wiki

Examples

Using dummy dataset

For a Sisyphus ReturnnTrainingJob, go into the work dir, run:

python3 .../returnn/rnn.py ../output/returnn.config \
  ++dry_run 1 ++use_dummy_datasets 1 ++batching default ++num_epochs 1000 \
  ++use_train_proc_manager 0 ++watch_memory 1 ++torch_distributed None ++use_horovod 0