Cornstarch is a multimodal model training framework, including distributed training features with 5D parallelism (PP, TP, CP, DP, and modality parallelism). You can create your own multimodal model with a set of HuggingFace unimodal models and train it.
Cornstarch provides
- Pipeline Template and Heterogeneous Pipeline Parallelism: specify different pipeline templates and combine them to deploy heterogeneous pipeline parallel execution
- Composable multimodal model creation: specify your own multimodal models from a set of HuggingFace transformers unimodal models
- MultimodalModel Generation and Parallelization: specify your own multimodal model and parallelize it with 5D parallelism (DP+PP+TP+CP+and modality parallelism)
Please refer to our document!
- Cornstarch: Distributed Multimodal Training Must Be Multimodality-Aware
- Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
- Insu Jang (insujang@umich.edu)