Model Parallelism
Model parallelism is the umbrella term for splitting a model's parameters across multiple GPUs — combining tensor and pipeline parallelism — when the model does not fit in a single GPU's memory.
What is Model Parallelism?
Model parallelism is the umbrella term for splitting a model's parameters across multiple GPUs — combining tensor and pipeline parallelism — when the model does not fit in a single GPU's memory.
Model parallelism is the umbrella term for splitting a model's parameters across multiple GPUs — combining tensor and pipeline parallelism — when the model does not fit in a single GPU's memory.
Where is it used?
GPT-3 (175B) and Llama-3-405B training combine TP and PP across thousands of GPUs; Megatron-LM's 3D parallelism (DP+TP+PP) is the industry standard for frontier-scale training.
How to build it
Configure Megatron-LM with `--tensor-model-parallel-size 8 --pipeline-model-parallel-size 4` across 32 GPUs, and inspect the memory per GPU to confirm the model is sharded as expected.