Pipeline Parallelism
Pipeline parallelism partitions the model's layers across GPUs and pipelines micro-batches through the stages, overlapping compute and communication to keep all GPUs busy.
What is Pipeline Parallelism?
Pipeline parallelism partitions the model's layers across GPUs and pipelines micro-batches through the stages, overlapping compute and communication to keep all GPUs busy.
Pipeline parallelism partitions the model's layers across GPUs and pipelines micro-batches through the stages, overlapping compute and communication to keep all GPUs busy.
Where is it used?
GPipe and PipeDream-Flush in Megatron-LM and DeepSpeed pipeline stages for 100B+ models; HF `Accelerate` `pipeline_parallel` config orchestrates simple pipelines.
How to build it
Use `torch.distributed.pipeline.sync.Pipe` with `DevicePlacement` to split an `nn.Sequential` across GPUs, feed micro-batches, and inspect the schedule to see bubble overhead.