MDSection 32 of 38 · 7 subtopics
Distributed Training
Multi-GPU training: data, tensor, and pipeline parallelism. Communication overhead.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
01
Distributed Computing
Multi-GPU cluster.
distributed-computing
02
Data Parallelism
Dataset replicas across GPUs.
data-parallelism
03
Tensor Parallelism
Matrix sharding visualization.
tensor-parallelism
04
Pipeline Parallelism
Layer stages across GPUs.
pipeline-parallelism
05
Model Parallelism
Model partition map.
model-parallelism
06
GPU Communication
GPU-to-GPU network diagram.
gpu-communication
07
Communication Overhead
Compute vs communication timeline.
communication-overhead