Data Parallelism
Data parallelism replicates the full model on every GPU, splits the input batch across GPUs, and synchronises gradients via all-reduce after backward, enabling larger effective batch sizes.
What is Data Parallelism?
Data parallelism replicates the full model on every GPU, splits the input batch across GPUs, and synchronises gradients via all-reduce after backward, enabling larger effective batch sizes.
Data parallelism replicates the full model on every GPU, splits the input batch across GPUs, and synchronises gradients via all-reduce after backward, enabling larger effective batch sizes.
Where is it used?
PyTorch `DistributedDataParallel` (DDP) is the default for single-node multi-GPU training of models that fit on one GPU; FSDP extends it by sharding parameters.
How to build it
Use `torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])`, wrap the dataloader with `DistributedSampler`, and launch with `torchrun --nproc_per_node=8 train.py`.