GPU Communication
GPU communication refers to the collective operations (all-reduce, all-gather, reduce-scatter) used to synchronise gradients or activations across GPUs, typically over NVLink or InfiniBand.
What is GPU Communication?
GPU communication refers to the collective operations (all-reduce, all-gather, reduce-scatter) used to synchronise gradients or activations across GPUs, typically over NVLink or InfiniBand.
GPU communication refers to the collective operations (all-reduce, all-gather, reduce-scatter) used to synchronise gradients or activations across GPUs, typically over NVLink or InfiniBand.
Where is it used?
NCCL powers PyTorch DDP/FSDP collectives; NVLink within a node and InfiniBand across nodes provide the high bandwidth needed for Llama-3 and GPT-4 scale training.
How to build it
Run `torch.distributed.all_reduce(tensor, op=ReduceOp.SUM)` across `torchrun` processes, profile with `torch.profiler` to see NCCL kernels, and compare NVLink vs PCIe bandwidth.