Communication Overhead
Communication overhead is the time spent on collective operations (all-reduce, all-gather) that does not contribute to forward/backward compute, often the bottleneck in large-scale distributed training.
What is Communication Overhead?
Communication overhead is the time spent on collective operations (all-reduce, all-gather) that does not contribute to forward/backward compute, often the bottleneck in large-scale distributed training.
Communication overhead is the time spent on collective operations (all-reduce, all-gather) that does not contribute to forward/backward compute, often the bottleneck in large-scale distributed training.
Where is it used?
FSDP's all-gather during forward and reduce-scatter during backward add overhead; gradient compression and overlap of compute with communication in Megatron-LM mitigate this.
How to build it
Profile DDP with `torch.profiler` and inspect NCCL kernel time as a fraction of total; enable `gradient_as_bucket_view` and `static_graph` in DDP to reduce overhead, re-profile to compare.