Tensor Parallelism
Tensor parallelism splits individual weight matrices across GPUs (e.g., column-split for QKV projections) and communicates partial results via all-reduce, enabling models too large for one GPU's VRAM.
What is Tensor Parallelism?
Tensor parallelism splits individual weight matrices across GPUs (e.g., column-split for QKV projections) and communicates partial results via all-reduce, enabling models too large for one GPU's VRAM.
Tensor parallelism splits individual weight matrices across GPUs (e.g., column-split for QKV projections) and communicates partial results via all-reduce, enabling models too large for one GPU's VRAM.
Where is it used?
Megatron-LM and vLLM use TP for Llama-3-70B training and serving; `torch.distributed.tensor_parallel` and DeepSpeed-Megatron provide production-grade TP implementations.
How to build it
Use `megatron.core.tensor_parallel.ColumnParallelLinear` for QKV and `RowParallelLinear` for the output projection, launch across 8 GPUs, and verify the all-reduce fusion in the trace.