CPU vs GPU
CPUs have few powerful cores optimized for sequential logic and low latency, while GPUs have thousands of simpler cores optimized for parallel arithmetic throughput. Deep learning is massively parallel, so GPUs dominate.
What is CPU vs GPU?
CPUs have few powerful cores optimized for sequential logic and low latency, while GPUs have thousands of simpler cores optimized for parallel arithmetic throughput. Deep learning is massively parallel, so GPUs dominate.
CPUs have few powerful cores optimized for sequential logic and low latency, while GPUs have thousands of simpler cores optimized for parallel arithmetic throughput. Deep learning is massively parallel, so GPUs dominate.
Where is it used?
Training GPT-4 class models requires thousands of NVIDIA H100 GPUs; inference on Llama 3 is also GPU-bound. CPUs handle data loading, tokenization, and small-model inference.
How to build it
Compare `torch.matmul` timing on CPU vs CUDA using `time.time()` around a large matmul, moving tensors with `.to('cuda')` and back, to empirically measure the 10-100x speedup.
Code
A practical example: