Tensor Operations
Tensor operations include element-wise math, reshaping, slicing, broadcasting, reduction (sum/mean), and matrix products performed on multi-dimensional arrays. These primitives compose into all neural network computations.
What is Tensor Operations?
Tensor operations include element-wise math, reshaping, slicing, broadcasting, reduction (sum/mean), and matrix products performed on multi-dimensional arrays. These primitives compose into all neural network computations.
Tensor operations include element-wise math, reshaping, slicing, broadcasting, reduction (sum/mean), and matrix products performed on multi-dimensional arrays. These primitives compose into all neural network computations.
Where is it used?
Reshaping and broadcasting are used constantly in transformers — e.g., splitting the embedding into multiple heads requires `.view()` and `.transpose()` on tensors in every attention layer.
How to build it
Practice with `torch.reshape`, `torch.cat`, `torch.sum(x, dim=-1)`, and `x.transpose(1,2)` on random tensors to understand how shape manipulation drives model architecture.
Code
A practical example: