Scalars
A scalar is a single number — a 0-dimensional tensor representing magnitude only, with no direction. Scalars are the atoms of all mathematical objects in deep learning.
What is Scalars?
A scalar is a single number — a 0-dimensional tensor representing magnitude only, with no direction. Scalars are the atoms of all mathematical objects in deep learning.
A scalar is a single number — a 0-dimensional tensor representing magnitude only, with no direction. Scalars are the atoms of all mathematical objects in deep learning.
Where is it used?
Scalars represent learning rates, loss values, temperature in softmax, bias terms, and every individual weight in a neural network. The final cross-entropy loss of an LLM is a scalar.
How to build it
Create a scalar tensor with `torch.tensor(3.14)` (shape `torch.Size([])`), perform arithmetic, and observe that reducing a tensor with `.sum()` or `.mean()` yields a scalar.
Code
A practical example: