Weights
Weights are learnable parameters that scale each input to a neuron, determining how much that input contributes to the output. They are what training optimizes.
What is Weights?
Weights are learnable parameters that scale each input to a neuron, determining how much that input contributes to the output. They are what training optimizes.
Weights are learnable parameters that scale each input to a neuron, determining how much that input contributes to the output. They are what training optimizes.
Where is it used?
Llama-3 70B has ~70 billion weights (parameters); each attention and MLP layer's weight matrices are tuned by gradient descent over trillions of tokens.
How to build it
Inspect weights via `model.parameters()`, print their shapes and `.requires_grad` flags, and initialize a custom layer with `nn.init.xavier_uniform_(layer.weight)`.