Weight Matrices
Weight matrices are the 2D tensors that perform linear transformations in a neural network. In transformers, every attention projection (Q, K, V, O) and FFN layer is a weight matrix; they are where the model's learned knowledge is stored.
What is Weight Matrices?
Weight matrices are the 2D tensors that perform linear transformations in a neural network. In transformers, every attention projection (Q, K, V, O) and FFN layer is a weight matrix; they are where the model's learned knowledge is stored.
Weight matrices are the 2D tensors that perform linear transformations in a neural network. In transformers, every attention projection (Q, K, V, O) and FFN layer is a weight matrix; they are where the model's learned knowledge is stored.
Where is it used?
GPT-4, LLaMA, and BERT contain thousands of weight matrices across their layers. During fine-tuning via LoRA, small low-rank matrices are added alongside frozen weight matrices to adapt the model cheaply.
How to build it
Create with `nn.Linear(in, out)` which stores `weight` of shape `(out, in)` and optional `bias`. Initialize with `nn.init.xavier_uniform_(layer.weight)` for stable training. Access via `model.layers[0].attn.q_proj.weight`.