Neuron
A neuron is a computational unit that takes weighted inputs, sums them with a bias, and applies a non-linear activation function. Stacking neurons in layers produces deep networks.
What is Neuron?
A neuron is a computational unit that takes weighted inputs, sums them with a bias, and applies a non-linear activation function. Stacking neurons in layers produces deep networks.
A neuron is a computational unit that takes weighted inputs, sums them with a bias, and applies a non-linear activation function. Stacking neurons in layers produces deep networks.
Where is it used?
Every feed-forward layer in a transformer contains thousands of neurons; the MLP in GPT-2 small has 3,072 neurons per layer that process each token's hidden state.
How to build it
Implement a neuron manually: `y = activation(torch.dot(w, x) + b)`, then compare with `nn.Linear(in, 1)` followed by `F.relu` to see the same computation abstracted.
Code
A practical example: