RNN
A Recurrent Neural Network (RNN) processes sequences one token at a time, maintaining a hidden state that carries information from previous steps. It was the dominant sequence model before transformers.
What is RNN?
A Recurrent Neural Network (RNN) processes sequences one token at a time, maintaining a hidden state that carries information from previous steps. It was the dominant sequence model before transformers.
A Recurrent Neural Network (RNN) processes sequences one token at a time, maintaining a hidden state that carries information from previous steps. It was the dominant sequence model before transformers.
Where is it used?
Early neural language models and translation systems (seq2seq with RNNs, Google's original NMT) used RNNs; they fell out of favor because they cannot parallelize and forget long context.
How to build it
Use `nn.RNN(input_size, hidden_size, num_layers)` in PyTorch, feed a sequence, and inspect the final hidden state; compare training speed with a transformer on the same data.
Code
A practical example: