Encoder
The encoder is a stack of transformer blocks that processes the entire input sequence bidirectionally, allowing each token to attend to all others. It produces contextualized representations but does not generate output autoregressively.
What is Encoder?
The encoder is a stack of transformer blocks that processes the entire input sequence bidirectionally, allowing each token to attend to all others. It produces contextualized representations but does not generate output autoregressively.
The encoder is a stack of transformer blocks that processes the entire input sequence bidirectionally, allowing each token to attend to all others. It produces contextualized representations but does not generate output autoregressively.
Where is it used?
The encoder stack is the core of BERT, used for classification, NER, and sentence embeddings. Encoder-decoder models like T5 and BART use the encoder to understand the input before the decoder generates the output.
How to build it
Stack N transformer blocks each containing self-attention (bidirectional, no causal mask) and an FFN. In PyTorch: `for blk in self.blocks: x = blk(x, src_mask=None)` — no causal mask means full bidirectional attention.