Forward Pass
The forward pass is the process of pushing input tokens through the model to produce output logits. It involves embedding lookup, all transformer layers (attention + FFN), and the final LM head projection. No gradients are computed during inference-only forward passes.
What is Forward Pass?
The forward pass is the process of pushing input tokens through the model to produce output logits. It involves embedding lookup, all transformer layers (attention + FFN), and the final LM head projection. No gradients are computed during inference-only forward passes.
The forward pass is the process of pushing input tokens through the model to produce output logits. It involves embedding lookup, all transformer layers (attention + FFN), and the final LM head projection. No gradients are computed during inference-only forward passes.
Where is it used?
Every LLM inference and training step begins with a forward pass. During training, the forward pass produces logits used for loss computation, followed by backpropagation. vLLM optimizes the forward pass with PagedAttention and continuous batching.
How to build it
Call the model: `logits = model(input_ids)` which returns shape `(B, T, vocab_size)`. For training: `loss = F.cross_entropy(logits[:, :-1].reshape(-1, V), labels[:, 1:].reshape(-1))`. For inference: `next_logits = logits[:, -1]`.