Sequence Length
Sequence length is the actual number of tokens in a specific input, as opposed to the context window which is the maximum allowed. It determines memory usage and compute for that particular forward pass.
What is Sequence Length?
Sequence length is the actual number of tokens in a specific input, as opposed to the context window which is the maximum allowed. It determines memory usage and compute for that particular forward pass.
Sequence length is the actual number of tokens in a specific input, as opposed to the context window which is the maximum allowed. It determines memory usage and compute for that particular forward pass.
Where is it used?
During training, sequence length is a batch hyperparameter (e.g., 2048 or 4096) that balances throughput and memory. During inference, it grows as tokens are generated. vLLM and FlashAttention optimize memory based on the actual sequence length.
How to build it
Read it: `seq_len = input_ids.shape[1]`. In training, pad or pack sequences to a fixed `seq_len` for efficient batching. In inference, track it as generation extends the sequence. Memory scales as O(seq_len²) for standard attention, O(seq_len) with FlashAttention.