Training Sequences
Training sequences are fixed-length token chunks extracted from the tokenized corpus, each serving as one training example. The model learns to predict each token given the preceding tokens within the sequence window.
What is Training Sequences?
Training sequences are fixed-length token chunks extracted from the tokenized corpus, each serving as one training example. The model learns to predict each token given the preceding tokens within the sequence window.
Training sequences are fixed-length token chunks extracted from the tokenized corpus, each serving as one training example. The model learns to predict each token given the preceding tokens within the sequence window.
Where is it used?
nanoGPT packs the Shakespeare token stream into sequences of `block_size=256` or `1024`. GPT-2 used sequences of 1024; LLaMA-2 uses 4096. Packing concatenates documents with EOS to avoid padding waste. Each sequence yields `block_size - 1` prediction targets.
How to build it
Tokenize all text: `data = encode(text)`. Chunk: `data = torch.tensor(data); sequences = data[:n * block_size].view(n, block_size)`. In the dataset: `return seq[:-1], seq[1:]` as (input, target). Use `DataLoader` with `shuffle=True`.