Training Samples
A training sample is a single input-target pair fed to the model. For language modeling, a sample is a token sequence of fixed length (e.g., 2048), and the target is the same sequence shifted by one token for next-token prediction.
What is Training Samples?
A training sample is a single input-target pair fed to the model. For language modeling, a sample is a token sequence of fixed length (e.g., 2048), and the target is the same sequence shifted by one token for next-token prediction.
A training sample is a single input-target pair fed to the model. For language modeling, a sample is a token sequence of fixed length (e.g., 2048), and the target is the same sequence shifted by one token for next-token prediction.
Where is it used?
During GPT-3 and LLaMA pretraining, the corpus is split into fixed-length chunks (samples) of 2048 tokens. Shorter documents are concatenated (packing) to avoid wasting compute on padding. Each sample contributes T-1 prediction targets.
How to build it
Tokenize the full corpus, concatenate all tokens, then chunk: `samples = tokens[:n_chunks * seq_len].reshape(n_chunks, seq_len)`. For packing: concatenate documents with EOS separators, then chunk. Create a `Dataset` returning `(input_ids, labels)` where `labels = input_ids`.