Batch Construction
Batch construction groups training samples into batches for parallel processing on the GPU. Key decisions include batch size, sequence length uniformity, and gradient accumulation to simulate larger effective batch sizes than GPU memory allows.
What is Batch Construction?
Batch construction groups training samples into batches for parallel processing on the GPU. Key decisions include batch size, sequence length uniformity, and gradient accumulation to simulate larger effective batch sizes than GPU memory allows.
Batch construction groups training samples into batches for parallel processing on the GPU. Key decisions include batch size, sequence length uniformity, and gradient accumulation to simulate larger effective batch sizes than GPU memory allows.
Where is it used?
GPT-3 used a batch size of 3.2M tokens (via gradient accumulation). LLaMA-2 70B used 4M tokens per batch. Large effective batch sizes stabilize training and improve throughput. Data loaders shuffle and pad to construct batches efficiently.
How to build it
Use `DataLoader(dataset, batch_size=B, shuffle=True, drop_last=True)`. For gradient accumulation: `for i, batch in enumerate(loader): loss = model(batch) / accum_steps; loss.backward(); if (i+1) % accum_steps == 0: optimizer.step(); optimizer.zero_grad()`.