Architecture Experiments
Architecture experiments systematically vary model components (attention type, activation, normalisation) to measure their impact on quality and efficiency.
What is Architecture Experiments?
Architecture experiments systematically vary model components (attention type, activation, normalisation) to measure their impact on quality and efficiency.
Architecture experiments systematically vary model components (attention type, activation, normalisation) to measure their impact on quality and efficiency.
Where is it used?
SwiGLU vs GELU (Llama), RoPE vs ALiBi, GQA vs MHA (Llama-3), and MoE vs dense (Mixtral) were validated through architecture experiments before adoption.
How to build it
Implement a config-driven transformer in PyTorch, sweep `attention: ["mha", "mqa", "gqa"]` with `hydra`, train each on the same data, and compare perplexity and throughput.