Sparse Activation
Sparse activation means only a subset of experts (and their parameters) is used per token, so total parameters grow while FLOPs stay roughly constant — the key efficiency benefit of MoE.
What is Sparse Activation?
Sparse activation means only a subset of experts (and their parameters) is used per token, so total parameters grow while FLOPs stay roughly constant — the key efficiency benefit of MoE.
Sparse activation means only a subset of experts (and their parameters) is used per token, so total parameters grow while FLOPs stay roughly constant — the key efficiency benefit of MoE.
Where is it used?
Mixtral 8x7B activates ~13B of 47B params per token; DeepSeek-V3 activates ~37B of 671B; sparse activation lets frontier models scale parameters without scaling inference cost linearly.
How to build it
Count active params by `sum(p.numel() for e in selected_experts for p in e.parameters())` vs total; log active/total ratio per step to verify sparsity is maintained.