Load Balancing
Load balancing ensures tokens are distributed evenly across experts, preventing a few experts from receiving most tokens (collapse) which would waste capacity and degrade quality.
What is Load Balancing?
Load balancing ensures tokens are distributed evenly across experts, preventing a few experts from receiving most tokens (collapse) which would waste capacity and degrade quality.
Load balancing ensures tokens are distributed evenly across experts, preventing a few experts from receiving most tokens (collapse) which would waste capacity and degrade quality.
Where is it used?
Mixtral and DeepSeek add an auxiliary load-balancing loss; Switch Transformer introduced the expert-choice routing variant; imbalance is monitored via expert utilisation histograms.
How to build it
Compute `load_loss = n_experts * sum(f_i * P_i)` where `f_i` is fraction of tokens routed to expert i and `P_i` is mean router probability for expert i; add `load_loss * alpha` to the main loss.