Routing
Routing is the per-token decision of which experts to activate, producing sparse expert selection that defines the MoE's compute pattern and load distribution across experts.
What is Routing?
Routing is the per-token decision of which experts to activate, producing sparse expert selection that defines the MoE's compute pattern and load distribution across experts.
Routing is the per-token decision of which experts to activate, producing sparse expert selection that defines the MoE's compute pattern and load distribution across experts.
Where is it used?
Token routing in Mixtral/DeepSeek determines which GPUs (in expert parallelism) receive each token; expert parallelism in vLLM and Megatron-DeepMind routes tokens across nodes.
How to build it
Add an auxiliary load-balancing loss `(n_experts * sum(mean(expert_load)^2) - 1)` to the training loss, route tokens via top-2, and print per-expert token counts to verify balance.