Router
The router is a small linear layer that produces a logit per expert for each token, then selects the top-k experts via softmax over the selected logits to weight their outputs.
What is Router?
The router is a small linear layer that produces a logit per expert for each token, then selects the top-k experts via softmax over the selected logits to weight their outputs.
The router is a small linear layer that produces a logit per expert for each token, then selects the top-k experts via softmax over the selected logits to weight their outputs.
Where is it used?
Mixtral's `MixtralRouter` and DeepSeek's router use top-2 selection; noisy top-k gating (adding noise before top-k) encourages exploration during training.
How to build it
Implement `logits = self.router(x); topk = logits.topk(k, dim=-1); weights = softmax(topk.values); experts = topk.indices`, then dispatch tokens to chosen experts and multiply by weights.