Expert Networks
Expert networks are the parallel feed-forward sub-networks in an MoE layer, each specialising on different token patterns; only the top-k experts selected by the router process a given token.
What is Expert Networks?
Expert networks are the parallel feed-forward sub-networks in an MoE layer, each specialising on different token patterns; only the top-k experts selected by the router process a given token.
Expert networks are the parallel feed-forward sub-networks in an MoE layer, each specialising on different token patterns; only the top-k experts selected by the router process a given token.
Where is it used?
Mixtral has 8 expert FFNs per layer, DeepSeek-V3 uses 256 experts with shared experts; the `transformers` library supports MoE via `MixtralSparseMoeBlock`.
How to build it
Create `self.experts = nn.ModuleList([FeedForward(d, d_ff) for _ in range(n_experts)])`, index with `selected = [self.experts[i](x) for i in topk_indices]`, and stack outputs.