MDSection 33 of 38 · 6 subtopics
Mixture of Experts
Scale capacity without scaling compute. Routing, sparse activation, load balancing.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
01
Mixture of Experts
Router + expert network.
mixture-of-experts
02
Expert Networks
Parallel expert modules.
expert-networks
03
Router
Routing probability bars.
router
04
Routing
Token-to-expert graph.
routing
05
Sparse Activation
Active/inactive expert map.
sparse-activation
06
Load Balancing
Expert utilization histogram.
load-balancing