Mixture of Experts (MoE)

Mixture of Experts is a neural network architecture that scales model capacity without proportionally increasing computation. It uses multiple "expert" sub-networks with a routing mechanism that activates only a subset per input.

How MoE Works

Experts — Multiple FFN sub-networks (e.g., 8-256 experts)
Router/Gate — A learned routing network that selects which experts to use
Sparse activation — Only top-k experts are activated per token (typically k=1-4)
Combination — Expert outputs are weighted by router probabilities and summed

Mathematical Formulation

Output = Σ g_i(x) · Expert_i(x)

Where g_i(x) is the router weight for expert i, and only the top-k g_i are non-zero.

Why MoE Matters

Massive capacity — Total parameters can be 10-100x activated parameters
Constant compute — FLOPs per token stays the same regardless of total parameters
Better scaling — MoE models outperform dense models at the same compute budget

Key Models

Model	Total Params	Active Params	Experts
DeepSeek V3	671B	37B	256
LLama 4 Maverick	17B MoE	~4B	16
Mixtral 8x7B	47B	13B	8
Qwen2.5-MoE	42B	8B	8

Challenges

Load balancing — Router may overuse some experts; auxiliary loss needed
Communication — Experts may be on different GPUs, requiring all-to-all communication
Memory — All expert parameters must be loaded into memory
Fine-tuning — More complex than dense models
Batch size — Needs larger batches to keep all experts busy

Related Concepts

Transformer Architecture — Foundation that MoE extends
distillation — Can compress MoE to dense models
quantization — Reduces memory of large expert parameters