LLM Architecture KB

Learning path  · Intermediate

Sparse MoE foundations

The four entries that built today's sparse mixture-of-experts FFN: the original sparse routing idea, Switch's k=1 simplification, Mixtral's open instantiation, and DeepSeek's shared-expert refinement.

Mixture-of-experts decouples parameter count from per-token FLOPs by activating only a small subset of experts for each token. Read these in order to see how the four moving pieces — the router, the top-k rule, load balancing, and the shared-expert design — stabilized into the modern sparse-MoE recipe used by Mixtral, DeepSeek, and their successors.

  1. 1 Sparsely-Gated MoE Sparse MoE Top-K gating with per-expert noise plus an importance-balancing loss. The 2017 paper that established that 'sparse activation + load balance' is a viable training-time recipe; every modern MoE descends from it.
  2. 2 Switch Transformer Switch Top-1 expert routing — each token goes to exactly one expert. Simpler than GShard's top-2 baseline. Trillion-parameter Switch-C model proved sparse MoE was tractable at unprecedented scale; established the load-balancing loss formulation everyone uses.
  3. 3 Mixtral-Style Coarse MoE Mixtral MoE Eight full-sized SwiGLU experts per MoE layer; top-2 routing with a standard load-balancing auxiliary loss. The Mistral-org variant that brought open-weights MoE into wide use, sitting on the coarse end of the granularity spectrum that DeepSeekMoE later contested.
  4. 4 DeepSeekMoE DeepSeekMoE Split the FFN into many small experts, route each token to top-K, and reserve a few always-on shared experts for common-knowledge work. Activated parameter count stays small; specialization sharpens.