Category
FFN & MoE
Dense MLPs to gated SwiGLU to sparse mixtures of experts.
- Gaussian Error Linear Unit GELU foundational
Replace ReLU's hard zero cutoff with a smooth, probabilistically-motivated activation. Improves transformer pretraining quality at no compute cost; the standard FFN activation through 2022.
- Sparsely-Gated MoE Sparse MoE foundational
Make conditional computation actually work at scale: sparsely gate each token through a small number of experts out of many, with per-expert noise and a load-balance loss that lets the gate train end-to-end.
- FFN with ReLU FFN-ReLU deprecated
Provide point-wise nonlinearity between attention layers via the simplest available activation — two linear projections with a ReLU between them.
- GELU-Gated Linear Unit GeGLU production-adopted
Same gated bilinear FFN as SwiGLU, with GELU as the gate activation instead of Swish. Adopted by the Gemma family; matches SwiGLU's quality within noise.
- ReLU-Gated Linear Unit ReGLU foundational
Same gated bilinear FFN as SwiGLU and GeGLU, with ReLU as the gate. The third evaluated variant in the original GLU-variants paper; rarely adopted in production but useful as a baseline.
- Swish-Gated Linear Unit SwiGLU production-adopted
Replace the FFN's ReLU/GELU with a gated activation that consistently improves perplexity at matched compute.
- GShard GShard foundational
First production-credible sparse MoE for transformer encoders: top-2 routing across hundreds of experts, automatic sharding via tensor-program annotations, formal expert-capacity dispatch primitives.
- Switch Transformer Switch foundational
Strip the MoE routing decision down to its simplest possible form: route each token to a single expert. Demonstrate that this 'simplification' actually trains stably at trillion-parameter scale.
- DeepSeekMoE DeepSeekMoE production-adopted
Get sharper expert specialization than vanilla MoE by combining fine-grained expert segmentation with isolated shared experts that absorb the common-knowledge load.
- Mixtral-Style Coarse MoE Mixtral MoE production-adopted
Sparsely activate a small number of large experts per token. Coarser than DeepSeekMoE but operationally simpler — the workhorse design that introduced production MoE to most open-weights users.
- Auxiliary-Loss-Free Load Balancing Aux-Loss-Free production-adopted
Load-balance MoE experts without paying the auxiliary-loss quality tax — by adjusting a per-expert bias that enters at top-K selection but does not enter the final output computation.
- Kimi K2 MoE K2 MoE production-adopted
Push the DeepSeekMoE shared-expert pattern to a much wider expert bank (384 routed experts) at trillion-parameter scale, while keeping training stable via the MuonClip optimizer.