Category
Normalization
LayerNorm to RMSNorm to QK-Norm; the placement debates that shaped frontier models.
- Layer Normalization LayerNorm foundational
Normalize the activations of a layer across the feature dimension rather than across the batch — letting recurrent and attention-based networks train stably at any batch size, including batch-of-one inference.
- Root Mean Square Layer Normalization RMSNorm foundational
Drop the mean-centering step of LayerNorm — keep almost all of the stability benefit at meaningfully less compute.
- Pre-Norm, Post-Norm, and Sandwich Placement Norm Placement foundational
Where to put the normalization layer relative to the residual addition — the decision that determines whether deep transformers train stably without warmup.
- Query-Key Normalization QK-Norm production-adopted
Stop attention logits from blowing up when a single outlier appears in Q or K — common at long contexts and in mixed-precision training.
- Sandwich-LN Sandwich-LN production-adopted
Cap the magnitude of each block's perturbation to the residual stream — keeping Pre-Norm's gradient-flow benefit while preventing the residual norm from growing unboundedly with depth.
- nGPT — Normalized Transformer on the Hypersphere nGPT research
Constrain every vector in the network — embeddings, attention outputs, FFN outputs, weight rows — to the unit hypersphere. Replace the standard normalize-once-per-block recipe with normalization-everywhere as a structural invariant.
- OLMo 2 Reordered Post-Norm OLMo 2 Post-Norm production-adopted
Recover Post-Norm's bounded residual-stream norm at production depth without the gradient instability that pushed everyone to Pre-Norm in the first place.
- Gemma 3 Norm-Everywhere Norm-Everywhere production-adopted
Take Sandwich-LN to its maximal form by applying it to both the attention and FFN sublayers — every signal entering or leaving any sublayer's compute graph passes through a norm.