Learning path · Intro
Norm placement and type
Where to put the normalization layer, and which variant to use. Four short entries to grasp the 2024+ consensus.
Normalization is the cheapest module in a transformer block and one of the most argued-about. This short path takes you from the original LayerNorm to RMSNorm, surveys where the norm goes (pre, post, sandwich), and finishes with QK-Norm — the small addition that has slipped into several recent frontier dense decoders.
- 1 Layer Normalization LayerNorm Per-token feature-axis normalization with learnable scale and shift. The 2016 building block the original transformer used; superseded in modern decoders by RMSNorm, which drops the mean-centering step at no quality cost.
- 2 Root Mean Square Layer Normalization RMSNorm Normalize by RMS instead of subtracting the mean — drop one statistic, one bias, and ~5-10% of normalization compute, with no measurable quality cost.
- 3 Pre-Norm, Post-Norm, and Sandwich Placement Norm Placement Pre-Norm became the default because it lets gradients flow unaltered through the residual stream and removes the need for warmup; sandwich placements like Gemma 3's norm-everywhere are the modern minority alternative.
- 4 Query-Key Normalization QK-Norm Normalize Q and K before the attention dot product so the logit magnitude stays bounded — a near-free fix for the outlier blow-ups that wreck attention at long context.