Skip to content

LLM Architecture KB

By category
By lens
Models
Compare
Timeline
About
Search

Training stability

Methods that help loss curves stay smooth and convergent.

8 techniques tagged with this facet.

Normalization

Layer Normalization LayerNorm

2016-07 Per-token feature-axis normalization with learnable scale and shift. The 2016 building block the original transformer used; superseded in modern decoders by RMSNorm, which drops the mean-centering step at no quality cost.
Root Mean Square Layer Normalization RMSNorm

2019-10 Normalize by RMS instead of subtracting the mean — drop one statistic, one bias, and ~5-10% of normalization compute, with no measurable quality cost.
Pre-Norm, Post-Norm, and Sandwich Placement Norm Placement

2020-02 Pre-Norm became the default because it lets gradients flow unaltered through the residual stream and removes the need for warmup; sandwich placements like Gemma 3's norm-everywhere are the modern minority alternative.
Query-Key Normalization QK-Norm

2020-10 Normalize Q and K before the attention dot product so the logit magnitude stays bounded — a near-free fix for the outlier blow-ups that wreck attention at long context.
Sandwich-LN Sandwich-LN

2021-05 Two RMSNorms per sublayer: one before, one after, both inside the residual branch. Bounds the per-block contribution to the residual stream while keeping the identity path unnormalized. Used by Gemma 2 and 3.
nGPT — Normalized Transformer on the Hypersphere nGPT

2024-10 Project every vector to unit norm at every step. Eliminates LayerNorm/RMSNorm by structural construction; cosine similarity replaces inner product everywhere. Trains 4-20× faster than baseline GPT at matched quality, per the paper.
OLMo 2 Reordered Post-Norm OLMo 2 Post-Norm

2024-12 Move both RMSNorms to sit after each sublayer output, inside the residual branch — recovering Post-Norm's bounded residual stream while keeping Pre-Norm's gradient flow. Trained stably at 13B by combining with QK-Norm and an output-side Z-loss.
Gemma 3 Norm-Everywhere Norm-Everywhere

2025-03 Sandwich-LN applied to BOTH the attention and FFN sublayers — four RMSNorms per block. Combined with QK-Norm and interleaved local/global attention, it stabilizes the entire Gemma 3 size family from a single recipe.

Content licensed CC BY 4.0. Code licensed MIT.

Source on GitHub · Methodology · RSS · Last built 2026-05-13