Pre-Norm, Post-Norm, and Sandwich Placement

§ 1 · Premise

The 2017 default that almost broke transformers

Vaswani et al. (arXiv 1706.03762, §3.1) wrote the original transformer with Post-Norm placement: every sublayer is wrapped in “Add & Norm” — first add the residual, then normalize. The choice was inherited from ResNet conventions and was not separately justified in the paper. It came with a famously sensitive recipe: a learning-rate warmup of 4000 steps and the Adam $\beta_2 = 0.98$ setting were both load-bearing. Skip either and the 6-layer base model fails to converge.

Through 2018 and 2019 multiple groups reported the same pattern at depth. BERT-Large (arXiv 1810.04805) needed careful warmup; deeper transformer-encoder variants for translation (arXiv 1906.01787) diverged at depth 24+ without extensive tuning. The shared cause was not isolated until Xiong et al. (arXiv 2002.04745) gave a clean analysis: with Post-Norm placement, the gradient at initialization has expected magnitude that grows with depth, so the first few optimizer steps overshoot catastrophically unless the learning rate is held small enough to compensate. Warmup is the workaround.

The proposed fix — moving the LayerNorm inside the residual branch, so it acts on the input to each sublayer rather than on the post-residual sum — became the Pre-Norm recipe, and almost every open transformer trained after 2020 uses it. A small minority of recent designs add a second norm also inside the residual branch (Sandwich-LN, see Sandwich-LN entry) or move back toward a modified Post-Norm with extra stabilizers (OLMo 2 reordered Post-Norm).

The contribution sentence: the placement of normalization relative to the residual is the single architectural choice that decides whether a deep transformer can be trained at all without warmup.

§ 2 · Derivation

Three placements, three gradient stories

Let $\mathbf{x}_\ell \in \mathbb{R}^d$ be the residual-stream state at layer $\ell$, $f_\ell$ a sublayer (attention or FFN), and $\mathrm{Norm}$ a fixed normalization with unit-variance output (LayerNorm or RMSNorm). The three placements differ in where $\mathrm{Norm}$ sits in the residual diagram:

The OLMo 2 variant (one norm inside, on the sublayer output) is a fourth choice in the same space; see its own entry for the analysis.

The Post-Norm gradient blow-up. Differentiate the loss $L$ with respect to the intermediate state $\mathbf{x}_\ell$. Under the Post-Norm recursion, applying the chain rule through $\mathrm{Norm}$ at each layer gives, schematically (Xiong et al. §2 and Theorem 1):

where $\mathbf{J}_g$ denotes the Jacobian of $g$. The key observation: the Jacobian of $\mathrm{Norm}$ is proportional to $1/\|\mathbf{x}_k + f_k(\mathbf{x}_k)\|$, but the unnormalized sum has norm that grows with the magnitude of $f_k$‘s output. At initialization, $f_k$ produces an output with the same order of magnitude as $\mathbf{x}_k$, so each layer’s post-residual sum has variance $\approx 2$ times the input. After $L$ layers the cumulative factor is $2^{L/2}$ — gradients explode.

Xiong et al. (Theorem 1, Proposition 1) make this precise: under standard Xavier init, the expected gradient magnitude at the last sublayer of a Post-Norm transformer is $\Theta(d \sqrt{\ln d})$ at depth $L$, independent of $L$, but the individual layer gradients are $\Theta(\sqrt{L})$ — the loss surface is poorly conditioned at depth.

The Pre-Norm identity path. Under the Pre-Norm recursion, $\mathbf{x}_{\ell+1} = \mathbf{x}_\ell + f_\ell(\mathrm{Norm}(\mathbf{x}_\ell))$, the residual addition operates on the unnormalized stream $\mathbf{x}_\ell$ directly. The Jacobian becomes (Xiong et al. Proposition 2):

The identity $\mathbf{I}$ dominates at initialization because the inner Jacobian product is small (the sublayer is randomly initialized; its contribution starts near zero). The product over $L$ layers stays close to $\mathbf{I}$ — gradients pass through depth almost unchanged. Xiong et al. (Theorem 1, also Wang et al. arXiv 1906.01787 Theorem 1) prove the resulting gradient magnitude is $\Theta(d \sqrt{\ln d} / \sqrt{L})$, decreasing with depth. Deeper Pre-Norm transformers train more stably than shallow ones at the same learning rate.

This is the entire mathematical case for Pre-Norm: the identity term in the residual stream is unmediated by the normalization, so the depth-wise Jacobian product collapses to nearly $\mathbf{I}$. Warmup becomes unnecessary because the first optimizer step is no longer riding a $2^{L/2}$-amplified gradient.

The Sandwich tradeoff. Sandwich-LN inserts a second normalization inside the residual branch, after $f_\ell$. The residual addition still operates on the unnormalized stream (Pre-Norm-like gradient flow), but each block’s contribution to the residual is now $\mathrm{Norm}_\text{post}(f_\ell(\mathrm{Norm}_\text{pre}(\mathbf{x}_\ell)))$ — a bounded-norm perturbation. The residual stream norm $\|\mathbf{x}_\ell\|_2$ no longer grows unboundedly with $\ell$, addressing a secondary problem with Pre-Norm: that the residual norm at deep layers is much larger than at shallow layers, making per-layer LayerNorm-style output statistics harder to condition for downstream layers and for the final unembedding LM head. The OLMo 2 entry covers a one-sided variant of the same idea.

Parameter and FLOP cost across placements. Per block: Pre-Norm uses 2 normalizations (one per sublayer), Sandwich-LN uses 4 (two per sublayer), Post-Norm uses 2. With RMSNorm at hidden width $d$, each normalization adds $d$ parameters and $O(d)$ FLOPs per token. For an $L$-layer model, Sandwich adds $2 L d$ parameters and $2 L T d$ FLOPs per forward pass relative to Pre-Norm — negligible vs the $O(L d^2)$ parameter count of the projections and the $O(L T^2 d)$ attention FLOPs.

§ 3 · Reference implementation

Reference implementation

# Post-Norm — Vaswani 2017, requires warmup.
def block_post(x, attn, ffn, n1, n2):
    x = n1(x + attn(x))          # norm AFTER residual add
    x = n2(x + ffn(x))
    return x

# Pre-Norm — Xiong 2020, modern open-weights default.
def block_pre(x, attn, ffn, n1, n2):
    x = x + attn(n1(x))          # norm BEFORE sublayer input
    x = x + ffn(n2(x))           # identity path stays unnormalized
    return x

# Sandwich — CogView 2021 / Gemma 2/3, two norms per sublayer inside the residual.
def block_sandwich(x, attn, ffn, n1, n2, n3, n4):
    x = x + n2(attn(n1(x)))      # both norms inside the residual branch
    x = x + n4(ffn(n3(x)))       # bounds the per-block perturbation magnitude
    return x

# OLMo 2 — one norm AFTER each sublayer, inside the residual.
def block_olmo2(x, attn, ffn, n1, n2):
    x = x + n1(attn(x))          # norm after sublayer, before residual add
    x = x + n2(ffn(x))
    return x

All four variants share the same parameter inventory for the sublayers; only the normalization layout differs. The final norm on the network output (x = n_out(x) before the unembedding) is universal across all four — it sits outside the per-block recursion.

§ 4 · Empirical evidence

Empirical evidence

Original Pre-Norm vs Post-Norm (Xiong et al. 2020). The paper trains transformer base and big variants on IWSLT and WMT translation tasks (Table 2). Headline results: Post-Norm without warmup diverges at the base size on WMT-EnDe (training loss does not decrease for the first 1000 steps, then explodes); Pre-Norm without warmup reaches the same BLEU as Post-Norm with warmup, in roughly half the wall-clock. With warmup tuned, Post-Norm BLEU is 0.4 higher than Pre-Norm at the base size, suggesting a small quality lift from the extra normalization on the residual — at the cost of fragile training.

Earlier empirical observation (Wang et al. 2019). Wang, Li, and Tu (arXiv 1906.01787) made the same observation a few months earlier, in a translation-specific paper that trained 30-layer encoders. Their Table 5 shows that Post-Norm transformers diverged at 20 layers, while Pre-Norm trained stably at 30 layers and beat shallow Post-Norm baselines on WMT-EnDe.

DeepNet (Wang et al. 2022). Wang et al. (arXiv 2203.00555) showed that modified Post-Norm with carefully scaled initialization can train to depth 1000 on translation — implying that the Pre-Norm vs Post-Norm choice is not absolute but is mediated by initialization. DeepNet multiplies the residual branch by a depth-dependent factor and reports stable training of encoder-decoders 200 layers deep. The technique did not spread into the open dense LLM stack, but it is the strongest existing evidence that Post-Norm’s gradient problem is a solvable engineering issue rather than a fundamental block.

OLMo 2 (AI2, 2024). OLMo 2 (arXiv 2501.00656, §3.1, Figure 5) deliberately reverts toward Post-Norm character (their “reordered Post-Norm” — RMSNorm after each sublayer, inside the residual). The team reports the choice eliminated periodic loss spikes that the OLMo 1 Pre-Norm baseline exhibited across a 5T-token training run on the 13B model. The caveat: OLMo 2 pairs the placement change with QK-Norm and a Z-loss; ablating any one of the three returns some instability.

Gemma 2 / 3 (Google, 2024–2025). The Gemma 2 report (arXiv 2408.00118) and Gemma 3 report (arXiv 2503.19786, Table 2) adopt the Sandwich-LN placement (a Pre-Norm and a Post-Norm per sublayer, both inside the residual branch). The reports attribute the stability of the unified 1B-to-27B Gemma 3 recipe in part to the sandwich choice, though they do not isolate the placement’s contribution from the other stability tools (QK-Norm, careful init) in an ablation.

The dominant open stack. Across the 2024–2026 production open-weight families tracked in this knowledge base — Llama, DeepSeek, Qwen, Mistral, Hunyuan, Kimi, MiniMax — plain Pre-Norm with RMSNorm remains the consensus. The two minority branches (Sandwich-LN in Gemma 2/3, reordered Post-Norm in OLMo 2/3) both occupy the “bounded per-block perturbation” end of the design space and are independently defensible. No publicly documented frontier model in this knowledge base has reverted to plain Vaswani-style Post-Norm; all such designs either modify the placement, the init (DeepNet style), or pair the change with additional stabilizers.

The honest gap. No public study has run a clean ablation isolating Pre-Norm vs Sandwich-LN vs reordered Post-Norm at fixed scale (say, 70B parameters with identical data, optimizer, and hyperparameters). The placement choices in production models are correlated with team-specific stability tools — the choices co-vary. The strongest cross- production claim that can be made from the public record is that all three placements work at frontier scale when paired with appropriate stabilizers; the choice between them is under-determined by the public ablations.

@article{arxiv2002_04745,
  title  = {On Layer Normalization in the Transformer Architecture},
  author = {Ruibin Xiong and others},
  year   = {2020},
  eprint = {2002.04745},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2002.04745}
}

@article{arxiv_2002_04745,
  title         = {On Layer Normalization in the Transformer Architecture},
  author        = {Ruibin Xiong and and others},
  year          = {2020},
  eprint        = {2002.04745},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2002.04745}
}

{
  "id": "arxiv_2002_04745",
  "type": "article-journal",
  "title": "On Layer Normalization in the Transformer Architecture",
  "author": [
    {
      "literal": "Ruibin Xiong"
    },
    {
      "literal": "et al."
    }
  ],
  "issued": {
    "date-parts": [
      [
        2020
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2002.04745",
  "number": "2002.04745",
  "source": "arXiv"
}

TY  - JOUR
TI  - On Layer Normalization in the Transformer Architecture
AU  - Ruibin Xiong
AU  - et al.
PY  - 2020
JO  - arXiv
AN  - arXiv:2002.04745
UR  - https://arxiv.org/abs/2002.04745
ER  -