Layer Normalization

§ 1 · Premise

What Batch Normalization could not do

By 2016 the standard normalizer for feed-forward and convolutional networks was Batch Normalization (Ioffe & Szegedy, arXiv 1502.03167). BatchNorm normalizes a single feature channel across the minibatch: for a channel $j$ at position $t$ in a batch of size $B$, it estimates the mean and variance $\mu_j = \tfrac{1}{B}\sum_{b} x^{(b)}_{j,t}$ and $\sigma^2_j = \tfrac{1}{B}\sum_{b}(x^{(b)}_{j,t} - \mu_j)^2$ and rescales each activation by these per-channel statistics. On ImageNet-scale ConvNets with batches of 256+ this works extremely well — the empirical statistics are stable, and BatchNorm reduces both training time and final error.

Three structural problems surfaced when the same idea was carried to sequence models, all documented in Ba, Kiros, and Hinton (arXiv 1607.06450, §2):

1. Recurrence breaks the moving-average assumption. An RNN unrolled across $T$ time steps sees a different activation distribution at each $t$. BatchNorm either has to maintain $T$ separate sets of statistics — wasteful and brittle for sequences of variable length — or share statistics across time and pay an accuracy cost from the resulting mismatch.
2. Variable batch size at inference. Autoregressive decoding produces one token at a time, often with batch size 1. BatchNorm’s training behavior (batch statistics) and inference behavior (cached moving averages) diverge sharply at $B=1$; small estimation error in the running averages compounds across many decode steps.
3. Padding pollution. When sentences of different length are batched together, padded positions either contaminate the per-batch statistics or have to be masked out with per-channel reductions that complicate the kernel.

Layer Normalization fixes all three by moving the reduction off the batch axis and onto the feature axis. Each token computes its own statistics from its own $d$-dimensional activation vector, independently of every other token in the batch. The contribution sentence: one normalizer that needs neither a batch nor a sequence-position context to be well-defined.

§ 2 · Derivation

From batch statistics to per-token statistics

Let $\mathbf{x} \in \mathbb{R}^d$ be the pre-normalization activation vector at a single token position, where $d$ is the hidden width (often called $H$ in the paper; we use $d$ to avoid collision with the head dimension in attention contexts). The components of $\mathbf{x}$ index the feature axis. LayerNorm reduces along that axis to produce two scalar summaries:

The mean $\mu \in \mathbb{R}$ measures the dc offset of the activation across features; the variance $\sigma^2 \in \mathbb{R}$ measures the average squared deviation around that offset. Both are functions of $\mathbf{x}$ alone — no other tokens, no other batch elements, no moving average over training history. The normalized activation is

where $\mathbf{1} \in \mathbb{R}^d$ is the all-ones vector and $\varepsilon > 0$ is a small constant (Ba et al. use $\varepsilon = 10^{-5}$ in the recurrent experiments of §6) added to the variance to keep the square root differentiable when $\sigma^2 \to 0$. By construction, $\hat{\mathbf{x}}$ has empirical mean $0$ and empirical variance $1$ over its $d$ components.

A pure normalization to zero mean and unit variance would destroy the network’s ability to represent feature scales that the upstream weights have learned. The fix, identical to BatchNorm’s, is a learnable affine reparameterization with gain $\boldsymbol{\gamma} \in \mathbb{R}^d$ and bias $\boldsymbol{\beta} \in \mathbb{R}^d$:

where $\odot$ is the elementwise (Hadamard) product. The gain vector restores per-feature scale and the bias vector restores per-feature offset; together they make LayerNorm a strict generalization of the identity (set $\boldsymbol{\gamma} = \mathbf{1}, \boldsymbol{\beta} = \mathbf{0}$ and apply only the normalization). Parameter count is $2d$ per LayerNorm instance, which is negligible compared to the $O(d^2)$ projection matrices it sits between.

Why this and not channel-only rescaling? A naive alternative would be to skip mean subtraction and only rescale by $\sigma$. Ba et al. (§5) argue from the invariance properties laid out in their Table 1: BatchNorm is invariant to per-channel shifts of the weights but not to per-token shifts of the inputs; LayerNorm flips that picture. Mean subtraction is what gives LayerNorm its input-shift invariance property — adding the same constant to every feature of $\mathbf{x}$ leaves $\mathrm{LN}(\mathbf{x})$ unchanged. This invariance is the reason LayerNorm tolerates input distributions whose dc level drifts between examples (as they do across RNN time steps and across batched-padded sequences). RMSNorm (Zhang & Sennrich, arXiv 1910.07467) would later show empirically that re-scaling alone is sufficient for transformer training, but the 2016 analysis correctly identifies why the centering step might matter and includes it.

Gradient picture. Differentiating the LayerNorm output with respect to the input yields, after a standard quotient-rule expansion (Ba et al. §3, also derived in Xu et al. arXiv 1911.07013 §3):

where $\delta_{ij}$ is the Kronecker delta. The first term is the dominant identity-like contribution; the $-1/d$ term subtracts the gradient leaking through $\mu$; the $-\hat{x}_i\hat{x}_j/d$ term subtracts the gradient leaking through $\sigma$. Two observations follow from this expression. First, all three terms are $O(1/\sigma)$, so very small variance inputs make gradients explode — hence the $\varepsilon$ floor. Second, the rank-$1$ correction terms enforce that the upstream gradient is projected onto the subspace orthogonal to $\mathbf{1}$ and $\hat{\mathbf{x}}$, which is the geometric statement of “gradients flow only through the directions the normalization did not already fix.”

Computational cost. A single LayerNorm call requires two reductions across the feature axis (one for $\mu$, one for $\sigma^2$), a subtraction, a square-root and reciprocal, an elementwise multiplication by $\boldsymbol{\gamma}$, and an elementwise addition of $\boldsymbol{\beta}$. Total operation count is $O(d)$ per token, dominated by the two reductions. On GPUs the cost is bounded by memory traffic — reading $\mathbf{x}$ once and writing the output once — rather than by arithmetic. For a transformer with $L$ layers, sequence length $T$, batch size $B$, and two LayerNorms per block, the LayerNorm budget over a forward pass is $4 B T L d$ floating-point reads and writes (gain and bias account for the constant 4).

§ 3 · Reference implementation

Reference implementation

def layer_norm(x, gamma, beta, eps=1e-5):
    # x:     [..., d]
    # gamma: [d]   per-feature learnable gain, init to 1
    # beta:  [d]   per-feature learnable bias, init to 0
    mu = x.mean(-1, keepdim=True)                  # [..., 1]
    var = x.var(-1, keepdim=True, unbiased=False)  # [..., 1]
    x_hat = (x - mu) * (var + eps).rsqrt()         # zero mean, unit var
    return gamma * x_hat + beta                    # per-feature affine

def transformer_block_pre_ln(x, attn, ffn, ln1_g, ln1_b, ln2_g, ln2_b):
    # Pre-Norm placement (see norm-placement entry for Post-Norm).
    x = x + attn(layer_norm(x, ln1_g, ln1_b))      # [B, T, d]
    x = x + ffn(layer_norm(x, ln2_g, ln2_b))
    return x

The contract on the input: any tensor whose last axis is the feature axis. The output has the same shape. Production kernels (Apex FusedLayerNorm, PyTorch’s built-in nn.LayerNorm, and the fused implementations in cuDNN / Triton) collapse the two reductions and the affine into a single pass over memory, but the math is what the four lines above describe.

§ 4 · Empirical evidence

Empirical evidence

The original paper (Ba et al. 2016, §6) evaluates LayerNorm primarily on RNN-based tasks where BatchNorm was known to struggle. Headline numbers:

• Image-sentence ranking on COCO (Order Embeddings RNN encoder, §6.1, Table 2): LayerNorm improves Recall@1 from 23.3 to 25.7 over baseline and converges roughly $2\times$ faster than the baseline LSTM, with BatchNorm placed in between.
• Question answering on the Children’s Book Test (Attentive Reader, §6.2): LayerNorm reduces training time by roughly $1.5\times$ to reach the same validation accuracy as the unnormalized baseline.
• Skip-thought vectors (sentence-embedding training, §6.4): the LayerNorm variant converges in 1 month wall-clock vs the BatchNorm variant’s 1 month plus, and matches its downstream evaluation scores on 5 of 6 transfer tasks.

The decisive cross-task pattern is convergence speedup, not final-accuracy headroom — the unnormalized baselines often catch up if trained long enough, but the LayerNorm runs get there faster and with less hyperparameter sensitivity.

The transformer adoption. Vaswani et al. (arXiv 1706.03762, §3.1) chose LayerNorm for the original transformer “Add & Norm” block without an ablation, citing the recurrent-network results above as sufficient evidence that batch-axis normalization was the wrong primitive for sequence models. Every subsequent transformer paper inherited that choice by default through roughly 2019. The placement they used (Post-Norm — normalize after the residual addition) turned out to be the load-bearing weakness; see the Pre/Post/Sandwich placement entry for the analysis from Xiong et al. (arXiv 2002.04745).

The 2019 simplification. Zhang and Sennrich (arXiv 1910.07467, Tables 1–5) ran a direct ablation of LayerNorm’s mean-subtraction step across six tasks (machine translation, reading comprehension, image-sentence ranking, two LM benchmarks, and a CIFAR classifier). They report no statistically significant quality difference between LayerNorm and RMSNorm — which drops $\mu$ and $\boldsymbol{\beta}$ entirely — across any of the six. The wall-clock saving attributed to removing one reduction and one elementwise operation ranged from 7% to 64% depending on the model and framework. This ablation is the strongest single piece of evidence that LayerNorm’s centering step was carrying very little weight for transformers, which in turn explains why every open frontier dense LLM from Llama 1 (arXiv 2302.13971) onward has shipped RMSNorm instead.

Where LayerNorm with bias still survives. The legacy decoder-only family — GPT-2 (Radford et al. 2019), GPT-NeoX-20B (arXiv 2204.06745), the BLOOM-176B release (arXiv 2211.05100) — uses plain LayerNorm. The encoder-decoder line (T5, mT5) uses a modified LayerNorm variant without bias, sitting between the two. Outside of these legacy lines, no recent open dense or MoE LLM in this knowledge base’s adoption tables retains the full LayerNorm formulation; the per-block normalizer is either RMSNorm, a sandwich variant of RMSNorm, or QK-Norm-augmented RMSNorm. No public study has, to this author’s knowledge, isolated whether the bias term in LayerNorm hurts modern transformer quality at all — the empirical case for dropping it rests on parameter count and wall-clock alone.

Subsequent theoretical work. Xu et al. (arXiv 1911.07013, “Understanding and Improving Layer Normalization”) dissect the gradient expression above and report that the derivatives of $\boldsymbol{\gamma}$ and $\boldsymbol{\beta}$ contribute non-trivially to optimization dynamics (their Table 3) — removing the affine entirely hurts convergence on machine-translation benchmarks even when the normalization itself is kept. Their AdaNorm variant rescales the affine adaptively and reports small but consistent BLEU gains on WMT En-De and En-Fr. Neither AdaNorm nor any other LayerNorm-with-modified-affine appears in the production open-weight stack tracked here; the field consolidated on the simpler RMSNorm rather than chasing affine improvements on top of LayerNorm.

@article{arxiv1607_06450,
  title  = {Layer Normalization},
  author = {Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton},
  year   = {2016},
  eprint = {1607.06450},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1607.06450}
}

@article{arxiv_1607_06450,
  title         = {Layer Normalization},
  author        = {Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E. Hinton},
  year          = {2016},
  eprint        = {1607.06450},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/1607.06450}
}

{
  "id": "arxiv_1607_06450",
  "type": "article-journal",
  "title": "Layer Normalization",
  "author": [
    {
      "literal": "Jimmy Lei Ba"
    },
    {
      "literal": "Jamie Ryan Kiros"
    },
    {
      "literal": "Geoffrey E. Hinton"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2016
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/1607.06450",
  "number": "1607.06450",
  "source": "arXiv"
}

TY  - JOUR
TI  - Layer Normalization
AU  - Jimmy Lei Ba
AU  - Jamie Ryan Kiros
AU  - Geoffrey E. Hinton
PY  - 2016
JO  - arXiv
AN  - arXiv:1607.06450
UR  - https://arxiv.org/abs/1607.06450
ER  -