ReZero — Residual With Learnable Skip Scale

§ 1 · Premise

Deep transformers cannot start anywhere

The residual entry shows that the identity term inside each block’s Jacobian $I + \partial f_\ell/\partial \mathbf{x}_\ell$ is what makes deep stacks trainable. But the identity is only one term — the magnitude of the sublayer Jacobian at initialization still governs how the product of layer Jacobians behaves at finite depth.

Bachlechner et al. (2020) frame the problem through the input–output Jacobian of an $L$-block stack at initialization. For Post-Norm transformers without warmup, the singular values of $\partial \mathbf{x}_L / \partial \mathbf{x}_0$ blow up exponentially in $L$ on the upper end and collapse toward zero on the lower end — past depth $\sim$12 layers the optimizer diverges in the first few steps (Bachlechner et al. 2020, §2 + Figure 3). Pre-Norm trains but its gradient distribution shifts unevenly across depth: early layers see much larger gradients than late ones (Xiong et al. 2020, §4.2). Standard remedies are LayerNorm (magnitude control), Xavier/He initialization (per-layer variance), and a learning-rate warmup schedule that keeps optimizer step sizes small until the gradient distribution settles — three knobs whose joint tuning is brittle past 64 layers.

The premise: if every block started as exactly the identity, none of these knobs would be load-bearing. Gradient norm through the block would be exactly one. The optimizer could safely take full step sizes from step zero, and the network would learn its way away from the identity function only as the loss demanded.

§ 2 · Derivation

One scalar per branch, initialized to zero

Start from the standard residual block with sublayer $f_\ell : \mathbb{R}^d \to \mathbb{R}^d$ (self-attention or FFN):

ReZero replaces this with a learnable scalar $\alpha_\ell \in \mathbb{R}$ gating the sublayer’s contribution, and drops the normalization:

Initialized at $\alpha_\ell^{(0)} = 0$, the block is literally the identity map at step zero; its sublayer parameters can be set arbitrarily without disturbing the forward pass. The Jacobian factor becomes

so at step zero the per-block Jacobian is exactly $I$ and the full input–output Jacobian $\partial \mathbf{x}_L / \partial \mathbf{x}_0 = I$ regardless of $L$. Singular values are all one. Backpropagated gradient norms are independent of depth, and the optimizer is free to take its full nominal step.

Why “init to zero” rather than “init to small.” A small but nonzero $\alpha$ still leaves the multiplicative product $\prod_\ell (1 + \alpha\, \sigma_k(\partial f_\ell))$ depth-sensitive on the order of $e^{L\alpha\sigma}$. Zero is the unique value that makes the product exactly one for every $L$. Bachlechner et al. (§3) prove that gradient signal at initialization is preserved with no dependence on layer count.

Why drop the norm. Once $\alpha_\ell$ controls the magnitude of the sublayer’s contribution end-to-end, normalization becomes redundant in the regime that matters: the addition can never push the stream into a numerical range that bottoms out gradients, because $\alpha_\ell$ starts small. The paper also argues (§3.1) that LayerNorm’s centering step actively interferes with the proof of dynamical isometry — it makes the per-block Jacobian’s expected singular value depend on properties of the input distribution. Removing it restores a clean analysis.

Dynamical-isometry reading. ReZero is a particular instance of dynamical isometry (Pennington et al. 2017, §2): the condition that the input–output Jacobian of a deep network has all singular values close to one at initialization. Standard initialization schemes (Xavier, He, orthogonal) achieve this in expectation for one layer but not after $L$-fold composition through nonlinearities. ReZero achieves exact dynamical isometry by construction, for arbitrary $L$, with no constraints on the sublayer’s internal weights.

Gradient and training-time scaling. As training proceeds, $\alpha_\ell$ moves away from zero — but the paper shows (Figure 4) that the $\alpha_\ell$ values spread: early layers grow $\alpha$ toward $\sim 0.1$–$1.0$ while deep layers stay near zero. The network self-allocates “effective depth” rather than using all $L$ blocks equally. This is impossible to achieve with fixed Pre-Norm because every block always contributes a nontrivial perturbation.

Parameter and compute cost. One additional scalar per sublayer — $2L$ total scalars for an $L$-layer transformer. Compute overhead is $L$ scalar multiplies, $\mathcal{O}(B \cdot T \cdot d)$ each, negligible against the $\mathcal{O}(B \cdot T \cdot d^2)$ sublayer compute. The normalization layers (and their parameters and FLOPs) are gone, which is a small net savings.

§ 3 · Reference implementation

ReZero block, sketch

class ReZeroBlock(nn.Module):
    def __init__(self, sublayer):
        super().__init__()
        self.sublayer = sublayer                          # attn or FFN
        self.alpha = nn.Parameter(torch.zeros(1))         # init to 0 — load-bearing
    def forward(self, x):
        # x: (B, T, d) — residual stream
        return x + self.alpha * self.sublayer(x)          # no LayerNorm

Two such blocks per transformer layer (attention + FFN), each with its own $\alpha_\ell$. The sublayer is unchanged from the standard transformer; the load-bearing edit is the * alpha and the absence of any norm.

§ 4 · Empirical evidence

What ReZero reports — and what others reproduced

The paper’s headline experiments (Bachlechner et al. 2020, §4):

• 128-layer transformer trains from scratch without warmup, without LayerNorm, on the enwiki8 character LM benchmark (§4.2, Figure 5). The matched 128-layer Post-Norm baseline diverges within the first few hundred steps; the Pre-Norm 128-layer baseline trains but reaches a higher loss at the same step budget.
• WikiText-103 perplexity at 12 to 64 layers matches Pre-Norm + LayerNorm baselines, with 56 % fewer iterations to reach the baseline’s final perplexity at 12 layers (Table 4, §4.3).
• Image classification (ResNet-110 on CIFAR-10) trains stably with ReZero applied to each residual block, matching the original ResNet recipe within statistical noise (§4.4).
• Recurrent networks with ReZero on the recurrence also benefit, suggesting the mechanism generalizes beyond transformers (§4.5).

Independent ablations. Liu et al. (2020), “Understanding the Difficulty of Training Transformers”, arrive at a closely related diagnosis: they propose Admin (§5, arXiv 2004.08249), a per-branch constant scaling factor $\omega_\ell$ derived from a forward-variance calculation, which plays the same role as ReZero’s $\alpha_\ell$ but is fixed rather than learned. Admin trains 12-layer Post-Norm transformers stably without warmup and converges to lower loss than the Pre-Norm baseline on WMT’14 EN-DE (their Table 2). The agreement on mechanism — some per-branch scaling controlling early-step Jacobian magnitude — strengthens the ReZero analysis.

SkipInit (De & Smith 2020). A concurrent ICLR submission shows that initializing the final BatchNorm $\gamma$ of each residual block to zero yields the same depth-stability effect for ResNets and removes the need for BatchNorm itself (§3, arXiv 2002.10444). The “skip init” name describes ReZero’s mechanism almost word-for-word, with the difference being which scalar is targeted — the final-layer normalization $\gamma$ vs an explicit branch multiplier. Both pieces of work identify the same fact: zero-initialized residual scale is the minimal condition for depth-independent gradient flow.

T-Fixup (Huang et al. 2020) provides the same deep-transformer recipe with a different implementation: rescaled initialization of attention and FFN weights, no LayerNorm, no warmup, producing stable training of 100+ layer transformers (§3, arXiv 2002.04745). T-Fixup and ReZero are different recipes for the same dynamical-isometry target; the comparison is in the T-Fixup paper’s §5.

Scaling-curve evidence at frontier scale. None. Bachlechner et al.’s largest experiment is a 128-layer character LM on enwiki8 — small by 2026 standards. No public open-frontier release has run a Llama-scale ablation of ReZero against Pre-Norm + RMSNorm, so “ReZero at 70B+ parameters” remains an open empirical question. The closed-frontier policy applies: speculating on which production labs may have tested it is out of scope.

Why production decoders didn’t adopt it. The consensus stack (Pre-Norm + RMSNorm at every depth tier from 7B to 671B) is a known quantity with charted failure modes and hyperparameter sweet spots across thousands of training runs. ReZero’s headline benefit applies at depths ($L \gtrsim 100$) past the production envelope: Llama 3.1 405B is 126 layers (Grattafiori et al. 2024, Table 3) and trains under Pre-Norm + RMSNorm; DeepSeek-V3 is 61 layers (DeepSeek-AI 2024, §2.1). Switching to ReZero is risk without a documented reward at those depths. DeepNet revisits the same problem with a different scaling and reportedly scales to 1000 layers; that lineage is the live one in research.

@article{arxiv2003_04887,
  title  = {ReZero is All You Need: Fast Convergence at Large Depth},
  author = {Thomas Bachlechner and others (UCSD)},
  year   = {2020},
  eprint = {2003.04887},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2003.04887}
}

@article{arxiv_2003_04887,
  title         = {ReZero is All You Need: Fast Convergence at Large Depth},
  author        = {Thomas Bachlechner et al. (UCSD)},
  year          = {2020},
  eprint        = {2003.04887},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2003.04887}
}

{
  "id": "arxiv_2003_04887",
  "type": "article-journal",
  "title": "ReZero is All You Need: Fast Convergence at Large Depth",
  "author": [
    {
      "literal": "Thomas Bachlechner et al. (UCSD)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2020
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2003.04887",
  "number": "2003.04887",
  "source": "arXiv"
}

TY  - JOUR
TI  - ReZero is All You Need: Fast Convergence at Large Depth
AU  - Thomas Bachlechner et al. (UCSD)
PY  - 2020
JO  - arXiv
AN  - arXiv:2003.04887
UR  - https://arxiv.org/abs/2003.04887
ER  -