The Residual Stream

§ 1 · Premise

The pre-residual depth wall

By 2014, the open question for image recognition was whether stacking more layers would continue to buy accuracy. VGG-19 had pushed plain-feedforward depth to nineteen weight layers (Simonyan & Zisserman 2015, §3); GoogLeNet had reached twenty-two via inception modules with auxiliary classifiers (Szegedy et al. 2015, §5). Past those depths, naive plain networks regressed on the training set itself. He et al. (2015) document this with the canonical plot of CIFAR-10 training error rising as a 56-layer plain net is compared to a 20-layer one — not a generalization gap, an optimization gap (He et al. 2015, Figure 1).

The diagnosis is sharp: deep stacks were being asked to transform activations through a long chain of nonlinearities, when most of those layers should be doing very little — close to the identity. A 50-layer plain net should be able to match its 20-layer prefix simply by setting the last 30 layers to identity, but plain optimization cannot find that solution at random initialization.

A 175-billion-parameter dense decoder running at fp16 weighs 350 GB; its 96-layer stack passes each token’s residual vector through 192 attention/FFN sublayers without losing the addressing structure the unembedding will eventually read. That this works at all is downstream of the one architectural choice this entry is about. The contribution: add the input back, after the sublayer.

§ 2 · Derivation

From transform to perturbation

Write a plain block as a learned function $H_\ell : \mathbb{R}^d \to \mathbb{R}^d$ acting on the activation $\mathbf{x}_\ell \in \mathbb{R}^d$:

The identity is a valid target for $H_\ell$ but not a default; to recover it the weights have to land in a specific nontrivial subspace. Reparameterize: let $f_\ell(\mathbf{x}_\ell) = H_\ell(\mathbf{x}_\ell) - \mathbf{x}_\ell$ and require the block to learn $f_\ell$ directly:

The “do nothing” baseline is now $f_\ell \equiv 0$, reachable by sending the last linear in the sublayer to zero — a single bias initialization rather than a coordinated weight pattern. He et al. argue (§3.1) that if the desired transformation is close to identity, learning the difference is easier than learning the full mapping.

The gradient through $L - \ell$ residual blocks unfolds as a product over $\ell, \ldots, L-1$:

Each factor is the identity plus the Jacobian of the local residual function. Expanded, the product becomes a sum over subsets of layers — the contribution that skips every block is the identity, and small-Jacobian blocks contribute additively rather than multiplicatively. As long as $\|\partial f_k / \partial \mathbf{x}_k\|$ stays bounded, the gradient norm cannot collapse to zero across depth, which is the failure mode Glorot & Bengio (2010) diagnosed for plain sigmoidal stacks (§4).

For a transformer block the same construction is applied twice — once for attention, once for the FFN:

The placement of the normalization inside or outside the addition is the Pre-Norm vs Post-Norm distinction; see the norm-placement entry for the zoo. The residual addition itself — the $+\mathbf{x}_\ell$ — is invariant across those choices.

Why elementwise addition and not concatenation. Concatenation would grow the activation dimension across depth and force every downstream weight matrix to widen, which is incompatible with the weight-sharing pattern of a transformer block. Addition keeps $d$ fixed and turns the sequence of sublayers into a sequence of reads and writes against a single $\mathbb{R}^d$ working memory. Highway Networks (Srivastava et al. 2015) tried a gated convex combination $\mathbf{x}_{\ell+1} = T \odot f(\mathbf{x}_\ell) + (1-T) \odot \mathbf{x}_\ell$ (§2.2); ResNet’s untruncated identity term turned out to be strictly better at depth.

Parameter and compute cost. The residual addition itself adds zero parameters and one elementwise sum per block — $\mathcal{O}(B \cdot T \cdot d)$ FLOPs alongside the $\mathcal{O}(B \cdot T \cdot d^2)$ sublayer compute. The cost is dominated by the sublayer it wraps, by orders of magnitude.

§ 3 · Reference implementation

The two-residual transformer block

def transformer_block(x, attn, ffn, norm1, norm2):
    # x: (B, T, d) — the residual stream
    # Pre-Norm placement: norm is inside the residual branch
    x = x + attn(norm1(x))    # attention writes a delta into the stream
    x = x + ffn(norm2(x))     # FFN writes another delta
    return x

Two residual additions per block, no gating, no learned skip scale, no per-branch scaling. This is the consensus across every dense open-frontier decoder cataloged in the adoption list above: Llama 3.1 (Grattafiori et al. 2024, §3.1), DeepSeek-V3 (DeepSeek-AI 2024, §2.1), Gemma 3 (Gemma Team 2025, §2), OLMo 2 (OLMo Team 2025, §3). Variants in the ReZero, DeepNet, NormFormer, and Hyper-Connections entries decorate this skeleton without removing the bare $+\mathbf{x}$.

§ 4 · Empirical evidence

What the identity skip bought

He et al. report the headline result on ImageNet (Table 4, §4.1): plain-34 lands at 28.54 % top-1 error, ResNet-34 at 25.03 %, and the same training recipe drives ResNet-152 to 21.43 %. The plain-34 training curve also sits above plain-18’s, confirming the optimization gap; the residual variants reverse the ordering as expected. CIFAR-10 results push the same recipe to 110 layers with stable training (Table 6, §4.2). The 1202-layer variant trains without divergence but overfits the small dataset — depth is no longer a stability ceiling, it is a regularization concern.

The transformer port is Vaswani et al. (2017): the original transformer block already includes residual additions around both sub-layers (§3.1, “We employ a residual connection … around each of the two sub-layers”). No subsequent decoder family removed them. GPT-2’s 48-layer stack (Radford et al. 2019, §2), GPT-3’s 96-layer stack (Brown et al. 2020, §2.1), Chinchilla’s 80-layer 70B (Hoffmann et al. 2022, §3), and the 61-layer DeepSeek-V3 all rely on the unchanged additive skip.

Mechanistic-interpretability framing. Elhage et al. (2021) named the ”$\mathbf{x}_\ell \in \mathbb{R}^d$ across depth” object the residual stream and showed it behaves as the model’s working memory: attention heads and FFN neurons are read/write operations against it, decomposable into linearly separable contributions (Anthropic transformer-circuits framework, §“Residual Stream as Communication Channel”). That framing is now the standard lens for circuit-level analyses of trained transformers.

Norm growth across depth. Pre-Norm residual stacks exhibit monotonic norm growth: each block adds a perturbation, nothing trims the stream. Xiong et al. (2020, §4.2) document the growth profile and show Pre-Norm transformers train stably without warmup precisely because of the controlled gradient term identified above (arXiv 2002.04745). Liu et al. (2020) — “Understanding the Difficulty of Training Transformers” — extend the analysis and attribute Post-Norm’s training instability past ~12 layers to the absence of the same identity term in the gradient product (arXiv 2004.08249, §3).

When the simple recipe is modified. Three directions appear in subsequent entries. ReZero (Bachlechner et al. 2020) inserts a learnable scalar at zero init so the network starts at exact identity. DeepNet (Wang et al. 2022) rescales the Post-Norm residual by a depth-derived constant to stabilize 1000-layer training. Hyper-Connections (Zhu et al. 2024) replaces the single stream with a bank of $n$ parallel streams plus learned mixing. None has displaced the unadorned $\mathbf{x} + f(\mathbf{x})$ in production decoders as of the May 2026 release window.

No public depth-vs-skip ablation in the modern frontier scale. Open frontier labs have not published a Llama-scale ablation that re-runs the no-residual baseline, presumably because plain-stack divergence at $L = 80$ is uncontroversial. The strongest available evidence remains He et al.’s ImageNet/CIFAR experiments and the transformer-era stability analyses above.

@article{arxiv1512_03385,
  title  = {Deep Residual Learning for Image Recognition},
  author = {Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun},
  year   = {2015},
  eprint = {1512.03385},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1512.03385}
}

@article{arxiv_1512_03385,
  title         = {Deep Residual Learning for Image Recognition},
  author        = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},
  year          = {2015},
  eprint        = {1512.03385},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/1512.03385}
}

{
  "id": "arxiv_1512_03385",
  "type": "article-journal",
  "title": "Deep Residual Learning for Image Recognition",
  "author": [
    {
      "literal": "Kaiming He"
    },
    {
      "literal": "Xiangyu Zhang"
    },
    {
      "literal": "Shaoqing Ren"
    },
    {
      "literal": "Jian Sun"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2015
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/1512.03385",
  "number": "1512.03385",
  "source": "arXiv"
}

TY  - JOUR
TI  - Deep Residual Learning for Image Recognition
AU  - Kaiming He
AU  - Xiangyu Zhang
AU  - Shaoqing Ren
AU  - Jian Sun
PY  - 2015
JO  - arXiv
AN  - arXiv:1512.03385
UR  - https://arxiv.org/abs/1512.03385
ER  -