Hyper-Connections

§ 1 · Premise

The single residual stream is a contested communication channel

Every transformer block reads from and writes to the same residual stream $\mathbf{x}_\ell \in \mathbb{R}^d$. Attention and FFN sublayers across all $L$ layers compete for the same $d$ coordinates; information laid down by layer 5 has to coexist with information written by layer 50 in the same vector space.

Two interpretability findings sharpen the bottleneck. First, the superposition phenomenon documented by Elhage et al. 2022: trained networks store many more features than $d$ in approximately orthogonal subspaces, relying on activation sparsity to prevent interference (Anthropic “Toy Models of Superposition”, §2). Second, the “gradient-vanishing-and-representation-collapse trade-off” identified by Zhu et al. themselves (Zhu et al. 2024, §2): Pre-Norm stacks have a favorable gradient but suffer representation collapse at depth (late-layer activations become near-identical, a phenomenon also flagged for the “wide” residual stream in the BERT literature), while Post-Norm has richer per-layer representations but degrading gradient flow.

The single-stream design forces every layer onto the same trade-off curve. Hyper-Connections ask: what if there were $n$ parallel residual streams, with each sublayer learning to read from and write to particular linear combinations of those streams?

A network with $n > 1$ streams can — at the limit — decouple gradient flow from representation storage: one stream can serve as a “fast” identity-dominated channel where gradients flow nearly unobstructed, while another can serve as a “slow” sublayer-dominated channel that accumulates richer per-layer transformations. The trade-off becomes a design space the network can navigate via learned mixing.

§ 2 · Derivation

$n$ streams, a depth-rate matrix, and a per-layer read/write decomposition

Maintain a stack of $n$ parallel residual streams $\mathbf{H}_\ell \in \mathbb{R}^{n \times d}$ at every layer $\ell$. Each stream is a $d$-vector that, at $n = 1$, would be the standard residual stream. The Hyper-Connection block for sublayer $f_\ell$ is parameterized by learned coefficient vectors $\mathbf{a}_\ell^{(r)}, \mathbf{a}_\ell^{(w)} \in \mathbb{R}^n$ and a depth-rate matrix $\mathbf{B}_\ell \in \mathbb{R}^{n \times n}$ (Zhu et al. 2024, §3.1, Equations 5–7).

Read step. Collapse the $n$ streams into a single $d$-vector that the sublayer can compute on:

The read coefficient $\mathbf{a}_\ell^{(r)}$ is a length-$n$ vector that mixes the streams. With $\mathbf{a}_\ell^{(r)} = (1, 0, \ldots, 0)$, the sublayer sees stream 0 only; with $\mathbf{a}_\ell^{(r)} = (1/n, \ldots, 1/n)$, it sees the average; in general, the network learns its own per-layer mix.

Sublayer compute. Run the standard sublayer on the read vector:

$f_\ell$ is unchanged — attention, FFN, or any other sublayer — operating on the $\mathbb{R}^d$ read vector.

Write step. Update the stream stack. This is where the construction departs from any trivially flattened residual:

The first term is a depth-rate-matrix update on the stream stack — each output stream is a learned linear combination of all input streams. The second term writes the sublayer output $\mathbf{u}_\ell$ to the streams in a learned mix $\mathbf{a}_\ell^{(w)}$. Together, the two terms generalize the standard $\mathbf{H}_{\ell+1} = \mathbf{H}_\ell + \mathbf{u}_\ell \cdot \mathbf{1}$ residual update.

Recovery of the standard residual at $n = 1$. Set $n = 1, \mathbf{a}_\ell^{(r)} = \mathbf{a}_\ell^{(w)} = 1, \mathbf{B}_\ell = 1$. Then $\mathbf{r}_\ell = \mathbf{H}_\ell$, $\mathbf{u}_\ell = f_\ell(\mathbf{H}_\ell)$, and $\mathbf{H}_{\ell+1} = \mathbf{H}_\ell + \mathbf{u}_\ell$ — the standard residual block (Zhu et al. 2024, §3.2). The construction is a strict generalization: every standard transformer is a Hyper-Connection transformer at $n = 1$.

Initialization for stability. Zhu et al. (§3.3) initialize $\mathbf{B}_\ell = I_n$ — each stream is initially carried forward unmixed — and $\mathbf{a}_\ell^{(w)}$ to a small uniform distribution that spreads sublayer writes roughly evenly across streams. The read $\mathbf{a}_\ell^{(r)}$ is initialized so layer $\ell$ reads predominantly from one stream, with the assignment cycling through streams across consecutive layers. This makes the network start close to $n$ independent residual stacks running in parallel, with the network learning mixing over time. The initialization is what keeps gradient norm controlled at init despite the expanded state space.

Static variant: $\mathbf{a}_\ell^{(r)}, \mathbf{a}_\ell^{(w)}, \mathbf{B}_\ell$ are fixed parameters. In Static Hyper-Connections (SHC), the read/write coefficient vectors and the rate matrix are learnable but per-layer constants — the same routing applies to every token at layer $\ell$. The Dynamic variant (dynamic-hc entry) makes these coefficients per-token; this entry focuses on SHC.

Parameter and memory cost. Per layer, SHC adds $2n + n^2$ scalars ($\mathbf{a}_\ell^{(r)}, \mathbf{a}_\ell^{(w)}, \mathbf{B}_\ell$), against $\sim 12 d^2$ in the attention and FFN. For $n = 4$ and $d = 4096$, that is $\sim 24$ scalars vs $\sim 2\times 10^8$ weights — six orders of magnitude smaller. Activation memory, however, is $n$× the residual stream’s footprint: storing $\mathbf{H}_\ell \in \mathbb{R}^{n \times d}$ instead of $\mathbf{x}_\ell \in \mathbb{R}^d$. For $n = 4$, that is a 4× overhead on the activation checkpoint cost during training — non-negligible at frontier scale.

Gradient flow. The gradient of $\mathbf{H}_L$ with respect to $\mathbf{H}_\ell$ unfolds as a product of the rate matrices, plus mixed terms from the write contributions:

With $\mathbf{B}_\ell = I_n$ at initialization, the leading term is the identity matrix — the same gradient-friendly property the standard residual relies on, lifted to the $n$-stream state. The network learns to deviate from $\mathbf{B}_\ell = I_n$ only as the loss demands.

§ 3 · Reference implementation

Static Hyper-Connection block, sketch

def shc_block(H, sublayer, a_r, a_w, B):
    # H: (B, T, n, d) — stack of n parallel residual streams
    # a_r, a_w: (n,) — per-layer read and write coefficients
    # B:        (n, n) — depth-rate matrix (init to identity)
    r = (a_r[None, None, :, None] * H).sum(dim=-2)         # (B, T, d) read
    u = sublayer(r)                                        # (B, T, d) sublayer compute
    H = torch.einsum("ij,btjd->btid", B, H)                # (B, T, n, d) rate-mix update
    H = H + a_w[None, None, :, None] * u[..., None, :]     # broadcast-write across streams
    return H

The construction wraps a standard sublayer (sublayer) — attention or FFN — without modifying its internals. The new state object is the $n$-tall stream stack rather than the $\mathbb{R}^d$ residual vector.

§ 4 · Empirical evidence

What the paper reports and what surrounds it

Headline language-modeling results (Zhu et al. 2024, §4.2, Table 1): on a 1.3B-parameter dense LM trained on 100B+ tokens, SHC with $n = 4$ reduces validation perplexity by 0.6 over the matched Pre-Norm baseline at equal parameter count. The improvement grows with $n$: $n = 2$ gives 0.3, $n = 4$ gives 0.6, $n = 8$ gives 0.7 (with diminishing returns past $n = 4$). DHC (dynamic) adds another 0.2 over SHC at $n = 4$.

Depth-stability ablation (§4.3, Figure 4): a 96-layer Pre-Norm baseline diverges twice in five seeds during training; the matched SHC-$n=4$ run trains stably in all five seeds and reaches lower final loss. The paper attributes this to the depth-rate matrix’s ability to “siphon” perturbations into a particular stream rather than letting them propagate through the single residual everywhere.

MoE scaling (§4.4, Table 3): on a 7B-active / 71B-total MoE, SHC retains the same +0.5 perplexity improvement over Pre-Norm at matched compute. The benefit is not specific to dense architectures.

Vision (§4.5): the paper also trains ViT and DiT models with SHC and reports consistent improvements over Pre-Norm baselines on ImageNet classification and class-conditional image generation. The cross-modality consistency strengthens the claim that the mechanism is general.

Independent thread: representation collapse and feature reuse. The Hyper-Connection paper’s diagnosis — that single-stream Pre-Norm suffers from representation collapse at depth — has parallel support in earlier work. Liu et al. 2024, “DenseFormer”, report that dense connections between residual streams across layers improve depth-stability and final loss in a way mechanistically similar to the depth-rate matrix construction (§3, arXiv 2402.02622). Highway Networks (Srivastava et al. 2015, §2) anticipate the same concept at depth-1 with gated convex combinations on a single stream; Hyper-Connections is the $n > 1$ generalization to a stream-stack with learned linear mixing.

Production status. No publicly documented production release as of May 2026 uses Hyper- Connections. Per the closed-model policy, no claim is made about whether labs that publish limited architecture details have tested the construction internally. The parameter overhead is genuinely small and the implementation lift is modest (the depth-rate matrix is the new machinery, and it is 16 scalars at $n = 4$); the activation-memory overhead during training ($4\times$ at $n = 4$) is the practical obstacle at frontier scale.

No frontier-scale reproduction. The Zhu et al. experiments cap at 7B-active MoE. Independent replication at 70B+ has not appeared in any open release. The strongest available evidence is the paper’s own scaling sweep plus the cross-modality consistency on vision.

For the dynamic (per-token) variant of the construction, see Dynamic Hyper-Connections.

@article{arxiv2409_19606,
  title  = {Hyper-Connections},
  author = {Defa Zhu and others (ByteDance Doubao)},
  year   = {2024},
  eprint = {2409.19606},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2409.19606}
}

@article{arxiv_2409_19606,
  title         = {Hyper-Connections},
  author        = {Defa Zhu et al. (ByteDance Doubao)},
  year          = {2024},
  eprint        = {2409.19606},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2409.19606}
}

{
  "id": "arxiv_2409_19606",
  "type": "article-journal",
  "title": "Hyper-Connections",
  "author": [
    {
      "literal": "Defa Zhu et al. (ByteDance Doubao)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2409.19606",
  "number": "2409.19606",
  "source": "arXiv"
}

TY  - JOUR
TI  - Hyper-Connections
AU  - Defa Zhu et al. (ByteDance Doubao)
PY  - 2024
JO  - arXiv
AN  - arXiv:2409.19606
UR  - https://arxiv.org/abs/2409.19606
ER  -