Dynamic Hyper-Connections

§ 1 · Premise

Static routing leaves expressivity on the table

Static Hyper-Connections (SHC) generalize the single residual stream to $n$ parallel streams plus learned per-layer read/write coefficient vectors $\mathbf{a}_\ell^{(r)}, \mathbf{a}_\ell^{(w)} \in \mathbb{R}^n$ and a depth-rate matrix $\mathbf{B}_\ell \in \mathbb{R}^{n \times n}$. The construction is strictly more expressive than the standard residual (Zhu et al. 2024, §3.1), but it applies the same routing pattern to every token at layer $\ell$.

The implicit assumption is that all tokens benefit from the same layer-$\ell$ stream mix. That is unlikely to hold uniformly. A noun-phrase token might benefit from carrying its semantic features through one stream while a function-word token routes positional information through another. Mixture-of-Experts routing — token-level expert selection at every layer — already concedes the same point about token-conditional structure (Shazeer et al. 2017, §2.1); applying the same principle to residual-stream routing is the Dynamic Hyper-Connections (DHC) contribution.

DHC is presented in the same paper as SHC and is the variant that produces the strongest empirical results in the paper’s tables; it can be read as the “real” Hyper-Connections proposal, with SHC as the conservative starting point that makes the gains attributable to the multi-stream construction rather than to per-token gating in general.

§ 2 · Derivation

Predict $\mathbf{a}^{(r)}, \mathbf{a}^{(w)}, \mathbf{B}$ from the per-token state

Start from the SHC update at layer $\ell$ for a single token at sequence position $t$:

In SHC, the three routing parameters $\mathbf{a}_\ell^{(r)}, \mathbf{a}_\ell^{(w)}, \mathbf{B}_\ell$ are layer-constants. DHC promotes them to functions of the per-token state $\mathbf{H}_\ell^{(t)} \in \mathbb{R}^{n \times d}$ via small linear projections.

Token summary. First, build a $d$-dimensional summary of the token’s $n$-stream state using a learned mix $\mathbf{m} \in \mathbb{R}^n$ that is shared across layers:

In the paper’s implementation, $\mathbf{m}$ is initialized to uniform so the summary starts as a stream average; the network learns to skew the summary toward whichever streams carry the most decision-relevant features.

Per-token read and write coefficients. Project the summary onto two $\mathbb{R}^n$ vectors:

where $W_\ell^{(r)}, W_\ell^{(w)} \in \mathbb{R}^{n \times d}$ are per-layer projection weights and $\mathbf{a}_\ell^{(r,0)}, \mathbf{a}_\ell^{(w,0)} \in \mathbb{R}^n$ are per-layer biases that recover SHC behavior when the projection-weight contribution is small. Each token thus picks its own read/write mix.

Per-token rate matrix. The depth-rate matrix is also predicted per token, via a low-rank projection that keeps the parameter count tractable:

The outer product of two predicted $\mathbb{R}^n$ vectors gives a rank-1 perturbation on top of the per-layer constant $\mathbf{B}_\ell^{(0)}$. (The paper allows the rank to be lifted by predicting multiple such outer products; the rank-1 form is the lightweight default.)

The DHC update. Substitute the per-token coefficients into the SHC update:

The sublayer $f_\ell$ is unchanged — DHC adds routing machinery around attention and FFN without touching their internals.

Reduction to SHC. Set $W_\ell^{(r)} = W_\ell^{(w)} = W_\ell^{(B,u)} = W_\ell^{(B,v)} = 0$. The per-token coefficients collapse to their per-layer biases and DHC becomes SHC. DHC is a strict generalization, just as SHC is a strict generalization of the standard residual.

Why this and not “predict the full $\mathbf{B}$ unconstrained.” A full per-token $\mathbb{R}^{n \times n}$ matrix would require $n^2 d$ parameters per layer ($\sim 2.6 \times 10^5$ weights at $n = 4$, $d = 4096$) just for the rate matrix — comparable to the rest of the block. The rank-1 (or low-rank) parameterization keeps the routing prediction much smaller than the sublayer compute while preserving the expressivity gain. The same low-rank-routing argument appears in MoE literature (Shazeer et al. 2017, §2.1).

Initialization for stability. The projection weights $W_\ell^{(\cdot)}$ are initialized to zero or near-zero; the biases recover SHC’s initialization (Zhu et al. 2024, §3.3). Network behavior at step zero is exactly SHC; per-token routing emerges as the projection weights grow during training. This guarantees that DHC inherits SHC’s depth-stability properties at initialization.

Parameter and compute overhead over SHC. Per layer, four additional $\mathbb{R}^{n \times d}$ projections ($W_\ell^{(r)}, W_\ell^{(w)}, W_\ell^{(B,u)}, W_\ell^{(B,v)}$) — total $4nd$ parameters. At $n = 4, d = 4096$: $\sim 65$K parameters per layer, against $\sim 200$M for the attention and FFN. Compute overhead is four small matmuls $\mathbb{R}^d \to \mathbb{R}^n$ per token per layer: $\mathcal{O}(B \cdot T \cdot n \cdot d)$ FLOPs, $\sim n / d$ times smaller than the sublayer compute and therefore negligible. Activation memory is the same as SHC: $n \times d$ per token per layer.

§ 3 · Reference implementation

Dynamic Hyper-Connection block, sketch

def dhc_block(H, sublayer, mix_m, W_r, W_w, W_Bu, W_Bv, a_r0, a_w0, B0):
    # H: (B, T, n, d) — stack of n parallel residual streams per token
    x_bar = (mix_m[None, None, :, None] * H).sum(dim=-2)        # (B, T, d) token summary
    a_r = W_r @ x_bar + a_r0                                    # (B, T, n) per-token read mix
    a_w = W_w @ x_bar + a_w0                                    # (B, T, n) per-token write mix
    B_u = W_Bu @ x_bar                                          # (B, T, n) low-rank rate factor
    B_v = W_Bv @ x_bar                                          # (B, T, n)
    B_t = B0 + torch.einsum("bti,btj->btij", B_u, B_v)          # (B, T, n, n) per-token rate matrix

    r = (a_r[..., None] * H).sum(dim=-2)                        # (B, T, d) read
    u = sublayer(r)                                             # (B, T, d) sublayer
    H = torch.einsum("btij,btjd->btid", B_t, H)                 # (B, T, n, d) rate-mix update
    H = H + a_w[..., None] * u[..., None, :]                    # write per-token
    return H

The routing parameters become per-token tensors rather than per-layer constants; the rest of the SHC machinery is preserved.

§ 4 · Empirical evidence

What DHC adds over SHC

Headline language-modeling result (Zhu et al. 2024, §4.2, Table 1): on a 1.3B dense LM at matched parameter count and training budget, DHC outperforms SHC ($n = 4$) by 0.2 perplexity and the Pre-Norm baseline by 0.8 perplexity. The DHC advantage over SHC grows modestly at the 7B-active MoE scale to 0.3 perplexity (Table 3), suggesting per-token routing is more load-bearing as the model has more representational capacity to specialize.

Depth stability (§4.3, Figure 4): DHC at 96 layers matches SHC’s improved seed-to-seed stability over the Pre-Norm baseline. The per-token routing does not destabilize training; the initialization (projection weights at zero) recovers SHC behavior at step zero.

Routing analysis (§4.6, Figure 6): the paper visualizes the learned per-token routing patterns and finds that DHC clusters tokens by syntactic / semantic role (high-frequency function words route differently from content words) at the routing-coefficient level. The mechanism is interpretable in a way SHC cannot be, since SHC’s routing is token-independent by construction.

Vision (§4.5): the consistent ranking DHC > SHC > Pre-Norm holds on ViT ImageNet classification and DiT image generation. Cross-modality consistency strengthens the per-token- routing benefit claim.

Comparison to MoE-style token routing. DHC’s routing mechanism is mechanistically similar to Mixture-of-Experts routing (Shazeer et al. 2017, §2.1) but applied at the residual-stream level rather than the FFN-expert level. The two are complementary: a MoE transformer with DHC residuals would have token-level routing both for which FFN expert to consult and for which residual stream to write back into. The Zhu et al. MoE experiment (§4.4) combines DHC and DeepSeekMoE-style expert routing; they do not interfere and the gains compose.

Independent reproduction. As with SHC, no independent frontier-scale reproduction has appeared in the open literature as of May 2026. The DenseFormer family (Liu et al. 2024, §3) is the closest related design — depth-wise dense connections between residual states — but DenseFormer’s mixing is static across tokens, more comparable to SHC than to DHC.

Production status. No publicly documented production deployment of DHC. The ByteDance Doubao team that introduced the technique does not publish enough architecture detail on its production models to confirm or rule out use; per the closed-model policy, no claim is made either way. Other open frontier labs continue with the standard residual or with placements (Gemma 3’s norm-everywhere, OLMo 2’s reordered norms) that operate on a single stream.

For the static rate-matrix construction that DHC sits on top of, see Hyper-Connections. The two entries cover the same paper (arXiv 2409.19606) but separate the multi-stream contribution from the per-token routing contribution so each can be evaluated on its own.

@article{arxiv2409_19606,
  title  = {Hyper-Connections},
  author = {Defa Zhu and others (ByteDance Doubao)},
  year   = {2024},
  eprint = {2409.19606},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2409.19606}
}

@article{arxiv_2409_19606,
  title         = {Hyper-Connections},
  author        = {Defa Zhu et al. (ByteDance Doubao)},
  year          = {2024},
  eprint        = {2409.19606},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2409.19606}
}

{
  "id": "arxiv_2409_19606",
  "type": "article-journal",
  "title": "Hyper-Connections",
  "author": [
    {
      "literal": "Defa Zhu et al. (ByteDance Doubao)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2409.19606",
  "number": "2409.19606",
  "source": "arXiv"
}

TY  - JOUR
TI  - Hyper-Connections
AU  - Defa Zhu et al. (ByteDance Doubao)
PY  - 2024
JO  - arXiv
AN  - arXiv:2409.19606
UR  - https://arxiv.org/abs/2409.19606
ER  -