Category
Residual Connections
Skip-connection geometry — from standard residuals to hyper-connections.
- The Residual Stream Residual foundational
Train arbitrarily deep networks without gradient vanishing — by letting each layer learn a perturbation on a clean linear stream rather than transforming activations in place.
- ReZero — Residual With Learnable Skip Scale ReZero research
Train arbitrarily deep transformers without LayerNorm, without warmup, without careful initialization — just one learnable scalar per residual branch, initialized to zero.
- NormFormer — Extra Normalization in the Residual NormFormer research
Add three small per-block normalization terms — a LayerNorm after self-attention, head-wise scaling of attention outputs, and a LayerNorm after the FFN's first linear — that speed up Pre-Norm transformer pretraining by ~24% with no downstream architectural changes.
- DeepNet — Scaling Transformers to 1000 Layers DeepNet research
Train 1000+ layer transformers stably by rescaling the residual addition by a depth-derived constant α and shrinking sublayer-weight initialization by a companion β, recovering Post-Norm's expressivity with Pre-Norm's stability.
- Dynamic Hyper-Connections DHC / mHC research
Make Hyper-Connections' read/write coefficients per-token rather than per-layer — letting the network route different tokens through different residual streams within the same layer.
- Hyper-Connections HC research
Replace the single residual stream with a small bank of parallel streams plus learned read/write matrices, letting different sublayers route information through different channels.