Browse
Timeline
Every technique in the knowledge base, in the order it appeared. 55 entries spanning 2015 to 2025.
2015
2016
- Jun Gaussian Error Linear Unit GELU FFN & MoE Multiply x by the standard normal CDF: GELU(x) = x · Φ(x). Smoother than ReLU, weighted by the input's z-score. The default transformer FFN activation in BERT, GPT-2, GPT-3 era; superseded by gated variants (SwiGLU, GeGLU) but still the substrate they sit on.
- Jul Layer Normalization LayerNorm Normalization Per-token feature-axis normalization with learnable scale and shift. The 2016 building block the original transformer used; superseded in modern decoders by RMSNorm, which drops the mean-centering step at no quality cost.
2017
- Jan Sparsely-Gated MoE Sparse MoE FFN & MoE Top-K gating with per-expert noise plus an importance-balancing loss. The 2017 paper that established that 'sparse activation + load balance' is a viable training-time recipe; every modern MoE descends from it.
- Jun FFN with ReLU FFN-ReLU FFN & MoE The original transformer FFN: ReLU squashing of a 4× hidden expansion, projected back. Foundational and historical — every later FFN variant (GELU, SwiGLU, GeGLU) is a refinement of this two-layer-with-nonlinearity pattern.
- Jun Multi-Head Attention MHA Attention Mechanisms The original attention layout: H independent heads, each computing scaled dot-product attention on a d_h-dim subspace, concatenated then projected. Now superseded by GQA, MLA in production decoders, but still the conceptual foundation.
- Jun Sinusoidal Position Encoding Sinusoidal Positional Encoding Add a fixed sinusoid of geometrically-spaced frequencies to every token embedding. Closed-form, no learned parameters, extrapolates moderately. The 2017 transformer's original answer to 'how does the model know where each token is'.
2019
- Apr Sparse Transformer Sparse Transformer Attention Mechanisms Factorize attention into two structured sparse patterns: strided (local window) and fixed (long-range hubs). Total cost O(T·√T) per layer; full reachability in two hops. Foundational for sparse-attention research; superseded by SWA, BigBird, MoE attention in production.
- Oct Root Mean Square Layer Normalization RMSNorm Normalization Normalize by RMS instead of subtracting the mean — drop one statistic, one bias, and ~5-10% of normalization compute, with no measurable quality cost.
- Nov Compressive Transformer Compressive Long Context Two-tier memory: a short-term FIFO cache plus a long-term compressed memory built by applying a learned compression function (mean pooling, conv, or attention) to evicted segments. The original long-context recipe that traded resolution for length.
- Nov Multi-Query Attention MQA Attention Mechanisms One K, V projection shared by every query head. H× smaller KV cache than MHA; small but real quality drop that motivated GQA. The first move in the MHA → MQA → GQA → MLA evolution.
2020
- Jan Reformer — LSH Attention Reformer Attention Mechanisms Hash Q and K to discrete buckets via random rotation; restrict attention to queries and keys in the same bucket. O(N · log N) compute via learned (not hand-designed) sparsity. Lost to FlashAttention's I/O-aware exact computation on practical grounds.
- Feb GELU-Gated Linear Unit GeGLU FFN & MoE Bilinear gated FFN with a GELU-activated gating path. Matches SwiGLU on quality; lineage difference is mostly historical, with the Gemma family carrying GeGLU forward and the Llama/DeepSeek family carrying SwiGLU.
- Feb Pre-Norm, Post-Norm, and Sandwich Placement Norm Placement Normalization Pre-Norm became the default because it lets gradients flow unaltered through the residual stream and removes the need for warmup; sandwich placements like Gemma 3's norm-everywhere are the modern minority alternative.
- Feb ReLU-Gated Linear Unit ReGLU FFN & MoE Bilinear gated FFN with a ReLU-activated gating path. Matches SwiGLU and GeGLU on quality within noise per Shazeer's original ablations; the rarely-shipped third GLU variant.
- Feb Swish-Gated Linear Unit SwiGLU FFN & MoE Replace the FFN with a Swish-gated bilinear form: one linear path multiplied element-wise by Swish of another. Consistent quality win over ReLU/GELU at matched parameter count; the default modern FFN.
- Mar ReZero — Residual With Learnable Skip Scale ReZero Residual Connections Scale each residual branch by a per-layer learnable scalar α initialized to zero. The network starts at identity (gradient = 1 through every block) and learns to integrate sublayer contributions gradually. Stable at depths where standard transformers diverge.
- Apr Sliding Window Attention SWA Attention Mechanisms Restrict each query to the last W keys. Per-layer compute becomes O(W) per token; stacked layers still cover the full context because the receptive field grows linearly with depth.
- Jun GShard GShard FFN & MoE Top-2 routing across hundreds of experts in a sharded transformer encoder. Established expert capacity, dispatch tensors, and the per-expert random-token-dropping policy. The conceptual ancestor of Switch, Mixtral, and DeepSeekMoE.
- Jun Linear Attention Linear Attention Attention Mechanisms Replace softmax(QK^T)V with φ(Q)·(φ(K)^T·V). The K^T·V term is a small (d×d) state independent of sequence length; the operator becomes a recurrence at inference. Approximate, but linear in T.
- Jun Linformer — Low-Rank Attention Projection Linformer Attention Mechanisms Multiply K and V by learned projection matrices E, F ∈ ℝ^{k×N} that compress the sequence axis to a fixed rank k. Attention becomes O(N·k) per query. Empirically near-softmax on encoder tasks; the sequence-length-dependent projection makes decoder use awkward.
- Jul BigBird BigBird Attention Mechanisms Each query attends to three sets of keys: a random sample, a local window, and a fixed global set. O(N) attention; provably universal-approximator and Turing-complete. The 2020 successor to Sparse Transformer that put sparse attention on a theoretical footing.
- Sep Performer — Random Feature Softmax Approximation Performer Attention Mechanisms Approximate the softmax kernel as ⟨φ(q), φ(k)⟩ for a random feature map φ. The decomposition reorders QKV matmuls so attention is O(N·d) per layer; unlike Linformer the approximation is sequence-length-independent and works for autoregressive decoders.
- Oct Query-Key Normalization QK-Norm Normalization Normalize Q and K before the attention dot product so the logit magnitude stays bounded — a near-free fix for the outlier blow-ups that wreck attention at long context.
2021
- Jan Switch Transformer Switch FFN & MoE Top-1 expert routing — each token goes to exactly one expert. Simpler than GShard's top-2 baseline. Trillion-parameter Switch-C model proved sparse MoE was tractable at unprecedented scale; established the load-balancing loss formulation everyone uses.
- Apr Rotary Position Embedding RoPE Positional Encoding Rotate the (2i, 2i+1) coordinate pairs of Q and K by an angle proportional to position. Relative position falls out of the inner product for free; the model never sees a learned position table.
- May Sandwich-LN Sandwich-LN Normalization Two RMSNorms per sublayer: one before, one after, both inside the residual branch. Bounds the per-block contribution to the residual stream while keeping the identity path unnormalized. Used by Gemma 2 and 3.
- Aug Attention with Linear Biases ALiBi Positional Encoding Bias attention scores by a per-head linear function of the query-key distance. No learnable position parameters; extrapolates beyond training length without any fine-tune. Lost the dominance race to RoPE for dense decoders but is mechanically illuminating.
- Oct NormFormer — Extra Normalization in the Residual NormFormer Residual Connections Three extra normalizations bolted onto Pre-Norm: a LayerNorm on the attention output, learnable per-head scaling of that output, and a LayerNorm inside the FFN after the first linear. 24% faster pretraining at matched final perplexity on the 1.3B baseline; not adopted in production but cleanly motivated.
2022
- Mar DeepNet — Scaling Transformers to 1000 Layers DeepNet Residual Connections Scale each Post-Norm residual addition by a constant α(N) and shrink sublayer-weight init by β(N), both derived from depth. Trains 1000-layer encoder-decoders without divergence; gives Post-Norm geometry back to deep transformers that had been pushed toward Pre-Norm purely for stability reasons.
- Mar Memorizing Transformers Memorizing Long Context Augment one mid-stack attention layer with a kNN lookup over a non-differentiable bank of past K, V pairs. Retrieved keys/values are merged into the layer's attention. Conceptually adjacent to RAG; mechanically a forerunner.
- Mar No Position Encoding NoPE Positional Encoding Drop the positional encoding entirely. The causal mask asymmetrically restricts each query to its left context; the model learns to extract token position from that asymmetry alone. Works surprisingly well; reveals what positional encodings are actually adding.
- May FlashAttention FlashAttention Attention Mechanisms Tile attention so QK^T and the softmax stay in SRAM rather than round-tripping through HBM. Exact (not approximate) attention, 2–4× faster, 5–20× less peak memory. The universal kernel under every modern transformer trainer.
2023
- May Grouped-Query Attention GQA Attention Mechanisms Group query heads so each group reads one shared K, V pair. 4-8× KV-cache reduction with quality near MHA; the dominant attention layout for dense LLMs from 2023 onward.
- May Landmark Attention Landmark Long Context Insert a learnable landmark token at the end of each chunk; train it to summarize the chunk's content. At inference, queries attend to landmarks to gate which chunks get full attention. Hierarchical chunked attention with O(C + K·W) cost where C is the chunk count, K the chunks selected, and W per-chunk size.
- Jun Position Interpolation PI Positional Encoding Divide position values by the extension factor s before applying RoPE. Position t becomes t/s; trained rotation angles never extrapolate. Simple, parameter-free, works — but loses resolution uniformly across all frequency bands.
- Jul LongNet — Dilated Attention LongNet Long Context Each attention head operates at a different dilation rate r ∈ {1, 2, 4, ...}: stride-r local windows. Combining log₂(N) such heads gives O(N · log N) total cost with full reachability across the sequence. Demonstrated training on 1B-token sequences.
- Jul NTK-Aware RoPE Scaling NTK-Aware Positional Encoding Multiply RoPE's base b by s^(d_h/(d_h-2)) for extension factor s. Fast dimensions (low index) are nearly untouched; slow dimensions get linearly interpolated. Zero fine-tuning needed at small extensions; the inspiration for YaRN's per-band approach.
- Aug YaRN — Yet Another RoPE eXtensioN YaRN Positional Encoding Treat RoPE's rotation bands as three regimes — preserve the fast ones, linearly interpolate the slow ones — and rescale the softmax temperature. The 2023 long-context extension recipe of choice.
- Sep StreamingLLM and Attention Sinks Attention Sinks Long Context The first 1-4 tokens of any pretrained decoder act as attention sinks — they absorb the softmax mass that has nowhere else to go. Pin them in the KV cache and you can slide the rest of the window over arbitrarily long input without quality collapse.
2024
- Jan Activation Beacon Activation Beacon Long Context Insert learned 'beacon' tokens that summarize past activation windows. The model attends to beacons instead of the full long-context activations; 100× context extension with a tiny fine-tune of just the beacon parameters.
- Jan DeepSeekMoE DeepSeekMoE FFN & MoE Split the FFN into many small experts, route each token to top-K, and reserve a few always-on shared experts for common-knowledge work. Activated parameter count stays small; specialization sharpens.
- Jan Lightning Attention Lightning Attention Mechanisms Tile and fuse linear attention's prefix-sum recurrence so it runs faster than FlashAttention at long context. Interleaved 7:1 with softmax attention in MiniMax-01 to recover what linear attention loses on absolute quality while keeping its linear-in-T scaling.
- Jan Mixtral-Style Coarse MoE Mixtral MoE FFN & MoE Eight full-sized SwiGLU experts per MoE layer; top-2 routing with a standard load-balancing auxiliary loss. The Mistral-org variant that brought open-weights MoE into wide use, sitting on the coarse end of the granularity spectrum that DeepSeekMoE later contested.
- Feb LongRoPE — Per-Dimension RoPE Search LongRoPE Positional Encoding Use evolutionary search to find per-dimension RoPE rescaling factors. Generalizes YaRN's closed-form frequency-band recipe to arbitrary non-monotone schedules. Demonstrated 2M+ context extension on Llama-2 with a short fine-tune.
- May Decoupled RoPE Decoupled RoPE Positional Encoding Split the K projection into a small position-aware head that carries RoPE and a larger position-free body that absorbs into the latent KV cache. The trick that lets MLA compress K, V without losing position.
- May Multi-Head Latent Attention MLA Attention Mechanisms Compress K and V to a small per-token latent; reconstruct heads at attention time. ~5–7× smaller KV cache than MHA on DeepSeek-V2 ablations.
- Aug Auxiliary-Loss-Free Load Balancing Aux-Loss-Free FFN & MoE Maintain a per-expert bias term that shifts the gate's top-K decision toward under-used experts. The bias enters at selection time only — the final expert outputs are weighted by the original (unbiased) gate values. Load balance without quality cost.
- Sep Dynamic Hyper-Connections DHC / mHC Residual Connections The per-token variant of Hyper-Connections: A_l and B_l are computed as small projections of the current input rather than learned constants. Each token gets its own routing pattern across the n streams, at the cost of one extra projection per layer.
- Sep Hyper-Connections HC Residual Connections Generalize the residual connection to n parallel streams. Each sublayer learns depth-wise read/write coefficients that mix streams in and out. Recovers the standard residual at n=1 and gives a strict superset of expressivity above that.
- Oct nGPT — Normalized Transformer on the Hypersphere nGPT Normalization Project every vector to unit norm at every step. Eliminates LayerNorm/RMSNorm by structural construction; cosine similarity replaces inner product everywhere. Trains 4-20× faster than baseline GPT at matched quality, per the paper.
- Dec OLMo 2 Reordered Post-Norm OLMo 2 Post-Norm Normalization Move both RMSNorms to sit after each sublayer output, inside the residual branch — recovering Post-Norm's bounded residual stream while keeping Pre-Norm's gradient flow. Trained stably at 13B by combining with QK-Norm and an output-side Z-loss.
2025
- Mar Gemma 3 Norm-Everywhere Norm-Everywhere Normalization Sandwich-LN applied to BOTH the attention and FFN sublayers — four RMSNorms per block. Combined with QK-Norm and interleaved local/global attention, it stabilizes the entire Gemma 3 size family from a single recipe.
- Jul Kimi K2 MoE K2 MoE FFN & MoE Trillion-parameter MoE with 384 routed experts plus one shared expert — wider than DeepSeek V3's 256 + 1 by ~50%. Stabilized at scale by the MuonClip optimizer (Muon family with weight clipping). Same shared-expert + aux-loss-free routing lineage as DeepSeekMoE.
- Dec DeepSeek Sparse Attention DSA Attention Mechanisms A two-stage attention: a fast Lightning Indexer scores how relevant each historical key is to the current query, then full attention runs only over the top-K keys. Built on top of MLA — the latent KV cache is unchanged; the sparsity is in which cached entries get attended.