Browse
Compare
Pick up to three models to see their architecture and adopted techniques side by side. Empty cells (—) mean the choice was not publicly disclosed, per the citation policy.
| DeepSeek V3DeepSeek-AI | Llama 3.1 70BMeta | Gemma 3 27BGoogle DeepMind | |
|---|---|---|---|
| 2024-12·671B (37B active)·128K ctx·open | 2024-07·70B·128K ctx·open | 2025-03·27B·128K ctx·open | |
| Architecture | |||
| Positional encoding | RoPE (decoupled head for MLA) | RoPE (theta scaled for 128K context) | RoPE (local SWA layers at 10K base, global layers at 1M base) |
| Norm placement | Pre-Norm | Pre-Norm | Pre-Norm + Post-Norm (norm-everywhere) |
| Norm type | RMSNorm | RMSNorm | RMSNorm |
| QK-Norm | no | no | yes |
| Activation | SwiGLU | SwiGLU | GeGLU |
| Attention | MLA | GQA (8 key-value heads, 64 query heads) | Interleaved SWA + global (5 local : 1 global) |
| MoE | DeepSeekMoE with aux-loss-free routing | — | — |
| Adopted techniques | |||
| Positional Encoding | |||
| Normalization | |||
| Residual Connections | |||
| FFN & MoE | |||
| Attention Mechanisms | |||
means the choice was not publicly disclosed by the model's developers, per the citation policy. Empty technique rows likewise mean no adoption claim has been documented for that category.
All models
Pick to compare (42 models)
All-models grid view — sortable matrices across every model and technique
Model architecture matrix
Each row is a production model with public architecture documentation. Each column is a design slot. Click a column header to sort; chips above filter by organization, disclosure level, or release-year range.
| GLM-130BTsinghua KEG / Zhipu AI | 2022-10-05 | 2D Rotary Position Embedding (RoPE on each axis of the 2D position grid) | Post-Norm with DeepNorm scaling | LayerNorm | no | GeLU | MHA (96 query heads, head_dim 128) | — |
| Llama 1 65BMeta | 2023-02-24 | RoPE (base 10000) | Pre-Norm | RMSNorm | no | SwiGLU | MHA (64 query heads = 64 KV heads) | — |
| GPT-4OpenAI | 2023-03-14 | — | — | — | — | — | — | — |
| Llama 2 70BMeta | 2023-07-18 | RoPE (base 10000) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (8 KV heads, 64 query heads) | — |
| Mistral 7BMistral AI | 2023-09-27 | RoPE | Pre-Norm | RMSNorm | no | SwiGLU | GQA (8 KV heads, 32 query heads) + Sliding Window Attention (W = 4096) | — |
| Mixtral 8x7BMistral AI | 2023-12-11 | RoPE (base 1000000) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (32 query heads, 8 KV heads) | 8 experts per MoE layer, top-2 routing, all FFNs are MoE; standard load-balancing auxiliary loss |
| DeepSeek LLM 67BDeepSeek-AI | 2024-01-05 | RoPE (base 10000) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (64 query heads, 8 KV heads) | — |
| OLMo 1 7BAllen Institute for AI (AI2) | 2024-02-01 | RoPE (base 10000) | Pre-Norm | LayerNorm (non-parametric) | no | SwiGLU | MHA (32 query heads, 32 KV heads, head dim 128) | — |
| Gemma 1 7BGoogle DeepMind | 2024-02-21 | RoPE (base 10000) | Pre-Norm | RMSNorm | no | GeGLU | MHA (16 query heads, 16 KV heads, head_dim 256) | — |
| Command R+ 104BCohere | 2024-04-04 | RoPE | Pre-Norm | — | — | — | GQA | — |
| RecurrentGemma 2BGoogle DeepMind | 2024-04-11 | Implicit (linear recurrence carries position) + RoPE on local attention blocks | Pre-Norm | RMSNorm | no | GeGLU | Hybrid: interleaved Griffin recurrent blocks + local Sliding Window Attention (window 2048) | — |
| DeepSeek V2DeepSeek-AI | 2024-05-07 | RoPE (decoupled head for MLA) | Pre-Norm | RMSNorm | no | SwiGLU | MLA | DeepSeekMoE with standard auxiliary balance loss |
| GLM-4 9BZhipu AI / Tsinghua KEG | 2024-06-04 | 2D Rotary Position Embedding (the GLM-family signature, carried from GLM-130B) | Post-Norm | RMSNorm | no | SwiGLU | GQA (32 query heads, 2 KV heads — G = 16; near-MQA aggressive sharing) | — |
| Gemma 2 27BGoogle DeepMind | 2024-06-27 | RoPE | Pre-Norm + Post-Norm (sandwich-style) | RMSNorm | no | GeGLU | GQA (16 KV heads, 32 query heads) + interleaved local/global SWA (window 4096) | — |
| Llama 3.1 70BMeta | 2024-07-23 | RoPE (theta scaled for 128K context) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (8 key-value heads, 64 query heads) | — |
| OLMoE 1B/7BAllen Institute for AI (AI2) | 2024-09-04 | RoPE (base 10000) | Pre-Norm | RMSNorm | yes | SwiGLU | MHA (16 query heads, 16 KV heads) | Top-8 routing across 64 routed experts; no shared expert; Switch-style aux balance loss |
| Qwen 2.5 72BAlibaba (Qwen Team) | 2024-09-19 | RoPE | Pre-Norm | RMSNorm | no | SwiGLU | GQA | — |
| Hunyuan-Large 389BTencent | 2024-11-05 | RoPE (base 10000 with dynamic scaling, paper reports an effective base near 1e9 at 256K context) | Pre-Norm | RMSNorm | — | SwiGLU | GQA (80 query heads, 8 KV heads) with Cross-Layer Attention (KV cache shared across 2 consecutive layers) | 16 specialized experts, top-1 routing + 1 always-on shared expert; expert-specific learning rate; recycling routing |
| OLMo 2 13BAllen Institute for AI (AI2) | 2024-11-26 | RoPE (base 500K) | Reordered Post-Norm (norm after attention output, after FFN output, inside residual) | RMSNorm | yes | SwiGLU | MHA (40 query heads, 40 KV heads, head dim 128) | — |
| Llama 3.3 70B InstructMeta | 2024-12-06 | RoPE (theta scaled for 128K context) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (8 key-value heads, 64 query heads) | — |
| Phi-4 14BMicrosoft Research | 2024-12-12 | RoPE (base 250000) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (40 query heads, 10 KV heads) | — |
| DeepSeek V3DeepSeek-AI | 2024-12-26 | RoPE (decoupled head for MLA) | Pre-Norm | RMSNorm | no | SwiGLU | MLA | DeepSeekMoE with aux-loss-free routing |
| MiniMax-Text-01MiniMax | 2025-01-14 | RoPE (base 1e7, applied to half of head dim) | Post-Norm | RMSNorm | no | SwiGLU | Lightning Attention (7 layers) interleaved with softmax attention (1 layer); ratio 7:1 | 32 routed experts (no shared expert); top-2 routing |
| DeepSeek R1DeepSeek-AI | 2025-01-20 | RoPE (decoupled head for MLA) | Pre-Norm | RMSNorm | no | SwiGLU | MLA | DeepSeekMoE with aux-loss-free routing |
| Kimi K1.5Moonshot AI | 2025-01-22 | — | — | — | — | — | — | — |
| Gemma 3 27BGoogle DeepMind | 2025-03-12 | RoPE (local SWA layers at 10K base, global layers at 1M base) | Pre-Norm + Post-Norm (norm-everywhere) | RMSNorm | yes | GeGLU | Interleaved SWA + global (5 local : 1 global) | — |
| Llama 4 ScoutMeta | 2025-04-05 | iRoPE (interleaved RoPE / NoPE-style no-position layers) | Pre-Norm | RMSNorm | no | SwiGLU | iRoPE (Llama-4) over GQA | 16 routed experts + 1 shared expert, top-1 routing |
| Qwen3 235B-A22BAlibaba (Qwen Team) | 2025-05-14 | RoPE (base 10K → 1M via ABF; YaRN + DCA for long context) | Pre-Norm | RMSNorm | yes | SwiGLU | GQA (64 query heads, 4 KV heads) | Sparse MoE — 128 experts, top-8 routing, no shared expert, global-batch load balancing |
| Qwen3 30B-A3BAlibaba (Qwen Team) | 2025-05-14 | RoPE (base 10K → 1M via ABF; YaRN + DCA for long context) | Pre-Norm | RMSNorm | yes | SwiGLU | GQA (32 query heads, 4 KV heads) | Sparse MoE — 128 experts, top-8 routing, no shared expert, global-batch load balancing |
| Qwen3 32BAlibaba (Qwen Team) | 2025-05-14 | RoPE (base 10K → 1M via ABF; YaRN + DCA for long context) | Pre-Norm | RMSNorm | yes | SwiGLU | GQA (64 query heads, 8 KV heads) | — |
| MiniMax-M1MiniMax | 2025-06-16 | RoPE (base 1e7) | Post-Norm | RMSNorm | no | SwiGLU | Lightning Attention (7 layers) interleaved with softmax attention (1 layer); ratio 7:1 | 32 routed experts (no shared expert); top-2 routing |
| Kimi K2Moonshot AI | 2025-07-11 | RoPE (decoupled head, MLA-style) | Pre-Norm | RMSNorm | no | SwiGLU | MLA-style latent attention | Sparse MoE with 384 routed experts + 1 shared expert; top-K routing |
| GLM-4.5Zhipu AI | 2025-07-28 | RoPE (theta 1e6; partial rotary factor 0.5) | Pre-Norm | RMSNorm | yes | SwiGLU (SiLU) | GQA (96 query heads, 8 KV heads) | Sparse MoE — 160 routed experts, top-8 routing, 1 shared expert; routed_scaling_factor 2.5 |
| gpt-oss-120bOpenAI | 2025-08-05 | RoPE (base 150K, YaRN scaling from a 4K base context to 131K) | Pre-Norm | RMSNorm | no | SwiGLU | GQA (64 query heads, 8 KV heads) with strict alternation of 128-token sliding-window and full-attention layers | Sparse MoE — 128 experts per layer, top-4 routing |
| DeepSeek V3.1DeepSeek-AI | 2025-08-21 | RoPE (decoupled head for MLA) | Pre-Norm | RMSNorm | no | SwiGLU | MLA | DeepSeekMoE with aux-loss-free routing |
| Qwen3-Next 80B-A3BAlibaba (Qwen Team) | 2025-09-12 | RoPE on Gated Attention layers (head-dim 64); no positional encoding on Gated DeltaNet layers | Pre-Norm | Zero-centered, weight-decayed RMSNorm | — | SwiGLU | Hybrid: 3 Gated DeltaNet (linear) layers per 1 Gated Attention (full) layer, repeated 12× = 48 layers total. Gated Attention: 16 Q heads / 2 KV heads, head-dim 256. | Sparse MoE — 512 experts, top-10 routing, 1 shared expert |
| DeepSeek V3.2-ExpDeepSeek-AI | 2025-09-29 | RoPE (decoupled head for MLA) | Pre-Norm | RMSNorm | no | SwiGLU | MLA + DeepSeek Sparse Attention (Lightning Indexer + top-K key selection) | DeepSeekMoE with aux-loss-free routing |
| Kimi Linear 48B-A3BMoonshot AI | 2025-10-30 | RoPE (base 10K) on a 64-dim decoupled head only; MLA body uses NoPE | Pre-Norm | RMSNorm | no | SwiGLU | Hybrid: 20 Kimi Delta Attention (KDA) linear layers + 7 MLA full-attention layers across 27 layers (3:1 KDA:MLA pattern) | Sparse MoE — 256 routed experts + 1 shared, top-8 routing, sigmoid gate with grouped top-K |
| OLMo 3 32BAllen Institute for AI (AI2) | 2025-11-20 | RoPE (base 500K) + YaRN scaling (factor 8, 8K → 64K) | Reordered Post-Norm (inherited from OLMo 2) | RMSNorm | yes | SwiGLU | GQA (40 query heads, 8 KV heads, head dim 128) with interleaved sliding-window (3 sliding : 1 full, window 4096) | — |
| Mistral Large 3 (675B)Mistral AI | 2025-12-02 | — | — | — | — | — | — | Granular MoE (exact expert count and routing not disclosed publicly) |
| Nemotron 3 Nano 30B-A3BNVIDIA | 2025-12-15 | — | — | — | — | — | Hybrid: 23 Mamba-2 + 6 GQA attention layers (2 KV groups); MoE FFN on 23 layers | Sparse MoE — 128 routed experts + 1 shared expert per MoE layer, top-6 routing |
| DeepSeek V4-ProDeepSeek-AI | 2026-05-06 | Partial RoPE (last 64 dims of queries / KV entries / core-attention outputs) + YaRN scaling (factor 16, 64K → 1M) | Pre-Norm | RMSNorm | yes | SwiGLU | Hybrid: Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA), interleaved, with a 128-token sliding-window branch and attention sink | DeepSeekMoE — 1 shared + 384 routed experts (intermediate dim 3072), top-6 per token, aux-loss-free (noaux_tc) with √Softplus scoring; FP4 quantization-aware training for routed experts; first 3 MoE layers use Hash routing |
Model × technique adoption matrix
Rows are models, columns are techniques grouped by category. A filled marker means the model's spec sheet cites that technique. Hover a cell to highlight its row and column.
Focus on models
Technique matrix
Each row is a documented technique. The "adopters" column counts how many of our spec-sheet'd models cite the technique. Click any column header to sort.
| Summary | |||||
|---|---|---|---|---|---|
| 2015-12 | Residual Connections | The Residual StreamResidual | foundational | 4 | Every modern transformer block adds its output to the running activation rather than replacing it. The 'residual stream' is the linear path through depth that carries information unaltered; each block contributes a perturbation. |
| 2016-06 | FFN & MoE | Gaussian Error Linear UnitGELU | foundational | 0 | Multiply x by the standard normal CDF: GELU(x) = x · Φ(x). Smoother than ReLU, weighted by the input's z-score. The default transformer FFN activation in BERT, GPT-2, GPT-3 era; superseded by gated variants (SwiGLU, GeGLU) but still the substrate they sit on. |
| 2016-07 | Normalization | Layer NormalizationLayerNorm | foundational | 0 | Per-token feature-axis normalization with learnable scale and shift. The 2016 building block the original transformer used; superseded in modern decoders by RMSNorm, which drops the mean-centering step at no quality cost. |
| 2017-01 | FFN & MoE | Sparsely-Gated MoESparse MoE | foundational | 4 | Top-K gating with per-expert noise plus an importance-balancing loss. The 2017 paper that established that 'sparse activation + load balance' is a viable training-time recipe; every modern MoE descends from it. |
| 2017-06 | FFN & MoE | FFN with ReLUFFN-ReLU | deprecated | 0 | The original transformer FFN: ReLU squashing of a 4× hidden expansion, projected back. Foundational and historical — every later FFN variant (GELU, SwiGLU, GeGLU) is a refinement of this two-layer-with-nonlinearity pattern. |
| 2017-06 | Attention Mechanisms | Multi-Head AttentionMHA | foundational | 3 | The original attention layout: H independent heads, each computing scaled dot-product attention on a d_h-dim subspace, concatenated then projected. Now superseded by GQA, MLA in production decoders, but still the conceptual foundation. |
| 2017-06 | Positional Encoding | Sinusoidal Position EncodingSinusoidal | foundational | 0 | Add a fixed sinusoid of geometrically-spaced frequencies to every token embedding. Closed-form, no learned parameters, extrapolates moderately. The 2017 transformer's original answer to 'how does the model know where each token is'. |
| 2019-04 | Attention Mechanisms | Sparse TransformerSparse Transformer | foundational | 0 | Factorize attention into two structured sparse patterns: strided (local window) and fixed (long-range hubs). Total cost O(T·√T) per layer; full reachability in two hops. Foundational for sparse-attention research; superseded by SWA, BigBird, MoE attention in production. |
| 2019-10 | Normalization | Root Mean Square Layer NormalizationRMSNorm | foundational | 15 | Normalize by RMS instead of subtracting the mean — drop one statistic, one bias, and ~5-10% of normalization compute, with no measurable quality cost. |
| 2019-11 | Long Context | Compressive TransformerCompressive | foundational | 0 | Two-tier memory: a short-term FIFO cache plus a long-term compressed memory built by applying a learned compression function (mean pooling, conv, or attention) to evicted segments. The original long-context recipe that traded resolution for length. |
| 2019-11 | Attention Mechanisms | Multi-Query AttentionMQA | production-adopted | 0 | One K, V projection shared by every query head. H× smaller KV cache than MHA; small but real quality drop that motivated GQA. The first move in the MHA → MQA → GQA → MLA evolution. |
| 2020-01 | Attention Mechanisms | Reformer — LSH AttentionReformer | research | 0 | Hash Q and K to discrete buckets via random rotation; restrict attention to queries and keys in the same bucket. O(N · log N) compute via learned (not hand-designed) sparsity. Lost to FlashAttention's I/O-aware exact computation on practical grounds. |
| 2020-02 | FFN & MoE | GELU-Gated Linear UnitGeGLU | production-adopted | 3 | Bilinear gated FFN with a GELU-activated gating path. Matches SwiGLU on quality; lineage difference is mostly historical, with the Gemma family carrying GeGLU forward and the Llama/DeepSeek family carrying SwiGLU. |
| 2020-02 | Normalization | Pre-Norm, Post-Norm, and Sandwich PlacementNorm Placement | foundational | 5 | Pre-Norm became the default because it lets gradients flow unaltered through the residual stream and removes the need for warmup; sandwich placements like Gemma 3's norm-everywhere are the modern minority alternative. |
| 2020-02 | FFN & MoE | ReLU-Gated Linear UnitReGLU | foundational | 0 | Bilinear gated FFN with a ReLU-activated gating path. Matches SwiGLU and GeGLU on quality within noise per Shazeer's original ablations; the rarely-shipped third GLU variant. |
| 2020-02 | FFN & MoE | Swish-Gated Linear UnitSwiGLU | production-adopted | 17 | Replace the FFN with a Swish-gated bilinear form: one linear path multiplied element-wise by Swish of another. Consistent quality win over ReLU/GELU at matched parameter count; the default modern FFN. |
| 2020-03 | Residual Connections | ReZero — Residual With Learnable Skip ScaleReZero | research | 0 | Scale each residual branch by a per-layer learnable scalar α initialized to zero. The network starts at identity (gradient = 1 through every block) and learns to integrate sublayer contributions gradually. Stable at depths where standard transformers diverge. |
| 2020-04 | Attention Mechanisms | Sliding Window AttentionSWA | production-adopted | 4 | Restrict each query to the last W keys. Per-layer compute becomes O(W) per token; stacked layers still cover the full context because the receptive field grows linearly with depth. |
| 2020-06 | FFN & MoE | GShardGShard | foundational | 0 | Top-2 routing across hundreds of experts in a sharded transformer encoder. Established expert capacity, dispatch tensors, and the per-expert random-token-dropping policy. The conceptual ancestor of Switch, Mixtral, and DeepSeekMoE. |
| 2020-06 | Attention Mechanisms | Linear AttentionLinear Attention | production-adopted | 3 | Replace softmax(QK^T)V with φ(Q)·(φ(K)^T·V). The K^T·V term is a small (d×d) state independent of sequence length; the operator becomes a recurrence at inference. Approximate, but linear in T. |
| 2020-06 | Attention Mechanisms | Linformer — Low-Rank Attention ProjectionLinformer | research | 0 | Multiply K and V by learned projection matrices E, F ∈ ℝ^{k×N} that compress the sequence axis to a fixed rank k. Attention becomes O(N·k) per query. Empirically near-softmax on encoder tasks; the sequence-length-dependent projection makes decoder use awkward. |
| 2020-07 | Attention Mechanisms | BigBirdBigBird | research | 0 | Each query attends to three sets of keys: a random sample, a local window, and a fixed global set. O(N) attention; provably universal-approximator and Turing-complete. The 2020 successor to Sparse Transformer that put sparse attention on a theoretical footing. |
| 2020-09 | Attention Mechanisms | Performer — Random Feature Softmax ApproximationPerformer | research | 0 | Approximate the softmax kernel as ⟨φ(q), φ(k)⟩ for a random feature map φ. The decomposition reorders QKV matmuls so attention is O(N·d) per layer; unlike Linformer the approximation is sequence-length-independent and works for autoregressive decoders. |
| 2020-10 | Normalization | Query-Key NormalizationQK-Norm | production-adopted | 7 | Normalize Q and K before the attention dot product so the logit magnitude stays bounded — a near-free fix for the outlier blow-ups that wreck attention at long context. |
| 2021-01 | FFN & MoE | Switch TransformerSwitch | foundational | 0 | Top-1 expert routing — each token goes to exactly one expert. Simpler than GShard's top-2 baseline. Trillion-parameter Switch-C model proved sparse MoE was tractable at unprecedented scale; established the load-balancing loss formulation everyone uses. |
| 2021-04 | Positional Encoding | Rotary Position EmbeddingRoPE | production-adopted | 18 | Rotate the (2i, 2i+1) coordinate pairs of Q and K by an angle proportional to position. Relative position falls out of the inner product for free; the model never sees a learned position table. |
| 2021-05 | Normalization | Sandwich-LNSandwich-LN | production-adopted | 2 | Two RMSNorms per sublayer: one before, one after, both inside the residual branch. Bounds the per-block contribution to the residual stream while keeping the identity path unnormalized. Used by Gemma 2 and 3. |
| 2021-08 | Positional Encoding | Attention with Linear BiasesALiBi | research | 0 | Bias attention scores by a per-head linear function of the query-key distance. No learnable position parameters; extrapolates beyond training length without any fine-tune. Lost the dominance race to RoPE for dense decoders but is mechanically illuminating. |
| 2021-10 | Residual Connections | NormFormer — Extra Normalization in the ResidualNormFormer | research | 0 | Three extra normalizations bolted onto Pre-Norm: a LayerNorm on the attention output, learnable per-head scaling of that output, and a LayerNorm inside the FFN after the first linear. 24% faster pretraining at matched final perplexity on the 1.3B baseline; not adopted in production but cleanly motivated. |
| 2022-03 | Residual Connections | DeepNet — Scaling Transformers to 1000 LayersDeepNet | research | 0 | Scale each Post-Norm residual addition by a constant α(N) and shrink sublayer-weight init by β(N), both derived from depth. Trains 1000-layer encoder-decoders without divergence; gives Post-Norm geometry back to deep transformers that had been pushed toward Pre-Norm purely for stability reasons. |
| 2022-03 | Long Context | Memorizing TransformersMemorizing | research | 0 | Augment one mid-stack attention layer with a kNN lookup over a non-differentiable bank of past K, V pairs. Retrieved keys/values are merged into the layer's attention. Conceptually adjacent to RAG; mechanically a forerunner. |
| 2022-03 | Positional Encoding | No Position EncodingNoPE | research | 2 | Drop the positional encoding entirely. The causal mask asymmetrically restricts each query to its left context; the model learns to extract token position from that asymmetry alone. Works surprisingly well; reveals what positional encodings are actually adding. |
| 2022-05 | Attention Mechanisms | FlashAttentionFlashAttention | production-adopted | 0 | Tile attention so QK^T and the softmax stay in SRAM rather than round-tripping through HBM. Exact (not approximate) attention, 2–4× faster, 5–20× less peak memory. The universal kernel under every modern transformer trainer. |
| 2023-05 | Attention Mechanisms | Grouped-Query AttentionGQA | production-adopted | 16 | Group query heads so each group reads one shared K, V pair. 4-8× KV-cache reduction with quality near MHA; the dominant attention layout for dense LLMs from 2023 onward. |
| 2023-05 | Long Context | Landmark AttentionLandmark | research | 0 | Insert a learnable landmark token at the end of each chunk; train it to summarize the chunk's content. At inference, queries attend to landmarks to gate which chunks get full attention. Hierarchical chunked attention with O(C + K·W) cost where C is the chunk count, K the chunks selected, and W per-chunk size. |
| 2023-06 | Positional Encoding | Position InterpolationPI | foundational | 0 | Divide position values by the extension factor s before applying RoPE. Position t becomes t/s; trained rotation angles never extrapolate. Simple, parameter-free, works — but loses resolution uniformly across all frequency bands. |
| 2023-07 | Long Context | LongNet — Dilated AttentionLongNet | research | 0 | Each attention head operates at a different dilation rate r ∈ {1, 2, 4, ...}: stride-r local windows. Combining log₂(N) such heads gives O(N · log N) total cost with full reachability across the sequence. Demonstrated training on 1B-token sequences. |
| 2023-07 | Positional Encoding | NTK-Aware RoPE ScalingNTK-Aware | production-adopted | 0 | Multiply RoPE's base b by s^(d_h/(d_h-2)) for extension factor s. Fast dimensions (low index) are nearly untouched; slow dimensions get linearly interpolated. Zero fine-tuning needed at small extensions; the inspiration for YaRN's per-band approach. |
| 2023-08 | Positional Encoding | YaRN — Yet Another RoPE eXtensioNYaRN | production-adopted | 6 | Treat RoPE's rotation bands as three regimes — preserve the fast ones, linearly interpolate the slow ones — and rescale the softmax temperature. The 2023 long-context extension recipe of choice. |
| 2023-09 | Long Context | StreamingLLM and Attention SinksAttention Sinks | production-adopted | 0 | The first 1-4 tokens of any pretrained decoder act as attention sinks — they absorb the softmax mass that has nowhere else to go. Pin them in the KV cache and you can slide the rest of the window over arbitrarily long input without quality collapse. |
| 2024-01 | Long Context | Activation BeaconActivation Beacon | research | 0 | Insert learned 'beacon' tokens that summarize past activation windows. The model attends to beacons instead of the full long-context activations; 100× context extension with a tiny fine-tune of just the beacon parameters. |
| 2024-01 | FFN & MoE | DeepSeekMoEDeepSeekMoE | production-adopted | 8 | Split the FFN into many small experts, route each token to top-K, and reserve a few always-on shared experts for common-knowledge work. Activated parameter count stays small; specialization sharpens. |
| 2024-01 | Attention Mechanisms | Lightning AttentionLightning | production-adopted | 2 | Tile and fuse linear attention's prefix-sum recurrence so it runs faster than FlashAttention at long context. Interleaved 7:1 with softmax attention in MiniMax-01 to recover what linear attention loses on absolute quality while keeping its linear-in-T scaling. |
| 2024-01 | FFN & MoE | Mixtral-Style Coarse MoEMixtral MoE | production-adopted | 2 | Eight full-sized SwiGLU experts per MoE layer; top-2 routing with a standard load-balancing auxiliary loss. The Mistral-org variant that brought open-weights MoE into wide use, sitting on the coarse end of the granularity spectrum that DeepSeekMoE later contested. |
| 2024-02 | Positional Encoding | LongRoPE — Per-Dimension RoPE SearchLongRoPE | research | 0 | Use evolutionary search to find per-dimension RoPE rescaling factors. Generalizes YaRN's closed-form frequency-band recipe to arbitrary non-monotone schedules. Demonstrated 2M+ context extension on Llama-2 with a short fine-tune. |
| 2024-05 | Positional Encoding | Decoupled RoPEDecoupled RoPE | production-adopted | 3 | Split the K projection into a small position-aware head that carries RoPE and a larger position-free body that absorbs into the latent KV cache. The trick that lets MLA compress K, V without losing position. |
| 2024-05 | Attention Mechanisms | Multi-Head Latent AttentionMLA | production-adopted | 6 | Compress K and V to a small per-token latent; reconstruct heads at attention time. ~5–7× smaller KV cache than MHA on DeepSeek-V2 ablations. |
| 2024-08 | FFN & MoE | Auxiliary-Loss-Free Load BalancingAux-Loss-Free | production-adopted | 3 | Maintain a per-expert bias term that shifts the gate's top-K decision toward under-used experts. The bias enters at selection time only — the final expert outputs are weighted by the original (unbiased) gate values. Load balance without quality cost. |
| 2024-09 | Residual Connections | Dynamic Hyper-ConnectionsDHC / mHC | research | 0 | The per-token variant of Hyper-Connections: A_l and B_l are computed as small projections of the current input rather than learned constants. Each token gets its own routing pattern across the n streams, at the cost of one extra projection per layer. |
| 2024-09 | Residual Connections | Hyper-ConnectionsHC | research | 0 | Generalize the residual connection to n parallel streams. Each sublayer learns depth-wise read/write coefficients that mix streams in and out. Recovers the standard residual at n=1 and gives a strict superset of expressivity above that. |
| 2024-10 | Normalization | nGPT — Normalized Transformer on the HyperspherenGPT | research | 0 | Project every vector to unit norm at every step. Eliminates LayerNorm/RMSNorm by structural construction; cosine similarity replaces inner product everywhere. Trains 4-20× faster than baseline GPT at matched quality, per the paper. |
| 2024-12 | Normalization | OLMo 2 Reordered Post-NormOLMo 2 Post-Norm | production-adopted | 2 | Move both RMSNorms to sit after each sublayer output, inside the residual branch — recovering Post-Norm's bounded residual stream while keeping Pre-Norm's gradient flow. Trained stably at 13B by combining with QK-Norm and an output-side Z-loss. |
| 2025-03 | Normalization | Gemma 3 Norm-EverywhereNorm-Everywhere | production-adopted | 1 | Sandwich-LN applied to BOTH the attention and FFN sublayers — four RMSNorms per block. Combined with QK-Norm and interleaved local/global attention, it stabilizes the entire Gemma 3 size family from a single recipe. |
| 2025-07 | FFN & MoE | Kimi K2 MoEK2 MoE | production-adopted | 1 | Trillion-parameter MoE with 384 routed experts plus one shared expert — wider than DeepSeek V3's 256 + 1 by ~50%. Stabilized at scale by the MuonClip optimizer (Muon family with weight clipping). Same shared-expert + aux-loss-free routing lineage as DeepSeekMoE. |
| 2025-12 | Attention Mechanisms | DeepSeek Sparse AttentionDSA | production-adopted | 1 | A two-stage attention: a fast Lightning Indexer scores how relevant each historical key is to the current query, then full attention runs only over the top-K keys. Built on top of MLA — the latent KV cache is unchanged; the sparsity is in which cached entries get attended. |
Coverage notes
v1 ship covers the open / documented frontier model families (Llama, OLMo, DeepSeek, Gemma); the v2 ship plans to extend to Kimi, MiniMax, GLM with the same disclosure standard. Closed-frontier families (GPT-*, Claude, Gemini) are intentionally absent — see the methodology for why.