Skip to content

LLM Architecture KB

By category
By lens
Models
Compare
Timeline
About
Search

Efficiency

Techniques that reduce parameters, FLOPs, or memory at fixed quality.

14 techniques tagged with this facet.

Attention Mechanisms

Sparse Transformer Sparse Transformer

2019-04 Factorize attention into two structured sparse patterns: strided (local window) and fixed (long-range hubs). Total cost O(T·√T) per layer; full reachability in two hops. Foundational for sparse-attention research; superseded by SWA, BigBird, MoE attention in production.
Multi-Query Attention MQA

2019-11 One K, V projection shared by every query head. H× smaller KV cache than MHA; small but real quality drop that motivated GQA. The first move in the MHA → MQA → GQA → MLA evolution.
Reformer — LSH Attention Reformer

2020-01 Hash Q and K to discrete buckets via random rotation; restrict attention to queries and keys in the same bucket. O(N · log N) compute via learned (not hand-designed) sparsity. Lost to FlashAttention's I/O-aware exact computation on practical grounds.
Sliding Window Attention SWA

2020-04 Restrict each query to the last W keys. Per-layer compute becomes O(W) per token; stacked layers still cover the full context because the receptive field grows linearly with depth.
Linear Attention Linear Attention

2020-06 Replace softmax(QK^T)V with φ(Q)·(φ(K)^T·V). The K^T·V term is a small (d×d) state independent of sequence length; the operator becomes a recurrence at inference. Approximate, but linear in T.
Linformer — Low-Rank Attention Projection Linformer

2020-06 Multiply K and V by learned projection matrices E, F ∈ ℝ^{k×N} that compress the sequence axis to a fixed rank k. Attention becomes O(N·k) per query. Empirically near-softmax on encoder tasks; the sequence-length-dependent projection makes decoder use awkward.
BigBird BigBird

2020-07 Each query attends to three sets of keys: a random sample, a local window, and a fixed global set. O(N) attention; provably universal-approximator and Turing-complete. The 2020 successor to Sparse Transformer that put sparse attention on a theoretical footing.
Performer — Random Feature Softmax Approximation Performer

2020-09 Approximate the softmax kernel as ⟨φ(q), φ(k)⟩ for a random feature map φ. The decomposition reorders QKV matmuls so attention is O(N·d) per layer; unlike Linformer the approximation is sequence-length-independent and works for autoregressive decoders.
FlashAttention FlashAttention

2022-05 Tile attention so QK^T and the softmax stay in SRAM rather than round-tripping through HBM. Exact (not approximate) attention, 2–4× faster, 5–20× less peak memory. The universal kernel under every modern transformer trainer.
Grouped-Query Attention GQA

2023-05 Group query heads so each group reads one shared K, V pair. 4-8× KV-cache reduction with quality near MHA; the dominant attention layout for dense LLMs from 2023 onward.
Lightning Attention Lightning

2024-01 Tile and fuse linear attention's prefix-sum recurrence so it runs faster than FlashAttention at long context. Interleaved 7:1 with softmax attention in MiniMax-01 to recover what linear attention loses on absolute quality while keeping its linear-in-T scaling.
Multi-Head Latent Attention MLA

2024-05 Compress K and V to a small per-token latent; reconstruct heads at attention time. ~5–7× smaller KV cache than MHA on DeepSeek-V2 ablations.
DeepSeek Sparse Attention DSA

2025-12 A two-stage attention: a fast Lightning Indexer scores how relevant each historical key is to the current query, then full attention runs only over the top-K keys. Built on top of MLA — the latent KV cache is unchanged; the sparsity is in which cached entries get attended.

Long Context

LongNet — Dilated Attention LongNet

2023-07 Each attention head operates at a different dilation rate r ∈ {1, 2, 4, ...}: stride-r local windows. Combining log₂(N) such heads gives O(N · log N) total cost with full reachability across the sequence. Demonstrated training on 1B-token sequences.

Content licensed CC BY 4.0. Code licensed MIT.

Source on GitHub · Methodology · RSS · Last built 2026-05-13