LLM Architecture Knowledge Base
A chronologically-organized reference of decoder-only LLM architecture innovations. Each entry: the math, a simple implementation sketch, tradeoffs, ablation results, and which disclosed production models adopt it — with citations.
This site catalogs meaningful architectural innovations in decoder-only LLMs from 2017 to today. Coverage stays at "medium deep-dive" depth: enough to understand the move and reimplement a sketch, not enough to be a paper substitute.
We follow a strict closed-model policy: no inferred architecture claims about undocumented models. Every adoption claim links to a paper, model card, or official report.
The 2026 consensus stack
One transformer block, six choices
The default frontier dense decoder of 2024–26 is a small, settled set of choices. Each labeled module below links to its entry.
The open frontier
Eight years of open-weight models
Every open-weight model with a disclosed parameter count tracked in this knowledge base, plotted by release date and total parameters. Color encodes the releasing organization; dense models are circles, mixture-of-experts models are stars.
Featured
Start here
- Multi-Head Latent Attention MLA
Cut KV cache memory below MQA/GQA while preserving or improving quality.
- Rotary Position Embedding RoPE
Encode position by rotating Q and K vectors — so that attention's inner product depends only on the relative offset between tokens, with no separate position-embedding lookup.
- Root Mean Square Layer Normalization RMSNorm
Drop the mean-centering step of LayerNorm — keep almost all of the stability benefit at meaningfully less compute.
- Grouped-Query Attention GQA
Get most of Multi-Query Attention's KV-cache savings without the quality drop — by sharing K, V across small groups of query heads instead of all heads.
- DeepSeekMoE DeepSeekMoE
Get sharper expert specialization than vanilla MoE by combining fine-grained expert segmentation with isolated shared experts that absorb the common-knowledge load.
Browse
By category
- Positional Encoding 9 entries most recent: Decoupled RoPE
- Normalization 8 entries most recent: Gemma 3 Norm-Everywhere
- Residual Connections 6 entries most recent: Hyper-Connections
- FFN & MoE 12 entries most recent: Kimi K2 MoE
- Attention Mechanisms 14 entries most recent: DeepSeek Sparse Attention
- Long Context 6 entries most recent: Activation Beacon