https://lizeman.github.io/llm-arch-kb/ A chronological reference of decoder-only LLM architecture innovations — math, simple implementations, tradeoffs, and adopting models with citations. en https://lizeman.github.io/llm-arch-kb/attention/dsa/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/dsa/ attention https://lizeman.github.io/llm-arch-kb/ffn-moe/kimi-k2-moe/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/kimi-k2-moe/ ffn-moe https://lizeman.github.io/llm-arch-kb/normalization/gemma3-norm-everywhere/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/gemma3-norm-everywhere/ normalization https://lizeman.github.io/llm-arch-kb/normalization/olmo2-post-norm/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/olmo2-post-norm/ normalization https://lizeman.github.io/llm-arch-kb/normalization/ngpt/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/ngpt/ normalization https://lizeman.github.io/llm-arch-kb/residual/hyper-connections/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/residual/hyper-connections/ residual https://lizeman.github.io/llm-arch-kb/residual/dynamic-hc/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/residual/dynamic-hc/ residual https://lizeman.github.io/llm-arch-kb/ffn-moe/aux-loss-free/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/aux-loss-free/ ffn-moe https://lizeman.github.io/llm-arch-kb/attention/mla/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/mla/ attention https://lizeman.github.io/llm-arch-kb/positional/decoupled-rope/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/decoupled-rope/ positional https://lizeman.github.io/llm-arch-kb/positional/longrope/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/longrope/ positional https://lizeman.github.io/llm-arch-kb/ffn-moe/mixtral-moe/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/mixtral-moe/ ffn-moe https://lizeman.github.io/llm-arch-kb/attention/lightning-attention/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/lightning-attention/ attention https://lizeman.github.io/llm-arch-kb/ffn-moe/deepseek-moe/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/deepseek-moe/ ffn-moe https://lizeman.github.io/llm-arch-kb/long-context/activation-beacon/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/long-context/activation-beacon/ long-context https://lizeman.github.io/llm-arch-kb/long-context/streaming-llm/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/long-context/streaming-llm/ long-context https://lizeman.github.io/llm-arch-kb/positional/yarn/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/yarn/ positional https://lizeman.github.io/llm-arch-kb/positional/ntk-aware-rope/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/ntk-aware-rope/ positional https://lizeman.github.io/llm-arch-kb/long-context/longnet/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/long-context/longnet/ long-context https://lizeman.github.io/llm-arch-kb/positional/position-interpolation/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/position-interpolation/ positional https://lizeman.github.io/llm-arch-kb/long-context/landmark-attention/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/long-context/landmark-attention/ long-context https://lizeman.github.io/llm-arch-kb/attention/gqa/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/gqa/ attention https://lizeman.github.io/llm-arch-kb/attention/flash-attention/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/flash-attention/ attention https://lizeman.github.io/llm-arch-kb/positional/nope/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/nope/ positional https://lizeman.github.io/llm-arch-kb/long-context/memorizing-transformers/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/long-context/memorizing-transformers/ long-context https://lizeman.github.io/llm-arch-kb/residual/deepnet/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/residual/deepnet/ residual https://lizeman.github.io/llm-arch-kb/residual/normformer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/residual/normformer/ residual https://lizeman.github.io/llm-arch-kb/positional/alibi/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/alibi/ positional https://lizeman.github.io/llm-arch-kb/normalization/sandwich-ln/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/sandwich-ln/ normalization https://lizeman.github.io/llm-arch-kb/positional/rope/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/rope/ positional https://lizeman.github.io/llm-arch-kb/ffn-moe/switch-transformer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/switch-transformer/ ffn-moe https://lizeman.github.io/llm-arch-kb/normalization/qk-norm/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/qk-norm/ normalization https://lizeman.github.io/llm-arch-kb/attention/performer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/performer/ attention https://lizeman.github.io/llm-arch-kb/attention/bigbird/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/bigbird/ attention https://lizeman.github.io/llm-arch-kb/attention/linformer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/linformer/ attention https://lizeman.github.io/llm-arch-kb/attention/linear-attention/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/linear-attention/ attention https://lizeman.github.io/llm-arch-kb/ffn-moe/gshard/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/gshard/ ffn-moe https://lizeman.github.io/llm-arch-kb/attention/sliding-window/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/sliding-window/ attention https://lizeman.github.io/llm-arch-kb/residual/rezero/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/residual/rezero/ residual https://lizeman.github.io/llm-arch-kb/ffn-moe/swiglu/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/swiglu/ ffn-moe https://lizeman.github.io/llm-arch-kb/ffn-moe/reglu/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/reglu/ ffn-moe https://lizeman.github.io/llm-arch-kb/normalization/norm-placement/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/norm-placement/ normalization https://lizeman.github.io/llm-arch-kb/ffn-moe/geglu/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/geglu/ ffn-moe https://lizeman.github.io/llm-arch-kb/attention/reformer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/reformer/ attention https://lizeman.github.io/llm-arch-kb/attention/mqa/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/mqa/ attention https://lizeman.github.io/llm-arch-kb/long-context/compressive-transformer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/long-context/compressive-transformer/ long-context https://lizeman.github.io/llm-arch-kb/normalization/rmsnorm/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/rmsnorm/ normalization https://lizeman.github.io/llm-arch-kb/attention/sparse-transformer/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/sparse-transformer/ attention https://lizeman.github.io/llm-arch-kb/positional/sinusoidal/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/positional/sinusoidal/ positional https://lizeman.github.io/llm-arch-kb/attention/mha/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/attention/mha/ attention https://lizeman.github.io/llm-arch-kb/ffn-moe/ffn-relu/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/ffn-relu/ ffn-moe https://lizeman.github.io/llm-arch-kb/ffn-moe/sparse-moe/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/sparse-moe/ ffn-moe https://lizeman.github.io/llm-arch-kb/normalization/layernorm/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/normalization/layernorm/ normalization https://lizeman.github.io/llm-arch-kb/ffn-moe/gelu/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/ffn-moe/gelu/ ffn-moe https://lizeman.github.io/llm-arch-kb/residual/residual-overview/ Tue, 12 May 2026 00:00:00 GMT https://lizeman.github.io/llm-arch-kb/residual/residual-overview/ residual