https://lizeman.github.io/llm-arch-kb/https://lizeman.github.io/llm-arch-kb/about/https://lizeman.github.io/llm-arch-kb/attention/https://lizeman.github.io/llm-arch-kb/attention/bigbird/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/dsa/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/flash-attention/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/gqa/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/lightning-attention/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/linear-attention/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/linformer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/mha/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/mla/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/mqa/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/performer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/reformer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/sliding-window/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/attention/sparse-transformer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/compare/https://lizeman.github.io/llm-arch-kb/facets/https://lizeman.github.io/llm-arch-kb/facets/efficiency/https://lizeman.github.io/llm-arch-kb/facets/hardware-aware/https://lizeman.github.io/llm-arch-kb/facets/inference-only/https://lizeman.github.io/llm-arch-kb/facets/kv-cache/https://lizeman.github.io/llm-arch-kb/facets/long-context/https://lizeman.github.io/llm-arch-kb/facets/parameter-free/https://lizeman.github.io/llm-arch-kb/facets/quality/https://lizeman.github.io/llm-arch-kb/facets/routing/https://lizeman.github.io/llm-arch-kb/facets/training-stability/https://lizeman.github.io/llm-arch-kb/ffn-moe/https://lizeman.github.io/llm-arch-kb/ffn-moe/aux-loss-free/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/deepseek-moe/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/ffn-relu/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/geglu/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/gelu/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/gshard/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/kimi-k2-moe/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/mixtral-moe/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/reglu/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/sparse-moe/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/swiglu/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/ffn-moe/switch-transformer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/glossary/https://lizeman.github.io/llm-arch-kb/long-context/https://lizeman.github.io/llm-arch-kb/long-context/activation-beacon/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/long-context/compressive-transformer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/long-context/landmark-attention/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/long-context/longnet/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/long-context/memorizing-transformers/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/long-context/streaming-llm/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/models/https://lizeman.github.io/llm-arch-kb/models/command-r-plus-104b/https://lizeman.github.io/llm-arch-kb/models/deepseek-r1/https://lizeman.github.io/llm-arch-kb/models/deepseek-v1-67b/https://lizeman.github.io/llm-arch-kb/models/deepseek-v2/https://lizeman.github.io/llm-arch-kb/models/deepseek-v3-1/https://lizeman.github.io/llm-arch-kb/models/deepseek-v3-2/https://lizeman.github.io/llm-arch-kb/models/deepseek-v3/https://lizeman.github.io/llm-arch-kb/models/deepseek-v4-pro/https://lizeman.github.io/llm-arch-kb/models/gemma-1-7b/https://lizeman.github.io/llm-arch-kb/models/gemma-2-27b/https://lizeman.github.io/llm-arch-kb/models/gemma-3-27b/https://lizeman.github.io/llm-arch-kb/models/glm-4-5/https://lizeman.github.io/llm-arch-kb/models/glm-4-9b/https://lizeman.github.io/llm-arch-kb/models/glm-130b/https://lizeman.github.io/llm-arch-kb/models/gpt-4/https://lizeman.github.io/llm-arch-kb/models/gpt-oss-120b/https://lizeman.github.io/llm-arch-kb/models/hunyuan-large-389b/https://lizeman.github.io/llm-arch-kb/models/kimi-k1-5/https://lizeman.github.io/llm-arch-kb/models/kimi-k2/https://lizeman.github.io/llm-arch-kb/models/kimi-linear-48b-a3b/https://lizeman.github.io/llm-arch-kb/models/llama-1-65b/https://lizeman.github.io/llm-arch-kb/models/llama-2-70b/https://lizeman.github.io/llm-arch-kb/models/llama-3-1-70b/https://lizeman.github.io/llm-arch-kb/models/llama-3-3-70b/https://lizeman.github.io/llm-arch-kb/models/llama-4-scout/https://lizeman.github.io/llm-arch-kb/models/minimax-01/https://lizeman.github.io/llm-arch-kb/models/minimax-m1/https://lizeman.github.io/llm-arch-kb/models/mistral-7b/https://lizeman.github.io/llm-arch-kb/models/mistral-large-3-675b/https://lizeman.github.io/llm-arch-kb/models/mixtral-8x7b/https://lizeman.github.io/llm-arch-kb/models/nemotron-3-nano-30b/https://lizeman.github.io/llm-arch-kb/models/olmo-1-7b/https://lizeman.github.io/llm-arch-kb/models/olmo-2-13b/https://lizeman.github.io/llm-arch-kb/models/olmo-3-32b/https://lizeman.github.io/llm-arch-kb/models/olmoe-1b-7b/https://lizeman.github.io/llm-arch-kb/models/phi-4-14b/https://lizeman.github.io/llm-arch-kb/models/qwen-2-5-72b/https://lizeman.github.io/llm-arch-kb/models/qwen3-30b-a3b/https://lizeman.github.io/llm-arch-kb/models/qwen3-32b/https://lizeman.github.io/llm-arch-kb/models/qwen3-235b-a22b/https://lizeman.github.io/llm-arch-kb/models/qwen3-next-80b-a3b/https://lizeman.github.io/llm-arch-kb/models/recurrent-gemma-2b/https://lizeman.github.io/llm-arch-kb/normalization/https://lizeman.github.io/llm-arch-kb/normalization/gemma3-norm-everywhere/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/layernorm/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/ngpt/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/norm-placement/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/olmo2-post-norm/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/qk-norm/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/rmsnorm/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/normalization/sandwich-ln/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/paths/https://lizeman.github.io/llm-arch-kb/paths/modern-attention/https://lizeman.github.io/llm-arch-kb/paths/modern-normalization/https://lizeman.github.io/llm-arch-kb/paths/moe-basics/https://lizeman.github.io/llm-arch-kb/positional/https://lizeman.github.io/llm-arch-kb/positional/alibi/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/decoupled-rope/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/longrope/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/nope/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/ntk-aware-rope/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/position-interpolation/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/rope/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/sinusoidal/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/positional/yarn/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/references/https://lizeman.github.io/llm-arch-kb/residual/https://lizeman.github.io/llm-arch-kb/residual/deepnet/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/residual/dynamic-hc/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/residual/hyper-connections/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/residual/normformer/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/residual/residual-overview/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/residual/rezero/2026-05-12T00:00:00.000Zhttps://lizeman.github.io/llm-arch-kb/search/https://lizeman.github.io/llm-arch-kb/timeline/