LLM Architecture KB

Learning path  · Intermediate

From MHA to MLA

Walk the four canonical attention layouts in chronological order — the path from full multi-head to grouped to latent compression.

Attention is the part of the transformer that has changed most under inference pressure. Each step on this path keeps the math recognizably “scaled dot product over heads” but trades a little quality for a lot of KV-cache compression. The endpoint, in 2026, is either GQA (most frontier dense decoders) or MLA (DeepSeek-line).

  1. 1 Multi-Head Attention MHA The original attention layout: H independent heads, each computing scaled dot-product attention on a d_h-dim subspace, concatenated then projected. Now superseded by GQA, MLA in production decoders, but still the conceptual foundation.
  2. 2 Multi-Query Attention MQA One K, V projection shared by every query head. H× smaller KV cache than MHA; small but real quality drop that motivated GQA. The first move in the MHA → MQA → GQA → MLA evolution.
  3. 3 Grouped-Query Attention GQA Group query heads so each group reads one shared K, V pair. 4-8× KV-cache reduction with quality near MHA; the dominant attention layout for dense LLMs from 2023 onward.
  4. 4 Multi-Head Latent Attention MLA Compress K and V to a small per-token latent; reconstruct heads at attention time. ~5–7× smaller KV cache than MHA on DeepSeek-V2 ablations.