DeepSeek V2

Architecture

DeepSeek V2 (May 2024) is the model that introduced MLA and established the DeepSeek architectural lineage. The four defining choices:

• MLA attention: latent-space K, V compression to a per-token bottleneck ($d_c = 512$), with a decoupled RoPE head ($d_R = 64$). 5–7× smaller KV cache than MHA on this scale.
• DeepSeekMoE: 160 routed experts plus 2 shared experts per MoE layer, top-6 routing. Established the fine-grained-plus-shared design pattern that V3 inherits.
• Standard auxiliary balance loss: V2 still uses Switch-style aux loss for MoE load balancing. V3 later replaced this with the aux-loss-free scheme.
• YaRN context extension: trained at 4K base, extended to 128K via YaRN post-pretraining.

The 60 layers and Pre-Norm RMSNorm + SwiGLU stack are otherwise canonical. The cumulative effect of MLA + fine-grained MoE was a sparse-activation model with quality comparable to dense 70B models at ~1/3 the inference cost.

V2 is included here as the original MLA + DeepSeekMoE adopter; for the latest in the lineage, see DeepSeek V3.

Sources

• https://arxiv.org/abs/2405.04434
• https://huggingface.co/deepseek-ai/DeepSeek-V2
• https://github.com/deepseek-ai/DeepSeek-V2

@article{arxiv_2405_04434,
  title         = {DeepSeek V2},
  author        = {DeepSeek-AI},
  year          = {2024},
  eprint        = {2405.04434},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2405.04434}
}

{
  "id": "arxiv_2405_04434",
  "type": "article-journal",
  "title": "DeepSeek V2",
  "author": [
    {
      "literal": "DeepSeek-AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2405.04434",
  "number": "2405.04434",
  "source": "arXiv"
}

TY  - JOUR
TI  - DeepSeek V2
AU  - DeepSeek-AI
PY  - 2024
JO  - arXiv
AN  - arXiv:2405.04434
UR  - https://arxiv.org/abs/2405.04434
ER  -