DeepSeek LLM 67B

Architecture

DeepSeek LLM 67B (January 2024) is the dense V1 of the DeepSeek family — the deliberate baseline against which V2’s architectural jump was measured. The architecture is the consensus open-weights stack of early 2024:

• Pre-Norm RMSNorm placement.
• RoPE with base 10000 and a 4K context window.
• SwiGLU FFN at standard 8/3 hidden ratio.
• GQA with 64 query heads grouped into 8 KV heads — the first DeepSeek model to use GQA (V1’s smaller 7B variant uses MHA).

There is no MLA, no MoE, no decoupled-RoPE head, no aux-loss-free routing — all those choices arrived with V2 (May 2024) and V3 (December 2024). The V1 paper’s contribution is empirical: disciplined scaling-law studies on bilingual data, with the architecture held constant at the 2024 consensus.

Reading the V1 paper alongside V2 makes the V2 architectural delta legible — V2’s MLA replaces V1’s GQA, V2’s DeepSeekMoE replaces the dense FFN, and V2’s YaRN extension replaces the fixed 4K context. Everything else (Pre-Norm, RMSNorm, SwiGLU base, RoPE base) carries over unchanged.

For later DeepSeek releases see DeepSeek V2 and DeepSeek V3.

Sources

• https://arxiv.org/abs/2401.02954
• https://huggingface.co/deepseek-ai/deepseek-llm-67b-base
• https://github.com/deepseek-ai/DeepSeek-LLM

@article{arxiv_2401_02954,
  title         = {DeepSeek LLM 67B},
  author        = {DeepSeek-AI},
  year          = {2024},
  eprint        = {2401.02954},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2401.02954}
}

{
  "id": "arxiv_2401_02954",
  "type": "article-journal",
  "title": "DeepSeek LLM 67B",
  "author": [
    {
      "literal": "DeepSeek-AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2401.02954",
  "number": "2401.02954",
  "source": "arXiv"
}

TY  - JOUR
TI  - DeepSeek LLM 67B
AU  - DeepSeek-AI
PY  - 2024
JO  - arXiv
AN  - arXiv:2401.02954
UR  - https://arxiv.org/abs/2401.02954
ER  -