Production model

DeepSeek V3

DeepSeek-AI · Released 2024-12-26 · 671B parameters (37B active) · 128,000 ctx · open weights

Architecture

Positional encoding	RoPE (decoupled head for MLA)
Norm placement	Pre-Norm
Norm type	RMSNorm
QK-Norm	no
Activation	SwiGLU
Attention	MLA
MoE	DeepSeekMoE with aux-loss-free routing
Other	Multi-Token Prediction auxiliary objective, FP8 training

Techniques used

Multi-Head Latent Attention — Same MLA design as V2 across 61 layers.
Auxiliary-Loss-Free Load Balancing — Used across all MoE layers; replaces the standard balance loss.
DeepSeekMoE — 256 routed experts + 1 shared expert per MoE layer; top-8 routing; aux-loss-free balancing on top.
Swish-Gated Linear Unit — SwiGLU in dense FFN; experts in DeepSeekMoE also use SwiGLU activations.
Pre-Norm, Post-Norm, and Sandwich Placement — Pre-Norm RMSNorm across 61 layers.
Root Mean Square Layer Normalization — RMSNorm, Pre-Norm, across all 61 layers.
Decoupled RoPE — Same decoupled-RoPE head shape carried over from V2.
Rotary Position Embedding — RoPE on the decoupled head of MLA; YaRN scaling for 128K.
YaRN — Yet Another RoPE eXtensioN — YaRN used for the 128K context extension from the base 4K window.
The Residual Stream — Standard Pre-Norm residual across 61 layers.

DeepSeek V3 is a 671B-parameter MoE language model with 37B active parameters per token, trained on 14.8T tokens. It uses MLA for attention compression, DeepSeekMoE with an auxiliary-loss-free load-balancing scheme, and a multi-token-prediction training objective on top of the standard next-token loss.

The architecture inherits MLA and DeepSeekMoE from V2 unchanged. The two notable V3 additions are the aux-loss-free router bias (avoids the usual MoE quality cost of an auxiliary balancing term) and FP8 mixed-precision training at scale.

Sources

Export

BibTeX

@article{arxiv_2412_19437,
  title         = {DeepSeek V3},
  author        = {DeepSeek-AI},
  year          = {2024},
  eprint        = {2412.19437},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2412.19437}
}

CSL JSON

{
  "id": "arxiv_2412_19437",
  "type": "article-journal",
  "title": "DeepSeek V3",
  "author": [
    {
      "literal": "DeepSeek-AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2412.19437",
  "number": "2412.19437",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - DeepSeek V3
AU  - DeepSeek-AI
PY  - 2024
JO  - arXiv
AN  - arXiv:2412.19437
UR  - https://arxiv.org/abs/2412.19437
ER  -

Models like this

Kimi Linear 48B-A3BMoonshot AI50% overlap
Llama 3.1 70BMeta45% overlap
Qwen3-Next 80B-A3BAlibaba (Qwen Team)42% overlap