Qwen 2.5 72B

Architecture

Qwen 2.5 72B (September 2024) is Alibaba’s largest open-weights dense model in the Qwen 2.x generation. Architecturally it follows the consensus open-weights stack of 2024: Pre-Norm RMSNorm + RoPE + GQA + SwiGLU. The technical report describes Qwen 2.5 as a model family spanning 0.5B–72B dense variants plus separate MoE configurations; this entry covers the 72B dense.

Headline characteristics:

• GQA attention throughout — the 72B uses grouped-query attention, sharing each KV head across multiple query heads (the precise group count for 72B is in the model config files).
• RoPE positional encoding with a base scaled to support a 128K-token context window. Longer contexts at inference time are reached with YaRN-style scaling per the team’s recommendations.
• SwiGLU FFN inside Pre-Norm RMSNorm blocks — the standard 2024 recipe.

Qwen 2.5’s distinguishing characteristics are mostly outside the architecture: the pretraining corpus (~18T tokens in the technical report), the tokenizer (byte-level BPE at ~152K vocabulary), and the post-training recipe that makes the Instruct variant competitive with much larger Western models on code and math benchmarks.

For exact head/layer/FFN dimensions, see the model config in the Hugging Face repository.

Sources

• https://arxiv.org/abs/2412.15115
• https://huggingface.co/Qwen/Qwen2.5-72B
• https://qwenlm.github.io/blog/qwen2.5/

@article{arxiv_2412_15115,
  title         = {Qwen 2.5 72B},
  author        = {Alibaba (Qwen Team)},
  year          = {2024},
  eprint        = {2412.15115},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2412.15115}
}

{
  "id": "arxiv_2412_15115",
  "type": "article-journal",
  "title": "Qwen 2.5 72B",
  "author": [
    {
      "literal": "Alibaba (Qwen Team)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2412.15115",
  "number": "2412.15115",
  "source": "arXiv"
}

TY  - JOUR
TI  - Qwen 2.5 72B
AU  - Alibaba (Qwen Team)
PY  - 2024
JO  - arXiv
AN  - arXiv:2412.15115
UR  - https://arxiv.org/abs/2412.15115
ER  -