GLM-4 9B

Architecture

GLM-4 9B (June 2024) is the openly-released variant of the GLM-4 family, the successor to GLM-130B. The architecture documents the family’s gradual convergence with the post-Llama consensus stack, while keeping two signature GLM choices:

• Retained from GLM-130B:
  • 2D RoPE position encoding — the GLM family’s hallmark since 2022. Each token’s position is a 2D coordinate (segment index, intra-segment position); RoPE is applied independently to each axis. This pairs naturally with the GLM blank-infilling pretraining objective, which the GLM-4 series continues to use alongside causal LM.
  • Post-Norm-flavored placement with depth-derived residual scaling (DeepNet-style). The placement choice keeps the GLM family on the minority side of the Pre vs Post debate.
• Updated to modern defaults:
  • RMSNorm replaces the LayerNorm used in GLM-130B.
  • SwiGLU FFN replaces the plain GeLU FFN of GLM-130B — adopting the GLU-variant FFN settled on by the Llama family.
  • GQA replaces MHA — the same KV-cache reduction adopted across nearly every open release in 2023–24.

The base GLM-4-9B ships with an 8K context window. A separate GLM-4-9B-Chat-1M variant extends this to 1M tokens through long-context continued training; the procedure is described in §3 of the GLM-4 tech report. The 1M variant is the family’s headline long-context capability but is a derivative checkpoint, not the base configuration.

GQA sharing is unusually aggressive — 32 query heads share just 2 KV heads (G = 16). This is closer to MQA than the G = 4 / G = 8 ratios common across Llama 3 and Qwen 2/3, and it pushes per-token KV-cache footprint very low at the cost of expressiveness in the KV projections.

For the family’s earlier and later releases see GLM-130B and GLM-4.5. The 32B GLM-4-Plus and other larger GLM-4 variants are not released openly; their architecture is similar to GLM-4-9B at increased scale per the tech report, but the precise specifications are not publicly documented.

Sources

• https://arxiv.org/abs/2406.12793
• https://huggingface.co/THUDM/glm-4-9b
• https://github.com/THUDM/GLM-4

@article{arxiv_2406_12793,
  title         = {GLM-4 9B},
  author        = {Zhipu AI / Tsinghua KEG},
  year          = {2024},
  eprint        = {2406.12793},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2406.12793}
}

{
  "id": "arxiv_2406_12793",
  "type": "article-journal",
  "title": "GLM-4 9B",
  "author": [
    {
      "literal": "Zhipu AI / Tsinghua KEG"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2406.12793",
  "number": "2406.12793",
  "source": "arXiv"
}

TY  - JOUR
TI  - GLM-4 9B
AU  - Zhipu AI / Tsinghua KEG
PY  - 2024
JO  - arXiv
AN  - arXiv:2406.12793
UR  - https://arxiv.org/abs/2406.12793
ER  -