GLM-130B

Architecture

GLM-130B (October 2022) is the first 100B+ open bilingual model, released by Tsinghua KEG and Zhipu AI under a non-commercial research license. The architecture is a snapshot of late-2021 / early-2022 design conventions, predating the Llama-driven settlement on Pre-Norm + RMSNorm + RoPE + SwiGLU + GQA. Reading GLM-130B against contemporary models is useful for seeing what the consensus stack quietly displaced.

The four architectural choices that age GLM-130B against modern models:

• Post-Norm with DeepNorm scaling. The Microsoft DeepNet paper showed that Post-Norm could be made stable at very deep stacks by rescaling residuals by a depth-derived constant $\beta = (8N)^{-1/4}$. GLM-130B uses this recipe at 70 layers. Modern open models settled on Pre-Norm instead.
• LayerNorm, not RMSNorm. RMSNorm was published in 2019 (Zhang & Sennrich) but didn’t become the open-stack default until Llama 1 made the choice in early 2023. See the LayerNorm and RMSNorm entries.
• GeLU FFN, not a GLU variant. Plain ReLU/GeLU FFN at 4× hidden expansion. Shazeer’s GLU-variant ablations (2020) had been published but adoption lagged; the GLU split came with Llama 1’s SwiGLU choice in 2023.
• 2D RoPE. GLM-130B encodes position as a 2D grid (segment index, intra-segment position) with RoPE applied on each axis. The 2D structure supports the GLM blank-infilling pretraining objective, which uses non-causal attention within infill spans. Single-axis RoPE became standard with Llama; 2D RoPE remains a GLM-family signature, carried into GLM-4 and GLM-4.5.

The GLM pretraining objective itself — autoregressive blank infilling — is also distinct from the pure causal-LM objective Llama and successors use. The model masks contiguous spans, then predicts each masked span autoregressively while the surrounding tokens have full bidirectional visibility. GLM-130B inference can be operated either causally or with the bidirectional infill mode active, depending on the task framing.

For later GLM family members see GLM-4 9B — which keeps the Post-Norm placement and 2D RoPE while moving to RMSNorm + SwiGLU + GQA — and GLM-4.5, the 355B / 32B-active MoE flagship.

Sources

• https://arxiv.org/abs/2210.02414
• https://github.com/THUDM/GLM-130B

@article{arxiv_2210_02414,
  title         = {GLM-130B},
  author        = {Tsinghua KEG / Zhipu AI},
  year          = {2022},
  eprint        = {2210.02414},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2210.02414}
}

{
  "id": "arxiv_2210_02414",
  "type": "article-journal",
  "title": "GLM-130B",
  "author": [
    {
      "literal": "Tsinghua KEG / Zhipu AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2022
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2210.02414",
  "number": "2210.02414",
  "source": "arXiv"
}

TY  - JOUR
TI  - GLM-130B
AU  - Tsinghua KEG / Zhipu AI
PY  - 2022
JO  - arXiv
AN  - arXiv:2210.02414
UR  - https://arxiv.org/abs/2210.02414
ER  -