Production model

OLMo 2 13B

Allen Institute for AI (AI2) · Released 2024-11-26 · 13B parameters · 4,096 ctx · open weights

Architecture

Positional encoding	RoPE (base 500K)
Norm placement	Reordered Post-Norm (norm after attention output, after FFN output, inside residual)
Norm type	RMSNorm
QK-Norm	yes
Activation	SwiGLU
Attention	MHA (40 query heads, 40 KV heads, head dim 128)
MoE	—
Other	40 layers, hidden 5120, FFN intermediate 13824, Z-loss for output-layer stability, Fully open: weights, training data (Dolma 1.7), training code, intermediate checkpoints

Techniques used

Grouped-Query Attention — GQA in the 13B dense model.
Swish-Gated Linear Unit — SwiGLU FFN.
Pre-Norm, Post-Norm, and Sandwich Placement — Deliberate revival of Post-Norm character with QK-Norm for training stability.
OLMo 2 Reordered Post-Norm — OLMo 2's distinctive normalization layout: RMSNorm after each sublayer's output, inside the residual branch. Combined with QK-Norm and Z-loss.
Query-Key Normalization — QK-Norm combined with reordered Post-Norm placement for 13B training stability.
Root Mean Square Layer Normalization — RMSNorm with reordered Post-Norm-style placement.
Rotary Position Embedding — RoPE with base 500K.
The Residual Stream — Standard residual with norms reordered toward Post-Norm character.

OLMo 2 13B is the Allen Institute for AI’s fully-open second-generation dense decoder. Compared to OLMo 1, the architectural changes were deliberately conservative — the team’s stated focus was training stability and reproducibility at long horizons. The two notable choices:

Reordered normalization placement. RMSNorms are placed after the attention output and after the FFN output, inside the residual branch — a layout that AI2 describes as a return toward Post-Norm character, in contrast to the standard Pre-Norm of Llama and most peers. Combined with QK-Norm, this gives the recipe its training-stability properties at the 13B scale without warmup-sensitivity.
Z-loss on the output layer. A small auxiliary penalty on the squared log-partition of the output softmax, encouraging output-layer logits to stay near zero mean.

Attention is plain MHA (40 query heads, 40 KV heads) rather than GQA — at the 13B scale the inference economics of all-equal KV heads are tolerable, and the team prioritized minimising unrelated architectural variables. GQA enters the OLMo line later at the 32B scale.

OLMo 2 is also notable for its openness: the weights, training data (Dolma 1.7), training code, and intermediate checkpoints across pretraining are all public. For research on long-horizon training dynamics, it is currently the most complete artifact in the open ecosystem.

Sources

Export

BibTeX

@article{arxiv_2501_00656,
  title         = {OLMo 2 13B},
  author        = {Allen Institute for AI (AI2)},
  year          = {2024},
  eprint        = {2501.00656},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2501.00656}
}

CSL JSON

{
  "id": "arxiv_2501_00656",
  "type": "article-journal",
  "title": "OLMo 2 13B",
  "author": [
    {
      "literal": "Allen Institute for AI (AI2)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2501.00656",
  "number": "2501.00656",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - OLMo 2 13B
AU  - Allen Institute for AI (AI2)
PY  - 2024
JO  - arXiv
AN  - arXiv:2501.00656
UR  - https://arxiv.org/abs/2501.00656
ER  -

Models like this

Llama 3.1 70BMeta75% overlap
OLMo 3 32BAllen Institute for AI (AI2)60% overlap
Gemma 3 27BGoogle DeepMind58% overlap