LLM Architecture KB

Production model

OLMo 1 7B

Allen Institute for AI (AI2) · Released 2024-02-01  ·  7B parameters  ·  2,048 ctx  ·  open weights

Architecture

Positional encoding RoPE (base 10000)
Norm placement Pre-Norm
Norm type LayerNorm (non-parametric)
QK-Norm no
Activation SwiGLU
Attention MHA (32 query heads, 32 KV heads, head dim 128)
MoE —
Other 32 layers, hidden dim 4096, FFN intermediate 11008, Non-parametric LayerNorm (no learned scale/bias) — distinctive vs Llama's RMSNorm, 2048-token context, no attention bias, Fully open: weights, training data (Dolma), training code, intermediate checkpoints, Apache 2.0 license

Techniques used

OLMo 1 7B (February 2024) is the Allen Institute for AI’s first fully-open language model and the start of a lineage that emphasizes reproducibility over architectural novelty. The recipe is close to Llama 1 — Pre-Norm + RoPE + SwiGLU + MHA — chosen not because the team had nothing to add but because the openness contribution was the priority. The one deliberate divergence from Llama is the normalization: OLMo 1 uses a non-parametric LayerNorm (no learned scale or bias) rather than Llama’s RMSNorm. The team cites training stability at long horizons as the motivation.

The “fully open” claim refers to four artifacts published together:

  1. Weights for all checkpoints (every ~250B tokens).
  2. Training data (Dolma, ~3T tokens, also fully open).
  3. Training code (the OLMo trainer in PyTorch / Megatron).
  4. Intermediate evaluations at every checkpoint.

For research on long-horizon training dynamics, hyperparameter ablations, and curriculum effects, OLMo 1 remains the most complete open artifact in the field — Llama and DeepSeek release weights and papers but not the training data or intermediate checkpoints.

For the next generation, see OLMo 2 13B — which switched to RMSNorm, added QK-Norm, and moved the layer-norms into a reordered Post-Norm placement — and OLMo 3 32B, AI2’s flagship dense release.

Sources

Export

BibTeX 
@article{arxiv_2402_00838,
  title         = {OLMo 1 7B},
  author        = {Allen Institute for AI (AI2)},
  year          = {2024},
  eprint        = {2402.00838},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2402.00838}
}
CSL JSON 
{
  "id": "arxiv_2402_00838",
  "type": "article-journal",
  "title": "OLMo 1 7B",
  "author": [
    {
      "literal": "Allen Institute for AI (AI2)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2402.00838",
  "number": "2402.00838",
  "source": "arXiv"
}
RIS 
TY  - JOUR
TI  - OLMo 1 7B
AU  - Allen Institute for AI (AI2)
PY  - 2024
JO  - arXiv
AN  - arXiv:2402.00838
UR  - https://arxiv.org/abs/2402.00838
ER  -

Models like this