Production model

OLMoE 1B/7B

Allen Institute for AI (AI2) · Released 2024-09-04 · 6.9B parameters (1.3B active) · 4,096 ctx · open weights

Architecture

Positional encoding	RoPE (base 10000)
Norm placement	Pre-Norm
Norm type	RMSNorm
QK-Norm	yes
Activation	SwiGLU
Attention	MHA (16 query heads, 16 KV heads)
MoE	Top-8 routing across 64 routed experts; no shared expert; Switch-style aux balance loss
Other	16 layers, hidden dim 2048, expert intermediate 1024, QK-Norm in attention (the OLMo 2 line later adopts it across dense variants too), Fully open: weights, training data (Dolma 1.7 + custom MoE mix), training code, intermediate checkpoints

Techniques used

Grouped-Query Attention — GQA across the 16-layer MoE.
Mixtral-Style Coarse MoE — Top-8 routing across 64 small experts; coarse-MoE recipe without DeepSeekMoE's shared experts.
Swish-Gated Linear Unit — SwiGLU inside each routed expert.
Rotary Position Embedding — RoPE (4K base context).

OLMoE 1B/7B is AI2’s first open-weights mixture-of-experts model, released September 2024 with the same openness pledge as OLMo 1 and 2: weights, training data, training code, and intermediate checkpoints all public. Architecturally it’s a 16-layer transformer with 64 small experts per MoE layer; each token routes to 8 of them (top-8 of 64 is a higher K than Mixtral’s top-2 of 8, with much smaller experts).

The choices here line up with the Mixtral MoE family rather than DeepSeekMoE: no shared expert, standard Switch-style auxiliary load-balancing loss. The paper explicitly evaluates DeepSeekMoE-style shared experts in its ablations (Table 7) and reports them as net-neutral at this scale — a notably different empirical conclusion than DeepSeek’s own ablations, possibly due to scale or training data differences. The “should you use shared experts” question therefore remains genuinely contested in the literature.

Context length is 4K base (later extended in derivatives). The dense Pre-Norm + RMSNorm + RoPE + MHA + SwiGLU stack is straightforward; the architectural interest is concentrated in the MoE routing choices above. (Note: KV heads equal query heads here — no GQA — so the inference economics come entirely from the sparse FFN, not from KV-cache compression.)

Sources

Export

BibTeX

@article{arxiv_2409_02060,
  title         = {OLMoE 1B/7B},
  author        = {Allen Institute for AI (AI2)},
  year          = {2024},
  eprint        = {2409.02060},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2409.02060}
}

CSL JSON

{
  "id": "arxiv_2409_02060",
  "type": "article-journal",
  "title": "OLMoE 1B/7B",
  "author": [
    {
      "literal": "Allen Institute for AI (AI2)"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2409.02060",
  "number": "2409.02060",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - OLMoE 1B/7B
AU  - Allen Institute for AI (AI2)
PY  - 2024
JO  - arXiv
AN  - arXiv:2409.02060
UR  - https://arxiv.org/abs/2409.02060
ER  -

Models like this