LLM Architecture KB

Production model

Mixtral 8x7B

Mistral AI · Released 2023-12-11  ·  46.7B parameters (12.9B active)  ·  32,768 ctx  ·  open weights

Architecture

Positional encoding RoPE (base 1000000)
Norm placement Pre-Norm
Norm type RMSNorm
QK-Norm no
Activation SwiGLU
Attention GQA (32 query heads, 8 KV heads)
MoE 8 experts per MoE layer, top-2 routing, all FFNs are MoE; standard load-balancing auxiliary loss
Other 32 layers, d_model 4096, d_ff 14336, Apache 2.0 license — the first open-weights production MoE

Techniques used

  • Grouped-Query Attention — Same GQA layout as Mistral 7B (32Q / 8KV) carried into the MoE descendant.
  • Mixtral-Style Coarse MoE — Reference implementation — 8 SwiGLU experts per layer, top-2 routing, all 32 transformer layers MoE.

Mixtral 8x7B (December 2023) is the model that brought sparse mixture-of-experts to the open-weights world at production quality. Each of its 32 transformer layers replaces the dense FFN with 8 experts of the same shape; a learned router selects the top 2 per token and mixes their outputs by routing weights. Total parameter count is 46.7B; active count per token is 12.9B — roughly a 3.6× sparsity factor.

The architectural choices outside the MoE layer match the Mistral 7B ancestor: Pre-Norm RMSNorm, RoPE (base 10⁶, an extended setting that supports 32K context natively), GQA with 32 query heads and 8 KV heads, and SwiGLU inside each expert. Unlike Mistral 7B, Mixtral 8x7B drops sliding-window attention — at 32K base context the cache cost was tolerated for the full-attention quality.

The routing recipe is what the Mixtral-MoE entry describes in detail: top-2 of 8 full-sized experts, standard Switch-style load-balancing auxiliary loss, no shared expert. DeepSeekMoE later contested this granularity — smaller, more numerous experts with a shared always-on expert — but in late 2023 Mixtral’s coarse recipe was the production-validated baseline.

Mixtral 8x7B’s release under Apache 2.0 set the licensing precedent that subsequent open-weights MoE models (OLMoE, DeepSeek V2, DeepSeek V3) followed. Inference-time the model competes with Llama 2 70B at roughly the compute of a 13B dense model.

Sources

Export

BibTeX 
@article{arxiv_2401_04088,
  title         = {Mixtral 8x7B},
  author        = {Mistral AI},
  year          = {2023},
  eprint        = {2401.04088},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2401.04088}
}
CSL JSON 
{
  "id": "arxiv_2401_04088",
  "type": "article-journal",
  "title": "Mixtral 8x7B",
  "author": [
    {
      "literal": "Mistral AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2023
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2401.04088",
  "number": "2401.04088",
  "source": "arXiv"
}
RIS 
TY  - JOUR
TI  - Mixtral 8x7B
AU  - Mistral AI
PY  - 2023
JO  - arXiv
AN  - arXiv:2401.04088
UR  - https://arxiv.org/abs/2401.04088
ER  -

Models like this