Mistral 7B

Architecture

Mistral 7B (September 2023) is the model that brought sliding window attention to the production frontier and gave Apache-licensed open weights to a model competitive with much larger dense baselines. Architecturally a 32-layer transformer with the Pre-Norm RMSNorm + RoPE

• GQA + SwiGLU canonical stack, plus the defining addition:

• Sliding window attention with $W = 4096$. Each layer’s attention is local; the stacked receptive field reaches $32 \times 4096 \approx 131K$ tokens. The recipe demonstrated that pure local attention is sufficient for a competitive 7B-class model. See the SWA entry.

The Mistral 7B paper’s empirical comparisons (Table 2) showed the model matching or beating Llama 2 13B on most benchmarks despite roughly half the parameters — a result mostly attributable to data and training rather than architecture, but the architectural simplicity made the recipe easily reproducible. The 32-layer configuration: hidden size 4096, FFN intermediate 14336, RoPE base 10000, 32K base context.

Subsequent open-weights efforts adopted variations of the same stack: Mixtral 8x7B from the same lab kept the Mistral-7B shape per expert while dropping SWA, and the OLMo family from AI2 picked up the GQA + RoPE + SwiGLU triple. Mistral 7B remains the canonical production reference for sliding window attention.

Sources

• https://arxiv.org/abs/2310.06825
• https://huggingface.co/mistralai/Mistral-7B-v0.1
• https://mistral.ai/news/announcing-mistral-7b/

@article{arxiv_2310_06825,
  title         = {Mistral 7B},
  author        = {Mistral AI},
  year          = {2023},
  eprint        = {2310.06825},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2310.06825}
}

{
  "id": "arxiv_2310_06825",
  "type": "article-journal",
  "title": "Mistral 7B",
  "author": [
    {
      "literal": "Mistral AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2023
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2310.06825",
  "number": "2310.06825",
  "source": "arXiv"
}

TY  - JOUR
TI  - Mistral 7B
AU  - Mistral AI
PY  - 2023
JO  - arXiv
AN  - arXiv:2310.06825
UR  - https://arxiv.org/abs/2310.06825
ER  -