Production model

MiniMax-Text-01

MiniMax · Released 2025-01-14 · 456B parameters (45.9B active) · 4,000,000 ctx · open weights

Architecture

Positional encoding	RoPE (base 1e7, applied to half of head dim)
Norm placement	Post-Norm
Norm type	RMSNorm
QK-Norm	no
Activation	SwiGLU
Attention	Lightning Attention (7 layers) interleaved with softmax attention (1 layer); ratio 7:1
MoE	32 routed experts (no shared expert); top-2 routing
Other	80 layers, hidden 6144, 64 attention heads (head dim 128), 8 KV heads (GQA on the softmax layers), 32 experts per MoE layer, top-2 routing, expert FFN dim 9216, Training context up to 1M tokens; inference extrapolates to 4M, Hybrid linear/softmax attention stack

Techniques used

Lightning Attention — 7 Lightning Attention layers : 1 softmax attention layer interleave; 4M context.
DeepSeekMoE — 32 routed experts + 1 shared expert; top-2 routing.
Swish-Gated Linear Unit — SwiGLU FFN inside each routed expert.
Rotary Position Embedding — RoPE; 4M context via Lightning Attention's linear cost.

MiniMax-Text-01 (released January 2025) is the first production decoder to use Lightning Attention at frontier scale. The defining choices:

Hybrid attention stack: 7 out of every 8 layers use Lightning Attention (tiled linear attention); the 8th uses standard softmax attention. The mix gives near-softmax recall on context-rich tasks plus near-linear compute scaling on long sequences.
4M token context: enabled by the linear-attention cost profile in the 7 Lightning layers per group; quadratic cost concentrates in the 1 softmax layer per group.
MoE architecture: 32 routed experts per layer, top-2 routing, no shared expert (shared_intermediate_size: 0 in the released config). 456B total parameters, 45.9B active per token across 80 layers.

The architectural bet — linear attention is viable in production if interleaved with periodic softmax for quality recovery — is the most concrete 2025 case for the linear-attention research line. Whether the 7:1 ratio is optimal across scales is contested, but the existence proof that it works at 456B is now in the record.

Sources

Export

BibTeX

@article{arxiv_2501_08313,
  title         = {MiniMax-Text-01},
  author        = {MiniMax},
  year          = {2025},
  eprint        = {2501.08313},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2501.08313}
}

CSL JSON

{
  "id": "arxiv_2501_08313",
  "type": "article-journal",
  "title": "MiniMax-Text-01",
  "author": [
    {
      "literal": "MiniMax"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2025
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2501.08313",
  "number": "2501.08313",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - MiniMax-Text-01
AU  - MiniMax
PY  - 2025
JO  - arXiv
AN  - arXiv:2501.08313
UR  - https://arxiv.org/abs/2501.08313
ER  -

Models like this