Kimi Linear 48B-A3B

Architecture

Kimi Linear (October 2025) is Moonshot AI’s hybrid linear-attention model and the first open-weights production release to combine Kimi Delta Attention (KDA) — a linear-attention variant — with Multi-Head Latent Attention at meaningful scale. The architectural bet is the same one MiniMax made with Lightning Attention: a few full-attention layers preserve quality, while a majority of cheap linear-attention layers cut the KV cache and the inference cost at long context.

The hybrid stack. 27 transformer layers split 20 / 7 between KDA and MLA. Layers 4, 8, 12, 16, 20, 24, and 27 run full MLA; everything else is KDA. The pattern is approximately three KDA layers for every one MLA layer — the same ratio Qwen3-Next uses for its Gated DeltaNet stack, and similar in spirit to MiniMax’s 7:1 Lightning : softmax mix but tighter on the full-attention budget.

What KDA changes vs Gated DeltaNet. Qwen3-Next’s Gated DeltaNet applies a single scalar gate per attention head to control the memory decay rate. KDA replaces that scalar with a channel-wise gate — one gating value per feature dimension. The paper argues this gives the recurrent state finer control over what to forget per channel, which matters more at long context where any single decay rate must compromise across token types.

MLA with NoPE. The seven full-attention layers run MLA in a configuration close to DeepSeek V3’s — kv_lora_rank 512, decoupled head dim 64, non-position head dim 128 — but with one twist: the MLA body uses NoPE (no positional encoding on the latent path), and RoPE is applied only to the small 64-dim head. The causal mask provides positional asymmetry on the non-rotated path; the decoupled rotated head carries the actual position signal. See decoupled RoPE.

MoE shape. DeepSeekMoE lineage — 256 routed experts plus one shared expert, top-8 routing per token. Routing uses a sigmoid gate (not the more common softmax) with grouped top-K selection. The first layer is dense; layers 2–27 are MoE.

Why this matters. The paper’s headline numbers come from the hybrid stack’s asymptotic cost: at 128K context the model is ~4× faster than a full-MLA baseline on RULER while matching quality, and at 1M context decoding TPOT is ~6× faster than the full-attention reference. The KV cache shrinks by up to 75% — proportional to the share of KDA layers (which carry a fixed-size recurrent state rather than a growing KV cache).

For the rest of Moonshot’s lineup see Kimi K2 (1T total / 32B active, full-MLA, no linear-attention layers).

Sources

• https://arxiv.org/abs/2510.26692
• https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct
• https://github.com/MoonshotAI/Kimi-Linear

@article{arxiv_2510_26692,
  title         = {Kimi Linear 48B-A3B},
  author        = {Moonshot AI},
  year          = {2025},
  eprint        = {2510.26692},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2510.26692}
}

{
  "id": "arxiv_2510_26692",
  "type": "article-journal",
  "title": "Kimi Linear 48B-A3B",
  "author": [
    {
      "literal": "Moonshot AI"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2025
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2510.26692",
  "number": "2510.26692",
  "source": "arXiv"
}

TY  - JOUR
TI  - Kimi Linear 48B-A3B
AU  - Moonshot AI
PY  - 2025
JO  - arXiv
AN  - arXiv:2510.26692
UR  - https://arxiv.org/abs/2510.26692
ER  -