Production model

Qwen3-Next 80B-A3B

Alibaba (Qwen Team) · Released 2025-09-12 · 80B parameters (3B active) · 262,144 ctx · open weights

Architecture

Positional encoding	RoPE on Gated Attention layers (head-dim 64); no positional encoding on Gated DeltaNet layers
Norm placement	Pre-Norm
Norm type	Zero-centered, weight-decayed RMSNorm
QK-Norm	—
Activation	SwiGLU
Attention	Hybrid: 3 Gated DeltaNet (linear) layers per 1 Gated Attention (full) layer, repeated 12× = 48 layers total. Gated Attention: 16 Q heads / 2 KV heads, head-dim 256.
MoE	Sparse MoE — 512 experts, top-10 routing, 1 shared expert
Other	Hybrid linear + full attention at a 3:1 layer ratio, Multi-Token Prediction (MTP) head for speculative decoding, 262K native context; up to ~1M with YaRN, Apache 2.0 license

Techniques used

Grouped-Query Attention — Gated Attention layers (1 in every 4 layers) use 16 query heads sharing 2 KV heads.
Linear Attention — Gated DeltaNet (a linear-attention variant) on 36 of 48 layers, interleaved 3:1 with Gated Attention; 75% of layers use linear cost.
DeepSeekMoE — 512 routed experts + 1 shared expert; top-10 routing — the widest expert pool in any open-weights release at entry's verified date.
Swish-Gated Linear Unit — SwiGLU FFN inside each routed expert (config.json).
Root Mean Square Layer Normalization — Zero-centered, weight-decayed RMSNorm — a stability tweak over standard RMSNorm; weight decay applied to the gain parameter.
Rotary Position Embedding — RoPE applied only to the Gated Attention layers (rotary dim 64); Gated DeltaNet layers carry no positional encoding.
YaRN — Yet Another RoPE eXtensioN — YaRN extension from the 262K native context to ~1M reported by the Qwen team.

Qwen3-Next 80B-A3B (September 2025) is Alibaba’s first production hybrid linear-attention decoder: 80B total parameters, 3B active per token, 48 layers arranged as 12 repetitions of 3 Gated DeltaNet layers followed by 1 Gated Attention layer.

Gated DeltaNet is a linear-attention variant in the linear-attention family. It provides $O(L)$ scaling and small KV cost; 75% of Qwen3-Next’s layers use it.
Gated Attention is a standard full-attention layer with 16 query heads sharing 2 KV heads (head-dim 256) and a 64-dim RoPE head. 25% of layers — every fourth — use it, providing the high-fidelity reasoning anchor that the linear layers alone underperform on.

The MoE shape is unusually wide: 512 experts with top-10 routing plus one always-on shared expert, intermediate dim 512 per expert. The wide pool, with the shared expert, puts Qwen3-Next closer to the DeepSeek-MoE / Kimi-K2 line than to the no-shared-expert Qwen3 flagship.

Other distinguishing choices in the public materials:

Zero-centered, weight-decayed RMSNorm as the normalization. The model card and vLLM writeup describe this as a stability tweak over standard RMSNorm; weight decay is applied to the gain parameter.
Multi-Token Prediction (MTP) head for speculative decoding at inference time.
262K native context with YaRN extension to ~1M reported by the team.

Quality and efficiency are framed in the team’s writeup as a 10× lower training cost and ~10× higher long-context throughput than Qwen3 32B at comparable downstream task quality. Per the citation policy, exact ablation deltas are taken from the team’s writeups and Hugging Face materials; details not surfaced in those sources (e.g. QK-norm presence) are left out rather than guessed.

Sources

Models like this