Production model

Kimi K2

Moonshot AI · Released 2025-07-11 · 1T parameters (32B active) · 128,000 ctx · open weights

Architecture

Positional encoding	RoPE (decoupled head, MLA-style)
Norm placement	Pre-Norm
Norm type	RMSNorm
QK-Norm	no
Activation	SwiGLU
Attention	MLA-style latent attention
MoE	Sparse MoE with 384 routed experts + 1 shared expert; top-K routing
Other	1T total parameters / 32B active per token — first publicly released open-weights model at the trillion-parameter scale, 61 layers (1 dense + 60 MoE), hidden 7168, 64 attention heads, 384 routed experts + 1 shared expert, top-8 routing, MoE intermediate 2048 per expert, Pre-trained on 15.5T tokens; block-FP8 default weight format, Trained with MuonClip optimizer (Muon family with weight clipping for large-scale stability), Modified MIT-style license

Techniques used

Multi-Head Latent Attention — MLA-style latent attention with a decoupled RoPE head — the same pattern introduced in DeepSeek-V2.
Kimi K2 MoE — 1T total / 32B active across 384 routed experts plus 1 shared expert.

Kimi K2 (July 2025) is Moonshot AI’s open-weights flagship — the first publicly released model at the trillion-parameter scale. The architecture is in the DeepSeek V3 family lineage:

MLA-style latent attention with a decoupled RoPE head — the same MLA pattern that compresses K and V to a per-token latent while carrying position information through a separate small head.
Sparse MoE with an unusually large expert count: 384 routed experts plus one always-on shared expert. By comparison, DeepSeek V3 uses 256 + 1. The wider expert pool changes the routing dynamics and training-stability tradeoffs without changing the fundamental DeepSeekMoE shape.
Pre-Norm RMSNorm + SwiGLU for the rest of the stack — the dominant 2024–25 open consensus.

The headline non-architectural difference from DeepSeek’s training recipe is the MuonClip optimizer — a variant of the Muon optimizer family (Jordan et al., a 2024 line of work on matrix-aware second-order updates) with weight clipping for large-scale stability. The Moonshot team’s writeup attributes part of K2’s training-cost efficiency at trillion-scale to this optimizer choice.

Specific architecture parameters not surfaced in the public model card and repo at release — exact layer count, exact head dim, exact routing top-K, exact load-balancing recipe — are not reproduced here (per the citation policy). The model card lists the high-level shape; the GitHub repo holds the inference code and confirms the MoE expert count.

For the smaller / earlier Moonshot release see the Kimi K1.5 entry (when added).

Sources

Models like this