Production model

Hunyuan-Large 389B

Tencent · Released 2024-11-05 · 389B parameters (52B active) · 256,000 ctx · open weights

Architecture

Positional encoding	RoPE (base 10000 with dynamic scaling, paper reports an effective base near 1e9 at 256K context)
Norm placement	Pre-Norm
Norm type	RMSNorm
QK-Norm	—
Activation	SwiGLU
Attention	GQA (80 query heads, 8 KV heads) with Cross-Layer Attention (KV cache shared across 2 consecutive layers)
MoE	16 specialized experts, top-1 routing + 1 always-on shared expert; expert-specific learning rate; recycling routing
Other	64 layers, hidden dim 6400, FFN intermediate 18304, Cross-Layer Attention with share factor 2 (every pair of layers shares KV), 128K vocabulary, ~7T training tokens, Custom Tencent license (research + commercial, with restrictions at very high revenue)

Techniques used

Grouped-Query Attention — 80 query heads grouped into 8 KV heads (G = 10) across 64 layers.
DeepSeekMoE — 1 shared expert + 16 specialized experts with top-1 routing; specialists train at ~0.31× the shared expert's learning rate (technical report §3).
Swish-Gated Linear Unit — SwiGLU FFN inside each expert (technical report Table 2).
Root Mean Square Layer Normalization — Pre-Norm RMSNorm across the 64-layer MoE.
Rotary Position Embedding — RoPE base pushed to 10⁹ to support the 256K context.

Hunyuan-Large (November 2024) is Tencent’s largest open-weights MoE: 389B total parameters, 52B active per token, 64 layers. The technical report (arXiv:2411.02265) discloses an architecture that follows the dominant 2024 stack with two distinctive MoE-side choices:

One shared + one specialized expert per token. The router selects top-1 out of 16 specialized experts; the chosen specialist runs alongside an always-on shared expert. This is the DeepSeekMoE shape transposed — fewer, larger experts with top-1 instead of many small experts with top-K.
Expert-specific learning rate. The specialized experts train with a learning rate scaled to ~0.31× the shared expert’s rate (report §3). Tencent argues this stabilizes early MoE training by letting the shared expert lead while the specialists differentiate.
Recycling routing. Tokens routed to overloaded experts are recycled to the next-best available expert rather than dropped — a load-balancing fallback distinct from the aux-loss-free approach used by DeepSeek V3.

The non-MoE components are mostly conventional: Pre-Norm RMSNorm, SwiGLU FFN inside each expert, GQA with 80 query heads sharing 8 KV heads, and RoPE with dynamic scaling for the 256K context window. On top of GQA the model adds Cross-Layer Attention (CLA) with share factor 2 — every two consecutive layers reuse the same KV projections — for an additional ~2× KV-cache reduction on top of GQA. This is the same compression idea (multiple layers sharing one KV cache) that later models use under different names.

QK-Norm presence is not surfaced in the technical report and is left out rather than guessed (per the citation policy).

Sources

Export

BibTeX

@article{arxiv_2411_02265,
  title         = {Hunyuan-Large 389B},
  author        = {Tencent},
  year          = {2024},
  eprint        = {2411.02265},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2411.02265}
}

CSL JSON

{
  "id": "arxiv_2411_02265",
  "type": "article-journal",
  "title": "Hunyuan-Large 389B",
  "author": [
    {
      "literal": "Tencent"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2411.02265",
  "number": "2411.02265",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - Hunyuan-Large 389B
AU  - Tencent
PY  - 2024
JO  - arXiv
AN  - arXiv:2411.02265
UR  - https://arxiv.org/abs/2411.02265
ER  -

Models like this

GLM-4.5Zhipu AI83% overlap
Llama 2 70BMeta80% overlap
Qwen3-Next 80B-A3BAlibaba (Qwen Team)71% overlap