Production model
gpt-oss-120b
Architecture
| Positional encoding | RoPE (base 150K, YaRN scaling from a 4K base context to 131K) |
|---|---|
| Norm placement | Pre-Norm |
| Norm type | RMSNorm |
| QK-Norm | no |
| Activation | SwiGLU |
| Attention | GQA (64 query heads, 8 KV heads) with strict alternation of 128-token sliding-window and full-attention layers |
| MoE | Sparse MoE — 128 experts per layer, top-4 routing |
| Other | 117B total parameters / 5.1B active per token, 36 layers, hidden 2880, head dim 64; 18 sliding + 18 full attention layers alternating, Vocab 201,088; max position 131,072, First OpenAI open-weights release since GPT-2, Apache 2.0 license, MXFP4 quantization of expert weights as the default inference format |
gpt-oss-120b (August 2025) is OpenAI’s first open-weights model release since GPT-2 in 2019. The introduction blog post and the Hugging Face model card disclose the architecture as a sparse Mixture-of-Experts transformer with the now-dominant open-weights backbone:
- 117B total parameters, 5.1B active per token. 128 experts per MoE layer with top-4 routing — a wider, finer-grained recipe than Mixtral’s 8-experts / top-2 baseline.
- Pre-Norm RMSNorm + SwiGLU + GQA backbone with RoPE base 150K and YaRN scaling from a 4K base context up to 131K.
- 36 layers, strict 1:1 alternation of 128-token banded sliding-window attention and full-attention layers (18 sliding + 18 full). GQA is configured with 64 query heads and 8 KV heads at head dim 64.
- MXFP4 expert quantization is documented as the default inference format — the expert weight matrices ship in the OCP Microscaling FP4 format, with the rest of the model in BF16/FP16.
The release is under Apache 2.0 license. The fields above come from the official
announcement, the Hugging Face model card, and the released config.json (which encodes
the layer types, head counts, and RoPE configuration).