Production model
Llama 4 Scout
Architecture
| Positional encoding | iRoPE (interleaved RoPE / NoPE-style no-position layers) |
|---|---|
| Norm placement | Pre-Norm |
| Norm type | RMSNorm |
| QK-Norm | no |
| Activation | SwiGLU |
| Attention | iRoPE (Llama-4) over GQA |
| MoE | 16 routed experts + 1 shared expert, top-1 routing |
| Other | First Llama with native Mixture of Experts, iRoPE: a fraction of layers omit RoPE entirely (NoPE-style) for length generalization, 10M-token context window in the Scout configuration, Native multimodal (text + image) via early fusion, Llama 4 Community License |
Techniques used
- No Position Encoding — iRoPE — a fraction of layers omit RoPE entirely (NoPE-style) for length generalization; the rest carry standard RoPE.
Llama 4 Scout (April 2025) is the first Llama generation to ship native Mixture of Experts and the first to abandon a single, uniform attention pattern across all layers. The two architectural pivots:
- MoE with shared expert. 16 routed experts plus one always-on shared expert; top-1 routing. The 109B total parameter count includes all experts; only ~17B are active per token. The shared-expert pattern is conceptually adjacent to DeepSeekMoE, though the precise routing recipe is not detailed publicly.
- iRoPE — interleaved RoPE / no-RoPE layers. Some layers carry RoPE position information, others omit positional encoding entirely (a NoPE-style layer). The motivation given in Meta’s announcement is length generalization: the no-RoPE layers don’t have a baked-in distance preference, which helps the model stay coherent at the 10M-token context window the Scout configuration targets.
The other components remain in the established Llama lineage: Pre-Norm RMSNorm, SwiGLU FFN, GQA. Multimodality is native (image tokens enter the same transformer, not via a separate encoder + projector).
Llama 4 was released without an arXiv tech report; the public information comes from Meta’s announcement post and the Hugging Face model card. Specifics that aren’t in those sources — exact head/layer/expert counts, FFN expansion ratio, RoPE base values, the precise iRoPE interleaving pattern — are left out rather than guessed (per the citation policy).
For prior Llama generations see Llama 1 65B, Llama 2 70B, and Llama 3.1 70B.
Sources
- https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct
Models like this
- Kimi Linear 48B-A3BMoonshot AI13% overlap