Phi-4 14B

Architecture

Phi-4 14B (December 2024) is Microsoft Research’s fourth-generation Phi model. The distinguishing claim of the Phi family has never been architectural novelty — it is the data-centric training recipe, and Phi-4’s technical report reinforces that framing. The architecture is described as conventional dense decoder transformer with Pre-Norm RMSNorm, RoPE, GQA, and SwiGLU.

What is novel in Phi-4:

• Synthetic-data-heavy training mix. The technical report details a curriculum that blends high-quality web data, curated educational sources, and synthetic data generated with strong frontier models — an evolution of the “textbooks are all you need” thesis from earlier Phi work.
• 14B parameter scale. Larger than Phi-3 (3.8B / 14B Mini), targeting GPT-4-class reasoning on a single-GPU footprint.
• 16K base context window.

Architectural details (40 layers, hidden size 5120, FFN 17920, GQA with 40 query heads and 10 KV heads, RoPE base 250000) are in the released model config.json rather than highlighted in the technical report — Phi-4’s paper is intentionally data-and-evaluation focused. The open-weights release under MIT license is the canonical artifact for verifying the architecture spec.

Sources

• https://arxiv.org/abs/2412.08905
• https://huggingface.co/microsoft/phi-4

@article{arxiv_2412_08905,
  title         = {Phi-4 14B},
  author        = {Microsoft Research},
  year          = {2024},
  eprint        = {2412.08905},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2412.08905}
}

{
  "id": "arxiv_2412_08905",
  "type": "article-journal",
  "title": "Phi-4 14B",
  "author": [
    {
      "literal": "Microsoft Research"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2412.08905",
  "number": "2412.08905",
  "source": "arXiv"
}

TY  - JOUR
TI  - Phi-4 14B
AU  - Microsoft Research
PY  - 2024
JO  - arXiv
AN  - arXiv:2412.08905
UR  - https://arxiv.org/abs/2412.08905
ER  -