Production model
Llama 3.3 70B Instruct
Architecture
| Positional encoding | RoPE (theta scaled for 128K context) |
|---|---|
| Norm placement | Pre-Norm |
| Norm type | RMSNorm |
| QK-Norm | no |
| Activation | SwiGLU |
| Attention | GQA (8 key-value heads, 64 query heads) |
| MoE | — |
| Other | Same base architecture as Llama 3.1 70B — 80 layers, 128K context, Instruction-tuned only release; the architectural deltas vs 3.1 are post-training, not pretraining, Tokenizer: tiktoken-based, 128K vocabulary |
Llama 3.3 70B Instruct (December 2024) is Meta’s late-2024 post-training refresh of the Llama 3.1 70B base. The pretraining architecture is unchanged — 80 layers, GQA with 8 KV heads / 64 query heads, Pre-Norm RMSNorm, SwiGLU FFN, and a 128K-context RoPE configuration. What changed is the post-training recipe: improved instruction tuning and alignment data lift the 70B-Instruct quality close to the 405B-Instruct on several reasoning benchmarks.
Because the architecture is identical to 3.1, this entry exists primarily to capture the lineage. For architectural detail see the Llama 3.1 70B spec sheet and the Llama 3 paper.
Meta released 3.3 as an instruction-tuned variant only — there is no separately released 3.3 base. The model card on Hugging Face is the canonical disclosure.
Sources
Export
BibTeX
@article{arxiv_2407_21783,
title = {Llama 3.3 70B Instruct},
author = {Meta},
year = {2024},
eprint = {2407.21783},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2407.21783}
}CSL JSON
{
"id": "arxiv_2407_21783",
"type": "article-journal",
"title": "Llama 3.3 70B Instruct",
"author": [
{
"literal": "Meta"
}
],
"issued": {
"date-parts": [
[
2024
]
]
},
"URL": "https://arxiv.org/abs/2407.21783",
"number": "2407.21783",
"source": "arXiv"
}RIS
TY - JOUR
TI - Llama 3.3 70B Instruct
AU - Meta
PY - 2024
JO - arXiv
AN - arXiv:2407.21783
UR - https://arxiv.org/abs/2407.21783
ER -