Production model

Llama 2 70B

Meta · Released 2023-07-18 · 70B parameters · 4,096 ctx · open weights

Architecture

Positional encoding	RoPE (base 10000)
Norm placement	Pre-Norm
Norm type	RMSNorm
QK-Norm	no
Activation	SwiGLU
Attention	GQA (8 KV heads, 64 query heads)
MoE	—
Other	80 layers, First Llama to ship GQA — established the consensus stack, Llama 2 license (commercial use permitted with restrictions)

Techniques used

Grouped-Query Attention — First production decoder to ship GQA; uptrained from an MHA checkpoint at 5% original-training compute.
Swish-Gated Linear Unit — SwiGLU; established the 8/3·d_model hidden-dim convention.
Root Mean Square Layer Normalization — RMSNorm throughout, Pre-Norm placement; established the open-weights consensus.
Rotary Position Embedding — RoPE base 10000, no scaling (4K context).

Llama 2 70B (July 2023) is the model that established the Pre-Norm RMSNorm + RoPE + GQA + SwiGLU stack as the consensus choice for open-weights dense decoders. Llama 1 had used MHA at 65B; Llama 2 was the first in the family to ship GQA, using the GQA paper’s uptraining recipe to convert an MHA-pretrained checkpoint to GQA at modest additional compute (~5% of original training).

The 70B configuration: 80 transformer layers, 8 KV heads shared across 64 query heads, head dim 128, hidden size 8192, FFN intermediate 28672. Base context 4K. RoPE base 10000 (standard, no scaling). Tokenizer: SentencePiece BPE, 32K vocabulary.

For the current Llama frontier with 128K context, see Llama 3.1 70B. The dense architecture is essentially identical; the context extension and tokenizer changes are the main 2.x → 3.x deltas.

Sources

Export

BibTeX

@article{arxiv_2307_09288,
  title         = {Llama 2 70B},
  author        = {Meta},
  year          = {2023},
  eprint        = {2307.09288},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2307.09288}
}

CSL JSON

{
  "id": "arxiv_2307_09288",
  "type": "article-journal",
  "title": "Llama 2 70B",
  "author": [
    {
      "literal": "Meta"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2023
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2307.09288",
  "number": "2307.09288",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - Llama 2 70B
AU  - Meta
PY  - 2023
JO  - arXiv
AN  - arXiv:2307.09288
UR  - https://arxiv.org/abs/2307.09288
ER  -

Models like this