LLM Architecture KB

Production model

Llama 3.1 70B

Meta · Released 2024-07-23  ·  70B parameters  ·  128,000 ctx  ·  open weights

Architecture

Positional encoding RoPE (theta scaled for 128K context)
Norm placement Pre-Norm
Norm type RMSNorm
QK-Norm no
Activation SwiGLU
Attention GQA (8 key-value heads, 64 query heads)
MoE —
Other Tied word embeddings: no, Tokenizer: tiktoken-based, 128K vocabulary

Techniques used

Llama 3.1 70B (July 2024) is Meta’s flagship dense decoder from the Llama 3 family. Architecturally it is the canonical “consensus stack” for dense models in 2024–25: Pre-Norm RMSNorm + RoPE + GQA + SwiGLU. The 70B model has 80 layers, 64 query heads sharing 8 KV heads (GQA group count 8), head dim 128.

Llama 3.1’s training extends context from 8K to 128K via continued pre-training with a scaled RoPE base rather than a parameter-free interpolation scheme like YaRN. The tokenizer changes from the SentencePiece BPE used in Llama 2 to a tiktoken-based BPE with a 128K vocabulary, roughly 4× larger than Llama 2’s 32K.

Relative to Llama 2 70B the pretraining architecture is essentially unchanged — the 2.x → 3.x deltas are the larger vocabulary, the longer trained context, and a substantially larger pretraining corpus (~15T tokens vs ~2T). The Llama 3 paper (“The Llama 3 Herd of Models”, arXiv 2407.21783) is the canonical disclosure.

Sources

Export

BibTeX 
@article{arxiv_2407_21783,
  title         = {Llama 3.1 70B},
  author        = {Meta},
  year          = {2024},
  eprint        = {2407.21783},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2407.21783}
}
CSL JSON 
{
  "id": "arxiv_2407_21783",
  "type": "article-journal",
  "title": "Llama 3.1 70B",
  "author": [
    {
      "literal": "Meta"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2407.21783",
  "number": "2407.21783",
  "source": "arXiv"
}
RIS 
TY  - JOUR
TI  - Llama 3.1 70B
AU  - Meta
PY  - 2024
JO  - arXiv
AN  - arXiv:2407.21783
UR  - https://arxiv.org/abs/2407.21783
ER  -

Models like this