Production model

Llama 1 65B

Meta · Released 2023-02-24 · 65B parameters · 2,048 ctx · open weights

Architecture

Positional encoding	RoPE (base 10000)
Norm placement	Pre-Norm
Norm type	RMSNorm
QK-Norm	no
Activation	SwiGLU
Attention	MHA (64 query heads = 64 KV heads)
MoE	—
Other	80 layers, First open-weights model to combine RMSNorm + RoPE + SwiGLU as a deliberate stack, Non-commercial research license (restrictions later relaxed in Llama 2)

Techniques used

Multi-Head Attention — Original Llama, before GQA was incorporated in Llama 2.
Swish-Gated Linear Unit — First open-weights SwiGLU production model.
Root Mean Square Layer Normalization — First production open-weights model to adopt RMSNorm + Pre-Norm at scale.
Rotary Position Embedding — RoPE base 10000; 2K base context.

Llama 1 65B (February 2023) is the original Llama and the model that established the Pre-Norm RMSNorm + RoPE + SwiGLU triple as the open-weights consensus stack. Per the Llama 1 paper’s Table 2, each of those choices was justified empirically against the alternatives available at the time:

RMSNorm over LayerNorm: cited Zhang & Sennrich 2019; ~7% wall-clock speedup at no measurable quality cost.
RoPE over learned or sinusoidal position embeddings: cited Su et al. 2021; cleaner long-context behavior and relative-position math via the rotation identity.
SwiGLU over GELU: cited Shazeer 2020; 0.5–1.3 perplexity points improvement at matched parameter count.

Attention is still MHA — 64 query heads, 64 KV heads, head dim 128. The decision to move to GQA came with Llama 2; the uptraining technique that made the switch cheap was from Ainslie et al. 2023, published a few months later.

Base context length is 2K. Subsequent extension to 32K and beyond came post-release via fine-tuning recipes like PI and YaRN.

For the GQA-era Llama, see Llama 2 70B and Llama 3.1 70B.

Sources

Export

BibTeX

@article{arxiv_2302_13971,
  title         = {Llama 1 65B},
  author        = {Meta},
  year          = {2023},
  eprint        = {2302.13971},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2302.13971}
}

CSL JSON

{
  "id": "arxiv_2302_13971",
  "type": "article-journal",
  "title": "Llama 1 65B",
  "author": [
    {
      "literal": "Meta"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2023
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2302.13971",
  "number": "2302.13971",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - Llama 1 65B
AU  - Meta
PY  - 2023
JO  - arXiv
AN  - arXiv:2302.13971
UR  - https://arxiv.org/abs/2302.13971
ER  -

Models like this