Gemma 1 7B

Architecture

Gemma 1 7B (February 2024) is the first open release in the Gemma family. The architecture follows the early-2024 consensus on three of four major axes — Pre-Norm, RMSNorm, RoPE — but diverges from the dominant Llama-aligned stack on the FFN activation and on attention geometry:

• GeGLU rather than SwiGLU. Gated GELU on the FFN inner activation, not Swish-gated. The difference in benchmark performance is small (~0.1–0.2 perplexity in published Shazeer-style ablations); it’s a fingerprint of Google’s preferred default rather than a fundamental divergence. See the GeGLU entry.
• Multi-Head Attention with very wide head dim — 16 heads at head_dim 256 (rather than the common 32 heads at 128 or 64 heads at 128). This trades the per-token KV-cache footprint upward (128 × 16 × 2 = 4096 floats per token per layer in fp16) in exchange for richer per-head representational capacity. GQA appears in Gemma 2; in Gemma 1, every query head has its own K, V.

The other components are standard: 28 transformer blocks, RoPE base 10000, 8K context window without scaling tricks, GeGLU at hidden_dim 24576 (4× the 6144 model dim).

The Gemma 2B variant from the same release uses MQA (one shared KV head) rather than MHA — a small-model concession to KV-cache pressure that the 7B doesn’t make. The architectural divergence between 2B and 7B was reduced in Gemma 2 and 3, which use the same GQA + sandwich placement pattern across their entire size families.

For the rest of the family see Gemma 2 27B (sandwich norm

• interleaved local/global SWA appears) and Gemma 3 27B (QK-Norm, norm-everywhere, multimodal).

Sources

• https://arxiv.org/abs/2403.08295
• https://huggingface.co/google/gemma-7b

@article{arxiv_2403_08295,
  title         = {Gemma 1 7B},
  author        = {Google DeepMind},
  year          = {2024},
  eprint        = {2403.08295},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2403.08295}
}

{
  "id": "arxiv_2403_08295",
  "type": "article-journal",
  "title": "Gemma 1 7B",
  "author": [
    {
      "literal": "Google DeepMind"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2403.08295",
  "number": "2403.08295",
  "source": "arXiv"
}

TY  - JOUR
TI  - Gemma 1 7B
AU  - Google DeepMind
PY  - 2024
JO  - arXiv
AN  - arXiv:2403.08295
UR  - https://arxiv.org/abs/2403.08295
ER  -