Production model

Gemma 2 27B

Google DeepMind · Released 2024-06-27 · 27B parameters · 8,192 ctx · open weights

Architecture

Positional encoding	RoPE
Norm placement	Pre-Norm + Post-Norm (sandwich-style)
Norm type	RMSNorm
QK-Norm	no
Activation	GeGLU
Attention	GQA (16 KV heads, 32 query heads) + interleaved local/global SWA (window 4096)
MoE	—
Other	Soft attention logit cap (tanh clamping at ±50), Knowledge distillation from a larger Gemma teacher model

Techniques used

Sliding Window Attention — 1:1 alternation of SWA and global attention layers; Gemma 3 tightens this to 5:1.
GELU-Gated Linear Unit — GeGLU FFN.
Pre-Norm, Post-Norm, and Sandwich Placement — First Gemma generation to use sandwich placement.
Sandwich-LN — First Gemma generation to use Sandwich placement: norm before and after each sublayer.

Gemma 2 27B (June 2024) is the Gemma family’s transition model — the release where Google’s “norm-everywhere” sandwich placement first appeared, and where the interleaved SWA + global attention pattern was established. The architectural delta from Gemma 1:

Sandwich norm placement: RMSNorm before and after the attention/FFN sub-layers, inside the residual branch. The “norm-everywhere” pattern that Gemma 3 inherits unchanged. See the norm-placement entry.
Interleaved SWA + global: most layers use sliding-window attention at window 4096; every other layer uses global attention. The 1:1 alternation was tightened to 5:1 in Gemma 3.
Soft attention logit cap: a tanh-based clamp at ±50 prevents attention logit blow-up without requiring QK-Norm. Gemma 3 adds QK-Norm on top of this cap.

The 27B model uses 32 query heads sharing 16 KV heads (G = 2, modest GQA). The context length is 8K base, with no positional extension beyond that in the released checkpoint.

For the Gemma family’s current frontier, see Gemma 3 27B. The two entries together let readers compare what changed between the family’s 2024 and 2025 generations.

Sources

Export

BibTeX

@article{arxiv_2408_00118,
  title         = {Gemma 2 27B},
  author        = {Google DeepMind},
  year          = {2024},
  eprint        = {2408.00118},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2408.00118}
}

CSL JSON

{
  "id": "arxiv_2408_00118",
  "type": "article-journal",
  "title": "Gemma 2 27B",
  "author": [
    {
      "literal": "Google DeepMind"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2408.00118",
  "number": "2408.00118",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - Gemma 2 27B
AU  - Google DeepMind
PY  - 2024
JO  - arXiv
AN  - arXiv:2408.00118
UR  - https://arxiv.org/abs/2408.00118
ER  -

Models like this