Production model

Gemma 3 27B

Google DeepMind · Released 2025-03-12 · 27B parameters · 128,000 ctx · open weights

Architecture

Positional encoding	RoPE (local SWA layers at 10K base, global layers at 1M base)
Norm placement	Pre-Norm + Post-Norm (norm-everywhere)
Norm type	RMSNorm
QK-Norm	yes
Activation	GeGLU
Attention	Interleaved SWA + global (5 local : 1 global)
MoE	—
Other	Soft attention logit cap, Multilingual + multimodal vision encoder (text-only entry omits vision)

Techniques used

Grouped-Query Attention — GQA at the head level across both local SWA and global attention layers.
Sliding Window Attention — Interleaved 5 SWA layers (window 4096) : 1 global attention layer; the dominant long-context strategy in Gemma 3.
GELU-Gated Linear Unit — GeGLU FFN; the dominant variant choice in the Gemma family.
Swish-Gated Linear Unit — GeGLU — same gated structure with GELU in place of Swish.
Gemma 3 Norm-Everywhere — Norm-everywhere applied across the entire 27B family (and the 1B/4B/12B variants), with QK-Norm on top.
Pre-Norm, Post-Norm, and Sandwich Placement — Norm-everywhere: Pre-Norm + Post-Norm per block (sandwich-style).
Query-Key Normalization — RMSNorm applied to Q and K before the attention dot product.
Root Mean Square Layer Normalization — RMSNorm, applied as both Pre-Norm and Post-Norm per block.
Sandwich-LN — Norm-everywhere — sandwich placement applied to both attention and FFN sublayers.
Rotary Position Embedding — Two RoPE bases: 10K for local SWA layers, 1M for global layers.
The Residual Stream — Sandwich/norm-everywhere residual with normalization inside the residual branch.

Gemma 3 27B (March 2025) is Google’s open-weights dense flagship and the multimodal successor to Gemma 2 27B. It retains the Gemma family’s GeGLU FFN and sandwich-style norm placement, and adds two stability/scale choices that distinguish it from the 2024 generation:

QK-Norm. An RMSNorm applied to Q and K before the attention dot product. Replaces Gemma 2’s reliance on a tanh attention-logit soft cap for stability — though Gemma 3 keeps the soft cap in place, this is a belt-and-braces combination.
Norm-everywhere placement. RMSNorm appears both before and after each attention and FFN sub-layer (inside the residual branch). The “sandwich” pattern Gemma 2 introduced is retained and tightened.

The defining long-context move is an aggressive 5:1 interleaving of local sliding-window attention (window 4096) with global full-attention layers. Most layers see only a 4K local neighborhood; every sixth layer runs full attention. The two layer types also use different RoPE bases — local layers at 10K, global layers at 1M — letting the rare global layers extrapolate over the full 128K context while the cheap local layers stay numerically well-behaved. See the sliding-window attention entry for the family-wide pattern.

Gemma 3 is also multimodal: the released checkpoints pair a SigLIP vision encoder with the text decoder. The architecture entry above describes only the language-model component; vision details are out of scope for this knowledge base.

Sources

Export

BibTeX

@article{arxiv_2503_19786,
  title         = {Gemma 3 27B},
  author        = {Google DeepMind},
  year          = {2025},
  eprint        = {2503.19786},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2503.19786}
}

CSL JSON

{
  "id": "arxiv_2503_19786",
  "type": "article-journal",
  "title": "Gemma 3 27B",
  "author": [
    {
      "literal": "Google DeepMind"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2025
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2503.19786",
  "number": "2503.19786",
  "source": "arXiv"
}

RIS

TY  - JOUR
TI  - Gemma 3 27B
AU  - Google DeepMind
PY  - 2025
JO  - arXiv
AN  - arXiv:2503.19786
UR  - https://arxiv.org/abs/2503.19786
ER  -

Models like this

OLMo 2 13BAllen Institute for AI (AI2)58% overlap
Llama 3.1 70BMeta55% overlap
OLMo 3 32BAllen Institute for AI (AI2)46% overlap