RecurrentGemma 2B

Architecture

RecurrentGemma 2B (April 2024) is the first open model based on the Griffin architecture (Botev et al. 2024) — a hybrid of input-gated linear recurrence (RG-LRU blocks) with conventional local sliding-window attention. The motivation is inference-time efficiency at long sequences: the recurrent blocks have bounded state regardless of context length, so per-token cost stays constant where a pure-attention model’s cost grows with sequence length.

The block layout interleaves two kinds of layer:

• Griffin recurrent blocks (RG-LRU). A diagonal linear recurrence with input-dependent gating. The state at each layer is a single $d$-dim vector per token, updated by an element-wise gate computed from the current input. No softmax, no quadratic cost. Position information is implicit — the recurrence’s update rule depends on token order but not on an explicit position embedding.
• Local Sliding Window Attention. Standard scaled-dot-product attention restricted to a fixed window (2048 tokens here), with RoPE on Q and K. Reintroduces explicit attention behavior over a recent neighborhood, which the recurrence alone struggles to do.

Other components match the Gemma 1 lineage: Pre-Norm RMSNorm, GeGLU FFN, the same tokenizer and pretraining data. The 2B parameter count and 8K training context are also matched to the Gemma 1 2B baseline, so the architectural difference can be read directly from comparable evaluations.

The paper’s headline (Table 4): RecurrentGemma 2B matches Gemma 1 2B on most language benchmarks and substantially exceeds it on long-context throughput (constant inference cost vs linearly growing attention cost). The downside is harder copying / retrieval at very long ranges, where the bounded recurrent state cannot perfectly recall an arbitrary token from 2K+ tokens ago.

RecurrentGemma is a strict architectural minority within the open frontier (the Mamba family and Jamba being adjacent attempts), but it is one of the cleanest published demonstrations of a hybrid recurrent + local-attention stack at modest production scale.

Sources

• https://arxiv.org/abs/2404.07839
• https://arxiv.org/abs/2402.19427
• https://huggingface.co/google/recurrentgemma-2b

@article{arxiv_2404_07839,
  title         = {RecurrentGemma 2B},
  author        = {Google DeepMind},
  year          = {2024},
  eprint        = {2404.07839},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2404.07839}
}

{
  "id": "arxiv_2404_07839",
  "type": "article-journal",
  "title": "RecurrentGemma 2B",
  "author": [
    {
      "literal": "Google DeepMind"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2024
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2404.07839",
  "number": "2404.07839",
  "source": "arXiv"
}

TY  - JOUR
TI  - RecurrentGemma 2B
AU  - Google DeepMind
PY  - 2024
JO  - arXiv
AN  - arXiv:2404.07839
UR  - https://arxiv.org/abs/2404.07839
ER  -