Rotary Position Embedding

§ 1 · Premise

Position information is not in the input

Self-attention as defined by Vaswani et al. (2017, §3.2) is permutation-equivariant: permuting the input tokens permutes the output tokens identically. The attention score between query at position $m$ and key at position $n$,

is a symmetric bilinear function of the two embeddings. It contains no information about $m$, $n$, or $n - m$. For language — where “the dog bit the man” and “the man bit the dog” share a multiset of tokens but mean different things — a positional signal must be injected somewhere.

The original Transformer added a fixed sinusoidal vector $\mathbf{p}_t$ to the token embedding $\mathbf{e}_{\text{tok}(t)}$ before the first attention layer (Vaswani et al. 2017, §3.5). BERT, GPT-2, and GPT-3 replaced the sinusoidal table with a learned matrix $P \in \mathbb{R}^{T_{\max} \times d_{\text{model}}}$ of the same shape (Devlin et al. 2018, §3.2; Radford et al. 2019). Both share the additive form

Three failures recur. First, the learned table is a hard cap. GPT-3’s $P$ matrix has shape $2048 \times 12288$, $\sim 100$ MB at fp32, and is exactly undefined at position $2049$ (Brown et al. 2020, §2.1). Sinusoidal embeddings, although mathematically defined for any $t$, in practice degrade sharply past the trained length: ALiBi Fig. 3 shows sinusoidal-trained models diverging in perplexity by $1.5\times$ training length. Second, the additive form leaks position into the entire residual stream: every token embedding carries an absolute-position fingerprint even through layers where position is irrelevant. Third, the absolute-position signal must be re-derived as a relative offset by every attention head in every layer; the network has to learn, from data, the trigonometric identity $\sin(a-b) = \sin a \cos b - \cos a \sin b$ in order to compare two positions. Shaw et al. (2018, §3.2), T5 (Raffel et al. 2019, §2.1), and ALiBi (Press et al. 2021) all sidestepped this by editing the attention score itself with a relative-position term — but at the cost of breaking the clean inner-product formulation that Flash Attention–style fused kernels rely on.

RoPE’s claim is that the relative-only attention score is a property of geometry, not something the network has to learn: rotate each query and key by an angle that scales with its absolute position, and the inner product collapses to a function of the offset alone.

§ 2 · Derivation

From the dot product to rotation

Let $\mathbf{q}_m, \mathbf{k}_n \in \mathbb{R}^{d_h}$ denote a query at position $m$ and a key at position $n$, where $d_h$ is the per-head dimension (even). The pre-softmax attention logit is $\langle \mathbf{q}_m, \mathbf{k}_n \rangle$. We seek functions $f_q, f_k$ such that the position-aware vectors $f_q(\mathbf{q}, m)$ and $f_k(\mathbf{k}, n)$ satisfy

for some function $g$ that depends on the two raw vectors and the relative offset alone. Su et al. 2021, §3.2 frame this as the defining property they want. The substitution $f_q(\mathbf{q}, m) = \mathbf{q}$, $f_k(\mathbf{k}, n) = \mathbf{k}$ — no positional information — trivially satisfies (2.1) with $g$ independent of $n - m$, which is the starting failure mode. The substitution $f_q(\mathbf{q}, m) = \mathbf{q} + \mathbf{p}_m$ — additive embedding — does not satisfy (2.1) in general, because cross terms $\mathbf{q}^\top \mathbf{p}_n$ and $\mathbf{p}_m^\top \mathbf{k}$ depend on absolute positions.

The 2-D case. Specialize to $d_h = 2$. Identify $\mathbb{R}^2$ with $\mathbb{C}$ by $\mathbf{q} = q_0 + i q_1$. Define

i.e. multiplication by a unit-modulus complex number that rotates the vector by angle $m\theta$. The Hermitian inner product is

and the real part — which is what the dot product over $\mathbb{R}^2$ recovers — is $\operatorname{Re}(\overline{\mathbf{q}} \mathbf{k}\, e^{i(n-m)\theta})$, a function of $\mathbf{q}, \mathbf{k}, n - m$ only. Property (2.1) is satisfied. Equivalently, in matrix form, the same map is the planar rotation

so $(R_m \mathbf{q})^\top (R_n \mathbf{k}) = \mathbf{q}^\top R_m^\top R_n \mathbf{k} = \mathbf{q}^\top R_{n - m} \mathbf{k}$. The dependence on $m$ and $n$ separately has cancelled.

Lifting to $d_h$ dimensions. A single 2-D rotation cannot encode more than one frequency. Su et al. (2021, §3.2.2) lift the construction by partitioning the $d_h$ coordinates of $\mathbf{q}, \mathbf{k}$ into $d_h/2$ disjoint pairs $(q_{2i}, q_{2i+1})$ for $i = 0, 1, \dots, d_h/2 - 1$, and rotating each pair by its own frequency $\theta_i$. The position-$t$ rotation is the block-diagonal matrix

Each $R_t^{(i)}$ is orthogonal, so $R_t$ is orthogonal; the product $R_m^\top R_n$ is block-diagonal with blocks $R_{n-m}^{(i)}$; the global identity $R_m^\top R_n = R_{n - m}$ holds dimension-pair by dimension-pair, and (2.1) holds in $\mathbb{R}^{d_h}$.

Frequency choice. The frequencies are

mirroring the wavelength geometric series of the sinusoidal embedding of Vaswani et al. 2017, §3.5. The pair $i = 0$ rotates at $1$ radian per step (wavelength $2\pi \approx 6.3$ tokens); the pair $i = d_h/2 - 1$ rotates at $\sim b^{-1} = 10^{-4}$ radians per step (wavelength $\sim 2\pi b \approx 6.28 \times 10^4$ tokens). Low-index pairs resolve short-range offsets; high-index pairs carry long-range phase. The choice of base $b$ is the single hyperparameter; long-context recipes (Position Interpolation Chen et al. 2023, NTK-aware scaling Peng & Quesnelle 2023, and YaRN) act entirely by reweighting these $\theta_i$.

Three structural properties follow directly from (2.4):

1. Linearity in the rotated vector. $R_t$ is a linear map. Scaling $\mathbf{q}$ by $c$ scales $R_t \mathbf{q}$ by $c$. This is what lets RoPE be applied as a post-projection mutation, independent of $W_Q, W_K$.
2. Norm preservation. $R_t$ is orthogonal, so $\|R_t \mathbf{q}\|_2 = \|\mathbf{q}\|_2$. RoPE does not change the magnitude of any pre-RoPE Q or K vector — only its phase. This matters for stability: norms set by $W_Q, W_K$ initialization survive the rotation unchanged.
3. Distance decay. Su et al. (2021, §3.4.3, Fig. 2) compute the upper bound on attention logits for two random vectors as a function of offset and show it decays roughly like $1/|n - m|^{\alpha}$ for moderate offsets — an inductive bias toward local attention, without an ALiBi-style hand-coded slope.

Crucially, RoPE does not rotate the value matrix $W_V$: only Q and K participate in (2.1). The value path remains a pure linear function of the token content.

Parameter and compute cost. RoPE introduces zero learnable parameters: $\theta_i$ is fixed by (2.5); $\cos(t\theta_i)$ and $\sin(t\theta_i)$ are deterministic. Per query or key vector the rotation is $d_h/2$ planar rotations — $2 d_h$ multiply-adds total — so the per-token cost is $O(d_h)$ FLOPs per head, $O(H d_h) = O(d_{\text{model}})$ per layer. For a Llama-2-70B attention head ($d_h = 128$) this is $\sim 256$ FMAs per token per head, negligible against the $O(d_h T)$ cost of the attention dot products themselves.

§ 3 · Reference implementation

Paired-coordinate rotation

def precompute_freqs(d_h, max_pos, base=10000.0):
    # theta_i = base^{-2i / d_h}, i = 0 ... d_h/2 - 1
    i = torch.arange(0, d_h, 2).float()                 # [d_h/2]
    theta = base ** (-i / d_h)                          # [d_h/2]
    t = torch.arange(max_pos).float()                   # [max_pos]
    freqs = torch.outer(t, theta)                       # [max_pos, d_h/2]
    return freqs.cos(), freqs.sin()                     # each [max_pos, d_h/2]

def apply_rope(x, cos, sin):
    # x:   [B, T, H, d_h]   one of Q or K, post-projection
    # cos: [T, d_h/2]       sin: [T, d_h/2]
    x_even = x[..., 0::2]                               # [B, T, H, d_h/2]
    x_odd  = x[..., 1::2]                               # [B, T, H, d_h/2]
    cos = cos[None, :, None, :]                         # broadcast over B, H
    sin = sin[None, :, None, :]
    # planar rotation per pair: (x_even, x_odd) -> (x_even*cos - x_odd*sin,
    #                                                x_even*sin + x_odd*cos)
    out_even = x_even * cos - x_odd * sin
    out_odd  = x_even * sin + x_odd * cos
    return torch.stack([out_even, out_odd], dim=-1).flatten(-2)   # [B, T, H, d_h]

# usage inside attention forward:
#   q = apply_rope(W_Q(x), cos, sin)
#   k = apply_rope(W_K(x), cos, sin)
#   v = W_V(x)                            # V is NOT rotated
#   logits = einsum("bthd,bshd->bths", q, k) / sqrt(d_h)

§ 4 · Empirical evidence

What RoPE actually buys you

RoFormer paper. Su et al. 2021 report three primary result sets. Table 1 compares RoPE to sinusoidal absolute embeddings on WMT 2014 En→De machine translation with a Transformer-Base: RoPE reaches 27.5 BLEU vs sinusoidal 27.3 BLEU (§4.1). Table 2 swaps in RoPE on a BERT-base architecture pretrained on the BookCorpus + Wikipedia mix and reports a $\sim$1-point MLM accuracy improvement over the absolute baseline at matched compute (§4.2). Table 3 fine-tunes the resulting RoFormer on GLUE; RoPE matches or narrowly beats BERT on 5 of 6 tasks. The gains are real but small; the paper’s contribution is the form of the encoding, not large absolute numbers.

Llama 1 / 2 / 3 adoption rationale. Llama 1 (Touvron et al. 2023a, §2) adopted RoPE without ablation, citing GPT-NeoX-20B (Black et al. 2022, §3.4) as the reference data point — GPT-NeoX had reported faster convergence and slightly lower validation loss vs learned absolute embeddings on the Pile. Llama 2 (Touvron et al. 2023b) retained RoPE unchanged. Llama 3 (Grattafiori et al. 2024, §3.1) kept RoPE and pushed the base $b$ to $5 \times 10^5$ to support an 128K-token context window — see also the partial-rotary and base-scaling configurations in the adoption list above.

Independent reproductions and follow-ups. Press et al. 2021, Fig. 4 (the ALiBi paper) include RoPE in their length-extrapolation benchmark: on WikiText-103 with training length 512 and evaluation lengths up to 3072, RoPE perplexity grows from 19.3 at training length to 28.1 at $6\times$ — better than learned absolute embeddings (which diverge above 36 perplexity) but worse than ALiBi (which holds near 19.7). This is the canonical “RoPE doesn’t extrapolate cleanly” data point that motivated the subsequent extension work. Chen et al. 2023, §3 (Position Interpolation) reproduce the RoPE perplexity blowup at $4\times$ training length on Llama-7B and recover it by rescaling positions $t \to t \cdot L_{\text{train}} / L_{\text{eval}}$ before applying (2.5). YaRN (Peng et al. 2023, Table 2) improves on this by reweighting $\theta_i$ frequency-by-frequency, taking Llama-2-7B from 5.4 perplexity at 8K context to 3.8 at 64K — a public, reproduced result that defines the practical ceiling of RoPE-based length extension as of 2024.

Length-extrapolation cliff. Across all three independent studies, the qualitative finding is consistent: without intervention, a RoPE model trained on context length $L$ loses calibration on context length $> 1.5 L$ to $2 L$, with perplexity climbing monotonically. The fast-rotating dimensions $i \approx 0$ wrap around the unit circle many times within the training window and generalize; the slow-rotating dimensions $i \approx d_h/2$ never complete a full rotation within the training window and have no signal beyond it. This frequency-ladder asymmetry is the lever that Position Interpolation, NTK-aware scaling, and YaRN all pull on.

@article{arxiv2104_09864,
  title  = {RoFormer: Enhanced Transformer with Rotary Position Embedding},
  author = {Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu},
  year   = {2021},
  eprint = {2104.09864},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2104.09864}
}

@article{arxiv_2104_09864,
  title         = {RoFormer: Enhanced Transformer with Rotary Position Embedding},
  author        = {Jianlin Su and Yu Lu and Shengfeng Pan and Ahmed Murtadha and Bo Wen and Yunfeng Liu},
  year          = {2021},
  eprint        = {2104.09864},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2104.09864}
}

{
  "id": "arxiv_2104_09864",
  "type": "article-journal",
  "title": "RoFormer: Enhanced Transformer with Rotary Position Embedding",
  "author": [
    {
      "literal": "Jianlin Su"
    },
    {
      "literal": "Yu Lu"
    },
    {
      "literal": "Shengfeng Pan"
    },
    {
      "literal": "Ahmed Murtadha"
    },
    {
      "literal": "Bo Wen"
    },
    {
      "literal": "Yunfeng Liu"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2021
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/2104.09864",
  "number": "2104.09864",
  "source": "arXiv"
}

TY  - JOUR
TI  - RoFormer: Enhanced Transformer with Rotary Position Embedding
AU  - Jianlin Su
AU  - Yu Lu
AU  - Shengfeng Pan
AU  - Ahmed Murtadha
AU  - Bo Wen
AU  - Yunfeng Liu
PY  - 2021
JO  - arXiv
AN  - arXiv:2104.09864
UR  - https://arxiv.org/abs/2104.09864
ER  -