Category
Positional Encoding
How tokens learn where they are — from sinusoidal embeddings to RoPE and beyond.
- Sinusoidal Position Encoding Sinusoidal foundational
Tell self-attention where each token lives in the sequence — without a learnable position table — using a closed-form sinusoidal embedding added to the token embedding.
- Rotary Position Embedding RoPE production-adopted
Encode position by rotating Q and K vectors — so that attention's inner product depends only on the relative offset between tokens, with no separate position-embedding lookup.
- Attention with Linear Biases ALiBi research
Encode position with no learnable parameters and no rotation — just a static linear bias on attention scores. Trades expressive flexibility for clean length extrapolation.
- No Position Encoding NoPE research
Show that decoder-only transformers don't need an explicit positional encoding at all — the causal mask alone leaks enough position information for the model to recover it implicitly.
- Position Interpolation PI foundational
Extend a pretrained RoPE model's context by simply *squishing* the position values into the originally-trained range. The 2023 technique that opened the RoPE-extension research line that YaRN and LongRoPE later refined.
- NTK-Aware RoPE Scaling NTK-Aware production-adopted
Rescale the RoPE base b instead of squishing positions — preserving the fast dimensions' resolution while only the slow ones interpolate. The bridge between PI and YaRN.
- YaRN — Yet Another RoPE eXtensioN YaRN production-adopted
Extend a pretrained RoPE model's usable context far beyond the training length with only a short fine-tune, by treating fast and slow rotation bands differently.
- LongRoPE — Per-Dimension RoPE Search LongRoPE research
Extend RoPE context past what closed-form schemes like YaRN can reach by treating per-dimension rescaling factors as search variables — letting evolutionary search find non-monotone schedules that simple analytical formulas miss.
- Decoupled RoPE Decoupled RoPE production-adopted
Carry RoPE position information through an attention layer whose K, V cache has been compressed to a position-free latent (as in MLA).