Category
Long Context
YaRN, LongRoPE, attention sinks, beacons, and the rest of the long-context toolbox.
- Compressive Transformer Compressive foundational
Avoid evicting old activations from a sliding-window cache — compress them into a smaller representation that can still be attended to. The 2019 conceptual ancestor of Activation Beacon and modern compression-based long-context.
- Memorizing Transformers Memorizing research
Extend a transformer's effective context to millions of tokens by adding a non-differentiable kNN memory that retrieves relevant past key/value pairs from outside the standard attention window.
- Landmark Attention Landmark research
Give attention 'random access' to a long context by summarizing each chunk with a learned 'landmark' token. Queries attend to landmarks first to decide which chunks to fully expand, then attend within only the selected chunks.
- LongNet — Dilated Attention LongNet research
Cover very long sequences via dilated attention — multiple attention heads operating at exponentially-increasing stride patterns, so the receptive field grows multiplicatively in depth and head count rather than linearly in either.
- StreamingLLM and Attention Sinks Attention Sinks production-adopted
Let a pretrained decoder serve infinite-length streaming generation without retraining — by keeping a tiny set of 'sink' tokens permanently in the KV cache.
- Activation Beacon Activation Beacon research
Extend a pretrained 4K-context model to 400K context by training a small set of 'beacon' tokens that compress past activations into a compact summary — without retraining the base model.