Facets Long context Tools to extend the usable sequence length. 15 techniques tagged with this facet. Positional Encoding Attention with Linear Biases ALiBi 2021-08 Bias attention scores by a per-head linear function of the query-key distance. No learnable position parameters; extrapolates beyond training length without any fine-tune. Lost the dominance race to RoPE for dense decoders but is mechanically illuminating. Position Interpolation PI 2023-06 Divide position values by the extension factor s before applying RoPE. Position t becomes t/s; trained rotation angles never extrapolate. Simple, parameter-free, works — but loses resolution uniformly across all frequency bands. NTK-Aware RoPE Scaling NTK-Aware 2023-07 Multiply RoPE's base b by s^(d_h/(d_h-2)) for extension factor s. Fast dimensions (low index) are nearly untouched; slow dimensions get linearly interpolated. Zero fine-tuning needed at small extensions; the inspiration for YaRN's per-band approach. YaRN — Yet Another RoPE eXtensioN YaRN 2023-08 Treat RoPE's rotation bands as three regimes — preserve the fast ones, linearly interpolate the slow ones — and rescale the softmax temperature. The 2023 long-context extension recipe of choice. LongRoPE — Per-Dimension RoPE Search LongRoPE 2024-02 Use evolutionary search to find per-dimension RoPE rescaling factors. Generalizes YaRN's closed-form frequency-band recipe to arbitrary non-monotone schedules. Demonstrated 2M+ context extension on Llama-2 with a short fine-tune. Attention Mechanisms Sparse Transformer Sparse Transformer 2019-04 Factorize attention into two structured sparse patterns: strided (local window) and fixed (long-range hubs). Total cost O(T·√T) per layer; full reachability in two hops. Foundational for sparse-attention research; superseded by SWA, BigBird, MoE attention in production. Sliding Window Attention SWA 2020-04 Restrict each query to the last W keys. Per-layer compute becomes O(W) per token; stacked layers still cover the full context because the receptive field grows linearly with depth. BigBird BigBird 2020-07 Each query attends to three sets of keys: a random sample, a local window, and a fixed global set. O(N) attention; provably universal-approximator and Turing-complete. The 2020 successor to Sparse Transformer that put sparse attention on a theoretical footing. DeepSeek Sparse Attention DSA 2025-12 A two-stage attention: a fast Lightning Indexer scores how relevant each historical key is to the current query, then full attention runs only over the top-K keys. Built on top of MLA — the latent KV cache is unchanged; the sparsity is in which cached entries get attended. Long Context Compressive Transformer Compressive 2019-11 Two-tier memory: a short-term FIFO cache plus a long-term compressed memory built by applying a learned compression function (mean pooling, conv, or attention) to evicted segments. The original long-context recipe that traded resolution for length. Memorizing Transformers Memorizing 2022-03 Augment one mid-stack attention layer with a kNN lookup over a non-differentiable bank of past K, V pairs. Retrieved keys/values are merged into the layer's attention. Conceptually adjacent to RAG; mechanically a forerunner. Landmark Attention Landmark 2023-05 Insert a learnable landmark token at the end of each chunk; train it to summarize the chunk's content. At inference, queries attend to landmarks to gate which chunks get full attention. Hierarchical chunked attention with O(C + K·W) cost where C is the chunk count, K the chunks selected, and W per-chunk size. LongNet — Dilated Attention LongNet 2023-07 Each attention head operates at a different dilation rate r ∈ {1, 2, 4, ...}: stride-r local windows. Combining log₂(N) such heads gives O(N · log N) total cost with full reachability across the sequence. Demonstrated training on 1B-token sequences. StreamingLLM and Attention Sinks Attention Sinks 2023-09 The first 1-4 tokens of any pretrained decoder act as attention sinks — they absorb the softmax mass that has nowhere else to go. Pin them in the KV cache and you can slide the rest of the window over arbitrarily long input without quality collapse. Activation Beacon Activation Beacon 2024-01 Insert learned 'beacon' tokens that summarize past activation windows. The model attends to beacons instead of the full long-context activations; 100× context extension with a tiny fine-tune of just the beacon parameters.