Reference

Glossary

Quick definitions for the recurring vocabulary used across entries. Each term links to itself for easy citation. For the full treatment, follow the term into its technique page.

Attention pattern: The matrix of which queries look at which keys. Full, causal, sliding-window, sparse, and linear-attention all differ in this pattern.
Decode: The autoregressive phase of generation: one token at a time, attending to the cached prefill via the KV cache. Memory-bandwidth bound on modern accelerators.
Embedding: The lookup that turns a discrete token id into a vector in the residual stream's d_model space. Often weight-tied to the output unembedding.
Expert: In a mixture-of-experts FFN, one of N parallel sub-MLPs. A router chooses a small subset per token, leaving the rest idle for that token.
FFN: Feed-forward network. The pointwise sub-layer of a transformer block — usually a two-layer MLP (or a gated variant like SwiGLU) applied to each position independently.
Gate: A learned scalar (or vector) that multiplies a signal, controlling how much of it passes. Central to GLU-family activations and to MoE routing.
GeGLU: GLU activation variant using GELU as the gate's nonlinearity. Used in some FFN designs prior to SwiGLU's dominance.
GQA: Grouped-query attention. Heads share key/value projections in groups (between MHA and MQA), shrinking the KV cache while keeping most of MHA's quality.
Group: In GQA, a cluster of query heads that all read from the same shared key/value pair.
Head: One of the H parallel attention computations inside a multi-head attention layer; each head projects Q/K/V into a lower-rank subspace.
KV cache: Per-token key/value tensors saved during prefill so each subsequent decode step can reuse them. Size grows linearly with context and dominates memory at long sequences.
LayerNorm: Mean-and-variance normalization applied across the d_model dimension at each position, followed by a learned gain and bias.
Load-balancing loss: An auxiliary MoE loss that pushes the router to spread tokens across experts evenly, preventing collapse to a few favorites.
Logits: The pre-softmax scores produced by the model's output projection — one per vocabulary token at each position.
MHA: Multi-head attention: the original Vaswani-et-al. formulation with H independent heads, each with its own Q, K, and V projections.
MLA: Multi-head latent attention: DeepSeek's KV-compression scheme that projects K/V to a low-rank latent and decompresses on the fly, shrinking the cache further than GQA.
MLP: Multi-layer perceptron. In transformer context, a synonym for the FFN sub-layer's two linear projections separated by a nonlinearity.
MoE: Mixture-of-experts. The FFN is replaced with N parallel experts plus a router that activates only k of them per token, decoupling parameter count from FLOPs.
MQA: Multi-query attention. All heads share a single key/value pair, maximally shrinking the KV cache at some quality cost. The endpoint of the MHA→GQA→MQA spectrum.
Perplexity: Exponentiated average negative log-likelihood per token on a held-out corpus. Lower is better; the standard intrinsic LM metric.
Prefill: The first generation phase: process the prompt in parallel and populate the KV cache. Compute-bound on modern accelerators.
Projection matrix: A learned linear map. In attention, the W_Q, W_K, W_V, and W_O matrices; in FFN, the up- and down-projections.
Residual stream: The shared d_model-wide vector that flows through the block, read from and written to by each sub-layer. The standard mental model for transformer internals.
RMSNorm: Root-mean-square normalization. Drops LayerNorm's mean-centering and bias, keeping just a learned scale. Cheaper and the default in modern frontier decoders.
RoPE: Rotary positional embedding. Encodes position by rotating pairs of Q and K coordinates by a frequency-dependent angle.
Router: In MoE, the small gating network that scores experts per token and selects the top-k to activate.
Sink token: An attention sink: the first few tokens (often BOS) which receive disproportionate attention mass. Streaming-LLM keeps them in the cache to stabilize long-context decoding.
Sliding window: An attention mask that restricts each query to a local window of recent keys, capping per-token cost at the window size rather than the full context.
Softmax: The exponentiate-and-normalize operation that turns a vector of scores into a probability distribution. Used inside attention and at the output head.
SwiGLU: GLU activation variant using Swish (SiLU) as the gate. The dominant modern FFN activation in frontier dense decoders.
Token: The discrete unit of input — typically a sub-word piece from a learned BPE/Unigram vocabulary. The model's context length is measured in tokens.
Top-k routing: The MoE rule: for each token, route it to only its top-k highest-scoring experts. k is usually 1 (Switch) or 2 (most modern MoEs).