Gaussian Error Linear Unit

§ 1 · Premise

What ReLU pays for its simplicity

The 2017 transformer’s FFN was a two-layer MLP with a ReLU between (Vaswani et al. 2017, §3.3). ReLU — $\max(0, x)$ — is a single comparison and a select, which is why the entire 2012–2016 generation of vision and language models used it almost without exception. The cost of that simplicity becomes visible at three places.

The kink at zero. ReLU is not differentiable at $x = 0$; the subgradient takes any value in $[0, 1]$. Optimization-theory analyses have to add an “almost-everywhere differentiable” caveat. In practice the kink is harmless, but it forces every smoothness-based regularization argument to add an extra step.

The dead-neuron regime. Once a hidden unit’s pre-activation $z$ has been driven negative for every training example, its output is permanently zero and its gradient is zero. The unit contributes nothing to the forward pass and cannot recover via gradient descent. Glorot, Bordes & Bengio (2011, “Deep Sparse Rectifier Neural Networks”, §3) documented this failure mode at the same time they introduced ReLU; subsequent work used leaky variants (Maas et al. 2013; He et al. 2015, PReLU; Clevert et al. 2015, ELU) to soften the negative half.

The stochastic-regularization disconnect. Dropout (Srivastava et al. 2014) multiplies each activation by an independent Bernoulli mask; zoneout, shake-shake, and other regularizers extend the idea. ReLU’s deterministic gate has no obvious continuous-relaxation analogue. Adaptive dropout (Ba & Frey 2013) explicitly conditions the Bernoulli rate on the activation, suggesting a deterministic activation that already encodes “how likely is this unit to fire” would line up better with the dropout family.

Hendrycks & Gimpel (2016) build the activation directly from this last observation. The contribution: replace the deterministic gate $\mathbf{1}\{x > 0\}$ with the expected value of a stochastic gate driven by $x$ itself, where the gating variable is the cumulative probability of a standard normal — and discover empirically that the resulting smooth activation outperforms ReLU at the same compute cost.

§ 2 · Derivation

From a stochastic gate to a deterministic activation

Start from the regularization perspective: imagine multiplying each pre-activation $x$ by a Bernoulli mask $m \in \{0, 1\}$ whose rate depends on $x$. The natural choice for “large $x$ likely to be kept, negative $x$ likely to be dropped” is

where $\Phi$ is the standard normal cumulative distribution function. Hendrycks & Gimpel (2016, §2) call this the “GELU” gating: the unit fires with probability equal to the chance that a standard normal draw lands below the current input. The connection to adaptive dropout (Ba & Frey 2013) is explicit — both condition the keep-rate on the activation.

The deterministic activation is the expectation of the stochastic mask applied to $x$:

This is the canonical GELU. The single-symbol $\Phi$ hides the structure; expanding the CDF gives

Reading the shape. $\mathrm{erf}(z)$ is the integral of a Gaussian density; it is odd-symmetric, saturates at $\pm 1$, and is roughly linear in a band around zero. Two regimes follow.

• Large positive $x$: $\Phi(x) \to 1$, so $\mathrm{GELU}(x) \to x$. The activation matches identity for inputs well above zero, just like ReLU.
• Large negative $x$: $\Phi(x) \to 0$, so $\mathrm{GELU}(x) \to 0$. The activation matches zero for inputs well below zero, just like ReLU.
• Near zero: $\Phi(x) \approx \tfrac{1}{2} + \tfrac{x}{\sqrt{2\pi}}$ to first order, so $\mathrm{GELU}(x) \approx \tfrac{x}{2} + \tfrac{x^2}{\sqrt{2\pi}}$ — the kink is replaced by a smooth quadratic transition.

The non-monotonic dip. A subtle but important feature is that GELU is not monotonic. For $x$ slightly negative, the activation dips below zero, reaching a minimum near $x \approx -0.75$ with value $\approx -0.17$, before climbing back to zero. The dip exists because $x \cdot \Phi(x)$ is the product of a negative number with a small-but-positive probability; for very negative $x$, $\Phi(x)$ kills the term faster than $|x|$ grows. Swish (Ramachandran, Zoph & Le 2017, arXiv 1710.05941) shares this non-monotonic feature; Mish (Misra 2019) is a close relative. ReLU, ELU, and Leaky ReLU are all monotonic and therefore lack this property. Whether the dip helps or hurts is unsettled; what is empirically clear is that smooth + slight non-monotonicity outperforms hard-zero ReLU at transformer scale.

The derivative. A closed-form gradient is required for autograd. Differentiating $x \cdot \Phi(x)$ via the product rule:

where $\phi(x) = \tfrac{1}{\sqrt{2\pi}} e^{-x^2/2}$ is the standard normal density. The derivative is bounded, smooth, and nowhere zero — no dead-unit failure mode. Compared to ReLU’s $\mathbf{1}\{x > 0\}$ this passes a small gradient even for negative inputs (the $x \cdot \phi(x)$ term), which is the mechanism by which GELU rescues units that ReLU would leave dead.

Approximations. Computing $\mathrm{erf}$ in hardware is expensive (typically a degree-7 rational polynomial). Hendrycks & Gimpel (2016) propose two cheaper forms.

The tanh approximation:

The sigmoid approximation, which exposes the family connection to Swish:

The 1.702 constant is the temperature that minimizes max absolute error against the exact $\Phi(x)$ form over $x \in [-10, 10]$ (Hendrycks & Gimpel 2016, §2). Swish at $\beta = 1$ is $x \cdot \sigma(x)$; setting $\beta = 1.702$ recovers approximate GELU, which means GELU and Swish are essentially the same family parameterized by inner-sigmoid temperature.

Computational complexity. A single GELU evaluation needs one erf (or one tanh, or one sigmoid) plus a few adds and a multiply: $O(1)$ per scalar input, same complexity class as ReLU but with a higher constant. The exact erf form is ~5× slower than ReLU on most hardware; the sigmoid approximation closes that gap to roughly 2×. In transformer training the FFN GELU is rarely the bottleneck — matmuls dominate — so the wall-clock penalty is typically below 1%.

§ 3 · Reference implementation

Three forms

# Exact: erf-based form. PyTorch's F.gelu(x, approximate="none") matches this.
def gelu_exact(x):
    # x: [..., d]
    return x * 0.5 * (1.0 + (x / 2**0.5).erf())

# tanh approximation: GPT-2 / BERT-base shipped with this for hardware-friendly tanh.
def gelu_tanh(x):
    inner = (2 / math.pi)**0.5 * (x + 0.044715 * x.pow(3))
    return 0.5 * x * (1.0 + inner.tanh())

# Sigmoid approximation: cheapest; equivalent to Swish with beta=1.702.
def gelu_sigmoid(x):
    return x * (1.702 * x).sigmoid()

Each variant is roughly equivalent in final-model quality; the tanh form is the OpenAI default (used in BERT, GPT-2, GPT-3 reference implementations), the exact form is PyTorch’s default since 1.4. Mixed-precision training is well-behaved for all three: GELU has bounded gradient, no $\log$, no division by activation-dependent quantities.

§ 4 · Empirical evidence

What the original paper showed, and what came after

Hendrycks & Gimpel (2016, §3) benchmarked GELU against ReLU and ELU on five small tasks: MNIST classification, MNIST autoencoder reconstruction, Tweet part-of-speech tagging, TIMIT phone-frame recognition, and CIFAR-10/100 image classification. GELU edged out ReLU in 4 of 5 tasks and ELU in 3 of 5 — by margins of 0.5–1.5 percentage points (Figure 2, Figure 3). The margin was small and the tasks were small; the result by itself would not have moved the field. What did was the next two years of empirical work on transformers.

BERT (Devlin et al. 2019, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, arXiv 1810.04805, §3.1) used GELU in every FFN, citing Hendrycks & Gimpel directly. GPT-2 (Radford et al. 2019, “Language Models are Unsupervised Multitask Learners”, §2) and GPT-3 (Brown et al. 2020, “Language Models are Few-Shot Learners”, arXiv 2005.14165, §2.1) did the same. RoBERTa (Liu et al. 2019, arXiv 1907.11692) kept GELU when redoing BERT pretraining. By 2020 GELU was the default for the dense transformer FFN — not because any single ablation made an overwhelming case, but because every serious replication kept it.

The first systematic FFN-activation ablation in the transformer era was Shazeer (2020, “GLU Variants Improve Transformer”, arXiv 2002.05202, Table 1). Pretraining T5-base on the C4 corpus, the paper measured log-perplexity at matched FLOPs:

• $\mathrm{FFN}_{\mathrm{ReLU}}$: 1.997
• $\mathrm{FFN}_{\mathrm{GELU}}$: 1.983
• $\mathrm{FFN}_{\mathrm{Swish}}$: 1.994

GELU beat ReLU by ~0.014 in log-perplexity at matched compute — about 1.4% relative — and edged Swish in the ungated FFN. The story does not end there: the gated variants in the same table (GEGLU 1.942, SwiGLU 1.944) cut another 0.04 in log-perplexity. The GeGLU entry and SwiGLU entry cover those numbers; what matters here is that standalone GELU is the strongest non-gated activation Shazeer measured.

Where GELU still lives. The 2024–2026 frontier moved to gated FFNs, but GELU is the gate in GeGLU (Gemma 1/2/3 all use GeGLU; see the GeGLU entry). The activation itself is not deprecated; what is deprecated is the ungated FFN that used to wrap it. Encoder-only models that have not migrated to gated FFNs — BERT-base, RoBERTa, DeBERTa — still ship plain GELU. Decoder-only LLMs from 2017–2022 (GPT-2, GPT-3, GPT-Neo, GPT-J, OPT, BLOOM) all use plain GELU; the migration to SwiGLU starts with Llama 1 in early 2023.

Public sensitivity studies. Beyond Shazeer (2020), there is no large-scale public ablation of approximation form (exact vs tanh vs sigmoid) on transformer perplexity. Folklore holds the three are within 0.1% perplexity; this matches the small-task numbers in Hendrycks & Gimpel (2016, §3) but has not been re-measured at modern scale. Closed labs (OpenAI, Anthropic, Google) do not publish FFN-activation ablations at frontier scale, so what GPT-4 or Claude or Gemini use internally is not a question this entry can answer.

@article{arxiv1606_08415,
  title  = {Gaussian Error Linear Units (GELUs)},
  author = {Dan Hendrycks, Kevin Gimpel},
  year   = {2016},
  eprint = {1606.08415},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1606.08415}
}

@article{arxiv_1606_08415,
  title         = {Gaussian Error Linear Units (GELUs)},
  author        = {Dan Hendrycks and Kevin Gimpel},
  year          = {2016},
  eprint        = {1606.08415},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/1606.08415}
}

{
  "id": "arxiv_1606_08415",
  "type": "article-journal",
  "title": "Gaussian Error Linear Units (GELUs)",
  "author": [
    {
      "literal": "Dan Hendrycks"
    },
    {
      "literal": "Kevin Gimpel"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2016
      ]
    ]
  },
  "URL": "https://arxiv.org/abs/1606.08415",
  "number": "1606.08415",
  "source": "arXiv"
}

TY  - JOUR
TI  - Gaussian Error Linear Units (GELUs)
AU  - Dan Hendrycks
AU  - Kevin Gimpel
PY  - 2016
JO  - arXiv
AN  - arXiv:1606.08415
UR  - https://arxiv.org/abs/1606.08415
ER  -