Methodology

§ 1 · Scope

Decoder-only LLMs from 2017 onward

This is a knowledge base of architecture innovations in decoder-only language models. Encoder-only, encoder-decoder, vision, and multimodal architectures are out of scope. So are training recipes, inference engines, and product features — the focus is on architectural moves like RoPE, RMSNorm, GQA, MLA, and MoE routing.

§ 2 · Depth

Medium deep-dive

Each entry covers: the math, a short pseudocode implementation, the tradeoffs, at least one quantitative ablation result where available, and the disclosed production models that adopt it. Enough to understand the move and write a working sketch; not enough to replace the paper.

§ 2a · How an entry is built

What you'll find on every page

A technique entry follows the same shape so that browsing across categories is predictable:

• Header — title, category, year, the original paper as a citation.
• Premise — what problem the technique addresses and what came before.
• The move — the math and a short PyTorch-style pseudocode sketch.
• Mechanism / why it works — the analysis that motivates the design.
• Results — at least one quantitative number from the paper's ablations.
• Adopted by — the disclosed production models that use it, each with a source URL.
• Lineage — predecessor and successor techniques, when relevant.
• Summary box — the entry in one breath.

Model spec sheets follow a different shape: a fixed architectural-slot table (positional encoding · norm placement · norm type · QK-Norm · activation · attention · MoE), a brief prose summary citing the paper, and a list of techniques used (auto-generated from the adoption graph). Empty cells mean the choice was not publicly disclosed — see §3.

§ 3 · Citation discipline

The closed-model policy

We make no inferred architecture claims about closed or undocumented models. GPT-3.5, GPT-4, GPT-4o, GPT-5, Claude (any version), Gemini (any version), and similar models are listed with disclosure_level: undisclosed and all architecture fields set to null. The adoption matrix renders empty cells as a dash.

Every adopted_by entry must include a source_url linking to a paper, official tech report, model card, or recorded talk. CI enforces this; the build fails if a claim lacks a citation.

The model families we cover in detail — drawn from open or documented disclosures — are: Llama, OLMo, DeepSeek, Gemma (incl. RecurrentGemma), Kimi, MiniMax, and GLM.

§ 4 · Not sources

What does not enter the base

Rumor threads, leaked screenshots, podcast speculation, and unconfirmed tweets never enter the knowledge base, regardless of how plausible they seem. This is a knowledge base, not a rumor mill.

§ 5 · Updates

How entries stay current

Each entry carries a last_verified date. We periodically re-check adoption claims against the cited source. Frontier releases are tracked via arXiv, official lab blogs, and Hugging Face model cards.

Source code is on GitHub. Corrections welcome via PR; see the entry template if you want to contribute a new technique entry.

§ 6 · Cite

Cite this knowledge base

If you reference this site in a paper or post, please use the citation below. Each individual technique and model entry also exposes BibTeX/CSL JSON/RIS exports for the underlying paper.

Zeman, L. (2026). LLM Architecture Knowledge Base. https://lizeman.github.io/llm-arch-kb/

DOI registration pending (Zenodo).

@misc{zeman2026,
  title         = {LLM Architecture Knowledge Base},
  author        = {Zeman and Li},
  year          = {2026},
  url           = {https://lizeman.github.io/llm-arch-kb/}
}

{
  "id": "zeman2026",
  "type": "webpage",
  "title": "LLM Architecture Knowledge Base",
  "author": [
    {
      "literal": "Zeman"
    },
    {
      "literal": "Li"
    }
  ],
  "issued": {
    "date-parts": [
      [
        2026
      ]
    ]
  },
  "URL": "https://lizeman.github.io/llm-arch-kb/"
}

TY  - ELEC
TI  - LLM Architecture Knowledge Base
AU  - Zeman
AU  - Li
PY  - 2026
UR  - https://lizeman.github.io/llm-arch-kb/
ER  -