Production model
Command R+ 104B
Architecture
| Positional encoding | RoPE |
|---|---|
| Norm placement | Pre-Norm |
| Norm type | — |
| QK-Norm | — |
| Activation | — |
| Attention | GQA |
| MoE | — |
| Other | 104B dense decoder, Multilingual: 10 primary languages explicitly trained for, Cohere release for non-commercial / research use (CC-BY-NC weights) |
Command R+ 104B (April 2024) is Cohere’s open-weights flagship — a 104B dense decoder with 128K-token context, released for research and non-commercial use under CC-BY-NC. It was positioned as Cohere’s frontier model for retrieval-augmented generation and tool use, with particular emphasis on multilingual performance across ten primary languages.
What the public materials and the released model config files confirm about the architecture:
- GQA attention — surfaced in the released model config.
- RoPE positional encoding.
- 128K context window.
- Dense — no Mixture-of-Experts component.
What is not in the publicly disclosed materials: exact layer count, head count, FFN
expansion ratio, the precise normalization type (RMSNorm vs LayerNorm) and activation
(SwiGLU vs GeGLU vs other). These fields are left null rather than inferred from the
released model config, per the citation policy.
For Cohere’s lighter sibling release see Command R 35B; for the broader Cohere training and deployment context see the company’s developer blog.