Cohere · Command R · 103.8B parameters

Command R+ 104B VRAM requirements

Command R+ 104B has 64 layers and uses grouped-query attention (8 KV heads). At Q4_K_M the weights come to 59.1 GB.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Won't fit

62.0 GB of 21.8 GB · 285%

065 GB

Weights 59.1 GB

KV cache 2.0 GB

Runtime overhead 1.0 GB

Over the limit 40.3 GB

Short by 40.3 GB. You can run it with 20 of 64 layers on the RTX 4090 and the rest in system RAM, at roughly 0.90 tok/s — usable for batch work, painful for chat. A smaller quantisation or a shorter context is usually the better trade.

Generation0.90tok/s

Prompt processing334tok/s

Max context0tokens

KV per 1K tokens0GB

Every quantisation of Command R+ 104B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	193.3 GB	196.3 GB	Won't fit	—	0.22
INT8 / W8A8	8.50	101.3 GB	104.3 GB	Won't fit	—	0.45
Q8_0 (GGUF)	8.50	101.3 GB	104.3 GB	Won't fit	—	0.45
FP8 (E4M3)	8.00	96.7 GB	99.6 GB	Won't fit	—	0.48
Q6_K	6.56	79.3 GB	82.2 GB	Won't fit	—	0.61
Q5_K_M	5.67	68.5 GB	71.5 GB	Won't fit	—	0.73
Q5_K_S	5.52	66.7 GB	69.7 GB	Won't fit	—	0.77
AWQ 4-bit	4.25	60.0 GB	62.9 GB	Won't fit	—	0.89
GPTQ 4-bit	4.25	60.0 GB	62.9 GB	Won't fit	—	0.89
MXFP4	4.25	60.0 GB	62.9 GB	Won't fit	—	0.89
Q4_K_M	4.85	59.1 GB	62.0 GB	Won't fit	—	0.90
Q4_K_S	4.58	56.0 GB	59.0 GB	Won't fit	—	0.97
Q4_0	4.55	55.7 GB	58.6 GB	Won't fit	—	0.97
IQ4_XS	4.25	52.3 GB	55.2 GB	Won't fit	—	1.08
Q3_K_M	3.91	48.4 GB	51.4 GB	Won't fit	—	1.19
IQ3_M	3.70	46.0 GB	49.0 GB	Won't fit	—	1.31
IQ3_XXS	3.06	38.8 GB	41.7 GB	Won't fit	—	1.76
Q2_K	2.63	33.9 GB	36.8 GB	Won't fit	—	2.25
IQ2_XXS	2.06	27.4 GB	30.4 GB	Won't fit	—	3.63
IQ1_M	1.75	23.9 GB	26.9 GB	Won't fit	—	5.72

Command R+ 104B on each GPU

Q4_K_M weights at 8K context, single card, monitor attached.

GPU	VRAM	GB/s	Verdict	Max ctx	tok/s
H100 SXM 80GB	80	3350	Runs comfortably	57K	38.5
A100 80GB	80	2039	Runs comfortably	57K	21.1
Mac Studio M3 Ultra 256GB	256	819	Runs comfortably	128K	7.85
Mac Studio M4 Max 128GB	128	546	Runs comfortably	128K	5.83
NVIDIA DGX Spark (GB10)	128	273	Runs comfortably	128K	3.00
Ryzen AI Max+ 395 128GB	128	256	Runs comfortably	128K	2.24
RTX A6000	48	768	Won't fit	—	1.80
L40S	48	864	Won't fit	—	1.75
RTX 5090	32	1792	Won't fit	—	1.21
Mac Mini M4 Pro 48GB	48	273	Won't fit	—	1.20
RTX 3090	24	936	Won't fit	—	0.94
RTX 4090	24	1008	Won't fit	—	0.90
Radeon RX 7900 XTX	24	960	Won't fit	—	0.85
RTX 5080	16	960	Won't fit	—	0.85
RTX 5070 Ti	16	896	Won't fit	—	0.85
RTX 5060 Ti 16GB	16	448	Won't fit	—	0.84
RTX 5070	12	672	Won't fit	—	0.79
RTX 4080 Super	16	736	Won't fit	—	0.77
RTX 4070 Ti Super	16	672	Won't fit	—	0.77
RTX 4060 Ti 16GB	16	288	Won't fit	—	0.75
RTX 3060 12GB	12	360	Won't fit	—	0.74
RTX 3080 10GB	10	760	Won't fit	—	0.73
RTX 4070 Super	12	504	Won't fit	—	0.71
RTX 4070	12	504	Won't fit	—	0.71
Arc B580	12	456	Won't fit	—	0.60

Architecture

Parameters	103.8B
Layers	64
Hidden size	12288
Attention heads / KV heads	96 / 8
Head dimension	128
Vocabulary	256,000
Trained context	128K
KV cache per 1K tokens	0 GB
Hugging Face	CohereForAI/c4ai-command-r-plus

The Cohere family

Command R 35B uses full multi-head attention rather than grouped-query, so its KV cache costs five times as much per token as Command R+ 104B. On this family it is context length, not weights, that usually runs you out of memory.

huggingface.co/CohereForAI · cohere.com · all 2 Cohere models

Command R 35B
35.0B · 40 layers

Direct answers

Command R+ 104B on RTX 5090
32 GB Command R+ 104B on RTX 4090
24 GB Command R+ 104B on RTX 3090
24 GB Command R+ 104B on RTX 5080
16 GB Command R+ 104B on RTX 5070 Ti
16 GB Command R+ 104B on RTX 5070
12 GB Command R+ 104B on RTX 4070 Ti Super
16 GB Command R+ 104B on RTX 4070 Super
12 GB Command R+ 104B on RTX 4070
12 GB