Llama · 3.2B parameters

Llama 3.2 3B VRAM requirements

Llama 3.2 3B has 28 layers and uses grouped-query attention (8 KV heads). At Q4_K_M the weights come to 1.8 GB, and the best quantisation that fits a 24 GB card is FP16 / BF16.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

3.5 GB of 21.8 GB · 16%

022 GB

Weights 1.8 GB

KV cache 0.9 GB

Runtime overhead 0.8 GB

Llama 3.2 3B at Q4_K_M leaves 18.2 GB spare on a RTX 4090. There is room to raise the context length or move up a quantisation level.

Generation254tok/s

Prompt processing10794tok/s

Max context128Ktokens

KV per 1K tokens0GB

Every quantisation of Llama 3.2 3B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	6.0 GB	7.7 GB	Runs comfortably	128K	92.7
INT8 / W8A8	8.50	3.1 GB	4.8 GB	Runs comfortably	128K	167
Q8_0 (GGUF)	8.50	3.1 GB	4.8 GB	Runs comfortably	128K	167
FP8 (E4M3)	8.00	3.0 GB	4.7 GB	Runs comfortably	128K	171
Q6_K	6.56	2.5 GB	4.1 GB	Runs comfortably	128K	202
AWQ 4-bit	4.25	2.1 GB	3.8 GB	Runs comfortably	128K	227
GPTQ 4-bit	4.25	2.1 GB	3.8 GB	Runs comfortably	128K	227
MXFP4	4.25	2.1 GB	3.8 GB	Runs comfortably	128K	227
Q5_K_M	5.67	2.1 GB	3.8 GB	Runs comfortably	128K	227
Q5_K_S	5.52	2.1 GB	3.7 GB	Runs comfortably	128K	232
Q4_K_M	4.85	1.8 GB	3.5 GB	Runs comfortably	128K	254
Q4_K_S	4.58	1.8 GB	3.4 GB	Runs comfortably	128K	264
Q4_0	4.55	1.7 GB	3.4 GB	Runs comfortably	128K	265
IQ4_XS	4.25	1.6 GB	3.3 GB	Runs comfortably	128K	277
Q3_K_M	3.91	1.5 GB	3.2 GB	Runs comfortably	128K	292
IQ3_M	3.70	1.5 GB	3.1 GB	Runs comfortably	128K	302
IQ3_XXS	3.06	1.3 GB	2.9 GB	Runs comfortably	128K	337
Q2_K	2.63	1.1 GB	2.8 GB	Runs comfortably	128K	366
IQ2_XXS	2.06	0.9 GB	2.6 GB	Runs comfortably	128K	412
IQ1_M	1.75	0.8 GB	2.5 GB	Runs comfortably	128K	443

Llama 3.2 3B on each GPU

Q4_K_M weights at 8K context, single card, monitor attached.

GPU	VRAM	GB/s	Verdict	Max ctx	tok/s
H100 SXM 80GB	80	3350	Runs comfortably	128K	877
A100 80GB	80	2039	Runs comfortably	128K	512
RTX 5090	32	1792	Runs comfortably	128K	478
RTX 5080	16	960	Runs comfortably	106K	266
RTX 4090	24	1008	Runs comfortably	128K	254
RTX 5070 Ti	16	896	Runs comfortably	106K	249
RTX 3090	24	936	Runs comfortably	128K	246
Radeon RX 7900 XTX	24	960	Runs comfortably	128K	230
L40S	48	864	Runs comfortably	128K	219
RTX A6000	48	768	Runs comfortably	128K	204
RTX 3080 10GB	10	760	Runs comfortably	54K	202
Mac Studio M3 Ultra 256GB	256	819	Runs comfortably	128K	201
RTX 5070	12	672	Runs comfortably	72K	189
RTX 4080 Super	16	736	Runs comfortably	106K	188
RTX 4070 Ti Super	16	672	Runs comfortably	106K	172
Mac Studio M4 Max 128GB	128	546	Runs comfortably	128K	150
RTX 4070 Super	12	504	Runs comfortably	72K	130
RTX 4070	12	504	Runs comfortably	72K	130
RTX 5060 Ti 16GB	16	448	Runs comfortably	106K	127
Arc B580	12	456	Runs comfortably	72K	99.9
RTX 3060 12GB	12	360	Runs comfortably	72K	97.2
NVIDIA DGX Spark (GB10)	128	273	Runs comfortably	128K	78.1
Mac Mini M4 Pro 48GB	48	273	Runs comfortably	128K	76.1
RTX 4060 Ti 16GB	16	288	Runs comfortably	106K	74.8
Ryzen AI Max+ 395 128GB	128	256	Runs comfortably	128K	58.4

Architecture

Parameters	3.2B
Layers	28
Hidden size	3072
Attention heads / KV heads	24 / 8
Head dimension	128
Vocabulary	128,256
Trained context	128K
KV cache per 1K tokens	0 GB
Hugging Face	meta-llama/Llama-3.2-3B-Instruct

Direct answers

Llama 3.2 3B on RTX 5090
See the verdict Llama 3.2 3B on RTX 4090
See the verdict Llama 3.2 3B on RTX 3090
See the verdict Llama 3.2 3B on RTX 5080
See the verdict Llama 3.2 3B on RTX 5070 Ti
See the verdict Llama 3.2 3B on RTX 5070
See the verdict Llama 3.2 3B on RTX 4070 Ti Super
See the verdict Llama 3.2 3B on RTX 4070 Super
See the verdict Llama 3.2 3B on RTX 4070
See the verdict