How much VRAM does DeepSeek-R1-Distill-Qwen 32B need?

At Q4_K_M the weights are 18.6 GB. With an 8K context the KV cache adds 2.0 GB and the runtime adds 0.8 GB, for 21.5 GB in total.

DeepSeek on NVIDIA Ada

Can I run DeepSeek-R1-Distill-Qwen 32B on an RTX 4090?

Yes. DeepSeek-R1-Distill-Qwen 32B at Q4_K_M uses 21.5 GB of the 21.8 GB available on a RTX 4090, and runs at about 30.5 tokens per second. You can push the context to 9K.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Fits, but tight

21.5 GB of 21.8 GB · 99%

022 GB

Weights 18.6 GB

KV cache 2.0 GB

Runtime overhead 0.8 GB

This fits with almost nothing to spare. A background application claiming VRAM will push it over. Drop to the next quantisation down, or quantise the KV cache to Q8_0 — that halves the cache for no meaningful quality loss.

Generation30.5tok/s

Prompt processing1058tok/s

Max context9Ktokens

KV per 1K tokens0GB

Every quantisation of DeepSeek-R1-Distill-Qwen 32B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	61.0 GB	63.9 GB	Won't fit	—	0.85
INT8 / W8A8	8.50	32.1 GB	34.9 GB	Won't fit	—	2.52
Q8_0 (GGUF)	8.50	32.1 GB	34.9 GB	Won't fit	—	2.52
FP8 (E4M3)	8.00	30.5 GB	33.3 GB	Won't fit	—	2.83
Q6_K	6.56	25.0 GB	27.9 GB	Won't fit	—	4.94
Q5_K_M	5.67	21.6 GB	24.5 GB	Won't fit	—	9.34
Q5_K_S	5.52	21.1 GB	23.9 GB	Won't fit	—	10.4
Q4_K_M	4.85	18.6 GB	21.5 GB	Fits, but tight	9K	30.5
AWQ 4-bit	4.25	18.3 GB	21.2 GB	Fits, but tight	10K	30.9
GPTQ 4-bit	4.25	18.3 GB	21.2 GB	Fits, but tight	10K	30.9
MXFP4	4.25	18.3 GB	21.2 GB	Fits, but tight	10K	30.9
Q4_K_S	4.58	17.6 GB	20.5 GB	Fits, but tight	13K	32.0
Q4_0	4.55	17.5 GB	20.4 GB	Fits, but tight	14K	32.2
IQ4_XS	4.25	16.4 GB	19.3 GB	Runs comfortably	18K	34.2
Q3_K_M	3.91	15.2 GB	18.0 GB	Runs comfortably	23K	36.8
IQ3_M	3.70	14.4 GB	17.3 GB	Runs comfortably	26K	38.6
IQ3_XXS	3.06	12.1 GB	15.0 GB	Runs comfortably	35K	45.3
Q2_K	2.63	10.6 GB	13.4 GB	Runs comfortably	41K	51.3
IQ2_XXS	2.06	8.5 GB	11.3 GB	Runs comfortably	50K	62.2
IQ1_M	1.75	7.4 GB	10.2 GB	Runs comfortably	54K	70.4

Also worth checking

DeepSeek-R1-Distill-Qwen 32B on every GPU
Full hardware table Everything that runs on a RTX 4090
Full model table Why the KV cache is the part that bites
Guide