How much VRAM does Qwen3 8B need?

At Q4_K_M the weights are 4.7 GB. With an 8K context the KV cache adds 1.1 GB and the runtime adds 0.8 GB, for 6.7 GB in total.

Qwen on NVIDIA Ada

Can I run Qwen3 8B on an L40S?

Yes. Qwen3 8B at Q4_K_M uses 6.7 GB of the 44.3 GB available on a L40S, and runs at about 96.1 tokens per second. You can push the context to 128K.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

6.7 GB of 44.3 GB · 15%

044 GB

Weights 4.7 GB

KV cache 1.1 GB

Runtime overhead 0.8 GB

Qwen3 8B at Q4_K_M leaves 37.7 GB spare on a L40S. There is room to raise the context length or move up a quantisation level.

Generation96.1tok/s

Prompt processing4641tok/s

Max context128Ktokens

KV per 1K tokens0GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	15.3 GB	17.2 GB	Runs comfortably	128K	32.5
INT8 / W8A8	8.50	7.8 GB	9.8 GB	Runs comfortably	128K	61.0
Q8_0 (GGUF)	8.50	7.8 GB	9.8 GB	Runs comfortably	128K	61.0
FP8 (E4M3)	8.00	7.6 GB	9.6 GB	Runs comfortably	128K	62.4
Q6_K	6.56	6.3 GB	8.2 GB	Runs comfortably	128K	74.8
AWQ 4-bit	4.25	5.8 GB	7.7 GB	Runs comfortably	128K	80.6
GPTQ 4-bit	4.25	5.8 GB	7.7 GB	Runs comfortably	128K	80.6
MXFP4	4.25	5.8 GB	7.7 GB	Runs comfortably	128K	80.6
Q5_K_M	5.67	5.4 GB	7.4 GB	Runs comfortably	128K	85.2
Q5_K_S	5.52	5.3 GB	7.2 GB	Runs comfortably	128K	87.3
Q4_K_M	4.85	4.7 GB	6.7 GB	Runs comfortably	128K	96.1
Q4_K_S	4.58	4.5 GB	6.4 GB	Runs comfortably	128K	100
Q4_0	4.55	4.5 GB	6.4 GB	Runs comfortably	128K	101
IQ4_XS	4.25	4.2 GB	6.2 GB	Runs comfortably	128K	106
Q3_K_M	3.91	4.0 GB	5.9 GB	Runs comfortably	128K	112
IQ3_M	3.70	3.8 GB	5.7 GB	Runs comfortably	128K	116
IQ3_XXS	3.06	3.3 GB	5.2 GB	Runs comfortably	128K	131
Q2_K	2.63	2.9 GB	4.9 GB	Runs comfortably	128K	144
IQ2_XXS	2.06	2.5 GB	4.4 GB	Runs comfortably	128K	165
IQ1_M	1.75	2.2 GB	4.2 GB	Runs comfortably	128K	180