How much VRAM does GLM-4 9B need?

At Q4_K_M the weights are 5.4 GB. With an 8K context the KV cache adds 0.3 GB and the runtime adds 0.8 GB, for 6.5 GB in total.

Zhipu on NVIDIA Ada

Can I run GLM-4 9B on an L40S?

Yes. GLM-4 9B at Q4_K_M uses 6.5 GB of the 44.3 GB available on a L40S, and runs at about 91.2 tokens per second. You can push the context to 128K.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

6.5 GB of 44.3 GB · 15%

044 GB

Weights 5.4 GB

KV cache 0.3 GB

Runtime overhead 0.8 GB

GLM-4 9B at Q4_K_M leaves 37.8 GB spare on a L40S. There is room to raise the context length or move up a quantisation level.

Generation91.2tok/s

Prompt processing4044tok/s

Max context128Ktokens

KV per 1K tokens0GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	17.5 GB	18.6 GB	Runs comfortably	128K	29.1
INT8 / W8A8	8.50	9.0 GB	10.2 GB	Runs comfortably	128K	55.7
Q8_0 (GGUF)	8.50	9.0 GB	10.2 GB	Runs comfortably	128K	55.7
FP8 (E4M3)	8.00	8.8 GB	9.9 GB	Runs comfortably	128K	57.3
Q6_K	6.56	7.2 GB	8.3 GB	Runs comfortably	128K	69.5
AWQ 4-bit	4.25	6.3 GB	7.5 GB	Runs comfortably	128K	78.2
GPTQ 4-bit	4.25	6.3 GB	7.5 GB	Runs comfortably	128K	78.2
MXFP4	4.25	6.3 GB	7.5 GB	Runs comfortably	128K	78.2
Q5_K_M	5.67	6.2 GB	7.3 GB	Runs comfortably	128K	79.9
Q5_K_S	5.52	6.0 GB	7.2 GB	Runs comfortably	128K	82.0
Q4_K_M	4.85	5.4 GB	6.5 GB	Runs comfortably	128K	91.2
Q4_K_S	4.58	5.1 GB	6.3 GB	Runs comfortably	128K	95.5
Q4_0	4.55	5.1 GB	6.2 GB	Runs comfortably	128K	96.0
IQ4_XS	4.25	4.8 GB	6.0 GB	Runs comfortably	128K	101
Q3_K_M	3.91	4.5 GB	5.6 GB	Runs comfortably	128K	108
IQ3_M	3.70	4.3 GB	5.4 GB	Runs comfortably	128K	113
IQ3_XXS	3.06	3.7 GB	4.8 GB	Runs comfortably	128K	130
Q2_K	2.63	3.3 GB	4.4 GB	Runs comfortably	128K	145
IQ2_XXS	2.06	2.8 GB	3.9 GB	Runs comfortably	128K	171
IQ1_M	1.75	2.5 GB	3.6 GB	Runs comfortably	128K	189