NVIDIA · Ada

What runs on a L4

24 GB at 300 GB/s. The largest model that fits at Q4_K_M is Qwen3 32B, at roughly 9.14 tokens per second.

Runs comfortably

6.4 GB of 21.8 GB · 30%
022 GB
Weights 4.6 GB
KV cache 1.0 GB
Runtime overhead 0.8 GB

Llama 3.1 8B at Q4_K_M leaves 15.3 GB spare on a L4. There is room to raise the context length or move up a quantisation level.

Generation34.9tok/s
Prompt processing3164tok/s
Max context128Ktokens
KV per 1K tokens0GB

Every quantisation of Llama 3.1 8B on a L4

Highlighted row is the highest quality that still fits at 8K context.

QuantisationbpwWeights TotalVerdictMax ctxtok/s
FP16 / BF16 16.00 15.0 GB 16.8 GB Runs comfortably 48K 11.6
INT8 / W8A8 8.50 7.7 GB 9.5 GB Runs comfortably 106K 21.8
Q8_0 (GGUF) 8.50 7.7 GB 9.5 GB Runs comfortably 106K 21.8
FP8 (E4M3) 8.00 7.5 GB 9.3 GB Runs comfortably 108K 22.5
Q6_K 6.56 6.1 GB 8.0 GB Runs comfortably 118K 27.0
AWQ 4-bit 4.25 5.4 GB 7.2 GB Runs comfortably 124K 30.3
GPTQ 4-bit 4.25 5.4 GB 7.2 GB Runs comfortably 124K 30.3
MXFP4 4.25 5.4 GB 7.2 GB Runs comfortably 124K 30.3
Q5_K_M 5.67 5.3 GB 7.1 GB Runs comfortably 125K 30.8
Q5_K_S 5.52 5.2 GB 7.0 GB Runs comfortably 126K 31.6
Q4_K_M 4.85 4.6 GB 6.4 GB Runs comfortably 128K 34.9
Q4_K_S 4.58 4.4 GB 6.2 GB Runs comfortably 128K 36.5
Q4_0 4.55 4.4 GB 6.2 GB Runs comfortably 128K 36.7
IQ4_XS 4.25 4.1 GB 5.9 GB Runs comfortably 128K 38.6
Q3_K_M 3.91 3.8 GB 5.7 GB Runs comfortably 128K 41.0
IQ3_M 3.70 3.7 GB 5.5 GB Runs comfortably 128K 42.7
IQ3_XXS 3.06 3.2 GB 5.0 GB Runs comfortably 128K 48.7
Q2_K 2.63 2.8 GB 4.6 GB Runs comfortably 128K 53.8
IQ2_XXS 2.06 2.3 GB 4.2 GB Runs comfortably 128K 62.4
IQ1_M 1.75 2.1 GB 3.9 GB Runs comfortably 128K 68.4

Models on a L4

Q4_K_M at 8K context. Bandwidth sets the speed; capacity sets the ceiling.

ModelParamsWeightsVerdictMax ctxtok/s
DeepSeek-R1 671B-A37B 671B 379.0 GB Won't fit — 1.86
Qwen3 235B-A22B 235.1B 132.8 GB Won't fit — 3.14
gpt-oss 120B-A5.1B 116.8B 66.0 GB Won't fit — 15.3
Llama 4 Scout 109B-A17B 109B 61.7 GB Won't fit — 4.81
Qwen2.5 72B 72.7B 41.2 GB Won't fit — 1.37
Llama 3.1 70B 70.5B 40.0 GB Won't fit — 1.44
Mixtral 8x7B 46.7B 26.4 GB Won't fit — 11.2
Command R 35B 35.0B 20.1 GB Won't fit 640 2.60
Yi-1.5 34B 34.4B 19.5 GB Won't fit 6K 7.82
Qwen3 32B 32.8B 18.6 GB Fits, but tight 9K 9.14
Qwen2.5-Coder 32B 32.8B 18.6 GB Fits, but tight 9K 9.14
DeepSeek-R1-Distill-Qwen 32B 32.8B 18.6 GB Fits, but tight 9K 9.14
Qwen3 30B-A3B 30.5B 17.3 GB Runs comfortably 39K 53.4
Gemma 3 27B 27.4B 15.6 GB Runs comfortably 57K 11.0
Mistral Small 24B 23.6B 13.4 GB Runs comfortably 32K 12.8
gpt-oss 20B-A3.6B 20.9B 11.9 GB Runs comfortably 128K 64.7
Qwen3 14B 14.8B 8.5 GB Runs comfortably 80K 19.7
Phi-4 14B 14.7B 8.4 GB Runs comfortably 16K 19.6
Mistral NeMo 12B 12.3B 7.0 GB Runs comfortably 89K 23.4
Gemma 3 12B 12.2B 7.0 GB Runs comfortably 128K 23.7
GLM-4 9B 9.4B 5.4 GB Runs comfortably 128K 32.1
Qwen3 8B 8.2B 4.7 GB Runs comfortably 115K 33.8
Llama 3.1 8B 8.0B 4.6 GB Runs comfortably 128K 34.9
DeepSeek-R1-Distill-Llama 8B 8.0B 4.6 GB Runs comfortably 128K 34.9
Qwen2.5 7B 7.6B 4.4 GB Runs comfortably 128K 38.8
Mistral 7B v0.3 7.3B 4.1 GB Runs comfortably 32K 38.7
Gemma 3 4B 4.3B 2.5 GB Runs comfortably 128K 66.2
Qwen3 4B 4.0B 2.3 GB Runs comfortably 128K 62.0
Llama 3.2 3B 3.2B 1.8 GB Runs comfortably 128K 77.9
Llama 3.2 1B 1.2B 0.7 GB Runs comfortably 128K 209

Direct answers

Llama 3.1 8B on L4
8.0BLlama 3.1 70B on L4
70.5BLlama 3.2 3B on L4
3.2BLlama 3.2 1B on L4
1.2BLlama 4 Scout 109B-A17B on L4
109BQwen3 8B on L4
8.2BQwen3 14B on L4
14.8BQwen3 32B on L4
32.8B