NVIDIA · Ada

What runs on a RTX 4060 Ti 16GB

16 GB at 288 GB/s. The largest model that fits at Q4_K_M is gpt-oss 20B-A3.6B, at roughly 62.2 tokens per second.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

6.4 GB of 14.2 GB · 45%

014 GB

Weights 4.6 GB

KV cache 1.0 GB

Runtime overhead 0.8 GB

Llama 3.1 8B at Q4_K_M leaves 7.8 GB spare on a RTX 4060 Ti 16GB. There is room to raise the context length or move up a quantisation level.

Generation33.6tok/s

Prompt processing1151tok/s

Max context70Ktokens

KV per 1K tokens0GB

Every quantisation of Llama 3.1 8B on a RTX 4060 Ti 16GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	15.0 GB	16.8 GB	Won't fit	—	6.73
INT8 / W8A8	8.50	7.7 GB	9.5 GB	Runs comfortably	46K	21.0
Q8_0 (GGUF)	8.50	7.7 GB	9.5 GB	Runs comfortably	46K	21.0
FP8 (E4M3)	8.00	7.5 GB	9.3 GB	Runs comfortably	48K	21.6
Q6_K	6.56	6.1 GB	8.0 GB	Runs comfortably	58K	25.9
AWQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	64K	29.1
GPTQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	64K	29.1
MXFP4	4.25	5.4 GB	7.2 GB	Runs comfortably	64K	29.1
Q5_K_M	5.67	5.3 GB	7.1 GB	Runs comfortably	65K	29.6
Q5_K_S	5.52	5.2 GB	7.0 GB	Runs comfortably	66K	30.3
Q4_K_M	4.85	4.6 GB	6.4 GB	Runs comfortably	70K	33.6
Q4_K_S	4.58	4.4 GB	6.2 GB	Runs comfortably	72K	35.1
Q4_0	4.55	4.4 GB	6.2 GB	Runs comfortably	72K	35.2
IQ4_XS	4.25	4.1 GB	5.9 GB	Runs comfortably	74K	37.1
Q3_K_M	3.91	3.8 GB	5.7 GB	Runs comfortably	77K	39.4
IQ3_M	3.70	3.7 GB	5.5 GB	Runs comfortably	78K	41.0
IQ3_XXS	3.06	3.2 GB	5.0 GB	Runs comfortably	82K	46.8
Q2_K	2.63	2.8 GB	4.6 GB	Runs comfortably	85K	51.7
IQ2_XXS	2.06	2.3 GB	4.2 GB	Runs comfortably	89K	60.0
IQ1_M	1.75	2.1 GB	3.9 GB	Runs comfortably	91K	65.7

Q4_K_M at 8K context. Bandwidth sets the speed; capacity sets the ceiling.

Model	Params	Weights	Verdict	Max ctx	tok/s
DeepSeek-R1 671B-A37B	671B	379.0 GB	Won't fit	—	1.84
Qwen3 235B-A22B	235.1B	132.8 GB	Won't fit	—	3.01
gpt-oss 120B-A5.1B	116.8B	66.0 GB	Won't fit	—	13.8
Llama 4 Scout 109B-A17B	109B	61.7 GB	Won't fit	—	4.28
Qwen2.5 72B	72.7B	41.2 GB	Won't fit	—	1.13
Llama 3.1 70B	70.5B	40.0 GB	Won't fit	—	1.17
Mixtral 8x7B	46.7B	26.4 GB	Won't fit	—	7.50
Command R 35B	35.0B	20.1 GB	Won't fit	—	1.73
Yi-1.5 34B	34.4B	19.5 GB	Won't fit	—	3.43
Qwen3 32B	32.8B	18.6 GB	Won't fit	—	3.71
Qwen2.5-Coder 32B	32.8B	18.6 GB	Won't fit	—	3.71
DeepSeek-R1-Distill-Qwen 32B	32.8B	18.6 GB	Won't fit	—	3.71
Qwen3 30B-A3B	30.5B	17.3 GB	Won't fit	—	32.0
Gemma 3 27B	27.4B	15.6 GB	Won't fit	—	5.89
Mistral Small 24B	23.6B	13.4 GB	Won't fit	—	9.09
gpt-oss 20B-A3.6B	20.9B	11.9 GB	Fits, but tight	33K	62.2
Qwen3 14B	14.8B	8.5 GB	Runs comfortably	32K	18.9
Phi-4 14B	14.7B	8.4 GB	Runs comfortably	16K	18.8
Mistral NeMo 12B	12.3B	7.0 GB	Runs comfortably	41K	22.5
Gemma 3 12B	12.2B	7.0 GB	Runs comfortably	98K	22.8
GLM-4 9B	9.4B	5.4 GB	Runs comfortably	128K	30.8
Qwen3 8B	8.2B	4.7 GB	Runs comfortably	62K	32.5
Llama 3.1 8B	8.0B	4.6 GB	Runs comfortably	70K	33.6
DeepSeek-R1-Distill-Llama 8B	8.0B	4.6 GB	Runs comfortably	70K	33.6
Qwen2.5 7B	7.6B	4.4 GB	Runs comfortably	128K	37.3
Mistral 7B v0.3	7.3B	4.1 GB	Runs comfortably	32K	37.2
Gemma 3 4B	4.3B	2.5 GB	Runs comfortably	128K	63.6
Qwen3 4B	4.0B	2.3 GB	Runs comfortably	79K	59.6
Llama 3.2 3B	3.2B	1.8 GB	Runs comfortably	106K	74.8
Llama 3.2 1B	1.2B	0.7 GB	Runs comfortably	128K	200