NVIDIA · Ampere

What runs on a RTX 3060 12GB

12 GB at 360 GB/s. The largest model that fits at Q4_K_M is Mistral NeMo 12B, at roughly 29.3 tokens per second.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

6.4 GB of 10.5 GB · 61%

010 GB

Weights 4.6 GB

KV cache 1.0 GB

Runtime overhead 0.8 GB

Llama 3.1 8B at Q4_K_M leaves 4.0 GB spare on a RTX 3060 12GB. There is room to raise the context length or move up a quantisation level.

Generation43.7tok/s

Prompt processing654tok/s

Max context40Ktokens

KV per 1K tokens0GB

Every quantisation of Llama 3.1 8B on a RTX 3060 12GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	15.0 GB	16.8 GB	Won't fit	—	4.81
INT8 / W8A8	8.50	7.7 GB	9.5 GB	Fits, but tight	16K	27.3
Q8_0 (GGUF)	8.50	7.7 GB	9.5 GB	Fits, but tight	16K	27.3
FP8 (E4M3)	8.00	7.5 GB	9.3 GB	Runs comfortably	17K	28.1
Q6_K	6.56	6.1 GB	8.0 GB	Runs comfortably	28K	33.8
AWQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	34K	37.9
GPTQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	34K	37.9
MXFP4	4.25	5.4 GB	7.2 GB	Runs comfortably	34K	37.9
Q5_K_M	5.67	5.3 GB	7.1 GB	Runs comfortably	35K	38.6
Q5_K_S	5.52	5.2 GB	7.0 GB	Runs comfortably	36K	39.5
Q4_K_M	4.85	4.6 GB	6.4 GB	Runs comfortably	40K	43.7
Q4_K_S	4.58	4.4 GB	6.2 GB	Runs comfortably	42K	45.6
Q4_0	4.55	4.4 GB	6.2 GB	Runs comfortably	42K	45.9
IQ4_XS	4.25	4.1 GB	5.9 GB	Runs comfortably	44K	48.3
Q3_K_M	3.91	3.8 GB	5.7 GB	Runs comfortably	46K	51.3
IQ3_M	3.70	3.7 GB	5.5 GB	Runs comfortably	48K	53.4
IQ3_XXS	3.06	3.2 GB	5.0 GB	Runs comfortably	52K	60.9
Q2_K	2.63	2.8 GB	4.6 GB	Runs comfortably	55K	67.2
IQ2_XXS	2.06	2.3 GB	4.2 GB	Runs comfortably	59K	78.0
IQ1_M	1.75	2.1 GB	3.9 GB	Runs comfortably	61K	85.4

Q4_K_M at 8K context. Bandwidth sets the speed; capacity sets the ceiling.

Model	Params	Weights	Verdict	Max ctx	tok/s
DeepSeek-R1 671B-A37B	671B	379.0 GB	Won't fit	—	1.89
Qwen3 235B-A22B	235.1B	132.8 GB	Won't fit	—	3.07
gpt-oss 120B-A5.1B	116.8B	66.0 GB	Won't fit	—	13.6
Llama 4 Scout 109B-A17B	109B	61.7 GB	Won't fit	—	4.27
Qwen2.5 72B	72.7B	41.2 GB	Won't fit	—	1.08
Llama 3.1 70B	70.5B	40.0 GB	Won't fit	—	1.13
Mixtral 8x7B	46.7B	26.4 GB	Won't fit	—	6.79
Command R 35B	35.0B	20.1 GB	Won't fit	—	1.59
Yi-1.5 34B	34.4B	19.5 GB	Won't fit	—	2.85
Qwen3 32B	32.8B	18.6 GB	Won't fit	—	3.06
Qwen2.5-Coder 32B	32.8B	18.6 GB	Won't fit	—	3.06
DeepSeek-R1-Distill-Qwen 32B	32.8B	18.6 GB	Won't fit	—	3.06
Qwen3 30B-A3B	30.5B	17.3 GB	Won't fit	—	26.1
Gemma 3 27B	27.4B	15.6 GB	Won't fit	—	4.45
Mistral Small 24B	23.6B	13.4 GB	Won't fit	—	5.85
gpt-oss 20B-A3.6B	20.9B	11.9 GB	Won't fit	—	42.3
Qwen3 14B	14.8B	8.5 GB	Won't fit	8K	22.1
Phi-4 14B	14.7B	8.4 GB	Won't fit	6K	19.9
Mistral NeMo 12B	12.3B	7.0 GB	Runs comfortably	17K	29.3
Gemma 3 12B	12.2B	7.0 GB	Runs comfortably	38K	29.7
GLM-4 9B	9.4B	5.4 GB	Runs comfortably	109K	40.2
Qwen3 8B	8.2B	4.7 GB	Runs comfortably	35K	42.3
Llama 3.1 8B	8.0B	4.6 GB	Runs comfortably	40K	43.7
DeepSeek-R1-Distill-Llama 8B	8.0B	4.6 GB	Runs comfortably	40K	43.7
Qwen2.5 7B	7.6B	4.4 GB	Runs comfortably	97K	48.5
Mistral 7B v0.3	7.3B	4.1 GB	Runs comfortably	32K	48.4
Gemma 3 4B	4.3B	2.5 GB	Runs comfortably	128K	82.6
Qwen3 4B	4.0B	2.3 GB	Runs comfortably	53K	77.4
Llama 3.2 3B	3.2B	1.8 GB	Runs comfortably	72K	97.2
Llama 3.2 1B	1.2B	0.7 GB	Runs comfortably	128K	260