NVIDIA · Hopper

What runs on a H100 PCIe 80GB

80 GB at 2000 GB/s. The largest model that fits at Q4_K_M is gpt-oss 120B-A5.1B, at roughly 241 tokens per second.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

6.4 GB of 74.4 GB · 9%

074 GB

Weights 4.6 GB

KV cache 1.0 GB

Runtime overhead 0.8 GB

Llama 3.1 8B at Q4_K_M leaves 68.0 GB spare on a H100 PCIe 80GB. There is room to raise the context length or move up a quantisation level.

Generation260tok/s

Prompt processing19771tok/s

Max context128Ktokens

KV per 1K tokens0GB

Every quantisation of Llama 3.1 8B on a H100 PCIe 80GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	15.0 GB	16.8 GB	Runs comfortably	128K	89.1
INT8 / W8A8	8.50	7.7 GB	9.5 GB	Runs comfortably	128K	165
Q8_0 (GGUF)	8.50	7.7 GB	9.5 GB	Runs comfortably	128K	165
FP8 (E4M3)	8.00	7.5 GB	9.3 GB	Runs comfortably	128K	170
Q6_K	6.56	6.1 GB	8.0 GB	Runs comfortably	128K	203
AWQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	128K	227
GPTQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	128K	227
MXFP4	4.25	5.4 GB	7.2 GB	Runs comfortably	128K	227
Q5_K_M	5.67	5.3 GB	7.1 GB	Runs comfortably	128K	231
Q5_K_S	5.52	5.2 GB	7.0 GB	Runs comfortably	128K	236
Q4_K_M	4.85	4.6 GB	6.4 GB	Runs comfortably	128K	260
Q4_K_S	4.58	4.4 GB	6.2 GB	Runs comfortably	128K	271
Q4_0	4.55	4.4 GB	6.2 GB	Runs comfortably	128K	273
IQ4_XS	4.25	4.1 GB	5.9 GB	Runs comfortably	128K	286
Q3_K_M	3.91	3.8 GB	5.7 GB	Runs comfortably	128K	303
IQ3_M	3.70	3.7 GB	5.5 GB	Runs comfortably	128K	315
IQ3_XXS	3.06	3.2 GB	5.0 GB	Runs comfortably	128K	357
Q2_K	2.63	2.8 GB	4.6 GB	Runs comfortably	128K	391
IQ2_XXS	2.06	2.3 GB	4.2 GB	Runs comfortably	128K	450
IQ1_M	1.75	2.1 GB	3.9 GB	Runs comfortably	128K	489

Q4_K_M at 8K context. Bandwidth sets the speed; capacity sets the ceiling.

Model	Params	Weights	Verdict	Max ctx	tok/s
DeepSeek-R1 671B-A37B	671B	379.0 GB	Won't fit	—	2.52
Qwen3 235B-A22B	235.1B	132.8 GB	Won't fit	—	6.69
gpt-oss 120B-A5.1B	116.8B	66.0 GB	Fits, but tight	108K	241
Llama 4 Scout 109B-A17B	109B	61.7 GB	Runs comfortably	63K	110
Qwen2.5 72B	72.7B	41.2 GB	Runs comfortably	103K	32.5
Llama 3.1 70B	70.5B	40.0 GB	Runs comfortably	107K	33.4
Mixtral 8x7B	46.7B	26.4 GB	Runs comfortably	32K	140
Command R 35B	35.0B	20.1 GB	Runs comfortably	43K	55.1
Yi-1.5 34B	34.4B	19.5 GB	Runs comfortably	32K	66.9
Qwen3 32B	32.8B	18.6 GB	Runs comfortably	128K	69.5
Qwen2.5-Coder 32B	32.8B	18.6 GB	Runs comfortably	128K	69.5
DeepSeek-R1-Distill-Qwen 32B	32.8B	18.6 GB	Runs comfortably	128K	69.5
Qwen3 30B-A3B	30.5B	17.3 GB	Runs comfortably	128K	168
Gemma 3 27B	27.4B	15.6 GB	Runs comfortably	128K	83.2
Mistral Small 24B	23.6B	13.4 GB	Runs comfortably	32K	97.5
gpt-oss 20B-A3.6B	20.9B	11.9 GB	Runs comfortably	128K	348
Qwen3 14B	14.8B	8.5 GB	Runs comfortably	128K	149
Phi-4 14B	14.7B	8.4 GB	Runs comfortably	16K	148
Mistral NeMo 12B	12.3B	7.0 GB	Runs comfortably	128K	176
Gemma 3 12B	12.2B	7.0 GB	Runs comfortably	128K	177
GLM-4 9B	9.4B	5.4 GB	Runs comfortably	128K	238
Qwen3 8B	8.2B	4.7 GB	Runs comfortably	128K	251
Llama 3.1 8B	8.0B	4.6 GB	Runs comfortably	128K	260
DeepSeek-R1-Distill-Llama 8B	8.0B	4.6 GB	Runs comfortably	128K	260
Qwen2.5 7B	7.6B	4.4 GB	Runs comfortably	128K	289
Mistral 7B v0.3	7.3B	4.1 GB	Runs comfortably	32K	287
Gemma 3 4B	4.3B	2.5 GB	Runs comfortably	128K	472
Qwen3 4B	4.0B	2.3 GB	Runs comfortably	128K	443
Llama 3.2 3B	3.2B	1.8 GB	Runs comfortably	128K	557
Llama 3.2 1B	1.2B	0.7 GB	Runs comfortably	128K	1430