01.AI on NVIDIA Ada

Can I run Yi-1.5 34B on an RTX 4090?

Not at Q4_K_M — it needs 22.2 GB against 21.8 GB available. Drop to Q4_K_S and it fits, at about 30.8 tokens per second.

Won't fit

22.2 GB of 21.8 GB · 102%
023 GB
Weights 19.5 GB
KV cache 1.9 GB
Runtime overhead 0.9 GB
Over the limit 0.5 GB

Short by 0.5 GB. You can run it with 58 of 60 layers on the RTX 4090 and the rest in system RAM, at roughly 19.7 tok/s — usable for batch work, painful for chat. A smaller quantisation or a shorter context is usually the better trade.

Generation19.7tok/s
Prompt processing1007tok/s
Max context6Ktokens
KV per 1K tokens0GB

Every quantisation of Yi-1.5 34B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

QuantisationbpwWeights TotalVerdictMax ctxtok/s
FP16 / BF16 16.00 64.1 GB 66.8 GB Won't fit — 0.80
INT8 / W8A8 8.50 33.8 GB 36.6 GB Won't fit — 2.27
Q8_0 (GGUF) 8.50 33.8 GB 36.6 GB Won't fit — 2.27
FP8 (E4M3) 8.00 32.0 GB 34.8 GB Won't fit — 2.56
Q6_K 6.56 26.3 GB 29.0 GB Won't fit — 4.27
Q5_K_M 5.67 22.7 GB 25.5 GB Won't fit — 7.36
Q5_K_S 5.52 22.1 GB 24.9 GB Won't fit — 8.13
Q4_K_M 4.85 19.5 GB 22.2 GB Won't fit 6K 19.7
Q4_K_S 4.58 18.4 GB 21.2 GB Fits, but tight 10K 30.8
Q4_0 4.55 18.3 GB 21.1 GB Fits, but tight 11K 31.0
AWQ 4-bit 4.25 18.3 GB 21.0 GB Fits, but tight 11K 31.1
GPTQ 4-bit 4.25 18.3 GB 21.0 GB Fits, but tight 11K 31.1
MXFP4 4.25 18.3 GB 21.0 GB Fits, but tight 11K 31.1
IQ4_XS 4.25 17.2 GB 19.9 GB Fits, but tight 16K 33.0
Q3_K_M 3.91 15.8 GB 18.6 GB Runs comfortably 22K 35.6
IQ3_M 3.70 15.0 GB 17.8 GB Runs comfortably 25K 37.4
IQ3_XXS 3.06 12.5 GB 15.3 GB Runs comfortably 32K 44.2
Q2_K 2.63 10.8 GB 13.6 GB Runs comfortably 32K 50.4
IQ2_XXS 2.06 8.6 GB 11.4 GB Runs comfortably 32K 61.9
IQ1_M 1.75 7.4 GB 10.2 GB Runs comfortably 32K 70.6

Also worth checking

Yi-1.5 34B on every GPU
Full hardware tableEverything that runs on a RTX 4090
Full model tableWhy the KV cache is the part that bites
Guide