Mistral · 122.6B parameters

Mistral Large 2 123B VRAM requirements

Mistral Large 2 123B has 88 layers and uses grouped-query attention (8 KV heads). At Q4_K_M the weights come to 69.3 GB.

Won't fit

73.0 GB of 21.8 GB · 335%
076 GB
Weights 69.3 GB
KV cache 2.8 GB
Runtime overhead 1.0 GB
Over the limit 51.2 GB

Short by 51.2 GB. You can run it with 22 of 88 layers on the RTX 4090 and the rest in system RAM, at roughly 0.71 tok/s — usable for batch work, painful for chat. A smaller quantisation or a shorter context is usually the better trade.

Generation0.71tok/s
Prompt processing283tok/s
Max context0tokens
KV per 1K tokens0GB

Every quantisation of Mistral Large 2 123B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

QuantisationbpwWeights TotalVerdictMax ctxtok/s
FP16 / BF16 16.00 228.4 GB 232.1 GB Won't fit — 0.18
INT8 / W8A8 8.50 121.1 GB 124.8 GB Won't fit — 0.36
Q8_0 (GGUF) 8.50 121.1 GB 124.8 GB Won't fit — 0.36
FP8 (E4M3) 8.00 114.2 GB 117.9 GB Won't fit — 0.38
Q6_K 6.56 93.6 GB 97.3 GB Won't fit — 0.49
Q5_K_M 5.67 80.9 GB 84.6 GB Won't fit — 0.58
Q5_K_S 5.52 78.8 GB 82.5 GB Won't fit — 0.61
Q4_K_M 4.85 69.3 GB 73.0 GB Won't fit — 0.71
Q4_K_S 4.58 65.5 GB 69.2 GB Won't fit — 0.77
Q4_0 4.55 65.0 GB 68.7 GB Won't fit — 0.77
AWQ 4-bit 4.25 61.8 GB 65.5 GB Won't fit — 0.83
GPTQ 4-bit 4.25 61.8 GB 65.5 GB Won't fit — 0.83
MXFP4 4.25 61.8 GB 65.5 GB Won't fit — 0.83
IQ4_XS 4.25 60.8 GB 64.5 GB Won't fit — 0.85
Q3_K_M 3.91 56.0 GB 59.7 GB Won't fit — 0.95
IQ3_M 3.70 53.0 GB 56.7 GB Won't fit — 1.02
IQ3_XXS 3.06 43.9 GB 47.6 GB Won't fit — 1.37
Q2_K 2.63 37.8 GB 41.5 GB Won't fit — 1.77
IQ2_XXS 2.06 29.7 GB 33.4 GB Won't fit — 2.82
IQ1_M 1.75 25.3 GB 29.0 GB Won't fit — 4.21

Mistral Large 2 123B on each GPU

Q4_K_M weights at 8K context, single card, monitor attached.

GPUVRAMGB/sVerdictMax ctxtok/s
H100 SXM 80GB 80 3350 Fits, but tight 12K 32.6
A100 80GB 80 2039 Fits, but tight 12K 17.9
Mac Studio M3 Ultra 256GB 256 819 Runs comfortably 128K 6.67
Mac Studio M4 Max 128GB 128 546 Runs comfortably 76K 4.96
NVIDIA DGX Spark (GB10) 128 273 Runs comfortably 76K 2.55
Ryzen AI Max+ 395 128GB 128 256 Runs comfortably 76K 1.90
RTX A6000 48 768 Won't fit — 1.21
L40S 48 864 Won't fit — 1.17
RTX 5090 32 1792 Won't fit — 0.92
Mac Mini M4 Pro 48GB 48 273 Won't fit — 0.90
RTX 3090 24 936 Won't fit — 0.74
RTX 4090 24 1008 Won't fit — 0.71
RTX 5080 16 960 Won't fit — 0.69
RTX 5070 Ti 16 896 Won't fit — 0.69
RTX 5060 Ti 16GB 16 448 Won't fit — 0.68
Radeon RX 7900 XTX 24 960 Won't fit — 0.67
RTX 5070 12 672 Won't fit — 0.65
RTX 4080 Super 16 736 Won't fit — 0.63
RTX 4070 Ti Super 16 672 Won't fit — 0.63
RTX 4060 Ti 16GB 16 288 Won't fit — 0.61
RTX 3060 12GB 12 360 Won't fit — 0.61
RTX 3080 10GB 10 760 Won't fit — 0.60
RTX 4070 Super 12 504 Won't fit — 0.59
RTX 4070 12 504 Won't fit — 0.59
Arc B580 12 456 Won't fit — 0.50

Architecture

Parameters122.6B
Layers88
Hidden size12288
Attention heads / KV heads96 / 8
Head dimension128
Vocabulary32,768
Trained context128K
KV cache per 1K tokens0 GB
Hugging Facemistralai/Mistral-Large-Instruct-2407

The Mistral family

Dense models — 7B, NeMo 12B, Small 24B and Large 123B — alongside the two Mixtral mixture-of-experts releases and Codestral for code. Vocabulary size is not consistent across the family: 32k on 7B, Large and Codestral, 131k on NeMo and Small, which changes how much of a small model is embedding table.

huggingface.co/mistralai · mistral.ai · all 7 Mistral models

Mistral 7B v0.3
7.3B · 32 layersMistral NeMo 12B
12.3B · 40 layersMistral Small 24B
23.6B · 40 layersMixtral 8x7B
46.7B · 12.9B active · 32 layersMixtral 8x22B
140.6B · 39B active · 56 layersCodestral 22B
22.2B · 56 layers

Direct answers

Mistral Large 2 123B on RTX 5090
32 GBMistral Large 2 123B on RTX 4090
24 GBMistral Large 2 123B on RTX 3090
24 GBMistral Large 2 123B on RTX 5080
16 GBMistral Large 2 123B on RTX 5070 Ti
16 GBMistral Large 2 123B on RTX 5070
12 GBMistral Large 2 123B on RTX 4070 Ti Super
16 GBMistral Large 2 123B on RTX 4070 Super
12 GBMistral Large 2 123B on RTX 4070
12 GB