Guide

Which quantisation should you actually use?

Q4_K_M for almost everyone, Q5 or Q6 if you have room, AWQ if you are serving with vLLM. The reasoning, briefly.

Quantisation stores each weight in fewer bits. The naming is unhelpfully cryptic, the size difference between formats is large, and the quality difference between most of them is small. Here is how to choose without reading a research paper.

The number in the name is not the size

Q4_K_M is not four bits per weight. K-quants store weights in super-blocks with their own scales and minimums, and those cost space too — the real figure is about 4.85 bits. Group-wise formats like AWQ and GPTQ come to around 4.25 bits for the transformer body, but they leave embeddings and the output layer in fp16 entirely. For a model with a 256K vocabulary that is a serious amount of memory that the advertised bit width does not mention.

What actually degrades

Between fp16 and Q6_K, the difference is hard to detect at all. Q5 is slightly softer. Q4_K_M is where most people settle, and it is genuinely fine for chat, summarisation and code. Below Q4 the losses become real: Q3 starts making arithmetic and instruction-following mistakes, and Q2 is only worth considering on very large models, where a heavily quantised 70B still beats a well-quantised 8B.

The general rule: prefer a larger model at lower precision over a smaller model at higher precision, until you reach about 3.5 bits. Below that, the ordering flips.

Pick by runtime, not by preference

  • llama.cpp and Ollama — GGUF. Q4_K_M by default, Q5_K_M or Q6_K if you have room. IQ formats are slightly better per byte but slower on CPU.
  • vLLM and SGLang — AWQ or GPTQ at 4 bits, or FP8 on Hopper and Blackwell cards where it is nearly lossless and genuinely fast.
  • Apple Silicon — MLX 4-bit, which is well tuned for unified memory.
  • ExLlamaV2 — EXL2, where you choose the bits per weight directly and can target a specific VRAM budget.

Native quantisation is different

Some models are trained or released quantised rather than converted afterwards. OpenAI's gpt-oss models ship in MXFP4, and re-quantising them upward to Q8 wastes space without recovering anything, because the precision was never there. Where a model has a native format, use it.

The practical procedure

Work out how much memory you have, subtract the KV cache for the context length you actually need, and take the largest quantisation that fits in what is left. That is the whole decision. The calculator does exactly this and highlights the answer.