Method

How these numbers are worked out

Every figure on this site comes from one open model. Here is what it does and where it will be wrong.

Memory

Three components are added together.

Weights. Parameters multiplied by the effective bits per weight, which includes the block scales and zero points that quantisation formats carry. Embedding and output tensors are handled separately, because GGUF k-quants store them at higher precision and AWQ and GPTQ leave them in fp16 entirely. For a model with a large vocabulary and few parameters, such as Gemma 3 1B, that difference is a third of the file.

KV cache. 2 × layers × context × kv_heads × head_dim × bytes, multiplied by the number of concurrent sequences. Models using multi-head latent attention store one compressed vector per layer instead, which is why DeepSeek-R1 has a smaller cache than a 70B model with grouped-query attention. Sliding-window models are counted layer by layer, since only some layers see the whole context.

Overhead. The CUDA or Metal context, the runtime's own buffers, the activation working set for one prefill chunk, and the logits tensor. vLLM reserves considerably more than llama.cpp.

Usable capacity is 94% of the advertised amount, less 800 MB if a monitor is attached to the card. Unified-memory systems are held to 75%, matching the default wired-memory limit on macOS.

Speed

Generating one token requires reading every active weight from memory, so single-stream decoding is bound by bandwidth rather than compute:

tokens/sec ≈ bandwidth × efficiency ÷ bytes read per token

Efficiency is the fraction of theoretical bandwidth a real kernel reaches, and it varies by architecture and runtime. It is calibrated against measured results on an RTX 5090 and an RTX 4070, then applied to other cards from their published bandwidth. Mixture-of-experts models read only their active experts, with a penalty for the gather. Prompt processing is compute-bound instead, and is estimated from 2 × active parameters FLOPs per token against about 42% of peak.

Where this will be wrong

  • Speed figures for cards other than the RTX 5090 and RTX 4070 are extrapolated from specifications, not measured. Treat them as within roughly 20%.
  • Driver version, PCIe topology, power limits and thermal throttling all move real results and none are modelled.
  • Batched serving throughput on vLLM scales better than the simple estimate here suggests.
  • Parameter counts come from model cards and are occasionally rounded by their authors.

Something wrong?

If you have measured a real number that disagrees with this site, that measurement is more valuable than the model. Send the model, quantisation, context length, runtime, GPU and tokens per second, and the calibration constants get updated.