Guide

Choosing --max-model-len in vLLM without guessing

vLLM claims almost all your VRAM at startup. Here is how to work out the context length that will actually load.

vLLM allocates its KV cache up front. It takes gpu_memory_utilization of the card, subtracts the weights and its own overhead, and turns whatever remains into cache blocks. If the context you asked for needs more blocks than that, it refuses to start — often after several minutes of loading weights.

The arithmetic it is doing

cache_budget = vram × gpu_memory_utilization − weights − overhead
max_len      = cache_budget ÷ (kv_bytes_per_token × max_num_seqs)

Two things surprise people here. First, gpu_memory_utilization is a fraction of the whole card, including whatever your desktop is already holding — on a machine with a monitor attached, 0.90 can already be more than is free. Second, the budget is divided by concurrency: --max-num-seqs 64 at 32K context needs sixty-four caches, not one.

Worked example

Qwen3 32B AWQ on an RTX 5090:

  • 32 GB × 0.92 utilisation = 29.4 GB claimed
  • Weights at 4.25 bpw, with fp16 embeddings: about 18.4 GB
  • vLLM overhead: about 1.9 GB
  • Cache budget: roughly 9.1 GB
  • KV per token: 256 KB, so about 37K tokens total

With --max-num-seqs 4 that is roughly 9K per sequence. Setting --max-model-len 32768 and expecting four concurrent users will fail, and the error message will not explain why.

What to change, in order

  • --kv-cache-dtype fp8 — doubles the effective cache, negligible quality cost.
  • --max-num-seqs — lower it. Most self-hosted setups do not need 256 concurrent sequences and the default is generous.
  • --gpu-memory-utilization — raise it toward 0.95 on a headless machine, lower it on a desktop.
  • --enable-chunked-prefill — reduces the activation spike during long prompts.
  • --swap-space — lets preempted sequences spill to host RAM instead of failing.

Check before you launch

Set the model, quantisation and concurrency in the calculator, switch the runtime to vLLM, and read the maximum context off the panel. It applies the same arithmetic vLLM does, in about a second rather than after a five-minute load.