How much VRAM do you need for LLM inference?
VRAM is usually the first constraint when sizing an inference server. Here is a practical, no-math way to reason about it before you commit to hardware.
Three things use VRAM
- The model weights themselves, which scale with parameter count and precision.
- The KV cache, which grows with context length and the number of concurrent requests.
- Overhead from the runtime and CUDA libraries.
Precision changes everything
Running a model at lower precision (for example 8-bit or 4-bit quantisation) dramatically reduces the memory the weights need, often letting a larger model fit on a smaller GPU, with some quality trade-off.
Concurrency and context
Serving many users at once, or long contexts, inflates the KV cache. If you plan high concurrency, budget extra VRAM headroom beyond just the weights.
Rule of thumb: size for the weights first, then add headroom for your expected concurrency and context length. We help size this during workload analysis.
When to go multi-GPU
If a model will not fit on one GPU, or you need more throughput, multi-GPU servers (2×, 4×, 8×) spread the load. We specify the right configuration around your target models and traffic.
In short: Size VRAM for model weights first, then add headroom for concurrency and context; quantisation buys room. Request a custom quote