It is an inference server built for throughput rather than for single-user chat. Its defining feature is PagedAttention, which manages the KV cache in fixed blocks the way an operating system manages virtual memory — so many requests can share the GPU without each reserving a worst-case contiguous slab. That is why vLLM serves dozens of concurrent users on hardware where a single-stream runner would serve one.

How much VRAM does vLLM need?

More than the model weights, and by design. vllm serve loads at the model's native precision — usually BF16, not a 4-bit quantization — and then claims a share of the remaining VRAM up front as a KV cache pool. The default gpu_memory_utilization is 0.90, meaning it takes 90% of the card whatever the model's size. The weight figures in the table are the floor, not the allocation you will observe.

Is vLLM better than Ollama?

They are built for different jobs. Ollama is a single-user local service: it loads a quantized model, answers you, and unloads when idle. vLLM is a production serving engine that assumes many simultaneous requests and keeps everything resident to serve them. For one person on one machine, vLLM asks for much more VRAM and returns nothing you would notice. For an application handling concurrent traffic, the batching is the entire point.

Can vLLM run quantized models?

Yes, but not GGUF in the way llama.cpp does. vLLM works with AWQ, GPTQ, FP8 and similar formats, loaded from a pre-quantized repository rather than converted on the fly. A 4-bit AWQ build lands near the Q4 figures listed on the Ollama and llama.cpp pages. What you cannot do is point vllm serve at a GGUF file and expect the same experience — support exists but is not the path the project optimises for.

Does vLLM work on Apple Silicon?

Not meaningfully. vLLM targets CUDA first, with ROCm and some accelerator backends alongside it; there is no Metal path that would make it a sensible choice on a Mac. On Apple Silicon the practical options are llama.cpp and the tools built on it, or MLX. The unified memory that makes a Mac good at holding large models does not help here, because the software will not use the GPU.

vLLM model VRAM requirements

vLLM is the one runner where the familiar Q4 numbers do not apply. vllm serve loads a model at its native precision and then reserves most of the card for its KV cache pool, so the requirement is several times what the same model costs under Ollama. The table below gives the full-precision weight cost for all 30 models — 7.5GB to 1400GB — and the section below explains the rest of the allocation.

Read this before comparing these numbers to the Ollama page

The figures here are FP16/BF16, because that is what vllm serve loads when you give it a plain Hugging Face repository. Our Ollama and llama.cpp pages quote Q4, which is what those tools default to. A model is not cheaper on one runner than another — you are looking at different precisions. Serve a pre-quantized AWQ or GPTQ build and vLLM lands near the Q4 figures instead.

All 30 models at full precision

Sorted by what vLLM loads by default. The Q4 column is there for comparison — it is what the same model costs as a pre-quantized AWQ or GPTQ build, and it is the difference between a model fitting on one card and needing three.

Model	FP16 VRAM	If quantized	Serve command
Meta Llama 3.2 3B Instruct 3.21B · General	7.5GB	2.5GB	`vllm serve meta-llama/Llama-3.2-3B-Instruct --dtype auto`
Qwen 2.5 Coder 7B Instruct 7.6B · Coding	17GB	5.5GB	`vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --dtype auto`
DeepSeek R1 Distill Llama 8B 8.0B · Reasoning	18GB	5.8GB	`vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-8B --dtype auto`
Llama 3.1 8B Instruct 8.03B · General	18GB	6GB	`vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --dtype auto`
Qwen3 8B 8.2B · General	19.4GB	6.4GB	`vllm serve Qwen/Qwen3-8B`
Google Gemma 2 9B 9.24B · General	21GB	7GB	`vllm serve google/gemma-2-9b-it --dtype auto`
Gemma 4 12B 12B · General	27GB	8.7GB	`vllm serve google/gemma-4-12B-it`
Mistral NeMo 12B Instruct 12.2B · General	27GB	9GB	`vllm serve mistralai/Mistral-Nemo-Instruct-2407 --dtype auto`
DeepSeek R1 Distill Qwen 14B 14.8B · Reasoning	32GB	10.5GB	`vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-14B --dtype auto --max-model-len 32768`
Qwen 2.5 Coder 14B Instruct 14.7B · Coding	32GB	10.5GB	`vllm serve Qwen/Qwen2.5-Coder-14B-Instruct --dtype auto`
Microsoft Phi-4 14B 14.7B · Reasoning	32GB	10.5GB	`vllm serve microsoft/phi-4 --dtype auto`
OpenAI gpt-oss-20b 20.9B (3.6B active) · Reasoning	44.8GB	15.3GB	`vllm serve openai/gpt-oss-20b`
Mistral Codestral 22B 22.2B · Coding	48GB	15GB	`vllm serve mistralai/Codestral-22B-v0.1 --dtype auto`
Mistral Small 3 24B 24.0B · General	52GB	16GB	`vllm serve mistralai/Mistral-Small-24B-Instruct-2501 --dtype auto`
Gemma 4 26B-A4B (MoE) 26.5B (3.8B active) · General	56GB	17.4GB	`vllm serve google/gemma-4-26B-A4B`
Qwen3.8 27B 27.8B · General	58.6GB	18.2GB	`vllm serve Qwen/Qwen3.8-27B`
Google Gemma 2 27B 27.2B · General	60GB	18GB	`vllm serve google/gemma-2-27b-it --dtype auto`
Qwen3-Coder 30B-A3B Instruct 30.5B (3.3B active) · Coding	64GB	19.8GB	`vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct`
Gemma 4 31B 31.3B · General	65.6GB	20.3GB	`vllm serve google/gemma-4-31B-it`
DeepSeek R1 Distill Qwen 32B 32.5B · Reasoning	70GB	21.5GB	`vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --dtype auto --max-model-len 16384`
Qwen 2.5 Coder 32B Instruct 32.5B · Coding	70GB	21.5GB	`vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --dtype auto --max-model-len 32768`
Qwen3.6 35B-A3B (MoE) 36B (3B active) · General	75GB	23.1GB	`vllm serve Qwen/Qwen3.6-35B-A3B`
Cohere Command R 35B 35.0B · RAG	75GB	23GB	`vllm serve CohereForAI/c4ai-command-r-v01 --dtype auto`
DeepSeek R1 Distill Llama 70B 70.6B · Reasoning	150GB	45GB	`vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B --tensor-parallel-size 2`
Llama 3.3 70B Instruct 70.6B · General	150GB	42GB	`vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2`
Qwen 2.5 72B Instruct 72.7B · General	155GB	45GB	`vllm serve Qwen/Qwen2.5-72B-Instruct --tensor-parallel-size 2`
Llama 4 Scout (17B active, 16 experts) 109B (17B active) · General	221GB	66.9GB	`vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct`
OpenAI gpt-oss-120b 116.8B (5.1B active) · Reasoning	236.6GB	66.5GB	`vllm serve openai/gpt-oss-120b`
DeepSeek V4-Flash 284B (13B active) · Reasoning	571GB	171.9GB	`vllm serve deepseek-ai/DeepSeek-V4-Flash --tensor-parallel-size 8`
DeepSeek V3 671B MoE 671B (37B active) · General	1400GB	400GB	`vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8`

Weight figures only. vLLM adds its KV cache pool on top — see below. The VRAM calculator breaks any single model down into weights, cache and overhead at a context length you choose.

Why vLLM reserves 90% of your card

The surprise for most people moving from Ollama is watching a 7B model take twenty-odd gigabytes. Nothing is wrong. gpu_memory_utilization defaults to 0.9, and vLLM claims that fraction of total VRAM at startup — weights first, then everything left over becomes the KV cache pool that PagedAttention hands out to concurrent requests.

It is a deliberate design choice rather than waste. A larger pool means more requests in flight at once, which is the entire reason to run vLLM instead of something lighter. But it means the allocation you observe is a function of your card, not of your model:

8GB

~7.2GB claimed

12GB

~10.8GB claimed

16GB

~14.4GB claimed

24GB

~21.6GB claimed

48GB

~43.2GB claimed

Lower it with --gpu-memory-utilization 0.8 when the GPU also drives a display, and cap context with --max-model-len when vLLM refuses to start because the model’s advertised context will not fit in what is left after the weights. Those two flags resolve most first-run failures.

Is vLLM the right runner for you?

Ollama

One user, any hardware, models bigger than your VRAM.

vLLM

Many users, NVIDIA-class GPU, whole model resident.

The short version: if one person is talking to the model, vLLM costs you a great deal of VRAM for a benefit you will not see. If an application is serving many people at once, batching is worth more than every other optimisation combined.

Ollama vs vLLM llama.cpp models & VRAM Self-hosting vs API cost

Choosing a quantized build for vLLM means choosing a format. GGUF vs AWQ covers which one each runner can actually load.

Common questions

What is vLLM?: It is an inference server built for throughput rather than for single-user chat. Its defining feature is PagedAttention, which manages the KV cache in fixed blocks the way an operating system manages virtual memory — so many requests can share the GPU without each reserving a worst-case contiguous slab. That is why vLLM serves dozens of concurrent users on hardware where a single-stream runner would serve one.
How much VRAM does vLLM need?: More than the model weights, and by design. vllm serve loads at the model's native precision — usually BF16, not a 4-bit quantization — and then claims a share of the remaining VRAM up front as a KV cache pool. The default gpu_memory_utilization is 0.90, meaning it takes 90% of the card whatever the model's size. The weight figures in the table are the floor, not the allocation you will observe.
Is vLLM better than Ollama?: They are built for different jobs. Ollama is a single-user local service: it loads a quantized model, answers you, and unloads when idle. vLLM is a production serving engine that assumes many simultaneous requests and keeps everything resident to serve them. For one person on one machine, vLLM asks for much more VRAM and returns nothing you would notice. For an application handling concurrent traffic, the batching is the entire point.
Can vLLM run quantized models?: Yes, but not GGUF in the way llama.cpp does. vLLM works with AWQ, GPTQ, FP8 and similar formats, loaded from a pre-quantized repository rather than converted on the fly. A 4-bit AWQ build lands near the Q4 figures listed on the Ollama and llama.cpp pages. What you cannot do is point vllm serve at a GGUF file and expect the same experience — support exists but is not the path the project optimises for.
Does vLLM work on Apple Silicon?: Not meaningfully. vLLM targets CUDA first, with ROCm and some accelerator backends alongside it; there is no Metal path that would make it a sensible choice on a Mac. On Apple Silicon the practical options are llama.cpp and the tools built on it, or MLX. The unified memory that makes a Mac good at holding large models does not help here, because the software will not use the GPU.

Guides that go deeper

Choosing a runner, sizing hardware for very large models, and how memory adds up.