It is the C/C++ inference engine that most local LLM tools are built on. Ollama, LM Studio, Jan and a long list of others wrap it rather than replace it — which means the memory figures on this page apply whichever of them you use, because the same engine is doing the work underneath. Running llama.cpp directly gives you every flag it exposes; running a wrapper gives you sensible defaults and a nicer interface.

What is the difference between llama-cli and llama-server?

llama-cli runs a single prompt or an interactive session in the terminal and exits. llama-server starts an HTTP server with an OpenAI-compatible API and a built-in web UI, so other applications can call it. Both load the same GGUF file with the same memory cost — the choice is about how you want to reach the model, not about what it needs to run.

Does llama.cpp use the GPU?

Only if you build it with the right backend and then tell it how much to offload. It compiles against CUDA, ROCm, Metal, Vulkan or SYCL, and the number of layers placed on the GPU is set with -ngl. Ollama and LM Studio make this decision for you; with llama.cpp it is yours, which is the point — you can deliberately split a model that does not fit, keeping most layers on the GPU and the rest in system RAM.

Which GGUF quantization should I use?

Q4_K_M is the usual starting point and what the commands below assume: roughly a quarter of the FP16 memory for a quality loss most people cannot pick out in ordinary use. Q8_0 roughly doubles the requirement for a difference that is real but small. Going below Q4 degrades noticeably on smaller models, though it is sometimes the only way to fit a very large one.

Is llama.cpp faster than Ollama?

Marginally, and for a specific reason: Ollama is a thin layer over llama.cpp, so it inherits its speed and adds a small amount of process overhead. The real difference is control. llama.cpp exposes every flag — batch size, layer offload, cache type, threading — and llama.cpp is where new quantization formats and model architectures land first. The cost is that you manage builds, files and flags yourself.

llama.cpp model VRAM requirements

llama.cpp is the engine underneath most local LLM tools, so these numbers hold whether you run it directly or through Ollama, LM Studio or Jan — the same GGUF is loaded by the same code. Each model below carries the memory it needs at Q4 and Q8, including KV cache and overhead rather than weights alone, and the command that runs it. They span 2.5GB to 400GB at Q4.

All 30 models, smallest first

Commands assume a Q4_K_M GGUF in the working directory and a build with GPU support, and are shown as templates — an example -p prompt is omitted, since llama-cli starts an interactive session without one. The binary differs by workload: 11 of these start llama-server because they are worth leaving running behind an API, and 19 use llama-cli for one-shot work. Either binary loads the same file at the same memory cost.

Model	Q4 VRAM	Q8 VRAM	Command
Meta Llama 3.2 3B Instruct 3.21B · General	2.5GB	4.2GB	`./llama-cli -m Llama-3.2-3B-Instruct-Q4_K_M.gguf -c 8192`
Qwen 2.5 Coder 7B Instruct 7.6B · Coding	5.5GB	9.5GB	`./llama-cli -m Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -c 8192`
DeepSeek R1 Distill Llama 8B 8.0B · Reasoning	5.8GB	9.8GB	`./llama-cli -m DeepSeek-R1-Distill-Llama-8B-Q4_K_M.gguf -c 8192`
Llama 3.1 8B Instruct 8.03B · General	6GB	10GB	`./llama-cli -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 8192`
Qwen3 8B 8.2B · General	6.4GB	10.7GB	`./llama-server -m Qwen3-8B-Q4_K_M.gguf -c 16384`
Google Gemma 2 9B 9.24B · General	7GB	11.5GB	`./llama-cli -m gemma-2-9b-it-Q4_K_M.gguf -c 8192`
Gemma 4 12B 12B · General	8.7GB	14.7GB	`./llama-server -m gemma-4-12B-it-Q4_K_M.gguf -c 16384`
Mistral NeMo 12B Instruct 12.2B · General	9GB	15GB	`./llama-cli -m Mistral-Nemo-Instruct-2407-Q4_K_M.gguf -c 8192`
DeepSeek R1 Distill Qwen 14B 14.8B · Reasoning	10.5GB	17.5GB	`./llama-cli -m DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf -c 8192`
Qwen 2.5 Coder 14B Instruct 14.7B · Coding	10.5GB	17GB	`./llama-cli -m Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf -c 8192`
Microsoft Phi-4 14B 14.7B · Reasoning	10.5GB	17.5GB	`./llama-cli -m phi-4-Q4_K_M.gguf -c 8192`
Mistral Codestral 22B 22.2B · Coding	15GB	26GB	`./llama-cli -m Codestral-22B-v0.1-Q4_K_M.gguf -c 8192`
OpenAI gpt-oss-20b 20.9B (3.6B active) · Reasoning	15.3GB	24.2GB	`./llama-server -m gpt-oss-20b-Q4_K_M.gguf -c 16384`
Mistral Small 3 24B 24.0B · General	16GB	28GB	`./llama-cli -m Mistral-Small-24B-Instruct-2501-Q4_K_M.gguf -c 8192`
Gemma 4 26B-A4B (MoE) 26.5B (3.8B active) · General	17.4GB	30.1GB	`./llama-server -m gemma-4-26B-A4B-Q4_K_M.gguf -c 16384`
Google Gemma 2 27B 27.2B · General	18GB	32GB	`./llama-cli -m gemma-2-27b-it-Q4_K_M.gguf -c 8192`
Qwen3.8 27B 27.8B · General	18.2GB	31.5GB	`./llama-server -m Qwen3.8-27B-Q4_K_M.gguf -c 16384`
Qwen3-Coder 30B-A3B Instruct 30.5B (3.3B active) · Coding	19.8GB	34.3GB	`./llama-server -m Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf -c 16384`
Gemma 4 31B 31.3B · General	20.3GB	35.2GB	`./llama-server -m gemma-4-31B-it-Q4_K_M.gguf -c 16384`
DeepSeek R1 Distill Qwen 32B 32.5B · Reasoning	21.5GB	38GB	`./llama-cli -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf -c 8192`
Qwen 2.5 Coder 32B Instruct 32.5B · Coding	21.5GB	38GB	`./llama-cli -m Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -c 16384`
Cohere Command R 35B 35.0B · RAG	23GB	40GB	`./llama-cli -m c4ai-command-r-v01-Q4_K_M.gguf -c 16384`
Qwen3.6 35B-A3B (MoE) 36B (3B active) · General	23.1GB	40.2GB	`./llama-server -m Qwen3.6-35B-A3B-Q4_K_M.gguf -c 16384`
Llama 3.3 70B Instruct 70.6B · General	42GB	78GB	`./llama-cli -m Llama-3.3-70B-Instruct-Q4_K_M.gguf -c 8192`
DeepSeek R1 Distill Llama 70B 70.6B · Reasoning	45GB	80GB	`./llama-cli -m DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf -c 8192`
Qwen 2.5 72B Instruct 72.7B · General	45GB	82GB	`./llama-cli -m Qwen2.5-72B-Instruct-Q4_K_M.gguf -c 8192`
OpenAI gpt-oss-120b 116.8B (5.1B active) · Reasoning	66.5GB	125.8GB	`./llama-server -m gpt-oss-120b-Q4_K_M.gguf -c 16384`
Llama 4 Scout (17B active, 16 experts) 109B (17B active) · General	66.9GB	117.5GB	`./llama-server -m Llama-4-Scout-17B-16E-Instruct-Q4_K_M.gguf -c 16384`
DeepSeek V4-Flash 284B (13B active) · Reasoning	171.9GB	303GB	`./llama-server -m DeepSeek-V4-Flash-Q4_K_M.gguf -c 16384`
DeepSeek V3 671B MoE 671B (37B active) · General	400GB	740GB	`./llama-cli -m DeepSeek-V3-Q4_K_M.gguf -c 8192`

Figures are the minimum to load at that quantization. The -c flag in these commands sets context length, and it is not free — the KV cache grows with it. The VRAM calculator splits the requirement into weights, KV cache and overhead so you can see what a longer context actually costs, and the GGUF size calculator gives the file size at every quantization level before you download one.

Running it as a server

llama-server exposes an OpenAI-compatible HTTP API and a small built-in web UI, which is what makes llama.cpp usable as a backend rather than a terminal tool. Anything that speaks the OpenAI chat-completions format can point at it — including Open WebUI, if you want a full chat interface in front of it.

The flag that matters most is -ngl, which sets how many layers go on the GPU. llama.cpp does not decide this for you the way Ollama does. Set it too high and the load fails on allocation; set it too low and you leave the GPU idle while the CPU does work it did not need to. Starting from the table above — pick a model that fits your card at Q4 and offload everything — avoids the guesswork entirely.

There are Python bindings as well, llama-cpp-python, which wrap the same engine and are the usual route when you are embedding a local model in an application rather than talking to one.

What fits your GPU

Start from the memory you have. Each tier scores every model against what a card that size realistically gives you once the display and the framework have taken their share.

llama.cpp is the reason a model larger than your card still runs at all — it splits layers between GPU and system RAM rather than refusing. That works, and the CPU-resident layers run at system memory bandwidth, which is where the slowdown comes from. The hardware matrix profiles real machines against every model if you want to start from a device instead.

llama.cpp or a wrapper?

llama.cpp

The engine. Total control, manual setup, newest features first.

Ollama

The wrapper. Good defaults, model registry, an API, no flags to learn.

llama.cpp vs Ollama Ollama models & VRAM vLLM models & VRAM

llama.cpp only reads GGUF, which is why so many model pages list two download formats. GGUF vs Safetensors explains which one you actually need.

Common questions

What is llama.cpp?: It is the C/C++ inference engine that most local LLM tools are built on. Ollama, LM Studio, Jan and a long list of others wrap it rather than replace it — which means the memory figures on this page apply whichever of them you use, because the same engine is doing the work underneath. Running llama.cpp directly gives you every flag it exposes; running a wrapper gives you sensible defaults and a nicer interface.
What is the difference between llama-cli and llama-server?: llama-cli runs a single prompt or an interactive session in the terminal and exits. llama-server starts an HTTP server with an OpenAI-compatible API and a built-in web UI, so other applications can call it. Both load the same GGUF file with the same memory cost — the choice is about how you want to reach the model, not about what it needs to run.
Does llama.cpp use the GPU?: Only if you build it with the right backend and then tell it how much to offload. It compiles against CUDA, ROCm, Metal, Vulkan or SYCL, and the number of layers placed on the GPU is set with -ngl. Ollama and LM Studio make this decision for you; with llama.cpp it is yours, which is the point — you can deliberately split a model that does not fit, keeping most layers on the GPU and the rest in system RAM.
Which GGUF quantization should I use?: Q4_K_M is the usual starting point and what the commands below assume: roughly a quarter of the FP16 memory for a quality loss most people cannot pick out in ordinary use. Q8_0 roughly doubles the requirement for a difference that is real but small. Going below Q4 degrades noticeably on smaller models, though it is sometimes the only way to fit a very large one.
Is llama.cpp faster than Ollama?: Marginally, and for a specific reason: Ollama is a thin layer over llama.cpp, so it inherits its speed and adds a small amount of process overhead. The real difference is control. llama.cpp exposes every flag — batch size, layer offload, cache type, threading — and llama.cpp is where new quantization formats and model architectures land first. The cost is that you manage builds, files and flags yourself.

Guides that go deeper

Choosing a runner, what each quantization costs, and how memory is sized.