Running a local LLM

How to run a local LLM in three steps

1. Check your memory. On Windows, Task Manager → Performance → GPU shows dedicated VRAM; on a Mac the GPU shares system RAM. At 8GB, 6 of the 30 models in our directory already load.
2. Install a runner. Ollama if you want a command line and an API, LM Studio if you want a desktop app. Both are free.
3. Run a model. ollama run llama3.2:3b downloads Meta Llama 3.2 3B Instruct (2.5GB at Q4_K_M) and opens a chat. After that first download it is an offline AI model: it needs no internet connection to answer.

The four sections below take each of those decisions properly.

What you gain, and what you give up

The full argument, with the cases where each side clearly wins: Local LLM vs ChatGPT and cloud APIs.

Capacity decides what fits; bandwidth decides how fast it runs. The entry point is lower than most people expect — at 8GB, roughly 7GB of which is actually usable once the display and the framework have taken their share, 6 of the 30 models in our directory will load.

For a specific model and context length, the VRAM calculator breaks the requirement into weights, KV cache and overhead. If you are on a Mac, unified memory works differently — the Mac VRAM optimizer shows how much of your RAM the GPU can actually address. To start from a machine rather than a number, the hardware matrix profiles real devices against every model.

All four run the same models on the same hardware, so this is not a capability choice. It is a question of whether you are the one talking to the model, or something else in your stack is.

• OllamaStart here

  A service and an API. Scriptable, embeddable, terminal-first.

  Ollama models & VRAM
• LM StudioStart here

  An application. Browse, download, chat, inspect — no terminal needed.

  LM Studio models & VRAM
• llama.cpp

  The engine. Total control, manual setup, newest features first.

  llama.cpp models & VRAM
• vLLM

  Many users, NVIDIA-class GPU, whole model resident.

  vLLM models & VRAM

A runner on its own gives you a command line and an API. If you want a chat window in front of it — history, document upload, more than one user — Open WebUI is the usual addition, and it costs essentially no VRAM of its own. For the privacy side of the decision rather than the hardware side, see self-hosted AI.

3. Pick a first model

The smallest capable model in each category, with the memory it needs at Q4_K_M and the command that fetches it. Start here, find the limit, then size up deliberately.

ollama run llama3.2:3b

ollama run qwen2.5-coder:7b

ollama run deepseek-r1:8b

All 30 models, with sizes and licences, are in the models directory. To go the other way and start from your hardware, the model recommender returns a shortlist.

4. Work out whether it pays

Local hardware is a fixed cost; an API is a metered one. Which wins depends entirely on how many tokens you actually push through it — at low volume a hosted model is cheaper for years, and at high volume the card pays for itself quickly. The API pricing and ROI tool puts your own volume against current provider rates and finds the crossover.

Common questions

What is a local LLM?

It is a language model whose weights are downloaded to your own machine and run on your own CPU or GPU. Nothing is sent to a provider, there is no per-token bill and no rate limit. The trade is capability: the models you can hold in consumer memory are smaller than the hosted frontier models.

Do I need a GPU to run a local LLM?

No, but it decides what is pleasant. A small model runs on CPU and system RAM, slowly. A GPU holds the weights in much faster memory, and on Apple Silicon the GPU shares unified memory with the CPU, so a Mac with enough RAM behaves like a card of roughly 70% that size.

How much VRAM do I need?

Enough for the weights at your chosen quantization, plus the KV cache for your context length, plus roughly 1GB the display and the framework reserve before your model loads. The VRAM calculator works this out per model, and the tier pages list what fits at each common capacity.

Which local LLM should I start with?

An 8B-class model at Q4_K_M. It will load on an 8GB card, though that is a tight fit once context is allocated — 12GB is where an 8B model gets real headroom. It answers quickly enough to be usable and tells you whether the workflow suits you before you commit to bigger hardware. Move up once you have hit a limit you can name.

Do local AI models work offline?

Yes. Once the weights are downloaded, generating a response is local computation and needs no internet connection — disconnect the machine and the model still answers. The download itself needs a connection, and so does any feature that deliberately reaches outward, such as web search in a chat interface.

Is a local LLM as good as ChatGPT?

Not on the hardest reasoning and coding work — the best hosted models are larger than anything that fits on consumer hardware. Local wins on privacy, on fixed cost after the hardware, on availability without a network, and on never being rate limited or deprecated underneath you.

Choosing a runner, sizing memory, and what each quantization level costs you.

• Best Local LLM Runners: Ollama vs LM Studio vs Jan vs llama.cpp vs vLLMWhich runner suits your hardware and workflow.Read the guide
• How Much RAM and VRAM Do You Need to Run Local LLMs?Model weights, KV cache and OS overhead, tier by tier.Read the guide
• Quantization Guide: Q4_K_M vs Q8_0 vs FP8What each quant level costs you in memory and quality.Read the guide