Self-hosted AI, end to end
A private AI assistant is three pieces, not one: weights on your disk, a server that loads them, and an interface you talk to. All three are free; the hardware is the cost. This page covers how the pieces fit and what self-hosting genuinely protects — if you want the hardware walkthrough instead, start with running a local LLM.
What it actually protects
What self-hosting guarantees
- No prompt leaves the machine. Inference is local computation — pull the network cable and it still answers.
- No third-party retention policy applies to your conversations, because no third party receives them.
- The model cannot be deprecated, re-priced or altered underneath you. The weights you downloaded are the weights you keep.
- No rate limits, and no per-token cost once the hardware is paid for.
What it does not
- Secure the machine. Chat history and uploaded documents sit in a database on disk like any other application’s data.
- Cover features that reach outward. Web search, external tools and remote APIs send data out the moment you enable them.
- Make the model safe to expose. A server reachable from the internet is reachable by everyone unless you put authentication in front of it.
- Match frontier quality at every size. A model that fits a 16GB card is good; it is not the largest hosted model.
The distinction matters because “private” is often sold as a property of the model. It is a property of where the computation happens. A local model reached through someone else’s hosted front end is not private at all.
The three layers
- 1
The model
Open-weight models you download once and keep. This is the layer where licence matters more than people assume — 'open weights' is not the same as open source, and some of the best models carry conditions on commercial use.
- 2
The server
The process that loads the weights and answers requests. Ollama is the least work; llama.cpp gives you every flag; vLLM is the one to choose when several people share the machine, because it batches concurrent requests properly.
- 3
The interface
What you actually talk to. A server on its own is an API; the interface adds conversation history, document upload and user accounts. Open WebUI is the common choice and adds essentially no VRAM of its own.
Worth reading before you commit to a model: Open weight vs Open source — the difference decides whether you can use it commercially.
Building a private ChatGPT
Most people asking for self-hosted AI want the ChatGPT experience without the data leaving the building. Each part of that experience has a self-hosted equivalent — none of them is a single download, and the capability ceiling is set by the model your hardware holds.
| What you use in ChatGPT | Self-hosted equivalent |
|---|---|
| Chat with history | Open WebUI in front of a model served by Ollama. Open WebUI |
| Asking about your documents | Open WebUI's document upload, which retrieves passages with a local embedding model. Local embedding models |
| Image generation | An open image model such as FLUX.1 or SDXL, run in ComfyUI. ComfyUI |
| Coding help | A local coding model connected to your editor. Local coding assistant |
| Access from other devices | Ollama bound to your network, with authentication in front of it. Ollama as a server |
The one thing it will not reproduce is the largest hosted model. Everything on the right works without an internet connection once the models are downloaded.
What the hardware buys you
VRAM is the constraint that decides everything else — it sets which models you can hold, and holding the model is the difference between an assistant that answers in seconds and one that answers in minutes.
16GB
A mainstream desktop card. Runs 12 of the 30 models — enough for drafting, summarising and code assistance.
48GB
Two cards, or a large unified-memory Mac. Runs 26 models, including the ones that approach hosted quality.
To go the other way — from a machine you already own to the models it runs — the hardware matrix profiles real devices, and the model recommender turns a spec into a shortlist.
Does it pay for itself?
Honestly: often not, on cost alone. Hardware is paid up front against a per-token charge that starts at nothing, and light use rarely clears the gap. The case for self-hosting is usually the privacy property above, or the independence from a vendor’s pricing and deprecation schedule — with cost as a bonus at high volume rather than the reason.
Common questions
- What is self-hosted AI?
- Running the model on hardware you control instead of calling someone else's API. In practice it means three things you install yourself: the model weights, a server that loads them and answers requests, and an interface you talk to. All three are free and open; the cost is the hardware and the time to set it up.
- Is a self-hosted LLM actually private?
- The inference is. Once the weights are on your disk, generating a response involves no network request — you can disconnect the machine and it still works, which is the strongest privacy property available and not something a hosted service can offer at any price. What it does not do is secure the rest of the system: a chat interface still writes history to a database, document upload still stores files, and anything you deliberately connect to the internet is connected. Self-hosting removes the third party, not the need to look after your own machine.
- What hardware do I need for a private AI chatbot?
- Less than most people expect for a usable assistant, and far more than expected for a frontier-class one. Capable 7B to 14B models run on a mainstream 16GB card at 4-bit, which is enough for drafting, summarising and code assistance. Matching the largest hosted models needs 48GB or more, which usually means multiple GPUs or a large unified-memory Mac. The VRAM tier pages score every model against each card size.
- Is self-hosting cheaper than paying for an API?
- It depends almost entirely on volume, and the break-even is further out than enthusiasm suggests. Hardware is paid up front and electricity is ongoing, against a per-token charge that starts at nothing. Light, occasional use rarely justifies the purchase on cost alone; heavy or sustained use often does. The API pricing tool runs the comparison against your actual token volume rather than a rule of thumb.
- Can AI work without the internet?
- A self-hosted model can. Once the weights are on your disk, generating a response needs no connection at all — the download is the only step that does. Hosted assistants such as ChatGPT cannot, because the model runs on the provider's servers. Features that deliberately reach outward, like web search, need a connection whichever you use.
- Can I self-host AI without a GPU?
- Yes, within limits. Models run on CPU against system RAM, and small ones are genuinely usable that way — but memory bandwidth rather than compute sets the speed, and system RAM is several times slower than a GPU's. Expect a few tokens per second on a modern desktop CPU with a small model. Apple Silicon is the exception, because its unified memory gives the GPU bandwidth that ordinary system RAM does not.
Guides that go deeper
Sizing memory, choosing a runner, and picking the card that holds the model.
- LiteLLM Alternatives in 2026, and How to Run LiteLLM Safely With Local ModelsGateway licences, maintenance and the defaults to change.Read the guide
- How Much RAM and VRAM Do You Need to Run Local LLMs?Model weights, KV cache and OS overhead, tier by tier.Read the guide
- Best Local LLM Runners: Ollama vs LM Studio vs Jan vs llama.cpp vs vLLMWhich runner suits your hardware and workflow.Read the guide