Build a Private RAG Chatbot for Your Documents (2026): Open WebUI, Ollama and PrivateGPT

A RAG chatbot answers questions from your own documents: contracts, manuals, research notes, a codebase. It works by finding the relevant passages first and handing them to the model with your question. Run the whole thing locally and none of those documents leave your machine. That closes the data-egress risk but not the others — any document you index is untrusted input, which is why prompt injection and the OWASP Top 10 matter more once the pipeline is yours. That’s the appeal of local RAG: you get the “chat with your files” experience of hosted assistants without uploading anything to a provider. This guide covers how it works, the two practical ways to build one in 2026 (Open WebUI with Ollama, or PrivateGPT), which embedding model to use, and the one setting that decides whether answers actually use your documents.

Quick answer

• Easiest private RAG chatbot: Open WebUI in front of Ollama, with a local embedding model. Upload documents and reference them with # in chat.
• Building an app instead: PrivateGPT is now an API layer for private AI applications. It connects to a local model server and adds document ingestion with citations, tools and MCP.
• Embedding model: a small one is enough. nomic-embed-text is 274MB through Ollama, and it’s what Open WebUI’s own docs recommend for self-hosted setups.
• The setting almost everyone misses: Ollama derives its default context from your GPU’s VRAM — 4096 tokens on cards below 24 GiB. Retrieved passages don’t fit in it, so the model ignores your documents. Raise it before you blame the model.

How local RAG works

1. Chunk: each document is split into passages of a set size, with some overlap.
2. Embed: an embedding model turns each chunk into a vector, a list of numbers representing its meaning.
3. Store: the vectors go into a vector database.
4. Retrieve: your question is embedded the same way, and the closest chunks are pulled back.
5. Answer: those chunks are placed in the prompt, and the chat model answers from them.

So a local RAG setup runs two models: a small embedding model and your chat model. The chat model is the one that decides your memory needs; the embedding model is tiny by comparison. Size the chat model with the VRAM calculator, and see what each card size runs on our VRAM tier pages.

Option A: Open WebUI + Ollama (the easiest path)

If you don’t have Open WebUI running yet, our Open WebUI setup guide covers the install. Then:

1. Pull a chat model and an embedding model in Ollama:

   ollama pull qwen3:8b
   ollama pull nomic-embed-text

2. Point Open WebUI at the embedding model. Go to Settings → Admin → Documents, choose Ollama as the embedding engine and select nomic-embed-text. The same page holds the text splitter, chunk size and overlap settings.
3. Raise the context length. This is a different screen, which is why it gets missed: Settings → Admin → Models, click the pencil on your Ollama model, open Advanced Params and raise the context length. See below for why.
4. Add your documents. Upload files from the Knowledge section of the Workspace. Knowledge collections group related files, and can be attached to a model or referenced ad hoc.
5. Ask about them. In a chat, type # before your query and pick the document or collection from the suggestion box. A document icon appears above the message box to confirm it’s attached.

The context-length trap

This is the single most common reason a local RAG chatbot appears to ignore your files, and it isn’t really a RAG problem at all. Open WebUI’s documentation puts a warning at the top of its RAG page: Ollama “picks a model’s default context length from your GPU’s VRAM, and on GPUs with less than 24 GiB that is 4096 tokens”, which “severely limits Retrieval-Augmented Generation (RAG) performance” because retrieved data “may not be used at all or only partially processed”.

The scale of the mismatch is easy to underestimate. The docs note that web pages “typically contain 4,000-8,000+ tokens even after content extraction” — so with 4096 tokens available you can still be short of a single page, and the fix is to raise the model’s context to 8192 or more, with the docs suggesting beyond 16,000 where the model supports it. That costs memory, so check your headroom first.

Other settings that change answer quality

• Hybrid search. An optional mode enhances retrieval “via BM25, with re-ranking powered by CrossEncoder, and configurable relevance score thresholds”. It helps when your questions use exact terms, like part numbers or function names.
• Changing the embedding model later. The docs are explicit: existing documents keep their original chunks and embeddings until you re-index. After switching models, go to Settings → Admin → Documents and click Reindex to re-embed everything. Vectors from two different models are not comparable, so choose before loading a large library.
• The vector database. Open WebUI ships with a local ChromaDB by default and supports 13 vector databases in total. If you ever run it with multiple workers, the docs require you to move off the local ChromaDB to pgvector, Milvus, Qdrant or mariadb-vector.

Option B: PrivateGPT (for building your own app)

PrivateGPT changed character over the years, and older tutorials describe a different project. Its GitHub repository now describes it as a “Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.”

The important structural point is stated plainly in its README: PrivateGPT “does not run models itself. It connects to any OpenAI-compatible inference server via OPENAI_API_BASE“ — if that server implements /v1/chat/completions and /v1/models, it works. What it adds on top is:

• A standard messages API with streaming, async and token counting
• File and artifact ingestion, and retrieval with citations and agentic RAG
• Built-in tools mirroring the Claude API (web search, web fetch, code execution)
• Custom tools and MCP connectors
• Structured access to databases and CSVs
• A built-in workbench UI for testing and demos — the README is direct that “the API is the actual product”

The project is maintained by Zylon, is Apache 2.0 licensed, and released v1.0.1 on 18 June 2026. The macOS quick start uses Homebrew:

brew tap zylon-ai/tap
brew install private-gpt

Then point it at your inference server. Note there are two variables, not one — PrivateGPT wants a chat endpoint and an embedding endpoint separately, and leaving the second unset is an easy way to get a server that starts but cannot ingest. Ollama serves both from port 11434:

OPENAI_API_BASE=http://localhost:11434/v1 \
OPENAI_EMBEDDING_API_BASE=http://localhost:11434/v1 \
private-gpt serve

The workbench UI is then at http://localhost:8080/ui and the API at http://localhost:8080. One thing worth knowing before you build against it: the API follows the Anthropic API spec, not the OpenAI one — PrivateGPT speaks OpenAI outward, to your model server, and Anthropic-style inward, to your application. Linux and Windows install paths are in the project’s documentation.

Which to choose: Open WebUI if you want to chat with your documents today. PrivateGPT if you’re building a private document-aware application and want an API with citations and tools already in place.

Choosing an embedding model

Figures from our guide to local embedding models, taken from each model’s official card and config:

Start with nomic-embed-text for English documents — Open WebUI’s own troubleshooting guide recommends it, or mxbai-embed-large, for air-gapped and self-hosted setups precisely because they need no external API calls. Use bge-m3 if your documents span several languages.

Keeping it private

Local RAG keeps documents on your machine only if every piece is local. Two things break that quietly:

• Web search: any enabled web search feature sends queries out.
• Network exposure: a model server reachable from other machines can be used by anyone who can reach it.

Our self-hosted AI page covers what self-hosting does and doesn’t protect. The Ollama server guide shows how to keep the API bound to localhost.

FAQ

Is local RAG really private?

The retrieval and the generation are, as long as the embedding model, vector database and chat model all run on your machine. Which store you pick barely changes that privacy position, but it does change your RAM budget and your licence obligations — see how the local vector databases compare. Features that reach outward, such as web search or a hosted embedding API, send data out when you turn them on.

Do I need a GPU for a RAG chatbot?

Not for the embedding model, which is small. The chat model is what benefits from a GPU. A small model runs on CPU, slowly — our guide to tokens per second explains what sets that speed.

Why does the chatbot ignore my documents?

Usually the context length. Ollama’s VRAM-derived default — 4096 tokens on cards below 24 GiB — is too small for retrieved passages, so they get truncated before the model sees them; raise it to 8192 or more. Then check that you actually referenced the document with #, and that the relevant text isn’t split awkwardly — try a different chunk size.

Can I change the embedding model after indexing?

Yes, but you must re-index. Open WebUI has a Reindex button under Settings → Admin → Documents. Without it, old documents keep their old vectors and retrieval quality drops, because embeddings from different models aren’t comparable.

What happened to the old PrivateGPT?

The project is still active under Zylon, but it’s now an API layer that sits on top of a model server, not a self-contained chat app. Follow its current documentation rather than older tutorials. If you would rather assemble the pieces yourself in Python, using Ollama with LangChain covers the chat and embedding classes directly.

Sources

• Open WebUI docs: Retrieval Augmented Generation, Knowledge, RAG troubleshooting
• Zylon: PrivateGPT GitHub repository and documentation
• Ollama: nomic-embed-text

Updated 17 September 2026. Settings paths, defaults and version details were checked against the projects’ own documentation on that date.