• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
Best Free Web Resources — local AI tools and hardware guides

Best Free Web Resources

Local AI Tools & Digital Engineering Resources

What can I run?
  • AI Tools
    • Start: Local LLMs
    • Ollama Models & VRAM
    • LM Studio Models
    • GPU Comparison
    • AI Hardware Matrix
    • Local LLM Directory
    • VRAM Calculator
    • Hardware Compare
    • Local Model Recommender
    • API Pricing & ROI
  • Local AI Guides
  • Web Resources
    • Inspiration
    • Photography
    • Games
    • SEO
    • Tutorial
      • Coding
      • WordPress
    • Resume/CV
    • Graphics
      • Logo
    • Freebies
Cover illustration: Jan With Local Models

Jan With Local Models: Setup, the llama.cpp Engine and the Local API Server

November 21, 2026 by BFWR Editorial

Jan is a free, Apache-2.0 desktop app from Menlo Research that downloads and runs GGUF models through a llama.cpp engine built into the app. Version 0.8.8, released on 28 October 2026, bundles llama.cpp build b11669 from 15 October, runs on Windows 10 or later (x64 with AVX2, or ARM64), Apple Silicon Macs on macOS 13.6 or later, and the main Linux distributions, and on Windows and Linux asks for 8GB of RAM and, for GPU offload, 6GB of VRAM. Install 0.8.8 or newer rather than anything older: GitHub lists seven security advisories, one rated critical, that affect 0.8.7 and earlier. Jan’s OpenAI-compatible server runs on http://127.0.0.1:1337/v1 once you start it, and it should stay on 127.0.0.1.

This guide covers installing Jan, its engine and GPU backends, importing models, where files live, the API server and its security settings, MCP, what leaves your machine, and the most common failures. BFWR has not run Jan. The steps and figures come from Jan’s documentation, release notes, source repository, GitHub issues and security advisories, and the Hugging Face API, all read on 15 November 2026.

Quick answer

  • Version: v0.8.8 (28 October 2026), a security release. Check the releases page for anything newer.
  • Engine: llama.cpp ships inside the app and updates with it. NVIDIA cards use CUDA 13; AMD, Intel and older NVIDIA cards use Vulkan; Macs use Metal, with an experimental MLX engine.
  • Your own GGUF: Settings, Model Providers, Llama.cpp, Import. Jan links the file in place.
  • API server: 127.0.0.1:1337, prefix /v1, no API key unless you set one. Set one anyway.
  • Not local by default: web search is on and uses Exa’s keyless endpoint; analytics are asked for at first launch.
  • Intel Macs: local models stopped working in v0.8.5.

Install and requirements

Download Jan from jan.ai or the GitHub release. The requirements below are from Jan’s install pages, and the file sizes from the v0.8.8 release assets (MB = 106 bytes).

System Requirements v0.8.8 file Size
Windows x64 Windows 10+, AVX2 CPU (Intel Haswell 2013+, AMD Excavator 2015+), 8GB RAM (16GB recommended), 6GB VRAM for GPU use, 10GB free Jan_0.8.8_x64-setup.exe 571.7 MB
Windows on ARM As above, ARM64 Jan_0.8.8_arm64-setup.exe 335.6 MB
macOS macOS 13.6+, Apple Silicon, 10GB+ free Jan_0.8.8_universal.dmg 115.9 MB
Linux Debian/Ubuntu, Fedora/RHEL, Arch, openSUSE; AVX2; 8GB RAM; 6GB VRAM Jan_0.8.8_amd64.AppImage / Jan_0.8.8_amd64.deb 712.8 MB / 1,062.7 MB

The Windows and Linux packages carry the CUDA 13 engine, so no CUDA Toolkit is needed: “Jan bundles its CUDA 13 engine and runtime libraries.”

On Macs, the docs’ rough guide is 3B models on 8GB, 7B on 16GB and 13B on 32GB. “Local models do not run on Intel Macs since v0.8.5”; the app still opens there for cloud providers.

How Jan runs models: the llama.cpp engine

Since v0.8.5 the engine is not a separate download. In the docs’ words, “The llama.cpp engine is built into Jan and updates together with the app, so there is no separate engine version to download or pick.” It runs in its own worker process, and v0.8.8 moved it to llama.cpp b11669. The version, build and commit are shown under Settings > Model Providers > Llama.cpp.

Backends are chosen from what is bundled, under Settings > Hardware > GPUs:

  • NVIDIA on Windows and Linux: CUDA 13. Cards that cannot run it, such as Pascal GTX 10-series, use Vulkan.
  • AMD and Intel Arc: Vulkan, with a current driver. There is no ROCm download; for HIP, run your own llama-server as a custom endpoint.
  • Apple Silicon: Metal by default. The separate MLX engine needs macOS 14+, and its page says “MLX support in Jan is experimental”; it has no embeddings, so use llama.cpp for document chat.

Two defaults matter for memory. The context size defaults to 8192 tokens per model, and can go up to 131072. The KV cache type defaults to f16 for keys and values; switching KV Cache K Type and KV Cache V Type to q8_0 roughly halves its size. Our context window guide and VRAM calculator show what each costs.

Getting models in

From the Hub. The Hub tab browses Hugging Face, and each model carries a fit label, Fits, May be slow or Won’t fit, worked out for the machine Jan is running on. The docs add: “No data is downloaded to determine fit status.” You can also paste a Hugging Face model ID, such as the docs’ example TheBloke/Mistral-7B-v0.1-GGUF, into the Hub search, or use Use this model > Jan on a Hugging Face model page. Gated models need a token under Settings > General > Others > HuggingFace Token.

Your own GGUF file. Open Settings > Model Providers > Llama.cpp and click Import. “Jan links to the file in place without copying it”, so a model you already use in LM Studio or llama.cpp does not take up space twice, and deleting it in Jan leaves the original alone. For a split GGUF, point Jan at shard 1 and it picks up the rest.

Jan’s own models are small: Jan-v3-4b-base-instruct-Q4_K_XL.gguf, the example in its data-folder docs, is 3.00 GB, tagged Apache-2.0, and built on Qwen/Qwen3-4B-Instruct-2507, also Apache-2.0. For a step up on an 8–12GB card, see our Gemma 4 12B vs Qwen3.5 9B comparison and the model directory.

Where Jan keeps its files

  • Windows: %APPDATA%\Jan\data, which is C:\Users\<YourUsername>\AppData\Roaming\Jan\data
  • macOS: ~/Library/Application Support/Jan/data
  • Linux (.deb or AppImage): ~/.local/share/Jan/data
  • Linux (Flatpak): ~/.var/app/ai.jan.Jan/data/Jan/data

Downloaded GGUF models sit under llamacpp/models/<org>/<repo>/ as model.gguf beside a model.yml. Chats are under threads/ as messages.jsonl and thread.json, and the log is logs/app.log. To move it, use Settings > General > Jan Data Folder; since v0.8.8 a drive root such as D:\ is refused, so pick a folder like D:\Jan.

The local API server

Jan can serve a local model to other programs over an OpenAI-compatible API, and needs one loaded to start. Go to Settings > Local API Server, choose the model under Default Model, and click Start Server. With the default settings the base URL is http://127.0.0.1:1337/v1. The docs’ test request:

curl http://127.0.0.1:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "YOUR_MODEL_ID",
    "messages": [{"role": "user", "content": "Tell me a joke."}]
  }'

Omit the Authorization line if you have not set a key. Any OpenAI-compatible client can then use that base URL.

Security settings, and what to change

The defaults, from the server page and the v0.8.8 release notes:

  • Server Host 127.0.0.1: “the most secure option for personal use”. 0.0.0.0 opens it to your network.
  • API Key empty. The docs say “Leave the field empty to disable authentication”, and that this is not recommended on 0.0.0.0.
  • Trusted Hosts: since v0.8.8, a web page can only call the server from your own machine, Jan itself, or a host you list.
  • CORS on. The v0.8.8 notes say that before this release the switch “was shown but had no effect”.
  • Execute Tools on Server off, so MCP tools do not run on requests that come in through the API.

Keep Server Host at 127.0.0.1 and set an API key anyway. If another device must reach it, set a key first and list only that device in Trusted Hosts. Two advisories show why. GHSA-x6p8-7cp8-c3p6 (high) found that on versions 0.8.0 to 0.8.4, binding to 0.0.0.0 “silently discards the Trusted Hosts allowlist”, fixed in 0.8.5. GHSA-66pg-h682-8cmf (high), a cross-site request forgery on the local API, affected 0.6.9 to 0.8.7. A further change, merged on 30 October in pull request #9505, makes Jan refuse a non-loopback address without a key. It is not in 0.8.8, so for now that is your job.

Of the eleven advisories published on 29 October 2026, seven affect 0.8.7 and earlier and are fixed in 0.8.8, among them GHSA-j6hv-vhmg-vpj6, a critical “One-click RCE” through malicious MCP or tool content.

MCP tools

Jan is an MCP host. Add servers under Settings > MCP Servers with the + button, giving a server name, a command and arguments; the docs’ example is Browser MCP, with command npx and argument @browsermcp/mcp, which needs Node.js. For a local model, turn on Tools under its capabilities.

By default “you must approve each tool call individually”, in a panel that shows the arguments the model wants to pass. Allow All MCP Tool Permissions pre-approves everything; leave it off, or a tool call prompted by text hidden in a web page runs unseen. Our prompt injection guide explains that risk. Since v0.8.7 the chat toolbar also has a Shell switch for the sandboxed shell tool, which is on by default for tool-capable models.

What leaves your machine

Three things can reach the internet:

  • Web search, on by default. “The default provider (Exa) uses a free, keyless endpoint.” When a tool-capable model calls web_search or web_fetch, the query or URL goes to that provider. Turn it off in Settings > Web Search, per chat with the globe icon, or switch the provider to your own SearXNG instance.
  • Analytics, by choice. The privacy page says “You’ll choose tracking preferences at first launch.” If allowed, it counts active users and retention against a random ID, held by PostHog in the EU.
  • Cloud providers. OpenAI, Anthropic, Google, Groq, Mistral and OpenRouter are built in under Settings > Model Providers, with your own API key. As the privacy page puts it, “Cloud models (like GPT, Claude) need to see your messages to work.”

Common failures and fixes

Errors appear as a banner in the chat, with the cause in logs/app.log on lines containing jan-llama-worker:. The troubleshooting page lists these:

  • “This model’s architecture is not supported by the current engine.” In the log: unknown model architecture: '<arch>'. The bundled llama.cpp is older than the model. Update Jan, or run a newer llama-server yourself (for example llama-server -m /path/to/model.gguf --alias my-model --port 8080 -ngl 99 --jinja) and add it as an OpenAI-compatible custom provider at http://localhost:8080/v1. Qwen3 hit this in April 2025 (#4948); v0.5.17 followed in May.
  • Out of memory (failed to allocate ... buffer, cudaMalloc): pick a smaller quantization, cut the context size, lower GPU Layers, or switch the KV cache to q8_0.
  • Corrupt or partial file (failed to load model, missing tensor): download it again.
  • “The model took too long to load and was stopped.” The load timeout is 600 seconds; raise Timeout for llamacpp or move models to faster storage.

Three hardware cases from the issue tracker. On a GTX 10-series card, v0.8.4 picked a CUDA 13 build that has no Pascal code, so every load failed (#9091); since v0.8.5 those cards use Vulkan. On a GTX 16 or RTX 20 card, the error “the provided PTX was compiled with an unsupported toolchain” in v0.8.5 is fixed by v0.8.6. On Linux, the v0.8.5 AppImage crashed in libvulkan.so.1 on every load (#9173) because it shipped its own Vulkan loader; v0.8.6 uses the system’s. The reporter still saw crashes on 0.8.8 with MangoHud installed, an open case; a maintainer’s suggested fallback is the .deb, which uses system libraries.

The licence

Jan’s LICENSE is the Apache License 2.0, “Copyright 2025 Menlo Research”, with one extra line: “Attribution is requested in user-facing documentation and materials, where appropriate.” That is a request, not a condition, so you can use, modify and redistribute Jan, including commercially, under the normal Apache terms. Each model carries its own licence, which Jan does not change; check it in our model licence checker. This is our reading of the published text, not legal advice.

FAQ

What port does Jan’s local API use?

1337 on 127.0.0.1, with the prefix /v1, so chat requests go to http://127.0.0.1:1337/v1/chat/completions. Port, host and prefix can be changed under the server’s Configuration button.

Can Jan use my Ollama models?

Yes, through a custom OpenAI-compatible provider with the base URL http://localhost:11434/v1, which Jan’s docs list for Ollama. Ollama then runs the model and Jan is the chat window. Plain GGUF files can be imported instead.

Why is Jan not using my GPU?

Check that the GPU’s switch is on in Settings > Hardware > GPUs, that a current driver is installed, and, for AMD and Intel, that it supports Vulkan.

Does Jan work on an Intel Mac?

Only with cloud or custom providers. Local models have not run on Intel Macs since v0.8.5.

Related guides and tools

  • Best local LLM runners: Ollama vs LM Studio vs Jan vs llama.cpp vs vLLM
  • Gemma 4 12B vs Qwen3.5 9B: which small model to run
  • LLM context windows and what they cost in VRAM
  • Securing a local LLM: prompt injection
  • Open WebUI: setup and backends
  • VRAM calculator
  • Model directory
  • llama.cpp models and commands

Sources

  • janhq/jan: LICENSE, README, the v0.8.8 release and its assets, release notes for v0.8.0 to v0.8.8 in docs/src/pages/changelog/, pull request #9505, and issues #4948, #9091 and #9173
  • Jan docs: Windows, Mac and Linux install, llama.cpp engine, MLX, model management, data folder, local API server, MCP, web search, privacy, custom endpoints and troubleshooting
  • Jan security advisories, read through the GitHub API
  • Hugging Face API: janhq/Jan-v3-4B-base-instruct-gguf, janhq/Jan-v3-4B-base-instruct and Qwen/Qwen3-4B-Instruct-2507
  • llama.cpp release b11669

Updated 15 November 2026. Jan v0.8.8, its release assets, release notes and security advisories, the live docs at docs.jan.ai, the issues cited and the Hugging Face model data were read on that date.

⚡ Free Interactive Tools

Work out what your hardware can actually run

Free calculators and directories, sized from real model weights — before you download gigabytes or order a card:

⚙️ llama.cpp Models The engine under most runners 💰 API Pricing & ROI Cloud cost vs local hardware 🚀 Start: Local LLMs Hardware, runner and first model 📊 What Can I Run? Models by VRAM tier, 8GB–128GB

Filed Under: Technology, Tutorial

About BFWR Editorial

BFWR Editorial is the byline for guides researched and written by Best Free Web Resources. These guides are built from primary sources — model cards, configuration files, official documentation and vendor pricing pages — and every figure is dated and linked so you can check it yourself. Size and memory numbers are arithmetic computed from published model data or from our own calculators, never measurements: BFWR does not operate a hardware test lab and publishes no benchmarks of its own. Where a figure comes from a vendor, we say so.

Primary Sidebar

⚡ FREE INTERACTIVE TOOLS

Local AI Hardware & VRAM Matrix

Benchmark your PC or Mac for local AI, calculate VRAM requirements, and compare cloud API costs vs local hardware ROI.

🚀 New here? Start with Local LLMs → 💻 Hardware Matrix (79 Devices) → 📚 Local LLM Directory (30 Models) → 🎮 AI GPU Comparison → ⚡ VRAM & KV Cache Calculator → 💰 API Pricing & Break-Even → 🍎 Mac VRAM Unlocker (90% Boost) →
📦 Ollama 🖥️ LM Studio ⚙️ llama.cpp 🚦 vLLM 💬 Open WebUI 🎨 ComfyUI 🔒 Self-Hosted 📊 VRAM Tiers 🎯 Recommender ⚖️ Comparisons

Search

  • Facebook
  • Instagram
  • RSS
  • Twitter
  • YouTube

Footer

Latest Guides

  • Jan With Local Models: Setup, the llama.cpp Engine and the Local API Server
  • Free Seamless Textures in 2026: CC0 Sources, Photoshop Use and What the Licences Allow
  • GPT4All in 2026: Is It Still Maintained, and What to Use Instead
  • ComfyUI Manager Not Showing: Why, and How to Fix It in 2026
  • MiniMax H3 Image in ComfyUI: Files, VRAM, Settings and the Licence
  • LFM3 1.7B Locally: 256K Context in About 5GB, and the Licence Limit

AI Tools & Calculators

  • What can I run?
  • VRAM Calculator
  • GGUF Size Calculator
  • Fine-tuning VRAM
  • LoRA Training VRAM
  • Image Generation VRAM
  • Mac VRAM Optimizer

Browse & Compare

  • Models Directory (30 models)
  • Hardware Matrix (79 devices)
  • Image Models
  • Model Licences
  • Comparisons
  • Model Recommender
  • API Pricing & ROI

Guides & Resources

  • Technology
  • Coding
  • Tutorials
  • Freebies
  • Graphics

Copyright © 2026 · Best Free Web Resources · All Rights Reserved.  |  About · Privacy Policy · Contact · AI Tools Hub