NVIDIA Nemotron 3.5 Locally: Which Model Fits Your GPU

Nemotron 3.5 is NVIDIA’s 2026 family of open models. Four of them are language models you can run at home: the small Bolt E4B, the 30B Lightning, the 74B Bolt squeezed into a 24GB package, and the 120B Super VL. Their names carry memory figures that do not match the downloads.

This guide lists each model with its real download sizes, explains what the 4GB, 24GB and DFlash labels mean, works out the memory to leave for context, and says which file to pick for a 4GB to 128GB GPU or Mac. BFWR has not run these models. File sizes come from the Hugging Face API and Ollama’s registry, hardware advice from NVIDIA’s model cards, and context figures are computed from each model’s config.json.

Quick answer

• 4GB to 12GB: Bolt E4B, NVIDIA’s Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf (6.18 GB). NVIDIA measured 3.1 to 5.0 GB of VRAM in use: borderline on 4GB, comfortable on 8GB.
• 24GB: Lightning 30B-A3B at Q4_0 from ggml-org (18.90 GB). Ollama’s 25.43 GB Q4_K_M build leaves almost no room.
• 32GB: Bolt 74B-A9B. Despite the 24GB name, NVIDIA’s llama.cpp recipes are 28GB and 32GB, for 32GB+ and 48GB+ cards.
• 64GB to 128GB: Bolt 74B with 256K context, or Super VL 120B-A12B (72 to 90 GB files) on a 96GB or 128GB machine.
• Mac: Bolt 74B is not supported on Apple Silicon, per Ollama; E4B, Lightning and Super have MLX builds.
• Licence: OpenMDW-1.1, permissive, with no commercial, revenue or user limits.

The Nemotron 3.5 family at a glance

Sizes are decimal GB (one billion bytes), read on 1 November 2026. Parameter counts are NVIDIA’s; the Hugging Face API counts 31.58B for Lightning and 124.45B for Super VL. Lightning, Bolt 74B and Super VL are hybrids of Mamba-2, mixture-of-experts and a few attention layers. Bolt E4B is Google’s Gemma 4 E4B, “further post-trained from Gemma-4-E4B-it” by NVIDIA.

What 24GB, 4GB and DFlash actually mean

Bolt 74B-A9B 24GB is Super VL, compressed and then quantized to 2 bits

NVIDIA shrank Super VL rather than training a new model. The architecture search “targets a deployment footprint below 24 GB at batch size 1 and 128K context”, and “expert top-K is fixed at 9, down from 22 in the parent model”. The experts are then quantized with quantization-aware distillation so that “each parameter has 2.0625 bits per element”; Mamba layers use NVFP4, and embedding and output layers Q8_0.

That explains an odd number: the Hugging Face API reports 22,367,086,600 parameters for the safetensors repo, not 74 billion. 20.93 billion of them are U8 entries, bytes that store packed low-bit weights, so the API is counting storage, not weights. The GGUF’s metadata reports 74.04B.

The 24GB label describes the weights, not the whole job. The GGUF card’s deployment line reads “llama.cpp on RTX/DGX Spark (64GB / 128GB) or RTX discrete GPUs (32GB+ / 48GB+)”, with recipes labelled 28GB (65,536-token context) and 32GB (262,144 tokens). There is no recipe for a 24GB graphics card.

Bolt E4B 4GB is a 6.18 GB file

The Q4_K_M file in NVIDIA’s repo is 6,180,446,208 bytes. NVIDIA’s announcement calls the name a workload figure: “as little as ~4GB in its target workloads”. Its table (llama.cpp, CUDA, Q4_K_M) shows 3.5 GB of VRAM on an RTX 5070 with 3K tokens of input, 4.3 GB with 32K tokens cached, 3.1 to 4.2 GB on an RTX Spark laptop, and 4.2 to 5.0 GB on Jetson boards.

NVIDIA does not explain why that is below the file size. Gemma’s card notes that E4B’s per-layer embedding tables “are large but are only used for quick lookups”. If your card is tight, plan for the file size.

DFlash and LiLiCorr are optional drafters

Repos ending in -DFlash or -LiLiCorr are draft models for speculative decoding, not models to chat with. NVIDIA: “It proposes all draft tokens in one forward pass, which the target verifies in parallel while preserving the target model’s output distribution.” The E4B drafter is 0.74 GB and Bolt 74B’s is 1.00 GB. Skip them when memory is short; output is the same, only slower.

Download sizes per quant

unsloth’s 1-bit and 2-bit Lightning files are all about 19.4 GB, no smaller than ggml-org’s Q4_0, so going below 4-bit buys nothing on Lightning. And only NVIDIA’s Bolt 74B IQ2 file was trained for its format; unsloth’s 1-bit Bolt files are separate quantizations (different file hashes) with no NVIDIA accuracy figures.

How much memory to leave for context

Only attention layers keep a growing key-value cache, and these hybrids have few of them. From each config.json: Lightning has 6 attention layers of 52, Bolt 74B and Super VL 8 of 88, each with 2 key-value heads of 128 dimensions. At the default 16-bit cache that is 6,144 bytes per token for Lightning and 8,192 for Bolt and Super.

Mamba layers add a fixed state of about 0.05 GB (Lightning) and 0.17 GB (Bolt) that does not grow. So most headroom goes to compute buffers, the vision projector (1.73 GB for Bolt 74B), the drafter, and your desktop. NVIDIA’s 28GB Bolt recipe loads 24.75 GB of files with a 65,536-token cache of about 0.29 GB at 8-bit, and still budgets 28GB. Card memory is binary: a 24GB card holds 25.77 GB in file-size units, a 32GB card 34.36 GB.

Which model fits your memory

4GB

Only Bolt E4B, and only just. NVIDIA’s 3.5 GB at short input suggests the Q4_K_M build fits short conversations; its 4.3 GB at 32K cached tokens suggests long ones will not. unsloth’s UD-IQ2_M (3.53 GB) is smaller, but NVIDIA benchmarked only Q4_K_M and BF16. Skip the drafter.

8GB

Bolt E4B at Q4_K_M with its full 64K context and the drafter. Ollama’s e4b package adds a vision projector too and totals 7.91 GB, tight on 8GB. Nothing larger fits: Lightning’s smallest file is 18.90 GB.

12GB to 16GB

Bolt E4B at Q8_0 (8.03 GB) for less quantization loss. Lightning fits only with its experts in system RAM, via llama.cpp’s --cpu-moe (“keep all Mixture of Experts (MoE) weights in the CPU”) or --n-cpu-moe N. That needs 32GB of system RAM and runs slower.

24GB

Lightning at Q4_0 (18.90 GB) leaves about 6.9 GB, enough for its 256K context (1.61 GB) plus buffers. Ollama’s Q4_K_M build, at 25.43 GB, is 23.7 binary GB before any context. Despite its name, Bolt 74B has no NVIDIA 24GB-card setup: the IQ2 file alone leaves about 3.7 GB, and whether a short-context, text-only Bolt runs well on an RTX 3090, RTX 4090 or RX 7900 XTX is unpublished. NVIDIA lists only NVIDIA GPUs as supported.

32GB

Bolt 74B with NVIDIA’s 28GB recipe (64K context, 8-bit cache, drafter, vision), which the card assigns to “RTX dGPU 32GB+” and RTX Spark 64GB. On a 32GB Mac, use Ollama’s MLX Lightning (30b-a3b-nvfp4, 19 GB); Ollama’s Bolt page says “Apple Silicon / Metal is not currently supported for 74B-A9B.”

64GB to 128GB

On a 48GB+ card or 128GB RTX Spark, NVIDIA’s 32GB Bolt recipe runs the full 262,144-token context. Super VL needs 72 to 90 GB at 4-bit; its card lists “1× DGX Spark, 1x NVIDIA RTX PRO 6000, 1× H200, 2× H100, or 1× GB200”. At 64GB, Super VL fits only as unsloth’s sub-4-bit files (46.14 to 59.76 GB), so Bolt 74B, NVIDIA’s own compressed Super trained for 2-bit, is the more defensible pick. At 96GB or 128GB, Super VL at Q4_0 or NVFP4 fits.

Running it: Ollama, llama.cpp, LM Studio and vLLM

Ollama

# Bolt E4B (the small one)
ollama run nemotron-3.5-bolt:e4b

# Lightning 30B-A3B
ollama run nemotron-3.5-lightning

Watch the tag: plain ollama run nemotron-3.5-bolt pulls the 74B model (24.75 GB). Ollama sets context from VRAM, per NVIDIA’s Lightning card “under 24GB -> 4K, 24-48GB -> 32K, 48GB+ -> 256K”; raise it with /set parameter num_ctx.

llama.cpp

Bolt 74B and Super VL use a new architecture that was merged into llama.cpp on 15 October 2026 (pull request #30520, “Add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support”), so use a build from after that date. NVIDIA’s download and 28GB recipe (Linux shell; the card also gives PowerShell):

export MODEL_DIR="$PWD/models/nemotron-bolt"

hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --local-dir "$MODEL_DIR"

hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf \
  --local-dir "$MODEL_DIR"

export MODEL="$MODEL_DIR/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
export MMPROJ="$MODEL_DIR/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
export SPEC_DRAFT="$MODEL_DIR/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"

llama-server \
  --model "$MODEL" \
  --mmproj "$MMPROJ" \
  --batch-size 2048 \
  --ubatch-size 512 \
  --ctx-size 65536 \
  --parallel 1 \
  -ngl 999 \
  --spec-type draft-dflash \
  --spec-draft-n-max 6 \
  --spec-draft-model "$SPEC_DRAFT" \
  --spec-draft-ngl 999 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --temp 1.0 \
  --top-p 0.95 \
  --flash-attn on \
  --jinja

For Lightning, NVIDIA’s card points to ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF with a :Q4_K_M tag, but that repo holds only Q4_0, Q8_0 and BF16. llama.cpp’s docs say -hf will “default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn’t exist”, so name the quant. NVIDIA’s command with only the tag changed:

llama-server \
  -hf ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_0 \
  --temp 1.0 --top-p 0.95 \
  -np 1 \
  -c 40960 \
  --port 8000 \
  -ngl 99 \
  -fa on \
  --jinja \
  --no-webui \
  --fit off

Security: llama-server listens on 127.0.0.1 by default, with no API key unless you set --api-key. NVIDIA’s Bolt E4B Docker example publishes the port with -p 8000:8000, which exposes it to your network. Publish to localhost instead:

docker run --rm --gpus all \
  -p 127.0.0.1:8000:8000 \
  -v "$MODEL_DIR:/models:ro" \
  ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459 \
  -m "/models/$MODEL_FILE" \
  --alias nemotron-bolt \
  --host 0.0.0.0 \
  --port 8000 \
  -c 65536 \
  -ngl 999 \
  --jinja

The only change is 127.0.0.1: in the port mapping; --host 0.0.0.0 applies inside the container.

LM Studio and vLLM

LM Studio’s catalogue lists Lightning (minimum system memory 21 GB) and Super VL (at least 70 GB), but had no Bolt page on 1 November. On Linux, vLLM is NVIDIA’s main route: Lightning’s card uses vllm/vllm-openai:v0.27.1, while safetensors Bolt 74B needs a pinned nightly image and extra kernels, and its card names A100, H100 and Blackwell in one place but only Blackwell SM120 and SM121 in another.

What NVIDIA measured

These are NVIDIA’s figures. On an RTX Spark laptop (Surface Laptop Ultra, 128GB) with llama.cpp, Bolt 74B with its drafter reached 83.99 tokens per second on NVIDIA’s coding throughput test, against 96.63 for Qwen3.6 35B-A3B and 100.56 for Gemma 4 26B-A4B at 4-bit with their own drafters. Bolt’s case is quality per GB, not speed: NVIDIA’s table has it ahead of both on 10 of 13 benchmarks. Bolt E4B at Q4_K_M managed 133.4 tokens per second on an RTX 5070 and 15.5 on a Jetson Orin Nano 8GB.

The licence: OpenMDW-1.1

Every repo shows license:other, Hugging Face’s label for licences it does not recognise. The cards name OpenMDW-1.1, a short text whose grant is “permission is hereby granted, free of charge, to deal in the Model Materials without restriction”. There is no commercial limit, revenue or user threshold, territory exclusion, or rule on derivatives. The conditions:

• Keep the notices. If you distribute the model, include the licence and copyright notices.
• Suing ends the grant. If you sue claiming the Model Materials infringe a patent or copyright, the licence terminates for you, unless you were sued first.
• Outputs are yours. “This agreement does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs generated by using the Model Materials.”

Clearing third-party rights is on you, and there is no warranty. Bolt E4B’s card adds “This model has been post-trained on the Google Gemma 4 model”, whose terms NVIDIA gives as Apache 2.0. Ollama’s Lightning package still carries the older NVIDIA Open Model License text, though NVIDIA’s repo has had an OpenMDW-1.1 LICENSE file since 7 August 2026. This is a reading of the published text, not legal advice.

The rest of the family

The family also has Transcribe 30B-A4B (speech to text; llama.cpp support was an open pull request on 1 November), Embed 1B and 8B and Rerank 8B for retrieval, Parse for document images, a 0.6B streaming speech model and a Content Safety classifier. All use OpenMDW-1.1; Embed and Rerank note Apache 2.0 Ministral 3 bases, and Content Safety, built on Gemma 3, adds Google’s Gemma Terms of Use.

FAQ

Does Nemotron 3.5 Bolt 74B run on a 24GB card?

NVIDIA does not say so. Its smallest recipe is 28GB, for 32GB+ cards. The 22.02 GB file fits on paper, but Lightning at Q4_0 is the safer 24GB choice.

Which Nemotron 3.5 model should I use on a Mac?

Bolt E4B, which Ollama supports on Apple Silicon, or Lightning through Ollama’s 19 GB MLX build on 32GB or more. Ollama says Bolt 74B does not currently support Apple Silicon.

Why is the 4GB model a 6.18 GB download?

The name is NVIDIA’s target memory use, not the file size. NVIDIA reports 3.1 to 5.0 GB of VRAM in use across its tests.

Is Nemotron 3.5 free for commercial use?

Yes. OpenMDW-1.1 sets no commercial limits, and the cards say the models are ready for commercial use.

Related guides and tools

• How much RAM and VRAM you need to run local LLMs
• LLM context windows explained: what they cost in VRAM
• Gemma 4 12B vs Qwen3.5 9B: which small local model to run
• Radeon RX 7900 XTX for local AI: 24GB, ROCm and what fits
• NVIDIA RTX Spark laptops for local AI
• VRAM tiers: which models fit each memory size and the 24GB tier
• llama.cpp models, memory and commands

Sources

• NVIDIA: Nemotron 3.5 Bolt 74B-A9B 24GB GGUF model card and safetensors model card
• NVIDIA: Nemotron 3.5 Bolt E4B 4GB model card and DFlash drafter
• NVIDIA: Nemotron 3.5 Lightning 30B-A3B NVFP4 model card
• NVIDIA: Nemotron 3.5 Super VL 120B-A12B NVFP4 model card
• NVIDIA Research: Nemotron 3.5 Bolt 74B-A9B 24GB announcement and Bolt E4B 4GB announcement
• Google: Gemma 4 E4B model card
• ggml-org: Lightning GGUF, Super VL GGUF
• unsloth: Bolt E4B GGUF, Bolt 74B GGUF, Lightning GGUF, Super VL GGUF
• Ollama: nemotron-3.5-bolt, nemotron-3.5-lightning, nemotron-3.5-super
• LM Studio: Nemotron 3.5 Lightning, Nemotron 3.5 Super VL
• llama.cpp pull request #30520 and llama-server documentation
• OpenMDW License Agreement, version 1.1
• Docker: port publishing

Updated 1 November 2026. Model cards, file sizes (Hugging Face API), Ollama registry manifests, LM Studio catalogue pages, llama.cpp pull requests and the OpenMDW-1.1 text were read on that date. No model was run for this guide.