Homelab AI Server (Dual RTX 3090 48GB VRAM)

Scored against every model in our directory at Q4_K_M, using the ~48GB of VRAM on this machine and its 936 GB/s of memory bandwidth. Of 30 models, 26 run comfortably, 0 are a tight fit, 3 need CPU offload, and 1 will not load. The largest model that fits well here is Qwen 2.5 72B Instruct.

How these are estimated: tokens/sec ≈ memory bandwidth × 0.65 ÷ active weights in GB. These are arithmetic estimates from published specifications, not lab measurements — BFWR does not run a hardware test lab. Real throughput varies with runner, quantization, context length and thermals.

Diffusion models size up differently from language models: the text encoder is often as large as the image model itself, and memory scales with resolution. Measured at 1024×1024, one image at a time, against the ~48GB of VRAM here, 11 of 11 open image models will run — the best being FLUX.1 [dev] at BF16 / FP16, peaking near 36.5GB — but that model is non-commercial; FLUX.1 [schnell] is the best option you can ship.

Weight sizes are read from the real .safetensors files in each model’s Hugging Face repository. Activation memory is an approximation that varies with the attention implementation and sampler, so leave headroom. These are planning figures, not benchmarks.

What is the largest model the Homelab AI Server (Dual RTX 3090 48GB VRAM) can run?

Qwen 2.5 72B Instruct (72.7B) is the largest model that fits, at roughly 42.5GB in Q4_K_M against 48GB of VRAM. It is rated "Comfortable" here, with an estimated 14.3 tokens/sec.

How many tokens per second should I expect?

Speed depends on the model, not just the machine. On this hardware the estimates range from about 338 tokens/sec for the smallest models down to single digits for the largest that still fit. The estimate uses memory bandwidth (936 GB/s here) × 0.65 ÷ active weights in GB, and is not a measured benchmark.

Can the Homelab AI Server (Dual RTX 3090 48GB VRAM) run a 70B model?

Yes — Qwen 2.5 72B Instruct is rated "Comfortable" here at roughly 42.5GB in Q4_K_M, with an estimated 14.3 tokens/sec. Expect to keep the context modest.

How much context can I use?

128k tokens is a sensible ceiling on this machine at FP16 for 8B; Q4_K_M for 70B. Context is stored in the KV cache, which grows with every token and competes with the model weights for the same memory, so a longer context means a smaller model.

Can the Homelab AI Server (Dual RTX 3090 48GB VRAM) generate images with Flux or Stable Diffusion?

Yes. 11 of 11 open image models fit at 1024x1024, the best being FLUX.1 [dev] at BF16 / FP16, peaking near 36.5GB. Note that FLUX.1 [dev] is licensed for non-commercial use only; FLUX.1 [schnell] is the best option you can actually ship. Higher resolutions and larger batches cost proportionally more.

Which runner works best on the Homelab AI Server (Dual RTX 3090 48GB VRAM)?

vLLM, Ollama, llama.cpp, TGI are all expected to run on this hardware. Ollama is the simplest starting point, and `ollama run qwen3.6:35b` is a good first model for this machine.

In-depth reads that pair with this machine — sizing memory, picking a runner, and choosing quantization for ~48GB of usable memory.

• Strata: Running Qwen3.8-Flash-Next on a 12GB Graphics CardThe engine that splits a 125B model across GPU, RAM and SSD.Read the guide
• Best Budget GPUs for Local LLMs16GB cards, used RTX 3090s and current street prices.Read the guide
• RTX 5080 vs RTX 4090: Why 16GB VRAM Is the LimitWhy VRAM capacity beats raw speed for local inference.Read the guide