← All machines

Hybrid GPU + RAM · DIY build

RTX 5090 + EPYC 9355P with 768 GB DDR5 (MoE offload)

The usual recipe for running 600B+ MoE models at home: expert weights in 12-channel DDR5, attention and KV cache on one fast GPU. Faster prompt processing than CPU-only for a fraction of an all-GPU build's VRAM bill.

Memory for models

32 GB

VRAM

Memory bandwidth

1,792 GB/s

Parts

~$3,763 + unpriced parts

GPU, motherboard, ram and cooling prices not included

Largest at 4-bit

Nemotron 3.5 Lightning 30B-A3B

25.3 GB at 8,192 tokens of context

gpt-oss-20b, 4-bit

290 tok/smedian of 4 runs

Parts

PartQtyEach
GPUNVIDIA GeForce RTX 50901not priced
CPUAMD EPYC 9355P (32-core, 8 CCDs, 280 W, SP5)1~$2,598
MotherboardSupermicro H13SSL-N rev 2.01 (3x PCIe 5.0 x16)1not priced
RAM12x 64GB DDR5-6400 ECC RDIMM12not priced
PSUCorsair HX1500i (1500 W at 115-240 V, 1200 W at 100-115 V)1~$390
CaseCorsair 7000D AIRFLOW full tower (450 mm GPU clearance, 8 slots)1~$290
CoolingSP5 tower cooler1not priced
StorageCorsair MP600 CORE XT 2TB PCIe 4.0 NVMe1~$485
Priced parts only$3,763GPU, motherboard, ram and cooling prices not included

New parts are priced at the maker’s own store or list price. Used cards have no price here yet, and a total with a missing GPU would understate the build, so none is shown.

What it runs at 4-bit

Every model in the catalog at its 4-bit file with 8,192 tokens of context. 39 of 41 fit. Speeds are the median of measured runs on the same GPUs where any exist; otherwise an estimate from memory bandwidth. Try other quants and context lengths.

ModelNeedsFitGeneration speed
Kimi K32.8T, 104B active · UD-Q4_K_XL1406.8 GBcontext memory estimatedDoes not fit
DeepSeek V4 Pro1.6T, 49B active · Q4_K_M890.5 GBcontext memory not countedDoes not fit
Kimi K2.61T, 32B active · UD-Q4_K_XL545.7 GBcontext memory estimatedGPU + RAM offloadassumed limit~8.5–12 tok/sestimate, part in RAM
DeepSeek V4.1 Flash552B, 16B active · Q4 (DwarfStar)484.5 GBGPU + RAM offloadassumed limit~11–15 tok/sestimate, part in RAM
GLM-5.3753.3B · UD-Q4_K_XL437.4 GBcontext memory estimatedGPU + RAM offloadassumed limit–
DeepSeek V3.2671B, 37B active · Q4_K_M379.6 GBcontext memory estimatedGPU + RAM offloadassumed limit~7.2–11 tok/sestimate, part in RAM
MiniMax M3428B, 23B active · Q4_K_M244.8 GBcontext memory not countedGPU + RAM offloadassumed limit~12–17 tok/sestimate, part in RAM
Qwen3.5 397B-A17B397B, 17B active · Q4_K_M229.1 GBcontext memory estimatedGPU + RAM offloadassumed limit~16–23 tok/sestimate, part in RAM
GLM-5.3 Flash320B, 18B active · UD-Q4_K_XL187.6 GBcontext memory estimatedGPU + RAM offloadassumed limit~15–22 tok/sestimate, part in RAM
Hy3295B, 21B active · Q4_K_M173.6 GBGPU + RAM offloadassumed limit~13–19 tok/sestimate, part in RAM
MiniMax M2.7228.7B · Q4_K_M132.7 GBGPU + RAM offloadassumed limit–
Qwen3.8 Flash-Next125B, 6B active · Q4_K_M113.1 GBcontext memory estimatedGPU + RAM offloadassumed limit~32–47 tok/sestimate, part in RAM
Nemotron 3 Super 120B-A12B120B, 12B active · Q4_K_M82.6 GBcontext memory estimatedGPU + RAM offloadassumed limit~23–34 tok/sestimate, part in RAM
Qwen3.5 122B-A10B122B, 10B active · Q4_K_M73.0 GBcontext memory estimatedGPU + RAM offloadassumed limit~34–49 tok/sestimate, part in RAM
Mistral Medium 3.5 128B128B · Q4_K_M74.0 GBGPU + RAM offloadassumed limit~5.4–8.6 tok/sestimate, part in RAM
GLM-4.5-Air106B, 12B active · Q4_K_M70.9 GBGPU + RAM offloadassumed limit~26–38 tok/sestimate, part in RAM
Mistral Small 4 119B (2603)119B, 6.5B active · Q4_K_M69.3 GBcontext memory estimatedGPU + RAM offloadassumed limit~55–80 tok/sestimate, part in RAM
Llama 4 Scout 17B-16E Instruct109B, 17B active · Q4_K_M62.4 GBcontext memory not countedGPU + RAM offloadassumed limit~22–33 tok/sestimate, part in RAM
gpt-oss-120b117B, 5.1B active · MXFP4 (native)60.2 GBcontext memory estimatedGPU + RAM offloadassumed limit~85–124 tok/sestimate, part in RAM
Qwen3 Coder Next80B, 3B active · Q4_K_M46.9 GBcontext memory estimatedGPU + RAM offloadassumed limit~150–219 tok/sestimate, part in RAM
Llama 3.3 70B Instruct70B · Q4_K_M43.6 GBGPU + RAM offloadassumed limit~13–21 tok/sestimate, part in RAM
Nemotron 3.5 Lightning 30B-A3B30B, 3B active · Q4_K_M25.3 GBcontext memory estimatedFits~175–256 tok/sestimate
Qwen3.6 35B-A3B35B, 3B active · Q4_K_M22.4 GBcontext memory estimatedFits117 tok/smedian of 1 run
Ornith 1.5 35B-A3B35B, 3B active · Q4_K_M21.9 GBcontext memory estimatedFits~240–350 tok/sestimate
Qwen3 Coder 30B-A3B Instruct30.5B, 3.3B active · Q4_K_M19.5 GBFits~222–324 tok/sestimate
Gemma 4 31B IT30.7B · Q4_K_M18.9 GBcontext memory estimatedFits61 tok/smedian of 1 run
Granite 4.2 30B29.3B · Q4_K_M20.3 GBFits~49–77 tok/sestimate
Qwen3.8 27B27B · Q4_K_M18.2 GBcontext memory estimatedFits~50–79 tok/sestimate
Gemma 4 26B-A4B IT25.2B, 3.8B active · Q4_K_M17.4 GBcontext memory estimatedFits122 tok/smedian of 2 runs
Qwen3.6 27B27B · Q4_K_M17.7 GBcontext memory estimatedFits~52–82 tok/sestimate
Devstral Small 2 24B (2512)24B · Q4_K_M16.1 GBFits~61–96 tok/sestimate
gpt-oss-20b21B, 3.6B active · MXFP4 (native)12.5 GBcontext memory estimatedFits290 tok/smedian of 4 runs
Phi-414.7B · Q4_K_M11.3 GBFits~99–155 tok/sestimate
Ministral 3 14B Reasoning (2512)13.9B · Q4_K_M10.4 GBFits~107–167 tok/sestimate
Gemma 4 12B IT11.95B · Q4_K_M8.2 GBcontext memory estimatedFits~123–194 tok/sestimate
Qwen3.5 9B9B · Q4_K_M7.0 GBcontext memory estimatedFits~155–243 tok/sestimate
Granite 4.2 8B8.8B · Q4_K_M7.9 GBFits~159–249 tok/sestimate
Gemma 4 E4B IT8B · Q4_K_M6.5 GBcontext memory not countedFits~289–454 tok/sestimate
Qwen3 8B8.2B · Q4_K_M7.3 GBFits200 tok/smedian of 1 run
Llama 3.1 8B Instruct8B · Q4_K_M7.1 GBFits200 tok/smedian of 3 runs
Qwen3.5 4B4B · Q4_K_M4.3 GBcontext memory estimatedFits~321–503 tok/sestimate

Measured speeds

15 published runs with one user, plain decoding and at most 4,096 tokens already in context, on RTX 5090. Runs on 1x RTX 5090 with the same GPUs count here too, and say so.

Plain single-user runs
ModelQuantBackendGenerationPromptContextMeasured onSource
Gemma 4 26B-A4B ITQ4_Kllama.cpp (build not stated)180 tok/s8,799 tok/s4,096-token prompt4,096RTX 5090review2026-03
Gemma 4 26B-A4B ITINT4 experts / HQQ4 attention / k4v4 KVKrasis (MoE runtime; version not stated in table)64 tok/s5,599 tok/s0RTX 5090independent benchmark2026-10
Gemma 4 31B ITQ4_Kllama.cpp (build not stated)61 tok/s3,395 tok/s4,096-token prompt4,096RTX 5090review2026-03
gpt-oss-20bMXFP4 MoEllama.cpp b6782 CUDA,BLAS299 tok/s13,002 tok/s2,048-token prompt0RTX 5090llama.cpp benchmark discussion2025-10
gpt-oss-20bQ4_Kllama.cpp (build not stated)298 tok/s9,444 tok/s4,096-token prompt4,096RTX 5090review2026-03
gpt-oss-20bMXFP4 MoEllama.cpp b6210 CUDA283 tok/s9,848 tok/s2,048-token prompt0RTX 5090llama.cpp benchmark discussion2025-08
gpt-oss-20bMXFP4Ollama (version not stated)205 tok/s8,519 tok/s0RTX 5090review2025-10
Llama 3.1 8B InstructQ4_K_Mllama.cpp b11741278 tok/s–0RTX 5090review2026-10
Llama 3.1 8B InstructQ4_K_MOllama (version not stated)200 tok/s30,982 tok/s0RTX 5090review2025-10
Llama 3.1 8B InstructQ4_K_MLocalScore (llamafile; version not stated on leaderboard)66 tok/s6,297 tok/s0RTX 5090LocalScore2026-10
Llama 3.1 8B InstructQ8_0Ollama (version not stated)145 tok/s31,442 tok/s0RTX 5090review2025-10
Llama 3.1 8B InstructFP8SGLang (lmsysorg/sglang:spark image)138 tok/s21,956 tok/s2,048-token prompt0RTX 5090review2025-10
Llama 3.3 70B InstructIQ3_XXSllama.cpp b1174153 tok/s–0RTX 5090review2026-10
Qwen3 8BQ4_Kllama.cpp b8189200 tok/s11,933 tok/s4,096-token prompt4,096RTX 5090review2026-03
Qwen3.6 35B-A3BINT4 experts / HQQ4 attention / k4v4 KVKrasis (MoE runtime; version not stated in table)117 tok/s10,670 tok/s0RTX 5090independent benchmark2026-10

With speculative decoding

A small draft model or extra prediction heads guess several tokens ahead. Generation is faster on text the guesses get right, so these are kept out of the medians above.

Runs with speculative decoding
ModelQuantBackendGenerationPromptContextMeasured onSource
Qwen3.8 27BNVFP4 (native, NVFP4 KV)vLLM 0.27.1 + heavy local patches101 tok/s–0RTX 5090independent benchmark2026-08

Deep into a long context

Measured with more than 4,096 tokens already in context, where every new token also reads the context memory.

Runs at long context
ModelQuantBackendGenerationPromptContextMeasured onSource
Gemma 4 26B-A4B ITQ4_Kllama.cpp (build not stated)167 tok/s7,733 tok/s16,384-token prompt16,384RTX 5090review2026-03
Gemma 4 26B-A4B ITQ4_Kllama.cpp (build not stated)159 tok/s6,293 tok/s32,768-token prompt32,768RTX 5090review2026-03
Gemma 4 31B ITQ4_Kllama.cpp (build not stated)59 tok/s2,794 tok/s16,384-token prompt16,384RTX 5090review2026-03
Gemma 4 31B ITQ4_Kllama.cpp (build not stated)55 tok/s2,229 tok/s32,768-token prompt32,768RTX 5090review2026-03
gpt-oss-20bQ4_Kllama.cpp (build not stated)249 tok/s7,168 tok/s16,384-token prompt16,384RTX 5090review2026-03
gpt-oss-20bQ4_Kllama.cpp (build not stated)215 tok/s5,183 tok/s32,768-token prompt32,768RTX 5090review2026-03
Qwen3 8BQ4_Kllama.cpp b8189162 tok/s8,538 tok/s16,384-token prompt16,384RTX 5090review2026-03
Qwen3 8BQ4_Kllama.cpp b8189130 tok/s6,034 tok/s32,768-token prompt32,768RTX 5090review2026-03

Specifications

GPUs
1 × NVIDIA GeForce RTX 5090
VRAM
32 GB
System RAM
768 GB
RAM bandwidth
12 channels x 6400 MT/s x 8 bytes = 614.4 GB/s
Power supply
1500 W
Estimated load
RTX 5090 575 W + EPYC 9355P 280 W + 100 W = 955 W
Software
CUDA, Vulkan

Before you build it

  • llama.cpp keeps MoE expert weights in system RAM with --cpu-moe (all layers) or --n-cpu-moe N (first N layers), leaving attention and shared weights on the GPU.
  • The H13SSL-N gives the 5090 a full PCIe 5.0 x16 slot from the CPU.
  • Server RDIMMs cost more per GB than desktop kits in the 2026 shortage: about $2,245 per 64 GB DDR5 RDIMM (~$35/GB) in the Oct 2026 retail tracker, so 12 sticks alone run well over $20,000.
  • A US 15 A / 120 V circuit is 1,800 W on paper, but the NEC 80% rule for continuous loads (3 h+) limits it to 12 A, about 1,440 W, shared with everything else on that circuit.

Sources

Compared with

Similar machines