root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Quadro RTX 5000 NVLink budget build guide and Qwen 3.5 models

Browse Source 
Research and document the $850 budget build: 2x Quadro RTX 5000 (16GB each)
connected via NVLink for 32GB unified VRAM. With Qwen 3.5's Gated Delta
Network architecture (Feb 2026), this setup runs the 35B-A3B MoE model
with full 262K context in ~25GB — the best price-to-capability ratio
for local AI coding available.

Includes:
- Exact NVLink bridge part numbers (RTX 5000 uses unique smaller connector)
- Motherboard/PSU requirements and slot spacing guidance
- llama.cpp and Ollama multi-GPU configuration
- VRAM budget calculations for all Qwen 3.5 model sizes
- Phased build plan (start with 1 card at $400, add second later)
- Updated model table with full Qwen 3.5 family specs
- Cost comparison vs RTX 8000, RTX 3090, Claude Max, and API pricing

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 15:57:24 +00:00
parent 8646dca2bb
commit 7fa91e9816
1 changed files with 227 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+227 -1
View File
@@ -25,10 +25,25 @@ coding sessions where the model needs to understand your entire codebase.
## Best Local Coding Models (2026)
### Qwen 3.5 Family (February 2026 — Gated Delta Networks)
Architecture breakthrough: 3 of every 4 layers use **linear attention** (O(n) scaling),
drastically reducing KV cache memory. These models need far less VRAM for long contexts
than traditional transformers.
| Model | Type | Active Params | Size at Q4_K_M | Max Context | Quality | Notes |
|-------|------|---------------|---------------|-------------|---------|-------|
| **Qwen3.5-35B-A3B** | **MoE** | **3B** | **~12GB** | **262K** | **A-** | Best bang for buck — 35B model, 3B active, fits 262K ctx in 25GB |
| **Qwen3.5-27B** | Dense | 27B | ~17GB | 262K | A- | 72.4% SWE-bench, ties GPT-5 mini |
| **Qwen3.5-122B-A10B** | MoE | 10B | ~76GB | 262K | A | Matches GPT-5 mini across the board |
| **Qwen3.5-9B** | Dense | 9B | ~6GB | 262K | B+ | Fits on any modern GPU |
| **Qwen3.5-4B** | Dense | 4B | ~3GB | 262K | B | Tiny but capable |
### Previous Generation (Still Relevant)
| Model | Size at Q4_K_M | Quality | Notes |
|-------|---------------|---------|-------|
| **Qwen2.5-Coder 32B** | ~20GB | 73.7 Aider (≈ GPT-4o) | FIM king, 92.7% HumanEval |
| **Qwen 3.5 27B** | ~16GB | 72.4% SWE-bench (ties GPT-5 mini) | 262K context, multimodal |
| **Qwen3-Coder 30B-A3B** (MoE) | ~18GB | #1 SWE-rebench (64.6%) | Only 3.3B active, very fast |
| **Qwen3-Coder-Next 80B** (MoE) | needs 64GB+ RAM offload | Beats Claude Opus 4.6 on SWE-rebench | Hybrid attention, 256K context |
@@ -100,6 +115,217 @@ The RTX 8000 has Turing Tensor Cores with native FP16 support. Unlike the P40, i
need `--force-fp32` workarounds. Image gen performance is significantly better than the P40,
though still behind Ampere/Ada cards.
## Budget Build: 2x Quadro RTX 5000 + NVLink ($850 Total)
*The best price-to-capability ratio for local AI coding in 2026.*
### Why This Works Now
Qwen 3.5 (February 2026) introduced **Gated Delta Networks** — 3 out of 4 layers use linear
attention (O(n) scaling) instead of quadratic. KV cache memory usage is dramatically lower
than traditional transformers. A 35B MoE model with 262K context now fits in ~25GB VRAM.
### Hardware
#### GPU: NVIDIA Quadro RTX 5000 (Turing, TU104)
| Spec | Value |
|------|-------|
| VRAM | 16GB GDDR6 |
| CUDA Cores | 3072 |
| Tensor Cores | 384 (Gen 2, FP16) |
| TDP | ~230W |
| NVLink | **Yes — 50 GB/s bidirectional** |
| Form Factor | Dual-slot, blower cooler (rack-friendly) |
| PCIe | 3.0 x16 |
| Used Price | **~$400** |
| Part Number | VCQRTX5000-PB |
#### NVLink Bridge (CRITICAL: RTX 5000 uses a unique smaller connector)
The Quadro RTX 5000 has a **shorter NVLink connector** than all other Quadro RTX cards.
Bridges from the RTX 6000/8000 will NOT physically fit. You must buy the RTX 5000-specific bridge.
| Detail | Value |
|--------|-------|
| Product | NVIDIA Quadro RTX 5000 NVLink HB Bridge 2-Slot |
| SKU | NVLINKX8-2SLOT-PB |
| Part Numbers | 1JF3K, 699-54934-0500-000, 900-54934-0100-000, P4934, 6FY12AA, L55997-001 |
| Price | **~$30-80** (eBay, Amazon) |
| Bandwidth | 50 GB/s total (25 GB/s per direction) |
| Sizing | 2-slot (cards adjacent) or 3-slot (one slot gap — better thermals) |
**Where to buy:**
- eBay: search "Quadro RTX 5000 NVLink" or part numbers P4934 / L55997-001 / 1JF3K
- Amazon: search part number 6FY12AA or 1JF3K
**WARNING:** The 3-slot bridge is recommended over 2-slot. With a 2-slot bridge the cards
sit directly adjacent — the top card's blower intake gets blocked by the bottom card.
A 3-slot bridge leaves an air gap for proper cooling.
#### Motherboard Requirements
| Requirement | Details |
|-------------|---------|
| PCIe slots | Two x16 slots (x8 electrical is fine — LLM inference is VRAM-bound, not PCIe-bound) |
| Slot spacing | Must match your NVLink bridge size (2-slot or 3-slot gap) |
| Power supply | 650W+ minimum (80 PLUS Gold recommended), 850W+ for headroom |
| Power connectors | 2x 8-pin PCIe power (one per card). Do NOT daisy-chain — use separate cables |
| CPU platform | Any modern platform works. Threadripper/Xeon not required |
**Recommended motherboards (workstation/server):**
- Any board with 2x PCIe x16 slots spaced 2-3 slots apart
- Server: Dell R730/R740 with GPU riser (but verify 3-slot bridge clearance in 2U)
- Workstation: MSI X399 Creation, ASUS WS series, Supermicro X11/X12 boards
- Desktop: Most ATX boards with 2 full-length x16 slots work
**Rack server note:** The Quadro RTX 5000's blower cooler exhausts out the bracket —
this works well in rack airflow. If using a 2U server, measure clearance for the NVLink
bridge sitting on top of the cards. A 4U chassis gives the most room.
### What Runs on 32GB Unified (2x RTX 5000 + NVLink)
| Model | Arch | Quant | Weights | Context | Total VRAM | Quality |
|-------|------|-------|---------|---------|------------|---------|
| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **A-** |
| Qwen3.5-27B | Dense | Q4_K_M | ~17GB | 128K+ | ~25GB | A- |
| Qwen3-Coder-Next (80B/3B active) | MoE | Q4 | ~20GB | 128K | ~28GB | A |
| Qwen2.5-Coder-14B | Dense | Q4_K_M | ~10GB | 128K | ~22GB | B+ |
| Qwen2.5-Coder-14B | Dense | Q8 | ~16GB | 64K | ~28GB | A- |
| Qwen2.5-Coder-32B | Dense | Q4_K_M | ~20GB | 16-24K | ~28GB | A- |
**The sweet spot: Qwen3.5-35B-A3B at Q4_K_M with 262K context.** This is a 35B parameter
model with only 3B active at inference (MoE). The Gated Delta Network architecture slashes
KV cache memory. The entire model + full 262K context fits in ~25GB — well within 32GB.
### What Runs on 16GB (Single RTX 5000 — Phase 1)
| Model | Quant | Context | Quality |
|-------|-------|---------|---------|
| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **A-** |
| Qwen3.5-9B | Q8 | 128K+ | B+ |
| Qwen3.5-4B | Q8 | 262K | B |
| Qwen2.5-Coder-7B | Q8 | 128K | B |
| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ |
Even a single card can run the Qwen3.5-35B-A3B MoE model — just with a smaller context window.
### Estimated Inference Speed
| Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) |
|-------|-------------|---------------------|
| Qwen3.5-35B-A3B Q4 (short ctx) | ~25-35 tok/s | ~25-35 tok/s |
| Qwen3.5-35B-A3B Q4 (128K ctx) | ~10-18 tok/s | ~15-25 tok/s |
| Qwen3.5-35B-A3B Q4 (262K ctx) | Won't fit | ~10-18 tok/s |
| Qwen2.5-Coder-14B Q4 | ~20-30 tok/s | ~25-35 tok/s |
NVLink matters most at large context windows where KV cache spans both cards.
At short contexts that fit on one card, the second GPU adds less benefit.
### Power Consumption & Cost
| Config | Idle | Load | Monthly (8hr/day @ $0.09/kWh) | Annual |
|--------|------|------|-------------------------------|--------|
| 1x Quadro RTX 5000 | ~15W | ~210W | **~$4.50** | ~$54 |
| 2x Quadro RTX 5000 | ~30W | ~420W | **~$9.00** | ~$108 |
### Software Setup
#### llama.cpp (Recommended — Best Multi-GPU Support)
```bash
# Build with CUDA support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
# Download Qwen3.5-35B-A3B GGUF (Q4_K_M)
# Get from: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF
# Run on dual GPU with NVLink
./build/bin/llama-server \
-m Qwen3.5-35B-A3B-Q4_K_M.gguf \
-ngl 999 \
-c 262144 \
--host 0.0.0.0 \
--port 8080
# llama.cpp auto-detects NVLink and splits layers across both GPUs
# Use -ts 1,1 to manually set equal split if needed
```
#### Ollama
```bash
# Requires Ollama v0.17+ for Qwen3.5 support
# NOTE: As of March 2026, some Qwen3.5 GGUFs have compatibility issues
# with Ollama due to mmproj vision files. llama.cpp may be more reliable.
# Environment variables for multi-GPU
export OLLAMA_GPU_SPLIT=16,16 # Equal split across both 16GB cards
export OLLAMA_KV_CACHE_TYPE=q8_0 # Halves KV cache VRAM with minimal quality loss
export OLLAMA_KEEP_ALIVE=24h # Keep model loaded in VRAM
export OLLAMA_FLASH_ATTENTION=1 # Enable flash attention for VRAM savings
# Pull and run
ollama pull qwen3.5:35b-a3b-q4_K_M
ollama run qwen3.5:35b-a3b-q4_K_M
```
#### Verify NVLink Is Working
```bash
# Check NVLink status
nvidia-smi nvlink --status
# Check NVLink bandwidth
nvidia-smi nvlink -gt d
# Monitor both GPUs during inference
watch -n 0.5 nvidia-smi
```
### Total Cost Summary
| Item | Cost |
|------|------|
| 1x Quadro RTX 5000 (Phase 1) | $400 |
| 1x Quadro RTX 5000 (Phase 2) | $400 |
| NVLink HB Bridge 3-slot | ~$50 |
| **Hardware total** | **$850** |
| Monthly power (2 cards, 8hr/day) | $9/mo |
| Claude Pro subscription | $20/mo |
| **Monthly operating cost** | **~$29/mo** |
| **3-year total cost of ownership** | **$850 + $1,044 = $1,894** |
### Comparison: This Build vs Alternatives
| Setup | Cost (3yr) | Best Model | Max Context | Quality |
|-------|-----------|------------|-------------|---------|
| **2x RTX 5000 + $20 Pro** | **$1,894** | Qwen3.5-35B-A3B + Opus | 262K local | **A- local, A+ cloud** |
| 1x RTX 3090 + $20 Pro | $1,420 | Qwen3.5-35B-A3B + Opus | ~128K local | A- local, A+ cloud |
| RTX 8000 (48GB) + $20 Pro | $3,220+ | Qwen3.5-35B-A3B + Opus | 262K+ local | A- local, A+ cloud |
| Claude Max only (no GPU) | $3,600 | Opus 4.6 | 200K | A+ cloud only |
| API-only (Opus heavy use) | $18,000+ | Opus 4.6 | 200K | A+ cloud only |
### Phased Build Plan
**Phase 1 — Start with one card ($400)**
1. Buy Quadro RTX 5000 (VCQRTX5000-PB) — ~$400 on eBay
2. Install in any PCIe x16 slot
3. Install llama.cpp or Ollama v0.17+
4. Run Qwen3.5-35B-A3B at Q4_K_L with 64-128K context
5. Already A- quality for coding — test if local inference fits your workflow
**Phase 2 — Add second card + NVLink ($450)**
1. Buy matching Quadro RTX 5000 — ~$400
2. Buy NVLink HB Bridge 3-slot (part: P4934 / 1JF3K / 6FY12AA) — ~$50
3. Install second card in adjacent/nearby x16 slot
4. Connect NVLink bridge
5. Verify with `nvidia-smi nvlink --status`
6. Now running 32GB unified — Qwen3.5-35B-A3B at Q4_K_M with full 262K context
## 24GB GPU Options (Previous Research — Still Valid for Tighter Budgets)
| GPU | VRAM | Price Range | Best Deals | Notes |