From 7fa91e98162f560061dae37720b64b7447439a7d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 22 Mar 2026 15:57:24 +0000 Subject: [PATCH] Add Quadro RTX 5000 NVLink budget build guide and Qwen 3.5 models MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Research and document the $850 budget build: 2x Quadro RTX 5000 (16GB each) connected via NVLink for 32GB unified VRAM. With Qwen 3.5's Gated Delta Network architecture (Feb 2026), this setup runs the 35B-A3B MoE model with full 262K context in ~25GB — the best price-to-capability ratio for local AI coding available. Includes: - Exact NVLink bridge part numbers (RTX 5000 uses unique smaller connector) - Motherboard/PSU requirements and slot spacing guidance - llama.cpp and Ollama multi-GPU configuration - VRAM budget calculations for all Qwen 3.5 model sizes - Phased build plan (start with 1 card at $400, add second later) - Updated model table with full Qwen 3.5 family specs - Cost comparison vs RTX 8000, RTX 3090, Claude Max, and API pricing https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu --- docs/gpu-setup-research.md | 228 ++++++++++++++++++++++++++++++++++++- 1 file changed, 227 insertions(+), 1 deletion(-) diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md index e72d231..acafb52 100644 --- a/docs/gpu-setup-research.md +++ b/docs/gpu-setup-research.md @@ -25,10 +25,25 @@ coding sessions where the model needs to understand your entire codebase. ## Best Local Coding Models (2026) +### Qwen 3.5 Family (February 2026 — Gated Delta Networks) + +Architecture breakthrough: 3 of every 4 layers use **linear attention** (O(n) scaling), +drastically reducing KV cache memory. These models need far less VRAM for long contexts +than traditional transformers. + +| Model | Type | Active Params | Size at Q4_K_M | Max Context | Quality | Notes | +|-------|------|---------------|---------------|-------------|---------|-------| +| **Qwen3.5-35B-A3B** | **MoE** | **3B** | **~12GB** | **262K** | **A-** | Best bang for buck — 35B model, 3B active, fits 262K ctx in 25GB | +| **Qwen3.5-27B** | Dense | 27B | ~17GB | 262K | A- | 72.4% SWE-bench, ties GPT-5 mini | +| **Qwen3.5-122B-A10B** | MoE | 10B | ~76GB | 262K | A | Matches GPT-5 mini across the board | +| **Qwen3.5-9B** | Dense | 9B | ~6GB | 262K | B+ | Fits on any modern GPU | +| **Qwen3.5-4B** | Dense | 4B | ~3GB | 262K | B | Tiny but capable | + +### Previous Generation (Still Relevant) + | Model | Size at Q4_K_M | Quality | Notes | |-------|---------------|---------|-------| | **Qwen2.5-Coder 32B** | ~20GB | 73.7 Aider (≈ GPT-4o) | FIM king, 92.7% HumanEval | -| **Qwen 3.5 27B** | ~16GB | 72.4% SWE-bench (ties GPT-5 mini) | 262K context, multimodal | | **Qwen3-Coder 30B-A3B** (MoE) | ~18GB | #1 SWE-rebench (64.6%) | Only 3.3B active, very fast | | **Qwen3-Coder-Next 80B** (MoE) | needs 64GB+ RAM offload | Beats Claude Opus 4.6 on SWE-rebench | Hybrid attention, 256K context | @@ -100,6 +115,217 @@ The RTX 8000 has Turing Tensor Cores with native FP16 support. Unlike the P40, i need `--force-fp32` workarounds. Image gen performance is significantly better than the P40, though still behind Ampere/Ada cards. +## Budget Build: 2x Quadro RTX 5000 + NVLink ($850 Total) + +*The best price-to-capability ratio for local AI coding in 2026.* + +### Why This Works Now + +Qwen 3.5 (February 2026) introduced **Gated Delta Networks** — 3 out of 4 layers use linear +attention (O(n) scaling) instead of quadratic. KV cache memory usage is dramatically lower +than traditional transformers. A 35B MoE model with 262K context now fits in ~25GB VRAM. + +### Hardware + +#### GPU: NVIDIA Quadro RTX 5000 (Turing, TU104) + +| Spec | Value | +|------|-------| +| VRAM | 16GB GDDR6 | +| CUDA Cores | 3072 | +| Tensor Cores | 384 (Gen 2, FP16) | +| TDP | ~230W | +| NVLink | **Yes — 50 GB/s bidirectional** | +| Form Factor | Dual-slot, blower cooler (rack-friendly) | +| PCIe | 3.0 x16 | +| Used Price | **~$400** | +| Part Number | VCQRTX5000-PB | + +#### NVLink Bridge (CRITICAL: RTX 5000 uses a unique smaller connector) + +The Quadro RTX 5000 has a **shorter NVLink connector** than all other Quadro RTX cards. +Bridges from the RTX 6000/8000 will NOT physically fit. You must buy the RTX 5000-specific bridge. + +| Detail | Value | +|--------|-------| +| Product | NVIDIA Quadro RTX 5000 NVLink HB Bridge 2-Slot | +| SKU | NVLINKX8-2SLOT-PB | +| Part Numbers | 1JF3K, 699-54934-0500-000, 900-54934-0100-000, P4934, 6FY12AA, L55997-001 | +| Price | **~$30-80** (eBay, Amazon) | +| Bandwidth | 50 GB/s total (25 GB/s per direction) | +| Sizing | 2-slot (cards adjacent) or 3-slot (one slot gap — better thermals) | + +**Where to buy:** +- eBay: search "Quadro RTX 5000 NVLink" or part numbers P4934 / L55997-001 / 1JF3K +- Amazon: search part number 6FY12AA or 1JF3K + +**WARNING:** The 3-slot bridge is recommended over 2-slot. With a 2-slot bridge the cards +sit directly adjacent — the top card's blower intake gets blocked by the bottom card. +A 3-slot bridge leaves an air gap for proper cooling. + +#### Motherboard Requirements + +| Requirement | Details | +|-------------|---------| +| PCIe slots | Two x16 slots (x8 electrical is fine — LLM inference is VRAM-bound, not PCIe-bound) | +| Slot spacing | Must match your NVLink bridge size (2-slot or 3-slot gap) | +| Power supply | 650W+ minimum (80 PLUS Gold recommended), 850W+ for headroom | +| Power connectors | 2x 8-pin PCIe power (one per card). Do NOT daisy-chain — use separate cables | +| CPU platform | Any modern platform works. Threadripper/Xeon not required | + +**Recommended motherboards (workstation/server):** +- Any board with 2x PCIe x16 slots spaced 2-3 slots apart +- Server: Dell R730/R740 with GPU riser (but verify 3-slot bridge clearance in 2U) +- Workstation: MSI X399 Creation, ASUS WS series, Supermicro X11/X12 boards +- Desktop: Most ATX boards with 2 full-length x16 slots work + +**Rack server note:** The Quadro RTX 5000's blower cooler exhausts out the bracket — +this works well in rack airflow. If using a 2U server, measure clearance for the NVLink +bridge sitting on top of the cards. A 4U chassis gives the most room. + +### What Runs on 32GB Unified (2x RTX 5000 + NVLink) + +| Model | Arch | Quant | Weights | Context | Total VRAM | Quality | +|-------|------|-------|---------|---------|------------|---------| +| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **A-** | +| Qwen3.5-27B | Dense | Q4_K_M | ~17GB | 128K+ | ~25GB | A- | +| Qwen3-Coder-Next (80B/3B active) | MoE | Q4 | ~20GB | 128K | ~28GB | A | +| Qwen2.5-Coder-14B | Dense | Q4_K_M | ~10GB | 128K | ~22GB | B+ | +| Qwen2.5-Coder-14B | Dense | Q8 | ~16GB | 64K | ~28GB | A- | +| Qwen2.5-Coder-32B | Dense | Q4_K_M | ~20GB | 16-24K | ~28GB | A- | + +**The sweet spot: Qwen3.5-35B-A3B at Q4_K_M with 262K context.** This is a 35B parameter +model with only 3B active at inference (MoE). The Gated Delta Network architecture slashes +KV cache memory. The entire model + full 262K context fits in ~25GB — well within 32GB. + +### What Runs on 16GB (Single RTX 5000 — Phase 1) + +| Model | Quant | Context | Quality | +|-------|-------|---------|---------| +| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **A-** | +| Qwen3.5-9B | Q8 | 128K+ | B+ | +| Qwen3.5-4B | Q8 | 262K | B | +| Qwen2.5-Coder-7B | Q8 | 128K | B | +| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ | + +Even a single card can run the Qwen3.5-35B-A3B MoE model — just with a smaller context window. + +### Estimated Inference Speed + +| Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) | +|-------|-------------|---------------------| +| Qwen3.5-35B-A3B Q4 (short ctx) | ~25-35 tok/s | ~25-35 tok/s | +| Qwen3.5-35B-A3B Q4 (128K ctx) | ~10-18 tok/s | ~15-25 tok/s | +| Qwen3.5-35B-A3B Q4 (262K ctx) | Won't fit | ~10-18 tok/s | +| Qwen2.5-Coder-14B Q4 | ~20-30 tok/s | ~25-35 tok/s | + +NVLink matters most at large context windows where KV cache spans both cards. +At short contexts that fit on one card, the second GPU adds less benefit. + +### Power Consumption & Cost + +| Config | Idle | Load | Monthly (8hr/day @ $0.09/kWh) | Annual | +|--------|------|------|-------------------------------|--------| +| 1x Quadro RTX 5000 | ~15W | ~210W | **~$4.50** | ~$54 | +| 2x Quadro RTX 5000 | ~30W | ~420W | **~$9.00** | ~$108 | + +### Software Setup + +#### llama.cpp (Recommended — Best Multi-GPU Support) + +```bash +# Build with CUDA support +git clone https://github.com/ggerganov/llama.cpp +cd llama.cpp +cmake -B build -DGGML_CUDA=ON +cmake --build build --config Release -j$(nproc) + +# Download Qwen3.5-35B-A3B GGUF (Q4_K_M) +# Get from: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF + +# Run on dual GPU with NVLink +./build/bin/llama-server \ + -m Qwen3.5-35B-A3B-Q4_K_M.gguf \ + -ngl 999 \ + -c 262144 \ + --host 0.0.0.0 \ + --port 8080 + +# llama.cpp auto-detects NVLink and splits layers across both GPUs +# Use -ts 1,1 to manually set equal split if needed +``` + +#### Ollama + +```bash +# Requires Ollama v0.17+ for Qwen3.5 support +# NOTE: As of March 2026, some Qwen3.5 GGUFs have compatibility issues +# with Ollama due to mmproj vision files. llama.cpp may be more reliable. + +# Environment variables for multi-GPU +export OLLAMA_GPU_SPLIT=16,16 # Equal split across both 16GB cards +export OLLAMA_KV_CACHE_TYPE=q8_0 # Halves KV cache VRAM with minimal quality loss +export OLLAMA_KEEP_ALIVE=24h # Keep model loaded in VRAM +export OLLAMA_FLASH_ATTENTION=1 # Enable flash attention for VRAM savings + +# Pull and run +ollama pull qwen3.5:35b-a3b-q4_K_M +ollama run qwen3.5:35b-a3b-q4_K_M +``` + +#### Verify NVLink Is Working + +```bash +# Check NVLink status +nvidia-smi nvlink --status + +# Check NVLink bandwidth +nvidia-smi nvlink -gt d + +# Monitor both GPUs during inference +watch -n 0.5 nvidia-smi +``` + +### Total Cost Summary + +| Item | Cost | +|------|------| +| 1x Quadro RTX 5000 (Phase 1) | $400 | +| 1x Quadro RTX 5000 (Phase 2) | $400 | +| NVLink HB Bridge 3-slot | ~$50 | +| **Hardware total** | **$850** | +| Monthly power (2 cards, 8hr/day) | $9/mo | +| Claude Pro subscription | $20/mo | +| **Monthly operating cost** | **~$29/mo** | +| **3-year total cost of ownership** | **$850 + $1,044 = $1,894** | + +### Comparison: This Build vs Alternatives + +| Setup | Cost (3yr) | Best Model | Max Context | Quality | +|-------|-----------|------------|-------------|---------| +| **2x RTX 5000 + $20 Pro** | **$1,894** | Qwen3.5-35B-A3B + Opus | 262K local | **A- local, A+ cloud** | +| 1x RTX 3090 + $20 Pro | $1,420 | Qwen3.5-35B-A3B + Opus | ~128K local | A- local, A+ cloud | +| RTX 8000 (48GB) + $20 Pro | $3,220+ | Qwen3.5-35B-A3B + Opus | 262K+ local | A- local, A+ cloud | +| Claude Max only (no GPU) | $3,600 | Opus 4.6 | 200K | A+ cloud only | +| API-only (Opus heavy use) | $18,000+ | Opus 4.6 | 200K | A+ cloud only | + +### Phased Build Plan + +**Phase 1 — Start with one card ($400)** +1. Buy Quadro RTX 5000 (VCQRTX5000-PB) — ~$400 on eBay +2. Install in any PCIe x16 slot +3. Install llama.cpp or Ollama v0.17+ +4. Run Qwen3.5-35B-A3B at Q4_K_L with 64-128K context +5. Already A- quality for coding — test if local inference fits your workflow + +**Phase 2 — Add second card + NVLink ($450)** +1. Buy matching Quadro RTX 5000 — ~$400 +2. Buy NVLink HB Bridge 3-slot (part: P4934 / 1JF3K / 6FY12AA) — ~$50 +3. Install second card in adjacent/nearby x16 slot +4. Connect NVLink bridge +5. Verify with `nvidia-smi nvlink --status` +6. Now running 32GB unified — Qwen3.5-35B-A3B at Q4_K_M with full 262K context + ## 24GB GPU Options (Previous Research — Still Valid for Tighter Budgets) | GPU | VRAM | Price Range | Best Deals | Notes |