diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md index c6a96fd..e0d1368 100644 --- a/docs/gpu-setup-research.md +++ b/docs/gpu-setup-research.md @@ -218,32 +218,39 @@ A 3-slot bridge leaves an air gap for proper cooling. this works well in rack airflow. If using a 2U server, measure clearance for the NVLink bridge sitting on top of the cards. A 4U chassis gives the most room. -### What Runs on 32GB Unified (2x RTX 5000 + NVLink) +### What Runs on 16GB (Single RTX 5000 — Start Here) -| Model | Arch | Quant | Weights | Context | Total VRAM | Quality | -|-------|------|-------|---------|---------|------------|---------| -| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **B+ to A-** | -| Qwen3.5-27B | Dense | Q4_K_M | ~17GB | 128K+ | ~25GB | A- | -| Qwen3-Coder-Next (80B/3B active) | MoE | Q4 | ~20GB | 128K | ~28GB | A | -| Qwen2.5-Coder-14B | Dense | Q4_K_M | ~10GB | 128K | ~22GB | B+ | -| Qwen2.5-Coder-14B | Dense | Q8 | ~16GB | 64K | ~28GB | A- | -| Qwen2.5-Coder-32B | Dense | Q4_K_M | ~20GB | 16-24K | ~28GB | A- | +| Model | Quant | Context | Quality | Notes | +|-------|-------|---------|---------|-------| +| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **B+** | MoE, 3B active. Good but VRAM is tight — context may be lower | +| Qwen3.5-9B | Q8 | 128K+ | B+ | Fits comfortably, high quant | +| Qwen3.5-4B | Q8 | 262K | B | Tiny model, long context | +| Qwen2.5-Coder-7B | Q8 | 128K | B | Solid for simple tasks | +| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ | Tight fit, limited context | -**The sweet spot: Qwen3.5-35B-A3B at Q4_K_M with 262K context.** This is a 35B parameter -model with only 3B active at inference (MoE). The Gated Delta Network architecture slashes -KV cache memory. The entire model + full 262K context fits in ~25GB — well within 32GB. +A single card is a solid start — B+ coding with decent context. But 16GB is the ceiling. +You can't run bigger dense models, can't use higher quantization, and context is squeezed. -### What Runs on 16GB (Single RTX 5000 — Phase 1) +### What the Second Card + NVLink Unlocks (32GB) -| Model | Quant | Context | Quality | -|-------|-------|---------|---------| -| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **A-** | -| Qwen3.5-9B | Q8 | 128K+ | B+ | -| Qwen3.5-4B | Q8 | 262K | B | -| Qwen2.5-Coder-7B | Q8 | 128K | B | -| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ | +The second card doesn't just double context — it opens models that **don't fit on 16GB at all:** -Even a single card can run the Qwen3.5-35B-A3B MoE model — just with a smaller context window. +| Model | Arch | Quant | Weights | Context | Total VRAM | Quality | **Why it needs 32GB** | +|-------|------|-------|---------|---------|------------|---------|----------------------| +| **Qwen3.5-27B** | **Dense** | **Q4_K_M** | **~17GB** | **128K+** | **~25GB** | **A-** | **17GB weights won't fit on 16GB** | +| **Qwen2.5-Coder-32B** | **Dense** | **Q4_K_M** | **~20GB** | **16-24K** | **~28GB** | **A-** | **20GB weights won't fit on 16GB** | +| Qwen2.5-Coder-14B | Dense | **Q8** | ~16GB | 64K | ~28GB | A- | Q8 quant = better output, needs 16GB for weights alone | +| Qwen3-Coder-Next (80B) | MoE | Q4 | ~20GB | 128K | ~28GB | A | 20GB weights won't fit on 16GB | +| Qwen3.5-35B-A3B | MoE (3B active) | Q4_K_M | ~12GB | 262K | ~25GB | B+ | Fits on 1 card at reduced context, but 32GB = full 262K + headroom | + +**The real upgrade isn't 262K context — it's access to dense 27B/32B models that are +genuinely A- quality.** The 35B-A3B MoE runs on both setups, but its 3B active params +limit quality. The Qwen3.5-27B dense model uses all 27B params on every token — that's +the quality jump. And its 17GB of weights physically can't fit on a single 16GB card. + +Think of it this way: +- **1 card**: B+ coding (MoE or small dense models, squeezed context) +- **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options) ### Estimated Inference Speed