root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix misleading single vs dual card comparison

Browse Source 
Single card table was showing A- for 35B-A3B which undercut the case
for a second card. Restructured to lead with 16GB (start here) at
honest B+ ratings, then show what 32GB unlocks: dense 27B/32B models
that physically can't fit on 16GB and are genuinely A- quality.
The upgrade isn't about 262K context - it's about accessing better
models (27B dense > 35B MoE with 3B active params).

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 17:15:00 +00:00
parent 6d6a47e028
commit f775b704ae
1 changed files with 28 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+28 -21
View File
@@ -218,32 +218,39 @@ A 3-slot bridge leaves an air gap for proper cooling.
this works well in rack airflow. If using a 2U server, measure clearance for the NVLink
bridge sitting on top of the cards. A 4U chassis gives the most room.
### What Runs on 32GB Unified (2x RTX 5000 + NVLink)
### What Runs on 16GB (Single RTX 5000 — Start Here)
| Model | Arch | Quant | Weights | Context | Total VRAM | Quality |
|-------|------|-------|---------|---------|------------|---------|
| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **B+ to A-** |
| Qwen3.5-27B | Dense | Q4_K_M | ~17GB | 128K+ | ~25GB | A- |
| Qwen3-Coder-Next (80B/3B active) | MoE | Q4 | ~20GB | 128K | ~28GB | A |
| Qwen2.5-Coder-14B | Dense | Q4_K_M | ~10GB | 128K | ~22GB | B+ |
| Qwen2.5-Coder-14B | Dense | Q8 | ~16GB | 64K | ~28GB | A- |
| Qwen2.5-Coder-32B | Dense | Q4_K_M | ~20GB | 16-24K | ~28GB | A- |
| Model | Quant | Context | Quality | Notes |
|-------|-------|---------|---------|-------|
| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **B+** | MoE, 3B active. Good but VRAM is tight — context may be lower |
| Qwen3.5-9B | Q8 | 128K+ | B+ | Fits comfortably, high quant |
| Qwen3.5-4B | Q8 | 262K | B | Tiny model, long context |
| Qwen2.5-Coder-7B | Q8 | 128K | B | Solid for simple tasks |
| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ | Tight fit, limited context |
**The sweet spot: Qwen3.5-35B-A3B at Q4_K_M with 262K context.** This is a 35B parameter
model with only 3B active at inference (MoE). The Gated Delta Network architecture slashes
KV cache memory. The entire model + full 262K context fits in ~25GB — well within 32GB.
A single card is a solid start — B+ coding with decent context. But 16GB is the ceiling.
You can't run bigger dense models, can't use higher quantization, and context is squeezed.
### What Runs on 16GB (Single RTX 5000 — Phase 1)
### What the Second Card + NVLink Unlocks (32GB)
| Model | Quant | Context | Quality |
|-------|-------|---------|---------|
| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **A-** |
| Qwen3.5-9B | Q8 | 128K+ | B+ |
| Qwen3.5-4B | Q8 | 262K | B |
| Qwen2.5-Coder-7B | Q8 | 128K | B |
| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ |
The second card doesn't just double context — it opens models that **don't fit on 16GB at all:**
Even a single card can run the Qwen3.5-35B-A3B MoE model — just with a smaller context window.
| Model | Arch | Quant | Weights | Context | Total VRAM | Quality | **Why it needs 32GB** |
|-------|------|-------|---------|---------|------------|---------|----------------------|
| **Qwen3.5-27B** | **Dense** | **Q4_K_M** | **~17GB** | **128K+** | **~25GB** | **A-** | **17GB weights won't fit on 16GB** |
| **Qwen2.5-Coder-32B** | **Dense** | **Q4_K_M** | **~20GB** | **16-24K** | **~28GB** | **A-** | **20GB weights won't fit on 16GB** |
| Qwen2.5-Coder-14B | Dense | **Q8** | ~16GB | 64K | ~28GB | A- | Q8 quant = better output, needs 16GB for weights alone |
| Qwen3-Coder-Next (80B) | MoE | Q4 | ~20GB | 128K | ~28GB | A | 20GB weights won't fit on 16GB |
| Qwen3.5-35B-A3B | MoE (3B active) | Q4_K_M | ~12GB | 262K | ~25GB | B+ | Fits on 1 card at reduced context, but 32GB = full 262K + headroom |
**The real upgrade isn't 262K context — it's access to dense 27B/32B models that are
genuinely A- quality.** The 35B-A3B MoE runs on both setups, but its 3B active params
limit quality. The Qwen3.5-27B dense model uses all 27B params on every token — that's
the quality jump. And its 17GB of weights physically can't fit on a single 16GB card.
Think of it this way:
- **1 card**: B+ coding (MoE or small dense models, squeezed context)
- **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options)
### Estimated Inference Speed