Fix misleading single vs dual card comparison

Single card table was showing A- for 35B-A3B which undercut the case
for a second card. Restructured to lead with 16GB (start here) at
honest B+ ratings, then show what 32GB unlocks: dense 27B/32B models
that physically can't fit on 16GB and are genuinely A- quality.
The upgrade isn't about 262K context - it's about accessing better
models (27B dense > 35B MoE with 3B active params).

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 17:15:00 +00:00
parent 6d6a47e028
commit f775b704ae
+28 -21
View File
@@ -218,32 +218,39 @@ A 3-slot bridge leaves an air gap for proper cooling.
this works well in rack airflow. If using a 2U server, measure clearance for the NVLink this works well in rack airflow. If using a 2U server, measure clearance for the NVLink
bridge sitting on top of the cards. A 4U chassis gives the most room. bridge sitting on top of the cards. A 4U chassis gives the most room.
### What Runs on 32GB Unified (2x RTX 5000 + NVLink) ### What Runs on 16GB (Single RTX 5000 — Start Here)
| Model | Arch | Quant | Weights | Context | Total VRAM | Quality | | Model | Quant | Context | Quality | Notes |
|-------|------|-------|---------|---------|------------|---------| |-------|-------|---------|---------|-------|
| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **B+ to A-** | | **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **B+** | MoE, 3B active. Good but VRAM is tight — context may be lower |
| Qwen3.5-27B | Dense | Q4_K_M | ~17GB | 128K+ | ~25GB | A- | | Qwen3.5-9B | Q8 | 128K+ | B+ | Fits comfortably, high quant |
| Qwen3-Coder-Next (80B/3B active) | MoE | Q4 | ~20GB | 128K | ~28GB | A | | Qwen3.5-4B | Q8 | 262K | B | Tiny model, long context |
| Qwen2.5-Coder-14B | Dense | Q4_K_M | ~10GB | 128K | ~22GB | B+ | | Qwen2.5-Coder-7B | Q8 | 128K | B | Solid for simple tasks |
| Qwen2.5-Coder-14B | Dense | Q8 | ~16GB | 64K | ~28GB | A- | | Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ | Tight fit, limited context |
| Qwen2.5-Coder-32B | Dense | Q4_K_M | ~20GB | 16-24K | ~28GB | A- |
**The sweet spot: Qwen3.5-35B-A3B at Q4_K_M with 262K context.** This is a 35B parameter A single card is a solid start — B+ coding with decent context. But 16GB is the ceiling.
model with only 3B active at inference (MoE). The Gated Delta Network architecture slashes You can't run bigger dense models, can't use higher quantization, and context is squeezed.
KV cache memory. The entire model + full 262K context fits in ~25GB — well within 32GB.
### What Runs on 16GB (Single RTX 5000 — Phase 1) ### What the Second Card + NVLink Unlocks (32GB)
| Model | Quant | Context | Quality | The second card doesn't just double context — it opens models that **don't fit on 16GB at all:**
|-------|-------|---------|---------|
| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **A-** |
| Qwen3.5-9B | Q8 | 128K+ | B+ |
| Qwen3.5-4B | Q8 | 262K | B |
| Qwen2.5-Coder-7B | Q8 | 128K | B |
| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ |
Even a single card can run the Qwen3.5-35B-A3B MoE model — just with a smaller context window. | Model | Arch | Quant | Weights | Context | Total VRAM | Quality | **Why it needs 32GB** |
|-------|------|-------|---------|---------|------------|---------|----------------------|
| **Qwen3.5-27B** | **Dense** | **Q4_K_M** | **~17GB** | **128K+** | **~25GB** | **A-** | **17GB weights won't fit on 16GB** |
| **Qwen2.5-Coder-32B** | **Dense** | **Q4_K_M** | **~20GB** | **16-24K** | **~28GB** | **A-** | **20GB weights won't fit on 16GB** |
| Qwen2.5-Coder-14B | Dense | **Q8** | ~16GB | 64K | ~28GB | A- | Q8 quant = better output, needs 16GB for weights alone |
| Qwen3-Coder-Next (80B) | MoE | Q4 | ~20GB | 128K | ~28GB | A | 20GB weights won't fit on 16GB |
| Qwen3.5-35B-A3B | MoE (3B active) | Q4_K_M | ~12GB | 262K | ~25GB | B+ | Fits on 1 card at reduced context, but 32GB = full 262K + headroom |
**The real upgrade isn't 262K context — it's access to dense 27B/32B models that are
genuinely A- quality.** The 35B-A3B MoE runs on both setups, but its 3B active params
limit quality. The Qwen3.5-27B dense model uses all 27B params on every token — that's
the quality jump. And its 17GB of weights physically can't fit on a single 16GB card.
Think of it this way:
- **1 card**: B+ coding (MoE or small dense models, squeezed context)
- **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options)
### Estimated Inference Speed ### Estimated Inference Speed