diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md index e0d1368..714e6ba 100644 --- a/docs/gpu-setup-research.md +++ b/docs/gpu-setup-research.md @@ -252,6 +252,50 @@ Think of it this way: - **1 card**: B+ coding (MoE or small dense models, squeezed context) - **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options) +### Practical Context Windows (Usability, Not Ceilings) + +Context window "support" is a ceiling, not what you actually get. VRAM must hold both the +model weights AND the KV cache. What's left after weights determines your real context. +Quality also degrades toward the edges of a model's context window. + +**Reference: A 10,000-line codebase ≈ 100-150K tokens** (varies by language/comments). +This Claude Opus session uses a **1 million token** context window for comparison. + +#### 1 Card (16GB) — Practical + +| Model | Weights | Free for KV | **Usable context** | 10K-line project? | +|-------|---------|-------------|-------------------|-------------------| +| Qwen3.5-35B-A3B (MoE) | ~12GB | ~3GB | **32-50K tokens** | **No — ~1/3 of it** | +| Qwen3.5-9B (dense) | ~6GB | ~9GB | **80-100K tokens** | **Mostly — but B+ quality** | +| Qwen2.5-Coder-14B | ~10GB | ~5GB | **16-24K tokens** | **No — a few files at a time** | + +**Workflow on 1 card:** You're feeding files in chunks. Good for "fix this function" or +"explain this file." Not for "read my whole project and refactor the auth system." + +#### 2 Cards (32GB via NVLink) — Practical + +| Model | Weights | Free for KV | **Usable context** | 10K-line project? | +|-------|---------|-------------|-------------------|-------------------| +| **Qwen3.5-27B (dense)** | ~17GB | ~14GB | **80-128K tokens** | **Yes — most/all of it at A-** | +| Qwen3.5-35B-A3B (MoE) | ~12GB | ~19GB | **128-180K tokens** | **Yes with room to spare (B+)** | +| Qwen2.5-Coder-32B | ~20GB | ~11GB | **32-48K tokens** | **Partial — but strong A- on what it sees** | + +**Workflow on 2 cards:** You can dump most/all of a 10K-line project in one shot with the +27B dense model. That's the real workflow change — "here's my whole project, find the bug" +becomes possible locally. + +#### vs This Claude Session + +| Setup | Usable context | vs Opus 1M | Whole-project workflow? | +|-------|---------------|------------|----------------------| +| 1x RTX 5000 (best) | ~50-100K | 5-10% | No — file by file | +| **2x RTX 5000 (best)** | **~128-180K** | **13-18%** | **Yes — for 10K-line projects** | +| Claude Opus (this session) | 1,000K | 100% | Yes — for anything | + +**Neither setup replaces this session** for complex multi-file work across a 50K+ line +codebase. That's why you keep Pro. But 2 cards handles the daily "read my project and +help me code" workflow locally with no rate limits — and that's 80% of the work. + ### Estimated Inference Speed | Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) |