Add practical context windows section — usability not ceilings

VRAM must hold weights AND KV cache. What's left after weights is
your real context window. For a 10K-line project (~100-150K tokens):
- 1 card: 32-100K usable, file-by-file workflow
- 2 cards: 80-180K usable, whole-project-in-one-shot workflow
- This Opus session: 1M tokens (neither setup comes close)

Comparison table vs Claude Opus 1M session for perspective.
Keep Pro for the hard stuff, use local for the daily 80%.

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 17:22:47 +00:00
parent f775b704ae
commit 1d78957083
+44
View File
@@ -252,6 +252,50 @@ Think of it this way:
- **1 card**: B+ coding (MoE or small dense models, squeezed context)
- **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options)
### Practical Context Windows (Usability, Not Ceilings)
Context window "support" is a ceiling, not what you actually get. VRAM must hold both the
model weights AND the KV cache. What's left after weights determines your real context.
Quality also degrades toward the edges of a model's context window.
**Reference: A 10,000-line codebase ≈ 100-150K tokens** (varies by language/comments).
This Claude Opus session uses a **1 million token** context window for comparison.
#### 1 Card (16GB) — Practical
| Model | Weights | Free for KV | **Usable context** | 10K-line project? |
|-------|---------|-------------|-------------------|-------------------|
| Qwen3.5-35B-A3B (MoE) | ~12GB | ~3GB | **32-50K tokens** | **No — ~1/3 of it** |
| Qwen3.5-9B (dense) | ~6GB | ~9GB | **80-100K tokens** | **Mostly — but B+ quality** |
| Qwen2.5-Coder-14B | ~10GB | ~5GB | **16-24K tokens** | **No — a few files at a time** |
**Workflow on 1 card:** You're feeding files in chunks. Good for "fix this function" or
"explain this file." Not for "read my whole project and refactor the auth system."
#### 2 Cards (32GB via NVLink) — Practical
| Model | Weights | Free for KV | **Usable context** | 10K-line project? |
|-------|---------|-------------|-------------------|-------------------|
| **Qwen3.5-27B (dense)** | ~17GB | ~14GB | **80-128K tokens** | **Yes — most/all of it at A-** |
| Qwen3.5-35B-A3B (MoE) | ~12GB | ~19GB | **128-180K tokens** | **Yes with room to spare (B+)** |
| Qwen2.5-Coder-32B | ~20GB | ~11GB | **32-48K tokens** | **Partial — but strong A- on what it sees** |
**Workflow on 2 cards:** You can dump most/all of a 10K-line project in one shot with the
27B dense model. That's the real workflow change — "here's my whole project, find the bug"
becomes possible locally.
#### vs This Claude Session
| Setup | Usable context | vs Opus 1M | Whole-project workflow? |
|-------|---------------|------------|----------------------|
| 1x RTX 5000 (best) | ~50-100K | 5-10% | No — file by file |
| **2x RTX 5000 (best)** | **~128-180K** | **13-18%** | **Yes — for 10K-line projects** |
| Claude Opus (this session) | 1,000K | 100% | Yes — for anything |
**Neither setup replaces this session** for complex multi-file work across a 50K+ line
codebase. That's why you keep Pro. But 2 cards handles the daily "read my project and
help me code" workflow locally with no rate limits — and that's 80% of the work.
### Estimated Inference Speed
| Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) |