Add practical context windows section — usability not ceilings
VRAM must hold weights AND KV cache. What's left after weights is your real context window. For a 10K-line project (~100-150K tokens): - 1 card: 32-100K usable, file-by-file workflow - 2 cards: 80-180K usable, whole-project-in-one-shot workflow - This Opus session: 1M tokens (neither setup comes close) Comparison table vs Claude Opus 1M session for perspective. Keep Pro for the hard stuff, use local for the daily 80%. https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
@@ -252,6 +252,50 @@ Think of it this way:
|
||||
- **1 card**: B+ coding (MoE or small dense models, squeezed context)
|
||||
- **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options)
|
||||
|
||||
### Practical Context Windows (Usability, Not Ceilings)
|
||||
|
||||
Context window "support" is a ceiling, not what you actually get. VRAM must hold both the
|
||||
model weights AND the KV cache. What's left after weights determines your real context.
|
||||
Quality also degrades toward the edges of a model's context window.
|
||||
|
||||
**Reference: A 10,000-line codebase ≈ 100-150K tokens** (varies by language/comments).
|
||||
This Claude Opus session uses a **1 million token** context window for comparison.
|
||||
|
||||
#### 1 Card (16GB) — Practical
|
||||
|
||||
| Model | Weights | Free for KV | **Usable context** | 10K-line project? |
|
||||
|-------|---------|-------------|-------------------|-------------------|
|
||||
| Qwen3.5-35B-A3B (MoE) | ~12GB | ~3GB | **32-50K tokens** | **No — ~1/3 of it** |
|
||||
| Qwen3.5-9B (dense) | ~6GB | ~9GB | **80-100K tokens** | **Mostly — but B+ quality** |
|
||||
| Qwen2.5-Coder-14B | ~10GB | ~5GB | **16-24K tokens** | **No — a few files at a time** |
|
||||
|
||||
**Workflow on 1 card:** You're feeding files in chunks. Good for "fix this function" or
|
||||
"explain this file." Not for "read my whole project and refactor the auth system."
|
||||
|
||||
#### 2 Cards (32GB via NVLink) — Practical
|
||||
|
||||
| Model | Weights | Free for KV | **Usable context** | 10K-line project? |
|
||||
|-------|---------|-------------|-------------------|-------------------|
|
||||
| **Qwen3.5-27B (dense)** | ~17GB | ~14GB | **80-128K tokens** | **Yes — most/all of it at A-** |
|
||||
| Qwen3.5-35B-A3B (MoE) | ~12GB | ~19GB | **128-180K tokens** | **Yes with room to spare (B+)** |
|
||||
| Qwen2.5-Coder-32B | ~20GB | ~11GB | **32-48K tokens** | **Partial — but strong A- on what it sees** |
|
||||
|
||||
**Workflow on 2 cards:** You can dump most/all of a 10K-line project in one shot with the
|
||||
27B dense model. That's the real workflow change — "here's my whole project, find the bug"
|
||||
becomes possible locally.
|
||||
|
||||
#### vs This Claude Session
|
||||
|
||||
| Setup | Usable context | vs Opus 1M | Whole-project workflow? |
|
||||
|-------|---------------|------------|----------------------|
|
||||
| 1x RTX 5000 (best) | ~50-100K | 5-10% | No — file by file |
|
||||
| **2x RTX 5000 (best)** | **~128-180K** | **13-18%** | **Yes — for 10K-line projects** |
|
||||
| Claude Opus (this session) | 1,000K | 100% | Yes — for anything |
|
||||
|
||||
**Neither setup replaces this session** for complex multi-file work across a 50K+ line
|
||||
codebase. That's why you keep Pro. But 2 cards handles the daily "read my project and
|
||||
help me code" workflow locally with no rate limits — and that's 80% of the work.
|
||||
|
||||
### Estimated Inference Speed
|
||||
|
||||
| Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) |
|
||||
|
||||
Reference in New Issue
Block a user