Six stackable techniques that compound:
1. KV cache quantization (Q8_0 = 2x context, asymmetric K=Q8/V=Q4 = 2.6x)
2. Flash attention (free VRAM + speed, zero quality loss)
3. Host-memory prompt caching (--cram, use the server's 128-384GB RAM)
4. KV to system RAM (-nkvo, last resort, 5-20x slower)
5. Architecture selection (GQA + MoE = tiny KV footprint)
6. NVMe mmap for model loading (fast cold starts, not inference)
Stacked result: single card goes from ~50K to ~130K usable context
with the MoE model. Updated llama.cpp config with all flags.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
VRAM must hold weights AND KV cache. What's left after weights is
your real context window. For a 10K-line project (~100-150K tokens):
- 1 card: 32-100K usable, file-by-file workflow
- 2 cards: 80-180K usable, whole-project-in-one-shot workflow
- This Opus session: 1M tokens (neither setup comes close)
Comparison table vs Claude Opus 1M session for perspective.
Keep Pro for the hard stuff, use local for the daily 80%.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Single card table was showing A- for 35B-A3B which undercut the case
for a second card. Restructured to lead with 16GB (start here) at
honest B+ ratings, then show what 32GB unlocks: dense 27B/32B models
that physically can't fit on 16GB and are genuinely A- quality.
The upgrade isn't about 262K context - it's about accessing better
models (27B dense > 35B MoE with 3B active params).
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The 35B-A3B MoE activates only 3B params per token - quality tracks
active params, making it more like a smart 7B than a true 35B. Revised
rating to B+ to A-. The 27B dense model is the real A- but needs 17GB
weights. Also notes: 262K is a VRAM ceiling not a quality guarantee,
Q4 quantization costs something, and context quality degrades at edges.
Practical high-quality context is more like 64-128K.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Both GPUs must be dedicated to the LLM for 262K context window.
Image gen and LLM run one at a time (swap takes seconds).
Simultaneous only possible with smaller context (~64K on one card).
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The real use case: drop Claude Max ($100/mo), keep Pro ($20/mo), offload
bulk codebase work to local. Local handles the 80% (reading 10K-line
projects, routine fixes, boilerplate) with no rate limits. Pro handles
the hard 20% where Opus quality matters. GPU pays for itself in 13 months,
saves $1,596 over 3 years vs Max.
Also adds: NVLink speed reality check, image gen capabilities (SDXL/Flux
included at no extra cost), hardware longevity estimate (3-5 years),
and updated cost comparison tables.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Complete shopping list with Dell-specific part numbers:
- GPU power cables: 9H6FV / N08NH (~$10-15 each, need 2)
- GPU Riser 3 required for second GPU slot
- Low-profile heatsinks needed on R720 (usually pre-installed on R730)
- 2x 1100W PSUs mandatory, non-redundant mode for full wattage
Documents riser layout, NVLink bridge clearance in 2U, potential issues
(CPU TDP limits, "unsupported" GPU warning, blower noise), and
R720 vs R730 comparison. Total build cost ~$940-960 with Dell parts.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Research and document the $850 budget build: 2x Quadro RTX 5000 (16GB each)
connected via NVLink for 32GB unified VRAM. With Qwen 3.5's Gated Delta
Network architecture (Feb 2026), this setup runs the 35B-A3B MoE model
with full 262K context in ~25GB — the best price-to-capability ratio
for local AI coding available.
Includes:
- Exact NVLink bridge part numbers (RTX 5000 uses unique smaller connector)
- Motherboard/PSU requirements and slot spacing guidance
- llama.cpp and Ollama multi-GPU configuration
- VRAM budget calculations for all Qwen 3.5 model sizes
- Phased build plan (start with 1 card at $400, add second later)
- Updated model table with full Qwen 3.5 family specs
- Cost comparison vs RTX 8000, RTX 3090, Claude Max, and API pricing
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The $750 listing was a single outlier. Actual market price for
Quadro RTX 8000 Passive 48GB is $2,000-2,900 on eBay. Updated
all recommendations and cost comparisons accordingly.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
- Added full 48GB GPU market comparison (RTX 8000, A40, A6000, L40, RTX 6000 Ada)
- Quadro RTX 8000 Passive at $750-1,400 is 4-5x cheaper than alternatives
- Added RTX 8000 LLM benchmarks (34 t/s on 30B models at 8K context)
- Explained why 48GB >> 24GB for coding: context window is the bottleneck
- Added 2026 coding model landscape (Qwen3.5 27B, Qwen3-Coder, etc.)
- Revised recommendations: RTX 8000 as primary, dual P40 as budget alt
- Updated config notes for 48GB (32K context, higher quantization options)
- All prices verified from real listings as of March 22, 2026
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
- Corrected all GPU prices to actual eBay/Newegg listings as of March 22, 2026
- Added analysis of 32B coding models (Qwen2.5-Coder 32B, Qwen3.5 27B) on 24GB VRAM
- Added honest comparison of local LLM quality vs Claude Code
- Revised recommendations: dual P40 ($400-500) or P40+T4 ($400-550)
- Added configuration notes for 32B models, dual-GPU, and newer Qwen3 models
- RTX A4000 at $700+ is too expensive for this build
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Research comparing P40, A2000, T4, and M40 GPUs for LLM inference
and image generation in Dell R720/R730 rack servers. Includes
benchmarks, compatibility notes, pricing, and recommendations.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Claude