diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md index 14ce3af..e72d231 100644 --- a/docs/gpu-setup-research.md +++ b/docs/gpu-setup-research.md @@ -41,19 +41,20 @@ Best strategy: use local models for routine tasks, save Claude credits for hard | GPU | Arch | Used Price | TDP | Cooling | Tensor Cores | Mem BW | |-----|------|------------|-----|---------|--------------|--------| -| **Quadro RTX 8000** | Turing (2018) | **$750–1,400** | 260W | Passive variant | Yes (576) | 672 GB/s | +| **Quadro RTX 8000** | Turing (2018) | **$2,000–2,900** | 260W | Passive variant | Yes (576) | 672 GB/s | | **A40** | Ampere (2020) | **~$5,050+** | 300W | Passive | Yes (336 3rd-gen) | 696 GB/s | | **RTX A6000** | Ampere (2020) | **~$5,400+** | 300W | Active (blower) | Yes (336 3rd-gen) | 768 GB/s | | **L40** | Ada (2022) | **~$6,500+** | 300W | Passive | Yes (568 4th-gen) | 864 GB/s | | **RTX 6000 Ada** | Ada (2022) | **~$6,500+** | 300W | Active | Yes (568 4th-gen) | 960 GB/s | Sources: eBay active/sold listings, GPUPoet price tracking, Pangoly, CamelCamelCamel (all March 2026) +Note: One outlier RTX 8000 listing at ~$750 exists but is not representative of the market. -### Winner: Quadro RTX 8000 Passive ($750–1,400) +### Cheapest 48GB Option: Quadro RTX 8000 Passive ($2,000–2,900) -The RTX 8000 is **4–5x cheaper** than every other 48GB option. The passive variant is -purpose-built for rack servers — no fan, relies on chassis airflow, designed for 24/7 operation -in 2U/4U systems. +The RTX 8000 is still the cheapest 48GB card — roughly half the price of an A40 and a +third of an A6000. The passive variant is purpose-built for rack servers — no fan, relies +on chassis airflow, designed for 24/7 operation in 2U/4U systems. Key advantages over the P40: - **48GB vs 24GB** — room for models + massive context @@ -61,6 +62,16 @@ Key advantages over the P40: - **NVLink support** — pair two for 96GB combined (100 GB/s bidirectional) - 10W idle power draw +### Cost Reality Check +At $2,000–2,900 the RTX 8000 is a significant investment. The key question: is unified +48GB VRAM worth 4–6x the cost of dual P40s ($400–500)? + +**Yes, if** you need large context windows (32K+) for complex coding — KV cache can't +be split across two GPUs without NVLink (which P40s don't have). + +**No, if** you're mostly doing short-prompt coding tasks and image gen — dual P40s give +you 48GB total (split) at a fraction of the cost, and each card can handle its own workload. + ## RTX 8000 Performance Benchmarks ### LLM Inference (Exllama, 5.0 bpw quantization) @@ -77,6 +88,7 @@ Key advantages over the P40: | Metric | P40 (24GB) | RTX 8000 (48GB) | |--------|-----------|-----------------| +| **Used price** | **$150–320** | **$2,000–2,900** | | 32B model fit | Barely (~2GB free) | Comfortable (~28GB free) | | 32B generation speed | ~5-12 t/s (est.) | ~20-34 t/s | | Max practical context | ~4K tokens | **32K+ tokens** | @@ -128,21 +140,22 @@ though still behind Ampere/Ada cards. ## Recommended Setups -### Best Overall: RTX 8000 Passive ($750–1,400) +### If budget allows ($2,000–2,900): RTX 8000 Passive Single card handles both coding and image gen. 48GB VRAM fits 32B models with massive -context windows. Passive cooling is rack-native. Tensor cores handle FP16 image gen properly. -One card, one slot, simple setup. +context windows (32K+). Passive cooling is rack-native. Tensor cores handle FP16 image gen +properly. One card, one slot, simple setup. The premium buys you unified VRAM = big context. -### Best Overall + Dedicated Image Gen: RTX 8000 + A2000 ($1,000–1,750) +### If budget allows + dedicated image gen ($2,300–3,250): RTX 8000 + A2000 RTX 8000 for coding with full 48GB dedicated to LLM context. A2000 for image gen (3x faster than Turing, 70W, bus-powered, blower cooled). Best separation of concerns — no model swapping needed. -### Budget Alternative: Dual P40 ($400–500) +### Best value ($400–500): Dual P40 Two P40s for 48GB total, but split across cards (can't combine for one model without -tensor parallelism). One for 32B coding (tight fit), one for image gen (slow, needs --force-fp32). +NVLink, which P40s lack). One for 32B coding (tight fit, ~4K context), one for image gen +(slow, needs --force-fp32). **5x cheaper than RTX 8000** but with significant context limitations. -### Cheapest Entry: Single P40 ($200–300) +### Cheapest entry ($200–300): Single P40 Run 32B coding model with very limited context (~4K tokens). Swap to image gen when needed. Good for testing whether local LLM coding works for your workflow before investing more.