root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Major update: 48GB GPU analysis with RTX 8000 as best value

Browse Source 
- Added full 48GB GPU market comparison (RTX 8000, A40, A6000, L40, RTX 6000 Ada)
- Quadro RTX 8000 Passive at $750-1,400 is 4-5x cheaper than alternatives
- Added RTX 8000 LLM benchmarks (34 t/s on 30B models at 8K context)
- Explained why 48GB >> 24GB for coding: context window is the bottleneck
- Added 2026 coding model landscape (Qwen3.5 27B, Qwen3-Coder, etc.)
- Revised recommendations: RTX 8000 as primary, dual P40 as budget alt
- Updated config notes for 48GB (32K context, higher quantization options)
- All prices verified from real listings as of March 22, 2026

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 14:29:58 +00:00
parent a4e0ebc1bd
commit a14a45bce6
1 changed files with 133 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+133 -112
View File
@@ -4,156 +4,176 @@
## Goal ## Goal
Cost-efficient rack-mountable GPU setup for: Cost-efficient rack-mountable GPU setup for:
1. **LLM coding inference** — Run 32B parameter coding models (Qwen2.5-Coder 32B, Qwen3 32B) for quality closest to Claude/GPT-4o 1. **LLM coding inference** — Run 32B+ parameter coding models with maximum context windows
2. **Image generation** — ComfyUI / InvokeAI with Stable Diffusion SDXL / Flux 2. **Image generation** — ComfyUI / InvokeAI with Stable Diffusion SDXL / Flux
Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack) Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
## Why 24GB VRAM Matters for Coding ## Why 48GB VRAM is the Right Target
32B parameter models are the sweet spot for local coding quality: ### The Problem with 24GB
- **Qwen2.5-Coder 32B** scores 73.7 on Aider (between GPT-4o at 71% and Claude 3.5 Haiku at 75%) 32B coding models at Q4_K_M quantization use ~20GB of weights, leaving only ~4GB for KV cache
- Competitive with GPT-4o on EvalPlus, LiveCodeBench, BigCodeBench on a 24GB card. This severely limits context window size — the key ingredient for complex
- At Q4_K_M quantization (~20GB), fits on 24GB with room for small context coding sessions where the model needs to understand your entire codebase.
- 14B models are noticeably worse for complex coding tasks
Newer models also fit 24GB: ### What 48GB Unlocks
- **Qwen 3.5 27B** (dense): ties GPT-5 mini on SWE-bench (72.4%), ~16GB at Q4 - **32B models at higher quantization** (Q6_K/Q8_0) = better output quality
- **Qwen3-Coder 30B-A3B** (MoE): only 3.3B active params, fast inference, fits easily - **28GB+ free for KV cache** = massive context windows (32K+ tokens)
- **Qwen2.5-Coder 32B** remains the FIM (autocomplete) king: 92.7% HumanEval - **70B models** in aggressive quantization (~12 t/s but functional)
- **Simultaneous model loading** — coding model + image gen model at once
- Room for future larger models without hardware changes
## Best Local Coding Models (2026)
| Model | Size at Q4_K_M | Quality | Notes |
|-------|---------------|---------|-------|
| **Qwen2.5-Coder 32B** | ~20GB | 73.7 Aider (≈ GPT-4o) | FIM king, 92.7% HumanEval |
| **Qwen 3.5 27B** | ~16GB | 72.4% SWE-bench (ties GPT-5 mini) | 262K context, multimodal |
| **Qwen3-Coder 30B-A3B** (MoE) | ~18GB | #1 SWE-rebench (64.6%) | Only 3.3B active, very fast |
| **Qwen3-Coder-Next 80B** (MoE) | needs 64GB+ RAM offload | Beats Claude Opus 4.6 on SWE-rebench | Hybrid attention, 256K context |
### Honest Assessment: Local vs Claude Code ### Honest Assessment: Local vs Claude Code
Nothing local approaches Claude Opus 4.6 quality for complex multi-file agentic coding. Nothing local approaches Claude Opus 4.6 quality for complex multi-file agentic coding.
These 32B models are competitive with **GPT-4o** — a tier below Claude Sonnet, two tiers below Opus. These 32B models are competitive with **GPT-4o** — a tier below Claude Sonnet, two tiers below Opus.
Best strategy: use local models for routine tasks, save Claude credits for hard problems. Best strategy: use local models for routine tasks, save Claude credits for hard problems.
## GPU Candidates Compared ## 48GB GPU Market (March 22, 2026 — Real Prices)
| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | RTX A4000 | | GPU | Arch | Used Price | TDP | Cooling | Tensor Cores | Mem BW |
|---------|-----------|----------------|----------|-----------| |-----|------|------------|-----|---------|--------------|--------|
| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Ampere (2020) | | **Quadro RTX 8000** | Turing (2018) | **$750–1,400** | 260W | Passive variant | Yes (576) | 672 GB/s |
| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 16GB GDDR6 | | **A40** | Ampere (2020) | **~$5,050+** | 300W | Passive | Yes (336 3rd-gen) | 696 GB/s |
| Tensor Cores | No | Yes | Yes | Yes | | **RTX A6000** | Ampere (2020) | **~$5,400+** | 300W | Active (blower) | Yes (336 3rd-gen) | 768 GB/s |
| TDP | 250W | 70W | 70W | 140W | | **L40** | Ada (2022) | **~$6,500+** | 300W | Passive | Yes (568 4th-gen) | 864 GB/s |
| Compute Capability | 6.1 | 8.6 | 7.5 | 8.6 | | **RTX 6000 Ada** | Ada (2022) | **~$6,500+** | 300W | Active | Yes (568 4th-gen) | 960 GB/s |
| Cooling | Passive (server fans) | Active (blower) | Passive (server fans) | Active (single-slot) |
| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (6-pin) |
| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen4 x16 |
| Can run 32B Q4? | Yes (tight) | No (12GB) | No (16GB) | No (16GB) |
## Current Prices (March 22, 2026) Sources: eBay active/sold listings, GPUPoet price tracking, Pangoly, CamelCamelCamel (all March 2026)
### Winner: Quadro RTX 8000 Passive ($750–1,400)
The RTX 8000 is **4–5x cheaper** than every other 48GB option. The passive variant is
purpose-built for rack servers — no fan, relies on chassis airflow, designed for 24/7 operation
in 2U/4U systems.
Key advantages over the P40:
- **48GB vs 24GB** — room for models + massive context
- **Has Tensor Cores** (576 Turing) — native FP16, no `--force-fp32` hacks for image gen
- **NVLink support** — pair two for 96GB combined (100 GB/s bidirectional)
- 10W idle power draw
## RTX 8000 Performance Benchmarks
### LLM Inference (Exllama, 5.0 bpw quantization)
| Model | Context | Prompt Processing | Generation |
|-------|---------|-------------------|------------|
| Qwen3 30B-A3B (MoE) | 8K | 950 t/s | **34 t/s** |
| Qwen3 30B-A3B (MoE) | 16K | 673 t/s | **21 t/s** |
| Qwen3 30B-A3B (MoE) | 32K | 345 t/s | **11 t/s** |
| Llama 3.3 70B | short | 36 t/s | **13 t/s** |
| Llama 3.1 8B | — | — | **72 t/s** |
### Compared to P40 (24GB)
| Metric | P40 (24GB) | RTX 8000 (48GB) |
|--------|-----------|-----------------|
| 32B model fit | Barely (~2GB free) | Comfortable (~28GB free) |
| 32B generation speed | ~5-12 t/s (est.) | ~20-34 t/s |
| Max practical context | ~4K tokens | **32K+ tokens** |
| Image gen (SDXL) | ~49s (`--force-fp32`) | Faster (native FP16) |
| Rack server ready | Yes (passive) | Yes (passive variant) |
### Image Generation
The RTX 8000 has Turing Tensor Cores with native FP16 support. Unlike the P40, it does NOT
need `--force-fp32` workarounds. Image gen performance is significantly better than the P40,
though still behind Ampere/Ada cards.
## 24GB GPU Options (Previous Research — Still Valid for Tighter Budgets)
| GPU | VRAM | Price Range | Best Deals | Notes | | GPU | VRAM | Price Range | Best Deals | Notes |
|-----|------|-------------|------------|-------| |-----|------|-------------|------------|-------|
| **Tesla P40** | 24GB | $150-320 | Newegg refurb $219-270; eBay used $150-200 | Best VRAM/$ ratio | | **Tesla P40** | 24GB | $150-320 | Newegg refurb $219-270; eBay used $150-200 | Best VRAM/$ at 24GB |
| **RTX A2000 12GB** | 12GB | $250-535 | eBay used ~$250-350; one listing at $490 | New retail ~$535 | | **RTX A2000 12GB** | 12GB | $250-535 | eBay used ~$250-350; one listing at $490 | Can't run 32B models |
| **Tesla T4** | 16GB | $150-350 | eBay used $150-250 | Great power efficiency | | **Tesla T4** | 16GB | $150-350 | eBay used $150-250 | Great power efficiency |
| **RTX A4000** | 16GB | $700-750+ | eBay used ~$700; new $720+ | Too expensive for this build | | **RTX A4000** | 16GB | $700-750+ | eBay used ~$700; new $720+ | Too expensive for 16GB |
Sources: eBay active listings, Newegg, Lowpi.com, Pangoly price history (all checked March 2026)
## Performance Benchmarks
### LLM Inference (tokens/sec via Ollama)
| Model | A2000 12GB | P40 24GB | RTX 3090 24GB (ref) |
|-------|------------|----------|---------------------|
| qwen2.5:14b | ~21 t/s | ~17 t/s | — |
| qwen2.5-coder:32b Q4_K_M | Won't fit | ~5-12 t/s (est.) | ~37-40 t/s |
| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | — |
**P40 reality check for 32B**: The model barely fits (~22GB for weights), leaving only ~2GB for KV cache.
Context window will be severely limited. Expect 5-12 tok/s — usable for short prompts, painful for long sessions.
### Image Generation (ComfyUI)
| GPU | SDXL 20 steps | Notes |
|-----|---------------|-------|
| P40 | ~49 seconds | Requires `--force-fp32` flag |
| A2000 | ~16 seconds | ~3x faster than P40 |
| RTX 4090 | ~3 seconds | Reference (16x faster than P40) |
## Rack Server Compatibility ## Rack Server Compatibility
### Quadro RTX 8000 Passive in R720/R730
- **Physical fit**: Full-length, dual-slot — fits in GPU riser slots
- **Power**: 260W, requires 8-pin aux power + GPU enablement kit
- **Cooling**: Passive — relies on server chassis fans (same as P40)
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
- **NVLink**: Can pair two RTX 8000s for 96GB combined VRAM
- Very similar physical/power requirements to the Tesla P40
### RTX A2000 in R720/R730 ### RTX A2000 in R720/R730
- **Physical fit**: Yes. Dual-slot, low-profile, 167mm length - **Physical fit**: Yes. Dual-slot, low-profile, 167mm length
- **Power**: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720) - **Power**: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720)
- **Cooling**: Blower-style fan exhausts out bracket — ideal for rack airflow - **Cooling**: Blower-style fan exhausts out bracket — ideal for rack airflow
- **Requirement**: Dual CPUs needed for GPU PCIe slots - **Requirement**: Dual CPUs needed for GPU PCIe slots
- **Confirmed working** in Dell R740XD (similar architecture) - **Confirmed working** in Dell R740XD (similar architecture)
- Third-party single-slot cooler available from n3rdware for tighter fits
### Tesla P40 in R720/R730 ### Tesla P40 in R720/R730
- **Physical fit**: Yes. Full-length, single-slot, designed for rack servers - **Physical fit**: Yes. Full-length, single-slot, designed for rack servers
- **Power**: 250W, requires 8-pin aux power. Needs GPU enablement kit (power cables + riser) - **Power**: 250W, requires 8-pin aux power. Needs GPU enablement kit
- **Cooling**: Passive — relies on server chassis fans (which R720/R730 have) - **Cooling**: Passive — relies on server chassis fans
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended - **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
- **Natively supported** in these servers - **Natively supported** in these servers
- Users report power-limiting to 140W with little performance impact
### R720 vs R730 ### R720 vs R730
- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound) - R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
- R730: PCIe Gen3, generally preferred - R730: PCIe Gen3, generally preferred
- Both support up to 2x double-wide or 4x single-wide GPUs - Both support up to 2x double-wide or 4x single-wide GPUs
## Setup Options Analysis ## Recommended Setups
### Option A: Dual P40 (~$400-500) ### Best Overall: RTX 8000 Passive ($750–1,400)
- **P40 #1**: Qwen2.5-Coder 32B (tight fit, 5-12 tok/s, limited context) Single card handles both coding and image gen. 48GB VRAM fits 32B models with massive
- **P40 #2**: Image gen with ComfyUI (`--force-fp32`, ~49s/image SDXL) context windows. Passive cooling is rack-native. Tensor cores handle FP16 image gen properly.
- Or: split 32B model across both P40s via tensor parallelism for better speed One card, one slot, simple setup.
- Both are native rack server GPUs (passive, designed for R720/R730)
- Total power: ~500W GPU (can power-limit to ~280W)
- **Best value for 32B + image gen**
### Option B: P40 + A2000 (~$550-810) ### Best Overall + Dedicated Image Gen: RTX 8000 + A2000 ($1,000–1,750)
- P40 for 32B coding model (24GB, tight but works) RTX 8000 for coding with full 48GB dedicated to LLM context.
- A2000 for image gen (3x faster than P40, tensor cores, 70W) A2000 for image gen (3x faster than Turing, 70W, bus-powered, blower cooled).
- Total power: ~320W GPU Best separation of concerns — no model swapping needed.
- A2000 at $490 is overpriced — shop for $250-350 range
- **Better image gen speed vs Option A**
### Option C: Single P40 (~$200-300) ### Budget Alternative: Dual P40 ($400–500)
- Run 32B coding model OR image gen (not both simultaneously) Two P40s for 48GB total, but split across cards (can't combine for one model without
- 32B model leaves no room for anything else in VRAM tensor parallelism). One for 32B coding (tight fit), one for image gen (slow, needs --force-fp32).
- Swap between tasks by unloading/loading models
- **Cheapest entry point**, upgrade later
### Option D: P40 (coding) + T4 (image gen) (~$400-550) ### Cheapest Entry: Single P40 ($200–300)
- P40: 32B coding model (24GB) Run 32B coding model with very limited context (~4K tokens). Swap to image gen when needed.
- T4: Image gen with tensor cores, 16GB, 70W, passive Good for testing whether local LLM coding works for your workflow before investing more.
- T4 is faster than P40 for image gen (Turing tensor cores, native FP16)
- Both passive-cooled = true rack-native
- **Good balance of price and image gen speed**
## Recommendations
### If 32B coding quality is the priority: Dual P40 ($400-500)
Two P40s give you 48GB total. Run 32B coding model on one, image gen on the other.
Or split the model across both for faster inference. Both fit natively in R720/R730.
### If image gen speed matters equally: P40 + T4 ($400-550)
P40 for 32B coding, T4 for image gen. Both passive-cooled, both rack-native.
T4 has tensor cores + FP16 for much faster image gen than P40.
### Cheapest possible: Single P40 ($200-300)
Run 32B coding model with limited context. Swap to image gen when needed.
Upgrade to dual-GPU later.
## Configuration Notes for local-ai stack ## Configuration Notes for local-ai stack
### For 32B models on P40 ### For 48GB RTX 8000
```bash
# Ollama — take advantage of the full 48GB
OLLAMA_NUM_GPU=999
OLLAMA_NUM_CTX=32768 # Large context window — 48GB can handle it
OLLAMA_KEEP_ALIVE=24h
# Pull best coding models
ollama pull qwen2.5-coder:32b-instruct-q4_K_M # ~20GB, leaves 28GB for context
ollama pull qwen3.5:27b # ~16GB at Q4, even more context room
ollama pull qwen3-coder:30b # MoE, very fast inference
# Higher quantization for better quality (48GB allows this)
# Look for Q6_K or Q8_0 variants on Ollama for better output quality
```
### For 32B models on P40 (24GB — tight fit)
```bash ```bash
# In Ollama environment
OLLAMA_NUM_GPU=999 OLLAMA_NUM_GPU=999
OLLAMA_NUM_CTX=4096 # Keep context small to fit in remaining VRAM OLLAMA_NUM_CTX=4096 # Keep context small to fit in remaining VRAM
OLLAMA_KEEP_ALIVE=24h OLLAMA_KEEP_ALIVE=24h
# Pull the right quantization
ollama pull qwen2.5-coder:32b-instruct-q4_K_M ollama pull qwen2.5-coder:32b-instruct-q4_K_M
``` ```
### For dual-GPU setup ### For dual-GPU setup (RTX 8000 + A2000 or P40 + anything)
```bash ```bash
# Assign GPU 0 to Ollama (coding), GPU 1 to InvokeAI (image gen) # Assign GPU 0 to Ollama (coding), GPU 1 to InvokeAI (image gen)
# In docker-compose.yml for Ollama: # In docker-compose.yml for Ollama:
@@ -163,7 +183,7 @@ CUDA_VISIBLE_DEVICES=0
CUDA_VISIBLE_DEVICES=1 CUDA_VISIBLE_DEVICES=1
``` ```
### For P40 image gen ### For image gen on P40 (no tensor cores)
```bash ```bash
# InvokeAI # InvokeAI
INVOKEAI_PRECISION=float32 INVOKEAI_PRECISION=float32
@@ -172,30 +192,31 @@ INVOKEAI_PRECISION=float32
--force-fp32 --force-fp32
``` ```
### For newer coding models (2026) ### For image gen on RTX 8000 / A2000 / T4 (has tensor cores)
```bash ```bash
# Qwen 3.5 27B — fits easily on 24GB at Q4, better quality than 2.5 # InvokeAI — native FP16 works fine
ollama pull qwen3.5:27b INVOKEAI_PRECISION=float16
# Qwen3-Coder 30B-A3B MoE — fast inference, agentic coding # ComfyUI — no special flags needed
ollama pull qwen3-coder:30b
``` ```
## Sources ## Sources
- [Quadro RTX 8000 for Local LLMs — Hardware Corner](https://www.hardware-corner.net/guides/quadro-rtx-8000-for-llm/)
- [RTX 8000 Passive — Network Outlet](https://networkoutlet.com/blogs/articles/nvidia-quadro-rtx-8000-48gb-passive-cooling-powering-ai-rendering-server-workloads)
- [LLM Benchmarks on Turing/Ampere GPUs — Stefandroid](https://blog.stefandroid.com/2025/06/02/benchmark-llm-performance-nvidia-gpus.html)
- [NVIDIA A40 Price Tracking — GPUPoet](https://gpupoet.com/gpu/learn/card/nvidia-a40)
- [NVIDIA L40 Price Tracking — GPUPoet](https://gpupoet.com/gpu/learn/card/nvidia-l40)
- [RTX A6000 Price History — CamelCamelCamel](https://camelcamelcamel.com/product/B09BDH8VZV)
- [RTX A6000 Price History — Pangoly](https://pangoly.com/en/price-history/pny-nvidia-quadro-rtx-a6000)
- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf) - [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf)
- [Lenovo ThinkSystem RTX A2000 Product Guide](https://lenovopress.lenovo.com/lp1919-thinksystem-nvidia-rtx-a2000-12gb-pcie-active-gpu) - [Dell R730 Owner's Manual — Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r730/r730_ompublication/expansion-card-installation-guidelines)
- [n3rdware Single-Slot A2000 Cooler](https://n3rdware.com/gpu-coolers/single-slot-rtx-a2000-cooler) - [Dell R720 Owner's Manual — Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom/expansion-card-installation-guidelines)
- [Dell R730 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r730/r730_ompublication/expansion-card-installation-guidelines)
- [Dell R720 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom/expansion-card-installation-guidelines)
- [ComfyUI GPU Benchmarks Discussion](https://github.com/Comfy-Org/ComfyUI/discussions/2970) - [ComfyUI GPU Benchmarks Discussion](https://github.com/Comfy-Org/ComfyUI/discussions/2970)
- [ComfyUI P40 FP32 Issue](https://github.com/Comfy-Org/ComfyUI/issues/4363) - [ComfyUI P40 FP32 Issue](https://github.com/Comfy-Org/ComfyUI/issues/4363)
- [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40)
- [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms)
- [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/)
- [Tesla P40 for Local LLMs 2026](https://like2byte.com/tesla-p40-local-llm-guide/)
- [Best Local LLMs for 24GB VRAM 2026](https://localllm.in/blog/best-local-llms-24gb-vram) - [Best Local LLMs for 24GB VRAM 2026](https://localllm.in/blog/best-local-llms-24gb-vram)
- [Best Coding Models 2026](https://localvram.com/en/guides/best-coding-models/) - [Best Coding Models 2026](https://localvram.com/en/guides/best-coding-models/)
- [Ollama VRAM Requirements Guide](https://localllm.in/blog/ollama-vram-requirements-for-local-llms) - [Ollama VRAM Requirements Guide](https://localllm.in/blog/ollama-vram-requirements-for-local-llms)
- [Local LLMs That Can Replace Claude Code](https://agentnativedev.medium.com/local-llms-that-can-replace-claude-code-6f5b6cac93bf) - [Local LLMs That Can Replace Claude Code](https://agentnativedev.medium.com/local-llms-that-can-replace-claude-code-6f5b6cac93bf)
- [7 Local LLM Families to Replace Claude/Codex](https://agentnativedev.medium.com/7-local-llm-families-to-replace-claude-codex-for-everyday-tasks-25ba74c3635d)
- [Qwen2.5-Coder 32B on Ollama](https://ollama.com/library/qwen2.5-coder:32b-instruct-q4_K_M) - [Qwen2.5-Coder 32B on Ollama](https://ollama.com/library/qwen2.5-coder:32b-instruct-q4_K_M)
- [Qwen2.5-Coder 32B HuggingFace Discussion](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/discussions/28) - [Qwen3-Coder — How to Run Locally](https://unsloth.ai/docs/models/qwen3-coder-how-to-run-locally)