root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update GPU research with real March 2026 prices and 32B model analysis

Browse Source 
- Corrected all GPU prices to actual eBay/Newegg listings as of March 22, 2026
- Added analysis of 32B coding models (Qwen2.5-Coder 32B, Qwen3.5 27B) on 24GB VRAM
- Added honest comparison of local LLM quality vs Claude Code
- Revised recommendations: dual P40 ($400-500) or P40+T4 ($400-550)
- Added configuration notes for 32B models, dual-GPU, and newer Qwen3 models
- RTX A4000 at $700+ is too expensive for this build

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 14:21:02 +00:00
parent 14da24c02b
commit a4e0ebc1bd
1 changed files with 133 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+133 -49
View File
@@ -1,35 +1,69 @@
# GPU Setup Research: Rack Server AI Workloads # GPU Setup Research: Rack Server AI Workloads
*Last updated: March 22, 2026*
## Goal ## Goal
Cost-efficient rack-mountable GPU setup for: Cost-efficient rack-mountable GPU setup for:
1. **LLM coding inference** (Ollama + qwen2.5-coder models) 1. **LLM coding inference** — Run 32B parameter coding models (Qwen2.5-Coder 32B, Qwen3 32B) for quality closest to Claude/GPT-4o
2. **Image generation** (ComfyUI / InvokeAI with Stable Diffusion) 2. **Image generation** — ComfyUI / InvokeAI with Stable Diffusion SDXL / Flux
Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack) Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
## Why 24GB VRAM Matters for Coding
32B parameter models are the sweet spot for local coding quality:
- **Qwen2.5-Coder 32B** scores 73.7 on Aider (between GPT-4o at 71% and Claude 3.5 Haiku at 75%)
- Competitive with GPT-4o on EvalPlus, LiveCodeBench, BigCodeBench
- At Q4_K_M quantization (~20GB), fits on 24GB with room for small context
- 14B models are noticeably worse for complex coding tasks
Newer models also fit 24GB:
- **Qwen 3.5 27B** (dense): ties GPT-5 mini on SWE-bench (72.4%), ~16GB at Q4
- **Qwen3-Coder 30B-A3B** (MoE): only 3.3B active params, fast inference, fits easily
- **Qwen2.5-Coder 32B** remains the FIM (autocomplete) king: 92.7% HumanEval
### Honest Assessment: Local vs Claude Code
Nothing local approaches Claude Opus 4.6 quality for complex multi-file agentic coding.
These 32B models are competitive with **GPT-4o** — a tier below Claude Sonnet, two tiers below Opus.
Best strategy: use local models for routine tasks, save Claude credits for hard problems.
## GPU Candidates Compared ## GPU Candidates Compared
| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | Tesla M40 | | Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | RTX A4000 |
|---------|-----------|----------------|----------|-----------| |---------|-----------|----------------|----------|-----------|
| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Maxwell (2015) | | Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Ampere (2020) |
| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 24GB GDDR5 | | VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 16GB GDDR6 |
| Tensor Cores | No | Yes | Yes | No | | Tensor Cores | No | Yes | Yes | Yes |
| TDP | 250W | 70W | 70W | 250W | | TDP | 250W | 70W | 70W | 140W |
| Compute Capability | 6.1 | 8.6 | 7.5 | 5.2 | | Compute Capability | 6.1 | 8.6 | 7.5 | 8.6 |
| Cooling | Passive (needs fan) | Active (blower) | Passive (needs fan) | Passive (needs fan) | | Cooling | Passive (server fans) | Active (blower) | Passive (server fans) | Active (single-slot) |
| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (8-pin) | | Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (6-pin) |
| Used Price (2026) | ~$300 | ~$490 | ~$800+ | ~$100-150 | | PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen4 x16 |
| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen3 x16 | | Can run 32B Q4? | Yes (tight) | No (12GB) | No (16GB) | No (16GB) |
## Current Prices (March 22, 2026)
| GPU | VRAM | Price Range | Best Deals | Notes |
|-----|------|-------------|------------|-------|
| **Tesla P40** | 24GB | $150-320 | Newegg refurb $219-270; eBay used $150-200 | Best VRAM/$ ratio |
| **RTX A2000 12GB** | 12GB | $250-535 | eBay used ~$250-350; one listing at $490 | New retail ~$535 |
| **Tesla T4** | 16GB | $150-350 | eBay used $150-250 | Great power efficiency |
| **RTX A4000** | 16GB | $700-750+ | eBay used ~$700; new $720+ | Too expensive for this build |
Sources: eBay active listings, Newegg, Lowpi.com, Pangoly price history (all checked March 2026)
## Performance Benchmarks ## Performance Benchmarks
### LLM Inference (tokens/sec via Ollama) ### LLM Inference (tokens/sec via Ollama)
| Model | A2000 12GB | P40 24GB | | Model | A2000 12GB | P40 24GB | RTX 3090 24GB (ref) |
|-------|------------|----------| |-------|------------|----------|---------------------|
| qwen2.5:14b | ~21 t/s | ~17 t/s | | qwen2.5:14b | ~21 t/s | ~17 t/s | — |
| llama3.2:3b-Q8 | ~50 t/s | ~40 t/s | | qwen2.5-coder:32b Q4_K_M | Won't fit | ~5-12 t/s (est.) | ~37-40 t/s |
| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | | llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | — |
**P40 reality check for 32B**: The model barely fits (~22GB for weights), leaving only ~2GB for KV cache.
Context window will be severely limited. Expect 5-12 tok/s — usable for short prompts, painful for long sessions.
### Image Generation (ComfyUI) ### Image Generation (ComfyUI)
@@ -55,6 +89,7 @@ Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
- **Cooling**: Passive — relies on server chassis fans (which R720/R730 have) - **Cooling**: Passive — relies on server chassis fans (which R720/R730 have)
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended - **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
- **Natively supported** in these servers - **Natively supported** in these servers
- Users report power-limiting to 140W with little performance impact
### R720 vs R730 ### R720 vs R730
- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound) - R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
@@ -63,46 +98,88 @@ Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
## Setup Options Analysis ## Setup Options Analysis
### Option A: P40 + A2000 (~$800) ### Option A: Dual P40 (~$400-500)
- P40 for LLM coding (24GB fits 14B models) - **P40 #1**: Qwen2.5-Coder 32B (tight fit, 5-12 tok/s, limited context)
- A2000 for image gen (3x faster than P40, tensor cores) - **P40 #2**: Image gen with ComfyUI (`--force-fp32`, ~49s/image SDXL)
- Or: split 32B model across both P40s via tensor parallelism for better speed
- Both are native rack server GPUs (passive, designed for R720/R730)
- Total power: ~500W GPU (can power-limit to ~280W)
- **Best value for 32B + image gen**
### Option B: P40 + A2000 (~$550-810)
- P40 for 32B coding model (24GB, tight but works)
- A2000 for image gen (3x faster than P40, tensor cores, 70W)
- Total power: ~320W GPU - Total power: ~320W GPU
- **Best performance split** - A2000 at $490 is overpriced — shop for $250-350 range
- **Better image gen speed vs Option A**
### Option B: Single P40 (~$300) ### Option C: Single P40 (~$200-300)
- Use for both LLM and image gen - Run 32B coding model OR image gen (not both simultaneously)
- 24GB handles 14B models well - 32B model leaves no room for anything else in VRAM
- Image gen slow but usable (~49s SDXL) - Swap between tasks by unloading/loading models
- **Best budget option**, upgrade later - **Cheapest entry point**, upgrade later
### Option C: Dual P40 (~$600) ### Option D: P40 (coding) + T4 (image gen) (~$400-550)
- One dedicated to LLM, one to image gen - P40: 32B coding model (24GB)
- 500W total GPU power draw - T4: Image gen with tensor cores, 16GB, 70W, passive
- Both need passive cooling (server fans handle this) - T4 is faster than P40 for image gen (Turing tensor cores, native FP16)
- Image gen still slow on P40 - Both passive-cooled = true rack-native
- **Good balance of price and image gen speed**
### Option D: P40 + P4 (~$350)
- P40 for LLM (24GB)
- P4 for light image gen (8GB, passive, 75W)
- P4 limited to SD 1.5 and smaller models
- Low total power: ~325W
## Recommendations ## Recommendations
### Budget Priority (under $500): Single P40 ### If 32B coding quality is the priority: Dual P40 ($400-500)
Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM. Two P40s give you 48GB total. Run 32B coding model on one, image gen on the other.
Image gen works with `--force-fp32` flag. Add a second GPU later. Or split the model across both for faster inference. Both fit natively in R720/R730.
### Performance Priority (~$800): P40 + A2000 ### If image gen speed matters equally: P40 + T4 ($400-550)
Best of both worlds. P40 handles LLM inference with full 24GB VRAM. P40 for 32B coding, T4 for image gen. Both passive-cooled, both rack-native.
A2000 handles image gen 3x faster with modern Ampere architecture. T4 has tensor cores + FP16 for much faster image gen than P40.
Both fit in R720/R730. A2000 blower cooling works well in rack.
### Configuration Notes for local-ai stack ### Cheapest possible: Single P40 ($200-300)
- Set `INVOKEAI_PRECISION=float32` when using P40 for image gen Run 32B coding model with limited context. Swap to image gen when needed.
- Ollama `OLLAMA_NUM_GPU=999` works with both GPUs Upgrade to dual-GPU later.
- For dual GPU: assign specific GPUs via `CUDA_VISIBLE_DEVICES`
- P40 needs `--force-fp32` in ComfyUI launch args ## Configuration Notes for local-ai stack
### For 32B models on P40
```bash
# In Ollama environment
OLLAMA_NUM_GPU=999
OLLAMA_NUM_CTX=4096 # Keep context small to fit in remaining VRAM
OLLAMA_KEEP_ALIVE=24h
# Pull the right quantization
ollama pull qwen2.5-coder:32b-instruct-q4_K_M
```
### For dual-GPU setup
```bash
# Assign GPU 0 to Ollama (coding), GPU 1 to InvokeAI (image gen)
# In docker-compose.yml for Ollama:
CUDA_VISIBLE_DEVICES=0
# In docker-compose.yml for InvokeAI:
CUDA_VISIBLE_DEVICES=1
```
### For P40 image gen
```bash
# InvokeAI
INVOKEAI_PRECISION=float32
# ComfyUI launch args
--force-fp32
```
### For newer coding models (2026)
```bash
# Qwen 3.5 27B — fits easily on 24GB at Q4, better quality than 2.5
ollama pull qwen3.5:27b
# Qwen3-Coder 30B-A3B MoE — fast inference, agentic coding
ollama pull qwen3-coder:30b
```
## Sources ## Sources
- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf) - [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf)
@@ -115,3 +192,10 @@ Both fit in R720/R730. A2000 blower cooling works well in rack.
- [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40) - [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40)
- [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms) - [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms)
- [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/) - [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/)
- [Tesla P40 for Local LLMs 2026](https://like2byte.com/tesla-p40-local-llm-guide/)
- [Best Local LLMs for 24GB VRAM 2026](https://localllm.in/blog/best-local-llms-24gb-vram)
- [Best Coding Models 2026](https://localvram.com/en/guides/best-coding-models/)
- [Ollama VRAM Requirements Guide](https://localllm.in/blog/ollama-vram-requirements-for-local-llms)
- [Local LLMs That Can Replace Claude Code](https://agentnativedev.medium.com/local-llms-that-can-replace-claude-code-6f5b6cac93bf)
- [Qwen2.5-Coder 32B on Ollama](https://ollama.com/library/qwen2.5-coder:32b-instruct-q4_K_M)
- [Qwen2.5-Coder 32B HuggingFace Discussion](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/discussions/28)