root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update GPU research with real March 2026 prices and 32B model analysis

Browse Source 
- Corrected all GPU prices to actual eBay/Newegg listings as of March 22, 2026
- Added analysis of 32B coding models (Qwen2.5-Coder 32B, Qwen3.5 27B) on 24GB VRAM
- Added honest comparison of local LLM quality vs Claude Code
- Revised recommendations: dual P40 ($400-500) or P40+T4 ($400-550)
- Added configuration notes for 32B models, dual-GPU, and newer Qwen3 models
- RTX A4000 at $700+ is too expensive for this build

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 14:21:02 +00:00
parent 14da24c02b
commit a4e0ebc1bd
1 changed files with 133 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+133 -49
View File
@@ -1,35 +1,69 @@
# GPU Setup Research: Rack Server AI Workloads
*Last updated: March 22, 2026*
## Goal
Cost-efficient rack-mountable GPU setup for:
1. **LLM coding inference** (Ollama + qwen2.5-coder models)
2. **Image generation** (ComfyUI / InvokeAI with Stable Diffusion)
1. **LLM coding inference** — Run 32B parameter coding models (Qwen2.5-Coder 32B, Qwen3 32B) for quality closest to Claude/GPT-4o
2. **Image generation** — ComfyUI / InvokeAI with Stable Diffusion SDXL / Flux
Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
## Why 24GB VRAM Matters for Coding
32B parameter models are the sweet spot for local coding quality:
- **Qwen2.5-Coder 32B** scores 73.7 on Aider (between GPT-4o at 71% and Claude 3.5 Haiku at 75%)
- Competitive with GPT-4o on EvalPlus, LiveCodeBench, BigCodeBench
- At Q4_K_M quantization (~20GB), fits on 24GB with room for small context
- 14B models are noticeably worse for complex coding tasks
Newer models also fit 24GB:
- **Qwen 3.5 27B** (dense): ties GPT-5 mini on SWE-bench (72.4%), ~16GB at Q4
- **Qwen3-Coder 30B-A3B** (MoE): only 3.3B active params, fast inference, fits easily
- **Qwen2.5-Coder 32B** remains the FIM (autocomplete) king: 92.7% HumanEval
### Honest Assessment: Local vs Claude Code
Nothing local approaches Claude Opus 4.6 quality for complex multi-file agentic coding.
These 32B models are competitive with **GPT-4o** — a tier below Claude Sonnet, two tiers below Opus.
Best strategy: use local models for routine tasks, save Claude credits for hard problems.
## GPU Candidates Compared
| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | Tesla M40 |
| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | RTX A4000 |
|---------|-----------|----------------|----------|-----------|
| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Maxwell (2015) |
| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 24GB GDDR5 |
| Tensor Cores | No | Yes | Yes | No |
| TDP | 250W | 70W | 70W | 250W |
| Compute Capability | 6.1 | 8.6 | 7.5 | 5.2 |
| Cooling | Passive (needs fan) | Active (blower) | Passive (needs fan) | Passive (needs fan) |
| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (8-pin) |
| Used Price (2026) | ~$300 | ~$490 | ~$800+ | ~$100-150 |
| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen3 x16 |
| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Ampere (2020) |
| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 16GB GDDR6 |
| Tensor Cores | No | Yes | Yes | Yes |
| TDP | 250W | 70W | 70W | 140W |
| Compute Capability | 6.1 | 8.6 | 7.5 | 8.6 |
| Cooling | Passive (server fans) | Active (blower) | Passive (server fans) | Active (single-slot) |
| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (6-pin) |
| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen4 x16 |
| Can run 32B Q4? | Yes (tight) | No (12GB) | No (16GB) | No (16GB) |
## Current Prices (March 22, 2026)
| GPU | VRAM | Price Range | Best Deals | Notes |
|-----|------|-------------|------------|-------|
| **Tesla P40** | 24GB | $150-320 | Newegg refurb $219-270; eBay used $150-200 | Best VRAM/$ ratio |
| **RTX A2000 12GB** | 12GB | $250-535 | eBay used ~$250-350; one listing at $490 | New retail ~$535 |
| **Tesla T4** | 16GB | $150-350 | eBay used $150-250 | Great power efficiency |
| **RTX A4000** | 16GB | $700-750+ | eBay used ~$700; new $720+ | Too expensive for this build |
Sources: eBay active listings, Newegg, Lowpi.com, Pangoly price history (all checked March 2026)
## Performance Benchmarks
### LLM Inference (tokens/sec via Ollama)
| Model | A2000 12GB | P40 24GB |
|-------|------------|----------|
| qwen2.5:14b | ~21 t/s | ~17 t/s |
| llama3.2:3b-Q8 | ~50 t/s | ~40 t/s |
| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s |
| Model | A2000 12GB | P40 24GB | RTX 3090 24GB (ref) |
|-------|------------|----------|---------------------|
| qwen2.5:14b | ~21 t/s | ~17 t/s | — |
| qwen2.5-coder:32b Q4_K_M | Won't fit | ~5-12 t/s (est.) | ~37-40 t/s |
| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | — |
**P40 reality check for 32B**: The model barely fits (~22GB for weights), leaving only ~2GB for KV cache.
Context window will be severely limited. Expect 5-12 tok/s — usable for short prompts, painful for long sessions.
### Image Generation (ComfyUI)
@@ -55,6 +89,7 @@ Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
- **Cooling**: Passive — relies on server chassis fans (which R720/R730 have)
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
- **Natively supported** in these servers
- Users report power-limiting to 140W with little performance impact
### R720 vs R730
- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
@@ -63,46 +98,88 @@ Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
## Setup Options Analysis
### Option A: P40 + A2000 (~$800)
- P40 for LLM coding (24GB fits 14B models)
- A2000 for image gen (3x faster than P40, tensor cores)
### Option A: Dual P40 (~$400-500)
- **P40 #1**: Qwen2.5-Coder 32B (tight fit, 5-12 tok/s, limited context)
- **P40 #2**: Image gen with ComfyUI (`--force-fp32`, ~49s/image SDXL)
- Or: split 32B model across both P40s via tensor parallelism for better speed
- Both are native rack server GPUs (passive, designed for R720/R730)
- Total power: ~500W GPU (can power-limit to ~280W)
- **Best value for 32B + image gen**
### Option B: P40 + A2000 (~$550-810)
- P40 for 32B coding model (24GB, tight but works)
- A2000 for image gen (3x faster than P40, tensor cores, 70W)
- Total power: ~320W GPU
- **Best performance split**
- A2000 at $490 is overpriced — shop for $250-350 range
- **Better image gen speed vs Option A**
### Option B: Single P40 (~$300)
- Use for both LLM and image gen
- 24GB handles 14B models well
- Image gen slow but usable (~49s SDXL)
- **Best budget option**, upgrade later
### Option C: Single P40 (~$200-300)
- Run 32B coding model OR image gen (not both simultaneously)
- 32B model leaves no room for anything else in VRAM
- Swap between tasks by unloading/loading models
- **Cheapest entry point**, upgrade later
### Option C: Dual P40 (~$600)
- One dedicated to LLM, one to image gen
- 500W total GPU power draw
- Both need passive cooling (server fans handle this)
- Image gen still slow on P40
### Option D: P40 + P4 (~$350)
- P40 for LLM (24GB)
- P4 for light image gen (8GB, passive, 75W)
- P4 limited to SD 1.5 and smaller models
- Low total power: ~325W
### Option D: P40 (coding) + T4 (image gen) (~$400-550)
- P40: 32B coding model (24GB)
- T4: Image gen with tensor cores, 16GB, 70W, passive
- T4 is faster than P40 for image gen (Turing tensor cores, native FP16)
- Both passive-cooled = true rack-native
- **Good balance of price and image gen speed**
## Recommendations
### Budget Priority (under $500): Single P40
Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM.
Image gen works with `--force-fp32` flag. Add a second GPU later.
### If 32B coding quality is the priority: Dual P40 ($400-500)
Two P40s give you 48GB total. Run 32B coding model on one, image gen on the other.
Or split the model across both for faster inference. Both fit natively in R720/R730.
### Performance Priority (~$800): P40 + A2000
Best of both worlds. P40 handles LLM inference with full 24GB VRAM.
A2000 handles image gen 3x faster with modern Ampere architecture.
Both fit in R720/R730. A2000 blower cooling works well in rack.
### If image gen speed matters equally: P40 + T4 ($400-550)
P40 for 32B coding, T4 for image gen. Both passive-cooled, both rack-native.
T4 has tensor cores + FP16 for much faster image gen than P40.
### Configuration Notes for local-ai stack
- Set `INVOKEAI_PRECISION=float32` when using P40 for image gen
- Ollama `OLLAMA_NUM_GPU=999` works with both GPUs
- For dual GPU: assign specific GPUs via `CUDA_VISIBLE_DEVICES`
- P40 needs `--force-fp32` in ComfyUI launch args
### Cheapest possible: Single P40 ($200-300)
Run 32B coding model with limited context. Swap to image gen when needed.
Upgrade to dual-GPU later.
## Configuration Notes for local-ai stack
### For 32B models on P40
```bash
# In Ollama environment
OLLAMA_NUM_GPU=999
OLLAMA_NUM_CTX=4096 # Keep context small to fit in remaining VRAM
OLLAMA_KEEP_ALIVE=24h
# Pull the right quantization
ollama pull qwen2.5-coder:32b-instruct-q4_K_M
```
### For dual-GPU setup
```bash
# Assign GPU 0 to Ollama (coding), GPU 1 to InvokeAI (image gen)
# In docker-compose.yml for Ollama:
CUDA_VISIBLE_DEVICES=0
# In docker-compose.yml for InvokeAI:
CUDA_VISIBLE_DEVICES=1
```
### For P40 image gen
```bash
# InvokeAI
INVOKEAI_PRECISION=float32
# ComfyUI launch args
--force-fp32
```
### For newer coding models (2026)
```bash
# Qwen 3.5 27B — fits easily on 24GB at Q4, better quality than 2.5
ollama pull qwen3.5:27b
# Qwen3-Coder 30B-A3B MoE — fast inference, agentic coding
ollama pull qwen3-coder:30b
```
## Sources
- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf)
@@ -115,3 +192,10 @@ Both fit in R720/R730. A2000 blower cooling works well in rack.
- [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40)
- [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms)
- [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/)
- [Tesla P40 for Local LLMs 2026](https://like2byte.com/tesla-p40-local-llm-guide/)
- [Best Local LLMs for 24GB VRAM 2026](https://localllm.in/blog/best-local-llms-24gb-vram)
- [Best Coding Models 2026](https://localvram.com/en/guides/best-coding-models/)
- [Ollama VRAM Requirements Guide](https://localllm.in/blog/ollama-vram-requirements-for-local-llms)
- [Local LLMs That Can Replace Claude Code](https://agentnativedev.medium.com/local-llms-that-can-replace-claude-code-6f5b6cac93bf)
- [Qwen2.5-Coder 32B on Ollama](https://ollama.com/library/qwen2.5-coder:32b-instruct-q4_K_M)
- [Qwen2.5-Coder 32B HuggingFace Discussion](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/discussions/28)