diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md index c0a9483..cfcdd58 100644 --- a/docs/gpu-setup-research.md +++ b/docs/gpu-setup-research.md @@ -1,35 +1,69 @@ # GPU Setup Research: Rack Server AI Workloads +*Last updated: March 22, 2026* + ## Goal Cost-efficient rack-mountable GPU setup for: -1. **LLM coding inference** (Ollama + qwen2.5-coder models) -2. **Image generation** (ComfyUI / InvokeAI with Stable Diffusion) +1. **LLM coding inference** — Run 32B parameter coding models (Qwen2.5-Coder 32B, Qwen3 32B) for quality closest to Claude/GPT-4o +2. **Image generation** — ComfyUI / InvokeAI with Stable Diffusion SDXL / Flux Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack) +## Why 24GB VRAM Matters for Coding + +32B parameter models are the sweet spot for local coding quality: +- **Qwen2.5-Coder 32B** scores 73.7 on Aider (between GPT-4o at 71% and Claude 3.5 Haiku at 75%) +- Competitive with GPT-4o on EvalPlus, LiveCodeBench, BigCodeBench +- At Q4_K_M quantization (~20GB), fits on 24GB with room for small context +- 14B models are noticeably worse for complex coding tasks + +Newer models also fit 24GB: +- **Qwen 3.5 27B** (dense): ties GPT-5 mini on SWE-bench (72.4%), ~16GB at Q4 +- **Qwen3-Coder 30B-A3B** (MoE): only 3.3B active params, fast inference, fits easily +- **Qwen2.5-Coder 32B** remains the FIM (autocomplete) king: 92.7% HumanEval + +### Honest Assessment: Local vs Claude Code +Nothing local approaches Claude Opus 4.6 quality for complex multi-file agentic coding. +These 32B models are competitive with **GPT-4o** — a tier below Claude Sonnet, two tiers below Opus. +Best strategy: use local models for routine tasks, save Claude credits for hard problems. + ## GPU Candidates Compared -| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | Tesla M40 | +| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | RTX A4000 | |---------|-----------|----------------|----------|-----------| -| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Maxwell (2015) | -| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 24GB GDDR5 | -| Tensor Cores | No | Yes | Yes | No | -| TDP | 250W | 70W | 70W | 250W | -| Compute Capability | 6.1 | 8.6 | 7.5 | 5.2 | -| Cooling | Passive (needs fan) | Active (blower) | Passive (needs fan) | Passive (needs fan) | -| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (8-pin) | -| Used Price (2026) | ~$300 | ~$490 | ~$800+ | ~$100-150 | -| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen3 x16 | +| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Ampere (2020) | +| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 16GB GDDR6 | +| Tensor Cores | No | Yes | Yes | Yes | +| TDP | 250W | 70W | 70W | 140W | +| Compute Capability | 6.1 | 8.6 | 7.5 | 8.6 | +| Cooling | Passive (server fans) | Active (blower) | Passive (server fans) | Active (single-slot) | +| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (6-pin) | +| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen4 x16 | +| Can run 32B Q4? | Yes (tight) | No (12GB) | No (16GB) | No (16GB) | + +## Current Prices (March 22, 2026) + +| GPU | VRAM | Price Range | Best Deals | Notes | +|-----|------|-------------|------------|-------| +| **Tesla P40** | 24GB | $150-320 | Newegg refurb $219-270; eBay used $150-200 | Best VRAM/$ ratio | +| **RTX A2000 12GB** | 12GB | $250-535 | eBay used ~$250-350; one listing at $490 | New retail ~$535 | +| **Tesla T4** | 16GB | $150-350 | eBay used $150-250 | Great power efficiency | +| **RTX A4000** | 16GB | $700-750+ | eBay used ~$700; new $720+ | Too expensive for this build | + +Sources: eBay active listings, Newegg, Lowpi.com, Pangoly price history (all checked March 2026) ## Performance Benchmarks ### LLM Inference (tokens/sec via Ollama) -| Model | A2000 12GB | P40 24GB | -|-------|------------|----------| -| qwen2.5:14b | ~21 t/s | ~17 t/s | -| llama3.2:3b-Q8 | ~50 t/s | ~40 t/s | -| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | +| Model | A2000 12GB | P40 24GB | RTX 3090 24GB (ref) | +|-------|------------|----------|---------------------| +| qwen2.5:14b | ~21 t/s | ~17 t/s | — | +| qwen2.5-coder:32b Q4_K_M | Won't fit | ~5-12 t/s (est.) | ~37-40 t/s | +| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | — | + +**P40 reality check for 32B**: The model barely fits (~22GB for weights), leaving only ~2GB for KV cache. +Context window will be severely limited. Expect 5-12 tok/s — usable for short prompts, painful for long sessions. ### Image Generation (ComfyUI) @@ -55,6 +89,7 @@ Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack) - **Cooling**: Passive — relies on server chassis fans (which R720/R730 have) - **Requirement**: Dual CPUs, redundant 1100W PSUs recommended - **Natively supported** in these servers +- Users report power-limiting to 140W with little performance impact ### R720 vs R730 - R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound) @@ -63,46 +98,88 @@ Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack) ## Setup Options Analysis -### Option A: P40 + A2000 (~$800) -- P40 for LLM coding (24GB fits 14B models) -- A2000 for image gen (3x faster than P40, tensor cores) +### Option A: Dual P40 (~$400-500) +- **P40 #1**: Qwen2.5-Coder 32B (tight fit, 5-12 tok/s, limited context) +- **P40 #2**: Image gen with ComfyUI (`--force-fp32`, ~49s/image SDXL) +- Or: split 32B model across both P40s via tensor parallelism for better speed +- Both are native rack server GPUs (passive, designed for R720/R730) +- Total power: ~500W GPU (can power-limit to ~280W) +- **Best value for 32B + image gen** + +### Option B: P40 + A2000 (~$550-810) +- P40 for 32B coding model (24GB, tight but works) +- A2000 for image gen (3x faster than P40, tensor cores, 70W) - Total power: ~320W GPU -- **Best performance split** +- A2000 at $490 is overpriced — shop for $250-350 range +- **Better image gen speed vs Option A** -### Option B: Single P40 (~$300) -- Use for both LLM and image gen -- 24GB handles 14B models well -- Image gen slow but usable (~49s SDXL) -- **Best budget option**, upgrade later +### Option C: Single P40 (~$200-300) +- Run 32B coding model OR image gen (not both simultaneously) +- 32B model leaves no room for anything else in VRAM +- Swap between tasks by unloading/loading models +- **Cheapest entry point**, upgrade later -### Option C: Dual P40 (~$600) -- One dedicated to LLM, one to image gen -- 500W total GPU power draw -- Both need passive cooling (server fans handle this) -- Image gen still slow on P40 - -### Option D: P40 + P4 (~$350) -- P40 for LLM (24GB) -- P4 for light image gen (8GB, passive, 75W) -- P4 limited to SD 1.5 and smaller models -- Low total power: ~325W +### Option D: P40 (coding) + T4 (image gen) (~$400-550) +- P40: 32B coding model (24GB) +- T4: Image gen with tensor cores, 16GB, 70W, passive +- T4 is faster than P40 for image gen (Turing tensor cores, native FP16) +- Both passive-cooled = true rack-native +- **Good balance of price and image gen speed** ## Recommendations -### Budget Priority (under $500): Single P40 -Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM. -Image gen works with `--force-fp32` flag. Add a second GPU later. +### If 32B coding quality is the priority: Dual P40 ($400-500) +Two P40s give you 48GB total. Run 32B coding model on one, image gen on the other. +Or split the model across both for faster inference. Both fit natively in R720/R730. -### Performance Priority (~$800): P40 + A2000 -Best of both worlds. P40 handles LLM inference with full 24GB VRAM. -A2000 handles image gen 3x faster with modern Ampere architecture. -Both fit in R720/R730. A2000 blower cooling works well in rack. +### If image gen speed matters equally: P40 + T4 ($400-550) +P40 for 32B coding, T4 for image gen. Both passive-cooled, both rack-native. +T4 has tensor cores + FP16 for much faster image gen than P40. -### Configuration Notes for local-ai stack -- Set `INVOKEAI_PRECISION=float32` when using P40 for image gen -- Ollama `OLLAMA_NUM_GPU=999` works with both GPUs -- For dual GPU: assign specific GPUs via `CUDA_VISIBLE_DEVICES` -- P40 needs `--force-fp32` in ComfyUI launch args +### Cheapest possible: Single P40 ($200-300) +Run 32B coding model with limited context. Swap to image gen when needed. +Upgrade to dual-GPU later. + +## Configuration Notes for local-ai stack + +### For 32B models on P40 +```bash +# In Ollama environment +OLLAMA_NUM_GPU=999 +OLLAMA_NUM_CTX=4096 # Keep context small to fit in remaining VRAM +OLLAMA_KEEP_ALIVE=24h + +# Pull the right quantization +ollama pull qwen2.5-coder:32b-instruct-q4_K_M +``` + +### For dual-GPU setup +```bash +# Assign GPU 0 to Ollama (coding), GPU 1 to InvokeAI (image gen) +# In docker-compose.yml for Ollama: +CUDA_VISIBLE_DEVICES=0 + +# In docker-compose.yml for InvokeAI: +CUDA_VISIBLE_DEVICES=1 +``` + +### For P40 image gen +```bash +# InvokeAI +INVOKEAI_PRECISION=float32 + +# ComfyUI launch args +--force-fp32 +``` + +### For newer coding models (2026) +```bash +# Qwen 3.5 27B — fits easily on 24GB at Q4, better quality than 2.5 +ollama pull qwen3.5:27b + +# Qwen3-Coder 30B-A3B MoE — fast inference, agentic coding +ollama pull qwen3-coder:30b +``` ## Sources - [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf) @@ -115,3 +192,10 @@ Both fit in R720/R730. A2000 blower cooling works well in rack. - [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40) - [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms) - [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/) +- [Tesla P40 for Local LLMs 2026](https://like2byte.com/tesla-p40-local-llm-guide/) +- [Best Local LLMs for 24GB VRAM 2026](https://localllm.in/blog/best-local-llms-24gb-vram) +- [Best Coding Models 2026](https://localvram.com/en/guides/best-coding-models/) +- [Ollama VRAM Requirements Guide](https://localllm.in/blog/ollama-vram-requirements-for-local-llms) +- [Local LLMs That Can Replace Claude Code](https://agentnativedev.medium.com/local-llms-that-can-replace-claude-code-6f5b6cac93bf) +- [Qwen2.5-Coder 32B on Ollama](https://ollama.com/library/qwen2.5-coder:32b-instruct-q4_K_M) +- [Qwen2.5-Coder 32B HuggingFace Discussion](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/discussions/28)