diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md new file mode 100644 index 0000000..c0a9483 --- /dev/null +++ b/docs/gpu-setup-research.md @@ -0,0 +1,117 @@ +# GPU Setup Research: Rack Server AI Workloads + +## Goal +Cost-efficient rack-mountable GPU setup for: +1. **LLM coding inference** (Ollama + qwen2.5-coder models) +2. **Image generation** (ComfyUI / InvokeAI with Stable Diffusion) + +Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack) + +## GPU Candidates Compared + +| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | Tesla M40 | +|---------|-----------|----------------|----------|-----------| +| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Maxwell (2015) | +| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 24GB GDDR5 | +| Tensor Cores | No | Yes | Yes | No | +| TDP | 250W | 70W | 70W | 250W | +| Compute Capability | 6.1 | 8.6 | 7.5 | 5.2 | +| Cooling | Passive (needs fan) | Active (blower) | Passive (needs fan) | Passive (needs fan) | +| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (8-pin) | +| Used Price (2026) | ~$300 | ~$490 | ~$800+ | ~$100-150 | +| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen3 x16 | + +## Performance Benchmarks + +### LLM Inference (tokens/sec via Ollama) + +| Model | A2000 12GB | P40 24GB | +|-------|------------|----------| +| qwen2.5:14b | ~21 t/s | ~17 t/s | +| llama3.2:3b-Q8 | ~50 t/s | ~40 t/s | +| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s | + +### Image Generation (ComfyUI) + +| GPU | SDXL 20 steps | Notes | +|-----|---------------|-------| +| P40 | ~49 seconds | Requires `--force-fp32` flag | +| A2000 | ~16 seconds | ~3x faster than P40 | +| RTX 4090 | ~3 seconds | Reference (16x faster than P40) | + +## Rack Server Compatibility + +### RTX A2000 in R720/R730 +- **Physical fit**: Yes. Dual-slot, low-profile, 167mm length +- **Power**: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720) +- **Cooling**: Blower-style fan exhausts out bracket — ideal for rack airflow +- **Requirement**: Dual CPUs needed for GPU PCIe slots +- **Confirmed working** in Dell R740XD (similar architecture) +- Third-party single-slot cooler available from n3rdware for tighter fits + +### Tesla P40 in R720/R730 +- **Physical fit**: Yes. Full-length, single-slot, designed for rack servers +- **Power**: 250W, requires 8-pin aux power. Needs GPU enablement kit (power cables + riser) +- **Cooling**: Passive — relies on server chassis fans (which R720/R730 have) +- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended +- **Natively supported** in these servers + +### R720 vs R730 +- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound) +- R730: PCIe Gen3, generally preferred +- Both support up to 2x double-wide or 4x single-wide GPUs + +## Setup Options Analysis + +### Option A: P40 + A2000 (~$800) +- P40 for LLM coding (24GB fits 14B models) +- A2000 for image gen (3x faster than P40, tensor cores) +- Total power: ~320W GPU +- **Best performance split** + +### Option B: Single P40 (~$300) +- Use for both LLM and image gen +- 24GB handles 14B models well +- Image gen slow but usable (~49s SDXL) +- **Best budget option**, upgrade later + +### Option C: Dual P40 (~$600) +- One dedicated to LLM, one to image gen +- 500W total GPU power draw +- Both need passive cooling (server fans handle this) +- Image gen still slow on P40 + +### Option D: P40 + P4 (~$350) +- P40 for LLM (24GB) +- P4 for light image gen (8GB, passive, 75W) +- P4 limited to SD 1.5 and smaller models +- Low total power: ~325W + +## Recommendations + +### Budget Priority (under $500): Single P40 +Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM. +Image gen works with `--force-fp32` flag. Add a second GPU later. + +### Performance Priority (~$800): P40 + A2000 +Best of both worlds. P40 handles LLM inference with full 24GB VRAM. +A2000 handles image gen 3x faster with modern Ampere architecture. +Both fit in R720/R730. A2000 blower cooling works well in rack. + +### Configuration Notes for local-ai stack +- Set `INVOKEAI_PRECISION=float32` when using P40 for image gen +- Ollama `OLLAMA_NUM_GPU=999` works with both GPUs +- For dual GPU: assign specific GPUs via `CUDA_VISIBLE_DEVICES` +- P40 needs `--force-fp32` in ComfyUI launch args + +## Sources +- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf) +- [Lenovo ThinkSystem RTX A2000 Product Guide](https://lenovopress.lenovo.com/lp1919-thinksystem-nvidia-rtx-a2000-12gb-pcie-active-gpu) +- [n3rdware Single-Slot A2000 Cooler](https://n3rdware.com/gpu-coolers/single-slot-rtx-a2000-cooler) +- [Dell R730 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r730/r730_ompublication/expansion-card-installation-guidelines) +- [Dell R720 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom/expansion-card-installation-guidelines) +- [ComfyUI GPU Benchmarks Discussion](https://github.com/Comfy-Org/ComfyUI/discussions/2970) +- [ComfyUI P40 FP32 Issue](https://github.com/Comfy-Org/ComfyUI/issues/4363) +- [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40) +- [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms) +- [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/)