root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add GPU setup research for rack server AI workloads

Browse Source 
Research comparing P40, A2000, T4, and M40 GPUs for LLM inference
and image generation in Dell R720/R730 rack servers. Includes
benchmarks, compatibility notes, pricing, and recommendations.

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 14:12:46 +00:00
parent 063a7b6db1
commit 14da24c02b
1 changed files with 117 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+117
View File
@@ -0,0 +1,117 @@
# GPU Setup Research: Rack Server AI Workloads
## Goal
Cost-efficient rack-mountable GPU setup for:
1. **LLM coding inference** (Ollama + qwen2.5-coder models)
2. **Image generation** (ComfyUI / InvokeAI with Stable Diffusion)
Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
## GPU Candidates Compared
| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | Tesla M40 |
|---------|-----------|----------------|----------|-----------|
| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Maxwell (2015) |
| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 24GB GDDR5 |
| Tensor Cores | No | Yes | Yes | No |
| TDP | 250W | 70W | 70W | 250W |
| Compute Capability | 6.1 | 8.6 | 7.5 | 5.2 |
| Cooling | Passive (needs fan) | Active (blower) | Passive (needs fan) | Passive (needs fan) |
| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (8-pin) |
| Used Price (2026) | ~$300 | ~$490 | ~$800+ | ~$100-150 |
| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen3 x16 |
## Performance Benchmarks
### LLM Inference (tokens/sec via Ollama)
| Model | A2000 12GB | P40 24GB |
|-------|------------|----------|
| qwen2.5:14b | ~21 t/s | ~17 t/s |
| llama3.2:3b-Q8 | ~50 t/s | ~40 t/s |
| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s |
### Image Generation (ComfyUI)
| GPU | SDXL 20 steps | Notes |
|-----|---------------|-------|
| P40 | ~49 seconds | Requires `--force-fp32` flag |
| A2000 | ~16 seconds | ~3x faster than P40 |
| RTX 4090 | ~3 seconds | Reference (16x faster than P40) |
## Rack Server Compatibility
### RTX A2000 in R720/R730
- **Physical fit**: Yes. Dual-slot, low-profile, 167mm length
- **Power**: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720)
- **Cooling**: Blower-style fan exhausts out bracket — ideal for rack airflow
- **Requirement**: Dual CPUs needed for GPU PCIe slots
- **Confirmed working** in Dell R740XD (similar architecture)
- Third-party single-slot cooler available from n3rdware for tighter fits
### Tesla P40 in R720/R730
- **Physical fit**: Yes. Full-length, single-slot, designed for rack servers
- **Power**: 250W, requires 8-pin aux power. Needs GPU enablement kit (power cables + riser)
- **Cooling**: Passive — relies on server chassis fans (which R720/R730 have)
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
- **Natively supported** in these servers
### R720 vs R730
- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
- R730: PCIe Gen3, generally preferred
- Both support up to 2x double-wide or 4x single-wide GPUs
## Setup Options Analysis
### Option A: P40 + A2000 (~$800)
- P40 for LLM coding (24GB fits 14B models)
- A2000 for image gen (3x faster than P40, tensor cores)
- Total power: ~320W GPU
- **Best performance split**
### Option B: Single P40 (~$300)
- Use for both LLM and image gen
- 24GB handles 14B models well
- Image gen slow but usable (~49s SDXL)
- **Best budget option**, upgrade later
### Option C: Dual P40 (~$600)
- One dedicated to LLM, one to image gen
- 500W total GPU power draw
- Both need passive cooling (server fans handle this)
- Image gen still slow on P40
### Option D: P40 + P4 (~$350)
- P40 for LLM (24GB)
- P4 for light image gen (8GB, passive, 75W)
- P4 limited to SD 1.5 and smaller models
- Low total power: ~325W
## Recommendations
### Budget Priority (under $500): Single P40
Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM.
Image gen works with `--force-fp32` flag. Add a second GPU later.
### Performance Priority (~$800): P40 + A2000
Best of both worlds. P40 handles LLM inference with full 24GB VRAM.
A2000 handles image gen 3x faster with modern Ampere architecture.
Both fit in R720/R730. A2000 blower cooling works well in rack.
### Configuration Notes for local-ai stack
- Set `INVOKEAI_PRECISION=float32` when using P40 for image gen
- Ollama `OLLAMA_NUM_GPU=999` works with both GPUs
- For dual GPU: assign specific GPUs via `CUDA_VISIBLE_DEVICES`
- P40 needs `--force-fp32` in ComfyUI launch args
## Sources
- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf)
- [Lenovo ThinkSystem RTX A2000 Product Guide](https://lenovopress.lenovo.com/lp1919-thinksystem-nvidia-rtx-a2000-12gb-pcie-active-gpu)
- [n3rdware Single-Slot A2000 Cooler](https://n3rdware.com/gpu-coolers/single-slot-rtx-a2000-cooler)
- [Dell R730 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r730/r730_ompublication/expansion-card-installation-guidelines)
- [Dell R720 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom/expansion-card-installation-guidelines)
- [ComfyUI GPU Benchmarks Discussion](https://github.com/Comfy-Org/ComfyUI/discussions/2970)
- [ComfyUI P40 FP32 Issue](https://github.com/Comfy-Org/ComfyUI/issues/4363)
- [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40)
- [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms)
- [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/)