# GPU Setup Research: Rack Server AI Workloads

## Goal
Cost-efficient rack-mountable GPU setup for:
1. **LLM coding inference** (Ollama + qwen2.5-coder models)
2. **Image generation** (ComfyUI / InvokeAI with Stable Diffusion)

Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)

## GPU Candidates Compared

| Feature | Tesla P40 | RTX A2000 12GB | Tesla T4 | Tesla M40 |
|---------|-----------|----------------|----------|-----------|
| Architecture | Pascal (2016) | Ampere (2020) | Turing (2018) | Maxwell (2015) |
| VRAM | 24GB GDDR5 | 12GB GDDR6 | 16GB GDDR6 | 24GB GDDR5 |
| Tensor Cores | No | Yes | Yes | No |
| TDP | 250W | 70W | 70W | 250W |
| Compute Capability | 6.1 | 8.6 | 7.5 | 5.2 |
| Cooling | Passive (needs fan) | Active (blower) | Passive (needs fan) | Passive (needs fan) |
| Aux Power Required | Yes (8-pin) | No (bus-powered) | No (bus-powered) | Yes (8-pin) |
| Used Price (2026) | ~$300 | ~$490 | ~$800+ | ~$100-150 |
| PCIe | Gen3 x16 | Gen4 x16 | Gen3 x16 | Gen3 x16 |

## Performance Benchmarks

### LLM Inference (tokens/sec via Ollama)

| Model | A2000 12GB | P40 24GB |
|-------|------------|----------|
| qwen2.5:14b | ~21 t/s | ~17 t/s |
| llama3.2:3b-Q8 | ~50 t/s | ~40 t/s |
| llama3.2:3b-Q4 | ~60 t/s | ~48 t/s |

### Image Generation (ComfyUI)

| GPU | SDXL 20 steps | Notes |
|-----|---------------|-------|
| P40 | ~49 seconds | Requires `--force-fp32` flag |
| A2000 | ~16 seconds | ~3x faster than P40 |
| RTX 4090 | ~3 seconds | Reference (16x faster than P40) |

## Rack Server Compatibility

### RTX A2000 in R720/R730
- **Physical fit**: Yes. Dual-slot, low-profile, 167mm length
- **Power**: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720)
- **Cooling**: Blower-style fan exhausts out bracket — ideal for rack airflow
- **Requirement**: Dual CPUs needed for GPU PCIe slots
- **Confirmed working** in Dell R740XD (similar architecture)
- Third-party single-slot cooler available from n3rdware for tighter fits

### Tesla P40 in R720/R730
- **Physical fit**: Yes. Full-length, single-slot, designed for rack servers
- **Power**: 250W, requires 8-pin aux power. Needs GPU enablement kit (power cables + riser)
- **Cooling**: Passive — relies on server chassis fans (which R720/R730 have)
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
- **Natively supported** in these servers

### R720 vs R730
- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
- R730: PCIe Gen3, generally preferred
- Both support up to 2x double-wide or 4x single-wide GPUs

## Setup Options Analysis

### Option A: P40 + A2000 (~$800)
- P40 for LLM coding (24GB fits 14B models)
- A2000 for image gen (3x faster than P40, tensor cores)
- Total power: ~320W GPU
- **Best performance split**

### Option B: Single P40 (~$300)
- Use for both LLM and image gen
- 24GB handles 14B models well
- Image gen slow but usable (~49s SDXL)
- **Best budget option**, upgrade later

### Option C: Dual P40 (~$600)
- One dedicated to LLM, one to image gen
- 500W total GPU power draw
- Both need passive cooling (server fans handle this)
- Image gen still slow on P40

### Option D: P40 + P4 (~$350)
- P40 for LLM (24GB)
- P4 for light image gen (8GB, passive, 75W)
- P4 limited to SD 1.5 and smaller models
- Low total power: ~325W

## Recommendations

### Budget Priority (under $500): Single P40
Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM.
Image gen works with `--force-fp32` flag. Add a second GPU later.

### Performance Priority (~$800): P40 + A2000
Best of both worlds. P40 handles LLM inference with full 24GB VRAM.
A2000 handles image gen 3x faster with modern Ampere architecture.
Both fit in R720/R730. A2000 blower cooling works well in rack.

### Configuration Notes for local-ai stack
- Set `INVOKEAI_PRECISION=float32` when using P40 for image gen
- Ollama `OLLAMA_NUM_GPU=999` works with both GPUs
- For dual GPU: assign specific GPUs via `CUDA_VISIBLE_DEVICES`
- P40 needs `--force-fp32` in ComfyUI launch args

## Sources
- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf)
- [Lenovo ThinkSystem RTX A2000 Product Guide](https://lenovopress.lenovo.com/lp1919-thinksystem-nvidia-rtx-a2000-12gb-pcie-active-gpu)
- [n3rdware Single-Slot A2000 Cooler](https://n3rdware.com/gpu-coolers/single-slot-rtx-a2000-cooler)
- [Dell R730 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r730/r730_ompublication/expansion-card-installation-guidelines)
- [Dell R720 Owner's Manual - Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom/expansion-card-installation-guidelines)
- [ComfyUI GPU Benchmarks Discussion](https://github.com/Comfy-Org/ComfyUI/discussions/2970)
- [ComfyUI P40 FP32 Issue](https://github.com/Comfy-Org/ComfyUI/issues/4363)
- [How to Use Tesla P40 Guide](https://github.com/JingShing/How-to-use-tesla-p40)
- [Build a Local LLM Server Under $1000](https://sanj.dev/post/affordable-ai-hardware-local-llms)
- [Best Budget GPUs for AI 2026](https://techtactician.com/best-budget-gpus-for-local-ai-workflows/)