Files

T

Claude 14da24c02b Add GPU setup research for rack server AI workloads

Research comparing P40, A2000, T4, and M40 GPUs for LLM inference
and image generation in Dell R720/R730 rack servers. Includes
benchmarks, compatibility notes, pricing, and recommendations.

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu

2026-03-22 14:12:46 +00:00

5.1 KiB

Raw Blame History

GPU Setup Research: Rack Server AI Workloads

Goal

Cost-efficient rack-mountable GPU setup for:

LLM coding inference (Ollama + qwen2.5-coder models)
Image generation (ComfyUI / InvokeAI with Stable Diffusion)

Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)

GPU Candidates Compared

Feature	Tesla P40	RTX A2000 12GB	Tesla T4	Tesla M40
Architecture	Pascal (2016)	Ampere (2020)	Turing (2018)	Maxwell (2015)
VRAM	24GB GDDR5	12GB GDDR6	16GB GDDR6	24GB GDDR5
Tensor Cores	No	Yes	Yes	No
TDP	250W	70W	70W	250W
Compute Capability	6.1	8.6	7.5	5.2
Cooling	Passive (needs fan)	Active (blower)	Passive (needs fan)	Passive (needs fan)
Aux Power Required	Yes (8-pin)	No (bus-powered)	No (bus-powered)	Yes (8-pin)
Used Price (2026)	~$300	~$490	~$800+	~$100-150
PCIe	Gen3 x16	Gen4 x16	Gen3 x16	Gen3 x16

Performance Benchmarks

LLM Inference (tokens/sec via Ollama)

Model	A2000 12GB	P40 24GB
qwen2.5:14b	~21 t/s	~17 t/s
llama3.2:3b-Q8	~50 t/s	~40 t/s
llama3.2:3b-Q4	~60 t/s	~48 t/s

Image Generation (ComfyUI)

GPU	SDXL 20 steps	Notes
P40	~49 seconds	Requires `--force-fp32` flag
A2000	~16 seconds	~3x faster than P40
RTX 4090	~3 seconds	Reference (16x faster than P40)

Rack Server Compatibility

RTX A2000 in R720/R730

Physical fit: Yes. Dual-slot, low-profile, 167mm length
Power: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720)
Cooling: Blower-style fan exhausts out bracket — ideal for rack airflow
Requirement: Dual CPUs needed for GPU PCIe slots
Confirmed working in Dell R740XD (similar architecture)
Third-party single-slot cooler available from n3rdware for tighter fits

Tesla P40 in R720/R730

Physical fit: Yes. Full-length, single-slot, designed for rack servers
Power: 250W, requires 8-pin aux power. Needs GPU enablement kit (power cables + riser)
Cooling: Passive — relies on server chassis fans (which R720/R730 have)
Requirement: Dual CPUs, redundant 1100W PSUs recommended
Natively supported in these servers

R720 vs R730

R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
R730: PCIe Gen3, generally preferred
Both support up to 2x double-wide or 4x single-wide GPUs

Setup Options Analysis

Option A: P40 + A2000 (~$800)

P40 for LLM coding (24GB fits 14B models)
A2000 for image gen (3x faster than P40, tensor cores)
Total power: ~320W GPU
Best performance split

Option B: Single P40 (~$300)

Use for both LLM and image gen
24GB handles 14B models well
Image gen slow but usable (~49s SDXL)
Best budget option, upgrade later

Option C: Dual P40 (~$600)

One dedicated to LLM, one to image gen
500W total GPU power draw
Both need passive cooling (server fans handle this)
Image gen still slow on P40

Option D: P40 + P4 (~$350)

P40 for LLM (24GB)
P4 for light image gen (8GB, passive, 75W)
P4 limited to SD 1.5 and smaller models
Low total power: ~325W

Recommendations

Budget Priority (under $500): Single P40

Start with one P40. Your local-ai stack auto-selects qwen2.5:14b at 24GB VRAM. Image gen works with --force-fp32 flag. Add a second GPU later.

Performance Priority (~$800): P40 + A2000

Best of both worlds. P40 handles LLM inference with full 24GB VRAM. A2000 handles image gen 3x faster with modern Ampere architecture. Both fit in R720/R730. A2000 blower cooling works well in rack.

Configuration Notes for local-ai stack

Set INVOKEAI_PRECISION=float32 when using P40 for image gen
Ollama OLLAMA_NUM_GPU=999 works with both GPUs
For dual GPU: assign specific GPUs via CUDA_VISIBLE_DEVICES
P40 needs --force-fp32 in ComfyUI launch args

5.1 KiB Raw Blame History