Add local GPU inference: auto-detect GPU, auto-download best diffusion models

Adds AI_PROVIDER=local_gpu — a fully self-contained GPU inference engine
using HuggingFace Diffusers that requires zero InvokeAI/ComfyUI setup.
All existing providers (InvokeAI, ComfyUI, OpenAI, Replicate) remain intact
and can be mixed with local GPU via per-operation overrides.

New features:
- GPU auto-detection (CUDA/NVIDIA, MPS/Apple Silicon, CPU fallback)
- VRAM-tiered model selection:
    ultra ≥16 GB → SDXL inpaint + SDXL base
    high  8-16 GB → SDXL inpaint + SDXL base
    medium 4-8 GB → SD 2.x inpaint + SD 2.1
    low  <4 GB   → SD 2.x (small)
- Auto-download model weights to HuggingFace disk cache at startup
  (background task; first request loads from local disk, not internet)
- LRU pipeline cache evicts oldest GPU pipeline when VRAM limit reached
- Per-operation model overrides via HF_MODEL_INPAINT / HF_MODEL_TXT2IMG etc.
- Optional HF_TOKEN for gated/private HuggingFace models

New files:
- backend/app/services/gpu_detect.py   — GPU detection + tier/model mapping
- backend/app/services/local_diffusion.py — Diffusers provider + LRU cache
- backend/app/routers/gpu_status.py    — GET /api/gpu/status, POST /api/gpu/prefetch
- backend/requirements.gpu.txt         — Diffusers ecosystem deps (GPU only)
- docker-compose.gpu.yml               — NVIDIA GPU compose (one-command startup)
- Dockerfile.gpu                       — pytorch/pytorch:2.1.0-cuda12.1 base image
- scripts/gpu_setup.py                 — Startup GPU info logger

Modified:
- backend/app/config.py                — local_gpu settings added
- backend/app/services/remote_provider.py — local_gpu registered as provider
- backend/app/routers/ai_tools.py      — /api/config exposes GPU tier + caps
- backend/app/main.py                  — GPU router + background prefetch task
- backend/entrypoint.sh                — runs gpu_setup.py at container start
- .env.example                         — local_gpu documented as first option

Quick start with GPU:
  docker compose -f docker-compose.gpu.yml up --build

https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
This commit is contained in:
Claude
2026-06-13 15:08:41 +00:00
parent 81d866efb8
commit 8fe8498df2
13 changed files with 1027 additions and 6 deletions
+9
View File
@@ -327,12 +327,21 @@ async def get_config():
# Default provider for display (used when no per-op override)
default_name = (settings.ai_provider or "").lower() or None
from app.services.gpu_detect import get_cached_gpu_info
gpu_info = get_cached_gpu_info()
return {
"local": {
"lama": lama_available(),
"rembg": rembg_available(),
"opencv": True,
"gpu_detected": gpu_available(),
"gpu_backend": gpu_info.backend,
"gpu_device": gpu_info.device_name,
"gpu_vram_gb": gpu_info.vram_gb,
"gpu_tier": gpu_info.tier,
"local_gpu_available": gpu_info.backend in ("cuda", "mps"),
"local_gpu_capabilities": gpu_info.capabilities,
},
"remote": {
"default_provider": default_name,