Add local GPU inference: auto-detect GPU, auto-download best diffusion models

Adds AI_PROVIDER=local_gpu — a fully self-contained GPU inference engine
using HuggingFace Diffusers that requires zero InvokeAI/ComfyUI setup.
All existing providers (InvokeAI, ComfyUI, OpenAI, Replicate) remain intact
and can be mixed with local GPU via per-operation overrides.

New features:
- GPU auto-detection (CUDA/NVIDIA, MPS/Apple Silicon, CPU fallback)
- VRAM-tiered model selection:
    ultra ≥16 GB → SDXL inpaint + SDXL base
    high  8-16 GB → SDXL inpaint + SDXL base
    medium 4-8 GB → SD 2.x inpaint + SD 2.1
    low  <4 GB   → SD 2.x (small)
- Auto-download model weights to HuggingFace disk cache at startup
  (background task; first request loads from local disk, not internet)
- LRU pipeline cache evicts oldest GPU pipeline when VRAM limit reached
- Per-operation model overrides via HF_MODEL_INPAINT / HF_MODEL_TXT2IMG etc.
- Optional HF_TOKEN for gated/private HuggingFace models

New files:
- backend/app/services/gpu_detect.py   — GPU detection + tier/model mapping
- backend/app/services/local_diffusion.py — Diffusers provider + LRU cache
- backend/app/routers/gpu_status.py    — GET /api/gpu/status, POST /api/gpu/prefetch
- backend/requirements.gpu.txt         — Diffusers ecosystem deps (GPU only)
- docker-compose.gpu.yml               — NVIDIA GPU compose (one-command startup)
- Dockerfile.gpu                       — pytorch/pytorch:2.1.0-cuda12.1 base image
- scripts/gpu_setup.py                 — Startup GPU info logger

Modified:
- backend/app/config.py                — local_gpu settings added
- backend/app/services/remote_provider.py — local_gpu registered as provider
- backend/app/routers/ai_tools.py      — /api/config exposes GPU tier + caps
- backend/app/main.py                  — GPU router + background prefetch task
- backend/entrypoint.sh                — runs gpu_setup.py at container start
- .env.example                         — local_gpu documented as first option

Quick start with GPU:
  docker compose -f docker-compose.gpu.yml up --build

https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
This commit is contained in:
Claude
2026-06-13 15:08:41 +00:00
parent 81d866efb8
commit 8fe8498df2
13 changed files with 1027 additions and 6 deletions
+26
View File
@@ -10,6 +10,7 @@ import os
from app.config import settings
from app.database import init_db
from app.routers import projects, edits, images, patches, generate, tools, ai_tools, print_tools
from app.routers import gpu_status
@asynccontextmanager
@@ -24,6 +25,30 @@ async def lifespan(app: FastAPI):
# Pre-download SAM model in background so first click is fast
from app.services.sam_service import ensure_sam_installed
asyncio.create_task(ensure_sam_installed())
# If local GPU provider is active, log GPU info at startup
if settings.ai_provider.lower() == "local_gpu" or any(
v.lower() == "local_gpu"
for v in [
settings.ai_provider_inpaint,
settings.ai_provider_txt2img,
settings.ai_provider_img2img,
settings.ai_provider_outpaint,
]
if v
):
from app.services.gpu_detect import get_cached_gpu_info
info = get_cached_gpu_info()
print(
f"[gpu] {info.device_name} | {info.vram_gb:.1f} GB | tier={info.tier} | "
f"backend={info.backend}"
)
if settings.auto_download_models:
# Download model weight files to disk cache in background so first
# user request loads from local disk instead of the internet.
from app.services.local_diffusion import prefetch_model_files
asyncio.create_task(prefetch_model_files())
yield
@@ -52,6 +77,7 @@ app.include_router(generate.router)
app.include_router(tools.router)
app.include_router(ai_tools.router)
app.include_router(print_tools.router)
app.include_router(gpu_status.router)
@app.get("/api")