Claude
8fe8498df2
Adds AI_PROVIDER=local_gpu — a fully self-contained GPU inference engine
using HuggingFace Diffusers that requires zero InvokeAI/ComfyUI setup.
All existing providers (InvokeAI, ComfyUI, OpenAI, Replicate) remain intact
and can be mixed with local GPU via per-operation overrides.
New features:
- GPU auto-detection (CUDA/NVIDIA, MPS/Apple Silicon, CPU fallback)
- VRAM-tiered model selection:
ultra ≥16 GB → SDXL inpaint + SDXL base
high 8-16 GB → SDXL inpaint + SDXL base
medium 4-8 GB → SD 2.x inpaint + SD 2.1
low <4 GB → SD 2.x (small)
- Auto-download model weights to HuggingFace disk cache at startup
(background task; first request loads from local disk, not internet)
- LRU pipeline cache evicts oldest GPU pipeline when VRAM limit reached
- Per-operation model overrides via HF_MODEL_INPAINT / HF_MODEL_TXT2IMG etc.
- Optional HF_TOKEN for gated/private HuggingFace models
New files:
- backend/app/services/gpu_detect.py — GPU detection + tier/model mapping
- backend/app/services/local_diffusion.py — Diffusers provider + LRU cache
- backend/app/routers/gpu_status.py — GET /api/gpu/status, POST /api/gpu/prefetch
- backend/requirements.gpu.txt — Diffusers ecosystem deps (GPU only)
- docker-compose.gpu.yml — NVIDIA GPU compose (one-command startup)
- Dockerfile.gpu — pytorch/pytorch:2.1.0-cuda12.1 base image
- scripts/gpu_setup.py — Startup GPU info logger
Modified:
- backend/app/config.py — local_gpu settings added
- backend/app/services/remote_provider.py — local_gpu registered as provider
- backend/app/routers/ai_tools.py — /api/config exposes GPU tier + caps
- backend/app/main.py — GPU router + background prefetch task
- backend/entrypoint.sh — runs gpu_setup.py at container start
- .env.example — local_gpu documented as first option
Quick start with GPU:
docker compose -f docker-compose.gpu.yml up --build
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
68 lines
2.2 KiB
Python
68 lines
2.2 KiB
Python
#!/usr/bin/env python3
|
||
"""
|
||
GPU setup script — runs at container startup.
|
||
Detects GPU, logs capabilities, triggers background model prefetch when
|
||
AI_PROVIDER=local_gpu and AUTO_DOWNLOAD_MODELS=true.
|
||
Non-fatal: any failure just prints a warning.
|
||
"""
|
||
import os
|
||
import sys
|
||
|
||
|
||
def main():
|
||
print("Detecting GPU…")
|
||
|
||
backend = "cpu"
|
||
device_name = "CPU"
|
||
vram_gb = 0.0
|
||
|
||
try:
|
||
import torch
|
||
|
||
if torch.cuda.is_available():
|
||
backend = "cuda"
|
||
props = torch.cuda.get_device_properties(0)
|
||
device_name = props.name
|
||
vram_gb = props.total_memory / (1024 ** 3)
|
||
print(f"✓ CUDA GPU: {device_name} ({vram_gb:.1f} GB VRAM)")
|
||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||
backend = "mps"
|
||
device_name = "Apple Silicon"
|
||
print("✓ Apple Silicon MPS GPU detected")
|
||
else:
|
||
print("⚠ No GPU detected — AI_PROVIDER=local_gpu will use CPU (inference will be slow)")
|
||
|
||
except ImportError:
|
||
print("⚠ PyTorch not installed — GPU detection skipped")
|
||
return
|
||
|
||
provider = os.environ.get("AI_PROVIDER", "").lower()
|
||
if provider != "local_gpu":
|
||
print(f" AI_PROVIDER={provider!r} — local GPU inference not active")
|
||
return
|
||
|
||
auto_dl = os.environ.get("AUTO_DOWNLOAD_MODELS", "true").lower()
|
||
if auto_dl != "true":
|
||
print(" AUTO_DOWNLOAD_MODELS=false — skipping model prefetch")
|
||
print(" Models will download on first request and cache to ~/.cache/huggingface")
|
||
return
|
||
|
||
# Determine tier for a helpful startup message
|
||
if vram_gb >= 16:
|
||
tier, models_hint = "ultra", "SDXL (best quality)"
|
||
elif vram_gb >= 8:
|
||
tier, models_hint = "high", "SDXL"
|
||
elif vram_gb >= 4:
|
||
tier, models_hint = "medium", "Stable Diffusion 2.x"
|
||
else:
|
||
tier, models_hint = "low", "Stable Diffusion 2.x (small)"
|
||
|
||
print(f" GPU tier: {tier} → will use {models_hint} models")
|
||
print(" Models will auto-download on first request (~2–7 GB per pipeline).")
|
||
print(" To pre-download now: POST /api/gpu/prefetch")
|
||
print(" Check progress at: GET /api/gpu/prefetch-status")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|