Add ComfyUI service + image generation docs for Open WebUI integration

- Add ComfyUI as optional service in setup wizard (alongside InvokeAI) - Configure Open WebUI env vars for ComfyUI integration when selected (ENABLE_IMAGE_GENERATION, IMAGE_GENERATION_ENGINE, COMFYUI_BASE_URL) - Add ComfyUI docker service (ai-dock/comfyui, port 8188, GPU access) - Add comprehensive image generation documentation to README: - ComfyUI + Open WebUI setup steps (model install, workflow export, node mapping) - AUTOMATIC1111 alternative setup - Environment variables reference table - VRAM considerations for simultaneous LLM + image gen - Update all touchpoints: UFW rules, Caddyfile, start.sh URLs, volumes, compose services, summary output, directory creation - Note: InvokeAI does NOT integrate with Open WebUI natively (no compatible API) ComfyUI is the recommended path for chat-integrated image generation https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
2026-03-22 19:02:22 +00:00
parent 7dac4c5e60
commit 8920a61156
2 changed files with 180 additions and 14 deletions
@@ -2,7 +2,7 @@

 A fully offline, self-hosted AI environment for Ubuntu 24.04. Runs on any NVIDIA GPU (or CPU-only).

-**Services:** Ollama · Open WebUI · RAG · MCP · ChromaDB · SearXNG · Kiwix · Gitea · InvokeAI · Portainer
+**Services:** Ollama · Open WebUI · RAG · MCP · ChromaDB · SearXNG · Kiwix · Gitea · InvokeAI · ComfyUI · Portainer

 ---

@@ -25,7 +25,8 @@ After setup, all services are available on your LAN:
 | Service    | URL                        | Purpose                        |
 |------------|----------------------------|--------------------------------|
 | Open WebUI | `http://<ip>:3000`         | Chat interface (Ollama + RAG)  |
-| InvokeAI   | `http://<ip>:9090`         | Image generation               |
+| InvokeAI   | `http://<ip>:9090`         | Image generation (standalone)  |
+| ComfyUI    | `http://<ip>:8188`         | Image generation (OWUI integration) |
 | SearXNG    | `http://<ip>:8888`         | Private web search             |
 | Kiwix      | `http://<ip>:8181`         | Offline Wikipedia / docs       |
 | Gitea      | `http://<ip>:3001`         | Self-hosted Git                |
@@ -98,6 +99,8 @@ Both scripts are **idempotent** — safe to re-run for updates. Config files are
 ├── kiwix/                  # ZIM files for Kiwix
 ├── gitea/                  # Gitea data
 ├── invokeai-outputs/       # InvokeAI generated images
+├── comfyui-output/         # ComfyUI generated images
+├── comfyui-data/           # ComfyUI custom nodes
 └── logs/
 ```

@@ -132,6 +135,112 @@ Both scripts are **idempotent** — safe to re-run for updates. Config files are

 ---

+## Image Generation from Open WebUI
+
+Open WebUI can generate images inline in chat conversations using ComfyUI as the backend. When configured, you can ask any model to "generate an image of..." and it will call ComfyUI to create the image.
+
+### How it works
+
+Open WebUI natively supports these image generation engines:
+- **ComfyUI** — Node-based, best Open WebUI integration, local
+- **AUTOMATIC1111** — Stable Diffusion WebUI, local
+- **OpenAI DALL-E** — Cloud API
+- **Gemini** — Cloud API
+
+**InvokeAI** does NOT have a compatible API for Open WebUI integration. It works great as a standalone tool at `http://<ip>:9090` but cannot be called from within Open WebUI chats. For chat-integrated image generation, use **ComfyUI**.
+
+### Setup: ComfyUI + Open WebUI (recommended)
+
+If you selected ComfyUI during setup, the environment variables are already configured. You just need to install a model and set up a workflow.
+
+#### Step 1: Install a Stable Diffusion model in ComfyUI
+
+```bash
+# Open ComfyUI at http://<ip>:8188
+# Use the built-in Model Manager to download a model, or manually:
+docker exec comfyui bash -c "cd /opt/ComfyUI/models/checkpoints && \
+  wget -q 'https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors'"
+```
+
+Or download any `.safetensors` checkpoint and copy it in:
+```bash
+docker cp ~/Downloads/my-model.safetensors comfyui:/opt/ComfyUI/models/checkpoints/
+```
+
+#### Step 2: Create and export a workflow
+
+1. Open ComfyUI at `http://<ip>:8188`
+2. Build or load a workflow (the default text-to-image workflow works)
+3. Click the **gear icon** → enable **Dev Mode**
+4. Click **Save (API Format)** — this downloads `workflow_api.json`
+
+#### Step 3: Configure Open WebUI
+
+1. Open WebUI → **Admin** → **Settings** → **Images**
+2. Set **Engine** to `ComfyUI`
+3. Set **URL** to `http://comfyui:8188` (container networking, already set via env vars)
+4. Click **Import Workflow** and upload your `workflow_api.json`
+5. Map the **prompt node** (usually the KSampler or CLIPTextEncode node)
+6. Save settings
+
+#### Step 4: Generate images in chat
+
+In any Open WebUI chat, type something like:
+- "Generate an image of a mountain landscape at sunset"
+- "Create a photo of a cyberpunk city"
+
+The model will detect the image generation request and pass it to ComfyUI.
+
+### Setup: AUTOMATIC1111 (alternative)
+
+If you prefer AUTOMATIC1111 over ComfyUI:
+
+1. Run AUTOMATIC1111 with the `--api` flag
+2. In Open WebUI → **Admin** → **Settings** → **Images**:
+   - Engine: `Automatic1111`
+   - URL: `http://host.docker.internal:7860` (or container name if in Docker)
+3. Environment variables (alternative to UI config):
+   ```
+   ENABLE_IMAGE_GENERATION=true
+   IMAGE_GENERATION_ENGINE=automatic1111
+   AUTOMATIC1111_BASE_URL=http://host.docker.internal:7860
+   ```
+
+### Environment variables reference
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `ENABLE_IMAGE_GENERATION` | `false` | Enable image generation feature |
+| `IMAGE_GENERATION_ENGINE` | — | `comfyui`, `automatic1111`, `openai`, or `gemini` |
+| `IMAGE_GENERATION_MODEL` | — | Model ID for generation |
+| `IMAGE_SIZE` | `512x512` | Default output size |
+| `COMFYUI_BASE_URL` | — | ComfyUI API URL (e.g. `http://comfyui:8188`) |
+| `COMFYUI_API_KEY` | — | ComfyUI API key (if auth enabled) |
+| `COMFYUI_WORKFLOW` | — | Custom workflow JSON (API format) |
+| `AUTOMATIC1111_BASE_URL` | — | AUTOMATIC1111 API URL |
+| `AUTOMATIC1111_API_AUTH` | — | Auth credentials (`user:pass`) |
+
+### Recommended Open WebUI Functions
+
+Install these from **Admin → Functions → ＋ → Import From Link** or search in Discover:
+
+- **Auto Memory** — Automatically stores relevant info as persistent memories across chats
+- **Generate Image** — Adds a "Generate Image" action button to messages for quick re-generation
+
+### VRAM considerations
+
+Image generation and LLM inference compete for GPU memory. With a single GPU:
+
+| VRAM | Recommendation |
+|------|---------------|
+| ≥ 24GB | Run both LLM + image gen simultaneously |
+| 12–24GB | Use smaller LLM when generating images, or stop Ollama first |
+| < 12GB | Run one at a time — stop Ollama before generating images |
+
+ComfyUI models typically need 4–8GB VRAM (SD 1.5: ~4GB, SDXL: ~7GB, Flux: ~12GB).
+
+---
+
 ## Using LoRA Models in InvokeAI

 LoRA (Low-Rank Adaptation) files let you customize image generation with fine-tuned styles or characters. If you trained a LoRA on RunPod or elsewhere, here's how to use it.