root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add ComfyUI service + image generation docs for Open WebUI integration

Browse Source 
- Add ComfyUI as optional service in setup wizard (alongside InvokeAI)
- Configure Open WebUI env vars for ComfyUI integration when selected
  (ENABLE_IMAGE_GENERATION, IMAGE_GENERATION_ENGINE, COMFYUI_BASE_URL)
- Add ComfyUI docker service (ai-dock/comfyui, port 8188, GPU access)
- Add comprehensive image generation documentation to README:
  - ComfyUI + Open WebUI setup steps (model install, workflow export, node mapping)
  - AUTOMATIC1111 alternative setup
  - Environment variables reference table
  - VRAM considerations for simultaneous LLM + image gen
- Update all touchpoints: UFW rules, Caddyfile, start.sh URLs, volumes,
  compose services, summary output, directory creation
- Note: InvokeAI does NOT integrate with Open WebUI natively (no compatible API)
  ComfyUI is the recommended path for chat-integrated image generation

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 19:02:22 +00:00
parent 7dac4c5e60
commit 8920a61156
2 changed files with 180 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+111 -2
View File
@@ -2,7 +2,7 @@
A fully offline, self-hosted AI environment for Ubuntu 24.04. Runs on any NVIDIA GPU (or CPU-only).
**Services:** Ollama · Open WebUI · RAG · MCP · ChromaDB · SearXNG · Kiwix · Gitea · InvokeAI · Portainer
**Services:** Ollama · Open WebUI · RAG · MCP · ChromaDB · SearXNG · Kiwix · Gitea · InvokeAI · ComfyUI · Portainer
---
@@ -25,7 +25,8 @@ After setup, all services are available on your LAN:
| Service | URL | Purpose |
|------------|----------------------------|--------------------------------|
| Open WebUI | `http://<ip>:3000` | Chat interface (Ollama + RAG) |
| InvokeAI | `http://<ip>:9090` | Image generation |
| InvokeAI | `http://<ip>:9090` | Image generation (standalone) |
| ComfyUI | `http://<ip>:8188` | Image generation (OWUI integration) |
| SearXNG | `http://<ip>:8888` | Private web search |
| Kiwix | `http://<ip>:8181` | Offline Wikipedia / docs |
| Gitea | `http://<ip>:3001` | Self-hosted Git |
@@ -98,6 +99,8 @@ Both scripts are **idempotent** — safe to re-run for updates. Config files are
├── kiwix/ # ZIM files for Kiwix
├── gitea/ # Gitea data
├── invokeai-outputs/ # InvokeAI generated images
├── comfyui-output/ # ComfyUI generated images
├── comfyui-data/ # ComfyUI custom nodes
└── logs/
```
@@ -132,6 +135,112 @@ Both scripts are **idempotent** — safe to re-run for updates. Config files are
---
## Image Generation from Open WebUI
Open WebUI can generate images inline in chat conversations using ComfyUI as the backend. When configured, you can ask any model to "generate an image of..." and it will call ComfyUI to create the image.
### How it works
Open WebUI natively supports these image generation engines:
- **ComfyUI** — Node-based, best Open WebUI integration, local
- **AUTOMATIC1111** — Stable Diffusion WebUI, local
- **OpenAI DALL-E** — Cloud API
- **Gemini** — Cloud API
**InvokeAI** does NOT have a compatible API for Open WebUI integration. It works great as a standalone tool at `http://<ip>:9090` but cannot be called from within Open WebUI chats. For chat-integrated image generation, use **ComfyUI**.
### Setup: ComfyUI + Open WebUI (recommended)
If you selected ComfyUI during setup, the environment variables are already configured. You just need to install a model and set up a workflow.
#### Step 1: Install a Stable Diffusion model in ComfyUI
```bash
# Open ComfyUI at http://<ip>:8188
# Use the built-in Model Manager to download a model, or manually:
docker exec comfyui bash -c "cd /opt/ComfyUI/models/checkpoints && \
wget -q 'https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors'"
```
Or download any `.safetensors` checkpoint and copy it in:
```bash
docker cp ~/Downloads/my-model.safetensors comfyui:/opt/ComfyUI/models/checkpoints/
```
#### Step 2: Create and export a workflow
1. Open ComfyUI at `http://<ip>:8188`
2. Build or load a workflow (the default text-to-image workflow works)
3. Click the **gear icon** → enable **Dev Mode**
4. Click **Save (API Format)** — this downloads `workflow_api.json`
#### Step 3: Configure Open WebUI
1. Open WebUI → **Admin** → **Settings** → **Images**
2. Set **Engine** to `ComfyUI`
3. Set **URL** to `http://comfyui:8188` (container networking, already set via env vars)
4. Click **Import Workflow** and upload your `workflow_api.json`
5. Map the **prompt node** (usually the KSampler or CLIPTextEncode node)
6. Save settings
#### Step 4: Generate images in chat
In any Open WebUI chat, type something like:
- "Generate an image of a mountain landscape at sunset"
- "Create a photo of a cyberpunk city"
The model will detect the image generation request and pass it to ComfyUI.
### Setup: AUTOMATIC1111 (alternative)
If you prefer AUTOMATIC1111 over ComfyUI:
1. Run AUTOMATIC1111 with the `--api` flag
2. In Open WebUI → **Admin** → **Settings** → **Images**:
- Engine: `Automatic1111`
- URL: `http://host.docker.internal:7860` (or container name if in Docker)
3. Environment variables (alternative to UI config):
```
ENABLE_IMAGE_GENERATION=true
IMAGE_GENERATION_ENGINE=automatic1111
AUTOMATIC1111_BASE_URL=http://host.docker.internal:7860
```
### Environment variables reference
| Variable | Default | Description |
|----------|---------|-------------|
| `ENABLE_IMAGE_GENERATION` | `false` | Enable image generation feature |
| `IMAGE_GENERATION_ENGINE` | — | `comfyui`, `automatic1111`, `openai`, or `gemini` |
| `IMAGE_GENERATION_MODEL` | — | Model ID for generation |
| `IMAGE_SIZE` | `512x512` | Default output size |
| `COMFYUI_BASE_URL` | — | ComfyUI API URL (e.g. `http://comfyui:8188`) |
| `COMFYUI_API_KEY` | — | ComfyUI API key (if auth enabled) |
| `COMFYUI_WORKFLOW` | — | Custom workflow JSON (API format) |
| `AUTOMATIC1111_BASE_URL` | — | AUTOMATIC1111 API URL |
| `AUTOMATIC1111_API_AUTH` | — | Auth credentials (`user:pass`) |
### Recommended Open WebUI Functions
Install these from **Admin → Functions → ＋ → Import From Link** or search in Discover:
- **Auto Memory** — Automatically stores relevant info as persistent memories across chats
- **Generate Image** — Adds a "Generate Image" action button to messages for quick re-generation
### VRAM considerations
Image generation and LLM inference compete for GPU memory. With a single GPU:
| VRAM | Recommendation |
|------|---------------|
| ≥ 24GB | Run both LLM + image gen simultaneously |
| 12–24GB | Use smaller LLM when generating images, or stop Ollama first |
| < 12GB | Run one at a time — stop Ollama before generating images |
ComfyUI models typically need 4–8GB VRAM (SD 1.5: ~4GB, SDXL: ~7GB, Flux: ~12GB).
---
## Using LoRA Models in InvokeAI
LoRA (Low-Rank Adaptation) files let you customize image generation with fine-tuned styles or characters. If you trained a LoRA on RunPod or elsewhere, here's how to use it.
laptop_full_setup.sh
+69 -12
View File
@@ -6,7 +6,7 @@
# Options (checklist at launch):
# • Full system setup — Ubuntu apps, security, backups (ubuntu-post-install.sh)
# • AI stack — Ollama · Open WebUI · RAG · MCP · ChromaDB
# Gitea · InvokeAI · Portainer
# Gitea · InvokeAI · ComfyUI · Portainer
# • Kiwix — Offline Wikipedia, Stack Overflow, Arch Wiki, etc.
#
# Usage:
@@ -103,7 +103,7 @@ INSTALL_AI=true
# Per-service flags (all default ON — toggled off in Q1a)
SVC_WEBUI=true; SVC_RAG=true; SVC_MCP=true
SVC_GITEA=true; SVC_INVOKEAI=true; SVC_PORTAINER=true; SVC_KIWIX=true
SVC_GITEA=true; SVC_INVOKEAI=true; SVC_COMFYUI=true; SVC_PORTAINER=true; SVC_KIWIX=true
POSTINSTALL_SCRIPT="$SCRIPT_DIR/ubuntu-post-install.sh"
HAS_POSTINSTALL=false
@@ -155,12 +155,13 @@ if $INSTALL_AI; then
if command -v whiptail &>/dev/null; then
SELECTED_SVCS=$(whiptail --title "AI Stack — Select Services" \
--checklist "Ollama always installed (core). SPACE = toggle ENTER = confirm" \
20 72 9 \
22 78 10 \
"open-webui" "Open WebUI — Chat interface (like ChatGPT, uses Ollama)" "$(existing_svc open-webui)" \
"rag" "RAG + ChromaDB— Code & document search (needed by MCP)" "$(existing_svc rag-server)" \
"mcp" "MCP Server — Claude Code tools (bash, files, git, search)" "$(existing_svc mcp-server)" \
"gitea" "Gitea — Self-hosted Git server" "$(existing_svc gitea)" \
"invokeai" "InvokeAI — Image generation (Stable Diffusion)" "$(existing_svc invokeai)" \
"comfyui" "ComfyUI — Node-based image gen (integrates w/ Open WebUI)" "$(existing_svc comfyui)" \
"portainer" "Portainer — Docker management web UI" "$(existing_svc portainer)" \
"kiwix" "Kiwix — Offline Wikipedia, Stack Overflow, Arch Wiki" "$(existing_svc kiwix)" \
3>&1 1>&2 2>&3) || die "Cancelled."
@@ -170,6 +171,7 @@ if $INSTALL_AI; then
SVC_MCP=false; [[ "$SELECTED_SVCS" == *'"mcp"'* ]] && SVC_MCP=true
SVC_GITEA=false; [[ "$SELECTED_SVCS" == *'"gitea"'* ]] && SVC_GITEA=true
SVC_INVOKEAI=false; [[ "$SELECTED_SVCS" == *'"invokeai"'* ]] && SVC_INVOKEAI=true
SVC_COMFYUI=false; [[ "$SELECTED_SVCS" == *'"comfyui"'* ]] && SVC_COMFYUI=true
SVC_PORTAINER=false;[[ "$SELECTED_SVCS" == *'"portainer"'* ]] && SVC_PORTAINER=true
SVC_KIWIX=false; [[ "$SELECTED_SVCS" == *'"kiwix"'* ]] && SVC_KIWIX=true
else
@@ -178,7 +180,7 @@ if $INSTALL_AI; then
echo -e " ${BOLD}[1a] AI Stack — select services to install${NC}"
echo " Ollama always included (required core). Type number to toggle, Enter to confirm."
_W=$SVC_WEBUI; _R=$SVC_RAG; _M=$SVC_MCP
_G=$SVC_GITEA; _I=$SVC_INVOKEAI; _P=$SVC_PORTAINER; _K=$SVC_KIWIX
_G=$SVC_GITEA; _I=$SVC_INVOKEAI; _C=$SVC_COMFYUI; _P=$SVC_PORTAINER; _K=$SVC_KIWIX
while true; do
echo ""
printf " [%s] 1. Open WebUI — Chat interface\n" "$($_W && echo '*' || echo ' ')"
@@ -186,8 +188,9 @@ if $INSTALL_AI; then
printf " [%s] 3. MCP Server — Claude Code tools (needs RAG)\n" "$($_M && echo '*' || echo ' ')"
printf " [%s] 4. Gitea — Self-hosted Git\n" "$($_G && echo '*' || echo ' ')"
printf " [%s] 5. InvokeAI — Image generation\n" "$($_I && echo '*' || echo ' ')"
printf " [%s] 6. Portainer — Docker management UI\n" "$($_P && echo '*' || echo ' ')"
printf " [%s] 7. Kiwix — Offline Wikipedia, Stack Overflow\n" "$($_K && echo '*' || echo ' ')"
printf " [%s] 6. ComfyUI — Node-based image gen (OWUI integration)\n" "$($_C && echo '*' || echo ' ')"
printf " [%s] 7. Portainer — Docker management UI\n" "$($_P && echo '*' || echo ' ')"
printf " [%s] 8. Kiwix — Offline Wikipedia, Stack Overflow\n" "$($_K && echo '*' || echo ' ')"
echo ""
read -rp " Toggle [number] or Enter to confirm: " T
case "$T" in
@@ -196,13 +199,14 @@ if $INSTALL_AI; then
3) $_M && _M=false || _M=true ;;
4) $_G && _G=false || _G=true ;;
5) $_I && _I=false || _I=true ;;
6) $_P && _P=false || _P=true ;;
7) $_K && _K=false || _K=true ;;
6) $_C && _C=false || _C=true ;;
7) $_P && _P=false || _P=true ;;
8) $_K && _K=false || _K=true ;;
"") break ;;
esac
done
SVC_WEBUI=$_W; SVC_RAG=$_R; SVC_MCP=$_M
SVC_GITEA=$_G; SVC_INVOKEAI=$_I; SVC_PORTAINER=$_P; SVC_KIWIX=$_K
SVC_GITEA=$_G; SVC_INVOKEAI=$_I; SVC_COMFYUI=$_C; SVC_PORTAINER=$_P; SVC_KIWIX=$_K
fi
# Enforce dependencies: MCP needs RAG
@@ -618,6 +622,7 @@ if $INSTALL_AI; then
$SVC_MCP && _svcs+=" · MCP"
$SVC_GITEA && _svcs+=" · Gitea"
$SVC_INVOKEAI && _svcs+=" · InvokeAI"
$SVC_COMFYUI && _svcs+=" · ComfyUI"
$SVC_PORTAINER && _svcs+=" · Portainer"
$SVC_KIWIX && _svcs+=" · Kiwix"
echo " ✓ AI stack → $_svcs"
@@ -743,7 +748,7 @@ fi
section "Directories"
# =============================================================================
for d in papers repos workspace index invokeai-data invokeai-outputs \
gitea portainer-data logs; do
comfyui-data comfyui-output gitea portainer-data logs; do
mkdir -p "$BASE/$d"
done
# Kiwix ZIM dir — may be on a different drive
@@ -844,7 +849,7 @@ cat > "$BASE/docker-compose.yml" << COMPOSE
# Show status: docker compose ps
#
# Services: ollama open-webui chromadb rag-server mcp-server
# kiwix gitea invokeai portainer
# kiwix gitea invokeai comfyui portainer
# ───────────────────────────────────────────────────────────────────────────────
services:
@@ -897,6 +902,12 @@ ${OLLAMA_VOLUME_LINE}
- WEBUI_URL=${WEBUI_URL:-}
- ENABLE_RAG_WEB_SEARCH=true
- RAG_WEB_SEARCH_ENGINE=duckduckgo
$( $SVC_COMFYUI && cat <<'IMGENV'
- ENABLE_IMAGE_GENERATION=true
- IMAGE_GENERATION_ENGINE=comfyui
- COMFYUI_BASE_URL=http://comfyui:8188
IMGENV
)
depends_on:
ollama:
condition: service_healthy
@@ -1031,6 +1042,27 @@ ${OLLAMA_VOLUME_LINE}
count: all
capabilities: [gpu]
# ── ComfyUI — Node-based image generation ──────────────────────────────────
comfyui:
image: ghcr.io/ai-dock/comfyui:latest
container_name: comfyui
restart: unless-stopped
ports:
- "0.0.0.0:8188:8188"
volumes:
- comfyui-models:/opt/ComfyUI/models
- $BASE/comfyui-output:/opt/ComfyUI/output
- $BASE/comfyui-data:/opt/ComfyUI/custom_nodes
environment:
- CLI_ARGS=--listen 0.0.0.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
# ── Portainer — Docker management UI ───────────────────────────────────────
portainer:
image: portainer/portainer-ce:latest
@@ -1047,6 +1079,7 @@ volumes:
${OLLAMA_VOLUMES_DECL}
open-webui-data:
invokeai-models:
comfyui-models:
COMPOSE
ok "docker-compose.yml written"
@@ -1059,7 +1092,7 @@ if command -v ufw &>/dev/null; then
"3000:Open WebUI" "11434:Ollama" "8001:RAG Server" \
"8002:MCP Server" "8000:ChromaDB" \
"8181:Kiwix" "3001:Gitea" "2222:Gitea SSH" \
"9090:InvokeAI" "9000:Portainer" "9443:Portainer S"; do
"9090:InvokeAI" "8188:ComfyUI" "9000:Portainer" "9443:Portainer S"; do
port="${port_comment%%:*}"
comment="${port_comment##*:}"
sudo ufw allow from "$LAN_SUBNET" to any port "$port" proto tcp \
@@ -1083,6 +1116,7 @@ section "Helper Scripts"
_START_URLS=""
$SVC_WEBUI && _START_URLS+=$'echo " Open WebUI → http://'"$LOCAL_IP"$':3000"\n'
$SVC_INVOKEAI && _START_URLS+=$'echo " InvokeAI → http://'"$LOCAL_IP"$':9090"\n'
$SVC_COMFYUI && _START_URLS+=$'echo " ComfyUI → http://'"$LOCAL_IP"$':8188"\n'
$SVC_GITEA && _START_URLS+=$'echo " Gitea → http://'"$LOCAL_IP"$':3001"\n'
$SVC_RAG && _START_URLS+=$'echo " RAG Health → http://'"$LOCAL_IP"$':8001/health"\n'
$SVC_MCP && _START_URLS+=$'echo " MCP SSE → http://'"$LOCAL_IP"$':8002/sse"\n'
@@ -1197,6 +1231,7 @@ webui.yourdomain.com {
}
}
invokeai.yourdomain.com { reverse_proxy $LOCAL_IP:9090 }
comfyui.yourdomain.com { reverse_proxy $LOCAL_IP:8188 }
git.yourdomain.com { reverse_proxy $LOCAL_IP:3001 }
kiwix.yourdomain.com { reverse_proxy $LOCAL_IP:8181 }
rag.yourdomain.com { reverse_proxy $LOCAL_IP:8001 }
@@ -1247,6 +1282,7 @@ if $INSTALL_AI; then
$SVC_MCP && COMPOSE_SERVICES+=" mcp-server"
$SVC_GITEA && COMPOSE_SERVICES+=" gitea"
$SVC_INVOKEAI && COMPOSE_SERVICES+=" invokeai"
$SVC_COMFYUI && COMPOSE_SERVICES+=" comfyui"
$SVC_PORTAINER && COMPOSE_SERVICES+=" portainer"
$SVC_KIWIX && COMPOSE_SERVICES+=" kiwix"
fi
@@ -1403,6 +1439,7 @@ echo ""
if $INSTALL_AI; then
$SVC_WEBUI && echo -e " ${CYAN}Open WebUI${NC} → http://$LOCAL_IP:3000"
$SVC_INVOKEAI && echo -e " ${CYAN}InvokeAI${NC} → http://$LOCAL_IP:9090"
$SVC_COMFYUI && echo -e " ${CYAN}ComfyUI${NC} → http://$LOCAL_IP:8188"
$SVC_GITEA && echo -e " ${CYAN}Gitea${NC} → http://$LOCAL_IP:3001"
$SVC_RAG && echo -e " ${CYAN}RAG Health${NC} → http://$LOCAL_IP:8001/health"
$SVC_MCP && echo -e " ${CYAN}MCP SSE${NC} → http://$LOCAL_IP:8002/sse"
@@ -1437,6 +1474,26 @@ if $INSTALL_AI; then
echo " → Shows tokens used vs available, progress bar, context % remaining"
echo " Context compaction: https://openwebui.com/f/projectmoon/checkpoint_summarization_filter"
echo " → Auto-summarizes old messages when context fills up (like Claude)"
echo " Auto Memory: Install from Admin → Functions → Discover → search 'Auto Memory'"
echo " → Automatically stores relevant info as persistent memories across chats"
$SVC_COMFYUI && {
echo ""
echo -e " ${YELLOW}Image Generation (ComfyUI → Open WebUI):${NC}"
echo " ComfyUI is pre-configured. To complete setup:"
echo " 1. Open ComfyUI at http://$LOCAL_IP:8188 and install a model (e.g. SD 1.5, SDXL)"
echo " 2. In ComfyUI: Settings (gear) → enable 'Dev Mode' → Save workflow as 'API Format'"
echo " 3. In Open WebUI: Admin → Settings → Images"
echo " Engine: ComfyUI | URL: http://comfyui:8188 (already set via env vars)"
echo " 4. Import your workflow JSON and map the prompt/output nodes"
echo " 5. Ask any model to 'generate an image of...' — it will use ComfyUI"
}
$SVC_INVOKEAI && ! $SVC_COMFYUI && {
echo ""
echo -e " ${YELLOW}Image Generation (InvokeAI — standalone):${NC}"
echo " InvokeAI runs at http://$LOCAL_IP:9090 with its own UI"
echo " Note: InvokeAI does NOT integrate with Open WebUI natively"
echo " For Open WebUI integration, enable ComfyUI in the setup wizard"
}
fi
if $SVC_KIWIX && [[ "$ZIM_CHOICE" == "3" ]]; then
echo -e " ${YELLOW}ZIM downloads:${NC} ./kiwix_download.sh (not started)"