diff --git a/README.md b/README.md index 7f6b3c3..36e66e8 100644 --- a/README.md +++ b/README.md @@ -784,35 +784,109 @@ ComfyUI models typically need 4–8GB VRAM (SD 1.5: ~4GB, SDXL: ~7GB, Flux: ~12G --- -## Using LoRA Models in InvokeAI +## InvokeAI vs ComfyUI — which to use when -LoRA (Low-Rank Adaptation) files let you customize image generation with fine-tuned styles or characters. If you trained a LoRA on RunPod or elsewhere, here's how to use it. +Both are installed by the setup script. Here's when to use each: -### Import a LoRA file +| Task | InvokeAI (`:9090`) | ComfyUI (`:8188`) | +|------|:---:|:---:| +| Friendly UI, sliders, drag-and-drop | Yes | No (node editor) | +| Use a LoRA you trained on RunPod | Yes — just import + select | Yes — but wire LoRA Loader node | +| Same face, different settings (img2img) | Yes — Image to Image tab | Yes — IP-Adapter nodes | +| Age/emotion/scene changes | Yes — img2img + prompt | Yes — IP-Adapter + prompt | +| Chat-integrated image gen (from Open WebUI) | No (no OWUI API) | Yes (workflow export) | +| Maximum flexibility / custom pipelines | No | Yes | +| Learning curve | Low | High | + +**Bottom line:** Use InvokeAI for the interactive "play with images" workflow. Use ComfyUI only when you need Open WebUI chat integration or advanced node pipelines. + +## Using InvokeAI for image iteration + +InvokeAI is the easier path for what you want — take a reference image and generate variations with different settings, ages, emotions, art styles. + +### Step 1: Install a base model (required first) + +Before LoRAs or img2img will work, InvokeAI needs a base Stable Diffusion model. ```bash -# Copy your LoRA into the InvokeAI Docker volume: -./invokeai-import-lora.sh ~/Downloads/my-lora.safetensors +# Option A: Download SD 1.5 (~4GB VRAM needed) via the container +docker exec invokeai invokeai-model-install --add stabilityai/stable-diffusion-v1-5 -# Optionally give it a display name: -./invokeai-import-lora.sh ~/Downloads/my-lora.safetensors "My Custom Style" +# Option B: Download SDXL (~7GB VRAM needed) +docker exec invokeai invokeai-model-install --add stabilityai/stable-diffusion-xl-base-1.0 ``` -### Use the LoRA in InvokeAI +Or use the UI: open `http://:9090` → **Model Manager** (cube icon) → **Starter Models** tab → install SD 1.5 or SDXL. -1. Open InvokeAI at `http://:9090` -2. Go to **Model Manager** (cube icon, left sidebar) and click **Scan for Models** / **Sync Models** -3. Your LoRA should appear in the model list -4. Switch to **Text to Image** tab -5. In the left panel, find the **LoRA** section (below the model selector) -6. Click **+** to add your LoRA, then adjust the **weight** slider (start at 0.7–0.85) +> **6GB GPU note:** SD 1.5 fits comfortably. SDXL is tight — it may work with float16 (already set) but could be slow. Start with SD 1.5. -### Troubleshooting greyed-out upload buttons +### Step 2: Import your RunPod LoRA -- **No base model installed:** You need a fully downloaded base model (e.g., SD 1.5) before InvokeAI enables LoRA uploads. Use Model Manager to install one first. -- **Model not synced:** After copying files, click **Scan for Models** in Model Manager. -- **Architecture mismatch:** A LoRA trained on SD 1.5 only works with SD 1.5 base models — not SDXL or SD 2.x. -- **Use the import script instead:** The greyed-out UI upload can be bypassed entirely by using `invokeai-import-lora.sh` to copy files directly into the model volume. +```bash +# Copy the .safetensors file from wherever you downloaded it: +./invokeai-import-lora.sh ~/Downloads/my-lora.safetensors "My Character" +``` + +Then in the InvokeAI UI: +1. **Model Manager** (cube icon) → **Scan for Models** +2. Your LoRA should appear in the list + +If it doesn't: +- Click **Add Model** → **Scan Folder** → enter `/invokeai/models/lora` +- Make sure the LoRA architecture matches your base model (SD 1.5 LoRA needs SD 1.5 base) + +### Step 3: Generate with your LoRA (text-to-image) + +1. Go to the **Text to Image** tab +2. Select your **base model** (must match the LoRA's training base) +3. In the left panel, expand the **LoRA** section (below model selector) +4. Click **+** → select your LoRA → set weight to **0.7–0.85** +5. Write a prompt: "portrait of [subject], smiling, studio lighting" +6. Click **Invoke** + +### Step 4: Iterate on an image (img2img) + +This is where InvokeAI shines for your use case — take an image and riff on it: + +1. Switch to the **Image to Image** tab +2. Drag your reference photo onto the canvas (or click to upload) +3. Keep your LoRA active (same as above) +4. Set the **Denoising Strength** slider: + +| Strength | Effect | +|----------|--------| +| 0.2–0.3 | Subtle tweaks — mostly keeps the original, minor style changes | +| 0.4–0.5 | Moderate changes — recognizable but different mood/lighting | +| 0.6–0.7 | Significant changes — same composition, new details/style | +| 0.8–1.0 | Major rewrite — loosely inspired by original, mostly new | + +5. Change the prompt to describe what you want different: + - **Age up:** "same person, elderly, wrinkles, grey hair, wise expression" + - **Age down:** "same person as a young child, bright eyes, playground" + - **Emotion:** "same person, laughing joyfully" or "same person, crying, dramatic lighting" + - **Setting:** "same person, sitting in a Parisian cafe, afternoon light" + - **Art style:** "same person, oil painting, renaissance style, dramatic chiaroscuro" +6. Click **Invoke** — iterate by adjusting strength and prompt + +### Step 5: Use the Unified Canvas for painting/inpainting + +For more control (paint over specific areas, extend an image): + +1. Switch to the **Unified Canvas** tab +2. Upload or paste your image +3. Use the **brush tool** to mask areas you want to change +4. Write a prompt for just the masked area +5. Invoke — only the masked area regenerates + +Example: mask just the background → prompt "tropical beach sunset" → keeps the face, replaces the background. + +### Troubleshooting + +- **Greyed-out upload/LoRA buttons:** Install a base model first (Step 1). InvokeAI disables most features until a checkpoint is loaded. +- **Model not synced:** After copying files via script, click **Scan for Models** in Model Manager. +- **Architecture mismatch:** A LoRA trained on SD 1.5 only works with SD 1.5 base models — not SDXL. Check what your RunPod training used. +- **Out of VRAM:** Try SD 1.5 instead of SDXL, or reduce image size to 512x512. +- **Use the import script:** The greyed-out UI upload can be bypassed entirely by using `invokeai-import-lora.sh` to copy files directly into the model volume. ---