local-ai/README.md

Local AI Stack

A fully offline, self-hosted AI environment for Ubuntu 24.04. Runs on any NVIDIA GPU (or CPU-only).

Services: Ollama · Open WebUI · RAG · MCP · ChromaDB · SearXNG · Kiwix · Gitea · InvokeAI · ComfyUI · Portainer

---

Quick Start

git clone <this-repo>
cd local-ai
./laptop_full_setup.sh

That's it. The script installs Docker, NVIDIA drivers (if needed), generates all config, starts the stack, and optionally pulls models.

---

Service URLs

After setup, all services are available on your LAN:

Service	URL	Purpose
Open WebUI	http://<ip>:3000	Chat interface (Ollama + RAG)
InvokeAI	http://<ip>:9090	Image generation (standalone)
ComfyUI	http://<ip>:8188	Image generation (OWUI integration)
SearXNG	http://<ip>:8888	Private web search
Kiwix	http://<ip>:8181	Offline Wikipedia / docs
Gitea	http://<ip>:3001	Self-hosted Git
RAG Health	http://<ip>:8001/health	RAG server status
MCP SSE	http://<ip>:8002/sse	MCP endpoint for Claude Code
Portainer	https://<ip>:9443	Docker management UI

---

Day-to-Day Commands

All generated into ~/docker/ai-stack/ by the setup script:

bash ~/docker/ai-stack/start.sh       # pull latest images + docker compose up -d
bash ~/docker/ai-stack/stop.sh        # docker compose down
bash ~/docker/ai-stack/status.sh      # GPU / container / RAG health
bash ~/docker/ai-stack/pull-models.sh # pull Ollama models (run once after first install)

The stack also registers as a systemd service that starts on boot:

sudo systemctl start local-ai
sudo systemctl stop local-ai
sudo systemctl status local-ai

---

Script Reference

Script	Lines	What it does
laptop_full_setup.sh	620	Main setup. Installs Docker + NVIDIA toolkit, creates ~/docker/ai-stack/, writes docker-compose.yml, starts stack, registers systemd service.
local-ai-setup.sh	837	Alternative setup script. Same as above but also auto-detects VRAM and selects models accordingly (14B for ≥14GB VRAM, 7B for CPU). Use this instead of laptop_full_setup.sh if you want VRAM-aware model selection.
ubuntu-post-install.sh	8,889	Full Ubuntu 24.04 post-install (dev tools, fonts, apps, tweaks). Run once on a fresh OS install. Independent of the AI stack.
configure-storage.sh	239	Storage/mount configuration helper. Run separately if you have a secondary drive for AI data.
kiwix_download.sh	198	Downloads ZIM files (Wikipedia, Stack Overflow, etc.) for offline use. Run separately — files are large.
invokeai-import-lora.sh	85	Copies a LoRA .safetensors file into InvokeAI's Docker model volume.
comfyui-import-lora.sh	97	Copies a LoRA into ComfyUI and prints workflow setup instructions.
comfyui-install-ipadapter.sh	185	Installs IP-Adapter nodes + models into ComfyUI for reference-image workflows (same face, different settings).
setup-image-models.sh	200	GPU-aware image model installer. Detects VRAM, offers appropriate SD/SDXL/Flux models, installs into InvokeAI and/or ComfyUI.

Which setup script should I use?

• laptop_full_setup.sh — fixed model selection (qwen2.5:14b / qwen2.5-coder:7b), simpler
• local-ai-setup.sh — detects your VRAM at runtime and picks appropriate models, also embeds server.py and mcp_server.py directly (doesn't need repo files copied separately)

Both scripts are idempotent — safe to re-run for updates. Config files are kept on re-run unless you pass --force.

---

Generated File Layout

~/docker/ai-stack/
├── docker-compose.yml      # generated by setup script
├── .env                    # API tokens — edit this, never overwritten
├── server.py               # RAG server (copied from repo)
├── mcp_server.py           # MCP server (copied from repo)
├── requirements.txt        # RAG Python deps
├── mcp_requirements.txt    # MCP Python deps
├── start.sh                # start the stack
├── stop.sh                 # stop the stack
├── status.sh               # GPU + container + RAG health
├── pull-models.sh          # pull Ollama models
├── Caddyfile.example       # reverse proxy config template
├── papers/                 # drop PDFs here for RAG indexing
├── repos/                  # git repos indexed by RAG
├── workspace/              # MCP working directory
├── index/                  # ChromaDB vector store (persistent)
├── kiwix/                  # ZIM files for Kiwix
├── gitea/                  # Gitea data
├── invokeai-outputs/       # InvokeAI generated images
├── comfyui-output/         # ComfyUI generated images
├── comfyui-data/           # ComfyUI custom nodes
└── logs/

---

First Run Checklist

1. Run setup:

   ./laptop_full_setup.sh
   

2. Pull models (prompted at end of setup, or run manually):

   bash ~/docker/ai-stack/pull-models.sh
   

   Downloads ~15-30GB. Takes 10-40 min depending on connection.

3. Add API tokens (optional — for Gitea/GitHub MCP tools):

   nano ~/docker/ai-stack/.env
   

4. Connect Claude Code to MCP:

   claude mcp add local http://<your-ip>:8002/sse
   

5. Download ZIMs for offline docs (optional, large):

   ./kiwix_download.sh
   

---

Querying Your Codebase from Open WebUI

The stack includes a code-aware RAG server that sits between Open WebUI and Ollama. When you chat in Open WebUI, the RAG server automatically retrieves relevant code from your indexed repos and injects it into the prompt — so the model answers with your actual code as context.

This is NOT the same as Open WebUI's built-in Knowledge Collections. This is a separate, always-on layer that understands code structure.

How it works (architecture)

You type a question in Open WebUI
         ↓
Open WebUI → RAG Server (port 8001) /v1/chat/completions
         ↓
RAG Server: searches ChromaDB for relevant code chunks
   (AST-parsed Python functions, regex-split JS/Go/Rust, etc.)
         ↓
RAG Server: prepends code snippets to your prompt:
   "### auth.py:authenticate
    def authenticate(user, password): ..."
         ↓
RAG Server → Ollama: generates response WITH your code as context
         ↓
Response sent back to Open WebUI

Open WebUI is already configured to route through the RAG server via:

OPENAI_API_BASE_URL=http://rag-server:8001/v1

Step 1: Index your code

Option A: Drop repos in the repos directory

cd ~/docker/ai-stack/repos
git clone http://localhost:3001/your-user/your-repo.git
# Restart RAG server to trigger indexing:
docker restart rag-server

Option B: Use the ingest API

# From Gitea:
curl -X POST http://localhost:8001/ingest/repo \
  -H 'Content-Type: application/json' \
  -d '{"url": "http://gitea:3000/user/repo", "name": "my-repo"}'

# From GitHub:
curl -X POST http://localhost:8001/ingest/repo \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://github.com/user/repo", "name": "my-repo", "branch": "main"}'

Option C: Auto-index on push (Gitea webhook)

1. In Gitea → your repo → Settings → Webhooks → Add Webhook → Gitea
2. Target URL: http://rag-server:8001/webhook/gitea
3. Trigger: Push events
4. Now every git push to Gitea auto-reindexes that repo

Step 2: Chat about your code

Just ask questions in Open WebUI. The RAG server automatically retrieves relevant code:

• "What does the authenticate function do?"
• "How is the database connection configured?"
• "Show me all the API endpoints"
• "What tests exist for the user model?"

The model sees the actual code snippets and responds based on them — not hallucinating.

What gets indexed

Supported file types: .py, .js, .ts, .tsx, .jsx, .go, .rs, .java, .c, .cpp, .h, .hpp, .cs, .rb, .sh, .yaml, .yml, .toml, .sql, .md

Smart chunking:

• Python: AST-parsed — each function and class is its own searchable chunk
• Other languages: Regex-split on function, class, const, func, impl, etc.
• Fallback: Sliding window (1200 chars, 200 char overlap)

Skipped directories: node_modules, .git, __pycache__, dist, build, .venv, venv, vendor, target, bin, obj

Checking RAG status

curl http://localhost:8001/health

Returns document counts per collection (code, papers) and overall status.

RAG server vs Knowledge Collections vs Memories

These are three separate systems in the stack. Understanding the difference matters:

	RAG Server	Knowledge Collections	Memories
What	Code-aware retrieval layer	Open WebUI's built-in document RAG	Short facts about the user
Content	Git repos (auto-indexed)	PDFs, text files you upload	Extracted from conversations
Chunking	AST/regex (code-aware)	Generic text chunking	Single sentences
Activation	Always on (every chat)	Per-chat (# tag)	Global (every chat)
Best for	"What does this function do?"	Project docs, research notes	"Remember I prefer Python"
Context cost	~1500-2500 tokens (top 6 chunks)	~1500-2500 tokens (top K chunks)	~200 tokens

For code questions: The RAG server handles this automatically — no setup needed beyond indexing your repos.

For project docs/notes: Use Knowledge Collections — type # to scope per-chat.

For personal preferences: Use Memories (sparingly — they're global).

---

Image Generation from Open WebUI

Open WebUI can generate images inline in chat conversations using ComfyUI as the backend. When configured, you can ask any model to "generate an image of..." and it will call ComfyUI to create the image.

How it works

Open WebUI natively supports these image generation engines:

• ComfyUI — Node-based, best Open WebUI integration, local
• AUTOMATIC1111 — Stable Diffusion WebUI, local
• OpenAI DALL-E — Cloud API
• Gemini — Cloud API

InvokeAI does NOT have a compatible API for Open WebUI integration. It works great as a standalone tool at http://<ip>:9090 but cannot be called from within Open WebUI chats. For chat-integrated image generation, use ComfyUI.

Setup: ComfyUI + Open WebUI (recommended)

If you selected ComfyUI during setup, the environment variables are already configured. You just need to install a model and set up a workflow.

Step 1: Install a Stable Diffusion model in ComfyUI

# Open ComfyUI at http://<ip>:8188
# Use the built-in Model Manager to download a model, or manually:
docker exec comfyui bash -c "cd /opt/ComfyUI/models/checkpoints && \
  wget -q 'https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors'"

Or download any .safetensors checkpoint and copy it in:

docker cp ~/Downloads/my-model.safetensors comfyui:/opt/ComfyUI/models/checkpoints/

Step 2: Create and export a workflow

1. Open ComfyUI at http://<ip>:8188
2. Build or load a workflow (the default text-to-image workflow works)
3. Click the gear icon → enable Dev Mode
4. Click Save (API Format) — this downloads workflow_api.json

Step 3: Configure Open WebUI

1. Open WebUI → Admin → Settings → Images
2. Set Engine to ComfyUI
3. Set URL to http://comfyui:8188 (container networking, already set via env vars)
4. Click Import Workflow and upload your workflow_api.json
5. Map the prompt node (usually the KSampler or CLIPTextEncode node)
6. Save settings

Step 4: Generate images in chat

In any Open WebUI chat, type something like:

• "Generate an image of a mountain landscape at sunset"
• "Create a photo of a cyberpunk city"

The model will detect the image generation request and pass it to ComfyUI.

Using LoRAs with Open WebUI (workflow-per-style)

LoRAs let you apply trained styles (anime, photorealistic, specific characters, etc.) to image generation. The trick: bake the LoRA into a ComfyUI workflow, export it, and import into Open WebUI. After that, you iterate from chat without touching ComfyUI.

Step 1: Import your LoRA file

./comfyui-import-lora.sh ~/Downloads/my-style-lora.safetensors "Anime Style"

Or manually:

docker cp ~/Downloads/my-style-lora.safetensors comfyui:/opt/ComfyUI/models/loras/

Step 2: Build a workflow with the LoRA (one-time)

1. Open ComfyUI at http://<ip>:8188
2. Load the default text-to-image workflow
3. Add a Load LoRA node: right-click → Add Node → loaders → Load LoRA
4. Wire it between the checkpoint and the rest of the pipeline:

   [Load Checkpoint] → MODEL → [Load LoRA] → MODEL → [KSampler]
                     → CLIP  →             → CLIP  → [CLIP Text Encode]
   

5. Select your LoRA file, set strength_model and strength_clip (start 0.7–0.85)
6. Test it — click Queue Prompt and verify it works
7. Enable Dev Mode (gear icon) → click Save (API Format)

Step 3: Import into Open WebUI

1. Open WebUI → Admin → Settings → Images
2. Click Import Workflow → upload the workflow_api.json
3. Map the prompt node (usually CLIPTextEncode)
4. Save

Now every "Generate an image of..." in chat uses your LoRA automatically.

Combining multiple LoRAs in one image

You can chain multiple Load LoRA nodes to blend styles:

[Load Checkpoint] → [Load LoRA 1] → [Load LoRA 2] → [KSampler]
                     (anime, 0.7)    (lighting, 0.5)

Each LoRA has its own strength_model / strength_clip sliders. Export the combined workflow as usual — both LoRAs are baked in and always applied.

Managing multiple LoRA styles

Each LoRA combo needs its own exported workflow. Practical approach:

Workflow file	LoRAs	Use case
anime-only.json	Anime @ 0.8	Anime style
anime-cinematic.json	Anime @ 0.7 + Cinematic @ 0.5	Anime + dramatic lighting
photorealistic.json	Photorealistic @ 0.9	Photo look
base-model.json	None	No LoRA, base checkpoint only

Switch between them in Admin → Settings → Images → Import Workflow. The active workflow applies to all image generation requests.

Important: There are no keywords or trigger words to activate LoRAs from chat. LoRAs are hardwired in the workflow JSON — whatever prompt you type, the LoRA(s) always apply. To change styles, swap the workflow in OWUI settings.

LoRA compatibility

LoRA trained on	Must use checkpoint
SD 1.5	Any SD 1.5 model (e.g. v1-5-pruned-emaonly.safetensors)
SDXL	Any SDXL model (e.g. sd_xl_base_1.0.safetensors)
Flux	Flux checkpoint

Mismatched architectures will produce errors or garbage output.

IP-Adapter: same face, different settings (reference image workflows)

IP-Adapter lets you upload a reference photo and generate new images that preserve the face/subject while changing everything else — age, emotion, setting, art style. This is different from LoRAs (which bake in a trained style). IP-Adapter works from a single image with no training needed.

What you can do

Task	How	Example prompt
Different setting	Reference face + new scene	"same person, sitting in a cafe in Paris"
Age up/down	Reference face + age prompt	"same person as an elderly man" or "same person as a child"
Emotions	Reference face + emotion	"same person, laughing" or "same person, crying"
Art style	Reference face + style	"same person, oil painting, renaissance style"
Pose change	Reference face + pose	"same person, looking over shoulder, dramatic lighting"

Quick install

./comfyui-install-ipadapter.sh              # SDXL models (~2.5GB)
./comfyui-install-ipadapter.sh --faceid     # + FaceID for stronger face lock
./comfyui-install-ipadapter.sh --all        # SDXL + SD 1.5 models

Then restart ComfyUI: docker restart comfyui

Basic workflow in ComfyUI

1. Open http://<ip>:8188
2. Add nodes: right-click → Add Node → ipadapter
3. Wire up:

   [Load Checkpoint] → MODEL → [IPAdapter Apply] → [KSampler] → [VAE Decode] → [Save Image]
                      → CLIP → [CLIP Text Encode] → positive →
   [Load Image (your photo)] → [IPAdapter Unified Loader] → ipadapter →
   

4. IPAdapter Unified Loader: set preset to PLUS FACE (portrait)
5. IPAdapter Apply: set weight 0.7–1.0 (higher = more faithful to reference)
6. Text prompt: describe the new scene/emotion/age
7. Click Queue Prompt

Adjusting face fidelity

Weight	Effect
0.5–0.6	Loose reference — inspired by the face but not a match
0.7–0.8	Good balance — recognizable face, creative freedom in scene
0.9–1.0	Strong lock — very close to reference face
FaceID preset	Strongest — uses face detection for identity lock

Important: ComfyUI only (not from Open WebUI chat)

IP-Adapter workflows require uploading a reference image to ComfyUI. Open WebUI's image generation integration only sends text prompts — it can't attach reference images. For this use case, work directly in ComfyUI at http://<ip>:8188.

For text-only image generation from Open WebUI chat, use LoRA workflows instead.

Setup: AUTOMATIC1111 (alternative)

If you prefer AUTOMATIC1111 over ComfyUI:

1. Run AUTOMATIC1111 with the --api flag
2. In Open WebUI → Admin → Settings → Images:
   • Engine: Automatic1111
   • URL: http://host.docker.internal:7860 (or container name if in Docker)
3. Environment variables (alternative to UI config):

   ENABLE_IMAGE_GENERATION=true
   IMAGE_GENERATION_ENGINE=automatic1111
   AUTOMATIC1111_BASE_URL=http://host.docker.internal:7860
   

Environment variables reference

Variable	Default	Description
ENABLE_IMAGE_GENERATION	false	Enable image generation feature
IMAGE_GENERATION_ENGINE	—	comfyui, automatic1111, openai, or gemini
IMAGE_GENERATION_MODEL	—	Model ID for generation
IMAGE_SIZE	512x512	Default output size
COMFYUI_BASE_URL	—	ComfyUI API URL (e.g. http://comfyui:8188)
COMFYUI_API_KEY	—	ComfyUI API key (if auth enabled)
COMFYUI_WORKFLOW	—	Custom workflow JSON (API format)
AUTOMATIC1111_BASE_URL	—	AUTOMATIC1111 API URL
AUTOMATIC1111_API_AUTH	—	Auth credentials (user:pass)

Installing Functions & Actions (without community signup)

The Open WebUI community hub (openwebui.com) requires a free account to download functions. If you get parse errors pasting URLs or don't want to sign up, you can install functions manually by pasting their source code directly.

Manual installation (no account needed)

1. Find the function's source on GitHub — most are in the open-webui/functions repo or linked from community pages
2. Copy the raw Python source code (the entire .py file)
3. In Open WebUI → Workspace → Functions → click ＋ (Create)
4. Paste the code into the editor
5. Give it a name and save
6. Enable it under Workspace → Functions — toggle it on globally or per-model

For Actions (like Generate Image), the process is the same but go to Workspace → Functions and set the function type to Action in the metadata.

Recommended functions

Function	Type	What it does
Auto Memory	Filter	Automatically extracts and stores facts from conversations as persistent memories
Generate Image	Action	Adds a "Generate Image" button to messages for quick image generation

Auto Memory setup

Auto Memory is a filter function that runs after each message exchange, uses an LLM call to extract noteworthy facts, and stores them as user memories in Open WebUI's built-in memory system.

Step 1: Install the function

Use the manual method above. The source code is at: https://github.com/open-webui/functions — search for auto_memory or adaptive_memory

Step 2: Configure the function settings

After installing, click the gear icon on the function to configure:

Setting	Value for local stack	Notes
openai_api_url	http://ollama:11434/v1	Ollama's OpenAI-compatible endpoint
model	Your chat model (e.g. qwen2.5:14b)	Used for memory extraction LLM calls
api_key	ollama	Any non-empty string — Ollama ignores auth
context_window_n	4 (default)	Number of related memories injected per message
similarity_score_filter	Leave default	Cosine similarity threshold for memory retrieval
messages_to_consider	Leave default	How many recent messages to analyze
allow_modify_user_memories	false	Keep off — lets LLM delete/edit memories via API if enabled

Step 3: Enable memories per user

Each user must enable the memory feature individually:

1. Click your profile icon → Settings → Personalization
2. Toggle Memory to ON

Without this, the function installs but does nothing.

Generate Image action setup

1. Install via manual method above (search for generate_image in the functions repo)
2. Configure the function settings:
   • Uses your existing Open WebUI image generation settings (ComfyUI/A1111)
   • No additional API configuration needed if image generation already works
3. After installing, a "Generate Image" button appears on AI messages
4. Clicking it sends the message content as an image generation prompt

Memory system: how it works and context impact

Open WebUI has two separate systems for persistent knowledge. Understanding the difference is critical for managing your context window.

Built-in Memories vs Knowledge Collections

	Memories	Knowledge
What	Short facts extracted from chats	Document collections (PDFs, text files)
How stored	Text + vector embeddings in DB	Chunked documents + embeddings in ChromaDB
How injected	Prepended to system prompt every message	Retrieved via RAG only when relevant chunks match
Scope	Global per user — injected into ALL chats	Per-chat — you choose which collection with #
Context cost	Always consumed, every message	Only consumed when you tag a collection
Control	All or nothing (on/off globally)	Fine-grained (pick per conversation)

Context window consumption by memories

Each memory is roughly 20–80 tokens (a sentence or two). With the default context_window_n=4 setting in Auto Memory, 4 related memories are injected per message.

Estimated context consumption per message:

Stored memories	Injected per msg	Tokens consumed	% of 8K context	% of 32K context
10	~4	~200	2.5%	0.6%
50	~4	~200	2.5%	0.6%
200	~4	~200	2.5%	0.6%

The key insight: only the top N similar memories are injected (default 4), not all of them. So having 200 memories doesn't consume more context than having 10 — the retrieval system picks the most relevant ones.

However, the memories are injected on every single message in every chat. This is the overhead cost.

Context consumed by chat history (the bigger problem)

The real context pressure comes from conversation history, not memories. Here's what actually fills your context window:

Messages in chat	~Tokens used	% of 8K	% of 32K
5 exchanges (10 msgs)	~2,000–4,000	25–50%	6–12%
20 exchanges (40 msgs)	~8,000–16,000	100%+ (truncated)	25–50%
100 exchanges (200 msgs)	~40,000–80,000	way over	100%+ (truncated)

With a small model running 8K context, you'll hit the limit after ~10-20 exchanges. The model starts dropping earlier messages. Memories add a small fixed overhead (~200 tokens) on top of this.

Across multiple separate chats

Good news: separate chats do NOT share context windows. Each chat starts fresh. The only cross-chat cost is the ~200 tokens of memories injected into each new chat's system prompt.

So 5, 20, or 100 separate chats don't accumulate — each one independently uses the context window. Memories are the only thing that carries over.

Project-scoped memory (avoiding global memory pollution)

Open WebUI does NOT have native project-scoped memory. Memories are global per user — every fact extracted from any chat gets injected into every other chat.

This is a known limitation. Here are workarounds:

Option 1: Use Knowledge Collections instead of Memories (recommended)

Knowledge collections give you the scoping you want:

1. Create a collection: Workspace → Knowledge → Create Collection (e.g. "GPU Research Project")
2. Add documents: Upload PDFs, text files, or paste notes into the collection
3. Use per-chat: In any chat, type # and select your collection — only that chat gets the context
4. One-off chats stay clean: Don't tag a collection, and no project context is injected

This is the closest thing to "projects" in Open WebUI. You can have separate knowledge collections for separate projects, and only pull them in when relevant.

Option 2: Disable Auto Memory, use manual memories

1. Turn off the Auto Memory function
2. Manually add memories via Profile → Settings → Personalization → Memories
3. Keep only universally useful facts (your name, preferences, etc.)
4. Use Knowledge collections for project-specific context

Option 3: Periodically clear memories

Profile → Settings → Personalization → Memories → Clear All — nuclear option, but keeps things clean between projects.

Working with Knowledge Collections (small context window survival guide)

With small local models (4B-9B, 4K-8K context), you'll hit the context window limit fast — often after 10-20 exchanges. Here's how to work effectively despite that.

How Knowledge Collections actually work

When you type # in chat and select a collection, Open WebUI does RAG retrieval — it finds the most relevant chunks from the collection, not the whole thing. This is efficient and doesn't blow up your context window.

• Default chunk size: ~500 tokens
• Typically 3-5 relevant chunks are retrieved per query (~1500-2500 tokens)
• The chunks are injected as context alongside your message

Creating Knowledge Collections from chat conversations

There is no built-in "export chat to knowledge" button — this is a requested but unimplemented feature. Here's the practical workflow:

The handoff method (like the ChatGPT dark ages, but structured):

1. When you're ~60-70% through your context window (you'll feel the model getting fuzzy), ask:

   Summarize everything we've discussed and decided so far. Include:
   - Key decisions made
   - Current state of the work
   - What still needs to be done
   - Any important details or constraints
   Format as a structured document I can use to continue this conversation.
   

2. Copy the summary output
3. Go to Workspace → Knowledge → Create Collection (e.g. "GPU Build - Session 1")
4. Click Add Content → paste the summary as a text file (.txt or .md)
5. Start a new chat, type # and select your collection, then continue where you left off

Ongoing project workflow:

Chat 1: Research phase
  → Ask for summary at end
  → Save summary to "Project X" knowledge collection

Chat 2: Type # → select "Project X" → continue
  → Model gets relevant context via RAG
  → Ask for updated summary at end
  → Add updated summary to collection (replace or append)

Chat 3: Type # → select "Project X" → continue
  → Repeat...

Each chat starts fresh with full context window available, but can pull in relevant history from previous sessions via RAG.

Useful community functions for context management

Install these manually (Workspace → Functions → Create, paste the code):

Function	What it does
Checkpoint Summarization Filter	Auto-summarizes conversation history when context gets long
Chat Context Clipper	Keeps only the last N messages, preserves system prompt and first message
Context Length Filter	Hard limits on turns (default 25) and tokens (default 10,000)

The Checkpoint Summarization Filter is the closest to automatic handoff — it summarizes older messages so the model can keep going without losing track.

How Open WebUI handles context overflow

By default, Open WebUI truncates old messages (drops them silently) — it does NOT auto-summarize. The model just loses access to earlier conversation. This is why you suddenly feel like the model "forgot" what you were talking about.

You can control this with:

• Context Length Filter function: set max turns and token limits explicitly
• Chat Context Clipper: keeps latest N messages, always preserves system prompt + first message pair
• Or do manual handoffs before you hit the limit

RAG tuning for better knowledge retrieval

If your knowledge collections aren't returning good results, tune these in Admin → Settings → Documents:

Setting	Default	Recommendation
Chunk Size	500	300 for factual docs, 800 for narrative/code
Chunk Overlap	100	50-100 (higher = better continuity, more tokens)
Top K	4	3-5 (more = more context consumed)
Relevance Threshold	0.0	0.3-0.5 (filters out low-quality matches)

Important: Chunk size cannot exceed your embedding model's token limit. nomic-embed-text supports up to 8192 tokens, so you have plenty of headroom.

The honest comparison: Local AI vs Claude Code

	Local AI (Open WebUI + Ollama)	Claude Code
Context window	4K-32K (model dependent)	200K
Cross-session memory	Manual handoffs or Auto Memory	Automatic (CLAUDE.md, project memory)
Project scoping	Knowledge collections (manual)	Built-in (each project has its own context)
Code awareness	RAG server auto-retrieves relevant chunks from indexed repos	Reads your entire codebase directly
Continuation	New chat + #collection handoff	"Let's finish X" just works
Code editing	Can't edit files (chat only)	Reads, writes, runs code directly
Cost	Free (your electricity)	API usage fees
Privacy	100% local	Cloud-based
Offline	Works without internet	Requires internet

Local AI requires more manual workflow management but has real code awareness via the RAG server. For code-heavy editing and multi-step tasks, Claude Code is dramatically better. For private code Q&A, document research, and learning — the local stack with RAG + knowledge collections is solid.

Channels (beta feature)

Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation.

Realistic expectations by model size

Not all models can do all tasks. Here's what to actually expect:

Task	4B (qwen3.5:4b)	9B (qwen3.5:9b)	35B MoE (qwen3.5-35b-a3b)	14B+ dense
Answer simple questions	OK	Good	Good	Good
Explain existing code (with RAG)	OK	Good	Good	Good
Fix a simple bug (typo, off-by-one)	Maybe	Usually	Usually	Yes
Write a small utility function	Shaky	OK	Good	Good
Fix logic error across 2-3 functions	No	Maybe	Usually	Usually
Write a new feature (multiple files)	No	Shaky	Maybe	Maybe
Refactor with style consistency	No	No	Sometimes	Sometimes
Summarize a conversation for handoff	OK	Good	Good	Good

The 35B MoE model (qwen3.5-35b-a3b) is the sweet spot for small GPUs. It was trained as a 35B model but only activates 3B parameters per token. This means it has the knowledge of a 35B model with the VRAM footprint closer to a 4B. On a 6GB card it may fit (VRAM usage varies with context length and KV cache settings).

Bottom line for a 6GB GPU:

• Use qwen3.5:4b for quick chat, explanations, and summarization
• Try qwen3.5-35b-a3b for code tasks — if it fits, it will be significantly better than 4B
• Use Claude Code for anything that requires reading/writing multiple files or complex reasoning
• The RAG server helps a lot — even a 4B model gives useful answers when it has the right code chunks in context

Fixing models that output code instead of natural language

If your model (especially smaller ones like Qwen 3.5) responds with Python code blocks instead of plain English answers (as shown in the screenshot), this is a common behavior with code-optimized models.

Why it happens

• Small models like qwen3.5:4b are heavily optimized for coding tasks
• They interpret ambiguous questions as "write me a program" instead of "answer my question"
• The model's training data skews toward code generation at smaller parameter counts
• Thinking/reasoning mode (if enabled) amplifies this tendency

Fix: Set a system prompt in Open WebUI

1. Go to Admin → Settings → Interface → Default System Prompt (for all chats)

   Or per-model: Workspace → Models → select model → System Prompt

2. Use a system prompt like:

You are a helpful assistant. Answer questions in clear, natural language.
Explain concepts conversationally. Only include code if the user explicitly
asks for code or a script. When discussing technical topics, use plain
English explanations with examples, not programs.

3. For the specific GPU pricing question in the screenshot, the model should have responded with a comparison table in text, not a Python script. A good system prompt prevents this.

Alternative: Use a chat-optimized model

Some models are better at conversational responses:

Model	Size	Better for chat?
qwen2.5:14b	14B	Yes — more balanced
qwen2.5:7b	7B	Yes — good general chat
llama3.1:8b	8B	Yes — conversational
qwen2.5-coder:7b	7B	No — code-focused
qwen3.5:4b	4B	No — too small, code-biased

Avoid using -coder variants or very small models (≤4B) for general chat. Use them only when you actually want code.

VRAM considerations

Image generation and LLM inference compete for GPU memory. With a single GPU:

VRAM	Recommendation
≥ 24GB	Run both LLM + image gen simultaneously
12–24GB	Use smaller LLM when generating images, or stop Ollama first
< 12GB	Run one at a time — stop Ollama before generating images

ComfyUI models typically need 4–8GB VRAM (SD 1.5: ~4GB, SDXL: ~7GB, Flux: ~12GB).

---

InvokeAI vs ComfyUI — which to use when

Both are installed by the setup script. Here's when to use each:

Task	InvokeAI (:9090)	ComfyUI (:8188)
Friendly UI, sliders, drag-and-drop	Yes	No (node editor)
Use a LoRA you trained on RunPod	Yes — just import + select	Yes — but wire LoRA Loader node
Same face, different settings (img2img)	Yes — Image to Image tab	Yes — IP-Adapter nodes
Age/emotion/scene changes	Yes — img2img + prompt	Yes — IP-Adapter + prompt
Chat-integrated image gen (from Open WebUI)	No (no OWUI API)	Yes (workflow export)
Maximum flexibility / custom pipelines	No	Yes
Learning curve	Low	High

Bottom line: Use InvokeAI for the interactive "play with images" workflow. Use ComfyUI only when you need Open WebUI chat integration or advanced node pipelines.

Using InvokeAI for image iteration

InvokeAI is the easier path for what you want — take a reference image and generate variations with different settings, ages, emotions, art styles.

Step 1: Install a base model (required first)

Before LoRAs or img2img will work, InvokeAI needs a base Stable Diffusion model.

# Option A: Download SD 1.5 (~4GB VRAM needed) via the container
docker exec invokeai invokeai-model-install --add stabilityai/stable-diffusion-v1-5

# Option B: Download SDXL (~7GB VRAM needed)
docker exec invokeai invokeai-model-install --add stabilityai/stable-diffusion-xl-base-1.0

Or use the UI: open http://<ip>:9090 → Model Manager (cube icon) → Starter Models tab → install SD 1.5 or SDXL.

6GB GPU note: SD 1.5 fits comfortably. SDXL is tight — it may work with float16 (already set) but could be slow. Start with SD 1.5.

Step 2: Import your RunPod LoRA

# Copy the .safetensors file from wherever you downloaded it:
./invokeai-import-lora.sh ~/Downloads/my-lora.safetensors "My Character"

Then in the InvokeAI UI:

1. Model Manager (cube icon) → Scan for Models
2. Your LoRA should appear in the list

If it doesn't:

• Click Add Model → Scan Folder → enter /invokeai/models/lora
• Make sure the LoRA architecture matches your base model (SD 1.5 LoRA needs SD 1.5 base)

Step 3: Generate with your LoRA (text-to-image)

1. Go to the Text to Image tab
2. Select your base model (must match the LoRA's training base)
3. In the left panel, expand the LoRA section (below model selector)
4. Click + → select your LoRA → set weight to 0.7–0.85
5. Write a prompt: "portrait of [subject], smiling, studio lighting"
6. Click Invoke

Step 4: Iterate on an image (img2img)

This is where InvokeAI shines for your use case — take an image and riff on it:

1. Switch to the Image to Image tab
2. Drag your reference photo onto the canvas (or click to upload)
3. Keep your LoRA active (same as above)
4. Set the Denoising Strength slider:

Strength	Effect
0.2–0.3	Subtle tweaks — mostly keeps the original, minor style changes
0.4–0.5	Moderate changes — recognizable but different mood/lighting
0.6–0.7	Significant changes — same composition, new details/style
0.8–1.0	Major rewrite — loosely inspired by original, mostly new

5. Change the prompt to describe what you want different:
   • Age up: "same person, elderly, wrinkles, grey hair, wise expression"
   • Age down: "same person as a young child, bright eyes, playground"
   • Emotion: "same person, laughing joyfully" or "same person, crying, dramatic lighting"
   • Setting: "same person, sitting in a Parisian cafe, afternoon light"
   • Art style: "same person, oil painting, renaissance style, dramatic chiaroscuro"
6. Click Invoke — iterate by adjusting strength and prompt

Step 5: Use the Unified Canvas for inpainting

This is the "fix this specific thing" workflow — brush over a hand, arm, face, background, whatever, and regenerate just that area while keeping everything else untouched.

1. Switch to the Unified Canvas tab
2. Upload or paste your image
3. Select the Mask brush tool (not the paint brush)
4. Brush over only the area you want to change — everything else stays locked
5. Write a prompt describing what the masked area should become
6. Set Denoising Strength to 0.6–0.8 (higher = more change)
7. Click Invoke — only the masked pixels regenerate

Common inpainting fixes:

Problem	Mask	Prompt
Hand in wrong position	Brush over the arm/hand	"natural hand resting at side, relaxed pose"
Extra fingers	Brush over the hand	"normal human hand, five fingers, anatomically correct"
Weird eyes	Brush over both eyes	"natural eyes, looking at camera, detailed iris"
Bad background	Brush over background only	"clean studio backdrop" or "forest trail, golden hour"
Wrong clothing	Brush over the clothing area	"wearing blue denim jacket, casual style"
Face swap / aging	Brush over the face	"same person, elderly, wrinkles" or "same person as child"

Tips for better inpainting results:

• Mask slightly larger than the problem area — gives the model room to blend edges
• Use soft brush edges (lower brush hardness) for more natural blending
• If the result has visible seams, increase your mask area and try again
• Lower denoising (0.4–0.5) for subtle fixes, higher (0.7–0.9) for major changes
• Keep your LoRA active during inpainting — it maintains the trained style/face consistency

Troubleshooting

• Greyed-out upload/LoRA buttons: Install a base model first (Step 1). InvokeAI disables most features until a checkpoint is loaded.
• Model not synced: After copying files via script, click Scan for Models in Model Manager.
• Architecture mismatch: A LoRA trained on SD 1.5 only works with SD 1.5 base models — not SDXL. Check what your RunPod training used.
• Out of VRAM: Try SD 1.5 instead of SDXL, or reduce image size to 512x512.
• Use the import script: The greyed-out UI upload can be bypassed entirely by using invokeai-import-lora.sh to copy files directly into the model volume.

---

Updating

Re-run the setup script — it detects an existing install and skips prereqs:

./laptop_full_setup.sh
# or
./laptop_full_setup.sh --force   # also overwrites config files

---

GPU / Model Tiers (local-ai-setup.sh)

VRAM	Chat model	Code model	Context
≥ 14 GB	qwen2.5:14b	qwen2.5-coder:14b	32k
8–14 GB	qwen2.5:14b	qwen2.5-coder:7b	16k
4–8 GB	qwen2.5:7b	qwen2.5-coder:7b	8k
CPU	qwen2.5:7b	qwen2.5-coder:7b	4k

Embed model is always nomic-embed-text (required for RAG).

VRAM reality check

Ollama will always try to run any model — it silently offloads layers to CPU when VRAM is insufficient. The model still works but gets significantly slower. The setup script's "fully in VRAM" label can be misleading.

Actual VRAM needed for common models (Q4_K_M quantization):

Model	Download size	VRAM for inference	Fits in 6GB?	Fits in 8GB?
qwen3.5:4b	~2.5 GB	~3.5–4 GB	Yes	Yes
qwen2.5:7b	~4.4 GB	~5.5–6 GB	Tight	Yes
qwen3.5:9b	~5.5 GB	~6.5–7 GB	No — partial CPU offload	Tight
qwen2.5:14b	~8.7 GB	~10–11 GB	No	No
qwen3.5-35b-a3b (MoE)	~20 GB	~3.5 GB active	Yes (only 3B active)	Yes

Why the file size != VRAM needed: Inference requires additional memory for KV cache, attention buffers, and CUDA overhead. Expect ~1-2 GB more than the model file size.

Signs of CPU offload (model too big for your VRAM):

• Tokens per second drops from 20-40 to 2-8
• nvidia-smi shows VRAM maxed out
• CPU usage spikes during generation
• First token takes much longer than usual

Image Generation Model Tiers

The setup-image-models.sh script detects your GPU and offers appropriate models:

VRAM	Available Models	Default	Notes
≥ 24GB	SD 1.5, SDXL, SDXL Turbo, Flux.1-schnell, Flux.1-dev	SDXL	All models, no constraints
12–23GB	SD 1.5, SDXL, SDXL Turbo, Flux.1-schnell	SDXL	Flux-dev too tight
8–11GB	SD 1.5, SDXL (tight), SDXL Turbo	SD 1.5	SDXL works at 512px, may be slow
4–7GB	SD 1.5 (float16)	SD 1.5	Only SD 1.5 fits
< 4GB	none	—	CPU generation not recommended

GPU sharing: Ollama and image generation share the GPU. Ollama auto-unloads models after its KEEP_ALIVE timeout (default 24h), so image gen gets full VRAM when the LLM is idle. For immediate unload: docker exec ollama ollama stop <model-name>

Multi-GPU scaling: With dual GPUs (e.g., 2× RTX 5000 = 32GB total), the VRAM is summed for tier selection. Both InvokeAI and ComfyUI will use all available GPUs.

# Install image models (auto-detects GPU):
./setup-image-models.sh

# Or auto-install the recommended default:
./setup-image-models.sh --auto