From 38d103fcc124edd86f37b1e0b1bdfc6f9d83e887 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 22 Mar 2026 19:13:40 +0000 Subject: [PATCH] Add memory, knowledge, function install, and model sizing docs - Manual function/action installation without community signup - Auto Memory setup with Ollama configuration - Memory vs Knowledge collections comparison (context impact) - Context window consumption analysis (memories are ~200 tokens fixed, conversation history is the real context hog) - Project-scoped memory workarounds (Knowledge collections recommended) - System prompt fix for models outputting code instead of natural language - VRAM reality check table (model file size != inference VRAM needed) - Qwen 9B does NOT fit in 6GB VRAM despite setup script claiming so https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu --- README.md | 208 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 204 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 76228ba..75ab8af 100644 --- a/README.md +++ b/README.md @@ -220,12 +220,190 @@ If you prefer AUTOMATIC1111 over ComfyUI: | `AUTOMATIC1111_BASE_URL` | — | AUTOMATIC1111 API URL | | `AUTOMATIC1111_API_AUTH` | — | Auth credentials (`user:pass`) | -### Recommended Open WebUI Functions +### Installing Functions & Actions (without community signup) -Install these from **Admin → Functions → ＋ → Import From Link** or search in Discover: +The Open WebUI community hub (`openwebui.com`) requires a free account to download functions. If you get parse errors pasting URLs or don't want to sign up, you can install functions manually by pasting their source code directly. -- **Auto Memory** — Automatically stores relevant info as persistent memories across chats -- **Generate Image** — Adds a "Generate Image" action button to messages for quick re-generation +#### Manual installation (no account needed) + +1. Find the function's source on GitHub — most are in the [open-webui/functions](https://github.com/open-webui/functions) repo or linked from community pages +2. Copy the **raw Python source code** (the entire `.py` file) +3. In Open WebUI → **Workspace** → **Functions** → click **＋** (Create) +4. Paste the code into the editor +5. Give it a name and save +6. Enable it under **Workspace** → **Functions** — toggle it on globally or per-model + +For **Actions** (like Generate Image), the process is the same but go to **Workspace** → **Functions** and set the function type to **Action** in the metadata. + +#### Recommended functions + +| Function | Type | What it does | +|----------|------|-------------| +| **Auto Memory** | Filter | Automatically extracts and stores facts from conversations as persistent memories | +| **Generate Image** | Action | Adds a "Generate Image" button to messages for quick image generation | + +### Auto Memory setup + +Auto Memory is a filter function that runs after each message exchange, uses an LLM call to extract noteworthy facts, and stores them as user memories in Open WebUI's built-in memory system. + +#### Step 1: Install the function + +Use the manual method above. The source code is at: +`https://github.com/open-webui/functions` — search for `auto_memory` or `adaptive_memory` + +#### Step 2: Configure the function settings + +After installing, click the **gear icon** on the function to configure: + +| Setting | Value for local stack | Notes | +|---------|----------------------|-------| +| `openai_api_url` | `http://ollama:11434/v1` | Ollama's OpenAI-compatible endpoint | +| `model` | Your chat model (e.g. `qwen2.5:14b`) | Used for memory extraction LLM calls | +| `api_key` | `ollama` | Any non-empty string — Ollama ignores auth | +| `context_window_n` | `4` (default) | Number of related memories injected per message | +| `similarity_score_filter` | Leave default | Cosine similarity threshold for memory retrieval | +| `messages_to_consider` | Leave default | How many recent messages to analyze | +| `allow_modify_user_memories` | `false` | **Keep off** — lets LLM delete/edit memories via API if enabled | + +#### Step 3: Enable memories per user + +Each user must enable the memory feature individually: +1. Click your **profile icon** → **Settings** → **Personalization** +2. Toggle **Memory** to **ON** + +Without this, the function installs but does nothing. + +### Generate Image action setup + +1. Install via manual method above (search for `generate_image` in the functions repo) +2. Configure the function settings: + - Uses your existing Open WebUI image generation settings (ComfyUI/A1111) + - No additional API configuration needed if image generation already works +3. After installing, a **"Generate Image"** button appears on AI messages +4. Clicking it sends the message content as an image generation prompt + +### Memory system: how it works and context impact + +Open WebUI has two separate systems for persistent knowledge. Understanding the difference is critical for managing your context window. + +#### Built-in Memories vs Knowledge Collections + +| | **Memories** | **Knowledge** | +|---|---|---| +| **What** | Short facts extracted from chats | Document collections (PDFs, text files) | +| **How stored** | Text + vector embeddings in DB | Chunked documents + embeddings in ChromaDB | +| **How injected** | Prepended to system prompt every message | Retrieved via RAG only when relevant chunks match | +| **Scope** | **Global per user** — injected into ALL chats | **Per-chat** — you choose which collection with `#` | +| **Context cost** | Always consumed, every message | Only consumed when you tag a collection | +| **Control** | All or nothing (on/off globally) | Fine-grained (pick per conversation) | + +#### Context window consumption by memories + +Each memory is roughly 20–80 tokens (a sentence or two). With the default `context_window_n=4` setting in Auto Memory, 4 related memories are injected per message. + +**Estimated context consumption per message:** + +| Stored memories | Injected per msg | Tokens consumed | % of 8K context | % of 32K context | +|----------------|-----------------|-----------------|-----------------|------------------| +| 10 | ~4 | ~200 | 2.5% | 0.6% | +| 50 | ~4 | ~200 | 2.5% | 0.6% | +| 200 | ~4 | ~200 | 2.5% | 0.6% | + +The key insight: **only the top N similar memories are injected** (default 4), not all of them. So having 200 memories doesn't consume more context than having 10 — the retrieval system picks the most relevant ones. + +However, **the memories are injected on every single message** in every chat. This is the overhead cost. + +#### Context consumed by chat history (the bigger problem) + +The real context pressure comes from **conversation history**, not memories. Here's what actually fills your context window: + +| Messages in chat | ~Tokens used | % of 8K | % of 32K | +|-----------------|-------------|---------|----------| +| 5 exchanges (10 msgs) | ~2,000–4,000 | 25–50% | 6–12% | +| 20 exchanges (40 msgs) | ~8,000–16,000 | **100%+ (truncated)** | 25–50% | +| 100 exchanges (200 msgs) | ~40,000–80,000 | **way over** | **100%+ (truncated)** | + +With a small model running 8K context, you'll hit the limit after ~10-20 exchanges. The model starts dropping earlier messages. Memories add a small fixed overhead (~200 tokens) on top of this. + +#### Across multiple separate chats + +**Good news:** separate chats do NOT share context windows. Each chat starts fresh. The only cross-chat cost is the ~200 tokens of memories injected into each new chat's system prompt. + +So 5, 20, or 100 separate chats don't accumulate — each one independently uses the context window. Memories are the only thing that carries over. + +### Project-scoped memory (avoiding global memory pollution) + +**Open WebUI does NOT have native project-scoped memory.** Memories are global per user — every fact extracted from any chat gets injected into every other chat. + +This is a known limitation. Here are workarounds: + +#### Option 1: Use Knowledge Collections instead of Memories (recommended) + +Knowledge collections give you the scoping you want: + +1. **Create a collection:** Workspace → Knowledge → Create Collection (e.g. "GPU Research Project") +2. **Add documents:** Upload PDFs, text files, or paste notes into the collection +3. **Use per-chat:** In any chat, type `#` and select your collection — only that chat gets the context +4. **One-off chats stay clean:** Don't tag a collection, and no project context is injected + +This is the closest thing to "projects" in Open WebUI. You can have separate knowledge collections for separate projects, and only pull them in when relevant. + +#### Option 2: Disable Auto Memory, use manual memories + +1. Turn off the Auto Memory function +2. Manually add memories via **Profile → Settings → Personalization → Memories** +3. Keep only universally useful facts (your name, preferences, etc.) +4. Use Knowledge collections for project-specific context + +#### Option 3: Periodically clear memories + +**Profile → Settings → Personalization → Memories → Clear All** — nuclear option, but keeps things clean between projects. + +### Channels (beta feature) + +Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation. + +### Fixing models that output code instead of natural language + +If your model (especially smaller ones like Qwen 3.5) responds with Python code blocks instead of plain English answers (as shown in the screenshot), this is a common behavior with code-optimized models. + +#### Why it happens + +- Small models like `qwen3.5:4b` are heavily optimized for coding tasks +- They interpret ambiguous questions as "write me a program" instead of "answer my question" +- The model's training data skews toward code generation at smaller parameter counts +- Thinking/reasoning mode (if enabled) amplifies this tendency + +#### Fix: Set a system prompt in Open WebUI + +1. Go to **Admin** → **Settings** → **Interface** → **Default System Prompt** (for all chats) + + Or per-model: **Workspace** → **Models** → select model → **System Prompt** + +2. Use a system prompt like: + +``` +You are a helpful assistant. Answer questions in clear, natural language. +Explain concepts conversationally. Only include code if the user explicitly +asks for code or a script. When discussing technical topics, use plain +English explanations with examples, not programs. +``` + +3. For the specific GPU pricing question in the screenshot, the model should have responded with a comparison table in text, not a Python script. A good system prompt prevents this. + +#### Alternative: Use a chat-optimized model + +Some models are better at conversational responses: + +| Model | Size | Better for chat? | +|-------|------|-----------------| +| `qwen2.5:14b` | 14B | Yes — more balanced | +| `qwen2.5:7b` | 7B | Yes — good general chat | +| `llama3.1:8b` | 8B | Yes — conversational | +| `qwen2.5-coder:7b` | 7B | No — code-focused | +| `qwen3.5:4b` | 4B | No — too small, code-biased | + +Avoid using `-coder` variants or very small models (≤4B) for general chat. Use them only when you actually want code. ### VRAM considerations @@ -295,3 +473,25 @@ Re-run the setup script — it detects an existing install and skips prereqs: | CPU | qwen2.5:7b | qwen2.5-coder:7b | 4k | Embed model is always `nomic-embed-text` (required for RAG). + +### VRAM reality check + +Ollama will **always try to run** any model — it silently offloads layers to CPU when VRAM is insufficient. The model still works but gets significantly slower. The setup script's "fully in VRAM" label can be misleading. + +**Actual VRAM needed for common models (Q4_K_M quantization):** + +| Model | Download size | VRAM for inference | Fits in 6GB? | Fits in 8GB? | +|-------|-------------|-------------------|-------------|-------------| +| `qwen3.5:4b` | ~2.5 GB | ~3.5–4 GB | Yes | Yes | +| `qwen2.5:7b` | ~4.4 GB | ~5.5–6 GB | Tight | Yes | +| `qwen3.5:9b` | ~5.5 GB | ~6.5–7 GB | **No — partial CPU offload** | Tight | +| `qwen2.5:14b` | ~8.7 GB | ~10–11 GB | No | No | +| `qwen3.5-35b-a3b` (MoE) | ~20 GB | ~3.5 GB active | Yes (only 3B active) | Yes | + +**Why the file size != VRAM needed:** Inference requires additional memory for KV cache, attention buffers, and CUDA overhead. Expect ~1-2 GB more than the model file size. + +**Signs of CPU offload (model too big for your VRAM):** +- Tokens per second drops from 20-40 to 2-8 +- `nvidia-smi` shows VRAM maxed out +- CPU usage spikes during generation +- First token takes much longer than usual