root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add memory, knowledge, function install, and model sizing docs

Browse Source 
- Manual function/action installation without community signup
- Auto Memory setup with Ollama configuration
- Memory vs Knowledge collections comparison (context impact)
- Context window consumption analysis (memories are ~200 tokens fixed,
  conversation history is the real context hog)
- Project-scoped memory workarounds (Knowledge collections recommended)
- System prompt fix for models outputting code instead of natural language
- VRAM reality check table (model file size != inference VRAM needed)
- Qwen 9B does NOT fit in 6GB VRAM despite setup script claiming so

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 19:13:40 +00:00
parent 8920a61156
commit 38d103fcc1
1 changed files with 204 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+204 -4
View File
@@ -220,12 +220,190 @@ If you prefer AUTOMATIC1111 over ComfyUI:
| `AUTOMATIC1111_BASE_URL` | — | AUTOMATIC1111 API URL |
| `AUTOMATIC1111_API_AUTH` | — | Auth credentials (`user:pass`) |
### Recommended Open WebUI Functions
### Installing Functions & Actions (without community signup)
Install these from **Admin → Functions → ＋ → Import From Link** or search in Discover:
The Open WebUI community hub (`openwebui.com`) requires a free account to download functions. If you get parse errors pasting URLs or don't want to sign up, you can install functions manually by pasting their source code directly.
- **Auto Memory** — Automatically stores relevant info as persistent memories across chats
- **Generate Image** — Adds a "Generate Image" action button to messages for quick re-generation
#### Manual installation (no account needed)
1. Find the function's source on GitHub — most are in the [open-webui/functions](https://github.com/open-webui/functions) repo or linked from community pages
2. Copy the **raw Python source code** (the entire `.py` file)
3. In Open WebUI → **Workspace** → **Functions** → click **＋** (Create)
4. Paste the code into the editor
5. Give it a name and save
6. Enable it under **Workspace** → **Functions** — toggle it on globally or per-model
For **Actions** (like Generate Image), the process is the same but go to **Workspace** → **Functions** and set the function type to **Action** in the metadata.
#### Recommended functions
| Function | Type | What it does |
|----------|------|-------------|
| **Auto Memory** | Filter | Automatically extracts and stores facts from conversations as persistent memories |
| **Generate Image** | Action | Adds a "Generate Image" button to messages for quick image generation |
### Auto Memory setup
Auto Memory is a filter function that runs after each message exchange, uses an LLM call to extract noteworthy facts, and stores them as user memories in Open WebUI's built-in memory system.
#### Step 1: Install the function
Use the manual method above. The source code is at:
`https://github.com/open-webui/functions` — search for `auto_memory` or `adaptive_memory`
#### Step 2: Configure the function settings
After installing, click the **gear icon** on the function to configure:
| Setting | Value for local stack | Notes |
|---------|----------------------|-------|
| `openai_api_url` | `http://ollama:11434/v1` | Ollama's OpenAI-compatible endpoint |
| `model` | Your chat model (e.g. `qwen2.5:14b`) | Used for memory extraction LLM calls |
| `api_key` | `ollama` | Any non-empty string — Ollama ignores auth |
| `context_window_n` | `4` (default) | Number of related memories injected per message |
| `similarity_score_filter` | Leave default | Cosine similarity threshold for memory retrieval |
| `messages_to_consider` | Leave default | How many recent messages to analyze |
| `allow_modify_user_memories` | `false` | **Keep off** — lets LLM delete/edit memories via API if enabled |
#### Step 3: Enable memories per user
Each user must enable the memory feature individually:
1. Click your **profile icon** → **Settings** → **Personalization**
2. Toggle **Memory** to **ON**
Without this, the function installs but does nothing.
### Generate Image action setup
1. Install via manual method above (search for `generate_image` in the functions repo)
2. Configure the function settings:
- Uses your existing Open WebUI image generation settings (ComfyUI/A1111)
- No additional API configuration needed if image generation already works
3. After installing, a **"Generate Image"** button appears on AI messages
4. Clicking it sends the message content as an image generation prompt
### Memory system: how it works and context impact
Open WebUI has two separate systems for persistent knowledge. Understanding the difference is critical for managing your context window.
#### Built-in Memories vs Knowledge Collections
| | **Memories** | **Knowledge** |
|---|---|---|
| **What** | Short facts extracted from chats | Document collections (PDFs, text files) |
| **How stored** | Text + vector embeddings in DB | Chunked documents + embeddings in ChromaDB |
| **How injected** | Prepended to system prompt every message | Retrieved via RAG only when relevant chunks match |
| **Scope** | **Global per user** — injected into ALL chats | **Per-chat** — you choose which collection with `#` |
| **Context cost** | Always consumed, every message | Only consumed when you tag a collection |
| **Control** | All or nothing (on/off globally) | Fine-grained (pick per conversation) |
#### Context window consumption by memories
Each memory is roughly 20–80 tokens (a sentence or two). With the default `context_window_n=4` setting in Auto Memory, 4 related memories are injected per message.
**Estimated context consumption per message:**
| Stored memories | Injected per msg | Tokens consumed | % of 8K context | % of 32K context |
|----------------|-----------------|-----------------|-----------------|------------------|
| 10 | ~4 | ~200 | 2.5% | 0.6% |
| 50 | ~4 | ~200 | 2.5% | 0.6% |
| 200 | ~4 | ~200 | 2.5% | 0.6% |
The key insight: **only the top N similar memories are injected** (default 4), not all of them. So having 200 memories doesn't consume more context than having 10 — the retrieval system picks the most relevant ones.
However, **the memories are injected on every single message** in every chat. This is the overhead cost.
#### Context consumed by chat history (the bigger problem)
The real context pressure comes from **conversation history**, not memories. Here's what actually fills your context window:
| Messages in chat | ~Tokens used | % of 8K | % of 32K |
|-----------------|-------------|---------|----------|
| 5 exchanges (10 msgs) | ~2,000–4,000 | 25–50% | 6–12% |
| 20 exchanges (40 msgs) | ~8,000–16,000 | **100%+ (truncated)** | 25–50% |
| 100 exchanges (200 msgs) | ~40,000–80,000 | **way over** | **100%+ (truncated)** |
With a small model running 8K context, you'll hit the limit after ~10-20 exchanges. The model starts dropping earlier messages. Memories add a small fixed overhead (~200 tokens) on top of this.
#### Across multiple separate chats
**Good news:** separate chats do NOT share context windows. Each chat starts fresh. The only cross-chat cost is the ~200 tokens of memories injected into each new chat's system prompt.
So 5, 20, or 100 separate chats don't accumulate — each one independently uses the context window. Memories are the only thing that carries over.
### Project-scoped memory (avoiding global memory pollution)
**Open WebUI does NOT have native project-scoped memory.** Memories are global per user — every fact extracted from any chat gets injected into every other chat.
This is a known limitation. Here are workarounds:
#### Option 1: Use Knowledge Collections instead of Memories (recommended)
Knowledge collections give you the scoping you want:
1. **Create a collection:** Workspace → Knowledge → Create Collection (e.g. "GPU Research Project")
2. **Add documents:** Upload PDFs, text files, or paste notes into the collection
3. **Use per-chat:** In any chat, type `#` and select your collection — only that chat gets the context
4. **One-off chats stay clean:** Don't tag a collection, and no project context is injected
This is the closest thing to "projects" in Open WebUI. You can have separate knowledge collections for separate projects, and only pull them in when relevant.
#### Option 2: Disable Auto Memory, use manual memories
1. Turn off the Auto Memory function
2. Manually add memories via **Profile → Settings → Personalization → Memories**
3. Keep only universally useful facts (your name, preferences, etc.)
4. Use Knowledge collections for project-specific context
#### Option 3: Periodically clear memories
**Profile → Settings → Personalization → Memories → Clear All** — nuclear option, but keeps things clean between projects.
### Channels (beta feature)
Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation.
### Fixing models that output code instead of natural language
If your model (especially smaller ones like Qwen 3.5) responds with Python code blocks instead of plain English answers (as shown in the screenshot), this is a common behavior with code-optimized models.
#### Why it happens
- Small models like `qwen3.5:4b` are heavily optimized for coding tasks
- They interpret ambiguous questions as "write me a program" instead of "answer my question"
- The model's training data skews toward code generation at smaller parameter counts
- Thinking/reasoning mode (if enabled) amplifies this tendency
#### Fix: Set a system prompt in Open WebUI
1. Go to **Admin** → **Settings** → **Interface** → **Default System Prompt** (for all chats)
Or per-model: **Workspace** → **Models** → select model → **System Prompt**
2. Use a system prompt like:
```
You are a helpful assistant. Answer questions in clear, natural language.
Explain concepts conversationally. Only include code if the user explicitly
asks for code or a script. When discussing technical topics, use plain
English explanations with examples, not programs.
```
3. For the specific GPU pricing question in the screenshot, the model should have responded with a comparison table in text, not a Python script. A good system prompt prevents this.
#### Alternative: Use a chat-optimized model
Some models are better at conversational responses:
| Model | Size | Better for chat? |
|-------|------|-----------------|
| `qwen2.5:14b` | 14B | Yes — more balanced |
| `qwen2.5:7b` | 7B | Yes — good general chat |
| `llama3.1:8b` | 8B | Yes — conversational |
| `qwen2.5-coder:7b` | 7B | No — code-focused |
| `qwen3.5:4b` | 4B | No — too small, code-biased |
Avoid using `-coder` variants or very small models (≤4B) for general chat. Use them only when you actually want code.
### VRAM considerations
@@ -295,3 +473,25 @@ Re-run the setup script — it detects an existing install and skips prereqs:
| CPU | qwen2.5:7b | qwen2.5-coder:7b | 4k |
Embed model is always `nomic-embed-text` (required for RAG).
### VRAM reality check
Ollama will **always try to run** any model — it silently offloads layers to CPU when VRAM is insufficient. The model still works but gets significantly slower. The setup script's "fully in VRAM" label can be misleading.
**Actual VRAM needed for common models (Q4_K_M quantization):**
| Model | Download size | VRAM for inference | Fits in 6GB? | Fits in 8GB? |
|-------|-------------|-------------------|-------------|-------------|
| `qwen3.5:4b` | ~2.5 GB | ~3.5–4 GB | Yes | Yes |
| `qwen2.5:7b` | ~4.4 GB | ~5.5–6 GB | Tight | Yes |
| `qwen3.5:9b` | ~5.5 GB | ~6.5–7 GB | **No — partial CPU offload** | Tight |
| `qwen2.5:14b` | ~8.7 GB | ~10–11 GB | No | No |
| `qwen3.5-35b-a3b` (MoE) | ~20 GB | ~3.5 GB active | Yes (only 3B active) | Yes |
**Why the file size != VRAM needed:** Inference requires additional memory for KV cache, attention buffers, and CUDA overhead. Expect ~1-2 GB more than the model file size.
**Signs of CPU offload (model too big for your VRAM):**
- Tokens per second drops from 20-40 to 2-8
- `nvidia-smi` shows VRAM maxed out
- CPU usage spikes during generation
- First token takes much longer than usual