diff --git a/README.md b/README.md index 75ab8af..c6c382b 100644 --- a/README.md +++ b/README.md @@ -359,6 +359,105 @@ This is the closest thing to "projects" in Open WebUI. You can have separate kno **Profile → Settings → Personalization → Memories → Clear All** — nuclear option, but keeps things clean between projects. +### Working with Knowledge Collections (small context window survival guide) + +With small local models (4B-9B, 4K-8K context), you'll hit the context window limit fast — often after 10-20 exchanges. Here's how to work effectively despite that. + +#### How Knowledge Collections actually work + +When you type `#` in chat and select a collection, Open WebUI does **RAG retrieval** — it finds the most relevant chunks from the collection, not the whole thing. This is efficient and doesn't blow up your context window. + +- Default chunk size: ~500 tokens +- Typically 3-5 relevant chunks are retrieved per query (~1500-2500 tokens) +- The chunks are injected as context alongside your message + +#### Creating Knowledge Collections from chat conversations + +There is **no built-in "export chat to knowledge" button** — this is a requested but unimplemented feature. Here's the practical workflow: + +**The handoff method (like the ChatGPT dark ages, but structured):** + +1. When you're ~60-70% through your context window (you'll feel the model getting fuzzy), ask: + ``` + Summarize everything we've discussed and decided so far. Include: + - Key decisions made + - Current state of the work + - What still needs to be done + - Any important details or constraints + Format as a structured document I can use to continue this conversation. + ``` +2. **Copy the summary output** +3. Go to **Workspace → Knowledge → Create Collection** (e.g. "GPU Build - Session 1") +4. Click **Add Content** → paste the summary as a text file (`.txt` or `.md`) +5. Start a **new chat**, type `#` and select your collection, then continue where you left off + +**Ongoing project workflow:** + +``` +Chat 1: Research phase + → Ask for summary at end + → Save summary to "Project X" knowledge collection + +Chat 2: Type # → select "Project X" → continue + → Model gets relevant context via RAG + → Ask for updated summary at end + → Add updated summary to collection (replace or append) + +Chat 3: Type # → select "Project X" → continue + → Repeat... +``` + +Each chat starts fresh with full context window available, but can pull in relevant history from previous sessions via RAG. + +#### Useful community functions for context management + +Install these manually (Workspace → Functions → Create, paste the code): + +| Function | What it does | +|----------|-------------| +| **Checkpoint Summarization Filter** | Auto-summarizes conversation history when context gets long | +| **Chat Context Clipper** | Keeps only the last N messages, preserves system prompt and first message | +| **Context Length Filter** | Hard limits on turns (default 25) and tokens (default 10,000) | + +The **Checkpoint Summarization Filter** is the closest to automatic handoff — it summarizes older messages so the model can keep going without losing track. + +#### How Open WebUI handles context overflow + +By default, Open WebUI **truncates** old messages (drops them silently) — it does NOT auto-summarize. The model just loses access to earlier conversation. This is why you suddenly feel like the model "forgot" what you were talking about. + +You can control this with: +- **Context Length Filter** function: set max turns and token limits explicitly +- **Chat Context Clipper**: keeps latest N messages, always preserves system prompt + first message pair +- Or do manual handoffs before you hit the limit + +#### RAG tuning for better knowledge retrieval + +If your knowledge collections aren't returning good results, tune these in **Admin → Settings → Documents**: + +| Setting | Default | Recommendation | +|---------|---------|---------------| +| Chunk Size | 500 | 300 for factual docs, 800 for narrative/code | +| Chunk Overlap | 100 | 50-100 (higher = better continuity, more tokens) | +| Top K | 4 | 3-5 (more = more context consumed) | +| Relevance Threshold | 0.0 | 0.3-0.5 (filters out low-quality matches) | + +**Important:** Chunk size cannot exceed your embedding model's token limit. `nomic-embed-text` supports up to 8192 tokens, so you have plenty of headroom. + +#### The honest comparison: Local AI vs Claude Code + +| | **Local AI (Open WebUI + Ollama)** | **Claude Code** | +|---|---|---| +| **Context window** | 4K-32K (model dependent) | 200K | +| **Cross-session memory** | Manual handoffs or Auto Memory | Automatic (CLAUDE.md, project memory) | +| **Project scoping** | Knowledge collections (manual) | Built-in (each project has its own context) | +| **Code awareness** | None — you paste code in | Reads your entire codebase | +| **Continuation** | New chat + `#collection` handoff | "Let's finish X" just works | +| **Cost** | Free (your electricity) | API usage fees | +| **Privacy** | 100% local | Cloud-based | +| **Offline** | Works without internet | Requires internet | + +Local AI requires more manual workflow management. The tradeoff is privacy, cost, and offline capability. For code-heavy work, Claude Code is dramatically better. For private research, document Q&A, and learning — local models with knowledge collections work well once you get the handoff workflow down. + ### Channels (beta feature) Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation.