Context management for local models with limited windows:
- Context Tracker (community function): shows tokens used vs available,
progress bar, percentage remaining — so you see the cliff coming
- Checkpoint Summarization Filter: auto-summarizes old messages when
context fills up, like Claude's auto-compaction
- Both added as recommended post-install links in setup output
(Open WebUI Functions install with one click from the UI)
6GB GPU tier (Quadro P3300, GTX 1060, etc.):
- Qwen 3.5 4B at Q4_K_M = ~2.5GB weights, leaves 3.5GB for KV cache
- With Q8 KV cache: ~32K usable context on 6GB
- Better than squeezing 9B into nothing — more context > slightly smarter
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Main added a basic kiwix_search tool. Our branch already has a unified
search() that does everything kiwix_search did plus: DDG fallback,
freshness detection, structured results, and read_doc() for full articles.
Keep our version, keep the sync tool from our branch.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Replaced three separate tools (search_docs, web_search, search) with
a smart unified search() that:
- Always hits Kiwix first (instant, offline, no rate limit)
- Checks query for freshness keywords (latest, 2026, release, CVE, etc)
- If time-sensitive: also hits DDG, flags "prefer live results"
- If timeless (algorithms, docs, concepts): Kiwix only, skips web hit
- If Kiwix returns nothing: falls back to DDG automatically
Model sees both result sets with clear guidance on which to trust.
Keeps read_doc() for reading full Kiwix articles and web_search()
for explicit live-only queries when verifying offline currency.
130GB of ZIMs earn their keep on timeless topics (no rate limit,
instant, complete articles). DDG covers everything else.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The model can now search before it generates:
MCP tools (available via Open WebUI + Claude Code):
- search_docs(query): searches Kiwix ZIM files (Wikipedia, Stack Overflow,
DevDocs, Arch Wiki) — instant, offline, no rate limits
- read_doc(path): reads full article content from Kiwix results
- web_search(query): DuckDuckGo search, no API key needed
Open WebUI native search:
- ENABLE_RAG_WEB_SEARCH=true + RAG_WEB_SEARCH_ENGINE=duckduckgo
- Switched from SearXNG (not in stack) to DDG (zero config)
Search priority: Kiwix first (offline, fast) → DDG fallback (live web)
All 3 setup scripts updated:
- duckduckgo-search added to mcp_requirements.txt
- KIWIX_URL=http://kiwix:80 added to MCP container env
- curl added to MCP container deps (for sync script)
- Open WebUI DDG search enabled by default
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Six stackable techniques that compound:
1. KV cache quantization (Q8_0 = 2x context, asymmetric K=Q8/V=Q4 = 2.6x)
2. Flash attention (free VRAM + speed, zero quality loss)
3. Host-memory prompt caching (--cram, use the server's 128-384GB RAM)
4. KV to system RAM (-nkvo, last resort, 5-20x slower)
5. Architecture selection (GQA + MoE = tiny KV footprint)
6. NVMe mmap for model loading (fast cold starts, not inference)
Stacked result: single card goes from ~50K to ~130K usable context
with the MoE model. Updated llama.cpp config with all flags.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
VRAM must hold weights AND KV cache. What's left after weights is
your real context window. For a 10K-line project (~100-150K tokens):
- 1 card: 32-100K usable, file-by-file workflow
- 2 cards: 80-180K usable, whole-project-in-one-shot workflow
- This Opus session: 1M tokens (neither setup comes close)
Comparison table vs Claude Opus 1M session for perspective.
Keep Pro for the hard stuff, use local for the daily 80%.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Single card table was showing A- for 35B-A3B which undercut the case
for a second card. Restructured to lead with 16GB (start here) at
honest B+ ratings, then show what 32GB unlocks: dense 27B/32B models
that physically can't fit on 16GB and are genuinely A- quality.
The upgrade isn't about 262K context - it's about accessing better
models (27B dense > 35B MoE with 3B active params).
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The 35B-A3B MoE activates only 3B params per token - quality tracks
active params, making it more like a smart 7B than a true 35B. Revised
rating to B+ to A-. The 27B dense model is the real A- but needs 17GB
weights. Also notes: 262K is a VRAM ceiling not a quality guarantee,
Q4 quantization costs something, and context quality degrades at edges.
Practical high-quality context is more like 64-128K.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Both GPUs must be dedicated to the LLM for 262K context window.
Image gen and LLM run one at a time (swap takes seconds).
Simultaneous only possible with smaller context (~64K on one card).
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The real use case: drop Claude Max ($100/mo), keep Pro ($20/mo), offload
bulk codebase work to local. Local handles the 80% (reading 10K-line
projects, routine fixes, boilerplate) with no rate limits. Pro handles
the hard 20% where Opus quality matters. GPU pays for itself in 13 months,
saves $1,596 over 3 years vs Max.
Also adds: NVLink speed reality check, image gen capabilities (SDXL/Flux
included at no extra cost), hardware longevity estimate (3-5 years),
and updated cost comparison tables.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Complete shopping list with Dell-specific part numbers:
- GPU power cables: 9H6FV / N08NH (~$10-15 each, need 2)
- GPU Riser 3 required for second GPU slot
- Low-profile heatsinks needed on R720 (usually pre-installed on R730)
- 2x 1100W PSUs mandatory, non-redundant mode for full wattage
Documents riser layout, NVLink bridge clearance in 2U, potential issues
(CPU TDP limits, "unsupported" GPU warning, blower noise), and
R720 vs R730 comparison. Total build cost ~$940-960 with Dell parts.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Research and document the $850 budget build: 2x Quadro RTX 5000 (16GB each)
connected via NVLink for 32GB unified VRAM. With Qwen 3.5's Gated Delta
Network architecture (Feb 2026), this setup runs the 35B-A3B MoE model
with full 262K context in ~25GB — the best price-to-capability ratio
for local AI coding available.
Includes:
- Exact NVLink bridge part numbers (RTX 5000 uses unique smaller connector)
- Motherboard/PSU requirements and slot spacing guidance
- llama.cpp and Ollama multi-GPU configuration
- VRAM budget calculations for all Qwen 3.5 model sizes
- Phased build plan (start with 1 card at $400, add second later)
- Updated model table with full Qwen 3.5 family specs
- Cost comparison vs RTX 8000, RTX 3090, Claude Max, and API pricing
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
The $750 listing was a single outlier. Actual market price for
Quadro RTX 8000 Passive 48GB is $2,000-2,900 on eBay. Updated
all recommendations and cost comparisons accordingly.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
- Added full 48GB GPU market comparison (RTX 8000, A40, A6000, L40, RTX 6000 Ada)
- Quadro RTX 8000 Passive at $750-1,400 is 4-5x cheaper than alternatives
- Added RTX 8000 LLM benchmarks (34 t/s on 30B models at 8K context)
- Explained why 48GB >> 24GB for coding: context window is the bottleneck
- Added 2026 coding model landscape (Qwen3.5 27B, Qwen3-Coder, etc.)
- Revised recommendations: RTX 8000 as primary, dual P40 as budget alt
- Updated config notes for 48GB (32K context, higher quantization options)
- All prices verified from real listings as of March 22, 2026
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
- Corrected all GPU prices to actual eBay/Newegg listings as of March 22, 2026
- Added analysis of 32B coding models (Qwen2.5-Coder 32B, Qwen3.5 27B) on 24GB VRAM
- Added honest comparison of local LLM quality vs Claude Code
- Revised recommendations: dual P40 ($400-500) or P40+T4 ($400-550)
- Added configuration notes for 32B models, dual-GPU, and newer Qwen3 models
- RTX A4000 at $700+ is too expensive for this build
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
Research comparing P40, A2000, T4, and M40 GPUs for LLM inference
and image generation in Dell R720/R730 rack servers. Includes
benchmarks, compatibility notes, pricing, and recommendations.
https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
- Add aider Docker service (paulgauthier/aider) with GUI on port 8080
pointing at local ollama code model (auto-detected by VRAM tier)
- Add aider.sh CLI wrapper: cd into any git repo and run files through
aider terminal mode against the same local model
- Add port 8080 to UFW firewall rules
- Surface Aider UI URL in start.sh output and final install summary
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
ollama pull compares digests server-side so no download happens if
a model is already current. Timer runs monthly offset 2h from kiwix.
install.sh updated to handle all units generically.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
User units in systemd/ with an install.sh that symlinks and enables.
Persistent=true means missed runs (machine off) catch up on next boot.
Logs via journald: journalctl --user -u kiwix-update.service
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
- kiwix_download.sh: add Ask Ubuntu, Super User, Unix & Linux SE, Server Fault
- mcp_server.py: add kiwix_search tool — models can now query all offline ZIMs
- local-ai-setup.sh: pass KIWIX_URL env to MCP container, add kiwix dependency
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
Replace subshell ls expansion (which produces no args when /data is empty)
with a conditional: serve ZIM files if they exist, otherwise sleep infinity
so the container stays up gracefully until ZIM files are added.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
SearXNG:
- Dropped from both setup scripts and docker-compose (Google/Startpage
block self-hosted instances by IP — not reliable enough to include)
- Removed all interactive safe-search and engine-selection prompts
- Removed Open WebUI RAG web-search env vars (ENABLE_RAG_WEB_SEARCH,
SEARXNG_QUERY_URL, etc.)
- Removed port 8888 from UFW rules, start.sh URLs, done output, and
the Caddyfile template
- Removed searxng/ from .gitignore (directory no longer created)
- configure-searxng-safesearch.sh kept in repo for optional manual use
Kiwix:
- Replace the blocking wait-loop (`until ls *.zim`) with a one-liner
that passes whatever ZIM files exist (or none) directly to kiwix-serve,
so the container starts immediately and shows an empty library page
rather than hanging until ZIMs are downloaded
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
All scripts now default BASE to their own directory (SCRIPT_DIR) instead
of ~/docker/ai-stack, so after a git clone the user never needs to change
folders — docker-compose.yml, .env, settings, and data dirs all live next
to the scripts.
configure-searxng-safesearch.sh and configure-storage.sh do the same for
standalone use (still honoured when called with BASE= from a parent script).
docker-compose.yml (generated by both setup scripts) gains a comment block
at the top listing the everyday docker compose commands:
up -d / down / restart / stop / logs -f / pull / ps
so the file itself is the reference for managing containers.
.gitignore added to exclude generated files (docker-compose.yml, .env,
requirements.txt, helper scripts) and data directories (workspace/, kiwix/,
searxng/, gitea/, etc.) from git tracking.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
mcp_server.py / local-ai-setup.sh:
FastMCP.get_asgi_app() was removed in mcp 1.x — replace with
sse_app(), which returns the same Starlette ASGI app for SSE
transport. Fixes the crash loop the container was stuck in.
laptop_full_setup.sh Caddyfile:
$(hostname).local requires mDNS (Avahi/Bonjour) to resolve from a
proxy machine, which is often not available on all LAN clients.
Replace every occurrence with $LOCAL_IP (the machine's LAN IP)
so the generated Caddyfile.example works reliably regardless of
mDNS support.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
Adds invokeai-import-lora.sh to copy LoRA .safetensors files directly
into InvokeAI's Docker model volume, bypassing the greyed-out UI upload
buttons. Also adds README documentation for LoRA usage and troubleshooting.
https://claude.ai/code/session_01RU7NQuTbA8S8NRoWojvhR5
SearXNG's preferences.lock list prevents users from changing a setting
via the UI — it greys out the option. Without this, safe_search in
settings.yml was only a server default that users could still override
in their preferences page.
For moderate/strict, add:
preferences:
lock:
- safesearch
For none, the lock is omitted so users can freely adjust it.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
set +u was added around the ENGINE_BLOCK for-loops but the reporting
lines after ok "Updated settings.yml" still ran under set -u.
bash < 4.4 throws "unbound variable" for ${#ENABLE_MAP[@]} when the
associative array is empty (no --enable-engines argument was given).
Add set +u before the Disabled/Re-enabled info lines; it already covers
the ${#DISABLE_MAP[@]} reference at the end of the script.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
Root cause of all 85 engines being disabled:
`3>&1 1>&2 2>&3 2>/dev/null` — the trailing `2>/dev/null` overwrites FD2
with /dev/null AFTER the FD swap, destroying the pipe that whiptail writes
selections to. User selections were silently discarded every time. Remove it.
Also: if whiptail exits 0 but returns empty string (all items unchecked),
fall back to pre-checked defaults rather than disabling everything.
ENABLE_MAP: unbound variable (configure-searxng-safesearch.sh):
bash < 4.4 treats `${!empty_assoc[@]}` as unbound under `set -u`.
Wrap for loops and mark_disabled key-check with set +u / set -u.
Add artic (Art Institute of Chicago) to image engine menus.
Add yandex images (defaults OFF for moderate/strict, same as yandex).
Update _SX_AUTOFF to include "yandex images" so re-enabling it works.
Update configure script engine category lists to match expanded menus.
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
- Radiolist first: Choose / All / Skip (ESC = skip)
- Checklist shows all 18 ZIMs with sizes; pre-checks any already
downloaded ZIMs so re-runs don't re-select what you have
- ZIM_PICKS now stores space-separated names (wikipedia, stackoverflow,
archlinux...) instead of numbers; dispatch updated to match
- Text fallback (no whiptail): same numbered list as before, converts
input numbers to names before dispatch
- Summary shows count and names of selected ZIMs
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
For both setup scripts and configure-searxng-safesearch.sh:
- Five whiptail checklist menus: Web · Images · Videos · News · Science
- Pre-populated ON/OFF based on the chosen safe-search level:
none → all engines ON
moderate/strict → no-safe-search engines (mojeek, yandex, baidu, naver,
invidious, piped, peertube, sepiasearch, flickr,
imgur, deviantart, dailymotion, vimeo) default OFF
- User can toggle any engine independently before confirming
- ESC on any menu restores defaults for that category (keeps them unchanged)
- If user re-enables an auto-disabled engine, --enable-engines is passed
to configure script to override the auto-disable
- Text fallback for non-interactive/headless installs
- Summary shows count of disabled engines instead of raw list
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
Setup (both scripts):
- Q6 asks safe-search level: none / moderate / strict
- Then asks which categories to disable: videos images news science social
- Then asks which engines to disable by name (duckduckgo, bing, etc.)
- Choices flow into SEARXNG_QUERY_URL &safesearch=N in docker-compose.yml
- SearXNG summary line added to laptop_full_setup.sh install plan
configure-searxng-safesearch.sh:
- Full rewrite: --disable-categories, --disable-engines, --enable-engines
- Category maps: videos / images / news / science / social engine lists
- --enable-engines overrides auto-disables (e.g. keep yandex on strict)
- Preserves existing secret key on update
- Creates settings.yml from scratch if missing (safe for setup use)
- Help flag (-h/--help)
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
- New configure-searxng-safesearch.sh: set strict/moderate/none level,
disable engines that can't enforce the chosen level (torrent sites,
mojeek, baidu, yandex, nvidious/piped/peertube video frontends),
updates &safesearch= in SEARXNG_QUERY_URL, and restarts searxng
- Add RAG_WEB_SEARCH_RESULT_COUNT=5 and RAG_WEB_SEARCH_CONCURRENT_REQUESTS=10
to Open WebUI env in both setup scripts
- Add &safesearch=0 to SEARXNG_QUERY_URL so the script can find/replace it
- Sync ENABLE_TOOL_SERVERS=true into laptop_full_setup.sh (was missing)
https://claude.ai/code/session_012gDnantBmFTWZGCiKyjazx
Claude