Add context awareness: token tracker + compaction + 6GB GPU tier

Context management for local models with limited windows:
- Context Tracker (community function): shows tokens used vs available,
  progress bar, percentage remaining — so you see the cliff coming
- Checkpoint Summarization Filter: auto-summarizes old messages when
  context fills up, like Claude's auto-compaction
- Both added as recommended post-install links in setup output
  (Open WebUI Functions install with one click from the UI)

6GB GPU tier (Quadro P3300, GTX 1060, etc.):
- Qwen 3.5 4B at Q4_K_M = ~2.5GB weights, leaves 3.5GB for KV cache
- With Q8 KV cache: ~32K usable context on 6GB
- Better than squeezing 9B into nothing — more context > slightly smarter

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 18:30:43 +00:00
parent 67b9f593bd
commit f7ef090d7c
3 changed files with 25 additions and 2 deletions
+11
View File
@@ -66,6 +66,9 @@ elif [[ "$TOTAL_VRAM" -ge 14 ]]; then
elif [[ "$TOTAL_VRAM" -ge 8 ]]; then
CHAT_MODEL="qwen3.5:9b"; CODE_MODEL="qwen3.5:9b"
CTX=32768; GPU_TIER="${TOTAL_VRAM}GB VRAM — 9B dense, 32K context"
elif [[ "$TOTAL_VRAM" -ge 6 ]]; then
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
CTX=32768; GPU_TIER="${TOTAL_VRAM}GB VRAM — 4B + KV quant, 32K context (3.5GB free for cache)"
elif [[ "$TOTAL_VRAM" -ge 4 ]]; then
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
CTX=16384; GPU_TIER="${TOTAL_VRAM}GB VRAM — 4B models, 16K context"
@@ -1391,6 +1394,14 @@ if $INSTALL_AI; then
echo " Via MCP: gitea_github_sync(mode='all')"
}
fi
if $INSTALL_AI; then
echo ""
echo -e " ${YELLOW}Recommended Open WebUI Functions (install from Admin → Functions → ):${NC}"
echo " Context tracker: https://openwebui.com/f/centrisic/context_tracker"
echo " → Shows tokens used vs available, progress bar, context % remaining"
echo " Context compaction: https://openwebui.com/f/projectmoon/checkpoint_summarization_filter"
echo " → Auto-summarizes old messages when context fills up (like Claude)"
fi
if $SVC_KIWIX && [[ "$ZIM_CHOICE" == "3" ]]; then
echo -e " ${YELLOW}ZIM downloads:${NC} ./kiwix_download.sh (not started)"
elif $SVC_KIWIX; then