Add context awareness: token tracker + compaction + 6GB GPU tier

Context management for local models with limited windows:
- Context Tracker (community function): shows tokens used vs available,
  progress bar, percentage remaining — so you see the cliff coming
- Checkpoint Summarization Filter: auto-summarizes old messages when
  context fills up, like Claude's auto-compaction
- Both added as recommended post-install links in setup output
  (Open WebUI Functions install with one click from the UI)

6GB GPU tier (Quadro P3300, GTX 1060, etc.):
- Qwen 3.5 4B at Q4_K_M = ~2.5GB weights, leaves 3.5GB for KV cache
- With Q8 KV cache: ~32K usable context on 6GB
- Better than squeezing 9B into nothing — more context > slightly smarter

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 18:30:43 +00:00
parent 67b9f593bd
commit f7ef090d7c
3 changed files with 25 additions and 2 deletions
+3
View File
@@ -1125,6 +1125,9 @@ elif [[ "$TOTAL_VRAM" -ge 14 ]]; then
elif [[ "$TOTAL_VRAM" -ge 8 ]]; then
CHAT_MODEL="qwen3.5:9b"; CODE_MODEL="qwen3.5:9b"
CTX=32768; TIER="${TOTAL_VRAM}GB — 9B dense, 32K context"
elif [[ "$TOTAL_VRAM" -ge 6 ]]; then
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
CTX=32768; TIER="${TOTAL_VRAM}GB — 4B + KV quant, 32K context"
elif [[ "$TOTAL_VRAM" -ge 4 ]]; then
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
CTX=16384; TIER="${TOTAL_VRAM}GB — 4B models, 16K context"