Add context awareness: token tracker + compaction + 6GB GPU tier
Context management for local models with limited windows: - Context Tracker (community function): shows tokens used vs available, progress bar, percentage remaining — so you see the cliff coming - Checkpoint Summarization Filter: auto-summarizes old messages when context fills up, like Claude's auto-compaction - Both added as recommended post-install links in setup output (Open WebUI Functions install with one click from the UI) 6GB GPU tier (Quadro P3300, GTX 1060, etc.): - Qwen 3.5 4B at Q4_K_M = ~2.5GB weights, leaves 3.5GB for KV cache - With Q8 KV cache: ~32K usable context on 6GB - Better than squeezing 9B into nothing — more context > slightly smarter https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
@@ -1125,6 +1125,9 @@ elif [[ "$TOTAL_VRAM" -ge 14 ]]; then
|
||||
elif [[ "$TOTAL_VRAM" -ge 8 ]]; then
|
||||
CHAT_MODEL="qwen3.5:9b"; CODE_MODEL="qwen3.5:9b"
|
||||
CTX=32768; TIER="${TOTAL_VRAM}GB — 9B dense, 32K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 6 ]]; then
|
||||
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
|
||||
CTX=32768; TIER="${TOTAL_VRAM}GB — 4B + KV quant, 32K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 4 ]]; then
|
||||
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
|
||||
CTX=16384; TIER="${TOTAL_VRAM}GB — 4B models, 16K context"
|
||||
|
||||
Reference in New Issue
Block a user