Convert all setup prompts to whiptail, fix VRAM estimates, add model expectations
Setup script changes: - All prompts now use whiptail dialogs with text fallback - Q1b (SSH), Q2 (storage), Q3 (Kiwix), Q4b (firewall), Q5 (models), Q6 (download), final confirm all converted - Model tier selection uses radiolist with recommended tier pre-selected - Custom model entry uses inputbox with current defaults pre-filled - Fix speed_label: now shows actual VRAM needed (file size + 2GB overhead) instead of misleading "fully in VRAM" for models that don't fit - qwen3.5-35b-a3b MoE already in tier list (was there, now with accurate VRAM estimate shown) README changes: - Add "Realistic expectations by model size" table - 35B MoE highlighted as sweet spot for small GPUs https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
@@ -575,6 +575,29 @@ Local AI requires more manual workflow management but has real code awareness vi
|
||||
|
||||
Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation.
|
||||
|
||||
### Realistic expectations by model size
|
||||
|
||||
Not all models can do all tasks. Here's what to actually expect:
|
||||
|
||||
| Task | 4B (qwen3.5:4b) | 9B (qwen3.5:9b) | 35B MoE (qwen3.5-35b-a3b) | 14B+ dense |
|
||||
|------|:---:|:---:|:---:|:---:|
|
||||
| Answer simple questions | OK | Good | Good | Good |
|
||||
| Explain existing code (with RAG) | OK | Good | Good | Good |
|
||||
| Fix a simple bug (typo, off-by-one) | Maybe | Usually | Usually | Yes |
|
||||
| Write a small utility function | Shaky | OK | Good | Good |
|
||||
| Fix logic error across 2-3 functions | No | Maybe | Usually | Usually |
|
||||
| Write a new feature (multiple files) | No | Shaky | Maybe | Maybe |
|
||||
| Refactor with style consistency | No | No | Sometimes | Sometimes |
|
||||
| Summarize a conversation for handoff | OK | Good | Good | Good |
|
||||
|
||||
**The 35B MoE model (`qwen3.5-35b-a3b`) is the sweet spot for small GPUs.** It was trained as a 35B model but only activates 3B parameters per token. This means it has the *knowledge* of a 35B model with the VRAM footprint closer to a 4B. On a 6GB card it may fit (VRAM usage varies with context length and KV cache settings).
|
||||
|
||||
**Bottom line for a 6GB GPU:**
|
||||
- Use `qwen3.5:4b` for quick chat, explanations, and summarization
|
||||
- Try `qwen3.5-35b-a3b` for code tasks — if it fits, it will be significantly better than 4B
|
||||
- Use Claude Code for anything that requires reading/writing multiple files or complex reasoning
|
||||
- The RAG server helps a lot — even a 4B model gives useful answers when it has the right code chunks in context
|
||||
|
||||
### Fixing models that output code instead of natural language
|
||||
|
||||
If your model (especially smaller ones like Qwen 3.5) responds with Python code blocks instead of plain English answers (as shown in the screenshot), this is a common behavior with code-optimized models.
|
||||
|
||||
Reference in New Issue
Block a user