Convert all setup prompts to whiptail, fix VRAM estimates, add model expectations

Setup script changes:
- All prompts now use whiptail dialogs with text fallback
- Q1b (SSH), Q2 (storage), Q3 (Kiwix), Q4b (firewall), Q5 (models),
  Q6 (download), final confirm all converted
- Model tier selection uses radiolist with recommended tier pre-selected
- Custom model entry uses inputbox with current defaults pre-filled
- Fix speed_label: now shows actual VRAM needed (file size + 2GB overhead)
  instead of misleading "fully in VRAM" for models that don't fit
- qwen3.5-35b-a3b MoE already in tier list (was there, now with accurate
  VRAM estimate shown)

README changes:
- Add "Realistic expectations by model size" table
- 35B MoE highlighted as sweet spot for small GPUs

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 19:39:39 +00:00
parent b3985ea053
commit 3bd714960f
2 changed files with 246 additions and 103 deletions
+23
View File
@@ -575,6 +575,29 @@ Local AI requires more manual workflow management but has real code awareness vi
Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation.
### Realistic expectations by model size
Not all models can do all tasks. Here's what to actually expect:
| Task | 4B (qwen3.5:4b) | 9B (qwen3.5:9b) | 35B MoE (qwen3.5-35b-a3b) | 14B+ dense |
|------|:---:|:---:|:---:|:---:|
| Answer simple questions | OK | Good | Good | Good |
| Explain existing code (with RAG) | OK | Good | Good | Good |
| Fix a simple bug (typo, off-by-one) | Maybe | Usually | Usually | Yes |
| Write a small utility function | Shaky | OK | Good | Good |
| Fix logic error across 2-3 functions | No | Maybe | Usually | Usually |
| Write a new feature (multiple files) | No | Shaky | Maybe | Maybe |
| Refactor with style consistency | No | No | Sometimes | Sometimes |
| Summarize a conversation for handoff | OK | Good | Good | Good |
**The 35B MoE model (`qwen3.5-35b-a3b`) is the sweet spot for small GPUs.** It was trained as a 35B model but only activates 3B parameters per token. This means it has the *knowledge* of a 35B model with the VRAM footprint closer to a 4B. On a 6GB card it may fit (VRAM usage varies with context length and KV cache settings).
**Bottom line for a 6GB GPU:**
- Use `qwen3.5:4b` for quick chat, explanations, and summarization
- Try `qwen3.5-35b-a3b` for code tasks — if it fits, it will be significantly better than 4B
- Use Claude Code for anything that requires reading/writing multiple files or complex reasoning
- The RAG server helps a lot — even a 4B model gives useful answers when it has the right code chunks in context
### Fixing models that output code instead of natural language
If your model (especially smaller ones like Qwen 3.5) responds with Python code blocks instead of plain English answers (as shown in the screenshot), this is a common behavior with code-optimized models.