root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Honest quality ratings: 35B-A3B is B+ not A-, add reality checks

Browse Source 
The 35B-A3B MoE activates only 3B params per token - quality tracks
active params, making it more like a smart 7B than a true 35B. Revised
rating to B+ to A-. The 27B dense model is the real A- but needs 17GB
weights. Also notes: 262K is a VRAM ceiling not a quality guarantee,
Q4 quantization costs something, and context quality degrades at edges.
Practical high-quality context is more like 64-128K.

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 17:11:49 +00:00
parent 209173e988
commit 6d6a47e028
1 changed files with 13 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+13 -3
View File
@@ -33,12 +33,22 @@ than traditional transformers.
| Model | Type | Active Params | Size at Q4_K_M | Max Context | Quality | Notes |
|-------|------|---------------|---------------|-------------|---------|-------|
| **Qwen3.5-35B-A3B** | **MoE** | **3B** | **~12GB** | **262K** | **A-** | Best bang for buck — 35B model, 3B active, fits 262K ctx in 25GB |
| **Qwen3.5-27B** | Dense | 27B | ~17GB | 262K | A- | 72.4% SWE-bench, ties GPT-5 mini |
| **Qwen3.5-35B-A3B** | **MoE** | **3B** | **~12GB** | **262K** | **B+ to A-** | 35B total but only 3B active — quality tracks active params |
| **Qwen3.5-27B** | Dense | 27B | ~17GB | 262K | **A-** | 72.4% SWE-bench, ties GPT-5 mini. The real A- option. |
| **Qwen3.5-122B-A10B** | MoE | 10B | ~76GB | 262K | A | Matches GPT-5 mini across the board |
| **Qwen3.5-9B** | Dense | 9B | ~6GB | 262K | B+ | Fits on any modern GPU |
| **Qwen3.5-4B** | Dense | 4B | ~3GB | 262K | B | Tiny but capable |
**Quality reality check:** MoE models route tokens through only a subset of parameters.
The 35B-A3B activates **3B params per token** — think of it as a smart 7B model, not a 35B.
Quality is closer to B+ for complex coding. The 27B dense model is genuinely A- but needs
17GB weights (leaving less room for context on 32GB). At Q4 quantization there's a further
small quality loss. And 262K is a VRAM ceiling, not a quality guarantee — models degrade
at the edges of their context window. Practical high-quality context is more like 64-128K.
**No local model approaches Claude Opus on hard problems.** The strategy isn't to replace
Opus — it's to offload the 80% of routine work so your Pro plan limits stop being an issue.
### Previous Generation (Still Relevant)
| Model | Size at Q4_K_M | Quality | Notes |
@@ -212,7 +222,7 @@ bridge sitting on top of the cards. A 4U chassis gives the most room.
| Model | Arch | Quant | Weights | Context | Total VRAM | Quality |
|-------|------|-------|---------|---------|------------|---------|
| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **A-** |
| **Qwen3.5-35B-A3B** | **MoE (3B active)** | **Q4_K_M** | **~12GB** | **262K** | **~25GB** | **B+ to A-** |
| Qwen3.5-27B | Dense | Q4_K_M | ~17GB | 128K+ | ~25GB | A- |
| Qwen3-Coder-Next (80B/3B active) | MoE | Q4 | ~20GB | 128K | ~28GB | A |
| Qwen2.5-Coder-14B | Dense | Q4_K_M | ~10GB | 128K | ~22GB | B+ |