root/local-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add single-card optimization guide — squeeze every byte from 16GB

Browse Source 
Six stackable techniques that compound:
1. KV cache quantization (Q8_0 = 2x context, asymmetric K=Q8/V=Q4 = 2.6x)
2. Flash attention (free VRAM + speed, zero quality loss)
3. Host-memory prompt caching (--cram, use the server's 128-384GB RAM)
4. KV to system RAM (-nkvo, last resort, 5-20x slower)
5. Architecture selection (GQA + MoE = tiny KV footprint)
6. NVMe mmap for model loading (fast cold starts, not inference)

Stacked result: single card goes from ~50K to ~130K usable context
with the MoE model. Updated llama.cpp config with all flags.

https://claude.ai/code/session_01PtYTPherSJaxDEVPgF6Nxu
This commit is contained in:
Claude
2026-03-22 17:36:06 +00:00
parent 1d78957083
commit 0f57690692
1 changed files with 165 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gpu-setup-research.md
+165 -1
View File
@@ -296,6 +296,151 @@ becomes possible locally.
codebase. That's why you keep Pro. But 2 cards handles the daily "read my project and
help me code" workflow locally with no rate limits — and that's 80% of the work.
### Squeezing Every Byte: Single-Card Optimization (16GB)
Before buying a second card, stack these techniques. They're cumulative — use all of them
together. The gains compound because they all free VRAM from the same bottleneck: KV cache.
#### 1. Quantize the KV Cache (Biggest Single Win)
By default, llama.cpp stores the KV cache in FP16. That's 2 bytes per value. You can
compress it with zero code changes — just flags:
| Cache Type | Bytes/value | vs FP16 | Quality Impact | Verdict |
|-----------|-------------|---------|----------------|---------|
| FP16 (default) | 2.0 | baseline | none | wasteful on 16GB |
| **Q8_0** | **1.0** | **50% smaller** | **~0.002-0.05 perplexity** | **Always use this** |
| Q4_0 | 0.5 | 75% smaller | ~0.2 perplexity (noticeable) | Use if desperate |
| **Asymmetric: K=Q8_0, V=Q4_0** | **0.75 avg** | **62% smaller** | **Better than uniform Q4** | **Best bang/buck** |
The K cache is more sensitive to quantization than V. Asymmetric (Q8 keys, Q4 values) gives
you ~62% savings with quality closer to Q8 than Q4.
**Concrete example — Qwen3.5-35B-A3B on 1 card (16GB):**
- Weights: ~12GB → 4GB free for KV cache
- FP16 KV cache: 4GB → **~50K context**
- Q8_0 KV cache: 4GB buys 2x → **~100K context**
- K=Q8/V=Q4 KV cache: 4GB buys 2.6x → **~130K context**
That's the difference between "a few files" and "a meaningful chunk of a project."
```bash
# llama.cpp — always use these three flags together
llama-server \
--cache-type-k q8_0 \
--cache-type-v q4_0 \
--flash-attn \
-m model.gguf -ngl 99 -c 131072
# Ollama — set environment variable before starting
export OLLAMA_KV_CACHE_TYPE=q8_0 # or q4_0 for aggressive
export OLLAMA_FLASH_ATTENTION=1
ollama serve
```
#### 2. Flash Attention (Free Speed + VRAM)
Flash attention restructures how attention is computed — instead of materializing the full
attention matrix in VRAM, it computes it in tiles. Result: less VRAM used during inference,
slightly faster, **zero quality loss**.
Always enable it. There's no downside on Turing GPUs with quantized KV cache.
```bash
# llama.cpp
--flash-attn
# Ollama
export OLLAMA_FLASH_ATTENTION=1
```
#### 3. Host-Memory Prompt Caching (`--cram`) — System RAM as L2 Cache
This is the smart use of system RAM. The `--cram` flag in llama-server stores pre-computed
prompt representations in host memory (system RAM). When you send the same system prompt
or reuse a conversation prefix, it skips reprocessing — hot-swaps the cached computation
back onto the GPU.
This doesn't increase context window size, but it **dramatically reduces time-to-first-token**
for repeated workflows (which is most coding — same system prompt, same project context).
```bash
# llama-server with 16GB RAM cache for prompts
llama-server \
--cram 16384 \
--cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \
-m model.gguf -ngl 99 -c 131072
```
Your R720/R730 has 128-384GB of DDR3/DDR4 RAM. Use it. `--cram 65536` (64GB) is reasonable
for a dedicated inference server — it costs nothing, and repeat prompts become near-instant.
#### 4. KV Cache to System RAM (`-nkvo`) — Last Resort for Context
The `-nkvo` (no KV offload) flag moves the entire KV cache to system RAM, freeing all 16GB
of VRAM for model weights. This sounds great but comes with a brutal speed penalty:
| Scenario | Speed Impact |
|----------|-------------|
| Full VRAM (normal) | Baseline (25-35 tok/s) |
| KV in system RAM via PCIe | **5-20x slower** (~2-7 tok/s) |
| KV on NVMe via mmap | **30x+ slower** (~1 tok/s) |
**When it makes sense:** Loading a model that barely doesn't fit (e.g., Qwen3.5-27B dense
at 17GB weights on a 16GB card). You'd get ~2-5 tok/s with KV in RAM — painfully slow, but
it's the difference between "runs slowly" and "doesn't run at all." Fine for a batch job
where you walk away and come back. Not viable for interactive coding.
**Don't do this routinely.** Quantized KV cache (technique #1) is 50-100x better because
the cache stays on the GPU. Only use `-nkvo` for models that literally can't fit otherwise.
#### 5. Pick the Right Architecture (GQA + MoE = VRAM Efficient)
Not all models consume KV cache equally. Modern architectures with **Grouped Query Attention
(GQA)** use far less KV cache than older Multi-Head Attention (MHA):
| Architecture | KV cache at 64K context | Examples |
|-------------|------------------------|---------|
| MHA (old) | ~8-12GB | LLaMA-1, GPT-J |
| **GQA (modern)** | **~1-3GB** | **Qwen3.5 series, LLaMA-3** |
| **GQA + MoE** | **~1.2GB** | **Qwen3.5-35B-A3B** |
The Qwen3.5-35B-A3B is almost purpose-built for your situation: 3B active params (fast on
Turing), MoE architecture (small memory footprint during inference), and GQA (tiny KV cache).
With quantized KV on top of that, 130K+ context on a single 16GB card is realistic.
#### 6. NVMe as mmap Backing Store
Your fast NVMe matters for **model loading**, not inference. llama.cpp uses mmap by default
to stream model weights from disk, so a fast NVMe means:
- Near-instant cold starts (weights stream in as needed)
- Graceful degradation if model slightly exceeds RAM (OS pages out unused layers)
But NVMe is **not** a viable substitute for VRAM during inference. The bandwidth gap is too
large: VRAM runs at ~400 GB/s (RTX 5000), system RAM at ~50-100 GB/s (DDR4 quad-channel),
NVMe at ~3-7 GB/s. Three orders of magnitude difference from VRAM.
**Practical use:** Keep all your GGUF model files on NVMe. Enable mmap (default). That's it.
Don't try to use NVMe as overflow for the KV cache — the latency kills interactive use.
#### Stacking Everything: Revised Single-Card Numbers
| Model | Optimization | Usable Context | Speed | Quality |
|-------|-------------|---------------|-------|---------|
| Qwen3.5-35B-A3B Q4 | None (defaults) | ~32-50K | 25-35 tok/s | B+ |
| Qwen3.5-35B-A3B Q4 | **KV Q8 + flash** | **~80-100K** | **25-35 tok/s** | **B+** |
| Qwen3.5-35B-A3B Q4 | **KV asym + flash** | **~100-130K** | **25-35 tok/s** | **B+ (tiny quality dip)** |
| Qwen3.5-9B Q8 | KV Q8 + flash | ~120-160K | 30-45 tok/s | B |
| Qwen2.5-Coder-14B Q4 | KV Q8 + flash | ~40-64K | 20-30 tok/s | B+ |
Add `--cram` on top for instant repeated prompts. That's your real single-card ceiling.
**The honest answer:** With all optimizations stacked, a single card goes from "a few files
at a time" to "maybe half a 10K-line project." That's a meaningful upgrade from the
unoptimized baseline, but it still doesn't match what 2 cards with a dense 27B model gives
you. The second card isn't about optimization tricks — it's about physics (more VRAM = more
data on the fast bus).
### Estimated Inference Speed
| Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) |
@@ -375,16 +520,35 @@ cmake --build build --config Release -j$(nproc)
# Download Qwen3.5-35B-A3B GGUF (Q4_K_M)
# Get from: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF
# Run on dual GPU with NVLink
# Run on dual GPU with NVLink (all optimizations on)
./build/bin/llama-server \
-m Qwen3.5-35B-A3B-Q4_K_M.gguf \
-ngl 999 \
-c 262144 \
--cache-type-k q8_0 \
--cache-type-v q4_0 \
--flash-attn \
--cram 65536 \
--host 0.0.0.0 \
--port 8080
# --cache-type-k q8_0 / --cache-type-v q4_0 = asymmetric KV quantization (62% smaller cache)
# --flash-attn = tiled attention (less VRAM, no quality loss)
# --cram 65536 = 64GB host RAM prompt cache (instant repeat prompts)
# llama.cpp auto-detects NVLink and splits layers across both GPUs
# Use -ts 1,1 to manually set equal split if needed
# Single card variant (no NVLink) — same flags, smaller context
./build/bin/llama-server \
-m Qwen3.5-35B-A3B-Q4_K_M.gguf \
-ngl 999 \
-c 131072 \
--cache-type-k q8_0 \
--cache-type-v q4_0 \
--flash-attn \
--cram 65536 \
--host 0.0.0.0 \
--port 8080
```
#### Ollama