diff --git a/docs/gpu-setup-research.md b/docs/gpu-setup-research.md index 714e6ba..9519d87 100644 --- a/docs/gpu-setup-research.md +++ b/docs/gpu-setup-research.md @@ -296,6 +296,151 @@ becomes possible locally. codebase. That's why you keep Pro. But 2 cards handles the daily "read my project and help me code" workflow locally with no rate limits — and that's 80% of the work. +### Squeezing Every Byte: Single-Card Optimization (16GB) + +Before buying a second card, stack these techniques. They're cumulative — use all of them +together. The gains compound because they all free VRAM from the same bottleneck: KV cache. + +#### 1. Quantize the KV Cache (Biggest Single Win) + +By default, llama.cpp stores the KV cache in FP16. That's 2 bytes per value. You can +compress it with zero code changes — just flags: + +| Cache Type | Bytes/value | vs FP16 | Quality Impact | Verdict | +|-----------|-------------|---------|----------------|---------| +| FP16 (default) | 2.0 | baseline | none | wasteful on 16GB | +| **Q8_0** | **1.0** | **50% smaller** | **~0.002-0.05 perplexity** | **Always use this** | +| Q4_0 | 0.5 | 75% smaller | ~0.2 perplexity (noticeable) | Use if desperate | +| **Asymmetric: K=Q8_0, V=Q4_0** | **0.75 avg** | **62% smaller** | **Better than uniform Q4** | **Best bang/buck** | + +The K cache is more sensitive to quantization than V. Asymmetric (Q8 keys, Q4 values) gives +you ~62% savings with quality closer to Q8 than Q4. + +**Concrete example — Qwen3.5-35B-A3B on 1 card (16GB):** +- Weights: ~12GB → 4GB free for KV cache +- FP16 KV cache: 4GB → **~50K context** +- Q8_0 KV cache: 4GB buys 2x → **~100K context** +- K=Q8/V=Q4 KV cache: 4GB buys 2.6x → **~130K context** + +That's the difference between "a few files" and "a meaningful chunk of a project." + +```bash +# llama.cpp — always use these three flags together +llama-server \ + --cache-type-k q8_0 \ + --cache-type-v q4_0 \ + --flash-attn \ + -m model.gguf -ngl 99 -c 131072 + +# Ollama — set environment variable before starting +export OLLAMA_KV_CACHE_TYPE=q8_0 # or q4_0 for aggressive +export OLLAMA_FLASH_ATTENTION=1 +ollama serve +``` + +#### 2. Flash Attention (Free Speed + VRAM) + +Flash attention restructures how attention is computed — instead of materializing the full +attention matrix in VRAM, it computes it in tiles. Result: less VRAM used during inference, +slightly faster, **zero quality loss**. + +Always enable it. There's no downside on Turing GPUs with quantized KV cache. + +```bash +# llama.cpp +--flash-attn + +# Ollama +export OLLAMA_FLASH_ATTENTION=1 +``` + +#### 3. Host-Memory Prompt Caching (`--cram`) — System RAM as L2 Cache + +This is the smart use of system RAM. The `--cram` flag in llama-server stores pre-computed +prompt representations in host memory (system RAM). When you send the same system prompt +or reuse a conversation prefix, it skips reprocessing — hot-swaps the cached computation +back onto the GPU. + +This doesn't increase context window size, but it **dramatically reduces time-to-first-token** +for repeated workflows (which is most coding — same system prompt, same project context). + +```bash +# llama-server with 16GB RAM cache for prompts +llama-server \ + --cram 16384 \ + --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \ + -m model.gguf -ngl 99 -c 131072 +``` + +Your R720/R730 has 128-384GB of DDR3/DDR4 RAM. Use it. `--cram 65536` (64GB) is reasonable +for a dedicated inference server — it costs nothing, and repeat prompts become near-instant. + +#### 4. KV Cache to System RAM (`-nkvo`) — Last Resort for Context + +The `-nkvo` (no KV offload) flag moves the entire KV cache to system RAM, freeing all 16GB +of VRAM for model weights. This sounds great but comes with a brutal speed penalty: + +| Scenario | Speed Impact | +|----------|-------------| +| Full VRAM (normal) | Baseline (25-35 tok/s) | +| KV in system RAM via PCIe | **5-20x slower** (~2-7 tok/s) | +| KV on NVMe via mmap | **30x+ slower** (~1 tok/s) | + +**When it makes sense:** Loading a model that barely doesn't fit (e.g., Qwen3.5-27B dense +at 17GB weights on a 16GB card). You'd get ~2-5 tok/s with KV in RAM — painfully slow, but +it's the difference between "runs slowly" and "doesn't run at all." Fine for a batch job +where you walk away and come back. Not viable for interactive coding. + +**Don't do this routinely.** Quantized KV cache (technique #1) is 50-100x better because +the cache stays on the GPU. Only use `-nkvo` for models that literally can't fit otherwise. + +#### 5. Pick the Right Architecture (GQA + MoE = VRAM Efficient) + +Not all models consume KV cache equally. Modern architectures with **Grouped Query Attention +(GQA)** use far less KV cache than older Multi-Head Attention (MHA): + +| Architecture | KV cache at 64K context | Examples | +|-------------|------------------------|---------| +| MHA (old) | ~8-12GB | LLaMA-1, GPT-J | +| **GQA (modern)** | **~1-3GB** | **Qwen3.5 series, LLaMA-3** | +| **GQA + MoE** | **~1.2GB** | **Qwen3.5-35B-A3B** | + +The Qwen3.5-35B-A3B is almost purpose-built for your situation: 3B active params (fast on +Turing), MoE architecture (small memory footprint during inference), and GQA (tiny KV cache). +With quantized KV on top of that, 130K+ context on a single 16GB card is realistic. + +#### 6. NVMe as mmap Backing Store + +Your fast NVMe matters for **model loading**, not inference. llama.cpp uses mmap by default +to stream model weights from disk, so a fast NVMe means: +- Near-instant cold starts (weights stream in as needed) +- Graceful degradation if model slightly exceeds RAM (OS pages out unused layers) + +But NVMe is **not** a viable substitute for VRAM during inference. The bandwidth gap is too +large: VRAM runs at ~400 GB/s (RTX 5000), system RAM at ~50-100 GB/s (DDR4 quad-channel), +NVMe at ~3-7 GB/s. Three orders of magnitude difference from VRAM. + +**Practical use:** Keep all your GGUF model files on NVMe. Enable mmap (default). That's it. +Don't try to use NVMe as overflow for the KV cache — the latency kills interactive use. + +#### Stacking Everything: Revised Single-Card Numbers + +| Model | Optimization | Usable Context | Speed | Quality | +|-------|-------------|---------------|-------|---------| +| Qwen3.5-35B-A3B Q4 | None (defaults) | ~32-50K | 25-35 tok/s | B+ | +| Qwen3.5-35B-A3B Q4 | **KV Q8 + flash** | **~80-100K** | **25-35 tok/s** | **B+** | +| Qwen3.5-35B-A3B Q4 | **KV asym + flash** | **~100-130K** | **25-35 tok/s** | **B+ (tiny quality dip)** | +| Qwen3.5-9B Q8 | KV Q8 + flash | ~120-160K | 30-45 tok/s | B | +| Qwen2.5-Coder-14B Q4 | KV Q8 + flash | ~40-64K | 20-30 tok/s | B+ | + +Add `--cram` on top for instant repeated prompts. That's your real single-card ceiling. + +**The honest answer:** With all optimizations stacked, a single card goes from "a few files +at a time" to "maybe half a 10K-line project." That's a meaningful upgrade from the +unoptimized baseline, but it still doesn't match what 2 cards with a dense 27B model gives +you. The second card isn't about optimization tricks — it's about physics (more VRAM = more +data on the fast bus). + ### Estimated Inference Speed | Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) | @@ -375,16 +520,35 @@ cmake --build build --config Release -j$(nproc) # Download Qwen3.5-35B-A3B GGUF (Q4_K_M) # Get from: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF -# Run on dual GPU with NVLink +# Run on dual GPU with NVLink (all optimizations on) ./build/bin/llama-server \ -m Qwen3.5-35B-A3B-Q4_K_M.gguf \ -ngl 999 \ -c 262144 \ + --cache-type-k q8_0 \ + --cache-type-v q4_0 \ + --flash-attn \ + --cram 65536 \ --host 0.0.0.0 \ --port 8080 +# --cache-type-k q8_0 / --cache-type-v q4_0 = asymmetric KV quantization (62% smaller cache) +# --flash-attn = tiled attention (less VRAM, no quality loss) +# --cram 65536 = 64GB host RAM prompt cache (instant repeat prompts) # llama.cpp auto-detects NVLink and splits layers across both GPUs # Use -ts 1,1 to manually set equal split if needed + +# Single card variant (no NVLink) — same flags, smaller context +./build/bin/llama-server \ + -m Qwen3.5-35B-A3B-Q4_K_M.gguf \ + -ngl 999 \ + -c 131072 \ + --cache-type-k q8_0 \ + --cache-type-v q4_0 \ + --flash-attn \ + --cram 65536 \ + --host 0.0.0.0 \ + --port 8080 ``` #### Ollama