ai-stack: vendor the local-ai full AI stack as a new service
Vendor the functional source of github.com/outis1one/local-ai into ./ai-stack (22 files) and add services/ai-stack.sh, which copies the source to ~/docker/ai-stack and hands off to the app's VRAM-aware installer (local-ai-setup.sh). The stack bundles Ollama, Open WebUI, RAG + MCP servers, ChromaDB, SearXNG, Kiwix, Gitea, InvokeAI, ComfyUI and Portainer. Cloud LLM providers (Groq/DeepInfra/OpenAI/OpenRouter) are optionally wired into Open WebUI via the plural OPENAI_API_BASE_URLS list, with the local RAG connection kept as the first entry so RAG keeps working. Open WebUI ships built-in auth, so Caddy is configured without Authelia. Excludes the upstream's two bundled copies of this very project (ubuntu-post-install.sh, ubuntu-post-install-main.zip) — stale and circular. Coexists with the existing ai-gpu service. Also fix the install-function names for ai-gpu and ai-stack: the dispatcher calls install_<raw-name>, so the function must be install_ai-gpu / install_ai-stack (hyphen), matching the working mail-archiver / wg-easy services. ai-gpu was previously uninstallable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nb2vJ8W7bHKx1JXVvpCraH
This commit is contained in:
Claude
@@ -67,7 +67,7 @@ Update them any time with `sudo ./setup.sh configure`.
|
||||
|-------|---------|
|
||||
| `base` | `net-tools`, `ncdu`, `git`, `curl`, `wget`, `htop`, `tree`, `zip`/`unzip`, `ca-certificates`, `gnupg`, `jq`, `rsync`; `glow` (terminal markdown reader, Charm apt repo) |
|
||||
| `homelab` | `caddy`, `crowdsec`, `authelia`, `homeassistant`, `asterisk`, `sunshine` |
|
||||
| `utilities` | `actualbudget`, `ai-gpu`, `archivebox`, `changedetection`, `ddclient`, `filebrowser`, `fmd`, `gatus`, `homebox`, `iopaint`, `joplin`, `koha`, `magicmirror`, `mail-archiver`, `mattermost`, `mealie`, `meshcentral`, `n8n`, `nextcloud`, `ntfy`, `onlyoffice`, `paintplus`, `portainer`, `rustdesk`, `stirling-pdf`, `syncthing`, `traccar`, `unifi`, `uptimekuma`, `vaultwarden`, `watchyourlan`, `watchtower`, `wg-easy` |
|
||||
| `utilities` | `actualbudget`, `ai-gpu`, `ai-stack`, `archivebox`, `changedetection`, `ddclient`, `filebrowser`, `fmd`, `gatus`, `homebox`, `iopaint`, `joplin`, `koha`, `magicmirror`, `mail-archiver`, `mattermost`, `mealie`, `meshcentral`, `n8n`, `nextcloud`, `ntfy`, `onlyoffice`, `paintplus`, `portainer`, `rustdesk`, `stirling-pdf`, `syncthing`, `traccar`, `unifi`, `uptimekuma`, `vaultwarden`, `watchyourlan`, `watchtower`, `wg-easy` |
|
||||
| `media` | `arm`, `audiobookshelf`, `calibre-web`, `emby`, `immich`, `jellyfin`, `lyrion` |
|
||||
| `cameras` | `frigate`, `frigate-audio`, `frigate-notify`, `sky-cam` |
|
||||
| `gaming` | `drum-rhythm-game`, `js99er`, `kyber-launcher`, `kyber-server`, `minecraft`, `wolf`, `wolf-pair` |
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# Generated by setup scripts — not tracked
|
||||
docker-compose.yml
|
||||
.env
|
||||
requirements.txt
|
||||
mcp_requirements.txt
|
||||
start.sh
|
||||
stop.sh
|
||||
status.sh
|
||||
pull-models.sh
|
||||
Caddyfile.example
|
||||
.ufw-done
|
||||
|
||||
# Data directories
|
||||
workspace/
|
||||
repos/
|
||||
papers/
|
||||
index/
|
||||
logs/
|
||||
kiwix/
|
||||
gitea/
|
||||
portainer-data/
|
||||
invokeai-data/
|
||||
invokeai-outputs/
|
||||
@@ -0,0 +1,988 @@
|
||||
# Local AI Stack
|
||||
|
||||
A fully offline, self-hosted AI environment for Ubuntu 24.04. Runs on any NVIDIA GPU (or CPU-only).
|
||||
|
||||
**Services:** Ollama · Open WebUI · RAG · MCP · ChromaDB · SearXNG · Kiwix · Gitea · InvokeAI · ComfyUI · Portainer
|
||||
|
||||
> **Vendored into [ubuntu-post-install](https://github.com/outis1one/ubuntu-post-install) as the `ai-stack` service.**
|
||||
> `services/ai-stack.sh` copies this source to `~/docker/ai-stack/` and hands off to `local-ai-setup.sh`
|
||||
> (the VRAM-aware installer below). The upstream `ubuntu-post-install.sh` and `ubuntu-post-install-main.zip`
|
||||
> — old standalone copies of that project, "independent of the AI stack" per this README — are omitted from
|
||||
> the vendored copy. Source: github.com/outis1one/local-ai
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
git clone <this-repo>
|
||||
cd local-ai
|
||||
./laptop_full_setup.sh
|
||||
```
|
||||
|
||||
That's it. The script installs Docker, NVIDIA drivers (if needed), generates all config, starts the stack, and optionally pulls models.
|
||||
|
||||
---
|
||||
|
||||
## Service URLs
|
||||
|
||||
After setup, all services are available on your LAN:
|
||||
|
||||
| Service | URL | Purpose |
|
||||
|------------|----------------------------|--------------------------------|
|
||||
| Open WebUI | `http://<ip>:3000` | Chat interface (Ollama + RAG) |
|
||||
| InvokeAI | `http://<ip>:9090` | Image generation (standalone) |
|
||||
| ComfyUI | `http://<ip>:8188` | Image generation (OWUI integration) |
|
||||
| SearXNG | `http://<ip>:8888` | Private web search |
|
||||
| Kiwix | `http://<ip>:8181` | Offline Wikipedia / docs |
|
||||
| Gitea | `http://<ip>:3001` | Self-hosted Git |
|
||||
| RAG Health | `http://<ip>:8001/health` | RAG server status |
|
||||
| MCP SSE | `http://<ip>:8002/sse` | MCP endpoint for Claude Code |
|
||||
| Portainer | `https://<ip>:9443` | Docker management UI |
|
||||
|
||||
---
|
||||
|
||||
## Day-to-Day Commands
|
||||
|
||||
All generated into `~/docker/ai-stack/` by the setup script:
|
||||
|
||||
```bash
|
||||
bash ~/docker/ai-stack/start.sh # pull latest images + docker compose up -d
|
||||
bash ~/docker/ai-stack/stop.sh # docker compose down
|
||||
bash ~/docker/ai-stack/status.sh # GPU / container / RAG health
|
||||
bash ~/docker/ai-stack/pull-models.sh # pull Ollama models (run once after first install)
|
||||
```
|
||||
|
||||
The stack also registers as a **systemd service** that starts on boot:
|
||||
|
||||
```bash
|
||||
sudo systemctl start local-ai
|
||||
sudo systemctl stop local-ai
|
||||
sudo systemctl status local-ai
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Script Reference
|
||||
|
||||
| Script | Lines | What it does |
|
||||
|---|---|---|
|
||||
| `laptop_full_setup.sh` | 620 | **Main setup.** Installs Docker + NVIDIA toolkit, creates `~/docker/ai-stack/`, writes `docker-compose.yml`, starts stack, registers systemd service. |
|
||||
| `local-ai-setup.sh` | 837 | Alternative setup script. Same as above but also **auto-detects VRAM** and selects models accordingly (14B for ≥14GB VRAM, 7B for CPU). Use this instead of `laptop_full_setup.sh` if you want VRAM-aware model selection. |
|
||||
| `ubuntu-post-install.sh` | 8,889 | Full Ubuntu 24.04 post-install (dev tools, fonts, apps, tweaks). Run once on a fresh OS install. Independent of the AI stack. |
|
||||
| `configure-storage.sh` | 239 | Storage/mount configuration helper. Run separately if you have a secondary drive for AI data. |
|
||||
| `kiwix_download.sh` | 198 | Downloads ZIM files (Wikipedia, Stack Overflow, etc.) for offline use. Run separately — files are large. |
|
||||
| `invokeai-import-lora.sh` | 85 | Copies a LoRA `.safetensors` file into InvokeAI's Docker model volume. |
|
||||
| `comfyui-import-lora.sh` | 97 | Copies a LoRA into ComfyUI and prints workflow setup instructions. |
|
||||
| `comfyui-install-ipadapter.sh` | 185 | Installs IP-Adapter nodes + models into ComfyUI for reference-image workflows (same face, different settings). |
|
||||
| `setup-image-models.sh` | 200 | **GPU-aware** image model installer. Detects VRAM, offers appropriate SD/SDXL/Flux models, installs into InvokeAI and/or ComfyUI. |
|
||||
|
||||
### Which setup script should I use?
|
||||
|
||||
- `laptop_full_setup.sh` — fixed model selection (`qwen2.5:14b` / `qwen2.5-coder:7b`), simpler
|
||||
- `local-ai-setup.sh` — detects your VRAM at runtime and picks appropriate models, also embeds `server.py` and `mcp_server.py` directly (doesn't need repo files copied separately)
|
||||
|
||||
Both scripts are **idempotent** — safe to re-run for updates. Config files are kept on re-run unless you pass `--force`.
|
||||
|
||||
---
|
||||
|
||||
## Generated File Layout
|
||||
|
||||
```
|
||||
~/docker/ai-stack/
|
||||
├── docker-compose.yml # generated by setup script
|
||||
├── .env # API tokens — edit this, never overwritten
|
||||
├── server.py # RAG server (copied from repo)
|
||||
├── mcp_server.py # MCP server (copied from repo)
|
||||
├── requirements.txt # RAG Python deps
|
||||
├── mcp_requirements.txt # MCP Python deps
|
||||
├── start.sh # start the stack
|
||||
├── stop.sh # stop the stack
|
||||
├── status.sh # GPU + container + RAG health
|
||||
├── pull-models.sh # pull Ollama models
|
||||
├── Caddyfile.example # reverse proxy config template
|
||||
├── papers/ # drop PDFs here for RAG indexing
|
||||
├── repos/ # git repos indexed by RAG
|
||||
├── workspace/ # MCP working directory
|
||||
├── index/ # ChromaDB vector store (persistent)
|
||||
├── kiwix/ # ZIM files for Kiwix
|
||||
├── gitea/ # Gitea data
|
||||
├── invokeai-outputs/ # InvokeAI generated images
|
||||
├── comfyui-output/ # ComfyUI generated images
|
||||
├── comfyui-data/ # ComfyUI custom nodes
|
||||
└── logs/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## First Run Checklist
|
||||
|
||||
1. **Run setup:**
|
||||
```bash
|
||||
./laptop_full_setup.sh
|
||||
```
|
||||
2. **Pull models** (prompted at end of setup, or run manually):
|
||||
```bash
|
||||
bash ~/docker/ai-stack/pull-models.sh
|
||||
```
|
||||
Downloads ~15-30GB. Takes 10-40 min depending on connection.
|
||||
|
||||
3. **Add API tokens** (optional — for Gitea/GitHub MCP tools):
|
||||
```bash
|
||||
nano ~/docker/ai-stack/.env
|
||||
```
|
||||
|
||||
4. **Connect Claude Code to MCP:**
|
||||
```bash
|
||||
claude mcp add local http://<your-ip>:8002/sse
|
||||
```
|
||||
|
||||
5. **Download ZIMs** for offline docs (optional, large):
|
||||
```bash
|
||||
./kiwix_download.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Querying Your Codebase from Open WebUI
|
||||
|
||||
The stack includes a **code-aware RAG server** that sits between Open WebUI and Ollama. When you chat in Open WebUI, the RAG server automatically retrieves relevant code from your indexed repos and injects it into the prompt — so the model answers with your actual code as context.
|
||||
|
||||
This is NOT the same as Open WebUI's built-in Knowledge Collections. This is a separate, always-on layer that understands code structure.
|
||||
|
||||
### How it works (architecture)
|
||||
|
||||
```
|
||||
You type a question in Open WebUI
|
||||
↓
|
||||
Open WebUI → RAG Server (port 8001) /v1/chat/completions
|
||||
↓
|
||||
RAG Server: searches ChromaDB for relevant code chunks
|
||||
(AST-parsed Python functions, regex-split JS/Go/Rust, etc.)
|
||||
↓
|
||||
RAG Server: prepends code snippets to your prompt:
|
||||
"### auth.py:authenticate
|
||||
def authenticate(user, password): ..."
|
||||
↓
|
||||
RAG Server → Ollama: generates response WITH your code as context
|
||||
↓
|
||||
Response sent back to Open WebUI
|
||||
```
|
||||
|
||||
Open WebUI is already configured to route through the RAG server via:
|
||||
```
|
||||
OPENAI_API_BASE_URL=http://rag-server:8001/v1
|
||||
```
|
||||
|
||||
### Step 1: Index your code
|
||||
|
||||
**Option A: Drop repos in the repos directory**
|
||||
```bash
|
||||
cd ~/docker/ai-stack/repos
|
||||
git clone http://localhost:3001/your-user/your-repo.git
|
||||
# Restart RAG server to trigger indexing:
|
||||
docker restart rag-server
|
||||
```
|
||||
|
||||
**Option B: Use the ingest API**
|
||||
```bash
|
||||
# From Gitea:
|
||||
curl -X POST http://localhost:8001/ingest/repo \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url": "http://gitea:3000/user/repo", "name": "my-repo"}'
|
||||
|
||||
# From GitHub:
|
||||
curl -X POST http://localhost:8001/ingest/repo \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url": "https://github.com/user/repo", "name": "my-repo", "branch": "main"}'
|
||||
```
|
||||
|
||||
**Option C: Auto-index on push (Gitea webhook)**
|
||||
1. In Gitea → your repo → **Settings** → **Webhooks** → **Add Webhook** → **Gitea**
|
||||
2. Target URL: `http://rag-server:8001/webhook/gitea`
|
||||
3. Trigger: Push events
|
||||
4. Now every `git push` to Gitea auto-reindexes that repo
|
||||
|
||||
### Step 2: Chat about your code
|
||||
|
||||
Just ask questions in Open WebUI. The RAG server automatically retrieves relevant code:
|
||||
|
||||
- "What does the authenticate function do?"
|
||||
- "How is the database connection configured?"
|
||||
- "Show me all the API endpoints"
|
||||
- "What tests exist for the user model?"
|
||||
|
||||
The model sees the actual code snippets and responds based on them — not hallucinating.
|
||||
|
||||
### What gets indexed
|
||||
|
||||
**Supported file types:**
|
||||
`.py`, `.js`, `.ts`, `.tsx`, `.jsx`, `.go`, `.rs`, `.java`, `.c`, `.cpp`, `.h`, `.hpp`, `.cs`, `.rb`, `.sh`, `.yaml`, `.yml`, `.toml`, `.sql`, `.md`
|
||||
|
||||
**Smart chunking:**
|
||||
- **Python:** AST-parsed — each function and class is its own searchable chunk
|
||||
- **Other languages:** Regex-split on `function`, `class`, `const`, `func`, `impl`, etc.
|
||||
- **Fallback:** Sliding window (1200 chars, 200 char overlap)
|
||||
|
||||
**Skipped directories:**
|
||||
`node_modules`, `.git`, `__pycache__`, `dist`, `build`, `.venv`, `venv`, `vendor`, `target`, `bin`, `obj`
|
||||
|
||||
### Checking RAG status
|
||||
|
||||
```bash
|
||||
curl http://localhost:8001/health
|
||||
```
|
||||
|
||||
Returns document counts per collection (code, papers) and overall status.
|
||||
|
||||
### RAG server vs Knowledge Collections vs Memories
|
||||
|
||||
These are **three separate systems** in the stack. Understanding the difference matters:
|
||||
|
||||
| | **RAG Server** | **Knowledge Collections** | **Memories** |
|
||||
|---|---|---|---|
|
||||
| **What** | Code-aware retrieval layer | Open WebUI's built-in document RAG | Short facts about the user |
|
||||
| **Content** | Git repos (auto-indexed) | PDFs, text files you upload | Extracted from conversations |
|
||||
| **Chunking** | AST/regex (code-aware) | Generic text chunking | Single sentences |
|
||||
| **Activation** | Always on (every chat) | Per-chat (`#` tag) | Global (every chat) |
|
||||
| **Best for** | "What does this function do?" | Project docs, research notes | "Remember I prefer Python" |
|
||||
| **Context cost** | ~1500-2500 tokens (top 6 chunks) | ~1500-2500 tokens (top K chunks) | ~200 tokens |
|
||||
|
||||
**For code questions:** The RAG server handles this automatically — no setup needed beyond indexing your repos.
|
||||
|
||||
**For project docs/notes:** Use Knowledge Collections — type `#` to scope per-chat.
|
||||
|
||||
**For personal preferences:** Use Memories (sparingly — they're global).
|
||||
|
||||
---
|
||||
|
||||
## Image Generation from Open WebUI
|
||||
|
||||
Open WebUI can generate images inline in chat conversations using ComfyUI as the backend. When configured, you can ask any model to "generate an image of..." and it will call ComfyUI to create the image.
|
||||
|
||||
### How it works
|
||||
|
||||
Open WebUI natively supports these image generation engines:
|
||||
- **ComfyUI** — Node-based, best Open WebUI integration, local
|
||||
- **AUTOMATIC1111** — Stable Diffusion WebUI, local
|
||||
- **OpenAI DALL-E** — Cloud API
|
||||
- **Gemini** — Cloud API
|
||||
|
||||
**InvokeAI** does NOT have a compatible API for Open WebUI integration. It works great as a standalone tool at `http://<ip>:9090` but cannot be called from within Open WebUI chats. For chat-integrated image generation, use **ComfyUI**.
|
||||
|
||||
### Setup: ComfyUI + Open WebUI (recommended)
|
||||
|
||||
If you selected ComfyUI during setup, the environment variables are already configured. You just need to install a model and set up a workflow.
|
||||
|
||||
#### Step 1: Install a Stable Diffusion model in ComfyUI
|
||||
|
||||
```bash
|
||||
# Open ComfyUI at http://<ip>:8188
|
||||
# Use the built-in Model Manager to download a model, or manually:
|
||||
docker exec comfyui bash -c "cd /opt/ComfyUI/models/checkpoints && \
|
||||
wget -q 'https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors'"
|
||||
```
|
||||
|
||||
Or download any `.safetensors` checkpoint and copy it in:
|
||||
```bash
|
||||
docker cp ~/Downloads/my-model.safetensors comfyui:/opt/ComfyUI/models/checkpoints/
|
||||
```
|
||||
|
||||
#### Step 2: Create and export a workflow
|
||||
|
||||
1. Open ComfyUI at `http://<ip>:8188`
|
||||
2. Build or load a workflow (the default text-to-image workflow works)
|
||||
3. Click the **gear icon** → enable **Dev Mode**
|
||||
4. Click **Save (API Format)** — this downloads `workflow_api.json`
|
||||
|
||||
#### Step 3: Configure Open WebUI
|
||||
|
||||
1. Open WebUI → **Admin** → **Settings** → **Images**
|
||||
2. Set **Engine** to `ComfyUI`
|
||||
3. Set **URL** to `http://comfyui:8188` (container networking, already set via env vars)
|
||||
4. Click **Import Workflow** and upload your `workflow_api.json`
|
||||
5. Map the **prompt node** (usually the KSampler or CLIPTextEncode node)
|
||||
6. Save settings
|
||||
|
||||
#### Step 4: Generate images in chat
|
||||
|
||||
In any Open WebUI chat, type something like:
|
||||
- "Generate an image of a mountain landscape at sunset"
|
||||
- "Create a photo of a cyberpunk city"
|
||||
|
||||
The model will detect the image generation request and pass it to ComfyUI.
|
||||
|
||||
### Using LoRAs with Open WebUI (workflow-per-style)
|
||||
|
||||
LoRAs let you apply trained styles (anime, photorealistic, specific characters, etc.) to image generation. The trick: **bake the LoRA into a ComfyUI workflow, export it, and import into Open WebUI.** After that, you iterate from chat without touching ComfyUI.
|
||||
|
||||
#### Step 1: Import your LoRA file
|
||||
|
||||
```bash
|
||||
./comfyui-import-lora.sh ~/Downloads/my-style-lora.safetensors "Anime Style"
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
docker cp ~/Downloads/my-style-lora.safetensors comfyui:/opt/ComfyUI/models/loras/
|
||||
```
|
||||
|
||||
#### Step 2: Build a workflow with the LoRA (one-time)
|
||||
|
||||
1. Open ComfyUI at `http://<ip>:8188`
|
||||
2. Load the default text-to-image workflow
|
||||
3. Add a **Load LoRA** node: right-click → Add Node → loaders → Load LoRA
|
||||
4. Wire it between the checkpoint and the rest of the pipeline:
|
||||
```
|
||||
[Load Checkpoint] → MODEL → [Load LoRA] → MODEL → [KSampler]
|
||||
→ CLIP → → CLIP → [CLIP Text Encode]
|
||||
```
|
||||
5. Select your LoRA file, set `strength_model` and `strength_clip` (start 0.7–0.85)
|
||||
6. Test it — click **Queue Prompt** and verify it works
|
||||
7. Enable **Dev Mode** (gear icon) → click **Save (API Format)**
|
||||
|
||||
#### Step 3: Import into Open WebUI
|
||||
|
||||
1. Open WebUI → **Admin** → **Settings** → **Images**
|
||||
2. Click **Import Workflow** → upload the `workflow_api.json`
|
||||
3. Map the prompt node (usually CLIPTextEncode)
|
||||
4. Save
|
||||
|
||||
Now every "Generate an image of..." in chat uses your LoRA automatically.
|
||||
|
||||
#### Combining multiple LoRAs in one image
|
||||
|
||||
You can chain multiple Load LoRA nodes to blend styles:
|
||||
|
||||
```
|
||||
[Load Checkpoint] → [Load LoRA 1] → [Load LoRA 2] → [KSampler]
|
||||
(anime, 0.7) (lighting, 0.5)
|
||||
```
|
||||
|
||||
Each LoRA has its own `strength_model` / `strength_clip` sliders. Export the combined workflow as usual — both LoRAs are baked in and always applied.
|
||||
|
||||
#### Managing multiple LoRA styles
|
||||
|
||||
Each LoRA combo needs its own exported workflow. Practical approach:
|
||||
|
||||
| Workflow file | LoRAs | Use case |
|
||||
|---------------|-------|----------|
|
||||
| `anime-only.json` | Anime @ 0.8 | Anime style |
|
||||
| `anime-cinematic.json` | Anime @ 0.7 + Cinematic @ 0.5 | Anime + dramatic lighting |
|
||||
| `photorealistic.json` | Photorealistic @ 0.9 | Photo look |
|
||||
| `base-model.json` | None | No LoRA, base checkpoint only |
|
||||
|
||||
Switch between them in **Admin → Settings → Images → Import Workflow**. The active workflow applies to all image generation requests.
|
||||
|
||||
> **Important:** There are no keywords or trigger words to activate LoRAs from chat. LoRAs are hardwired in the workflow JSON — whatever prompt you type, the LoRA(s) always apply. To change styles, swap the workflow in OWUI settings.
|
||||
|
||||
#### LoRA compatibility
|
||||
|
||||
| LoRA trained on | Must use checkpoint |
|
||||
|----------------|-------------------|
|
||||
| SD 1.5 | Any SD 1.5 model (e.g. `v1-5-pruned-emaonly.safetensors`) |
|
||||
| SDXL | Any SDXL model (e.g. `sd_xl_base_1.0.safetensors`) |
|
||||
| Flux | Flux checkpoint |
|
||||
|
||||
Mismatched architectures will produce errors or garbage output.
|
||||
|
||||
### IP-Adapter: same face, different settings (reference image workflows)
|
||||
|
||||
IP-Adapter lets you upload a reference photo and generate new images that preserve the face/subject while changing everything else — age, emotion, setting, art style. This is different from LoRAs (which bake in a trained style). IP-Adapter works from a single image with no training needed.
|
||||
|
||||
#### What you can do
|
||||
|
||||
| Task | How | Example prompt |
|
||||
|------|-----|----------------|
|
||||
| Different setting | Reference face + new scene | "same person, sitting in a cafe in Paris" |
|
||||
| Age up/down | Reference face + age prompt | "same person as an elderly man" or "same person as a child" |
|
||||
| Emotions | Reference face + emotion | "same person, laughing" or "same person, crying" |
|
||||
| Art style | Reference face + style | "same person, oil painting, renaissance style" |
|
||||
| Pose change | Reference face + pose | "same person, looking over shoulder, dramatic lighting" |
|
||||
|
||||
#### Quick install
|
||||
|
||||
```bash
|
||||
./comfyui-install-ipadapter.sh # SDXL models (~2.5GB)
|
||||
./comfyui-install-ipadapter.sh --faceid # + FaceID for stronger face lock
|
||||
./comfyui-install-ipadapter.sh --all # SDXL + SD 1.5 models
|
||||
```
|
||||
|
||||
Then restart ComfyUI: `docker restart comfyui`
|
||||
|
||||
#### Basic workflow in ComfyUI
|
||||
|
||||
1. Open `http://<ip>:8188`
|
||||
2. Add nodes: right-click → Add Node → ipadapter
|
||||
3. Wire up:
|
||||
```
|
||||
[Load Checkpoint] → MODEL → [IPAdapter Apply] → [KSampler] → [VAE Decode] → [Save Image]
|
||||
→ CLIP → [CLIP Text Encode] → positive →
|
||||
[Load Image (your photo)] → [IPAdapter Unified Loader] → ipadapter →
|
||||
```
|
||||
4. **IPAdapter Unified Loader**: set preset to `PLUS FACE (portrait)`
|
||||
5. **IPAdapter Apply**: set weight 0.7–1.0 (higher = more faithful to reference)
|
||||
6. Text prompt: describe the new scene/emotion/age
|
||||
7. Click **Queue Prompt**
|
||||
|
||||
#### Adjusting face fidelity
|
||||
|
||||
| Weight | Effect |
|
||||
|--------|--------|
|
||||
| 0.5–0.6 | Loose reference — inspired by the face but not a match |
|
||||
| 0.7–0.8 | Good balance — recognizable face, creative freedom in scene |
|
||||
| 0.9–1.0 | Strong lock — very close to reference face |
|
||||
| FaceID preset | Strongest — uses face detection for identity lock |
|
||||
|
||||
#### Important: ComfyUI only (not from Open WebUI chat)
|
||||
|
||||
IP-Adapter workflows require uploading a reference image to ComfyUI. Open WebUI's image generation integration only sends text prompts — it can't attach reference images. For this use case, work directly in ComfyUI at `http://<ip>:8188`.
|
||||
|
||||
For text-only image generation from Open WebUI chat, use [LoRA workflows](#using-loras-with-open-webui-workflow-per-style) instead.
|
||||
|
||||
### Setup: AUTOMATIC1111 (alternative)
|
||||
|
||||
If you prefer AUTOMATIC1111 over ComfyUI:
|
||||
|
||||
1. Run AUTOMATIC1111 with the `--api` flag
|
||||
2. In Open WebUI → **Admin** → **Settings** → **Images**:
|
||||
- Engine: `Automatic1111`
|
||||
- URL: `http://host.docker.internal:7860` (or container name if in Docker)
|
||||
3. Environment variables (alternative to UI config):
|
||||
```
|
||||
ENABLE_IMAGE_GENERATION=true
|
||||
IMAGE_GENERATION_ENGINE=automatic1111
|
||||
AUTOMATIC1111_BASE_URL=http://host.docker.internal:7860
|
||||
```
|
||||
|
||||
### Environment variables reference
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `ENABLE_IMAGE_GENERATION` | `false` | Enable image generation feature |
|
||||
| `IMAGE_GENERATION_ENGINE` | — | `comfyui`, `automatic1111`, `openai`, or `gemini` |
|
||||
| `IMAGE_GENERATION_MODEL` | — | Model ID for generation |
|
||||
| `IMAGE_SIZE` | `512x512` | Default output size |
|
||||
| `COMFYUI_BASE_URL` | — | ComfyUI API URL (e.g. `http://comfyui:8188`) |
|
||||
| `COMFYUI_API_KEY` | — | ComfyUI API key (if auth enabled) |
|
||||
| `COMFYUI_WORKFLOW` | — | Custom workflow JSON (API format) |
|
||||
| `AUTOMATIC1111_BASE_URL` | — | AUTOMATIC1111 API URL |
|
||||
| `AUTOMATIC1111_API_AUTH` | — | Auth credentials (`user:pass`) |
|
||||
|
||||
### Installing Functions & Actions (without community signup)
|
||||
|
||||
The Open WebUI community hub (`openwebui.com`) requires a free account to download functions. If you get parse errors pasting URLs or don't want to sign up, you can install functions manually by pasting their source code directly.
|
||||
|
||||
#### Manual installation (no account needed)
|
||||
|
||||
1. Find the function's source on GitHub — most are in the [open-webui/functions](https://github.com/open-webui/functions) repo or linked from community pages
|
||||
2. Copy the **raw Python source code** (the entire `.py` file)
|
||||
3. In Open WebUI → **Workspace** → **Functions** → click **＋** (Create)
|
||||
4. Paste the code into the editor
|
||||
5. Give it a name and save
|
||||
6. Enable it under **Workspace** → **Functions** — toggle it on globally or per-model
|
||||
|
||||
For **Actions** (like Generate Image), the process is the same but go to **Workspace** → **Functions** and set the function type to **Action** in the metadata.
|
||||
|
||||
#### Recommended functions
|
||||
|
||||
| Function | Type | What it does |
|
||||
|----------|------|-------------|
|
||||
| **Auto Memory** | Filter | Automatically extracts and stores facts from conversations as persistent memories |
|
||||
| **Generate Image** | Action | Adds a "Generate Image" button to messages for quick image generation |
|
||||
|
||||
### Auto Memory setup
|
||||
|
||||
Auto Memory is a filter function that runs after each message exchange, uses an LLM call to extract noteworthy facts, and stores them as user memories in Open WebUI's built-in memory system.
|
||||
|
||||
#### Step 1: Install the function
|
||||
|
||||
Use the manual method above. The source code is at:
|
||||
`https://github.com/open-webui/functions` — search for `auto_memory` or `adaptive_memory`
|
||||
|
||||
#### Step 2: Configure the function settings
|
||||
|
||||
After installing, click the **gear icon** on the function to configure:
|
||||
|
||||
| Setting | Value for local stack | Notes |
|
||||
|---------|----------------------|-------|
|
||||
| `openai_api_url` | `http://ollama:11434/v1` | Ollama's OpenAI-compatible endpoint |
|
||||
| `model` | Your chat model (e.g. `qwen2.5:14b`) | Used for memory extraction LLM calls |
|
||||
| `api_key` | `ollama` | Any non-empty string — Ollama ignores auth |
|
||||
| `context_window_n` | `4` (default) | Number of related memories injected per message |
|
||||
| `similarity_score_filter` | Leave default | Cosine similarity threshold for memory retrieval |
|
||||
| `messages_to_consider` | Leave default | How many recent messages to analyze |
|
||||
| `allow_modify_user_memories` | `false` | **Keep off** — lets LLM delete/edit memories via API if enabled |
|
||||
|
||||
#### Step 3: Enable memories per user
|
||||
|
||||
Each user must enable the memory feature individually:
|
||||
1. Click your **profile icon** → **Settings** → **Personalization**
|
||||
2. Toggle **Memory** to **ON**
|
||||
|
||||
Without this, the function installs but does nothing.
|
||||
|
||||
### Generate Image action setup
|
||||
|
||||
1. Install via manual method above (search for `generate_image` in the functions repo)
|
||||
2. Configure the function settings:
|
||||
- Uses your existing Open WebUI image generation settings (ComfyUI/A1111)
|
||||
- No additional API configuration needed if image generation already works
|
||||
3. After installing, a **"Generate Image"** button appears on AI messages
|
||||
4. Clicking it sends the message content as an image generation prompt
|
||||
|
||||
### Memory system: how it works and context impact
|
||||
|
||||
Open WebUI has two separate systems for persistent knowledge. Understanding the difference is critical for managing your context window.
|
||||
|
||||
#### Built-in Memories vs Knowledge Collections
|
||||
|
||||
| | **Memories** | **Knowledge** |
|
||||
|---|---|---|
|
||||
| **What** | Short facts extracted from chats | Document collections (PDFs, text files) |
|
||||
| **How stored** | Text + vector embeddings in DB | Chunked documents + embeddings in ChromaDB |
|
||||
| **How injected** | Prepended to system prompt every message | Retrieved via RAG only when relevant chunks match |
|
||||
| **Scope** | **Global per user** — injected into ALL chats | **Per-chat** — you choose which collection with `#` |
|
||||
| **Context cost** | Always consumed, every message | Only consumed when you tag a collection |
|
||||
| **Control** | All or nothing (on/off globally) | Fine-grained (pick per conversation) |
|
||||
|
||||
#### Context window consumption by memories
|
||||
|
||||
Each memory is roughly 20–80 tokens (a sentence or two). With the default `context_window_n=4` setting in Auto Memory, 4 related memories are injected per message.
|
||||
|
||||
**Estimated context consumption per message:**
|
||||
|
||||
| Stored memories | Injected per msg | Tokens consumed | % of 8K context | % of 32K context |
|
||||
|----------------|-----------------|-----------------|-----------------|------------------|
|
||||
| 10 | ~4 | ~200 | 2.5% | 0.6% |
|
||||
| 50 | ~4 | ~200 | 2.5% | 0.6% |
|
||||
| 200 | ~4 | ~200 | 2.5% | 0.6% |
|
||||
|
||||
The key insight: **only the top N similar memories are injected** (default 4), not all of them. So having 200 memories doesn't consume more context than having 10 — the retrieval system picks the most relevant ones.
|
||||
|
||||
However, **the memories are injected on every single message** in every chat. This is the overhead cost.
|
||||
|
||||
#### Context consumed by chat history (the bigger problem)
|
||||
|
||||
The real context pressure comes from **conversation history**, not memories. Here's what actually fills your context window:
|
||||
|
||||
| Messages in chat | ~Tokens used | % of 8K | % of 32K |
|
||||
|-----------------|-------------|---------|----------|
|
||||
| 5 exchanges (10 msgs) | ~2,000–4,000 | 25–50% | 6–12% |
|
||||
| 20 exchanges (40 msgs) | ~8,000–16,000 | **100%+ (truncated)** | 25–50% |
|
||||
| 100 exchanges (200 msgs) | ~40,000–80,000 | **way over** | **100%+ (truncated)** |
|
||||
|
||||
With a small model running 8K context, you'll hit the limit after ~10-20 exchanges. The model starts dropping earlier messages. Memories add a small fixed overhead (~200 tokens) on top of this.
|
||||
|
||||
#### Across multiple separate chats
|
||||
|
||||
**Good news:** separate chats do NOT share context windows. Each chat starts fresh. The only cross-chat cost is the ~200 tokens of memories injected into each new chat's system prompt.
|
||||
|
||||
So 5, 20, or 100 separate chats don't accumulate — each one independently uses the context window. Memories are the only thing that carries over.
|
||||
|
||||
### Project-scoped memory (avoiding global memory pollution)
|
||||
|
||||
**Open WebUI does NOT have native project-scoped memory.** Memories are global per user — every fact extracted from any chat gets injected into every other chat.
|
||||
|
||||
This is a known limitation. Here are workarounds:
|
||||
|
||||
#### Option 1: Use Knowledge Collections instead of Memories (recommended)
|
||||
|
||||
Knowledge collections give you the scoping you want:
|
||||
|
||||
1. **Create a collection:** Workspace → Knowledge → Create Collection (e.g. "GPU Research Project")
|
||||
2. **Add documents:** Upload PDFs, text files, or paste notes into the collection
|
||||
3. **Use per-chat:** In any chat, type `#` and select your collection — only that chat gets the context
|
||||
4. **One-off chats stay clean:** Don't tag a collection, and no project context is injected
|
||||
|
||||
This is the closest thing to "projects" in Open WebUI. You can have separate knowledge collections for separate projects, and only pull them in when relevant.
|
||||
|
||||
#### Option 2: Disable Auto Memory, use manual memories
|
||||
|
||||
1. Turn off the Auto Memory function
|
||||
2. Manually add memories via **Profile → Settings → Personalization → Memories**
|
||||
3. Keep only universally useful facts (your name, preferences, etc.)
|
||||
4. Use Knowledge collections for project-specific context
|
||||
|
||||
#### Option 3: Periodically clear memories
|
||||
|
||||
**Profile → Settings → Personalization → Memories → Clear All** — nuclear option, but keeps things clean between projects.
|
||||
|
||||
### Working with Knowledge Collections (small context window survival guide)
|
||||
|
||||
With small local models (4B-9B, 4K-8K context), you'll hit the context window limit fast — often after 10-20 exchanges. Here's how to work effectively despite that.
|
||||
|
||||
#### How Knowledge Collections actually work
|
||||
|
||||
When you type `#` in chat and select a collection, Open WebUI does **RAG retrieval** — it finds the most relevant chunks from the collection, not the whole thing. This is efficient and doesn't blow up your context window.
|
||||
|
||||
- Default chunk size: ~500 tokens
|
||||
- Typically 3-5 relevant chunks are retrieved per query (~1500-2500 tokens)
|
||||
- The chunks are injected as context alongside your message
|
||||
|
||||
#### Creating Knowledge Collections from chat conversations
|
||||
|
||||
There is **no built-in "export chat to knowledge" button** — this is a requested but unimplemented feature. Here's the practical workflow:
|
||||
|
||||
**The handoff method (like the ChatGPT dark ages, but structured):**
|
||||
|
||||
1. When you're ~60-70% through your context window (you'll feel the model getting fuzzy), ask:
|
||||
```
|
||||
Summarize everything we've discussed and decided so far. Include:
|
||||
- Key decisions made
|
||||
- Current state of the work
|
||||
- What still needs to be done
|
||||
- Any important details or constraints
|
||||
Format as a structured document I can use to continue this conversation.
|
||||
```
|
||||
2. **Copy the summary output**
|
||||
3. Go to **Workspace → Knowledge → Create Collection** (e.g. "GPU Build - Session 1")
|
||||
4. Click **Add Content** → paste the summary as a text file (`.txt` or `.md`)
|
||||
5. Start a **new chat**, type `#` and select your collection, then continue where you left off
|
||||
|
||||
**Ongoing project workflow:**
|
||||
|
||||
```
|
||||
Chat 1: Research phase
|
||||
→ Ask for summary at end
|
||||
→ Save summary to "Project X" knowledge collection
|
||||
|
||||
Chat 2: Type # → select "Project X" → continue
|
||||
→ Model gets relevant context via RAG
|
||||
→ Ask for updated summary at end
|
||||
→ Add updated summary to collection (replace or append)
|
||||
|
||||
Chat 3: Type # → select "Project X" → continue
|
||||
→ Repeat...
|
||||
```
|
||||
|
||||
Each chat starts fresh with full context window available, but can pull in relevant history from previous sessions via RAG.
|
||||
|
||||
#### Useful community functions for context management
|
||||
|
||||
Install these manually (Workspace → Functions → Create, paste the code):
|
||||
|
||||
| Function | What it does |
|
||||
|----------|-------------|
|
||||
| **Checkpoint Summarization Filter** | Auto-summarizes conversation history when context gets long |
|
||||
| **Chat Context Clipper** | Keeps only the last N messages, preserves system prompt and first message |
|
||||
| **Context Length Filter** | Hard limits on turns (default 25) and tokens (default 10,000) |
|
||||
|
||||
The **Checkpoint Summarization Filter** is the closest to automatic handoff — it summarizes older messages so the model can keep going without losing track.
|
||||
|
||||
#### How Open WebUI handles context overflow
|
||||
|
||||
By default, Open WebUI **truncates** old messages (drops them silently) — it does NOT auto-summarize. The model just loses access to earlier conversation. This is why you suddenly feel like the model "forgot" what you were talking about.
|
||||
|
||||
You can control this with:
|
||||
- **Context Length Filter** function: set max turns and token limits explicitly
|
||||
- **Chat Context Clipper**: keeps latest N messages, always preserves system prompt + first message pair
|
||||
- Or do manual handoffs before you hit the limit
|
||||
|
||||
#### RAG tuning for better knowledge retrieval
|
||||
|
||||
If your knowledge collections aren't returning good results, tune these in **Admin → Settings → Documents**:
|
||||
|
||||
| Setting | Default | Recommendation |
|
||||
|---------|---------|---------------|
|
||||
| Chunk Size | 500 | 300 for factual docs, 800 for narrative/code |
|
||||
| Chunk Overlap | 100 | 50-100 (higher = better continuity, more tokens) |
|
||||
| Top K | 4 | 3-5 (more = more context consumed) |
|
||||
| Relevance Threshold | 0.0 | 0.3-0.5 (filters out low-quality matches) |
|
||||
|
||||
**Important:** Chunk size cannot exceed your embedding model's token limit. `nomic-embed-text` supports up to 8192 tokens, so you have plenty of headroom.
|
||||
|
||||
#### The honest comparison: Local AI vs Claude Code
|
||||
|
||||
| | **Local AI (Open WebUI + Ollama)** | **Claude Code** |
|
||||
|---|---|---|
|
||||
| **Context window** | 4K-32K (model dependent) | 200K |
|
||||
| **Cross-session memory** | Manual handoffs or Auto Memory | Automatic (CLAUDE.md, project memory) |
|
||||
| **Project scoping** | Knowledge collections (manual) | Built-in (each project has its own context) |
|
||||
| **Code awareness** | RAG server auto-retrieves relevant chunks from indexed repos | Reads your entire codebase directly |
|
||||
| **Continuation** | New chat + `#collection` handoff | "Let's finish X" just works |
|
||||
| **Code editing** | Can't edit files (chat only) | Reads, writes, runs code directly |
|
||||
| **Cost** | Free (your electricity) | API usage fees |
|
||||
| **Privacy** | 100% local | Cloud-based |
|
||||
| **Offline** | Works without internet | Requires internet |
|
||||
|
||||
Local AI requires more manual workflow management but has real code awareness via the RAG server. For code-heavy editing and multi-step tasks, Claude Code is dramatically better. For private code Q&A, document research, and learning — the local stack with RAG + knowledge collections is solid.
|
||||
|
||||
### Channels (beta feature)
|
||||
|
||||
Channels are persistent chat rooms (like Slack/Discord channels) with multi-model support. They do NOT scope memories differently — memories are still global per user. Channels are useful for team collaboration, not memory isolation.
|
||||
|
||||
### Realistic expectations by model size
|
||||
|
||||
Not all models can do all tasks. Here's what to actually expect:
|
||||
|
||||
| Task | 4B (qwen3.5:4b) | 9B (qwen3.5:9b) | 35B MoE (qwen3.5-35b-a3b) | 14B+ dense |
|
||||
|------|:---:|:---:|:---:|:---:|
|
||||
| Answer simple questions | OK | Good | Good | Good |
|
||||
| Explain existing code (with RAG) | OK | Good | Good | Good |
|
||||
| Fix a simple bug (typo, off-by-one) | Maybe | Usually | Usually | Yes |
|
||||
| Write a small utility function | Shaky | OK | Good | Good |
|
||||
| Fix logic error across 2-3 functions | No | Maybe | Usually | Usually |
|
||||
| Write a new feature (multiple files) | No | Shaky | Maybe | Maybe |
|
||||
| Refactor with style consistency | No | No | Sometimes | Sometimes |
|
||||
| Summarize a conversation for handoff | OK | Good | Good | Good |
|
||||
|
||||
**The 35B MoE model (`qwen3.5-35b-a3b`) is the sweet spot for small GPUs.** It was trained as a 35B model but only activates 3B parameters per token. This means it has the *knowledge* of a 35B model with the VRAM footprint closer to a 4B. On a 6GB card it may fit (VRAM usage varies with context length and KV cache settings).
|
||||
|
||||
**Bottom line for a 6GB GPU:**
|
||||
- Use `qwen3.5:4b` for quick chat, explanations, and summarization
|
||||
- Try `qwen3.5-35b-a3b` for code tasks — if it fits, it will be significantly better than 4B
|
||||
- Use Claude Code for anything that requires reading/writing multiple files or complex reasoning
|
||||
- The RAG server helps a lot — even a 4B model gives useful answers when it has the right code chunks in context
|
||||
|
||||
### Fixing models that output code instead of natural language
|
||||
|
||||
If your model (especially smaller ones like Qwen 3.5) responds with Python code blocks instead of plain English answers (as shown in the screenshot), this is a common behavior with code-optimized models.
|
||||
|
||||
#### Why it happens
|
||||
|
||||
- Small models like `qwen3.5:4b` are heavily optimized for coding tasks
|
||||
- They interpret ambiguous questions as "write me a program" instead of "answer my question"
|
||||
- The model's training data skews toward code generation at smaller parameter counts
|
||||
- Thinking/reasoning mode (if enabled) amplifies this tendency
|
||||
|
||||
#### Fix: Set a system prompt in Open WebUI
|
||||
|
||||
1. Go to **Admin** → **Settings** → **Interface** → **Default System Prompt** (for all chats)
|
||||
|
||||
Or per-model: **Workspace** → **Models** → select model → **System Prompt**
|
||||
|
||||
2. Use a system prompt like:
|
||||
|
||||
```
|
||||
You are a helpful assistant. Answer questions in clear, natural language.
|
||||
Explain concepts conversationally. Only include code if the user explicitly
|
||||
asks for code or a script. When discussing technical topics, use plain
|
||||
English explanations with examples, not programs.
|
||||
```
|
||||
|
||||
3. For the specific GPU pricing question in the screenshot, the model should have responded with a comparison table in text, not a Python script. A good system prompt prevents this.
|
||||
|
||||
#### Alternative: Use a chat-optimized model
|
||||
|
||||
Some models are better at conversational responses:
|
||||
|
||||
| Model | Size | Better for chat? |
|
||||
|-------|------|-----------------|
|
||||
| `qwen2.5:14b` | 14B | Yes — more balanced |
|
||||
| `qwen2.5:7b` | 7B | Yes — good general chat |
|
||||
| `llama3.1:8b` | 8B | Yes — conversational |
|
||||
| `qwen2.5-coder:7b` | 7B | No — code-focused |
|
||||
| `qwen3.5:4b` | 4B | No — too small, code-biased |
|
||||
|
||||
Avoid using `-coder` variants or very small models (≤4B) for general chat. Use them only when you actually want code.
|
||||
|
||||
### VRAM considerations
|
||||
|
||||
Image generation and LLM inference compete for GPU memory. With a single GPU:
|
||||
|
||||
| VRAM | Recommendation |
|
||||
|------|---------------|
|
||||
| ≥ 24GB | Run both LLM + image gen simultaneously |
|
||||
| 12–24GB | Use smaller LLM when generating images, or stop Ollama first |
|
||||
| < 12GB | Run one at a time — stop Ollama before generating images |
|
||||
|
||||
ComfyUI models typically need 4–8GB VRAM (SD 1.5: ~4GB, SDXL: ~7GB, Flux: ~12GB).
|
||||
|
||||
---
|
||||
|
||||
## InvokeAI vs ComfyUI — which to use when
|
||||
|
||||
Both are installed by the setup script. Here's when to use each:
|
||||
|
||||
| Task | InvokeAI (`:9090`) | ComfyUI (`:8188`) |
|
||||
|------|:---:|:---:|
|
||||
| Friendly UI, sliders, drag-and-drop | Yes | No (node editor) |
|
||||
| Use a LoRA you trained on RunPod | Yes — just import + select | Yes — but wire LoRA Loader node |
|
||||
| Same face, different settings (img2img) | Yes — Image to Image tab | Yes — IP-Adapter nodes |
|
||||
| Age/emotion/scene changes | Yes — img2img + prompt | Yes — IP-Adapter + prompt |
|
||||
| Chat-integrated image gen (from Open WebUI) | No (no OWUI API) | Yes (workflow export) |
|
||||
| Maximum flexibility / custom pipelines | No | Yes |
|
||||
| Learning curve | Low | High |
|
||||
|
||||
**Bottom line:** Use InvokeAI for the interactive "play with images" workflow. Use ComfyUI only when you need Open WebUI chat integration or advanced node pipelines.
|
||||
|
||||
## Using InvokeAI for image iteration
|
||||
|
||||
InvokeAI is the easier path for what you want — take a reference image and generate variations with different settings, ages, emotions, art styles.
|
||||
|
||||
### Step 1: Install a base model (required first)
|
||||
|
||||
Before LoRAs or img2img will work, InvokeAI needs a base Stable Diffusion model.
|
||||
|
||||
```bash
|
||||
# Option A: Download SD 1.5 (~4GB VRAM needed) via the container
|
||||
docker exec invokeai invokeai-model-install --add stabilityai/stable-diffusion-v1-5
|
||||
|
||||
# Option B: Download SDXL (~7GB VRAM needed)
|
||||
docker exec invokeai invokeai-model-install --add stabilityai/stable-diffusion-xl-base-1.0
|
||||
```
|
||||
|
||||
Or use the UI: open `http://<ip>:9090` → **Model Manager** (cube icon) → **Starter Models** tab → install SD 1.5 or SDXL.
|
||||
|
||||
> **6GB GPU note:** SD 1.5 fits comfortably. SDXL is tight — it may work with float16 (already set) but could be slow. Start with SD 1.5.
|
||||
|
||||
### Step 2: Import your RunPod LoRA
|
||||
|
||||
```bash
|
||||
# Copy the .safetensors file from wherever you downloaded it:
|
||||
./invokeai-import-lora.sh ~/Downloads/my-lora.safetensors "My Character"
|
||||
```
|
||||
|
||||
Then in the InvokeAI UI:
|
||||
1. **Model Manager** (cube icon) → **Scan for Models**
|
||||
2. Your LoRA should appear in the list
|
||||
|
||||
If it doesn't:
|
||||
- Click **Add Model** → **Scan Folder** → enter `/invokeai/models/lora`
|
||||
- Make sure the LoRA architecture matches your base model (SD 1.5 LoRA needs SD 1.5 base)
|
||||
|
||||
### Step 3: Generate with your LoRA (text-to-image)
|
||||
|
||||
1. Go to the **Text to Image** tab
|
||||
2. Select your **base model** (must match the LoRA's training base)
|
||||
3. In the left panel, expand the **LoRA** section (below model selector)
|
||||
4. Click **+** → select your LoRA → set weight to **0.7–0.85**
|
||||
5. Write a prompt: "portrait of [subject], smiling, studio lighting"
|
||||
6. Click **Invoke**
|
||||
|
||||
### Step 4: Iterate on an image (img2img)
|
||||
|
||||
This is where InvokeAI shines for your use case — take an image and riff on it:
|
||||
|
||||
1. Switch to the **Image to Image** tab
|
||||
2. Drag your reference photo onto the canvas (or click to upload)
|
||||
3. Keep your LoRA active (same as above)
|
||||
4. Set the **Denoising Strength** slider:
|
||||
|
||||
| Strength | Effect |
|
||||
|----------|--------|
|
||||
| 0.2–0.3 | Subtle tweaks — mostly keeps the original, minor style changes |
|
||||
| 0.4–0.5 | Moderate changes — recognizable but different mood/lighting |
|
||||
| 0.6–0.7 | Significant changes — same composition, new details/style |
|
||||
| 0.8–1.0 | Major rewrite — loosely inspired by original, mostly new |
|
||||
|
||||
5. Change the prompt to describe what you want different:
|
||||
- **Age up:** "same person, elderly, wrinkles, grey hair, wise expression"
|
||||
- **Age down:** "same person as a young child, bright eyes, playground"
|
||||
- **Emotion:** "same person, laughing joyfully" or "same person, crying, dramatic lighting"
|
||||
- **Setting:** "same person, sitting in a Parisian cafe, afternoon light"
|
||||
- **Art style:** "same person, oil painting, renaissance style, dramatic chiaroscuro"
|
||||
6. Click **Invoke** — iterate by adjusting strength and prompt
|
||||
|
||||
### Step 5: Use the Unified Canvas for inpainting
|
||||
|
||||
This is the "fix this specific thing" workflow — brush over a hand, arm, face, background, whatever, and regenerate just that area while keeping everything else untouched.
|
||||
|
||||
1. Switch to the **Unified Canvas** tab
|
||||
2. Upload or paste your image
|
||||
3. Select the **Mask** brush tool (not the paint brush)
|
||||
4. Brush over **only the area you want to change** — everything else stays locked
|
||||
5. Write a prompt describing what the masked area should become
|
||||
6. Set **Denoising Strength** to 0.6–0.8 (higher = more change)
|
||||
7. Click **Invoke** — only the masked pixels regenerate
|
||||
|
||||
**Common inpainting fixes:**
|
||||
|
||||
| Problem | Mask | Prompt |
|
||||
|---------|------|--------|
|
||||
| Hand in wrong position | Brush over the arm/hand | "natural hand resting at side, relaxed pose" |
|
||||
| Extra fingers | Brush over the hand | "normal human hand, five fingers, anatomically correct" |
|
||||
| Weird eyes | Brush over both eyes | "natural eyes, looking at camera, detailed iris" |
|
||||
| Bad background | Brush over background only | "clean studio backdrop" or "forest trail, golden hour" |
|
||||
| Wrong clothing | Brush over the clothing area | "wearing blue denim jacket, casual style" |
|
||||
| Face swap / aging | Brush over the face | "same person, elderly, wrinkles" or "same person as child" |
|
||||
|
||||
**Tips for better inpainting results:**
|
||||
- **Mask slightly larger** than the problem area — gives the model room to blend edges
|
||||
- **Use soft brush edges** (lower brush hardness) for more natural blending
|
||||
- If the result has visible seams, increase your mask area and try again
|
||||
- **Lower denoising (0.4–0.5)** for subtle fixes, **higher (0.7–0.9)** for major changes
|
||||
- Keep your LoRA active during inpainting — it maintains the trained style/face consistency
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- **Greyed-out upload/LoRA buttons:** Install a base model first (Step 1). InvokeAI disables most features until a checkpoint is loaded.
|
||||
- **Model not synced:** After copying files via script, click **Scan for Models** in Model Manager.
|
||||
- **Architecture mismatch:** A LoRA trained on SD 1.5 only works with SD 1.5 base models — not SDXL. Check what your RunPod training used.
|
||||
- **Out of VRAM:** Try SD 1.5 instead of SDXL, or reduce image size to 512x512.
|
||||
- **Use the import script:** The greyed-out UI upload can be bypassed entirely by using `invokeai-import-lora.sh` to copy files directly into the model volume.
|
||||
|
||||
---
|
||||
|
||||
## Updating
|
||||
|
||||
Re-run the setup script — it detects an existing install and skips prereqs:
|
||||
|
||||
```bash
|
||||
./laptop_full_setup.sh
|
||||
# or
|
||||
./laptop_full_setup.sh --force # also overwrites config files
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GPU / Model Tiers (`local-ai-setup.sh`)
|
||||
|
||||
| VRAM | Chat model | Code model | Context |
|
||||
|---------|-----------------|-----------------------|---------|
|
||||
| ≥ 14 GB | qwen2.5:14b | qwen2.5-coder:14b | 32k |
|
||||
| 8–14 GB | qwen2.5:14b | qwen2.5-coder:7b | 16k |
|
||||
| 4–8 GB | qwen2.5:7b | qwen2.5-coder:7b | 8k |
|
||||
| CPU | qwen2.5:7b | qwen2.5-coder:7b | 4k |
|
||||
|
||||
Embed model is always `nomic-embed-text` (required for RAG).
|
||||
|
||||
### VRAM reality check
|
||||
|
||||
Ollama will **always try to run** any model — it silently offloads layers to CPU when VRAM is insufficient. The model still works but gets significantly slower. The setup script's "fully in VRAM" label can be misleading.
|
||||
|
||||
**Actual VRAM needed for common models (Q4_K_M quantization):**
|
||||
|
||||
| Model | Download size | VRAM for inference | Fits in 6GB? | Fits in 8GB? |
|
||||
|-------|-------------|-------------------|-------------|-------------|
|
||||
| `qwen3.5:4b` | ~2.5 GB | ~3.5–4 GB | Yes | Yes |
|
||||
| `qwen2.5:7b` | ~4.4 GB | ~5.5–6 GB | Tight | Yes |
|
||||
| `qwen3.5:9b` | ~5.5 GB | ~6.5–7 GB | **No — partial CPU offload** | Tight |
|
||||
| `qwen2.5:14b` | ~8.7 GB | ~10–11 GB | No | No |
|
||||
| `qwen3.5-35b-a3b` (MoE) | ~20 GB | ~3.5 GB active | Yes (only 3B active) | Yes |
|
||||
|
||||
**Why the file size != VRAM needed:** Inference requires additional memory for KV cache, attention buffers, and CUDA overhead. Expect ~1-2 GB more than the model file size.
|
||||
|
||||
**Signs of CPU offload (model too big for your VRAM):**
|
||||
- Tokens per second drops from 20-40 to 2-8
|
||||
- `nvidia-smi` shows VRAM maxed out
|
||||
- CPU usage spikes during generation
|
||||
- First token takes much longer than usual
|
||||
|
||||
### Image Generation Model Tiers
|
||||
|
||||
The `setup-image-models.sh` script detects your GPU and offers appropriate models:
|
||||
|
||||
| VRAM | Available Models | Default | Notes |
|
||||
|------|-----------------|---------|-------|
|
||||
| ≥ 24GB | SD 1.5, SDXL, SDXL Turbo, Flux.1-schnell, Flux.1-dev | SDXL | All models, no constraints |
|
||||
| 12–23GB | SD 1.5, SDXL, SDXL Turbo, Flux.1-schnell | SDXL | Flux-dev too tight |
|
||||
| 8–11GB | SD 1.5, SDXL (tight), SDXL Turbo | SD 1.5 | SDXL works at 512px, may be slow |
|
||||
| 4–7GB | SD 1.5 (float16) | SD 1.5 | Only SD 1.5 fits |
|
||||
| < 4GB | none | — | CPU generation not recommended |
|
||||
|
||||
**GPU sharing:** Ollama and image generation share the GPU. Ollama auto-unloads models
|
||||
after its `KEEP_ALIVE` timeout (default 24h), so image gen gets full VRAM when the LLM
|
||||
is idle. For immediate unload: `docker exec ollama ollama stop <model-name>`
|
||||
|
||||
**Multi-GPU scaling:** With dual GPUs (e.g., 2× RTX 5000 = 32GB total), the VRAM
|
||||
is summed for tier selection. Both InvokeAI and ComfyUI will use all available GPUs.
|
||||
|
||||
```bash
|
||||
# Install image models (auto-detects GPU):
|
||||
./setup-image-models.sh
|
||||
|
||||
# Or auto-install the recommended default:
|
||||
./setup-image-models.sh --auto
|
||||
```
|
||||
Executable
+97
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env bash
|
||||
# Import a LoRA (.safetensors) into ComfyUI's Docker volume
|
||||
# Usage: ./comfyui-import-lora.sh /path/to/my-lora.safetensors
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo -e "${BOLD}Usage:${NC} $0 <lora-file.safetensors> [display-name]"
|
||||
echo ""
|
||||
echo " Copies a LoRA file into ComfyUI's models/loras/ directory so it"
|
||||
echo " can be used in workflows (including via Open WebUI)."
|
||||
echo ""
|
||||
echo " Examples:"
|
||||
echo " $0 ~/Downloads/my-style-lora.safetensors"
|
||||
echo " $0 ~/Downloads/my-style-lora.safetensors \"Anime Style\""
|
||||
echo ""
|
||||
echo " After importing, create a workflow in ComfyUI that uses this LoRA,"
|
||||
echo " export it as API format, and import into Open WebUI for chat-based"
|
||||
echo " image generation with this LoRA applied automatically."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
LORA_FILE="$1"
|
||||
DISPLAY_NAME="${2:-$(basename "$LORA_FILE" .safetensors)}"
|
||||
|
||||
# Validate file exists
|
||||
if [[ ! -f "$LORA_FILE" ]]; then
|
||||
echo -e "${RED}Error:${NC} File not found: $LORA_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate file extension
|
||||
if [[ "$LORA_FILE" != *.safetensors && "$LORA_FILE" != *.ckpt && "$LORA_FILE" != *.pt ]]; then
|
||||
echo -e "${YELLOW}Warning:${NC} File doesn't have a typical LoRA extension (.safetensors, .ckpt, .pt)"
|
||||
read -rp "Continue anyway? [y/N] " yn
|
||||
[[ "$yn" != [yY]* ]] && exit 1
|
||||
fi
|
||||
|
||||
# Check if ComfyUI container exists
|
||||
if ! docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q '^comfyui$'; then
|
||||
echo -e "${RED}Error:${NC} ComfyUI container not found. Is the AI stack running?"
|
||||
echo " Try: docker compose up -d comfyui"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if container is running
|
||||
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^comfyui$'; then
|
||||
echo -e "${YELLOW}ComfyUI container is stopped. Starting it...${NC}"
|
||||
docker start comfyui
|
||||
sleep 3
|
||||
fi
|
||||
|
||||
FILENAME=$(basename "$LORA_FILE")
|
||||
|
||||
echo -e "${CYAN}[..]${NC} Copying ${BOLD}$FILENAME${NC} into ComfyUI..."
|
||||
|
||||
# Copy the LoRA file into the container's loras directory
|
||||
docker exec comfyui mkdir -p /opt/ComfyUI/models/loras
|
||||
docker cp "$LORA_FILE" "comfyui:/opt/ComfyUI/models/loras/$FILENAME"
|
||||
|
||||
echo -e "${GREEN}[OK]${NC} LoRA '${DISPLAY_NAME}' copied to ComfyUI!"
|
||||
echo ""
|
||||
echo -e "${BOLD}Next: Create a workflow that uses this LoRA${NC}"
|
||||
echo ""
|
||||
echo " 1. Open ComfyUI at http://localhost:8188"
|
||||
echo " 2. Load or create a text-to-image workflow"
|
||||
echo " 3. Add a ${BOLD}LoRA Loader${NC} node:"
|
||||
echo " Right-click canvas → Add Node → loaders → Load LoRA"
|
||||
echo " 4. Wire it between the ${BOLD}checkpoint loader${NC} and the ${BOLD}CLIP/sampler${NC}:"
|
||||
echo ""
|
||||
echo " [Load Checkpoint] → MODEL → [Load LoRA] → MODEL → [KSampler]"
|
||||
echo " → CLIP → → CLIP → [CLIP Text Encode]"
|
||||
echo ""
|
||||
echo " 5. In the LoRA Loader node, select ${BOLD}${FILENAME}${NC}"
|
||||
echo " 6. Set ${BOLD}strength_model${NC} and ${BOLD}strength_clip${NC} (start with 0.7–0.85)"
|
||||
echo ""
|
||||
echo -e "${BOLD}To use from Open WebUI chat (no more ComfyUI interaction needed):${NC}"
|
||||
echo ""
|
||||
echo " 7. Click the ${BOLD}gear icon${NC} → enable ${BOLD}Dev Mode${NC}"
|
||||
echo " 8. Click ${BOLD}Save (API Format)${NC} → saves workflow_api.json"
|
||||
echo " 9. In Open WebUI → Admin → Settings → Images → ${BOLD}Import Workflow${NC}"
|
||||
echo " 10. Upload the workflow_api.json and map the prompt node"
|
||||
echo " 11. Now just chat: \"Generate an image of a forest in ${DISPLAY_NAME} style\""
|
||||
echo ""
|
||||
echo -e "${YELLOW}Tip:${NC} The LoRA must match your base model architecture."
|
||||
echo " SD 1.5 LoRA → SD 1.5 checkpoint. SDXL LoRA → SDXL checkpoint."
|
||||
echo ""
|
||||
echo -e "${YELLOW}Tip:${NC} You can ${BOLD}chain multiple LoRAs${NC} in one workflow:"
|
||||
echo " [Checkpoint] → [Load LoRA 1] → [Load LoRA 2] → [KSampler]"
|
||||
echo " Each has its own strength slider so you can blend styles."
|
||||
echo ""
|
||||
echo -e "${YELLOW}Tip:${NC} Export multiple workflows (one per LoRA combo) and switch"
|
||||
echo " between them in Open WebUI's image settings as needed."
|
||||
echo " LoRAs are baked into the workflow — there's no keyword to"
|
||||
echo " toggle them on/off from chat."
|
||||
Executable
+206
@@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install IP-Adapter custom nodes + models into ComfyUI's Docker container
|
||||
# Enables: same face/different settings, age up/down, emotion changes, style transfer
|
||||
#
|
||||
# Usage:
|
||||
# ./comfyui-install-ipadapter.sh — install for SDXL (recommended)
|
||||
# ./comfyui-install-ipadapter.sh --sd15 — install for SD 1.5
|
||||
# ./comfyui-install-ipadapter.sh --all — install both SDXL + SD 1.5
|
||||
# ./comfyui-install-ipadapter.sh --faceid — also install FaceID (better face lock)
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[..]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[!!]${NC} $*"; }
|
||||
die() { echo -e "${RED}[XX]${NC} $*" >&2; exit 1; }
|
||||
|
||||
# ── parse args ──────────────────────────────────────────────────────────────
|
||||
INSTALL_SD15=false
|
||||
INSTALL_SDXL=true
|
||||
INSTALL_FACEID=false
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--sd15) INSTALL_SD15=true; INSTALL_SDXL=false ;;
|
||||
--all) INSTALL_SD15=true; INSTALL_SDXL=true ;;
|
||||
--faceid) INSTALL_FACEID=true ;;
|
||||
--help|-h)
|
||||
echo -e "${BOLD}Usage:${NC} $0 [--sd15] [--all] [--faceid]"
|
||||
echo ""
|
||||
echo " Installs IP-Adapter custom nodes and models into ComfyUI."
|
||||
echo " Enables reference-image workflows: same face in different"
|
||||
echo " settings, age changes, emotions, style transfer."
|
||||
echo ""
|
||||
echo " Options:"
|
||||
echo " --sd15 Install SD 1.5 models (instead of SDXL)"
|
||||
echo " --all Install both SDXL + SD 1.5 models"
|
||||
echo " --faceid Also install FaceID models (better face lock,"
|
||||
echo " requires insightface — adds ~1GB)"
|
||||
echo ""
|
||||
echo " Default: SDXL models only (~2.5GB download)"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── check container ─────────────────────────────────────────────────────────
|
||||
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^comfyui$'; then
|
||||
if docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q '^comfyui$'; then
|
||||
warn "ComfyUI container is stopped. Starting it..."
|
||||
docker start comfyui
|
||||
sleep 5
|
||||
else
|
||||
die "ComfyUI container not found. Is the AI stack running?"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── helper: download into container ─────────────────────────────────────────
|
||||
# Usage: dl_model <url> <container_path>
|
||||
dl_model() {
|
||||
local url="$1" dest="$2"
|
||||
local filename; filename=$(basename "$dest")
|
||||
if docker exec comfyui test -f "$dest" 2>/dev/null; then
|
||||
ok "Already exists: $filename"
|
||||
return 0
|
||||
fi
|
||||
local dir; dir=$(dirname "$dest")
|
||||
docker exec comfyui mkdir -p "$dir"
|
||||
info "Downloading $filename..."
|
||||
if ! docker exec comfyui wget -q --show-progress -O "$dest" "$url" 2>&1; then
|
||||
# wget --show-progress not always available
|
||||
docker exec comfyui wget -q -O "$dest" "$url"
|
||||
fi
|
||||
ok "Downloaded $filename"
|
||||
}
|
||||
|
||||
HF_IPA="https://huggingface.co/h94/IP-Adapter/resolve/main"
|
||||
MODELS="/opt/ComfyUI/models"
|
||||
|
||||
# ── Step 1: Install custom nodes ────────────────────────────────────────────
|
||||
echo ""
|
||||
echo -e "${BOLD}━━━ IP-Adapter Installation for ComfyUI ━━━${NC}"
|
||||
echo ""
|
||||
|
||||
info "Installing ComfyUI_IPAdapter_plus custom nodes..."
|
||||
if docker exec comfyui test -d /opt/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus 2>/dev/null; then
|
||||
info "Custom nodes already installed — pulling updates..."
|
||||
docker exec comfyui bash -c "cd /opt/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus && git pull -q"
|
||||
ok "Custom nodes updated"
|
||||
else
|
||||
docker exec comfyui bash -c "cd /opt/ComfyUI/custom_nodes && git clone --depth 1 https://github.com/cubiq/ComfyUI_IPAdapter_plus.git"
|
||||
ok "Custom nodes installed"
|
||||
fi
|
||||
|
||||
# Install Python dependencies if requirements.txt exists
|
||||
if docker exec comfyui test -f /opt/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/requirements.txt 2>/dev/null; then
|
||||
info "Installing Python dependencies..."
|
||||
docker exec comfyui pip install -q -r /opt/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/requirements.txt 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ── Step 2: CLIP Vision models (required by all variants) ───────────────────
|
||||
echo ""
|
||||
info "Downloading CLIP Vision encoders..."
|
||||
|
||||
dl_model \
|
||||
"https://huggingface.co/h94/IP-Adapter/resolve/main/models/image_encoder/model.safetensors" \
|
||||
"$MODELS/clip_vision/CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors"
|
||||
|
||||
if $INSTALL_SDXL; then
|
||||
dl_model \
|
||||
"https://huggingface.co/h94/IP-Adapter/resolve/main/sdxl_models/image_encoder/model.safetensors" \
|
||||
"$MODELS/clip_vision/CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors"
|
||||
fi
|
||||
|
||||
# ── Step 3: IP-Adapter models ──────────────────────────────────────────────
|
||||
echo ""
|
||||
docker exec comfyui mkdir -p "$MODELS/ipadapter"
|
||||
|
||||
if $INSTALL_SDXL; then
|
||||
info "Downloading SDXL IP-Adapter models..."
|
||||
dl_model "$HF_IPA/sdxl_models/ip-adapter-plus_sdxl_vit-h.safetensors" \
|
||||
"$MODELS/ipadapter/ip-adapter-plus_sdxl_vit-h.safetensors"
|
||||
dl_model "$HF_IPA/sdxl_models/ip-adapter-plus-face_sdxl_vit-h.safetensors" \
|
||||
"$MODELS/ipadapter/ip-adapter-plus-face_sdxl_vit-h.safetensors"
|
||||
fi
|
||||
|
||||
if $INSTALL_SD15; then
|
||||
info "Downloading SD 1.5 IP-Adapter models..."
|
||||
dl_model "$HF_IPA/models/ip-adapter-plus_sd15.safetensors" \
|
||||
"$MODELS/ipadapter/ip-adapter-plus_sd15.safetensors"
|
||||
dl_model "$HF_IPA/models/ip-adapter-plus-face_sd15.safetensors" \
|
||||
"$MODELS/ipadapter/ip-adapter-plus-face_sd15.safetensors"
|
||||
fi
|
||||
|
||||
# ── Step 4 (optional): FaceID models ───────────────────────────────────────
|
||||
if $INSTALL_FACEID; then
|
||||
echo ""
|
||||
info "Installing FaceID dependencies (insightface, onnxruntime)..."
|
||||
docker exec comfyui pip install -q insightface onnxruntime 2>/dev/null \
|
||||
|| warn "Could not install insightface — FaceID nodes may not work"
|
||||
|
||||
HF_FACEID="https://huggingface.co/h94/IP-Adapter-FaceID/resolve/main"
|
||||
info "Downloading FaceID models..."
|
||||
|
||||
if $INSTALL_SDXL; then
|
||||
dl_model "$HF_FACEID/ip-adapter-faceid-plusv2_sdxl.bin" \
|
||||
"$MODELS/ipadapter/ip-adapter-faceid-plusv2_sdxl.bin"
|
||||
fi
|
||||
if $INSTALL_SD15; then
|
||||
dl_model "$HF_FACEID/ip-adapter-faceid-plusv2_sd15.bin" \
|
||||
"$MODELS/ipadapter/ip-adapter-faceid-plusv2_sd15.bin"
|
||||
fi
|
||||
|
||||
# FaceID LoRAs (required for FaceID models)
|
||||
docker exec comfyui mkdir -p "$MODELS/loras"
|
||||
if $INSTALL_SDXL; then
|
||||
dl_model "$HF_FACEID/ip-adapter-faceid-plusv2_sdxl_lora.safetensors" \
|
||||
"$MODELS/loras/ip-adapter-faceid-plusv2_sdxl_lora.safetensors"
|
||||
fi
|
||||
if $INSTALL_SD15; then
|
||||
dl_model "$HF_FACEID/ip-adapter-faceid-plusv2_sd15_lora.safetensors" \
|
||||
"$MODELS/loras/ip-adapter-faceid-plusv2_sd15_lora.safetensors"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Done ────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo -e "${GREEN}━━━ Installation complete! ━━━${NC}"
|
||||
echo ""
|
||||
echo -e "${BOLD}Restart ComfyUI to load new nodes:${NC}"
|
||||
echo " docker restart comfyui"
|
||||
echo ""
|
||||
echo -e "${BOLD}Then open ComfyUI at http://localhost:8188 and try it:${NC}"
|
||||
echo ""
|
||||
echo " Basic reference image (style/scene transfer):"
|
||||
echo " ┌──────────────────────────────────────────────────────────────────┐"
|
||||
echo " │ [Load Checkpoint]──→[Load Image]──→[IPAdapter Unified Loader] │"
|
||||
echo " │ │ │ │"
|
||||
echo " │ ├── MODEL ──────────────→ [IPAdapter Apply] → [KSampler]│"
|
||||
echo " │ └── CLIP → [CLIP Text Encode]──→ positive ──→ │"
|
||||
echo " │ \"a castle at sunset\" │"
|
||||
echo " └──────────────────────────────────────────────────────────────────┘"
|
||||
echo ""
|
||||
echo " 1. Add nodes: right-click → Add Node → ipadapter"
|
||||
echo " 2. ${BOLD}IPAdapter Unified Loader${NC}: set preset to '${BOLD}PLUS FACE (portrait)${NC}'"
|
||||
echo " 3. ${BOLD}Load Image${NC}: upload your reference photo"
|
||||
echo " 4. ${BOLD}IPAdapter Apply${NC}: connect model + image, set weight 0.7–1.0"
|
||||
echo " 5. Text prompt controls the new scene: \"elderly, sitting in cafe, smiling\""
|
||||
echo ""
|
||||
echo -e "${BOLD}What each preset does:${NC}"
|
||||
echo " PLUS — general style/scene transfer"
|
||||
echo " PLUS FACE — preserves face likeness (best for your use case)"
|
||||
if $INSTALL_FACEID; then
|
||||
echo " FACEID PLUSV2 — strongest face lock (uses insightface for detection)"
|
||||
fi
|
||||
echo ""
|
||||
echo -e "${BOLD}Example prompts with a reference face:${NC}"
|
||||
echo " • \"same person, elderly, wise expression, studio lighting\""
|
||||
echo " • \"same person as a child, playing in a park, happy\""
|
||||
echo " • \"same person, crying, dramatic lighting, black and white\""
|
||||
echo " • \"same person, oil painting style, renaissance setting\""
|
||||
echo ""
|
||||
echo -e "${YELLOW}Note:${NC} This works directly in ComfyUI. Open WebUI's ComfyUI integration"
|
||||
echo " only sends text prompts — it can't attach a reference image."
|
||||
echo " For chat-based image gen (without reference images), use LoRAs instead."
|
||||
Executable
+234
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env bash
|
||||
# configure-searxng-safesearch.sh
|
||||
# Set SearXNG safe-search level; optionally disable categories or engines.
|
||||
#
|
||||
# Usage:
|
||||
# ./configure-searxng-safesearch.sh [strict|moderate|none] [OPTIONS]
|
||||
#
|
||||
# Options:
|
||||
# --disable-categories cat1,cat2 videos images news science social
|
||||
# --disable-engines eng1,eng2 e.g. duckduckgo,bing,yandex
|
||||
# --enable-engines eng1,eng2 re-enable engines auto-disabled by level
|
||||
#
|
||||
# Examples:
|
||||
# ./configure-searxng-safesearch.sh strict
|
||||
# ./configure-searxng-safesearch.sh moderate --disable-categories videos,images
|
||||
# ./configure-searxng-safesearch.sh none --disable-engines bing,duckduckgo
|
||||
# ./configure-searxng-safesearch.sh strict --disable-categories videos \
|
||||
# --disable-engines bing --enable-engines yandex
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
red() { printf '\e[31m%s\e[0m\n' "$*"; }
|
||||
grn() { printf '\e[32m%s\e[0m\n' "$*"; }
|
||||
blu() { printf '\e[34m%s\e[0m\n' "$*"; }
|
||||
yel() { printf '\e[33m%s\e[0m\n' "$*"; }
|
||||
die() { red "ERROR: $*"; exit 1; }
|
||||
ok() { grn " ✓ $*"; }
|
||||
info() { blu " → $*"; }
|
||||
warn() { yel " ! $*"; }
|
||||
|
||||
# ── Defaults ──────────────────────────────────────────────────────────────────
|
||||
LEVEL="moderate"
|
||||
DISABLE_CATS=""
|
||||
DISABLE_ENGINES_EXTRA=""
|
||||
ENABLE_ENGINES_EXTRA=""
|
||||
BASE="${BASE:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}"
|
||||
|
||||
# ── Parse arguments ───────────────────────────────────────────────────────────
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
strict|moderate|none) LEVEL="$1"; shift ;;
|
||||
--disable-categories) DISABLE_CATS="$2"; shift 2 ;;
|
||||
--disable-engines) DISABLE_ENGINES_EXTRA="$2"; shift 2 ;;
|
||||
--enable-engines) ENABLE_ENGINES_EXTRA="$2"; shift 2 ;;
|
||||
--base) BASE="$2"; shift 2 ;;
|
||||
-h|--help)
|
||||
sed -n '2,20p' "$0" | sed 's/^# \?//'
|
||||
exit 0
|
||||
;;
|
||||
*) die "Unknown argument: $1 (run with --help)" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SETTINGS="$BASE/searxng/settings.yml"
|
||||
COMPOSE="$BASE/docker-compose.yml"
|
||||
|
||||
# Ensure settings directory exists (fix ownership if Docker created it as root)
|
||||
mkdir -p "$(dirname "$SETTINGS")" 2>/dev/null || \
|
||||
sudo mkdir -p "$(dirname "$SETTINGS")"
|
||||
if [[ -e "$SETTINGS" ]] && [[ ! -w "$SETTINGS" ]]; then
|
||||
warn "settings.yml not writable — fixing ownership"
|
||||
sudo chown "$(id -u):$(id -g)" "$SETTINGS" || \
|
||||
die "Cannot write to $SETTINGS — run: sudo chown $USER $SETTINGS"
|
||||
fi
|
||||
|
||||
# ── Level → integer ───────────────────────────────────────────────────────────
|
||||
case "$LEVEL" in
|
||||
none) SAFE_INT=0 ;;
|
||||
moderate) SAFE_INT=1 ;;
|
||||
strict) SAFE_INT=2 ;;
|
||||
esac
|
||||
|
||||
info "Safe search: $LEVEL (${SAFE_INT})"
|
||||
|
||||
# ── Engines with no safe-search support ───────────────────────────────────────
|
||||
# Auto-disabled when level is moderate or strict.
|
||||
NO_SAFESEARCH_ENGINES=(
|
||||
# Torrent / P2P — no filtering possible
|
||||
"1337x" "piratebay" "nyaa" "torrentz" "kickass torrents"
|
||||
# Web engines without safe-search API
|
||||
"mojeek" "naver" "baidu"
|
||||
# Yandex: parameter exists but not reliably enforced for non-Russian queries
|
||||
"yandex" "yandex images"
|
||||
# Video frontends — no safe-search passthrough
|
||||
"invidious" "piped" "peertube" "sepiasearch"
|
||||
)
|
||||
|
||||
# ── Category → engine lists ───────────────────────────────────────────────────
|
||||
VIDEOS_ENGINES=(
|
||||
"youtube" "bing videos" "brave videos" "duckduckgo videos" "google videos" "qwant videos"
|
||||
"dailymotion" "media.ccc.de" "wikcommons.videos"
|
||||
"vimeo" "odysee" "rumble" "bitchute"
|
||||
"invidious" "piped" "peertube" "sepiasearch"
|
||||
)
|
||||
IMAGES_ENGINES=(
|
||||
"google images" "bing images" "duckduckgo images" "brave images" "qwant images"
|
||||
"startpage images" "mojeek images" "presearch images"
|
||||
"openverse" "unsplash" "pexels" "pixabay images" "pinterest" "flickr"
|
||||
"wikcommons.images" "artic" "yandex images"
|
||||
"imgur" "deviantart" "artstation" "adobe stock"
|
||||
)
|
||||
NEWS_ENGINES=(
|
||||
"google news" "bing news" "duckduckgo news" "brave news" "qwant news"
|
||||
"startpage news" "presearch news" "mojeek news"
|
||||
"reuters" "yahoo news" "wikinews" "yep news"
|
||||
)
|
||||
SCIENCE_ENGINES=(
|
||||
"arxiv" "semantic scholar" "pubmed" "crossref" "base"
|
||||
)
|
||||
SOCIAL_ENGINES=(
|
||||
"reddit" "lemmy" "mastodon"
|
||||
)
|
||||
|
||||
# ── Build disable/enable maps ─────────────────────────────────────────────────
|
||||
declare -A DISABLE_MAP # engine → 1
|
||||
declare -A ENABLE_MAP # engine → 1 (overrides everything)
|
||||
|
||||
# Parse --enable-engines
|
||||
if [[ -n "$ENABLE_ENGINES_EXTRA" ]]; then
|
||||
IFS=',' read -ra _engs <<< "$ENABLE_ENGINES_EXTRA"
|
||||
for e in "${_engs[@]}"; do
|
||||
e="${e#"${e%%[![:space:]]*}"}"; e="${e%"${e##*[![:space:]]}"}" # trim
|
||||
[[ -n "$e" ]] && ENABLE_MAP["$e"]=1
|
||||
done
|
||||
fi
|
||||
|
||||
# Helper: add to DISABLE_MAP unless explicitly re-enabled
|
||||
mark_disabled() {
|
||||
local eng="$1"
|
||||
# Use set +u to safely check array key existence on bash < 4.4 (set -u quirk)
|
||||
set +u; local _chk="${ENABLE_MAP[$eng]+x}"; set -u
|
||||
[[ -n "$_chk" ]] && return # user said keep it
|
||||
DISABLE_MAP["$eng"]=1
|
||||
}
|
||||
|
||||
# Auto-disable no-safesearch engines for moderate/strict
|
||||
if [[ "$LEVEL" != "none" ]]; then
|
||||
for eng in "${NO_SAFESEARCH_ENGINES[@]}"; do
|
||||
mark_disabled "$eng"
|
||||
done
|
||||
fi
|
||||
|
||||
# Category disables
|
||||
if [[ -n "$DISABLE_CATS" ]]; then
|
||||
IFS=',' read -ra _cats <<< "$DISABLE_CATS"
|
||||
for cat in "${_cats[@]}"; do
|
||||
cat="${cat#"${cat%%[![:space:]]*}"}"; cat="${cat%"${cat##*[![:space:]]}"}"
|
||||
cat="${cat,,}"
|
||||
case "$cat" in
|
||||
videos) for e in "${VIDEOS_ENGINES[@]}"; do mark_disabled "$e"; done ;;
|
||||
images) for e in "${IMAGES_ENGINES[@]}"; do mark_disabled "$e"; done ;;
|
||||
news) for e in "${NEWS_ENGINES[@]}"; do mark_disabled "$e"; done ;;
|
||||
science) for e in "${SCIENCE_ENGINES[@]}"; do mark_disabled "$e"; done ;;
|
||||
social) for e in "${SOCIAL_ENGINES[@]}"; do mark_disabled "$e"; done ;;
|
||||
"") ;;
|
||||
*) warn "Unknown category '$cat' — valid: videos images news science social" ;;
|
||||
esac
|
||||
done
|
||||
fi
|
||||
|
||||
# Extra engine disables
|
||||
if [[ -n "$DISABLE_ENGINES_EXTRA" ]]; then
|
||||
IFS=',' read -ra _engs <<< "$DISABLE_ENGINES_EXTRA"
|
||||
for e in "${_engs[@]}"; do
|
||||
e="${e#"${e%%[![:space:]]*}"}"; e="${e%"${e##*[![:space:]]}"}"
|
||||
[[ -n "$e" ]] && mark_disabled "$e"
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Preserve existing secret key ─────────────────────────────────────────────
|
||||
SECRET_KEY=$(grep -oP '(?<=secret_key: ")[^"]+' "$SETTINGS" 2>/dev/null || true)
|
||||
[[ -z "$SECRET_KEY" ]] && SECRET_KEY=$(openssl rand -hex 32)
|
||||
|
||||
# ── Build engine override block ───────────────────────────────────────────────
|
||||
ENGINE_BLOCK=""
|
||||
# set +u: iterating empty associative arrays throws "unbound variable" on bash <4.4
|
||||
set +u
|
||||
for eng in "${!DISABLE_MAP[@]}"; do
|
||||
ENGINE_BLOCK+=" - name: ${eng}\n disabled: true\n"
|
||||
done
|
||||
for eng in "${!ENABLE_MAP[@]}"; do
|
||||
ENGINE_BLOCK+=" - name: ${eng}\n disabled: false\n"
|
||||
done
|
||||
set -u
|
||||
|
||||
# ── Write settings.yml ────────────────────────────────────────────────────────
|
||||
{
|
||||
printf 'use_default_settings: true\n'
|
||||
printf 'general:\n instance_name: "Local Search"\n'
|
||||
printf 'server:\n secret_key: "%s"\n limiter: false\n' "$SECRET_KEY"
|
||||
printf 'search:\n safe_search: %d\n default_lang: "en"\n formats: [html, json]\n' "$SAFE_INT"
|
||||
# Lock the safe-search preference so users cannot override it via the UI
|
||||
if [[ "$LEVEL" != "none" ]]; then
|
||||
printf 'preferences:\n lock:\n - safesearch\n'
|
||||
fi
|
||||
if [[ -n "$ENGINE_BLOCK" ]]; then
|
||||
printf 'engines:\n'
|
||||
printf '%b' "$ENGINE_BLOCK"
|
||||
fi
|
||||
} > "$SETTINGS"
|
||||
|
||||
ok "Updated settings.yml (safe_search: $SAFE_INT)"
|
||||
# bash < 4.4: ${#assoc[@]} on an empty declared array throws "unbound variable"
|
||||
# under set -u — disable nounset for the rest of the reporting section
|
||||
set +u
|
||||
if [[ ${#DISABLE_MAP[@]} -gt 0 ]]; then
|
||||
info "Disabled (${#DISABLE_MAP[@]}): $(printf '%s, ' "${!DISABLE_MAP[@]}" | sed 's/, $//')"
|
||||
fi
|
||||
if [[ ${#ENABLE_MAP[@]} -gt 0 ]]; then
|
||||
info "Re-enabled: $(printf '%s, ' "${!ENABLE_MAP[@]}" | sed 's/, $//')"
|
||||
fi
|
||||
|
||||
# ── Update &safesearch= in SEARXNG_QUERY_URL inside docker-compose.yml ────────
|
||||
if [[ -f "$COMPOSE" ]]; then
|
||||
sed -i -E \
|
||||
"s|(SEARXNG_QUERY_URL=http://searxng:[0-9]+/search\?[^&[:space:]]*)(&safesearch=[0-9])?|\1\&safesearch=${SAFE_INT}|g" \
|
||||
"$COMPOSE"
|
||||
ok "Updated SEARXNG_QUERY_URL (&safesearch=${SAFE_INT})"
|
||||
fi
|
||||
|
||||
# ── Restart SearXNG ───────────────────────────────────────────────────────────
|
||||
if docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^searxng$'; then
|
||||
info "Restarting SearXNG..."
|
||||
docker restart searxng
|
||||
ok "SearXNG restarted"
|
||||
else
|
||||
info "SearXNG not running — changes take effect on next start"
|
||||
fi
|
||||
|
||||
echo
|
||||
grn "Done — safe search: $LEVEL"
|
||||
[[ "$LEVEL" != "none" ]] && \
|
||||
info "Engines skipped (can't enforce '$LEVEL'): ${#DISABLE_MAP[@]} total"
|
||||
Executable
+239
@@ -0,0 +1,239 @@
|
||||
#!/usr/bin/env bash
|
||||
# configure-storage.sh — interactively assign drives for ZIM and model storage
|
||||
# then patches docker-compose.yml in place
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[..]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[!!]${NC} $*"; }
|
||||
die() { echo -e "${RED}[ERR]${NC} $*"; exit 1; }
|
||||
|
||||
BASE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
COMPOSE="$BASE/docker-compose.yml"
|
||||
|
||||
[[ -f "$COMPOSE" ]] || die "docker-compose.yml not found at $BASE — run local-ai-setup.sh first"
|
||||
|
||||
# ── collect drives ─────────────────────────────────────────────────────────────
|
||||
echo -e "\n${BOLD}━━━ Detected Drives ━━━${NC}\n"
|
||||
|
||||
# build array of mountpoints excluding tiny/system ones
|
||||
mapfile -t MOUNTS < <(
|
||||
df -h --output=target,size,avail,pcent 2>/dev/null \
|
||||
| tail -n +2 \
|
||||
| grep -v -E '^\s*(/$|/boot|/sys|/proc|/dev|/run|/snap|/var/lib/docker|/etc|tmpfs)' \
|
||||
| awk '$3 ~ /[0-9]/ {print}' \
|
||||
| sort -u
|
||||
)
|
||||
|
||||
if [[ ${#MOUNTS[@]} -eq 0 ]]; then
|
||||
warn "No additional mounted drives found beyond system disk."
|
||||
warn "Mount your drives first (e.g. /mnt/storage), then re-run this script."
|
||||
echo ""
|
||||
echo "Quick mount example:"
|
||||
echo " sudo mkdir -p /mnt/storage"
|
||||
echo " sudo mount /dev/sdb1 /mnt/storage"
|
||||
echo " # To make permanent, add to /etc/fstab"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# display numbered list
|
||||
echo -e " ${BOLD}# Mount Point Size Free Used${NC}"
|
||||
echo " ─────────────────────────────────────────────────────"
|
||||
IDX=0
|
||||
declare -a MOUNT_PATHS
|
||||
for line in "${MOUNTS[@]}"; do
|
||||
mp=$(echo "$line" | awk '{print $1}')
|
||||
sz=$(echo "$line" | awk '{print $2}')
|
||||
av=$(echo "$line" | awk '{print $3}')
|
||||
pc=$(echo "$line" | awk '{print $4}')
|
||||
printf " ${CYAN}%-3d${NC} %-30s %-7s %-7s %s\n" "$IDX" "$mp" "$sz" "$av" "$pc"
|
||||
MOUNT_PATHS[$IDX]="$mp"
|
||||
(( IDX++ ))
|
||||
done
|
||||
echo ""
|
||||
|
||||
pick_drive() {
|
||||
local purpose="$1"
|
||||
local varname="$2"
|
||||
local skip_ok="${3:-true}"
|
||||
|
||||
echo -e "${BOLD}$purpose storage${NC}"
|
||||
if $skip_ok; then
|
||||
read -rp " Enter drive number (or Enter to skip): " choice
|
||||
else
|
||||
read -rp " Enter drive number: " choice
|
||||
fi
|
||||
|
||||
if [[ -z "$choice" ]]; then
|
||||
eval "$varname=''"
|
||||
info "Skipping $purpose storage."
|
||||
elif [[ "$choice" =~ ^[0-9]+$ ]] && [[ "$choice" -lt "$IDX" ]]; then
|
||||
eval "$varname='${MOUNT_PATHS[$choice]}'"
|
||||
ok "Selected: ${MOUNT_PATHS[$choice]}"
|
||||
else
|
||||
warn "Invalid choice — skipping $purpose."
|
||||
eval "$varname=''"
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ── ZIM storage ────────────────────────────────────────────────────────────────
|
||||
echo -e "${BOLD}━━━ Kiwix ZIM Storage ━━━${NC}\n"
|
||||
echo " ZIM files can be large (Wikipedia alone is ~100GB)."
|
||||
echo " Pick a drive with plenty of space."
|
||||
echo ""
|
||||
pick_drive "ZIM" ZIM_DRIVE
|
||||
|
||||
ZIM_PATH=""
|
||||
if [[ -n "$ZIM_DRIVE" ]]; then
|
||||
read -rp " Folder name on that drive [zims]: " ZIM_FOLDER
|
||||
ZIM_FOLDER="${ZIM_FOLDER:-zims}"
|
||||
ZIM_PATH="$ZIM_DRIVE/$ZIM_FOLDER"
|
||||
mkdir -p "$ZIM_PATH"
|
||||
ok "ZIM path: $ZIM_PATH"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ── Model storage ──────────────────────────────────────────────────────────────
|
||||
echo -e "${BOLD}━━━ Ollama Model Storage ━━━${NC}\n"
|
||||
echo " Models range from 4GB (7B) to 30GB+ (70B)."
|
||||
echo " Pick a fast drive (SSD preferred) with 50–200GB free."
|
||||
echo ""
|
||||
pick_drive "Model" MODEL_DRIVE
|
||||
|
||||
MODEL_PATH=""
|
||||
if [[ -n "$MODEL_DRIVE" ]]; then
|
||||
read -rp " Folder name on that drive [ollama-models]: " MODEL_FOLDER
|
||||
MODEL_FOLDER="${MODEL_FOLDER:-ollama-models}"
|
||||
MODEL_PATH="$MODEL_DRIVE/$MODEL_FOLDER"
|
||||
mkdir -p "$MODEL_PATH"
|
||||
ok "Model path: $MODEL_PATH"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ── bail if nothing selected ───────────────────────────────────────────────────
|
||||
if [[ -z "$ZIM_PATH" && -z "$MODEL_PATH" ]]; then
|
||||
info "Nothing selected — no changes made."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── summary + confirm ──────────────────────────────────────────────────────────
|
||||
echo -e "${BOLD}━━━ Planned Changes ━━━${NC}\n"
|
||||
[[ -n "$ZIM_PATH" ]] && echo " Kiwix ZIMs → $ZIM_PATH"
|
||||
[[ -n "$MODEL_PATH" ]] && echo " Ollama models → $MODEL_PATH"
|
||||
echo ""
|
||||
read -rp "Apply these changes to docker-compose.yml? [Y/n]: " CONFIRM
|
||||
[[ "${CONFIRM,,}" == "n" ]] && { info "Aborted — no changes made."; exit 0; }
|
||||
|
||||
# ── backup ────────────────────────────────────────────────────────────────────
|
||||
cp "$COMPOSE" "$COMPOSE.bak"
|
||||
info "Backed up to docker-compose.yml.bak"
|
||||
|
||||
# ── patch kiwix volumes ────────────────────────────────────────────────────────
|
||||
if [[ -n "$ZIM_PATH" ]]; then
|
||||
# Replace the kiwix volumes line to add extra ZIM mounts
|
||||
# Current: volumes: [$BASE/kiwix:/data]
|
||||
# New: volumes:
|
||||
# - $BASE/kiwix:/data
|
||||
# - /mnt/drive/zims:/data/external
|
||||
|
||||
python3 - "$COMPOSE" "$BASE" "$ZIM_PATH" << 'PY'
|
||||
import sys, re
|
||||
|
||||
compose_file = sys.argv[1]
|
||||
base = sys.argv[2]
|
||||
zim_path = sys.argv[3]
|
||||
|
||||
text = open(compose_file).read()
|
||||
|
||||
# Find the kiwix service volumes line (single-line array style)
|
||||
old = f" volumes: [{base}/kiwix:/data]"
|
||||
new = (
|
||||
f" volumes:\n"
|
||||
f" - {base}/kiwix:/data\n"
|
||||
f" - {zim_path}:/data/external"
|
||||
)
|
||||
|
||||
if old in text:
|
||||
text = text.replace(old, new)
|
||||
open(compose_file, "w").write(text)
|
||||
print(f"[OK] Kiwix volumes updated")
|
||||
else:
|
||||
# Already multi-line — append the new mount if not already there
|
||||
marker = "container_name: kiwix"
|
||||
if marker in text and zim_path not in text:
|
||||
# find the volumes block under kiwix and append
|
||||
lines = text.splitlines()
|
||||
in_kiwix = False
|
||||
in_vols = False
|
||||
insert_after = -1
|
||||
for i, line in enumerate(lines):
|
||||
if marker in line:
|
||||
in_kiwix = True
|
||||
if in_kiwix and "volumes:" in line:
|
||||
in_vols = True
|
||||
if in_vols and line.strip().startswith("- ") and ":/data" in line:
|
||||
insert_after = i
|
||||
if in_vols and insert_after > 0 and not line.strip().startswith("- "):
|
||||
break
|
||||
if insert_after > 0:
|
||||
lines.insert(insert_after + 1, f" - {zim_path}:/data/external")
|
||||
open(compose_file, "w").write("\n".join(lines) + "\n")
|
||||
print(f"[OK] Kiwix extra ZIM volume appended")
|
||||
else:
|
||||
print(f"[!!] Could not patch kiwix volumes — edit manually: add - {zim_path}:/data/external")
|
||||
else:
|
||||
print(f"[!!] Kiwix volumes line not found in expected format — edit manually")
|
||||
PY
|
||||
fi
|
||||
|
||||
# ── patch ollama volumes ───────────────────────────────────────────────────────
|
||||
if [[ -n "$MODEL_PATH" ]]; then
|
||||
python3 - "$COMPOSE" "$MODEL_PATH" << 'PY'
|
||||
import sys
|
||||
|
||||
compose_file = sys.argv[1]
|
||||
model_path = sys.argv[2]
|
||||
|
||||
text = open(compose_file).read()
|
||||
|
||||
# Replace Docker-managed volume with host path
|
||||
old = " volumes: [ollama-models:/root/.ollama]"
|
||||
new = f" volumes: [{model_path}:/root/.ollama]"
|
||||
|
||||
if old in text:
|
||||
text = text.replace(old, new)
|
||||
# Remove ollama-models from the top-level volumes section if present
|
||||
text = text.replace(" ollama-models:\n", "")
|
||||
open(compose_file, "w").write(text)
|
||||
print(f"[OK] Ollama volume → {model_path}")
|
||||
else:
|
||||
print(f"[!!] ollama volumes line not in expected format — edit manually:")
|
||||
print(f" change ollama-models:/root/.ollama to {model_path}:/root/.ollama")
|
||||
PY
|
||||
fi
|
||||
|
||||
# ── restart if running ─────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
if docker compose -f "$COMPOSE" ps --quiet 2>/dev/null | grep -q .; then
|
||||
read -rp "Stack is running — restart now to apply changes? [Y/n]: " RESTART
|
||||
if [[ "${RESTART,,}" != "n" ]]; then
|
||||
cd "$BASE"
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
ok "Stack restarted"
|
||||
else
|
||||
warn "Remember to restart: cd $BASE && docker compose down && docker compose up -d"
|
||||
fi
|
||||
else
|
||||
info "Stack not running — changes will apply on next start."
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}${BOLD}Done!${NC}"
|
||||
[[ -n "$ZIM_PATH" ]] && echo " Drop .zim files into: $ZIM_PATH"
|
||||
[[ -n "$MODEL_PATH" ]] && echo " Ollama models stored in: $MODEL_PATH"
|
||||
echo ""
|
||||
echo " Undo: cp $COMPOSE.bak $COMPOSE"
|
||||
@@ -0,0 +1,868 @@
|
||||
# GPU Setup Research: Rack Server AI Workloads
|
||||
|
||||
*Last updated: March 22, 2026*
|
||||
|
||||
## Goal
|
||||
Cost-efficient rack-mountable GPU setup for:
|
||||
1. **LLM coding inference** — Run 32B+ parameter coding models with maximum context windows
|
||||
2. **Image generation** — ComfyUI / InvokeAI with Stable Diffusion SDXL / Flux
|
||||
|
||||
Target servers: Dell R720/R730 or HP DL380 equivalent (2U rack)
|
||||
|
||||
## Why 48GB VRAM is the Right Target
|
||||
|
||||
### The Problem with 24GB
|
||||
32B coding models at Q4_K_M quantization use ~20GB of weights, leaving only ~4GB for KV cache
|
||||
on a 24GB card. This severely limits context window size — the key ingredient for complex
|
||||
coding sessions where the model needs to understand your entire codebase.
|
||||
|
||||
### What 48GB Unlocks
|
||||
- **32B models at higher quantization** (Q6_K/Q8_0) = better output quality
|
||||
- **28GB+ free for KV cache** = massive context windows (32K+ tokens)
|
||||
- **70B models** in aggressive quantization (~12 t/s but functional)
|
||||
- **Simultaneous model loading** — coding model + image gen model at once
|
||||
- Room for future larger models without hardware changes
|
||||
|
||||
## Best Local Coding Models (2026)
|
||||
|
||||
### Qwen 3.5 Family (February 2026 — Gated Delta Networks)
|
||||
|
||||
Architecture breakthrough: 3 of every 4 layers use **linear attention** (O(n) scaling),
|
||||
drastically reducing KV cache memory. These models need far less VRAM for long contexts
|
||||
than traditional transformers.
|
||||
|
||||
| Model | Type | Active Params | Size at Q4_K_M | Max Context | Quality | Notes |
|
||||
|-------|------|---------------|---------------|-------------|---------|-------|
|
||||
| **Qwen3.5-35B-A3B** | **MoE** | **3B** | **~12GB** | **262K** | **B+ to A-** | 35B total but only 3B active — quality tracks active params |
|
||||
| **Qwen3.5-27B** | Dense | 27B | ~17GB | 262K | **A-** | 72.4% SWE-bench, ties GPT-5 mini. The real A- option. |
|
||||
| **Qwen3.5-122B-A10B** | MoE | 10B | ~76GB | 262K | A | Matches GPT-5 mini across the board |
|
||||
| **Qwen3.5-9B** | Dense | 9B | ~6GB | 262K | B+ | Fits on any modern GPU |
|
||||
| **Qwen3.5-4B** | Dense | 4B | ~3GB | 262K | B | Tiny but capable |
|
||||
|
||||
**Quality reality check:** MoE models route tokens through only a subset of parameters.
|
||||
The 35B-A3B activates **3B params per token** — think of it as a smart 7B model, not a 35B.
|
||||
Quality is closer to B+ for complex coding. The 27B dense model is genuinely A- but needs
|
||||
17GB weights (leaving less room for context on 32GB). At Q4 quantization there's a further
|
||||
small quality loss. And 262K is a VRAM ceiling, not a quality guarantee — models degrade
|
||||
at the edges of their context window. Practical high-quality context is more like 64-128K.
|
||||
|
||||
**No local model approaches Claude Opus on hard problems.** The strategy isn't to replace
|
||||
Opus — it's to offload the 80% of routine work so your Pro plan limits stop being an issue.
|
||||
|
||||
### Previous Generation (Still Relevant)
|
||||
|
||||
| Model | Size at Q4_K_M | Quality | Notes |
|
||||
|-------|---------------|---------|-------|
|
||||
| **Qwen2.5-Coder 32B** | ~20GB | 73.7 Aider (≈ GPT-4o) | FIM king, 92.7% HumanEval |
|
||||
| **Qwen3-Coder 30B-A3B** (MoE) | ~18GB | #1 SWE-rebench (64.6%) | Only 3.3B active, very fast |
|
||||
| **Qwen3-Coder-Next 80B** (MoE) | needs 64GB+ RAM offload | Beats Claude Opus 4.6 on SWE-rebench | Hybrid attention, 256K context |
|
||||
|
||||
### Honest Assessment: Local vs Claude Code
|
||||
|
||||
Nothing local approaches Claude Opus 4.6 quality for complex multi-file agentic coding.
|
||||
These 32B models are competitive with **GPT-4o** — a tier below Claude Sonnet, two tiers below Opus.
|
||||
|
||||
**The real strategy: Drop Max ($100/mo), keep Pro ($20/mo), offload bulk work to local.**
|
||||
|
||||
The problem with Pro for large projects: rate limits. A 10,000-line codebase needs the model
|
||||
to read, understand, and hold context across many files. On Pro you'll hit usage caps mid-session
|
||||
on complex multi-file work. Max ($100/mo) removes those limits — but that's $80/mo extra.
|
||||
|
||||
Local AI eliminates this problem differently:
|
||||
- **Local model (262K context)**: Reads your entire 10K-line project at once. No rate limits,
|
||||
no usage caps, runs 24/7. Handles the bulk work — understanding codebase structure, routine
|
||||
bug fixes, simple refactors, code explanation, test writing, boilerplate generation.
|
||||
- **Claude Pro ($20/mo)**: Reserved for the hard problems — complex multi-file architectural
|
||||
changes, subtle bugs that need Opus-level reasoning, code review on critical paths.
|
||||
Pro limits are fine when you're only sending Claude the *hard* 20% instead of everything.
|
||||
|
||||
This is the unlock: local doesn't replace Claude, it **reduces your Claude usage enough
|
||||
that Pro limits stop being a problem.** The 80% of routine work that was burning through
|
||||
your Max quota now runs locally with zero limits.
|
||||
|
||||
| Plan | Monthly | What You Get | Limit Problem |
|
||||
|------|---------|--------------|---------------|
|
||||
| Max only | $100 | Opus unlimited | Paying $80/mo for unlimited when you don't need it |
|
||||
| Pro only | $20 | Opus with rate limits | **Hits caps on 10K-line projects** |
|
||||
| **Pro + Local GPU** | **$29** | Opus for hard stuff + unlimited local | **No caps — bulk work is local** |
|
||||
| Local only (no Claude) | $9 | A- quality only | Stuck on hard problems with no escape hatch |
|
||||
|
||||
## 48GB GPU Market (March 22, 2026 — Real Prices)
|
||||
|
||||
| GPU | Arch | Used Price | TDP | Cooling | Tensor Cores | Mem BW |
|
||||
|-----|------|------------|-----|---------|--------------|--------|
|
||||
| **Quadro RTX 8000** | Turing (2018) | **$2,000–2,900** | 260W | Passive variant | Yes (576) | 672 GB/s |
|
||||
| **A40** | Ampere (2020) | **~$5,050+** | 300W | Passive | Yes (336 3rd-gen) | 696 GB/s |
|
||||
| **RTX A6000** | Ampere (2020) | **~$5,400+** | 300W | Active (blower) | Yes (336 3rd-gen) | 768 GB/s |
|
||||
| **L40** | Ada (2022) | **~$6,500+** | 300W | Passive | Yes (568 4th-gen) | 864 GB/s |
|
||||
| **RTX 6000 Ada** | Ada (2022) | **~$6,500+** | 300W | Active | Yes (568 4th-gen) | 960 GB/s |
|
||||
|
||||
Sources: eBay active/sold listings, GPUPoet price tracking, Pangoly, CamelCamelCamel (all March 2026)
|
||||
Note: One outlier RTX 8000 listing at ~$750 exists but is not representative of the market.
|
||||
|
||||
### Cheapest 48GB Option: Quadro RTX 8000 Passive ($2,000–2,900)
|
||||
|
||||
The RTX 8000 is still the cheapest 48GB card — roughly half the price of an A40 and a
|
||||
third of an A6000. The passive variant is purpose-built for rack servers — no fan, relies
|
||||
on chassis airflow, designed for 24/7 operation in 2U/4U systems.
|
||||
|
||||
Key advantages over the P40:
|
||||
- **48GB vs 24GB** — room for models + massive context
|
||||
- **Has Tensor Cores** (576 Turing) — native FP16, no `--force-fp32` hacks for image gen
|
||||
- **NVLink support** — pair two for 96GB combined (100 GB/s bidirectional)
|
||||
- 10W idle power draw
|
||||
|
||||
### Cost Reality Check
|
||||
At $2,000–2,900 the RTX 8000 is a significant investment. The key question: is unified
|
||||
48GB VRAM worth 4–6x the cost of dual P40s ($400–500)?
|
||||
|
||||
**Yes, if** you need large context windows (32K+) for complex coding — KV cache can't
|
||||
be split across two GPUs without NVLink (which P40s don't have).
|
||||
|
||||
**No, if** you're mostly doing short-prompt coding tasks and image gen — dual P40s give
|
||||
you 48GB total (split) at a fraction of the cost, and each card can handle its own workload.
|
||||
|
||||
## RTX 8000 Performance Benchmarks
|
||||
|
||||
### LLM Inference (Exllama, 5.0 bpw quantization)
|
||||
|
||||
| Model | Context | Prompt Processing | Generation |
|
||||
|-------|---------|-------------------|------------|
|
||||
| Qwen3 30B-A3B (MoE) | 8K | 950 t/s | **34 t/s** |
|
||||
| Qwen3 30B-A3B (MoE) | 16K | 673 t/s | **21 t/s** |
|
||||
| Qwen3 30B-A3B (MoE) | 32K | 345 t/s | **11 t/s** |
|
||||
| Llama 3.3 70B | short | 36 t/s | **13 t/s** |
|
||||
| Llama 3.1 8B | — | — | **72 t/s** |
|
||||
|
||||
### Compared to P40 (24GB)
|
||||
|
||||
| Metric | P40 (24GB) | RTX 8000 (48GB) |
|
||||
|--------|-----------|-----------------|
|
||||
| **Used price** | **$150–320** | **$2,000–2,900** |
|
||||
| 32B model fit | Barely (~2GB free) | Comfortable (~28GB free) |
|
||||
| 32B generation speed | ~5-12 t/s (est.) | ~20-34 t/s |
|
||||
| Max practical context | ~4K tokens | **32K+ tokens** |
|
||||
| Image gen (SDXL) | ~49s (`--force-fp32`) | Faster (native FP16) |
|
||||
| Rack server ready | Yes (passive) | Yes (passive variant) |
|
||||
|
||||
### Image Generation
|
||||
The RTX 8000 has Turing Tensor Cores with native FP16 support. Unlike the P40, it does NOT
|
||||
need `--force-fp32` workarounds. Image gen performance is significantly better than the P40,
|
||||
though still behind Ampere/Ada cards.
|
||||
|
||||
## Budget Build: 2x Quadro RTX 5000 + NVLink ($850 Total)
|
||||
|
||||
*The best price-to-capability ratio for local AI coding in 2026.*
|
||||
|
||||
### Why This Works Now
|
||||
|
||||
Qwen 3.5 (February 2026) introduced **Gated Delta Networks** — 3 out of 4 layers use linear
|
||||
attention (O(n) scaling) instead of quadratic. KV cache memory usage is dramatically lower
|
||||
than traditional transformers. A 35B MoE model with 262K context now fits in ~25GB VRAM.
|
||||
|
||||
### Hardware
|
||||
|
||||
#### GPU: NVIDIA Quadro RTX 5000 (Turing, TU104)
|
||||
|
||||
| Spec | Value |
|
||||
|------|-------|
|
||||
| VRAM | 16GB GDDR6 |
|
||||
| CUDA Cores | 3072 |
|
||||
| Tensor Cores | 384 (Gen 2, FP16) |
|
||||
| TDP | ~230W |
|
||||
| NVLink | **Yes — 50 GB/s bidirectional** |
|
||||
| Form Factor | Dual-slot, blower cooler (rack-friendly) |
|
||||
| PCIe | 3.0 x16 |
|
||||
| Used Price | **~$400** |
|
||||
| Part Number | VCQRTX5000-PB |
|
||||
|
||||
#### NVLink Bridge (CRITICAL: RTX 5000 uses a unique smaller connector)
|
||||
|
||||
The Quadro RTX 5000 has a **shorter NVLink connector** than all other Quadro RTX cards.
|
||||
Bridges from the RTX 6000/8000 will NOT physically fit. You must buy the RTX 5000-specific bridge.
|
||||
|
||||
| Detail | Value |
|
||||
|--------|-------|
|
||||
| Product | NVIDIA Quadro RTX 5000 NVLink HB Bridge 2-Slot |
|
||||
| SKU | NVLINKX8-2SLOT-PB |
|
||||
| Part Numbers | 1JF3K, 699-54934-0500-000, 900-54934-0100-000, P4934, 6FY12AA, L55997-001 |
|
||||
| Price | **~$30-80** (eBay, Amazon) |
|
||||
| Bandwidth | 50 GB/s total (25 GB/s per direction) |
|
||||
| Sizing | 2-slot (cards adjacent) or 3-slot (one slot gap — better thermals) |
|
||||
|
||||
**Where to buy:**
|
||||
- eBay: search "Quadro RTX 5000 NVLink" or part numbers P4934 / L55997-001 / 1JF3K
|
||||
- Amazon: search part number 6FY12AA or 1JF3K
|
||||
|
||||
**WARNING:** The 3-slot bridge is recommended over 2-slot. With a 2-slot bridge the cards
|
||||
sit directly adjacent — the top card's blower intake gets blocked by the bottom card.
|
||||
A 3-slot bridge leaves an air gap for proper cooling.
|
||||
|
||||
#### Motherboard Requirements
|
||||
|
||||
| Requirement | Details |
|
||||
|-------------|---------|
|
||||
| PCIe slots | Two x16 slots (x8 electrical is fine — LLM inference is VRAM-bound, not PCIe-bound) |
|
||||
| Slot spacing | Must match your NVLink bridge size (2-slot or 3-slot gap) |
|
||||
| Power supply | 650W+ minimum (80 PLUS Gold recommended), 850W+ for headroom |
|
||||
| Power connectors | 2x 8-pin PCIe power (one per card). Do NOT daisy-chain — use separate cables |
|
||||
| CPU platform | Any modern platform works. Threadripper/Xeon not required |
|
||||
|
||||
**Recommended motherboards (workstation/server):**
|
||||
- Any board with 2x PCIe x16 slots spaced 2-3 slots apart
|
||||
- Server: Dell R730/R740 with GPU riser (but verify 3-slot bridge clearance in 2U)
|
||||
- Workstation: MSI X399 Creation, ASUS WS series, Supermicro X11/X12 boards
|
||||
- Desktop: Most ATX boards with 2 full-length x16 slots work
|
||||
|
||||
**Rack server note:** The Quadro RTX 5000's blower cooler exhausts out the bracket —
|
||||
this works well in rack airflow. If using a 2U server, measure clearance for the NVLink
|
||||
bridge sitting on top of the cards. A 4U chassis gives the most room.
|
||||
|
||||
### What Runs on 16GB (Single RTX 5000 — Start Here)
|
||||
|
||||
| Model | Quant | Context | Quality | Notes |
|
||||
|-------|-------|---------|---------|-------|
|
||||
| **Qwen3.5-35B-A3B** | Q4_K_L | ~64-128K | **B+** | MoE, 3B active. Good but VRAM is tight — context may be lower |
|
||||
| Qwen3.5-9B | Q8 | 128K+ | B+ | Fits comfortably, high quant |
|
||||
| Qwen3.5-4B | Q8 | 262K | B | Tiny model, long context |
|
||||
| Qwen2.5-Coder-7B | Q8 | 128K | B | Solid for simple tasks |
|
||||
| Qwen2.5-Coder-14B | Q4_K_M | 16-32K | B+ | Tight fit, limited context |
|
||||
|
||||
A single card is a solid start — B+ coding with decent context. But 16GB is the ceiling.
|
||||
You can't run bigger dense models, can't use higher quantization, and context is squeezed.
|
||||
|
||||
### What the Second Card + NVLink Unlocks (32GB)
|
||||
|
||||
The second card doesn't just double context — it opens models that **don't fit on 16GB at all:**
|
||||
|
||||
| Model | Arch | Quant | Weights | Context | Total VRAM | Quality | **Why it needs 32GB** |
|
||||
|-------|------|-------|---------|---------|------------|---------|----------------------|
|
||||
| **Qwen3.5-27B** | **Dense** | **Q4_K_M** | **~17GB** | **128K+** | **~25GB** | **A-** | **17GB weights won't fit on 16GB** |
|
||||
| **Qwen2.5-Coder-32B** | **Dense** | **Q4_K_M** | **~20GB** | **16-24K** | **~28GB** | **A-** | **20GB weights won't fit on 16GB** |
|
||||
| Qwen2.5-Coder-14B | Dense | **Q8** | ~16GB | 64K | ~28GB | A- | Q8 quant = better output, needs 16GB for weights alone |
|
||||
| Qwen3-Coder-Next (80B) | MoE | Q4 | ~20GB | 128K | ~28GB | A | 20GB weights won't fit on 16GB |
|
||||
| Qwen3.5-35B-A3B | MoE (3B active) | Q4_K_M | ~12GB | 262K | ~25GB | B+ | Fits on 1 card at reduced context, but 32GB = full 262K + headroom |
|
||||
|
||||
**The real upgrade isn't 262K context — it's access to dense 27B/32B models that are
|
||||
genuinely A- quality.** The 35B-A3B MoE runs on both setups, but its 3B active params
|
||||
limit quality. The Qwen3.5-27B dense model uses all 27B params on every token — that's
|
||||
the quality jump. And its 17GB of weights physically can't fit on a single 16GB card.
|
||||
|
||||
Think of it this way:
|
||||
- **1 card**: B+ coding (MoE or small dense models, squeezed context)
|
||||
- **2 cards**: **A- coding** (full dense 27B/32B models, comfortable context, higher quant options)
|
||||
|
||||
### Practical Context Windows (Usability, Not Ceilings)
|
||||
|
||||
Context window "support" is a ceiling, not what you actually get. VRAM must hold both the
|
||||
model weights AND the KV cache. What's left after weights determines your real context.
|
||||
Quality also degrades toward the edges of a model's context window.
|
||||
|
||||
**Reference: A 10,000-line codebase ≈ 100-150K tokens** (varies by language/comments).
|
||||
This Claude Opus session uses a **1 million token** context window for comparison.
|
||||
|
||||
#### 1 Card (16GB) — Practical
|
||||
|
||||
| Model | Weights | Free for KV | **Usable context** | 10K-line project? |
|
||||
|-------|---------|-------------|-------------------|-------------------|
|
||||
| Qwen3.5-35B-A3B (MoE) | ~12GB | ~3GB | **32-50K tokens** | **No — ~1/3 of it** |
|
||||
| Qwen3.5-9B (dense) | ~6GB | ~9GB | **80-100K tokens** | **Mostly — but B+ quality** |
|
||||
| Qwen2.5-Coder-14B | ~10GB | ~5GB | **16-24K tokens** | **No — a few files at a time** |
|
||||
|
||||
**Workflow on 1 card:** You're feeding files in chunks. Good for "fix this function" or
|
||||
"explain this file." Not for "read my whole project and refactor the auth system."
|
||||
|
||||
#### 2 Cards (32GB via NVLink) — Practical
|
||||
|
||||
| Model | Weights | Free for KV | **Usable context** | 10K-line project? |
|
||||
|-------|---------|-------------|-------------------|-------------------|
|
||||
| **Qwen3.5-27B (dense)** | ~17GB | ~14GB | **80-128K tokens** | **Yes — most/all of it at A-** |
|
||||
| Qwen3.5-35B-A3B (MoE) | ~12GB | ~19GB | **128-180K tokens** | **Yes with room to spare (B+)** |
|
||||
| Qwen2.5-Coder-32B | ~20GB | ~11GB | **32-48K tokens** | **Partial — but strong A- on what it sees** |
|
||||
|
||||
**Workflow on 2 cards:** You can dump most/all of a 10K-line project in one shot with the
|
||||
27B dense model. That's the real workflow change — "here's my whole project, find the bug"
|
||||
becomes possible locally.
|
||||
|
||||
#### vs This Claude Session
|
||||
|
||||
| Setup | Usable context | vs Opus 1M | Whole-project workflow? |
|
||||
|-------|---------------|------------|----------------------|
|
||||
| 1x RTX 5000 (best) | ~50-100K | 5-10% | No — file by file |
|
||||
| **2x RTX 5000 (best)** | **~128-180K** | **13-18%** | **Yes — for 10K-line projects** |
|
||||
| Claude Opus (this session) | 1,000K | 100% | Yes — for anything |
|
||||
|
||||
**Neither setup replaces this session** for complex multi-file work across a 50K+ line
|
||||
codebase. That's why you keep Pro. But 2 cards handles the daily "read my project and
|
||||
help me code" workflow locally with no rate limits — and that's 80% of the work.
|
||||
|
||||
### Squeezing Every Byte: Single-Card Optimization (16GB)
|
||||
|
||||
Before buying a second card, stack these techniques. They're cumulative — use all of them
|
||||
together. The gains compound because they all free VRAM from the same bottleneck: KV cache.
|
||||
|
||||
#### 1. Quantize the KV Cache (Biggest Single Win)
|
||||
|
||||
By default, llama.cpp stores the KV cache in FP16. That's 2 bytes per value. You can
|
||||
compress it with zero code changes — just flags:
|
||||
|
||||
| Cache Type | Bytes/value | vs FP16 | Quality Impact | Verdict |
|
||||
|-----------|-------------|---------|----------------|---------|
|
||||
| FP16 (default) | 2.0 | baseline | none | wasteful on 16GB |
|
||||
| **Q8_0** | **1.0** | **50% smaller** | **~0.002-0.05 perplexity** | **Always use this** |
|
||||
| Q4_0 | 0.5 | 75% smaller | ~0.2 perplexity (noticeable) | Use if desperate |
|
||||
| **Asymmetric: K=Q8_0, V=Q4_0** | **0.75 avg** | **62% smaller** | **Better than uniform Q4** | **Best bang/buck** |
|
||||
|
||||
The K cache is more sensitive to quantization than V. Asymmetric (Q8 keys, Q4 values) gives
|
||||
you ~62% savings with quality closer to Q8 than Q4.
|
||||
|
||||
**Concrete example — Qwen3.5-35B-A3B on 1 card (16GB):**
|
||||
- Weights: ~12GB → 4GB free for KV cache
|
||||
- FP16 KV cache: 4GB → **~50K context**
|
||||
- Q8_0 KV cache: 4GB buys 2x → **~100K context**
|
||||
- K=Q8/V=Q4 KV cache: 4GB buys 2.6x → **~130K context**
|
||||
|
||||
That's the difference between "a few files" and "a meaningful chunk of a project."
|
||||
|
||||
```bash
|
||||
# llama.cpp — always use these three flags together
|
||||
llama-server \
|
||||
--cache-type-k q8_0 \
|
||||
--cache-type-v q4_0 \
|
||||
--flash-attn \
|
||||
-m model.gguf -ngl 99 -c 131072
|
||||
|
||||
# Ollama — set environment variable before starting
|
||||
export OLLAMA_KV_CACHE_TYPE=q8_0 # or q4_0 for aggressive
|
||||
export OLLAMA_FLASH_ATTENTION=1
|
||||
ollama serve
|
||||
```
|
||||
|
||||
#### 2. Flash Attention (Free Speed + VRAM)
|
||||
|
||||
Flash attention restructures how attention is computed — instead of materializing the full
|
||||
attention matrix in VRAM, it computes it in tiles. Result: less VRAM used during inference,
|
||||
slightly faster, **zero quality loss**.
|
||||
|
||||
Always enable it. There's no downside on Turing GPUs with quantized KV cache.
|
||||
|
||||
```bash
|
||||
# llama.cpp
|
||||
--flash-attn
|
||||
|
||||
# Ollama
|
||||
export OLLAMA_FLASH_ATTENTION=1
|
||||
```
|
||||
|
||||
#### 3. Host-Memory Prompt Caching (`--cram`) — System RAM as L2 Cache
|
||||
|
||||
This is the smart use of system RAM. The `--cram` flag in llama-server stores pre-computed
|
||||
prompt representations in host memory (system RAM). When you send the same system prompt
|
||||
or reuse a conversation prefix, it skips reprocessing — hot-swaps the cached computation
|
||||
back onto the GPU.
|
||||
|
||||
This doesn't increase context window size, but it **dramatically reduces time-to-first-token**
|
||||
for repeated workflows (which is most coding — same system prompt, same project context).
|
||||
|
||||
```bash
|
||||
# llama-server with 16GB RAM cache for prompts
|
||||
llama-server \
|
||||
--cram 16384 \
|
||||
--cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \
|
||||
-m model.gguf -ngl 99 -c 131072
|
||||
```
|
||||
|
||||
Your R720/R730 has 128-384GB of DDR3/DDR4 RAM. Use it. `--cram 65536` (64GB) is reasonable
|
||||
for a dedicated inference server — it costs nothing, and repeat prompts become near-instant.
|
||||
|
||||
#### 4. KV Cache to System RAM (`-nkvo`) — Last Resort for Context
|
||||
|
||||
The `-nkvo` (no KV offload) flag moves the entire KV cache to system RAM, freeing all 16GB
|
||||
of VRAM for model weights. This sounds great but comes with a brutal speed penalty:
|
||||
|
||||
| Scenario | Speed Impact |
|
||||
|----------|-------------|
|
||||
| Full VRAM (normal) | Baseline (25-35 tok/s) |
|
||||
| KV in system RAM via PCIe | **5-20x slower** (~2-7 tok/s) |
|
||||
| KV on NVMe via mmap | **30x+ slower** (~1 tok/s) |
|
||||
|
||||
**When it makes sense:** Loading a model that barely doesn't fit (e.g., Qwen3.5-27B dense
|
||||
at 17GB weights on a 16GB card). You'd get ~2-5 tok/s with KV in RAM — painfully slow, but
|
||||
it's the difference between "runs slowly" and "doesn't run at all." Fine for a batch job
|
||||
where you walk away and come back. Not viable for interactive coding.
|
||||
|
||||
**Don't do this routinely.** Quantized KV cache (technique #1) is 50-100x better because
|
||||
the cache stays on the GPU. Only use `-nkvo` for models that literally can't fit otherwise.
|
||||
|
||||
#### 5. Pick the Right Architecture (GQA + MoE = VRAM Efficient)
|
||||
|
||||
Not all models consume KV cache equally. Modern architectures with **Grouped Query Attention
|
||||
(GQA)** use far less KV cache than older Multi-Head Attention (MHA):
|
||||
|
||||
| Architecture | KV cache at 64K context | Examples |
|
||||
|-------------|------------------------|---------|
|
||||
| MHA (old) | ~8-12GB | LLaMA-1, GPT-J |
|
||||
| **GQA (modern)** | **~1-3GB** | **Qwen3.5 series, LLaMA-3** |
|
||||
| **GQA + MoE** | **~1.2GB** | **Qwen3.5-35B-A3B** |
|
||||
|
||||
The Qwen3.5-35B-A3B is almost purpose-built for your situation: 3B active params (fast on
|
||||
Turing), MoE architecture (small memory footprint during inference), and GQA (tiny KV cache).
|
||||
With quantized KV on top of that, 130K+ context on a single 16GB card is realistic.
|
||||
|
||||
#### 6. NVMe as mmap Backing Store
|
||||
|
||||
Your fast NVMe matters for **model loading**, not inference. llama.cpp uses mmap by default
|
||||
to stream model weights from disk, so a fast NVMe means:
|
||||
- Near-instant cold starts (weights stream in as needed)
|
||||
- Graceful degradation if model slightly exceeds RAM (OS pages out unused layers)
|
||||
|
||||
But NVMe is **not** a viable substitute for VRAM during inference. The bandwidth gap is too
|
||||
large: VRAM runs at ~400 GB/s (RTX 5000), system RAM at ~50-100 GB/s (DDR4 quad-channel),
|
||||
NVMe at ~3-7 GB/s. Three orders of magnitude difference from VRAM.
|
||||
|
||||
**Practical use:** Keep all your GGUF model files on NVMe. Enable mmap (default). That's it.
|
||||
Don't try to use NVMe as overflow for the KV cache — the latency kills interactive use.
|
||||
|
||||
#### Stacking Everything: Revised Single-Card Numbers
|
||||
|
||||
| Model | Optimization | Usable Context | Speed | Quality |
|
||||
|-------|-------------|---------------|-------|---------|
|
||||
| Qwen3.5-35B-A3B Q4 | None (defaults) | ~32-50K | 25-35 tok/s | B+ |
|
||||
| Qwen3.5-35B-A3B Q4 | **KV Q8 + flash** | **~80-100K** | **25-35 tok/s** | **B+** |
|
||||
| Qwen3.5-35B-A3B Q4 | **KV asym + flash** | **~100-130K** | **25-35 tok/s** | **B+ (tiny quality dip)** |
|
||||
| Qwen3.5-9B Q8 | KV Q8 + flash | ~120-160K | 30-45 tok/s | B |
|
||||
| Qwen2.5-Coder-14B Q4 | KV Q8 + flash | ~40-64K | 20-30 tok/s | B+ |
|
||||
|
||||
Add `--cram` on top for instant repeated prompts. That's your real single-card ceiling.
|
||||
|
||||
**The honest answer:** With all optimizations stacked, a single card goes from "a few files
|
||||
at a time" to "maybe half a 10K-line project." That's a meaningful upgrade from the
|
||||
unoptimized baseline, but it still doesn't match what 2 cards with a dense 27B model gives
|
||||
you. The second card isn't about optimization tricks — it's about physics (more VRAM = more
|
||||
data on the fast bus).
|
||||
|
||||
### Estimated Inference Speed
|
||||
|
||||
| Model | 1x RTX 5000 | 2x RTX 5000 (NVLink) |
|
||||
|-------|-------------|---------------------|
|
||||
| Qwen3.5-35B-A3B Q4 (short ctx) | ~25-35 tok/s | ~25-35 tok/s |
|
||||
| Qwen3.5-35B-A3B Q4 (128K ctx) | ~10-18 tok/s | ~15-25 tok/s |
|
||||
| Qwen3.5-35B-A3B Q4 (262K ctx) | Won't fit | ~10-18 tok/s |
|
||||
| Qwen2.5-Coder-14B Q4 | ~20-30 tok/s | ~25-35 tok/s |
|
||||
|
||||
NVLink matters most at large context windows where KV cache spans both cards.
|
||||
At short contexts that fit on one card, the second GPU adds less benefit.
|
||||
|
||||
**Speed reality check:** NVLink doesn't make it faster — it prevents the slowdown you'd get
|
||||
from PCIe when the model spans both cards. The base speed is still Turing (2018 silicon).
|
||||
10-35 tok/s is fast enough for coding (you read slower than that), but it's not instant.
|
||||
The MoE architecture (only 3B active params at inference) is what makes it viable on older
|
||||
hardware — NVLink just removes the inter-GPU bottleneck for 262K context.
|
||||
|
||||
### Image Generation (Included — No Extra Cost)
|
||||
|
||||
The RTX 5000 has **384 Tensor Cores with native FP16** — full SDXL/Flux support, no hacks.
|
||||
|
||||
| Workload | VRAM Needed | Where It Runs |
|
||||
|----------|-------------|---------------|
|
||||
| SDXL (1024x1024) | ~8-10GB | Either card alone |
|
||||
| Flux Dev | ~12-14GB | Single card (16GB) |
|
||||
| Flux Dev (high-res / batched) | ~18-24GB | Both cards via NVLink (32GB) |
|
||||
| ComfyUI / InvokeAI | Works natively | No `--force-fp32` needed |
|
||||
|
||||
**Important: 262K context requires both cards unified.** You can't split one off for image
|
||||
gen and keep 262K. It's one task at a time:
|
||||
|
||||
```bash
|
||||
# CODING SESSION: Both cards unified → 32GB → 262K context
|
||||
ollama run qwen3.5:35b-a3b-q4_K_M # Uses both GPUs via NVLink
|
||||
|
||||
# IMAGE GEN SESSION: Stop LLM, run image gen on one card (16GB is plenty)
|
||||
ollama stop # Frees VRAM
|
||||
comfyui --listen 0.0.0.0 # SDXL/Flux fits easily in 16GB
|
||||
|
||||
# Swap takes a few seconds, not simultaneous but not painful
|
||||
```
|
||||
|
||||
If you want simultaneous coding + image gen, you'd run a smaller model at shorter context
|
||||
on one card (e.g., Qwen3.5-35B-A3B at ~64K on 16GB) and image gen on the other. But for
|
||||
full 262K context, both cards must be dedicated to the LLM.
|
||||
|
||||
### Hardware Longevity: 3-5 Years Realistic
|
||||
|
||||
- **2026-2027**: Sweet spot. MoE + linear attention models are getting smaller active params.
|
||||
32GB unified handles the best coding models at full context. Peak value.
|
||||
- **2028-2029**: Still useful. The trend is more efficient models, not bigger ones.
|
||||
32GB likely still runs the best ~35-70B MoE coding models of that era.
|
||||
- **2030+**: Questionable. New architectures may need FP8, newer tensor core ops that
|
||||
Turing lacks. But VRAM is VRAM — something useful will always run on 32GB.
|
||||
- **The cards themselves won't die** — Quadro-grade, designed for 24/7 data center use.
|
||||
They'll be outclassed before they fail.
|
||||
|
||||
### Power Consumption & Cost
|
||||
|
||||
| Config | Idle | Load | Monthly (8hr/day @ $0.09/kWh) | Annual |
|
||||
|--------|------|------|-------------------------------|--------|
|
||||
| 1x Quadro RTX 5000 | ~15W | ~210W | **~$4.50** | ~$54 |
|
||||
| 2x Quadro RTX 5000 | ~30W | ~420W | **~$9.00** | ~$108 |
|
||||
|
||||
### Software Setup
|
||||
|
||||
#### llama.cpp (Recommended — Best Multi-GPU Support)
|
||||
|
||||
```bash
|
||||
# Build with CUDA support
|
||||
git clone https://github.com/ggerganov/llama.cpp
|
||||
cd llama.cpp
|
||||
cmake -B build -DGGML_CUDA=ON
|
||||
cmake --build build --config Release -j$(nproc)
|
||||
|
||||
# Download Qwen3.5-35B-A3B GGUF (Q4_K_M)
|
||||
# Get from: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF
|
||||
|
||||
# Run on dual GPU with NVLink (all optimizations on)
|
||||
./build/bin/llama-server \
|
||||
-m Qwen3.5-35B-A3B-Q4_K_M.gguf \
|
||||
-ngl 999 \
|
||||
-c 262144 \
|
||||
--cache-type-k q8_0 \
|
||||
--cache-type-v q4_0 \
|
||||
--flash-attn \
|
||||
--cram 65536 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080
|
||||
|
||||
# --cache-type-k q8_0 / --cache-type-v q4_0 = asymmetric KV quantization (62% smaller cache)
|
||||
# --flash-attn = tiled attention (less VRAM, no quality loss)
|
||||
# --cram 65536 = 64GB host RAM prompt cache (instant repeat prompts)
|
||||
# llama.cpp auto-detects NVLink and splits layers across both GPUs
|
||||
# Use -ts 1,1 to manually set equal split if needed
|
||||
|
||||
# Single card variant (no NVLink) — same flags, smaller context
|
||||
./build/bin/llama-server \
|
||||
-m Qwen3.5-35B-A3B-Q4_K_M.gguf \
|
||||
-ngl 999 \
|
||||
-c 131072 \
|
||||
--cache-type-k q8_0 \
|
||||
--cache-type-v q4_0 \
|
||||
--flash-attn \
|
||||
--cram 65536 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080
|
||||
```
|
||||
|
||||
#### Ollama
|
||||
|
||||
```bash
|
||||
# Requires Ollama v0.17+ for Qwen3.5 support
|
||||
# NOTE: As of March 2026, some Qwen3.5 GGUFs have compatibility issues
|
||||
# with Ollama due to mmproj vision files. llama.cpp may be more reliable.
|
||||
|
||||
# Environment variables for multi-GPU
|
||||
export OLLAMA_GPU_SPLIT=16,16 # Equal split across both 16GB cards
|
||||
export OLLAMA_KV_CACHE_TYPE=q8_0 # Halves KV cache VRAM with minimal quality loss
|
||||
export OLLAMA_KEEP_ALIVE=24h # Keep model loaded in VRAM
|
||||
export OLLAMA_FLASH_ATTENTION=1 # Enable flash attention for VRAM savings
|
||||
|
||||
# Pull and run
|
||||
ollama pull qwen3.5:35b-a3b-q4_K_M
|
||||
ollama run qwen3.5:35b-a3b-q4_K_M
|
||||
```
|
||||
|
||||
#### Verify NVLink Is Working
|
||||
|
||||
```bash
|
||||
# Check NVLink status
|
||||
nvidia-smi nvlink --status
|
||||
|
||||
# Check NVLink bandwidth
|
||||
nvidia-smi nvlink -gt d
|
||||
|
||||
# Monitor both GPUs during inference
|
||||
watch -n 0.5 nvidia-smi
|
||||
```
|
||||
|
||||
### Total Cost Summary
|
||||
|
||||
| Item | Cost |
|
||||
|------|------|
|
||||
| 2x Quadro RTX 5000 | ~$800 |
|
||||
| NVLink HB Bridge 2-slot (P4934) | ~$50 |
|
||||
| Dell cables/riser (R720/R730) | ~$60 |
|
||||
| Dell 1100W PSUs (if needed) | ~$60-100 |
|
||||
| **Hardware total** | **~$960** |
|
||||
| Monthly power (2 cards, 8hr/day) | $9/mo |
|
||||
| Claude Pro subscription (keep) | $20/mo |
|
||||
| Claude Max subscription (drop) | -$100/mo saved |
|
||||
| **Net monthly cost** | **$29/mo (was $100/mo)** |
|
||||
|
||||
### The Math: Drop Max, Keep Pro, Add Local
|
||||
|
||||
| | Year 1 | Year 2 | Year 3 | **3-Year Total** |
|
||||
|---|--------|--------|--------|-----------------|
|
||||
| **Claude Max (current)** | $1,200 | $1,200 | $1,200 | **$3,600** |
|
||||
| **Pro + Local GPU** | $960 + $348 | $348 | $348 | **$2,004** |
|
||||
| **Savings** | | | | **$1,596** |
|
||||
|
||||
You save ~$71/mo after hardware payoff. The GPU pays for itself in **13 months**.
|
||||
After that, you're saving $80/mo vs Max with no usage limits on bulk work.
|
||||
|
||||
### Comparison: This Build vs Alternatives
|
||||
|
||||
| Setup | Monthly | 3yr Total | Limits? | Quality |
|
||||
|-------|---------|-----------|---------|---------|
|
||||
| **Pro + 2x RTX 5000** | **$29** | **$2,004** | **Unlimited local, Pro limits for Opus** | **A- local, A+ cloud** |
|
||||
| Pro + 1x RTX 3090 | $25 | $1,620 | Unlimited local (128K ctx), Pro limits | A- local, A+ cloud |
|
||||
| Pro + RTX 8000 (48GB) | $29 | $3,040+ | Unlimited local, Pro limits | A- local, A+ cloud |
|
||||
| **Claude Max (no GPU)** | **$100** | **$3,600** | **Unlimited Opus** | **A+ cloud only** |
|
||||
| Claude Pro only (no GPU) | $20 | $720 | **Hits caps on large projects** | A+ cloud, limited |
|
||||
| API-only (Opus heavy use) | $500+ | $18,000+ | Pay per token | A+ cloud only |
|
||||
|
||||
### Phased Build Plan
|
||||
|
||||
**Phase 1 — Start with one card ($400)**
|
||||
1. Buy Quadro RTX 5000 (VCQRTX5000-PB) — ~$400 on eBay
|
||||
2. Install in any PCIe x16 slot
|
||||
3. Install llama.cpp or Ollama v0.17+
|
||||
4. Run Qwen3.5-35B-A3B at Q4_K_L with 64-128K context
|
||||
5. Already A- quality for coding — test if local inference fits your workflow
|
||||
|
||||
**Phase 2 — Add second card + NVLink ($450)**
|
||||
1. Buy matching Quadro RTX 5000 — ~$400
|
||||
2. Buy NVLink HB Bridge 2-slot (part: P4934 / 1JF3K / 6FY12AA) — ~$50
|
||||
3. Install second card in adjacent/nearby x16 slot
|
||||
4. Connect NVLink bridge
|
||||
5. Verify with `nvidia-smi nvlink --status`
|
||||
6. Now running 32GB unified — Qwen3.5-35B-A3B at Q4_K_M with full 262K context
|
||||
|
||||
### Dell R720/R730 Installation Guide
|
||||
|
||||
#### Prerequisites (MUST HAVE before buying GPUs)
|
||||
|
||||
| Requirement | R720 | R730 | Why |
|
||||
|-------------|-------|-------|-----|
|
||||
| **Dual CPUs** | Required | Required | GPU riser slots are wired to CPU2 — dead without it |
|
||||
| **2x 1100W PSUs** | Required | Required | 2x 230W GPUs + system = ~600W+ under load |
|
||||
| **GPU Riser 3** | Required for 2nd GPU | Required for GPUs | Provides the PCIe x16 slot + 8-pin power |
|
||||
| **GPU Power Cable** | Required | Required | Riser-to-GPU power, not included by default |
|
||||
| **Low-Profile Heatsinks** | Must swap (part of enablement kit) | Usually pre-installed | Standard heatsinks block GPU riser clearance |
|
||||
| **Max ambient temp** | 30°C (not the usual 35°C) | 30°C | High GPU TDP restricts cooling headroom |
|
||||
|
||||
#### Shopping List: Dell-Specific Parts
|
||||
|
||||
| Part | Dell P/N | What It Is | Price | Where |
|
||||
|------|----------|------------|-------|-------|
|
||||
| **GPU Power Cable** | **9H6FV** (09H6FV) | 8-pin EPS (riser) → 6-pin + 6+2-pin PCIe. One cable powers one GPU | ~$10-15 | Amazon, eBay |
|
||||
| **GPU Power Cable (alt)** | **N08NH** (0N08NH) | Same function, alternate Dell part number | ~$10-15 | Amazon, eBay |
|
||||
| **GPU Riser 3** (R720) | Check eBay for "R720 riser 3" or "R720 GPU riser" | Second riser card that provides GPU-capable x16 slot | ~$15-30 | eBay |
|
||||
| **GPU Riser 3** (R730) | Check eBay for "R730 riser 3" or "R730 GPU riser" | R730 version — NOT interchangeable with R720 | ~$15-30 | eBay |
|
||||
| **Low-Profile Heatsinks** (R720 only) | Part of original GPU enablement kit | Shorter heatsinks that clear the GPU riser. Search "R720 low profile heatsink" | ~$10-20/pair | eBay |
|
||||
|
||||
**You need 2x power cables** (one per GPU). Search Amazon for "Dell R720 R730 GPU power cable 9H6FV" — multiple sellers (COMeap, ZAHARA, BestParts) stock them for ~$10-15 each.
|
||||
|
||||
#### How It Fits
|
||||
|
||||
```
|
||||
Dell R720/R730 Riser Layout (rear view):
|
||||
┌─────────────────────────────────────┐
|
||||
│ Riser 1 Riser 2 Riser 3│
|
||||
│ (network/ (GPU 1) (GPU 2)│
|
||||
│ storage) PCIe x16 PCIe x16│
|
||||
│ Gen2(720) Gen2(720)│
|
||||
│ Gen3(730) Gen3(730)│
|
||||
└─────────────────────────────────────┘
|
||||
↑ RTX 5000 ↑ ↑ RTX 5000 ↑
|
||||
└── NVLink Bridge ──┘
|
||||
```
|
||||
|
||||
- Both GPUs sit on **adjacent risers** (Riser 2 + Riser 3) — this is 2-slot spacing
|
||||
- The **2-slot NVLink bridge** (P4934) is the correct size for R720/R730
|
||||
- The cards mount **vertically** via risers, parallel to each other
|
||||
- NVLink bridge connects across the top of both cards
|
||||
|
||||
#### R720 vs R730
|
||||
|
||||
| Feature | R720 | R730 |
|
||||
|---------|------|------|
|
||||
| **PCIe** | Gen2 x16 | **Gen3 x16** |
|
||||
| **Impact on LLM** | None — VRAM-bound | None — VRAM-bound |
|
||||
| **Impact on NVLink** | None — NVLink bypasses PCIe | None — NVLink bypasses PCIe |
|
||||
| **GPU power delivery** | Same 8-pin from riser | Same 8-pin from riser |
|
||||
| **Heatsink swap** | Usually required | Usually already low-profile |
|
||||
| **Used price** | ~$100-150 cheaper | Preferred if budget allows |
|
||||
| **Recommendation** | Fine if you already have one | **Buy this one** if shopping new |
|
||||
|
||||
#### Potential Issues
|
||||
|
||||
1. **NVLink bridge clearance in 2U** — The bridge sits on top of both GPUs. In a 2U chassis
|
||||
this is tight. The R720/R730 riser design mounts cards vertically which actually helps —
|
||||
the bridge faces the chassis side panel, not the lid. Should fit, but measure before buying.
|
||||
|
||||
2. **Blower fan noise** — The RTX 5000 has an active blower (unlike passive Tesla cards).
|
||||
The server's own fans may spin higher to compensate. The blower exhausts out the bracket
|
||||
which is correct for rack airflow.
|
||||
|
||||
3. **"Unsupported" GPU warning** — Dell officially supports Tesla/Quadro cards from their era.
|
||||
The Quadro RTX 5000 is a later generation than R720/R730 was designed for, but community
|
||||
reports confirm Quadro RTX and even consumer RTX cards work fine. You won't get Dell support
|
||||
if something goes wrong, but electrically it's standard PCIe.
|
||||
|
||||
4. **CPU TDP limit** — Dell requires CPUs of 115W or less when GPUs are installed (R720).
|
||||
Check your CPU model. Most common Xeon E5-2600 v1/v2 (R720) and E5-2600 v3/v4 (R730)
|
||||
processors are within this range, but some high-core-count variants exceed it.
|
||||
|
||||
5. **PSU mode** — With dual 300W GPUs, set PSU configuration to **non-redundant mode**
|
||||
to use combined wattage from both PSUs. In redundant mode, you're limited to one PSU's
|
||||
capacity (1100W) which may not be enough under full GPU + CPU load.
|
||||
|
||||
#### Complete R720/R730 Shopping List
|
||||
|
||||
```
|
||||
GPUS + NVLINK
|
||||
2x Quadro RTX 5000 ~$800
|
||||
1x NVLink Bridge 2-slot (P4934 / L55997-001) ~$50
|
||||
|
||||
DELL-SPECIFIC PARTS
|
||||
2x GPU Power Cable (9H6FV or N08NH) ~$25
|
||||
1x GPU Riser 3 (match your server model!) ~$20
|
||||
2x Low-Profile Heatsinks (R720 only) ~$15
|
||||
|
||||
POWER (if not already installed)
|
||||
2x Dell 1100W PSU ~$30-50 ea
|
||||
|
||||
TOTAL (assuming you have the server + dual CPUs) ~$940-960
|
||||
```
|
||||
|
||||
## 24GB GPU Options (Previous Research — Still Valid for Tighter Budgets)
|
||||
|
||||
| GPU | VRAM | Price Range | Best Deals | Notes |
|
||||
|-----|------|-------------|------------|-------|
|
||||
| **Tesla P40** | 24GB | $150-320 | Newegg refurb $219-270; eBay used $150-200 | Best VRAM/$ at 24GB |
|
||||
| **RTX A2000 12GB** | 12GB | $250-535 | eBay used ~$250-350; one listing at $490 | Can't run 32B models |
|
||||
| **Tesla T4** | 16GB | $150-350 | eBay used $150-250 | Great power efficiency |
|
||||
| **RTX A4000** | 16GB | $700-750+ | eBay used ~$700; new $720+ | Too expensive for 16GB |
|
||||
|
||||
## Rack Server Compatibility
|
||||
|
||||
### Quadro RTX 8000 Passive in R720/R730
|
||||
- **Physical fit**: Full-length, dual-slot — fits in GPU riser slots
|
||||
- **Power**: 260W, requires 8-pin aux power + GPU enablement kit
|
||||
- **Cooling**: Passive — relies on server chassis fans (same as P40)
|
||||
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
|
||||
- **NVLink**: Can pair two RTX 8000s for 96GB combined VRAM
|
||||
- Very similar physical/power requirements to the Tesla P40
|
||||
|
||||
### RTX A2000 in R720/R730
|
||||
- **Physical fit**: Yes. Dual-slot, low-profile, 167mm length
|
||||
- **Power**: 70W bus-powered, no aux cable needed. Must use 75W slots (slots 4-7 on R720)
|
||||
- **Cooling**: Blower-style fan exhausts out bracket — ideal for rack airflow
|
||||
- **Requirement**: Dual CPUs needed for GPU PCIe slots
|
||||
- **Confirmed working** in Dell R740XD (similar architecture)
|
||||
|
||||
### Tesla P40 in R720/R730
|
||||
- **Physical fit**: Yes. Full-length, single-slot, designed for rack servers
|
||||
- **Power**: 250W, requires 8-pin aux power. Needs GPU enablement kit
|
||||
- **Cooling**: Passive — relies on server chassis fans
|
||||
- **Requirement**: Dual CPUs, redundant 1100W PSUs recommended
|
||||
- **Natively supported** in these servers
|
||||
|
||||
### R720 vs R730
|
||||
- R720: PCIe Gen2 (not a bottleneck for LLM inference, which is VRAM-bound)
|
||||
- R730: PCIe Gen3, generally preferred
|
||||
- Both support up to 2x double-wide or 4x single-wide GPUs
|
||||
|
||||
## Recommended Setups
|
||||
|
||||
### If budget allows ($2,000–2,900): RTX 8000 Passive
|
||||
Single card handles both coding and image gen. 48GB VRAM fits 32B models with massive
|
||||
context windows (32K+). Passive cooling is rack-native. Tensor cores handle FP16 image gen
|
||||
properly. One card, one slot, simple setup. The premium buys you unified VRAM = big context.
|
||||
|
||||
### If budget allows + dedicated image gen ($2,300–3,250): RTX 8000 + A2000
|
||||
RTX 8000 for coding with full 48GB dedicated to LLM context.
|
||||
A2000 for image gen (3x faster than Turing, 70W, bus-powered, blower cooled).
|
||||
Best separation of concerns — no model swapping needed.
|
||||
|
||||
### Best value ($400–500): Dual P40
|
||||
Two P40s for 48GB total, but split across cards (can't combine for one model without
|
||||
NVLink, which P40s lack). One for 32B coding (tight fit, ~4K context), one for image gen
|
||||
(slow, needs --force-fp32). **5x cheaper than RTX 8000** but with significant context limitations.
|
||||
|
||||
### Cheapest entry ($200–300): Single P40
|
||||
Run 32B coding model with very limited context (~4K tokens). Swap to image gen when needed.
|
||||
Good for testing whether local LLM coding works for your workflow before investing more.
|
||||
|
||||
## Configuration Notes for local-ai stack
|
||||
|
||||
### For 48GB RTX 8000
|
||||
```bash
|
||||
# Ollama — take advantage of the full 48GB
|
||||
OLLAMA_NUM_GPU=999
|
||||
OLLAMA_NUM_CTX=32768 # Large context window — 48GB can handle it
|
||||
OLLAMA_KEEP_ALIVE=24h
|
||||
|
||||
# Pull best coding models
|
||||
ollama pull qwen2.5-coder:32b-instruct-q4_K_M # ~20GB, leaves 28GB for context
|
||||
ollama pull qwen3.5:27b # ~16GB at Q4, even more context room
|
||||
ollama pull qwen3-coder:30b # MoE, very fast inference
|
||||
|
||||
# Higher quantization for better quality (48GB allows this)
|
||||
# Look for Q6_K or Q8_0 variants on Ollama for better output quality
|
||||
```
|
||||
|
||||
### For 32B models on P40 (24GB — tight fit)
|
||||
```bash
|
||||
OLLAMA_NUM_GPU=999
|
||||
OLLAMA_NUM_CTX=4096 # Keep context small to fit in remaining VRAM
|
||||
OLLAMA_KEEP_ALIVE=24h
|
||||
|
||||
ollama pull qwen2.5-coder:32b-instruct-q4_K_M
|
||||
```
|
||||
|
||||
### For dual-GPU setup (RTX 8000 + A2000 or P40 + anything)
|
||||
```bash
|
||||
# Assign GPU 0 to Ollama (coding), GPU 1 to InvokeAI (image gen)
|
||||
# In docker-compose.yml for Ollama:
|
||||
CUDA_VISIBLE_DEVICES=0
|
||||
|
||||
# In docker-compose.yml for InvokeAI:
|
||||
CUDA_VISIBLE_DEVICES=1
|
||||
```
|
||||
|
||||
### For image gen on P40 (no tensor cores)
|
||||
```bash
|
||||
# InvokeAI
|
||||
INVOKEAI_PRECISION=float32
|
||||
|
||||
# ComfyUI launch args
|
||||
--force-fp32
|
||||
```
|
||||
|
||||
### For image gen on RTX 8000 / A2000 / T4 (has tensor cores)
|
||||
```bash
|
||||
# InvokeAI — native FP16 works fine
|
||||
INVOKEAI_PRECISION=float16
|
||||
|
||||
# ComfyUI — no special flags needed
|
||||
```
|
||||
|
||||
## Sources
|
||||
- [Quadro RTX 8000 for Local LLMs — Hardware Corner](https://www.hardware-corner.net/guides/quadro-rtx-8000-for-llm/)
|
||||
- [RTX 8000 Passive — Network Outlet](https://networkoutlet.com/blogs/articles/nvidia-quadro-rtx-8000-48gb-passive-cooling-powering-ai-rendering-server-workloads)
|
||||
- [LLM Benchmarks on Turing/Ampere GPUs — Stefandroid](https://blog.stefandroid.com/2025/06/02/benchmark-llm-performance-nvidia-gpus.html)
|
||||
- [NVIDIA A40 Price Tracking — GPUPoet](https://gpupoet.com/gpu/learn/card/nvidia-a40)
|
||||
- [NVIDIA L40 Price Tracking — GPUPoet](https://gpupoet.com/gpu/learn/card/nvidia-l40)
|
||||
- [RTX A6000 Price History — CamelCamelCamel](https://camelcamelcamel.com/product/B09BDH8VZV)
|
||||
- [RTX A6000 Price History — Pangoly](https://pangoly.com/en/price-history/pny-nvidia-quadro-rtx-a6000)
|
||||
- [NVIDIA RTX A2000 Datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-a2000/nvidia-rtx-a2000-datasheet-1987439-r5.pdf)
|
||||
- [Dell R730 Owner's Manual — Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r730/r730_ompublication/expansion-card-installation-guidelines)
|
||||
- [Dell R720 Owner's Manual — Expansion Cards](https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom/expansion-card-installation-guidelines)
|
||||
- [ComfyUI GPU Benchmarks Discussion](https://github.com/Comfy-Org/ComfyUI/discussions/2970)
|
||||
- [ComfyUI P40 FP32 Issue](https://github.com/Comfy-Org/ComfyUI/issues/4363)
|
||||
- [Best Local LLMs for 24GB VRAM 2026](https://localllm.in/blog/best-local-llms-24gb-vram)
|
||||
- [Best Coding Models 2026](https://localvram.com/en/guides/best-coding-models/)
|
||||
- [Ollama VRAM Requirements Guide](https://localllm.in/blog/ollama-vram-requirements-for-local-llms)
|
||||
- [Local LLMs That Can Replace Claude Code](https://agentnativedev.medium.com/local-llms-that-can-replace-claude-code-6f5b6cac93bf)
|
||||
- [7 Local LLM Families to Replace Claude/Codex](https://agentnativedev.medium.com/7-local-llm-families-to-replace-claude-codex-for-everyday-tasks-25ba74c3635d)
|
||||
- [Qwen2.5-Coder 32B on Ollama](https://ollama.com/library/qwen2.5-coder:32b-instruct-q4_K_M)
|
||||
- [Qwen3-Coder — How to Run Locally](https://unsloth.ai/docs/models/qwen3-coder-how-to-run-locally)
|
||||
Executable
+464
@@ -0,0 +1,464 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# Gitea ↔ GitHub Mirror Sync
|
||||
#
|
||||
# Mirrors repos between your local Gitea and GitHub in both directions:
|
||||
# GitHub → Gitea: Pulls repos you own on GitHub into Gitea (backup/offline use)
|
||||
# Gitea → GitHub: Pushes Gitea repos to GitHub (remote backup)
|
||||
#
|
||||
# Usage:
|
||||
# ./gitea-github-sync.sh — sync all configured repos
|
||||
# ./gitea-github-sync.sh --pull-only — GitHub → Gitea only
|
||||
# ./gitea-github-sync.sh --push-only — Gitea → GitHub only
|
||||
# ./gitea-github-sync.sh --repo owner/name — sync one specific repo
|
||||
# ./gitea-github-sync.sh --list — list what would sync (dry run)
|
||||
# ./gitea-github-sync.sh --init — interactive first-time setup
|
||||
#
|
||||
# Config: ~/.config/gitea-github-sync/config
|
||||
# Tokens: reads from .env in the same directory as this script (or $SYNC_ENV)
|
||||
#
|
||||
# Schedule: install the systemd timer with --install-timer
|
||||
# ./gitea-github-sync.sh --install-timer — every 6 hours (default)
|
||||
# ./gitea-github-sync.sh --install-timer 1h — custom interval
|
||||
# ./gitea-github-sync.sh --remove-timer — remove the timer
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[sync]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[ ok ]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[warn]${NC} $*"; }
|
||||
err() { echo -e "${RED}[err ]${NC} $*" >&2; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CONFIG_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/gitea-github-sync"
|
||||
CONFIG_FILE="$CONFIG_DIR/config"
|
||||
WORK_DIR="$CONFIG_DIR/repos"
|
||||
LOG_FILE="$CONFIG_DIR/sync.log"
|
||||
|
||||
# ── load tokens from .env ───────────────────────────────────────────────────
|
||||
ENV_FILE="${SYNC_ENV:-$SCRIPT_DIR/.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
set -a; source <(grep -E '^(GITEA_TOKEN|GITHUB_TOKEN|GITEA_URL)=' "$ENV_FILE" | sed 's/ *#.*//'); set +a
|
||||
fi
|
||||
|
||||
GITEA_URL="${GITEA_URL:-http://localhost:3001}"
|
||||
GITEA_TOKEN="${GITEA_TOKEN:-}"
|
||||
GITHUB_TOKEN="${GITHUB_TOKEN:-}"
|
||||
|
||||
# ── parse args ──────────────────────────────────────────────────────────────
|
||||
MODE="all" # all | pull | push | list | init | install-timer | remove-timer
|
||||
SINGLE_REPO=""
|
||||
TIMER_INTERVAL="6h"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--pull-only) MODE="pull"; shift ;;
|
||||
--push-only) MODE="push"; shift ;;
|
||||
--list) MODE="list"; shift ;;
|
||||
--init) MODE="init"; shift ;;
|
||||
--install-timer) MODE="install-timer"; shift; [[ "${1:-}" =~ ^[0-9]+[smhd]$ ]] && { TIMER_INTERVAL="$1"; shift; } ;;
|
||||
--remove-timer) MODE="remove-timer"; shift ;;
|
||||
--repo) shift; SINGLE_REPO="${1:-}"; shift ;;
|
||||
-h|--help)
|
||||
sed -n '2,/^# =====/{ /^# =====/d; s/^# \?//p; }' "$0"; exit 0 ;;
|
||||
*) err "Unknown arg: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────────────────────
|
||||
_gitea_api() {
|
||||
local method="$1" path="$2"; shift 2
|
||||
curl -sfL -X "$method" \
|
||||
-H "Authorization: token $GITEA_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
"$GITEA_URL/api/v1$path" "$@"
|
||||
}
|
||||
|
||||
_github_api() {
|
||||
local method="$1" path="$2"; shift 2
|
||||
curl -sfL -X "$method" \
|
||||
-H "Authorization: Bearer $GITHUB_TOKEN" \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com$path" "$@"
|
||||
}
|
||||
|
||||
_log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG_FILE"; }
|
||||
|
||||
# ── config management ──────────────────────────────────────────────────────
|
||||
load_config() {
|
||||
mkdir -p "$CONFIG_DIR" "$WORK_DIR"
|
||||
GITHUB_USER=""
|
||||
GITEA_USER=""
|
||||
SYNC_REPOS=() # explicit list (empty = auto-discover)
|
||||
EXCLUDE_REPOS=() # repos to skip
|
||||
PUSH_PRIVATE=false # push private Gitea repos to GitHub?
|
||||
PULL_PRIVATE=true # pull private GitHub repos to Gitea?
|
||||
PULL_FORKS=false # pull forked repos from GitHub?
|
||||
|
||||
if [[ -f "$CONFIG_FILE" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
source "$CONFIG_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
save_config() {
|
||||
mkdir -p "$CONFIG_DIR"
|
||||
cat > "$CONFIG_FILE" << EOF
|
||||
# Gitea-GitHub Sync — configuration
|
||||
# Generated $(date '+%Y-%m-%d %H:%M:%S')
|
||||
|
||||
# GitHub username (for discovering repos to pull)
|
||||
GITHUB_USER="$GITHUB_USER"
|
||||
|
||||
# Gitea username (for discovering repos to push)
|
||||
GITEA_USER="$GITEA_USER"
|
||||
|
||||
# Explicit repo list — if set, only these sync. Format: owner/repo
|
||||
# Leave empty () to auto-discover from both platforms.
|
||||
SYNC_REPOS=($(printf '"%s" ' "${SYNC_REPOS[@]}"))
|
||||
|
||||
# Repos to skip (pattern matched against owner/repo)
|
||||
EXCLUDE_REPOS=($(printf '"%s" ' "${EXCLUDE_REPOS[@]}"))
|
||||
|
||||
# Push private Gitea repos to GitHub as private repos?
|
||||
PUSH_PRIVATE=$PUSH_PRIVATE
|
||||
|
||||
# Pull private GitHub repos to Gitea?
|
||||
PULL_PRIVATE=$PULL_PRIVATE
|
||||
|
||||
# Pull forked repos from GitHub?
|
||||
PULL_FORKS=$PULL_FORKS
|
||||
EOF
|
||||
ok "Config saved: $CONFIG_FILE"
|
||||
}
|
||||
|
||||
# ── init (first-time setup) ────────────────────────────────────────────────
|
||||
do_init() {
|
||||
echo -e "\n${BOLD}Gitea ↔ GitHub Sync — First-Time Setup${NC}\n"
|
||||
|
||||
# Check tokens
|
||||
if [[ -z "$GITEA_TOKEN" || "$GITEA_TOKEN" == "your-gitea-token-here" ]]; then
|
||||
err "GITEA_TOKEN not set. Add it to $ENV_FILE first."
|
||||
echo " Generate at: $GITEA_URL/user/settings/applications"
|
||||
exit 1
|
||||
fi
|
||||
if [[ -z "$GITHUB_TOKEN" || "$GITHUB_TOKEN" == "your-github-token-here" ]]; then
|
||||
err "GITHUB_TOKEN not set. Add it to $ENV_FILE first."
|
||||
echo " Generate at: https://github.com/settings/tokens"
|
||||
echo " Scopes needed: repo (full control)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Discover usernames
|
||||
info "Detecting GitHub user..."
|
||||
GITHUB_USER=$(_github_api GET /user | python3 -c "import sys,json; print(json.load(sys.stdin)['login'])" 2>/dev/null) \
|
||||
|| { err "Failed to reach GitHub API. Check GITHUB_TOKEN."; exit 1; }
|
||||
ok "GitHub user: $GITHUB_USER"
|
||||
|
||||
info "Detecting Gitea user..."
|
||||
GITEA_USER=$(_gitea_api GET /user | python3 -c "import sys,json; print(json.load(sys.stdin)['login'])" 2>/dev/null) \
|
||||
|| { err "Failed to reach Gitea API. Check GITEA_TOKEN and GITEA_URL ($GITEA_URL)."; exit 1; }
|
||||
ok "Gitea user: $GITEA_USER"
|
||||
|
||||
# Ask about sync scope
|
||||
echo ""
|
||||
read -rp "Pull private GitHub repos to Gitea? [Y/n] " ans
|
||||
PULL_PRIVATE=true; [[ "${ans,,}" == "n" ]] && PULL_PRIVATE=false
|
||||
|
||||
read -rp "Pull forked repos from GitHub? [y/N] " ans
|
||||
PULL_FORKS=false; [[ "${ans,,}" == "y" ]] && PULL_FORKS=true
|
||||
|
||||
read -rp "Push private Gitea repos to GitHub? [y/N] " ans
|
||||
PUSH_PRIVATE=false; [[ "${ans,,}" == "y" ]] && PUSH_PRIVATE=true
|
||||
|
||||
save_config
|
||||
|
||||
echo ""
|
||||
info "Run '$(basename "$0") --list' to preview what would sync."
|
||||
info "Run '$(basename "$0")' to sync now."
|
||||
info "Run '$(basename "$0") --install-timer' to sync automatically."
|
||||
}
|
||||
|
||||
# ── discover repos ─────────────────────────────────────────────────────────
|
||||
get_github_repos() {
|
||||
local page=1 repos=()
|
||||
while true; do
|
||||
local batch
|
||||
batch=$(_github_api GET "/user/repos?per_page=100&page=$page&affiliation=owner" \
|
||||
| python3 -c "
|
||||
import sys, json
|
||||
for r in json.load(sys.stdin):
|
||||
if r.get('fork') and not $($PULL_FORKS && echo True || echo False):
|
||||
continue
|
||||
if r.get('private') and not $($PULL_PRIVATE && echo True || echo False):
|
||||
continue
|
||||
print(r['full_name'] + '|' + r['clone_url'] + '|' + str(r.get('private',False)).lower())
|
||||
" 2>/dev/null) || break
|
||||
[[ -z "$batch" ]] && break
|
||||
while IFS= read -r line; do repos+=("$line"); done <<< "$batch"
|
||||
((page++))
|
||||
done
|
||||
printf '%s\n' "${repos[@]}"
|
||||
}
|
||||
|
||||
get_gitea_repos() {
|
||||
local page=1 repos=()
|
||||
while true; do
|
||||
local batch
|
||||
batch=$(_gitea_api GET "/repos/search?limit=50&page=$page" \
|
||||
| python3 -c "
|
||||
import sys, json
|
||||
for r in json.load(sys.stdin).get('data', []):
|
||||
if r.get('private') and not $($PUSH_PRIVATE && echo True || echo False):
|
||||
continue
|
||||
print(r['full_name'] + '|' + r['clone_url'] + '|' + str(r.get('private',False)).lower())
|
||||
" 2>/dev/null) || break
|
||||
[[ -z "$batch" ]] && break
|
||||
while IFS= read -r line; do repos+=("$line"); done <<< "$batch"
|
||||
((page++))
|
||||
done
|
||||
printf '%s\n' "${repos[@]}"
|
||||
}
|
||||
|
||||
is_excluded() {
|
||||
local repo="$1"
|
||||
for pat in "${EXCLUDE_REPOS[@]}"; do
|
||||
[[ "$repo" == $pat ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# ── sync: GitHub → Gitea (pull) ───────────────────────────────────────────
|
||||
sync_github_to_gitea() {
|
||||
local full_name="$1" clone_url="$2" is_private="$3"
|
||||
local repo_name="${full_name#*/}"
|
||||
local local_path="$WORK_DIR/$full_name"
|
||||
|
||||
# Clone or fetch from GitHub
|
||||
if [[ -d "$local_path" ]]; then
|
||||
info "Fetching $full_name from GitHub..."
|
||||
git -C "$local_path" fetch --all --prune --quiet 2>/dev/null || {
|
||||
err "Failed to fetch $full_name"; return 1; }
|
||||
else
|
||||
info "Cloning $full_name from GitHub..."
|
||||
mkdir -p "$(dirname "$local_path")"
|
||||
local auth_url="${clone_url/https:\/\//https:\/\/$GITHUB_TOKEN@}"
|
||||
git clone --bare --quiet "$auth_url" "$local_path" 2>/dev/null || {
|
||||
err "Failed to clone $full_name"; return 1; }
|
||||
fi
|
||||
|
||||
# Ensure repo exists on Gitea
|
||||
local gitea_check
|
||||
gitea_check=$(_gitea_api GET "/repos/$GITEA_USER/$repo_name" 2>/dev/null) || true
|
||||
if ! echo "$gitea_check" | python3 -c "import sys,json; json.load(sys.stdin)['id']" &>/dev/null; then
|
||||
info "Creating $repo_name on Gitea..."
|
||||
_gitea_api POST "/user/repos" \
|
||||
-d "{\"name\":\"$repo_name\",\"private\":$is_private,\"description\":\"Mirror of $full_name from GitHub\"}" \
|
||||
>/dev/null || { err "Failed to create $repo_name on Gitea"; return 1; }
|
||||
fi
|
||||
|
||||
# Push to Gitea
|
||||
local gitea_push_url="${GITEA_URL/https:\/\//https:\/\/$GITEA_USER:$GITEA_TOKEN@}"
|
||||
gitea_push_url="${gitea_push_url/http:\/\//http:\/\/$GITEA_USER:$GITEA_TOKEN@}"
|
||||
gitea_push_url="$gitea_push_url/$GITEA_USER/$repo_name.git"
|
||||
|
||||
git -C "$local_path" push --mirror "$gitea_push_url" --quiet 2>/dev/null || {
|
||||
err "Failed to push $full_name to Gitea"; return 1; }
|
||||
ok "GitHub → Gitea: $full_name"
|
||||
_log "PULL $full_name OK"
|
||||
}
|
||||
|
||||
# ── sync: Gitea → GitHub (push) ───────────────────────────────────────────
|
||||
sync_gitea_to_github() {
|
||||
local full_name="$1" clone_url="$2" is_private="$3"
|
||||
local repo_name="${full_name#*/}"
|
||||
local local_path="$WORK_DIR/gitea/$full_name"
|
||||
|
||||
# Clone or fetch from Gitea
|
||||
local gitea_auth_url="${clone_url/https:\/\//https:\/\/$GITEA_USER:$GITEA_TOKEN@}"
|
||||
gitea_auth_url="${gitea_auth_url/http:\/\//http:\/\/$GITEA_USER:$GITEA_TOKEN@}"
|
||||
|
||||
if [[ -d "$local_path" ]]; then
|
||||
info "Fetching $full_name from Gitea..."
|
||||
git -C "$local_path" fetch --all --prune --quiet 2>/dev/null || {
|
||||
err "Failed to fetch $full_name from Gitea"; return 1; }
|
||||
else
|
||||
info "Cloning $full_name from Gitea..."
|
||||
mkdir -p "$(dirname "$local_path")"
|
||||
git clone --bare --quiet "$gitea_auth_url" "$local_path" 2>/dev/null || {
|
||||
err "Failed to clone $full_name from Gitea"; return 1; }
|
||||
fi
|
||||
|
||||
# Ensure repo exists on GitHub
|
||||
local gh_check
|
||||
gh_check=$(_github_api GET "/repos/$GITHUB_USER/$repo_name" 2>/dev/null) || true
|
||||
if ! echo "$gh_check" | python3 -c "import sys,json; json.load(sys.stdin)['id']" &>/dev/null; then
|
||||
info "Creating $repo_name on GitHub..."
|
||||
_github_api POST "/user/repos" \
|
||||
-d "{\"name\":\"$repo_name\",\"private\":$is_private,\"description\":\"Mirror from Gitea\"}" \
|
||||
>/dev/null || { err "Failed to create $repo_name on GitHub"; return 1; }
|
||||
fi
|
||||
|
||||
# Push to GitHub
|
||||
local github_push_url="https://$GITHUB_TOKEN@github.com/$GITHUB_USER/$repo_name.git"
|
||||
git -C "$local_path" push --mirror "$github_push_url" --quiet 2>/dev/null || {
|
||||
err "Failed to push $full_name to GitHub"; return 1; }
|
||||
ok "Gitea → GitHub: $full_name"
|
||||
_log "PUSH $full_name OK"
|
||||
}
|
||||
|
||||
# ── list (dry run) ─────────────────────────────────────────────────────────
|
||||
do_list() {
|
||||
echo -e "\n${BOLD}Repos that would sync:${NC}\n"
|
||||
|
||||
if [[ ${#SYNC_REPOS[@]} -gt 0 ]]; then
|
||||
echo -e "${CYAN}Explicit list:${NC}"
|
||||
printf ' %s\n' "${SYNC_REPOS[@]}"
|
||||
else
|
||||
if [[ "$MODE" != "push" ]]; then
|
||||
echo -e "${CYAN}GitHub → Gitea (pull):${NC}"
|
||||
get_github_repos | while IFS='|' read -r name url priv; do
|
||||
is_excluded "$name" && echo " $name (excluded)" && continue
|
||||
echo " $name $([ "$priv" = "true" ] && echo "[private]")"
|
||||
done
|
||||
fi
|
||||
echo ""
|
||||
if [[ "$MODE" != "pull" ]]; then
|
||||
echo -e "${CYAN}Gitea → GitHub (push):${NC}"
|
||||
get_gitea_repos | while IFS='|' read -r name url priv; do
|
||||
is_excluded "$name" && echo " $name (excluded)" && continue
|
||||
echo " $name $([ "$priv" = "true" ] && echo "[private]")"
|
||||
done
|
||||
fi
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ── main sync ──────────────────────────────────────────────────────────────
|
||||
do_sync() {
|
||||
local pull_count=0 push_count=0 fail_count=0
|
||||
|
||||
_log "=== Sync started (mode=$MODE) ==="
|
||||
|
||||
# GitHub → Gitea
|
||||
if [[ "$MODE" == "all" || "$MODE" == "pull" ]]; then
|
||||
info "Discovering GitHub repos..."
|
||||
while IFS='|' read -r name url priv; do
|
||||
[[ -z "$name" ]] && continue
|
||||
[[ -n "$SINGLE_REPO" && "$name" != "$SINGLE_REPO" ]] && continue
|
||||
is_excluded "$name" && continue
|
||||
if sync_github_to_gitea "$name" "$url" "$priv"; then
|
||||
((pull_count++))
|
||||
else
|
||||
((fail_count++))
|
||||
fi
|
||||
done < <(get_github_repos)
|
||||
fi
|
||||
|
||||
# Gitea → GitHub
|
||||
if [[ "$MODE" == "all" || "$MODE" == "push" ]]; then
|
||||
info "Discovering Gitea repos..."
|
||||
while IFS='|' read -r name url priv; do
|
||||
[[ -z "$name" ]] && continue
|
||||
[[ -n "$SINGLE_REPO" && "${name#*/}" != "${SINGLE_REPO#*/}" ]] && continue
|
||||
is_excluded "$name" && continue
|
||||
# Skip repos that came from GitHub (already mirrored)
|
||||
local repo_name="${name#*/}"
|
||||
if [[ -d "$WORK_DIR/$GITHUB_USER/$repo_name" ]]; then
|
||||
info "Skipping $name (already a GitHub mirror)"
|
||||
continue
|
||||
fi
|
||||
if sync_gitea_to_github "$name" "$url" "$priv"; then
|
||||
((push_count++))
|
||||
else
|
||||
((fail_count++))
|
||||
fi
|
||||
done < <(get_gitea_repos)
|
||||
fi
|
||||
|
||||
echo ""
|
||||
ok "Sync complete: ${pull_count} pulled, ${push_count} pushed, ${fail_count} failed"
|
||||
_log "=== Sync complete: pull=$pull_count push=$push_count fail=$fail_count ==="
|
||||
}
|
||||
|
||||
# ── systemd timer ──────────────────────────────────────────────────────────
|
||||
install_timer() {
|
||||
local service_file="/etc/systemd/system/gitea-github-sync.service"
|
||||
local timer_file="/etc/systemd/system/gitea-github-sync.timer"
|
||||
local script_path
|
||||
script_path="$(readlink -f "$0")"
|
||||
|
||||
info "Installing systemd timer (interval: $TIMER_INTERVAL)..."
|
||||
|
||||
sudo tee "$service_file" > /dev/null << EOF
|
||||
[Unit]
|
||||
Description=Gitea-GitHub Mirror Sync
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=$USER
|
||||
ExecStart=$script_path
|
||||
Environment=HOME=$HOME
|
||||
StandardOutput=append:$LOG_FILE
|
||||
StandardError=append:$LOG_FILE
|
||||
EOF
|
||||
|
||||
sudo tee "$timer_file" > /dev/null << EOF
|
||||
[Unit]
|
||||
Description=Gitea-GitHub Sync Timer
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec=$TIMER_INTERVAL
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
EOF
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now gitea-github-sync.timer
|
||||
ok "Timer installed: every $TIMER_INTERVAL"
|
||||
ok "Check status: systemctl status gitea-github-sync.timer"
|
||||
ok "Run now: sudo systemctl start gitea-github-sync.service"
|
||||
ok "Logs: $LOG_FILE"
|
||||
}
|
||||
|
||||
remove_timer() {
|
||||
info "Removing systemd timer..."
|
||||
sudo systemctl disable --now gitea-github-sync.timer 2>/dev/null || true
|
||||
sudo rm -f /etc/systemd/system/gitea-github-sync.{service,timer}
|
||||
sudo systemctl daemon-reload
|
||||
ok "Timer removed"
|
||||
}
|
||||
|
||||
# ── preflight checks ──────────────────────────────────────────────────────
|
||||
preflight() {
|
||||
local ok=true
|
||||
if [[ -z "$GITEA_TOKEN" || "$GITEA_TOKEN" == "your-gitea-token-here" ]]; then
|
||||
err "GITEA_TOKEN not set. Edit $ENV_FILE"; ok=false
|
||||
fi
|
||||
if [[ -z "$GITHUB_TOKEN" || "$GITHUB_TOKEN" == "your-github-token-here" ]]; then
|
||||
err "GITHUB_TOKEN not set. Edit $ENV_FILE"; ok=false
|
||||
fi
|
||||
if [[ -z "$GITEA_USER" || -z "$GITHUB_USER" ]]; then
|
||||
err "Run --init first to configure usernames"; ok=false
|
||||
fi
|
||||
$ok || exit 1
|
||||
}
|
||||
|
||||
# ── main ───────────────────────────────────────────────────────────────────
|
||||
load_config
|
||||
|
||||
case "$MODE" in
|
||||
init) do_init ;;
|
||||
install-timer) install_timer ;;
|
||||
remove-timer) remove_timer ;;
|
||||
list) preflight; do_list ;;
|
||||
*) preflight; do_sync ;;
|
||||
esac
|
||||
Executable
+81
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bash
|
||||
# Import a LoRA (.safetensors) file into InvokeAI's Docker volume
|
||||
# Usage: ./invokeai-import-lora.sh /path/to/my-lora.safetensors
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo -e "${BOLD}Usage:${NC} $0 <lora-file.safetensors> [display-name]"
|
||||
echo ""
|
||||
echo " Copies a LoRA file into InvokeAI's model volume so it appears"
|
||||
echo " in the Model Manager automatically."
|
||||
echo ""
|
||||
echo " Examples:"
|
||||
echo " $0 ~/Downloads/my-character-lora.safetensors"
|
||||
echo " $0 ~/Downloads/my-character-lora.safetensors \"My Character\""
|
||||
exit 1
|
||||
fi
|
||||
|
||||
LORA_FILE="$1"
|
||||
DISPLAY_NAME="${2:-$(basename "$LORA_FILE" .safetensors)}"
|
||||
|
||||
# Validate file exists
|
||||
if [[ ! -f "$LORA_FILE" ]]; then
|
||||
echo -e "${RED}Error:${NC} File not found: $LORA_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate file extension
|
||||
if [[ "$LORA_FILE" != *.safetensors && "$LORA_FILE" != *.ckpt && "$LORA_FILE" != *.pt ]]; then
|
||||
echo -e "${YELLOW}Warning:${NC} File doesn't have a typical LoRA extension (.safetensors, .ckpt, .pt)"
|
||||
read -rp "Continue anyway? [y/N] " yn
|
||||
[[ "$yn" != [yY]* ]] && exit 1
|
||||
fi
|
||||
|
||||
# Check if InvokeAI container exists
|
||||
if ! docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q '^invokeai$'; then
|
||||
echo -e "${RED}Error:${NC} InvokeAI container not found. Is the AI stack running?"
|
||||
echo " Try: bash ~/docker/ai-stack/start.sh"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if container is running
|
||||
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^invokeai$'; then
|
||||
echo -e "${YELLOW}InvokeAI container is stopped. Starting it...${NC}"
|
||||
docker start invokeai
|
||||
sleep 3
|
||||
fi
|
||||
|
||||
FILENAME=$(basename "$LORA_FILE")
|
||||
|
||||
echo -e "${CYAN}[..]${NC} Copying ${BOLD}$FILENAME${NC} into InvokeAI..."
|
||||
|
||||
# Copy the LoRA file into the container's models directory
|
||||
# InvokeAI looks for LoRA files in /invokeai/models/lora/
|
||||
docker exec invokeai mkdir -p /invokeai/models/lora
|
||||
docker cp "$LORA_FILE" "invokeai:/invokeai/models/lora/$FILENAME"
|
||||
|
||||
echo -e "${GREEN}[OK]${NC} LoRA file copied successfully!"
|
||||
echo ""
|
||||
echo -e "${BOLD}Next steps in InvokeAI (http://localhost:9090):${NC}"
|
||||
echo ""
|
||||
echo " 1. Open the ${BOLD}Model Manager${NC} (cube icon in the left sidebar)"
|
||||
echo " 2. Click ${BOLD}\"Scan for Models\"${NC} or ${BOLD}\"Sync Models\"${NC} button"
|
||||
echo " - Your LoRA '${DISPLAY_NAME}' should appear in the list"
|
||||
echo " 3. If it doesn't auto-detect, click ${BOLD}\"Add Model\" > \"Scan Folder\"${NC}"
|
||||
echo " and enter: ${BOLD}/invokeai/models/lora${NC}"
|
||||
echo ""
|
||||
echo -e "${BOLD}To use the LoRA when generating images:${NC}"
|
||||
echo ""
|
||||
echo " 1. Go to the ${BOLD}Text to Image${NC} or ${BOLD}Image to Image${NC} tab"
|
||||
echo " 2. In the left panel, find the ${BOLD}\"LoRA\"${NC} section"
|
||||
echo " (expand it if collapsed — it's below the main model selector)"
|
||||
echo " 3. Click ${BOLD}\"+\"${NC} to add your LoRA from the dropdown"
|
||||
echo " 4. Adjust the ${BOLD}weight${NC} slider (start with 0.7–0.85)"
|
||||
echo " 5. Make sure your ${BOLD}base model${NC} matches what the LoRA was trained on"
|
||||
echo " (e.g., if trained on SD 1.5, select a SD 1.5 checkpoint)"
|
||||
echo ""
|
||||
echo -e "${YELLOW}Tip:${NC} If the LoRA was trained on SD 1.5, you MUST use an SD 1.5"
|
||||
echo " base model — it won't work with SDXL or other architectures."
|
||||
@@ -0,0 +1,218 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# Kiwix ZIM Download Script
|
||||
# Downloads all ZIM files to ~/docker/ai-stack/kiwix/
|
||||
# Finds latest version of each file automatically
|
||||
# Usage: bash kiwix-download.sh
|
||||
# =============================================================================
|
||||
|
||||
KIWIX_DIR="$HOME/docker/ai-stack/kiwix"
|
||||
MIRROR="https://ftp.fau.de/kiwix/zim"
|
||||
MIRROR2="https://download.kiwix.org/zim" # fallback for files missing on fau.de
|
||||
LOG="$HOME/docker/ai-stack/logs/kiwix-download.log"
|
||||
mkdir -p "$KIWIX_DIR" "$(dirname "$LOG")"
|
||||
|
||||
G='\033[0;32m'; Y='\033[1;33m'; C='\033[0;36m'; N='\033[0m'
|
||||
ok() { echo -e "${G}[OK]${N} $1" | tee -a "$LOG"; }
|
||||
inf() { echo -e "${C}[..]${N} $1" | tee -a "$LOG"; }
|
||||
wrn() { echo -e "${Y}[!!]${N} $1" | tee -a "$LOG"; }
|
||||
|
||||
# Find latest version of a ZIM file from mirror
|
||||
# Usage: latest_zim "base_url" "pattern"
|
||||
latest_zim() {
|
||||
local base_url="$1"
|
||||
local pattern="$2"
|
||||
curl -s "$base_url/" | \
|
||||
grep -oP "${pattern}_[\d-]+\.zim" | \
|
||||
sort -u | tail -1
|
||||
}
|
||||
|
||||
# Download a ZIM file if not already present
|
||||
# Usage: download_zim "category" "filename" "description" "size" ["mirror"]
|
||||
download_zim() {
|
||||
local category="$1"
|
||||
local filename="$2"
|
||||
local description="$3"
|
||||
local size="$4"
|
||||
local mirror="${5:-$MIRROR}"
|
||||
local url="$mirror/$category/$filename"
|
||||
local dest="$KIWIX_DIR/$filename"
|
||||
|
||||
if [[ -f "$dest" ]]; then
|
||||
ok "$description already downloaded — skipping"
|
||||
return
|
||||
fi
|
||||
|
||||
inf "Queuing: $description (~$size)"
|
||||
inf " URL: $url"
|
||||
|
||||
nohup wget -c "$url" -O "$dest" \
|
||||
>> "$LOG" 2>&1 &
|
||||
echo $! >> "$KIWIX_DIR/.download_pids"
|
||||
ok "Started download PID $! — $description"
|
||||
}
|
||||
|
||||
echo "" | tee -a "$LOG"
|
||||
echo "=== Kiwix Download Started: $(date) ===" | tee -a "$LOG"
|
||||
echo "" | tee -a "$LOG"
|
||||
echo -e "${C}Finding latest versions...${N}"
|
||||
echo ""
|
||||
|
||||
# ── Find latest filenames ──────────────────────────────────────────────────
|
||||
inf "Checking Wikipedia..."
|
||||
WIKI=$(latest_zim "$MIRROR/wikipedia" "wikipedia_en_all_nopic")
|
||||
|
||||
inf "Checking Wiktionary..."
|
||||
WIKT=$(latest_zim "$MIRROR/wiktionary" "wiktionary_en_all_nopic")
|
||||
|
||||
inf "Checking Wikiquote..."
|
||||
WIKQ=$(latest_zim "$MIRROR/wikiquote" "wikiquote_en_all_nopic")
|
||||
|
||||
inf "Checking Wikisource..."
|
||||
WIKS=$(latest_zim "$MIRROR/wikisource" "wikisource_en_all_nopic")
|
||||
|
||||
inf "Checking Wikibooks..."
|
||||
WIKB=$(latest_zim "$MIRROR/wikibooks" "wikibooks_en_all_nopic")
|
||||
|
||||
inf "Checking Wikivoyage..."
|
||||
WIKV=$(latest_zim "$MIRROR/wikivoyage" "wikivoyage_en_all_nopic")
|
||||
|
||||
inf "Checking Wikiversity..."
|
||||
WIKUNI=$(latest_zim "$MIRROR/wikiversity" "wikiversity_en_all_nopic")
|
||||
|
||||
inf "Checking WikiNews..."
|
||||
WIKNEWS=$(latest_zim "$MIRROR/wikinews" "wikinews_en_all_nopic")
|
||||
|
||||
inf "Checking Vikidia (kids K-8)..."
|
||||
VIKIDIA=$(latest_zim "$MIRROR/vikidia" "vikidia_en_all_nopic")
|
||||
|
||||
# Stack Exchange sites: domain-style filenames in stack_exchange/
|
||||
inf "Checking Stack Overflow..."
|
||||
SO=$(latest_zim "$MIRROR/stack_exchange" "stackoverflow.com_en_all")
|
||||
|
||||
inf "Checking Ask Ubuntu..."
|
||||
ASKUBUNTU=$(latest_zim "$MIRROR/stack_exchange" "askubuntu.com_en_all")
|
||||
|
||||
inf "Checking Super User..."
|
||||
SUPERUSER=$(latest_zim "$MIRROR/stack_exchange" "superuser.com_en_all")
|
||||
|
||||
inf "Checking Unix & Linux SE..."
|
||||
UNIX=$(latest_zim "$MIRROR/stack_exchange" "unix.stackexchange.com_en_all")
|
||||
|
||||
inf "Checking Server Fault..."
|
||||
SERVERFAULT=$(latest_zim "$MIRROR/stack_exchange" "serverfault.com_en_all")
|
||||
|
||||
# Arch Wiki: _maxi is part of the base name, not the date stamp
|
||||
inf "Checking Arch Wiki..."
|
||||
ARCH=$(latest_zim "$MIRROR/other" "archlinux_en_all_maxi")
|
||||
|
||||
# TED: lives in ted/ folder, mul_youth variant
|
||||
inf "Checking TED Talks..."
|
||||
TED=$(latest_zim "$MIRROR/ted" "ted_mul_youth")
|
||||
|
||||
# PhET: own folder
|
||||
inf "Checking PhET Simulations..."
|
||||
PHET=$(latest_zim "$MIRROR/phet" "phet_en_all")
|
||||
|
||||
# DevDocs: variant is zig not all
|
||||
inf "Checking DevDocs..."
|
||||
DEVDOCS=$(latest_zim "$MIRROR/devdocs" "devdocs_en_zig")
|
||||
|
||||
inf "Checking FreeCodeCamp..."
|
||||
FCC=$(latest_zim "$MIRROR/freecodecamp" "freecodecamp_en_all")
|
||||
|
||||
inf "Checking iFixit..."
|
||||
IFIX=$(latest_zim "$MIRROR/ifixit" "ifixit_en_all")
|
||||
|
||||
# LibreTexts: own folder, workforce variant (no _all on this mirror)
|
||||
inf "Checking LibreTexts..."
|
||||
LIBRE=$(latest_zim "$MIRROR/libretexts" "libretexts.org_en_workforce")
|
||||
|
||||
# Gutenberg: en_all exists on download.kiwix.org, not ftp.fau.de
|
||||
inf "Checking Project Gutenberg..."
|
||||
GUT=$(latest_zim "$MIRROR2/gutenberg" "gutenberg_en_all")
|
||||
|
||||
echo ""
|
||||
echo -e "${C}═══════════════════════════════════════════════════${N}"
|
||||
echo -e "${C} Download Queue${N}"
|
||||
echo -e "${C}═══════════════════════════════════════════════════${N}"
|
||||
echo ""
|
||||
echo " Wikipedia (no images) ${WIKI:-NOT FOUND} ~46GB"
|
||||
echo " Wiktionary ${WIKT:-NOT FOUND} ~2GB"
|
||||
echo " Wikiquote ${WIKQ:-NOT FOUND} ~300MB"
|
||||
echo " Wikisource ${WIKS:-NOT FOUND} ~4GB"
|
||||
echo " Wikibooks ${WIKB:-NOT FOUND} ~500MB"
|
||||
echo " Wikivoyage ${WIKV:-NOT FOUND} ~200MB"
|
||||
echo " Wikiversity ${WIKUNI:-NOT FOUND} ~500MB"
|
||||
echo " WikiNews ${WIKNEWS:-NOT FOUND} ~300MB"
|
||||
echo " Vikidia (kids K-8) ${VIKIDIA:-NOT FOUND} ~66MB"
|
||||
echo " Stack Overflow ${SO:-NOT FOUND} ~3GB"
|
||||
echo " Ask Ubuntu ${ASKUBUNTU:-NOT FOUND} ~1GB"
|
||||
echo " Super User ${SUPERUSER:-NOT FOUND} ~1GB"
|
||||
echo " Unix & Linux SE ${UNIX:-NOT FOUND} ~500MB"
|
||||
echo " Server Fault ${SERVERFAULT:-NOT FOUND} ~500MB"
|
||||
echo " Arch Linux Wiki ${ARCH:-NOT FOUND} ~30MB"
|
||||
echo " TED Talks ${TED:-NOT FOUND} ~5GB"
|
||||
echo " PhET Simulations ${PHET:-NOT FOUND} ~500MB"
|
||||
echo " DevDocs ${DEVDOCS:-NOT FOUND} ~1GB"
|
||||
echo " FreeCodeCamp ${FCC:-NOT FOUND} ~small"
|
||||
echo " iFixit (repair guides) ${IFIX:-NOT FOUND} ~2GB"
|
||||
echo " LibreTexts (textbooks) ${LIBRE:-NOT FOUND} ~varies"
|
||||
echo " Project Gutenberg ${GUT:-NOT FOUND} ~60GB"
|
||||
echo ""
|
||||
echo -e "${Y} Total estimate: ~133GB — make sure you have space!${N}"
|
||||
echo ""
|
||||
|
||||
df -h "$KIWIX_DIR" | tail -1 | awk '{print " Available disk space: " $4}'
|
||||
echo ""
|
||||
|
||||
read -rp " Proceed with all downloads? (Y/n): " CONFIRM
|
||||
[[ "${CONFIRM,,}" == "n" ]] && exit 0
|
||||
|
||||
# ── Start downloads ────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
rm -f "$KIWIX_DIR/.download_pids"
|
||||
|
||||
[[ -n "$WIKI" ]] && download_zim "wikipedia" "$WIKI" "Wikipedia (no images)" "46GB"
|
||||
[[ -n "$WIKT" ]] && download_zim "wiktionary" "$WIKT" "Wiktionary" "2GB"
|
||||
[[ -n "$WIKQ" ]] && download_zim "wikiquote" "$WIKQ" "Wikiquote" "300MB"
|
||||
[[ -n "$WIKS" ]] && download_zim "wikisource" "$WIKS" "Wikisource" "4GB"
|
||||
[[ -n "$WIKB" ]] && download_zim "wikibooks" "$WIKB" "Wikibooks" "500MB"
|
||||
[[ -n "$WIKV" ]] && download_zim "wikivoyage" "$WIKV" "Wikivoyage" "200MB"
|
||||
[[ -n "$WIKUNI" ]] && download_zim "wikiversity" "$WIKUNI" "Wikiversity" "500MB"
|
||||
[[ -n "$WIKNEWS" ]] && download_zim "wikinews" "$WIKNEWS" "WikiNews" "300MB"
|
||||
[[ -n "$VIKIDIA" ]] && download_zim "vikidia" "$VIKIDIA" "Vikidia (kids K-8)" "66MB"
|
||||
[[ -n "$SO" ]] && download_zim "stack_exchange" "$SO" "Stack Overflow" "3GB"
|
||||
[[ -n "$ASKUBUNTU" ]] && download_zim "stack_exchange" "$ASKUBUNTU" "Ask Ubuntu" "1GB"
|
||||
[[ -n "$SUPERUSER" ]] && download_zim "stack_exchange" "$SUPERUSER" "Super User" "1GB"
|
||||
[[ -n "$UNIX" ]] && download_zim "stack_exchange" "$UNIX" "Unix & Linux SE" "500MB"
|
||||
[[ -n "$SERVERFAULT" ]] && download_zim "stack_exchange" "$SERVERFAULT" "Server Fault" "500MB"
|
||||
[[ -n "$ARCH" ]] && download_zim "other" "$ARCH" "Arch Linux Wiki" "30MB"
|
||||
[[ -n "$TED" ]] && download_zim "ted" "$TED" "TED Talks" "5GB"
|
||||
[[ -n "$PHET" ]] && download_zim "phet" "$PHET" "PhET Simulations" "500MB"
|
||||
[[ -n "$DEVDOCS" ]] && download_zim "devdocs" "$DEVDOCS" "DevDocs" "1GB"
|
||||
[[ -n "$FCC" ]] && download_zim "freecodecamp" "$FCC" "FreeCodeCamp" "small"
|
||||
[[ -n "$IFIX" ]] && download_zim "ifixit" "$IFIX" "iFixit (repair guides)" "2GB"
|
||||
[[ -n "$LIBRE" ]] && download_zim "libretexts" "$LIBRE" "LibreTexts (textbooks)" "varies"
|
||||
[[ -n "$GUT" ]] && download_zim "gutenberg" "$GUT" "Project Gutenberg" "60GB" "$MIRROR2"
|
||||
|
||||
echo ""
|
||||
echo -e "${G}════════════════════════════════════════════════════${N}"
|
||||
echo -e "${G} All downloads started in background${N}"
|
||||
echo -e "${G}════════════════════════════════════════════════════${N}"
|
||||
echo ""
|
||||
echo " Monitor progress:"
|
||||
echo " tail -f $LOG"
|
||||
echo ""
|
||||
echo " Check file sizes growing:"
|
||||
echo " watch -n 60 'ls -lh $KIWIX_DIR/'"
|
||||
echo ""
|
||||
echo " Check active downloads:"
|
||||
echo " jobs -l"
|
||||
echo " ps aux | grep wget"
|
||||
echo ""
|
||||
echo " Once all downloads complete, restart Kiwix:"
|
||||
echo " cd ~/docker/ai-stack"
|
||||
echo " docker compose up -d kiwix"
|
||||
echo ""
|
||||
echo -e "${Y} NOTE: Downloads resume automatically if interrupted (-c flag)${N}"
|
||||
Executable
+130
@@ -0,0 +1,130 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# Kiwix ZIM Auto-Updater
|
||||
# Checks each ZIM for a newer version on the mirror, downloads it, removes old.
|
||||
# Safe to run as a cron job — only acts when a newer version exists.
|
||||
# Usage: bash kiwix_update.sh
|
||||
# Cron example (monthly): 0 3 1 * * /bin/bash /path/to/kiwix_update.sh
|
||||
# =============================================================================
|
||||
|
||||
KIWIX_DIR="$HOME/docker/ai-stack/kiwix"
|
||||
MIRROR="https://ftp.fau.de/kiwix/zim"
|
||||
MIRROR2="https://download.kiwix.org/zim"
|
||||
LOG="$HOME/docker/ai-stack/logs/kiwix-update.log"
|
||||
mkdir -p "$KIWIX_DIR" "$(dirname "$LOG")"
|
||||
|
||||
G='\033[0;32m'; Y='\033[1;33m'; C='\033[0;36m'; R='\033[0;31m'; N='\033[0m'
|
||||
ok() { echo -e "${G}[OK]${N} $1" | tee -a "$LOG"; }
|
||||
inf() { echo -e "${C}[..]${N} $1" | tee -a "$LOG"; }
|
||||
wrn() { echo -e "${Y}[!!]${N} $1" | tee -a "$LOG"; }
|
||||
err() { echo -e "${R}[XX]${N} $1" | tee -a "$LOG"; }
|
||||
|
||||
echo "" | tee -a "$LOG"
|
||||
echo "=== Kiwix Update Check: $(date) ===" | tee -a "$LOG"
|
||||
echo "" | tee -a "$LOG"
|
||||
|
||||
# Find latest filename on mirror
|
||||
latest_zim() {
|
||||
local base_url="$1"
|
||||
local pattern="$2"
|
||||
curl -s "$base_url/" | \
|
||||
grep -oP "${pattern}_[\d-]+\.zim" | \
|
||||
sort -u | tail -1
|
||||
}
|
||||
|
||||
# Find what's currently on disk matching a pattern
|
||||
current_zim() {
|
||||
local pattern="$1"
|
||||
ls "$KIWIX_DIR"/${pattern}_*.zim 2>/dev/null | sort | tail -1 | xargs basename 2>/dev/null
|
||||
}
|
||||
|
||||
# Check and update one ZIM
|
||||
# Usage: check_update "category" "pattern" "description" ["mirror"]
|
||||
UPDATED=0
|
||||
check_update() {
|
||||
local category="$1"
|
||||
local pattern="$2"
|
||||
local description="$3"
|
||||
local mirror="${4:-$MIRROR}"
|
||||
|
||||
inf "Checking $description..."
|
||||
local latest
|
||||
latest=$(latest_zim "$mirror/$category" "$pattern")
|
||||
|
||||
if [[ -z "$latest" ]]; then
|
||||
wrn "$description: could not find latest on mirror — skipping"
|
||||
return
|
||||
fi
|
||||
|
||||
local current
|
||||
current=$(current_zim "$pattern")
|
||||
|
||||
if [[ -z "$current" ]]; then
|
||||
inf "$description: not downloaded yet — skipping (run kiwix_download.sh first)"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$latest" == "$current" ]]; then
|
||||
ok "$description: up to date ($current)"
|
||||
return
|
||||
fi
|
||||
|
||||
echo ""
|
||||
inf "$description: update available"
|
||||
inf " Current: $current"
|
||||
inf " Latest: $latest"
|
||||
|
||||
local url="$mirror/$category/$latest"
|
||||
local dest="$KIWIX_DIR/$latest"
|
||||
local old="$KIWIX_DIR/$current"
|
||||
|
||||
inf " Downloading $latest..."
|
||||
if wget -q --show-progress -c "$url" -O "$dest" >> "$LOG" 2>&1; then
|
||||
ok " Download complete — removing old file"
|
||||
rm -f "$old"
|
||||
UPDATED=$((UPDATED + 1))
|
||||
else
|
||||
err " Download failed — keeping old file"
|
||||
rm -f "$dest"
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ── Check each ZIM ─────────────────────────────────────────────────────────
|
||||
check_update "wikipedia" "wikipedia_en_all_nopic" "Wikipedia"
|
||||
check_update "wiktionary" "wiktionary_en_all_nopic" "Wiktionary"
|
||||
check_update "wikiquote" "wikiquote_en_all_nopic" "Wikiquote"
|
||||
check_update "wikisource" "wikisource_en_all_nopic" "Wikisource"
|
||||
check_update "wikibooks" "wikibooks_en_all_nopic" "Wikibooks"
|
||||
check_update "wikivoyage" "wikivoyage_en_all_nopic" "Wikivoyage"
|
||||
check_update "wikiversity" "wikiversity_en_all_nopic" "Wikiversity"
|
||||
check_update "wikinews" "wikinews_en_all_nopic" "WikiNews"
|
||||
check_update "vikidia" "vikidia_en_all_nopic" "Vikidia"
|
||||
check_update "stack_exchange" "stackoverflow.com_en_all" "Stack Overflow"
|
||||
check_update "stack_exchange" "askubuntu.com_en_all" "Ask Ubuntu"
|
||||
check_update "stack_exchange" "superuser.com_en_all" "Super User"
|
||||
check_update "stack_exchange" "unix.stackexchange.com_en_all" "Unix & Linux SE"
|
||||
check_update "stack_exchange" "serverfault.com_en_all" "Server Fault"
|
||||
check_update "other" "archlinux_en_all_maxi" "Arch Wiki"
|
||||
check_update "ted" "ted_mul_youth" "TED Talks"
|
||||
check_update "phet" "phet_en_all" "PhET Simulations"
|
||||
check_update "devdocs" "devdocs_en_zig" "DevDocs"
|
||||
check_update "freecodecamp" "freecodecamp_en_all" "FreeCodeCamp"
|
||||
check_update "ifixit" "ifixit_en_all" "iFixit"
|
||||
check_update "libretexts" "libretexts.org_en_workforce" "LibreTexts"
|
||||
check_update "gutenberg" "gutenberg_en_all" "Project Gutenberg" "$MIRROR2"
|
||||
|
||||
# ── Restart Kiwix if anything changed ─────────────────────────────────────
|
||||
echo ""
|
||||
if [[ $UPDATED -gt 0 ]]; then
|
||||
ok "$UPDATED ZIM(s) updated — restarting Kiwix container"
|
||||
docker compose -f "$HOME/docker/ai-stack/docker-compose.yml" restart kiwix >> "$LOG" 2>&1 \
|
||||
&& ok "Kiwix restarted" \
|
||||
|| wrn "Could not restart Kiwix — do it manually: docker compose restart kiwix"
|
||||
else
|
||||
ok "All ZIMs are up to date — no restart needed"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Update check complete: $(date) ===" | tee -a "$LOG"
|
||||
echo ""
|
||||
Executable
+1689
File diff suppressed because it is too large
Load Diff
Executable
+986
@@ -0,0 +1,986 @@
|
||||
#!/usr/bin/env bash
|
||||
# Local AI Stack — single script, new install or update
|
||||
# Usage: ./local-ai-setup.sh [--force] [--no-pull]
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[..]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[!!]${NC} $*"; }
|
||||
section() { echo -e "\n${BOLD}━━━ $* ━━━${NC}"; }
|
||||
|
||||
FORCE=false; NO_PULL=false
|
||||
for a in "$@"; do [[ "$a" == "--force" ]] && FORCE=true; [[ "$a" == "--no-pull" ]] && NO_PULL=true; done
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
BASE="$SCRIPT_DIR"
|
||||
LOCAL_IP=$(ip route get 1.1.1.1 2>/dev/null | grep -oP 'src \K\S+' || hostname -I | awk '{print $1}')
|
||||
[[ -z "$LOCAL_IP" ]] && read -rp "Enter LAN IP: " LOCAL_IP
|
||||
|
||||
IS_UPDATE=false; [[ -f "$BASE/docker-compose.yml" ]] && IS_UPDATE=true
|
||||
|
||||
# ── detect VRAM and set models accordingly ────────────────────────────────────
|
||||
VRAM_GB=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null \
|
||||
| head -1 | awk '{printf "%d", $1/1024}' 2>/dev/null || echo "0")
|
||||
GPU_COUNT=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | wc -l || echo "0")
|
||||
TOTAL_VRAM=$((VRAM_GB * GPU_COUNT))
|
||||
|
||||
# Ollama optimization flags (stacked — see docs/gpu-setup-research.md)
|
||||
OLLAMA_KV_CACHE="q8_0" # halves KV cache VRAM (q4_0 for aggressive)
|
||||
OLLAMA_FLASH="1" # flash attention: less VRAM, no quality loss
|
||||
|
||||
if [[ "$TOTAL_VRAM" -ge 40 ]]; then
|
||||
CHAT_MODEL="qwen3.5:27b"; CODE_MODEL="qwen3.5:27b"
|
||||
CTX=131072; TIER="${TOTAL_VRAM}GB — 27B dense, 128K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 28 ]]; then
|
||||
CHAT_MODEL="qwen3.5-35b-a3b"; CODE_MODEL="qwen3.5-35b-a3b"
|
||||
CTX=131072; TIER="${TOTAL_VRAM}GB — 35B MoE, 128K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 14 ]]; then
|
||||
CHAT_MODEL="qwen3.5-35b-a3b"; CODE_MODEL="qwen3.5-35b-a3b"
|
||||
CTX=65536; TIER="${TOTAL_VRAM}GB — 35B MoE + KV quant, 64K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 8 ]]; then
|
||||
CHAT_MODEL="qwen3.5:9b"; CODE_MODEL="qwen3.5:9b"
|
||||
CTX=32768; TIER="${TOTAL_VRAM}GB — 9B dense, 32K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 6 ]]; then
|
||||
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
|
||||
CTX=32768; TIER="${TOTAL_VRAM}GB — 4B + KV quant, 32K context"
|
||||
elif [[ "$TOTAL_VRAM" -ge 4 ]]; then
|
||||
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
|
||||
CTX=16384; TIER="${TOTAL_VRAM}GB — 4B models, 16K context"
|
||||
else
|
||||
CHAT_MODEL="qwen3.5:4b"; CODE_MODEL="qwen3.5:4b"
|
||||
CTX=4096; OLLAMA_KV_CACHE="q4_0"; TIER="CPU-only — 4B models, 4K context"
|
||||
fi
|
||||
EMBED_MODEL="nomic-embed-text"
|
||||
|
||||
# ── Image generation model tiers (VRAM-aware) ────────────────────────────────
|
||||
# These vars are used by setup-image-models.sh and printed in status output.
|
||||
# Image gen shares GPU with Ollama — Ollama unloads after KEEP_ALIVE timeout,
|
||||
# so image gen gets full VRAM when Ollama is idle.
|
||||
if [[ "$TOTAL_VRAM" -ge 24 ]]; then
|
||||
IMG_MODELS="SD 1.5, SDXL, SDXL Turbo, Flux.1-dev, Flux.1-schnell"
|
||||
IMG_TIER="all models including Flux"
|
||||
IMG_DEFAULT="SDXL"
|
||||
elif [[ "$TOTAL_VRAM" -ge 12 ]]; then
|
||||
IMG_MODELS="SD 1.5, SDXL, SDXL Turbo, Flux.1-schnell (tight)"
|
||||
IMG_TIER="SDXL + Flux-schnell"
|
||||
IMG_DEFAULT="SDXL"
|
||||
elif [[ "$TOTAL_VRAM" -ge 8 ]]; then
|
||||
IMG_MODELS="SD 1.5, SDXL (tight at 512px), SDXL Turbo"
|
||||
IMG_TIER="SD 1.5 comfortable, SDXL possible"
|
||||
IMG_DEFAULT="SD 1.5"
|
||||
elif [[ "$TOTAL_VRAM" -ge 4 ]]; then
|
||||
IMG_MODELS="SD 1.5 (float16)"
|
||||
IMG_TIER="SD 1.5 only"
|
||||
IMG_DEFAULT="SD 1.5"
|
||||
else
|
||||
IMG_MODELS="none (CPU generation extremely slow)"
|
||||
IMG_TIER="CPU only — not recommended"
|
||||
IMG_DEFAULT=""
|
||||
fi
|
||||
|
||||
# ── InvokeAI precision (GPU-aware) ───────────────────────────────────────────
|
||||
GPU_COMPUTE=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader 2>/dev/null \
|
||||
| head -1 | tr -d '.' || echo "0")
|
||||
if [[ "$GPU_COMPUTE" -ge 80 ]]; then
|
||||
INVOKEAI_PRECISION="bfloat16" # Ampere+ (RTX 30xx/40xx/50xx, A-series)
|
||||
elif [[ "$GPU_COMPUTE" -ge 60 ]]; then
|
||||
INVOKEAI_PRECISION="float16" # Pascal+ (GTX 10xx, RTX 20xx, Tesla P40/V100)
|
||||
else
|
||||
INVOKEAI_PRECISION="auto" # Let InvokeAI decide
|
||||
fi
|
||||
|
||||
section "Local AI Stack — $($IS_UPDATE && echo UPDATE || echo NEW INSTALL)"
|
||||
info "Base : $BASE"
|
||||
info "IP : $LOCAL_IP"
|
||||
info "GPU : ${VRAM_GB}GB VRAM → $TIER"
|
||||
info "Image : $IMG_TIER ($IMG_MODELS)"
|
||||
|
||||
|
||||
write_if_new() {
|
||||
local dest="$1"; local body; body=$(cat)
|
||||
if [[ ! -f "$dest" ]] || $FORCE; then
|
||||
printf '%s\n' "$body" > "$dest"; ok "Wrote $(basename "$dest")"
|
||||
else
|
||||
info "Kept $(basename "$dest") (--force to overwrite)"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── prereqs (new install only) ────────────────────────────────────────────────
|
||||
if ! $IS_UPDATE; then
|
||||
section "Prerequisites"
|
||||
if ! command -v docker &>/dev/null; then
|
||||
info "Installing Docker..."
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
sudo usermod -aG docker "$USER"
|
||||
warn "Run: newgrp docker (or log out/in)"
|
||||
else
|
||||
ok "Docker: $(docker --version | cut -d' ' -f3)"
|
||||
fi
|
||||
|
||||
if command -v nvidia-smi &>/dev/null && ! dpkg -l 2>/dev/null | grep -q nvidia-container-toolkit; then
|
||||
info "Installing NVIDIA Container Toolkit..."
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
||||
| sudo gpg --dearmor --yes -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
||||
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
|
||||
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||
sudo apt-get update -qq && sudo apt-get install -y nvidia-container-toolkit
|
||||
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
|
||||
ok "NVIDIA Container Toolkit installed"
|
||||
fi
|
||||
command -v rg &>/dev/null || sudo apt-get install -y ripgrep
|
||||
fi
|
||||
|
||||
# ── directories ───────────────────────────────────────────────────────────────
|
||||
section "Directories"
|
||||
for d in papers repos workspace index invokeai-data invokeai-outputs comfyui-data comfyui-output kiwix gitea portainer-data logs; do
|
||||
mkdir -p "$BASE/$d"
|
||||
done
|
||||
ok "Ready under $BASE"
|
||||
|
||||
# =============================================================================
|
||||
section "Writing server.py (RAG)"
|
||||
# =============================================================================
|
||||
write_if_new "$BASE/server.py" << 'PY'
|
||||
import ast, fnmatch, hashlib, json, logging, os, re, subprocess, threading, time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
import chromadb, httpx
|
||||
from chromadb.utils.embedding_functions import OllamaEmbeddingFunction
|
||||
from fastapi import FastAPI, HTTPException, Request, BackgroundTasks
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
log = logging.getLogger("rag")
|
||||
|
||||
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
|
||||
CHROMA_URL = os.getenv("CHROMA_URL", "http://chromadb:8000")
|
||||
EMBED_MODEL = os.getenv("EMBED_MODEL", "nomic-embed-text")
|
||||
CHAT_MODEL = os.getenv("CHAT_MODEL", "qwen2.5:14b")
|
||||
PAPERS_DIR = Path(os.getenv("PAPERS_DIR", "/papers"))
|
||||
REPOS_DIR = Path(os.getenv("REPOS_DIR", "/repos"))
|
||||
TOP_K = int(os.getenv("TOP_K", "6"))
|
||||
|
||||
CODE_EXTS = {".py",".js",".ts",".tsx",".jsx",".go",".rs",".java",".c",".cpp",
|
||||
".h",".cs",".rb",".sh",".yaml",".yml",".toml",".sql",".md"}
|
||||
SKIP_DIRS = {"node_modules",".git","__pycache__","dist","build",".venv","venv","target"}
|
||||
MAX_BYTES = 400_000
|
||||
|
||||
app = FastAPI(title="RAG Server")
|
||||
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
|
||||
|
||||
def _embed_fn():
|
||||
return OllamaEmbeddingFunction(url=f"{OLLAMA_URL}/api/embeddings", model_name=EMBED_MODEL)
|
||||
|
||||
def _chroma():
|
||||
host, port = CHROMA_URL.replace("http://","").split(":")
|
||||
return chromadb.HttpClient(host=host, port=int(port))
|
||||
|
||||
def get_col(name):
|
||||
return _chroma().get_or_create_collection(name, embedding_function=_embed_fn())
|
||||
|
||||
def _doc_id(text, key):
|
||||
return hashlib.md5(f"{key}|{text[:200]}".encode()).hexdigest()
|
||||
|
||||
def _sliding(text, size=1000, overlap=150):
|
||||
chunks, i = [], 0
|
||||
while i < len(text):
|
||||
chunks.append(text[i:i+size]); i += size - overlap
|
||||
return [c for c in chunks if c.strip()]
|
||||
|
||||
def _chunk_python(src):
|
||||
try: tree = ast.parse(src)
|
||||
except SyntaxError: return []
|
||||
lines = src.splitlines(); out = []
|
||||
for node in ast.iter_child_nodes(tree):
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
out.append((node.name, "\n".join(lines[node.lineno-1:node.end_lineno])[:4000]))
|
||||
return out
|
||||
|
||||
def _chunk_file(path, src):
|
||||
if path.suffix == ".py":
|
||||
pairs = _chunk_python(src)
|
||||
if pairs: return pairs
|
||||
pat = re.compile(r'(?:^|\n)(?=(?:export\s+)?(?:async\s+)?(?:function|class)|^func |^type |^impl |^pub fn |^fn )', re.M)
|
||||
parts = [p.strip() for p in pat.split(src) if p.strip()]
|
||||
if len(parts) > 1: return [(f"s{i}", p[:4000]) for i,p in enumerate(parts)]
|
||||
return [(f"c{i}", c) for i,c in enumerate(_sliding(src, 1200, 200))]
|
||||
|
||||
def ingest_file(col, fpath, repo=""):
|
||||
if fpath.stat().st_size > MAX_BYTES: return
|
||||
if any(fnmatch.fnmatch(fpath.name, p) for p in ("*.min.js","*.map","package-lock.json","yarn.lock")): return
|
||||
try: src = fpath.read_text(encoding="utf-8", errors="ignore")
|
||||
except: return
|
||||
if not src.strip(): return
|
||||
rel = str(fpath)
|
||||
pairs = _chunk_file(fpath, src) if fpath.suffix in CODE_EXTS else \
|
||||
[(f"c{i}",c) for i,c in enumerate(_sliding(src))]
|
||||
ids,docs,metas = [],[],[]
|
||||
for label,chunk in pairs:
|
||||
if not chunk.strip(): continue
|
||||
ids.append(_doc_id(chunk, rel+label)); docs.append(chunk)
|
||||
metas.append({"source":rel,"label":label,"repo":repo,"lang":fpath.suffix.lstrip(".")})
|
||||
if ids: col.upsert(ids=ids, documents=docs, metadatas=metas)
|
||||
|
||||
def ingest_dir(col, directory, repo=""):
|
||||
count = 0
|
||||
for f in directory.rglob("*"):
|
||||
if not f.is_file() or any(p in f.parts for p in SKIP_DIRS): continue
|
||||
ingest_file(col, f, repo); count += 1
|
||||
log.info("Indexed %d files from %s", count, directory); return count
|
||||
|
||||
def ingest_pdfs(col):
|
||||
try: import pypdf
|
||||
except ImportError: return 0
|
||||
n = 0
|
||||
for pdf in PAPERS_DIR.glob("*.pdf"):
|
||||
try:
|
||||
text = "\n".join(p.extract_text() or "" for p in pypdf.PdfReader(str(pdf)).pages)
|
||||
for i,chunk in enumerate(_sliding(text)):
|
||||
col.upsert(ids=[_doc_id(chunk,str(pdf)+str(i))], documents=[chunk],
|
||||
metadatas=[{"source":str(pdf),"label":f"p{i}","repo":"","lang":"pdf"}])
|
||||
n += 1
|
||||
except Exception as e: log.warning("PDF %s: %s", pdf.name, e)
|
||||
return n
|
||||
|
||||
def _startup():
|
||||
for _ in range(40):
|
||||
try:
|
||||
r = httpx.get(f"{OLLAMA_URL}/api/tags", timeout=5)
|
||||
if any(EMBED_MODEL in m["name"] for m in r.json().get("models",[])): break
|
||||
except: pass
|
||||
log.info("Waiting for embed model..."); time.sleep(5)
|
||||
code_col = get_col("code"); papers_col = get_col("papers")
|
||||
for d in REPOS_DIR.iterdir():
|
||||
if d.is_dir(): ingest_dir(code_col, d, d.name)
|
||||
ingest_pdfs(papers_col)
|
||||
for f in PAPERS_DIR.glob("*.txt"): ingest_file(papers_col, f)
|
||||
log.info("Startup index done")
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup(): threading.Thread(target=_startup, daemon=True).start()
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
try:
|
||||
cc = _chroma()
|
||||
return {"status":"ok",
|
||||
"code": cc.get_collection("code", embedding_function=_embed_fn()).count(),
|
||||
"papers": cc.get_collection("papers", embedding_function=_embed_fn()).count()}
|
||||
except Exception as e: return {"status":"error","detail":str(e)}
|
||||
|
||||
class RepoReq(BaseModel):
|
||||
url: str; name: str = ""; branch: str = "main"
|
||||
|
||||
@app.post("/ingest/repo")
|
||||
async def ingest_repo(req: RepoReq):
|
||||
name = req.name or req.url.rstrip("/").split("/")[-1].removesuffix(".git")
|
||||
dest = REPOS_DIR / name
|
||||
try:
|
||||
if dest.exists(): subprocess.run(["git","pull"], cwd=dest, check=True, timeout=120)
|
||||
else: subprocess.run(["git","clone","--depth=1","-b",req.branch,req.url,str(dest)], check=True, timeout=300)
|
||||
except subprocess.CalledProcessError as e: raise HTTPException(400, str(e))
|
||||
return {"status":"ok","repo":name,"files":ingest_dir(get_col("code"),dest,name)}
|
||||
|
||||
@app.post("/ingest/papers")
|
||||
async def trigger_papers(bg: BackgroundTasks):
|
||||
bg.add_task(ingest_pdfs, get_col("papers")); return {"status":"queued"}
|
||||
|
||||
async def _webhook(payload):
|
||||
repo = payload.get("repository") or {}
|
||||
url = repo.get("clone_url") or repo.get("html_url",""); name = repo.get("name","unknown")
|
||||
if not url: return {"status":"ignored"}
|
||||
dest = REPOS_DIR / name
|
||||
if dest.exists(): subprocess.run(["git","pull"], cwd=dest, timeout=120)
|
||||
else: subprocess.run(["git","clone","--depth=1",url,str(dest)], timeout=300)
|
||||
return {"status":"ok","repo":name,"files":ingest_dir(get_col("code"),dest,name)}
|
||||
|
||||
@app.post("/webhook/gitea")
|
||||
async def wh_gitea(r: Request): return await _webhook(await r.json())
|
||||
@app.post("/webhook/github")
|
||||
async def wh_github(r: Request): return await _webhook(await r.json())
|
||||
|
||||
def _ctx(query, cols):
|
||||
parts = []
|
||||
for cn in cols:
|
||||
try:
|
||||
col = get_col(cn)
|
||||
if col.count() == 0: continue
|
||||
res = col.query(query_texts=[query], n_results=min(TOP_K, col.count()))
|
||||
for doc,meta in zip(res["documents"][0], res["metadatas"][0]):
|
||||
parts.append(f"### {meta.get('source','')}:{meta.get('label','')}\n```{meta.get('lang','')}\n{doc}\n```")
|
||||
except Exception as e: log.warning("col %s: %s", cn, e)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
class ChatReq(BaseModel):
|
||||
model: str = CHAT_MODEL; messages: list[dict[str,Any]]
|
||||
stream: bool = False; collections: list[str] = ["code","papers"]
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat(req: ChatReq):
|
||||
query = next((m["content"] for m in reversed(req.messages) if m.get("role")=="user"), "")
|
||||
ctx = _ctx(query, req.collections)
|
||||
msgs = [{"role":"system","content":f"You are a coding assistant. Use context below.\n\n## Context\n{ctx}"}] + req.messages
|
||||
payload = {"model":req.model,"messages":msgs,"stream":req.stream}
|
||||
if req.stream:
|
||||
async def gen():
|
||||
async with httpx.AsyncClient(timeout=300) as c:
|
||||
async with c.stream("POST",f"{OLLAMA_URL}/v1/chat/completions",json=payload) as r:
|
||||
async for chunk in r.aiter_bytes(): yield chunk
|
||||
return StreamingResponse(gen(), media_type="text/event-stream")
|
||||
async with httpx.AsyncClient(timeout=300) as c:
|
||||
r = await c.post(f"{OLLAMA_URL}/v1/chat/completions", json=payload)
|
||||
return r.json()
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn; uvicorn.run("server:app", host="0.0.0.0", port=8001, reload=False)
|
||||
PY
|
||||
|
||||
# =============================================================================
|
||||
section "Writing mcp_server.py"
|
||||
# =============================================================================
|
||||
write_if_new "$BASE/mcp_server.py" << 'PY'
|
||||
import json, os, re, subprocess
|
||||
from pathlib import Path
|
||||
import httpx
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
|
||||
WORKSPACE = Path(os.getenv("WORKSPACE_DIR", "/workspace"))
|
||||
REPOS_DIR = Path(os.getenv("REPOS_DIR", "/repos"))
|
||||
GITEA_URL = os.getenv("GITEA_URL", "http://gitea:3000")
|
||||
GITEA_TOKEN = os.getenv("GITEA_TOKEN", "")
|
||||
GITHUB_TOKEN = os.getenv("GITHUB_TOKEN","")
|
||||
RAG_URL = os.getenv("RAG_URL", "http://rag-server:8001")
|
||||
|
||||
mcp = FastMCP("local-dev-tools")
|
||||
|
||||
@mcp.tool()
|
||||
def bash(command: str, cwd: str = "") -> str:
|
||||
"""Run a shell command. Default cwd is /workspace."""
|
||||
work = Path(cwd) if cwd else WORKSPACE
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
r = subprocess.run(command, shell=True, cwd=work, timeout=120, capture_output=True, text=True)
|
||||
out = r.stdout + (f"\n[stderr]\n{r.stderr}" if r.stderr else "")
|
||||
if r.returncode != 0: out += f"\n[exit {r.returncode}]"
|
||||
return out or "(no output)"
|
||||
except subprocess.TimeoutExpired: return "[timeout]"
|
||||
except Exception as e: return f"[error] {e}"
|
||||
|
||||
@mcp.tool()
|
||||
def read_file(path: str) -> str:
|
||||
"""Read a file. Absolute or relative to /workspace."""
|
||||
p = Path(path) if Path(path).is_absolute() else WORKSPACE / path
|
||||
if not p.exists(): return f"[not found] {p}"
|
||||
if p.stat().st_size > 500_000: return f"[too large: {p.stat().st_size//1024}KB]"
|
||||
return p.read_text(encoding="utf-8", errors="replace")
|
||||
|
||||
@mcp.tool()
|
||||
def write_file(path: str, content: str) -> str:
|
||||
"""Write content to a file. Relative to /workspace."""
|
||||
p = Path(path) if Path(path).is_absolute() else WORKSPACE / path
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(content, encoding="utf-8")
|
||||
return f"Wrote {len(content)} chars to {p}"
|
||||
|
||||
@mcp.tool()
|
||||
def list_files(path: str = "", pattern: str = "**/*") -> str:
|
||||
"""List files matching a glob pattern."""
|
||||
base = Path(path) if path else WORKSPACE
|
||||
if not base.exists(): return f"[not found] {base}"
|
||||
files = sorted(str(f.relative_to(base)) for f in base.glob(pattern) if f.is_file())
|
||||
return "\n".join(files[:500]) or "(empty)"
|
||||
|
||||
@mcp.tool()
|
||||
def search_code(query: str, path: str = "", glob: str = "", case_sensitive: bool = False) -> str:
|
||||
"""Search file contents with ripgrep."""
|
||||
base = path or str(WORKSPACE)
|
||||
cmd = ["rg", "--line-number", "--no-heading"]
|
||||
if not case_sensitive: cmd.append("-i")
|
||||
if glob: cmd += ["-g", glob]
|
||||
cmd += [query, base]
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
lines = r.stdout.strip().splitlines()
|
||||
if len(lines) > 200: lines = lines[:200] + [f"...({len(r.stdout.splitlines())-200} more)"]
|
||||
return "\n".join(lines) or "(no matches)"
|
||||
except FileNotFoundError:
|
||||
r = subprocess.run(["grep","-rn",query,base], capture_output=True, text=True, timeout=30)
|
||||
return r.stdout[:8000] or "(no matches)"
|
||||
|
||||
@mcp.tool()
|
||||
def fetch_url(url: str, extract_text: bool = True) -> str:
|
||||
"""Fetch the content of a URL."""
|
||||
try:
|
||||
r = httpx.get(url, timeout=30, follow_redirects=True, headers={"User-Agent":"Mozilla/5.0"})
|
||||
content = r.text
|
||||
if extract_text:
|
||||
content = re.sub(r'<script[^>]*>.*?</script>', '', content, flags=re.DOTALL)
|
||||
content = re.sub(r'<style[^>]*>.*?</style>', '', content, flags=re.DOTALL)
|
||||
content = re.sub(r'<[^>]+>', '', content)
|
||||
content = re.sub(r'\n{3,}', '\n\n', content).strip()
|
||||
return content[:20000]
|
||||
except Exception as e: return f"[error] {e}"
|
||||
|
||||
def _git(args, repo=""):
|
||||
cwd = Path(repo) if repo else WORKSPACE
|
||||
r = subprocess.run(["git"]+args, cwd=cwd, capture_output=True, text=True, timeout=60)
|
||||
return (r.stdout + r.stderr).strip() or "(no output)"
|
||||
|
||||
@mcp.tool()
|
||||
def git_status(repo: str = "") -> str:
|
||||
"""Show git status."""
|
||||
return _git(["status","--short"], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_diff(repo: str = "", cached: bool = False) -> str:
|
||||
"""Show git diff."""
|
||||
flag = ["--cached"] if cached else []
|
||||
return _git(["diff","--stat"]+flag, repo) + "\n\n" + _git(["diff"]+flag, repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_log(repo: str = "", n: int = 10) -> str:
|
||||
"""Show last n commits."""
|
||||
return _git(["log",f"-{n}","--oneline","--decorate"], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_commit(message: str, repo: str = "", add_all: bool = True) -> str:
|
||||
"""Stage all and commit."""
|
||||
if add_all: _git(["add","-A"], repo)
|
||||
return _git(["commit","-m",message], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_checkout(branch: str, repo: str = "", create: bool = False) -> str:
|
||||
"""Checkout or create a branch."""
|
||||
return _git(["checkout","-b",branch] if create else ["checkout",branch], repo)
|
||||
|
||||
def _gitea(method, path, body=None):
|
||||
if not GITEA_TOKEN: return {"error":"GITEA_TOKEN not set in .env"}
|
||||
r = httpx.request(method, f"{GITEA_URL}/api/v1{path}",
|
||||
json=body, headers={"Authorization":f"token {GITEA_TOKEN}"}, timeout=30)
|
||||
try: return r.json()
|
||||
except: return {"status":r.status_code,"text":r.text}
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_list_repos() -> str:
|
||||
"""List your Gitea repos."""
|
||||
d = _gitea("GET", "/repos/search?limit=50")
|
||||
if "error" in d: return d["error"]
|
||||
return "\n".join(f"{r['full_name']} — {r.get('description','')}" for r in d.get("data",[]))
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_create_repo(name: str, private: bool = True, description: str = "") -> str:
|
||||
"""Create a Gitea repo."""
|
||||
r = _gitea("POST","/user/repos",{"name":name,"private":private,"description":description,"auto_init":True,"default_branch":"main"})
|
||||
return r.get("html_url") or str(r)
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_create_issue(repo: str, title: str, body: str = "") -> str:
|
||||
"""Create a Gitea issue (owner/repo)."""
|
||||
r = _gitea("POST",f"/repos/{repo}/issues",{"title":title,"body":body})
|
||||
return r.get("html_url") or str(r)
|
||||
|
||||
@mcp.tool()
|
||||
def github_api(method: str, endpoint: str, body: str = "") -> str:
|
||||
"""Call GitHub REST API. endpoint e.g. /repos/owner/repo/issues"""
|
||||
if not GITHUB_TOKEN: return "GITHUB_TOKEN not set in .env"
|
||||
r = httpx.request(method.upper(), f"https://api.github.com{endpoint}",
|
||||
json=json.loads(body) if body else None,
|
||||
headers={"Authorization":f"Bearer {GITHUB_TOKEN}","Accept":"application/vnd.github+json"},
|
||||
timeout=30)
|
||||
try: return json.dumps(r.json(), indent=2)
|
||||
except: return r.text
|
||||
|
||||
@mcp.tool()
|
||||
def ingest_repo(url: str, name: str = "", branch: str = "main") -> str:
|
||||
"""Clone a repo and index it in RAG."""
|
||||
r = httpx.post(f"{RAG_URL}/ingest/repo", json={"url":url,"name":name,"branch":branch}, timeout=300)
|
||||
return r.text
|
||||
|
||||
@mcp.tool()
|
||||
def rag_health() -> str:
|
||||
"""Check RAG server status."""
|
||||
try: return httpx.get(f"{RAG_URL}/health", timeout=10).text
|
||||
except Exception as e: return f"RAG unreachable: {e}"
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(mcp.sse_app(), host="0.0.0.0", port=8002)
|
||||
PY
|
||||
|
||||
# =============================================================================
|
||||
section "Requirements"
|
||||
# =============================================================================
|
||||
cat > "$BASE/requirements.txt" << 'REQ'
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
httpx
|
||||
pydantic
|
||||
chromadb
|
||||
pypdf
|
||||
python-multipart
|
||||
REQ
|
||||
|
||||
cat > "$BASE/mcp_requirements.txt" << 'REQ'
|
||||
mcp[cli]
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
httpx
|
||||
duckduckgo-search
|
||||
REQ
|
||||
ok "requirements.txt + mcp_requirements.txt"
|
||||
|
||||
# =============================================================================
|
||||
section ".env (tokens — never overwritten)"
|
||||
# =============================================================================
|
||||
if [[ ! -f "$BASE/.env" ]]; then
|
||||
cat > "$BASE/.env" << ENV
|
||||
# Local AI Stack — edit to add your API tokens
|
||||
GITEA_TOKEN=your-gitea-token-here
|
||||
GITHUB_TOKEN=your-github-token-here
|
||||
GITEA_URL=http://$LOCAL_IP:3001
|
||||
ENV
|
||||
ok "Created .env"
|
||||
else
|
||||
info "Kept .env"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "Docker Compose"
|
||||
# =============================================================================
|
||||
cat > "$BASE/docker-compose.yml" << COMPOSE
|
||||
# Local AI Stack — generated $(date '+%Y-%m-%d')
|
||||
# GPU: OLLAMA_NUM_GPU=999 uses all available VRAM automatically (V100/RTX/any)
|
||||
# Context: OLLAMA_NUM_CTX= set by detected VRAM (GB)
|
||||
#
|
||||
# ── Common commands (run from this folder) ─────────────────────────────────────
|
||||
# Start everything: docker compose up -d
|
||||
# Stop everything: docker compose down
|
||||
# Restart one service: docker compose restart <service>
|
||||
# Stop one service: docker compose stop <service>
|
||||
# Start one service: docker compose up -d <service>
|
||||
# Follow all logs: docker compose logs -f
|
||||
# Follow one service logs: docker compose logs -f <service>
|
||||
# Pull latest images: docker compose pull && docker compose up -d
|
||||
# Show status: docker compose ps
|
||||
#
|
||||
# Services: ollama open-webui chromadb rag-server mcp-server aider
|
||||
# kiwix gitea invokeai portainer
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
services:
|
||||
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
container_name: ollama
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:11434:11434"]
|
||||
volumes: [ollama-models:/root/.ollama]
|
||||
environment:
|
||||
- OLLAMA_NUM_GPU=999
|
||||
- OLLAMA_NUM_CTX=$CTX
|
||||
- OLLAMA_KEEP_ALIVE=24h
|
||||
- OLLAMA_MAX_LOADED_MODELS=1
|
||||
- OLLAMA_KV_CACHE_TYPE=$OLLAMA_KV_CACHE
|
||||
- OLLAMA_FLASH_ATTENTION=$OLLAMA_FLASH
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD","ollama","list"]
|
||||
interval: 30s; timeout: 10s; retries: 5
|
||||
|
||||
open-webui:
|
||||
image: ghcr.io/open-webui/open-webui:main
|
||||
container_name: open-webui
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:3000:8080"]
|
||||
volumes: [open-webui-data:/app/backend/data]
|
||||
environment:
|
||||
- OLLAMA_BASE_URL=http://ollama:11434
|
||||
- OPENAI_API_BASE_URL=http://rag-server:8001/v1
|
||||
- OPENAI_API_KEY=local-rag
|
||||
- ENABLE_OPENAI_API=true
|
||||
- ENABLE_TOOL_SERVERS=true
|
||||
- WEBUI_AUTH=true
|
||||
- ENABLE_RAG_WEB_SEARCH=true
|
||||
- RAG_WEB_SEARCH_ENGINE=duckduckgo
|
||||
- ENABLE_IMAGE_GENERATION=true
|
||||
- IMAGE_GENERATION_ENGINE=comfyui
|
||||
- COMFYUI_BASE_URL=http://comfyui:8188
|
||||
depends_on:
|
||||
ollama: {condition: service_healthy}
|
||||
|
||||
chromadb:
|
||||
image: chromadb/chroma:latest
|
||||
container_name: chromadb
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:8000:8000"]
|
||||
volumes: [$BASE/index:/chroma/chroma]
|
||||
environment:
|
||||
- IS_PERSISTENT=TRUE
|
||||
- ANONYMIZED_TELEMETRY=FALSE
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL","wget -qO- http://localhost:8000/api/v2/heartbeat || exit 1"]
|
||||
interval: 15s; timeout: 5s; retries: 5
|
||||
|
||||
rag-server:
|
||||
image: python:3.11-slim
|
||||
container_name: rag-server
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:8001:8001"]
|
||||
volumes:
|
||||
- $BASE/papers:/papers
|
||||
- $BASE/repos:/repos
|
||||
- $BASE/index:/index
|
||||
- $BASE/server.py:/app/server.py
|
||||
- $BASE/requirements.txt:/app/requirements.txt
|
||||
working_dir: /app
|
||||
environment:
|
||||
- OLLAMA_URL=http://ollama:11434
|
||||
- CHROMA_URL=http://chromadb:8000
|
||||
- EMBED_MODEL=nomic-embed-text
|
||||
- CHAT_MODEL=$CHAT_MODEL
|
||||
command: >
|
||||
bash -c "apt-get update -qq && apt-get install -y --no-install-recommends git &&
|
||||
pip install --no-cache-dir -r requirements.txt &&
|
||||
uvicorn server:app --host 0.0.0.0 --port 8001"
|
||||
depends_on:
|
||||
chromadb: {condition: service_healthy}
|
||||
ollama: {condition: service_healthy}
|
||||
|
||||
mcp-server:
|
||||
image: python:3.11-slim
|
||||
container_name: mcp-server
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:8002:8002"]
|
||||
volumes:
|
||||
- $BASE/workspace:/workspace
|
||||
- $BASE/repos:/repos
|
||||
- $BASE/mcp_server.py:/app/mcp_server.py
|
||||
- $BASE/mcp_requirements.txt:/app/mcp_requirements.txt
|
||||
- $SCRIPT_DIR/gitea-github-sync.sh:/app/gitea-github-sync.sh:ro
|
||||
working_dir: /app
|
||||
env_file: $BASE/.env
|
||||
environment:
|
||||
- WORKSPACE_DIR=/workspace
|
||||
- REPOS_DIR=/repos
|
||||
- GITEA_URL=http://gitea:3000
|
||||
- RAG_URL=http://rag-server:8001
|
||||
- KIWIX_URL=http://kiwix:80
|
||||
command: >
|
||||
bash -c "apt-get update -qq && apt-get install -y --no-install-recommends git ripgrep curl &&
|
||||
pip install --no-cache-dir -r mcp_requirements.txt &&
|
||||
python mcp_server.py"
|
||||
depends_on: [rag-server, kiwix]
|
||||
|
||||
aider:
|
||||
image: paulgauthier/aider:latest
|
||||
container_name: aider
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:8080:8501"]
|
||||
volumes:
|
||||
- $BASE/workspace:/workspace
|
||||
- $BASE/repos:/repos
|
||||
working_dir: /workspace
|
||||
environment:
|
||||
- OLLAMA_API_BASE=http://ollama:11434
|
||||
- GIT_AUTHOR_NAME=aider
|
||||
- GIT_AUTHOR_EMAIL=aider@local
|
||||
- GIT_COMMITTER_NAME=aider
|
||||
- GIT_COMMITTER_EMAIL=aider@local
|
||||
command: >
|
||||
--gui --no-auto-commits --no-check-update
|
||||
--model ollama/$CODE_MODEL
|
||||
depends_on:
|
||||
ollama: {condition: service_healthy}
|
||||
|
||||
kiwix:
|
||||
image: ghcr.io/kiwix/kiwix-serve:latest
|
||||
container_name: kiwix
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:8181:80"]
|
||||
volumes: [$BASE/kiwix:/data]
|
||||
entrypoint: ["sh", "-c"]
|
||||
command: ["ls /data/*.zim >/dev/null 2>&1 && exec kiwix-serve /data/*.zim || { echo 'No ZIM files in /data yet - sleeping. Add .zim files and restart kiwix.'; exec sleep infinity; }"]
|
||||
|
||||
gitea:
|
||||
image: gitea/gitea:latest
|
||||
container_name: gitea
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:3001:3000","0.0.0.0:2222:22"]
|
||||
volumes:
|
||||
- $BASE/gitea:/data
|
||||
- /etc/timezone:/etc/timezone:ro
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
environment:
|
||||
- USER_UID=1000
|
||||
- USER_GID=1000
|
||||
- GITEA__database__DB_TYPE=sqlite3
|
||||
- GITEA__database__PATH=/data/gitea/gitea.db
|
||||
- GITEA__webhook__ALLOWED_HOST_LIST=rag-server,mcp-server
|
||||
|
||||
invokeai:
|
||||
image: ghcr.io/invoke-ai/invokeai:latest
|
||||
container_name: invokeai
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:9090:9090"]
|
||||
volumes:
|
||||
- invokeai-models:/invokeai/models
|
||||
- $BASE/invokeai-outputs:/invokeai/outputs
|
||||
- $BASE/invokeai-data:/invokeai/databases
|
||||
environment:
|
||||
- INVOKEAI_HOST=0.0.0.0
|
||||
- INVOKEAI_PORT=9090
|
||||
- INVOKEAI_PRECISION=$INVOKEAI_PRECISION
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
|
||||
comfyui:
|
||||
image: ghcr.io/ai-dock/comfyui:latest
|
||||
container_name: comfyui
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:8188:8188"]
|
||||
volumes:
|
||||
- comfyui-models:/opt/ComfyUI/models
|
||||
- $BASE/comfyui-output:/opt/ComfyUI/output
|
||||
- $BASE/comfyui-data:/opt/ComfyUI/custom_nodes
|
||||
environment:
|
||||
- CLI_ARGS=--listen 0.0.0.0
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
|
||||
portainer:
|
||||
image: portainer/portainer-ce:latest
|
||||
container_name: portainer
|
||||
restart: unless-stopped
|
||||
ports: ["0.0.0.0:9000:9000","0.0.0.0:9443:9443"]
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- $BASE/portainer-data:/data
|
||||
|
||||
volumes:
|
||||
ollama-models:
|
||||
open-webui-data:
|
||||
invokeai-models:
|
||||
comfyui-models:
|
||||
COMPOSE
|
||||
ok "docker-compose.yml"
|
||||
|
||||
# =============================================================================
|
||||
section "Firewall"
|
||||
# =============================================================================
|
||||
if command -v ufw &>/dev/null && [[ ! -f "$BASE/.ufw-done" ]] || $FORCE; then
|
||||
read -rp " LAN subnet [192.168.1.0/24]: " LAN; LAN="${LAN:-192.168.1.0/24}"
|
||||
[[ "$LAN" =~ /[0-9]+$ ]] || LAN="${LAN}/24"
|
||||
for pc in "3000:Open WebUI" "11434:Ollama" "8001:RAG" "8002:MCP" \
|
||||
"8000:ChromaDB" "8080:Aider" "8181:Kiwix" \
|
||||
"3001:Gitea" "2222:Gitea SSH" "9090:InvokeAI" "8188:ComfyUI" \
|
||||
"9000:Portainer" "9443:Portainer S"; do
|
||||
sudo ufw allow from "$LAN" to any port "${pc%%:*}" proto tcp comment "${pc##*:}" >/dev/null
|
||||
done
|
||||
sudo ufw reload >/dev/null; ok "UFW rules set for $LAN"; touch "$BASE/.ufw-done"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "Helper Scripts"
|
||||
# =============================================================================
|
||||
cat > "$BASE/start.sh" << STARTSH
|
||||
#!/bin/bash
|
||||
cd "$BASE"
|
||||
docker compose pull --quiet 2>/dev/null
|
||||
docker compose up -d
|
||||
echo ""
|
||||
echo " Open WebUI → http://$LOCAL_IP:3000"
|
||||
echo " Aider UI → http://$LOCAL_IP:8080 (Claude Code-like editor)"
|
||||
echo " InvokeAI → http://$LOCAL_IP:9090 (inpainting, img2img)"
|
||||
echo " ComfyUI → http://$LOCAL_IP:8188 (chat-integrated image gen)"
|
||||
echo " Kiwix → http://$LOCAL_IP:8181 (run kiwix_download.sh first)"
|
||||
echo " Gitea → http://$LOCAL_IP:3001"
|
||||
echo " RAG → http://$LOCAL_IP:8001/health"
|
||||
echo " MCP SSE → http://$LOCAL_IP:8002/sse"
|
||||
echo " Portainer → https://$LOCAL_IP:9443"
|
||||
echo ""
|
||||
echo " claude mcp add local http://$LOCAL_IP:8002/sse"
|
||||
STARTSH
|
||||
chmod +x "$BASE/start.sh"
|
||||
|
||||
cat > "$BASE/stop.sh" << STOPSH
|
||||
#!/bin/bash
|
||||
cd "$BASE" && docker compose down
|
||||
STOPSH
|
||||
chmod +x "$BASE/stop.sh"
|
||||
|
||||
cat > "$BASE/status.sh" << 'STATUSSH'
|
||||
#!/bin/bash
|
||||
echo "=== GPU ===" && nvidia-smi --query-gpu=name,memory.used,memory.total \
|
||||
--format=csv,noheader 2>/dev/null || echo "(no GPU)"
|
||||
echo "" && echo "=== Containers ===" && docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
|
||||
echo "" && echo "=== Ollama ===" && docker exec ollama ollama ps 2>/dev/null || echo "(not running)"
|
||||
echo "" && echo "=== RAG ===" && curl -s http://localhost:8001/health | python3 -m json.tool 2>/dev/null
|
||||
STATUSSH
|
||||
chmod +x "$BASE/status.sh"
|
||||
|
||||
cat > "$BASE/pull-models.sh" << PULLSH
|
||||
#!/bin/bash
|
||||
echo "Waiting for Ollama..."
|
||||
until docker exec ollama ollama list &>/dev/null; do sleep 3; done
|
||||
|
||||
echo "Embed model (RAG — required)..."
|
||||
docker exec ollama ollama pull $EMBED_MODEL
|
||||
|
||||
echo "Fast chat model..."
|
||||
docker exec ollama ollama pull qwen2.5:7b
|
||||
|
||||
echo "Smart chat model..."
|
||||
docker exec ollama ollama pull $CHAT_MODEL
|
||||
|
||||
echo "Code model..."
|
||||
docker exec ollama ollama pull $CODE_MODEL
|
||||
|
||||
echo "Reasoning model (DeepSeek-R1 14B — optional)..."
|
||||
read -rp "Pull DeepSeek-R1:14b for planning/reasoning? [y/N]: " DR
|
||||
[[ "\${DR,,}" == "y" ]] && docker exec ollama ollama pull deepseek-r1:14b
|
||||
|
||||
echo "" && docker exec ollama ollama list
|
||||
PULLSH
|
||||
chmod +x "$BASE/pull-models.sh"
|
||||
ok "start.sh stop.sh status.sh pull-models.sh"
|
||||
|
||||
cat > "$BASE/aider.sh" << AIDSH
|
||||
#!/bin/bash
|
||||
# Aider CLI — Claude Code-like terminal experience against any local git repo.
|
||||
# Usage:
|
||||
# ./aider.sh # interactive, files from stdin
|
||||
# ./aider.sh src/main.py # open specific files
|
||||
# ./aider.sh --model ollama/qwen2.5-coder:7b src/foo.py # override model
|
||||
#
|
||||
# Tip: clone Gitea repos into $BASE/repos/, then:
|
||||
# cd $BASE/repos/my-project && $BASE/aider.sh <files>
|
||||
|
||||
MODEL="\${AIDER_MODEL:-ollama/$CODE_MODEL}"
|
||||
REPO_ROOT="\$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
|
||||
docker run --rm -it \\
|
||||
--network local-ai_default \\
|
||||
-v "\$REPO_ROOT:\$REPO_ROOT" \\
|
||||
-v "$BASE/repos:/repos" \\
|
||||
-w "\$REPO_ROOT" \\
|
||||
-e OLLAMA_API_BASE=http://ollama:11434 \\
|
||||
-e GIT_AUTHOR_NAME=aider \\
|
||||
-e GIT_AUTHOR_EMAIL=aider@local \\
|
||||
-e GIT_COMMITTER_NAME=aider \\
|
||||
-e GIT_COMMITTER_EMAIL=aider@local \\
|
||||
paulgauthier/aider:latest \\
|
||||
--no-auto-commits \\
|
||||
--no-check-update \\
|
||||
--model "\$MODEL" \\
|
||||
"\$@"
|
||||
AIDSH
|
||||
chmod +x "$BASE/aider.sh"
|
||||
ok "aider.sh"
|
||||
|
||||
# =============================================================================
|
||||
section "Systemd"
|
||||
# =============================================================================
|
||||
sudo tee /etc/systemd/system/local-ai.service >/dev/null << SYSD
|
||||
[Unit]
|
||||
Description=Local AI Stack
|
||||
After=docker.service network-online.target
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
User=$USER
|
||||
WorkingDirectory=$BASE
|
||||
ExecStart=/bin/bash $BASE/start.sh
|
||||
ExecStop=/bin/bash $BASE/stop.sh
|
||||
TimeoutStartSec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
SYSD
|
||||
sudo systemctl daemon-reload && sudo systemctl enable local-ai.service
|
||||
ok "Systemd: local-ai.service enabled"
|
||||
|
||||
# =============================================================================
|
||||
section "Starting Stack"
|
||||
# =============================================================================
|
||||
cd "$BASE"
|
||||
info "Pulling images..."
|
||||
docker compose pull --quiet
|
||||
docker compose up -d
|
||||
ok "Stack running"
|
||||
|
||||
if ! $IS_UPDATE && ! $NO_PULL; then
|
||||
echo ""
|
||||
read -rp "Pull Ollama models now? (~15-30 min) [Y/n]: " DO_PULL
|
||||
[[ "${DO_PULL,,}" != "n" ]] && bash "$BASE/pull-models.sh"
|
||||
|
||||
# Install image generation base model (GPU-aware)
|
||||
if [[ -n "$IMG_DEFAULT" ]] && [[ -x "$SCRIPT_DIR/setup-image-models.sh" ]]; then
|
||||
echo ""
|
||||
section "Image Generation Models"
|
||||
info "GPU: ${TOTAL_VRAM}GB VRAM → $IMG_TIER"
|
||||
info "Auto-installing recommended model: $IMG_DEFAULT"
|
||||
bash "$SCRIPT_DIR/setup-image-models.sh" --auto
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}${BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e "${GREEN}${BOLD} Done! GPU: ${VRAM_GB}GB → $TIER${NC}"
|
||||
echo -e "${GREEN}${BOLD} Image gen: $IMG_TIER${NC}"
|
||||
echo -e "${GREEN}${BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo ""
|
||||
echo -e " ${CYAN}Open WebUI${NC} → http://$LOCAL_IP:3000"
|
||||
echo -e " ${CYAN}Aider UI${NC} → http://$LOCAL_IP:8080 (browser coding assistant)"
|
||||
echo -e " ${CYAN}InvokeAI${NC} → http://$LOCAL_IP:9090 (inpainting, img2img)"
|
||||
echo -e " ${CYAN}ComfyUI${NC} → http://$LOCAL_IP:8188 (chat-integrated image gen)"
|
||||
echo -e " ${CYAN}Kiwix${NC} → http://$LOCAL_IP:8181"
|
||||
echo -e " ${CYAN}Gitea${NC} → http://$LOCAL_IP:3001"
|
||||
echo -e " ${CYAN}RAG${NC} → http://$LOCAL_IP:8001/health"
|
||||
echo -e " ${CYAN}MCP SSE${NC} → http://$LOCAL_IP:8002/sse"
|
||||
echo -e " ${CYAN}Portainer${NC} → https://$LOCAL_IP:9443"
|
||||
echo ""
|
||||
echo -e " ${YELLOW}Add MCP to Claude Code:${NC}"
|
||||
echo " claude mcp add local http://$LOCAL_IP:8002/sse"
|
||||
echo ""
|
||||
echo -e " ${YELLOW}Tokens:${NC} $BASE/.env"
|
||||
echo -e " ${YELLOW}PDFs:${NC} $BASE/papers/"
|
||||
echo -e " ${YELLOW}Workspace:${NC} $BASE/workspace/"
|
||||
echo -e " ${YELLOW}ZIMs:${NC} ./kiwix_download.sh"
|
||||
echo ""
|
||||
echo -e " ${YELLOW}Aider CLI:${NC} cd your-repo && $BASE/aider.sh <files>"
|
||||
echo " (or open browser UI above — both use your local code model)"
|
||||
echo ""
|
||||
echo -e " ${YELLOW}Recommended Open WebUI Functions${NC} (install from Admin → Functions → ＋):"
|
||||
echo " Context tracker: https://openwebui.com/f/centrisic/context_tracker"
|
||||
echo " → Shows tokens used vs available, progress bar, context % remaining"
|
||||
echo " Context compaction: https://openwebui.com/f/projectmoon/checkpoint_summarization_filter"
|
||||
echo " → Auto-summarizes old messages when context fills up (like Claude)"
|
||||
echo ""
|
||||
echo -e " ${YELLOW}Save Claude usage:${NC} use local models for boilerplate, docs,"
|
||||
echo " simple fixes. Use Claude for hard bugs,"
|
||||
echo " multi-file refactoring, architecture decisions."
|
||||
echo ""
|
||||
@@ -0,0 +1,366 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
MCP Server — Claude Code-equivalent tools for Open WebUI / Claude Code CLI.
|
||||
Tools: bash, file read/write/list, code search, git ops, Gitea API, repo ingest,
|
||||
offline doc search (Kiwix), web search (DuckDuckGo), Gitea↔GitHub sync.
|
||||
Connects via SSE on port 8002 — add to Open WebUI Tools or ~/.claude/mcp.json
|
||||
"""
|
||||
import os, subprocess, textwrap
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
|
||||
WORKSPACE = Path(os.getenv("WORKSPACE_DIR", "/workspace"))
|
||||
REPOS_DIR = Path(os.getenv("REPOS_DIR", "/repos"))
|
||||
GITEA_URL = os.getenv("GITEA_URL", "http://gitea:3000")
|
||||
GITEA_TOKEN = os.getenv("GITEA_TOKEN", "")
|
||||
GITHUB_TOKEN= os.getenv("GITHUB_TOKEN","")
|
||||
RAG_URL = os.getenv("RAG_URL", "http://rag-server:8001")
|
||||
KIWIX_URL = os.getenv("KIWIX_URL", "http://kiwix:80")
|
||||
|
||||
mcp = FastMCP("local-dev-tools")
|
||||
|
||||
# ── bash ──────────────────────────────────────────────────────────────────────
|
||||
@mcp.tool()
|
||||
def bash(command: str, cwd: str = "") -> str:
|
||||
"""Run a shell command. Default cwd is /workspace."""
|
||||
work = Path(cwd) if cwd else WORKSPACE
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
r = subprocess.run(command, shell=True, cwd=work, timeout=120,
|
||||
capture_output=True, text=True)
|
||||
out = r.stdout + (f"\n[stderr]\n{r.stderr}" if r.stderr else "")
|
||||
if r.returncode != 0:
|
||||
out += f"\n[exit {r.returncode}]"
|
||||
return out or "(no output)"
|
||||
except subprocess.TimeoutExpired:
|
||||
return "[timeout after 120s]"
|
||||
except Exception as e:
|
||||
return f"[error] {e}"
|
||||
|
||||
# ── file ops ──────────────────────────────────────────────────────────────────
|
||||
@mcp.tool()
|
||||
def read_file(path: str) -> str:
|
||||
"""Read a file. Use absolute path or relative to /workspace."""
|
||||
p = Path(path) if Path(path).is_absolute() else WORKSPACE / path
|
||||
if not p.exists():
|
||||
return f"[not found] {p}"
|
||||
if p.stat().st_size > 500_000:
|
||||
return f"[too large — {p.stat().st_size//1024}KB]"
|
||||
return p.read_text(encoding="utf-8", errors="replace")
|
||||
|
||||
@mcp.tool()
|
||||
def write_file(path: str, content: str) -> str:
|
||||
"""Write content to a file (creates parent dirs). Relative to /workspace."""
|
||||
p = Path(path) if Path(path).is_absolute() else WORKSPACE / path
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(content, encoding="utf-8")
|
||||
return f"Wrote {len(content)} chars to {p}"
|
||||
|
||||
@mcp.tool()
|
||||
def list_files(path: str = "", pattern: str = "**/*") -> str:
|
||||
"""List files matching a glob pattern."""
|
||||
base = Path(path) if path else WORKSPACE
|
||||
if not base.exists():
|
||||
return f"[not found] {base}"
|
||||
files = sorted(str(f.relative_to(base)) for f in base.glob(pattern) if f.is_file())
|
||||
return "\n".join(files[:500]) or "(empty)"
|
||||
|
||||
@mcp.tool()
|
||||
def search_code(query: str, path: str = "", glob: str = "",
|
||||
case_sensitive: bool = False) -> str:
|
||||
"""Search file contents with ripgrep. Returns file:line matches."""
|
||||
base = path or str(WORKSPACE)
|
||||
cmd = ["rg", "--line-number", "--no-heading"]
|
||||
if not case_sensitive:
|
||||
cmd.append("-i")
|
||||
if glob:
|
||||
cmd += ["-g", glob]
|
||||
cmd += [query, base]
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
lines = r.stdout.strip().splitlines()
|
||||
if len(lines) > 200:
|
||||
lines = lines[:200] + [f"… ({len(r.stdout.splitlines())-200} more)"]
|
||||
return "\n".join(lines) or "(no matches)"
|
||||
except FileNotFoundError:
|
||||
# ripgrep not installed, fall back to grep
|
||||
r = subprocess.run(["grep", "-rn", query, base],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
return r.stdout[:8000] or "(no matches)"
|
||||
except Exception as e:
|
||||
return f"[error] {e}"
|
||||
|
||||
# ── git ───────────────────────────────────────────────────────────────────────
|
||||
def _git(args: list[str], repo: str = "") -> str:
|
||||
cwd = Path(repo) if repo else WORKSPACE
|
||||
r = subprocess.run(["git"] + args, cwd=cwd,
|
||||
capture_output=True, text=True, timeout=60)
|
||||
return (r.stdout + r.stderr).strip() or "(no output)"
|
||||
|
||||
@mcp.tool()
|
||||
def git_status(repo: str = "") -> str:
|
||||
"""Show git status of a repo (default: /workspace)."""
|
||||
return _git(["status", "--short"], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_diff(repo: str = "", cached: bool = False) -> str:
|
||||
"""Show git diff (staged if cached=True)."""
|
||||
args = ["diff", "--stat", "--cached"] if cached else ["diff", "--stat"]
|
||||
return _git(args, repo) + "\n\n" + _git(
|
||||
["diff", "--cached"] if cached else ["diff"], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_log(repo: str = "", n: int = 10) -> str:
|
||||
"""Show last n git commits."""
|
||||
return _git(["log", f"-{n}", "--oneline", "--decorate"], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_commit(message: str, repo: str = "", add_all: bool = True) -> str:
|
||||
"""Stage all changes and create a commit."""
|
||||
if add_all:
|
||||
_git(["add", "-A"], repo)
|
||||
return _git(["commit", "-m", message], repo)
|
||||
|
||||
@mcp.tool()
|
||||
def git_checkout(branch: str, repo: str = "", create: bool = False) -> str:
|
||||
"""Checkout a branch, optionally creating it."""
|
||||
args = ["checkout", "-b", branch] if create else ["checkout", branch]
|
||||
return _git(args, repo)
|
||||
|
||||
# ── Search: unified (Kiwix offline + DuckDuckGo live) ───────────────────────
|
||||
# Kiwix ZIMs have complete, high-quality articles but may be months old.
|
||||
# DDG has live results but lower signal-to-noise. The unified search tool
|
||||
# checks both and lets the model see freshness info to judge which to trust.
|
||||
#
|
||||
# Heuristic: topics that change fast (releases, CVEs, "latest X") get flagged
|
||||
# as potentially stale in offline results. Timeless topics (algorithms, language
|
||||
# docs, math) are fine from Kiwix and skip the web hit entirely.
|
||||
|
||||
import re as _re
|
||||
from datetime import datetime as _dt
|
||||
|
||||
# Words that suggest the query needs fresh data
|
||||
_FRESH_KEYWORDS = _re.compile(
|
||||
r'\b(latest|newest|recent|2025|2026|update|release|version|changelog|CVE|vulnerability|'
|
||||
r'breaking change|deprecat|current|today|this year|this month|announce|just released)\b',
|
||||
_re.IGNORECASE
|
||||
)
|
||||
|
||||
def _kiwix_search(query: str, limit: int = 5) -> list[dict]:
|
||||
"""Search Kiwix, return list of {title, snippet, path, source}."""
|
||||
try:
|
||||
r = httpx.get(f"{KIWIX_URL}/search",
|
||||
params={"pattern": query, "pageLength": limit},
|
||||
timeout=15, follow_redirects=True)
|
||||
if r.status_code != 200:
|
||||
return []
|
||||
html = r.text
|
||||
results = []
|
||||
# Try structured parse first
|
||||
articles = _re.findall(
|
||||
r'<a[^>]+href="(/[^"]+)"[^>]*>\s*<span[^>]*>([^<]*)</span>.*?'
|
||||
r'(?:<cite[^>]*>([^<]*)</cite>)?.*?'
|
||||
r'(?:<p[^>]*>(.*?)</p>)?',
|
||||
html, _re.DOTALL
|
||||
)
|
||||
if articles:
|
||||
for path, title, cite, snippet in articles[:limit]:
|
||||
snippet_clean = _re.sub(r'<[^>]+>', '', snippet or '').strip()[:300]
|
||||
results.append({"title": title.strip(), "snippet": snippet_clean,
|
||||
"path": path, "source": cite.strip() if cite else "kiwix"})
|
||||
else:
|
||||
# Fallback: grab any links
|
||||
for path, title in _re.findall(r'<a[^>]+href="(/[^"]+)"[^>]*>([^<]+)</a>', html)[:limit]:
|
||||
results.append({"title": title.strip(), "snippet": "",
|
||||
"path": path, "source": "kiwix"})
|
||||
return results
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def _ddg_search(query: str, limit: int = 5) -> list[dict]:
|
||||
"""Search DuckDuckGo, return list of {title, snippet, url}."""
|
||||
try:
|
||||
from duckduckgo_search import DDGS
|
||||
results = []
|
||||
with DDGS() as ddgs:
|
||||
for r in ddgs.text(query, max_results=limit):
|
||||
results.append({"title": r["title"], "snippet": r["body"], "url": r["href"]})
|
||||
return results
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
@mcp.tool()
|
||||
def search(query: str, limit: int = 5) -> str:
|
||||
"""Unified search: checks offline docs (Kiwix) AND live web (DuckDuckGo).
|
||||
Returns results from both with freshness guidance.
|
||||
For timeless topics (algorithms, docs): offline results are sufficient.
|
||||
For time-sensitive topics (releases, CVEs): live results are flagged as preferred."""
|
||||
needs_fresh = bool(_FRESH_KEYWORDS.search(query))
|
||||
output_parts = []
|
||||
|
||||
# Always search Kiwix (fast, local)
|
||||
kiwix_results = _kiwix_search(query, limit)
|
||||
if kiwix_results:
|
||||
header = "## Offline Docs (Kiwix)"
|
||||
if needs_fresh:
|
||||
header += " ⚠️ POSSIBLY STALE — query looks time-sensitive, prefer live results below"
|
||||
output_parts.append(header)
|
||||
for i, r in enumerate(kiwix_results, 1):
|
||||
entry = f"{i}. **{r['title']}**"
|
||||
if r["source"] and r["source"] != "kiwix":
|
||||
entry += f" ({r['source']})"
|
||||
if r["snippet"]:
|
||||
entry += f"\n {r['snippet']}"
|
||||
entry += f"\n → read_doc('{r['path']}')"
|
||||
output_parts.append(entry)
|
||||
|
||||
# Search DDG if: query needs fresh data, OR Kiwix returned nothing, OR always (to compare)
|
||||
do_web = needs_fresh or not kiwix_results
|
||||
ddg_results = []
|
||||
if do_web:
|
||||
ddg_results = _ddg_search(query, limit)
|
||||
|
||||
if ddg_results:
|
||||
header = "## Live Web (DuckDuckGo)"
|
||||
if needs_fresh:
|
||||
header += " ✓ PREFER THESE for this query"
|
||||
output_parts.append(header)
|
||||
for i, r in enumerate(ddg_results, 1):
|
||||
output_parts.append(f"{i}. **{r['title']}**\n {r['snippet']}\n {r['url']}")
|
||||
elif do_web:
|
||||
output_parts.append("## Live Web (DuckDuckGo)\n(no results or DDG unreachable)")
|
||||
|
||||
if not kiwix_results and not ddg_results:
|
||||
return f"No results for '{query}' from either offline docs or web search."
|
||||
|
||||
# Freshness note
|
||||
if kiwix_results and not needs_fresh and not ddg_results:
|
||||
output_parts.append("\n_Offline results look sufficient for this topic. "
|
||||
"Use web_search() if you need to verify currency._")
|
||||
|
||||
return "\n\n".join(output_parts)
|
||||
|
||||
@mcp.tool()
|
||||
def read_doc(path: str) -> str:
|
||||
"""Read a full article from Kiwix by its path (from search results).
|
||||
Example: read_doc('/wikipedia_en_all/A/Python_(programming_language)')"""
|
||||
try:
|
||||
r = httpx.get(f"{KIWIX_URL}{path}", timeout=15, follow_redirects=True)
|
||||
if r.status_code != 200:
|
||||
return f"Not found: {path} (HTTP {r.status_code})"
|
||||
# Strip HTML tags, keep text content
|
||||
text = _re.sub(r'<script[^>]*>.*?</script>', '', r.text, flags=_re.DOTALL)
|
||||
text = _re.sub(r'<style[^>]*>.*?</style>', '', text, flags=_re.DOTALL)
|
||||
text = _re.sub(r'<[^>]+>', ' ', text)
|
||||
text = _re.sub(r'\s+', ' ', text).strip()
|
||||
if len(text) > 8000:
|
||||
text = text[:8000] + "\n\n[... truncated — article continues ...]"
|
||||
return text
|
||||
except Exception as e:
|
||||
return f"Error reading doc: {e}"
|
||||
|
||||
@mcp.tool()
|
||||
def web_search(query: str, num_results: int = 5) -> str:
|
||||
"""Search ONLY the live web via DuckDuckGo. Use search() instead for most queries —
|
||||
it checks both offline and live. Use this directly only when you specifically need
|
||||
live-only results (e.g., verifying if offline info is current)."""
|
||||
results = _ddg_search(query, num_results)
|
||||
if not results:
|
||||
return f"No web results for '{query}'"
|
||||
return "\n\n".join(f"**{r['title']}**\n {r['snippet']}\n {r['url']}" for r in results)
|
||||
|
||||
# ── Gitea API ─────────────────────────────────────────────────────────────────
|
||||
def _gitea(method: str, path: str, body: dict = {}) -> dict:
|
||||
if not GITEA_TOKEN:
|
||||
return {"error": "GITEA_TOKEN not set in .env"}
|
||||
url = f"{GITEA_URL}/api/v1{path}"
|
||||
headers = {"Authorization": f"token {GITEA_TOKEN}",
|
||||
"Content-Type": "application/json"}
|
||||
r = httpx.request(method, url, json=body or None, headers=headers, timeout=30)
|
||||
try:
|
||||
return r.json()
|
||||
except Exception:
|
||||
return {"status": r.status_code, "text": r.text}
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_list_repos() -> str:
|
||||
"""List your Gitea repos."""
|
||||
repos = _gitea("GET", "/repos/search?limit=50")
|
||||
if "error" in repos:
|
||||
return repos["error"]
|
||||
return "\n".join(f"{r['full_name']} — {r.get('description','')}"
|
||||
for r in repos.get("data", []))
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_create_repo(name: str, private: bool = True, description: str = "") -> str:
|
||||
"""Create a new Gitea repository."""
|
||||
r = _gitea("POST", "/user/repos",
|
||||
{"name": name, "private": private, "description": description,
|
||||
"auto_init": True, "default_branch": "main"})
|
||||
return r.get("html_url") or str(r)
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_create_issue(repo: str, title: str, body: str = "") -> str:
|
||||
"""Create an issue on a Gitea repo (format: owner/repo)."""
|
||||
r = _gitea("POST", f"/repos/{repo}/issues", {"title": title, "body": body})
|
||||
return r.get("html_url") or str(r)
|
||||
|
||||
# ── GitHub API ────────────────────────────────────────────────────────────────
|
||||
@mcp.tool()
|
||||
def github_api(method: str, endpoint: str, body: str = "") -> str:
|
||||
"""Call the GitHub REST API. endpoint e.g. /repos/owner/repo/issues"""
|
||||
if not GITHUB_TOKEN:
|
||||
return "GITHUB_TOKEN not set in .env"
|
||||
import json as _json
|
||||
headers = {"Authorization": f"Bearer {GITHUB_TOKEN}",
|
||||
"Accept": "application/vnd.github+json"}
|
||||
r = httpx.request(method.upper(), f"https://api.github.com{endpoint}",
|
||||
json=_json.loads(body) if body else None,
|
||||
headers=headers, timeout=30)
|
||||
try:
|
||||
return _json.dumps(r.json(), indent=2)
|
||||
except Exception:
|
||||
return r.text
|
||||
|
||||
# ── Gitea ↔ GitHub sync ──────────────────────────────────────────────────────
|
||||
@mcp.tool()
|
||||
def gitea_github_sync(mode: str = "all", repo: str = "") -> str:
|
||||
"""Run Gitea↔GitHub mirror sync. mode: all|pull|push|list. repo: optional owner/name."""
|
||||
cmd = ["/app/gitea-github-sync.sh"]
|
||||
if mode == "pull": cmd.append("--pull-only")
|
||||
elif mode == "push": cmd.append("--push-only")
|
||||
elif mode == "list": cmd.append("--list")
|
||||
if repo:
|
||||
cmd.extend(["--repo", repo])
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=600,
|
||||
env={**os.environ, "SYNC_ENV": "/app/.env"})
|
||||
return (r.stdout + r.stderr).strip() or "Sync completed (no output)"
|
||||
except subprocess.TimeoutExpired:
|
||||
return "Sync timed out after 10 minutes"
|
||||
except Exception as e:
|
||||
return f"Sync failed: {e}"
|
||||
|
||||
# ── RAG ingest ────────────────────────────────────────────────────────────────
|
||||
@mcp.tool()
|
||||
def ingest_repo(url: str, name: str = "", branch: str = "main") -> str:
|
||||
"""Clone a git repo and index it in the RAG code collection."""
|
||||
r = httpx.post(f"{RAG_URL}/ingest/repo",
|
||||
json={"url": url, "name": name, "branch": branch}, timeout=300)
|
||||
return r.text
|
||||
|
||||
@mcp.tool()
|
||||
def rag_health() -> str:
|
||||
"""Check RAG server status and indexed document counts."""
|
||||
try:
|
||||
r = httpx.get(f"{RAG_URL}/health", timeout=10)
|
||||
return r.text
|
||||
except Exception as e:
|
||||
return f"RAG server unreachable: {e}"
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
app = mcp.sse_app()
|
||||
uvicorn.run(app, host="0.0.0.0", port=8002)
|
||||
Executable
+67
@@ -0,0 +1,67 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# Ollama Model Auto-Updater
|
||||
# Pulls latest version of every installed model. Ollama compares digests
|
||||
# server-side — no download happens if the model is already current.
|
||||
# Safe to run as a systemd timer.
|
||||
# =============================================================================
|
||||
|
||||
OLLAMA_URL="${OLLAMA_HOST:-http://localhost:11434}"
|
||||
LOG="$HOME/docker/ai-stack/logs/ollama-update.log"
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
G='\033[0;32m'; Y='\033[1;33m'; C='\033[0;36m'; R='\033[0;31m'; N='\033[0m'
|
||||
ok() { echo -e "${G}[OK]${N} $1" | tee -a "$LOG"; }
|
||||
inf() { echo -e "${C}[..]${N} $1" | tee -a "$LOG"; }
|
||||
wrn() { echo -e "${Y}[!!]${N} $1" | tee -a "$LOG"; }
|
||||
err() { echo -e "${R}[XX]${N} $1" | tee -a "$LOG"; }
|
||||
|
||||
echo "" | tee -a "$LOG"
|
||||
echo "=== Ollama Model Update: $(date) ===" | tee -a "$LOG"
|
||||
echo "" | tee -a "$LOG"
|
||||
|
||||
# Check Ollama is reachable
|
||||
if ! curl -sf "$OLLAMA_URL/api/tags" > /dev/null; then
|
||||
err "Ollama not reachable at $OLLAMA_URL — is the container running?"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get installed model names from the API
|
||||
mapfile -t MODELS < <(
|
||||
curl -sf "$OLLAMA_URL/api/tags" | \
|
||||
grep -oP '"name"\s*:\s*"\K[^"]+' | \
|
||||
sort -u
|
||||
)
|
||||
|
||||
if [[ ${#MODELS[@]} -eq 0 ]]; then
|
||||
wrn "No models found — nothing to update"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
inf "Found ${#MODELS[@]} model(s): ${MODELS[*]}"
|
||||
echo "" | tee -a "$LOG"
|
||||
|
||||
UPDATED=0
|
||||
FAILED=0
|
||||
|
||||
for model in "${MODELS[@]}"; do
|
||||
inf "Pulling $model..."
|
||||
output=$(ollama pull "$model" 2>&1)
|
||||
exit_code=$?
|
||||
|
||||
if [[ $exit_code -ne 0 ]]; then
|
||||
err "$model: pull failed"
|
||||
echo "$output" >> "$LOG"
|
||||
FAILED=$((FAILED + 1))
|
||||
elif echo "$output" | grep -q "up to date"; then
|
||||
ok "$model: already up to date"
|
||||
else
|
||||
ok "$model: updated"
|
||||
UPDATED=$((UPDATED + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
echo "" | tee -a "$LOG"
|
||||
inf "Done — $UPDATED updated, $FAILED failed"
|
||||
echo "=== Complete: $(date) ===" | tee -a "$LOG"
|
||||
echo "" | tee -a "$LOG"
|
||||
@@ -0,0 +1,281 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
RAG Server — code-aware chunking, multi-collection, repo ingest, webhooks.
|
||||
Collections: papers (PDFs/text), code (source files, AST-split for Python)
|
||||
"""
|
||||
import ast, fnmatch, hashlib, json, logging, os, re, subprocess, threading, time
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
import chromadb
|
||||
import httpx
|
||||
from chromadb.utils.embedding_functions import OllamaEmbeddingFunction
|
||||
from fastapi import FastAPI, HTTPException, Request, BackgroundTasks
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
log = logging.getLogger("rag")
|
||||
|
||||
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
|
||||
CHROMA_URL = os.getenv("CHROMA_URL", "http://chromadb:8000")
|
||||
EMBED_MODEL = os.getenv("EMBED_MODEL", "nomic-embed-text")
|
||||
CHAT_MODEL = os.getenv("CHAT_MODEL", "qwen2.5:14b")
|
||||
PAPERS_DIR = Path(os.getenv("PAPERS_DIR", "/papers"))
|
||||
REPOS_DIR = Path(os.getenv("REPOS_DIR", "/repos"))
|
||||
TOP_K = int(os.getenv("TOP_K", "6"))
|
||||
|
||||
CODE_EXTS = {".py",".js",".ts",".tsx",".jsx",".go",".rs",".java",".c",".cpp",
|
||||
".h",".hpp",".cs",".rb",".sh",".yaml",".yml",".toml",".sql",".md"}
|
||||
SKIP_DIRS = {"node_modules",".git","__pycache__","dist","build",".venv",
|
||||
"venv","env",".next","vendor","target","bin","obj"}
|
||||
SKIP_FILES = {"package-lock.json","yarn.lock","pnpm-lock.yaml","Cargo.lock"}
|
||||
MAX_BYTES = 400_000
|
||||
|
||||
app = FastAPI(title="RAG Server")
|
||||
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
|
||||
|
||||
# ── ChromaDB ──────────────────────────────────────────────────────────────────
|
||||
def _embed_fn():
|
||||
return OllamaEmbeddingFunction(
|
||||
url=f"{OLLAMA_URL}/api/embeddings", model_name=EMBED_MODEL)
|
||||
|
||||
def _chroma():
|
||||
host, port = CHROMA_URL.replace("http://","").split(":")
|
||||
return chromadb.HttpClient(host=host, port=int(port))
|
||||
|
||||
def get_col(name: str):
|
||||
return _chroma().get_or_create_collection(name, embedding_function=_embed_fn())
|
||||
|
||||
# ── chunkers ─────────────────────────────────────────────────────────────────
|
||||
def _doc_id(text: str, key: str) -> str:
|
||||
return hashlib.md5(f"{key}|{text[:200]}".encode()).hexdigest()
|
||||
|
||||
def _sliding(text: str, size=1000, overlap=150) -> list[str]:
|
||||
chunks, i = [], 0
|
||||
while i < len(text):
|
||||
chunks.append(text[i:i+size])
|
||||
i += size - overlap
|
||||
return [c for c in chunks if c.strip()]
|
||||
|
||||
def _chunk_python(src: str) -> list[tuple[str,str]]:
|
||||
try:
|
||||
tree = ast.parse(src)
|
||||
except SyntaxError:
|
||||
return []
|
||||
lines = src.splitlines()
|
||||
out = []
|
||||
for node in ast.iter_child_nodes(tree):
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
chunk = "\n".join(lines[node.lineno-1:node.end_lineno])
|
||||
out.append((node.name, chunk[:4000]))
|
||||
return out
|
||||
|
||||
def _chunk_file(path: Path, src: str) -> list[tuple[str,str]]:
|
||||
if path.suffix == ".py":
|
||||
pairs = _chunk_python(src)
|
||||
if pairs:
|
||||
return pairs
|
||||
# function/class boundary split for JS/TS/Go/Rust etc.
|
||||
pat = re.compile(
|
||||
r'(?:^|\n)(?=(?:export\s+)?(?:async\s+)?(?:function|class|const\s+\w+\s*=\s*(?:async\s+)?\()'
|
||||
r'|^func |^type |^impl |^pub fn |^fn )',
|
||||
re.MULTILINE)
|
||||
parts = [p.strip() for p in pat.split(src) if p.strip()]
|
||||
if len(parts) > 1:
|
||||
return [(f"s{i}", p[:4000]) for i, p in enumerate(parts)]
|
||||
return [(f"c{i}", c) for i, c in enumerate(_sliding(src, 1200, 200))]
|
||||
|
||||
# ── ingest helpers ────────────────────────────────────────────────────────────
|
||||
def ingest_file(col, fpath: Path, repo: str = ""):
|
||||
if fpath.stat().st_size > MAX_BYTES or fpath.name in SKIP_FILES:
|
||||
return
|
||||
if any(fnmatch.fnmatch(fpath.name, p) for p in ("*.min.js","*.min.css","*.map")):
|
||||
return
|
||||
try:
|
||||
src = fpath.read_text(encoding="utf-8", errors="ignore")
|
||||
except Exception:
|
||||
return
|
||||
if not src.strip():
|
||||
return
|
||||
rel = str(fpath)
|
||||
pairs = _chunk_file(fpath, src) if fpath.suffix in CODE_EXTS else \
|
||||
[(f"c{i}", c) for i, c in enumerate(_sliding(src))]
|
||||
ids, docs, metas = [], [], []
|
||||
for label, chunk in pairs:
|
||||
if not chunk.strip():
|
||||
continue
|
||||
ids.append(_doc_id(chunk, rel+label))
|
||||
docs.append(chunk)
|
||||
metas.append({"source": rel, "label": label, "repo": repo,
|
||||
"lang": fpath.suffix.lstrip(".")})
|
||||
if ids:
|
||||
col.upsert(ids=ids, documents=docs, metadatas=metas)
|
||||
|
||||
def ingest_dir(col, directory: Path, repo: str = "") -> int:
|
||||
count = 0
|
||||
for f in directory.rglob("*"):
|
||||
if not f.is_file():
|
||||
continue
|
||||
if any(p in f.parts for p in SKIP_DIRS):
|
||||
continue
|
||||
ingest_file(col, f, repo)
|
||||
count += 1
|
||||
log.info("Indexed %d files from %s", count, directory)
|
||||
return count
|
||||
|
||||
def ingest_pdfs(col) -> int:
|
||||
try:
|
||||
import pypdf
|
||||
except ImportError:
|
||||
log.warning("pypdf not installed — skipping PDFs")
|
||||
return 0
|
||||
n = 0
|
||||
for pdf in PAPERS_DIR.glob("*.pdf"):
|
||||
try:
|
||||
text = "\n".join(p.extract_text() or ""
|
||||
for p in pypdf.PdfReader(str(pdf)).pages)
|
||||
for i, chunk in enumerate(_sliding(text)):
|
||||
col.upsert(ids=[_doc_id(chunk, str(pdf)+str(i))],
|
||||
documents=[chunk],
|
||||
metadatas=[{"source": str(pdf), "label": f"p{i}",
|
||||
"repo": "", "lang": "pdf"}])
|
||||
n += 1
|
||||
except Exception as e:
|
||||
log.warning("PDF %s: %s", pdf.name, e)
|
||||
return n
|
||||
|
||||
# ── startup ───────────────────────────────────────────────────────────────────
|
||||
def _startup_index():
|
||||
# wait for embed model
|
||||
for _ in range(40):
|
||||
try:
|
||||
r = httpx.get(f"{OLLAMA_URL}/api/tags", timeout=5)
|
||||
if any(EMBED_MODEL in m["name"] for m in r.json().get("models", [])):
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
log.info("Waiting for embed model %s…", EMBED_MODEL)
|
||||
time.sleep(5)
|
||||
|
||||
code_col = get_col("code")
|
||||
papers_col = get_col("papers")
|
||||
for d in REPOS_DIR.iterdir():
|
||||
if d.is_dir():
|
||||
ingest_dir(code_col, d, d.name)
|
||||
ingest_pdfs(papers_col)
|
||||
for f in PAPERS_DIR.glob("*.txt"):
|
||||
ingest_file(papers_col, f)
|
||||
log.info("Startup index complete")
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup():
|
||||
threading.Thread(target=_startup_index, daemon=True).start()
|
||||
|
||||
# ── endpoints ─────────────────────────────────────────────────────────────────
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
try:
|
||||
cc = _chroma()
|
||||
return {"status": "ok",
|
||||
"code": cc.get_collection("code", embedding_function=_embed_fn()).count(),
|
||||
"papers": cc.get_collection("papers", embedding_function=_embed_fn()).count(),
|
||||
"embed": EMBED_MODEL, "chat": CHAT_MODEL}
|
||||
except Exception as e:
|
||||
return {"status": "error", "detail": str(e)}
|
||||
|
||||
class RepoRequest(BaseModel):
|
||||
url: str
|
||||
name: str = ""
|
||||
branch: str = "main"
|
||||
|
||||
@app.post("/ingest/repo")
|
||||
async def ingest_repo(req: RepoRequest):
|
||||
name = req.name or req.url.rstrip("/").split("/")[-1].removesuffix(".git")
|
||||
dest = REPOS_DIR / name
|
||||
try:
|
||||
if dest.exists():
|
||||
subprocess.run(["git","pull"], cwd=dest, check=True, timeout=120)
|
||||
else:
|
||||
subprocess.run(["git","clone","--depth=1","-b",req.branch,
|
||||
req.url, str(dest)], check=True, timeout=300)
|
||||
except subprocess.CalledProcessError as e:
|
||||
raise HTTPException(400, str(e))
|
||||
n = ingest_dir(get_col("code"), dest, name)
|
||||
return {"status": "ok", "repo": name, "files": n}
|
||||
|
||||
@app.post("/ingest/papers")
|
||||
async def trigger_papers(bg: BackgroundTasks):
|
||||
bg.add_task(ingest_pdfs, get_col("papers"))
|
||||
return {"status": "queued"}
|
||||
|
||||
async def _webhook(payload: dict):
|
||||
repo = payload.get("repository") or {}
|
||||
url = repo.get("clone_url") or repo.get("html_url","")
|
||||
name = repo.get("name","unknown")
|
||||
if not url:
|
||||
return {"status": "ignored"}
|
||||
dest = REPOS_DIR / name
|
||||
if dest.exists():
|
||||
subprocess.run(["git","pull"], cwd=dest, timeout=120)
|
||||
else:
|
||||
subprocess.run(["git","clone","--depth=1",url,str(dest)], timeout=300)
|
||||
n = ingest_dir(get_col("code"), dest, name)
|
||||
return {"status": "ok", "repo": name, "files": n}
|
||||
|
||||
@app.post("/webhook/gitea")
|
||||
async def webhook_gitea(r: Request): return await _webhook(await r.json())
|
||||
|
||||
@app.post("/webhook/github")
|
||||
async def webhook_github(r: Request): return await _webhook(await r.json())
|
||||
|
||||
# ── RAG chat ──────────────────────────────────────────────────────────────────
|
||||
class ChatRequest(BaseModel):
|
||||
model: str = CHAT_MODEL
|
||||
messages: list[dict[str,Any]]
|
||||
stream: bool = False
|
||||
collections: list[str] = ["code","papers"]
|
||||
|
||||
def _context(query: str, cols: list[str]) -> str:
|
||||
parts = []
|
||||
for cname in cols:
|
||||
try:
|
||||
col = get_col(cname)
|
||||
if col.count() == 0:
|
||||
continue
|
||||
res = col.query(query_texts=[query], n_results=min(TOP_K, col.count()))
|
||||
for doc, meta in zip(res["documents"][0], res["metadatas"][0]):
|
||||
parts.append(f"### {meta.get('source','')}:{meta.get('label','')}\n"
|
||||
f"```{meta.get('lang','')}\n{doc}\n```")
|
||||
except Exception as e:
|
||||
log.warning("col %s: %s", cname, e)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat(req: ChatRequest):
|
||||
query = next((m["content"] for m in reversed(req.messages)
|
||||
if m.get("role")=="user"), "")
|
||||
context = _context(query, req.collections)
|
||||
msgs = [{"role":"system","content":
|
||||
"You are a helpful coding assistant. Use the retrieved context below.\n\n"
|
||||
f"## Context\n{context}"}] + req.messages
|
||||
|
||||
payload = {"model": req.model, "messages": msgs, "stream": req.stream}
|
||||
|
||||
if req.stream:
|
||||
async def gen():
|
||||
async with httpx.AsyncClient(timeout=300) as client:
|
||||
async with client.stream("POST",
|
||||
f"{OLLAMA_URL}/v1/chat/completions", json=payload) as r:
|
||||
async for chunk in r.aiter_bytes():
|
||||
yield chunk
|
||||
return StreamingResponse(gen(), media_type="text/event-stream")
|
||||
|
||||
async with httpx.AsyncClient(timeout=300) as client:
|
||||
r = await client.post(f"{OLLAMA_URL}/v1/chat/completions", json=payload)
|
||||
return r.json()
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run("server:app", host="0.0.0.0", port=8001, reload=False)
|
||||
Executable
+275
@@ -0,0 +1,275 @@
|
||||
#!/usr/bin/env bash
|
||||
# Detect GPU VRAM and install appropriate Stable Diffusion models for
|
||||
# InvokeAI and/or ComfyUI. Run anytime — safe to re-run.
|
||||
#
|
||||
# Usage: ./setup-image-models.sh [--auto]
|
||||
# --auto Skip prompts, install the recommended default for your GPU
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[..]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[!!]${NC} $*"; }
|
||||
|
||||
AUTO=false
|
||||
[[ "${1:-}" == "--auto" ]] && AUTO=true
|
||||
|
||||
# ── Detect GPU ────────────────────────────────────────────────────────────────
|
||||
VRAM_GB=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null \
|
||||
| head -1 | awk '{printf "%d", $1/1024}' 2>/dev/null || echo "0")
|
||||
GPU_COUNT=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | wc -l || echo "0")
|
||||
GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1 || echo "None")
|
||||
TOTAL_VRAM=$((VRAM_GB * GPU_COUNT))
|
||||
|
||||
echo ""
|
||||
echo -e "${BOLD}━━━ Image Generation Model Setup ━━━${NC}"
|
||||
echo ""
|
||||
info "GPU : $GPU_NAME"
|
||||
[[ "$GPU_COUNT" -gt 1 ]] && info "GPU count : $GPU_COUNT"
|
||||
info "VRAM/card : ${VRAM_GB}GB"
|
||||
info "Total VRAM: ${TOTAL_VRAM}GB"
|
||||
echo ""
|
||||
|
||||
# ── Determine which containers are available ──────────────────────────────────
|
||||
HAS_INVOKEAI=false
|
||||
HAS_COMFYUI=false
|
||||
docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q '^invokeai$' && HAS_INVOKEAI=true
|
||||
docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q '^comfyui$' && HAS_COMFYUI=true
|
||||
|
||||
if ! $HAS_INVOKEAI && ! $HAS_COMFYUI; then
|
||||
echo -e "${RED}Error:${NC} Neither InvokeAI nor ComfyUI containers found."
|
||||
echo " Run the setup script first to deploy the AI stack."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
$HAS_INVOKEAI && info "InvokeAI : found"
|
||||
$HAS_COMFYUI && info "ComfyUI : found"
|
||||
echo ""
|
||||
|
||||
# ── Build model menu based on VRAM ────────────────────────────────────────────
|
||||
# Model VRAM requirements (generation, not just loading):
|
||||
# SD 1.5 ~4GB 512x512 native
|
||||
# SDXL ~7GB 1024x1024 native
|
||||
# SDXL Turbo ~7GB 512x512 (4-step)
|
||||
# Flux.1-schnell ~12GB fast, high quality
|
||||
# Flux.1-dev ~20GB best quality, slow
|
||||
|
||||
declare -a MODEL_IDS=()
|
||||
declare -a MODEL_NAMES=()
|
||||
declare -a MODEL_VRAM=()
|
||||
declare -a MODEL_NOTES=()
|
||||
|
||||
add_model() {
|
||||
MODEL_IDS+=("$1"); MODEL_NAMES+=("$2"); MODEL_VRAM+=("$3"); MODEL_NOTES+=("$4")
|
||||
}
|
||||
|
||||
# Always offer SD 1.5 if any GPU exists
|
||||
if [[ "$TOTAL_VRAM" -ge 4 ]]; then
|
||||
add_model "sd15" "Stable Diffusion 1.5" "4" "512px native, most LoRA compatible, fast"
|
||||
fi
|
||||
|
||||
if [[ "$TOTAL_VRAM" -ge 8 ]]; then
|
||||
add_model "sdxl" "Stable Diffusion XL" "7" "1024px native, better quality, more detail"
|
||||
add_model "sdxl-turbo" "SDXL Turbo" "7" "4-step generation, very fast, good quality"
|
||||
fi
|
||||
|
||||
if [[ "$TOTAL_VRAM" -ge 12 ]]; then
|
||||
add_model "flux-schnell" "Flux.1-schnell" "12" "fast Flux variant, excellent quality"
|
||||
fi
|
||||
|
||||
if [[ "$TOTAL_VRAM" -ge 20 ]]; then
|
||||
add_model "flux-dev" "Flux.1-dev" "20" "best quality, slower, needs lots of VRAM"
|
||||
fi
|
||||
|
||||
if [[ ${#MODEL_IDS[@]} -eq 0 ]]; then
|
||||
warn "No GPU with sufficient VRAM detected (need at least 4GB)."
|
||||
warn "CPU-only image generation is extremely slow and not recommended."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Determine default recommendation ─────────────────────────────────────────
|
||||
if [[ "$TOTAL_VRAM" -ge 20 ]]; then DEFAULT_ID="flux-dev"
|
||||
elif [[ "$TOTAL_VRAM" -ge 12 ]]; then DEFAULT_ID="sdxl"
|
||||
elif [[ "$TOTAL_VRAM" -ge 8 ]]; then DEFAULT_ID="sdxl"
|
||||
elif [[ "$TOTAL_VRAM" -ge 4 ]]; then DEFAULT_ID="sd15"
|
||||
else DEFAULT_ID="sd15"
|
||||
fi
|
||||
|
||||
echo -e "${BOLD}Available models for your ${TOTAL_VRAM}GB GPU:${NC}"
|
||||
echo ""
|
||||
for i in "${!MODEL_IDS[@]}"; do
|
||||
DEFAULT_TAG=""
|
||||
[[ "${MODEL_IDS[$i]}" == "$DEFAULT_ID" ]] && DEFAULT_TAG=" ${GREEN}← recommended${NC}"
|
||||
printf " ${BOLD}%d)${NC} %-25s ~%sGB VRAM %s%b\n" \
|
||||
$((i+1)) "${MODEL_NAMES[$i]}" "${MODEL_VRAM[$i]}" "${MODEL_NOTES[$i]}" "$DEFAULT_TAG"
|
||||
done
|
||||
echo ""
|
||||
|
||||
if $AUTO; then
|
||||
SELECTED="$DEFAULT_ID"
|
||||
info "Auto mode: installing $SELECTED"
|
||||
else
|
||||
echo -e " Enter number(s) separated by spaces, or press Enter for recommended."
|
||||
echo -e " Example: ${BOLD}1 2${NC} to install both SD 1.5 and SDXL"
|
||||
echo ""
|
||||
read -rp " Selection [recommended]: " CHOICE
|
||||
|
||||
if [[ -z "$CHOICE" ]]; then
|
||||
SELECTED="$DEFAULT_ID"
|
||||
else
|
||||
SELECTED=""
|
||||
for num in $CHOICE; do
|
||||
idx=$((num - 1))
|
||||
if [[ $idx -ge 0 && $idx -lt ${#MODEL_IDS[@]} ]]; then
|
||||
SELECTED+=" ${MODEL_IDS[$idx]}"
|
||||
else
|
||||
warn "Invalid selection: $num (skipping)"
|
||||
fi
|
||||
done
|
||||
SELECTED="${SELECTED# }"
|
||||
fi
|
||||
fi
|
||||
|
||||
[[ -z "$SELECTED" ]] && { warn "No models selected."; exit 1; }
|
||||
|
||||
echo ""
|
||||
info "Will install: $SELECTED"
|
||||
echo ""
|
||||
|
||||
# ── HuggingFace model identifiers ────────────────────────────────────────────
|
||||
declare -A HF_MODELS=(
|
||||
[sd15]="stabilityai/stable-diffusion-v1-5"
|
||||
[sdxl]="stabilityai/stable-diffusion-xl-base-1.0"
|
||||
[sdxl-turbo]="stabilityai/sdxl-turbo"
|
||||
[flux-schnell]="black-forest-labs/FLUX.1-schnell"
|
||||
[flux-dev]="black-forest-labs/FLUX.1-dev"
|
||||
)
|
||||
|
||||
declare -A MODEL_SIZES=(
|
||||
[sd15]="~4GB"
|
||||
[sdxl]="~7GB"
|
||||
[sdxl-turbo]="~7GB"
|
||||
[flux-schnell]="~12GB"
|
||||
[flux-dev]="~24GB"
|
||||
)
|
||||
|
||||
# ── Install into InvokeAI ────────────────────────────────────────────────────
|
||||
if $HAS_INVOKEAI; then
|
||||
echo -e "${BOLD}━━━ Installing into InvokeAI ━━━${NC}"
|
||||
|
||||
# Make sure container is running
|
||||
if ! docker ps --format '{{.Names}}' | grep -q '^invokeai$'; then
|
||||
info "Starting InvokeAI container..."
|
||||
docker start invokeai
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
for model_id in $SELECTED; do
|
||||
hf_id="${HF_MODELS[$model_id]:-}"
|
||||
[[ -z "$hf_id" ]] && { warn "Unknown model: $model_id"; continue; }
|
||||
info "Installing $model_id (${MODEL_SIZES[$model_id]}) → ${hf_id}..."
|
||||
info " This may take a while depending on your connection."
|
||||
|
||||
if docker exec invokeai invokeai-model-install --add "$hf_id" 2>&1; then
|
||||
ok "$model_id installed in InvokeAI"
|
||||
else
|
||||
warn "$model_id install failed in InvokeAI — try manually via Model Manager at :9090"
|
||||
fi
|
||||
echo ""
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Install into ComfyUI ─────────────────────────────────────────────────────
|
||||
if $HAS_COMFYUI; then
|
||||
echo -e "${BOLD}━━━ Installing into ComfyUI ━━━${NC}"
|
||||
info "ComfyUI downloads models on first use via its UI."
|
||||
info "To pre-download, use the ComfyUI Manager at http://localhost:8188"
|
||||
echo ""
|
||||
|
||||
# Make sure container is running
|
||||
if ! docker ps --format '{{.Names}}' | grep -q '^comfyui$'; then
|
||||
info "Starting ComfyUI container..."
|
||||
docker start comfyui
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
# For ComfyUI, download checkpoints into the models volume
|
||||
for model_id in $SELECTED; do
|
||||
hf_id="${HF_MODELS[$model_id]:-}"
|
||||
[[ -z "$hf_id" ]] && continue
|
||||
|
||||
# Check if model already exists
|
||||
CKPT_DIR="/opt/ComfyUI/models/checkpoints"
|
||||
if docker exec comfyui ls "$CKPT_DIR" 2>/dev/null | grep -qi "${model_id//-/_}"; then
|
||||
ok "$model_id already present in ComfyUI"
|
||||
continue
|
||||
fi
|
||||
|
||||
info "Downloading $model_id for ComfyUI (${MODEL_SIZES[$model_id]})..."
|
||||
info " Downloading from HuggingFace: $hf_id"
|
||||
|
||||
# Use ComfyUI's built-in download mechanism via python
|
||||
case "$model_id" in
|
||||
sd15)
|
||||
docker exec comfyui bash -c \
|
||||
"cd /opt/ComfyUI && python -c \"
|
||||
from huggingface_hub import hf_hub_download
|
||||
hf_hub_download('$hf_id', 'v1-5-pruned-emaonly.safetensors', local_dir='models/checkpoints')
|
||||
\" 2>&1" && ok "$model_id downloaded for ComfyUI" \
|
||||
|| warn "$model_id download failed — install via ComfyUI Manager UI"
|
||||
;;
|
||||
sdxl)
|
||||
docker exec comfyui bash -c \
|
||||
"cd /opt/ComfyUI && python -c \"
|
||||
from huggingface_hub import hf_hub_download
|
||||
hf_hub_download('$hf_id', 'sd_xl_base_1.0.safetensors', local_dir='models/checkpoints')
|
||||
\" 2>&1" && ok "$model_id downloaded for ComfyUI" \
|
||||
|| warn "$model_id download failed — install via ComfyUI Manager UI"
|
||||
;;
|
||||
*)
|
||||
info "$model_id: use ComfyUI Manager to install (complex model structure)"
|
||||
;;
|
||||
esac
|
||||
echo ""
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo -e "${GREEN}${BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e "${GREEN}${BOLD} Image model setup complete!${NC}"
|
||||
echo -e "${GREEN}${BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo ""
|
||||
echo -e " ${BOLD}Your GPU:${NC} $GPU_NAME — ${TOTAL_VRAM}GB VRAM"
|
||||
echo -e " ${BOLD}Installed:${NC} $SELECTED"
|
||||
echo ""
|
||||
$HAS_INVOKEAI && echo -e " ${CYAN}InvokeAI${NC} → http://localhost:9090"
|
||||
$HAS_COMFYUI && echo -e " ${CYAN}ComfyUI${NC} → http://localhost:8188"
|
||||
echo ""
|
||||
|
||||
if $HAS_INVOKEAI; then
|
||||
echo -e " ${YELLOW}InvokeAI quick start:${NC}"
|
||||
echo " 1. Open Model Manager → verify your model appears"
|
||||
echo " 2. Go to Text to Image → select the model"
|
||||
echo " 3. For inpainting: use the Unified Canvas tab"
|
||||
echo " - Upload image → brush over the area to change"
|
||||
echo " - Write what you want in that area → Invoke"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
if $HAS_COMFYUI; then
|
||||
echo -e " ${YELLOW}ComfyUI quick start:${NC}"
|
||||
echo " 1. Open ComfyUI → load a basic txt2img workflow"
|
||||
echo " 2. Select your checkpoint in the Load Checkpoint node"
|
||||
echo " 3. For Open WebUI integration: enable Dev Mode → export API workflow"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
echo -e " ${YELLOW}GPU sharing:${NC} Ollama and image gen share the GPU."
|
||||
echo " Ollama auto-unloads models after 24h idle (KEEP_ALIVE=24h)."
|
||||
echo " For immediate unload before heavy image gen:"
|
||||
echo " docker exec ollama ollama stop <model-name>"
|
||||
echo ""
|
||||
echo -e " ${YELLOW}Import LoRAs:${NC} ./invokeai-import-lora.sh <file.safetensors>"
|
||||
echo ""
|
||||
Executable
+33
@@ -0,0 +1,33 @@
|
||||
#!/bin/bash
|
||||
# Install all local-ai systemd user units and enable timers
|
||||
set -e
|
||||
|
||||
UNIT_DIR="$HOME/.config/systemd/user"
|
||||
SRC="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
mkdir -p "$UNIT_DIR"
|
||||
|
||||
for unit in "$SRC"/*.service "$SRC"/*.timer; do
|
||||
name=$(basename "$unit")
|
||||
ln -sf "$unit" "$UNIT_DIR/$name"
|
||||
echo "Linked $name"
|
||||
done
|
||||
|
||||
systemctl --user daemon-reload
|
||||
|
||||
for timer in "$SRC"/*.timer; do
|
||||
name=$(basename "$timer")
|
||||
systemctl --user enable --now "$name"
|
||||
echo "Enabled $name"
|
||||
done
|
||||
|
||||
echo ""
|
||||
systemctl --user list-timers kiwix-update.timer ollama-update.timer
|
||||
echo ""
|
||||
echo "Run manually:"
|
||||
echo " systemctl --user start kiwix-update.service"
|
||||
echo " systemctl --user start ollama-update.service"
|
||||
echo ""
|
||||
echo "Logs:"
|
||||
echo " journalctl --user -u kiwix-update.service -f"
|
||||
echo " journalctl --user -u ollama-update.service -f"
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Kiwix ZIM auto-updater
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/bash %h/local-ai/kiwix_update.sh
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Monthly Kiwix ZIM update check
|
||||
Requires=kiwix-update.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=monthly
|
||||
RandomizedDelaySec=1h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Ollama model auto-updater
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/bash %h/local-ai/ollama_update.sh
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Monthly Ollama model update check
|
||||
Requires=ollama-update.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=monthly
|
||||
RandomizedDelaySec=2h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
+2
-2
@@ -195,7 +195,7 @@ fi
|
||||
|
||||
register_service ai-gpu utilities "GPU AI stack — InvokeAI image gen + Ollama/OpenWebUI LLM (6 GB VRAM)" 9090
|
||||
|
||||
install_ai_gpu() {
|
||||
install_ai-gpu() {
|
||||
require_docker || return 1
|
||||
log_info "Installing AI GPU stack (InvokeAI + Ollama/OpenWebUI + portal)..."
|
||||
log_info "Requires: nvidia GPU with 6 GB+ VRAM, nvidia-container-toolkit installed."
|
||||
@@ -681,4 +681,4 @@ MD
|
||||
echo ""
|
||||
}
|
||||
|
||||
[[ "${_RUN_STANDALONE:-0}" == 1 ]] && install_ai_gpu
|
||||
[[ "${_RUN_STANDALONE:-0}" == 1 ]] && install_ai-gpu
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
#!/bin/bash
|
||||
# services/ai-stack.sh — Local AI Stack: a full self-hosted AI environment.
|
||||
#
|
||||
# Vendored from github.com/outis1one/local-ai into this repo under ./ai-stack and
|
||||
# copied to ~/docker/ai-stack at install time (no network clone). Bundles:
|
||||
# Ollama · Open WebUI · RAG + MCP servers · ChromaDB · SearXNG · Kiwix ·
|
||||
# Gitea · InvokeAI · ComfyUI · Portainer
|
||||
#
|
||||
# Script-driven (unlike most services here): the app ships its own VRAM-aware
|
||||
# installer (local-ai-setup.sh) that generates docker-compose.yml/.env, starts the
|
||||
# stack, and registers a `local-ai` systemd unit. This wrapper copies the vendored
|
||||
# source into place and hands off to that installer, then optionally wires cloud
|
||||
# LLM providers into Open WebUI alongside the local RAG connection.
|
||||
#
|
||||
# Open WebUI ships with built-in auth (WEBUI_AUTH=true) — no Authelia needed.
|
||||
# Distinct from `ai-gpu` (the ai-6gb-gpu repo: a leaner 3-stack GPU-swap setup for
|
||||
# 6 GB cards). Both can coexist.
|
||||
# Part of the modular post-install system (sourced by setup.sh).
|
||||
|
||||
register_service ai-stack utilities "Full self-hosted AI stack — Ollama/OpenWebUI + RAG + ComfyUI + more (local-ai)" 3000
|
||||
|
||||
install_ai-stack() {
|
||||
require_docker || return 1
|
||||
log_info "Installing Local AI Stack (Ollama + Open WebUI + RAG + image gen + more)..."
|
||||
|
||||
# Vendored application source lives in this repo at <repo>/ai-stack
|
||||
local SELF_DIR SRC_DIR
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SRC_DIR="$(cd "$SELF_DIR/.." && pwd)/ai-stack"
|
||||
|
||||
local AS_DIR="$DOCKER_DIR/ai-stack"
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "[DRY-RUN] Would copy vendored source $SRC_DIR -> $AS_DIR"
|
||||
echo "[DRY-RUN] Would optionally collect cloud LLM provider keys (Groq/DeepInfra/OpenAI/OpenRouter)"
|
||||
echo "[DRY-RUN] Would run the app installer local-ai-setup.sh (Docker/NVIDIA toolkit, VRAM-aware models, generates compose/.env, starts stack, registers systemd 'local-ai')"
|
||||
echo "[DRY-RUN] Would wire cloud providers into Open WebUI (OPENAI_API_BASE_URLS) preserving the local RAG connection"
|
||||
echo "[DRY-RUN] Would attach Open WebUI to caddy_net and configure Caddy (open-webui:8080, host port 3000)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ ! -d "$SRC_DIR" ]; then
|
||||
log_error "Vendored Local AI Stack source not found at $SRC_DIR"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# ── Copy vendored source into the docker dir ──────────────────────────────
|
||||
# The installer generates its compose/.env/systemd as siblings here (it uses
|
||||
# its own dir as BASE), matching the upstream layout. The installer never
|
||||
# overwrites a user-edited .env on re-run.
|
||||
mkdir -p "$AS_DIR"
|
||||
cp -a "$SRC_DIR/." "$AS_DIR/"
|
||||
ensure_docker_dir_ownership "$AS_DIR"
|
||||
cd "$AS_DIR" || return 1
|
||||
chmod +x ./*.sh systemd/*.sh 2>/dev/null || true
|
||||
|
||||
# ── Cloud LLM providers (optional) ────────────────────────────────────────
|
||||
# Open WebUI already uses the *singular* OPENAI_API_* slot for the local RAG
|
||||
# server. To add cloud providers we switch it to the *plural* list form and
|
||||
# keep RAG as the first entry, so RAG keeps working.
|
||||
echo ""
|
||||
log_info "Cloud LLM providers — optional, added to Open WebUI alongside local Ollama + RAG."
|
||||
log_info "All are OpenAI-compatible. Pick any combination (you enter a key for each):"
|
||||
echo ""
|
||||
log_info " 1) Groq Fast LPU inference, generous free tier. Key: https://console.groq.com/keys"
|
||||
log_info " 2) DeepInfra Cheapest host for open models, zero-retention. Key: https://deepinfra.com/dash/api_keys"
|
||||
log_info " 3) OpenAI GPT-5.x, o-series, gpt-image. Key: https://platform.openai.com/api-keys"
|
||||
log_info " 4) OpenRouter One key, 300+ models. Key: https://openrouter.ai/keys"
|
||||
echo ""
|
||||
log_info " Example: '1 2' wires Groq + DeepInfra. Leave blank to stay fully local."
|
||||
echo ""
|
||||
|
||||
local CLOUD_CHOICES=""
|
||||
prompt_text "Cloud providers to add []:" "" CLOUD_CHOICES
|
||||
|
||||
# Parallel arrays: display name, OpenAI-compatible base URL, and entered key
|
||||
declare -a CLOUD_NAMES=() CLOUD_URLS=() CLOUD_KEYS=()
|
||||
local _c _cname _curl _ckey
|
||||
for _c in $CLOUD_CHOICES; do
|
||||
_cname="" ; _curl=""
|
||||
case "$_c" in
|
||||
1) _cname="Groq"; _curl="https://api.groq.com/openai/v1" ;;
|
||||
2) _cname="DeepInfra"; _curl="https://api.deepinfra.com/v1/openai" ;;
|
||||
3) _cname="OpenAI"; _curl="https://api.openai.com/v1" ;;
|
||||
4) _cname="OpenRouter"; _curl="https://openrouter.ai/api/v1" ;;
|
||||
*) log_warning "Ignoring unknown choice '$_c'"; continue ;;
|
||||
esac
|
||||
_ckey=""
|
||||
prompt_text "$_cname API key (enter to skip):" "" _ckey
|
||||
if [ -n "$_ckey" ]; then
|
||||
CLOUD_NAMES+=("$_cname"); CLOUD_URLS+=("$_curl"); CLOUD_KEYS+=("$_ckey")
|
||||
else
|
||||
log_warning "No key for $_cname — skipping."
|
||||
fi
|
||||
done
|
||||
|
||||
# ── Hand off to the app's own installer ───────────────────────────────────
|
||||
echo ""
|
||||
log_warning "The Local AI Stack installer is heavy: it can install Docker + the NVIDIA"
|
||||
log_warning "container toolkit, pulls several GB of images, and registers a systemd unit."
|
||||
local RUN_NOW=""
|
||||
prompt_yn "Run the Local AI Stack installer now? (y/n):" "y" RUN_NOW
|
||||
|
||||
local INSTALLER_RAN=false
|
||||
if [[ "$RUN_NOW" =~ ^[Yy]$ ]]; then
|
||||
# --no-pull skips the (large, slow) Ollama model downloads when unattended.
|
||||
local _flags=""
|
||||
[ "$UNATTENDED" = true ] && _flags="--no-pull"
|
||||
if [ -f local-ai-setup.sh ]; then
|
||||
if bash local-ai-setup.sh $_flags; then
|
||||
INSTALLER_RAN=true
|
||||
log_success "Local AI Stack installer finished"
|
||||
else
|
||||
log_warning "local-ai-setup.sh reported an error — see output above"
|
||||
fi
|
||||
else
|
||||
log_error "local-ai-setup.sh missing from vendored source"
|
||||
fi
|
||||
else
|
||||
log_info "Skipped. Run later: cd $AS_DIR && bash local-ai-setup.sh"
|
||||
fi
|
||||
|
||||
# ── Wire cloud providers into the generated compose ───────────────────────
|
||||
if [ ${#CLOUD_NAMES[@]} -gt 0 ] && [ -f "$AS_DIR/docker-compose.yml" ]; then
|
||||
# Prepend the local RAG connection so RAG keeps working, then the clouds.
|
||||
local URLS="http://rag-server:8001/v1" KEYS="local-rag" _i
|
||||
for _i in "${!CLOUD_NAMES[@]}"; do
|
||||
URLS+=";${CLOUD_URLS[$_i]}"; KEYS+=";${CLOUD_KEYS[$_i]}"
|
||||
done
|
||||
|
||||
# Upsert into the stack .env (compose interpolates these; keys stay out of
|
||||
# the committed-looking compose file). Drop any existing line, then append.
|
||||
_as_set_env() {
|
||||
sed -i -E "/^#?[[:space:]]*$1=/d" "$AS_DIR/.env" 2>/dev/null
|
||||
printf '%s=%s\n' "$1" "$2" >> "$AS_DIR/.env"
|
||||
}
|
||||
touch "$AS_DIR/.env"
|
||||
_as_set_env OPENAI_API_BASE_URLS "$URLS"
|
||||
_as_set_env OPENAI_API_KEYS "$KEYS"
|
||||
chmod 600 "$AS_DIR/.env"
|
||||
|
||||
# Swap Open WebUI's singular RAG slot to the plural list form (idempotent).
|
||||
if grep -q 'OPENAI_API_BASE_URL=http://rag-server' "$AS_DIR/docker-compose.yml"; then
|
||||
sed -i 's|- OPENAI_API_BASE_URL=http://rag-server:8001/v1|- OPENAI_API_BASE_URLS=${OPENAI_API_BASE_URLS}|' "$AS_DIR/docker-compose.yml"
|
||||
sed -i 's|- OPENAI_API_KEY=local-rag|- OPENAI_API_KEYS=${OPENAI_API_KEYS}|' "$AS_DIR/docker-compose.yml"
|
||||
log_success "Cloud providers wired into Open WebUI: ${CLOUD_NAMES[*]} (local RAG preserved)"
|
||||
(cd "$AS_DIR" && docker compose up -d open-webui) \
|
||||
&& log_success "Open WebUI recreated with cloud providers" \
|
||||
|| log_warning "Could not recreate Open WebUI — run: cd $AS_DIR && docker compose up -d"
|
||||
else
|
||||
log_warning "Open WebUI RAG env not found in compose — add cloud providers via Open WebUI → Settings → Connections instead."
|
||||
fi
|
||||
ensure_docker_dir_ownership "$AS_DIR"
|
||||
elif [ ${#CLOUD_NAMES[@]} -gt 0 ]; then
|
||||
log_warning "No generated docker-compose.yml yet — add ${CLOUD_NAMES[*]} via Open WebUI → Settings → Connections after first start."
|
||||
fi
|
||||
|
||||
# ── Caddy (Open WebUI has built-in auth — no Authelia) ────────────────────
|
||||
# The generated compose doesn't join caddy_net, so attach the container by name.
|
||||
if [ -d "$DOCKER_DIR/caddy" ] && [ "$INSTALLER_RAN" = true ]; then
|
||||
docker network connect "$SITE_CADDY_NET" open-webui 2>/dev/null || true
|
||||
fi
|
||||
configure_caddy_for_service "Open WebUI" "open-webui:8080" "ai"
|
||||
|
||||
# ── Deploy notes (the app's own docs stay at $AS_DIR/README.md) ───────────
|
||||
cat > "$AS_DIR/POST-INSTALL-NOTES.md" << MD
|
||||
# Local AI Stack — deployment notes (ubuntu-post-install)
|
||||
|
||||
Vendored app source copied here from the \`ai-stack\` service. Full app docs:
|
||||
\`README.md\` in this directory. Source: github.com/outis1one/local-ai
|
||||
|
||||
## Service URLs
|
||||
| Service | URL | Auth |
|
||||
|------------|---------------------------|---------------------|
|
||||
| Open WebUI | http://localhost:3000 | built-in (first visit = admin) |
|
||||
| InvokeAI | http://localhost:9090 | none |
|
||||
| ComfyUI | http://localhost:8188 | none |
|
||||
| SearXNG | http://localhost:8888 | none |
|
||||
| Kiwix | http://localhost:8181 | none |
|
||||
| Gitea | http://localhost:3001 | built-in |
|
||||
| Portainer | https://localhost:9443 | built-in |
|
||||
|
||||
## Manage the stack
|
||||
\`\`\`bash
|
||||
cd $AS_DIR
|
||||
bash start.sh # pull latest images + docker compose up -d
|
||||
bash stop.sh # docker compose down
|
||||
bash status.sh # GPU / container / RAG health
|
||||
bash pull-models.sh # pull Ollama models (run once after first install)
|
||||
\`\`\`
|
||||
Also a systemd unit: \`sudo systemctl {start,stop,status} local-ai\`
|
||||
|
||||
## Cloud LLM providers (Open WebUI)
|
||||
Open WebUI uses an OpenAI-compatible connection list. The local RAG server is the
|
||||
first entry; any cloud providers added at install follow it. Two semicolon-separated
|
||||
lists in \`.env\`, matched by position (RAG must stay first):
|
||||
\`\`\`bash
|
||||
# $AS_DIR/.env
|
||||
OPENAI_API_BASE_URLS=http://rag-server:8001/v1;https://api.groq.com/openai/v1
|
||||
OPENAI_API_KEYS=local-rag;gsk_xxx
|
||||
cd $AS_DIR && docker compose up -d open-webui # apply
|
||||
\`\`\`
|
||||
| Provider | Base URL | Key |
|
||||
|----------|----------|-----|
|
||||
| Groq | \`https://api.groq.com/openai/v1\` | https://console.groq.com/keys |
|
||||
| DeepInfra | \`https://api.deepinfra.com/v1/openai\` | https://deepinfra.com/dash/api_keys |
|
||||
| OpenAI | \`https://api.openai.com/v1\` | https://platform.openai.com/api-keys |
|
||||
| OpenRouter | \`https://openrouter.ai/api/v1\` | https://openrouter.ai/keys |
|
||||
|
||||
Alternatively, add them at runtime in Open WebUI → Settings → Admin → Connections
|
||||
(no file edits, survives image upgrades).
|
||||
|
||||
## Update
|
||||
Re-run the \`ai-stack\` installer (refreshes vendored source, keeps your \`.env\`),
|
||||
then \`bash $AS_DIR/start.sh\`. Or in place: \`cd $AS_DIR && bash local-ai-setup.sh --force\`.
|
||||
|
||||
## Caddy
|
||||
Open WebUI is reverse-proxied as \`open-webui:8080\` on \`${SITE_CADDY_NET:-caddy_net}\`
|
||||
(attached with \`docker network connect\` after start). Other services are LAN-only by
|
||||
default — add Caddy site blocks for them if you want remote access.
|
||||
MD
|
||||
ensure_docker_dir_ownership "$AS_DIR"
|
||||
|
||||
echo ""
|
||||
echo " Open WebUI: http://localhost:3000 (chat — Ollama + RAG, built-in login)"
|
||||
echo " InvokeAI: http://localhost:9090 ComfyUI: http://localhost:8188"
|
||||
echo " SearXNG: http://localhost:8888 Kiwix: http://localhost:8181"
|
||||
echo " Gitea: http://localhost:3001 Portainer: https://localhost:9443"
|
||||
echo " App dir: $AS_DIR (app docs: README.md · deploy notes: POST-INSTALL-NOTES.md)"
|
||||
if [ ${#CLOUD_NAMES[@]} -gt 0 ]; then
|
||||
echo " Cloud LLM: ${CLOUD_NAMES[*]} (wired into Open WebUI)"
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
Reference in New Issue
Block a user