# ai-6gb-gpu

Local AI stack — image generation and LLM chat on a single NVIDIA GPU machine.
Portal at `localai.mydomain.com` switches between stacks to share VRAM.

## What this does

- **InvokeAI** — image generation with canvas inpainting (`images.mydomain.com`)
- **Ollama + Open WebUI** — local LLM chat with web search (`chat.mydomain.com`)
- **SearXNG** — local search engine for Open WebUI (no API key needed, or use Brave Search API)
- **ai-portal** — landing page that swaps GPU between image and chat stacks

## Machine requirements

- Ubuntu 24.04 LTS
- NVIDIA GPU (tested on Quadro RTX 3000 6GB)
- Docker + NVIDIA Container Toolkit (see setup below)
- homelab-auth running on Caddy machine (for SSO)

## Directory structure

```
~/docker
├── ai-image-gen/
│   ├── docker-compose.yml     ← InvokeAI
│   └── archive.sh             ← saves Docker images for offline restore
├── ai-llm/
│   └── docker-compose.yml     ← Ollama + Open WebUI + SearXNG
├── ai-portal/
│   ├── docker-compose.yml
│   ├── Dockerfile
│   ├── app.py                 ← Flask portal + GPU swap logic
│   └── templates/
│       ├── portal.html
│       └── switching.html
└── README.md
```

## First time setup

### 1. Install Docker
```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
```

### 2. Install NVIDIA driver
```bash
sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot
```

### 3. Install NVIDIA Container Toolkit
```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-ct-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-ct-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

### 4. Verify GPU visible in Docker
```bash
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```
Must show your GPU. If not, check driver installation.

### 5. Install linux-modules-extra (required for WiFi and some GPU features)
```bash
sudo apt install -y linux-modules-extra-$(uname -r)
```

### 6. Create data directories
```bash
mkdir -p ~/docker/ai-image-gen/data/invokeai
mkdir -p ~/docker/ai-llm/data/ollama
mkdir -p ~/docker/ai-llm/data/open-webui
mkdir -p ~/docker/ai-llm/data/searxng
mkdir -p ~/docker/ai-portal
```

### 7. Copy files from repo
Copy each directory from this repo to `~/docker/` on the machine.
Replace `mydomain.com` with your real domain in:
- `ai-portal/app.py`
- `ai-portal/templates/switching.html`
- `ai-portal/templates/portal.html`

### 8. Start the portal (runs permanently)
```bash
cd ~/docker/ai-portal && docker compose up -d
```

### 9. Start whichever stack you want first
```bash
# For LLM/chat
cd ~/docker/ai-llm && docker compose up -d

# For image generation
cd ~/docker/ai-image-gen && docker compose up -d
```

After that, use the portal at `localai.mydomain.com` to switch between them.

## Models

Pull once after first starting the LLM stack:
```bash
docker exec ai-llm-ollama-1 ollama pull qwen2.5-coder:7b   # coding
docker exec ai-llm-ollama-1 ollama pull qwen2.5:7b          # paper writing
docker exec ai-llm-ollama-1 ollama pull llava:7b            # vision/screenshots
```

First model for InvokeAI — go to `images.mydomain.com` → Model Manager → search `sd-1.5-inpainting` → Install.

## Open WebUI web search setup
In Open WebUI Admin Panel → Settings → Web Search:
- Engine: Brave
- API Key: get free key at `api.search.brave.com` (free $5/month credits, set spending cap to Free)
- Results Count: 3

## GPU swap behavior
The portal stops/starts individual containers to free VRAM:
- Switching to Images: stops Ollama, starts InvokeAI
- Switching to Chat: stops InvokeAI, starts Ollama
- Open WebUI stays up always (uses no VRAM)
- Authelia sessions stay intact (networks never go down)

## Archiving images for offline restore
Run after first successful boot:
```bash
chmod +x ~/docker/ai-image-gen/archive.sh
cd ~/docker/ai-image-gen && ./archive.sh
```
Store the `archive/` directory and `data/` on external drive.
To restore on a new machine: `./archive/restore.sh` then `docker compose up -d`.

## Updating
Only update when you have a specific reason. Run from the relevant stack directory:
```bash
./update.sh
```

## Notes
- Models survive container restarts — they live in `data/` volumes on the host
- VRAM is 6GB — only one GPU-heavy stack can run at a time
- InvokeAI takes ~10 seconds after start before the UI is ready
- Ollama automatically unloads models from VRAM after 5 minutes of inactivity
- WiFi requires `linux-modules-extra` — if WiFi is missing after kernel update, reinstall it