157 lines
5.0 KiB
Markdown
157 lines
5.0 KiB
Markdown
# ai-6gb-gpu
|
|
|
|
Local AI stack — image generation and LLM chat on a single NVIDIA GPU machine.
|
|
Portal at `localai.mydomain.com` switches between stacks to share VRAM.
|
|
|
|
## What this does
|
|
|
|
- **InvokeAI** — image generation with canvas inpainting (`images.mydomain.com`)
|
|
- **Ollama + Open WebUI** — local LLM chat with web search (`chat.mydomain.com`)
|
|
- **SearXNG** — local search engine for Open WebUI (no API key needed, or use Brave Search API)
|
|
- **ai-portal** — landing page that swaps GPU between image and chat stacks
|
|
|
|
## Machine requirements
|
|
|
|
- Ubuntu 24.04 LTS
|
|
- NVIDIA GPU (tested on Quadro RTX 3000 6GB)
|
|
- Docker + NVIDIA Container Toolkit (see setup below)
|
|
- homelab-auth running on Caddy machine (for SSO)
|
|
|
|
## Directory structure
|
|
|
|
```
|
|
~/docker
|
|
├── ai-image-gen/
|
|
│ ├── docker-compose.yml ← InvokeAI
|
|
│ └── archive.sh ← saves Docker images for offline restore
|
|
├── ai-llm/
|
|
│ └── docker-compose.yml ← Ollama + Open WebUI + SearXNG
|
|
├── ai-portal/
|
|
│ ├── docker-compose.yml
|
|
│ ├── Dockerfile
|
|
│ ├── app.py ← Flask portal + GPU swap logic
|
|
│ └── templates/
|
|
│ ├── portal.html
|
|
│ └── switching.html
|
|
└── README.md
|
|
```
|
|
|
|
## First time setup
|
|
|
|
### 1. Install Docker
|
|
```bash
|
|
curl -fsSL https://get.docker.com | sh
|
|
sudo usermod -aG docker $USER
|
|
newgrp docker
|
|
```
|
|
|
|
### 2. Install NVIDIA driver
|
|
```bash
|
|
sudo apt update && sudo apt install -y ubuntu-drivers-common
|
|
sudo ubuntu-drivers install
|
|
sudo reboot
|
|
```
|
|
|
|
### 3. Install NVIDIA Container Toolkit
|
|
```bash
|
|
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
|
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-ct-keyring.gpg
|
|
|
|
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
|
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-ct-keyring.gpg] https://#g' \
|
|
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
|
|
|
sudo apt update && sudo apt install -y nvidia-container-toolkit
|
|
sudo nvidia-ctk runtime configure --runtime=docker
|
|
sudo systemctl restart docker
|
|
```
|
|
|
|
### 4. Verify GPU visible in Docker
|
|
```bash
|
|
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
|
|
```
|
|
Must show your GPU. If not, check driver installation.
|
|
|
|
### 5. Install linux-modules-extra (required for WiFi and some GPU features)
|
|
```bash
|
|
sudo apt install -y linux-modules-extra-$(uname -r)
|
|
```
|
|
|
|
### 6. Create data directories
|
|
```bash
|
|
mkdir -p ~/docker/ai-image-gen/data/invokeai
|
|
mkdir -p ~/docker/ai-llm/data/ollama
|
|
mkdir -p ~/docker/ai-llm/data/open-webui
|
|
mkdir -p ~/docker/ai-llm/data/searxng
|
|
mkdir -p ~/docker/ai-portal
|
|
```
|
|
|
|
### 7. Copy files from repo
|
|
Copy each directory from this repo to `~/docker/` on the machine.
|
|
Replace `mydomain.com` with your real domain in:
|
|
- `ai-portal/app.py`
|
|
- `ai-portal/templates/switching.html`
|
|
- `ai-portal/templates/portal.html`
|
|
|
|
### 8. Start the portal (runs permanently)
|
|
```bash
|
|
cd ~/docker/ai-portal && docker compose up -d
|
|
```
|
|
|
|
### 9. Start whichever stack you want first
|
|
```bash
|
|
# For LLM/chat
|
|
cd ~/docker/ai-llm && docker compose up -d
|
|
|
|
# For image generation
|
|
cd ~/docker/ai-image-gen && docker compose up -d
|
|
```
|
|
|
|
After that, use the portal at `localai.mydomain.com` to switch between them.
|
|
|
|
## Models
|
|
|
|
Pull once after first starting the LLM stack:
|
|
```bash
|
|
docker exec ai-llm-ollama-1 ollama pull qwen2.5-coder:7b # coding
|
|
docker exec ai-llm-ollama-1 ollama pull qwen2.5:7b # paper writing
|
|
docker exec ai-llm-ollama-1 ollama pull llava:7b # vision/screenshots
|
|
```
|
|
|
|
First model for InvokeAI — go to `images.mydomain.com` → Model Manager → search `sd-1.5-inpainting` → Install.
|
|
|
|
## Open WebUI web search setup
|
|
In Open WebUI Admin Panel → Settings → Web Search:
|
|
- Engine: Brave
|
|
- API Key: get free key at `api.search.brave.com` (free $5/month credits, set spending cap to Free)
|
|
- Results Count: 3
|
|
|
|
## GPU swap behavior
|
|
The portal stops/starts individual containers to free VRAM:
|
|
- Switching to Images: stops Ollama, starts InvokeAI
|
|
- Switching to Chat: stops InvokeAI, starts Ollama
|
|
- Open WebUI stays up always (uses no VRAM)
|
|
- Authelia sessions stay intact (networks never go down)
|
|
|
|
## Archiving images for offline restore
|
|
Run after first successful boot:
|
|
```bash
|
|
chmod +x ~/docker/ai-image-gen/archive.sh
|
|
cd ~/docker/ai-image-gen && ./archive.sh
|
|
```
|
|
Store the `archive/` directory and `data/` on external drive.
|
|
To restore on a new machine: `./archive/restore.sh` then `docker compose up -d`.
|
|
|
|
## Updating
|
|
Only update when you have a specific reason. Run from the relevant stack directory:
|
|
```bash
|
|
./update.sh
|
|
```
|
|
|
|
## Notes
|
|
- Models survive container restarts — they live in `data/` volumes on the host
|
|
- VRAM is 6GB — only one GPU-heavy stack can run at a time
|
|
- InvokeAI takes ~10 seconds after start before the UI is ready
|
|
- Ollama automatically unloads models from VRAM after 5 minutes of inactivity
|
|
- WiFi requires `linux-modules-extra` — if WiFi is missing after kernel update, reinstall it
|