ai-6gb-gpu

Local AI stack — image generation and LLM chat on a single NVIDIA GPU machine. Portal at localai.mydomain.com switches between stacks to share VRAM.

What this does

• InvokeAI — image generation with canvas inpainting (images.mydomain.com)
• Ollama + Open WebUI — local LLM chat with web search (chat.mydomain.com)
• SearXNG — local search engine for Open WebUI (no API key needed, or use Brave Search API)
• ai-portal — landing page that swaps GPU between image and chat stacks

Machine requirements

• Ubuntu 24.04 LTS
• NVIDIA GPU (tested on Quadro RTX 3000 6GB)
• Docker + NVIDIA Container Toolkit (see setup below)
• homelab-auth running on Caddy machine (for SSO)

Directory structure

~/docker
├── ai-image-gen/
│   ├── docker-compose.yml     ← InvokeAI
│   └── archive.sh             ← saves Docker images for offline restore
├── ai-llm/
│   └── docker-compose.yml     ← Ollama + Open WebUI + SearXNG
├── ai-portal/
│   ├── docker-compose.yml
│   ├── Dockerfile
│   ├── app.py                 ← Flask portal + GPU swap logic
│   └── templates/
│       ├── portal.html
│       └── switching.html
└── README.md

First time setup

1. Install Docker

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

2. Install NVIDIA driver

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

3. Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-ct-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-ct-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

4. Verify GPU visible in Docker

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Must show your GPU. If not, check driver installation.

5. Install linux-modules-extra (required for WiFi and some GPU features)

sudo apt install -y linux-modules-extra-$(uname -r)

6. Create data directories

mkdir -p ~/docker/ai-image-gen/data/invokeai
mkdir -p ~/docker/ai-llm/data/ollama
mkdir -p ~/docker/ai-llm/data/open-webui
mkdir -p ~/docker/ai-llm/data/searxng
mkdir -p ~/docker/ai-portal

7. Copy files from repo

Copy each directory from this repo to ~/docker/ on the machine. Replace mydomain.com with your real domain in:

• ai-portal/app.py
• ai-portal/templates/switching.html
• ai-portal/templates/portal.html

8. Start the portal (runs permanently)

cd ~/docker/ai-portal && docker compose up -d

9. Start whichever stack you want first

# For LLM/chat
cd ~/docker/ai-llm && docker compose up -d

# For image generation
cd ~/docker/ai-image-gen && docker compose up -d

After that, use the portal at localai.mydomain.com to switch between them.

Models

Pull once after first starting the LLM stack:

docker exec ai-llm-ollama-1 ollama pull qwen2.5-coder:7b   # coding
docker exec ai-llm-ollama-1 ollama pull qwen2.5:7b          # paper writing
docker exec ai-llm-ollama-1 ollama pull llava:7b            # vision/screenshots

First model for InvokeAI — go to images.mydomain.com → Model Manager → search sd-1.5-inpainting → Install.

Open WebUI web search setup

In Open WebUI Admin Panel → Settings → Web Search:

• Engine: Brave
• API Key: get free key at api.search.brave.com (free $5/month credits, set spending cap to Free)
• Results Count: 3

GPU swap behavior

The portal stops/starts individual containers to free VRAM:

• Switching to Images: stops Ollama, starts InvokeAI
• Switching to Chat: stops InvokeAI, starts Ollama
• Open WebUI stays up always (uses no VRAM)
• Authelia sessions stay intact (networks never go down)

Archiving images for offline restore

Run after first successful boot:

chmod +x ~/docker/ai-image-gen/archive.sh
cd ~/docker/ai-image-gen && ./archive.sh

Store the archive/ directory and data/ on external drive. To restore on a new machine: ./archive/restore.sh then docker compose up -d.

Updating

Only update when you have a specific reason. Run from the relevant stack directory:

./update.sh

Notes

• Models survive container restarts — they live in data/ volumes on the host
• VRAM is 6GB — only one GPU-heavy stack can run at a time
• InvokeAI takes ~10 seconds after start before the UI is ready
• Ollama automatically unloads models from VRAM after 5 minutes of inactivity
• WiFi requires linux-modules-extra — if WiFi is missing after kernel update, reinstall it