diff --git a/README.md b/README.md index 652704a..bf44f6b 100644 --- a/README.md +++ b/README.md @@ -1 +1,31 @@ -# local_proxmox \ No newline at end of file +# local_proxmox + +Proxmox VE 9.1 configuration, scripts, and VM layouts for a Dell R730xd homelab. + +## Hardware + +- **Host:** Dell PowerEdge R730xd +- **RAID controller:** PERC H730 (configured in Non-RAID/per-disk passthrough mode) +- **GPUs:** 2x Quadro P3300 4GB (VFIO passthrough, one per VM) +- **TPU:** Google Coral USB (Frigate object detection) +- **Bays:** 12x (not all populated) + +## VM Layout + +| VM ID | Role | Drives | GPU | Coral | +|-------|------|--------|-----|-------| +| 100 | Frigate NVR | Bays 1–8 | Quadro #1 (VFIO) | Yes | +| 101 | TBD | Bays 9–10 | Quadro #2 (VFIO) | No | +| 102 | TBD | Bays 11–12 | None | No | + +## Setup Order + +1. [`scripts/perc-nonraid.sh`](scripts/perc-nonraid.sh) — convert H730 to per-disk non-RAID mode +2. [`scripts/build-bay-map.sh`](scripts/build-bay-map.sh) — map physical bays to stable `/dev/disk/by-id` paths +3. [`scripts/gpu-passthrough-setup.sh`](scripts/gpu-passthrough-setup.sh) — configure IOMMU + VFIO for both Quadros +4. Apply VM configs from [`vm-configs/`](vm-configs/) +5. Deploy Frigate from [`frigate/`](frigate/) + +## Docs + +- [`docs/hardware-layout.md`](docs/hardware-layout.md) — bay/device/VM mapping worksheet diff --git a/docs/hardware-layout.md b/docs/hardware-layout.md new file mode 100644 index 0000000..bbd1040 --- /dev/null +++ b/docs/hardware-layout.md @@ -0,0 +1,65 @@ +# Hardware Layout + +Dell R730xd — fill this in after running `scripts/build-bay-map.sh`. + +## Bay → Device → VM Map + +Run `scripts/build-bay-map.sh` on the Proxmox host to auto-populate the by-id column. +Confirm bay numbers by running `ledctl locate=` to blink the drive LED. + +| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) | Notes | +|-----|-------------|------|---------------------------|-------| +| 1 | 100 | | | | +| 2 | 100 | | | | +| 3 | 100 | | | | +| 4 | 100 | | | | +| 5 | 100 | | | | +| 6 | 100 | | | | +| 7 | 100 | | | | +| 8 | 100 | | | | +| 9 | 101 | | | | +| 10 | 101 | | | | +| 11 | 102 | | | | +| 12 | 102 | | | | + +## GPU Map + +Run `lspci -nn | grep -i nvidia` on the Proxmox host and fill in the PCI addresses. + +| Slot | PCI Address (fill in) | Assigned VM | Notes | +|------|-----------------------|-------------|-------| +| GPU 1 | | 100 | Frigate decode + display | +| GPU 2 | | 101 | | + +The GPU's HDMI audio function (same address, function 1) must be passed through alongside the GPU. +Example: GPU at `01:00.0` → also pass `01:00.1`. + +## USB / TPU Map + +| Device | VID:PID | Assigned VM | Notes | +|--------|---------|-------------|-------| +| Coral (pre-init) | 1a6e:089a | 100 | Global Unichip — before first inference | +| Coral (post-init) | 18d1:9302 | 100 | Google — after first inference; pass both | + +## IOMMU Groups + +Run on Proxmox host to check groupings before passthrough: + +```bash +for d in /sys/kernel/iommu_groups/*/devices/*; do + n=${d#*/iommu_groups/*}; n=${n%%/*} + printf 'IOMMU Group %s ' "$n" + lspci -nns "${d##*/}" +done | sort -V +``` + +Each GPU should appear in its own group (with only its HDMI audio sibling). +If a GPU shares a group with other devices, those must be passed through together. + +## Network + +| VM | Interface | Bridge | Notes | +|-----|-----------|--------|-------| +| 100 | net0 | vmbr0 | | +| 101 | net0 | vmbr0 | | +| 102 | net0 | vmbr0 | | diff --git a/docs/setup-guide.md b/docs/setup-guide.md new file mode 100644 index 0000000..5010dde --- /dev/null +++ b/docs/setup-guide.md @@ -0,0 +1,457 @@ +# Dell R730xd — Proxmox VE 9.1 Setup Guide + +Complete walkthrough from bare metal to running VMs with GPU passthrough, +Coral USB, and per-VM disk assignments. + +--- + +## What You Need Before Starting + +- USB drive (8GB+) for the Proxmox installer +- A second machine or phone to read this guide and SSH +- The iDRAC IP or physical access to a monitor + keyboard +- Internet connection on the server + +--- + +## Phase 1 — Firmware Updates (do this first) + +Outdated firmware causes mysterious IOMMU failures and fan issues. Do this +before anything else. + +### Option A: Lifecycle Controller (no extra tools needed) + +1. Power on the server, press **F10** when you see the Dell splash screen +2. Lifecycle Controller → **Firmware Update** → Check for updates +3. Point it at downloads.dell.com or a local repo +4. Update **iDRAC**, **BIOS**, **PERC H730**, and any NIC/HBA firmware +5. Let it reboot as many times as it needs + +### Option B: Dell System Update (DSU) from bootable USB + +A faster alternative if Lifecycle Controller is slow or unavailable — Dell +provides a bootable ISO that auto-detects and applies all updates. + +--- + +## Phase 2 — iDRAC Setup + +iDRAC is the out-of-band management interface. You need it configured for the +fan control script to work later. + +1. Power on → press **F2** to enter System Setup → **iDRAC Settings** +2. **Network:** + - Set a static IP (easier than DHCP for a server) + - Note the IP — you'll use it for the fan control ipmitool commands +3. **User configuration:** + - Change the default `root` password +4. **IPMI over LAN:** + - iDRAC Settings → Network → IPMI Settings → **Enable IPMI over LAN: On** + - This is required for `ipmitool` fan control from the Proxmox host + +--- + +## Phase 3 — BIOS Settings + +Still in F2 System Setup. These settings are **required** for GPU and USB +passthrough to work. Missing VT-d is the #1 reason passthrough fails silently. + +| Menu path | Setting | Value | +|-----------|---------|-------| +| Processor Settings | Virtualization Technology (VT-x) | **Enabled** | +| Processor Settings | VT for Direct I/O (VT-d) | **Enabled** | +| PCI Configuration | SR-IOV Global Enable | **Enabled** | +| Boot Settings → BIOS Boot Settings | Boot Mode | **UEFI** (not Legacy/BIOS) | +| Power Management | Hard Disk Drive Sequencing | **Enabled** | +| Power Management | Power Management Policy | **Performance** | + +> **Boot Mode must be UEFI.** Proxmox's EFI boot tool (`proxmox-boot-tool`) +> only works with UEFI. Legacy BIOS mode breaks the GPU passthrough script. + +**Apply and exit. The server will reboot.** + +--- + +## Phase 4 — H730 RAID Configuration (pre-Proxmox) + +The two 2.5" rear drives are your Proxmox OS drives. You want them mirrored +so a single drive failure doesn't take down the hypervisor. + +The 3.5" drives will be converted to non-RAID later **from within Proxmox** +using the `perc-nonraid.sh` script — do not touch them here. + +### Configure the 2.5" OS drives + +1. Reboot → press **Ctrl+R** during POST to enter the H730 configuration + utility (or use Lifecycle Controller → RAID Configuration) +2. Select the controller +3. Find the two 2.5" rear drives +4. **Create new virtual disk:** + - RAID level: **RAID 1** (mirror) + - Select both 2.5" drives + - Strip size: 64KB (default) + - Name: `OS-Mirror` (optional) + - Initialize: **Fast Initialize** +5. Press **Ctrl+Alt+Delete** to reboot + +The H730 now presents a single ~X GB RAID1 virtual disk to the OS. Proxmox +will install onto this and never know there are two physical drives behind it. + +> The 3.5" drives will show as "Unconfigured Good" in the H730 — that is fine. +> Leave them alone. `perc-nonraid.sh` handles them after Proxmox is installed. + +--- + +## Phase 5 — Install Proxmox VE 9.1 + +### Prepare the USB installer + +On another machine, download the Proxmox VE ISO from proxmox.com and write it +to a USB drive: + +```bash +# Linux/macOS +dd if=proxmox-ve_*.iso of=/dev/sdX bs=1M status=progress conv=fsync +# or use Balena Etcher (Windows/Mac/Linux GUI) +``` + +### Boot and install + +1. Plug the USB into the R730xd +2. Power on → press **F11** for the one-time boot menu +3. Select the USB drive +4. At the Proxmox boot menu: **Install Proxmox VE (Graphical)** +5. **Target disk:** select the RAID1 virtual disk (`OS-Mirror`) + - Filesystem: **ext4** is simplest — the H730 RAID1 already gives you + redundancy, so ZFS mirroring here would be double-redundant overkill +6. **Location and timezone:** set to your region +7. **Password and email:** set a strong root password, enter an email +8. **Network configuration:** + - Management interface: the built-in NIC (usually `em1` or `eno1`) + - Hostname: e.g. `pve.local` + - IP: choose a static IP on your LAN (e.g. `192.168.1.10/24`) + - Gateway and DNS: your router's IP +9. Click **Install** +10. Remove USB when prompted, let it reboot + +### First login + +Open a browser on your LAN machine and go to: + +``` +https://192.168.1.10:8006 +``` + +Accept the self-signed certificate warning. Login: `root` / (your password), +Realm: **Linux PAM**. + +--- + +## Phase 6 — Proxmox Post-Install (SSH) + +SSH into the host from now on — it's faster than the web console for these +steps. + +```bash +ssh root@192.168.1.10 +``` + +### 6a. Fix the apt repositories + +Proxmox shows "no valid subscription" warnings when using the enterprise repo +without a license. Switch to the free repo: + +```bash +# Disable enterprise repo +echo "# disabled" > /etc/apt/sources.list.d/pve-enterprise.list +echo "# disabled" > /etc/apt/sources.list.d/ceph.list + +# Add no-subscription repo +echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \ + > /etc/apt/sources.list.d/pve-no-subscription.list + +apt update && apt dist-upgrade -y +reboot +``` + +### 6b. Install tools used by the scripts + +```bash +apt install -y ipmitool lsscsi ledmon hdparm git +``` + +### 6c. Clone this repo onto the host + +```bash +git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox +cd /opt/local_proxmox +chmod +x scripts/*.sh +``` + +--- + +## Phase 7 — H730: Set 3.5" Drives to Non-RAID Mode + +Now that Proxmox is running, convert the 3.5" drives to per-disk (non-RAID) +mode. The OS drives (RAID1 virtual disk) are **not affected** — the script only +targets unconfigured physical disks. + +### Install perccli + +Download from Dell support (search "PERCCLI") or copy the `.deb` to the host: + +```bash +dpkg -i perccli_*.deb +``` + +### Run the script + +```bash +bash /opt/local_proxmox/scripts/perc-nonraid.sh +``` + +Review the output, confirm when prompted. **Reboot after completion.** + +```bash +reboot +``` + +After rebooting, Proxmox will see the 3.5" drives as individual block devices +(`/dev/sdb`, `/dev/sdc`, etc.). + +--- + +## Phase 8 — Map Physical Bays to Drives + +```bash +bash /opt/local_proxmox/scripts/build-bay-map.sh +``` + +Walk the bays with `ledctl` to confirm which physical slot is which device: + +```bash +ledctl locate=/dev/sdb # LED blinks on the matching bay +ledctl locate_off=/dev/sdb +``` + +Fill in the `by-id` paths in `docs/hardware-layout.md`. You will need these +in Phase 11 when creating VMs. + +--- + +## Phase 9 — GPU Passthrough Setup + +```bash +bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh +``` + +The script: +- Adds `intel_iommu=on iommu=pt` to the kernel command line +- Blacklists `nouveau`/`nvidia` on the host +- Binds both Quadro P2200s to `vfio-pci` +- Rebuilds initramfs + +**Note the PCI addresses it prints at the end** — you'll need them in Phase 11. + +```bash +reboot +``` + +### Verify after reboot + +```bash +lspci -nnk | grep -A3 -i nvidia +``` + +Both GPUs should show `Kernel driver in use: vfio-pci`. If they still show +`nouveau`, check that `/etc/modprobe.d/blacklist-gpu.conf` exists and +`update-initramfs -u` was run. + +--- + +## Phase 10 — Fan Control and Staggered Spin-Up + +### Fan control (prevents jet-engine noise from non-Dell GPUs) + +```bash +cp /opt/local_proxmox/scripts/fan-control.sh /usr/local/sbin/fan-control.sh +cp /opt/local_proxmox/scripts/fan-control.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable --now fan-control.service +systemctl status fan-control.service +``` + +Test it worked — the fans should audibly slow down within 30 seconds. +To check current speed: `ipmitool sdr type Fan` + +### Staggered spin-up + +**Layer 1 — iDRAC BIOS** (fires at every boot, before OS): + +```bash +bash /opt/local_proxmox/scripts/stagger-spinup.sh --idrac +# If racadm is not available, the script prints the iDRAC web UI path instead. +``` + +**Layer 2 — Linux service** (handles drives coming up from standby): + +```bash +cp /opt/local_proxmox/scripts/stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh +cp /opt/local_proxmox/scripts/stagger-spinup.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable stagger-spinup.service +``` + +--- + +## Phase 11 — Create the VMs + +### Prepare the VM configs + +Edit the three example configs and replace the placeholders: + +```bash +cd /opt/local_proxmox/vm-configs + +# Replace PLACEHOLDER_BAYx with real by-id paths from Phase 8 +# Replace XX:00 with real PCI addresses from Phase 9 +nano 100-frigate.conf.example +nano 101.conf.example +nano 102.conf.example +``` + +### Create OS disks for each VM + +In the Proxmox web UI or via CLI, create the base OS disk for each VM: + +```bash +# Creates the disk slots — Proxmox generates the correct scsi0 line +qm create 100 --memory 16384 --cores 8 --name frigate --net0 virtio,bridge=vmbr0 +qm create 101 --memory 16384 --cores 8 --name vm101 --net0 virtio,bridge=vmbr0 +qm create 102 --memory 8192 --cores 4 --name vm102 --net0 virtio,bridge=vmbr0 +``` + +Then merge your edited conf into the generated config: + +```bash +# Backup generated config, then append your hardware lines +cp /etc/pve/qemu-server/100.conf /etc/pve/qemu-server/100.conf.bak +cat 100-frigate.conf.example >> /etc/pve/qemu-server/100.conf +``` + +Or just open each VM in the Proxmox web UI → **Hardware** and add: +- PCI Device → your GPU (enable PCIe, enable Primary GPU for VM 100) +- USB Device → host device → `1a6e:089a` and `18d1:9302` (VM 100 only) +- Hard Disk → (use disk passthrough, SCSI controller, path = your by-id) + +### Install a guest OS + +Boot each VM from an ISO (upload ISOs to Proxmox under local storage → +ISO Images). Ubuntu Server 22.04 LTS is a good choice for the Frigate VM. + +--- + +## Phase 12 — Set Up Frigate in VM 100 + +From inside VM 100 (SSH into the guest OS): + +### Install Docker + +```bash +apt update && apt install -y ca-certificates curl +curl -fsSL https://get.docker.com | sh +``` + +### Install NVIDIA driver + Container Toolkit + +```bash +# Add NVIDIA apt repo +curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg + +curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list + +apt update +apt install -y nvidia-driver-535 nvidia-container-toolkit + +# Configure Docker to use the NVIDIA runtime +nvidia-ctk runtime configure --runtime=docker +systemctl restart docker + +# Verify the GPU is visible +docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi +``` + +### Deploy Frigate + +```bash +mkdir -p /opt/frigate && cd /opt/frigate + +# Copy configs from the repo (or clone it inside the VM) +cp /path/to/local_proxmox/frigate/docker-compose.yml . +cp /path/to/local_proxmox/frigate/config.yml . + +# Create the recordings directory (on your passthrough data drive) +# Mount your data drives first — if using ZFS: +zpool import # or create a new pool from the raw drives +mkdir -p /mnt/frigate + +# Edit config.yml and add your camera RTSP URLs +nano config.yml + +docker compose up -d +docker compose logs -f # watch for errors on first start +``` + +Frigate UI: `http://:5000` + +--- + +## Troubleshooting + +### GPU not passing through — still shows `nouveau` + +```bash +update-initramfs -u -k all && reboot +# After reboot: +lspci -nnk | grep -A3 -i nvidia # must show vfio-pci +``` + +### IOMMU not enabled + +```bash +dmesg | grep -i iommu +# Should show: "DMAR: IOMMU enabled" +# If not: re-check Phase 3 BIOS settings (VT-d) and Phase 9 +cat /etc/kernel/cmdline # must contain intel_iommu=on iommu=pt +``` + +### Fan control not working + +```bash +# Check iDRAC IP is reachable and IPMI over LAN is enabled (Phase 2) +ipmitool -I lan -H -U root -P sdr type Fan +# Then check the service: +systemctl status fan-control.service +journalctl -u fan-control.service -n 50 +``` + +### Coral not detected in Frigate + +```bash +# In VM 100, check both USB IDs are present +lsusb | grep -E "1a6e|18d1" +# If missing, check the USB passthrough lines in the VM config +# Both usb0 (1a6e:089a) and usb1 (18d1:9302) must be present +``` + +### Drive not appearing after perc-nonraid.sh + +```bash +lsblk +# If the drive is missing, check its state in perccli: +perccli /c0 /eall /sall show +# State should be "JBOD" or "UGood" — not "Offln" or "Msng" +``` diff --git a/frigate/config.yml b/frigate/config.yml new file mode 100644 index 0000000..6f4eb55 --- /dev/null +++ b/frigate/config.yml @@ -0,0 +1,94 @@ +# Frigate NVR configuration — VM 100 (Quadro P2200 + Google Coral USB) +# Docs: https://docs.frigate.video/configuration/ + +mqtt: + enabled: false # set to true and fill in host/port if you use Home Assistant or MQTT + +# ── Object detection ───────────────────────────────────────────────────────── +# Coral USB handles all inference; CPU usage for detection is near zero. +detectors: + coral: + type: edgetpu + device: usb + +# ── FFmpeg hardware decode (Quadro P2200 NVDEC) ─────────────────────────────── +# These presets tell ffmpeg to use the GPU for H.264/H.265 decode. +# CPU usage per stream drops from ~15–30% to ~1–2%. +ffmpeg: + hwaccel_args: preset-nvidia-h264 # default for H.264 cameras + # Per-camera override available if some cameras use H.265 — see cameras section below + +# ── Global detect settings ─────────────────────────────────────────────────── +detect: + enabled: true + width: 1280 + height: 720 + fps: 5 # Coral processes 5 fps for detection; recording captures full stream + +# ── Recording ──────────────────────────────────────────────────────────────── +record: + enabled: true + retain: + days: 7 + mode: motion # only keep segments with motion (saves disk) + events: + retain: + default: 14 # keep event clips for 14 days regardless of motion-only rule + mode: active_objects + +# ── Snapshots ──────────────────────────────────────────────────────────────── +snapshots: + enabled: true + timestamp: true + bounding_box: true + retain: + default: 14 + +# ── Object filter defaults ──────────────────────────────────────────────────── +objects: + track: + - person + - car + - dog + - cat + filters: + person: + min_area: 1500 # ignore very small detections (reduces false positives) + min_score: 0.6 + threshold: 0.7 + +# ── Cameras ────────────────────────────────────────────────────────────────── +# Add one entry per camera. Duplicate and adjust as needed. +# Use {FRIGATE_RTSP_PASSWORD} to reference the env var from docker-compose.yml. + +cameras: + front_door: + ffmpeg: + inputs: + - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1 + roles: + - detect + - record + # Uncomment to override hwaccel for H.265 cameras: + # ffmpeg: + # hwaccel_args: preset-nvidia-h265 + + # back_yard: + # ffmpeg: + # inputs: + # - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1 + # roles: + # - detect + # - record + +# ── Birdseye view (optional multi-camera overview) ──────────────────────────── +birdseye: + enabled: true + mode: motion # only show cameras with recent motion + +# ── Telemetry / stats ───────────────────────────────────────────────────────── +telemetry: + stats: + amd_gpu_stats: false + intel_gpu_stats: false + network_bandwidth: false diff --git a/frigate/docker-compose.yml b/frigate/docker-compose.yml new file mode 100644 index 0000000..fdf558d --- /dev/null +++ b/frigate/docker-compose.yml @@ -0,0 +1,39 @@ +services: + frigate: + container_name: frigate + image: ghcr.io/blakeblackshear/frigate:stable + restart: unless-stopped + privileged: true # needed for Coral USB device access + runtime: nvidia # NVIDIA Container Toolkit — enables NVDEC in ffmpeg + + shm_size: "256mb" # shared memory for decoded frame buffers + # increase to 512mb if running 8+ cameras + + environment: + NVIDIA_VISIBLE_DEVICES: all + NVIDIA_DRIVER_CAPABILITIES: compute,utility,video + # Set RTSP credentials here and reference as {FRIGATE_RTSP_PASSWORD} in config.yml + FRIGATE_RTSP_PASSWORD: "changeme" + + devices: + # Coral USB — exposes full USB bus so both pre/post-init VID:PIDs work + - /dev/bus/usb:/dev/bus/usb + + volumes: + - /etc/localtime:/etc/localtime:ro + - ./config.yml:/config/config.yml:ro + - /mnt/frigate:/media/frigate # recordings and snapshots — point at your drive mount + + ports: + - "5000:5000" # Frigate web UI + - "8554:8554" # RTSP restream + - "8555:8555/tcp" # WebRTC + - "8555:8555/udp" # WebRTC + + # healthcheck so docker knows when Frigate is actually ready + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:5000/api/version"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s diff --git a/scripts/build-bay-map.sh b/scripts/build-bay-map.sh new file mode 100755 index 0000000..e07d98f --- /dev/null +++ b/scripts/build-bay-map.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# Map physical drive bays to stable /dev/disk/by-id paths. +# Run on the Proxmox host after perc-nonraid.sh and a reboot. +# Output is a table you can paste into docs/hardware-layout.md. +# +# Requires: lsscsi, ledmon (apt install lsscsi ledmon) +# Optional: perccli for enclosure/slot info + +set -euo pipefail + +for cmd in lsscsi; do + command -v "$cmd" &>/dev/null || { echo "Missing: $cmd — run: apt install $cmd"; exit 1; } +done + +echo "=== Drive inventory ===" +echo "" +printf "%-12s %-10s %-30s %-20s %s\n" "DEVICE" "SIZE" "MODEL" "SERIAL" "BY-ID PATH" +printf "%-12s %-10s %-30s %-20s %s\n" "------" "----" "-----" "------" "---------" + +for dev in /dev/sd?; do + [[ -b "$dev" ]] || continue + + size=$(lsblk -dn -o SIZE "$dev" 2>/dev/null || echo "?") + model=$(cat "/sys/block/$(basename "$dev")/device/model" 2>/dev/null | tr -d ' ' || echo "?") + serial=$(cat "/sys/block/$(basename "$dev")/device/serial" 2>/dev/null | tr -d ' ' || echo "?") + + # Prefer WWN-based by-id, fall back to scsi- or ata- + byid=$(ls -1 /dev/disk/by-id/ 2>/dev/null \ + | grep -v "\-part" \ + | while read -r link; do + target=$(readlink -f "/dev/disk/by-id/$link") + [[ "$target" == "$dev" ]] && echo "$link" && break + done | head -1 || echo "not found") + + printf "%-12s %-10s %-30s %-20s %s\n" "$dev" "$size" "$model" "$serial" "/dev/disk/by-id/$byid" +done + +echo "" +echo "=== Bay identification via LED blink ===" +echo "" +echo "To confirm which physical bay a device is in, blink its LED:" +echo " apt install ledmon" +echo " ledctl locate=/dev/sdX # LED on" +echo " ledctl locate_off=/dev/sdX # LED off" +echo "" + +if command -v perccli &>/dev/null || command -v perccli64 &>/dev/null; then + PERCCLI=$(command -v perccli || command -v perccli64) + echo "=== PERC slot info ===" + $PERCCLI /c0 /eall /sall show | grep -E "^[0-9]|Drive's position|SN|WWN" || true +fi + +echo "" +echo "Copy the BY-ID paths into docs/hardware-layout.md." diff --git a/scripts/fan-control.service b/scripts/fan-control.service new file mode 100644 index 0000000..4e87a38 --- /dev/null +++ b/scripts/fan-control.service @@ -0,0 +1,19 @@ +[Unit] +Description=Dell R730xd fan speed control (third-party GPU) +# iDRAC sets fans to 100% when non-Dell PCIe cards are detected. +# This service overrides that and manages fan speed by inlet temperature. +After=network.target +# Restart if ipmitool fails transiently (e.g. iDRAC busy at boot) +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=simple +ExecStart=/usr/local/sbin/fan-control.sh +# On stop, re-enable iDRAC auto control so fans are safe if service is removed +ExecStop=/usr/local/sbin/fan-control.sh --auto +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/fan-control.sh b/scripts/fan-control.sh new file mode 100755 index 0000000..dd901a2 --- /dev/null +++ b/scripts/fan-control.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Dell R730xd fan speed control for third-party PCIe cards (Quadro P2200, etc.) +# +# iDRAC detects non-Dell GPUs and slams fans to 100% indefinitely. +# This script disables iDRAC automatic fan control and manages speed based +# on inlet temperature, keeping the server quiet under normal load. +# +# Install: +# apt install ipmitool +# cp fan-control.sh /usr/local/sbin/fan-control.sh +# chmod +x /usr/local/sbin/fan-control.sh +# cp fan-control.service /etc/systemd/system/ +# systemctl daemon-reload && systemctl enable --now fan-control.service +# +# Manual speed test (without running as daemon): +# fan-control.sh --set 25 # set fans to 25% and exit +# fan-control.sh --auto # restore iDRAC automatic control and exit + +set -euo pipefail + +IPMI="ipmitool raw 0x30 0x30" + +# Fan speed thresholds by inlet temperature (°C → % speed) +# Tune these for your environment. Inlet temp sensor reads ambient air entering front. +declare -A SPEED_MAP=( + [0]=15 # < 30°C → 15% (near-silent) + [30]=20 # 30–39°C → 20% + [40]=30 # 40–44°C → 30% + [45]=40 # 45–49°C → 40% + [50]=55 # 50–54°C → 55% + [55]=75 # 55–59°C → 75% + [60]=100 # ≥ 60°C → 100% (safety) +) + +# Minimum speed floor — never go below this (protects drives and CPUs) +MIN_SPEED=15 + +get_inlet_temp() { + ipmitool sdr type Temperature 2>/dev/null \ + | grep -i "Inlet Temp\|Ambient\|Inlet" \ + | grep -oP '\d+(?= degrees)' \ + | head -1 || echo "35" # safe default if sensor read fails +} + +pct_to_hex() { + printf '0x%02x' "$(( $1 < 100 ? $1 : 100 ))" +} + +set_fan_speed() { + local pct=$1 + [[ $pct -lt $MIN_SPEED ]] && pct=$MIN_SPEED + local hex + hex=$(pct_to_hex "$pct") + $IPMI 0x02 0xff "$hex" +} + +disable_auto_fan() { + $IPMI 0x01 0x00 + echo "$(date): iDRAC automatic fan control DISABLED" +} + +enable_auto_fan() { + $IPMI 0x01 0x01 + echo "$(date): iDRAC automatic fan control RE-ENABLED" +} + +speed_for_temp() { + local temp=$1 + local speed=$MIN_SPEED + for threshold in $(echo "${!SPEED_MAP[@]}" | tr ' ' '\n' | sort -n); do + [[ $temp -ge $threshold ]] && speed=${SPEED_MAP[$threshold]} + done + echo "$speed" +} + +# ── Argument handling ───────────────────────────────────────────────────────── + +case "${1:-}" in + --auto) + enable_auto_fan + exit 0 + ;; + --set) + pct="${2:?Usage: fan-control.sh --set <0-100>}" + disable_auto_fan + set_fan_speed "$pct" + echo "$(date): Fans set to ${pct}%" + exit 0 + ;; + --temp) + echo "Inlet temp: $(get_inlet_temp)°C" + exit 0 + ;; + "") + # Daemon mode — fall through to loop + ;; + *) + echo "Usage: $0 [--auto | --set | --temp]" + exit 1 + ;; +esac + +# ── Daemon loop ─────────────────────────────────────────────────────────────── + +trap 'enable_auto_fan; exit 0' SIGTERM SIGINT + +echo "$(date): Fan control daemon starting" +disable_auto_fan + +last_speed=-1 + +while true; do + temp=$(get_inlet_temp) + target=$(speed_for_temp "$temp") + + if [[ $target -ne $last_speed ]]; then + set_fan_speed "$target" + echo "$(date): Inlet ${temp}°C → fans ${target}%" + last_speed=$target + fi + + sleep 30 +done diff --git a/scripts/gpu-passthrough-setup.sh b/scripts/gpu-passthrough-setup.sh new file mode 100755 index 0000000..8c20ca9 --- /dev/null +++ b/scripts/gpu-passthrough-setup.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Configure IOMMU + VFIO passthrough for two Quadro P2200 GPUs on Proxmox VE 9.x. +# Run on the Proxmox host. Requires a reboot to take effect. +# +# After running this script: +# 1. Reboot the host +# 2. Verify with: lspci -nnk | grep -A3 -i nvidia +# Driver should show 'vfio-pci', not 'nouveau' or 'nvidia' +# 3. Assign GPUs to VMs via qm set or the Proxmox UI + +set -euo pipefail + +CMDLINE_FILE="/etc/kernel/cmdline" +MODPROBE_VFIO="/etc/modprobe.d/vfio.conf" +MODPROBE_BLACKLIST="/etc/modprobe.d/blacklist-gpu.conf" +INITRAMFS_MODULES="/etc/initramfs-tools/modules" + +# ── Step 1: Check IOMMU groups ──────────────────────────────────────────────── + +echo "=== Current IOMMU groups (GPUs) ===" +for d in /sys/kernel/iommu_groups/*/devices/*; do + n=${d#*/iommu_groups/*}; n=${n%%/*} + dev=$(lspci -nns "${d##*/}" 2>/dev/null || true) + [[ "$dev" =~ VGA|3D|Display|Audio ]] && printf 'Group %3s %s\n' "$n" "$dev" +done +echo "" + +# ── Step 2: Collect GPU PCI IDs ─────────────────────────────────────────────── + +echo "=== Detected NVIDIA devices ===" +lspci -nn | grep -i nvidia +echo "" + +# Grab all NVIDIA PCI IDs (vendor:device) for vfio-pci binding +# This captures both the GPU (VGA) and its HDMI audio sibling +NVIDIA_IDS=$(lspci -nn | grep -i nvidia | grep -oP '\[\K[0-9a-f]{4}:[0-9a-f]{4}(?=\])' | sort -u | tr '\n' ',' | sed 's/,$//') + +if [[ -z "$NVIDIA_IDS" ]]; then + echo "ERROR: No NVIDIA devices found. Is the GPU installed and visible to lspci?" + exit 1 +fi + +echo "GPU PCI IDs to bind to vfio-pci: $NVIDIA_IDS" +echo "" +read -rp "Proceed with configuring VFIO passthrough? [y/N] " confirm +[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; } + +# ── Step 3: Enable IOMMU in kernel cmdline ──────────────────────────────────── + +echo "" +echo "--- Configuring kernel cmdline for IOMMU ---" + +if [[ ! -f "$CMDLINE_FILE" ]]; then + echo "ERROR: $CMDLINE_FILE not found. Is this a Proxmox EFI system?" + echo "For legacy GRUB: edit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT instead." + exit 1 +fi + +current_cmdline=$(cat "$CMDLINE_FILE") +new_cmdline="$current_cmdline" + +[[ "$new_cmdline" =~ intel_iommu=on ]] || new_cmdline="$new_cmdline intel_iommu=on" +[[ "$new_cmdline" =~ iommu=pt ]] || new_cmdline="$new_cmdline iommu=pt" + +# Deduplicate spaces +new_cmdline=$(echo "$new_cmdline" | tr -s ' ' | sed 's/^ //;s/ $//') + +echo "$new_cmdline" > "$CMDLINE_FILE" +echo "Written: $CMDLINE_FILE" +echo " $new_cmdline" + +proxmox-boot-tool refresh +echo "Boot tool refreshed." + +# ── Step 4: Blacklist host GPU drivers ──────────────────────────────────────── + +echo "" +echo "--- Blacklisting nouveau and nvidia on host ---" +cat > "$MODPROBE_BLACKLIST" < "$MODPROBE_VFIO" </dev/null || echo "$mod" >> "$INITRAMFS_MODULES" +done +echo "Updated: $INITRAMFS_MODULES" + +update-initramfs -u -k all +echo "Initramfs updated." + +# ── Step 7: Print PCI addresses for VM config ───────────────────────────────── + +echo "" +echo "=== GPU PCI addresses for VM assignment ===" +echo "Use these in qm set or the Proxmox UI (Hardware → Add → PCI Device):" +echo "" +lspci -nn | grep -i nvidia | while read -r line; do + addr=$(echo "$line" | awk '{print $1}') + desc=$(echo "$line" | cut -d' ' -f2-) + printf " hostpciN: 0000:%s,pcie=1 # %s\n" "$addr" "$desc" +done + +echo "" +echo "IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM." +echo "Example for GPU at 01:00.0 (audio at 01:00.1):" +echo " hostpci0: 0000:01:00,pcie=1,x-vga=1" +echo " (Proxmox will auto-include 01:00.1 when you use the .0 address)" +echo "" +echo "Done. Reboot the host to activate IOMMU and vfio-pci binding." diff --git a/scripts/perc-nonraid.sh b/scripts/perc-nonraid.sh new file mode 100755 index 0000000..5dbd131 --- /dev/null +++ b/scripts/perc-nonraid.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Convert PERC H730 drives to Non-RAID (per-disk passthrough) mode. +# Run once on the Proxmox host before assigning drives to VMs. +# Requires: perccli (install from Dell's website or local .deb) +# +# Dell PERC H730 does not have a true HBA/IT mode. Non-RAID mode is the +# equivalent — each disk is presented directly to the OS with SMART intact. + +set -euo pipefail + +PERCCLI=$(command -v perccli || command -v perccli64 || true) +if [[ -z "$PERCCLI" ]]; then + echo "perccli not found. Install from:" + echo " https://www.dell.com/support (search 'PERCCLI')" + echo " or: dpkg -i perccli_*.deb" + exit 1 +fi + +echo "=== Controller overview ===" +$PERCCLI show + +echo "" +echo "=== Drives on controller 0 ===" +$PERCCLI /c0 /eall /sall show + +echo "" +read -rp "Proceed with converting all non-OS drives to Non-RAID mode? [y/N] " confirm +[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; } + +# Identify enclosure IDs (typically 8 for the internal backplane on R730xd) +ENCLOSURES=$($PERCCLI /c0 /eall show | awk '/^[0-9]/{print $1}' | sort -u) + +for enc in $ENCLOSURES; do + echo "" + echo "=== Processing enclosure $enc ===" + SLOTS=$($PERCCLI /c0 /e"$enc" /sall show | awk '/UGood|Onln|JBOD|DHS/{print $2}' | sort -u) + + for slot in $SLOTS; do + echo -n " Slot $slot: setting to Good... " + $PERCCLI /c0 /e"$enc" /s"$slot" set good force 2>&1 | grep -i "success\|error\|already" || true + + echo -n " Slot $slot: setting to Non-RAID... " + $PERCCLI /c0 /e"$enc" /s"$slot" set nonraid 2>&1 | grep -i "success\|error\|already" || true + done +done + +echo "" +echo "=== Final drive state ===" +$PERCCLI /c0 /eall /sall show + +echo "" +echo "Done. Reboot the host for changes to take full effect." +echo "After reboot, run build-bay-map.sh to map bays to /dev/disk/by-id paths." diff --git a/scripts/stagger-spinup.service b/scripts/stagger-spinup.service new file mode 100644 index 0000000..067aade --- /dev/null +++ b/scripts/stagger-spinup.service @@ -0,0 +1,16 @@ +[Unit] +Description=Stagger hard drive spin-up to limit PSU current surge +# Runs early in boot, before storage services and VMs start, so drives +# are already spun up and staggered before any heavy I/O begins. +DefaultDependencies=no +After=local-fs-pre.target +Before=local-fs.target sysinit.target + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/stagger-spinup.sh --linux +RemainAfterExit=yes +StandardOutput=journal + +[Install] +WantedBy=sysinit.target diff --git a/scripts/stagger-spinup.sh b/scripts/stagger-spinup.sh new file mode 100755 index 0000000..15477ac --- /dev/null +++ b/scripts/stagger-spinup.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# Stagger hard drive spin-up on Dell R730xd to avoid PSU current surge at boot. +# +# 12 x 3.5" HDDs spinning up simultaneously can spike ~240W for 2-3 seconds. +# Staggering them 2-3 seconds apart keeps the surge well within PSU limits. +# +# TWO layers of protection: +# 1. iDRAC BIOS setting — fires at POST, before the OS loads (preferred) +# 2. Linux systemd service — staggers drives that weren't spun up at POST +# +# SETUP +# ───── +# Layer 1 (iDRAC, run once): +# bash stagger-spinup.sh --idrac +# # Requires racadm and iDRAC network access, then reboot to apply. +# +# Layer 2 (Linux service, run once): +# cp stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh +# chmod +x /usr/local/sbin/stagger-spinup.sh +# Install the systemd unit below, then: +# systemctl daemon-reload && systemctl enable stagger-spinup.service + +set -euo pipefail + +STAGGER_SECONDS=3 # delay between each drive spin-up +HDPARM=$(command -v hdparm || true) + +# ── iDRAC BIOS method (Layer 1) ─────────────────────────────────────────────── + +configure_idrac() { + if ! command -v racadm &>/dev/null; then + echo "racadm not found." + echo "Option A: Run from iDRAC SSH:" + echo " ssh root@" + echo " racadm set BIOS.StorageSettings.HddSeq Enabled" + echo " racadm jobqueue create BIOS.Setup.1-1" + echo " # Then reboot to apply." + echo "" + echo "Option B: iDRAC web UI:" + echo " System BIOS → Power Management → Hard Disk Drive Sequencing → Enabled" + echo " Apply and reboot." + return 0 + fi + + echo "Enabling iDRAC hard disk drive sequencing (staggered spin-up)..." + racadm set BIOS.StorageSettings.HddSeq Enabled + racadm jobqueue create BIOS.Setup.1-1 + echo "Job queued. Reboot for the BIOS setting to take effect." + echo "Verify after reboot with: racadm get BIOS.StorageSettings.HddSeq" +} + +# ── Linux spin-up stagger (Layer 2) ────────────────────────────────────────── +# Called by the systemd service early in boot. +# Reads each block device in sequence with a short delay, causing drives in +# standby to spin up one at a time rather than simultaneously. + +stagger_linux() { + if [[ -z "$HDPARM" ]]; then + echo "hdparm not found — apt install hdparm" + exit 1 + fi + + mapfile -t drives < <(lsblk -dn -o NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}' | sort) + + if [[ ${#drives[@]} -eq 0 ]]; then + echo "No block devices found." + exit 0 + fi + + echo "$(date): Staggering spin-up for ${#drives[@]} drives (${STAGGER_SECONDS}s apart)..." + + for dev in "${drives[@]}"; do + # Skip if device is already active (check power mode) + state=$($HDPARM -C "$dev" 2>/dev/null | grep -oP '(?<=drive state is: )\S+' || echo "unknown") + + if [[ "$state" == "standby" || "$state" == "sleeping" ]]; then + echo "$(date): Waking $dev (was: $state)" + # A zero-length read is enough to trigger spin-up + dd if="$dev" of=/dev/null bs=512 count=1 status=none 2>/dev/null || true + sleep "$STAGGER_SECONDS" + else + echo "$(date): $dev already active (state: $state) — skipping" + fi + done + + echo "$(date): Stagger complete." +} + +# ── Argument dispatch ───────────────────────────────────────────────────────── + +case "${1:-}" in + --idrac) + configure_idrac + ;; + --linux|"") + stagger_linux + ;; + *) + echo "Usage: $0 [--idrac | --linux]" + echo " --idrac Configure iDRAC BIOS staggered spin-up (one-time setup)" + echo " --linux Stagger drives via hdparm now (run by systemd service)" + exit 1 + ;; +esac diff --git a/vm-configs/100-frigate.conf.example b/vm-configs/100-frigate.conf.example new file mode 100644 index 0000000..c595f9a --- /dev/null +++ b/vm-configs/100-frigate.conf.example @@ -0,0 +1,55 @@ +# Proxmox VM 100 — Frigate NVR +# +# Copy to /etc/pve/qemu-server/100.conf after filling in: +# - hostpci0: real PCI address from gpu-passthrough-setup.sh output +# - scsi1–scsi8: real /dev/disk/by-id paths from build-bay-map.sh output +# - net0: real MAC (Proxmox generates one; leave it if creating via UI) +# +# Requirements on the Proxmox host before starting this VM: +# 1. scripts/perc-nonraid.sh — drives in non-RAID mode +# 2. scripts/gpu-passthrough-setup.sh — IOMMU + vfio-pci active, host rebooted +# 3. Coral USB plugged in + +agent: 1 +bios: ovmf +boot: order=scsi0 +cores: 8 +cpu: host +machine: q35 +memory: 16384 +name: frigate +numa: 0 +ostype: l26 +scsihw: virtio-scsi-single +sockets: 1 + +# OS disk — on Proxmox local storage, adjust pool name as needed +scsi0: local-lvm:vm-100-disk-0,cache=writeback,size=64G + +# Data drives — bays 1–8, raw disk passthrough +# Fill in by-id paths from build-bay-map.sh +scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0 +scsi2: /dev/disk/by-id/PLACEHOLDER_BAY2,size=0 +scsi3: /dev/disk/by-id/PLACEHOLDER_BAY3,size=0 +scsi4: /dev/disk/by-id/PLACEHOLDER_BAY4,size=0 +scsi5: /dev/disk/by-id/PLACEHOLDER_BAY5,size=0 +scsi6: /dev/disk/by-id/PLACEHOLDER_BAY6,size=0 +scsi7: /dev/disk/by-id/PLACEHOLDER_BAY7,size=0 +scsi8: /dev/disk/by-id/PLACEHOLDER_BAY8,size=0 + +# Quadro P2200 #1 — GPU passthrough +# Replace XX:00 with real PCI address (e.g. 03:00) +# pcie=1 uses PCIe bus; x-vga=1 passes primary display output +# Proxmox auto-includes the HDMI audio sibling (XX:00.1) +hostpci0: 0000:XX:00,pcie=1,x-vga=1 + +# Google Coral USB — pass both VID:PID values +# Coral re-enumerates after first inference load: 1a6e:089a → 18d1:9302 +usb0: host=1a6e:089a +usb1: host=18d1:9302 + +# Network +net0: virtio=BC:24:11:00:00:64,bridge=vmbr0,firewall=1 + +# EFI disk (required for OVMF/UEFI boot with q35 machine) +efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M diff --git a/vm-configs/101.conf.example b/vm-configs/101.conf.example new file mode 100644 index 0000000..f763295 --- /dev/null +++ b/vm-configs/101.conf.example @@ -0,0 +1,35 @@ +# Proxmox VM 101 — General purpose with GPU +# +# Copy to /etc/pve/qemu-server/101.conf after filling in: +# - hostpci0: real PCI address of Quadro P2200 #2 +# - scsi1–scsi2: real /dev/disk/by-id paths for bays 9–10 + +agent: 1 +bios: ovmf +boot: order=scsi0 +cores: 8 +cpu: host +machine: q35 +memory: 16384 +name: vm101 +numa: 0 +ostype: l26 +scsihw: virtio-scsi-single +sockets: 1 + +# OS disk +scsi0: local-lvm:vm-101-disk-0,cache=writeback,size=64G + +# Data drives — bays 9–10 +scsi1: /dev/disk/by-id/PLACEHOLDER_BAY9,size=0 +scsi2: /dev/disk/by-id/PLACEHOLDER_BAY10,size=0 + +# Quadro P2200 #2 +# Replace XX:00 with real PCI address from gpu-passthrough-setup.sh output +hostpci0: 0000:XX:00,pcie=1,x-vga=1 + +# Network +net0: virtio=BC:24:11:00:00:65,bridge=vmbr0,firewall=1 + +# EFI disk +efidisk0: local-lvm:vm-101-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M diff --git a/vm-configs/102.conf.example b/vm-configs/102.conf.example new file mode 100644 index 0000000..674e285 --- /dev/null +++ b/vm-configs/102.conf.example @@ -0,0 +1,30 @@ +# Proxmox VM 102 — Storage / utility VM +# +# Copy to /etc/pve/qemu-server/102.conf after filling in: +# - scsi1–scsi2: real /dev/disk/by-id paths for bays 11–12 + +agent: 1 +bios: ovmf +boot: order=scsi0 +cores: 4 +cpu: host +machine: q35 +memory: 8192 +name: vm102 +numa: 0 +ostype: l26 +scsihw: virtio-scsi-single +sockets: 1 + +# OS disk +scsi0: local-lvm:vm-102-disk-0,cache=writeback,size=32G + +# Data drives — bays 11–12 +scsi1: /dev/disk/by-id/PLACEHOLDER_BAY11,size=0 +scsi2: /dev/disk/by-id/PLACEHOLDER_BAY12,size=0 + +# Network +net0: virtio=BC:24:11:00:00:66,bridge=vmbr0,firewall=1 + +# EFI disk +efidisk0: local-lvm:vm-102-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M