Merge pull request #1 from outis1one/claude/proxmox-hardware-passthrough-ylAYx

Claude/proxmox hardware passthrough yl a yx
This commit is contained in:
Outis
2026-04-24 05:35:44 -04:00
committed by GitHub
15 changed files with 1306 additions and 1 deletions
+31 -1
View File
@@ -1 +1,31 @@
# local_proxmox
# local_proxmox
Proxmox VE 9.1 configuration, scripts, and VM layouts for a Dell R730xd homelab.
## Hardware
- **Host:** Dell PowerEdge R730xd
- **RAID controller:** PERC H730 (configured in Non-RAID/per-disk passthrough mode)
- **GPUs:** 2x Quadro P3300 4GB (VFIO passthrough, one per VM)
- **TPU:** Google Coral USB (Frigate object detection)
- **Bays:** 12x (not all populated)
## VM Layout
| VM ID | Role | Drives | GPU | Coral |
|-------|------|--------|-----|-------|
| 100 | Frigate NVR | Bays 18 | Quadro #1 (VFIO) | Yes |
| 101 | TBD | Bays 910 | Quadro #2 (VFIO) | No |
| 102 | TBD | Bays 1112 | None | No |
## Setup Order
1. [`scripts/perc-nonraid.sh`](scripts/perc-nonraid.sh) — convert H730 to per-disk non-RAID mode
2. [`scripts/build-bay-map.sh`](scripts/build-bay-map.sh) — map physical bays to stable `/dev/disk/by-id` paths
3. [`scripts/gpu-passthrough-setup.sh`](scripts/gpu-passthrough-setup.sh) — configure IOMMU + VFIO for both Quadros
4. Apply VM configs from [`vm-configs/`](vm-configs/)
5. Deploy Frigate from [`frigate/`](frigate/)
## Docs
- [`docs/hardware-layout.md`](docs/hardware-layout.md) — bay/device/VM mapping worksheet
+65
View File
@@ -0,0 +1,65 @@
# Hardware Layout
Dell R730xd — fill this in after running `scripts/build-bay-map.sh`.
## Bay → Device → VM Map
Run `scripts/build-bay-map.sh` on the Proxmox host to auto-populate the by-id column.
Confirm bay numbers by running `ledctl locate=<device>` to blink the drive LED.
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) | Notes |
|-----|-------------|------|---------------------------|-------|
| 1 | 100 | | | |
| 2 | 100 | | | |
| 3 | 100 | | | |
| 4 | 100 | | | |
| 5 | 100 | | | |
| 6 | 100 | | | |
| 7 | 100 | | | |
| 8 | 100 | | | |
| 9 | 101 | | | |
| 10 | 101 | | | |
| 11 | 102 | | | |
| 12 | 102 | | | |
## GPU Map
Run `lspci -nn | grep -i nvidia` on the Proxmox host and fill in the PCI addresses.
| Slot | PCI Address (fill in) | Assigned VM | Notes |
|------|-----------------------|-------------|-------|
| GPU 1 | | 100 | Frigate decode + display |
| GPU 2 | | 101 | |
The GPU's HDMI audio function (same address, function 1) must be passed through alongside the GPU.
Example: GPU at `01:00.0` → also pass `01:00.1`.
## USB / TPU Map
| Device | VID:PID | Assigned VM | Notes |
|--------|---------|-------------|-------|
| Coral (pre-init) | 1a6e:089a | 100 | Global Unichip — before first inference |
| Coral (post-init) | 18d1:9302 | 100 | Google — after first inference; pass both |
## IOMMU Groups
Run on Proxmox host to check groupings before passthrough:
```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done | sort -V
```
Each GPU should appear in its own group (with only its HDMI audio sibling).
If a GPU shares a group with other devices, those must be passed through together.
## Network
| VM | Interface | Bridge | Notes |
|-----|-----------|--------|-------|
| 100 | net0 | vmbr0 | |
| 101 | net0 | vmbr0 | |
| 102 | net0 | vmbr0 | |
+457
View File
@@ -0,0 +1,457 @@
# Dell R730xd — Proxmox VE 9.1 Setup Guide
Complete walkthrough from bare metal to running VMs with GPU passthrough,
Coral USB, and per-VM disk assignments.
---
## What You Need Before Starting
- USB drive (8GB+) for the Proxmox installer
- A second machine or phone to read this guide and SSH
- The iDRAC IP or physical access to a monitor + keyboard
- Internet connection on the server
---
## Phase 1 — Firmware Updates (do this first)
Outdated firmware causes mysterious IOMMU failures and fan issues. Do this
before anything else.
### Option A: Lifecycle Controller (no extra tools needed)
1. Power on the server, press **F10** when you see the Dell splash screen
2. Lifecycle Controller → **Firmware Update** → Check for updates
3. Point it at downloads.dell.com or a local repo
4. Update **iDRAC**, **BIOS**, **PERC H730**, and any NIC/HBA firmware
5. Let it reboot as many times as it needs
### Option B: Dell System Update (DSU) from bootable USB
A faster alternative if Lifecycle Controller is slow or unavailable — Dell
provides a bootable ISO that auto-detects and applies all updates.
---
## Phase 2 — iDRAC Setup
iDRAC is the out-of-band management interface. You need it configured for the
fan control script to work later.
1. Power on → press **F2** to enter System Setup → **iDRAC Settings**
2. **Network:**
- Set a static IP (easier than DHCP for a server)
- Note the IP — you'll use it for the fan control ipmitool commands
3. **User configuration:**
- Change the default `root` password
4. **IPMI over LAN:**
- iDRAC Settings → Network → IPMI Settings → **Enable IPMI over LAN: On**
- This is required for `ipmitool` fan control from the Proxmox host
---
## Phase 3 — BIOS Settings
Still in F2 System Setup. These settings are **required** for GPU and USB
passthrough to work. Missing VT-d is the #1 reason passthrough fails silently.
| Menu path | Setting | Value |
|-----------|---------|-------|
| Processor Settings | Virtualization Technology (VT-x) | **Enabled** |
| Processor Settings | VT for Direct I/O (VT-d) | **Enabled** |
| PCI Configuration | SR-IOV Global Enable | **Enabled** |
| Boot Settings → BIOS Boot Settings | Boot Mode | **UEFI** (not Legacy/BIOS) |
| Power Management | Hard Disk Drive Sequencing | **Enabled** |
| Power Management | Power Management Policy | **Performance** |
> **Boot Mode must be UEFI.** Proxmox's EFI boot tool (`proxmox-boot-tool`)
> only works with UEFI. Legacy BIOS mode breaks the GPU passthrough script.
**Apply and exit. The server will reboot.**
---
## Phase 4 — H730 RAID Configuration (pre-Proxmox)
The two 2.5" rear drives are your Proxmox OS drives. You want them mirrored
so a single drive failure doesn't take down the hypervisor.
The 3.5" drives will be converted to non-RAID later **from within Proxmox**
using the `perc-nonraid.sh` script — do not touch them here.
### Configure the 2.5" OS drives
1. Reboot → press **Ctrl+R** during POST to enter the H730 configuration
utility (or use Lifecycle Controller → RAID Configuration)
2. Select the controller
3. Find the two 2.5" rear drives
4. **Create new virtual disk:**
- RAID level: **RAID 1** (mirror)
- Select both 2.5" drives
- Strip size: 64KB (default)
- Name: `OS-Mirror` (optional)
- Initialize: **Fast Initialize**
5. Press **Ctrl+Alt+Delete** to reboot
The H730 now presents a single ~X GB RAID1 virtual disk to the OS. Proxmox
will install onto this and never know there are two physical drives behind it.
> The 3.5" drives will show as "Unconfigured Good" in the H730 — that is fine.
> Leave them alone. `perc-nonraid.sh` handles them after Proxmox is installed.
---
## Phase 5 — Install Proxmox VE 9.1
### Prepare the USB installer
On another machine, download the Proxmox VE ISO from proxmox.com and write it
to a USB drive:
```bash
# Linux/macOS
dd if=proxmox-ve_*.iso of=/dev/sdX bs=1M status=progress conv=fsync
# or use Balena Etcher (Windows/Mac/Linux GUI)
```
### Boot and install
1. Plug the USB into the R730xd
2. Power on → press **F11** for the one-time boot menu
3. Select the USB drive
4. At the Proxmox boot menu: **Install Proxmox VE (Graphical)**
5. **Target disk:** select the RAID1 virtual disk (`OS-Mirror`)
- Filesystem: **ext4** is simplest — the H730 RAID1 already gives you
redundancy, so ZFS mirroring here would be double-redundant overkill
6. **Location and timezone:** set to your region
7. **Password and email:** set a strong root password, enter an email
8. **Network configuration:**
- Management interface: the built-in NIC (usually `em1` or `eno1`)
- Hostname: e.g. `pve.local`
- IP: choose a static IP on your LAN (e.g. `192.168.1.10/24`)
- Gateway and DNS: your router's IP
9. Click **Install**
10. Remove USB when prompted, let it reboot
### First login
Open a browser on your LAN machine and go to:
```
https://192.168.1.10:8006
```
Accept the self-signed certificate warning. Login: `root` / (your password),
Realm: **Linux PAM**.
---
## Phase 6 — Proxmox Post-Install (SSH)
SSH into the host from now on — it's faster than the web console for these
steps.
```bash
ssh root@192.168.1.10
```
### 6a. Fix the apt repositories
Proxmox shows "no valid subscription" warnings when using the enterprise repo
without a license. Switch to the free repo:
```bash
# Disable enterprise repo
echo "# disabled" > /etc/apt/sources.list.d/pve-enterprise.list
echo "# disabled" > /etc/apt/sources.list.d/ceph.list
# Add no-subscription repo
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
> /etc/apt/sources.list.d/pve-no-subscription.list
apt update && apt dist-upgrade -y
reboot
```
### 6b. Install tools used by the scripts
```bash
apt install -y ipmitool lsscsi ledmon hdparm git
```
### 6c. Clone this repo onto the host
```bash
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
cd /opt/local_proxmox
chmod +x scripts/*.sh
```
---
## Phase 7 — H730: Set 3.5" Drives to Non-RAID Mode
Now that Proxmox is running, convert the 3.5" drives to per-disk (non-RAID)
mode. The OS drives (RAID1 virtual disk) are **not affected** — the script only
targets unconfigured physical disks.
### Install perccli
Download from Dell support (search "PERCCLI") or copy the `.deb` to the host:
```bash
dpkg -i perccli_*.deb
```
### Run the script
```bash
bash /opt/local_proxmox/scripts/perc-nonraid.sh
```
Review the output, confirm when prompted. **Reboot after completion.**
```bash
reboot
```
After rebooting, Proxmox will see the 3.5" drives as individual block devices
(`/dev/sdb`, `/dev/sdc`, etc.).
---
## Phase 8 — Map Physical Bays to Drives
```bash
bash /opt/local_proxmox/scripts/build-bay-map.sh
```
Walk the bays with `ledctl` to confirm which physical slot is which device:
```bash
ledctl locate=/dev/sdb # LED blinks on the matching bay
ledctl locate_off=/dev/sdb
```
Fill in the `by-id` paths in `docs/hardware-layout.md`. You will need these
in Phase 11 when creating VMs.
---
## Phase 9 — GPU Passthrough Setup
```bash
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
```
The script:
- Adds `intel_iommu=on iommu=pt` to the kernel command line
- Blacklists `nouveau`/`nvidia` on the host
- Binds both Quadro P2200s to `vfio-pci`
- Rebuilds initramfs
**Note the PCI addresses it prints at the end** — you'll need them in Phase 11.
```bash
reboot
```
### Verify after reboot
```bash
lspci -nnk | grep -A3 -i nvidia
```
Both GPUs should show `Kernel driver in use: vfio-pci`. If they still show
`nouveau`, check that `/etc/modprobe.d/blacklist-gpu.conf` exists and
`update-initramfs -u` was run.
---
## Phase 10 — Fan Control and Staggered Spin-Up
### Fan control (prevents jet-engine noise from non-Dell GPUs)
```bash
cp /opt/local_proxmox/scripts/fan-control.sh /usr/local/sbin/fan-control.sh
cp /opt/local_proxmox/scripts/fan-control.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now fan-control.service
systemctl status fan-control.service
```
Test it worked — the fans should audibly slow down within 30 seconds.
To check current speed: `ipmitool sdr type Fan`
### Staggered spin-up
**Layer 1 — iDRAC BIOS** (fires at every boot, before OS):
```bash
bash /opt/local_proxmox/scripts/stagger-spinup.sh --idrac
# If racadm is not available, the script prints the iDRAC web UI path instead.
```
**Layer 2 — Linux service** (handles drives coming up from standby):
```bash
cp /opt/local_proxmox/scripts/stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
cp /opt/local_proxmox/scripts/stagger-spinup.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable stagger-spinup.service
```
---
## Phase 11 — Create the VMs
### Prepare the VM configs
Edit the three example configs and replace the placeholders:
```bash
cd /opt/local_proxmox/vm-configs
# Replace PLACEHOLDER_BAYx with real by-id paths from Phase 8
# Replace XX:00 with real PCI addresses from Phase 9
nano 100-frigate.conf.example
nano 101.conf.example
nano 102.conf.example
```
### Create OS disks for each VM
In the Proxmox web UI or via CLI, create the base OS disk for each VM:
```bash
# Creates the disk slots — Proxmox generates the correct scsi0 line
qm create 100 --memory 16384 --cores 8 --name frigate --net0 virtio,bridge=vmbr0
qm create 101 --memory 16384 --cores 8 --name vm101 --net0 virtio,bridge=vmbr0
qm create 102 --memory 8192 --cores 4 --name vm102 --net0 virtio,bridge=vmbr0
```
Then merge your edited conf into the generated config:
```bash
# Backup generated config, then append your hardware lines
cp /etc/pve/qemu-server/100.conf /etc/pve/qemu-server/100.conf.bak
cat 100-frigate.conf.example >> /etc/pve/qemu-server/100.conf
```
Or just open each VM in the Proxmox web UI → **Hardware** and add:
- PCI Device → your GPU (enable PCIe, enable Primary GPU for VM 100)
- USB Device → host device → `1a6e:089a` and `18d1:9302` (VM 100 only)
- Hard Disk → (use disk passthrough, SCSI controller, path = your by-id)
### Install a guest OS
Boot each VM from an ISO (upload ISOs to Proxmox under local storage →
ISO Images). Ubuntu Server 22.04 LTS is a good choice for the Frigate VM.
---
## Phase 12 — Set Up Frigate in VM 100
From inside VM 100 (SSH into the guest OS):
### Install Docker
```bash
apt update && apt install -y ca-certificates curl
curl -fsSL https://get.docker.com | sh
```
### Install NVIDIA driver + Container Toolkit
```bash
# Add NVIDIA apt repo
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install -y nvidia-driver-535 nvidia-container-toolkit
# Configure Docker to use the NVIDIA runtime
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker
# Verify the GPU is visible
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```
### Deploy Frigate
```bash
mkdir -p /opt/frigate && cd /opt/frigate
# Copy configs from the repo (or clone it inside the VM)
cp /path/to/local_proxmox/frigate/docker-compose.yml .
cp /path/to/local_proxmox/frigate/config.yml .
# Create the recordings directory (on your passthrough data drive)
# Mount your data drives first — if using ZFS:
zpool import <poolname> # or create a new pool from the raw drives
mkdir -p /mnt/frigate
# Edit config.yml and add your camera RTSP URLs
nano config.yml
docker compose up -d
docker compose logs -f # watch for errors on first start
```
Frigate UI: `http://<vm100-ip>:5000`
---
## Troubleshooting
### GPU not passing through — still shows `nouveau`
```bash
update-initramfs -u -k all && reboot
# After reboot:
lspci -nnk | grep -A3 -i nvidia # must show vfio-pci
```
### IOMMU not enabled
```bash
dmesg | grep -i iommu
# Should show: "DMAR: IOMMU enabled"
# If not: re-check Phase 3 BIOS settings (VT-d) and Phase 9
cat /etc/kernel/cmdline # must contain intel_iommu=on iommu=pt
```
### Fan control not working
```bash
# Check iDRAC IP is reachable and IPMI over LAN is enabled (Phase 2)
ipmitool -I lan -H <idrac-ip> -U root -P <password> sdr type Fan
# Then check the service:
systemctl status fan-control.service
journalctl -u fan-control.service -n 50
```
### Coral not detected in Frigate
```bash
# In VM 100, check both USB IDs are present
lsusb | grep -E "1a6e|18d1"
# If missing, check the USB passthrough lines in the VM config
# Both usb0 (1a6e:089a) and usb1 (18d1:9302) must be present
```
### Drive not appearing after perc-nonraid.sh
```bash
lsblk
# If the drive is missing, check its state in perccli:
perccli /c0 /eall /sall show
# State should be "JBOD" or "UGood" — not "Offln" or "Msng"
```
+94
View File
@@ -0,0 +1,94 @@
# Frigate NVR configuration — VM 100 (Quadro P2200 + Google Coral USB)
# Docs: https://docs.frigate.video/configuration/
mqtt:
enabled: false # set to true and fill in host/port if you use Home Assistant or MQTT
# ── Object detection ─────────────────────────────────────────────────────────
# Coral USB handles all inference; CPU usage for detection is near zero.
detectors:
coral:
type: edgetpu
device: usb
# ── FFmpeg hardware decode (Quadro P2200 NVDEC) ───────────────────────────────
# These presets tell ffmpeg to use the GPU for H.264/H.265 decode.
# CPU usage per stream drops from ~1530% to ~12%.
ffmpeg:
hwaccel_args: preset-nvidia-h264 # default for H.264 cameras
# Per-camera override available if some cameras use H.265 — see cameras section below
# ── Global detect settings ───────────────────────────────────────────────────
detect:
enabled: true
width: 1280
height: 720
fps: 5 # Coral processes 5 fps for detection; recording captures full stream
# ── Recording ────────────────────────────────────────────────────────────────
record:
enabled: true
retain:
days: 7
mode: motion # only keep segments with motion (saves disk)
events:
retain:
default: 14 # keep event clips for 14 days regardless of motion-only rule
mode: active_objects
# ── Snapshots ────────────────────────────────────────────────────────────────
snapshots:
enabled: true
timestamp: true
bounding_box: true
retain:
default: 14
# ── Object filter defaults ────────────────────────────────────────────────────
objects:
track:
- person
- car
- dog
- cat
filters:
person:
min_area: 1500 # ignore very small detections (reduces false positives)
min_score: 0.6
threshold: 0.7
# ── Cameras ──────────────────────────────────────────────────────────────────
# Add one entry per camera. Duplicate and adjust as needed.
# Use {FRIGATE_RTSP_PASSWORD} to reference the env var from docker-compose.yml.
cameras:
front_door:
ffmpeg:
inputs:
- path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1
roles:
- detect
- record
# Uncomment to override hwaccel for H.265 cameras:
# ffmpeg:
# hwaccel_args: preset-nvidia-h265
# back_yard:
# ffmpeg:
# inputs:
# - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1
# roles:
# - detect
# - record
# ── Birdseye view (optional multi-camera overview) ────────────────────────────
birdseye:
enabled: true
mode: motion # only show cameras with recent motion
# ── Telemetry / stats ─────────────────────────────────────────────────────────
telemetry:
stats:
amd_gpu_stats: false
intel_gpu_stats: false
network_bandwidth: false
+39
View File
@@ -0,0 +1,39 @@
services:
frigate:
container_name: frigate
image: ghcr.io/blakeblackshear/frigate:stable
restart: unless-stopped
privileged: true # needed for Coral USB device access
runtime: nvidia # NVIDIA Container Toolkit — enables NVDEC in ffmpeg
shm_size: "256mb" # shared memory for decoded frame buffers
# increase to 512mb if running 8+ cameras
environment:
NVIDIA_VISIBLE_DEVICES: all
NVIDIA_DRIVER_CAPABILITIES: compute,utility,video
# Set RTSP credentials here and reference as {FRIGATE_RTSP_PASSWORD} in config.yml
FRIGATE_RTSP_PASSWORD: "changeme"
devices:
# Coral USB — exposes full USB bus so both pre/post-init VID:PIDs work
- /dev/bus/usb:/dev/bus/usb
volumes:
- /etc/localtime:/etc/localtime:ro
- ./config.yml:/config/config.yml:ro
- /mnt/frigate:/media/frigate # recordings and snapshots — point at your drive mount
ports:
- "5000:5000" # Frigate web UI
- "8554:8554" # RTSP restream
- "8555:8555/tcp" # WebRTC
- "8555:8555/udp" # WebRTC
# healthcheck so docker knows when Frigate is actually ready
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5000/api/version"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# Map physical drive bays to stable /dev/disk/by-id paths.
# Run on the Proxmox host after perc-nonraid.sh and a reboot.
# Output is a table you can paste into docs/hardware-layout.md.
#
# Requires: lsscsi, ledmon (apt install lsscsi ledmon)
# Optional: perccli for enclosure/slot info
set -euo pipefail
for cmd in lsscsi; do
command -v "$cmd" &>/dev/null || { echo "Missing: $cmd — run: apt install $cmd"; exit 1; }
done
echo "=== Drive inventory ==="
echo ""
printf "%-12s %-10s %-30s %-20s %s\n" "DEVICE" "SIZE" "MODEL" "SERIAL" "BY-ID PATH"
printf "%-12s %-10s %-30s %-20s %s\n" "------" "----" "-----" "------" "---------"
for dev in /dev/sd?; do
[[ -b "$dev" ]] || continue
size=$(lsblk -dn -o SIZE "$dev" 2>/dev/null || echo "?")
model=$(cat "/sys/block/$(basename "$dev")/device/model" 2>/dev/null | tr -d ' ' || echo "?")
serial=$(cat "/sys/block/$(basename "$dev")/device/serial" 2>/dev/null | tr -d ' ' || echo "?")
# Prefer WWN-based by-id, fall back to scsi- or ata-
byid=$(ls -1 /dev/disk/by-id/ 2>/dev/null \
| grep -v "\-part" \
| while read -r link; do
target=$(readlink -f "/dev/disk/by-id/$link")
[[ "$target" == "$dev" ]] && echo "$link" && break
done | head -1 || echo "not found")
printf "%-12s %-10s %-30s %-20s %s\n" "$dev" "$size" "$model" "$serial" "/dev/disk/by-id/$byid"
done
echo ""
echo "=== Bay identification via LED blink ==="
echo ""
echo "To confirm which physical bay a device is in, blink its LED:"
echo " apt install ledmon"
echo " ledctl locate=/dev/sdX # LED on"
echo " ledctl locate_off=/dev/sdX # LED off"
echo ""
if command -v perccli &>/dev/null || command -v perccli64 &>/dev/null; then
PERCCLI=$(command -v perccli || command -v perccli64)
echo "=== PERC slot info ==="
$PERCCLI /c0 /eall /sall show | grep -E "^[0-9]|Drive's position|SN|WWN" || true
fi
echo ""
echo "Copy the BY-ID paths into docs/hardware-layout.md."
+19
View File
@@ -0,0 +1,19 @@
[Unit]
Description=Dell R730xd fan speed control (third-party GPU)
# iDRAC sets fans to 100% when non-Dell PCIe cards are detected.
# This service overrides that and manages fan speed by inlet temperature.
After=network.target
# Restart if ipmitool fails transiently (e.g. iDRAC busy at boot)
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
Type=simple
ExecStart=/usr/local/sbin/fan-control.sh
# On stop, re-enable iDRAC auto control so fans are safe if service is removed
ExecStop=/usr/local/sbin/fan-control.sh --auto
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
+123
View File
@@ -0,0 +1,123 @@
#!/usr/bin/env bash
# Dell R730xd fan speed control for third-party PCIe cards (Quadro P2200, etc.)
#
# iDRAC detects non-Dell GPUs and slams fans to 100% indefinitely.
# This script disables iDRAC automatic fan control and manages speed based
# on inlet temperature, keeping the server quiet under normal load.
#
# Install:
# apt install ipmitool
# cp fan-control.sh /usr/local/sbin/fan-control.sh
# chmod +x /usr/local/sbin/fan-control.sh
# cp fan-control.service /etc/systemd/system/
# systemctl daemon-reload && systemctl enable --now fan-control.service
#
# Manual speed test (without running as daemon):
# fan-control.sh --set 25 # set fans to 25% and exit
# fan-control.sh --auto # restore iDRAC automatic control and exit
set -euo pipefail
IPMI="ipmitool raw 0x30 0x30"
# Fan speed thresholds by inlet temperature (°C → % speed)
# Tune these for your environment. Inlet temp sensor reads ambient air entering front.
declare -A SPEED_MAP=(
[0]=15 # < 30°C → 15% (near-silent)
[30]=20 # 3039°C → 20%
[40]=30 # 4044°C → 30%
[45]=40 # 4549°C → 40%
[50]=55 # 5054°C → 55%
[55]=75 # 5559°C → 75%
[60]=100 # ≥ 60°C → 100% (safety)
)
# Minimum speed floor — never go below this (protects drives and CPUs)
MIN_SPEED=15
get_inlet_temp() {
ipmitool sdr type Temperature 2>/dev/null \
| grep -i "Inlet Temp\|Ambient\|Inlet" \
| grep -oP '\d+(?= degrees)' \
| head -1 || echo "35" # safe default if sensor read fails
}
pct_to_hex() {
printf '0x%02x' "$(( $1 < 100 ? $1 : 100 ))"
}
set_fan_speed() {
local pct=$1
[[ $pct -lt $MIN_SPEED ]] && pct=$MIN_SPEED
local hex
hex=$(pct_to_hex "$pct")
$IPMI 0x02 0xff "$hex"
}
disable_auto_fan() {
$IPMI 0x01 0x00
echo "$(date): iDRAC automatic fan control DISABLED"
}
enable_auto_fan() {
$IPMI 0x01 0x01
echo "$(date): iDRAC automatic fan control RE-ENABLED"
}
speed_for_temp() {
local temp=$1
local speed=$MIN_SPEED
for threshold in $(echo "${!SPEED_MAP[@]}" | tr ' ' '\n' | sort -n); do
[[ $temp -ge $threshold ]] && speed=${SPEED_MAP[$threshold]}
done
echo "$speed"
}
# ── Argument handling ─────────────────────────────────────────────────────────
case "${1:-}" in
--auto)
enable_auto_fan
exit 0
;;
--set)
pct="${2:?Usage: fan-control.sh --set <0-100>}"
disable_auto_fan
set_fan_speed "$pct"
echo "$(date): Fans set to ${pct}%"
exit 0
;;
--temp)
echo "Inlet temp: $(get_inlet_temp)°C"
exit 0
;;
"")
# Daemon mode — fall through to loop
;;
*)
echo "Usage: $0 [--auto | --set <pct> | --temp]"
exit 1
;;
esac
# ── Daemon loop ───────────────────────────────────────────────────────────────
trap 'enable_auto_fan; exit 0' SIGTERM SIGINT
echo "$(date): Fan control daemon starting"
disable_auto_fan
last_speed=-1
while true; do
temp=$(get_inlet_temp)
target=$(speed_for_temp "$temp")
if [[ $target -ne $last_speed ]]; then
set_fan_speed "$target"
echo "$(date): Inlet ${temp}°C → fans ${target}%"
last_speed=$target
fi
sleep 30
done
+131
View File
@@ -0,0 +1,131 @@
#!/usr/bin/env bash
# Configure IOMMU + VFIO passthrough for two Quadro P2200 GPUs on Proxmox VE 9.x.
# Run on the Proxmox host. Requires a reboot to take effect.
#
# After running this script:
# 1. Reboot the host
# 2. Verify with: lspci -nnk | grep -A3 -i nvidia
# Driver should show 'vfio-pci', not 'nouveau' or 'nvidia'
# 3. Assign GPUs to VMs via qm set or the Proxmox UI
set -euo pipefail
CMDLINE_FILE="/etc/kernel/cmdline"
MODPROBE_VFIO="/etc/modprobe.d/vfio.conf"
MODPROBE_BLACKLIST="/etc/modprobe.d/blacklist-gpu.conf"
INITRAMFS_MODULES="/etc/initramfs-tools/modules"
# ── Step 1: Check IOMMU groups ────────────────────────────────────────────────
echo "=== Current IOMMU groups (GPUs) ==="
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
dev=$(lspci -nns "${d##*/}" 2>/dev/null || true)
[[ "$dev" =~ VGA|3D|Display|Audio ]] && printf 'Group %3s %s\n' "$n" "$dev"
done
echo ""
# ── Step 2: Collect GPU PCI IDs ───────────────────────────────────────────────
echo "=== Detected NVIDIA devices ==="
lspci -nn | grep -i nvidia
echo ""
# Grab all NVIDIA PCI IDs (vendor:device) for vfio-pci binding
# This captures both the GPU (VGA) and its HDMI audio sibling
NVIDIA_IDS=$(lspci -nn | grep -i nvidia | grep -oP '\[\K[0-9a-f]{4}:[0-9a-f]{4}(?=\])' | sort -u | tr '\n' ',' | sed 's/,$//')
if [[ -z "$NVIDIA_IDS" ]]; then
echo "ERROR: No NVIDIA devices found. Is the GPU installed and visible to lspci?"
exit 1
fi
echo "GPU PCI IDs to bind to vfio-pci: $NVIDIA_IDS"
echo ""
read -rp "Proceed with configuring VFIO passthrough? [y/N] " confirm
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
# ── Step 3: Enable IOMMU in kernel cmdline ────────────────────────────────────
echo ""
echo "--- Configuring kernel cmdline for IOMMU ---"
if [[ ! -f "$CMDLINE_FILE" ]]; then
echo "ERROR: $CMDLINE_FILE not found. Is this a Proxmox EFI system?"
echo "For legacy GRUB: edit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT instead."
exit 1
fi
current_cmdline=$(cat "$CMDLINE_FILE")
new_cmdline="$current_cmdline"
[[ "$new_cmdline" =~ intel_iommu=on ]] || new_cmdline="$new_cmdline intel_iommu=on"
[[ "$new_cmdline" =~ iommu=pt ]] || new_cmdline="$new_cmdline iommu=pt"
# Deduplicate spaces
new_cmdline=$(echo "$new_cmdline" | tr -s ' ' | sed 's/^ //;s/ $//')
echo "$new_cmdline" > "$CMDLINE_FILE"
echo "Written: $CMDLINE_FILE"
echo " $new_cmdline"
proxmox-boot-tool refresh
echo "Boot tool refreshed."
# ── Step 4: Blacklist host GPU drivers ────────────────────────────────────────
echo ""
echo "--- Blacklisting nouveau and nvidia on host ---"
cat > "$MODPROBE_BLACKLIST" <<EOF
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia_drm
blacklist nvidia_modeset
options nouveau modeset=0
EOF
echo "Written: $MODPROBE_BLACKLIST"
# ── Step 5: Bind GPUs to vfio-pci ────────────────────────────────────────────
echo ""
echo "--- Binding GPU IDs to vfio-pci ---"
cat > "$MODPROBE_VFIO" <<EOF
options vfio-pci ids=$NVIDIA_IDS
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
EOF
echo "Written: $MODPROBE_VFIO"
echo " IDs: $NVIDIA_IDS"
# ── Step 6: Load vfio modules early in initramfs ─────────────────────────────
echo ""
echo "--- Adding vfio modules to initramfs ---"
for mod in vfio vfio_iommu_type1 vfio_pci vfio_pci_core; do
grep -qxF "$mod" "$INITRAMFS_MODULES" 2>/dev/null || echo "$mod" >> "$INITRAMFS_MODULES"
done
echo "Updated: $INITRAMFS_MODULES"
update-initramfs -u -k all
echo "Initramfs updated."
# ── Step 7: Print PCI addresses for VM config ─────────────────────────────────
echo ""
echo "=== GPU PCI addresses for VM assignment ==="
echo "Use these in qm set or the Proxmox UI (Hardware → Add → PCI Device):"
echo ""
lspci -nn | grep -i nvidia | while read -r line; do
addr=$(echo "$line" | awk '{print $1}')
desc=$(echo "$line" | cut -d' ' -f2-)
printf " hostpciN: 0000:%s,pcie=1 # %s\n" "$addr" "$desc"
done
echo ""
echo "IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM."
echo "Example for GPU at 01:00.0 (audio at 01:00.1):"
echo " hostpci0: 0000:01:00,pcie=1,x-vga=1"
echo " (Proxmox will auto-include 01:00.1 when you use the .0 address)"
echo ""
echo "Done. Reboot the host to activate IOMMU and vfio-pci binding."
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# Convert PERC H730 drives to Non-RAID (per-disk passthrough) mode.
# Run once on the Proxmox host before assigning drives to VMs.
# Requires: perccli (install from Dell's website or local .deb)
#
# Dell PERC H730 does not have a true HBA/IT mode. Non-RAID mode is the
# equivalent — each disk is presented directly to the OS with SMART intact.
set -euo pipefail
PERCCLI=$(command -v perccli || command -v perccli64 || true)
if [[ -z "$PERCCLI" ]]; then
echo "perccli not found. Install from:"
echo " https://www.dell.com/support (search 'PERCCLI')"
echo " or: dpkg -i perccli_*.deb"
exit 1
fi
echo "=== Controller overview ==="
$PERCCLI show
echo ""
echo "=== Drives on controller 0 ==="
$PERCCLI /c0 /eall /sall show
echo ""
read -rp "Proceed with converting all non-OS drives to Non-RAID mode? [y/N] " confirm
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
# Identify enclosure IDs (typically 8 for the internal backplane on R730xd)
ENCLOSURES=$($PERCCLI /c0 /eall show | awk '/^[0-9]/{print $1}' | sort -u)
for enc in $ENCLOSURES; do
echo ""
echo "=== Processing enclosure $enc ==="
SLOTS=$($PERCCLI /c0 /e"$enc" /sall show | awk '/UGood|Onln|JBOD|DHS/{print $2}' | sort -u)
for slot in $SLOTS; do
echo -n " Slot $slot: setting to Good... "
$PERCCLI /c0 /e"$enc" /s"$slot" set good force 2>&1 | grep -i "success\|error\|already" || true
echo -n " Slot $slot: setting to Non-RAID... "
$PERCCLI /c0 /e"$enc" /s"$slot" set nonraid 2>&1 | grep -i "success\|error\|already" || true
done
done
echo ""
echo "=== Final drive state ==="
$PERCCLI /c0 /eall /sall show
echo ""
echo "Done. Reboot the host for changes to take full effect."
echo "After reboot, run build-bay-map.sh to map bays to /dev/disk/by-id paths."
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=Stagger hard drive spin-up to limit PSU current surge
# Runs early in boot, before storage services and VMs start, so drives
# are already spun up and staggered before any heavy I/O begins.
DefaultDependencies=no
After=local-fs-pre.target
Before=local-fs.target sysinit.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/stagger-spinup.sh --linux
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=sysinit.target
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# Stagger hard drive spin-up on Dell R730xd to avoid PSU current surge at boot.
#
# 12 x 3.5" HDDs spinning up simultaneously can spike ~240W for 2-3 seconds.
# Staggering them 2-3 seconds apart keeps the surge well within PSU limits.
#
# TWO layers of protection:
# 1. iDRAC BIOS setting — fires at POST, before the OS loads (preferred)
# 2. Linux systemd service — staggers drives that weren't spun up at POST
#
# SETUP
# ─────
# Layer 1 (iDRAC, run once):
# bash stagger-spinup.sh --idrac
# # Requires racadm and iDRAC network access, then reboot to apply.
#
# Layer 2 (Linux service, run once):
# cp stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
# chmod +x /usr/local/sbin/stagger-spinup.sh
# Install the systemd unit below, then:
# systemctl daemon-reload && systemctl enable stagger-spinup.service
set -euo pipefail
STAGGER_SECONDS=3 # delay between each drive spin-up
HDPARM=$(command -v hdparm || true)
# ── iDRAC BIOS method (Layer 1) ───────────────────────────────────────────────
configure_idrac() {
if ! command -v racadm &>/dev/null; then
echo "racadm not found."
echo "Option A: Run from iDRAC SSH:"
echo " ssh root@<idrac-ip>"
echo " racadm set BIOS.StorageSettings.HddSeq Enabled"
echo " racadm jobqueue create BIOS.Setup.1-1"
echo " # Then reboot to apply."
echo ""
echo "Option B: iDRAC web UI:"
echo " System BIOS → Power Management → Hard Disk Drive Sequencing → Enabled"
echo " Apply and reboot."
return 0
fi
echo "Enabling iDRAC hard disk drive sequencing (staggered spin-up)..."
racadm set BIOS.StorageSettings.HddSeq Enabled
racadm jobqueue create BIOS.Setup.1-1
echo "Job queued. Reboot for the BIOS setting to take effect."
echo "Verify after reboot with: racadm get BIOS.StorageSettings.HddSeq"
}
# ── Linux spin-up stagger (Layer 2) ──────────────────────────────────────────
# Called by the systemd service early in boot.
# Reads each block device in sequence with a short delay, causing drives in
# standby to spin up one at a time rather than simultaneously.
stagger_linux() {
if [[ -z "$HDPARM" ]]; then
echo "hdparm not found — apt install hdparm"
exit 1
fi
mapfile -t drives < <(lsblk -dn -o NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}' | sort)
if [[ ${#drives[@]} -eq 0 ]]; then
echo "No block devices found."
exit 0
fi
echo "$(date): Staggering spin-up for ${#drives[@]} drives (${STAGGER_SECONDS}s apart)..."
for dev in "${drives[@]}"; do
# Skip if device is already active (check power mode)
state=$($HDPARM -C "$dev" 2>/dev/null | grep -oP '(?<=drive state is: )\S+' || echo "unknown")
if [[ "$state" == "standby" || "$state" == "sleeping" ]]; then
echo "$(date): Waking $dev (was: $state)"
# A zero-length read is enough to trigger spin-up
dd if="$dev" of=/dev/null bs=512 count=1 status=none 2>/dev/null || true
sleep "$STAGGER_SECONDS"
else
echo "$(date): $dev already active (state: $state) — skipping"
fi
done
echo "$(date): Stagger complete."
}
# ── Argument dispatch ─────────────────────────────────────────────────────────
case "${1:-}" in
--idrac)
configure_idrac
;;
--linux|"")
stagger_linux
;;
*)
echo "Usage: $0 [--idrac | --linux]"
echo " --idrac Configure iDRAC BIOS staggered spin-up (one-time setup)"
echo " --linux Stagger drives via hdparm now (run by systemd service)"
exit 1
;;
esac
+55
View File
@@ -0,0 +1,55 @@
# Proxmox VM 100 — Frigate NVR
#
# Copy to /etc/pve/qemu-server/100.conf after filling in:
# - hostpci0: real PCI address from gpu-passthrough-setup.sh output
# - scsi1scsi8: real /dev/disk/by-id paths from build-bay-map.sh output
# - net0: real MAC (Proxmox generates one; leave it if creating via UI)
#
# Requirements on the Proxmox host before starting this VM:
# 1. scripts/perc-nonraid.sh — drives in non-RAID mode
# 2. scripts/gpu-passthrough-setup.sh — IOMMU + vfio-pci active, host rebooted
# 3. Coral USB plugged in
agent: 1
bios: ovmf
boot: order=scsi0
cores: 8
cpu: host
machine: q35
memory: 16384
name: frigate
numa: 0
ostype: l26
scsihw: virtio-scsi-single
sockets: 1
# OS disk — on Proxmox local storage, adjust pool name as needed
scsi0: local-lvm:vm-100-disk-0,cache=writeback,size=64G
# Data drives — bays 18, raw disk passthrough
# Fill in by-id paths from build-bay-map.sh
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY2,size=0
scsi3: /dev/disk/by-id/PLACEHOLDER_BAY3,size=0
scsi4: /dev/disk/by-id/PLACEHOLDER_BAY4,size=0
scsi5: /dev/disk/by-id/PLACEHOLDER_BAY5,size=0
scsi6: /dev/disk/by-id/PLACEHOLDER_BAY6,size=0
scsi7: /dev/disk/by-id/PLACEHOLDER_BAY7,size=0
scsi8: /dev/disk/by-id/PLACEHOLDER_BAY8,size=0
# Quadro P2200 #1 — GPU passthrough
# Replace XX:00 with real PCI address (e.g. 03:00)
# pcie=1 uses PCIe bus; x-vga=1 passes primary display output
# Proxmox auto-includes the HDMI audio sibling (XX:00.1)
hostpci0: 0000:XX:00,pcie=1,x-vga=1
# Google Coral USB — pass both VID:PID values
# Coral re-enumerates after first inference load: 1a6e:089a → 18d1:9302
usb0: host=1a6e:089a
usb1: host=18d1:9302
# Network
net0: virtio=BC:24:11:00:00:64,bridge=vmbr0,firewall=1
# EFI disk (required for OVMF/UEFI boot with q35 machine)
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
+35
View File
@@ -0,0 +1,35 @@
# Proxmox VM 101 — General purpose with GPU
#
# Copy to /etc/pve/qemu-server/101.conf after filling in:
# - hostpci0: real PCI address of Quadro P2200 #2
# - scsi1scsi2: real /dev/disk/by-id paths for bays 910
agent: 1
bios: ovmf
boot: order=scsi0
cores: 8
cpu: host
machine: q35
memory: 16384
name: vm101
numa: 0
ostype: l26
scsihw: virtio-scsi-single
sockets: 1
# OS disk
scsi0: local-lvm:vm-101-disk-0,cache=writeback,size=64G
# Data drives — bays 910
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY9,size=0
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY10,size=0
# Quadro P2200 #2
# Replace XX:00 with real PCI address from gpu-passthrough-setup.sh output
hostpci0: 0000:XX:00,pcie=1,x-vga=1
# Network
net0: virtio=BC:24:11:00:00:65,bridge=vmbr0,firewall=1
# EFI disk
efidisk0: local-lvm:vm-101-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
+30
View File
@@ -0,0 +1,30 @@
# Proxmox VM 102 — Storage / utility VM
#
# Copy to /etc/pve/qemu-server/102.conf after filling in:
# - scsi1scsi2: real /dev/disk/by-id paths for bays 1112
agent: 1
bios: ovmf
boot: order=scsi0
cores: 4
cpu: host
machine: q35
memory: 8192
name: vm102
numa: 0
ostype: l26
scsihw: virtio-scsi-single
sockets: 1
# OS disk
scsi0: local-lvm:vm-102-disk-0,cache=writeback,size=32G
# Data drives — bays 1112
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY11,size=0
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY12,size=0
# Network
net0: virtio=BC:24:11:00:00:66,bridge=vmbr0,firewall=1
# EFI disk
efidisk0: local-lvm:vm-102-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M