Merge pull request #1 from outis1one/claude/proxmox-hardware-passthrough-ylAYx
Claude/proxmox hardware passthrough yl a yx
This commit is contained in:
@@ -1 +1,31 @@
|
||||
# local_proxmox
|
||||
# local_proxmox
|
||||
|
||||
Proxmox VE 9.1 configuration, scripts, and VM layouts for a Dell R730xd homelab.
|
||||
|
||||
## Hardware
|
||||
|
||||
- **Host:** Dell PowerEdge R730xd
|
||||
- **RAID controller:** PERC H730 (configured in Non-RAID/per-disk passthrough mode)
|
||||
- **GPUs:** 2x Quadro P3300 4GB (VFIO passthrough, one per VM)
|
||||
- **TPU:** Google Coral USB (Frigate object detection)
|
||||
- **Bays:** 12x (not all populated)
|
||||
|
||||
## VM Layout
|
||||
|
||||
| VM ID | Role | Drives | GPU | Coral |
|
||||
|-------|------|--------|-----|-------|
|
||||
| 100 | Frigate NVR | Bays 1–8 | Quadro #1 (VFIO) | Yes |
|
||||
| 101 | TBD | Bays 9–10 | Quadro #2 (VFIO) | No |
|
||||
| 102 | TBD | Bays 11–12 | None | No |
|
||||
|
||||
## Setup Order
|
||||
|
||||
1. [`scripts/perc-nonraid.sh`](scripts/perc-nonraid.sh) — convert H730 to per-disk non-RAID mode
|
||||
2. [`scripts/build-bay-map.sh`](scripts/build-bay-map.sh) — map physical bays to stable `/dev/disk/by-id` paths
|
||||
3. [`scripts/gpu-passthrough-setup.sh`](scripts/gpu-passthrough-setup.sh) — configure IOMMU + VFIO for both Quadros
|
||||
4. Apply VM configs from [`vm-configs/`](vm-configs/)
|
||||
5. Deploy Frigate from [`frigate/`](frigate/)
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/hardware-layout.md`](docs/hardware-layout.md) — bay/device/VM mapping worksheet
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# Hardware Layout
|
||||
|
||||
Dell R730xd — fill this in after running `scripts/build-bay-map.sh`.
|
||||
|
||||
## Bay → Device → VM Map
|
||||
|
||||
Run `scripts/build-bay-map.sh` on the Proxmox host to auto-populate the by-id column.
|
||||
Confirm bay numbers by running `ledctl locate=<device>` to blink the drive LED.
|
||||
|
||||
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) | Notes |
|
||||
|-----|-------------|------|---------------------------|-------|
|
||||
| 1 | 100 | | | |
|
||||
| 2 | 100 | | | |
|
||||
| 3 | 100 | | | |
|
||||
| 4 | 100 | | | |
|
||||
| 5 | 100 | | | |
|
||||
| 6 | 100 | | | |
|
||||
| 7 | 100 | | | |
|
||||
| 8 | 100 | | | |
|
||||
| 9 | 101 | | | |
|
||||
| 10 | 101 | | | |
|
||||
| 11 | 102 | | | |
|
||||
| 12 | 102 | | | |
|
||||
|
||||
## GPU Map
|
||||
|
||||
Run `lspci -nn | grep -i nvidia` on the Proxmox host and fill in the PCI addresses.
|
||||
|
||||
| Slot | PCI Address (fill in) | Assigned VM | Notes |
|
||||
|------|-----------------------|-------------|-------|
|
||||
| GPU 1 | | 100 | Frigate decode + display |
|
||||
| GPU 2 | | 101 | |
|
||||
|
||||
The GPU's HDMI audio function (same address, function 1) must be passed through alongside the GPU.
|
||||
Example: GPU at `01:00.0` → also pass `01:00.1`.
|
||||
|
||||
## USB / TPU Map
|
||||
|
||||
| Device | VID:PID | Assigned VM | Notes |
|
||||
|--------|---------|-------------|-------|
|
||||
| Coral (pre-init) | 1a6e:089a | 100 | Global Unichip — before first inference |
|
||||
| Coral (post-init) | 18d1:9302 | 100 | Google — after first inference; pass both |
|
||||
|
||||
## IOMMU Groups
|
||||
|
||||
Run on Proxmox host to check groupings before passthrough:
|
||||
|
||||
```bash
|
||||
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
||||
n=${d#*/iommu_groups/*}; n=${n%%/*}
|
||||
printf 'IOMMU Group %s ' "$n"
|
||||
lspci -nns "${d##*/}"
|
||||
done | sort -V
|
||||
```
|
||||
|
||||
Each GPU should appear in its own group (with only its HDMI audio sibling).
|
||||
If a GPU shares a group with other devices, those must be passed through together.
|
||||
|
||||
## Network
|
||||
|
||||
| VM | Interface | Bridge | Notes |
|
||||
|-----|-----------|--------|-------|
|
||||
| 100 | net0 | vmbr0 | |
|
||||
| 101 | net0 | vmbr0 | |
|
||||
| 102 | net0 | vmbr0 | |
|
||||
@@ -0,0 +1,457 @@
|
||||
# Dell R730xd — Proxmox VE 9.1 Setup Guide
|
||||
|
||||
Complete walkthrough from bare metal to running VMs with GPU passthrough,
|
||||
Coral USB, and per-VM disk assignments.
|
||||
|
||||
---
|
||||
|
||||
## What You Need Before Starting
|
||||
|
||||
- USB drive (8GB+) for the Proxmox installer
|
||||
- A second machine or phone to read this guide and SSH
|
||||
- The iDRAC IP or physical access to a monitor + keyboard
|
||||
- Internet connection on the server
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Firmware Updates (do this first)
|
||||
|
||||
Outdated firmware causes mysterious IOMMU failures and fan issues. Do this
|
||||
before anything else.
|
||||
|
||||
### Option A: Lifecycle Controller (no extra tools needed)
|
||||
|
||||
1. Power on the server, press **F10** when you see the Dell splash screen
|
||||
2. Lifecycle Controller → **Firmware Update** → Check for updates
|
||||
3. Point it at downloads.dell.com or a local repo
|
||||
4. Update **iDRAC**, **BIOS**, **PERC H730**, and any NIC/HBA firmware
|
||||
5. Let it reboot as many times as it needs
|
||||
|
||||
### Option B: Dell System Update (DSU) from bootable USB
|
||||
|
||||
A faster alternative if Lifecycle Controller is slow or unavailable — Dell
|
||||
provides a bootable ISO that auto-detects and applies all updates.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — iDRAC Setup
|
||||
|
||||
iDRAC is the out-of-band management interface. You need it configured for the
|
||||
fan control script to work later.
|
||||
|
||||
1. Power on → press **F2** to enter System Setup → **iDRAC Settings**
|
||||
2. **Network:**
|
||||
- Set a static IP (easier than DHCP for a server)
|
||||
- Note the IP — you'll use it for the fan control ipmitool commands
|
||||
3. **User configuration:**
|
||||
- Change the default `root` password
|
||||
4. **IPMI over LAN:**
|
||||
- iDRAC Settings → Network → IPMI Settings → **Enable IPMI over LAN: On**
|
||||
- This is required for `ipmitool` fan control from the Proxmox host
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — BIOS Settings
|
||||
|
||||
Still in F2 System Setup. These settings are **required** for GPU and USB
|
||||
passthrough to work. Missing VT-d is the #1 reason passthrough fails silently.
|
||||
|
||||
| Menu path | Setting | Value |
|
||||
|-----------|---------|-------|
|
||||
| Processor Settings | Virtualization Technology (VT-x) | **Enabled** |
|
||||
| Processor Settings | VT for Direct I/O (VT-d) | **Enabled** |
|
||||
| PCI Configuration | SR-IOV Global Enable | **Enabled** |
|
||||
| Boot Settings → BIOS Boot Settings | Boot Mode | **UEFI** (not Legacy/BIOS) |
|
||||
| Power Management | Hard Disk Drive Sequencing | **Enabled** |
|
||||
| Power Management | Power Management Policy | **Performance** |
|
||||
|
||||
> **Boot Mode must be UEFI.** Proxmox's EFI boot tool (`proxmox-boot-tool`)
|
||||
> only works with UEFI. Legacy BIOS mode breaks the GPU passthrough script.
|
||||
|
||||
**Apply and exit. The server will reboot.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — H730 RAID Configuration (pre-Proxmox)
|
||||
|
||||
The two 2.5" rear drives are your Proxmox OS drives. You want them mirrored
|
||||
so a single drive failure doesn't take down the hypervisor.
|
||||
|
||||
The 3.5" drives will be converted to non-RAID later **from within Proxmox**
|
||||
using the `perc-nonraid.sh` script — do not touch them here.
|
||||
|
||||
### Configure the 2.5" OS drives
|
||||
|
||||
1. Reboot → press **Ctrl+R** during POST to enter the H730 configuration
|
||||
utility (or use Lifecycle Controller → RAID Configuration)
|
||||
2. Select the controller
|
||||
3. Find the two 2.5" rear drives
|
||||
4. **Create new virtual disk:**
|
||||
- RAID level: **RAID 1** (mirror)
|
||||
- Select both 2.5" drives
|
||||
- Strip size: 64KB (default)
|
||||
- Name: `OS-Mirror` (optional)
|
||||
- Initialize: **Fast Initialize**
|
||||
5. Press **Ctrl+Alt+Delete** to reboot
|
||||
|
||||
The H730 now presents a single ~X GB RAID1 virtual disk to the OS. Proxmox
|
||||
will install onto this and never know there are two physical drives behind it.
|
||||
|
||||
> The 3.5" drives will show as "Unconfigured Good" in the H730 — that is fine.
|
||||
> Leave them alone. `perc-nonraid.sh` handles them after Proxmox is installed.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Install Proxmox VE 9.1
|
||||
|
||||
### Prepare the USB installer
|
||||
|
||||
On another machine, download the Proxmox VE ISO from proxmox.com and write it
|
||||
to a USB drive:
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
dd if=proxmox-ve_*.iso of=/dev/sdX bs=1M status=progress conv=fsync
|
||||
# or use Balena Etcher (Windows/Mac/Linux GUI)
|
||||
```
|
||||
|
||||
### Boot and install
|
||||
|
||||
1. Plug the USB into the R730xd
|
||||
2. Power on → press **F11** for the one-time boot menu
|
||||
3. Select the USB drive
|
||||
4. At the Proxmox boot menu: **Install Proxmox VE (Graphical)**
|
||||
5. **Target disk:** select the RAID1 virtual disk (`OS-Mirror`)
|
||||
- Filesystem: **ext4** is simplest — the H730 RAID1 already gives you
|
||||
redundancy, so ZFS mirroring here would be double-redundant overkill
|
||||
6. **Location and timezone:** set to your region
|
||||
7. **Password and email:** set a strong root password, enter an email
|
||||
8. **Network configuration:**
|
||||
- Management interface: the built-in NIC (usually `em1` or `eno1`)
|
||||
- Hostname: e.g. `pve.local`
|
||||
- IP: choose a static IP on your LAN (e.g. `192.168.1.10/24`)
|
||||
- Gateway and DNS: your router's IP
|
||||
9. Click **Install**
|
||||
10. Remove USB when prompted, let it reboot
|
||||
|
||||
### First login
|
||||
|
||||
Open a browser on your LAN machine and go to:
|
||||
|
||||
```
|
||||
https://192.168.1.10:8006
|
||||
```
|
||||
|
||||
Accept the self-signed certificate warning. Login: `root` / (your password),
|
||||
Realm: **Linux PAM**.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Proxmox Post-Install (SSH)
|
||||
|
||||
SSH into the host from now on — it's faster than the web console for these
|
||||
steps.
|
||||
|
||||
```bash
|
||||
ssh root@192.168.1.10
|
||||
```
|
||||
|
||||
### 6a. Fix the apt repositories
|
||||
|
||||
Proxmox shows "no valid subscription" warnings when using the enterprise repo
|
||||
without a license. Switch to the free repo:
|
||||
|
||||
```bash
|
||||
# Disable enterprise repo
|
||||
echo "# disabled" > /etc/apt/sources.list.d/pve-enterprise.list
|
||||
echo "# disabled" > /etc/apt/sources.list.d/ceph.list
|
||||
|
||||
# Add no-subscription repo
|
||||
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
|
||||
> /etc/apt/sources.list.d/pve-no-subscription.list
|
||||
|
||||
apt update && apt dist-upgrade -y
|
||||
reboot
|
||||
```
|
||||
|
||||
### 6b. Install tools used by the scripts
|
||||
|
||||
```bash
|
||||
apt install -y ipmitool lsscsi ledmon hdparm git
|
||||
```
|
||||
|
||||
### 6c. Clone this repo onto the host
|
||||
|
||||
```bash
|
||||
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
|
||||
cd /opt/local_proxmox
|
||||
chmod +x scripts/*.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — H730: Set 3.5" Drives to Non-RAID Mode
|
||||
|
||||
Now that Proxmox is running, convert the 3.5" drives to per-disk (non-RAID)
|
||||
mode. The OS drives (RAID1 virtual disk) are **not affected** — the script only
|
||||
targets unconfigured physical disks.
|
||||
|
||||
### Install perccli
|
||||
|
||||
Download from Dell support (search "PERCCLI") or copy the `.deb` to the host:
|
||||
|
||||
```bash
|
||||
dpkg -i perccli_*.deb
|
||||
```
|
||||
|
||||
### Run the script
|
||||
|
||||
```bash
|
||||
bash /opt/local_proxmox/scripts/perc-nonraid.sh
|
||||
```
|
||||
|
||||
Review the output, confirm when prompted. **Reboot after completion.**
|
||||
|
||||
```bash
|
||||
reboot
|
||||
```
|
||||
|
||||
After rebooting, Proxmox will see the 3.5" drives as individual block devices
|
||||
(`/dev/sdb`, `/dev/sdc`, etc.).
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Map Physical Bays to Drives
|
||||
|
||||
```bash
|
||||
bash /opt/local_proxmox/scripts/build-bay-map.sh
|
||||
```
|
||||
|
||||
Walk the bays with `ledctl` to confirm which physical slot is which device:
|
||||
|
||||
```bash
|
||||
ledctl locate=/dev/sdb # LED blinks on the matching bay
|
||||
ledctl locate_off=/dev/sdb
|
||||
```
|
||||
|
||||
Fill in the `by-id` paths in `docs/hardware-layout.md`. You will need these
|
||||
in Phase 11 when creating VMs.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 — GPU Passthrough Setup
|
||||
|
||||
```bash
|
||||
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
|
||||
```
|
||||
|
||||
The script:
|
||||
- Adds `intel_iommu=on iommu=pt` to the kernel command line
|
||||
- Blacklists `nouveau`/`nvidia` on the host
|
||||
- Binds both Quadro P2200s to `vfio-pci`
|
||||
- Rebuilds initramfs
|
||||
|
||||
**Note the PCI addresses it prints at the end** — you'll need them in Phase 11.
|
||||
|
||||
```bash
|
||||
reboot
|
||||
```
|
||||
|
||||
### Verify after reboot
|
||||
|
||||
```bash
|
||||
lspci -nnk | grep -A3 -i nvidia
|
||||
```
|
||||
|
||||
Both GPUs should show `Kernel driver in use: vfio-pci`. If they still show
|
||||
`nouveau`, check that `/etc/modprobe.d/blacklist-gpu.conf` exists and
|
||||
`update-initramfs -u` was run.
|
||||
|
||||
---
|
||||
|
||||
## Phase 10 — Fan Control and Staggered Spin-Up
|
||||
|
||||
### Fan control (prevents jet-engine noise from non-Dell GPUs)
|
||||
|
||||
```bash
|
||||
cp /opt/local_proxmox/scripts/fan-control.sh /usr/local/sbin/fan-control.sh
|
||||
cp /opt/local_proxmox/scripts/fan-control.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now fan-control.service
|
||||
systemctl status fan-control.service
|
||||
```
|
||||
|
||||
Test it worked — the fans should audibly slow down within 30 seconds.
|
||||
To check current speed: `ipmitool sdr type Fan`
|
||||
|
||||
### Staggered spin-up
|
||||
|
||||
**Layer 1 — iDRAC BIOS** (fires at every boot, before OS):
|
||||
|
||||
```bash
|
||||
bash /opt/local_proxmox/scripts/stagger-spinup.sh --idrac
|
||||
# If racadm is not available, the script prints the iDRAC web UI path instead.
|
||||
```
|
||||
|
||||
**Layer 2 — Linux service** (handles drives coming up from standby):
|
||||
|
||||
```bash
|
||||
cp /opt/local_proxmox/scripts/stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
|
||||
cp /opt/local_proxmox/scripts/stagger-spinup.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable stagger-spinup.service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 11 — Create the VMs
|
||||
|
||||
### Prepare the VM configs
|
||||
|
||||
Edit the three example configs and replace the placeholders:
|
||||
|
||||
```bash
|
||||
cd /opt/local_proxmox/vm-configs
|
||||
|
||||
# Replace PLACEHOLDER_BAYx with real by-id paths from Phase 8
|
||||
# Replace XX:00 with real PCI addresses from Phase 9
|
||||
nano 100-frigate.conf.example
|
||||
nano 101.conf.example
|
||||
nano 102.conf.example
|
||||
```
|
||||
|
||||
### Create OS disks for each VM
|
||||
|
||||
In the Proxmox web UI or via CLI, create the base OS disk for each VM:
|
||||
|
||||
```bash
|
||||
# Creates the disk slots — Proxmox generates the correct scsi0 line
|
||||
qm create 100 --memory 16384 --cores 8 --name frigate --net0 virtio,bridge=vmbr0
|
||||
qm create 101 --memory 16384 --cores 8 --name vm101 --net0 virtio,bridge=vmbr0
|
||||
qm create 102 --memory 8192 --cores 4 --name vm102 --net0 virtio,bridge=vmbr0
|
||||
```
|
||||
|
||||
Then merge your edited conf into the generated config:
|
||||
|
||||
```bash
|
||||
# Backup generated config, then append your hardware lines
|
||||
cp /etc/pve/qemu-server/100.conf /etc/pve/qemu-server/100.conf.bak
|
||||
cat 100-frigate.conf.example >> /etc/pve/qemu-server/100.conf
|
||||
```
|
||||
|
||||
Or just open each VM in the Proxmox web UI → **Hardware** and add:
|
||||
- PCI Device → your GPU (enable PCIe, enable Primary GPU for VM 100)
|
||||
- USB Device → host device → `1a6e:089a` and `18d1:9302` (VM 100 only)
|
||||
- Hard Disk → (use disk passthrough, SCSI controller, path = your by-id)
|
||||
|
||||
### Install a guest OS
|
||||
|
||||
Boot each VM from an ISO (upload ISOs to Proxmox under local storage →
|
||||
ISO Images). Ubuntu Server 22.04 LTS is a good choice for the Frigate VM.
|
||||
|
||||
---
|
||||
|
||||
## Phase 12 — Set Up Frigate in VM 100
|
||||
|
||||
From inside VM 100 (SSH into the guest OS):
|
||||
|
||||
### Install Docker
|
||||
|
||||
```bash
|
||||
apt update && apt install -y ca-certificates curl
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
```
|
||||
|
||||
### Install NVIDIA driver + Container Toolkit
|
||||
|
||||
```bash
|
||||
# Add NVIDIA apt repo
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
||||
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||
|
||||
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
||||
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
|
||||
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||
|
||||
apt update
|
||||
apt install -y nvidia-driver-535 nvidia-container-toolkit
|
||||
|
||||
# Configure Docker to use the NVIDIA runtime
|
||||
nvidia-ctk runtime configure --runtime=docker
|
||||
systemctl restart docker
|
||||
|
||||
# Verify the GPU is visible
|
||||
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
|
||||
```
|
||||
|
||||
### Deploy Frigate
|
||||
|
||||
```bash
|
||||
mkdir -p /opt/frigate && cd /opt/frigate
|
||||
|
||||
# Copy configs from the repo (or clone it inside the VM)
|
||||
cp /path/to/local_proxmox/frigate/docker-compose.yml .
|
||||
cp /path/to/local_proxmox/frigate/config.yml .
|
||||
|
||||
# Create the recordings directory (on your passthrough data drive)
|
||||
# Mount your data drives first — if using ZFS:
|
||||
zpool import <poolname> # or create a new pool from the raw drives
|
||||
mkdir -p /mnt/frigate
|
||||
|
||||
# Edit config.yml and add your camera RTSP URLs
|
||||
nano config.yml
|
||||
|
||||
docker compose up -d
|
||||
docker compose logs -f # watch for errors on first start
|
||||
```
|
||||
|
||||
Frigate UI: `http://<vm100-ip>:5000`
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### GPU not passing through — still shows `nouveau`
|
||||
|
||||
```bash
|
||||
update-initramfs -u -k all && reboot
|
||||
# After reboot:
|
||||
lspci -nnk | grep -A3 -i nvidia # must show vfio-pci
|
||||
```
|
||||
|
||||
### IOMMU not enabled
|
||||
|
||||
```bash
|
||||
dmesg | grep -i iommu
|
||||
# Should show: "DMAR: IOMMU enabled"
|
||||
# If not: re-check Phase 3 BIOS settings (VT-d) and Phase 9
|
||||
cat /etc/kernel/cmdline # must contain intel_iommu=on iommu=pt
|
||||
```
|
||||
|
||||
### Fan control not working
|
||||
|
||||
```bash
|
||||
# Check iDRAC IP is reachable and IPMI over LAN is enabled (Phase 2)
|
||||
ipmitool -I lan -H <idrac-ip> -U root -P <password> sdr type Fan
|
||||
# Then check the service:
|
||||
systemctl status fan-control.service
|
||||
journalctl -u fan-control.service -n 50
|
||||
```
|
||||
|
||||
### Coral not detected in Frigate
|
||||
|
||||
```bash
|
||||
# In VM 100, check both USB IDs are present
|
||||
lsusb | grep -E "1a6e|18d1"
|
||||
# If missing, check the USB passthrough lines in the VM config
|
||||
# Both usb0 (1a6e:089a) and usb1 (18d1:9302) must be present
|
||||
```
|
||||
|
||||
### Drive not appearing after perc-nonraid.sh
|
||||
|
||||
```bash
|
||||
lsblk
|
||||
# If the drive is missing, check its state in perccli:
|
||||
perccli /c0 /eall /sall show
|
||||
# State should be "JBOD" or "UGood" — not "Offln" or "Msng"
|
||||
```
|
||||
@@ -0,0 +1,94 @@
|
||||
# Frigate NVR configuration — VM 100 (Quadro P2200 + Google Coral USB)
|
||||
# Docs: https://docs.frigate.video/configuration/
|
||||
|
||||
mqtt:
|
||||
enabled: false # set to true and fill in host/port if you use Home Assistant or MQTT
|
||||
|
||||
# ── Object detection ─────────────────────────────────────────────────────────
|
||||
# Coral USB handles all inference; CPU usage for detection is near zero.
|
||||
detectors:
|
||||
coral:
|
||||
type: edgetpu
|
||||
device: usb
|
||||
|
||||
# ── FFmpeg hardware decode (Quadro P2200 NVDEC) ───────────────────────────────
|
||||
# These presets tell ffmpeg to use the GPU for H.264/H.265 decode.
|
||||
# CPU usage per stream drops from ~15–30% to ~1–2%.
|
||||
ffmpeg:
|
||||
hwaccel_args: preset-nvidia-h264 # default for H.264 cameras
|
||||
# Per-camera override available if some cameras use H.265 — see cameras section below
|
||||
|
||||
# ── Global detect settings ───────────────────────────────────────────────────
|
||||
detect:
|
||||
enabled: true
|
||||
width: 1280
|
||||
height: 720
|
||||
fps: 5 # Coral processes 5 fps for detection; recording captures full stream
|
||||
|
||||
# ── Recording ────────────────────────────────────────────────────────────────
|
||||
record:
|
||||
enabled: true
|
||||
retain:
|
||||
days: 7
|
||||
mode: motion # only keep segments with motion (saves disk)
|
||||
events:
|
||||
retain:
|
||||
default: 14 # keep event clips for 14 days regardless of motion-only rule
|
||||
mode: active_objects
|
||||
|
||||
# ── Snapshots ────────────────────────────────────────────────────────────────
|
||||
snapshots:
|
||||
enabled: true
|
||||
timestamp: true
|
||||
bounding_box: true
|
||||
retain:
|
||||
default: 14
|
||||
|
||||
# ── Object filter defaults ────────────────────────────────────────────────────
|
||||
objects:
|
||||
track:
|
||||
- person
|
||||
- car
|
||||
- dog
|
||||
- cat
|
||||
filters:
|
||||
person:
|
||||
min_area: 1500 # ignore very small detections (reduces false positives)
|
||||
min_score: 0.6
|
||||
threshold: 0.7
|
||||
|
||||
# ── Cameras ──────────────────────────────────────────────────────────────────
|
||||
# Add one entry per camera. Duplicate and adjust as needed.
|
||||
# Use {FRIGATE_RTSP_PASSWORD} to reference the env var from docker-compose.yml.
|
||||
|
||||
cameras:
|
||||
front_door:
|
||||
ffmpeg:
|
||||
inputs:
|
||||
- path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1
|
||||
roles:
|
||||
- detect
|
||||
- record
|
||||
# Uncomment to override hwaccel for H.265 cameras:
|
||||
# ffmpeg:
|
||||
# hwaccel_args: preset-nvidia-h265
|
||||
|
||||
# back_yard:
|
||||
# ffmpeg:
|
||||
# inputs:
|
||||
# - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1
|
||||
# roles:
|
||||
# - detect
|
||||
# - record
|
||||
|
||||
# ── Birdseye view (optional multi-camera overview) ────────────────────────────
|
||||
birdseye:
|
||||
enabled: true
|
||||
mode: motion # only show cameras with recent motion
|
||||
|
||||
# ── Telemetry / stats ─────────────────────────────────────────────────────────
|
||||
telemetry:
|
||||
stats:
|
||||
amd_gpu_stats: false
|
||||
intel_gpu_stats: false
|
||||
network_bandwidth: false
|
||||
@@ -0,0 +1,39 @@
|
||||
services:
|
||||
frigate:
|
||||
container_name: frigate
|
||||
image: ghcr.io/blakeblackshear/frigate:stable
|
||||
restart: unless-stopped
|
||||
privileged: true # needed for Coral USB device access
|
||||
runtime: nvidia # NVIDIA Container Toolkit — enables NVDEC in ffmpeg
|
||||
|
||||
shm_size: "256mb" # shared memory for decoded frame buffers
|
||||
# increase to 512mb if running 8+ cameras
|
||||
|
||||
environment:
|
||||
NVIDIA_VISIBLE_DEVICES: all
|
||||
NVIDIA_DRIVER_CAPABILITIES: compute,utility,video
|
||||
# Set RTSP credentials here and reference as {FRIGATE_RTSP_PASSWORD} in config.yml
|
||||
FRIGATE_RTSP_PASSWORD: "changeme"
|
||||
|
||||
devices:
|
||||
# Coral USB — exposes full USB bus so both pre/post-init VID:PIDs work
|
||||
- /dev/bus/usb:/dev/bus/usb
|
||||
|
||||
volumes:
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
- ./config.yml:/config/config.yml:ro
|
||||
- /mnt/frigate:/media/frigate # recordings and snapshots — point at your drive mount
|
||||
|
||||
ports:
|
||||
- "5000:5000" # Frigate web UI
|
||||
- "8554:8554" # RTSP restream
|
||||
- "8555:8555/tcp" # WebRTC
|
||||
- "8555:8555/udp" # WebRTC
|
||||
|
||||
# healthcheck so docker knows when Frigate is actually ready
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:5000/api/version"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
Executable
+54
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
# Map physical drive bays to stable /dev/disk/by-id paths.
|
||||
# Run on the Proxmox host after perc-nonraid.sh and a reboot.
|
||||
# Output is a table you can paste into docs/hardware-layout.md.
|
||||
#
|
||||
# Requires: lsscsi, ledmon (apt install lsscsi ledmon)
|
||||
# Optional: perccli for enclosure/slot info
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
for cmd in lsscsi; do
|
||||
command -v "$cmd" &>/dev/null || { echo "Missing: $cmd — run: apt install $cmd"; exit 1; }
|
||||
done
|
||||
|
||||
echo "=== Drive inventory ==="
|
||||
echo ""
|
||||
printf "%-12s %-10s %-30s %-20s %s\n" "DEVICE" "SIZE" "MODEL" "SERIAL" "BY-ID PATH"
|
||||
printf "%-12s %-10s %-30s %-20s %s\n" "------" "----" "-----" "------" "---------"
|
||||
|
||||
for dev in /dev/sd?; do
|
||||
[[ -b "$dev" ]] || continue
|
||||
|
||||
size=$(lsblk -dn -o SIZE "$dev" 2>/dev/null || echo "?")
|
||||
model=$(cat "/sys/block/$(basename "$dev")/device/model" 2>/dev/null | tr -d ' ' || echo "?")
|
||||
serial=$(cat "/sys/block/$(basename "$dev")/device/serial" 2>/dev/null | tr -d ' ' || echo "?")
|
||||
|
||||
# Prefer WWN-based by-id, fall back to scsi- or ata-
|
||||
byid=$(ls -1 /dev/disk/by-id/ 2>/dev/null \
|
||||
| grep -v "\-part" \
|
||||
| while read -r link; do
|
||||
target=$(readlink -f "/dev/disk/by-id/$link")
|
||||
[[ "$target" == "$dev" ]] && echo "$link" && break
|
||||
done | head -1 || echo "not found")
|
||||
|
||||
printf "%-12s %-10s %-30s %-20s %s\n" "$dev" "$size" "$model" "$serial" "/dev/disk/by-id/$byid"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=== Bay identification via LED blink ==="
|
||||
echo ""
|
||||
echo "To confirm which physical bay a device is in, blink its LED:"
|
||||
echo " apt install ledmon"
|
||||
echo " ledctl locate=/dev/sdX # LED on"
|
||||
echo " ledctl locate_off=/dev/sdX # LED off"
|
||||
echo ""
|
||||
|
||||
if command -v perccli &>/dev/null || command -v perccli64 &>/dev/null; then
|
||||
PERCCLI=$(command -v perccli || command -v perccli64)
|
||||
echo "=== PERC slot info ==="
|
||||
$PERCCLI /c0 /eall /sall show | grep -E "^[0-9]|Drive's position|SN|WWN" || true
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Copy the BY-ID paths into docs/hardware-layout.md."
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Dell R730xd fan speed control (third-party GPU)
|
||||
# iDRAC sets fans to 100% when non-Dell PCIe cards are detected.
|
||||
# This service overrides that and manages fan speed by inlet temperature.
|
||||
After=network.target
|
||||
# Restart if ipmitool fails transiently (e.g. iDRAC busy at boot)
|
||||
StartLimitIntervalSec=60
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/local/sbin/fan-control.sh
|
||||
# On stop, re-enable iDRAC auto control so fans are safe if service is removed
|
||||
ExecStop=/usr/local/sbin/fan-control.sh --auto
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+123
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env bash
|
||||
# Dell R730xd fan speed control for third-party PCIe cards (Quadro P2200, etc.)
|
||||
#
|
||||
# iDRAC detects non-Dell GPUs and slams fans to 100% indefinitely.
|
||||
# This script disables iDRAC automatic fan control and manages speed based
|
||||
# on inlet temperature, keeping the server quiet under normal load.
|
||||
#
|
||||
# Install:
|
||||
# apt install ipmitool
|
||||
# cp fan-control.sh /usr/local/sbin/fan-control.sh
|
||||
# chmod +x /usr/local/sbin/fan-control.sh
|
||||
# cp fan-control.service /etc/systemd/system/
|
||||
# systemctl daemon-reload && systemctl enable --now fan-control.service
|
||||
#
|
||||
# Manual speed test (without running as daemon):
|
||||
# fan-control.sh --set 25 # set fans to 25% and exit
|
||||
# fan-control.sh --auto # restore iDRAC automatic control and exit
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
IPMI="ipmitool raw 0x30 0x30"
|
||||
|
||||
# Fan speed thresholds by inlet temperature (°C → % speed)
|
||||
# Tune these for your environment. Inlet temp sensor reads ambient air entering front.
|
||||
declare -A SPEED_MAP=(
|
||||
[0]=15 # < 30°C → 15% (near-silent)
|
||||
[30]=20 # 30–39°C → 20%
|
||||
[40]=30 # 40–44°C → 30%
|
||||
[45]=40 # 45–49°C → 40%
|
||||
[50]=55 # 50–54°C → 55%
|
||||
[55]=75 # 55–59°C → 75%
|
||||
[60]=100 # ≥ 60°C → 100% (safety)
|
||||
)
|
||||
|
||||
# Minimum speed floor — never go below this (protects drives and CPUs)
|
||||
MIN_SPEED=15
|
||||
|
||||
get_inlet_temp() {
|
||||
ipmitool sdr type Temperature 2>/dev/null \
|
||||
| grep -i "Inlet Temp\|Ambient\|Inlet" \
|
||||
| grep -oP '\d+(?= degrees)' \
|
||||
| head -1 || echo "35" # safe default if sensor read fails
|
||||
}
|
||||
|
||||
pct_to_hex() {
|
||||
printf '0x%02x' "$(( $1 < 100 ? $1 : 100 ))"
|
||||
}
|
||||
|
||||
set_fan_speed() {
|
||||
local pct=$1
|
||||
[[ $pct -lt $MIN_SPEED ]] && pct=$MIN_SPEED
|
||||
local hex
|
||||
hex=$(pct_to_hex "$pct")
|
||||
$IPMI 0x02 0xff "$hex"
|
||||
}
|
||||
|
||||
disable_auto_fan() {
|
||||
$IPMI 0x01 0x00
|
||||
echo "$(date): iDRAC automatic fan control DISABLED"
|
||||
}
|
||||
|
||||
enable_auto_fan() {
|
||||
$IPMI 0x01 0x01
|
||||
echo "$(date): iDRAC automatic fan control RE-ENABLED"
|
||||
}
|
||||
|
||||
speed_for_temp() {
|
||||
local temp=$1
|
||||
local speed=$MIN_SPEED
|
||||
for threshold in $(echo "${!SPEED_MAP[@]}" | tr ' ' '\n' | sort -n); do
|
||||
[[ $temp -ge $threshold ]] && speed=${SPEED_MAP[$threshold]}
|
||||
done
|
||||
echo "$speed"
|
||||
}
|
||||
|
||||
# ── Argument handling ─────────────────────────────────────────────────────────
|
||||
|
||||
case "${1:-}" in
|
||||
--auto)
|
||||
enable_auto_fan
|
||||
exit 0
|
||||
;;
|
||||
--set)
|
||||
pct="${2:?Usage: fan-control.sh --set <0-100>}"
|
||||
disable_auto_fan
|
||||
set_fan_speed "$pct"
|
||||
echo "$(date): Fans set to ${pct}%"
|
||||
exit 0
|
||||
;;
|
||||
--temp)
|
||||
echo "Inlet temp: $(get_inlet_temp)°C"
|
||||
exit 0
|
||||
;;
|
||||
"")
|
||||
# Daemon mode — fall through to loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 [--auto | --set <pct> | --temp]"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# ── Daemon loop ───────────────────────────────────────────────────────────────
|
||||
|
||||
trap 'enable_auto_fan; exit 0' SIGTERM SIGINT
|
||||
|
||||
echo "$(date): Fan control daemon starting"
|
||||
disable_auto_fan
|
||||
|
||||
last_speed=-1
|
||||
|
||||
while true; do
|
||||
temp=$(get_inlet_temp)
|
||||
target=$(speed_for_temp "$temp")
|
||||
|
||||
if [[ $target -ne $last_speed ]]; then
|
||||
set_fan_speed "$target"
|
||||
echo "$(date): Inlet ${temp}°C → fans ${target}%"
|
||||
last_speed=$target
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
Executable
+131
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env bash
|
||||
# Configure IOMMU + VFIO passthrough for two Quadro P2200 GPUs on Proxmox VE 9.x.
|
||||
# Run on the Proxmox host. Requires a reboot to take effect.
|
||||
#
|
||||
# After running this script:
|
||||
# 1. Reboot the host
|
||||
# 2. Verify with: lspci -nnk | grep -A3 -i nvidia
|
||||
# Driver should show 'vfio-pci', not 'nouveau' or 'nvidia'
|
||||
# 3. Assign GPUs to VMs via qm set or the Proxmox UI
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
CMDLINE_FILE="/etc/kernel/cmdline"
|
||||
MODPROBE_VFIO="/etc/modprobe.d/vfio.conf"
|
||||
MODPROBE_BLACKLIST="/etc/modprobe.d/blacklist-gpu.conf"
|
||||
INITRAMFS_MODULES="/etc/initramfs-tools/modules"
|
||||
|
||||
# ── Step 1: Check IOMMU groups ────────────────────────────────────────────────
|
||||
|
||||
echo "=== Current IOMMU groups (GPUs) ==="
|
||||
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
||||
n=${d#*/iommu_groups/*}; n=${n%%/*}
|
||||
dev=$(lspci -nns "${d##*/}" 2>/dev/null || true)
|
||||
[[ "$dev" =~ VGA|3D|Display|Audio ]] && printf 'Group %3s %s\n' "$n" "$dev"
|
||||
done
|
||||
echo ""
|
||||
|
||||
# ── Step 2: Collect GPU PCI IDs ───────────────────────────────────────────────
|
||||
|
||||
echo "=== Detected NVIDIA devices ==="
|
||||
lspci -nn | grep -i nvidia
|
||||
echo ""
|
||||
|
||||
# Grab all NVIDIA PCI IDs (vendor:device) for vfio-pci binding
|
||||
# This captures both the GPU (VGA) and its HDMI audio sibling
|
||||
NVIDIA_IDS=$(lspci -nn | grep -i nvidia | grep -oP '\[\K[0-9a-f]{4}:[0-9a-f]{4}(?=\])' | sort -u | tr '\n' ',' | sed 's/,$//')
|
||||
|
||||
if [[ -z "$NVIDIA_IDS" ]]; then
|
||||
echo "ERROR: No NVIDIA devices found. Is the GPU installed and visible to lspci?"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "GPU PCI IDs to bind to vfio-pci: $NVIDIA_IDS"
|
||||
echo ""
|
||||
read -rp "Proceed with configuring VFIO passthrough? [y/N] " confirm
|
||||
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
|
||||
|
||||
# ── Step 3: Enable IOMMU in kernel cmdline ────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Configuring kernel cmdline for IOMMU ---"
|
||||
|
||||
if [[ ! -f "$CMDLINE_FILE" ]]; then
|
||||
echo "ERROR: $CMDLINE_FILE not found. Is this a Proxmox EFI system?"
|
||||
echo "For legacy GRUB: edit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT instead."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
current_cmdline=$(cat "$CMDLINE_FILE")
|
||||
new_cmdline="$current_cmdline"
|
||||
|
||||
[[ "$new_cmdline" =~ intel_iommu=on ]] || new_cmdline="$new_cmdline intel_iommu=on"
|
||||
[[ "$new_cmdline" =~ iommu=pt ]] || new_cmdline="$new_cmdline iommu=pt"
|
||||
|
||||
# Deduplicate spaces
|
||||
new_cmdline=$(echo "$new_cmdline" | tr -s ' ' | sed 's/^ //;s/ $//')
|
||||
|
||||
echo "$new_cmdline" > "$CMDLINE_FILE"
|
||||
echo "Written: $CMDLINE_FILE"
|
||||
echo " $new_cmdline"
|
||||
|
||||
proxmox-boot-tool refresh
|
||||
echo "Boot tool refreshed."
|
||||
|
||||
# ── Step 4: Blacklist host GPU drivers ────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Blacklisting nouveau and nvidia on host ---"
|
||||
cat > "$MODPROBE_BLACKLIST" <<EOF
|
||||
blacklist nouveau
|
||||
blacklist nvidia
|
||||
blacklist nvidiafb
|
||||
blacklist nvidia_drm
|
||||
blacklist nvidia_modeset
|
||||
options nouveau modeset=0
|
||||
EOF
|
||||
echo "Written: $MODPROBE_BLACKLIST"
|
||||
|
||||
# ── Step 5: Bind GPUs to vfio-pci ────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Binding GPU IDs to vfio-pci ---"
|
||||
cat > "$MODPROBE_VFIO" <<EOF
|
||||
options vfio-pci ids=$NVIDIA_IDS
|
||||
softdep nouveau pre: vfio-pci
|
||||
softdep nvidia pre: vfio-pci
|
||||
EOF
|
||||
echo "Written: $MODPROBE_VFIO"
|
||||
echo " IDs: $NVIDIA_IDS"
|
||||
|
||||
# ── Step 6: Load vfio modules early in initramfs ─────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Adding vfio modules to initramfs ---"
|
||||
for mod in vfio vfio_iommu_type1 vfio_pci vfio_pci_core; do
|
||||
grep -qxF "$mod" "$INITRAMFS_MODULES" 2>/dev/null || echo "$mod" >> "$INITRAMFS_MODULES"
|
||||
done
|
||||
echo "Updated: $INITRAMFS_MODULES"
|
||||
|
||||
update-initramfs -u -k all
|
||||
echo "Initramfs updated."
|
||||
|
||||
# ── Step 7: Print PCI addresses for VM config ─────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "=== GPU PCI addresses for VM assignment ==="
|
||||
echo "Use these in qm set or the Proxmox UI (Hardware → Add → PCI Device):"
|
||||
echo ""
|
||||
lspci -nn | grep -i nvidia | while read -r line; do
|
||||
addr=$(echo "$line" | awk '{print $1}')
|
||||
desc=$(echo "$line" | cut -d' ' -f2-)
|
||||
printf " hostpciN: 0000:%s,pcie=1 # %s\n" "$addr" "$desc"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM."
|
||||
echo "Example for GPU at 01:00.0 (audio at 01:00.1):"
|
||||
echo " hostpci0: 0000:01:00,pcie=1,x-vga=1"
|
||||
echo " (Proxmox will auto-include 01:00.1 when you use the .0 address)"
|
||||
echo ""
|
||||
echo "Done. Reboot the host to activate IOMMU and vfio-pci binding."
|
||||
Executable
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env bash
|
||||
# Convert PERC H730 drives to Non-RAID (per-disk passthrough) mode.
|
||||
# Run once on the Proxmox host before assigning drives to VMs.
|
||||
# Requires: perccli (install from Dell's website or local .deb)
|
||||
#
|
||||
# Dell PERC H730 does not have a true HBA/IT mode. Non-RAID mode is the
|
||||
# equivalent — each disk is presented directly to the OS with SMART intact.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PERCCLI=$(command -v perccli || command -v perccli64 || true)
|
||||
if [[ -z "$PERCCLI" ]]; then
|
||||
echo "perccli not found. Install from:"
|
||||
echo " https://www.dell.com/support (search 'PERCCLI')"
|
||||
echo " or: dpkg -i perccli_*.deb"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Controller overview ==="
|
||||
$PERCCLI show
|
||||
|
||||
echo ""
|
||||
echo "=== Drives on controller 0 ==="
|
||||
$PERCCLI /c0 /eall /sall show
|
||||
|
||||
echo ""
|
||||
read -rp "Proceed with converting all non-OS drives to Non-RAID mode? [y/N] " confirm
|
||||
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
|
||||
|
||||
# Identify enclosure IDs (typically 8 for the internal backplane on R730xd)
|
||||
ENCLOSURES=$($PERCCLI /c0 /eall show | awk '/^[0-9]/{print $1}' | sort -u)
|
||||
|
||||
for enc in $ENCLOSURES; do
|
||||
echo ""
|
||||
echo "=== Processing enclosure $enc ==="
|
||||
SLOTS=$($PERCCLI /c0 /e"$enc" /sall show | awk '/UGood|Onln|JBOD|DHS/{print $2}' | sort -u)
|
||||
|
||||
for slot in $SLOTS; do
|
||||
echo -n " Slot $slot: setting to Good... "
|
||||
$PERCCLI /c0 /e"$enc" /s"$slot" set good force 2>&1 | grep -i "success\|error\|already" || true
|
||||
|
||||
echo -n " Slot $slot: setting to Non-RAID... "
|
||||
$PERCCLI /c0 /e"$enc" /s"$slot" set nonraid 2>&1 | grep -i "success\|error\|already" || true
|
||||
done
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=== Final drive state ==="
|
||||
$PERCCLI /c0 /eall /sall show
|
||||
|
||||
echo ""
|
||||
echo "Done. Reboot the host for changes to take full effect."
|
||||
echo "After reboot, run build-bay-map.sh to map bays to /dev/disk/by-id paths."
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Stagger hard drive spin-up to limit PSU current surge
|
||||
# Runs early in boot, before storage services and VMs start, so drives
|
||||
# are already spun up and staggered before any heavy I/O begins.
|
||||
DefaultDependencies=no
|
||||
After=local-fs-pre.target
|
||||
Before=local-fs.target sysinit.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/stagger-spinup.sh --linux
|
||||
RemainAfterExit=yes
|
||||
StandardOutput=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=sysinit.target
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stagger hard drive spin-up on Dell R730xd to avoid PSU current surge at boot.
|
||||
#
|
||||
# 12 x 3.5" HDDs spinning up simultaneously can spike ~240W for 2-3 seconds.
|
||||
# Staggering them 2-3 seconds apart keeps the surge well within PSU limits.
|
||||
#
|
||||
# TWO layers of protection:
|
||||
# 1. iDRAC BIOS setting — fires at POST, before the OS loads (preferred)
|
||||
# 2. Linux systemd service — staggers drives that weren't spun up at POST
|
||||
#
|
||||
# SETUP
|
||||
# ─────
|
||||
# Layer 1 (iDRAC, run once):
|
||||
# bash stagger-spinup.sh --idrac
|
||||
# # Requires racadm and iDRAC network access, then reboot to apply.
|
||||
#
|
||||
# Layer 2 (Linux service, run once):
|
||||
# cp stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
|
||||
# chmod +x /usr/local/sbin/stagger-spinup.sh
|
||||
# Install the systemd unit below, then:
|
||||
# systemctl daemon-reload && systemctl enable stagger-spinup.service
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
STAGGER_SECONDS=3 # delay between each drive spin-up
|
||||
HDPARM=$(command -v hdparm || true)
|
||||
|
||||
# ── iDRAC BIOS method (Layer 1) ───────────────────────────────────────────────
|
||||
|
||||
configure_idrac() {
|
||||
if ! command -v racadm &>/dev/null; then
|
||||
echo "racadm not found."
|
||||
echo "Option A: Run from iDRAC SSH:"
|
||||
echo " ssh root@<idrac-ip>"
|
||||
echo " racadm set BIOS.StorageSettings.HddSeq Enabled"
|
||||
echo " racadm jobqueue create BIOS.Setup.1-1"
|
||||
echo " # Then reboot to apply."
|
||||
echo ""
|
||||
echo "Option B: iDRAC web UI:"
|
||||
echo " System BIOS → Power Management → Hard Disk Drive Sequencing → Enabled"
|
||||
echo " Apply and reboot."
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "Enabling iDRAC hard disk drive sequencing (staggered spin-up)..."
|
||||
racadm set BIOS.StorageSettings.HddSeq Enabled
|
||||
racadm jobqueue create BIOS.Setup.1-1
|
||||
echo "Job queued. Reboot for the BIOS setting to take effect."
|
||||
echo "Verify after reboot with: racadm get BIOS.StorageSettings.HddSeq"
|
||||
}
|
||||
|
||||
# ── Linux spin-up stagger (Layer 2) ──────────────────────────────────────────
|
||||
# Called by the systemd service early in boot.
|
||||
# Reads each block device in sequence with a short delay, causing drives in
|
||||
# standby to spin up one at a time rather than simultaneously.
|
||||
|
||||
stagger_linux() {
|
||||
if [[ -z "$HDPARM" ]]; then
|
||||
echo "hdparm not found — apt install hdparm"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mapfile -t drives < <(lsblk -dn -o NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}' | sort)
|
||||
|
||||
if [[ ${#drives[@]} -eq 0 ]]; then
|
||||
echo "No block devices found."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "$(date): Staggering spin-up for ${#drives[@]} drives (${STAGGER_SECONDS}s apart)..."
|
||||
|
||||
for dev in "${drives[@]}"; do
|
||||
# Skip if device is already active (check power mode)
|
||||
state=$($HDPARM -C "$dev" 2>/dev/null | grep -oP '(?<=drive state is: )\S+' || echo "unknown")
|
||||
|
||||
if [[ "$state" == "standby" || "$state" == "sleeping" ]]; then
|
||||
echo "$(date): Waking $dev (was: $state)"
|
||||
# A zero-length read is enough to trigger spin-up
|
||||
dd if="$dev" of=/dev/null bs=512 count=1 status=none 2>/dev/null || true
|
||||
sleep "$STAGGER_SECONDS"
|
||||
else
|
||||
echo "$(date): $dev already active (state: $state) — skipping"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "$(date): Stagger complete."
|
||||
}
|
||||
|
||||
# ── Argument dispatch ─────────────────────────────────────────────────────────
|
||||
|
||||
case "${1:-}" in
|
||||
--idrac)
|
||||
configure_idrac
|
||||
;;
|
||||
--linux|"")
|
||||
stagger_linux
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 [--idrac | --linux]"
|
||||
echo " --idrac Configure iDRAC BIOS staggered spin-up (one-time setup)"
|
||||
echo " --linux Stagger drives via hdparm now (run by systemd service)"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,55 @@
|
||||
# Proxmox VM 100 — Frigate NVR
|
||||
#
|
||||
# Copy to /etc/pve/qemu-server/100.conf after filling in:
|
||||
# - hostpci0: real PCI address from gpu-passthrough-setup.sh output
|
||||
# - scsi1–scsi8: real /dev/disk/by-id paths from build-bay-map.sh output
|
||||
# - net0: real MAC (Proxmox generates one; leave it if creating via UI)
|
||||
#
|
||||
# Requirements on the Proxmox host before starting this VM:
|
||||
# 1. scripts/perc-nonraid.sh — drives in non-RAID mode
|
||||
# 2. scripts/gpu-passthrough-setup.sh — IOMMU + vfio-pci active, host rebooted
|
||||
# 3. Coral USB plugged in
|
||||
|
||||
agent: 1
|
||||
bios: ovmf
|
||||
boot: order=scsi0
|
||||
cores: 8
|
||||
cpu: host
|
||||
machine: q35
|
||||
memory: 16384
|
||||
name: frigate
|
||||
numa: 0
|
||||
ostype: l26
|
||||
scsihw: virtio-scsi-single
|
||||
sockets: 1
|
||||
|
||||
# OS disk — on Proxmox local storage, adjust pool name as needed
|
||||
scsi0: local-lvm:vm-100-disk-0,cache=writeback,size=64G
|
||||
|
||||
# Data drives — bays 1–8, raw disk passthrough
|
||||
# Fill in by-id paths from build-bay-map.sh
|
||||
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
|
||||
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY2,size=0
|
||||
scsi3: /dev/disk/by-id/PLACEHOLDER_BAY3,size=0
|
||||
scsi4: /dev/disk/by-id/PLACEHOLDER_BAY4,size=0
|
||||
scsi5: /dev/disk/by-id/PLACEHOLDER_BAY5,size=0
|
||||
scsi6: /dev/disk/by-id/PLACEHOLDER_BAY6,size=0
|
||||
scsi7: /dev/disk/by-id/PLACEHOLDER_BAY7,size=0
|
||||
scsi8: /dev/disk/by-id/PLACEHOLDER_BAY8,size=0
|
||||
|
||||
# Quadro P2200 #1 — GPU passthrough
|
||||
# Replace XX:00 with real PCI address (e.g. 03:00)
|
||||
# pcie=1 uses PCIe bus; x-vga=1 passes primary display output
|
||||
# Proxmox auto-includes the HDMI audio sibling (XX:00.1)
|
||||
hostpci0: 0000:XX:00,pcie=1,x-vga=1
|
||||
|
||||
# Google Coral USB — pass both VID:PID values
|
||||
# Coral re-enumerates after first inference load: 1a6e:089a → 18d1:9302
|
||||
usb0: host=1a6e:089a
|
||||
usb1: host=18d1:9302
|
||||
|
||||
# Network
|
||||
net0: virtio=BC:24:11:00:00:64,bridge=vmbr0,firewall=1
|
||||
|
||||
# EFI disk (required for OVMF/UEFI boot with q35 machine)
|
||||
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||
@@ -0,0 +1,35 @@
|
||||
# Proxmox VM 101 — General purpose with GPU
|
||||
#
|
||||
# Copy to /etc/pve/qemu-server/101.conf after filling in:
|
||||
# - hostpci0: real PCI address of Quadro P2200 #2
|
||||
# - scsi1–scsi2: real /dev/disk/by-id paths for bays 9–10
|
||||
|
||||
agent: 1
|
||||
bios: ovmf
|
||||
boot: order=scsi0
|
||||
cores: 8
|
||||
cpu: host
|
||||
machine: q35
|
||||
memory: 16384
|
||||
name: vm101
|
||||
numa: 0
|
||||
ostype: l26
|
||||
scsihw: virtio-scsi-single
|
||||
sockets: 1
|
||||
|
||||
# OS disk
|
||||
scsi0: local-lvm:vm-101-disk-0,cache=writeback,size=64G
|
||||
|
||||
# Data drives — bays 9–10
|
||||
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY9,size=0
|
||||
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY10,size=0
|
||||
|
||||
# Quadro P2200 #2
|
||||
# Replace XX:00 with real PCI address from gpu-passthrough-setup.sh output
|
||||
hostpci0: 0000:XX:00,pcie=1,x-vga=1
|
||||
|
||||
# Network
|
||||
net0: virtio=BC:24:11:00:00:65,bridge=vmbr0,firewall=1
|
||||
|
||||
# EFI disk
|
||||
efidisk0: local-lvm:vm-101-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||
@@ -0,0 +1,30 @@
|
||||
# Proxmox VM 102 — Storage / utility VM
|
||||
#
|
||||
# Copy to /etc/pve/qemu-server/102.conf after filling in:
|
||||
# - scsi1–scsi2: real /dev/disk/by-id paths for bays 11–12
|
||||
|
||||
agent: 1
|
||||
bios: ovmf
|
||||
boot: order=scsi0
|
||||
cores: 4
|
||||
cpu: host
|
||||
machine: q35
|
||||
memory: 8192
|
||||
name: vm102
|
||||
numa: 0
|
||||
ostype: l26
|
||||
scsihw: virtio-scsi-single
|
||||
sockets: 1
|
||||
|
||||
# OS disk
|
||||
scsi0: local-lvm:vm-102-disk-0,cache=writeback,size=32G
|
||||
|
||||
# Data drives — bays 11–12
|
||||
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY11,size=0
|
||||
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY12,size=0
|
||||
|
||||
# Network
|
||||
net0: virtio=BC:24:11:00:00:66,bridge=vmbr0,firewall=1
|
||||
|
||||
# EFI disk
|
||||
efidisk0: local-lvm:vm-102-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||
Reference in New Issue
Block a user