Merge pull request #1 from outis1one/claude/proxmox-hardware-passthrough-ylAYx
Claude/proxmox hardware passthrough yl a yx
This commit is contained in:
Outis
@@ -1 +1,31 @@
|
|||||||
# local_proxmox
|
# local_proxmox
|
||||||
|
|
||||||
|
Proxmox VE 9.1 configuration, scripts, and VM layouts for a Dell R730xd homelab.
|
||||||
|
|
||||||
|
## Hardware
|
||||||
|
|
||||||
|
- **Host:** Dell PowerEdge R730xd
|
||||||
|
- **RAID controller:** PERC H730 (configured in Non-RAID/per-disk passthrough mode)
|
||||||
|
- **GPUs:** 2x Quadro P3300 4GB (VFIO passthrough, one per VM)
|
||||||
|
- **TPU:** Google Coral USB (Frigate object detection)
|
||||||
|
- **Bays:** 12x (not all populated)
|
||||||
|
|
||||||
|
## VM Layout
|
||||||
|
|
||||||
|
| VM ID | Role | Drives | GPU | Coral |
|
||||||
|
|-------|------|--------|-----|-------|
|
||||||
|
| 100 | Frigate NVR | Bays 1–8 | Quadro #1 (VFIO) | Yes |
|
||||||
|
| 101 | TBD | Bays 9–10 | Quadro #2 (VFIO) | No |
|
||||||
|
| 102 | TBD | Bays 11–12 | None | No |
|
||||||
|
|
||||||
|
## Setup Order
|
||||||
|
|
||||||
|
1. [`scripts/perc-nonraid.sh`](scripts/perc-nonraid.sh) — convert H730 to per-disk non-RAID mode
|
||||||
|
2. [`scripts/build-bay-map.sh`](scripts/build-bay-map.sh) — map physical bays to stable `/dev/disk/by-id` paths
|
||||||
|
3. [`scripts/gpu-passthrough-setup.sh`](scripts/gpu-passthrough-setup.sh) — configure IOMMU + VFIO for both Quadros
|
||||||
|
4. Apply VM configs from [`vm-configs/`](vm-configs/)
|
||||||
|
5. Deploy Frigate from [`frigate/`](frigate/)
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [`docs/hardware-layout.md`](docs/hardware-layout.md) — bay/device/VM mapping worksheet
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
# Hardware Layout
|
||||||
|
|
||||||
|
Dell R730xd — fill this in after running `scripts/build-bay-map.sh`.
|
||||||
|
|
||||||
|
## Bay → Device → VM Map
|
||||||
|
|
||||||
|
Run `scripts/build-bay-map.sh` on the Proxmox host to auto-populate the by-id column.
|
||||||
|
Confirm bay numbers by running `ledctl locate=<device>` to blink the drive LED.
|
||||||
|
|
||||||
|
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) | Notes |
|
||||||
|
|-----|-------------|------|---------------------------|-------|
|
||||||
|
| 1 | 100 | | | |
|
||||||
|
| 2 | 100 | | | |
|
||||||
|
| 3 | 100 | | | |
|
||||||
|
| 4 | 100 | | | |
|
||||||
|
| 5 | 100 | | | |
|
||||||
|
| 6 | 100 | | | |
|
||||||
|
| 7 | 100 | | | |
|
||||||
|
| 8 | 100 | | | |
|
||||||
|
| 9 | 101 | | | |
|
||||||
|
| 10 | 101 | | | |
|
||||||
|
| 11 | 102 | | | |
|
||||||
|
| 12 | 102 | | | |
|
||||||
|
|
||||||
|
## GPU Map
|
||||||
|
|
||||||
|
Run `lspci -nn | grep -i nvidia` on the Proxmox host and fill in the PCI addresses.
|
||||||
|
|
||||||
|
| Slot | PCI Address (fill in) | Assigned VM | Notes |
|
||||||
|
|------|-----------------------|-------------|-------|
|
||||||
|
| GPU 1 | | 100 | Frigate decode + display |
|
||||||
|
| GPU 2 | | 101 | |
|
||||||
|
|
||||||
|
The GPU's HDMI audio function (same address, function 1) must be passed through alongside the GPU.
|
||||||
|
Example: GPU at `01:00.0` → also pass `01:00.1`.
|
||||||
|
|
||||||
|
## USB / TPU Map
|
||||||
|
|
||||||
|
| Device | VID:PID | Assigned VM | Notes |
|
||||||
|
|--------|---------|-------------|-------|
|
||||||
|
| Coral (pre-init) | 1a6e:089a | 100 | Global Unichip — before first inference |
|
||||||
|
| Coral (post-init) | 18d1:9302 | 100 | Google — after first inference; pass both |
|
||||||
|
|
||||||
|
## IOMMU Groups
|
||||||
|
|
||||||
|
Run on Proxmox host to check groupings before passthrough:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
||||||
|
n=${d#*/iommu_groups/*}; n=${n%%/*}
|
||||||
|
printf 'IOMMU Group %s ' "$n"
|
||||||
|
lspci -nns "${d##*/}"
|
||||||
|
done | sort -V
|
||||||
|
```
|
||||||
|
|
||||||
|
Each GPU should appear in its own group (with only its HDMI audio sibling).
|
||||||
|
If a GPU shares a group with other devices, those must be passed through together.
|
||||||
|
|
||||||
|
## Network
|
||||||
|
|
||||||
|
| VM | Interface | Bridge | Notes |
|
||||||
|
|-----|-----------|--------|-------|
|
||||||
|
| 100 | net0 | vmbr0 | |
|
||||||
|
| 101 | net0 | vmbr0 | |
|
||||||
|
| 102 | net0 | vmbr0 | |
|
||||||
@@ -0,0 +1,457 @@
|
|||||||
|
# Dell R730xd — Proxmox VE 9.1 Setup Guide
|
||||||
|
|
||||||
|
Complete walkthrough from bare metal to running VMs with GPU passthrough,
|
||||||
|
Coral USB, and per-VM disk assignments.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What You Need Before Starting
|
||||||
|
|
||||||
|
- USB drive (8GB+) for the Proxmox installer
|
||||||
|
- A second machine or phone to read this guide and SSH
|
||||||
|
- The iDRAC IP or physical access to a monitor + keyboard
|
||||||
|
- Internet connection on the server
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — Firmware Updates (do this first)
|
||||||
|
|
||||||
|
Outdated firmware causes mysterious IOMMU failures and fan issues. Do this
|
||||||
|
before anything else.
|
||||||
|
|
||||||
|
### Option A: Lifecycle Controller (no extra tools needed)
|
||||||
|
|
||||||
|
1. Power on the server, press **F10** when you see the Dell splash screen
|
||||||
|
2. Lifecycle Controller → **Firmware Update** → Check for updates
|
||||||
|
3. Point it at downloads.dell.com or a local repo
|
||||||
|
4. Update **iDRAC**, **BIOS**, **PERC H730**, and any NIC/HBA firmware
|
||||||
|
5. Let it reboot as many times as it needs
|
||||||
|
|
||||||
|
### Option B: Dell System Update (DSU) from bootable USB
|
||||||
|
|
||||||
|
A faster alternative if Lifecycle Controller is slow or unavailable — Dell
|
||||||
|
provides a bootable ISO that auto-detects and applies all updates.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — iDRAC Setup
|
||||||
|
|
||||||
|
iDRAC is the out-of-band management interface. You need it configured for the
|
||||||
|
fan control script to work later.
|
||||||
|
|
||||||
|
1. Power on → press **F2** to enter System Setup → **iDRAC Settings**
|
||||||
|
2. **Network:**
|
||||||
|
- Set a static IP (easier than DHCP for a server)
|
||||||
|
- Note the IP — you'll use it for the fan control ipmitool commands
|
||||||
|
3. **User configuration:**
|
||||||
|
- Change the default `root` password
|
||||||
|
4. **IPMI over LAN:**
|
||||||
|
- iDRAC Settings → Network → IPMI Settings → **Enable IPMI over LAN: On**
|
||||||
|
- This is required for `ipmitool` fan control from the Proxmox host
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — BIOS Settings
|
||||||
|
|
||||||
|
Still in F2 System Setup. These settings are **required** for GPU and USB
|
||||||
|
passthrough to work. Missing VT-d is the #1 reason passthrough fails silently.
|
||||||
|
|
||||||
|
| Menu path | Setting | Value |
|
||||||
|
|-----------|---------|-------|
|
||||||
|
| Processor Settings | Virtualization Technology (VT-x) | **Enabled** |
|
||||||
|
| Processor Settings | VT for Direct I/O (VT-d) | **Enabled** |
|
||||||
|
| PCI Configuration | SR-IOV Global Enable | **Enabled** |
|
||||||
|
| Boot Settings → BIOS Boot Settings | Boot Mode | **UEFI** (not Legacy/BIOS) |
|
||||||
|
| Power Management | Hard Disk Drive Sequencing | **Enabled** |
|
||||||
|
| Power Management | Power Management Policy | **Performance** |
|
||||||
|
|
||||||
|
> **Boot Mode must be UEFI.** Proxmox's EFI boot tool (`proxmox-boot-tool`)
|
||||||
|
> only works with UEFI. Legacy BIOS mode breaks the GPU passthrough script.
|
||||||
|
|
||||||
|
**Apply and exit. The server will reboot.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4 — H730 RAID Configuration (pre-Proxmox)
|
||||||
|
|
||||||
|
The two 2.5" rear drives are your Proxmox OS drives. You want them mirrored
|
||||||
|
so a single drive failure doesn't take down the hypervisor.
|
||||||
|
|
||||||
|
The 3.5" drives will be converted to non-RAID later **from within Proxmox**
|
||||||
|
using the `perc-nonraid.sh` script — do not touch them here.
|
||||||
|
|
||||||
|
### Configure the 2.5" OS drives
|
||||||
|
|
||||||
|
1. Reboot → press **Ctrl+R** during POST to enter the H730 configuration
|
||||||
|
utility (or use Lifecycle Controller → RAID Configuration)
|
||||||
|
2. Select the controller
|
||||||
|
3. Find the two 2.5" rear drives
|
||||||
|
4. **Create new virtual disk:**
|
||||||
|
- RAID level: **RAID 1** (mirror)
|
||||||
|
- Select both 2.5" drives
|
||||||
|
- Strip size: 64KB (default)
|
||||||
|
- Name: `OS-Mirror` (optional)
|
||||||
|
- Initialize: **Fast Initialize**
|
||||||
|
5. Press **Ctrl+Alt+Delete** to reboot
|
||||||
|
|
||||||
|
The H730 now presents a single ~X GB RAID1 virtual disk to the OS. Proxmox
|
||||||
|
will install onto this and never know there are two physical drives behind it.
|
||||||
|
|
||||||
|
> The 3.5" drives will show as "Unconfigured Good" in the H730 — that is fine.
|
||||||
|
> Leave them alone. `perc-nonraid.sh` handles them after Proxmox is installed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5 — Install Proxmox VE 9.1
|
||||||
|
|
||||||
|
### Prepare the USB installer
|
||||||
|
|
||||||
|
On another machine, download the Proxmox VE ISO from proxmox.com and write it
|
||||||
|
to a USB drive:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux/macOS
|
||||||
|
dd if=proxmox-ve_*.iso of=/dev/sdX bs=1M status=progress conv=fsync
|
||||||
|
# or use Balena Etcher (Windows/Mac/Linux GUI)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Boot and install
|
||||||
|
|
||||||
|
1. Plug the USB into the R730xd
|
||||||
|
2. Power on → press **F11** for the one-time boot menu
|
||||||
|
3. Select the USB drive
|
||||||
|
4. At the Proxmox boot menu: **Install Proxmox VE (Graphical)**
|
||||||
|
5. **Target disk:** select the RAID1 virtual disk (`OS-Mirror`)
|
||||||
|
- Filesystem: **ext4** is simplest — the H730 RAID1 already gives you
|
||||||
|
redundancy, so ZFS mirroring here would be double-redundant overkill
|
||||||
|
6. **Location and timezone:** set to your region
|
||||||
|
7. **Password and email:** set a strong root password, enter an email
|
||||||
|
8. **Network configuration:**
|
||||||
|
- Management interface: the built-in NIC (usually `em1` or `eno1`)
|
||||||
|
- Hostname: e.g. `pve.local`
|
||||||
|
- IP: choose a static IP on your LAN (e.g. `192.168.1.10/24`)
|
||||||
|
- Gateway and DNS: your router's IP
|
||||||
|
9. Click **Install**
|
||||||
|
10. Remove USB when prompted, let it reboot
|
||||||
|
|
||||||
|
### First login
|
||||||
|
|
||||||
|
Open a browser on your LAN machine and go to:
|
||||||
|
|
||||||
|
```
|
||||||
|
https://192.168.1.10:8006
|
||||||
|
```
|
||||||
|
|
||||||
|
Accept the self-signed certificate warning. Login: `root` / (your password),
|
||||||
|
Realm: **Linux PAM**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 6 — Proxmox Post-Install (SSH)
|
||||||
|
|
||||||
|
SSH into the host from now on — it's faster than the web console for these
|
||||||
|
steps.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh root@192.168.1.10
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6a. Fix the apt repositories
|
||||||
|
|
||||||
|
Proxmox shows "no valid subscription" warnings when using the enterprise repo
|
||||||
|
without a license. Switch to the free repo:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Disable enterprise repo
|
||||||
|
echo "# disabled" > /etc/apt/sources.list.d/pve-enterprise.list
|
||||||
|
echo "# disabled" > /etc/apt/sources.list.d/ceph.list
|
||||||
|
|
||||||
|
# Add no-subscription repo
|
||||||
|
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
|
||||||
|
> /etc/apt/sources.list.d/pve-no-subscription.list
|
||||||
|
|
||||||
|
apt update && apt dist-upgrade -y
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6b. Install tools used by the scripts
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apt install -y ipmitool lsscsi ledmon hdparm git
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6c. Clone this repo onto the host
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
|
||||||
|
cd /opt/local_proxmox
|
||||||
|
chmod +x scripts/*.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 7 — H730: Set 3.5" Drives to Non-RAID Mode
|
||||||
|
|
||||||
|
Now that Proxmox is running, convert the 3.5" drives to per-disk (non-RAID)
|
||||||
|
mode. The OS drives (RAID1 virtual disk) are **not affected** — the script only
|
||||||
|
targets unconfigured physical disks.
|
||||||
|
|
||||||
|
### Install perccli
|
||||||
|
|
||||||
|
Download from Dell support (search "PERCCLI") or copy the `.deb` to the host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dpkg -i perccli_*.deb
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run the script
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash /opt/local_proxmox/scripts/perc-nonraid.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Review the output, confirm when prompted. **Reboot after completion.**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
After rebooting, Proxmox will see the 3.5" drives as individual block devices
|
||||||
|
(`/dev/sdb`, `/dev/sdc`, etc.).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 8 — Map Physical Bays to Drives
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash /opt/local_proxmox/scripts/build-bay-map.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Walk the bays with `ledctl` to confirm which physical slot is which device:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ledctl locate=/dev/sdb # LED blinks on the matching bay
|
||||||
|
ledctl locate_off=/dev/sdb
|
||||||
|
```
|
||||||
|
|
||||||
|
Fill in the `by-id` paths in `docs/hardware-layout.md`. You will need these
|
||||||
|
in Phase 11 when creating VMs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 9 — GPU Passthrough Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
The script:
|
||||||
|
- Adds `intel_iommu=on iommu=pt` to the kernel command line
|
||||||
|
- Blacklists `nouveau`/`nvidia` on the host
|
||||||
|
- Binds both Quadro P2200s to `vfio-pci`
|
||||||
|
- Rebuilds initramfs
|
||||||
|
|
||||||
|
**Note the PCI addresses it prints at the end** — you'll need them in Phase 11.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify after reboot
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lspci -nnk | grep -A3 -i nvidia
|
||||||
|
```
|
||||||
|
|
||||||
|
Both GPUs should show `Kernel driver in use: vfio-pci`. If they still show
|
||||||
|
`nouveau`, check that `/etc/modprobe.d/blacklist-gpu.conf` exists and
|
||||||
|
`update-initramfs -u` was run.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 10 — Fan Control and Staggered Spin-Up
|
||||||
|
|
||||||
|
### Fan control (prevents jet-engine noise from non-Dell GPUs)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp /opt/local_proxmox/scripts/fan-control.sh /usr/local/sbin/fan-control.sh
|
||||||
|
cp /opt/local_proxmox/scripts/fan-control.service /etc/systemd/system/
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable --now fan-control.service
|
||||||
|
systemctl status fan-control.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Test it worked — the fans should audibly slow down within 30 seconds.
|
||||||
|
To check current speed: `ipmitool sdr type Fan`
|
||||||
|
|
||||||
|
### Staggered spin-up
|
||||||
|
|
||||||
|
**Layer 1 — iDRAC BIOS** (fires at every boot, before OS):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash /opt/local_proxmox/scripts/stagger-spinup.sh --idrac
|
||||||
|
# If racadm is not available, the script prints the iDRAC web UI path instead.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Layer 2 — Linux service** (handles drives coming up from standby):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp /opt/local_proxmox/scripts/stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
|
||||||
|
cp /opt/local_proxmox/scripts/stagger-spinup.service /etc/systemd/system/
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable stagger-spinup.service
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 11 — Create the VMs
|
||||||
|
|
||||||
|
### Prepare the VM configs
|
||||||
|
|
||||||
|
Edit the three example configs and replace the placeholders:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/local_proxmox/vm-configs
|
||||||
|
|
||||||
|
# Replace PLACEHOLDER_BAYx with real by-id paths from Phase 8
|
||||||
|
# Replace XX:00 with real PCI addresses from Phase 9
|
||||||
|
nano 100-frigate.conf.example
|
||||||
|
nano 101.conf.example
|
||||||
|
nano 102.conf.example
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create OS disks for each VM
|
||||||
|
|
||||||
|
In the Proxmox web UI or via CLI, create the base OS disk for each VM:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Creates the disk slots — Proxmox generates the correct scsi0 line
|
||||||
|
qm create 100 --memory 16384 --cores 8 --name frigate --net0 virtio,bridge=vmbr0
|
||||||
|
qm create 101 --memory 16384 --cores 8 --name vm101 --net0 virtio,bridge=vmbr0
|
||||||
|
qm create 102 --memory 8192 --cores 4 --name vm102 --net0 virtio,bridge=vmbr0
|
||||||
|
```
|
||||||
|
|
||||||
|
Then merge your edited conf into the generated config:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backup generated config, then append your hardware lines
|
||||||
|
cp /etc/pve/qemu-server/100.conf /etc/pve/qemu-server/100.conf.bak
|
||||||
|
cat 100-frigate.conf.example >> /etc/pve/qemu-server/100.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Or just open each VM in the Proxmox web UI → **Hardware** and add:
|
||||||
|
- PCI Device → your GPU (enable PCIe, enable Primary GPU for VM 100)
|
||||||
|
- USB Device → host device → `1a6e:089a` and `18d1:9302` (VM 100 only)
|
||||||
|
- Hard Disk → (use disk passthrough, SCSI controller, path = your by-id)
|
||||||
|
|
||||||
|
### Install a guest OS
|
||||||
|
|
||||||
|
Boot each VM from an ISO (upload ISOs to Proxmox under local storage →
|
||||||
|
ISO Images). Ubuntu Server 22.04 LTS is a good choice for the Frigate VM.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 12 — Set Up Frigate in VM 100
|
||||||
|
|
||||||
|
From inside VM 100 (SSH into the guest OS):
|
||||||
|
|
||||||
|
### Install Docker
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apt update && apt install -y ca-certificates curl
|
||||||
|
curl -fsSL https://get.docker.com | sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Install NVIDIA driver + Container Toolkit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Add NVIDIA apt repo
|
||||||
|
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
||||||
|
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||||
|
|
||||||
|
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
||||||
|
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
|
||||||
|
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||||
|
|
||||||
|
apt update
|
||||||
|
apt install -y nvidia-driver-535 nvidia-container-toolkit
|
||||||
|
|
||||||
|
# Configure Docker to use the NVIDIA runtime
|
||||||
|
nvidia-ctk runtime configure --runtime=docker
|
||||||
|
systemctl restart docker
|
||||||
|
|
||||||
|
# Verify the GPU is visible
|
||||||
|
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deploy Frigate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p /opt/frigate && cd /opt/frigate
|
||||||
|
|
||||||
|
# Copy configs from the repo (or clone it inside the VM)
|
||||||
|
cp /path/to/local_proxmox/frigate/docker-compose.yml .
|
||||||
|
cp /path/to/local_proxmox/frigate/config.yml .
|
||||||
|
|
||||||
|
# Create the recordings directory (on your passthrough data drive)
|
||||||
|
# Mount your data drives first — if using ZFS:
|
||||||
|
zpool import <poolname> # or create a new pool from the raw drives
|
||||||
|
mkdir -p /mnt/frigate
|
||||||
|
|
||||||
|
# Edit config.yml and add your camera RTSP URLs
|
||||||
|
nano config.yml
|
||||||
|
|
||||||
|
docker compose up -d
|
||||||
|
docker compose logs -f # watch for errors on first start
|
||||||
|
```
|
||||||
|
|
||||||
|
Frigate UI: `http://<vm100-ip>:5000`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### GPU not passing through — still shows `nouveau`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
update-initramfs -u -k all && reboot
|
||||||
|
# After reboot:
|
||||||
|
lspci -nnk | grep -A3 -i nvidia # must show vfio-pci
|
||||||
|
```
|
||||||
|
|
||||||
|
### IOMMU not enabled
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dmesg | grep -i iommu
|
||||||
|
# Should show: "DMAR: IOMMU enabled"
|
||||||
|
# If not: re-check Phase 3 BIOS settings (VT-d) and Phase 9
|
||||||
|
cat /etc/kernel/cmdline # must contain intel_iommu=on iommu=pt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fan control not working
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check iDRAC IP is reachable and IPMI over LAN is enabled (Phase 2)
|
||||||
|
ipmitool -I lan -H <idrac-ip> -U root -P <password> sdr type Fan
|
||||||
|
# Then check the service:
|
||||||
|
systemctl status fan-control.service
|
||||||
|
journalctl -u fan-control.service -n 50
|
||||||
|
```
|
||||||
|
|
||||||
|
### Coral not detected in Frigate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In VM 100, check both USB IDs are present
|
||||||
|
lsusb | grep -E "1a6e|18d1"
|
||||||
|
# If missing, check the USB passthrough lines in the VM config
|
||||||
|
# Both usb0 (1a6e:089a) and usb1 (18d1:9302) must be present
|
||||||
|
```
|
||||||
|
|
||||||
|
### Drive not appearing after perc-nonraid.sh
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lsblk
|
||||||
|
# If the drive is missing, check its state in perccli:
|
||||||
|
perccli /c0 /eall /sall show
|
||||||
|
# State should be "JBOD" or "UGood" — not "Offln" or "Msng"
|
||||||
|
```
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
# Frigate NVR configuration — VM 100 (Quadro P2200 + Google Coral USB)
|
||||||
|
# Docs: https://docs.frigate.video/configuration/
|
||||||
|
|
||||||
|
mqtt:
|
||||||
|
enabled: false # set to true and fill in host/port if you use Home Assistant or MQTT
|
||||||
|
|
||||||
|
# ── Object detection ─────────────────────────────────────────────────────────
|
||||||
|
# Coral USB handles all inference; CPU usage for detection is near zero.
|
||||||
|
detectors:
|
||||||
|
coral:
|
||||||
|
type: edgetpu
|
||||||
|
device: usb
|
||||||
|
|
||||||
|
# ── FFmpeg hardware decode (Quadro P2200 NVDEC) ───────────────────────────────
|
||||||
|
# These presets tell ffmpeg to use the GPU for H.264/H.265 decode.
|
||||||
|
# CPU usage per stream drops from ~15–30% to ~1–2%.
|
||||||
|
ffmpeg:
|
||||||
|
hwaccel_args: preset-nvidia-h264 # default for H.264 cameras
|
||||||
|
# Per-camera override available if some cameras use H.265 — see cameras section below
|
||||||
|
|
||||||
|
# ── Global detect settings ───────────────────────────────────────────────────
|
||||||
|
detect:
|
||||||
|
enabled: true
|
||||||
|
width: 1280
|
||||||
|
height: 720
|
||||||
|
fps: 5 # Coral processes 5 fps for detection; recording captures full stream
|
||||||
|
|
||||||
|
# ── Recording ────────────────────────────────────────────────────────────────
|
||||||
|
record:
|
||||||
|
enabled: true
|
||||||
|
retain:
|
||||||
|
days: 7
|
||||||
|
mode: motion # only keep segments with motion (saves disk)
|
||||||
|
events:
|
||||||
|
retain:
|
||||||
|
default: 14 # keep event clips for 14 days regardless of motion-only rule
|
||||||
|
mode: active_objects
|
||||||
|
|
||||||
|
# ── Snapshots ────────────────────────────────────────────────────────────────
|
||||||
|
snapshots:
|
||||||
|
enabled: true
|
||||||
|
timestamp: true
|
||||||
|
bounding_box: true
|
||||||
|
retain:
|
||||||
|
default: 14
|
||||||
|
|
||||||
|
# ── Object filter defaults ────────────────────────────────────────────────────
|
||||||
|
objects:
|
||||||
|
track:
|
||||||
|
- person
|
||||||
|
- car
|
||||||
|
- dog
|
||||||
|
- cat
|
||||||
|
filters:
|
||||||
|
person:
|
||||||
|
min_area: 1500 # ignore very small detections (reduces false positives)
|
||||||
|
min_score: 0.6
|
||||||
|
threshold: 0.7
|
||||||
|
|
||||||
|
# ── Cameras ──────────────────────────────────────────────────────────────────
|
||||||
|
# Add one entry per camera. Duplicate and adjust as needed.
|
||||||
|
# Use {FRIGATE_RTSP_PASSWORD} to reference the env var from docker-compose.yml.
|
||||||
|
|
||||||
|
cameras:
|
||||||
|
front_door:
|
||||||
|
ffmpeg:
|
||||||
|
inputs:
|
||||||
|
- path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1
|
||||||
|
roles:
|
||||||
|
- detect
|
||||||
|
- record
|
||||||
|
# Uncomment to override hwaccel for H.265 cameras:
|
||||||
|
# ffmpeg:
|
||||||
|
# hwaccel_args: preset-nvidia-h265
|
||||||
|
|
||||||
|
# back_yard:
|
||||||
|
# ffmpeg:
|
||||||
|
# inputs:
|
||||||
|
# - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.XXX/stream1
|
||||||
|
# roles:
|
||||||
|
# - detect
|
||||||
|
# - record
|
||||||
|
|
||||||
|
# ── Birdseye view (optional multi-camera overview) ────────────────────────────
|
||||||
|
birdseye:
|
||||||
|
enabled: true
|
||||||
|
mode: motion # only show cameras with recent motion
|
||||||
|
|
||||||
|
# ── Telemetry / stats ─────────────────────────────────────────────────────────
|
||||||
|
telemetry:
|
||||||
|
stats:
|
||||||
|
amd_gpu_stats: false
|
||||||
|
intel_gpu_stats: false
|
||||||
|
network_bandwidth: false
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
services:
|
||||||
|
frigate:
|
||||||
|
container_name: frigate
|
||||||
|
image: ghcr.io/blakeblackshear/frigate:stable
|
||||||
|
restart: unless-stopped
|
||||||
|
privileged: true # needed for Coral USB device access
|
||||||
|
runtime: nvidia # NVIDIA Container Toolkit — enables NVDEC in ffmpeg
|
||||||
|
|
||||||
|
shm_size: "256mb" # shared memory for decoded frame buffers
|
||||||
|
# increase to 512mb if running 8+ cameras
|
||||||
|
|
||||||
|
environment:
|
||||||
|
NVIDIA_VISIBLE_DEVICES: all
|
||||||
|
NVIDIA_DRIVER_CAPABILITIES: compute,utility,video
|
||||||
|
# Set RTSP credentials here and reference as {FRIGATE_RTSP_PASSWORD} in config.yml
|
||||||
|
FRIGATE_RTSP_PASSWORD: "changeme"
|
||||||
|
|
||||||
|
devices:
|
||||||
|
# Coral USB — exposes full USB bus so both pre/post-init VID:PIDs work
|
||||||
|
- /dev/bus/usb:/dev/bus/usb
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
- /etc/localtime:/etc/localtime:ro
|
||||||
|
- ./config.yml:/config/config.yml:ro
|
||||||
|
- /mnt/frigate:/media/frigate # recordings and snapshots — point at your drive mount
|
||||||
|
|
||||||
|
ports:
|
||||||
|
- "5000:5000" # Frigate web UI
|
||||||
|
- "8554:8554" # RTSP restream
|
||||||
|
- "8555:8555/tcp" # WebRTC
|
||||||
|
- "8555:8555/udp" # WebRTC
|
||||||
|
|
||||||
|
# healthcheck so docker knows when Frigate is actually ready
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:5000/api/version"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
start_period: 60s
|
||||||
Executable
+54
@@ -0,0 +1,54 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Map physical drive bays to stable /dev/disk/by-id paths.
|
||||||
|
# Run on the Proxmox host after perc-nonraid.sh and a reboot.
|
||||||
|
# Output is a table you can paste into docs/hardware-layout.md.
|
||||||
|
#
|
||||||
|
# Requires: lsscsi, ledmon (apt install lsscsi ledmon)
|
||||||
|
# Optional: perccli for enclosure/slot info
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
for cmd in lsscsi; do
|
||||||
|
command -v "$cmd" &>/dev/null || { echo "Missing: $cmd — run: apt install $cmd"; exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "=== Drive inventory ==="
|
||||||
|
echo ""
|
||||||
|
printf "%-12s %-10s %-30s %-20s %s\n" "DEVICE" "SIZE" "MODEL" "SERIAL" "BY-ID PATH"
|
||||||
|
printf "%-12s %-10s %-30s %-20s %s\n" "------" "----" "-----" "------" "---------"
|
||||||
|
|
||||||
|
for dev in /dev/sd?; do
|
||||||
|
[[ -b "$dev" ]] || continue
|
||||||
|
|
||||||
|
size=$(lsblk -dn -o SIZE "$dev" 2>/dev/null || echo "?")
|
||||||
|
model=$(cat "/sys/block/$(basename "$dev")/device/model" 2>/dev/null | tr -d ' ' || echo "?")
|
||||||
|
serial=$(cat "/sys/block/$(basename "$dev")/device/serial" 2>/dev/null | tr -d ' ' || echo "?")
|
||||||
|
|
||||||
|
# Prefer WWN-based by-id, fall back to scsi- or ata-
|
||||||
|
byid=$(ls -1 /dev/disk/by-id/ 2>/dev/null \
|
||||||
|
| grep -v "\-part" \
|
||||||
|
| while read -r link; do
|
||||||
|
target=$(readlink -f "/dev/disk/by-id/$link")
|
||||||
|
[[ "$target" == "$dev" ]] && echo "$link" && break
|
||||||
|
done | head -1 || echo "not found")
|
||||||
|
|
||||||
|
printf "%-12s %-10s %-30s %-20s %s\n" "$dev" "$size" "$model" "$serial" "/dev/disk/by-id/$byid"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== Bay identification via LED blink ==="
|
||||||
|
echo ""
|
||||||
|
echo "To confirm which physical bay a device is in, blink its LED:"
|
||||||
|
echo " apt install ledmon"
|
||||||
|
echo " ledctl locate=/dev/sdX # LED on"
|
||||||
|
echo " ledctl locate_off=/dev/sdX # LED off"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if command -v perccli &>/dev/null || command -v perccli64 &>/dev/null; then
|
||||||
|
PERCCLI=$(command -v perccli || command -v perccli64)
|
||||||
|
echo "=== PERC slot info ==="
|
||||||
|
$PERCCLI /c0 /eall /sall show | grep -E "^[0-9]|Drive's position|SN|WWN" || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Copy the BY-ID paths into docs/hardware-layout.md."
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Dell R730xd fan speed control (third-party GPU)
|
||||||
|
# iDRAC sets fans to 100% when non-Dell PCIe cards are detected.
|
||||||
|
# This service overrides that and manages fan speed by inlet temperature.
|
||||||
|
After=network.target
|
||||||
|
# Restart if ipmitool fails transiently (e.g. iDRAC busy at boot)
|
||||||
|
StartLimitIntervalSec=60
|
||||||
|
StartLimitBurst=5
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
ExecStart=/usr/local/sbin/fan-control.sh
|
||||||
|
# On stop, re-enable iDRAC auto control so fans are safe if service is removed
|
||||||
|
ExecStop=/usr/local/sbin/fan-control.sh --auto
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=10
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
Executable
+123
@@ -0,0 +1,123 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Dell R730xd fan speed control for third-party PCIe cards (Quadro P2200, etc.)
|
||||||
|
#
|
||||||
|
# iDRAC detects non-Dell GPUs and slams fans to 100% indefinitely.
|
||||||
|
# This script disables iDRAC automatic fan control and manages speed based
|
||||||
|
# on inlet temperature, keeping the server quiet under normal load.
|
||||||
|
#
|
||||||
|
# Install:
|
||||||
|
# apt install ipmitool
|
||||||
|
# cp fan-control.sh /usr/local/sbin/fan-control.sh
|
||||||
|
# chmod +x /usr/local/sbin/fan-control.sh
|
||||||
|
# cp fan-control.service /etc/systemd/system/
|
||||||
|
# systemctl daemon-reload && systemctl enable --now fan-control.service
|
||||||
|
#
|
||||||
|
# Manual speed test (without running as daemon):
|
||||||
|
# fan-control.sh --set 25 # set fans to 25% and exit
|
||||||
|
# fan-control.sh --auto # restore iDRAC automatic control and exit
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
IPMI="ipmitool raw 0x30 0x30"
|
||||||
|
|
||||||
|
# Fan speed thresholds by inlet temperature (°C → % speed)
|
||||||
|
# Tune these for your environment. Inlet temp sensor reads ambient air entering front.
|
||||||
|
declare -A SPEED_MAP=(
|
||||||
|
[0]=15 # < 30°C → 15% (near-silent)
|
||||||
|
[30]=20 # 30–39°C → 20%
|
||||||
|
[40]=30 # 40–44°C → 30%
|
||||||
|
[45]=40 # 45–49°C → 40%
|
||||||
|
[50]=55 # 50–54°C → 55%
|
||||||
|
[55]=75 # 55–59°C → 75%
|
||||||
|
[60]=100 # ≥ 60°C → 100% (safety)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Minimum speed floor — never go below this (protects drives and CPUs)
|
||||||
|
MIN_SPEED=15
|
||||||
|
|
||||||
|
get_inlet_temp() {
|
||||||
|
ipmitool sdr type Temperature 2>/dev/null \
|
||||||
|
| grep -i "Inlet Temp\|Ambient\|Inlet" \
|
||||||
|
| grep -oP '\d+(?= degrees)' \
|
||||||
|
| head -1 || echo "35" # safe default if sensor read fails
|
||||||
|
}
|
||||||
|
|
||||||
|
pct_to_hex() {
|
||||||
|
printf '0x%02x' "$(( $1 < 100 ? $1 : 100 ))"
|
||||||
|
}
|
||||||
|
|
||||||
|
set_fan_speed() {
|
||||||
|
local pct=$1
|
||||||
|
[[ $pct -lt $MIN_SPEED ]] && pct=$MIN_SPEED
|
||||||
|
local hex
|
||||||
|
hex=$(pct_to_hex "$pct")
|
||||||
|
$IPMI 0x02 0xff "$hex"
|
||||||
|
}
|
||||||
|
|
||||||
|
disable_auto_fan() {
|
||||||
|
$IPMI 0x01 0x00
|
||||||
|
echo "$(date): iDRAC automatic fan control DISABLED"
|
||||||
|
}
|
||||||
|
|
||||||
|
enable_auto_fan() {
|
||||||
|
$IPMI 0x01 0x01
|
||||||
|
echo "$(date): iDRAC automatic fan control RE-ENABLED"
|
||||||
|
}
|
||||||
|
|
||||||
|
speed_for_temp() {
|
||||||
|
local temp=$1
|
||||||
|
local speed=$MIN_SPEED
|
||||||
|
for threshold in $(echo "${!SPEED_MAP[@]}" | tr ' ' '\n' | sort -n); do
|
||||||
|
[[ $temp -ge $threshold ]] && speed=${SPEED_MAP[$threshold]}
|
||||||
|
done
|
||||||
|
echo "$speed"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Argument handling ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
case "${1:-}" in
|
||||||
|
--auto)
|
||||||
|
enable_auto_fan
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
--set)
|
||||||
|
pct="${2:?Usage: fan-control.sh --set <0-100>}"
|
||||||
|
disable_auto_fan
|
||||||
|
set_fan_speed "$pct"
|
||||||
|
echo "$(date): Fans set to ${pct}%"
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
--temp)
|
||||||
|
echo "Inlet temp: $(get_inlet_temp)°C"
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
"")
|
||||||
|
# Daemon mode — fall through to loop
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 [--auto | --set <pct> | --temp]"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# ── Daemon loop ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
trap 'enable_auto_fan; exit 0' SIGTERM SIGINT
|
||||||
|
|
||||||
|
echo "$(date): Fan control daemon starting"
|
||||||
|
disable_auto_fan
|
||||||
|
|
||||||
|
last_speed=-1
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
temp=$(get_inlet_temp)
|
||||||
|
target=$(speed_for_temp "$temp")
|
||||||
|
|
||||||
|
if [[ $target -ne $last_speed ]]; then
|
||||||
|
set_fan_speed "$target"
|
||||||
|
echo "$(date): Inlet ${temp}°C → fans ${target}%"
|
||||||
|
last_speed=$target
|
||||||
|
fi
|
||||||
|
|
||||||
|
sleep 30
|
||||||
|
done
|
||||||
Executable
+131
@@ -0,0 +1,131 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Configure IOMMU + VFIO passthrough for two Quadro P2200 GPUs on Proxmox VE 9.x.
|
||||||
|
# Run on the Proxmox host. Requires a reboot to take effect.
|
||||||
|
#
|
||||||
|
# After running this script:
|
||||||
|
# 1. Reboot the host
|
||||||
|
# 2. Verify with: lspci -nnk | grep -A3 -i nvidia
|
||||||
|
# Driver should show 'vfio-pci', not 'nouveau' or 'nvidia'
|
||||||
|
# 3. Assign GPUs to VMs via qm set or the Proxmox UI
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
CMDLINE_FILE="/etc/kernel/cmdline"
|
||||||
|
MODPROBE_VFIO="/etc/modprobe.d/vfio.conf"
|
||||||
|
MODPROBE_BLACKLIST="/etc/modprobe.d/blacklist-gpu.conf"
|
||||||
|
INITRAMFS_MODULES="/etc/initramfs-tools/modules"
|
||||||
|
|
||||||
|
# ── Step 1: Check IOMMU groups ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "=== Current IOMMU groups (GPUs) ==="
|
||||||
|
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
||||||
|
n=${d#*/iommu_groups/*}; n=${n%%/*}
|
||||||
|
dev=$(lspci -nns "${d##*/}" 2>/dev/null || true)
|
||||||
|
[[ "$dev" =~ VGA|3D|Display|Audio ]] && printf 'Group %3s %s\n' "$n" "$dev"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# ── Step 2: Collect GPU PCI IDs ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "=== Detected NVIDIA devices ==="
|
||||||
|
lspci -nn | grep -i nvidia
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Grab all NVIDIA PCI IDs (vendor:device) for vfio-pci binding
|
||||||
|
# This captures both the GPU (VGA) and its HDMI audio sibling
|
||||||
|
NVIDIA_IDS=$(lspci -nn | grep -i nvidia | grep -oP '\[\K[0-9a-f]{4}:[0-9a-f]{4}(?=\])' | sort -u | tr '\n' ',' | sed 's/,$//')
|
||||||
|
|
||||||
|
if [[ -z "$NVIDIA_IDS" ]]; then
|
||||||
|
echo "ERROR: No NVIDIA devices found. Is the GPU installed and visible to lspci?"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "GPU PCI IDs to bind to vfio-pci: $NVIDIA_IDS"
|
||||||
|
echo ""
|
||||||
|
read -rp "Proceed with configuring VFIO passthrough? [y/N] " confirm
|
||||||
|
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
|
||||||
|
|
||||||
|
# ── Step 3: Enable IOMMU in kernel cmdline ────────────────────────────────────
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- Configuring kernel cmdline for IOMMU ---"
|
||||||
|
|
||||||
|
if [[ ! -f "$CMDLINE_FILE" ]]; then
|
||||||
|
echo "ERROR: $CMDLINE_FILE not found. Is this a Proxmox EFI system?"
|
||||||
|
echo "For legacy GRUB: edit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT instead."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
current_cmdline=$(cat "$CMDLINE_FILE")
|
||||||
|
new_cmdline="$current_cmdline"
|
||||||
|
|
||||||
|
[[ "$new_cmdline" =~ intel_iommu=on ]] || new_cmdline="$new_cmdline intel_iommu=on"
|
||||||
|
[[ "$new_cmdline" =~ iommu=pt ]] || new_cmdline="$new_cmdline iommu=pt"
|
||||||
|
|
||||||
|
# Deduplicate spaces
|
||||||
|
new_cmdline=$(echo "$new_cmdline" | tr -s ' ' | sed 's/^ //;s/ $//')
|
||||||
|
|
||||||
|
echo "$new_cmdline" > "$CMDLINE_FILE"
|
||||||
|
echo "Written: $CMDLINE_FILE"
|
||||||
|
echo " $new_cmdline"
|
||||||
|
|
||||||
|
proxmox-boot-tool refresh
|
||||||
|
echo "Boot tool refreshed."
|
||||||
|
|
||||||
|
# ── Step 4: Blacklist host GPU drivers ────────────────────────────────────────
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- Blacklisting nouveau and nvidia on host ---"
|
||||||
|
cat > "$MODPROBE_BLACKLIST" <<EOF
|
||||||
|
blacklist nouveau
|
||||||
|
blacklist nvidia
|
||||||
|
blacklist nvidiafb
|
||||||
|
blacklist nvidia_drm
|
||||||
|
blacklist nvidia_modeset
|
||||||
|
options nouveau modeset=0
|
||||||
|
EOF
|
||||||
|
echo "Written: $MODPROBE_BLACKLIST"
|
||||||
|
|
||||||
|
# ── Step 5: Bind GPUs to vfio-pci ────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- Binding GPU IDs to vfio-pci ---"
|
||||||
|
cat > "$MODPROBE_VFIO" <<EOF
|
||||||
|
options vfio-pci ids=$NVIDIA_IDS
|
||||||
|
softdep nouveau pre: vfio-pci
|
||||||
|
softdep nvidia pre: vfio-pci
|
||||||
|
EOF
|
||||||
|
echo "Written: $MODPROBE_VFIO"
|
||||||
|
echo " IDs: $NVIDIA_IDS"
|
||||||
|
|
||||||
|
# ── Step 6: Load vfio modules early in initramfs ─────────────────────────────
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- Adding vfio modules to initramfs ---"
|
||||||
|
for mod in vfio vfio_iommu_type1 vfio_pci vfio_pci_core; do
|
||||||
|
grep -qxF "$mod" "$INITRAMFS_MODULES" 2>/dev/null || echo "$mod" >> "$INITRAMFS_MODULES"
|
||||||
|
done
|
||||||
|
echo "Updated: $INITRAMFS_MODULES"
|
||||||
|
|
||||||
|
update-initramfs -u -k all
|
||||||
|
echo "Initramfs updated."
|
||||||
|
|
||||||
|
# ── Step 7: Print PCI addresses for VM config ─────────────────────────────────
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== GPU PCI addresses for VM assignment ==="
|
||||||
|
echo "Use these in qm set or the Proxmox UI (Hardware → Add → PCI Device):"
|
||||||
|
echo ""
|
||||||
|
lspci -nn | grep -i nvidia | while read -r line; do
|
||||||
|
addr=$(echo "$line" | awk '{print $1}')
|
||||||
|
desc=$(echo "$line" | cut -d' ' -f2-)
|
||||||
|
printf " hostpciN: 0000:%s,pcie=1 # %s\n" "$addr" "$desc"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM."
|
||||||
|
echo "Example for GPU at 01:00.0 (audio at 01:00.1):"
|
||||||
|
echo " hostpci0: 0000:01:00,pcie=1,x-vga=1"
|
||||||
|
echo " (Proxmox will auto-include 01:00.1 when you use the .0 address)"
|
||||||
|
echo ""
|
||||||
|
echo "Done. Reboot the host to activate IOMMU and vfio-pci binding."
|
||||||
Executable
+53
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Convert PERC H730 drives to Non-RAID (per-disk passthrough) mode.
|
||||||
|
# Run once on the Proxmox host before assigning drives to VMs.
|
||||||
|
# Requires: perccli (install from Dell's website or local .deb)
|
||||||
|
#
|
||||||
|
# Dell PERC H730 does not have a true HBA/IT mode. Non-RAID mode is the
|
||||||
|
# equivalent — each disk is presented directly to the OS with SMART intact.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
PERCCLI=$(command -v perccli || command -v perccli64 || true)
|
||||||
|
if [[ -z "$PERCCLI" ]]; then
|
||||||
|
echo "perccli not found. Install from:"
|
||||||
|
echo " https://www.dell.com/support (search 'PERCCLI')"
|
||||||
|
echo " or: dpkg -i perccli_*.deb"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Controller overview ==="
|
||||||
|
$PERCCLI show
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== Drives on controller 0 ==="
|
||||||
|
$PERCCLI /c0 /eall /sall show
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
read -rp "Proceed with converting all non-OS drives to Non-RAID mode? [y/N] " confirm
|
||||||
|
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
|
||||||
|
|
||||||
|
# Identify enclosure IDs (typically 8 for the internal backplane on R730xd)
|
||||||
|
ENCLOSURES=$($PERCCLI /c0 /eall show | awk '/^[0-9]/{print $1}' | sort -u)
|
||||||
|
|
||||||
|
for enc in $ENCLOSURES; do
|
||||||
|
echo ""
|
||||||
|
echo "=== Processing enclosure $enc ==="
|
||||||
|
SLOTS=$($PERCCLI /c0 /e"$enc" /sall show | awk '/UGood|Onln|JBOD|DHS/{print $2}' | sort -u)
|
||||||
|
|
||||||
|
for slot in $SLOTS; do
|
||||||
|
echo -n " Slot $slot: setting to Good... "
|
||||||
|
$PERCCLI /c0 /e"$enc" /s"$slot" set good force 2>&1 | grep -i "success\|error\|already" || true
|
||||||
|
|
||||||
|
echo -n " Slot $slot: setting to Non-RAID... "
|
||||||
|
$PERCCLI /c0 /e"$enc" /s"$slot" set nonraid 2>&1 | grep -i "success\|error\|already" || true
|
||||||
|
done
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== Final drive state ==="
|
||||||
|
$PERCCLI /c0 /eall /sall show
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Done. Reboot the host for changes to take full effect."
|
||||||
|
echo "After reboot, run build-bay-map.sh to map bays to /dev/disk/by-id paths."
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Stagger hard drive spin-up to limit PSU current surge
|
||||||
|
# Runs early in boot, before storage services and VMs start, so drives
|
||||||
|
# are already spun up and staggered before any heavy I/O begins.
|
||||||
|
DefaultDependencies=no
|
||||||
|
After=local-fs-pre.target
|
||||||
|
Before=local-fs.target sysinit.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/usr/local/sbin/stagger-spinup.sh --linux
|
||||||
|
RemainAfterExit=yes
|
||||||
|
StandardOutput=journal
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=sysinit.target
|
||||||
Executable
+104
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Stagger hard drive spin-up on Dell R730xd to avoid PSU current surge at boot.
|
||||||
|
#
|
||||||
|
# 12 x 3.5" HDDs spinning up simultaneously can spike ~240W for 2-3 seconds.
|
||||||
|
# Staggering them 2-3 seconds apart keeps the surge well within PSU limits.
|
||||||
|
#
|
||||||
|
# TWO layers of protection:
|
||||||
|
# 1. iDRAC BIOS setting — fires at POST, before the OS loads (preferred)
|
||||||
|
# 2. Linux systemd service — staggers drives that weren't spun up at POST
|
||||||
|
#
|
||||||
|
# SETUP
|
||||||
|
# ─────
|
||||||
|
# Layer 1 (iDRAC, run once):
|
||||||
|
# bash stagger-spinup.sh --idrac
|
||||||
|
# # Requires racadm and iDRAC network access, then reboot to apply.
|
||||||
|
#
|
||||||
|
# Layer 2 (Linux service, run once):
|
||||||
|
# cp stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
|
||||||
|
# chmod +x /usr/local/sbin/stagger-spinup.sh
|
||||||
|
# Install the systemd unit below, then:
|
||||||
|
# systemctl daemon-reload && systemctl enable stagger-spinup.service
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
STAGGER_SECONDS=3 # delay between each drive spin-up
|
||||||
|
HDPARM=$(command -v hdparm || true)
|
||||||
|
|
||||||
|
# ── iDRAC BIOS method (Layer 1) ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
configure_idrac() {
|
||||||
|
if ! command -v racadm &>/dev/null; then
|
||||||
|
echo "racadm not found."
|
||||||
|
echo "Option A: Run from iDRAC SSH:"
|
||||||
|
echo " ssh root@<idrac-ip>"
|
||||||
|
echo " racadm set BIOS.StorageSettings.HddSeq Enabled"
|
||||||
|
echo " racadm jobqueue create BIOS.Setup.1-1"
|
||||||
|
echo " # Then reboot to apply."
|
||||||
|
echo ""
|
||||||
|
echo "Option B: iDRAC web UI:"
|
||||||
|
echo " System BIOS → Power Management → Hard Disk Drive Sequencing → Enabled"
|
||||||
|
echo " Apply and reboot."
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Enabling iDRAC hard disk drive sequencing (staggered spin-up)..."
|
||||||
|
racadm set BIOS.StorageSettings.HddSeq Enabled
|
||||||
|
racadm jobqueue create BIOS.Setup.1-1
|
||||||
|
echo "Job queued. Reboot for the BIOS setting to take effect."
|
||||||
|
echo "Verify after reboot with: racadm get BIOS.StorageSettings.HddSeq"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Linux spin-up stagger (Layer 2) ──────────────────────────────────────────
|
||||||
|
# Called by the systemd service early in boot.
|
||||||
|
# Reads each block device in sequence with a short delay, causing drives in
|
||||||
|
# standby to spin up one at a time rather than simultaneously.
|
||||||
|
|
||||||
|
stagger_linux() {
|
||||||
|
if [[ -z "$HDPARM" ]]; then
|
||||||
|
echo "hdparm not found — apt install hdparm"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mapfile -t drives < <(lsblk -dn -o NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}' | sort)
|
||||||
|
|
||||||
|
if [[ ${#drives[@]} -eq 0 ]]; then
|
||||||
|
echo "No block devices found."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$(date): Staggering spin-up for ${#drives[@]} drives (${STAGGER_SECONDS}s apart)..."
|
||||||
|
|
||||||
|
for dev in "${drives[@]}"; do
|
||||||
|
# Skip if device is already active (check power mode)
|
||||||
|
state=$($HDPARM -C "$dev" 2>/dev/null | grep -oP '(?<=drive state is: )\S+' || echo "unknown")
|
||||||
|
|
||||||
|
if [[ "$state" == "standby" || "$state" == "sleeping" ]]; then
|
||||||
|
echo "$(date): Waking $dev (was: $state)"
|
||||||
|
# A zero-length read is enough to trigger spin-up
|
||||||
|
dd if="$dev" of=/dev/null bs=512 count=1 status=none 2>/dev/null || true
|
||||||
|
sleep "$STAGGER_SECONDS"
|
||||||
|
else
|
||||||
|
echo "$(date): $dev already active (state: $state) — skipping"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "$(date): Stagger complete."
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Argument dispatch ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
case "${1:-}" in
|
||||||
|
--idrac)
|
||||||
|
configure_idrac
|
||||||
|
;;
|
||||||
|
--linux|"")
|
||||||
|
stagger_linux
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 [--idrac | --linux]"
|
||||||
|
echo " --idrac Configure iDRAC BIOS staggered spin-up (one-time setup)"
|
||||||
|
echo " --linux Stagger drives via hdparm now (run by systemd service)"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Proxmox VM 100 — Frigate NVR
|
||||||
|
#
|
||||||
|
# Copy to /etc/pve/qemu-server/100.conf after filling in:
|
||||||
|
# - hostpci0: real PCI address from gpu-passthrough-setup.sh output
|
||||||
|
# - scsi1–scsi8: real /dev/disk/by-id paths from build-bay-map.sh output
|
||||||
|
# - net0: real MAC (Proxmox generates one; leave it if creating via UI)
|
||||||
|
#
|
||||||
|
# Requirements on the Proxmox host before starting this VM:
|
||||||
|
# 1. scripts/perc-nonraid.sh — drives in non-RAID mode
|
||||||
|
# 2. scripts/gpu-passthrough-setup.sh — IOMMU + vfio-pci active, host rebooted
|
||||||
|
# 3. Coral USB plugged in
|
||||||
|
|
||||||
|
agent: 1
|
||||||
|
bios: ovmf
|
||||||
|
boot: order=scsi0
|
||||||
|
cores: 8
|
||||||
|
cpu: host
|
||||||
|
machine: q35
|
||||||
|
memory: 16384
|
||||||
|
name: frigate
|
||||||
|
numa: 0
|
||||||
|
ostype: l26
|
||||||
|
scsihw: virtio-scsi-single
|
||||||
|
sockets: 1
|
||||||
|
|
||||||
|
# OS disk — on Proxmox local storage, adjust pool name as needed
|
||||||
|
scsi0: local-lvm:vm-100-disk-0,cache=writeback,size=64G
|
||||||
|
|
||||||
|
# Data drives — bays 1–8, raw disk passthrough
|
||||||
|
# Fill in by-id paths from build-bay-map.sh
|
||||||
|
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
|
||||||
|
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY2,size=0
|
||||||
|
scsi3: /dev/disk/by-id/PLACEHOLDER_BAY3,size=0
|
||||||
|
scsi4: /dev/disk/by-id/PLACEHOLDER_BAY4,size=0
|
||||||
|
scsi5: /dev/disk/by-id/PLACEHOLDER_BAY5,size=0
|
||||||
|
scsi6: /dev/disk/by-id/PLACEHOLDER_BAY6,size=0
|
||||||
|
scsi7: /dev/disk/by-id/PLACEHOLDER_BAY7,size=0
|
||||||
|
scsi8: /dev/disk/by-id/PLACEHOLDER_BAY8,size=0
|
||||||
|
|
||||||
|
# Quadro P2200 #1 — GPU passthrough
|
||||||
|
# Replace XX:00 with real PCI address (e.g. 03:00)
|
||||||
|
# pcie=1 uses PCIe bus; x-vga=1 passes primary display output
|
||||||
|
# Proxmox auto-includes the HDMI audio sibling (XX:00.1)
|
||||||
|
hostpci0: 0000:XX:00,pcie=1,x-vga=1
|
||||||
|
|
||||||
|
# Google Coral USB — pass both VID:PID values
|
||||||
|
# Coral re-enumerates after first inference load: 1a6e:089a → 18d1:9302
|
||||||
|
usb0: host=1a6e:089a
|
||||||
|
usb1: host=18d1:9302
|
||||||
|
|
||||||
|
# Network
|
||||||
|
net0: virtio=BC:24:11:00:00:64,bridge=vmbr0,firewall=1
|
||||||
|
|
||||||
|
# EFI disk (required for OVMF/UEFI boot with q35 machine)
|
||||||
|
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Proxmox VM 101 — General purpose with GPU
|
||||||
|
#
|
||||||
|
# Copy to /etc/pve/qemu-server/101.conf after filling in:
|
||||||
|
# - hostpci0: real PCI address of Quadro P2200 #2
|
||||||
|
# - scsi1–scsi2: real /dev/disk/by-id paths for bays 9–10
|
||||||
|
|
||||||
|
agent: 1
|
||||||
|
bios: ovmf
|
||||||
|
boot: order=scsi0
|
||||||
|
cores: 8
|
||||||
|
cpu: host
|
||||||
|
machine: q35
|
||||||
|
memory: 16384
|
||||||
|
name: vm101
|
||||||
|
numa: 0
|
||||||
|
ostype: l26
|
||||||
|
scsihw: virtio-scsi-single
|
||||||
|
sockets: 1
|
||||||
|
|
||||||
|
# OS disk
|
||||||
|
scsi0: local-lvm:vm-101-disk-0,cache=writeback,size=64G
|
||||||
|
|
||||||
|
# Data drives — bays 9–10
|
||||||
|
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY9,size=0
|
||||||
|
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY10,size=0
|
||||||
|
|
||||||
|
# Quadro P2200 #2
|
||||||
|
# Replace XX:00 with real PCI address from gpu-passthrough-setup.sh output
|
||||||
|
hostpci0: 0000:XX:00,pcie=1,x-vga=1
|
||||||
|
|
||||||
|
# Network
|
||||||
|
net0: virtio=BC:24:11:00:00:65,bridge=vmbr0,firewall=1
|
||||||
|
|
||||||
|
# EFI disk
|
||||||
|
efidisk0: local-lvm:vm-101-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Proxmox VM 102 — Storage / utility VM
|
||||||
|
#
|
||||||
|
# Copy to /etc/pve/qemu-server/102.conf after filling in:
|
||||||
|
# - scsi1–scsi2: real /dev/disk/by-id paths for bays 11–12
|
||||||
|
|
||||||
|
agent: 1
|
||||||
|
bios: ovmf
|
||||||
|
boot: order=scsi0
|
||||||
|
cores: 4
|
||||||
|
cpu: host
|
||||||
|
machine: q35
|
||||||
|
memory: 8192
|
||||||
|
name: vm102
|
||||||
|
numa: 0
|
||||||
|
ostype: l26
|
||||||
|
scsihw: virtio-scsi-single
|
||||||
|
sockets: 1
|
||||||
|
|
||||||
|
# OS disk
|
||||||
|
scsi0: local-lvm:vm-102-disk-0,cache=writeback,size=32G
|
||||||
|
|
||||||
|
# Data drives — bays 11–12
|
||||||
|
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY11,size=0
|
||||||
|
scsi2: /dev/disk/by-id/PLACEHOLDER_BAY12,size=0
|
||||||
|
|
||||||
|
# Network
|
||||||
|
net0: virtio=BC:24:11:00:00:66,bridge=vmbr0,firewall=1
|
||||||
|
|
||||||
|
# EFI disk
|
||||||
|
efidisk0: local-lvm:vm-102-disk-1,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||||
Reference in New Issue
Block a user