root/local_proxmox
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #4 from outis1one/claude/proxmox-hardware-passthrough-ylAYx

Browse Source 
Add detailed button-by-button walkthrough for phases 7-9
This commit is contained in:
Outis
2026-04-24 16:47:18 -04:00
committed by GitHub
parent 46acd2677c 677a85e908
commit e78a3ec0bc
2 changed files with 678 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/handoff.md
+143
View File
@@ -0,0 +1,143 @@
# Session Hand-off Document
## Dell R730xd — Proxmox VE 9.1 Homelab Setup
**Branch:** `claude/proxmox-hardware-passthrough-ylAYx`
**Repo:** `outis1one/local_proxmox`
---
## Hardware Summary
| Component | Detail |
|-----------|--------|
| Server | Dell PowerEdge R730xd |
| RAID controller | PERC H730 (configured in per-disk non-RAID mode for 3.5" drives) |
| OS drives | 2x 2.5" rear drives — RAID1 mirror via H730 |
| Data drives | Up to 12x 3.5" bays (not all populated) |
| GPUs | 2x Quadro P2200 5GB (VFIO passthrough, one per VM) |
| TPU | Google Coral USB (Frigate object detection, VM 100) |
| iDRAC | iDRAC8 **Express** — no virtual console, no virtual media |
| PSU | 750W (single or redundant pair — stagger spin-up configured to protect PSU) |
## VM Layout
| VM ID | Role | Drives | GPU | Coral |
|-------|------|--------|-----|-------|
| 100 | Frigate NVR | Bays 1–8 | P2200 #1 | Yes |
| 101 | General purpose | Bays 9–10 | P2200 #2 | No |
| 102 | Utility/storage | Bays 11–12 | None | No |
---
## What Is In The Repo
### scripts/
| File | Purpose | Status |
|------|---------|--------|
| `perc-nonraid.sh` | Converts H730 3.5" drives to per-disk non-RAID mode | Done |
| `build-bay-map.sh` | Maps physical bay numbers to /dev/disk/by-id paths via LED blink | Done |
| `gpu-passthrough-setup.sh` | Configures IOMMU + vfio-pci for both Quadro P2200s | Done |
| `fan-control.sh` | Manages fan speed via ipmitool — prevents 100% fan from non-Dell GPU | Done |
| `fan-control.service` | Systemd unit for fan-control.sh | Done |
| `stagger-spinup.sh` | Staggers drive spin-up at boot to limit PSU current surge | Done |
| `stagger-spinup.service` | Systemd unit for stagger-spinup.sh | Done |
### vm-configs/
| File | Purpose | Status |
|------|---------|--------|
| `100-frigate.conf.example` | VM 100 skeleton — GPU, Coral USB, 8 drive slots | **Needs PLACEHOLDER_BAYx filled in after Phase 8** |
| `101.conf.example` | VM 101 skeleton — GPU #2, 2 drive slots | **Needs PLACEHOLDER_BAYx filled in after Phase 8** |
| `102.conf.example` | VM 102 skeleton — 2 drive slots | **Needs PLACEHOLDER_BAYx filled in after Phase 8** |
### frigate/
| File | Purpose | Status |
|------|---------|--------|
| `docker-compose.yml` | Frigate container with nvidia runtime + Coral USB | Done |
| `config.yml` | Frigate config — NVDEC hwaccel, Coral detector, recording | **Needs real camera RTSP URLs added** |
### docs/
| File | Purpose | Status |
|------|---------|--------|
| `hardware-layout.md` | Bay/VM/GPU mapping worksheet | **Needs filling in after Phase 8** |
| `setup-guide.md` | High-level 12-phase overview | Done |
| `walkthrough-phases-1-3.md` | Full button-by-button: Firmware, iDRAC, BIOS | Done |
| `walkthrough-phases-4-6.md` | Full button-by-button: H730 RAID, Proxmox install, post-install | Done |
| `walkthrough-phases-7-9.md` | Full button-by-button: non-RAID drives, bay map, GPU passthrough | Done |
| `walkthrough-phases-10-12.md` | Fan control, VM creation, Frigate deploy | **NOT WRITTEN YET** |
| `complete-walkthrough.md` | All phases combined into one document | **NOT WRITTEN YET** |
---
## What Still Needs To Be Done
### 1. Write `docs/walkthrough-phases-10-12.md`
Three phases to cover in full button-by-button detail:
**Phase 10 — Fan control + stagger spin-up services**
- Copy fan-control.sh to /usr/local/sbin/, install and enable systemd service
- Verify fans slow down within 30 seconds (ipmitool sdr type Fan)
- Run stagger-spinup.sh --idrac for BIOS-level config (or iDRAC web UI path)
- Install stagger-spinup.service, enable (don't start manually — runs at boot)
**Phase 11 — Create VMs in Proxmox web UI**
- Upload Ubuntu 24.04 LTS ISO to Proxmox local storage
- Create VM 100 via web UI: General → OS → System (q35, OVMF, VirtIO SCSI Single) → Disks → CPU (host type) → Memory → Network
- Add hardware to VM 100: PCI device (GPU, All Functions, Primary GPU, PCI-Express), two USB devices (Coral 1a6e:089a and 18d1:9302), data drives via CLI (qm set)
- Install Ubuntu inside each VM via Proxmox console
- Repeat for VMs 101 and 102 (simpler — no Coral, different GPU or no GPU)
- Find VM IP addresses
**Phase 12 — Frigate inside VM 100**
- SSH into VM 100 guest OS
- Install Docker (get.docker.com script)
- Install nvidia-driver-535, reboot, verify with nvidia-smi
- Install nvidia-container-toolkit, configure Docker runtime, test with docker run nvidia-smi
- Create ZFS pool from 8 data drives (raidz2 recommended)
- Clone repo or copy frigate/ directory
- Edit config.yml with real camera RTSP URLs
- docker compose up -d, watch logs
- Access web UI at http://vm100-ip:5000
### 2. Write `docs/complete-walkthrough.md`
Combine walkthrough-phases-1-3.md + 4-6.md + 7-9.md + 10-12.md (once written)
into a single document. Remove the "next file" references at the end of each
phase section. Add a table of contents at the top.
---
## Key Facts To Know
- **iDRAC Express** — no KVM, no virtual media. Physical monitor + keyboard
required through Phase 5 (Proxmox installer). After that SSH only.
- **iDRAC default credentials:** root / calvin — must be changed (Phase 2)
- **iDRAC static IP** set in Phase 2 — needed for fan control ipmitool commands
- **IPMI over LAN** must be enabled in iDRAC (Phase 2) for fan control to work
- **Boot mode must be UEFI** (Phase 3) — legacy BIOS breaks gpu-passthrough-setup.sh
- **VT-d must be Enabled** (Phase 3) — most common reason passthrough silently fails
- **H730 OS drives** — the 2x 2.5" rear drives are RAID1 and must NOT be touched
by perc-nonraid.sh. The script targets only UGood (unconfigured) drives.
- **Quadro P2200** — 75W TDP, no external power connector needed, no Code 43
issue, NVDEC handles H.264/H.265 decode for Frigate with near-zero CPU usage
- **Coral USB re-enumerates** — passes through as 1a6e:089a before first
inference, then 18d1:9302 after. Both USB entries must be in the VM config.
- **Fan control** uses raw IPMI commands to iDRAC over LAN — works on Express
---
## Script Run Order (for reference)
When Proxmox is installed and SSH is working:
```
1. dpkg -i perccli_*.deb
2. bash scripts/perc-nonraid.sh → reboot
3. bash scripts/build-bay-map.sh → fill in hardware-layout.md and vm-configs
4. bash scripts/gpu-passthrough-setup.sh → note PCI addresses, reboot
5. install fan-control.sh + .service → verify fans slow down
6. install stagger-spinup.sh + .service
7. create VMs via Proxmox web UI
8. install Ubuntu in each VM
9. set up Frigate in VM 100
```
docs/walkthrough-phases-7-9.md
+535
View File
@@ -0,0 +1,535 @@
# Phases 7–9: Drive Non-RAID, Bay Mapping, GPU Passthrough
### Dell R730xd — Button-by-button walkthrough
---
## Phase 7 — Convert 3.5" Drives to Non-RAID Mode
**What this does:** Right now the H730 sees your 3.5" drives as "Unconfigured
Good" — they exist but the controller is not doing anything with them. We need
to put them into **Non-RAID mode**, which tells the H730 to present each drive
directly to Linux as an individual block device, with SMART health data intact.
Without this step, Linux cannot see the drives at all — the H730 hides them.
**The 2.5" OS drives are not affected** — they are a RAID1 virtual disk and the
script only targets unconfigured physical drives.
---
### Step 7.1 — Download and install perccli
`perccli` is Dell's command-line tool for managing the H730. The `perc-nonraid.sh`
script uses it to talk to the controller.
SSH into the Proxmox host if you are not already connected:
```bash
ssh root@192.168.1.10
```
Download perccli from Dell. On the Proxmox host, run:
```bash
cd /tmp
# Download directly from Dell (this URL is stable — it redirects to the latest version)
curl -L "https://dl.dell.com/FOLDER06708442M/1/perccli_7.1-007.0318_linux.tar.gz" \
-o perccli.tar.gz
```
> **If that URL does not work** (Dell occasionally moves files):
> On your other computer go to **https://www.dell.com/support/home**, search
> for your Service Tag, click **Drivers & Downloads**, filter Category to
> **Systems Management**, and search for "PERCCLI". Download the Linux version.
> Copy the `.tar.gz` or `.deb` file to the Proxmox host with:
> ```bash
> scp perccli_*.tar.gz root@192.168.1.10:/tmp/
> ```
Extract and install:
```bash
cd /tmp
tar xzf perccli.tar.gz
# The .deb file is inside the extracted folder:
find /tmp -name "*.deb" | head -5
# You will see something like: /tmp/perccli_007.1907.0000.0000_amd64.deb
dpkg -i /tmp/perccli_*.deb
```
Verify it installed:
```bash
perccli show
```
You should see output starting with something like:
```
CLI Version = 007.1907.0000.0000
Operating system = Linux5.x
Controller = 0
Status = Success
...
```
If you see `command not found`, try `perccli64` instead — some versions install
under that name.
---
### Step 7.2 — Run the non-RAID conversion script
```bash
bash /opt/local_proxmox/scripts/perc-nonraid.sh
```
The script first prints a summary of your controller and all drives:
```
=== Controller overview ===
...
=== Drives on controller 0 ===
-------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-------------------------------------------------------------
8:0 7 UGood - 3.637 TB SATA HDD N N 512B WD... U
8:1 8 UGood - 3.637 TB SATA HDD N N 512B WD... U
...
8:9 16 Onln 0 136.73 GB SATA SSD N N 512B TOSHIBA U ← OS drive (in VD)
8:10 17 Onln 0 136.73 GB SATA SSD N N 512B TOSHIBA U ← OS drive (in VD)
```
The columns that matter:
- **EID:Slt** — Enclosure ID and slot number (physical location in the backplane)
- **State** — `UGood` means unconfigured and ready. `Onln` means it is part of
a virtual disk (your OS mirror). The script only touches `UGood` drives.
- **Size** — confirms which are 3.5" data drives vs the small OS drives
The script then asks:
```
Proceed with converting all non-OS drives to Non-RAID mode? [y/N]
```
Type `y` and press Enter.
The script works through each drive. For each one you will see:
```
Slot 0: setting to Good... Status = Success
Slot 0: setting to Non-RAID... Status = Success
Slot 1: setting to Good... Status = Success
Slot 1: setting to Non-RAID... Status = Success
...
```
When all drives are done the script prints the final state — drives that were
`UGood` should now show `JBOD` (which is Dell's term for non-RAID/per-disk mode):
```
=== Final drive state ===
EID:Slt State
8:0 JBOD ←
8:1 JBOD ←
8:2 JBOD ←
...
8:9 Onln ← OS drive, untouched
8:10 Onln ← OS drive, untouched
```
---
### Step 7.3 — Reboot
```bash
reboot
```
Wait about 60 seconds, then SSH back in:
```bash
ssh root@192.168.1.10
```
### Step 7.4 — Verify drives are visible
```bash
lsblk -d -o NAME,SIZE,TYPE,ROTA
```
You should see your drives listed. Something like:
```
NAME SIZE TYPE ROTA
sda 136.7G disk 0 ← OS-Mirror RAID1 virtual disk (the 2.5" drives)
sdb 3.6T disk 1 ← 3.5" data drive
sdc 3.6T disk 1
sdd 3.6T disk 1
...
```
If you see only `sda` and nothing else, the non-RAID conversion did not take
effect — reboot again and re-run `lsblk`. If still missing, re-run
`perc-nonraid.sh` and check the output for any lines that say `Failed`.
---
## Phase 8 — Map Physical Bays to Drives
**The problem:** Linux names drives `sdb`, `sdc`, `sdd` etc. based on the order
it finds them at boot — that order can change. What you need is the **stable
hardware ID** (`/dev/disk/by-id/...`) for each drive, tied to its physical bay
number, so you can reliably assign "bay 3" to a specific VM forever.
This phase generates that map.
---
### Step 8.1 — Run the bay mapping script
```bash
bash /opt/local_proxmox/scripts/build-bay-map.sh
```
Output looks like this (truncated example):
```
=== Drive inventory ===
DEVICE SIZE MODEL SERIAL BY-ID PATH
------ ---- ----- ------ ---------
/dev/sdb 3.6T WDC_WD4000FYYZ WD-XXXXXXXXXXXX /dev/disk/by-id/scsi-35000cca23b7d4eb8
/dev/sdc 3.6T WDC_WD4000FYYZ WD-XXXXXXXXXXXX /dev/disk/by-id/scsi-35000cca23b5e1234
/dev/sdd 3.6T ST4000NM0023 Z1Z2XXXXXX /dev/disk/by-id/scsi-35000c500a0000001
...
=== Bay identification via LED blink ===
To confirm which physical bay a device is in, blink its LED:
ledctl locate=/dev/sdX # LED on
ledctl locate_off=/dev/sdX # LED off
```
The script gives you the by-id path for each device. Now you need to figure out
**which physical bay each device is in**.
---
### Step 8.2 — Walk the bays with LED blink
This is the physical part. You need to be at the server (or have someone there).
For each drive, run the blink command, walk to the server, see which bay's amber
LED is lit, note the bay number, then turn it off:
```bash
# Blink sdb
ledctl locate=/dev/sdb
# Walk to the server, find the lit bay — write down: sdb = bay X
ledctl locate_off=/dev/sdb
# Blink sdc
ledctl locate=/dev/sdc
# Walk to server, find the lit bay — write down: sdc = bay X
ledctl locate_off=/dev/sdc
# Repeat for each drive
```
> **Bay numbering on the R730xd:** Bays are numbered left to right, top to
> bottom when facing the front of the server. Bay 1 is top-left. The exact
> labeling depends on your bezel — some models label them 0–11, others 1–12.
> Use whatever number is printed or silk-screened on the chassis next to the bay.
> **If the LED does not blink:** The `ledmon` daemon must be running. Start it:
> ```bash
> systemctl start ledmon
> ledctl locate=/dev/sdb
> ```
---
### Step 8.3 — Fill in the hardware layout document
Open the layout document:
```bash
nano /opt/local_proxmox/docs/hardware-layout.md
```
Fill in the table using what you noted in Step 8.2. It looks like this — fill
in the `by-id` column from the script output and the bay number from the LED walk:
```
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) |
|-----|-------------|------|---------------------------------------------|
| 1 | 100 | 4TB | scsi-35000cca23b7d4eb8 |
| 2 | 100 | 4TB | scsi-35000cca23b5e1234 |
...
```
Save with **Ctrl+O**, Enter, then **Ctrl+X** to exit nano.
---
### Step 8.4 — Fill in the VM config placeholders
Now open each VM config example and replace the `PLACEHOLDER_BAYx` entries with
the real by-id paths:
```bash
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
```
Change lines like:
```
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
```
To the real path:
```
scsi1: /dev/disk/by-id/scsi-35000cca23b7d4eb8,size=0
```
Do this for all 8 drives in VM 100, drives 9–10 in VM 101, and drives 11–12 in
VM 102. Save each file.
> **The full path starts with `/dev/disk/by-id/`** — but in the Proxmox VM
> config you write the full path. Double-check by running:
> ```bash
> ls -la /dev/disk/by-id/ | grep -v part
> ```
> You will see the symlinks and the drives they point to.
---
## Phase 9 — GPU Passthrough Setup
**What this does:** Tells the Linux kernel to stop trying to use the two Quadro
P2200s itself and instead hand them over to the VFIO driver, which holds them
ready to be claimed by a virtual machine.
Three things happen:
1. IOMMU is turned on in the kernel (the hardware feature VT-d enables)
2. The host's NVIDIA/nouveau GPU drivers are blocked from loading
3. The VFIO driver claims both GPUs at boot, before any other driver can
---
### Step 9.1 — Check your IOMMU groups first
Before running the script, verify that VT-d is actually active:
```bash
dmesg | grep -i iommu | head -10
```
You should see lines like:
```
DMAR: IOMMU enabled
Intel-IOMMU: enabled
```
If you see nothing or see "disabled", IOMMU is not active. Go back to Phase 3
and re-check that VT-d is set to **Enabled** in BIOS, then reboot and try again.
Also check that each GPU is in its own IOMMU group:
```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done | sort -V | grep -i nvidia
```
Example good output — each GPU is in a different group:
```
IOMMU Group 24 03:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
IOMMU Group 24 03:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
IOMMU Group 31 04:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
IOMMU Group 31 04:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
```
Each GPU (`03:00.0` and `04:00.0`) and its audio sibling (`03:00.1` and
`04:00.1`) are together in their own group — that is exactly what you want.
> **If both GPUs are in the same IOMMU group as other devices** (chipset, NICs,
> etc.), you may need to enable **ACS** (Access Control Services). This is rare
> on server hardware like the R730xd which has good IOMMU separation. If you
> hit this, ask before proceeding.
---
### Step 9.2 — Run the GPU passthrough script
```bash
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
```
The script first prints your NVIDIA devices:
```
=== Detected NVIDIA devices ===
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
03:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
04:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
GPU PCI IDs to bind to vfio-pci: 10de:1c35,10de:10f1
```
Then asks:
```
Proceed with configuring VFIO passthrough? [y/N]
```
Type `y` and press Enter.
The script runs through these steps — you will see each one printed:
```
--- Configuring kernel cmdline for IOMMU ---
Written: /etc/kernel/cmdline
Boot tool refreshed.
--- Blacklisting nouveau and nvidia on host ---
Written: /etc/modprobe.d/blacklist-gpu.conf
--- Binding GPU IDs to vfio-pci ---
Written: /etc/modprobe.d/vfio.conf
IDs: 10de:1c35,10de:10f1
--- Adding vfio modules to initramfs ---
Updated: /etc/initramfs-tools/modules
Initramfs updated.
```
At the end it prints the PCI addresses you need:
```
=== GPU PCI addresses for VM assignment ===
hostpci0: 0000:03:00,pcie=1 # VGA: NVIDIA GP106GL [Quadro P2200]
hostpci0: 0000:04:00,pcie=1 # VGA: NVIDIA GP106GL [Quadro P2200]
IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM.
```
**Write down or copy these addresses** — you need them in Phase 11 when
creating the VMs. In this example:
- GPU 1 is at `03:00` → goes in VM 100 (Frigate)
- GPU 2 is at `04:00` → goes in VM 101
---
### Step 9.3 — Update the VM configs with the GPU addresses
While you have the addresses, add them to the VM configs now:
```bash
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
```
Find the line:
```
hostpci0: 0000:XX:00,pcie=1,x-vga=1
```
Replace `XX:00` with your GPU 1 address, e.g.:
```
hostpci0: 0000:03:00,pcie=1,x-vga=1
```
Save, then do the same for VM 101:
```bash
nano /opt/local_proxmox/vm-configs/101.conf.example
# Change XX:00 to 04:00 (GPU 2)
```
---
### Step 9.4 — Reboot
```bash
reboot
```
Wait about 60 seconds, then SSH back in:
```bash
ssh root@192.168.1.10
```
---
### Step 9.5 — Verify passthrough is working
```bash
lspci -nnk | grep -A3 -i nvidia
```
For each GPU, look for the driver line. It **must** say `vfio-pci`:
```
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
Subsystem: ...
Kernel driver in use: vfio-pci ← correct
Kernel modules: nouveau
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
Subsystem: ...
Kernel driver in use: vfio-pci ← correct
Kernel modules: nouveau
```
**If it shows `nouveau` or `nvidia` instead of `vfio-pci`:** The blacklist did
not take effect. Run:
```bash
update-initramfs -u -k all
reboot
```
Then check again. If still wrong, verify the blacklist file exists:
```bash
cat /etc/modprobe.d/blacklist-gpu.conf
# Should show: blacklist nouveau, blacklist nvidia, etc.
cat /etc/modprobe.d/vfio.conf
# Should show: options vfio-pci ids=10de:1c35,10de:10f1
```
Also confirm IOMMU is in the kernel command line:
```bash
cat /etc/kernel/cmdline
# Should contain: intel_iommu=on iommu=pt
```
---
### Phase 9 complete — where you are now
| What is done | Status |
|---|---|
| 3.5" drives in non-RAID mode, visible to Linux | ✓ |
| Physical bay → by-id map documented | ✓ |
| VM configs updated with real drive paths and GPU addresses | ✓ |
| IOMMU active, both GPUs claimed by vfio-pci | ✓ |
**Next:** Phase 10 — Install the fan control service (stops iDRAC from running
fans at 100% because of the non-Dell GPUs) and the stagger spin-up service.
That walkthrough is in `docs/walkthrough-phases-10-12.md`.