root/local_proxmox
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #11 from outis1one/claude/bios-vt-power-settings-mxjh0x

Browse Source 
Claude/bios vt power settings mxjh0x
This commit is contained in:
Outis
2026-06-27 00:39:09 -04:00
committed by GitHub
parent b56e4c82ff 05bcb747ae
commit b4290923cc
7 changed files with 2979 additions and 6561 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2979 -37
View File
File diff suppressed because it is too large Load Diff
docs/complete-walkthrough.md
-3043
View File
File diff suppressed because it is too large Load Diff
docs/setup-guide.md
-485
View File
@@ -1,485 +0,0 @@
# Dell R730xd — Proxmox VE 9.1 Setup Guide
Complete walkthrough from bare metal to running VMs with GPU passthrough,
Coral USB, and per-VM disk assignments.
---
## What You Need Before Starting
- USB drive (8GB+) for the Proxmox installer
- A second machine or phone to read this guide and SSH
- The iDRAC IP or physical access to a monitor + keyboard
- Internet connection on the server
---
## Phase 1 — Firmware Updates (do this first)
Outdated firmware causes mysterious IOMMU failures and fan issues. Do this
before anything else.
### Option A: Lifecycle Controller (no extra tools needed)
1. Power on the server, press **F10** when you see the Dell splash screen
2. Lifecycle Controller → **Firmware Update** → Check for updates
3. Point it at downloads.dell.com or a local repo
4. Update **iDRAC**, **BIOS**, **PERC H730**, and any NIC/HBA firmware
5. Let it reboot as many times as it needs
### Option B: Dell System Update (DSU) from bootable USB
A faster alternative if Lifecycle Controller is slow or unavailable — Dell
provides a bootable ISO that auto-detects and applies all updates.
---
## Phase 2 — iDRAC Setup
iDRAC is the out-of-band management interface. You need it configured for the
fan control script to work later.
1. Power on → press **F2** to enter System Setup → **iDRAC Settings**
2. **Network:**
- Set a static IP (easier than DHCP for a server)
- Note the IP — you'll use it for the fan control ipmitool commands
3. **User configuration:**
- Change the default `root` password
4. **IPMI over LAN:**
- iDRAC Settings → Network → IPMI Settings → **Enable IPMI over LAN: On**
- This is required for `ipmitool` fan control from the Proxmox host
---
## Phase 3 — BIOS Settings
Still in F2 System Setup. These settings are **required** for GPU and USB
passthrough to work.
| Menu path | Setting | Value |
|-----------|---------|-------|
| Processor Settings | Virtualization Technology | **Enabled** |
| Processor Settings | C States | **Disabled** |
| PCI Configuration | SR-IOV Global Enable | **Enabled** |
| Boot Settings → BIOS Boot Settings | Boot Mode | **UEFI** (not Legacy/BIOS) |
| System Profile Settings | System Profile | **Custom** |
| System Profile Settings | CPU Power Management | **Maximum Performance** |
> **VT for Direct I/O (VT-d):** Newer R730xd BIOS versions (2.19+) removed
> this toggle — VT-d is enabled by default. Don't worry if you can't find it.
> Verify it's active after Proxmox is installed: `dmesg | grep -i iommu`
> should show `DMAR: IOMMU enabled`. The kernel cmdline (`intel_iommu=on
> iommu=pt`) in Phase 9 is still required.
>
> **Hard Disk Drive Sequencing** was also removed in newer BIOS — it is no
> longer present and is not needed.
> **Boot Mode must be UEFI.** Proxmox's EFI boot tool (`proxmox-boot-tool`)
> only works with UEFI. Legacy BIOS mode breaks the GPU passthrough script.
**Apply and exit. The server will reboot.**
---
## Phase 4 — H730 RAID Configuration (pre-Proxmox)
The two 2.5" rear drives are your Proxmox OS drives. You want them mirrored
so a single drive failure doesn't take down the hypervisor.
The 3.5" drives will be converted to non-RAID later **from within Proxmox**
using the `perc-nonraid.sh` script — do not touch them here.
### Configure the 2.5" OS drives
1. Reboot → press **Ctrl+R** during POST to enter the H730 configuration
utility (or use Lifecycle Controller → RAID Configuration)
2. Select the controller
3. Find the two 2.5" rear drives
4. **Create new virtual disk:**
- RAID level: **RAID 1** (mirror)
- Select both 2.5" drives
- Strip size: 64KB (default)
- Name: `OS-Mirror` (optional)
- Initialize: **Fast Initialize**
5. Press **Ctrl+Alt+Delete** to reboot
The H730 now presents a single ~X GB RAID1 virtual disk to the OS. Proxmox
will install onto this and never know there are two physical drives behind it.
> The 3.5" drives will show as "Unconfigured Good" in the H730 — that is fine.
> Leave them alone. `perc-nonraid.sh` handles them after Proxmox is installed.
---
## Phase 5 — Install Proxmox VE 9.1
### Prepare the USB installer
On another machine, download the Proxmox VE ISO from proxmox.com and write it
to a USB drive:
```bash
# Linux/macOS
dd if=proxmox-ve_*.iso of=/dev/sdX bs=1M status=progress conv=fsync
# or use Balena Etcher (Windows/Mac/Linux GUI)
```
### Boot and install
1. Plug the USB into the R730xd
2. Power on → press **F11** for the one-time boot menu
3. Select the USB drive
4. At the Proxmox boot menu: **Install Proxmox VE (Graphical)**
5. **Target disk:** select the RAID1 virtual disk (`OS-Mirror`)
- Filesystem: **ext4** is simplest — the H730 RAID1 already gives you
redundancy, so ZFS mirroring here would be double-redundant overkill
6. **Location and timezone:** set to your region
7. **Password and email:** set a strong root password, enter an email
8. **Network configuration:**
- Management interface: the built-in NIC (usually `em1` or `eno1`)
- Hostname: e.g. `pve.local`
- IP: choose a static IP on your LAN (e.g. `192.168.1.10/24`)
- Gateway and DNS: your router's IP
9. Click **Install**
10. Remove USB when prompted, let it reboot
### First login
Open a browser on your LAN machine and go to:
```
https://192.168.1.10:8006
```
Accept the self-signed certificate warning. Login: `root` / (your password),
Realm: **Linux PAM**.
---
## Phase 6 — Proxmox Post-Install (SSH)
SSH into the host from now on — it's faster than the web console for these
steps.
```bash
ssh root@192.168.1.10
```
### 6a. Fix the apt repositories
Proxmox shows "no valid subscription" warnings when using the enterprise repo
without a license. Switch to the free repo:
```bash
# Disable enterprise repos — PVE9 uses .sources (DEB822 format), not .list
echo "# disabled - no subscription" > /etc/apt/sources.list.d/pve-enterprise.sources
echo "# disabled - no subscription" > /etc/apt/sources.list.d/ceph.sources
# Also disable any legacy .list versions if present
echo "# disabled - no subscription" > /etc/apt/sources.list.d/pve-enterprise.list
echo "# disabled - no subscription" > /etc/apt/sources.list.d/ceph.list
# Add no-subscription repos (trixie = Proxmox 9 / Debian 13)
echo "deb http://download.proxmox.com/debian/pve trixie pve-no-subscription" \
> /etc/apt/sources.list.d/pve-no-subscription.list
echo "deb http://download.proxmox.com/debian/ceph-squid trixie no-subscription" \
> /etc/apt/sources.list.d/ceph-no-subscription.list
apt update && apt dist-upgrade -y
reboot
```
### 6b. Install tools used by the scripts
```bash
apt install -y ipmitool lsscsi ledmon hdparm git
```
### 6c. Clone this repo onto the host
```bash
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
cd /opt/local_proxmox
chmod +x scripts/*.sh
```
---
## Phase 7 — H730: Set 3.5" Drives to Non-RAID Mode
Now that Proxmox is running, convert the 3.5" drives to per-disk (non-RAID)
mode. The OS drives (RAID1 virtual disk) are **not affected** — the script only
targets unconfigured physical disks.
### Install perccli
`apt` does not carry perccli. Download the RPM from Dell and convert it:
```bash
# Download (referer header required — Dell blocks plain curl)
curl -L \
--referer "https://www.dell.com/" \
--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
"https://dl.dell.com/FOLDER03559396M/1/perccli-1.17.10-1.noarch.rpm" \
-o /tmp/perccli.rpm
# Convert RPM → deb and install
apt install -y alien
alien --to-deb /tmp/perccli.rpm
dpkg -i /tmp/perccli_*.deb
# Binary lands in /opt/MegaRAID/perccli/ — symlink it into PATH
ln -s /opt/MegaRAID/perccli/perccli64 /usr/local/bin/perccli
# Verify
perccli show
```
### Run the script
```bash
bash /opt/local_proxmox/scripts/perc-nonraid.sh
```
Review the output, confirm when prompted. **Reboot after completion.**
```bash
reboot
```
After rebooting, Proxmox will see the 3.5" drives as individual block devices
(`/dev/sdb`, `/dev/sdc`, etc.).
---
## Phase 8 — Map Physical Bays to Drives
```bash
bash /opt/local_proxmox/scripts/build-bay-map.sh
```
Walk the bays with `ledctl` to confirm which physical slot is which device:
```bash
ledctl locate=/dev/sdb # LED blinks on the matching bay
ledctl locate_off=/dev/sdb
```
Fill in the `by-id` paths in `docs/hardware-layout.md`. You will need these
in Phase 11 when creating VMs.
---
## Phase 9 — GPU Passthrough Setup
```bash
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
```
The script:
- Adds `intel_iommu=on iommu=pt` to the kernel command line
- Blacklists `nouveau`/`nvidia` on the host
- Binds both Quadro P2200s to `vfio-pci`
- Rebuilds initramfs
**Note the PCI addresses it prints at the end** — you'll need them in Phase 11.
```bash
reboot
```
### Verify after reboot
```bash
lspci -nnk | grep -A3 -i nvidia
```
Both GPUs should show `Kernel driver in use: vfio-pci`. If they still show
`nouveau`, check that `/etc/modprobe.d/blacklist-gpu.conf` exists and
`update-initramfs -u` was run.
---
## Phase 10 — Fan Control and Staggered Spin-Up
### Fan control (prevents jet-engine noise from non-Dell GPUs)
```bash
cp /opt/local_proxmox/scripts/fan-control.sh /usr/local/sbin/fan-control.sh
cp /opt/local_proxmox/scripts/fan-control.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now fan-control.service
systemctl status fan-control.service
```
Test it worked — the fans should audibly slow down within 30 seconds.
To check current speed: `ipmitool sdr type Fan`
### Staggered spin-up
> **BIOS 2.19+ note:** `BIOS.StorageSettings.HddSeq` was removed from the
> R730xd firmware. The iDRAC/BIOS layer (Layer 1) no longer works — skip it
> and go straight to the Linux service below.
**Linux service** (staggers drives in standby at OS boot):
```bash
cp /opt/local_proxmox/scripts/stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
chmod +x /usr/local/sbin/stagger-spinup.sh
cp /opt/local_proxmox/scripts/stagger-spinup.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable stagger-spinup.service
```
---
## Phase 11 — Create the VMs
### Prepare the VM configs
Edit the three example configs and replace the placeholders:
```bash
cd /opt/local_proxmox/vm-configs
# Replace PLACEHOLDER_BAYx with real by-id paths from Phase 8
# Replace XX:00 with real PCI addresses from Phase 9
nano 100-frigate.conf.example
nano 101.conf.example
nano 102.conf.example
```
### Create OS disks for each VM
In the Proxmox web UI or via CLI, create the base OS disk for each VM:
```bash
# Creates the disk slots — Proxmox generates the correct scsi0 line
qm create 100 --memory 16384 --cores 8 --name frigate --net0 virtio,bridge=vmbr0
qm create 101 --memory 16384 --cores 8 --name vm101 --net0 virtio,bridge=vmbr0
qm create 102 --memory 8192 --cores 4 --name vm102 --net0 virtio,bridge=vmbr0
```
Then merge your edited conf into the generated config:
```bash
# Backup generated config, then append your hardware lines
cp /etc/pve/qemu-server/100.conf /etc/pve/qemu-server/100.conf.bak
cat 100-frigate.conf.example >> /etc/pve/qemu-server/100.conf
```
Or just open each VM in the Proxmox web UI → **Hardware** and add:
- PCI Device → your GPU (enable PCIe, enable Primary GPU for VM 100)
- USB Device → host device → `1a6e:089a` and `18d1:9302` (VM 100 only)
- Hard Disk → (use disk passthrough, SCSI controller, path = your by-id)
### Install a guest OS
Boot each VM from an ISO (upload ISOs to Proxmox under local storage →
ISO Images). Ubuntu Server 22.04 LTS is a good choice for the Frigate VM.
---
## Phase 12 — Set Up Frigate in VM 100
From inside VM 100 (SSH into the guest OS):
### Install Docker
```bash
apt update && apt install -y ca-certificates curl
curl -fsSL https://get.docker.com | sh
```
### Install NVIDIA driver + Container Toolkit
```bash
# Add NVIDIA apt repo
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install -y nvidia-driver-535 nvidia-container-toolkit
# Configure Docker to use the NVIDIA runtime
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker
# Verify the GPU is visible
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```
### Deploy Frigate
```bash
mkdir -p /opt/frigate && cd /opt/frigate
# Copy configs from the repo (or clone it inside the VM)
cp /path/to/local_proxmox/frigate/docker-compose.yml .
cp /path/to/local_proxmox/frigate/config.yml .
# Create the recordings directory (on your passthrough data drive)
# Mount your data drives first — if using ZFS:
zpool import <poolname> # or create a new pool from the raw drives
mkdir -p /mnt/frigate
# Edit config.yml and add your camera RTSP URLs
nano config.yml
docker compose up -d
docker compose logs -f # watch for errors on first start
```
Frigate UI: `http://<vm100-ip>:5000`
---
## Troubleshooting
### GPU not passing through — still shows `nouveau`
```bash
update-initramfs -u -k all && reboot
# After reboot:
lspci -nnk | grep -A3 -i nvidia # must show vfio-pci
```
### IOMMU not enabled
```bash
dmesg | grep -i iommu
# Should show: "DMAR: IOMMU enabled"
# If not: re-check Phase 9 cmdline (intel_iommu=on); VT-d is on by default in newer BIOS
cat /etc/kernel/cmdline # must contain intel_iommu=on iommu=pt
```
### Fan control not working
```bash
# Check iDRAC IP is reachable and IPMI over LAN is enabled (Phase 2)
ipmitool -I lan -H <idrac-ip> -U root -P <password> sdr type Fan
# Then check the service:
systemctl status fan-control.service
journalctl -u fan-control.service -n 50
```
### Coral not detected in Frigate
```bash
# In VM 100, check both USB IDs are present
lsusb | grep -E "1a6e|18d1"
# If missing, check the USB passthrough lines in the VM config
# Both usb0 (1a6e:089a) and usb1 (18d1:9302) must be present
```
### Drive not appearing after perc-nonraid.sh
```bash
lsblk
# If the drive is missing, check its state in perccli:
perccli /c0 /eall /sall show
# State should be "JBOD" or "UGood" — not "Offln" or "Msng"
```
docs/walkthrough-phases-1-3.md
-563
View File
@@ -1,563 +0,0 @@
# Phases 1–3: Firmware, iDRAC, and BIOS
### Dell R730xd — Button-by-button walkthrough
---
## Before You Touch Anything — Read This First
### What is iDRAC and do I need to pay for it?
iDRAC stands for **Integrated Dell Remote Access Controller**. Ignore the name.
Here is what it actually is:
There is a **second tiny computer built into your server's motherboard**. It has
its own processor, its own RAM, its own network port, and it runs 24 hours a day
as long as the server has power — even when the server is "off". Dell calls this
second computer iDRAC.
This server has **iDRAC8 Express**, which is the version built into the board
at no cost. There is a paid upgrade called iDRAC8 Enterprise but you do not
need it. Here is exactly what Express gives you and what it does not:
**Express includes (everything this guide uses):**
- iDRAC web UI — hardware health dashboard, temperatures, fan speeds, event logs
- Power the server on and off remotely from the web UI
- IPMI over LAN — lets `ipmitool` send fan control commands from Proxmox
- racadm — command-line control used by the stagger spin-up script
- SSH directly into iDRAC for management
**Express does NOT include:**
- Virtual Console — you cannot see the server's screen in a browser window.
A **physical monitor and keyboard are required** for all setup steps until
Proxmox is installed and SSH is working. After that you will never need the
monitor again.
- Virtual Media — you cannot mount an ISO file over the network. A **physical
USB drive** is required for the Proxmox installer.
Nothing in this guide requires Enterprise. The fan control script and stagger
spin-up both use IPMI over LAN and racadm, which are both available on Express.
**Do you need to pay for anything?** No. iDRAC Express is already there and
covers everything we need. The only other "subscription" people sometimes ask
about is the **Proxmox subscription** (about €100/year for enterprise update
servers and commercial support). You do not need that either — we configure the
free community repositories in Phase 6. The two are completely unrelated.
**Summary: pay nothing, skip nothing. Keep a monitor plugged in through Phase 6.**
---
### What you need on the table before starting
- The server plugged into power and a network switch/router
- A **separate network cable** for the iDRAC port (the small RJ45 on the back
labeled "iDRAC" — it is separate from the four main NIC ports)
- A **monitor and USB keyboard** plugged into the server — required through
Phase 5 (the Proxmox installer). iDRAC Express does not include a remote
KVM, so you cannot see the server's screen from another computer. After
Proxmox is installed and SSH is working (end of Phase 6) the monitor
can be unplugged permanently.
- A USB drive (8 GB or larger) — needed later for the Proxmox installer
- Another computer or phone to read this guide
---
## Phase 1 — Firmware Updates
**Why first?** Old firmware has bugs. BIOS from 2016 may not correctly expose
the VT-d settings that GPU passthrough requires. The H730 firmware has had fixes
for JBOD/non-RAID mode. Updating now prevents chasing ghosts later.
**Time required:** 30–90 minutes. The server will reboot several times on its own.
---
### Step 1.1 — Power on and watch the screen
Press the power button on the front of the server.
The screen will show a **Dell splash screen** with the PowerEdge logo. At the
bottom of the screen you will see a line of options, something like:
```
F2 = System Setup F10 = Lifecycle Controller F11 = Boot Manager F12 = PXE Boot
```
These options are only available for about **5–8 seconds** before the server
continues booting. If you miss the window the server will try to boot an OS
(and fail if nothing is installed yet). Just power it off and back on and try again.
---
### Step 1.2 — Enter Lifecycle Controller
Press **F10** when you see the splash screen.
The screen will go blank for 10–30 seconds, then show the **Lifecycle Controller**
loading screen. This is a mini operating system built into the server's firmware.
It has nothing to do with your server's main OS. Wait for it to finish loading —
it can take up to 90 seconds the first time.
When it finishes you will see the **Lifecycle Controller Home** screen with a menu
on the left side:
```
Home
Firmware Update
OS Deployment
RAID Configuration
Hardware Configuration
Settings
```
If it asks you to complete initial setup (language, network), do that first —
use Tab and arrow keys to navigate, Enter to confirm.
---
### Step 1.3 — Go to Firmware Update
Click or arrow-key to **Firmware Update** on the left menu, then press Enter.
You will see:
```
Launch Firmware Update
```
Press Enter on that.
---
### Step 1.4 — Choose your update method
You will be asked how you want to get the firmware. You have two options:
#### Option A — Server has internet access (easiest)
If your network cable is plugged in and the server can reach the internet:
1. Select **HTTPS** (it will say something like "downloads.dell.com")
2. Press **Next**
3. Lifecycle Controller will connect to Dell's servers and **automatically scan
your server** to find what firmware versions are installed vs. what is available
4. This takes 2–5 minutes — the screen will show a progress bar
5. Skip to Step 1.5
#### Option B — No internet on the server
You need to download the firmware files on another computer first.
1. On your other computer, open a browser and go to:
**https://www.dell.com/support/home**
2. You need your server's **Service Tag** — it is a 7-character code. Find it:
- On the **front of the server**: there is a small pull-out plastic tab (like a
credit card slot) on the left side of the front panel. Pull it out — the
Service Tag is printed on it.
- Or: on a sticker on the **top** of the server chassis.
- Or: it was displayed briefly during POST on the splash screen.
3. On the Dell support page, click **"View products"** or type the Service Tag
into the search bar at the top. Select your server from the results.
4. Click **"Drivers & Downloads"**
5. You will see a filter panel. Under **"Category"**, select each of these one
at a time and download the latest version:
- **iDRAC** — look for "iDRAC8 firmware" — file will end in `.exe` or `.d9`
- **BIOS** — look for "BIOS" — file ends in `.exe`
- **RAID** — look for "PERC H730" — file ends in `.exe`
6. Copy all downloaded files onto a **FAT32-formatted USB drive**
7. Plug the USB drive into the server
8. In Lifecycle Controller: select **Local Drive (USB)** and press Next
9. Navigate to the USB drive and select the firmware files
---
### Step 1.5 — Review and apply updates
Lifecycle Controller shows a table of all components with columns like:
```
Component Current Version Available Version Select
--------- --------------- ----------------- ------
iDRAC8 2.40.40.40 2.85.85.85 [ ]
BIOS 1.2.10 2.14.0 [ ]
PERC H730 Mini 25.4.0.0018 25.5.9.0001 [ ]
```
1. Check the box next to **every component** that shows a newer Available Version
(you can usually click "Select All")
2. Click **Install and Reboot** (or **Apply**)
3. A warning will say the server will reboot — click **Yes**
The server will now:
- Apply each firmware update
- Reboot automatically between some updates
- Return to the Lifecycle Controller when finished
**Do not power off the server during this process.** Just wait. It can take
30–60 minutes if there are many updates. The screen will show progress.
---
### Step 1.6 — Confirm completion
When all updates are done, Lifecycle Controller returns to its home screen.
You may see a success summary screen first — click **OK** or **Finish**.
To verify everything worked:
1. In Lifecycle Controller, go to **Firmware Update** → **Launch Firmware Update**
again
2. Run the check one more time
3. The Available Version column should now match Current Version for everything
you updated (or show no newer version)
Press **Exit** (or **Finish**) to leave Lifecycle Controller. The server will
reboot normally.
---
## Phase 2 — iDRAC Setup
You need to do two things with iDRAC:
1. Give it a **static IP address** so it is always reachable at the same address
2. Enable **IPMI over LAN** so the fan control script can send commands to it
There are two ways to do this — through the server's BIOS-like setup (F2),
or through the iDRAC web UI once you find its current IP. We will use F2 first
to set a static IP, then use the web UI for the rest.
---
### Step 2.1 — Enter System Setup (F2)
Power on (or reboot) the server.
When the Dell splash screen appears, press **F2**.
The screen will go blank briefly, then show the **System Setup Main Menu**:
```
System Setup Main Menu
System BIOS
iDRAC Settings
Device Settings
Service Tag Settings
```
Use the **arrow keys** to highlight **iDRAC Settings** and press **Enter**.
---
### Step 2.2 — Set a static IP for iDRAC
Inside iDRAC Settings you will see another menu:
```
iDRAC Settings
Network
User Configuration
Smart Card
Update and Rollback
...
```
Arrow down to **Network** and press **Enter**.
The Network screen has several sections. You are looking for the **IPv4 Settings**
section. Use Tab or arrow keys to navigate to these fields:
| Field | What to set |
|-------|-------------|
| Enable NIC | **Enabled** |
| NIC Selection | **Dedicated** (this uses the dedicated iDRAC port on the back) |
| Enable IPv4 | **Enabled** |
| Enable DHCP | **Disabled** ← change this |
| Static IP Address | e.g. `192.168.1.5` (pick an IP outside your router's DHCP range) |
| Static Gateway | your router's IP, e.g. `192.168.1.1` |
| Static Subnet Mask | `255.255.255.0` |
| DNS Server 1 | your router's IP, e.g. `192.168.1.1` |
> **What IP to use?** Log into your router and find its DHCP range — for example
> if DHCP hands out `192.168.1.100` to `192.168.1.200`, pick something outside
> that range like `192.168.1.5`. Write this IP down — you will use it often.
When done, press **Back** or navigate to the bottom and click **Apply**, then **OK**.
---
### Step 2.3 — Enable IPMI over LAN
Still inside iDRAC Settings → Network.
Scroll down past the IPv4 section until you see **IPMI Settings**.
Find:
```
Enable IPMI over LAN: [ Disabled ]
```
Change this to **Enabled**.
This is the setting that allows `ipmitool` commands (used by the fan control
script) to reach iDRAC over the network.
Click **Apply** at the bottom, then **OK** when asked to confirm.
---
### Step 2.4 — Change the default password
Go back to the iDRAC Settings main menu (press **Back**).
Arrow down to **User Configuration** and press **Enter**.
You will see a list of user slots. **User 1** is the built-in `root` account.
Select it (press Enter).
> **Why change it?** The factory default password is `calvin`. This is printed
> in Dell's public documentation and is widely known. Any device on your network
> could log into your iDRAC if you leave it as-is.
Change:
- **User Name:** leave as `root`
- **Change Password:** set to **Enabled**
- **Password:** enter a strong password
- **Confirm Password:** enter it again
Click **Apply**, then **OK**.
Press **Finish** to exit iDRAC Settings and return to the System Setup Main Menu.
**Do not reboot yet** — you still have BIOS settings to configure in Phase 3.
---
### Step 2.5 — Verify iDRAC from your browser (do after Phase 3)
After you finish Phase 3 and let the server reboot, test iDRAC from your other
computer:
1. Make sure the **dedicated iDRAC network port** (small RJ45 labeled "iDRAC" on
the server's back panel) is plugged into your switch or router
2. Open a browser on your other computer
3. Go to: `https://192.168.1.5` (whatever static IP you set in Step 2.2)
4. You will see a **certificate warning** — this is normal, iDRAC uses a
self-signed certificate. Click **Advanced** → **Proceed anyway** (wording
varies by browser)
5. The iDRAC8 login page appears — a dark Dell-branded page
6. Log in: Username `root`, Password (the one you set in Step 2.4)
You should see the iDRAC dashboard showing system health, temperatures, and fans.
This is your remote window into the server — you can now manage it from your desk.
---
## Phase 3 — BIOS Settings
These settings enable the CPU features that allow devices (GPUs, USB controllers)
to be handed directly to virtual machines. **Every setting below is required.**
Skipping VT-d is the single most common reason GPU passthrough silently fails.
You are still in System Setup from Phase 2 (or press F2 again on reboot).
---
### Step 3.1 — System BIOS → Processor Settings
From the System Setup Main Menu, arrow to **System BIOS** and press Enter.
You will see the System BIOS Settings menu:
```
System BIOS Settings
System Information
Memory Settings
Processor Settings ← go here
SATA Settings
Boot Settings
Integrated Devices
Serial Communication
System Profile Settings
Power Management
Security
Miscellaneous Settings
```
Arrow to **Processor Settings** and press **Enter**.
Find and set these two options:
---
**Virtualization Technology**
This enables the CPU to run virtual machines efficiently. Without it Proxmox
still works but performance is worse.
```
Virtualization Technology: [ Disabled ]
↓ change to
Virtualization Technology: [ Enabled ]
```
Use the arrow keys or spacebar to toggle the value.
---
**VT for Direct I/O** (also shown as "Virtualization Technology for Directed I/O")
This is the critical one. It enables **IOMMU** — the hardware feature that lets
the CPU safely hand a real PCIe device (like your GPU) directly to a virtual
machine. Without this, passthrough is impossible.
```
VT for Direct I/O: [ Disabled ]
↓ change to
VT for Direct I/O: [ Enabled ]
```
Click **Back** to return to System BIOS Settings.
---
### Step 3.2 — System BIOS → Integrated Devices
From the System BIOS Settings menu, arrow to **Integrated Devices** and press Enter.
Find:
**SR-IOV Global Enable**
SR-IOV is a PCIe feature that allows a single physical device to appear as
multiple devices. Even though the Quadro P2200 does not use SR-IOV, enabling
this globally avoids a class of IOMMU grouping problems.
```
SR-IOV Global Enable: [ Disabled ]
↓ change to
SR-IOV Global Enable: [ Enabled ]
```
Click **Back**.
---
### Step 3.3 — System BIOS → Boot Settings
From System BIOS Settings, arrow to **Boot Settings** and press Enter.
You will see:
```
Boot Settings
Boot Mode: [ BIOS ] ← must change this
Boot Sequence
...
```
**Boot Mode** — change from BIOS to UEFI:
```
Boot Mode: [ BIOS ]
↓ change to
Boot Mode: [ UEFI ]
```
> **Why UEFI?** Proxmox uses a tool called `proxmox-boot-tool` to manage the
> kernel boot parameters (including the IOMMU settings the GPU passthrough script
> writes). This tool only works with UEFI boot. If the server is in Legacy/BIOS
> mode the passthrough script will fail and you will have to reinstall.
A warning may appear saying the boot sequence will be cleared — click **Yes** or
**OK**. That is fine since we haven't installed anything yet.
Click **Back**.
---
### Step 3.4 — System BIOS → Power Management
From System BIOS Settings, arrow to **Power Management** and press Enter.
Find:
**Hard Disk Drive Sequencing** — this is the built-in stagger that fires at
power-on before the OS loads. It staggers when each drive spins up so they do
not all surge at the same time and overload the PSU.
```
Hard Disk Drive Sequencing: [ Disabled ]
↓ change to
Hard Disk Drive Sequencing: [ Enabled ]
```
Also check:
**C States** — these are CPU power-saving sleep states. They can add latency
to VM workloads. Optional but worth setting for a server:
```
C States: [ Enabled ]
↓ change to
C States: [ Disabled ]
```
Click **Back**.
---
### Step 3.5 — Apply and exit
From System BIOS Settings, click **Finish** (at the bottom of the menu, you may
need to scroll).
A dialog will appear:
```
Confirm changes and exit?
[ Yes ] [ No ]
```
Click **Yes**.
The server will reboot.
---
### Step 3.6 — Confirm the settings took
After the reboot, press **F2** again to re-enter System Setup and spot-check:
- **Processor Settings:** VT and VT for Direct I/O should both show **Enabled**
- **Boot Settings:** Boot Mode should show **UEFI**
If either shows Disabled, set it again and apply. Some BIOS versions have a bug
where the first Apply does not persist — applying a second time fixes it.
---
### Phase 3 complete — what you have now
| Setting | Why it matters |
|---------|---------------|
| VT-x (Virtualization Technology) | Lets the CPU run VMs at near-native speed |
| VT-d (Direct I/O) | Enables IOMMU — the foundation of all device passthrough |
| SR-IOV | Prevents IOMMU group problems with PCIe devices |
| UEFI boot mode | Required for Proxmox's boot management and EFI GPU passthrough |
| HDD sequencing | Staggers drive spin-up at power-on to protect the PSU |
**Next:** Phase 4 — Configure the H730 RAID controller to mirror the two
2.5" OS drives, then leave the 3.5" drives alone for Proxmox to manage.
That walkthrough is in `docs/walkthrough-phases-4-6.md` (coming next).
docs/walkthrough-phases-10-12.md
-1297
View File
File diff suppressed because it is too large Load Diff
docs/walkthrough-phases-4-6.md
-611
View File
@@ -1,611 +0,0 @@
# Phases 4–6: RAID Config, Proxmox Install, Post-Install
### Dell R730xd — Button-by-button walkthrough
---
## Phase 4 — H730 RAID Configuration
**Goal:** Mirror the two 2.5" rear drives together so a single drive failure
cannot take down Proxmox. Leave every 3.5" drive untouched — the `perc-nonraid.sh`
script handles those later from inside Proxmox.
**Why mirror the OS drives here instead of using ZFS?** The H730 presents the
RAID1 mirror as a single virtual disk to the OS. Proxmox installs onto it and
never needs to think about the fact that two physical drives are underneath.
This is the simplest and most reliable approach for an OS volume.
---
### Step 4.1 — Enter Lifecycle Controller
Power on (or reboot) the server and press **F10** at the Dell splash screen.
Wait for Lifecycle Controller to load. You will see the main menu on the left:
```
Home
Firmware Update
OS Deployment
RAID Configuration ← go here
Hardware Configuration
Settings
```
Arrow to **RAID Configuration** and press **Enter**.
---
### Step 4.2 — Open the RAID configuration wizard
You will see:
```
RAID Configuration
View Current Configuration
Create New VD
...
```
First, click **View Current Configuration** to see what is already there.
You are looking at the physical drives the H730 can see. Drives are listed under
their controller — it will say something like:
```
PERC H730 Mini (Slot 0)
Physical Disks:
Port 0: SEAGATE ST4000NM0023 4.0 TB State: Unconfigured Good
Port 1: SEAGATE ST4000NM0023 4.0 TB State: Unconfigured Good
...
Port 8: TOSHIBA MK1401GRRB 146 GB State: Unconfigured Good ← 2.5"
Port 9: TOSHIBA MK1401GRRB 146 GB State: Unconfigured Good ← 2.5"
```
> **Identifying the 2.5" OS drives:** They will be noticeably smaller than the
> 3.5" drives — likely 146 GB, 300 GB, 600 GB (SAS) or 120–480 GB (SSD).
> The 3.5" drives will be 1 TB, 2 TB, 4 TB, or larger. Size is the giveaway.
> **If drives show "Foreign Configuration":** They were previously part of a
> RAID array. See Step 4.3a before continuing.
> **If the 2.5" drives appear under a different controller** (e.g., "PERC H330
> Mini"): The R730xd sometimes puts the rear bays on a separate mini controller.
> That is fine — just create the RAID1 on whichever controller owns those drives.
Press **Back** to return to the RAID Configuration menu.
---
### Step 4.3a — Clear foreign configurations (only if needed)
If any drives showed "Foreign Configuration" in Step 4.2, you need to clear
them before you can create a new virtual disk.
In the RAID Configuration menu, look for:
```
Clear Foreign Configuration
```
Select it, choose **All Foreign Configurations**, click **Apply**.
A warning will appear saying all data on those drives will be lost — click **Yes**.
(We are building a new system so there is nothing to keep.)
After clearing, go back to **View Current Configuration** and confirm the drives
now show **Unconfigured Good**.
---
### Step 4.4 — Create the RAID 1 virtual disk
In the RAID Configuration menu, select **Create New VD** and press **Enter**.
The wizard walks you through four screens:
---
**Screen 1 — Select RAID Level**
```
Select RAID Level:
RAID 0 (no redundancy, faster)
RAID 1 (mirror, one drive can fail) ← select this
RAID 5 (requires 3+ drives)
RAID 6 (requires 4+ drives)
RAID 10 (requires 4+ drives)
```
Select **RAID 1** and click **Next**.
---
**Screen 2 — Select Physical Disks**
A list of all available physical disks appears. You need to select **only the
two 2.5" drives**. Do not select any 3.5" drives.
Click the checkbox next to each of the two 2.5" drives (identified by their
smaller size). Leave all 3.5" drives unchecked.
Click **Next**.
---
**Screen 3 — Virtual Disk Attributes**
```
Virtual Disk Name: OS-Mirror
Virtual Disk Size: (auto-filled — leave it)
Strip Element Size: 64KB (leave default)
Read Policy: Adaptive Read Ahead (leave default)
Write Policy: Write Back (leave default)
Disk Cache Policy: Enabled (leave default)
```
The only thing you need to change is the name — type `OS-Mirror` so it is easy
to identify. Everything else can stay at defaults.
Click **Next**.
---
**Screen 4 — Confirm**
Review the summary. It should show:
- RAID Level: 1
- Physical Disks: 2 (your 2.5" drives)
- Virtual Disk Size: whatever the smaller of the two drives is
Click **Finish**.
---
### Step 4.5 — Initialize the virtual disk
After creating the VD, Lifecycle Controller will ask about initialization:
```
Initialize virtual disk?
Fast Initialize ← select this
Full Initialize (writes zeros to every sector — takes hours, not needed)
Skip
```
Select **Fast Initialize** and click **OK**.
Fast initialization takes 30–60 seconds. A progress bar will appear. Wait for it.
---
### Step 4.6 — Verify and exit
When initialization finishes, go back to **View Current Configuration**.
You should now see:
```
PERC H730 Mini
Virtual Disks:
VD 0: OS-Mirror RAID 1 ~146 GB State: Optimal
Physical Disks:
Port 0–7: 3.5" drives State: Unconfigured Good
Port 8–9: 2.5" drives State: Online (member of VD 0)
```
The 3.5" drives should all show **Unconfigured Good**. That is exactly what you
want. Leave them that way.
Click **Back** repeatedly until you reach the Lifecycle Controller home screen,
then click **Exit**. The server will reboot.
---
## Phase 5 — Install Proxmox VE 9.1
### Step 5.1 — Download the Proxmox ISO
On your other computer, open a browser and go to:
```
https://www.proxmox.com/en/downloads
```
Click **Proxmox Virtual Environment** → find the latest **Proxmox VE 9.x ISO
Installer** and click **Download**.
The file will be named something like `proxmox-ve_9.1-1.iso` and is about 1 GB.
---
### Step 5.2 — Write the ISO to a USB drive
You need to write the ISO as a disk image (not copy the file). Use one of these
tools depending on your computer:
#### Windows — Rufus (free, no install needed)
1. Download Rufus from: **https://rufus.ie** (click the first .exe link)
2. Plug in your USB drive (8 GB or larger — all data on it will be erased)
3. Open Rufus
4. **Device:** select your USB drive from the dropdown
5. **Boot selection:** click **SELECT** and choose the Proxmox ISO file
6. **Partition scheme:** select **GPT**
7. **Target system:** select **UEFI (non CSM)**
8. Click **START**
9. A dialog appears asking about ISO mode vs DD mode — select **Write in DD Image
mode** and click **OK**
10. Click **OK** again when warned that the USB will be wiped
11. Wait for it to finish (1–3 minutes), then click **CLOSE**
#### Mac — Balena Etcher (free)
1. Download from: **https://etcher.balena.io** — click **Download for macOS**
2. Open Etcher
3. Click **Flash from file** → select the Proxmox ISO
4. Click **Select target** → select your USB drive
5. Click **Flash** — enter your Mac password if prompted
6. Wait for it to finish
#### Linux — terminal
```bash
# Find your USB drive device name (look for your drive size)
lsblk
# Write the ISO (replace sdX with your actual USB device, e.g. sdb — NOT sdb1)
dd if=proxmox-ve_9.1-1.iso of=/dev/sdX bs=1M status=progress conv=fsync
sync
```
---
### Step 5.3 — Boot the server from USB
1. Plug the USB drive into one of the USB ports on the front or back of the server
2. Power on (or reboot) the server
3. At the Dell splash screen, press **F11**
The **Boot Manager** screen appears:
```
Boot Manager
BIOS Boot Menu
UEFI Boot Menu ← go here
One-shot BIOS Boot Menu
...
```
Arrow to **UEFI Boot Menu** and press **Enter**.
You will see a list of bootable devices. Look for your USB drive — it will be
listed as something like:
```
UEFI: SanDisk Ultra USB 3.0, Partition 1
```
Arrow to it and press **Enter**.
> **If the USB does not appear in the UEFI Boot Menu:** Make sure Rufus wrote in
> DD mode and GPT was selected (Step 5.2). Legacy/MBR USB drives will not appear
> in the UEFI menu.
---
### Step 5.4 — Proxmox installer boot menu
The server boots from the USB and shows the Proxmox boot menu — white text on
a blue/dark background:
```
Proxmox VE Installer
Install Proxmox VE (Graphical) ← select this
Install Proxmox VE (Terminal UI)
Advanced Options
...
```
Arrow to **Install Proxmox VE (Graphical)** and press **Enter**.
The graphical installer loads. This takes about 30–60 seconds.
---
### Step 5.5 — End User License Agreement
The EULA screen appears. Read it or don't — click **I agree** at the bottom right.
---
### Step 5.6 — Target disk selection
This is the most important screen. You are choosing where Proxmox installs.
You will see a dropdown labeled **Target Harddisk**. Click it.
The list shows all visible storage. You are looking for the RAID1 virtual disk
you created in Phase 4. It will appear as a single disk — something like:
```
/dev/sda (146.00 GB) ← this is the OS-Mirror RAID1 VD
/dev/sdb (4.00 TB)
/dev/sdc (4.00 TB)
...
```
Select the small one — your OS-Mirror virtual disk (146 GB or whatever size your
2.5" drives are).
> **Do not select a 3.5" drive.** Those are your VM data drives.
**Filesystem:** Click the **Options** button next to the disk selector.
```
Filesystem: ext4 ← leave this as ext4
```
Leave it as `ext4`. The H730 RAID1 already gives you drive redundancy. Adding
ZFS here would be redundant overhead with no benefit.
Click **Next**.
---
### Step 5.7 — Location and timezone
```
Country: [type your country, e.g. United States]
Time Zone: [auto-filled based on country, e.g. America/New_York]
Keyboard: [your keyboard layout, e.g. U.S. English]
```
Adjust if needed. Click **Next**.
---
### Step 5.8 — Password and email
```
Password: [choose a strong root password — write it down]
Confirm: [same password again]
Email: [any email address — used for system alerts]
```
> The root password is how you log into Proxmox. If you forget it, recovery is
> painful. Write it on a piece of paper and put it somewhere safe.
Click **Next**.
---
### Step 5.9 — Network configuration
```
Management Interface: [auto-selected — usually em1 or eno1, the first NIC]
Hostname (FQDN): pve.local
IP Address: 192.168.1.10 ← change to an IP on your network
Netmask: 255.255.255.0
Gateway: 192.168.1.1 ← your router's IP
DNS Server: 192.168.1.1 ← your router's IP (or 8.8.8.8)
```
**Choosing a static IP for Proxmox:**
- Log into your router and find its DHCP range
- Pick an IP outside that range (same subnet)
- Example: if DHCP is 192.168.1.100–200, use 192.168.1.10
- This IP is how you will reach the Proxmox web UI and SSH from now on
Click **Next**.
---
### Step 5.10 — Summary and install
The installer shows a summary of everything you chose. Confirm:
- Target disk: your small RAID1 virtual disk
- Filesystem: ext4
- Hostname and IP look correct
Click **Install**.
The installation takes 5–10 minutes. A progress bar shows the steps:
- Formatting disk
- Copying files
- Setting up bootloader
- Configuring system
When it finishes you will see:
```
Installation successful!
Remove the installation medium and press Enter to reboot.
```
**Pull the USB drive out**, then press **Enter**.
---
### Step 5.11 — First login to the Proxmox web UI
The server reboots. After about 60 seconds it shows a text console with
Proxmox's login prompt and this message:
```
Welcome to the Proxmox Virtual Environment.
Please use your web browser to configure this server -
connect to: https://192.168.1.10:8006/
```
On your other computer, open a browser and go to:
```
https://192.168.1.10:8006
```
(Use the IP you set in Step 5.9.)
**Certificate warning:** Your browser will show a security warning because
Proxmox uses a self-signed SSL certificate. This is expected and safe on
your local network.
- Chrome/Edge: click **Advanced** → **Proceed to 192.168.1.10 (unsafe)**
- Firefox: click **Advanced** → **Accept the Risk and Continue**
The Proxmox login page appears — a dark interface with two fields:
```
User name: root
Password: [the password you set in Step 5.8]
Realm: Linux PAM standard authentication ← leave this as-is
```
Click **Login**.
**Subscription nag:** A popup immediately appears saying "No valid subscription".
Click **OK** to dismiss it. This appears every login until you fix the repos in
Phase 6. It does not affect anything.
You are now looking at the Proxmox web interface — the main dashboard.
---
## Phase 6 — Proxmox Post-Install
From this point you will use **SSH** instead of the web console for most tasks.
SSH is faster and lets you paste commands directly.
---
### Step 6.1 — Connect via SSH
On your other computer, open a terminal (Mac/Linux) or PuTTY/Windows Terminal
(Windows) and run:
```bash
ssh root@192.168.1.10
```
Type `yes` when asked to confirm the host fingerprint. Enter your root password.
You will see the Proxmox shell prompt:
```
root@pve:~#
```
All the commands in the rest of this guide are typed here.
---
### Step 6.2 — Fix the package repositories
Proxmox installs with "enterprise" apt repositories configured. These require
a paid subscription and will fail with an authentication error when you try to
update. We switch them to the free community repositories.
Run these commands one at a time:
```bash
# Disable the enterprise repo (comment it out)
echo "# disabled - no subscription" > /etc/apt/sources.list.d/pve-enterprise.list
# Disable the enterprise Ceph repo
echo "# disabled - no subscription" > /etc/apt/sources.list.d/ceph.list
# Add the no-subscription community repo
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
> /etc/apt/sources.list.d/pve-no-subscription.list
```
Now update and upgrade all packages:
```bash
apt update && apt dist-upgrade -y
```
This will download and install updates. It may take 5–15 minutes depending on
your internet speed. Answer **Y** if prompted about any config file changes
(just press Enter to accept the default).
When it finishes, reboot:
```bash
reboot
```
Wait about 60 seconds, then SSH back in:
```bash
ssh root@192.168.1.10
```
---
### Step 6.3 — Install the tools the scripts need
```bash
apt install -y ipmitool lsscsi ledmon hdparm git curl
```
What each tool does:
| Tool | Used for |
|------|---------|
| ipmitool | Fan control — sends IPMI commands to iDRAC |
| lsscsi | build-bay-map.sh — lists SCSI drives with details |
| ledmon | build-bay-map.sh — blinks drive bay LEDs for identification |
| hdparm | stagger-spinup.sh — checks drive power state, wakes drives |
| git | Cloning this repo onto the host |
| curl | Downloading packages and health checks |
---
### Step 6.4 — Clone this repo onto the Proxmox host
```bash
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
chmod +x /opt/local_proxmox/scripts/*.sh
cd /opt/local_proxmox
```
All scripts and configs are now at `/opt/local_proxmox/`.
---
### Step 6.5 — Dismiss the subscription warning permanently (optional)
If the "No valid subscription" popup annoys you every time you log into the
web UI, you can remove it with one command. This does not affect functionality:
```bash
sed -i.bak "s/data.status !== 'Active'/false/g" \
/usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
systemctl restart pveproxy
```
After running this, refresh the Proxmox web UI — the popup will be gone.
---
### Phase 6 complete — where you are now
| What is done | Status |
|---|---|
| Proxmox VE 9.1 installed on RAID1 OS mirror | ✓ |
| Free community repos configured | ✓ |
| All packages up to date | ✓ |
| Tools installed | ✓ |
| Repo cloned to /opt/local_proxmox | ✓ |
**Next:** Phase 7 — Run `perc-nonraid.sh` to convert the 3.5" drives to
non-RAID/per-disk mode, then map the physical bays. That walkthrough is in
`docs/walkthrough-phases-7-9.md`.
docs/walkthrough-phases-7-9.md
-525
View File
@@ -1,525 +0,0 @@
# Phases 7–9: Drive Non-RAID, Bay Mapping, GPU Passthrough
### Dell R730xd — Button-by-button walkthrough
---
## Phase 7 — Convert 3.5" Drives to Non-RAID Mode
**What this does:** Right now the H730 sees your 3.5" drives as "Unconfigured
Good" — they exist but the controller is not doing anything with them. We need
to put them into **Non-RAID mode**, which tells the H730 to present each drive
directly to Linux as an individual block device, with SMART health data intact.
Without this step, Linux cannot see the drives at all — the H730 hides them.
**The 2.5" OS drives are not affected** — they are a RAID1 virtual disk and the
script only targets unconfigured physical drives.
---
### Step 7.1 — Download and install perccli
`perccli` is Dell's command-line tool for managing the H730. The `perc-nonraid.sh`
script uses it to talk to the controller.
SSH into the Proxmox host if you are not already connected:
```bash
ssh root@192.168.1.10
```
Download and install perccli on the Proxmox host. Dell only distributes an
RPM, so we convert it with `alien`:
```bash
# Download (referer header required — Dell blocks plain curl)
curl -L \
--referer "https://www.dell.com/" \
--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
"https://dl.dell.com/FOLDER03559396M/1/perccli-1.17.10-1.noarch.rpm" \
-o /tmp/perccli.rpm
# Convert RPM → deb and install
apt install -y alien
alien --to-deb /tmp/perccli.rpm
dpkg -i /tmp/perccli_*.deb
# Binary lands in /opt/MegaRAID/perccli/ — symlink it into PATH
ln -s /opt/MegaRAID/perccli/perccli64 /usr/local/bin/perccli
```
Verify it installed:
```bash
perccli show
```
You should see output starting with something like:
```
CLI Version = 007.1907.0000.0000
Operating system = Linux5.x
Controller = 0
Status = Success
...
```
If you see `command not found`, try `perccli64` instead — some versions install
under that name.
---
### Step 7.2 — Run the non-RAID conversion script
```bash
bash /opt/local_proxmox/scripts/perc-nonraid.sh
```
The script first prints a summary of your controller and all drives:
```
=== Controller overview ===
...
=== Drives on controller 0 ===
-------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-------------------------------------------------------------
8:0 7 UGood - 3.637 TB SATA HDD N N 512B WD... U
8:1 8 UGood - 3.637 TB SATA HDD N N 512B WD... U
...
8:9 16 Onln 0 136.73 GB SATA SSD N N 512B TOSHIBA U ← OS drive (in VD)
8:10 17 Onln 0 136.73 GB SATA SSD N N 512B TOSHIBA U ← OS drive (in VD)
```
The columns that matter:
- **EID:Slt** — Enclosure ID and slot number (physical location in the backplane)
- **State** — `UGood` means unconfigured and ready. `Onln` means it is part of
a virtual disk (your OS mirror). The script only touches `UGood` drives.
- **Size** — confirms which are 3.5" data drives vs the small OS drives
The script then asks:
```
Proceed with converting all non-OS drives to Non-RAID mode? [y/N]
```
Type `y` and press Enter.
The script works through each drive. For each one you will see:
```
Slot 0: setting to Good... Status = Success
Slot 0: setting to Non-RAID... Status = Success
Slot 1: setting to Good... Status = Success
Slot 1: setting to Non-RAID... Status = Success
...
```
When all drives are done the script prints the final state — drives that were
`UGood` should now show `JBOD` (which is Dell's term for non-RAID/per-disk mode):
```
=== Final drive state ===
EID:Slt State
8:0 JBOD ←
8:1 JBOD ←
8:2 JBOD ←
...
8:9 Onln ← OS drive, untouched
8:10 Onln ← OS drive, untouched
```
---
### Step 7.3 — Reboot
```bash
reboot
```
Wait about 60 seconds, then SSH back in:
```bash
ssh root@192.168.1.10
```
### Step 7.4 — Verify drives are visible
```bash
lsblk -d -o NAME,SIZE,TYPE,ROTA
```
You should see your drives listed. Something like:
```
NAME SIZE TYPE ROTA
sda 136.7G disk 0 ← OS-Mirror RAID1 virtual disk (the 2.5" drives)
sdb 3.6T disk 1 ← 3.5" data drive
sdc 3.6T disk 1
sdd 3.6T disk 1
...
```
If you see only `sda` and nothing else, the non-RAID conversion did not take
effect — reboot again and re-run `lsblk`. If still missing, re-run
`perc-nonraid.sh` and check the output for any lines that say `Failed`.
---
## Phase 8 — Map Physical Bays to Drives
**The problem:** Linux names drives `sdb`, `sdc`, `sdd` etc. based on the order
it finds them at boot — that order can change. What you need is the **stable
hardware ID** (`/dev/disk/by-id/...`) for each drive, tied to its physical bay
number, so you can reliably assign "bay 3" to a specific VM forever.
This phase generates that map.
---
### Step 8.1 — Run the bay mapping script
```bash
bash /opt/local_proxmox/scripts/build-bay-map.sh
```
Output looks like this (truncated example):
```
=== Drive inventory ===
DEVICE SIZE MODEL SERIAL BY-ID PATH
------ ---- ----- ------ ---------
/dev/sdb 3.6T WDC_WD4000FYYZ WD-XXXXXXXXXXXX /dev/disk/by-id/scsi-35000cca23b7d4eb8
/dev/sdc 3.6T WDC_WD4000FYYZ WD-XXXXXXXXXXXX /dev/disk/by-id/scsi-35000cca23b5e1234
/dev/sdd 3.6T ST4000NM0023 Z1Z2XXXXXX /dev/disk/by-id/scsi-35000c500a0000001
...
=== Bay identification via LED blink ===
To confirm which physical bay a device is in, blink its LED:
ledctl locate=/dev/sdX # LED on
ledctl locate_off=/dev/sdX # LED off
```
The script gives you the by-id path for each device. Now you need to figure out
**which physical bay each device is in**.
---
### Step 8.2 — Walk the bays with LED blink
This is the physical part. You need to be at the server (or have someone there).
For each drive, run the blink command, walk to the server, see which bay's amber
LED is lit, note the bay number, then turn it off:
```bash
# Blink sdb
ledctl locate=/dev/sdb
# Walk to the server, find the lit bay — write down: sdb = bay X
ledctl locate_off=/dev/sdb
# Blink sdc
ledctl locate=/dev/sdc
# Walk to server, find the lit bay — write down: sdc = bay X
ledctl locate_off=/dev/sdc
# Repeat for each drive
```
> **Bay numbering on the R730xd:** Bays are numbered left to right, top to
> bottom when facing the front of the server. Bay 1 is top-left. The exact
> labeling depends on your bezel — some models label them 0–11, others 1–12.
> Use whatever number is printed or silk-screened on the chassis next to the bay.
> **If the LED does not blink:** The `ledmon` daemon must be running. Start it:
> ```bash
> systemctl start ledmon
> ledctl locate=/dev/sdb
> ```
---
### Step 8.3 — Fill in the hardware layout document
Open the layout document:
```bash
nano /opt/local_proxmox/docs/hardware-layout.md
```
Fill in the table using what you noted in Step 8.2. It looks like this — fill
in the `by-id` column from the script output and the bay number from the LED walk:
```
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) |
|-----|-------------|------|---------------------------------------------|
| 1 | 100 | 4TB | scsi-35000cca23b7d4eb8 |
| 2 | 100 | 4TB | scsi-35000cca23b5e1234 |
...
```
Save with **Ctrl+O**, Enter, then **Ctrl+X** to exit nano.
---
### Step 8.4 — Fill in the VM config placeholders
Now open each VM config example and replace the `PLACEHOLDER_BAYx` entries with
the real by-id paths:
```bash
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
```
Change lines like:
```
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
```
To the real path:
```
scsi1: /dev/disk/by-id/scsi-35000cca23b7d4eb8,size=0
```
Do this for all 8 drives in VM 100, drives 9–10 in VM 101, and drives 11–12 in
VM 102. Save each file.
> **The full path starts with `/dev/disk/by-id/`** — but in the Proxmox VM
> config you write the full path. Double-check by running:
> ```bash
> ls -la /dev/disk/by-id/ | grep -v part
> ```
> You will see the symlinks and the drives they point to.
---
## Phase 9 — GPU Passthrough Setup
**What this does:** Tells the Linux kernel to stop trying to use the two Quadro
P2200s itself and instead hand them over to the VFIO driver, which holds them
ready to be claimed by a virtual machine.
Three things happen:
1. IOMMU is turned on in the kernel (the hardware feature VT-d enables)
2. The host's NVIDIA/nouveau GPU drivers are blocked from loading
3. The VFIO driver claims both GPUs at boot, before any other driver can
---
### Step 9.1 — Check your IOMMU groups first
Before running the script, verify that VT-d is actually active:
```bash
dmesg | grep -i iommu | head -10
```
You should see lines like:
```
DMAR: IOMMU enabled
Intel-IOMMU: enabled
```
If you see nothing or see "disabled", IOMMU is not active. Go back to Phase 3
and re-check that VT-d is set to **Enabled** in BIOS, then reboot and try again.
Also check that each GPU is in its own IOMMU group:
```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done | sort -V | grep -i nvidia
```
Example good output — each GPU is in a different group:
```
IOMMU Group 24 03:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
IOMMU Group 24 03:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
IOMMU Group 31 04:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
IOMMU Group 31 04:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
```
Each GPU (`03:00.0` and `04:00.0`) and its audio sibling (`03:00.1` and
`04:00.1`) are together in their own group — that is exactly what you want.
> **If both GPUs are in the same IOMMU group as other devices** (chipset, NICs,
> etc.), you may need to enable **ACS** (Access Control Services). This is rare
> on server hardware like the R730xd which has good IOMMU separation. If you
> hit this, ask before proceeding.
---
### Step 9.2 — Run the GPU passthrough script
```bash
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
```
The script first prints your NVIDIA devices:
```
=== Detected NVIDIA devices ===
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
03:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
04:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
GPU PCI IDs to bind to vfio-pci: 10de:1c35,10de:10f1
```
Then asks:
```
Proceed with configuring VFIO passthrough? [y/N]
```
Type `y` and press Enter.
The script runs through these steps — you will see each one printed:
```
--- Configuring kernel cmdline for IOMMU ---
Written: /etc/kernel/cmdline
Boot tool refreshed.
--- Blacklisting nouveau and nvidia on host ---
Written: /etc/modprobe.d/blacklist-gpu.conf
--- Binding GPU IDs to vfio-pci ---
Written: /etc/modprobe.d/vfio.conf
IDs: 10de:1c35,10de:10f1
--- Adding vfio modules to initramfs ---
Updated: /etc/initramfs-tools/modules
Initramfs updated.
```
At the end it prints the PCI addresses you need:
```
=== GPU PCI addresses for VM assignment ===
hostpci0: 0000:03:00,pcie=1 # VGA: NVIDIA GP106GL [Quadro P2200]
hostpci0: 0000:04:00,pcie=1 # VGA: NVIDIA GP106GL [Quadro P2200]
IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM.
```
**Write down or copy these addresses** — you need them in Phase 11 when
creating the VMs. In this example:
- GPU 1 is at `03:00` → goes in VM 100 (Frigate)
- GPU 2 is at `04:00` → goes in VM 101
---
### Step 9.3 — Update the VM configs with the GPU addresses
While you have the addresses, add them to the VM configs now:
```bash
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
```
Find the line:
```
hostpci0: 0000:XX:00,pcie=1,x-vga=1
```
Replace `XX:00` with your GPU 1 address, e.g.:
```
hostpci0: 0000:03:00,pcie=1,x-vga=1
```
Save, then do the same for VM 101:
```bash
nano /opt/local_proxmox/vm-configs/101.conf.example
# Change XX:00 to 04:00 (GPU 2)
```
---
### Step 9.4 — Reboot
```bash
reboot
```
Wait about 60 seconds, then SSH back in:
```bash
ssh root@192.168.1.10
```
---
### Step 9.5 — Verify passthrough is working
```bash
lspci -nnk | grep -A3 -i nvidia
```
For each GPU, look for the driver line. It **must** say `vfio-pci`:
```
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
Subsystem: ...
Kernel driver in use: vfio-pci ← correct
Kernel modules: nouveau
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
Subsystem: ...
Kernel driver in use: vfio-pci ← correct
Kernel modules: nouveau
```
**If it shows `nouveau` or `nvidia` instead of `vfio-pci`:** The blacklist did
not take effect. Run:
```bash
update-initramfs -u -k all
reboot
```
Then check again. If still wrong, verify the blacklist file exists:
```bash
cat /etc/modprobe.d/blacklist-gpu.conf
# Should show: blacklist nouveau, blacklist nvidia, etc.
cat /etc/modprobe.d/vfio.conf
# Should show: options vfio-pci ids=10de:1c35,10de:10f1
```
Also confirm IOMMU is in the kernel command line:
```bash
cat /etc/kernel/cmdline
# Should contain: intel_iommu=on iommu=pt
```
---
### Phase 9 complete — where you are now
| What is done | Status |
|---|---|
| 3.5" drives in non-RAID mode, visible to Linux | ✓ |
| Physical bay → by-id map documented | ✓ |
| VM configs updated with real drive paths and GPU addresses | ✓ |
| IOMMU active, both GPUs claimed by vfio-pci | ✓ |
**Next:** Phase 10 — Install the fan control service (stops iDRAC from running
fans at 100% because of the non-Dell GPUs) and the stagger spin-up service.
That walkthrough is in `docs/walkthrough-phases-10-12.md`.