docs: consolidate to single README, move fan control to Phase 7
This commit is contained in:
Claude
File diff suppressed because it is too large
Load Diff
@@ -1,485 +0,0 @@
|
|||||||
# Dell R730xd — Proxmox VE 9.1 Setup Guide
|
|
||||||
|
|
||||||
Complete walkthrough from bare metal to running VMs with GPU passthrough,
|
|
||||||
Coral USB, and per-VM disk assignments.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What You Need Before Starting
|
|
||||||
|
|
||||||
- USB drive (8GB+) for the Proxmox installer
|
|
||||||
- A second machine or phone to read this guide and SSH
|
|
||||||
- The iDRAC IP or physical access to a monitor + keyboard
|
|
||||||
- Internet connection on the server
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 1 — Firmware Updates (do this first)
|
|
||||||
|
|
||||||
Outdated firmware causes mysterious IOMMU failures and fan issues. Do this
|
|
||||||
before anything else.
|
|
||||||
|
|
||||||
### Option A: Lifecycle Controller (no extra tools needed)
|
|
||||||
|
|
||||||
1. Power on the server, press **F10** when you see the Dell splash screen
|
|
||||||
2. Lifecycle Controller → **Firmware Update** → Check for updates
|
|
||||||
3. Point it at downloads.dell.com or a local repo
|
|
||||||
4. Update **iDRAC**, **BIOS**, **PERC H730**, and any NIC/HBA firmware
|
|
||||||
5. Let it reboot as many times as it needs
|
|
||||||
|
|
||||||
### Option B: Dell System Update (DSU) from bootable USB
|
|
||||||
|
|
||||||
A faster alternative if Lifecycle Controller is slow or unavailable — Dell
|
|
||||||
provides a bootable ISO that auto-detects and applies all updates.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2 — iDRAC Setup
|
|
||||||
|
|
||||||
iDRAC is the out-of-band management interface. You need it configured for the
|
|
||||||
fan control script to work later.
|
|
||||||
|
|
||||||
1. Power on → press **F2** to enter System Setup → **iDRAC Settings**
|
|
||||||
2. **Network:**
|
|
||||||
- Set a static IP (easier than DHCP for a server)
|
|
||||||
- Note the IP — you'll use it for the fan control ipmitool commands
|
|
||||||
3. **User configuration:**
|
|
||||||
- Change the default `root` password
|
|
||||||
4. **IPMI over LAN:**
|
|
||||||
- iDRAC Settings → Network → IPMI Settings → **Enable IPMI over LAN: On**
|
|
||||||
- This is required for `ipmitool` fan control from the Proxmox host
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3 — BIOS Settings
|
|
||||||
|
|
||||||
Still in F2 System Setup. These settings are **required** for GPU and USB
|
|
||||||
passthrough to work.
|
|
||||||
|
|
||||||
| Menu path | Setting | Value |
|
|
||||||
|-----------|---------|-------|
|
|
||||||
| Processor Settings | Virtualization Technology | **Enabled** |
|
|
||||||
| Processor Settings | C States | **Disabled** |
|
|
||||||
| PCI Configuration | SR-IOV Global Enable | **Enabled** |
|
|
||||||
| Boot Settings → BIOS Boot Settings | Boot Mode | **UEFI** (not Legacy/BIOS) |
|
|
||||||
| System Profile Settings | System Profile | **Custom** |
|
|
||||||
| System Profile Settings | CPU Power Management | **Maximum Performance** |
|
|
||||||
|
|
||||||
> **VT for Direct I/O (VT-d):** Newer R730xd BIOS versions (2.19+) removed
|
|
||||||
> this toggle — VT-d is enabled by default. Don't worry if you can't find it.
|
|
||||||
> Verify it's active after Proxmox is installed: `dmesg | grep -i iommu`
|
|
||||||
> should show `DMAR: IOMMU enabled`. The kernel cmdline (`intel_iommu=on
|
|
||||||
> iommu=pt`) in Phase 9 is still required.
|
|
||||||
>
|
|
||||||
> **Hard Disk Drive Sequencing** was also removed in newer BIOS — it is no
|
|
||||||
> longer present and is not needed.
|
|
||||||
|
|
||||||
> **Boot Mode must be UEFI.** Proxmox's EFI boot tool (`proxmox-boot-tool`)
|
|
||||||
> only works with UEFI. Legacy BIOS mode breaks the GPU passthrough script.
|
|
||||||
|
|
||||||
**Apply and exit. The server will reboot.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4 — H730 RAID Configuration (pre-Proxmox)
|
|
||||||
|
|
||||||
The two 2.5" rear drives are your Proxmox OS drives. You want them mirrored
|
|
||||||
so a single drive failure doesn't take down the hypervisor.
|
|
||||||
|
|
||||||
The 3.5" drives will be converted to non-RAID later **from within Proxmox**
|
|
||||||
using the `perc-nonraid.sh` script — do not touch them here.
|
|
||||||
|
|
||||||
### Configure the 2.5" OS drives
|
|
||||||
|
|
||||||
1. Reboot → press **Ctrl+R** during POST to enter the H730 configuration
|
|
||||||
utility (or use Lifecycle Controller → RAID Configuration)
|
|
||||||
2. Select the controller
|
|
||||||
3. Find the two 2.5" rear drives
|
|
||||||
4. **Create new virtual disk:**
|
|
||||||
- RAID level: **RAID 1** (mirror)
|
|
||||||
- Select both 2.5" drives
|
|
||||||
- Strip size: 64KB (default)
|
|
||||||
- Name: `OS-Mirror` (optional)
|
|
||||||
- Initialize: **Fast Initialize**
|
|
||||||
5. Press **Ctrl+Alt+Delete** to reboot
|
|
||||||
|
|
||||||
The H730 now presents a single ~X GB RAID1 virtual disk to the OS. Proxmox
|
|
||||||
will install onto this and never know there are two physical drives behind it.
|
|
||||||
|
|
||||||
> The 3.5" drives will show as "Unconfigured Good" in the H730 — that is fine.
|
|
||||||
> Leave them alone. `perc-nonraid.sh` handles them after Proxmox is installed.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 5 — Install Proxmox VE 9.1
|
|
||||||
|
|
||||||
### Prepare the USB installer
|
|
||||||
|
|
||||||
On another machine, download the Proxmox VE ISO from proxmox.com and write it
|
|
||||||
to a USB drive:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Linux/macOS
|
|
||||||
dd if=proxmox-ve_*.iso of=/dev/sdX bs=1M status=progress conv=fsync
|
|
||||||
# or use Balena Etcher (Windows/Mac/Linux GUI)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Boot and install
|
|
||||||
|
|
||||||
1. Plug the USB into the R730xd
|
|
||||||
2. Power on → press **F11** for the one-time boot menu
|
|
||||||
3. Select the USB drive
|
|
||||||
4. At the Proxmox boot menu: **Install Proxmox VE (Graphical)**
|
|
||||||
5. **Target disk:** select the RAID1 virtual disk (`OS-Mirror`)
|
|
||||||
- Filesystem: **ext4** is simplest — the H730 RAID1 already gives you
|
|
||||||
redundancy, so ZFS mirroring here would be double-redundant overkill
|
|
||||||
6. **Location and timezone:** set to your region
|
|
||||||
7. **Password and email:** set a strong root password, enter an email
|
|
||||||
8. **Network configuration:**
|
|
||||||
- Management interface: the built-in NIC (usually `em1` or `eno1`)
|
|
||||||
- Hostname: e.g. `pve.local`
|
|
||||||
- IP: choose a static IP on your LAN (e.g. `192.168.1.10/24`)
|
|
||||||
- Gateway and DNS: your router's IP
|
|
||||||
9. Click **Install**
|
|
||||||
10. Remove USB when prompted, let it reboot
|
|
||||||
|
|
||||||
### First login
|
|
||||||
|
|
||||||
Open a browser on your LAN machine and go to:
|
|
||||||
|
|
||||||
```
|
|
||||||
https://192.168.1.10:8006
|
|
||||||
```
|
|
||||||
|
|
||||||
Accept the self-signed certificate warning. Login: `root` / (your password),
|
|
||||||
Realm: **Linux PAM**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 6 — Proxmox Post-Install (SSH)
|
|
||||||
|
|
||||||
SSH into the host from now on — it's faster than the web console for these
|
|
||||||
steps.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh root@192.168.1.10
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6a. Fix the apt repositories
|
|
||||||
|
|
||||||
Proxmox shows "no valid subscription" warnings when using the enterprise repo
|
|
||||||
without a license. Switch to the free repo:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Disable enterprise repos — PVE9 uses .sources (DEB822 format), not .list
|
|
||||||
echo "# disabled - no subscription" > /etc/apt/sources.list.d/pve-enterprise.sources
|
|
||||||
echo "# disabled - no subscription" > /etc/apt/sources.list.d/ceph.sources
|
|
||||||
# Also disable any legacy .list versions if present
|
|
||||||
echo "# disabled - no subscription" > /etc/apt/sources.list.d/pve-enterprise.list
|
|
||||||
echo "# disabled - no subscription" > /etc/apt/sources.list.d/ceph.list
|
|
||||||
|
|
||||||
# Add no-subscription repos (trixie = Proxmox 9 / Debian 13)
|
|
||||||
echo "deb http://download.proxmox.com/debian/pve trixie pve-no-subscription" \
|
|
||||||
> /etc/apt/sources.list.d/pve-no-subscription.list
|
|
||||||
echo "deb http://download.proxmox.com/debian/ceph-squid trixie no-subscription" \
|
|
||||||
> /etc/apt/sources.list.d/ceph-no-subscription.list
|
|
||||||
|
|
||||||
apt update && apt dist-upgrade -y
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6b. Install tools used by the scripts
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt install -y ipmitool lsscsi ledmon hdparm git
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6c. Clone this repo onto the host
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
|
|
||||||
cd /opt/local_proxmox
|
|
||||||
chmod +x scripts/*.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 7 — H730: Set 3.5" Drives to Non-RAID Mode
|
|
||||||
|
|
||||||
Now that Proxmox is running, convert the 3.5" drives to per-disk (non-RAID)
|
|
||||||
mode. The OS drives (RAID1 virtual disk) are **not affected** — the script only
|
|
||||||
targets unconfigured physical disks.
|
|
||||||
|
|
||||||
### Install perccli
|
|
||||||
|
|
||||||
`apt` does not carry perccli. Download the RPM from Dell and convert it:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Download (referer header required — Dell blocks plain curl)
|
|
||||||
curl -L \
|
|
||||||
--referer "https://www.dell.com/" \
|
|
||||||
--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
|
|
||||||
"https://dl.dell.com/FOLDER03559396M/1/perccli-1.17.10-1.noarch.rpm" \
|
|
||||||
-o /tmp/perccli.rpm
|
|
||||||
|
|
||||||
# Convert RPM → deb and install
|
|
||||||
apt install -y alien
|
|
||||||
alien --to-deb /tmp/perccli.rpm
|
|
||||||
dpkg -i /tmp/perccli_*.deb
|
|
||||||
|
|
||||||
# Binary lands in /opt/MegaRAID/perccli/ — symlink it into PATH
|
|
||||||
ln -s /opt/MegaRAID/perccli/perccli64 /usr/local/bin/perccli
|
|
||||||
|
|
||||||
# Verify
|
|
||||||
perccli show
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run the script
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bash /opt/local_proxmox/scripts/perc-nonraid.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Review the output, confirm when prompted. **Reboot after completion.**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
After rebooting, Proxmox will see the 3.5" drives as individual block devices
|
|
||||||
(`/dev/sdb`, `/dev/sdc`, etc.).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 8 — Map Physical Bays to Drives
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bash /opt/local_proxmox/scripts/build-bay-map.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Walk the bays with `ledctl` to confirm which physical slot is which device:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ledctl locate=/dev/sdb # LED blinks on the matching bay
|
|
||||||
ledctl locate_off=/dev/sdb
|
|
||||||
```
|
|
||||||
|
|
||||||
Fill in the `by-id` paths in `docs/hardware-layout.md`. You will need these
|
|
||||||
in Phase 11 when creating VMs.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 9 — GPU Passthrough Setup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
The script:
|
|
||||||
- Adds `intel_iommu=on iommu=pt` to the kernel command line
|
|
||||||
- Blacklists `nouveau`/`nvidia` on the host
|
|
||||||
- Binds both Quadro P2200s to `vfio-pci`
|
|
||||||
- Rebuilds initramfs
|
|
||||||
|
|
||||||
**Note the PCI addresses it prints at the end** — you'll need them in Phase 11.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
### Verify after reboot
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lspci -nnk | grep -A3 -i nvidia
|
|
||||||
```
|
|
||||||
|
|
||||||
Both GPUs should show `Kernel driver in use: vfio-pci`. If they still show
|
|
||||||
`nouveau`, check that `/etc/modprobe.d/blacklist-gpu.conf` exists and
|
|
||||||
`update-initramfs -u` was run.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 10 — Fan Control and Staggered Spin-Up
|
|
||||||
|
|
||||||
### Fan control (prevents jet-engine noise from non-Dell GPUs)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp /opt/local_proxmox/scripts/fan-control.sh /usr/local/sbin/fan-control.sh
|
|
||||||
cp /opt/local_proxmox/scripts/fan-control.service /etc/systemd/system/
|
|
||||||
systemctl daemon-reload
|
|
||||||
systemctl enable --now fan-control.service
|
|
||||||
systemctl status fan-control.service
|
|
||||||
```
|
|
||||||
|
|
||||||
Test it worked — the fans should audibly slow down within 30 seconds.
|
|
||||||
To check current speed: `ipmitool sdr type Fan`
|
|
||||||
|
|
||||||
### Staggered spin-up
|
|
||||||
|
|
||||||
> **BIOS 2.19+ note:** `BIOS.StorageSettings.HddSeq` was removed from the
|
|
||||||
> R730xd firmware. The iDRAC/BIOS layer (Layer 1) no longer works — skip it
|
|
||||||
> and go straight to the Linux service below.
|
|
||||||
|
|
||||||
**Linux service** (staggers drives in standby at OS boot):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp /opt/local_proxmox/scripts/stagger-spinup.sh /usr/local/sbin/stagger-spinup.sh
|
|
||||||
chmod +x /usr/local/sbin/stagger-spinup.sh
|
|
||||||
cp /opt/local_proxmox/scripts/stagger-spinup.service /etc/systemd/system/
|
|
||||||
systemctl daemon-reload
|
|
||||||
systemctl enable stagger-spinup.service
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 11 — Create the VMs
|
|
||||||
|
|
||||||
### Prepare the VM configs
|
|
||||||
|
|
||||||
Edit the three example configs and replace the placeholders:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /opt/local_proxmox/vm-configs
|
|
||||||
|
|
||||||
# Replace PLACEHOLDER_BAYx with real by-id paths from Phase 8
|
|
||||||
# Replace XX:00 with real PCI addresses from Phase 9
|
|
||||||
nano 100-frigate.conf.example
|
|
||||||
nano 101.conf.example
|
|
||||||
nano 102.conf.example
|
|
||||||
```
|
|
||||||
|
|
||||||
### Create OS disks for each VM
|
|
||||||
|
|
||||||
In the Proxmox web UI or via CLI, create the base OS disk for each VM:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Creates the disk slots — Proxmox generates the correct scsi0 line
|
|
||||||
qm create 100 --memory 16384 --cores 8 --name frigate --net0 virtio,bridge=vmbr0
|
|
||||||
qm create 101 --memory 16384 --cores 8 --name vm101 --net0 virtio,bridge=vmbr0
|
|
||||||
qm create 102 --memory 8192 --cores 4 --name vm102 --net0 virtio,bridge=vmbr0
|
|
||||||
```
|
|
||||||
|
|
||||||
Then merge your edited conf into the generated config:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Backup generated config, then append your hardware lines
|
|
||||||
cp /etc/pve/qemu-server/100.conf /etc/pve/qemu-server/100.conf.bak
|
|
||||||
cat 100-frigate.conf.example >> /etc/pve/qemu-server/100.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
Or just open each VM in the Proxmox web UI → **Hardware** and add:
|
|
||||||
- PCI Device → your GPU (enable PCIe, enable Primary GPU for VM 100)
|
|
||||||
- USB Device → host device → `1a6e:089a` and `18d1:9302` (VM 100 only)
|
|
||||||
- Hard Disk → (use disk passthrough, SCSI controller, path = your by-id)
|
|
||||||
|
|
||||||
### Install a guest OS
|
|
||||||
|
|
||||||
Boot each VM from an ISO (upload ISOs to Proxmox under local storage →
|
|
||||||
ISO Images). Ubuntu Server 22.04 LTS is a good choice for the Frigate VM.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 12 — Set Up Frigate in VM 100
|
|
||||||
|
|
||||||
From inside VM 100 (SSH into the guest OS):
|
|
||||||
|
|
||||||
### Install Docker
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt update && apt install -y ca-certificates curl
|
|
||||||
curl -fsSL https://get.docker.com | sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Install NVIDIA driver + Container Toolkit
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Add NVIDIA apt repo
|
|
||||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
|
||||||
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
|
||||||
|
|
||||||
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
|
||||||
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
|
|
||||||
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
|
||||||
|
|
||||||
apt update
|
|
||||||
apt install -y nvidia-driver-535 nvidia-container-toolkit
|
|
||||||
|
|
||||||
# Configure Docker to use the NVIDIA runtime
|
|
||||||
nvidia-ctk runtime configure --runtime=docker
|
|
||||||
systemctl restart docker
|
|
||||||
|
|
||||||
# Verify the GPU is visible
|
|
||||||
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
|
|
||||||
```
|
|
||||||
|
|
||||||
### Deploy Frigate
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /opt/frigate && cd /opt/frigate
|
|
||||||
|
|
||||||
# Copy configs from the repo (or clone it inside the VM)
|
|
||||||
cp /path/to/local_proxmox/frigate/docker-compose.yml .
|
|
||||||
cp /path/to/local_proxmox/frigate/config.yml .
|
|
||||||
|
|
||||||
# Create the recordings directory (on your passthrough data drive)
|
|
||||||
# Mount your data drives first — if using ZFS:
|
|
||||||
zpool import <poolname> # or create a new pool from the raw drives
|
|
||||||
mkdir -p /mnt/frigate
|
|
||||||
|
|
||||||
# Edit config.yml and add your camera RTSP URLs
|
|
||||||
nano config.yml
|
|
||||||
|
|
||||||
docker compose up -d
|
|
||||||
docker compose logs -f # watch for errors on first start
|
|
||||||
```
|
|
||||||
|
|
||||||
Frigate UI: `http://<vm100-ip>:5000`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### GPU not passing through — still shows `nouveau`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
update-initramfs -u -k all && reboot
|
|
||||||
# After reboot:
|
|
||||||
lspci -nnk | grep -A3 -i nvidia # must show vfio-pci
|
|
||||||
```
|
|
||||||
|
|
||||||
### IOMMU not enabled
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i iommu
|
|
||||||
# Should show: "DMAR: IOMMU enabled"
|
|
||||||
# If not: re-check Phase 9 cmdline (intel_iommu=on); VT-d is on by default in newer BIOS
|
|
||||||
cat /etc/kernel/cmdline # must contain intel_iommu=on iommu=pt
|
|
||||||
```
|
|
||||||
|
|
||||||
### Fan control not working
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check iDRAC IP is reachable and IPMI over LAN is enabled (Phase 2)
|
|
||||||
ipmitool -I lan -H <idrac-ip> -U root -P <password> sdr type Fan
|
|
||||||
# Then check the service:
|
|
||||||
systemctl status fan-control.service
|
|
||||||
journalctl -u fan-control.service -n 50
|
|
||||||
```
|
|
||||||
|
|
||||||
### Coral not detected in Frigate
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# In VM 100, check both USB IDs are present
|
|
||||||
lsusb | grep -E "1a6e|18d1"
|
|
||||||
# If missing, check the USB passthrough lines in the VM config
|
|
||||||
# Both usb0 (1a6e:089a) and usb1 (18d1:9302) must be present
|
|
||||||
```
|
|
||||||
|
|
||||||
### Drive not appearing after perc-nonraid.sh
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk
|
|
||||||
# If the drive is missing, check its state in perccli:
|
|
||||||
perccli /c0 /eall /sall show
|
|
||||||
# State should be "JBOD" or "UGood" — not "Offln" or "Msng"
|
|
||||||
```
|
|
||||||
@@ -1,563 +0,0 @@
|
|||||||
# Phases 1–3: Firmware, iDRAC, and BIOS
|
|
||||||
### Dell R730xd — Button-by-button walkthrough
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Before You Touch Anything — Read This First
|
|
||||||
|
|
||||||
### What is iDRAC and do I need to pay for it?
|
|
||||||
|
|
||||||
iDRAC stands for **Integrated Dell Remote Access Controller**. Ignore the name.
|
|
||||||
Here is what it actually is:
|
|
||||||
|
|
||||||
There is a **second tiny computer built into your server's motherboard**. It has
|
|
||||||
its own processor, its own RAM, its own network port, and it runs 24 hours a day
|
|
||||||
as long as the server has power — even when the server is "off". Dell calls this
|
|
||||||
second computer iDRAC.
|
|
||||||
|
|
||||||
This server has **iDRAC8 Express**, which is the version built into the board
|
|
||||||
at no cost. There is a paid upgrade called iDRAC8 Enterprise but you do not
|
|
||||||
need it. Here is exactly what Express gives you and what it does not:
|
|
||||||
|
|
||||||
**Express includes (everything this guide uses):**
|
|
||||||
- iDRAC web UI — hardware health dashboard, temperatures, fan speeds, event logs
|
|
||||||
- Power the server on and off remotely from the web UI
|
|
||||||
- IPMI over LAN — lets `ipmitool` send fan control commands from Proxmox
|
|
||||||
- racadm — command-line control used by the stagger spin-up script
|
|
||||||
- SSH directly into iDRAC for management
|
|
||||||
|
|
||||||
**Express does NOT include:**
|
|
||||||
- Virtual Console — you cannot see the server's screen in a browser window.
|
|
||||||
A **physical monitor and keyboard are required** for all setup steps until
|
|
||||||
Proxmox is installed and SSH is working. After that you will never need the
|
|
||||||
monitor again.
|
|
||||||
- Virtual Media — you cannot mount an ISO file over the network. A **physical
|
|
||||||
USB drive** is required for the Proxmox installer.
|
|
||||||
|
|
||||||
Nothing in this guide requires Enterprise. The fan control script and stagger
|
|
||||||
spin-up both use IPMI over LAN and racadm, which are both available on Express.
|
|
||||||
|
|
||||||
**Do you need to pay for anything?** No. iDRAC Express is already there and
|
|
||||||
covers everything we need. The only other "subscription" people sometimes ask
|
|
||||||
about is the **Proxmox subscription** (about €100/year for enterprise update
|
|
||||||
servers and commercial support). You do not need that either — we configure the
|
|
||||||
free community repositories in Phase 6. The two are completely unrelated.
|
|
||||||
|
|
||||||
**Summary: pay nothing, skip nothing. Keep a monitor plugged in through Phase 6.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### What you need on the table before starting
|
|
||||||
|
|
||||||
- The server plugged into power and a network switch/router
|
|
||||||
- A **separate network cable** for the iDRAC port (the small RJ45 on the back
|
|
||||||
labeled "iDRAC" — it is separate from the four main NIC ports)
|
|
||||||
- A **monitor and USB keyboard** plugged into the server — required through
|
|
||||||
Phase 5 (the Proxmox installer). iDRAC Express does not include a remote
|
|
||||||
KVM, so you cannot see the server's screen from another computer. After
|
|
||||||
Proxmox is installed and SSH is working (end of Phase 6) the monitor
|
|
||||||
can be unplugged permanently.
|
|
||||||
- A USB drive (8 GB or larger) — needed later for the Proxmox installer
|
|
||||||
- Another computer or phone to read this guide
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 1 — Firmware Updates
|
|
||||||
|
|
||||||
**Why first?** Old firmware has bugs. BIOS from 2016 may not correctly expose
|
|
||||||
the VT-d settings that GPU passthrough requires. The H730 firmware has had fixes
|
|
||||||
for JBOD/non-RAID mode. Updating now prevents chasing ghosts later.
|
|
||||||
|
|
||||||
**Time required:** 30–90 minutes. The server will reboot several times on its own.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 1.1 — Power on and watch the screen
|
|
||||||
|
|
||||||
Press the power button on the front of the server.
|
|
||||||
|
|
||||||
The screen will show a **Dell splash screen** with the PowerEdge logo. At the
|
|
||||||
bottom of the screen you will see a line of options, something like:
|
|
||||||
|
|
||||||
```
|
|
||||||
F2 = System Setup F10 = Lifecycle Controller F11 = Boot Manager F12 = PXE Boot
|
|
||||||
```
|
|
||||||
|
|
||||||
These options are only available for about **5–8 seconds** before the server
|
|
||||||
continues booting. If you miss the window the server will try to boot an OS
|
|
||||||
(and fail if nothing is installed yet). Just power it off and back on and try again.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 1.2 — Enter Lifecycle Controller
|
|
||||||
|
|
||||||
Press **F10** when you see the splash screen.
|
|
||||||
|
|
||||||
The screen will go blank for 10–30 seconds, then show the **Lifecycle Controller**
|
|
||||||
loading screen. This is a mini operating system built into the server's firmware.
|
|
||||||
It has nothing to do with your server's main OS. Wait for it to finish loading —
|
|
||||||
it can take up to 90 seconds the first time.
|
|
||||||
|
|
||||||
When it finishes you will see the **Lifecycle Controller Home** screen with a menu
|
|
||||||
on the left side:
|
|
||||||
|
|
||||||
```
|
|
||||||
Home
|
|
||||||
Firmware Update
|
|
||||||
OS Deployment
|
|
||||||
RAID Configuration
|
|
||||||
Hardware Configuration
|
|
||||||
Settings
|
|
||||||
```
|
|
||||||
|
|
||||||
If it asks you to complete initial setup (language, network), do that first —
|
|
||||||
use Tab and arrow keys to navigate, Enter to confirm.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 1.3 — Go to Firmware Update
|
|
||||||
|
|
||||||
Click or arrow-key to **Firmware Update** on the left menu, then press Enter.
|
|
||||||
|
|
||||||
You will see:
|
|
||||||
|
|
||||||
```
|
|
||||||
Launch Firmware Update
|
|
||||||
```
|
|
||||||
|
|
||||||
Press Enter on that.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 1.4 — Choose your update method
|
|
||||||
|
|
||||||
You will be asked how you want to get the firmware. You have two options:
|
|
||||||
|
|
||||||
#### Option A — Server has internet access (easiest)
|
|
||||||
|
|
||||||
If your network cable is plugged in and the server can reach the internet:
|
|
||||||
|
|
||||||
1. Select **HTTPS** (it will say something like "downloads.dell.com")
|
|
||||||
2. Press **Next**
|
|
||||||
3. Lifecycle Controller will connect to Dell's servers and **automatically scan
|
|
||||||
your server** to find what firmware versions are installed vs. what is available
|
|
||||||
4. This takes 2–5 minutes — the screen will show a progress bar
|
|
||||||
5. Skip to Step 1.5
|
|
||||||
|
|
||||||
#### Option B — No internet on the server
|
|
||||||
|
|
||||||
You need to download the firmware files on another computer first.
|
|
||||||
|
|
||||||
1. On your other computer, open a browser and go to:
|
|
||||||
**https://www.dell.com/support/home**
|
|
||||||
|
|
||||||
2. You need your server's **Service Tag** — it is a 7-character code. Find it:
|
|
||||||
- On the **front of the server**: there is a small pull-out plastic tab (like a
|
|
||||||
credit card slot) on the left side of the front panel. Pull it out — the
|
|
||||||
Service Tag is printed on it.
|
|
||||||
- Or: on a sticker on the **top** of the server chassis.
|
|
||||||
- Or: it was displayed briefly during POST on the splash screen.
|
|
||||||
|
|
||||||
3. On the Dell support page, click **"View products"** or type the Service Tag
|
|
||||||
into the search bar at the top. Select your server from the results.
|
|
||||||
|
|
||||||
4. Click **"Drivers & Downloads"**
|
|
||||||
|
|
||||||
5. You will see a filter panel. Under **"Category"**, select each of these one
|
|
||||||
at a time and download the latest version:
|
|
||||||
- **iDRAC** — look for "iDRAC8 firmware" — file will end in `.exe` or `.d9`
|
|
||||||
- **BIOS** — look for "BIOS" — file ends in `.exe`
|
|
||||||
- **RAID** — look for "PERC H730" — file ends in `.exe`
|
|
||||||
|
|
||||||
6. Copy all downloaded files onto a **FAT32-formatted USB drive**
|
|
||||||
|
|
||||||
7. Plug the USB drive into the server
|
|
||||||
|
|
||||||
8. In Lifecycle Controller: select **Local Drive (USB)** and press Next
|
|
||||||
|
|
||||||
9. Navigate to the USB drive and select the firmware files
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 1.5 — Review and apply updates
|
|
||||||
|
|
||||||
Lifecycle Controller shows a table of all components with columns like:
|
|
||||||
|
|
||||||
```
|
|
||||||
Component Current Version Available Version Select
|
|
||||||
--------- --------------- ----------------- ------
|
|
||||||
iDRAC8 2.40.40.40 2.85.85.85 [ ]
|
|
||||||
BIOS 1.2.10 2.14.0 [ ]
|
|
||||||
PERC H730 Mini 25.4.0.0018 25.5.9.0001 [ ]
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Check the box next to **every component** that shows a newer Available Version
|
|
||||||
(you can usually click "Select All")
|
|
||||||
2. Click **Install and Reboot** (or **Apply**)
|
|
||||||
3. A warning will say the server will reboot — click **Yes**
|
|
||||||
|
|
||||||
The server will now:
|
|
||||||
- Apply each firmware update
|
|
||||||
- Reboot automatically between some updates
|
|
||||||
- Return to the Lifecycle Controller when finished
|
|
||||||
|
|
||||||
**Do not power off the server during this process.** Just wait. It can take
|
|
||||||
30–60 minutes if there are many updates. The screen will show progress.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 1.6 — Confirm completion
|
|
||||||
|
|
||||||
When all updates are done, Lifecycle Controller returns to its home screen.
|
|
||||||
You may see a success summary screen first — click **OK** or **Finish**.
|
|
||||||
|
|
||||||
To verify everything worked:
|
|
||||||
1. In Lifecycle Controller, go to **Firmware Update** → **Launch Firmware Update**
|
|
||||||
again
|
|
||||||
2. Run the check one more time
|
|
||||||
3. The Available Version column should now match Current Version for everything
|
|
||||||
you updated (or show no newer version)
|
|
||||||
|
|
||||||
Press **Exit** (or **Finish**) to leave Lifecycle Controller. The server will
|
|
||||||
reboot normally.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2 — iDRAC Setup
|
|
||||||
|
|
||||||
You need to do two things with iDRAC:
|
|
||||||
1. Give it a **static IP address** so it is always reachable at the same address
|
|
||||||
2. Enable **IPMI over LAN** so the fan control script can send commands to it
|
|
||||||
|
|
||||||
There are two ways to do this — through the server's BIOS-like setup (F2),
|
|
||||||
or through the iDRAC web UI once you find its current IP. We will use F2 first
|
|
||||||
to set a static IP, then use the web UI for the rest.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 2.1 — Enter System Setup (F2)
|
|
||||||
|
|
||||||
Power on (or reboot) the server.
|
|
||||||
|
|
||||||
When the Dell splash screen appears, press **F2**.
|
|
||||||
|
|
||||||
The screen will go blank briefly, then show the **System Setup Main Menu**:
|
|
||||||
|
|
||||||
```
|
|
||||||
System Setup Main Menu
|
|
||||||
|
|
||||||
System BIOS
|
|
||||||
iDRAC Settings
|
|
||||||
Device Settings
|
|
||||||
Service Tag Settings
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the **arrow keys** to highlight **iDRAC Settings** and press **Enter**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 2.2 — Set a static IP for iDRAC
|
|
||||||
|
|
||||||
Inside iDRAC Settings you will see another menu:
|
|
||||||
|
|
||||||
```
|
|
||||||
iDRAC Settings
|
|
||||||
|
|
||||||
Network
|
|
||||||
User Configuration
|
|
||||||
Smart Card
|
|
||||||
Update and Rollback
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Arrow down to **Network** and press **Enter**.
|
|
||||||
|
|
||||||
The Network screen has several sections. You are looking for the **IPv4 Settings**
|
|
||||||
section. Use Tab or arrow keys to navigate to these fields:
|
|
||||||
|
|
||||||
| Field | What to set |
|
|
||||||
|-------|-------------|
|
|
||||||
| Enable NIC | **Enabled** |
|
|
||||||
| NIC Selection | **Dedicated** (this uses the dedicated iDRAC port on the back) |
|
|
||||||
| Enable IPv4 | **Enabled** |
|
|
||||||
| Enable DHCP | **Disabled** ← change this |
|
|
||||||
| Static IP Address | e.g. `192.168.1.5` (pick an IP outside your router's DHCP range) |
|
|
||||||
| Static Gateway | your router's IP, e.g. `192.168.1.1` |
|
|
||||||
| Static Subnet Mask | `255.255.255.0` |
|
|
||||||
| DNS Server 1 | your router's IP, e.g. `192.168.1.1` |
|
|
||||||
|
|
||||||
> **What IP to use?** Log into your router and find its DHCP range — for example
|
|
||||||
> if DHCP hands out `192.168.1.100` to `192.168.1.200`, pick something outside
|
|
||||||
> that range like `192.168.1.5`. Write this IP down — you will use it often.
|
|
||||||
|
|
||||||
When done, press **Back** or navigate to the bottom and click **Apply**, then **OK**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 2.3 — Enable IPMI over LAN
|
|
||||||
|
|
||||||
Still inside iDRAC Settings → Network.
|
|
||||||
|
|
||||||
Scroll down past the IPv4 section until you see **IPMI Settings**.
|
|
||||||
|
|
||||||
Find:
|
|
||||||
|
|
||||||
```
|
|
||||||
Enable IPMI over LAN: [ Disabled ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Change this to **Enabled**.
|
|
||||||
|
|
||||||
This is the setting that allows `ipmitool` commands (used by the fan control
|
|
||||||
script) to reach iDRAC over the network.
|
|
||||||
|
|
||||||
Click **Apply** at the bottom, then **OK** when asked to confirm.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 2.4 — Change the default password
|
|
||||||
|
|
||||||
Go back to the iDRAC Settings main menu (press **Back**).
|
|
||||||
|
|
||||||
Arrow down to **User Configuration** and press **Enter**.
|
|
||||||
|
|
||||||
You will see a list of user slots. **User 1** is the built-in `root` account.
|
|
||||||
Select it (press Enter).
|
|
||||||
|
|
||||||
> **Why change it?** The factory default password is `calvin`. This is printed
|
|
||||||
> in Dell's public documentation and is widely known. Any device on your network
|
|
||||||
> could log into your iDRAC if you leave it as-is.
|
|
||||||
|
|
||||||
Change:
|
|
||||||
- **User Name:** leave as `root`
|
|
||||||
- **Change Password:** set to **Enabled**
|
|
||||||
- **Password:** enter a strong password
|
|
||||||
- **Confirm Password:** enter it again
|
|
||||||
|
|
||||||
Click **Apply**, then **OK**.
|
|
||||||
|
|
||||||
Press **Finish** to exit iDRAC Settings and return to the System Setup Main Menu.
|
|
||||||
|
|
||||||
**Do not reboot yet** — you still have BIOS settings to configure in Phase 3.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 2.5 — Verify iDRAC from your browser (do after Phase 3)
|
|
||||||
|
|
||||||
After you finish Phase 3 and let the server reboot, test iDRAC from your other
|
|
||||||
computer:
|
|
||||||
|
|
||||||
1. Make sure the **dedicated iDRAC network port** (small RJ45 labeled "iDRAC" on
|
|
||||||
the server's back panel) is plugged into your switch or router
|
|
||||||
2. Open a browser on your other computer
|
|
||||||
3. Go to: `https://192.168.1.5` (whatever static IP you set in Step 2.2)
|
|
||||||
4. You will see a **certificate warning** — this is normal, iDRAC uses a
|
|
||||||
self-signed certificate. Click **Advanced** → **Proceed anyway** (wording
|
|
||||||
varies by browser)
|
|
||||||
5. The iDRAC8 login page appears — a dark Dell-branded page
|
|
||||||
6. Log in: Username `root`, Password (the one you set in Step 2.4)
|
|
||||||
|
|
||||||
You should see the iDRAC dashboard showing system health, temperatures, and fans.
|
|
||||||
This is your remote window into the server — you can now manage it from your desk.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3 — BIOS Settings
|
|
||||||
|
|
||||||
These settings enable the CPU features that allow devices (GPUs, USB controllers)
|
|
||||||
to be handed directly to virtual machines. **Every setting below is required.**
|
|
||||||
Skipping VT-d is the single most common reason GPU passthrough silently fails.
|
|
||||||
|
|
||||||
You are still in System Setup from Phase 2 (or press F2 again on reboot).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 3.1 — System BIOS → Processor Settings
|
|
||||||
|
|
||||||
From the System Setup Main Menu, arrow to **System BIOS** and press Enter.
|
|
||||||
|
|
||||||
You will see the System BIOS Settings menu:
|
|
||||||
|
|
||||||
```
|
|
||||||
System BIOS Settings
|
|
||||||
|
|
||||||
System Information
|
|
||||||
Memory Settings
|
|
||||||
Processor Settings ← go here
|
|
||||||
SATA Settings
|
|
||||||
Boot Settings
|
|
||||||
Integrated Devices
|
|
||||||
Serial Communication
|
|
||||||
System Profile Settings
|
|
||||||
Power Management
|
|
||||||
Security
|
|
||||||
Miscellaneous Settings
|
|
||||||
```
|
|
||||||
|
|
||||||
Arrow to **Processor Settings** and press **Enter**.
|
|
||||||
|
|
||||||
Find and set these two options:
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Virtualization Technology**
|
|
||||||
|
|
||||||
This enables the CPU to run virtual machines efficiently. Without it Proxmox
|
|
||||||
still works but performance is worse.
|
|
||||||
|
|
||||||
```
|
|
||||||
Virtualization Technology: [ Disabled ]
|
|
||||||
↓ change to
|
|
||||||
Virtualization Technology: [ Enabled ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the arrow keys or spacebar to toggle the value.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**VT for Direct I/O** (also shown as "Virtualization Technology for Directed I/O")
|
|
||||||
|
|
||||||
This is the critical one. It enables **IOMMU** — the hardware feature that lets
|
|
||||||
the CPU safely hand a real PCIe device (like your GPU) directly to a virtual
|
|
||||||
machine. Without this, passthrough is impossible.
|
|
||||||
|
|
||||||
```
|
|
||||||
VT for Direct I/O: [ Disabled ]
|
|
||||||
↓ change to
|
|
||||||
VT for Direct I/O: [ Enabled ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Click **Back** to return to System BIOS Settings.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 3.2 — System BIOS → Integrated Devices
|
|
||||||
|
|
||||||
From the System BIOS Settings menu, arrow to **Integrated Devices** and press Enter.
|
|
||||||
|
|
||||||
Find:
|
|
||||||
|
|
||||||
**SR-IOV Global Enable**
|
|
||||||
|
|
||||||
SR-IOV is a PCIe feature that allows a single physical device to appear as
|
|
||||||
multiple devices. Even though the Quadro P2200 does not use SR-IOV, enabling
|
|
||||||
this globally avoids a class of IOMMU grouping problems.
|
|
||||||
|
|
||||||
```
|
|
||||||
SR-IOV Global Enable: [ Disabled ]
|
|
||||||
↓ change to
|
|
||||||
SR-IOV Global Enable: [ Enabled ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Click **Back**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 3.3 — System BIOS → Boot Settings
|
|
||||||
|
|
||||||
From System BIOS Settings, arrow to **Boot Settings** and press Enter.
|
|
||||||
|
|
||||||
You will see:
|
|
||||||
|
|
||||||
```
|
|
||||||
Boot Settings
|
|
||||||
|
|
||||||
Boot Mode: [ BIOS ] ← must change this
|
|
||||||
Boot Sequence
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
**Boot Mode** — change from BIOS to UEFI:
|
|
||||||
|
|
||||||
```
|
|
||||||
Boot Mode: [ BIOS ]
|
|
||||||
↓ change to
|
|
||||||
Boot Mode: [ UEFI ]
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Why UEFI?** Proxmox uses a tool called `proxmox-boot-tool` to manage the
|
|
||||||
> kernel boot parameters (including the IOMMU settings the GPU passthrough script
|
|
||||||
> writes). This tool only works with UEFI boot. If the server is in Legacy/BIOS
|
|
||||||
> mode the passthrough script will fail and you will have to reinstall.
|
|
||||||
|
|
||||||
A warning may appear saying the boot sequence will be cleared — click **Yes** or
|
|
||||||
**OK**. That is fine since we haven't installed anything yet.
|
|
||||||
|
|
||||||
Click **Back**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 3.4 — System BIOS → Power Management
|
|
||||||
|
|
||||||
From System BIOS Settings, arrow to **Power Management** and press Enter.
|
|
||||||
|
|
||||||
Find:
|
|
||||||
|
|
||||||
**Hard Disk Drive Sequencing** — this is the built-in stagger that fires at
|
|
||||||
power-on before the OS loads. It staggers when each drive spins up so they do
|
|
||||||
not all surge at the same time and overload the PSU.
|
|
||||||
|
|
||||||
```
|
|
||||||
Hard Disk Drive Sequencing: [ Disabled ]
|
|
||||||
↓ change to
|
|
||||||
Hard Disk Drive Sequencing: [ Enabled ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Also check:
|
|
||||||
|
|
||||||
**C States** — these are CPU power-saving sleep states. They can add latency
|
|
||||||
to VM workloads. Optional but worth setting for a server:
|
|
||||||
|
|
||||||
```
|
|
||||||
C States: [ Enabled ]
|
|
||||||
↓ change to
|
|
||||||
C States: [ Disabled ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Click **Back**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 3.5 — Apply and exit
|
|
||||||
|
|
||||||
From System BIOS Settings, click **Finish** (at the bottom of the menu, you may
|
|
||||||
need to scroll).
|
|
||||||
|
|
||||||
A dialog will appear:
|
|
||||||
|
|
||||||
```
|
|
||||||
Confirm changes and exit?
|
|
||||||
[ Yes ] [ No ]
|
|
||||||
```
|
|
||||||
|
|
||||||
Click **Yes**.
|
|
||||||
|
|
||||||
The server will reboot.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 3.6 — Confirm the settings took
|
|
||||||
|
|
||||||
After the reboot, press **F2** again to re-enter System Setup and spot-check:
|
|
||||||
|
|
||||||
- **Processor Settings:** VT and VT for Direct I/O should both show **Enabled**
|
|
||||||
- **Boot Settings:** Boot Mode should show **UEFI**
|
|
||||||
|
|
||||||
If either shows Disabled, set it again and apply. Some BIOS versions have a bug
|
|
||||||
where the first Apply does not persist — applying a second time fixes it.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Phase 3 complete — what you have now
|
|
||||||
|
|
||||||
| Setting | Why it matters |
|
|
||||||
|---------|---------------|
|
|
||||||
| VT-x (Virtualization Technology) | Lets the CPU run VMs at near-native speed |
|
|
||||||
| VT-d (Direct I/O) | Enables IOMMU — the foundation of all device passthrough |
|
|
||||||
| SR-IOV | Prevents IOMMU group problems with PCIe devices |
|
|
||||||
| UEFI boot mode | Required for Proxmox's boot management and EFI GPU passthrough |
|
|
||||||
| HDD sequencing | Staggers drive spin-up at power-on to protect the PSU |
|
|
||||||
|
|
||||||
**Next:** Phase 4 — Configure the H730 RAID controller to mirror the two
|
|
||||||
2.5" OS drives, then leave the 3.5" drives alone for Proxmox to manage.
|
|
||||||
That walkthrough is in `docs/walkthrough-phases-4-6.md` (coming next).
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,611 +0,0 @@
|
|||||||
# Phases 4–6: RAID Config, Proxmox Install, Post-Install
|
|
||||||
### Dell R730xd — Button-by-button walkthrough
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4 — H730 RAID Configuration
|
|
||||||
|
|
||||||
**Goal:** Mirror the two 2.5" rear drives together so a single drive failure
|
|
||||||
cannot take down Proxmox. Leave every 3.5" drive untouched — the `perc-nonraid.sh`
|
|
||||||
script handles those later from inside Proxmox.
|
|
||||||
|
|
||||||
**Why mirror the OS drives here instead of using ZFS?** The H730 presents the
|
|
||||||
RAID1 mirror as a single virtual disk to the OS. Proxmox installs onto it and
|
|
||||||
never needs to think about the fact that two physical drives are underneath.
|
|
||||||
This is the simplest and most reliable approach for an OS volume.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 4.1 — Enter Lifecycle Controller
|
|
||||||
|
|
||||||
Power on (or reboot) the server and press **F10** at the Dell splash screen.
|
|
||||||
|
|
||||||
Wait for Lifecycle Controller to load. You will see the main menu on the left:
|
|
||||||
|
|
||||||
```
|
|
||||||
Home
|
|
||||||
Firmware Update
|
|
||||||
OS Deployment
|
|
||||||
RAID Configuration ← go here
|
|
||||||
Hardware Configuration
|
|
||||||
Settings
|
|
||||||
```
|
|
||||||
|
|
||||||
Arrow to **RAID Configuration** and press **Enter**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 4.2 — Open the RAID configuration wizard
|
|
||||||
|
|
||||||
You will see:
|
|
||||||
|
|
||||||
```
|
|
||||||
RAID Configuration
|
|
||||||
|
|
||||||
View Current Configuration
|
|
||||||
Create New VD
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
First, click **View Current Configuration** to see what is already there.
|
|
||||||
|
|
||||||
You are looking at the physical drives the H730 can see. Drives are listed under
|
|
||||||
their controller — it will say something like:
|
|
||||||
|
|
||||||
```
|
|
||||||
PERC H730 Mini (Slot 0)
|
|
||||||
|
|
||||||
Physical Disks:
|
|
||||||
Port 0: SEAGATE ST4000NM0023 4.0 TB State: Unconfigured Good
|
|
||||||
Port 1: SEAGATE ST4000NM0023 4.0 TB State: Unconfigured Good
|
|
||||||
...
|
|
||||||
Port 8: TOSHIBA MK1401GRRB 146 GB State: Unconfigured Good ← 2.5"
|
|
||||||
Port 9: TOSHIBA MK1401GRRB 146 GB State: Unconfigured Good ← 2.5"
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Identifying the 2.5" OS drives:** They will be noticeably smaller than the
|
|
||||||
> 3.5" drives — likely 146 GB, 300 GB, 600 GB (SAS) or 120–480 GB (SSD).
|
|
||||||
> The 3.5" drives will be 1 TB, 2 TB, 4 TB, or larger. Size is the giveaway.
|
|
||||||
|
|
||||||
> **If drives show "Foreign Configuration":** They were previously part of a
|
|
||||||
> RAID array. See Step 4.3a before continuing.
|
|
||||||
|
|
||||||
> **If the 2.5" drives appear under a different controller** (e.g., "PERC H330
|
|
||||||
> Mini"): The R730xd sometimes puts the rear bays on a separate mini controller.
|
|
||||||
> That is fine — just create the RAID1 on whichever controller owns those drives.
|
|
||||||
|
|
||||||
Press **Back** to return to the RAID Configuration menu.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 4.3a — Clear foreign configurations (only if needed)
|
|
||||||
|
|
||||||
If any drives showed "Foreign Configuration" in Step 4.2, you need to clear
|
|
||||||
them before you can create a new virtual disk.
|
|
||||||
|
|
||||||
In the RAID Configuration menu, look for:
|
|
||||||
|
|
||||||
```
|
|
||||||
Clear Foreign Configuration
|
|
||||||
```
|
|
||||||
|
|
||||||
Select it, choose **All Foreign Configurations**, click **Apply**.
|
|
||||||
|
|
||||||
A warning will appear saying all data on those drives will be lost — click **Yes**.
|
|
||||||
(We are building a new system so there is nothing to keep.)
|
|
||||||
|
|
||||||
After clearing, go back to **View Current Configuration** and confirm the drives
|
|
||||||
now show **Unconfigured Good**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 4.4 — Create the RAID 1 virtual disk
|
|
||||||
|
|
||||||
In the RAID Configuration menu, select **Create New VD** and press **Enter**.
|
|
||||||
|
|
||||||
The wizard walks you through four screens:
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Screen 1 — Select RAID Level**
|
|
||||||
|
|
||||||
```
|
|
||||||
Select RAID Level:
|
|
||||||
|
|
||||||
RAID 0 (no redundancy, faster)
|
|
||||||
RAID 1 (mirror, one drive can fail) ← select this
|
|
||||||
RAID 5 (requires 3+ drives)
|
|
||||||
RAID 6 (requires 4+ drives)
|
|
||||||
RAID 10 (requires 4+ drives)
|
|
||||||
```
|
|
||||||
|
|
||||||
Select **RAID 1** and click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Screen 2 — Select Physical Disks**
|
|
||||||
|
|
||||||
A list of all available physical disks appears. You need to select **only the
|
|
||||||
two 2.5" drives**. Do not select any 3.5" drives.
|
|
||||||
|
|
||||||
Click the checkbox next to each of the two 2.5" drives (identified by their
|
|
||||||
smaller size). Leave all 3.5" drives unchecked.
|
|
||||||
|
|
||||||
Click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Screen 3 — Virtual Disk Attributes**
|
|
||||||
|
|
||||||
```
|
|
||||||
Virtual Disk Name: OS-Mirror
|
|
||||||
Virtual Disk Size: (auto-filled — leave it)
|
|
||||||
Strip Element Size: 64KB (leave default)
|
|
||||||
Read Policy: Adaptive Read Ahead (leave default)
|
|
||||||
Write Policy: Write Back (leave default)
|
|
||||||
Disk Cache Policy: Enabled (leave default)
|
|
||||||
```
|
|
||||||
|
|
||||||
The only thing you need to change is the name — type `OS-Mirror` so it is easy
|
|
||||||
to identify. Everything else can stay at defaults.
|
|
||||||
|
|
||||||
Click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Screen 4 — Confirm**
|
|
||||||
|
|
||||||
Review the summary. It should show:
|
|
||||||
- RAID Level: 1
|
|
||||||
- Physical Disks: 2 (your 2.5" drives)
|
|
||||||
- Virtual Disk Size: whatever the smaller of the two drives is
|
|
||||||
|
|
||||||
Click **Finish**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 4.5 — Initialize the virtual disk
|
|
||||||
|
|
||||||
After creating the VD, Lifecycle Controller will ask about initialization:
|
|
||||||
|
|
||||||
```
|
|
||||||
Initialize virtual disk?
|
|
||||||
Fast Initialize ← select this
|
|
||||||
Full Initialize (writes zeros to every sector — takes hours, not needed)
|
|
||||||
Skip
|
|
||||||
```
|
|
||||||
|
|
||||||
Select **Fast Initialize** and click **OK**.
|
|
||||||
|
|
||||||
Fast initialization takes 30–60 seconds. A progress bar will appear. Wait for it.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 4.6 — Verify and exit
|
|
||||||
|
|
||||||
When initialization finishes, go back to **View Current Configuration**.
|
|
||||||
|
|
||||||
You should now see:
|
|
||||||
|
|
||||||
```
|
|
||||||
PERC H730 Mini
|
|
||||||
|
|
||||||
Virtual Disks:
|
|
||||||
VD 0: OS-Mirror RAID 1 ~146 GB State: Optimal
|
|
||||||
|
|
||||||
Physical Disks:
|
|
||||||
Port 0–7: 3.5" drives State: Unconfigured Good
|
|
||||||
Port 8–9: 2.5" drives State: Online (member of VD 0)
|
|
||||||
```
|
|
||||||
|
|
||||||
The 3.5" drives should all show **Unconfigured Good**. That is exactly what you
|
|
||||||
want. Leave them that way.
|
|
||||||
|
|
||||||
Click **Back** repeatedly until you reach the Lifecycle Controller home screen,
|
|
||||||
then click **Exit**. The server will reboot.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 5 — Install Proxmox VE 9.1
|
|
||||||
|
|
||||||
### Step 5.1 — Download the Proxmox ISO
|
|
||||||
|
|
||||||
On your other computer, open a browser and go to:
|
|
||||||
|
|
||||||
```
|
|
||||||
https://www.proxmox.com/en/downloads
|
|
||||||
```
|
|
||||||
|
|
||||||
Click **Proxmox Virtual Environment** → find the latest **Proxmox VE 9.x ISO
|
|
||||||
Installer** and click **Download**.
|
|
||||||
|
|
||||||
The file will be named something like `proxmox-ve_9.1-1.iso` and is about 1 GB.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.2 — Write the ISO to a USB drive
|
|
||||||
|
|
||||||
You need to write the ISO as a disk image (not copy the file). Use one of these
|
|
||||||
tools depending on your computer:
|
|
||||||
|
|
||||||
#### Windows — Rufus (free, no install needed)
|
|
||||||
|
|
||||||
1. Download Rufus from: **https://rufus.ie** (click the first .exe link)
|
|
||||||
2. Plug in your USB drive (8 GB or larger — all data on it will be erased)
|
|
||||||
3. Open Rufus
|
|
||||||
4. **Device:** select your USB drive from the dropdown
|
|
||||||
5. **Boot selection:** click **SELECT** and choose the Proxmox ISO file
|
|
||||||
6. **Partition scheme:** select **GPT**
|
|
||||||
7. **Target system:** select **UEFI (non CSM)**
|
|
||||||
8. Click **START**
|
|
||||||
9. A dialog appears asking about ISO mode vs DD mode — select **Write in DD Image
|
|
||||||
mode** and click **OK**
|
|
||||||
10. Click **OK** again when warned that the USB will be wiped
|
|
||||||
11. Wait for it to finish (1–3 minutes), then click **CLOSE**
|
|
||||||
|
|
||||||
#### Mac — Balena Etcher (free)
|
|
||||||
|
|
||||||
1. Download from: **https://etcher.balena.io** — click **Download for macOS**
|
|
||||||
2. Open Etcher
|
|
||||||
3. Click **Flash from file** → select the Proxmox ISO
|
|
||||||
4. Click **Select target** → select your USB drive
|
|
||||||
5. Click **Flash** — enter your Mac password if prompted
|
|
||||||
6. Wait for it to finish
|
|
||||||
|
|
||||||
#### Linux — terminal
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Find your USB drive device name (look for your drive size)
|
|
||||||
lsblk
|
|
||||||
|
|
||||||
# Write the ISO (replace sdX with your actual USB device, e.g. sdb — NOT sdb1)
|
|
||||||
dd if=proxmox-ve_9.1-1.iso of=/dev/sdX bs=1M status=progress conv=fsync
|
|
||||||
sync
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.3 — Boot the server from USB
|
|
||||||
|
|
||||||
1. Plug the USB drive into one of the USB ports on the front or back of the server
|
|
||||||
2. Power on (or reboot) the server
|
|
||||||
3. At the Dell splash screen, press **F11**
|
|
||||||
|
|
||||||
The **Boot Manager** screen appears:
|
|
||||||
|
|
||||||
```
|
|
||||||
Boot Manager
|
|
||||||
|
|
||||||
BIOS Boot Menu
|
|
||||||
UEFI Boot Menu ← go here
|
|
||||||
One-shot BIOS Boot Menu
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Arrow to **UEFI Boot Menu** and press **Enter**.
|
|
||||||
|
|
||||||
You will see a list of bootable devices. Look for your USB drive — it will be
|
|
||||||
listed as something like:
|
|
||||||
|
|
||||||
```
|
|
||||||
UEFI: SanDisk Ultra USB 3.0, Partition 1
|
|
||||||
```
|
|
||||||
|
|
||||||
Arrow to it and press **Enter**.
|
|
||||||
|
|
||||||
> **If the USB does not appear in the UEFI Boot Menu:** Make sure Rufus wrote in
|
|
||||||
> DD mode and GPT was selected (Step 5.2). Legacy/MBR USB drives will not appear
|
|
||||||
> in the UEFI menu.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.4 — Proxmox installer boot menu
|
|
||||||
|
|
||||||
The server boots from the USB and shows the Proxmox boot menu — white text on
|
|
||||||
a blue/dark background:
|
|
||||||
|
|
||||||
```
|
|
||||||
Proxmox VE Installer
|
|
||||||
|
|
||||||
Install Proxmox VE (Graphical) ← select this
|
|
||||||
Install Proxmox VE (Terminal UI)
|
|
||||||
Advanced Options
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Arrow to **Install Proxmox VE (Graphical)** and press **Enter**.
|
|
||||||
|
|
||||||
The graphical installer loads. This takes about 30–60 seconds.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.5 — End User License Agreement
|
|
||||||
|
|
||||||
The EULA screen appears. Read it or don't — click **I agree** at the bottom right.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.6 — Target disk selection
|
|
||||||
|
|
||||||
This is the most important screen. You are choosing where Proxmox installs.
|
|
||||||
|
|
||||||
You will see a dropdown labeled **Target Harddisk**. Click it.
|
|
||||||
|
|
||||||
The list shows all visible storage. You are looking for the RAID1 virtual disk
|
|
||||||
you created in Phase 4. It will appear as a single disk — something like:
|
|
||||||
|
|
||||||
```
|
|
||||||
/dev/sda (146.00 GB) ← this is the OS-Mirror RAID1 VD
|
|
||||||
/dev/sdb (4.00 TB)
|
|
||||||
/dev/sdc (4.00 TB)
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Select the small one — your OS-Mirror virtual disk (146 GB or whatever size your
|
|
||||||
2.5" drives are).
|
|
||||||
|
|
||||||
> **Do not select a 3.5" drive.** Those are your VM data drives.
|
|
||||||
|
|
||||||
**Filesystem:** Click the **Options** button next to the disk selector.
|
|
||||||
|
|
||||||
```
|
|
||||||
Filesystem: ext4 ← leave this as ext4
|
|
||||||
```
|
|
||||||
|
|
||||||
Leave it as `ext4`. The H730 RAID1 already gives you drive redundancy. Adding
|
|
||||||
ZFS here would be redundant overhead with no benefit.
|
|
||||||
|
|
||||||
Click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.7 — Location and timezone
|
|
||||||
|
|
||||||
```
|
|
||||||
Country: [type your country, e.g. United States]
|
|
||||||
Time Zone: [auto-filled based on country, e.g. America/New_York]
|
|
||||||
Keyboard: [your keyboard layout, e.g. U.S. English]
|
|
||||||
```
|
|
||||||
|
|
||||||
Adjust if needed. Click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.8 — Password and email
|
|
||||||
|
|
||||||
```
|
|
||||||
Password: [choose a strong root password — write it down]
|
|
||||||
Confirm: [same password again]
|
|
||||||
Email: [any email address — used for system alerts]
|
|
||||||
```
|
|
||||||
|
|
||||||
> The root password is how you log into Proxmox. If you forget it, recovery is
|
|
||||||
> painful. Write it on a piece of paper and put it somewhere safe.
|
|
||||||
|
|
||||||
Click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.9 — Network configuration
|
|
||||||
|
|
||||||
```
|
|
||||||
Management Interface: [auto-selected — usually em1 or eno1, the first NIC]
|
|
||||||
Hostname (FQDN): pve.local
|
|
||||||
IP Address: 192.168.1.10 ← change to an IP on your network
|
|
||||||
Netmask: 255.255.255.0
|
|
||||||
Gateway: 192.168.1.1 ← your router's IP
|
|
||||||
DNS Server: 192.168.1.1 ← your router's IP (or 8.8.8.8)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Choosing a static IP for Proxmox:**
|
|
||||||
- Log into your router and find its DHCP range
|
|
||||||
- Pick an IP outside that range (same subnet)
|
|
||||||
- Example: if DHCP is 192.168.1.100–200, use 192.168.1.10
|
|
||||||
- This IP is how you will reach the Proxmox web UI and SSH from now on
|
|
||||||
|
|
||||||
Click **Next**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.10 — Summary and install
|
|
||||||
|
|
||||||
The installer shows a summary of everything you chose. Confirm:
|
|
||||||
- Target disk: your small RAID1 virtual disk
|
|
||||||
- Filesystem: ext4
|
|
||||||
- Hostname and IP look correct
|
|
||||||
|
|
||||||
Click **Install**.
|
|
||||||
|
|
||||||
The installation takes 5–10 minutes. A progress bar shows the steps:
|
|
||||||
- Formatting disk
|
|
||||||
- Copying files
|
|
||||||
- Setting up bootloader
|
|
||||||
- Configuring system
|
|
||||||
|
|
||||||
When it finishes you will see:
|
|
||||||
|
|
||||||
```
|
|
||||||
Installation successful!
|
|
||||||
Remove the installation medium and press Enter to reboot.
|
|
||||||
```
|
|
||||||
|
|
||||||
**Pull the USB drive out**, then press **Enter**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 5.11 — First login to the Proxmox web UI
|
|
||||||
|
|
||||||
The server reboots. After about 60 seconds it shows a text console with
|
|
||||||
Proxmox's login prompt and this message:
|
|
||||||
|
|
||||||
```
|
|
||||||
Welcome to the Proxmox Virtual Environment.
|
|
||||||
|
|
||||||
Please use your web browser to configure this server -
|
|
||||||
connect to: https://192.168.1.10:8006/
|
|
||||||
```
|
|
||||||
|
|
||||||
On your other computer, open a browser and go to:
|
|
||||||
|
|
||||||
```
|
|
||||||
https://192.168.1.10:8006
|
|
||||||
```
|
|
||||||
|
|
||||||
(Use the IP you set in Step 5.9.)
|
|
||||||
|
|
||||||
**Certificate warning:** Your browser will show a security warning because
|
|
||||||
Proxmox uses a self-signed SSL certificate. This is expected and safe on
|
|
||||||
your local network.
|
|
||||||
|
|
||||||
- Chrome/Edge: click **Advanced** → **Proceed to 192.168.1.10 (unsafe)**
|
|
||||||
- Firefox: click **Advanced** → **Accept the Risk and Continue**
|
|
||||||
|
|
||||||
The Proxmox login page appears — a dark interface with two fields:
|
|
||||||
|
|
||||||
```
|
|
||||||
User name: root
|
|
||||||
Password: [the password you set in Step 5.8]
|
|
||||||
Realm: Linux PAM standard authentication ← leave this as-is
|
|
||||||
```
|
|
||||||
|
|
||||||
Click **Login**.
|
|
||||||
|
|
||||||
**Subscription nag:** A popup immediately appears saying "No valid subscription".
|
|
||||||
Click **OK** to dismiss it. This appears every login until you fix the repos in
|
|
||||||
Phase 6. It does not affect anything.
|
|
||||||
|
|
||||||
You are now looking at the Proxmox web interface — the main dashboard.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 6 — Proxmox Post-Install
|
|
||||||
|
|
||||||
From this point you will use **SSH** instead of the web console for most tasks.
|
|
||||||
SSH is faster and lets you paste commands directly.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 6.1 — Connect via SSH
|
|
||||||
|
|
||||||
On your other computer, open a terminal (Mac/Linux) or PuTTY/Windows Terminal
|
|
||||||
(Windows) and run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh root@192.168.1.10
|
|
||||||
```
|
|
||||||
|
|
||||||
Type `yes` when asked to confirm the host fingerprint. Enter your root password.
|
|
||||||
|
|
||||||
You will see the Proxmox shell prompt:
|
|
||||||
|
|
||||||
```
|
|
||||||
root@pve:~#
|
|
||||||
```
|
|
||||||
|
|
||||||
All the commands in the rest of this guide are typed here.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 6.2 — Fix the package repositories
|
|
||||||
|
|
||||||
Proxmox installs with "enterprise" apt repositories configured. These require
|
|
||||||
a paid subscription and will fail with an authentication error when you try to
|
|
||||||
update. We switch them to the free community repositories.
|
|
||||||
|
|
||||||
Run these commands one at a time:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Disable the enterprise repo (comment it out)
|
|
||||||
echo "# disabled - no subscription" > /etc/apt/sources.list.d/pve-enterprise.list
|
|
||||||
|
|
||||||
# Disable the enterprise Ceph repo
|
|
||||||
echo "# disabled - no subscription" > /etc/apt/sources.list.d/ceph.list
|
|
||||||
|
|
||||||
# Add the no-subscription community repo
|
|
||||||
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
|
|
||||||
> /etc/apt/sources.list.d/pve-no-subscription.list
|
|
||||||
```
|
|
||||||
|
|
||||||
Now update and upgrade all packages:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt update && apt dist-upgrade -y
|
|
||||||
```
|
|
||||||
|
|
||||||
This will download and install updates. It may take 5–15 minutes depending on
|
|
||||||
your internet speed. Answer **Y** if prompted about any config file changes
|
|
||||||
(just press Enter to accept the default).
|
|
||||||
|
|
||||||
When it finishes, reboot:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
Wait about 60 seconds, then SSH back in:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh root@192.168.1.10
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 6.3 — Install the tools the scripts need
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt install -y ipmitool lsscsi ledmon hdparm git curl
|
|
||||||
```
|
|
||||||
|
|
||||||
What each tool does:
|
|
||||||
|
|
||||||
| Tool | Used for |
|
|
||||||
|------|---------|
|
|
||||||
| ipmitool | Fan control — sends IPMI commands to iDRAC |
|
|
||||||
| lsscsi | build-bay-map.sh — lists SCSI drives with details |
|
|
||||||
| ledmon | build-bay-map.sh — blinks drive bay LEDs for identification |
|
|
||||||
| hdparm | stagger-spinup.sh — checks drive power state, wakes drives |
|
|
||||||
| git | Cloning this repo onto the host |
|
|
||||||
| curl | Downloading packages and health checks |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 6.4 — Clone this repo onto the Proxmox host
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/outis1one/local_proxmox.git /opt/local_proxmox
|
|
||||||
chmod +x /opt/local_proxmox/scripts/*.sh
|
|
||||||
cd /opt/local_proxmox
|
|
||||||
```
|
|
||||||
|
|
||||||
All scripts and configs are now at `/opt/local_proxmox/`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 6.5 — Dismiss the subscription warning permanently (optional)
|
|
||||||
|
|
||||||
If the "No valid subscription" popup annoys you every time you log into the
|
|
||||||
web UI, you can remove it with one command. This does not affect functionality:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sed -i.bak "s/data.status !== 'Active'/false/g" \
|
|
||||||
/usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
|
|
||||||
systemctl restart pveproxy
|
|
||||||
```
|
|
||||||
|
|
||||||
After running this, refresh the Proxmox web UI — the popup will be gone.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Phase 6 complete — where you are now
|
|
||||||
|
|
||||||
| What is done | Status |
|
|
||||||
|---|---|
|
|
||||||
| Proxmox VE 9.1 installed on RAID1 OS mirror | ✓ |
|
|
||||||
| Free community repos configured | ✓ |
|
|
||||||
| All packages up to date | ✓ |
|
|
||||||
| Tools installed | ✓ |
|
|
||||||
| Repo cloned to /opt/local_proxmox | ✓ |
|
|
||||||
|
|
||||||
**Next:** Phase 7 — Run `perc-nonraid.sh` to convert the 3.5" drives to
|
|
||||||
non-RAID/per-disk mode, then map the physical bays. That walkthrough is in
|
|
||||||
`docs/walkthrough-phases-7-9.md`.
|
|
||||||
@@ -1,525 +0,0 @@
|
|||||||
# Phases 7–9: Drive Non-RAID, Bay Mapping, GPU Passthrough
|
|
||||||
### Dell R730xd — Button-by-button walkthrough
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 7 — Convert 3.5" Drives to Non-RAID Mode
|
|
||||||
|
|
||||||
**What this does:** Right now the H730 sees your 3.5" drives as "Unconfigured
|
|
||||||
Good" — they exist but the controller is not doing anything with them. We need
|
|
||||||
to put them into **Non-RAID mode**, which tells the H730 to present each drive
|
|
||||||
directly to Linux as an individual block device, with SMART health data intact.
|
|
||||||
|
|
||||||
Without this step, Linux cannot see the drives at all — the H730 hides them.
|
|
||||||
|
|
||||||
**The 2.5" OS drives are not affected** — they are a RAID1 virtual disk and the
|
|
||||||
script only targets unconfigured physical drives.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 7.1 — Download and install perccli
|
|
||||||
|
|
||||||
`perccli` is Dell's command-line tool for managing the H730. The `perc-nonraid.sh`
|
|
||||||
script uses it to talk to the controller.
|
|
||||||
|
|
||||||
SSH into the Proxmox host if you are not already connected:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh root@192.168.1.10
|
|
||||||
```
|
|
||||||
|
|
||||||
Download and install perccli on the Proxmox host. Dell only distributes an
|
|
||||||
RPM, so we convert it with `alien`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Download (referer header required — Dell blocks plain curl)
|
|
||||||
curl -L \
|
|
||||||
--referer "https://www.dell.com/" \
|
|
||||||
--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
|
|
||||||
"https://dl.dell.com/FOLDER03559396M/1/perccli-1.17.10-1.noarch.rpm" \
|
|
||||||
-o /tmp/perccli.rpm
|
|
||||||
|
|
||||||
# Convert RPM → deb and install
|
|
||||||
apt install -y alien
|
|
||||||
alien --to-deb /tmp/perccli.rpm
|
|
||||||
dpkg -i /tmp/perccli_*.deb
|
|
||||||
|
|
||||||
# Binary lands in /opt/MegaRAID/perccli/ — symlink it into PATH
|
|
||||||
ln -s /opt/MegaRAID/perccli/perccli64 /usr/local/bin/perccli
|
|
||||||
```
|
|
||||||
|
|
||||||
Verify it installed:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
perccli show
|
|
||||||
```
|
|
||||||
|
|
||||||
You should see output starting with something like:
|
|
||||||
|
|
||||||
```
|
|
||||||
CLI Version = 007.1907.0000.0000
|
|
||||||
Operating system = Linux5.x
|
|
||||||
Controller = 0
|
|
||||||
Status = Success
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
If you see `command not found`, try `perccli64` instead — some versions install
|
|
||||||
under that name.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 7.2 — Run the non-RAID conversion script
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bash /opt/local_proxmox/scripts/perc-nonraid.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
The script first prints a summary of your controller and all drives:
|
|
||||||
|
|
||||||
```
|
|
||||||
=== Controller overview ===
|
|
||||||
...
|
|
||||||
|
|
||||||
=== Drives on controller 0 ===
|
|
||||||
-------------------------------------------------------------
|
|
||||||
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
|
|
||||||
-------------------------------------------------------------
|
|
||||||
8:0 7 UGood - 3.637 TB SATA HDD N N 512B WD... U
|
|
||||||
8:1 8 UGood - 3.637 TB SATA HDD N N 512B WD... U
|
|
||||||
...
|
|
||||||
8:9 16 Onln 0 136.73 GB SATA SSD N N 512B TOSHIBA U ← OS drive (in VD)
|
|
||||||
8:10 17 Onln 0 136.73 GB SATA SSD N N 512B TOSHIBA U ← OS drive (in VD)
|
|
||||||
```
|
|
||||||
|
|
||||||
The columns that matter:
|
|
||||||
- **EID:Slt** — Enclosure ID and slot number (physical location in the backplane)
|
|
||||||
- **State** — `UGood` means unconfigured and ready. `Onln` means it is part of
|
|
||||||
a virtual disk (your OS mirror). The script only touches `UGood` drives.
|
|
||||||
- **Size** — confirms which are 3.5" data drives vs the small OS drives
|
|
||||||
|
|
||||||
The script then asks:
|
|
||||||
|
|
||||||
```
|
|
||||||
Proceed with converting all non-OS drives to Non-RAID mode? [y/N]
|
|
||||||
```
|
|
||||||
|
|
||||||
Type `y` and press Enter.
|
|
||||||
|
|
||||||
The script works through each drive. For each one you will see:
|
|
||||||
|
|
||||||
```
|
|
||||||
Slot 0: setting to Good... Status = Success
|
|
||||||
Slot 0: setting to Non-RAID... Status = Success
|
|
||||||
Slot 1: setting to Good... Status = Success
|
|
||||||
Slot 1: setting to Non-RAID... Status = Success
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
When all drives are done the script prints the final state — drives that were
|
|
||||||
`UGood` should now show `JBOD` (which is Dell's term for non-RAID/per-disk mode):
|
|
||||||
|
|
||||||
```
|
|
||||||
=== Final drive state ===
|
|
||||||
EID:Slt State
|
|
||||||
8:0 JBOD ←
|
|
||||||
8:1 JBOD ←
|
|
||||||
8:2 JBOD ←
|
|
||||||
...
|
|
||||||
8:9 Onln ← OS drive, untouched
|
|
||||||
8:10 Onln ← OS drive, untouched
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 7.3 — Reboot
|
|
||||||
|
|
||||||
```bash
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
Wait about 60 seconds, then SSH back in:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh root@192.168.1.10
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 7.4 — Verify drives are visible
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk -d -o NAME,SIZE,TYPE,ROTA
|
|
||||||
```
|
|
||||||
|
|
||||||
You should see your drives listed. Something like:
|
|
||||||
|
|
||||||
```
|
|
||||||
NAME SIZE TYPE ROTA
|
|
||||||
sda 136.7G disk 0 ← OS-Mirror RAID1 virtual disk (the 2.5" drives)
|
|
||||||
sdb 3.6T disk 1 ← 3.5" data drive
|
|
||||||
sdc 3.6T disk 1
|
|
||||||
sdd 3.6T disk 1
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
If you see only `sda` and nothing else, the non-RAID conversion did not take
|
|
||||||
effect — reboot again and re-run `lsblk`. If still missing, re-run
|
|
||||||
`perc-nonraid.sh` and check the output for any lines that say `Failed`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 8 — Map Physical Bays to Drives
|
|
||||||
|
|
||||||
**The problem:** Linux names drives `sdb`, `sdc`, `sdd` etc. based on the order
|
|
||||||
it finds them at boot — that order can change. What you need is the **stable
|
|
||||||
hardware ID** (`/dev/disk/by-id/...`) for each drive, tied to its physical bay
|
|
||||||
number, so you can reliably assign "bay 3" to a specific VM forever.
|
|
||||||
|
|
||||||
This phase generates that map.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 8.1 — Run the bay mapping script
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bash /opt/local_proxmox/scripts/build-bay-map.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Output looks like this (truncated example):
|
|
||||||
|
|
||||||
```
|
|
||||||
=== Drive inventory ===
|
|
||||||
|
|
||||||
DEVICE SIZE MODEL SERIAL BY-ID PATH
|
|
||||||
------ ---- ----- ------ ---------
|
|
||||||
/dev/sdb 3.6T WDC_WD4000FYYZ WD-XXXXXXXXXXXX /dev/disk/by-id/scsi-35000cca23b7d4eb8
|
|
||||||
/dev/sdc 3.6T WDC_WD4000FYYZ WD-XXXXXXXXXXXX /dev/disk/by-id/scsi-35000cca23b5e1234
|
|
||||||
/dev/sdd 3.6T ST4000NM0023 Z1Z2XXXXXX /dev/disk/by-id/scsi-35000c500a0000001
|
|
||||||
...
|
|
||||||
|
|
||||||
=== Bay identification via LED blink ===
|
|
||||||
|
|
||||||
To confirm which physical bay a device is in, blink its LED:
|
|
||||||
ledctl locate=/dev/sdX # LED on
|
|
||||||
ledctl locate_off=/dev/sdX # LED off
|
|
||||||
```
|
|
||||||
|
|
||||||
The script gives you the by-id path for each device. Now you need to figure out
|
|
||||||
**which physical bay each device is in**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 8.2 — Walk the bays with LED blink
|
|
||||||
|
|
||||||
This is the physical part. You need to be at the server (or have someone there).
|
|
||||||
|
|
||||||
For each drive, run the blink command, walk to the server, see which bay's amber
|
|
||||||
LED is lit, note the bay number, then turn it off:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Blink sdb
|
|
||||||
ledctl locate=/dev/sdb
|
|
||||||
# Walk to the server, find the lit bay — write down: sdb = bay X
|
|
||||||
ledctl locate_off=/dev/sdb
|
|
||||||
|
|
||||||
# Blink sdc
|
|
||||||
ledctl locate=/dev/sdc
|
|
||||||
# Walk to server, find the lit bay — write down: sdc = bay X
|
|
||||||
ledctl locate_off=/dev/sdc
|
|
||||||
|
|
||||||
# Repeat for each drive
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Bay numbering on the R730xd:** Bays are numbered left to right, top to
|
|
||||||
> bottom when facing the front of the server. Bay 1 is top-left. The exact
|
|
||||||
> labeling depends on your bezel — some models label them 0–11, others 1–12.
|
|
||||||
> Use whatever number is printed or silk-screened on the chassis next to the bay.
|
|
||||||
|
|
||||||
> **If the LED does not blink:** The `ledmon` daemon must be running. Start it:
|
|
||||||
> ```bash
|
|
||||||
> systemctl start ledmon
|
|
||||||
> ledctl locate=/dev/sdb
|
|
||||||
> ```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 8.3 — Fill in the hardware layout document
|
|
||||||
|
|
||||||
Open the layout document:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /opt/local_proxmox/docs/hardware-layout.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Fill in the table using what you noted in Step 8.2. It looks like this — fill
|
|
||||||
in the `by-id` column from the script output and the bay number from the LED walk:
|
|
||||||
|
|
||||||
```
|
|
||||||
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in) |
|
|
||||||
|-----|-------------|------|---------------------------------------------|
|
|
||||||
| 1 | 100 | 4TB | scsi-35000cca23b7d4eb8 |
|
|
||||||
| 2 | 100 | 4TB | scsi-35000cca23b5e1234 |
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Save with **Ctrl+O**, Enter, then **Ctrl+X** to exit nano.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 8.4 — Fill in the VM config placeholders
|
|
||||||
|
|
||||||
Now open each VM config example and replace the `PLACEHOLDER_BAYx` entries with
|
|
||||||
the real by-id paths:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
|
|
||||||
```
|
|
||||||
|
|
||||||
Change lines like:
|
|
||||||
|
|
||||||
```
|
|
||||||
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
|
|
||||||
```
|
|
||||||
|
|
||||||
To the real path:
|
|
||||||
|
|
||||||
```
|
|
||||||
scsi1: /dev/disk/by-id/scsi-35000cca23b7d4eb8,size=0
|
|
||||||
```
|
|
||||||
|
|
||||||
Do this for all 8 drives in VM 100, drives 9–10 in VM 101, and drives 11–12 in
|
|
||||||
VM 102. Save each file.
|
|
||||||
|
|
||||||
> **The full path starts with `/dev/disk/by-id/`** — but in the Proxmox VM
|
|
||||||
> config you write the full path. Double-check by running:
|
|
||||||
> ```bash
|
|
||||||
> ls -la /dev/disk/by-id/ | grep -v part
|
|
||||||
> ```
|
|
||||||
> You will see the symlinks and the drives they point to.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 9 — GPU Passthrough Setup
|
|
||||||
|
|
||||||
**What this does:** Tells the Linux kernel to stop trying to use the two Quadro
|
|
||||||
P2200s itself and instead hand them over to the VFIO driver, which holds them
|
|
||||||
ready to be claimed by a virtual machine.
|
|
||||||
|
|
||||||
Three things happen:
|
|
||||||
1. IOMMU is turned on in the kernel (the hardware feature VT-d enables)
|
|
||||||
2. The host's NVIDIA/nouveau GPU drivers are blocked from loading
|
|
||||||
3. The VFIO driver claims both GPUs at boot, before any other driver can
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 9.1 — Check your IOMMU groups first
|
|
||||||
|
|
||||||
Before running the script, verify that VT-d is actually active:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i iommu | head -10
|
|
||||||
```
|
|
||||||
|
|
||||||
You should see lines like:
|
|
||||||
|
|
||||||
```
|
|
||||||
DMAR: IOMMU enabled
|
|
||||||
Intel-IOMMU: enabled
|
|
||||||
```
|
|
||||||
|
|
||||||
If you see nothing or see "disabled", IOMMU is not active. Go back to Phase 3
|
|
||||||
and re-check that VT-d is set to **Enabled** in BIOS, then reboot and try again.
|
|
||||||
|
|
||||||
Also check that each GPU is in its own IOMMU group:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
|
||||||
n=${d#*/iommu_groups/*}; n=${n%%/*}
|
|
||||||
printf 'IOMMU Group %s ' "$n"
|
|
||||||
lspci -nns "${d##*/}"
|
|
||||||
done | sort -V | grep -i nvidia
|
|
||||||
```
|
|
||||||
|
|
||||||
Example good output — each GPU is in a different group:
|
|
||||||
|
|
||||||
```
|
|
||||||
IOMMU Group 24 03:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
|
|
||||||
IOMMU Group 24 03:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
|
|
||||||
IOMMU Group 31 04:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
|
|
||||||
IOMMU Group 31 04:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
|
|
||||||
```
|
|
||||||
|
|
||||||
Each GPU (`03:00.0` and `04:00.0`) and its audio sibling (`03:00.1` and
|
|
||||||
`04:00.1`) are together in their own group — that is exactly what you want.
|
|
||||||
|
|
||||||
> **If both GPUs are in the same IOMMU group as other devices** (chipset, NICs,
|
|
||||||
> etc.), you may need to enable **ACS** (Access Control Services). This is rare
|
|
||||||
> on server hardware like the R730xd which has good IOMMU separation. If you
|
|
||||||
> hit this, ask before proceeding.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 9.2 — Run the GPU passthrough script
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
The script first prints your NVIDIA devices:
|
|
||||||
|
|
||||||
```
|
|
||||||
=== Detected NVIDIA devices ===
|
|
||||||
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
|
|
||||||
03:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
|
|
||||||
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
|
|
||||||
04:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
|
|
||||||
|
|
||||||
GPU PCI IDs to bind to vfio-pci: 10de:1c35,10de:10f1
|
|
||||||
```
|
|
||||||
|
|
||||||
Then asks:
|
|
||||||
|
|
||||||
```
|
|
||||||
Proceed with configuring VFIO passthrough? [y/N]
|
|
||||||
```
|
|
||||||
|
|
||||||
Type `y` and press Enter.
|
|
||||||
|
|
||||||
The script runs through these steps — you will see each one printed:
|
|
||||||
|
|
||||||
```
|
|
||||||
--- Configuring kernel cmdline for IOMMU ---
|
|
||||||
Written: /etc/kernel/cmdline
|
|
||||||
Boot tool refreshed.
|
|
||||||
|
|
||||||
--- Blacklisting nouveau and nvidia on host ---
|
|
||||||
Written: /etc/modprobe.d/blacklist-gpu.conf
|
|
||||||
|
|
||||||
--- Binding GPU IDs to vfio-pci ---
|
|
||||||
Written: /etc/modprobe.d/vfio.conf
|
|
||||||
IDs: 10de:1c35,10de:10f1
|
|
||||||
|
|
||||||
--- Adding vfio modules to initramfs ---
|
|
||||||
Updated: /etc/initramfs-tools/modules
|
|
||||||
Initramfs updated.
|
|
||||||
```
|
|
||||||
|
|
||||||
At the end it prints the PCI addresses you need:
|
|
||||||
|
|
||||||
```
|
|
||||||
=== GPU PCI addresses for VM assignment ===
|
|
||||||
|
|
||||||
hostpci0: 0000:03:00,pcie=1 # VGA: NVIDIA GP106GL [Quadro P2200]
|
|
||||||
hostpci0: 0000:04:00,pcie=1 # VGA: NVIDIA GP106GL [Quadro P2200]
|
|
||||||
|
|
||||||
IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM.
|
|
||||||
```
|
|
||||||
|
|
||||||
**Write down or copy these addresses** — you need them in Phase 11 when
|
|
||||||
creating the VMs. In this example:
|
|
||||||
- GPU 1 is at `03:00` → goes in VM 100 (Frigate)
|
|
||||||
- GPU 2 is at `04:00` → goes in VM 101
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 9.3 — Update the VM configs with the GPU addresses
|
|
||||||
|
|
||||||
While you have the addresses, add them to the VM configs now:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
|
|
||||||
```
|
|
||||||
|
|
||||||
Find the line:
|
|
||||||
|
|
||||||
```
|
|
||||||
hostpci0: 0000:XX:00,pcie=1,x-vga=1
|
|
||||||
```
|
|
||||||
|
|
||||||
Replace `XX:00` with your GPU 1 address, e.g.:
|
|
||||||
|
|
||||||
```
|
|
||||||
hostpci0: 0000:03:00,pcie=1,x-vga=1
|
|
||||||
```
|
|
||||||
|
|
||||||
Save, then do the same for VM 101:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /opt/local_proxmox/vm-configs/101.conf.example
|
|
||||||
# Change XX:00 to 04:00 (GPU 2)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 9.4 — Reboot
|
|
||||||
|
|
||||||
```bash
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
Wait about 60 seconds, then SSH back in:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh root@192.168.1.10
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Step 9.5 — Verify passthrough is working
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lspci -nnk | grep -A3 -i nvidia
|
|
||||||
```
|
|
||||||
|
|
||||||
For each GPU, look for the driver line. It **must** say `vfio-pci`:
|
|
||||||
|
|
||||||
```
|
|
||||||
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
|
|
||||||
Subsystem: ...
|
|
||||||
Kernel driver in use: vfio-pci ← correct
|
|
||||||
Kernel modules: nouveau
|
|
||||||
|
|
||||||
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
|
|
||||||
Subsystem: ...
|
|
||||||
Kernel driver in use: vfio-pci ← correct
|
|
||||||
Kernel modules: nouveau
|
|
||||||
```
|
|
||||||
|
|
||||||
**If it shows `nouveau` or `nvidia` instead of `vfio-pci`:** The blacklist did
|
|
||||||
not take effect. Run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
update-initramfs -u -k all
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
Then check again. If still wrong, verify the blacklist file exists:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cat /etc/modprobe.d/blacklist-gpu.conf
|
|
||||||
# Should show: blacklist nouveau, blacklist nvidia, etc.
|
|
||||||
|
|
||||||
cat /etc/modprobe.d/vfio.conf
|
|
||||||
# Should show: options vfio-pci ids=10de:1c35,10de:10f1
|
|
||||||
```
|
|
||||||
|
|
||||||
Also confirm IOMMU is in the kernel command line:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cat /etc/kernel/cmdline
|
|
||||||
# Should contain: intel_iommu=on iommu=pt
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Phase 9 complete — where you are now
|
|
||||||
|
|
||||||
| What is done | Status |
|
|
||||||
|---|---|
|
|
||||||
| 3.5" drives in non-RAID mode, visible to Linux | ✓ |
|
|
||||||
| Physical bay → by-id map documented | ✓ |
|
|
||||||
| VM configs updated with real drive paths and GPU addresses | ✓ |
|
|
||||||
| IOMMU active, both GPUs claimed by vfio-pci | ✓ |
|
|
||||||
|
|
||||||
**Next:** Phase 10 — Install the fan control service (stops iDRAC from running
|
|
||||||
fans at 100% because of the non-Dell GPUs) and the stagger spin-up service.
|
|
||||||
That walkthrough is in `docs/walkthrough-phases-10-12.md`.
|
|
||||||
Reference in New Issue
Block a user