local_proxmox/docs/walkthrough-phases-7-9.md

# Phases 7–9: Drive Non-RAID, Bay Mapping, GPU Passthrough
### Dell R730xd — Button-by-button walkthrough

---

## Phase 7 — Convert 3.5" Drives to Non-RAID Mode

**What this does:** Right now the H730 sees your 3.5" drives as "Unconfigured
Good" — they exist but the controller is not doing anything with them. We need
to put them into **Non-RAID mode**, which tells the H730 to present each drive
directly to Linux as an individual block device, with SMART health data intact.

Without this step, Linux cannot see the drives at all — the H730 hides them.

**The 2.5" OS drives are not affected** — they are a RAID1 virtual disk and the
script only targets unconfigured physical drives.

---

### Step 7.1 — Download and install perccli

`perccli` is Dell's command-line tool for managing the H730. The `perc-nonraid.sh`
script uses it to talk to the controller.

SSH into the Proxmox host if you are not already connected:

```bash
ssh root@192.168.1.10
```

Download perccli from Dell. On the Proxmox host, run:

```bash
cd /tmp

# Download directly from Dell (this URL is stable — it redirects to the latest version)
curl -L "https://dl.dell.com/FOLDER06708442M/1/perccli_7.1-007.0318_linux.tar.gz" \
  -o perccli.tar.gz
```

> **If that URL does not work** (Dell occasionally moves files):
> On your other computer go to **https://www.dell.com/support/home**, search
> for your Service Tag, click **Drivers & Downloads**, filter Category to
> **Systems Management**, and search for "PERCCLI". Download the Linux version.
> Copy the `.tar.gz` or `.deb` file to the Proxmox host with:
> ```bash
> scp perccli_*.tar.gz root@192.168.1.10:/tmp/
> ```

Extract and install:

```bash
cd /tmp
tar xzf perccli.tar.gz
# The .deb file is inside the extracted folder:
find /tmp -name "*.deb" | head -5
# You will see something like: /tmp/perccli_007.1907.0000.0000_amd64.deb
dpkg -i /tmp/perccli_*.deb
```

Verify it installed:

```bash
perccli show
```

You should see output starting with something like:

```
CLI Version = 007.1907.0000.0000
Operating system = Linux5.x
Controller = 0
Status = Success
...
```

If you see `command not found`, try `perccli64` instead — some versions install
under that name.

---

### Step 7.2 — Run the non-RAID conversion script

```bash
bash /opt/local_proxmox/scripts/perc-nonraid.sh
```

The script first prints a summary of your controller and all drives:

```
=== Controller overview ===
...

=== Drives on controller 0 ===
-------------------------------------------------------------
EID:Slt DID State DG      Size Intf Med SED PI SeSz Model   Sp
-------------------------------------------------------------
8:0      7   UGood -  3.637 TB SATA HDD N   N  512B WD...   U
8:1      8   UGood -  3.637 TB SATA HDD N   N  512B WD...   U
...
8:9      16  Onln  0  136.73 GB SATA SSD N   N  512B TOSHIBA U   ← OS drive (in VD)
8:10     17  Onln  0  136.73 GB SATA SSD N   N  512B TOSHIBA U   ← OS drive (in VD)
```

The columns that matter:
- **EID:Slt** — Enclosure ID and slot number (physical location in the backplane)
- **State** — `UGood` means unconfigured and ready. `Onln` means it is part of
  a virtual disk (your OS mirror). The script only touches `UGood` drives.
- **Size** — confirms which are 3.5" data drives vs the small OS drives

The script then asks:

```
Proceed with converting all non-OS drives to Non-RAID mode? [y/N]
```

Type `y` and press Enter.

The script works through each drive. For each one you will see:

```
  Slot 0: setting to Good...   Status = Success
  Slot 0: setting to Non-RAID... Status = Success
  Slot 1: setting to Good...   Status = Success
  Slot 1: setting to Non-RAID... Status = Success
  ...
```

When all drives are done the script prints the final state — drives that were
`UGood` should now show `JBOD` (which is Dell's term for non-RAID/per-disk mode):

```
=== Final drive state ===
EID:Slt  State
8:0      JBOD    ← 
8:1      JBOD    ← 
8:2      JBOD    ← 
...
8:9      Onln    ← OS drive, untouched
8:10     Onln    ← OS drive, untouched
```

---

### Step 7.3 — Reboot

```bash
reboot
```

Wait about 60 seconds, then SSH back in:

```bash
ssh root@192.168.1.10
```

### Step 7.4 — Verify drives are visible

```bash
lsblk -d -o NAME,SIZE,TYPE,ROTA
```

You should see your drives listed. Something like:

```
NAME    SIZE TYPE ROTA
sda   136.7G disk    0   ← OS-Mirror RAID1 virtual disk (the 2.5" drives)
sdb     3.6T disk    1   ← 3.5" data drive
sdc     3.6T disk    1
sdd     3.6T disk    1
...
```

If you see only `sda` and nothing else, the non-RAID conversion did not take
effect — reboot again and re-run `lsblk`. If still missing, re-run
`perc-nonraid.sh` and check the output for any lines that say `Failed`.

---

## Phase 8 — Map Physical Bays to Drives

**The problem:** Linux names drives `sdb`, `sdc`, `sdd` etc. based on the order
it finds them at boot — that order can change. What you need is the **stable
hardware ID** (`/dev/disk/by-id/...`) for each drive, tied to its physical bay
number, so you can reliably assign "bay 3" to a specific VM forever.

This phase generates that map.

---

### Step 8.1 — Run the bay mapping script

```bash
bash /opt/local_proxmox/scripts/build-bay-map.sh
```

Output looks like this (truncated example):

```
=== Drive inventory ===

DEVICE       SIZE       MODEL                          SERIAL               BY-ID PATH
------       ----       -----                          ------               ---------
/dev/sdb     3.6T       WDC_WD4000FYYZ                WD-XXXXXXXXXXXX      /dev/disk/by-id/scsi-35000cca23b7d4eb8
/dev/sdc     3.6T       WDC_WD4000FYYZ                WD-XXXXXXXXXXXX      /dev/disk/by-id/scsi-35000cca23b5e1234
/dev/sdd     3.6T       ST4000NM0023                  Z1Z2XXXXXX           /dev/disk/by-id/scsi-35000c500a0000001
...

=== Bay identification via LED blink ===

To confirm which physical bay a device is in, blink its LED:
  ledctl locate=/dev/sdX      # LED on
  ledctl locate_off=/dev/sdX  # LED off
```

The script gives you the by-id path for each device. Now you need to figure out
**which physical bay each device is in**.

---

### Step 8.2 — Walk the bays with LED blink

This is the physical part. You need to be at the server (or have someone there).

For each drive, run the blink command, walk to the server, see which bay's amber
LED is lit, note the bay number, then turn it off:

```bash
# Blink sdb
ledctl locate=/dev/sdb
# Walk to the server, find the lit bay — write down: sdb = bay X
ledctl locate_off=/dev/sdb

# Blink sdc
ledctl locate=/dev/sdc
# Walk to server, find the lit bay — write down: sdc = bay X
ledctl locate_off=/dev/sdc

# Repeat for each drive
```

> **Bay numbering on the R730xd:** Bays are numbered left to right, top to
> bottom when facing the front of the server. Bay 1 is top-left. The exact
> labeling depends on your bezel — some models label them 0–11, others 1–12.
> Use whatever number is printed or silk-screened on the chassis next to the bay.

> **If the LED does not blink:** The `ledmon` daemon must be running. Start it:
> ```bash
> systemctl start ledmon
> ledctl locate=/dev/sdb
> ```

---

### Step 8.3 — Fill in the hardware layout document

Open the layout document:

```bash
nano /opt/local_proxmox/docs/hardware-layout.md
```

Fill in the table using what you noted in Step 8.2. It looks like this — fill
in the `by-id` column from the script output and the bay number from the LED walk:

```
| Bay | Assigned VM | Size | /dev/disk/by-id (fill in)                  |
|-----|-------------|------|---------------------------------------------|
| 1   | 100         | 4TB  | scsi-35000cca23b7d4eb8                      |
| 2   | 100         | 4TB  | scsi-35000cca23b5e1234                      |
...
```

Save with **Ctrl+O**, Enter, then **Ctrl+X** to exit nano.

---

### Step 8.4 — Fill in the VM config placeholders

Now open each VM config example and replace the `PLACEHOLDER_BAYx` entries with
the real by-id paths:

```bash
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
```

Change lines like:

```
scsi1: /dev/disk/by-id/PLACEHOLDER_BAY1,size=0
```

To the real path:

```
scsi1: /dev/disk/by-id/scsi-35000cca23b7d4eb8,size=0
```

Do this for all 8 drives in VM 100, drives 9–10 in VM 101, and drives 11–12 in
VM 102. Save each file.

> **The full path starts with `/dev/disk/by-id/`** — but in the Proxmox VM
> config you write the full path. Double-check by running:
> ```bash
> ls -la /dev/disk/by-id/ | grep -v part
> ```
> You will see the symlinks and the drives they point to.

---

## Phase 9 — GPU Passthrough Setup

**What this does:** Tells the Linux kernel to stop trying to use the two Quadro
P2200s itself and instead hand them over to the VFIO driver, which holds them
ready to be claimed by a virtual machine.

Three things happen:
1. IOMMU is turned on in the kernel (the hardware feature VT-d enables)
2. The host's NVIDIA/nouveau GPU drivers are blocked from loading
3. The VFIO driver claims both GPUs at boot, before any other driver can

---

### Step 9.1 — Check your IOMMU groups first

Before running the script, verify that VT-d is actually active:

```bash
dmesg | grep -i iommu | head -10
```

You should see lines like:

```
DMAR: IOMMU enabled
Intel-IOMMU: enabled
```

If you see nothing or see "disabled", IOMMU is not active. Go back to Phase 3
and re-check that VT-d is set to **Enabled** in BIOS, then reboot and try again.

Also check that each GPU is in its own IOMMU group:

```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done | sort -V | grep -i nvidia
```

Example good output — each GPU is in a different group:

```
IOMMU Group 24  03:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
IOMMU Group 24  03:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
IOMMU Group 31  04:00.0 VGA compatible controller [0300]: NVIDIA GP106GL [Quadro P2200] [10de:1c35]
IOMMU Group 31  04:00.1 Audio device [0403]: NVIDIA GP106 High Definition Audio [10de:10f1]
```

Each GPU (`03:00.0` and `04:00.0`) and its audio sibling (`03:00.1` and
`04:00.1`) are together in their own group — that is exactly what you want.

> **If both GPUs are in the same IOMMU group as other devices** (chipset, NICs,
> etc.), you may need to enable **ACS** (Access Control Services). This is rare
> on server hardware like the R730xd which has good IOMMU separation. If you
> hit this, ask before proceeding.

---

### Step 9.2 — Run the GPU passthrough script

```bash
bash /opt/local_proxmox/scripts/gpu-passthrough-setup.sh
```

The script first prints your NVIDIA devices:

```
=== Detected NVIDIA devices ===
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
03:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio
04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
04:00.1 Audio device [10de:10f1]: NVIDIA GP106 High Definition Audio

GPU PCI IDs to bind to vfio-pci: 10de:1c35,10de:10f1
```

Then asks:

```
Proceed with configuring VFIO passthrough? [y/N]
```

Type `y` and press Enter.

The script runs through these steps — you will see each one printed:

```
--- Configuring kernel cmdline for IOMMU ---
Written: /etc/kernel/cmdline
Boot tool refreshed.

--- Blacklisting nouveau and nvidia on host ---
Written: /etc/modprobe.d/blacklist-gpu.conf

--- Binding GPU IDs to vfio-pci ---
Written: /etc/modprobe.d/vfio.conf
  IDs: 10de:1c35,10de:10f1

--- Adding vfio modules to initramfs ---
Updated: /etc/initramfs-tools/modules
Initramfs updated.
```

At the end it prints the PCI addresses you need:

```
=== GPU PCI addresses for VM assignment ===

  hostpci0: 0000:03:00,pcie=1   # VGA: NVIDIA GP106GL [Quadro P2200]
  hostpci0: 0000:04:00,pcie=1   # VGA: NVIDIA GP106GL [Quadro P2200]

IMPORTANT: Pass each GPU + its HDMI audio sibling to the same VM.
```

**Write down or copy these addresses** — you need them in Phase 11 when
creating the VMs. In this example:
- GPU 1 is at `03:00` → goes in VM 100 (Frigate)
- GPU 2 is at `04:00` → goes in VM 101

---

### Step 9.3 — Update the VM configs with the GPU addresses

While you have the addresses, add them to the VM configs now:

```bash
nano /opt/local_proxmox/vm-configs/100-frigate.conf.example
```

Find the line:

```
hostpci0: 0000:XX:00,pcie=1,x-vga=1
```

Replace `XX:00` with your GPU 1 address, e.g.:

```
hostpci0: 0000:03:00,pcie=1,x-vga=1
```

Save, then do the same for VM 101:

```bash
nano /opt/local_proxmox/vm-configs/101.conf.example
# Change XX:00 to 04:00 (GPU 2)
```

---

### Step 9.4 — Reboot

```bash
reboot
```

Wait about 60 seconds, then SSH back in:

```bash
ssh root@192.168.1.10
```

---

### Step 9.5 — Verify passthrough is working

```bash
lspci -nnk | grep -A3 -i nvidia
```

For each GPU, look for the driver line. It **must** say `vfio-pci`:

```
03:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
        Subsystem: ...
        Kernel driver in use: vfio-pci     ← correct
        Kernel modules: nouveau

04:00.0 VGA compatible controller [10de:1c35]: NVIDIA GP106GL [Quadro P2200]
        Subsystem: ...
        Kernel driver in use: vfio-pci     ← correct
        Kernel modules: nouveau
```

**If it shows `nouveau` or `nvidia` instead of `vfio-pci`:** The blacklist did
not take effect. Run:

```bash
update-initramfs -u -k all
reboot
```

Then check again. If still wrong, verify the blacklist file exists:

```bash
cat /etc/modprobe.d/blacklist-gpu.conf
# Should show: blacklist nouveau, blacklist nvidia, etc.

cat /etc/modprobe.d/vfio.conf
# Should show: options vfio-pci ids=10de:1c35,10de:10f1
```

Also confirm IOMMU is in the kernel command line:

```bash
cat /etc/kernel/cmdline
# Should contain: intel_iommu=on iommu=pt
```

---

### Phase 9 complete — where you are now

| What is done | Status |
|---|---|
| 3.5" drives in non-RAID mode, visible to Linux | ✓ |
| Physical bay → by-id map documented | ✓ |
| VM configs updated with real drive paths and GPU addresses | ✓ |
| IOMMU active, both GPUs claimed by vfio-pci | ✓ |

**Next:** Phase 10 — Install the fan control service (stops iDRAC from running
fans at 100% because of the non-Dell GPUs) and the stagger spin-up service.
That walkthrough is in `docs/walkthrough-phases-10-12.md`.