PVE 创建 Ubuntu 虚拟机

概览：准备 ISO、安装 Ubuntu、加大容量数据盘前置：已经有 vmdata 的 LVM-Thin （需要正确选择它来放系统盘）

一、准备 Ubuntu 22.04 ISO 镜像

在自己电脑中下载 Ubuntu 22.04 LTS：
- Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  - https://releases.ubuntu.com/jammy/
- Desktop image
  - https://releases.ubuntu.com/jammy/ubuntu-22.04.5-desktop-amd64.iso
把下载好的 ISO 上传到 PVE 的 local 存储：
- 在 PVE 网页左侧点击 Datacenter → 节点 pve → 选中 local (pve)
- 点击选项卡 ISO Images → 点击左上角 Upload
- Select File → 选择刚下载的 ubuntu-22.04.5-desktop-amd64.iso
- 点击 Upload，等待上传完成
- 传完后在 ISO Images 列表里可以看到这个 ISO 文件
- 也可以用 Download from URL，但现在有 ISO 可以直接用 Upload
- iso 文件默认存储在：/var/lib/vz/template/iso

二、创建 Ubuntu 22.04 虚拟机

要求：系统盘放在 vmdata 上。

在左侧 Server View 的树状结构中，选中节点 pve
右上角点击 Create VM

1. General 选项卡

Node：默认就是 pve
VM ID：默认是 101（保证不和现有的冲突）
Name：比如 AI-122
- AI 表示用途
- 122 表示后面要在内部分配的 IP 后缀
其他保持默认，点 Next

2. OS 选项卡

勾选 Use CD/DVD disc image file (iso)
- Storage：选 local（就是刚刚上传 ISO 的存储）
- ISO image：选 ubuntu-22.04.5-desktop-amd64.iso
- Guest OS：
  - Type 选 Linux
  - Version：选 6.x - 2.6 Kernel 或 Ubuntu（有的话）
点 Next

3. System 选项卡

这里主要是启动方式和控制器。

Graphic card：默认 Default
Machine：选择 q35
- q35 模拟更新的 Intel Q35 芯片组，支持 GPU 直通、NVMe 直通，多种 PCIe 设备拓扑时更好
- i440fx 兼容性比较好，但是不支持原生 PCIe 拓扑，做直通和现代设备比较麻烦
BIOS：
- 推荐 OVMF (UEFI)，方便以后用 UEFI；
- 如果你有特别要求，也可以保留 SeaBIOS。
SCSI Controller：选 VirtIO SCSI single（性能好，也是官方推荐）
勾选 QEMU Agent（非常推荐，后面方便看到 IP、优雅关机等）
- 后续可能还需要在 Ubuntu 中运行如下命令：
  sh
```
sudo apt update
sudo apt install qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent
```
点 Next

4. Disks 选项卡

关键：把系统盘放到 vmdata（3.84TB SSD）

Bus/Device：选 SCSI。
Storage：选择 vmdata
- 这样这个虚拟磁盘会建在 /dev/nvme1n1 上的 LVM-Thin 里
Disk size：按需设置，比如 2048 GB
- 将来可以扩大，但是不能缩小
- 扩容还是有点麻烦的，需要做分区和扩容操作，尽量第一次就给够
勾选 Discard：TRIM/UNMAP，当虚拟机中删除文件时，PVE 可以把释放的空间真正还给底层 LVM-Thin
Cache：默认 No cache，稳定安全
点开 Advanced
- 勾选 SSD emulation：可以让虚拟机识别这块盘是 SSD，可以优化性能
点 Next

5. CPU 选项卡

Sockets：默认为 1
- 对于 Linux，绝大多数情况下，1 个 Socket + 多个 Core 就可以了
Cores：
- 用如下命令查看宿主机 CPU 核心数：
sh
```
egrep '^processor' /proc/cpuinfo | sort -u | wc -l
```
- 假设上面命令输出为 104，可以考虑给个 32
Type：
- 推荐选 host：性能好，尤其是目前是单节点用 PVE，又是较新的 Ubuntu 22.04
- 如果以后有多节点、要做热迁移，可以选 x86-64-v2-AES，兼容性更好
点开 Advanced：
- vCPUs：保持默认即可（可以删除掉值，会自动计算为 Sockets × Cores）
- NUMA：不用开，不必引入复杂性
点 Next

6. Memory 选项卡

Memory (MiB)：比如设为 1048576 (1TB)
点开 Advanced：
- Ballooning：默认勾选
- Allow KSM：默认勾选
点 Next

大内存 VM 的默认值经验

如果 VM 类似 bj123，主要用于 GPU/AI 任务，可以给较大的内存，但不一定要把宿主机内存几乎全部分给 VM。

经验值：

默认建议：98304 MiB，即 96 GiB。
需要更大内存任务时再临时调大，例如 120 GiB。
大内存 VM 启动时 QEMU 初始化和 guest 内存上报会更慢；从 120 GiB 调回 96 GiB 可以缩短启动等待。
宿主机应保留足够内存给 PVE、存储、网络和日志服务，避免管理面在 VM 压力下不可用。

命令行修改示例：

qm set <VM_ID> --memory 98304

如果 VM 正在运行，配置会写入 PVE；是否立即影响运行态取决于当前 balloon/hotplug 设置。最稳妥的判断方式是下一次重启 VM 后检查：

qm config <VM_ID> | grep '^memory:'
qm status <VM_ID> --verbose | grep -E '^(maxmem|mem|freemem):'

7. Network 选项卡

Bridge：选择默认的即可
- 默认是已经配置好的 Linux Bridge，比如左边树结构中的 localnetwork (pve) 一般对应 vmbr0
Model：选 VirtIO (paravirtualized)，性能更好
点 Next

8. Confirm 选项卡

检查一下：
- ide2 是 local:iso/ubuntu-22.04.5-desktop-amd64.iso,media=cdrom
- efidisk0 是 vmdata
确认无误后，点 Finish 创建 VM

此时 VM 已经建好，但系统还没装。

三、在 VM 里安装 Ubuntu 22.04

在左侧的树结构中，选择刚创建的 VM（比如 101 (AI-122)）
点击上方 Start 启动
点击 Console 下拉列表，选择 noVNC 打开控制台
会从 ISO 启动进入 Ubuntu 安装界面：
- 如果电脑的分辨率不够，需要滚动右侧的滚动条来看下方的选项
- 语言选择 English，点击 Install Ubuntu
- 选择键盘布局 English (US)
- 选择 Normal installation
- 取消勾选 Download updates while installing Ubuntu（后面可以手动更新）
- 分区时选 Erase disk and install Ubuntu：
  - 这里看到的“磁盘”是刚刚在 vmdata 上创建的虚拟磁盘，不是宿主机的物理盘，放心选
- 选择时区：Shanghai
- 设置主机名、用户、密码等
  - Your Name：
  - Your computer's name：ai122
  - Pick a username：
  - Choose a password：
  - Confirm your password：
  - 勾选 Log in automatically（方便使用）
- 等待安装完成
- 可选：勾选安装 OpenSSH Server（以后方便用 SSH 登入）
- 等待安装完成，点击 Restart Now 重启
- 此时会提示 Please remove the installation medium, then press ENTER
移除 ISO / 调整启动顺序（避免下次还从光驱启动）
- 在 PVE 左侧选中该 VM → 点击选项卡 Hardware
- 找到 CD/DVD Drive → Edit：
  - 选 Do not use any media，点 OK
- 然后点击选项卡 Options → Boot Order → Edit：
  - 确保 scsi0（系统盘）排在第一行
- 回到 Console 的那个提示界面，回车，应该就可以直接从安装好的系统启动
登录系统，点击 Activities → 搜索 Terminal 打开终端，并且添加到 Favorites 方便以后打开
点击右上角电源图标，选择 Settings
- Appearance → 选择 Dark 主题
- Power → Power Saving Options
  - Screen Blank 设为 Never
  - Automatic suspend 设为 Off（避免虚拟机自动休眠）
- Network → Wired → 点击 Connected 右边的设置图标 → IPv4：
  - IPv4 Method 选 Manual
  - Addresses：
    - Address：192.168.31.122
    - Netmask：255.255.255.0
    - Gateway：192.168.31.1
  - DNS：取消勾选 Automatic，填入 192.168.31.1
  - 点击 Apply 保存
  - 重启以使得静态 IP 地址设置生效
- 或者在命令行中修改网络：参考 Ubuntu 设置静态 IP
  sh
```
# 查看连接名称
nmcli connection show

# 假如输出 Name: Wired connection 2 (DEVICE：enp10s18)
CONN_NAME="Wired connection 2"

# 设置静态 IP
sudo nmcli connection modify "$CONN_NAME" ipv4.addresses 192.168.31.122/24 ipv4.gateway 192.168.31.1 ipv4.dns 192.168.31.1 ipv4.method manual

# 重启连接以使设置生效
sudo nmcli connection down "$CONN_NAME" && sudo nmcli connection up "$CONN_NAME"
```

三a、软件环境配置

换源

参考：Ubuntu 换国内源

USTC Mirror Help

https://mirrors.ustc.edu.cn/help/ubuntu.html

sudo sed -i 's@//.*archive.ubuntu.com@//mirrors.ustc.edu.cn@g' /etc/apt/sources.list

# 一般不建议替换 security 源
# 镜像站同步有延迟，可能会导致生产环境不能及时安装上最新的安全更新
sudo sed -i 's/security.ubuntu.com/mirrors.ustc.edu.cn/g' /etc/apt/sources.list

# 使用 HTTPS 避免运营商缓存劫持
sudo sed -i 's/http:/https:/g' /etc/apt/sources.list

更新软件包列表：

sudo apt update

开启SSH

参考：Ubuntu 开启 SSH服务

安装：

sudo apt install openssh-server

启动：

sudo systemctl enable ssh --now

查看服务状态：

sudo systemctl status ssh

之后就可以通过 SSH 登录这台虚拟机了。

安装 tailscale

参考：使用 Tailscale 组网

之后就可以通过 Tailscale 远程访问这台虚拟机了。

安装 tmux

参考：安装 tmux

tmux-resurrect 恢复失败排查

在 bj123 这类纯 SSH/算力 VM 中，常用下面命令在登录后恢复 tmux：

tmux has-session 2>/dev/null || (tmux new-session -d && tmux run-shell ~/.tmux/plugins/tmux-resurrect/scripts/restore.sh)

如果预期是恢复上一次保存的所有 session/window/pane，但实际没有恢复，按下面顺序查：

tmux ls
ls -la ~/.tmux/plugins/tmux-resurrect/scripts/
ls -la ~/.local/share/tmux/resurrect/
readlink -f ~/.local/share/tmux/resurrect/last
sed -n '1,220p' ~/.tmux.conf

重点看两类问题：

last 指向的保存文件是否本身就不完整。
- tmux-resurrect 只会从 ~/.local/share/tmux/resurrect/last 指向的文件恢复。
- 如果某次崩溃后只保存了一个空 session 或很少的 pane，last 可能已经被覆盖成缩水版本。
- 可以查看旧文件，选择最近一个完整文件重新指向：

cd ~/.local/share/tmux/resurrect
ls -lh tmux_resurrect_*.txt
ln -sfn tmux_resurrect_<TIMESTAMP>.txt last

.tmux.conf 中是否有会破坏恢复流程的 hook。
- tmux-resurrect 自己会处理从空 session 0 恢复时的清理。
- 不要额外添加类似下面的 hook：

tmux

set -g @resurrect-hook-pre-restore-pane-processes 'tmux kill-session -t=0 2>/dev/null || true'

这类 hook 会在 pane 恢复后、pane process 恢复前杀掉 session 0。而 tmux new-session -d 默认创建的 session 往往就是 0，结果可能是唯一 tmux server 被杀掉，恢复过程提前中断。

修复后可验证：

tmux kill-server 2>/dev/null || true
tmux has-session 2>/dev/null || (tmux new-session -d && tmux run-shell ~/.tmux/plugins/tmux-resurrect/scripts/restore.sh)
sleep 3
tmux ls
tmux list-windows -a
tmux list-panes -a -F 'session=#{session_name} window=#{window_index} pane=#{pane_index} path=#{pane_current_path} cmd=#{pane_current_command}'

四、在 Ubuntu 内安装 QEMU Guest Agent（建议）

如果在创建 VM 时已经勾了 QEMU Guest Agent，现在只需要在 VM 里安装软件。

在 PVE 里确认选项：
- 选中 VM （AI-122）→ Options → QEMU Guest Agent
- 确保状态为 Enabled，如果不是就 Edit 勾上

在 Ubuntu 里执行（通过 Console 或 SSH）：

bash

sudo apt update
sudo apt install qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

稍等几秒，在 PVE 中，选择 VM（比如 ai122）的 Summary 选项卡，就能看到 IP 等信息自动显示

五、启用显卡直通，并将分配给 VM

在 PVE 9 上启用 IOMMU

nano /etc/kernel/cmdline

添加：

intel_iommu=on iommu=pt

运行：

proxmox-boot-tool refresh

重启 PVE：

reboot

确认 IOMMU 状态：

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi | grep -i ioomu

如果输出中看到类似 IOMMU enabled 的内容，就说明 IOMMU 启用成功。

输出形如：

[    9.191140] DMAR-IR: IOAPIC id 12 under DRHD base  0xc5ffc000 IOMMU 6
[    9.191142] DMAR-IR: IOAPIC id 11 under DRHD base  0xb87fc000 IOMMU 5
[    9.191144] DMAR-IR: IOAPIC id 10 under DRHD base  0xaaffc000 IOMMU 4
[    9.191146] DMAR-IR: IOAPIC id 18 under DRHD base  0xfbffc000 IOMMU 3
[    9.191147] DMAR-IR: IOAPIC id 17 under DRHD base  0xee7fc000 IOMMU 2
[    9.191149] DMAR-IR: IOAPIC id 16 under DRHD base  0xe0ffc000 IOMMU 1
[    9.191151] DMAR-IR: IOAPIC id 15 under DRHD base  0xd37fc000 IOMMU 0
[    9.191152] DMAR-IR: IOAPIC id  8 under DRHD base  0x9d7fc000 IOMMU 7
[    9.191154] DMAR-IR: IOAPIC id  9 under DRHD base  0x9d7fc000 IOMMU 7

加载 VFIO 模块

nano /etc/modules

在末尾添加：

vfio
vfio_pci
vfio_iommu_type1
vfio_virqfd

黑名单宿主机显卡驱动

WARNING

注意：如果宿主机还需要用某块卡输出图形，就不要把那一块卡对应的驱动全黑名单。理想情况是宿主机用主板自带 iGPU 或 IPMI，把 8 块独显全部给 VM。

对于 NVIDIA 显卡：

echo "blacklist nouveau"  > /etc/modprobe.d/blacklist-nouveau.conf
echo "blacklist nvidia"   > /etc/modprobe.d/blacklist-nvidia.conf
echo "blacklist nvidiafb" > /etc/modprobe.d/blacklist-nvidiafb.conf

将显卡全部绑定到 vfio-pci

查看显卡列表：

lspci -nn | grep -E "VGA|3D|Display"

输出形如：

03:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
1a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
1b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
3d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
3e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
88:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
89:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
b1:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
b2:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
d8:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe DC SSD [3DNAND, Sentinel Rock Controller] [8086:0b60]
d9:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe DC SSD [3DNAND, Sentinel Rock Controller] [8086:0b60]

方括号里的 10de:2206 就是 vendor:device ID。
对同型号的 8 块卡，这个 ID 往往都是一样的。

# 很多显卡有独立的音频功能
lspci -nn | grep -i audio

输出形如：

1a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
1b:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
3d:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
3e:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
88:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
89:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
b1:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
b2:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)

这里的 10de:1aef 就是音频部分的 vendor:device ID。

查看 IOMMU 组：

find /sys/kernel/iommu_groups/ -type l

确认没有和别的重要设备绑在一起
因为是“8 块卡都给同一台 VM”，即便多块卡在同一个 IOMMU 组里，问题也不大
只要组里别混着 SATA 控制器之类宿主机必须用的设备

将这些卡全部绑定到 vfio-pci：

nano /etc/modprobe.d/vfio.conf

添加：

options vfio-pci ids=10de:2206,10de:1aef

重新生成 initramfs：

update-initramfs -u

重启：

reboot

重启之后确认每块 GPU 已经绑定到 vfio-pci：

# lspci -nnk | grep -A3 -E "VGA|3D|Display"
lspci -nnk | grep -A3 -E "NVIDIA" | grep -i kernel

输出形如：

Kernel driver in use: vfio-pci
# Kernel modules: nvidiafb, nouveau
# Kernel modules: snd_hda_intel

这就说明宿主机已经把显卡让出来了。

将显卡全部直通给 VM

现在开始在 PVE 的 Web 界面操作。

左侧树结构选中 VM （比如 ai122）→ Hardware 选项卡：

确认 BIOS: OVMF (UEFI)
确认 Machine: q35
点击 Add 下拉列表 → 选择 PCI Device
选择 Raw Device，点击出现列表，点选 IOMMU Group 正向排序（一般 GPU 都排在靠前的组）
在列表里选第一块 GPU
- 同一块 GPU 通常会有一个 VGA + 一个 Audio
- 勾选 All Functions，让 PVE 自动把同一张卡的所有函数一起直通
- 不勾 Primary GPU：
  - 如果只是算力卡，用远程 SSH，不用勾选
  - 如果想用这块卡做虚拟机的显示输出（接显示器），可以在其中一块卡上勾
- 点开 Advanced
  - 勾选 PCI-Express（q35 + 现代 GPU）
- 确认无误，点击 Add
- 添加好后，可以看到信息栏 PCI Device 多了一条记录，类似 0000:3d:00,pcie=1
重复上一步，把剩下的 7 块 GPU 都按同样方式加进来：
- 每次 Add → PCI Device，选不同的 GPU / IOMMU 组。
- 如果某几块卡在同一个 IOMMU 组里，PVE 会强制你把整个组都直通过去，这对“全给 ai122”来说是OK的
- 可以先加 1 块卡，确认没问题后再加剩下的

命令行一键直通

参考：SLOT 和 GPU 对应关系

清空旧的 hostpci 0-7：

for i in {0..7}; do qm set 101 -delete hostpci$i; done

设置新的 hostpci 0-7：

buses=(88 89 b1 b2 3d 3e 1a 1b); args=()
for i in "${!buses[@]}"; do args+=("-hostpci$i" "0000:${buses[$i]}:00,pcie=1"); done
qm set 101 "${args[@]}"

查看 VM 当前 PCI 设备：

qm config 101 | grep -E '^hostpci'

常见问题：`0 <= irq_num && irq_num < PCI_NUM_PINS`

问题详情：

kvm: ../hw/pci/pci.c:1815: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
TASK ERROR: start failed: QEMU exited with code 1

原因一般是掉卡。临时方案：

解决方法（暂时无效）

解决方法：禁用上游端口省电。

nano /etc/kernel/cmdline

添加如下内容：

pcie_port_pm=off pcie_aspm=off vfio-pci.disable_idle_d3=1

pcie_port_pm=off: 禁止 PCIe ports runtime PM（一般用于解决 device inaccessible）
pcie_aspm=off: 关闭链路 ASPM（PLX/switch/riser 很多时需要该参数保证稳定）
vfio-pci.disable_idle_d3=1: 不让 VFIO 管的设备在 idle 时进入 D3（避免 D3hot/D3cold → D0 失败）

也即修改后是：

intel_iommu=on iommu=pt pcie_port_pm=off pcie_aspm=off vfio-pci.disable_idle_d3=1

然后运行：

# proxmox-boot-tool refresh
# update-initramfs -u -k all
reboot

验证 VM 中显卡是否已经直通

在 PVE 中选择 AI-122，点击 Start。

运行：

lspci -nn | grep -E "VGA|3D|Display"

输出形如：

00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)

lspci -nn | grep -i audio

输出形如：

00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)

这就表示已经成功直通了。

如果要添加新的 GPU，得先关闭 VM，再重复上面的“将显卡全部直通给 VM”的步骤。

安装 NVDIA 驱动和 NVCC+CUDA

参考：Ubuntu 安装 NVIDIA 驱动和 CUDA (NVCC)

五a、常见问题

启动时间太长

例如：超过5分钟。

【待解决】启用 IOMMU / Passthrough（直通）后，启动慢似乎是个已知问题

【Windows】PVE直通下的Windows开机巨慢的解决方案之一

https://blog.csdn.net/Freesia_2350/article/details/146205627

Extremely slow VM startup when IOMMU/Passthrough is enabled

https://www.reddit.com/r/Proxmox/comments/wowj61/extremely_slow_vm_startup_when_iommupassthrough

【已验证】似乎启动慢是因为给 VM 分配的内存太大，初始化需要很久

试试把 VM 的内存调小一些，比如从 1TB (1048576) 调到 128GB (131072) 或者 64GB (65536)。

启动失败

【已验证】试试 Remove 几张显卡。

五b、智能启动脚本

启动 VM 时自动诊断和排除故障显卡

start_vm101.sh

#!/usr/bin/env bash

set -Eeuo pipefail

VMID=101
DRY_RUN=0
INCLUDE_QUARANTINED=0
REVALIDATE_QUARANTINED=0

CONF=""
GPU_TOOL="${GPU_TOOL:-/root/qm_gpus.sh}"
STATE_FILE="${STATE_FILE:-/root/start_vm101.json}"
STATE_DIR="${STATE_DIR:-/root/.vm-gpu-state}"
RUN_ROOT="${RUN_ROOT:-/root/.vm-start-runs}"
CONF_ROOT="${CONF_ROOT:-/etc/pve/qemu-server}"
LOCK_ROOT="${LOCK_ROOT:-/run/lock}"
LOCK_FILE=""
RUN_DIR=""
REQUEST_ID=""
PVE_NODE=""

QM_TIMEOUT="${QM_TIMEOUT:-30}"
START_TIMEOUT_REQUESTED="${START_TIMEOUT:-auto}"
START_TIMEOUT=""
START_TIMEOUT_MODE=""
START_PROGRESS_INTERVAL="${START_PROGRESS_INTERVAL:-30}"
STORAGE_WAIT_TIMEOUT="${STORAGE_WAIT_TIMEOUT:-600}"
ENUMERATION_TIMEOUT="${ENUMERATION_TIMEOUT:-60}"
STOP_WAIT_SECONDS="${STOP_WAIT_SECONDS:-90}"
MAX_PRODUCTION_STARTS="${MAX_PRODUCTION_STARTS:-2}"

CURRENT_PHASE="initializing"
FINAL_MESSAGE=""
GPU_SOURCE="none"
last_start_log=""
last_gpu_log=""
last_enumerated_gpu_count=""
storage_state_lines=""
storage_all_ready=0
production_start_count=0
state_initialized=0
state_finalized=0
VM_MEMORY_MIB=0
last_start_duration_seconds=""
last_start_failure_kind=""
start_progress_pid=""

declare -a static_healthy_gpus=()
declare -a cached_gpus=()
declare -a configured_gpus=()
declare -a revalidation_candidates=()
declare -a recovered_gpus=()

export STATE_DIR

log() {
	printf '[%s] %s\n' "$(date '+%F %T')" "$*"
}

warn() {
	printf '[%s] WARN: %s\n' "$(date '+%F %T')" "$*" >&2
}

die() {
	FINAL_MESSAGE="$*"
	printf '[%s] ERROR: %s\n' "$(date '+%F %T')" "$*" >&2
	exit 1
}

usage() {
	cat <<'EOF'
Usage: start_vm101.sh [VMID] [--dry-run]
                      [--include-quarantined|--revalidate-quarantined]

Normal startup:
  1. Record the request and current host/VM/storage/GPU state in
     /root/start_vm101.json.
  2. If the last successful GPU list is still the complete healthy visible
     set, compact it to hostpci0..N and start VM101 once.
  3. If the cached set is stale or startup fails, diagnose PCI realization
     with a small 256 MiB QEMU/VFIO probe, update the GPU configuration, and
     make at most one further production-VM start.
  4. Promote a configuration to last_success only after QEMU enumerates every
     configured NVIDIA GPU.

The production-start deadline is sized automatically from configured VM
memory. Set START_TIMEOUT to a positive number of seconds to override it.

--dry-run performs read-only storage and PCI checks. It writes only diagnostic
logs and the JSON state record; it does not mount storage, bind PCI devices,
change VM configuration, or start/stop the VM.

--include-quarantined temporarily includes quarantined GPUs in this run but
never clears their quarantine records.

--revalidate-quarantined requires a stopped VM for a real run. It forces the
full static and combined VFIO probe path, includes quarantined GPUs, and clears
a recovered GPU's quarantine record only after the production VM starts and
QEMU enumerates every configured NVIDIA GPU. With --dry-run it is preview-only.
The script never stops a running VM automatically.
EOF
}

require_cmd() {
	command -v "$1" >/dev/null 2>&1 || die "missing required command: $1"
}

resolve_start_timeout() {
	local memory_gib
	local calculated

	VM_MEMORY_MIB="$(timeout "$QM_TIMEOUT" qm config "$VMID" |
		awk '$1 == "memory:" { print $2; exit }')"
	[[ "$VM_MEMORY_MIB" =~ ^[1-9][0-9]*$ ]] ||
		die "cannot determine configured memory for VM ${VMID}"

	if [[ "$START_TIMEOUT_REQUESTED" == "auto" ]]; then
		memory_gib=$(((VM_MEMORY_MIB + 1023) / 1024))
		calculated=$((300 + memory_gib * 2))
		((calculated < 900)) && calculated=900
		((calculated > 3600)) && calculated=3600
		START_TIMEOUT="$calculated"
		START_TIMEOUT_MODE="auto"
		log "Production-start timeout: ${START_TIMEOUT}s (auto-sized for ${memory_gib} GiB VM memory)"
	elif [[ "$START_TIMEOUT_REQUESTED" =~ ^[1-9][0-9]*$ ]]; then
		START_TIMEOUT="$START_TIMEOUT_REQUESTED"
		START_TIMEOUT_MODE="explicit"
		log "Production-start timeout: ${START_TIMEOUT}s (explicit override)"
	else
		die "START_TIMEOUT must be 'auto' or a positive integer"
	fi
}

parse_args() {
	if [[ $# -gt 0 && "$1" != --* ]]; then
		VMID="$1"
		shift
	fi

	while [[ $# -gt 0 ]]; do
		case "$1" in
			--dry-run)
				DRY_RUN=1
				shift
				;;
			--include-quarantined)
				INCLUDE_QUARANTINED=1
				shift
				;;
			--revalidate-quarantined)
				REVALIDATE_QUARANTINED=1
				INCLUDE_QUARANTINED=1
				shift
				;;
			-h|--help)
				usage
				exit 0
				;;
			*)
				die "unknown argument: $1"
				;;
		esac
	done
}

normalize_bdf() {
	local raw="${1%%,*}"

	raw="${raw,,}"
	if [[ "$raw" =~ ^[0-9a-f]{2}:[0-9a-f]{2}(\.[0-7])?$ ]]; then
		raw="0000:${raw}"
	fi
	if [[ "$raw" =~ ^[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}(\.[0-7])?$ ]]; then
		printf '%s\n' "${raw%.*}"
		return 0
	fi
	return 1
}

join_csv() {
	local IFS=','
	printf '%s' "$*"
}

arrays_equal() {
	local left_name="$1"
	local right_name="$2"
	local -n left="$left_name"
	local -n right="$right_name"
	local index

	[[ ${#left[@]} -eq ${#right[@]} ]] || return 1
	for index in "${!left[@]}"; do
		[[ "${left[$index]}" == "${right[$index]}" ]] || return 1
	done
}

array_contains() {
	local needle="$1"
	shift
	local item

	for item in "$@"; do
		[[ "$item" == "$needle" ]] && return 0
	done
	return 1
}

load_revalidation_candidates() {
	local quarantine_file="${STATE_DIR}/vm${VMID}.quarantine"
	local raw
	local bdf

	revalidation_candidates=()
	[[ "$REVALIDATE_QUARANTINED" -eq 1 ]] || return 0
	if [[ ! -f "$quarantine_file" ]]; then
		log "No quarantine records exist for VM ${VMID}; the full healthy inventory will still be probed"
		return 0
	fi

	while IFS=$'\t' read -r raw _; do
		[[ "$raw" =~ ^[[:space:]]*(#|$) ]] && continue
		bdf="$(normalize_bdf "$raw" || true)"
		[[ -n "$bdf" ]] || continue
		array_contains "$bdf" "${revalidation_candidates[@]}" ||
			revalidation_candidates+=("$bdf")
	done <"$quarantine_file"
	log "Quarantined GPUs scheduled for full revalidation (${#revalidation_candidates[@]}): ${revalidation_candidates[*]:-none}"
}

vm_status() {
	local output

	if output="$(timeout "$QM_TIMEOUT" perl -MPVE::QemuServer::Helpers -e '
		my $vmid = shift;
		print PVE::QemuServer::Helpers::vm_running_locally($vmid)
			? "running\n" : "stopped\n";
	' "$VMID" 2>/dev/null)"; then
		printf '%s\n' "$output"
		return 0
	fi
	output="$(timeout "$QM_TIMEOUT" qm status "$VMID" 2>/dev/null)" || return 1
	awk '{print $2}' <<<"$output"
}

list_vm_volumes() {
	timeout "$QM_TIMEOUT" qm config "$VMID" | awk -F': ' '
		/^(efidisk|scsi|sata|virtio|ide|tpmstate)[0-9]+:/ {
			split($2, parts, ",")
			volume = parts[1]
			if (volume != "none" && volume ~ /^[^:]+:/) {
				print volume
			}
		}
	' | sort -u
}

collect_storage_state() {
	local volume
	local storage
	local path
	local mount_root
	local active
	local ready
	local storage_status_text
	local -a volumes=()
	local -a lines=()

	mapfile -t volumes < <(list_vm_volumes)
	storage_status_text="$(timeout "$QM_TIMEOUT" pvesm status 2>/dev/null || true)"
	storage_all_ready=1
	for volume in "${volumes[@]}"; do
		storage="${volume%%:*}"
		mount_root="/mnt/pve/${storage}"
		path="$(timeout "$QM_TIMEOUT" pvesm path "$volume" 2>/dev/null || true)"
		active=0
		ready=0
		if awk -v target="$storage" '
			NR > 1 && $1 == target && $3 == "active" { found = 1 }
			END { exit !found }
		' <<<"$storage_status_text"; then
			active=1
		fi
		if [[ "$active" -eq 1 && -n "$path" && -e "$path" ]]; then
			if [[ "$path" == "${mount_root}/"* || "$path" == "$mount_root" ]]; then
				mountpoint -q "$mount_root" && ready=1
			else
				ready=1
			fi
		fi
		[[ "$ready" -eq 1 ]] || storage_all_ready=0
		lines+=("$(printf '%s\t%s\t%s\t%s' "$volume" "$path" "$active" "$ready")")
	done

	storage_state_lines=""
	if [[ ${#lines[@]} -gt 0 ]]; then
		printf -v storage_state_lines '%s\n' "${lines[@]}"
		storage_state_lines="${storage_state_lines%$'\n'}"
	fi
}

activate_volume_mount() {
	local volume="$1"
	local storage="${volume%%:*}"
	local mount_root="/mnt/pve/${storage}"
	local mount_unit

	# pvesm status only proves that the storage backend is online. For block
	# backends such as LVM-thin, the individual LV can still be inactive and
	# its /dev path absent while the VM is stopped. Activate through PVE's
	# storage layer, the same mechanism used by qm start.
	log "Activating PVE storage volume ${volume}"
	timeout "$STORAGE_WAIT_TIMEOUT" perl -MPVE::Storage -e '
		my $volume = shift;
		my $cfg = PVE::Storage::config();
		PVE::Storage::activate_volumes($cfg, [$volume]);
	' "$volume" || die "failed to activate PVE storage volume ${volume}"

	# Keep support for externally managed directory mount units. In most PVE
	# storage plugins the call above already activates the backend, so this is
	# intentionally idempotent.
	mount_unit="$(systemd-escape --path --suffix=mount "$mount_root")"
	if systemctl cat "$mount_unit" >/dev/null 2>&1; then
		log "Ensuring storage mount ${mount_unit} is active for ${volume}"
		timeout "$STORAGE_WAIT_TIMEOUT" systemctl start "$mount_unit" ||
			die "failed to activate ${mount_unit}"
	fi
}

wait_for_vm_storage() {
	local deadline
	local next_report=0
	local now
	local volume
	local -a volumes=()
	local -a missing=()

	mapfile -t volumes < <(list_vm_volumes)
	log "Checking ${#volumes[@]} configured VM storage volumes"
	for volume in "${volumes[@]}"; do
		activate_volume_mount "$volume"
	done

	CURRENT_PHASE="storage_wait"
	collect_storage_state
	state_update "storage_check" "$CURRENT_PHASE" "in_progress" \
		"waiting for every configured volume to become active and readable" "" 0 0 0

	deadline=$((SECONDS + STORAGE_WAIT_TIMEOUT))
	while true; do
		collect_storage_state
		if [[ "$storage_all_ready" -eq 1 ]]; then
			log "All configured VM storage volumes are active and readable"
			state_update "storage_ready" "storage_ready" "success" \
				"all configured volumes are active, mounted where required, and readable" "" 0 0 0
			return 0
		fi

		missing=()
		while IFS=$'\t' read -r volume _ _ ready; do
			[[ "$ready" == "1" ]] || missing+=("$volume")
		done <<<"$storage_state_lines"

		now="$SECONDS"
		((now < deadline)) ||
			die "storage readiness timeout; unavailable: ${missing[*]}"
		if ((now >= next_report)); then
			warn "Waiting for storage: ${missing[*]}"
			state_update "storage_wait" "$CURRENT_PHASE" "in_progress" \
				"waiting for unavailable volumes: ${missing[*]}" "" 0 0 0
			next_report=$((now + 15))
		fi
		sleep 2
	done
}

configured_gpu_list() {
	local line
	local value
	local bdf

	while IFS= read -r line; do
		[[ "$line" =~ ^hostpci[0-9]+:[[:space:]]*(.+)$ ]] || continue
		value="${BASH_REMATCH[1]}"
		bdf="$(normalize_bdf "$value" || true)"
		[[ -n "$bdf" ]] && printf '%s\n' "$bdf"
	done < <(
		timeout "$QM_TIMEOUT" qm config "$VMID" |
			awk -F: '/^hostpci[0-9]+:/ { key=$1; sub(/^hostpci/, "", key); print key "\t" $0 }' |
			sort -n -k1,1 |
			cut -f2-
	)
}

qemu_monitor_gpu_count() {
	local output

	output="$(timeout "$QM_TIMEOUT" pvesh create \
		"/nodes/${PVE_NODE}/qemu/${VMID}/monitor" \
		--command "info pci" 2>/dev/null || true)"
	grep -Eic '(VGA|3D|Display) controller: PCI device 10de:' <<<"$output" || true
}

report_start_progress() {
	local started_epoch="$1"
	local elapsed
	local status
	local pid=""
	local rss_kib=""
	local rss_gib=""
	local target_gib

	elapsed=$(($(date +%s) - started_epoch))
	status="$(vm_status || true)"
	[[ -n "$status" ]] || status="unknown"
	if [[ -r "/run/qemu-server/${VMID}.pid" ]]; then
		pid="$(<"/run/qemu-server/${VMID}.pid")"
	fi
	if [[ "$pid" =~ ^[1-9][0-9]*$ && -r "/proc/${pid}/status" ]]; then
		rss_kib="$(awk '$1 == "VmRSS:" { print $2; exit }' "/proc/${pid}/status")"
	fi
	target_gib="$(awk -v mib="$VM_MEMORY_MIB" 'BEGIN { printf "%.1f", mib / 1024 }')"
	if [[ "$rss_kib" =~ ^[0-9]+$ ]]; then
		rss_gib="$(awk -v kib="$rss_kib" 'BEGIN { printf "%.1f", kib / 1048576 }')"
		log "VM ${VMID} is still initializing: elapsed=${elapsed}s/${START_TIMEOUT}s, status=${status}, QEMU_RSS=${rss_gib}/${target_gib} GiB"
	else
		log "VM ${VMID} is still initializing: elapsed=${elapsed}s/${START_TIMEOUT}s, status=${status}, QEMU process/RSS not available yet"
	fi
}

start_progress_monitor() {
	local started_epoch="$1"

	while sleep "$START_PROGRESS_INTERVAL"; do
		report_start_progress "$started_epoch"
	done
}

stop_progress_monitor() {
	if [[ "$start_progress_pid" =~ ^[1-9][0-9]*$ ]]; then
		kill "$start_progress_pid" >/dev/null 2>&1 || true
		wait "$start_progress_pid" 2>/dev/null || true
	fi
	start_progress_pid=""
}

state_update() {
	local action="$1"
	local phase="$2"
	local result="$3"
	local message="$4"
	local log_path="$5"
	local exit_code="$6"
	local is_attempt="$7"
	local promote_success="$8"
	local status
	local config_text
	local config_sha=""
	local boot_id=""
	local kernel=""
	local pid=""
	local enum_count="$last_enumerated_gpu_count"

	status="$(vm_status || true)"
	[[ -n "$status" ]] || status="unknown"
	config_text="$(timeout "$QM_TIMEOUT" qm config "$VMID" 2>/dev/null || true)"
	[[ -f "$CONF" ]] && config_sha="$(sha256sum "$CONF" | awk '{print $1}')"
	[[ -r /proc/sys/kernel/random/boot_id ]] &&
		boot_id="$(</proc/sys/kernel/random/boot_id)"
	kernel="$(uname -r)"
	if [[ "$status" == "running" ]]; then
		[[ -r "/run/qemu-server/${VMID}.pid" ]] &&
			pid="$(<"/run/qemu-server/${VMID}.pid")"
	else
		enum_count=""
	fi

	STATE_PATH="$STATE_FILE" \
	STATE_NOW="$(date -Is)" \
	STATE_ACTION="$action" \
	STATE_PHASE_VALUE="$phase" \
	STATE_RESULT="$result" \
	STATE_MESSAGE="$message" \
	STATE_LOG_PATH="$log_path" \
	STATE_EXIT_CODE="$exit_code" \
	STATE_IS_ATTEMPT="$is_attempt" \
	STATE_PROMOTE_SUCCESS="$promote_success" \
	STATE_VMID="$VMID" \
	STATE_REQUEST_ID="$REQUEST_ID" \
	STATE_RUN_DIR="$RUN_DIR" \
	STATE_DRY_RUN="$DRY_RUN" \
	STATE_INCLUDE_QUARANTINED="$INCLUDE_QUARANTINED" \
	STATE_REVALIDATE_QUARANTINED="$REVALIDATE_QUARANTINED" \
	STATE_REVALIDATION_CANDIDATES="$(join_csv "${revalidation_candidates[@]}")" \
	STATE_RECOVERED_GPUS="$(join_csv "${recovered_gpus[@]}")" \
	STATE_QM_TIMEOUT="$QM_TIMEOUT" \
	STATE_START_TIMEOUT="$START_TIMEOUT" \
	STATE_START_TIMEOUT_MODE="$START_TIMEOUT_MODE" \
	STATE_START_PROGRESS_INTERVAL="$START_PROGRESS_INTERVAL" \
	STATE_VM_MEMORY_MIB="$VM_MEMORY_MIB" \
	STATE_START_DURATION="$last_start_duration_seconds" \
	STATE_START_FAILURE_KIND="$last_start_failure_kind" \
	STATE_STORAGE_TIMEOUT="$STORAGE_WAIT_TIMEOUT" \
	STATE_ENUM_TIMEOUT="$ENUMERATION_TIMEOUT" \
	STATE_STOP_TIMEOUT="$STOP_WAIT_SECONDS" \
	STATE_MAX_STARTS="$MAX_PRODUCTION_STARTS" \
	STATE_GPU_SOURCE="$GPU_SOURCE" \
	STATE_QUARANTINE_FILE="${STATE_DIR}/vm${VMID}.quarantine" \
	STATE_STORAGE_LINES="$storage_state_lines" \
	STATE_STORAGE_READY="$storage_all_ready" \
	STATE_VM_STATUS="$status" \
	STATE_VM_PID="$pid" \
	STATE_ENUM_GPU_COUNT="$enum_count" \
	STATE_CONFIG_TEXT="$config_text" \
	STATE_CONFIG_SHA="$config_sha" \
	STATE_BOOT_ID="$boot_id" \
	STATE_KERNEL="$kernel" \
	perl -MJSON::PP -MFile::Basename=dirname -e '
		use strict;
		use warnings;

		sub boolean {
			return ($_[0] // q{}) ne q{} && ($_[0] // q{}) ne q{0}
				? JSON::PP::true : JSON::PP::false;
		}

		sub normalize_bdf {
			my ($value) = @_;
			$value = lc($value // q{});
			$value =~ s/,.*$//;
			if ($value =~ /^([0-9a-f]{2}):([0-9a-f]{2})(?:\.[0-7])?$/) {
				return "0000:$1:$2";
			}
			if ($value =~ /^([0-9a-f]{4}):([0-9a-f]{2}):([0-9a-f]{2})(?:\.[0-7])?$/) {
				return "$1:$2:$3";
			}
			return undef;
		}

		sub csv_array {
			my ($value) = @_;
			return [] if !defined($value) || $value eq q{};
			return [grep { $_ ne q{} } split /,/, $value];
		}

		my $path = $ENV{STATE_PATH};
		my $json = JSON::PP->new->canonical(1)->pretty(1);
		my $state = {};
		if (-e $path) {
			if (open my $in, q{<}, $path) {
				local $/;
				my $raw = <$in>;
				close $in;
				my $decoded = eval { $json->decode($raw) };
				if (!$@ && ref($decoded) eq q{HASH}) {
					$state = $decoded;
				} else {
					my $corrupt = $path . q{.corrupt.} . time() . q{.} . $$;
					rename $path, $corrupt
						or die "cannot preserve corrupt state as $corrupt: $!";
				}
			} else {
				die "cannot read $path: $!";
			}
		}

		my %raw_config;
		for my $line (split /\n/, ($ENV{STATE_CONFIG_TEXT} // q{})) {
			next unless $line =~ /^([^:\s]+):\s*(.*)$/;
			$raw_config{$1} = $2;
		}

		my %safe_config;
		my %allowed = map { $_ => 1 } qw(
			name memory balloon cores sockets cpu numa machine bios ostype
			scsihw boot onboot agent hugepages
		);
		for my $key (sort keys %allowed) {
			$safe_config{$key} = $raw_config{$key} if exists $raw_config{$key};
		}

		my @hostpci_keys = sort {
			($a =~ /(\d+)$/)[0] <=> ($b =~ /(\d+)$/)[0]
		} grep { /^hostpci\d+$/ } keys %raw_config;
		my %hostpci;
		my @configured_gpus;
		for my $key (@hostpci_keys) {
			my $bdf = normalize_bdf($raw_config{$key});
			$hostpci{$key} = {
				value => $raw_config{$key},
				(defined $bdf ? (bdf => $bdf) : ()),
			};
			push @configured_gpus, $bdf if defined $bdf;
		}

		my @volumes;
		for my $line (split /\n/, ($ENV{STATE_STORAGE_LINES} // q{})) {
			next if $line eq q{};
			my ($volid, $volume_path, $active, $ready) = split /\t/, $line, 4;
			push @volumes, {
				volid => ($volid // q{}),
				path => ($volume_path // q{}),
				active => boolean($active),
				ready => boolean($ready),
			};
		}

		my @quarantine;
		my $quarantine_path = $ENV{STATE_QUARANTINE_FILE} // q{};
		if ($quarantine_path ne q{} && open my $qf, q{<}, $quarantine_path) {
			while (my $line = <$qf>) {
				chomp $line;
				next if $line =~ /^\s*(?:#|$)/;
				my ($bdf, $timestamp, $reason) = split /\t/, $line, 3;
				push @quarantine, {
					bdf => ($bdf // q{}),
					timestamp => ($timestamp // q{}),
					reason => ($reason // q{}),
				};
			}
			close $qf;
		}

		my $parameters = {
			dry_run => boolean($ENV{STATE_DRY_RUN}),
			include_quarantined => boolean($ENV{STATE_INCLUDE_QUARANTINED}),
			revalidate_quarantined => boolean($ENV{STATE_REVALIDATE_QUARANTINED}),
			qm_timeout_seconds => 0 + ($ENV{STATE_QM_TIMEOUT} // 0),
			start_timeout_seconds => 0 + ($ENV{STATE_START_TIMEOUT} // 0),
			start_timeout_mode => ($ENV{STATE_START_TIMEOUT_MODE} // q{}),
			start_progress_interval_seconds =>
				0 + ($ENV{STATE_START_PROGRESS_INTERVAL} // 0),
			vm_memory_mib => 0 + ($ENV{STATE_VM_MEMORY_MIB} // 0),
			storage_wait_timeout_seconds => 0 + ($ENV{STATE_STORAGE_TIMEOUT} // 0),
			enumeration_timeout_seconds => 0 + ($ENV{STATE_ENUM_TIMEOUT} // 0),
			stop_wait_seconds => 0 + ($ENV{STATE_STOP_TIMEOUT} // 0),
			max_production_starts => 0 + ($ENV{STATE_MAX_STARTS} // 0),
		};

		$state->{schema_version} = 1;
		$state->{vmid} = 0 + $ENV{STATE_VMID};
		$state->{updated_at} = $ENV{STATE_NOW};
		$state->{phase} = $ENV{STATE_PHASE_VALUE};
		$state->{result} = $ENV{STATE_RESULT};
		$state->{message} = $ENV{STATE_MESSAGE};
		$state->{host} = {
			boot_id => ($ENV{STATE_BOOT_ID} // q{}),
			kernel => ($ENV{STATE_KERNEL} // q{}),
		};
		$state->{parameters} = $parameters;
		$state->{storage} = {
			status => boolean($ENV{STATE_STORAGE_READY}) ? q{ready} : q{not_ready},
			volumes => \@volumes,
		};
		$state->{gpu} = {
			source => ($ENV{STATE_GPU_SOURCE} // q{none}),
			quarantine => \@quarantine,
			revalidation_candidates => csv_array($ENV{STATE_REVALIDATION_CANDIDATES}),
			recovered_from_quarantine => csv_array($ENV{STATE_RECOVERED_GPUS}),
		};
		$state->{vm} = {
			status => ($ENV{STATE_VM_STATUS} // q{unknown}),
			config_sha256 => ($ENV{STATE_CONFIG_SHA} // q{}),
			config => \%safe_config,
			hostpci => \%hostpci,
			configured_gpus => \@configured_gpus,
			configured_gpu_count => scalar(@configured_gpus),
		};
		if (($ENV{STATE_VM_PID} // q{}) =~ /^\d+$/) {
			$state->{vm}{pid} = 0 + $ENV{STATE_VM_PID};
		}
		if (($ENV{STATE_ENUM_GPU_COUNT} // q{}) =~ /^\d+$/) {
			my $count = 0 + $ENV{STATE_ENUM_GPU_COUNT};
			$state->{vm}{enumerated_gpu_count} = $count;
			$state->{vm}{enumeration_complete} =
				$count == scalar(@configured_gpus)
				? JSON::PP::true : JSON::PP::false;
		}

		if (($ENV{STATE_ACTION} // q{}) eq q{init}) {
			$state->{request} = {
				id => ($ENV{STATE_REQUEST_ID} // q{}),
				started_at => $ENV{STATE_NOW},
				log_dir => ($ENV{STATE_RUN_DIR} // q{}),
				parameters => $parameters,
			};
		}

		if (boolean($ENV{STATE_IS_ATTEMPT})) {
			$state->{last_attempt} = {
				timestamp => $ENV{STATE_NOW},
				result => $ENV{STATE_RESULT},
				exit_code => 0 + ($ENV{STATE_EXIT_CODE} // 0),
				log => ($ENV{STATE_LOG_PATH} // q{}),
				message => ($ENV{STATE_MESSAGE} // q{}),
				vm_status => ($ENV{STATE_VM_STATUS} // q{unknown}),
				configured_gpus => [@configured_gpus],
				source => ($ENV{STATE_GPU_SOURCE} // q{none}),
			};
			if (($ENV{STATE_START_DURATION} // q{}) =~ /^\d+$/) {
				$state->{last_attempt}{duration_seconds} =
					0 + $ENV{STATE_START_DURATION};
			}
			if (($ENV{STATE_START_FAILURE_KIND} // q{}) ne q{}) {
				$state->{last_attempt}{failure_kind} =
					$ENV{STATE_START_FAILURE_KIND};
			}
		}

		if (boolean($ENV{STATE_PROMOTE_SUCCESS})) {
			$state->{last_success} = {
				timestamp => $ENV{STATE_NOW},
				config_sha256 => ($ENV{STATE_CONFIG_SHA} // q{}),
				configured_gpus => [@configured_gpus],
				hostpci => {%hostpci},
				enumerated_gpu_count =>
					0 + ($ENV{STATE_ENUM_GPU_COUNT} // scalar(@configured_gpus)),
				parameters => $parameters,
				source => ($ENV{STATE_GPU_SOURCE} // q{none}),
				host_boot_id => ($ENV{STATE_BOOT_ID} // q{}),
			};
			if (($ENV{STATE_START_DURATION} // q{}) =~ /^\d+$/) {
				$state->{last_success}{duration_seconds} =
					0 + $ENV{STATE_START_DURATION};
			}
		}

		if (($ENV{STATE_RESULT} // q{}) eq q{failed}) {
			$state->{last_failure} = {
				timestamp => $ENV{STATE_NOW},
				phase => ($ENV{STATE_PHASE_VALUE} // q{}),
				message => ($ENV{STATE_MESSAGE} // q{}),
				log => ($ENV{STATE_LOG_PATH} // q{}),
				exit_code => 0 + ($ENV{STATE_EXIT_CODE} // 0),
			};
		}

		my $history = ref($state->{history}) eq q{ARRAY} ? $state->{history} : [];
		push @$history, {
			timestamp => $ENV{STATE_NOW},
			action => ($ENV{STATE_ACTION} // q{}),
			phase => ($ENV{STATE_PHASE_VALUE} // q{}),
			result => ($ENV{STATE_RESULT} // q{}),
			message => ($ENV{STATE_MESSAGE} // q{}),
			exit_code => 0 + ($ENV{STATE_EXIT_CODE} // 0),
			vm_status => ($ENV{STATE_VM_STATUS} // q{unknown}),
			gpu_source => ($ENV{STATE_GPU_SOURCE} // q{none}),
		};
		if (($ENV{STATE_START_DURATION} // q{}) =~ /^\d+$/) {
			$history->[-1]{duration_seconds} =
				0 + $ENV{STATE_START_DURATION};
		}
		if (($ENV{STATE_START_FAILURE_KIND} // q{}) ne q{}) {
			$history->[-1]{failure_kind} =
				$ENV{STATE_START_FAILURE_KIND};
		}
		splice @$history, 0, @$history - 30 if @$history > 30;
		$state->{history} = $history;

		my $dir = dirname($path);
		-d $dir or mkdir $dir, 0700 or die "cannot create $dir: $!";
		my $tmp = $path . q{.tmp.} . $$;
		open my $out, q{>}, $tmp or die "cannot create $tmp: $!";
		chmod 0600, $tmp or die "cannot chmod $tmp: $!";
		print {$out} $json->encode($state)
			or die "cannot write $tmp: $!";
		close $out or die "cannot close $tmp: $!";
		rename $tmp, $path or die "cannot replace $path: $!";
		chmod 0600, $path or die "cannot chmod $path: $!";
	'
}

state_has_last_success() {
	[[ -f "$STATE_FILE" ]] || return 1
	perl -MJSON::PP -0777 -e '
		my $s = eval { JSON::PP->new->decode(<>) };
		exit 1 unless ref($s) eq q{HASH};
		exit 1 unless ref($s->{last_success}) eq q{HASH};
		exit 1 unless ref($s->{last_success}{configured_gpus}) eq q{ARRAY};
		exit 0;
	' "$STATE_FILE"
}

read_last_success_gpus() {
	perl -MJSON::PP -0777 -e '
		my $s = JSON::PP->new->decode(<>);
		print "$_\n" for @{$s->{last_success}{configured_gpus}};
	' "$STATE_FILE"
}

load_static_healthy_gpus() {
	local list_file="${RUN_DIR}/gpu-static.list"
	local error_file="${RUN_DIR}/gpu-static.log"
	local -a args=("$VMID" "--list")

	[[ "$INCLUDE_QUARANTINED" -eq 1 ]] && args+=("--include-quarantined")
	if ! "$GPU_TOOL" "${args[@]}" >"$list_file" 2>"$error_file"; then
		sed -n '1,160p' "$error_file" >&2 || true
		return 1
	fi
	sed -n '1,160p' "$error_file" >&2 || true
	mapfile -t static_healthy_gpus <"$list_file"
	log "Statically healthy visible GPUs (${#static_healthy_gpus[@]}): ${static_healthy_gpus[*]:-none}"
}

run_gpu_tool_logged() {
	local label="$1"
	shift
	local log_file="${RUN_DIR}/${label}.log"
	local rc=0

	last_gpu_log="$log_file"
	if "$GPU_TOOL" "$@" >"$log_file" 2>&1; then
		rc=0
	else
		rc=$?
	fi
	sed -n '1,220p' "$log_file" || true
	return "$rc"
}

apply_cached_gpu_config() {
	local only="none"
	local -a args=("$VMID" "--apply" "--no-vfio-probe")

	if [[ ${#cached_gpus[@]} -gt 0 ]]; then
		only="$(join_csv "${cached_gpus[@]}")"
	fi
	args+=("--only" "$only")
	[[ "$INCLUDE_QUARANTINED" -eq 1 ]] && args+=("--include-quarantined")

	CURRENT_PHASE="gpu_cached_apply"
	GPU_SOURCE="last_success"
	log "Reusing the complete last-success GPU set: ${cached_gpus[*]:-none}"
	run_gpu_tool_logged "gpu-cached-apply" "${args[@]}" ||
		return 1
	mapfile -t configured_gpus < <(configured_gpu_list)
	state_update "gpu_config" "$CURRENT_PHASE" "success" \
		"applied last-success GPUs as contiguous hostpci0..N" "$last_gpu_log" 0 0 0
}

run_gpu_diagnosis() {
	local -a args=("$VMID" "--apply" "--vfio-probe")

	[[ "$INCLUDE_QUARANTINED" -eq 1 ]] && args+=("--include-quarantined")
	CURRENT_PHASE="gpu_diagnosis"
	GPU_SOURCE="diagnosed"
	log "Running lightweight PCI/VFIO diagnosis without booting VM ${VMID}"
	state_update "gpu_diagnosis" "$CURRENT_PHASE" "in_progress" \
		"static PCI checks and a 256 MiB QEMU/VFIO realization probe are running" "" 0 0 0
	if ! run_gpu_tool_logged "gpu-diagnosis-$(date '+%H%M%S')" "${args[@]}"; then
		state_update "gpu_diagnosis" "$CURRENT_PHASE" "failed" \
			"lightweight GPU diagnosis or configuration failed" "$last_gpu_log" 1 0 0
		return 1
	fi
	mapfile -t configured_gpus < <(configured_gpu_list)
	log "Diagnosed GPU configuration (${#configured_gpus[@]}): ${configured_gpus[*]:-none}"
	state_update "gpu_diagnosis" "$CURRENT_PHASE" "success" \
		"lightweight diagnosis completed and hostpci entries were compacted" "$last_gpu_log" 0 0 0
}

quarantine_gpu() {
	local bdf="$1"
	local reason="$2"

	run_gpu_tool_logged "gpu-quarantine-${bdf//:/-}" \
		"$VMID" --quarantine "$bdf" --reason "$reason"
}

promote_revalidated_gpus() {
	local bdf
	local log_file="${RUN_DIR}/gpu-revalidation-promote.log"

	recovered_gpus=()
	[[ "$REVALIDATE_QUARANTINED" -eq 1 ]] || return 0
	mapfile -t configured_gpus < <(configured_gpu_list)
	if [[ ! "$last_enumerated_gpu_count" =~ ^[0-9]+$ ]] ||
		[[ "$last_enumerated_gpu_count" -ne ${#configured_gpus[@]} ]]; then
		warn "Refusing to clear quarantine before complete QEMU GPU enumeration"
		return 1
	fi

	: >"$log_file"
	chmod 600 "$log_file"
	for bdf in "${revalidation_candidates[@]}"; do
		if ! array_contains "$bdf" "${configured_gpus[@]}"; then
			warn "Keeping ${bdf} quarantined: it was not in the verified production GPU set"
			printf 'kept %s quarantined: not in verified production set\n' "$bdf" >>"$log_file"
			continue
		fi

		log "Promoting recovered GPU ${bdf} after full production enumeration"
		if "$GPU_TOOL" "$VMID" --unquarantine "$bdf" >>"$log_file" 2>&1; then
			recovered_gpus+=("$bdf")
		else
			warn "Failed to persist quarantine removal for recovered GPU ${bdf}"
			sed -n '1,160p' "$log_file" >&2 || true
			return 1
		fi
	done
	last_gpu_log="$log_file"
	if [[ ${#recovered_gpus[@]} -gt 0 ]]; then
		log "Recovered GPUs removed from quarantine (${#recovered_gpus[@]}): ${recovered_gpus[*]}"
	else
		log "No quarantined GPU completed the full production revalidation path"
	fi
}

verify_gpu_enumeration() {
	local expected="$1"
	local label="$2"
	local deadline=$((SECONDS + ENUMERATION_TIMEOUT))
	local count=0
	local output
	local monitor_log="${RUN_DIR}/monitor-${label}.log"

	while [[ "$SECONDS" -lt "$deadline" ]]; do
		output="$(timeout "$QM_TIMEOUT" pvesh create \
			"/nodes/${PVE_NODE}/qemu/${VMID}/monitor" \
			--command "info pci" 2>/dev/null || true)"
		printf '%s\n' "$output" >"$monitor_log"
		count="$(grep -Eic '(VGA|3D|Display) controller: PCI device 10de:' <<<"$output" || true)"
		last_enumerated_gpu_count="$count"
		if [[ "$count" -eq "$expected" ]]; then
			log "QEMU enumerated all ${expected} configured NVIDIA GPUs"
			return 0
		fi
		sleep 2
	done

	warn "GPU enumeration incomplete: configured=${expected}, QEMU-visible=${count}"
	return 1
}

stop_failed_start() {
	local status
	local deadline=$((SECONDS + STOP_WAIT_SECONDS))

	status="$(vm_status || true)"
	[[ "$status" == "running" ]] || return 0
	log "Stopping the failed VM ${VMID} start attempt"
	timeout "$STOP_WAIT_SECONDS" qm stop "$VMID" --skiplock 1 >/dev/null 2>&1 || true
	while [[ "$SECONDS" -lt "$deadline" ]]; do
		status="$(vm_status || true)"
		if [[ "$status" != "running" ]]; then
			last_enumerated_gpu_count=""
			state_update "vm_stop" "stopped_after_failure" "success" \
				"failed production start was stopped before further diagnosis" "$last_start_log" 0 0 0
			return 0
		fi
		sleep 1
	done
	die "VM ${VMID} did not stop within ${STOP_WAIT_SECONDS} seconds"
}

production_start() {
	local label="$1"
	local expected_gpus="$2"
	local output_file
	local rc=0
	local status
	local started_epoch
	local success_message

	production_start_count=$((production_start_count + 1))
	((production_start_count <= MAX_PRODUCTION_STARTS)) ||
		die "refusing production start ${production_start_count}; limit is ${MAX_PRODUCTION_STARTS}"

	CURRENT_PHASE="production_start"
	last_enumerated_gpu_count=""
	last_start_duration_seconds=""
	last_start_failure_kind=""
	output_file="${RUN_DIR}/production-$(printf '%02d' "$production_start_count")-${label}.log"
	last_start_log="$output_file"
	log "Production start ${production_start_count}/${MAX_PRODUCTION_STARTS}: ${label} (${expected_gpus} GPUs)"
	state_update "production_start" "$CURRENT_PHASE" "in_progress" \
		"starting VM ${VMID} with ${expected_gpus} configured GPUs" "$output_file" 0 0 0

	started_epoch="$(date +%s)"
	start_progress_monitor "$started_epoch" &
	start_progress_pid=$!
	# Do not let the long-lived QEMU process inherit the workflow lock. The
	# parent script keeps fd 9 until orchestration is fully finalized.
	timeout --foreground --kill-after=10 "$START_TIMEOUT" \
		qm start "$VMID" 9>&- >"$output_file" 2>&1 || rc=$?
	stop_progress_monitor
	last_start_duration_seconds=$(($(date +%s) - started_epoch))
	status="$(vm_status || true)"
	if [[ "$status" == "running" ]] &&
		verify_gpu_enumeration "$expected_gpus" "${production_start_count}-${label}"; then
		if [[ "$rc" -ne 0 ]]; then
			warn "qm start returned ${rc}, but the VM is running with complete GPU enumeration; checking stability"
			sleep 10
			status="$(vm_status || true)"
			last_enumerated_gpu_count="$(qemu_monitor_gpu_count)"
			if [[ "$status" != "running" ||
				"$last_enumerated_gpu_count" -ne "$expected_gpus" ]]; then
				last_start_failure_kind="command_failure"
			else
				warn "Preserving the healthy running VM after a non-zero qm start result"
			fi
		fi
	fi
	if [[ "$status" == "running" &&
		"$last_enumerated_gpu_count" -eq "$expected_gpus" &&
		"$last_start_failure_kind" != "command_failure" ]]; then
		success_message="VM started in ${last_start_duration_seconds}s and QEMU enumerated every configured GPU"
		state_update "production_start" "running" "success" \
			"$success_message" "$output_file" "$rc" 1 0
		return 0
	fi

	if [[ "$rc" -eq 124 ]]; then
		last_start_failure_kind="orchestration_timeout"
	else
		last_start_failure_kind="start_or_enumeration_failure"
	fi
	warn "Production start failed: command_rc=${rc}, VM_status=${status:-unknown}"
	sed -n '1,160p' "$output_file" | sed 's/^/  /' >&2 || true
	state_update "production_start" "$CURRENT_PHASE" "failed" \
		"production start or complete GPU enumeration failed after ${last_start_duration_seconds}s" \
		"$output_file" "$rc" 1 0
	stop_failed_start
	return 1
}

extract_unique_fault_gpu() {
	local log_file="$1"
	local line
	local token
	local bdf
	local configured
	local -a candidates=()
	local -a strong_lines=()

	[[ -f "$log_file" ]] || return 1
	mapfile -t strong_lines < <(
		grep -Ei \
			'(vfio|iommu|bar|reset|pcie?|device).*(fail|error|cannot|unable|not available|timeout|invalid)|(fail|error|cannot|unable|not available|timeout|invalid).*(vfio|iommu|bar|reset|pcie?|device)' \
			"$log_file" || true
	)
	for line in "${strong_lines[@]}"; do
		while IFS= read -r token; do
			bdf="$(normalize_bdf "$token" || true)"
			[[ -n "$bdf" ]] || continue
			for configured in "${configured_gpus[@]}"; do
				if [[ "$bdf" == "$configured" ]]; then
					if ! printf '%s\n' "${candidates[@]:-}" | grep -Fxq "$bdf"; then
						candidates+=("$bdf")
					fi
				fi
			done
		done < <(grep -Eio '([0-9a-f]{4}:)?[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]' <<<"$line" || true)
	done

	[[ ${#candidates[@]} -eq 1 ]] || return 1
	printf '%s\n' "${candidates[0]}"
}

prepare_after_identified_failure() {
	local bdf="$1"
	local reason="$2"

	warn "Production log identifies one configured GPU as faulty: ${bdf}"
	quarantine_gpu "$bdf" "$reason" ||
		return 1
	run_gpu_diagnosis
}

finalize_success() {
	CURRENT_PHASE="running"
	if [[ "$REVALIDATE_QUARANTINED" -eq 1 ]]; then
		GPU_SOURCE="revalidated"
		promote_revalidated_gpus ||
			die "VM ${VMID} is healthy and running, but recovered GPU quarantine state could not be persisted"
	fi
	state_update "complete" "$CURRENT_PHASE" "success" \
		"VM ${VMID} is running with the verified GPU configuration" "$last_start_log" 0 0 1
	state_finalized=1
	log "VM ${VMID} startup succeeded; last_success was updated in ${STATE_FILE}"
}

handle_final_start_failure() {
	local bad_gpu=""

	bad_gpu="$(extract_unique_fault_gpu "$last_start_log" || true)"
	if [[ -n "$bad_gpu" ]]; then
		prepare_after_identified_failure "$bad_gpu" \
			"production QEMU log identified this device after lightweight validation" || true
		warn "Configuration was updated for the next controlled start; no third production start will be attempted"
	fi
	die "VM ${VMID} did not reach a fully enumerated running state; see ${last_start_log}"
}

run_dry_run() {
	local -a args=("$VMID" "--dry-run")
	local status

	CURRENT_PHASE="dry_run"
	GPU_SOURCE="static_check"
	collect_storage_state
	[[ "$INCLUDE_QUARANTINED" -eq 1 ]] && args+=("--include-quarantined")
	if ! run_gpu_tool_logged "gpu-dry-run" "${args[@]}"; then
		die "GPU dry-run failed; see ${last_gpu_log}"
	fi
	mapfile -t configured_gpus < <(configured_gpu_list)
	status="$(vm_status || true)"
	if [[ "$status" == "running" ]]; then
		last_enumerated_gpu_count="$(qemu_monitor_gpu_count)"
	fi
	state_update "dry_run" "$CURRENT_PHASE" "success" \
		"read-only storage and static PCI checks completed; only logs and JSON state were written" \
		"$last_gpu_log" 0 0 0
	state_finalized=1
	log "Dry-run complete; runtime, storage mounts, PCI bindings, and VM configuration were unchanged"
}

on_exit() {
	local rc=$?

	trap - EXIT
	stop_progress_monitor
	if [[ "$rc" -ne 0 && "$state_initialized" -eq 1 && "$state_finalized" -eq 0 ]]; then
		state_update "exit" "$CURRENT_PHASE" "failed" \
			"${FINAL_MESSAGE:-startup workflow exited unexpectedly}" \
			"${last_start_log:-${last_gpu_log:-}}" "$rc" 0 0 || true
	fi
	exit "$rc"
}

main() {
	local status
	local cache_reason=""
	local bad_gpu=""

	parse_args "$@"
	[[ "$VMID" =~ ^[1-9][0-9]*$ ]] || die "invalid VMID: ${VMID}"
	[[ "$MAX_PRODUCTION_STARTS" =~ ^[12]$ ]] ||
		die "MAX_PRODUCTION_STARTS must be 1 or 2"
	[[ "$START_PROGRESS_INTERVAL" =~ ^[1-9][0-9]*$ ]] ||
		die "START_PROGRESS_INTERVAL must be a positive integer"

	CONF="${CONF_ROOT}/${VMID}.conf"
	LOCK_FILE="${LOCK_ROOT}/start-vm${VMID}.lock"

	[[ $EUID -eq 0 ]] || die "this script must run as root"
	require_cmd qm
	require_cmd pvesm
	require_cmd pvesh
	require_cmd timeout
	require_cmd flock
	require_cmd systemd-escape
	require_cmd systemctl
	require_cmd mountpoint
	require_cmd perl
	require_cmd sha256sum
	require_cmd hostname
	[[ -x "$GPU_TOOL" ]] || die "GPU tool is not executable: ${GPU_TOOL}"
	[[ -f "$CONF" ]] || die "VM config not found: ${CONF}"
	PVE_NODE="$(hostname)"
	resolve_start_timeout

	mkdir -p "$LOCK_ROOT"
	exec 9>"$LOCK_FILE"
	flock -n 9 || die "another startup/diagnostic job is active for VM ${VMID}"

	mkdir -p -m 700 "$RUN_ROOT"
	RUN_DIR="${RUN_ROOT}/vm${VMID}-$(date '+%F-%H%M%S-%N')"
	mkdir -m 700 "$RUN_DIR"
	REQUEST_ID="${RUN_DIR##*/}"
	log "Run logs: ${RUN_DIR}"
	load_revalidation_candidates

	CURRENT_PHASE="initialized"
	state_initialized=1
	state_update "init" "$CURRENT_PHASE" "in_progress" \
		"startup request accepted" "" 0 0 0

	if [[ "$DRY_RUN" -eq 1 ]]; then
		run_dry_run
		return 0
	fi

	status="$(vm_status || true)"
	if [[ "$status" == "running" ]]; then
		if [[ "$REVALIDATE_QUARANTINED" -eq 1 ]]; then
			CURRENT_PHASE="revalidation_requires_stopped_vm"
			die "--revalidate-quarantined requires VM ${VMID} to be stopped; refusing to stop a running VM automatically"
		fi
		GPU_SOURCE="current_running"
		collect_storage_state
		mapfile -t configured_gpus < <(configured_gpu_list)
		if [[ -z "$last_enumerated_gpu_count" ]]; then
			last_enumerated_gpu_count="$(qemu_monitor_gpu_count)"
		fi
		CURRENT_PHASE="already_running"
		if [[ "$last_enumerated_gpu_count" -ne ${#configured_gpus[@]} ]]; then
			state_update "already_running" "$CURRENT_PHASE" "running" \
				"VM is already running, but complete GPU enumeration is not verified; live configuration was not modified" \
				"" 0 0 0
			state_finalized=1
			warn "Current VM GPU enumeration is incomplete: configured=${#configured_gpus[@]}, QEMU-visible=${last_enumerated_gpu_count}"
			warn "The compacted configuration will be applied at the next stopped startup"
		elif [[ "$storage_all_ready" -ne 1 ]]; then
			state_update "already_running" "$CURRENT_PHASE" "running" \
				"VM and GPU enumeration are healthy, but configured storage is not fully ready; live configuration was not modified" \
				"" 0 0 0
			state_finalized=1
			warn "VM GPU enumeration is complete, but configured storage is not fully ready"
		else
			state_update "already_running" "$CURRENT_PHASE" "success" \
				"observed a healthy running VM with complete storage and QEMU GPU enumeration; live configuration was not modified" \
				"" 0 0 1
			state_finalized=1
			log "VM ${VMID} is already running with complete QEMU GPU enumeration; last_success was updated"
		fi
		return 0
	fi

	wait_for_vm_storage
	load_static_healthy_gpus ||
		die "static GPU inventory/health discovery failed"

	if [[ "$REVALIDATE_QUARANTINED" -eq 1 ]]; then
		cache_reason="explicit full quarantine revalidation requested"
	elif state_has_last_success; then
		mapfile -t cached_gpus < <(read_last_success_gpus)
		if arrays_equal cached_gpus static_healthy_gpus; then
			if apply_cached_gpu_config; then
				mapfile -t configured_gpus < <(configured_gpu_list)
				if production_start "last-success" "${#configured_gpus[@]}"; then
					finalize_success
					return 0
				fi
				if [[ "$last_start_failure_kind" == "orchestration_timeout" ]]; then
					die "production start exceeded the ${START_TIMEOUT}s orchestration deadline; PCI diagnosis would not make memory initialization faster"
				fi
				warn "Cached production start failed; switching to lightweight PCI diagnosis"
				run_gpu_diagnosis ||
					die "lightweight diagnosis failed after cached startup failure"
				if ((production_start_count < MAX_PRODUCTION_STARTS)); then
					if production_start "diagnosed-retry" "${#configured_gpus[@]}"; then
						finalize_success
						return 0
					fi
				fi
				handle_final_start_failure
			fi
			cache_reason="last-success configuration could not be applied"
		else
			cache_reason="healthy visible GPU set differs from last_success"
		fi
	else
		cache_reason="no verified last_success GPU configuration exists"
	fi

	warn "${cache_reason}; starting lightweight PCI diagnosis"
	run_gpu_diagnosis ||
		die "lightweight GPU diagnosis failed"
	if production_start "diagnosed" "${#configured_gpus[@]}"; then
		finalize_success
		return 0
	fi

	bad_gpu="$(extract_unique_fault_gpu "$last_start_log" || true)"
	if [[ -n "$bad_gpu" && "$production_start_count" -lt "$MAX_PRODUCTION_STARTS" ]]; then
		prepare_after_identified_failure "$bad_gpu" \
			"production QEMU log identified this device after lightweight validation" ||
			die "failed to quarantine and reconfigure after identifying ${bad_gpu}"
		if production_start "after-identified-quarantine" "${#configured_gpus[@]}"; then
			finalize_success
			return 0
		fi
	fi

	handle_final_start_failure
}

trap on_exit EXIT
main "$@"

qm_gpus.sh

#!/usr/bin/env bash

set -Eeuo pipefail

VMID=101
MODE="apply"
ONLY_SET=0
ONLY_CSV=""
INCLUDE_QUARANTINED=0
VFIO_PROBE=1
QUARANTINE_BDF=""
QUARANTINE_REASON="manual quarantine"
UNQUARANTINE_BDF=""

STATE_DIR="${STATE_DIR:-/root/.vm-gpu-state}"
BACKUP_DIR="${BACKUP_DIR:-/root/.vm-start-backups}"
LSPCI_TIMEOUT="${LSPCI_TIMEOUT:-3}"
QM_TIMEOUT="${QM_TIMEOUT:-30}"
MAX_HOSTPCI_DEVICES="${MAX_HOSTPCI_DEVICES:-16}"
GPU_VENDOR_ID="${GPU_VENDOR_ID:-10de}"
ALLOW_NON_VFIO="${ALLOW_NON_VFIO:-0}"
QEMU_BIN="${QEMU_BIN:-/usr/bin/kvm}"
VFIO_PROBE_SECONDS="${VFIO_PROBE_SECONDS:-5}"
MAX_VFIO_PROBES="${MAX_VFIO_PROBES:-20}"

CONF=""
QUARANTINE_FILE=""
PROBE_DIR=""
CONFIG_LOCK_FILE=""
rollback_needed=0
backup_file=""
vfio_probe_count=0
VERIFY_REASON=""

declare -a visible_gpus=()
declare -a requested_gpus=()
declare -a selected_gpus=()
declare -a skipped_gpus=()
declare -a compatible_gpus=()
declare -a combination_excluded_gpus=()
declare -a current_keys=()
declare -a original_keys=()
declare -A current_values=()
declare -A current_by_bdf=()
declare -A original_values=()
declare -A skip_reasons=()
declare -A gpu_warnings=()

log() {
	printf '[%s] %s\n' "$(date '+%F %T')" "$*" >&2
}

warn() {
	printf '[%s] WARN: %s\n' "$(date '+%F %T')" "$*" >&2
}

die() {
	printf '[%s] ERROR: %s\n' "$(date '+%F %T')" "$*" >&2
	exit 1
}

usage() {
	cat <<'EOF'
Usage:
  qm_gpus.sh [VMID] [--apply|--dry-run|--list]
                         [--only BDF[,BDF...]]
                         [--include-quarantined]
                         [--vfio-probe|--no-vfio-probe]
  qm_gpus.sh [VMID] --quarantine BDF [--reason TEXT]
  qm_gpus.sh [VMID] --unquarantine BDF

Modes:
  --apply       Discover healthy GPUs and replace hostpci entries with a
                contiguous hostpci0..N configuration. This is the default.
  --dry-run     Print the desired contiguous configuration without changing it.
  --list        Print only selected base BDFs, one per line.

Quarantined GPUs are excluded by default. --include-quarantined temporarily
includes them in selection and probing but never clears quarantine records.
Use start_vm101.sh --revalidate-quarantined for verified recovery and promotion.

Apply mode uses a small 256 MiB QEMU/VFIO realization probe by default. It
validates the whole candidate set without booting the large production VM and
uses the same lightweight probe for binary isolation if realization fails.
EOF
}

require_cmd() {
	command -v "$1" >/dev/null 2>&1 || die "missing required command: $1"
}

normalize_bdf() {
	local raw="${1%%,*}"

	raw="${raw,,}"
	if [[ "$raw" =~ ^[0-9a-f]{2}:[0-9a-f]{2}(\.[0-7])?$ ]]; then
		raw="0000:${raw}"
	fi

	if [[ "$raw" =~ ^[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}(\.[0-7])?$ ]]; then
		printf '%s\n' "${raw%.*}"
		return 0
	fi

	return 1
}

contains_bdf() {
	local needle="$1"
	shift
	local item

	for item in "$@"; do
		[[ "$item" == "$needle" ]] && return 0
	done
	return 1
}

join_csv() {
	local IFS=','
	printf '%s' "$*"
}

vm_status() {
	local output

	if output="$(timeout "$QM_TIMEOUT" perl -MPVE::QemuServer::Helpers -e '
		my $vmid = shift;
		print PVE::QemuServer::Helpers::vm_running_locally($vmid)
			? "running\n" : "stopped\n";
	' "$VMID" 2>/dev/null)"; then
		printf '%s\n' "$output"
		return 0
	fi
	output="$(timeout "$QM_TIMEOUT" qm status "$VMID" 2>/dev/null)" || return 1
	awk '{print $2}' <<<"$output"
}

discover_visible_gpus() {
	local output
	local line
	local bdf
	local base
	local -a found=()

	output="$(timeout "$LSPCI_TIMEOUT" lspci -Dnn 2>/dev/null)" ||
		die "lspci failed or timed out while discovering GPUs"

	while IFS= read -r line; do
		[[ "$line" == *"[${GPU_VENDOR_ID}:"* ]] || continue
		[[ "$line" =~ VGA\ compatible\ controller|3D\ controller|Display\ controller ]] || continue
		bdf="${line%% *}"
		base="$(normalize_bdf "$bdf" || true)"
		[[ -n "$base" ]] || continue
		contains_bdf "$base" "${found[@]}" || found+=("$base")
	done <<<"$output"

	if [[ ${#found[@]} -gt 0 ]]; then
		printf '%s\n' "${found[@]}" | sort -V
	fi
}

quarantine_reason() {
	local bdf="$1"

	[[ -f "$QUARANTINE_FILE" ]] || return 1
	awk -v target="$bdf" '
		$0 !~ /^[[:space:]]*(#|$)/ && $1 == target {
			$1 = ""
			sub(/^[[:space:]]+/, "")
			print
			found = 1
			exit
		}
		END { if (!found) exit 1 }
	' "$QUARANTINE_FILE"
}

quarantine_gpu() {
	local bdf="$1"
	local reason="$2"
	local tmp

	reason="${reason//$'\t'/ }"
	reason="${reason//$'\n'/ }"
	mkdir -p -m 700 "$STATE_DIR"
	touch "$QUARANTINE_FILE"
	chmod 600 "$QUARANTINE_FILE"
	exec 8>"${STATE_DIR}/vm${VMID}.lock"
	flock 8

	tmp="$(mktemp "${STATE_DIR}/vm${VMID}.quarantine.XXXXXX")"
	awk -v target="$bdf" '
		$0 ~ /^[[:space:]]*(#|$)/ || $1 != target { print }
	' "$QUARANTINE_FILE" >"$tmp"
	printf '%s\t%s\t%s\n' "$bdf" "$(date -Is)" "$reason" >>"$tmp"
	install -m 600 "$tmp" "$QUARANTINE_FILE"
	rm -f "$tmp"
	log "Quarantined ${bdf}: ${reason}"
}

unquarantine_gpu() {
	local bdf="$1"
	local tmp

	[[ -f "$QUARANTINE_FILE" ]] || {
		log "No quarantine state exists for VM ${VMID}"
		return 0
	}

	exec 8>"${STATE_DIR}/vm${VMID}.lock"
	flock 8
	tmp="$(mktemp "${STATE_DIR}/vm${VMID}.quarantine.XXXXXX")"
	awk -v target="$bdf" '
		$0 ~ /^[[:space:]]*(#|$)/ || $1 != target { print }
	' "$QUARANTINE_FILE" >"$tmp"
	install -m 600 "$tmp" "$QUARANTINE_FILE"
	rm -f "$tmp"
	log "Removed ${bdf} from VM ${VMID} quarantine"
}

assigned_to_other_vm() {
	local bdf="$1"
	local config
	local other_vmid
	local line
	local value
	local other_bdf

	for config in /etc/pve/qemu-server/*.conf; do
		[[ -e "$config" ]] || continue
		other_vmid="${config##*/}"
		other_vmid="${other_vmid%.conf}"
		[[ "$other_vmid" == "$VMID" ]] && continue

		while IFS= read -r line; do
			[[ "$line" =~ ^hostpci[0-9]+:[[:space:]]*(.+)$ ]] || continue
			value="${BASH_REMATCH[1]}"
			other_bdf="$(normalize_bdf "$value" || true)"
			[[ "$other_bdf" == "$bdf" ]] && return 0
		done < <(awk '/^\[/ { exit } { print }' "$config")
	done

	return 1
}

HEALTH_REASON=""
HEALTH_WARNING=""

gpu_is_healthy() {
	local bdf="$1"
	local endpoint="${bdf}.0"
	local sysfs="/sys/bus/pci/devices/${endpoint}"
	local vendor
	local class
	local config_vendor
	local details
	local link_line
	local driver=""
	local width

	HEALTH_REASON=""
	HEALTH_WARNING=""

	[[ -d "$sysfs" ]] || {
		HEALTH_REASON="PCI endpoint is absent"
		return 1
	}

	vendor="$(<"${sysfs}/vendor")"
	class="$(<"${sysfs}/class")"
	[[ "${vendor,,}" == "0x${GPU_VENDOR_ID}" ]] || {
		HEALTH_REASON="unexpected vendor ${vendor}"
		return 1
	}
	[[ "$class" == 0x03* ]] || {
		HEALTH_REASON="endpoint class ${class} is not a display controller"
		return 1
	}

	config_vendor="$(timeout "$LSPCI_TIMEOUT" setpci -s "$endpoint" VENDOR_ID.w 2>/dev/null || true)"
	config_vendor="${config_vendor,,}"
	[[ "$config_vendor" == "$GPU_VENDOR_ID" ]] || {
		HEALTH_REASON="PCI config space is unreadable (${config_vendor:-no response})"
		return 1
	}

	details="$(timeout "$LSPCI_TIMEOUT" lspci -Dvv -s "$endpoint" 2>/dev/null || true)"
	[[ -n "$details" ]] || {
		HEALTH_REASON="lspci detail read failed"
		return 1
	}

	if grep -Eq 'DevSta:.*FatalErr\+|UESta:.*SDES\+' <<<"$details"; then
		HEALTH_REASON="fatal or Surprise Down status is asserted"
		return 1
	fi

	link_line="$(awk '/LnkSta:/ { print; exit }' <<<"$details")"
	if [[ "$link_line" =~ Width[[:space:]]+x([0-9]+) ]]; then
		width="${BASH_REMATCH[1]}"
		((width > 0)) || {
			HEALTH_REASON="PCIe link width is x0"
			return 1
		}
	else
		HEALTH_REASON="PCIe link status is unavailable"
		return 1
	fi

	[[ -e "${sysfs}/iommu_group" ]] || {
		HEALTH_REASON="IOMMU group is unavailable"
		return 1
	}

	if [[ -L "${sysfs}/driver" ]]; then
		driver="$(basename "$(readlink -f "${sysfs}/driver")")"
	fi
	if [[ -n "$driver" && "$driver" != "vfio-pci" && "$ALLOW_NON_VFIO" != "1" ]]; then
		HEALTH_REASON="function 0 is bound to ${driver}, not vfio-pci"
		return 1
	fi

	if grep -Eq 'CESta:.*(RxErr|BadTLP|BadDLLP|Rollover|Timeout|AdvNonFatalErr)\+' <<<"$details"; then
		HEALTH_WARNING="sticky correctable PCIe status is present"
	fi

	return 0
}

parse_requested_gpus() {
	local item
	local base
	local key
	local -a raw=()

	if [[ "$ONLY_SET" -eq 0 ]]; then
		for key in "${current_keys[@]}"; do
			base="$(normalize_bdf "${current_values[$key]}" || true)"
			[[ -n "$base" ]] || continue
			contains_bdf "$base" "${requested_gpus[@]}" || requested_gpus+=("$base")
		done
		for base in "${visible_gpus[@]}"; do
			contains_bdf "$base" "${requested_gpus[@]}" || requested_gpus+=("$base")
		done
		return 0
	fi

	[[ -z "$ONLY_CSV" || "$ONLY_CSV" == "none" ]] && {
		requested_gpus=()
		return 0
	}

	IFS=',' read -r -a raw <<<"$ONLY_CSV"
	for item in "${raw[@]}"; do
		base="$(normalize_bdf "$item" || true)"
		[[ -n "$base" ]] || die "invalid GPU BDF in --only: ${item}"
		contains_bdf "$base" "${requested_gpus[@]}" || requested_gpus+=("$base")
	done
}

select_gpus() {
	local bdf
	local reason

	mapfile -t visible_gpus < <(discover_visible_gpus)
	parse_requested_gpus

	for bdf in "${requested_gpus[@]}"; do
		if ! contains_bdf "$bdf" "${visible_gpus[@]}"; then
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="not visible in the current host PCI inventory"
			continue
		fi

		if [[ "$INCLUDE_QUARANTINED" -eq 0 ]] && reason="$(quarantine_reason "$bdf" 2>/dev/null)"; then
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="quarantined: ${reason}"
			continue
		fi

		if assigned_to_other_vm "$bdf"; then
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="assigned to another VM"
			continue
		fi

		if gpu_is_healthy "$bdf"; then
			selected_gpus+=("$bdf")
			[[ -n "$HEALTH_WARNING" ]] && gpu_warnings["$bdf"]="$HEALTH_WARNING"
		else
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="$HEALTH_REASON"
		fi
	done

	if [[ "$ONLY_SET" -eq 1 && ${#skipped_gpus[@]} -gt 0 ]]; then
		for bdf in "${skipped_gpus[@]}"; do
			warn "Requested GPU ${bdf} rejected: ${skip_reasons[$bdf]}"
		done
		return 1
	fi
}

prepare_vfio_gpu() {
	local bdf="$1"
	local function_path
	local function_bdf
	local driver=""
	local -a function_paths=()

	shopt -s nullglob
	function_paths=(/sys/bus/pci/devices/"${bdf}".*)
	shopt -u nullglob
	[[ ${#function_paths[@]} -gt 0 ]] || return 1

	for function_path in "${function_paths[@]}"; do
		function_bdf="${function_path##*/}"
		driver=""
		if [[ -L "${function_path}/driver" ]]; then
			driver="$(basename "$(readlink -f "${function_path}/driver")")"
		fi
		[[ "$driver" == "vfio-pci" ]] && continue

		printf 'vfio-pci' >"${function_path}/driver_override" || return 1
		if [[ -n "$driver" ]]; then
			printf '%s' "$function_bdf" >"${function_path}/driver/unbind" || return 1
		fi
		printf '%s' "$function_bdf" >/sys/bus/pci/drivers_probe || return 1

		if [[ ! -L "${function_path}/driver" ]] ||
			[[ "$(basename "$(readlink -f "${function_path}/driver")")" != "vfio-pci" ]]; then
			return 1
		fi
	done
}

prepare_selected_gpus_for_vfio() {
	local bdf
	local -a prepared=()

	modprobe vfio-pci
	for bdf in "${selected_gpus[@]}"; do
		if prepare_vfio_gpu "$bdf"; then
			prepared+=("$bdf")
		else
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="could not bind every PCI function to vfio-pci"
		fi
	done
	selected_gpus=("${prepared[@]}")
}

vfio_probe_subset() {
	local label="$1"
	shift
	local -a subset=("$@")
	local -a command=()
	local -a function_paths=()
	local bdf
	local function_path
	local function_bdf
	local function_number
	local root_port
	local root_addr
	local device_arg
	local index
	local rc=0
	local log_file
	local safe_label="${label//[^a-zA-Z0-9_.-]/_}"

	vfio_probe_count=$((vfio_probe_count + 1))
	((vfio_probe_count <= MAX_VFIO_PROBES)) ||
		die "reached lightweight VFIO probe limit ${MAX_VFIO_PROBES}"

	log_file="${PROBE_DIR}/$(printf '%02d' "$vfio_probe_count")-${safe_label}.log"
	command=(
		"$QEMU_BIN"
		-name "vm${VMID}-vfio-probe"
		-machine "q35,accel=kvm"
		-cpu host
		-m 256M
		-nodefaults
		-display none
		-serial none
		-monitor none
		-no-reboot
		-S
	)

	for index in "${!subset[@]}"; do
		bdf="${subset[$index]}"
		root_port="rp${index}"
		printf -v root_addr '0x%x' "$((index + 2))"
		command+=(
			-device "pcie-root-port,id=${root_port},bus=pcie.0,addr=${root_addr},chassis=$((index + 1))"
		)

		shopt -s nullglob
		function_paths=(/sys/bus/pci/devices/"${bdf}".*)
		shopt -u nullglob
		mapfile -t function_paths < <(printf '%s\n' "${function_paths[@]}" | sort -V)

		for function_path in "${function_paths[@]}"; do
			function_bdf="${function_path##*/}"
			function_number="${function_bdf##*.}"
			device_arg="vfio-pci,host=${function_bdf},id=gpu${index}f${function_number},bus=${root_port},addr=0x0.${function_number}"
			if [[ "$function_number" == "0" && ${#function_paths[@]} -gt 1 ]]; then
				device_arg+=",multifunction=on"
			fi
			command+=(-device "$device_arg")
		done
	done

	log "Lightweight VFIO probe ${vfio_probe_count}: ${label} (${#subset[@]} GPUs, 256 MiB)"
	if timeout --foreground --signal=TERM --kill-after=2 "$VFIO_PROBE_SECONDS" \
		"${command[@]}" >"$log_file" 2>&1; then
		rc=0
	else
		rc=$?
	fi

	if [[ "$rc" -ne 124 ]]; then
		warn "Lightweight VFIO probe failed: ${label} (rc=${rc}, log=${log_file})"
		sed -n '1,100p' "$log_file" | sed 's/^/  /' >&2 || true
		return 1
	fi

	for bdf in "${subset[@]}"; do
		if ! gpu_is_healthy "$bdf"; then
			warn "Post-probe health check failed for ${bdf}: ${HEALTH_REASON}"
			return 1
		fi
	done
	return 0
}

remove_bdf_from_list() {
	local target="$1"
	shift
	local item

	for item in "$@"; do
		[[ "$item" == "$target" ]] || printf '%s\n' "$item"
	done
}

VFIO_FOUND_BAD=""

find_one_vfio_failure() {
	local -a suspects=("$@")
	local -a first_half=()
	local -a second_half=()
	local split
	local candidate

	VFIO_FOUND_BAD=""
	while [[ ${#suspects[@]} -gt 1 ]]; do
		split=$(( (${#suspects[@]} + 1) / 2 ))
		first_half=("${suspects[@]:0:split}")
		second_half=("${suspects[@]:split}")

		if vfio_probe_subset "bisect-${#first_half[@]}-of-${#suspects[@]}" "${first_half[@]}"; then
			suspects=("${second_half[@]}")
		else
			suspects=("${first_half[@]}")
		fi
	done

	[[ ${#suspects[@]} -eq 1 ]] || return 2
	candidate="${suspects[0]}"
	if vfio_probe_subset "confirm-${candidate//:/-}" "$candidate"; then
		return 2
	fi

	VFIO_FOUND_BAD="$candidate"
}

vfio_greedy_compatible_set() {
	local bdf
	local -a trial=()

	compatible_gpus=()
	combination_excluded_gpus=()
	for bdf in "${selected_gpus[@]}"; do
		trial=("${compatible_gpus[@]}" "$bdf")
		if vfio_probe_subset "greedy-${#trial[@]}" "${trial[@]}"; then
			compatible_gpus=("${trial[@]}")
			continue
		fi

		if [[ ${#compatible_gpus[@]} -eq 0 ]] ||
			! vfio_probe_subset "greedy-single-${bdf//:/-}" "$bdf"; then
			quarantine_gpu "$bdf" "confirmed singleton lightweight VFIO probe failure"
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="quarantined after singleton VFIO probe failure"
		else
			combination_excluded_gpus+=("$bdf")
			skipped_gpus+=("$bdf")
			skip_reasons["$bdf"]="excluded for this run after combination-only VFIO failure"
		fi
	done
	selected_gpus=("${compatible_gpus[@]}")
}

run_vfio_diagnostics() {
	local -a remaining=()

	[[ -x "$QEMU_BIN" ]] || die "QEMU binary is not executable: ${QEMU_BIN}"
	mkdir -p -m 700 "$STATE_DIR" "${STATE_DIR}/probes"
	chmod 700 "$STATE_DIR" "${STATE_DIR}/probes"
	PROBE_DIR="${STATE_DIR}/probes/vm${VMID}-$(date '+%F-%H%M%S-%N')"
	mkdir -m 700 "$PROBE_DIR"

	prepare_selected_gpus_for_vfio
	remaining=("${selected_gpus[@]}")
	if [[ ${#remaining[@]} -eq 0 ]]; then
		log "No GPU candidates remain after static and VFIO binding checks"
		return 0
	fi

	if vfio_probe_subset "combined-all" "${remaining[@]}"; then
		log "All ${#remaining[@]} candidates passed the lightweight combined VFIO probe"
		return 0
	fi

	if ! vfio_probe_subset "baseline-no-device"; then
		die "lightweight QEMU baseline failed without PCI devices; refusing to quarantine GPUs"
	fi

	while [[ ${#remaining[@]} -gt 0 ]]; do
		if find_one_vfio_failure "${remaining[@]}"; then
			quarantine_gpu "$VFIO_FOUND_BAD" "confirmed singleton lightweight VFIO probe failure"
			skipped_gpus+=("$VFIO_FOUND_BAD")
			skip_reasons["$VFIO_FOUND_BAD"]="quarantined after singleton VFIO probe failure"
			mapfile -t remaining < <(
				remove_bdf_from_list "$VFIO_FOUND_BAD" "${remaining[@]}"
			)
			if vfio_probe_subset "combined-after-quarantine" "${remaining[@]}"; then
				selected_gpus=("${remaining[@]}")
				return 0
			fi
			continue
		fi

		warn "VFIO failure is combination-dependent; finding the largest compatible ordered set"
		selected_gpus=("${remaining[@]}")
		vfio_greedy_compatible_set
		return 0
	done

	selected_gpus=()
}

load_current_hostpci() {
	local line
	local key
	local value
	local bdf

	current_keys=()
	current_values=()
	current_by_bdf=()

	while IFS= read -r line; do
		[[ "$line" =~ ^(hostpci[0-9]+):[[:space:]]*(.+)$ ]] || continue
		key="${BASH_REMATCH[1]}"
		value="${BASH_REMATCH[2]}"
		current_keys+=("$key")
		current_values["$key"]="$value"
		bdf="$(normalize_bdf "$value" || true)"
		[[ -n "$bdf" ]] && current_by_bdf["$bdf"]="$value"
	done < <(qm config "$VMID")
}

value_for_gpu() {
	local bdf="$1"
	local value="${current_by_bdf[$bdf]:-}"

	if [[ -z "$value" ]]; then
		printf '%s,pcie=1\n' "$bdf"
		return 0
	fi

	if [[ "$value" != *",pcie="* ]]; then
		value="${value},pcie=1"
	fi
	printf '%s\n' "$value"
}

restore_original_config() {
	local key
	local -a active_keys=()
	local -a delete_keys=()
	local -a command=(timeout "$QM_TIMEOUT" qm set "$VMID")
	local -A original_key_set=()

	rollback_needed=0
	warn "Restoring original hostpci configuration for VM ${VMID}"
	mapfile -t active_keys < <(
		qm config "$VMID" | awk -F: '/^hostpci[0-9]+:/ { print $1 }'
	)
	for key in "${original_keys[@]}"; do
		original_key_set["$key"]=1
	done
	for key in "${active_keys[@]}"; do
		[[ -n "${original_key_set[$key]:-}" ]] || delete_keys+=("$key")
	done
	if [[ ${#delete_keys[@]} -gt 0 ]]; then
		command+=(-delete "$(join_csv "${delete_keys[@]}")")
	fi
	for key in "${original_keys[@]}"; do
		command+=("-${key}" "${original_values[$key]}")
	done

	if [[ ${#command[@]} -gt 5 ]] &&
		! "${command[@]}" >/dev/null; then
		warn "Rollback transaction failed for VM ${VMID}"
		return 1
	fi
	if ! verify_original_config; then
		warn "Rollback validation failed: ${VERIFY_REASON}"
		qm config "$VMID" |
			awk '/^hostpci[0-9]+:/ { print "  actual " $0 }' >&2
		return 1
	fi
	log "Rollback restored the original hostpci configuration"
}

cleanup() {
	local rc=$?

	if [[ "$rollback_needed" -eq 1 ]]; then
		if ! restore_original_config; then
			rc=2
		fi
	fi
	trap - EXIT
	exit "$rc"
}

verify_contiguous_config() {
	local index
	local key
	local value
	local actual_bdf
	local expected_bdf
	local -A actual=()

	VERIFY_REASON=""
	while IFS=$'\t' read -r key value; do
		[[ -n "$key" ]] || continue
		actual["$key"]="$value"
	done < <(
		qm config "$VMID" |
			awk -F': ' '/^hostpci[0-9]+:/ { printf "%s\t%s\n", $1, $2 }'
	)

	if [[ ${#actual[@]} -ne ${#selected_gpus[@]} ]]; then
		VERIFY_REASON="entry count is ${#actual[@]}, expected ${#selected_gpus[@]}"
		return 1
	fi
	for index in "${!selected_gpus[@]}"; do
		key="hostpci${index}"
		if [[ -z "${actual[$key]:-}" ]]; then
			VERIFY_REASON="missing ${key}"
			return 1
		fi
		actual_bdf="$(normalize_bdf "${actual[$key]}" || true)"
		expected_bdf="${selected_gpus[$index]}"
		if [[ "$actual_bdf" != "$expected_bdf" ]]; then
			VERIFY_REASON="${key} has ${actual_bdf:-invalid BDF}, expected ${expected_bdf}"
			return 1
		fi
		if [[ "${actual[$key]}" != *",pcie=1"* ]]; then
			VERIFY_REASON="${key} is missing pcie=1: ${actual[$key]}"
			return 1
		fi
	done
}

verify_original_config() {
	local key
	local value
	local -A actual=()

	VERIFY_REASON=""
	while IFS=$'\t' read -r key value; do
		[[ -n "$key" ]] || continue
		actual["$key"]="$value"
	done < <(
		qm config "$VMID" |
			awk -F': ' '/^hostpci[0-9]+:/ { printf "%s\t%s\n", $1, $2 }'
	)

	if [[ ${#actual[@]} -ne ${#original_keys[@]} ]]; then
		VERIFY_REASON="rollback entry count is ${#actual[@]}, expected ${#original_keys[@]}"
		return 1
	fi
	for key in "${original_keys[@]}"; do
		if [[ -z "${actual[$key]:-}" ]]; then
			VERIFY_REASON="rollback is missing ${key}"
			return 1
		fi
		if [[ "${actual[$key]}" != "${original_values[$key]}" ]]; then
			VERIFY_REASON="${key} rollback value differs: actual=${actual[$key]} expected=${original_values[$key]}"
			return 1
		fi
	done
}

apply_hostpci_transaction() {
	local key
	local bdf
	local value
	local index
	local -a delete_keys=()
	local -a command=(timeout "$QM_TIMEOUT" qm set "$VMID")
	local -A desired_key_set=()

	for index in "${!selected_gpus[@]}"; do
		desired_key_set["hostpci${index}"]=1
	done
	for key in "${current_keys[@]}"; do
		[[ -n "${desired_key_set[$key]:-}" ]] || delete_keys+=("$key")
	done
	if [[ ${#delete_keys[@]} -gt 0 ]]; then
		command+=(-delete "$(join_csv "${delete_keys[@]}")")
	fi
	for index in "${!selected_gpus[@]}"; do
		bdf="${selected_gpus[$index]}"
		value="$(value_for_gpu "$bdf")"
		command+=("-hostpci${index}" "$value")
	done

	[[ ${#command[@]} -gt 5 ]] || return 0
	"${command[@]}" >/dev/null
}

apply_config() {
	local status
	local key

	status="$(vm_status || true)"
	[[ "$status" != "running" ]] || die "VM ${VMID} is running; stop it before applying GPU changes"
	((${#selected_gpus[@]} <= MAX_HOSTPCI_DEVICES)) ||
		die "selected ${#selected_gpus[@]} GPUs, exceeding hostpci limit ${MAX_HOSTPCI_DEVICES}"

	load_current_hostpci
	if verify_contiguous_config; then
		log "VM ${VMID} already has the desired contiguous hostpci configuration"
		return 0
	fi
	original_keys=("${current_keys[@]}")
	for key in "${original_keys[@]}"; do
		original_values["$key"]="${current_values[$key]}"
	done

	mkdir -p "$BACKUP_DIR"
	backup_file="${BACKUP_DIR}/vm${VMID}-gpus-$(date '+%F-%H%M%S-%N').conf"
	cp -a "$CONF" "$backup_file"
	log "Saved VM config backup: ${backup_file}"

	rollback_needed=1
	apply_hostpci_transaction ||
		die "atomic hostpci configuration transaction failed"

	if ! verify_contiguous_config; then
		warn "Post-write validation failed: ${VERIFY_REASON}"
		qm config "$VMID" |
			awk '/^hostpci[0-9]+:/ { print "  actual " $0 }' >&2
		die "post-write validation failed; hostpci entries are not contiguous or complete"
	fi
	rollback_needed=0
	if [[ ${#selected_gpus[@]} -eq 0 ]]; then
		log "Applied an empty hostpci GPU set"
	else
		log "Applied ${#selected_gpus[@]} healthy GPUs as contiguous hostpci0..$(( ${#selected_gpus[@]} - 1 ))"
	fi
}

print_plan() {
	local bdf
	local index

	log "Visible ${GPU_VENDOR_ID} display GPUs: ${#visible_gpus[@]}"
	log "Selected healthy GPUs: ${#selected_gpus[@]}"
	for index in "${!selected_gpus[@]}"; do
		bdf="${selected_gpus[$index]}"
		printf 'hostpci%s: %s\n' "$index" "$(value_for_gpu "$bdf")"
		[[ -n "${gpu_warnings[$bdf]:-}" ]] &&
			warn "${bdf}: ${gpu_warnings[$bdf]}"
	done

	for bdf in "${skipped_gpus[@]}"; do
		warn "Skipping ${bdf}: ${skip_reasons[$bdf]}"
	done
}

parse_args() {
	if [[ $# -gt 0 && "$1" != --* ]]; then
		VMID="$1"
		shift
	fi

	while [[ $# -gt 0 ]]; do
		case "$1" in
			--apply)
				MODE="apply"
				shift
				;;
			--dry-run)
				MODE="dry-run"
				shift
				;;
			--list)
				MODE="list"
				shift
				;;
			--only)
				[[ $# -ge 2 ]] || die "--only requires a comma-separated BDF list or 'none'"
				ONLY_SET=1
				ONLY_CSV="$2"
				shift 2
				;;
			--include-quarantined)
				INCLUDE_QUARANTINED=1
				shift
				;;
			--vfio-probe)
				VFIO_PROBE=1
				shift
				;;
			--no-vfio-probe)
				VFIO_PROBE=0
				shift
				;;
			--quarantine)
				[[ $# -ge 2 ]] || die "--quarantine requires a BDF"
				MODE="quarantine"
				QUARANTINE_BDF="$2"
				shift 2
				;;
			--unquarantine)
				[[ $# -ge 2 ]] || die "--unquarantine requires a BDF"
				MODE="unquarantine"
				UNQUARANTINE_BDF="$2"
				shift 2
				;;
			--reason)
				[[ $# -ge 2 ]] || die "--reason requires text"
				QUARANTINE_REASON="$2"
				shift 2
				;;
			-h|--help)
				usage
				exit 0
				;;
			*)
				die "unknown argument: $1"
				;;
		esac
	done
}

main() {
	local bdf
	local status

	parse_args "$@"
	[[ "$VMID" =~ ^[1-9][0-9]*$ ]] || die "invalid VMID: ${VMID}"
	CONF="/etc/pve/qemu-server/${VMID}.conf"
	QUARANTINE_FILE="${STATE_DIR}/vm${VMID}.quarantine"
	CONFIG_LOCK_FILE="${STATE_DIR}/vm${VMID}.config.lock"

	[[ $EUID -eq 0 ]] || die "this script must run as root"
	require_cmd qm
	require_cmd lspci
	require_cmd setpci
	require_cmd timeout
	require_cmd flock
	require_cmd modprobe
	require_cmd perl
	[[ -f "$CONF" ]] || die "VM config not found: ${CONF}"

	case "$MODE" in
		quarantine)
			bdf="$(normalize_bdf "$QUARANTINE_BDF" || true)"
			[[ -n "$bdf" ]] || die "invalid quarantine BDF: ${QUARANTINE_BDF}"
			quarantine_gpu "$bdf" "$QUARANTINE_REASON"
			return 0
			;;
		unquarantine)
			bdf="$(normalize_bdf "$UNQUARANTINE_BDF" || true)"
			[[ -n "$bdf" ]] || die "invalid unquarantine BDF: ${UNQUARANTINE_BDF}"
			unquarantine_gpu "$bdf"
			return 0
			;;
	esac

	load_current_hostpci
	select_gpus || die "one or more explicitly requested GPUs failed health policy"

	case "$MODE" in
		list)
			if [[ ${#selected_gpus[@]} -gt 0 ]]; then
				printf '%s\n' "${selected_gpus[@]}"
			fi
			;;
		dry-run)
			print_plan
			log "Dry-run performs static checks only; lightweight VFIO realization is not executed"
			;;
		apply)
			status="$(vm_status || true)"
			[[ "$status" != "running" ]] ||
				die "VM ${VMID} is running; stop it before applying GPU changes"
			mkdir -p -m 700 "$STATE_DIR"
			exec 7>"$CONFIG_LOCK_FILE"
			flock -n 7 ||
				die "another GPU configuration job is active for VM ${VMID}"
			if [[ "$VFIO_PROBE" -eq 1 ]]; then
				run_vfio_diagnostics
			fi
			print_plan
			apply_config
			;;
		*)
			die "unsupported mode: ${MODE}"
			;;
	esac
}

trap cleanup EXIT
main "$@"

六、把 20TB HDD 配置成大容量数据存储并挂给 VM

在 PVE 中查看磁盘信息

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,SERIAL

txt

NAME                            SIZE TYPE FSTYPE      MOUNTPOINT MODEL               SERIAL
sda                            18.2T disk                        WUH722020CLE604     PP****8P
nvme0n1                       894.3G disk                        INTEL SSDPF2KX960HZ PHA************QGN
├─nvme0n1p1                    1007K part
├─nvme0n1p2                       1G part vfat        /boot/efi
└─nvme0n1p3                     893G part LVM2_member
nvme1n1                         3.5T disk LVM2_member            INTEL SSDPF2KX038TZ PHA************AGN
├─vmdata-vmdata_tmeta          15.9G lvm
└─vmdata-vmdata_tdata           3.5T lvm

这里的 /dev/sda 就是 20TB 的 HDD。

查看详细信息：

fdisk -l /dev/sda

txt

Disk /dev/sda: 18.19 TiB, 20000588955648 bytes, 39063650304 sectors
Disk model: WUH722020CLE604
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

清理磁盘

如果是全新空盘，这一步可以跳过。

wipefs -a /dev/sda

创建 GPT 分区和单一大分区

大容量适合 Directory 存储 + GPT 分区。

# apt install -y parted
parted -a optimal /dev/sda --script mklabel gpt
parted -a optimal /dev/sda --script mkpart primary ext4 1MiB 100%
partprobe /dev/sda

应当看到新分区 /dev/sda1：

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT /dev/sda

txt

NAME    SIZE TYPE FSTYPE MOUNTPOINT
sda    18.2T disk
└─sda1 18.2T part

格式化为 ext4

mkfs.ext4 -L hdd20t /dev/sda1

等待运行完成，然后检查：

lsblk --fs /dev/sda

txt

NAME   FSTYPE FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
└─sda1 ext4   1.0   hdd20t 9e******-****-****-****-**********7b
root@pve:~# blkid /dev/sda1

或者：

blkid /dev/sda1

txt

/dev/sda1: LABEL="hdd20t" UUID="9e******-****-****-****-**********7b" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="e6******-****-****-****-**********5e"

这里的 UUID 就是后面要用来挂载的标识符。

挂载

创建挂载点：

mkdir -p /mnt/pve/hdd20t

启动时自动挂载：

nano /etc/fstab

添加一行：

UUID=9e******-****-****-****-**********7b /mnt/pve/hdd20t ext4 defaults,nofail 0 2

系统重新读取并挂载：

systemctl daemon-reload
mount -a

查看挂载情况：

findmnt /mnt/pve/hdd20t

txt

TARGET          SOURCE    FSTYPE OPTIONS
/mnt/pve/hdd20t /dev/sda1 ext4   rw,relatime

df -h /mnt/pve/hdd20t

txt

ilesystem      Size  Used Avail Use% Mounted on
/dev/sda1        19T  2.1M   18T   1% /mnt/pve/hdd20t

注册为 Directory 存储

pvesm add dir hdd20t --path /mnt/pve/hdd20t --content images,backup,iso,vztmpl,rootdir

images: VM 磁盘
rootdir: LXC 容器
backup: 备份
iso: ISO 镜像
vztmpl: 容器模板

查看状态：

pvesm status

txt

Name             Type     Status     Total (KiB)      Used (KiB) Available (KiB)        %
hdd20t            dir     active     19453053208            2096     18476443576    0.00%
local             dir     active        98497780        53327176        40121056   54.14%
local-lvm     lvmthin     active       794337280               0       794337280    0.00%
vmdata        lvmthin     active      3717050368       405901900      3311148467   10.92%

cat /etc/pve/storage.cfg

txt

...
dir: hdd20t
        path /mnt/pve/hdd20t
        content iso,vztmpl,backup,rootdir,images

将这个存储加给 VM

在配置中查看槽位信息：

qm config 101

...
parent: AI122-2025-1204-0606
scsi0: vmdata:vm-101-disk-1,discard=on,iothread=1,size=2T,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=f9******-****-****-****-**********bf
sockets: 1
vmgenid: ac******-****-****-****-**********f5

可以看到：

scsi0 是系统盘，已经在 vmdata 上
scsihw 是 virtio-scsi-single
目前还有一个空闲的 SCSI 插槽 scsi1

因此可以把这个新的存储挂在 scsi1 上：

qm set 101 --scsi1 hdd20t:4096,format=raw,iothread=1

给 VM 101
新增一块挂在 scsi1 的磁盘
存储位置在 hdd20t
大小 4096 GiB，也就是约 4TB，可以按需调整，见下一小节
格式 raw
开启 iothread=1

txt

update VM 101: -scsi1 hdd20t:4096,format=raw,iothread=1
Formatting '/mnt/pve/hdd20t/images/101/vm-101-disk-0.raw', fmt=raw size=4398046511104 preallocation=off
scsi1: successfully created disk 'hdd20t:101/vm-101-disk-0.raw,iothread=1,size=4T'

再次查看配置：

qm config 101 | grep scsi

txt

...
scsi1: hdd20t:101/vm-101-disk-0.raw,iothread=1,size=4T

表明已经成功添加了新的磁盘。

优化磁盘占用

将 ext4 保留块比例降到 1%，提高空间利用率：

tune2fs -m 1 /dev/sda1

tune2fs 1.47.2 (1-Jan-2025)
Setting reserved blocks percentage to 1% (48829557 blocks)

查看当前保留块比例：

tune2fs -l /dev/sda1 | egrep 'Reserved block count|Block size'

Reserved block count:     48829557
Block size:               4096

查看当前磁盘使用情况：

df -h /mnt/pve/hdd20t

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        19T  2.1M   18T   1% /mnt/pve/hdd20t

预留 100GB 给宿主机和文件系统缓冲，剩下的都给 VM：

avail_gib=$(df --output=avail -BG /mnt/pve/hdd20t | tail -1 | tr -dc '0-9')
target_gib=$((avail_gib - 100))
echo "$target_gib"

考虑到 ext4 + 标准 4KiB 块大小，单文件大小上限是 16TB。不能直接将整个 18TB 分配给 VM，否则可能会遇到下面的报错：

txt

# qm resize 101 scsi1 ${target_gib}G
VM 101 qmp command 'block_resize' failed - Could not resize file: File too large

因此 VM 分配 16TB：

qm resize 101 scsi1 16380G

查看配置：

qm config 101 | grep scsi1

scsi1: hdd20t:101/vm-101-disk-0.raw,iothread=1,size=16380G

在 VM 中添加磁盘

上面的命令都是在 PVE 宿主机上执行的。下面的命令是在 VM 里执行的。

下面的很多命令可能似曾相识，但是需要注意区分。

上面的工作是在 PVE 中格式化 hdd20t 这个宿主机存储池，下面的工作是在 VM 中格式化 scsi1 这个虚拟磁盘。

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL

...
sda        2T disk                                              QEMU HARDDISK
├─sda1   512M part vfat     /boot/efi
└─sda2     2T part ext4     /
sdb       16T disk                                              QEMU HARDDISK

这里的 sdb 就是新加的 16TB 磁盘。

在 VM 中创建 GPT 分区和单一大分区

sudo parted -a optimal /dev/sdb --script mklabel gpt
sudo parted -a optimal /dev/sdb --script mkpart primary ext4 1MiB 100%
sudo partprobe /dev/sdb

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT /dev/sdb

NAME   SIZE TYPE FSTYPE MOUNTPOINT
sdb     16T disk
└─sdb1  16T part

在 VM 中格式化文件系统

sudo mkfs.ext4 -L data /dev/sdb1

等待一会，运行完成。然后查看：

lsblk --fs /dev/sdb

NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sdb
└─sdb1 ext4   1.0   data  a6******-****-****-****-**********b2

在 VM 中挂载

sudo mkdir -p /media/data
sudo mount /dev/sdb1 /media/data

查看挂载情况：

df -h /media/data

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        16T   28K   16T   1% /media/data

设置开机自动挂载

查看 uuid：

sudo blkid /dev/sdb1

/dev/sdb1: LABEL="data" UUID="a6******-****-****-****-**********b2" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="fb******-****-****-****-**********1c"

sudo nano /etc/fstab，添加一行：

UUID=a6******-****-****-****-**********b2 /media/data ext4 defaults,nofail 0 2

挂载：

sudo mount -a

查看挂载情况：

df -h /media/data

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        16T   28K   16T   1% /media/data

七、使用自动化脚本和配置

本节记录一套可复用的自动化脚本，用来在 PVE 宿主机上创建 Ubuntu VM，并在 VM 内继续完成基础环境、Tailscale、v2ray、HDD 挂载、NVIDIA 驱动、CUDA/NVCC、zsh/tmux、.gd.sh、conda、Python 工具、Docker 和 NVIDIA Container Toolkit 安装。

脚本模板保存在：

txt

docs/notes/scripts/pve-ubuntu/
├── pve_ubuntu.sh
├── pve_ubuntu.yaml
├── setup_ubuntu.sh
└── setup_ubuntu.yaml

.chats/pve-ubuntu/ 中保留的是本次实际运行用的完整脚本和配置，可能包含真实密码、IP、磁盘 by-id、GPU PCI 地址等信息，并且不会随文档提交。docs/notes/scripts/pve-ubuntu/ 中保存的是脱敏后的模板备份，用于以后在其他 PVE 宿主机或 VM 上参考和改写。

其中 pve_ubuntu.sh 在 PVE 宿主机上运行，负责创建 VM、准备 cloud-init、配置 GPU 直通、格式化并挂载 HDD、创建 PVE Directory Storage、给 VM 挂载 HDD-backed 数据盘。setup_ubuntu.sh 会被写入 VM，并在 Ubuntu 内运行，负责安装软件包、启动 SSH/Tailscale/v2ray、挂载数据盘、安装 NVIDIA 驱动和 CUDA Toolkit，并按照 xeon 的通用习惯生成 zsh/tmux/conda/Docker 配置。模板中不包含 xeon 上的私有环境变量。

v2ray 的通用安装脚本和客户端配置不在本目录重复保存，直接复用：

docs/notes/scripts/v2ray-install-release.sh
docs/notes/configs/v2ray-client-config.json
详细说明见：安装 v2ray

运行前写好的配置

运行前需要准备并修改这些文件：

pve_ubuntu.yaml：PVE 宿主机侧配置，包含源宿主机、VM 参数、网络、GPU、HDD。
setup_ubuntu.yaml：Ubuntu VM 内部配置，包含用户、软件包、Tailscale、v2ray、HDD、NVIDIA/CUDA、Git、dotfiles、conda、Python 工具、Docker、NVIDIA Container Toolkit。
pve_ubuntu.sh：PVE 侧执行脚本。
setup_ubuntu.sh：Ubuntu 侧执行脚本，会被 pve_ubuntu.sh 写入 cloud-init。

这些模板中的真实 IP、密码、磁盘序列、GPU PCI 地址、Git 信息都已经用占位符脱敏。xeon、pve、qve、ai122、bj123 只是用于区分宿主机或 VM 的标识，可以按场景保留或修改。

占位符说明

pve_ubuntu.yaml 中常见占位符：

占位符	含义
`<SOURCE_PVE_LAN_IP>`	用来复制 ISO 和 v2ray 配置的源 PVE 宿主机 LAN IP，例如 `pve` 的局域网地址。
`<SOURCE_PVE_ROOT_PASSWORD>`	源 PVE 宿主机 root 密码；脚本通过 `sshpass/scp` 复制文件。
`<SOURCE_UBUNTU_ISO_PATH>`	源 PVE 上已有的 Ubuntu ISO 路径。
`<VM_ID>`	新 VM 的 PVE ID，例如 `123`。
`<VM_LAN_IP>`	新 Ubuntu VM 的静态 LAN IP。
`<LAN_GATEWAY_IP>`	局域网网关。
`<LAN_DNS_IP>` / `<PUBLIC_DNS_IP>`	DNS 服务器。
`<UBUNTU_USER>` / `<UBUNTU_USER_PASSWORD>` / `<UBUNTU_FULL_NAME>`	VM 内创建的 Ubuntu 用户信息。
`<GPU_PCI_ADDRESS_*>`	要直通给 VM 的 GPU PCI 地址，例如通过 `lspci -Dnn
`<GPU_VENDOR_DEVICE_ID>`	GPU 显卡功能的 vendor/device id，例如 `10de:2206`。
`<GPU_AUDIO_VENDOR_DEVICE_ID>`	GPU HDMI/DP Audio 功能的 vendor/device id，例如 `10de:1aef`。
`<GPU_PCI_ADDRESS_TO_EXCLUDE>`	已知会导致 VM 启动失败或暂不直通的 GPU，可留作记录。
`<HDD_DISK_BY_ID>`	HDD 的 `/dev/disk/by-id/` 稳定设备名，必须在目标 PVE 上重新确认。
`<PVE_HDD_LABEL>`	PVE 宿主机上 HDD 分区的文件系统标签。
`<PVE_HDD_MOUNTPOINT>`	PVE 宿主机上 HDD 的挂载点，例如 `/mnt/hdd-data`。
`<PVE_HDD_STORAGE_NAME>`	PVE Directory Storage 名称。
`<VM_DATA_DISK_SIZE_GB>`	挂给 VM 的数据盘大小，单位是 GB，例如 `7000`。

setup_ubuntu.yaml 中常见占位符：

占位符	含义
`tailscale.auth_key`	可选 Tailscale auth key；默认留空并把 `tailscale.up` 设为 `false`，后续手动运行 `tailscale up`。
`<GIT_USER_NAME>` / `<GIT_USER_EMAIL>`	VM 内 Git 全局配置。
`<VM_HDD_LABEL>`	Ubuntu VM 内数据盘分区的文件系统标签。
`<VM_HDD_MOUNTPOINT>`	Ubuntu VM 内数据盘挂载点，例如 `/media/data`。
`dotfiles.gd_url`	`.gd.sh` 的下载地址；如果 cloud-init 已写入 `dotfiles.gd_source`，优先使用本地文件。
`conda.installer_url`	Miniconda 安装脚本地址，模板默认使用 TUNA 镜像。
`conda.env_name` / `conda.python_version`	自动创建的 conda 环境名和 Python 版本。
`docker.http_proxy` / `docker.https_proxy`	Docker daemon 代理。若 v2ray `new.json` 已开启 `11119`，可填 `http://127.0.0.1:11119`。

v2ray 配置不在这里另写一份模板。pve_ubuntu.sh 会从 source.v2ray_config_dir 复制源宿主机上的 config.json 和 new.json，再通过 cloud-init 写入 VM；如果需要从头生成客户端配置，参考安装 v2ray 和 docs/notes/configs/v2ray-client-config.json。

阶段开关

两份 YAML 都支持阶段开关：

yaml

global:
  mode: auto

stages:
  gpu_passthrough:
    mode: auto

可选值：

auto：自动运行该阶段。
manual 或 skip：跳过该阶段。
confirm：运行时询问是否执行。

如果某个阶段已经完成，可以把它改成 skip。如果只想补跑某一阶段，可以把其他阶段设为 skip，目标阶段设为 auto。

PVE 侧主要阶段：

txt

install_host_packages
copy_iso
copy_v2ray_config
create_vm
hdd_storage
attach_hdd
gpu_passthrough
start_vm

Ubuntu 侧主要阶段：

txt

apt_sources
base_packages
qemu_guest_agent
ssh
tailscale
v2ray
hdd_mount
nvidia_driver
cuda
git
dotfiles
conda
python_tools
docker
nvidia_container
zsh
desktop

其中 NVIDIA 相关阶段的职责边界如下：

nvidia_driver：只安装 Ubuntu VM 内核态 NVIDIA 驱动，目标是让 nvidia-smi 正常。
cuda：只安装 CUDA Toolkit / nvcc，不负责 Docker runtime。
docker：只安装 Docker Engine、Compose、registry mirror、daemon 代理和用户组。
nvidia_container：只在 Docker 和 nvidia-smi 都正常后，安装 NVIDIA Container Toolkit，并通过 nvidia-ctk runtime configure --runtime=docker 合并更新 Docker runtime。

在 PVE 宿主机上运行

先把模板复制到目标 PVE 宿主机，例如 qve：

scp -r docs/notes/scripts/pve-ubuntu root@qve:/root/pve-ubuntu
scp docs/notes/scripts/v2ray-install-release.sh root@qve:/root/pve-ubuntu/

登录目标 PVE：

ssh root@qve
cd /root/pve-ubuntu

按目标机器修改配置：

nano pve_ubuntu.yaml
nano setup_ubuntu.yaml
chmod +x v2ray-install-release.sh

至少需要确认：

source.ip、source.password、source.iso_path
vm.id、vm.name、vm.hostname
vm.vga，纯 SSH/算力卡场景可设为 none
network.ipv4、network.gateway4、network.dns
ubuntu.user、ubuntu.password
gpu_passthrough.pci_addresses、gpu_passthrough.vfio_ids
hdd.disk_by_id、hdd.partition
hdd.wipe_existing
hdd.vm_disk.size
source.v2ray_config_dir，以及源宿主机中是否存在 config.json 和 new.json

如果某张 GPU 是宿主机的 boot VGA，但仍希望作为 VM 的算力卡直通，可以先确认：

cat /sys/bus/pci/devices/<GPU_PCI_ADDRESS>.0/boot_vga

值为 1 表示它是宿主机启动显卡。纯计算用途下，可以在 gpu_passthrough.pci_addresses 中给这张卡加 rombar=0，并把 vm.vga 设为 none：

yaml

vm:
  vga: none

gpu_passthrough:
  pci_addresses:
    - "0000:02:00"
    - "0000:03:00"
    - "0000:04:00,rombar=0"

这会让 PVE 的图形 Console 预期不可用或黑屏，但 SSH、guest agent、NVIDIA/CUDA 计算不受影响。

WARNING

hdd.wipe_existing: true 会格式化目标 HDD。换机器前必须用下面命令确认设备确实是要清空的数据盘：

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,SERIAL
ls -l /dev/disk/by-id/

运行 PVE 侧脚本：

bash pve_ubuntu.sh pve_ubuntu.yaml

如果脚本配置了 GPU 直通并提示需要重启，执行：

reboot

宿主机重新上线后再次运行同一命令。已经完成的阶段会按配置和现有状态跳过。

cd /root/pve-ubuntu
bash pve_ubuntu.sh pve_ubuntu.yaml

在 Ubuntu VM 内运行或补跑

正常情况下，setup_ubuntu.sh 和 setup_ubuntu.yaml 会通过 cloud-init 写入 VM 的 /opt/bj123-setup/，并在首次启动时自动运行。

如果需要手动补跑：

ssh <UBUNTU_USER>@<VM_LAN_IP>
sudo bash /opt/bj123-setup/setup_ubuntu.sh /opt/bj123-setup/setup_ubuntu.yaml

如果没有配置 Tailscale auth key，手动运行：

sudo tailscale up --hostname=bj123

命令会输出一个授权链接。复制链接到浏览器中确认授权后，VM 会加入 tailnet。

Ubuntu 22.04 Kernel 版本

Ubuntu 22.04.5 LTS 可能运行 GA kernel 5.15，也可能运行 HWE kernel 6.8。这不是 /etc/os-release 里的 Ubuntu 发行版号，而是内核分支差异。

对纯 SSH 和算力卡 VM，只要下面内容正常，就不需要为了和其他机器一致而升级 kernel：

nvidia-smi 能看到全部 GPU。
nvcc --version 正常。
nvidia-container-cli info 正常。
SSH、Tailscale、qemu guest agent、Docker 正常。

升级到 HWE kernel 会触发 NVIDIA DKMS 重新构建并需要重启，反而会增加变量。只有在明确需要新内核功能或当前驱动/硬件存在内核相关问题时，再考虑升级。

运行后自动生成的内容

PVE 宿主机上会生成或修改：

/root/pve-ubuntu/：运行目录，包含脚本、配置、复制来的 ISO 和 v2ray 配置。
/root/pve-ubuntu/seed/：cloud-init 的 user-data、meta-data、network-config。
/var/lib/vz/template/iso/<vm>-cidata.iso：cloud-init seed ISO。
/etc/default/grub 或 /etc/kernel/cmdline：IOMMU/VFIO 内核参数。
/etc/modules-load.d/vfio.conf
/etc/modprobe.d/vfio.conf
/etc/modprobe.d/blacklist-nvidia-passthrough.conf
/etc/fstab：HDD 自动挂载。
PVE storage：例如 <PVE_HDD_STORAGE_NAME>。
VM 配置：scsi0 系统盘、scsi1 HDD-backed 数据盘、hostpci* GPU 直通、efidisk0。

Ubuntu VM 内会生成或修改：

/opt/bj123-setup/：VM 内脚本和配置。
/var/log/setup_ubuntu.log
/usr/local/etc/v2ray/config.json
/usr/local/etc/v2ray/new.json
v2ray.service
v2ray@new.service
/etc/fstab：数据盘自动挂载。
<VM_HDD_MOUNTPOINT>：数据盘挂载点。
/etc/profile.d/cuda.sh
/usr/local/cuda
~/.zshrc、~/.zshenv：zsh prompt、常用 alias、.gd.sh、conda、CUDA、Hugging Face mirror、数据盘环境变量。
~/.tmux.conf：以 xeon 通用配置为参考的 tmux 配置和插件入口。
~/.gd.sh
~/.pip/pip.conf
~/.condarc
~/miniconda3/ 和 conda env，例如 ai。
/etc/docker/daemon.json
/etc/systemd/system/docker.service.d/proxy.conf
/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
/etc/apt/sources.list.d/nvidia-container-toolkit.list
Tailscale 状态和主机名。

验证命令

PVE 宿主机上检查：

cat /proc/cmdline
dmesg | grep -m1 -E "DMAR: IOMMU enabled|IOMMU enabled"
lspci -Dnnk | awk '/NVIDIA/{print; n=1; next} n && /Kernel driver in use/{print; n=0}'
qm config <VM_ID>
pvesm status
findmnt <PVE_HDD_MOUNTPOINT>

Ubuntu VM 内检查：

hostname
tailscale ip -4
systemctl is-active qemu-guest-agent ssh tailscaled v2ray v2ray@new
ss -ltnp | grep -E ":1111(0|1|8|9)"
findmnt <VM_HDD_MOUNTPOINT>
df -h <VM_HDD_MOUNTPOINT>
nvidia-smi
nvcc --version
nvidia-container-cli info
docker info | grep -i runtime
zsh -ic 'echo $CONDA_DEFAULT_ENV; python --version; command -v gd gpustat pipreqs'
mokutil --sb-state

样例脚本和配置

pve_ubuntu.sh

#!/usr/bin/env bash
set -euo pipefail

CONFIG="${1:-$(dirname "$0")/pve_ubuntu.yaml}"
BASE_DIR="$(cd "$(dirname "$0")" && pwd)"
LOG_FILE="/var/log/pve_ubuntu.log"

exec > >(tee -a "$LOG_FILE") 2>&1

log() {
  printf '[%s] %s\n' "$(date '+%F %T')" "$*"
}

need_root() {
  if [[ "${EUID}" -ne 0 ]]; then
    echo "Run as root on the PVE host." >&2
    exit 1
  fi
}

ensure_yaml() {
  if python3 - <<'PY' >/dev/null 2>&1
import yaml
PY
  then
    return
  fi
  apt-get update
  DEBIAN_FRONTEND=noninteractive apt-get install -y python3-yaml
}

yaml_get() {
  local path="$1"
  local default="${2:-}"
  python3 - "$CONFIG" "$path" "$default" <<'PY'
import sys, yaml
cfg_path, key_path, default = sys.argv[1:4]
with open(cfg_path, "r", encoding="utf-8") as f:
    data = yaml.safe_load(f) or {}
cur = data
for part in key_path.split("."):
    if isinstance(cur, dict) and part in cur:
        cur = cur[part]
    else:
        print(default)
        sys.exit(0)
if cur is None:
    print(default)
elif isinstance(cur, bool):
    print("true" if cur else "false")
elif isinstance(cur, list):
    print("\n".join(str(x) for x in cur))
else:
    print(cur)
PY
}

stage_mode() {
  local stage="$1"
  local mode
  mode="$(yaml_get "stages.${stage}.mode" "")"
  if [[ -z "$mode" ]]; then
    mode="$(yaml_get "global.mode" "confirm")"
  fi
  printf '%s' "$mode"
}

run_stage() {
  local stage="$1"
  local mode
  mode="$(stage_mode "$stage")"
  case "$mode" in
    auto) return 0 ;;
    manual|skip) log "skip stage ${stage} (mode=${mode})"; return 1 ;;
    confirm)
      read -r -p "Run stage ${stage}? [y/N] " answer
      [[ "${answer,,}" == y* ]]
      ;;
    *) log "skip stage ${stage} (unknown mode=${mode})"; return 1 ;;
  esac
}

install_host_packages() {
  DEBIAN_FRONTEND=noninteractive apt-get update
  DEBIAN_FRONTEND=noninteractive apt-get install -y python3-yaml sshpass wget curl genisoimage parted e2fsprogs
}

copy_iso_from_source() {
  local src_ip src_user src_pass src_iso iso_dir dst_iso
  src_ip="$(yaml_get source.ip)"
  src_user="$(yaml_get source.user root)"
  src_pass="$(yaml_get source.password)"
  src_iso="$(yaml_get source.iso_path)"
  iso_dir="$(yaml_get vm.iso_storage_dir /var/lib/vz/template/iso)"
  dst_iso="${iso_dir}/$(basename "$src_iso")"
  mkdir -p "$iso_dir"
  if [[ -f "$dst_iso" ]]; then
    log "ISO already exists: ${dst_iso}"
    return
  fi
  log "copy ISO from ${src_user}@${src_ip}:${src_iso} to ${dst_iso}"
  sshpass -p "$src_pass" scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "${src_user}@${src_ip}:${src_iso}" "$dst_iso"
}

copy_v2ray_config_from_source() {
  local src_ip src_user src_pass src_dir workdir dst_dir
  src_ip="$(yaml_get source.ip)"
  src_user="$(yaml_get source.user root)"
  src_pass="$(yaml_get source.password)"
  src_dir="$(yaml_get source.v2ray_config_dir /usr/local/etc/v2ray)"
  workdir="$(yaml_get global.workdir /root/pve-ubuntu)"
  dst_dir="${workdir}/v2ray"
  mkdir -p "$dst_dir"
  log "copy v2ray config from ${src_user}@${src_ip}:${src_dir}"
  sshpass -p "$src_pass" scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "${src_user}@${src_ip}:${src_dir}/"* "$dst_dir/" || true
}

download_cloud_image() {
  local url img
  url="$(yaml_get vm.cloud_image_url)"
  img="$(yaml_get vm.cloud_image_path)"
  mkdir -p "$(dirname "$img")"
  if [[ -s "$img" ]]; then
    log "cloud image already exists: ${img}"
    return
  fi
  log "download cloud image: ${url}"
  wget -O "${img}.tmp" "$url"
  mv "${img}.tmp" "$img"
}

password_hash() {
  local password="$1"
  openssl passwd -6 "$password"
}

write_b64_file_entry() {
  local path="$1"
  local dst="$2"
  local perms="${3:-0644}"
  [[ -f "$path" ]] || return
  {
    echo "  - path: ${dst}"
    echo "    permissions: '${perms}'"
    echo "    encoding: b64"
    echo "    content: $(base64 -w0 "$path")"
  } >>"$USER_DATA"
}

create_cloud_init_seed() {
  local workdir seed_dir user_data meta_data network_config cidata_iso vmid hostname username password full_name timezone ssh_auth ip prefix gateway dns password_hash_value
  workdir="$(yaml_get global.workdir /root/pve-ubuntu)"
  vmid="$(yaml_get vm.id)"
  hostname="$(yaml_get vm.hostname)"
  username="$(yaml_get ubuntu.user)"
  password="$(yaml_get ubuntu.password)"
  full_name="$(yaml_get ubuntu.full_name "$username")"
  timezone="$(yaml_get ubuntu.timezone Asia/Shanghai)"
  ssh_auth="$(yaml_get ubuntu.ssh_password_auth true)"
  ip="$(yaml_get network.ipv4)"
  prefix="$(yaml_get network.prefix 24)"
  gateway="$(yaml_get network.gateway4)"
  mapfile -t dns < <(yaml_get network.dns "192.168.31.1")
  password_hash_value="$(password_hash "$password")"
  seed_dir="${workdir}/seed-${vmid}"
  cidata_iso="$(yaml_get vm.iso_storage_dir /var/lib/vz/template/iso)/${hostname}-cidata.iso"
  mkdir -p "$seed_dir" "$(dirname "$cidata_iso")"

  USER_DATA="${seed_dir}/user-data"
  meta_data="${seed_dir}/meta-data"
  network_config="${seed_dir}/network-config"

  cat >"$USER_DATA" <<EOF
#cloud-config
hostname: ${hostname}
manage_etc_hosts: true
timezone: ${timezone}
locale: $(yaml_get ubuntu.locale en_US.UTF-8)
ssh_pwauth: ${ssh_auth}
disable_root: false
users:
  - default
  - name: ${username}
    gecos: ${full_name}
    shell: /bin/bash
    lock_passwd: false
    passwd: '${password_hash_value}'
    groups: [adm, cdrom, dip, lxd, plugdev, sudo]
    sudo: ['ALL=(ALL) ALL']
package_update: true
packages:
  - openssh-server
  - qemu-guest-agent
  - python3-yaml
write_files:
EOF
  write_b64_file_entry "${BASE_DIR}/setup_ubuntu.sh" "/opt/bj123-setup/setup_ubuntu.sh" "0755"
  write_b64_file_entry "${BASE_DIR}/setup_ubuntu.yaml" "/opt/bj123-setup/setup_ubuntu.yaml" "0644"
  write_b64_file_entry "${BASE_DIR}/v2ray-install-release.sh" "/opt/bj123-setup/v2ray-install-release.sh" "0755"
  if [[ -f "${BASE_DIR}/dotfiles/.gd.sh" ]]; then
    write_b64_file_entry "${BASE_DIR}/dotfiles/.gd.sh" "/opt/bj123-setup/dotfiles/.gd.sh" "0644"
  fi
  write_b64_file_entry "${workdir}/v2ray/config.json" "/opt/bj123-setup/v2ray/config.json" "0644"
  write_b64_file_entry "${workdir}/v2ray/new.json" "/opt/bj123-setup/v2ray/new.json" "0644"
  cat >>"$USER_DATA" <<EOF
runcmd:
  - systemctl enable --now ssh
  - systemctl enable --now qemu-guest-agent
  - chmod +x /opt/bj123-setup/setup_ubuntu.sh
  - [ bash, /opt/bj123-setup/setup_ubuntu.sh, /opt/bj123-setup/setup_ubuntu.yaml ]
EOF

  cat >"$meta_data" <<EOF
instance-id: ${hostname}-${vmid}
local-hostname: ${hostname}
EOF

  cat >"$network_config" <<EOF
version: 2
ethernets:
  lan0:
    match:
      name: "en*"
    dhcp4: false
    addresses:
      - ${ip}/${prefix}
    routes:
      - to: default
        via: ${gateway}
    nameservers:
      addresses: [$(printf '%s,' "${dns[@]}" | sed 's/,$//')]
EOF
  genisoimage -output "$cidata_iso" -volid cidata -joliet -rock "$USER_DATA" "$meta_data" "$network_config"
  log "cloud-init seed created: ${cidata_iso}"
}

create_vm() {
  local vmid name storage bridge memory cores sockets cpu machine bios ostype vga disk_size img overwrite cidata_iso
  vmid="$(yaml_get vm.id)"
  name="$(yaml_get vm.name)"
  storage="$(yaml_get vm.storage local-lvm)"
  bridge="$(yaml_get vm.bridge vmbr0)"
  memory="$(yaml_get vm.memory_mib 65536)"
  cores="$(yaml_get vm.cores 16)"
  sockets="$(yaml_get vm.sockets 1)"
  cpu="$(yaml_get vm.cpu host)"
  machine="$(yaml_get vm.machine q35)"
  bios="$(yaml_get vm.bios ovmf)"
  ostype="$(yaml_get vm.ostype l26)"
  vga="$(yaml_get vm.vga std)"
  disk_size="$(yaml_get vm.disk_size 256G)"
  img="$(yaml_get vm.cloud_image_path)"
  overwrite="$(yaml_get vm.overwrite_existing false)"
  cidata_iso="$(yaml_get vm.iso_storage_dir /var/lib/vz/template/iso)/$(yaml_get vm.hostname)-cidata.iso"

  if qm status "$vmid" >/dev/null 2>&1; then
    if [[ "$overwrite" != "true" ]]; then
      log "VM ${vmid} already exists; overwrite_existing=false"
      return
    fi
    qm stop "$vmid" --skiplock 1 || true
    qm destroy "$vmid" --purge 1 --destroy-unreferenced-disks 1
  fi

  qm create "$vmid" \
    --name "$name" \
    --memory "$memory" \
    --cores "$cores" \
    --sockets "$sockets" \
    --cpu "$cpu" \
    --machine "$machine" \
    --bios "$bios" \
    --ostype "$ostype" \
    --agent enabled=1 \
    --scsihw virtio-scsi-single \
    --net0 "virtio,bridge=${bridge}"

  qm importdisk "$vmid" "$img" "$storage"
  qm set "$vmid" --vga "$vga"
  qm set "$vmid" --scsi0 "${storage}:vm-${vmid}-disk-0,discard=on,ssd=1,iothread=1"
  qm set "$vmid" --efidisk0 "${storage}:0,efitype=4m,pre-enrolled-keys=0"
  qm set "$vmid" --ide2 "local:iso/$(basename "$cidata_iso"),media=cdrom"
  qm set "$vmid" --boot "order=scsi0;ide2;net0"
  qm set "$vmid" --serial0 socket
  qm resize "$vmid" scsi0 "$disk_size"
  if [[ "$(yaml_get vm.start_on_boot false)" == "true" ]]; then
    qm set "$vmid" --onboot 1
  fi
  log "VM ${vmid}/${name} created"
}

configure_gpu_passthrough() {
  if [[ "$(yaml_get gpu_passthrough.enabled false)" != "true" ]]; then
    log "gpu_passthrough.enabled=false; skip host VFIO changes"
    return
  fi
  local ids vmid idx changed arg current grub_line spec
  vmid="$(yaml_get vm.id)"
  ids="$(yaml_get gpu_passthrough.vfio_ids "" | paste -sd, -)"
  changed=0
  if [[ -f /etc/kernel/cmdline ]]; then
    current="$(cat /etc/kernel/cmdline)"
    while read -r arg; do
      [[ -z "$arg" ]] && continue
      if ! grep -qw -- "$arg" <<<"$current"; then
        sed -i "s/$/ ${arg}/" /etc/kernel/cmdline
        current="${current} ${arg}"
        changed=1
      fi
    done < <(yaml_get gpu_passthrough.kernel_args "intel_iommu=on"$'\n'"iommu=pt")
  elif [[ -f /etc/default/grub ]]; then
    grub_line="$(grep -E '^GRUB_CMDLINE_LINUX_DEFAULT=' /etc/default/grub || true)"
    current="${grub_line#*=}"
    current="${current%\"}"
    current="${current#\"}"
    while read -r arg; do
      [[ -z "$arg" ]] && continue
      if ! grep -qw -- "$arg" <<<"$current"; then
        current="${current} ${arg}"
        changed=1
      fi
    done < <(yaml_get gpu_passthrough.kernel_args "intel_iommu=on"$'\n'"iommu=pt")
    if [[ $changed -eq 1 ]]; then
      if grep -qE '^GRUB_CMDLINE_LINUX_DEFAULT=' /etc/default/grub; then
        sed -i "s|^GRUB_CMDLINE_LINUX_DEFAULT=.*|GRUB_CMDLINE_LINUX_DEFAULT=\"${current# }\"|" /etc/default/grub
      else
        echo "GRUB_CMDLINE_LINUX_DEFAULT=\"${current# }\"" >>/etc/default/grub
      fi
      update-grub
    fi
  else
    echo "Neither /etc/kernel/cmdline nor /etc/default/grub exists; cannot set IOMMU kernel args" >&2
    exit 1
  fi
  cat >/etc/modules-load.d/vfio.conf <<'EOF'
vfio
vfio_pci
vfio_iommu_type1
EOF
  cat >/etc/modprobe.d/blacklist-nvidia-passthrough.conf <<'EOF'
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
EOF
  if [[ -n "$ids" ]]; then
    {
      echo "options vfio-pci ids=${ids}"
      echo "softdep snd_hda_intel pre: vfio-pci"
    } >/etc/modprobe.d/vfio.conf
  fi
  proxmox-boot-tool refresh || true
  update-initramfs -u -k all
  qm stop "$vmid" --skiplock 1 >/dev/null 2>&1 || true
  qm set "$vmid" --vga "$(yaml_get vm.vga std)"
  idx=0
  while read -r pci; do
    [[ -z "$pci" ]] && continue
    if [[ "$pci" == *,* ]]; then
      spec="$pci"
      [[ "$spec" != *pcie=* ]] && spec="${spec},pcie=1"
    else
      spec="${pci},pcie=1"
    fi
    qm set "$vmid" "--hostpci${idx}" "$spec"
    idx=$((idx + 1))
  done < <(yaml_get gpu_passthrough.pci_addresses "")
  if qm config "$vmid" | grep -q '^efidisk0: .*pre-enrolled-keys=1'; then
    log "VM ${vmid} still has OVMF secure boot keys enrolled; recreate efidisk0 manually if NVIDIA module signing blocks driver loading"
  fi
  if [[ $changed -eq 1 || "$(lspci -Dnnk | awk '/NVIDIA/{n=1} n&&/Kernel driver in use/{print; n=0}' | grep -c vfio-pci || true)" -eq 0 ]]; then
    touch /run/pve_ubuntu_reboot_required
    log "GPU passthrough host configuration changed; reboot required"
  fi
  log "GPU passthrough configured; reboot qve before starting GPU workload"
}

configure_hdd_storage() {
  [[ "$(yaml_get hdd.enabled false)" == "true" ]] || return 0
  local disk part fs label mountpoint storage uuid existing_fs existing_label
  disk="$(yaml_get hdd.disk_by_id)"
  part="$(yaml_get hdd.partition)"
  fs="$(yaml_get hdd.filesystem ext4)"
  label="$(yaml_get hdd.label hdd8t)"
  mountpoint="$(yaml_get hdd.mountpoint /mnt/hdd8t)"
  storage="$(yaml_get hdd.storage_name hdd8t)"

  if mountpoint -q "$mountpoint" && pvesm status | awk '{print $1}' | grep -qx "$storage"; then
    log "HDD storage ${storage} already mounted at ${mountpoint}"
    return
  fi

  if [[ ! -b "$disk" ]]; then
    echo "HDD disk not found: $disk" >&2
    exit 1
  fi
  if [[ "$(yaml_get hdd.wipe_existing false)" != "true" ]] && [[ ! -b "$part" ]]; then
    echo "HDD partition missing and hdd.wipe_existing=false: $part" >&2
    exit 1
  fi

  existing_fs="$(blkid -s TYPE -o value "$part" 2>/dev/null || true)"
  existing_label="$(blkid -s LABEL -o value "$part" 2>/dev/null || true)"
  if [[ "$existing_fs" == "$fs" && "$existing_label" == "$label" ]]; then
    log "HDD partition ${part} already formatted as ${fs} with label ${label}; skip format"
  elif [[ "$(yaml_get hdd.wipe_existing false)" == "true" ]]; then
    log "Formatting HDD ${disk} as ${fs}; existing data will be destroyed"
    umount "$part" >/dev/null 2>&1 || true
    wipefs -a "$disk"
    parted -s "$disk" mklabel gpt
    parted -s "$disk" mkpart primary "$fs" 0% 100%
    partprobe "$disk" || true
    udevadm settle
    mkfs -t "$fs" -F -L "$label" "$part"
  fi

  mkdir -p "$mountpoint"
  uuid="$(blkid -s UUID -o value "$part")"
  grep -q " ${mountpoint} " /etc/fstab || echo "UUID=${uuid} ${mountpoint} ${fs} defaults,nofail 0 2" >>/etc/fstab
  mountpoint -q "$mountpoint" || mount "$mountpoint"
  if ! pvesm status | awk '{print $1}' | grep -qx "$storage"; then
    pvesm add dir "$storage" --path "$mountpoint" --content "$(yaml_get hdd.storage_content images,backup,iso)" --is_mountpoint 1
  fi
  log "HDD storage ${storage} ready at ${mountpoint}"
}

attach_hdd_to_vm() {
  [[ "$(yaml_get hdd.vm_disk.enabled false)" == "true" ]] || return 0
  local vmid storage bus size opts
  vmid="$(yaml_get vm.id)"
  storage="$(yaml_get hdd.storage_name hdd8t)"
  bus="$(yaml_get hdd.vm_disk.bus scsi1)"
  size="$(yaml_get hdd.vm_disk.size 7000)"
  if qm config "$vmid" | grep -q "^${bus}:"; then
    log "VM ${vmid} already has ${bus}; skip HDD attach"
    return
  fi
  opts="${storage}:${size},format=$(yaml_get hdd.vm_disk.format raw)"
  [[ "$(yaml_get hdd.vm_disk.iothread true)" == "true" ]] && opts="${opts},iothread=1"
  [[ "$(yaml_get hdd.vm_disk.discard false)" == "true" ]] && opts="${opts},discard=on"
  [[ "$(yaml_get hdd.vm_disk.ssd false)" == "true" ]] && opts="${opts},ssd=1"
  qm set "$vmid" "--${bus}" "$opts"
  log "Attached HDD-backed disk to VM ${vmid}: ${bus}=${opts}"
}

start_vm() {
  local vmid
  vmid="$(yaml_get vm.id)"
  if [[ -f /run/pve_ubuntu_reboot_required ]]; then
    log "Host reboot is required before starting VM ${vmid}; skip start"
    return
  fi
  if qm status "$vmid" | grep -q 'status: running'; then
    log "VM ${vmid} already running"
    return
  fi
  qm start "$vmid"
  log "VM ${vmid} started"
}

main() {
  need_root
  ensure_yaml
  log "pve_ubuntu started with config=${CONFIG}"
  if run_stage install_host_packages; then install_host_packages; fi
  if run_stage copy_iso; then copy_iso_from_source; fi
  if run_stage copy_v2ray_config; then copy_v2ray_config_from_source; fi
  download_cloud_image
  create_cloud_init_seed
  if run_stage create_vm; then create_vm; fi
  if run_stage hdd_storage; then configure_hdd_storage; fi
  if run_stage attach_hdd; then attach_hdd_to_vm; fi
  if run_stage gpu_passthrough; then configure_gpu_passthrough; fi
  if run_stage start_vm; then start_vm; fi
  log "pve_ubuntu finished"
}

main "$@"

pve_ubuntu.yaml

yaml

global:
  mode: auto
  workdir: /root/pve-ubuntu

source:
  host: pve
  ip: <SOURCE_PVE_LAN_IP>
  user: root
  password: <SOURCE_PVE_ROOT_PASSWORD>
  iso_path: <SOURCE_UBUNTU_ISO_PATH>
  v2ray_config_dir: /usr/local/etc/v2ray

stages:
  install_host_packages:
    mode: auto
  copy_iso:
    mode: auto
  copy_v2ray_config:
    mode: auto
  create_vm:
    mode: auto
  hdd_storage:
    mode: auto
  attach_hdd:
    mode: auto
  gpu_passthrough:
    mode: auto
  start_vm:
    mode: auto

vm:
  id: <VM_ID>
  name: bj123
  hostname: bj123
  bridge: vmbr0
  storage: local-lvm
  iso_storage_dir: /var/lib/vz/template/iso
  snippets_storage_dir: /var/lib/vz/snippets
  cloud_image_url: https://mirrors.tuna.tsinghua.edu.cn/ubuntu-cloud-images/jammy/current/jammy-server-cloudimg-amd64.img
  cloud_image_path: /var/lib/vz/template/cache/jammy-server-cloudimg-amd64.img
  disk_size: 256G
  memory_mib: 65536
  cores: 16
  sockets: 1
  cpu: host
  machine: q35
  bios: ovmf
  vga: none
  ostype: l26
  agent: true
  start_on_boot: false
  overwrite_existing: false

network:
  ipv4: <VM_LAN_IP>
  prefix: 24
  gateway4: <LAN_GATEWAY_IP>
  dns:
    - <LAN_DNS_IP>
    - <PUBLIC_DNS_IP>

ubuntu:
  user: <UBUNTU_USER>
  password: <UBUNTU_USER_PASSWORD>
  full_name: <UBUNTU_FULL_NAME>
  timezone: Asia/Shanghai
  locale: en_US.UTF-8
  ssh_password_auth: true

guest_setup:
  config: setup_ubuntu.yaml
  script: setup_ubuntu.sh
  run_on_first_boot: true

gpu_passthrough:
  enabled: true
  auto_detect_nvidia: true
  pci_addresses:
    - "<GPU_PCI_ADDRESS_1>"
    - "<GPU_PCI_ADDRESS_2>"
    - "<BOOT_VGA_GPU_PCI_ADDRESS>,rombar=0"
  excluded_pci_addresses:
    - "<GPU_PCI_ADDRESS_TO_EXCLUDE> # optional; keep problematic GPUs out of the VM"
  vfio_ids:
    - "<GPU_VENDOR_DEVICE_ID>"
    - "<GPU_AUDIO_VENDOR_DEVICE_ID>"
  kernel_args:
    - intel_iommu=on
    - iommu=pt
    - pcie_port_pm=off
    - pcie_aspm=off
    - vfio-pci.disable_idle_d3=1
  reboot_after_config: true

hdd:
  enabled: true
  disk_by_id: /dev/disk/by-id/<HDD_DISK_BY_ID>
  partition: /dev/disk/by-id/<HDD_DISK_BY_ID>-part1
  filesystem: ext4
  label: <PVE_HDD_LABEL>
  mountpoint: <PVE_HDD_MOUNTPOINT>
  storage_name: <PVE_HDD_STORAGE_NAME>
  storage_content: images,backup,iso
  wipe_existing: true
  vm_disk:
    enabled: true
    bus: scsi1
    size: <VM_DATA_DISK_SIZE_GB>
    format: raw
    discard: false
    ssd: false
    iothread: true

setup_ubuntu.sh

#!/usr/bin/env bash
set -euo pipefail

CONFIG="${1:-/opt/bj123-setup/setup_ubuntu.yaml}"
LOG_FILE="/var/log/setup_ubuntu.log"

exec > >(tee -a "$LOG_FILE") 2>&1

log() {
  printf '[%s] %s\n' "$(date '+%F %T')" "$*"
}

ensure_yaml() {
  if python3 - <<'PY' >/dev/null 2>&1
import yaml
PY
  then
    return
  fi
  apt-get update
  DEBIAN_FRONTEND=noninteractive apt-get install -y python3-yaml
}

yaml_get() {
  local path="$1"
  local default="${2:-}"
  python3 - "$CONFIG" "$path" "$default" <<'PY'
import sys, yaml
cfg_path, key_path, default = sys.argv[1:4]
with open(cfg_path, "r", encoding="utf-8") as f:
    data = yaml.safe_load(f) or {}
cur = data
for part in key_path.split("."):
    if isinstance(cur, dict) and part in cur:
        cur = cur[part]
    else:
        print(default)
        sys.exit(0)
if cur is None:
    print(default)
elif isinstance(cur, bool):
    print("true" if cur else "false")
elif isinstance(cur, list):
    print("\n".join(str(x) for x in cur))
else:
    print(cur)
PY
}

stage_mode() {
  local stage="$1"
  local mode
  mode="$(yaml_get "stages.${stage}.mode" "")"
  if [[ -z "$mode" ]]; then
    mode="$(yaml_get "global.mode" "confirm")"
  fi
  printf '%s' "$mode"
}

run_stage() {
  local stage="$1"
  local mode
  mode="$(stage_mode "$stage")"
  case "$mode" in
    auto) return 0 ;;
    manual|skip) log "skip stage ${stage} (mode=${mode})"; return 1 ;;
    confirm)
      read -r -p "Run stage ${stage}? [y/N] " answer
      [[ "${answer,,}" == y* ]]
      ;;
    *) log "skip stage ${stage} (unknown mode=${mode})"; return 1 ;;
  esac
}

install_packages() {
  mapfile -t pkgs < <(yaml_get packages.base "")
  if [[ "${#pkgs[@]}" -gt 0 ]]; then
    DEBIAN_FRONTEND=noninteractive apt-get install -y "${pkgs[@]}"
  fi
}

configure_apt_sources() {
  local mirror
  mirror="$(yaml_get system.apt_mirror "")"
  [[ -z "$mirror" ]] && return
  if [[ -f /etc/apt/sources.list ]]; then
    sed -i "s@http://.*archive.ubuntu.com@${mirror}@g; s@https://.*archive.ubuntu.com@${mirror}@g; s@http://security.ubuntu.com@${mirror}@g; s@https://security.ubuntu.com@${mirror}@g" /etc/apt/sources.list
    sed -i 's@http://@https://@g' /etc/apt/sources.list
  fi
}

configure_git() {
  local user email http_proxy https_proxy target_user home_dir
  target_user="$(yaml_get user.name ubuntu)"
  home_dir="$(getent passwd "$target_user" | cut -d: -f6)"
  user="$(yaml_get git.user_name "")"
  email="$(yaml_get git.user_email "")"
  http_proxy="$(yaml_get git.http_proxy "")"
  https_proxy="$(yaml_get git.https_proxy "")"
  [[ -z "$home_dir" ]] && return
  sudo -u "$target_user" git config --global user.name "$user"
  sudo -u "$target_user" git config --global user.email "$email"
  [[ -n "$http_proxy" ]] && sudo -u "$target_user" git config --global http.proxy "$http_proxy"
  [[ -n "$https_proxy" ]] && sudo -u "$target_user" git config --global https.proxy "$https_proxy"
  git lfs install --system || true
}

configure_zsh() {
  local target_user shell_path home_dir
  target_user="$(yaml_get user.name ubuntu)"
  home_dir="$(getent passwd "$target_user" | cut -d: -f6)"
  shell_path="$(yaml_get user.shell /usr/bin/zsh)"
  if [[ -x "$shell_path" && -n "$home_dir" ]] && id "$target_user" >/dev/null 2>&1; then
    chsh -s "$shell_path" "$target_user" || true
    sudo -u "$target_user" mkdir -p "$home_dir/.zsh"
    if [[ ! -d "$home_dir/.zsh/zsh-autocomplete/.git" ]]; then
      sudo -u "$target_user" git clone --depth 1 https://github.com/marlonrichert/zsh-autocomplete.git "$home_dir/.zsh/zsh-autocomplete" || true
    fi
    if [[ ! -f "$home_dir/.zsh/zsh-autosuggestions.zsh" ]]; then
      if [[ -d "$home_dir/.zsh/zsh-autosuggestions/.git" ]]; then
        cp "$home_dir/.zsh/zsh-autosuggestions/zsh-autosuggestions.zsh" "$home_dir/.zsh/zsh-autosuggestions.zsh" || true
      else
        timeout 60 wget -q https://raw.staticdn.net/zsh-users/zsh-autosuggestions/master/zsh-autosuggestions.zsh -O "$home_dir/.zsh/zsh-autosuggestions.zsh" || true
      fi
    fi
    cat >"$home_dir/.zshrc" <<'EOF'
autoload -Uz promptinit
promptinit
PROMPT='%F{yellow}%~ # %f'

setopt histignorealldups sharehistory
bindkey -e
HISTSIZE=1000
SAVEHIST=1000
HISTFILE=~/.zsh_history

zstyle ':completion:*' auto-description 'specify: %d'
zstyle ':completion:*' completer _expand _complete _correct _approximate
zstyle ':completion:*' format 'Completing %d'
zstyle ':completion:*' group-name ''
zstyle ':completion:*' menu select=2
eval "$(dircolors -b)"
zstyle ':completion:*:default' list-colors ${(s.:.)LS_COLORS}
zstyle ':completion:*' list-colors ''
zstyle ':completion:*' list-prompt %SAt %p: Hit TAB for more, or the character to insert%s
zstyle ':completion:*' matcher-list '' 'm:{a-z}={A-Z}' 'm:{a-zA-Z}={A-Za-z}' 'r:|[._-]=* r:|=* l:|=*'
zstyle ':completion:*' menu select=long
zstyle ':completion:*' select-prompt %SScrolling active: current selection at %p%s
zstyle ':completion:*' use-compctl false
zstyle ':completion:*' verbose true
zstyle ':completion:*:*:kill:*:processes' list-colors '=(#b) #([0-9]#)*=0=01;31'
zstyle ':completion:*:kill:*' command 'ps -u $USER -o pid,%cpu,tty,cputime,cmd'

alias ls="ls --color"
alias gs="git status"
alias gb="git rev-parse --abbrev-ref HEAD"
alias gba="git -P branch"
alias gdp="git -P diff"
alias gdh="git diff HEAD^ HEAD"
alias gl="git log"
alias gn="git --no-pager log --pretty='format:%Cgreen[%h] %Cblue[%ai] %Creset[%an]%C(Red)%d %n  %Creset%s %n' -n5"
alias ga="git add"
alias gas="git add . && git status"
alias gc="git commit"
alias gk="git checkout"
alias gau="git add -u"
alias gcm="git commit -m"
alias gcan="git commit --amend --no-edit"
alias gp="git push"
alias gpf="git push -f"
alias gacp="git add -u && git commit --amend --no-edit && git push -f"
[[ -f ~/.gd.sh ]] && source ~/.gd.sh

alias ta="tmux a"
alias td="tmux detach"
alias tn="tmux new -s x"
alias tl="tmux ls"
alias ts="tmux select-pane -T"
alias tm="top -o %MEM -d 2 -c"
alias tc="top -o %CPU -d 2 -c"
alias k9="kill -9"
alias lt="ls -lt"
alias hi="hostname -i"

bindkey "^[[1;5C" forward-word
bindkey "^[[1;3C" forward-word
bindkey "^[[1;5D" backward-word
bindkey "^[[1;3D" backward-word
bindkey "^[[1~"   beginning-of-line
bindkey "^[[4~"   end-of-line
bindkey "^[[3~"   delete-char
bindkey "^[^[[3~" delete-word

if [[ -f ~/.zsh/zsh-autosuggestions.zsh ]]; then
  ZSH_AUTOSUGGEST_HIGHLIGHT_STYLE="fg=#ff00ff"
  source ~/.zsh/zsh-autosuggestions.zsh
fi
if [[ -f ~/.zsh/zsh-autocomplete/zsh-autocomplete.plugin.zsh ]]; then
  source ~/.zsh/zsh-autocomplete/zsh-autocomplete.plugin.zsh 2>/dev/null
  zstyle ':completion:*' list-colors '=*=96'
fi

if [[ -f "$HOME/miniconda3/etc/profile.d/conda.sh" ]]; then
  . "$HOME/miniconda3/etc/profile.d/conda.sh"
elif [[ -x "$HOME/miniconda3/bin/conda" ]]; then
  export PATH="$HOME/miniconda3/bin:$PATH"
fi
alias cda="conda activate ai"
alias cdd="conda deactivate"
if command -v conda >/dev/null 2>&1 && conda env list | awk '{print $1}' | grep -qx ai; then
  conda activate ai
fi

alias nu="gpustat -cpu -i -F -P"
alias nsd="nvidia-smi | grep Default"
export HF_ENDPOINT=https://hf-mirror.com
export REPOS=$HOME/repos
export DATA=/media/data1
export PATH=/usr/local/cuda/bin:$HOME/.local/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}

ulimit -n 1048576 2>/dev/null || true
fpath+=~/.zfunc
autoload -Uz compinit
compinit
EOF
    cat >"$home_dir/.zshenv" <<'EOF'
skip_global_compinit=1
EOF
    chown "$target_user:$target_user" "$home_dir/.zshrc" "$home_dir/.zshenv"
  fi
}

configure_tmux() {
  local target_user home_dir
  target_user="$(yaml_get user.name ubuntu)"
  home_dir="$(getent passwd "$target_user" | cut -d: -f6)"
  [[ -z "$home_dir" ]] && return
  sudo -u "$target_user" mkdir -p "$home_dir/.tmux/plugins" "$home_dir/.config/systemd/user"
  if [[ ! -d "$home_dir/.tmux/plugins/tpm/.git" ]]; then
    sudo -u "$target_user" git clone --depth 1 https://github.com/tmux-plugins/tpm "$home_dir/.tmux/plugins/tpm" || true
  fi
  if [[ ! -d "$home_dir/.tmux/plugins/tmux-resurrect/.git" ]]; then
    sudo -u "$target_user" git clone --depth 1 https://github.com/tmux-plugins/tmux-resurrect "$home_dir/.tmux/plugins/tmux-resurrect" || true
  fi
  cat >"$home_dir/.tmux.conf" <<'EOF'
unbind C-b
set -g prefix M-z
bind M-z send-prefix
bind r source-file ~/.tmux.conf \; display ".tmux.conf reloaded!"
set -g mouse on
set -g status-interval 1
set-option -g status-position bottom
set-option -g status-style bg=default
set-option -g status-left ""
set-option -g window-status-format ""
set-option -g window-status-separator ""
set -g window-status-current-format "#[fg=cyan] #{pane_title}: [#{pane_current_path}]"
set-option -g status-right "#[fg=cyan,bold] [ww%V.%w] %m-%d %H:%M:%S"
set -g pane-border-status top
set -g pane-border-lines heavy
set -g pane-border-style bg=default,fg=cyan
set -g pane-active-border-style bg=cyan,fg=black
setw -g pane-border-format ' #{pane_index}: [#{pane_current_path}] '
unbind -n a
unbind-key -T root MouseDrag1Pane
unbind-key -T copy-mode-vi MouseDrag1Pane
unbind-key -T copy-mode MouseDrag1Pane
set-option -g default-shell /usr/bin/zsh
set-option -g history-limit 100000
set -g @plugin 'tmux-plugins/tpm'
set -g @plugin 'tmux-plugins/tmux-resurrect'
set -g @resurrect-hook-pre-restore-pane-processes 'tmux kill-session -t=0 2>/dev/null || true'
set -g @resurrect-processes '\
    ssh mongosh \
    "~npx->npx *" \
    "~npm->npm *" \
    "~python->python *" \
    "~docker->docker *" \
    "~gpustat->gpustat *" \
'
run '~/.tmux/plugins/tpm/tpm'
EOF
  chown "$target_user:$target_user" "$home_dir/.tmux.conf"
}

install_dotfiles() {
  local target_user home_dir gd_src
  target_user="$(yaml_get user.name ubuntu)"
  home_dir="$(getent passwd "$target_user" | cut -d: -f6)"
  [[ -z "$home_dir" ]] && return
  gd_src="$(yaml_get dotfiles.gd_source /opt/bj123-setup/dotfiles/.gd.sh)"
  if [[ -f "$gd_src" ]]; then
    install -m 0644 -o "$target_user" -g "$target_user" "$gd_src" "$home_dir/.gd.sh"
  elif [[ ! -f "$home_dir/.gd.sh" ]]; then
    timeout 60 wget -q "$(yaml_get dotfiles.gd_url https://raw.staticdn.net/Hansimov/blog/main/docs/notes/scripts/.gd.sh)" -O "$home_dir/.gd.sh" || true
    chown "$target_user:$target_user" "$home_dir/.gd.sh" 2>/dev/null || true
  fi
  sudo -u "$target_user" mkdir -p "$home_dir/.pip"
  cat >"$home_dir/.pip/pip.conf" <<'EOF'
[global]
index-url = https://mirrors.ustc.edu.cn/pypi/simple

[install]
trusted-host = mirrors.ustc.edu.cn
EOF
cat >"$home_dir/.condarc" <<'EOF'
channels:
  - conda-forge
  - bioconda
  - nodefaults
custom_channels:
  conda-forge: https://mirrors.ustc.edu.cn/anaconda/cloud
  bioconda: https://mirrors.ustc.edu.cn/anaconda/cloud
show_channel_urls: true
EOF
  chown -R "$target_user:$target_user" "$home_dir/.pip" "$home_dir/.condarc"
  configure_zsh
  configure_tmux
}

install_conda() {
  [[ "$(yaml_get conda.install false)" == "true" ]] || return
  local target_user home_dir installer url python_version env_name
  target_user="$(yaml_get user.name ubuntu)"
  home_dir="$(getent passwd "$target_user" | cut -d: -f6)"
  [[ -z "$home_dir" ]] && return
  url="$(yaml_get conda.installer_url https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-latest-Linux-x86_64.sh)"
  installer="/tmp/miniconda.sh"
  if [[ ! -x "$home_dir/miniconda3/bin/conda" ]]; then
    wget -O "$installer" "$url"
    sudo -u "$target_user" bash "$installer" -b -u -p "$home_dir/miniconda3"
  fi
  cat >"$home_dir/.condarc" <<'EOF'
channels:
  - conda-forge
  - bioconda
  - nodefaults
custom_channels:
  conda-forge: https://mirrors.ustc.edu.cn/anaconda/cloud
  bioconda: https://mirrors.ustc.edu.cn/anaconda/cloud
show_channel_urls: true
EOF
  chown "$target_user:$target_user" "$home_dir/.condarc"
  sudo -u "$target_user" "$home_dir/miniconda3/bin/conda" config --set show_channel_urls true || true
  env_name="$(yaml_get conda.env_name ai)"
  python_version="$(yaml_get conda.python_version 3.13)"
  if [[ "$(yaml_get conda.create_env true)" == "true" ]]; then
    if ! sudo -u "$target_user" "$home_dir/miniconda3/bin/conda" env list | awk '{print $1}' | grep -qx "$env_name"; then
      sudo -u "$target_user" "$home_dir/miniconda3/bin/conda" create -y -n "$env_name" "python=${python_version}" --override-channels -c https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge || true
    fi
  fi
  configure_zsh
}

install_python_tools() {
  [[ "$(yaml_get python_tools.install true)" == "true" ]] || return
  local target_user home_dir pip_bin env_name
  target_user="$(yaml_get user.name ubuntu)"
  home_dir="$(getent passwd "$target_user" | cut -d: -f6)"
  [[ -z "$home_dir" ]] && return
  DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip python3-venv
  sudo -u "$target_user" python3 -m pip install --user -U pip pipreqs gpustat || true
  env_name="$(yaml_get conda.env_name ai)"
  if [[ -x "$home_dir/miniconda3/envs/${env_name}/bin/pip" ]]; then
    pip_bin="$home_dir/miniconda3/envs/${env_name}/bin/pip"
    sudo -u "$target_user" "$pip_bin" install -U pip pipreqs gpustat || true
  fi
}

install_docker() {
  [[ "$(yaml_get docker.install false)" == "true" ]] || return
  local target_user mirror http_proxy https_proxy no_proxy
  target_user="$(yaml_get user.name ubuntu)"
  mirror="$(yaml_get docker.repo_mirror https://mirrors.ustc.edu.cn/docker-ce)"
  DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg
  install -m 0755 -d /etc/apt/keyrings
  rm -f /etc/apt/keyrings/docker.gpg
  curl -fsSL "${mirror}/linux/ubuntu/gpg" | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  chmod a+r /etc/apt/keyrings/docker.gpg
  echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] ${mirror}/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" >/etc/apt/sources.list.d/docker.list
  apt-get update
  DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  usermod -aG docker "$target_user" || true
  gpasswd -a "$target_user" docker || true
  mkdir -p /etc/docker
  python3 - <<'PY'
import json, pathlib
path = pathlib.Path("/etc/docker/daemon.json")
data = {}
if path.exists():
    try:
        data = json.loads(path.read_text())
    except Exception:
        data = {}
data.setdefault("registry-mirrors", [
    "https://docker.1ms.run",
    "https://docker.1panel.live",
    "https://docker.m.daocloud.io",
])
path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
PY
  http_proxy="$(yaml_get docker.http_proxy "")"
  https_proxy="$(yaml_get docker.https_proxy "$http_proxy")"
  no_proxy="$(yaml_get docker.no_proxy localhost,127.0.0.1)"
  if [[ -n "$http_proxy" ]]; then
    mkdir -p /etc/systemd/system/docker.service.d
    cat >/etc/systemd/system/docker.service.d/proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}"
EOF
  fi
  systemctl daemon-reload
  systemctl enable --now docker
  systemctl restart docker
}

install_nvidia_container() {
  [[ "$(yaml_get nvidia_container.install false)" == "true" ]] || return
  local base_url
  command -v docker >/dev/null 2>&1 || install_docker
  if ! command -v docker >/dev/null 2>&1; then
    log "Docker is not installed; skip NVIDIA Container Toolkit"
    return
  fi
  if ! command -v nvidia-smi >/dev/null 2>&1 || ! nvidia-smi >/dev/null 2>&1; then
    log "NVIDIA driver is not ready; skip NVIDIA Container Toolkit"
    return
  fi
  if [[ "$(yaml_get nvidia_container.use_ustc_mirror true)" == "true" ]]; then
    base_url="https://mirrors.ustc.edu.cn/libnvidia-container"
  else
    base_url="https://nvidia.github.io/libnvidia-container"
  fi
  rm -f /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
  curl -fsSL "${base_url}/gpgkey" | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
  curl -fsSL "${base_url}/stable/deb/nvidia-container-toolkit.list" | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' >/etc/apt/sources.list.d/nvidia-container-toolkit.list
  if [[ "$(yaml_get nvidia_container.use_ustc_mirror true)" == "true" ]]; then
    sed -i 's#nvidia.github.io#mirrors.ustc.edu.cn#g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
  fi
  apt-get update
  DEBIAN_FRONTEND=noninteractive apt-get install -y nvidia-container-toolkit
  if command -v nvidia-ctk >/dev/null 2>&1; then
    nvidia-ctk runtime configure --runtime=docker
  fi
  systemctl daemon-reload
  systemctl restart docker
}

install_tailscale() {
  [[ "$(yaml_get tailscale.install false)" != "true" ]] && return
  if ! command -v tailscale >/dev/null 2>&1; then
    curl -fsSL https://tailscale.com/install.sh | sh
  fi
  systemctl enable --now tailscaled
  local auth_key
  auth_key="$(yaml_get tailscale.auth_key "")"
  if [[ "$(yaml_get tailscale.up false)" == "true" ]]; then
    if [[ -n "$auth_key" ]]; then
      tailscale up --auth-key "$auth_key"
    else
      tailscale up
    fi
  fi
}

install_v2ray() {
  [[ "$(yaml_get v2ray.install true)" != "true" ]] && return
  local script config_src config_dst
  script="$(yaml_get v2ray.install_script /opt/bj123-setup/v2ray-install-release.sh)"
  config_src="$(yaml_get v2ray.config_src /opt/bj123-setup/v2ray/config.json)"
  config_dst="$(yaml_get v2ray.config_dst /usr/local/etc/v2ray/config.json)"
  if [[ -x "$script" ]]; then
    "$script" || true
  fi
  if [[ "$(yaml_get v2ray.install_dat true)" == "true" ]]; then
    mkdir -p /usr/local/share/v2ray
    timeout 60 wget -q https://githubfast.com/v2fly/geoip/releases/latest/download/geoip.dat -O /usr/local/share/v2ray/geoip.dat || true
    timeout 60 wget -q https://githubfast.com/v2fly/domain-list-community/releases/latest/download/dlc.dat -O /usr/local/share/v2ray/geosite.dat || true
  fi
  if [[ -f "$config_src" ]]; then
    mkdir -p "$(dirname "$config_dst")"
    install -m 0644 "$config_src" "$config_dst"
  fi
  while IFS=$'\t' read -r name src dst service; do
    [[ -z "$name" ]] && continue
    if [[ -f "$src" ]]; then
      mkdir -p "$(dirname "$dst")"
      install -m 0644 "$src" "$dst"
      systemctl enable --now "$service" || true
    fi
  done < <(python3 - "$CONFIG" <<'PY'
import sys, yaml
with open(sys.argv[1], "r", encoding="utf-8") as f:
    data = yaml.safe_load(f) or {}
for item in (((data.get("v2ray") or {}).get("extra_configs")) or []):
    name = str(item.get("name", "") or "")
    if not name:
        continue
    src = str(item.get("src", f"/opt/bj123-setup/v2ray/{name}.json"))
    dst = str(item.get("dst", f"/usr/local/etc/v2ray/{name}.json"))
    service = str(item.get("service", f"v2ray@{name}"))
    print("\t".join([name, src, dst, service]))
PY
  )
  if [[ "$(yaml_get v2ray.enable_service true)" == "true" ]]; then
    systemctl enable --now v2ray || true
  fi
}

mount_hdd() {
  [[ "$(yaml_get hdd.enabled false)" == "true" ]] || return
  local dev part fs label mountpoint uuid existing_fs existing_label
  dev="$(yaml_get hdd.device /dev/sdb)"
  part="$(yaml_get hdd.partition /dev/sdb1)"
  fs="$(yaml_get hdd.filesystem ext4)"
  label="$(yaml_get hdd.label data1)"
  mountpoint="$(yaml_get hdd.mountpoint /media/data1)"
  if mountpoint -q "$mountpoint"; then
    log "HDD already mounted at ${mountpoint}"
    return
  fi
  if [[ ! -b "$dev" ]]; then
    log "HDD device ${dev} is not present; skip guest HDD mount"
    return
  fi
  if [[ "$(findmnt -no SOURCE / 2>/dev/null)" == "$dev"* ]]; then
    log "Refusing to format root disk ${dev}"
    return 1
  fi
  existing_fs="$(blkid -s TYPE -o value "$part" 2>/dev/null || true)"
  existing_label="$(blkid -s LABEL -o value "$part" 2>/dev/null || true)"
  if [[ "$existing_fs" == "$fs" && "$existing_label" == "$label" ]]; then
    log "HDD partition ${part} already formatted as ${fs} with label ${label}; skip format"
  elif [[ "$(yaml_get hdd.wipe_existing false)" == "true" || ! -b "$part" ]]; then
    umount "$part" >/dev/null 2>&1 || true
    wipefs -a "$dev"
    parted -s "$dev" mklabel gpt
    parted -s "$dev" mkpart primary "$fs" 0% 100%
    partprobe "$dev" || true
    udevadm settle
    mkfs -t "$fs" -F -L "$label" "$part"
  fi
  mkdir -p "$mountpoint"
  uuid="$(blkid -s UUID -o value "$part")"
  grep -q " ${mountpoint} " /etc/fstab || echo "UUID=${uuid} ${mountpoint} ${fs} defaults,nofail 0 2" >>/etc/fstab
  mountpoint -q "$mountpoint" || mount "$mountpoint"
  log "HDD mounted at ${mountpoint}"
}

install_nvidia_driver() {
  [[ "$(yaml_get nvidia.install_driver false)" == "true" ]] || return
  if ! lspci -nn | grep -Eq 'NVIDIA.*(VGA|3D|Display)|VGA.*NVIDIA|3D.*NVIDIA|Display.*NVIDIA'; then
    log "No NVIDIA GPU visible in guest; skip NVIDIA driver"
    return
  fi
  if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi >/dev/null 2>&1; then
    log "NVIDIA driver already works"
    return
  fi
  DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-drivers-common
  local pkg
  pkg="$(yaml_get nvidia.driver_package auto)"
  if [[ "$pkg" == "auto" || -z "$pkg" ]]; then
    pkg="$(ubuntu-drivers devices 2>/dev/null | sed -n 's/.*driver *: *\\([^ ]*\\).*recommended.*/\\1/p' | head -1)"
  fi
  [[ -z "$pkg" ]] && pkg="nvidia-driver-535"
  log "Installing NVIDIA driver package: ${pkg}"
  DEBIAN_FRONTEND=noninteractive apt-get install -y "$pkg"
}

install_cuda() {
  [[ "$(yaml_get nvidia.install_cuda false)" == "true" ]] || return
  if command -v nvcc >/dev/null 2>&1; then
    log "CUDA nvcc already installed: $(command -v nvcc)"
    return
  fi
  if ! lspci -nn | grep -Eq 'NVIDIA.*(VGA|3D|Display)|VGA.*NVIDIA|3D.*NVIDIA|Display.*NVIDIA'; then
    log "No NVIDIA GPU visible in guest; skip CUDA"
    return
  fi
  local method package keyring_url tmpdeb
  method="$(yaml_get nvidia.cuda_method nvidia_repo)"
  package="$(yaml_get nvidia.cuda_package cuda-toolkit-13-0)"
  if [[ "$method" == "apt" ]]; then
    DEBIAN_FRONTEND=noninteractive apt-get install -y nvidia-cuda-toolkit
  else
    keyring_url="$(yaml_get nvidia.cuda_keyring_url)"
    tmpdeb="/tmp/cuda-keyring.deb"
    if [[ ! -f /etc/apt/sources.list.d/cuda-ubuntu2204-x86_64.list ]]; then
      wget -O "$tmpdeb" "$keyring_url"
      dpkg -i "$tmpdeb"
      apt-get update
    fi
    DEBIAN_FRONTEND=noninteractive apt-get install -y "$package"
  fi
  cat >/etc/profile.d/cuda.sh <<'EOF'
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}
EOF
}

install_desktop() {
  [[ "$(yaml_get desktop.install false)" != "true" ]] && return
  local package
  package="$(yaml_get desktop.package ubuntu-desktop-minimal)"
  DEBIAN_FRONTEND=noninteractive apt-get install -y "$package"
}

main() {
  log "setup_ubuntu started with config=${CONFIG}"
  ensure_yaml
  if run_stage apt_sources; then
    configure_apt_sources
  fi
  apt-get update
  if run_stage base_packages; then
    install_packages
  fi
  if run_stage qemu_guest_agent; then
    systemctl enable --now qemu-guest-agent
  fi
  if run_stage ssh; then
    systemctl enable --now ssh
  fi
  if run_stage tailscale; then
    install_tailscale
  fi
  if run_stage v2ray; then
    install_v2ray
  fi
  if run_stage hdd_mount; then
    mount_hdd
  fi
  if run_stage nvidia_driver; then
    install_nvidia_driver
  fi
  if run_stage cuda; then
    install_cuda
  fi
  if run_stage git; then
    configure_git
  fi
  if run_stage dotfiles; then
    install_dotfiles
  fi
  if run_stage conda; then
    install_conda
  fi
  if run_stage python_tools; then
    install_python_tools
  fi
  if run_stage docker; then
    install_docker
  fi
  if run_stage nvidia_container; then
    install_nvidia_container
  fi
  if run_stage zsh; then
    configure_zsh
  fi
  if run_stage desktop; then
    install_desktop
  fi
  log "setup_ubuntu finished"
}

main "$@"

setup_ubuntu.yaml

yaml

global:
  mode: auto

stages:
  apt_sources:
    mode: auto
  base_packages:
    mode: auto
  qemu_guest_agent:
    mode: auto
  ssh:
    mode: auto
  tailscale:
    mode: auto
  v2ray:
    mode: auto
  hdd_mount:
    mode: auto
  nvidia_driver:
    mode: auto
  cuda:
    mode: auto
  git:
    mode: auto
  dotfiles:
    mode: auto
  conda:
    mode: auto
  python_tools:
    mode: auto
  docker:
    mode: auto
  nvidia_container:
    mode: auto
  zsh:
    mode: auto
  desktop:
    mode: manual

system:
  hostname: bj123
  timezone: Asia/Shanghai
  apt_mirror: https://mirrors.ustc.edu.cn

user:
  name: <UBUNTU_USER>
  password: <UBUNTU_USER_PASSWORD>
  shell: /usr/bin/zsh

packages:
  base:
    - ca-certificates
    - curl
    - wget
    - gnupg
    - lsb-release
    - software-properties-common
    - build-essential
    - net-tools
    - pciutils
    - htop
    - tmux
    - unzip
    - qemu-guest-agent
    - openssh-server
    - git
    - git-lfs
    - zsh
    - python3-pip
    - python3-venv
    - lm-sensors

dotfiles:
  gd_source: /opt/bj123-setup/dotfiles/.gd.sh
  gd_url: https://raw.staticdn.net/Hansimov/blog/main/docs/notes/scripts/.gd.sh

conda:
  install: true
  installer_url: https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-latest-Linux-x86_64.sh
  env_name: ai
  python_version: "3.13"
  create_env: true

python_tools:
  install: true

docker:
  install: true
  repo_mirror: https://mirrors.ustc.edu.cn/docker-ce
  http_proxy: http://127.0.0.1:11119
  https_proxy: http://127.0.0.1:11119
  no_proxy: localhost,127.0.0.1

nvidia_container:
  install: true
  use_ustc_mirror: true

tailscale:
  install: true
  up: false
  auth_key: ""

v2ray:
  install: true
  install_script: /opt/bj123-setup/v2ray-install-release.sh
  config_src: /opt/bj123-setup/v2ray/config.json
  config_dst: /usr/local/etc/v2ray/config.json
  extra_configs:
    - name: new
      src: /opt/bj123-setup/v2ray/new.json
      dst: /usr/local/etc/v2ray/new.json
      service: v2ray@new
  install_dat: true
  enable_service: true

hdd:
  enabled: true
  device: /dev/sdb
  partition: /dev/sdb1
  filesystem: ext4
  label: <VM_HDD_LABEL>
  mountpoint: <VM_HDD_MOUNTPOINT>
  wipe_existing: true

nvidia:
  install_driver: true
  driver_package: auto
  install_cuda: true
  cuda_method: nvidia_repo
  cuda_package: cuda-toolkit-13-0
  cuda_keyring_url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb

git:
  user_name: <GIT_USER_NAME>
  user_email: <GIT_USER_EMAIL>
  http_proxy: http://127.0.0.1:11119
  https_proxy: http://127.0.0.1:11119

desktop:
  install: false
  package: ubuntu-desktop-minimal

PVE 创建 Ubuntu 虚拟机 ​

一、准备 Ubuntu 22.04 ISO 镜像 ​

二、创建 Ubuntu 22.04 虚拟机 ​

1. General 选项卡 ​

2. OS 选项卡 ​

3. System 选项卡 ​

4. Disks 选项卡 ​

5. CPU 选项卡 ​

6. Memory 选项卡 ​

大内存 VM 的默认值经验 ​

7. Network 选项卡 ​

8. Confirm 选项卡 ​

三、在 VM 里安装 Ubuntu 22.04 ​

三a、软件环境配置 ​

换源 ​

开启SSH ​

安装 tailscale ​

安装 tmux ​

tmux-resurrect 恢复失败排查 ​

安装 zsh ​

安装 v2ray ​

安装 conda + Python ​

安装 git ​

安装 docker ​

四、在 Ubuntu 内安装 QEMU Guest Agent（建议） ​

五、启用显卡直通，并将分配给 VM ​

在 PVE 9 上启用 IOMMU ​

加载 VFIO 模块 ​

黑名单宿主机显卡驱动 ​

将显卡全部绑定到 vfio-pci ​

将显卡全部直通给 VM ​

命令行一键直通 ​

常见问题：0 <= irq_num && irq_num < PCI_NUM_PINS ​

验证 VM 中显卡是否已经直通 ​

安装 NVDIA 驱动和 NVCC+CUDA ​

五a、常见问题 ​

启动时间太长 ​

启动失败 ​

五b、智能启动脚本 ​

启动 VM 时自动诊断和排除故障显卡 ​

六、把 20TB HDD 配置成大容量数据存储并挂给 VM ​

在 PVE 中查看磁盘信息 ​

清理磁盘 ​

创建 GPT 分区和单一大分区 ​

格式化为 ext4 ​

挂载 ​

注册为 Directory 存储 ​

将这个存储加给 VM ​

优化磁盘占用 ​

在 VM 中添加磁盘 ​

在 VM 中创建 GPT 分区和单一大分区 ​

在 VM 中格式化文件系统 ​

在 VM 中挂载 ​

设置开机自动挂载 ​

七、使用自动化脚本和配置 ​

运行前写好的配置 ​

占位符说明 ​

阶段开关 ​

在 PVE 宿主机上运行 ​

在 Ubuntu VM 内运行或补跑 ​

Ubuntu 22.04 Kernel 版本 ​

运行后自动生成的内容 ​

验证命令 ​

样例脚本和配置 ​

PVE 创建 Ubuntu 虚拟机

一、准备 Ubuntu 22.04 ISO 镜像

二、创建 Ubuntu 22.04 虚拟机

1. General 选项卡

2. OS 选项卡

3. System 选项卡

4. Disks 选项卡

5. CPU 选项卡

6. Memory 选项卡

大内存 VM 的默认值经验

7. Network 选项卡

8. Confirm 选项卡

三、在 VM 里安装 Ubuntu 22.04

三a、软件环境配置

换源

开启SSH

安装 tailscale

安装 tmux

tmux-resurrect 恢复失败排查

安装 zsh

安装 v2ray

安装 conda + Python

安装 git

安装 docker

四、在 Ubuntu 内安装 QEMU Guest Agent（建议）

五、启用显卡直通，并将分配给 VM

在 PVE 9 上启用 IOMMU

加载 VFIO 模块

黑名单宿主机显卡驱动

将显卡全部绑定到 vfio-pci

将显卡全部直通给 VM

命令行一键直通

常见问题：`0 <= irq_num && irq_num < PCI_NUM_PINS`

验证 VM 中显卡是否已经直通

安装 NVDIA 驱动和 NVCC+CUDA

五a、常见问题

启动时间太长

启动失败

五b、智能启动脚本

启动 VM 时自动诊断和排除故障显卡

六、把 20TB HDD 配置成大容量数据存储并挂给 VM

在 PVE 中查看磁盘信息

清理磁盘

创建 GPT 分区和单一大分区

格式化为 ext4

挂载

注册为 Directory 存储

将这个存储加给 VM

优化磁盘占用

在 VM 中添加磁盘

在 VM 中创建 GPT 分区和单一大分区

在 VM 中格式化文件系统

在 VM 中挂载

设置开机自动挂载

七、使用自动化脚本和配置

运行前写好的配置

占位符说明

阶段开关

在 PVE 宿主机上运行

在 Ubuntu VM 内运行或补跑

Ubuntu 22.04 Kernel 版本

运行后自动生成的内容

验证命令

样例脚本和配置