What began as a routine Proxmox upgrade turned into a surprisingly useful infrastructure-recovery exercise.
My primary homelab server lab0 was still running Proxmox VE 8.4 on Debian 12 Bookworm. Since Proxmox VE 9 had been out for a while and moves the underlying distribution to Debian 13 Trixie, I decided it was time to make the jump.
This is the same host running much of the infrastructure from my homelab series, including my Wazuh VM.
So before changing anything, I started where every major upgrade should start:
pve8to9 --fullThe initial report was better than expected: 43 checks total, 29 passing, three warnings, and one failure.
The failure was the one that mattered most:
FAIL: systemd-boot meta-package installed.There were a few other cleanup items too:
AMD CPU microcode was missing.
Legacy settings remained in
/etc/sysctl.conf.LVM volumes were using the old autoactivation behavior.
Wazuh’s third-party repository appeared as an unusual APT suite.
One VM was still running.
None of that was catastrophic. But this is precisely why pre-upgrade checks exist: they turn hidden technical debt into an actionable checklist.
Full unedited upgrade process (with the help of ChatGPT).
Confirming the Boot Path
The first thing I wanted to understand was the bootloader warning.
Having the systemd-boot package installed does not necessarily mean the host actually boots using systemd-boot, so I checked the relevant state:
proxmox-boot-tool status
efibootmgr -v
dpkg -l | grep -E 'systemd-boot|grub|proxmox-boot'The key detail was this EFI entry:
Boot0004* proxmox ... \EFI\proxmox\shimx64.efiThe host also had the full GRUB stack installed. In other words, it was booting through Proxmox’s GRUB/shim path—not systemd-boot. The package was simply an unnecessary meta-package left behind on the system.
I removed it, reran the checker, and the failure disappeared.
That took the host from one failure to zero.
Cleaning Up Before the Upgrade
Next, I worked through the remaining warnings.
Updating LVM autoactivation
Proxmox 9 changes how newly created LVM and LVM-thin guest volumes use autoactivation. One of my existing VM disks still had the older behavior enabled.
Proxmox provides a migration script for this:
/usr/share/pve-manager/migrations/pve-lvm-disable-autoactivationIt found the affected volume and corrected it.
One important lesson: I had to run this script a second time later in the upgrade
because another LVM-backed VM state volume appeared after the new LVM configuration was installed.
A clean pre-upgrade check does not guarantee that nothing changes during the distribution upgrade itself. Run the checker again afterward.
Installing AMD microcode
lab0 uses an AMD Ryzen CPU, and the upgrade checker correctly identified that amd64-microcode was not installed.
The reason was straightforward: my Debian repositories only included:
main contribThe microcode package lives in:
non-free-firmwareAfter updating the Bookworm repository definitions, I installed the package:
apt install amd64-microcodeThat rebuilt the initramfs and queued the updated microcode for activation at the next reboot.
Another warning resolved.
Moving sysctl settings out of /etc/sysctl.conf
I also had these forwarding settings in the legacy /etc/sysctl.conf:
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1They are intentional. This host participates in networking within the lab—but Proxmox 9 wanted them moved into a dedicated file under /etc/sysctl.d/.
I created:
/etc/sysctl.d/99-homelab-forwarding.confWith:
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1Then applied the configuration:
sysctl --systemBoth forwarding values remained enabled.
After another upgrade-check run, I was down to:
WARNINGS: 1
FAILURES: 0The remaining warning was simply that a VM was still running.
At that point, the host was ready.
Moving to Proxmox 9
Before changing repositories, I shut down Wazuh and temporarily disabled the Wazuh and Tailscale APT sources.
For a major Debian upgrade, I prefer to keep third-party repositories out of the transaction until the base OS and platform packages are stable. It reduces the number of variables involved if something goes sideways.
I then migrated the Debian repositories from Bookworm to Trixie and moved to the newer deb822-style APT configuration.
The Debian suites became:
trixie
trixie-updates
trixie-securityAnd the Proxmox repository became:
trixie pve-no-subscriptionAfter updating package metadata:
apt updateI verified the intended upgrade path:
apt policy proxmox-veThe output showed exactly what I wanted:
Installed: 8.4.0
Candidate: 9.2.0Then came the real upgrade:
apt dist-upgradeAnd that is where things became interesting.
The APT Terminal Disappeared
During the upgrade, Debian prompted for configuration-file decisions—normal things such as:
/etc/issue
/etc/lvm/lvm.confAt some point, though, the shell running apt dist-upgrade disappeared due to a random glitch.
The upgrade process itself did not.
When I checked the Proxmox package state, I found a strange hybrid environment. Proxmox 9 packages were present:
pve-manager: 9.2.11
proxmox-kernel-7.0.14-12-pveBut many packages were listed as:
not correctly installedThat looks alarming, but it is exactly the moment when you should avoid making things worse.
My first check was:
dpkg --auditIt showed a long list of packages that had been unpacked but not yet configured.
I tried:
dpkg --configure -aBut APT still owned the package database.
So instead of deleting lock files or rebooting, I checked the process tree:
ps auxf | grep -E '[a]pt|[d]pkg'There it was:
apt dist-upgrade
└── dpkg --configure --pendingThe original APT transaction was still running.
I inspected its file descriptors:
ls -l /proc/<PID>/fd/{0,1,2}That revealed the actual problem: APT’s input and output were connected to a deleted pseudo-terminal.
/dev/pts/0 (deleted)The terminal was gone. The process was not.
The upgrade had effectively become stranded while holding the package manager state.
Recovering Safely
Once I knew what I was looking at, the recovery path was clear.
I terminated the orphaned APT process normally:
kill <PID>I did not delete the package manager lock files.
I did not remove files from /var/lib/dpkg.
I did not reboot halfway through the package configuration process.
After the orphaned process exited, I ran:
dpkg --configure -aThis time it completed successfully.
LVM configured. GRUB configured. The new Proxmox kernel was detected:
Found linux image: /boot/vmlinuz-7.0.14-12-pveThen the Proxmox packages finished configuring:
pve-manager configured
proxmox-ve configuredI followed with:
apt -f installWhich returned:
Upgrading: 0
Installing: 0
Removing: 0Then:
apt-get dist-upgradeReturned:
0 upgraded, 0 newly installed, 0 to removeThat was the confirmation I was looking for: the package database had returned to a healthy state.
pveversion -v Now showed:
proxmox-ve: 9.2.0
pve-manager: 9.2.11
proxmox-kernel-7.0.14-12-pve-signedAnd the “not correctly installed” messages were gone.
One Last Weird Service Failure
There was still one failed service:
chrony.serviceChrony is responsible for time synchronization on this host. Its error was:
Fatal error: Could not send notification to $NOTIFY_SOCKETAt that moment, the host had a newly upgraded userspace (including systemd), but was still running the pre-upgrade kernel and runtime environment.
Rather than overcorrecting a failure that could be transient, I validated the essentials:
dpkg --auditReturned clean.
The new Proxmox kernel was installed and set as the default:
7.0.14-12-pveAnd the Proxmox upgrade checker showed no failures.
So I rebooted.
The Moment of Truth
The server came back cleanly.
uname -rReturned:
7.0.14-12-pveAnd:
pveversion -vConfirmed:
proxmox-ve: 9.2.0
pve-manager: 9.2.11Most importantly:
systemctl --failedReturned:
0 loaded units listed.Chrony had recovered after the reboot. A few seconds later:
chronyc trackingShowed:
Leap status : NormalWith an active NTP source.
Storage was healthy too:
pvesm statusBoth local and local-lvm were active.
Finally, I started Wazuh again:
qm start 100
qm status 100And got:
status: runningThe final validation was the best one:
pve8to9 --fullWARNINGS: 0
FAILURES: 0Beautiful.
Repository Cleanup
The final step was restoring the repositories I had temporarily disabled.
Tailscale had a Debian 13 Trixie repository available, so I changed its suite from:
bookwormTo:
trixieThen I re-enabled it.
Wazuh uses its own stable repository suite, so I re-enabled that source as well.
One last APT issue appeared: an enterprise Proxmox repository was still enabled.
https://enterprise.proxmox.com/debian/pve trixieSince this lab uses the no-subscription repository, APT correctly returned:
401 UnauthorizedThe fix was simple: disable the enterprise source and run:
apt updateThis time:
All packages are up to date.The Actual Lesson
This upgrade turned out to be a more valuable homelab exercise than I anticipated, mainly due to the circumstances surrounding it, not because Proxmox 9 introduced a significant breaking change.
Bootloaders. Repository management. CPU microcode. LVM behavior. Sysctl configuration. GRUB. systemd. NTP. Package state. Process trees. Terminal file descriptors. Recovering an interrupted package transaction.
Installing a SIEM is cool. Deploying Kubernetes is cool. Spinning up twenty containers is cool.
But maintaining infrastructure when the platform beneath it changes is where you build a different kind of engineering muscle.
At one point, I had:
A Proxmox 9 userspace.
A Proxmox 8 kernel still running.
Dozens of unpacked-but-unconfigured packages.
An orphaned APT process holding the package database lock.
A deleted pseudo-terminal attached to that process.
The negative reaction would have been:
Delete the lock file and reboot.
The right response was:
Observe the actual state.
Identify what owns that state.
Fix the underlying failure.
Let the package manager finish the work it was already trying to do.
That is infrastructure engineering.
lab0 is now running:
Debian 13 Trixie
Proxmox VE 9.2
Kernel 7.0.14-12-pve
0 failed systemd services
0 pve8to9 warnings
0 pve8to9 failuresAnd Wazuh is back online.
Not a bad Friday afternoon in the homelab.
Until Next Time
That’s it for this one.
As always, keep learning, keep building, and keep thinking deeply about the systems we’re trusted to defend.
See you in the next one.
~ Day 💙




