This is the sequel to «I thought I'd been hacked». In that post I cleaned up the box, reclaimed 13 TB of storage — and, without knowing it, planted a time bomb in the boot config. Here it goes off.
May 22, 2026. My Hetzner dedicated box has been running my agentic AI stack — Coolify, Docker-in-Docker devcontainers, the lot — for seven days straight. At 21:43 it dies in flames: kernel page allocation failure of order:10 (a 4 MB contiguous chunk), three OOM-killer invocations against runaway Next.js dev servers (28 GB virtual ×2) and Node processes (55 GB virtual), and a reboot.
That part I expected to recover from in minutes. What actually happened next ate the rest of my night — and the root cause was a change I had made deliberately, with care, five days earlier.
Five days earlier: the partition decision
A week before the crash I'd noticed Hetzner's installimage had configured all three disks (one ~954 GB NVMe and two 7.3 TB HDDs) as a single RAID5 — limiting usable storage to ~1.9 TB and leaving roughly 13 TB of HDD capacity unaddressed. So I did the work to recover it: convert the HDDs from MBR to GPT in-place, build a new RAID1 array, mount as /mnt/bulk.
I did this carefully. A full block-level backup to a Synology NAS via Tailscale first. I checked the boot mode (Legacy BIOS, not UEFI). And I reasoned, correctly as far as it went, that the MBR→GPT conversion would destroy GRUB on sda and sdb (the GPT structure overlaps the MBR boot code), but that the NVMe (which I wasn't touching) would still boot, and BIOS would fall through to it after sda/sdb failed.
The conversion went smoothly. RAID stayed [UUU]. The box hummed along for five days.
May 22: it died
The OOM cascade above. The interesting bit is what came next.
Hardware reset from the Hetzner Robot panel. Wait three minutes. SSH:
$ ssh root@projects.si14agents.com
ssh: connect to host projects.si14agents.com port 22: Connection timed out
OK, sometimes RAID assembly is slow after an unclean shutdown. Wait five minutes. Same. Ten. Same.
Activate Hetzner's Rescue system (a small Debian, PXE-booted by their network). It comes up fine — confirming the box is alive, just not booting from disk. From rescue I mount the disks. RAID healthy. Filesystems clean. Kernels and initramfs present in /boot. grub.cfg looks fine.
The hypothesis I gravitated to: a couple of weeks earlier apt had installed a new kernel the box had never rebooted into, plus a Synology-modified initramfs. Easy story: new kernel + modified initramfs = boot failure. I hand-edited grub.cfg to point at the old kernel. Rebooted.
Nothing.
Hours of mass-masking
Convinced the boot was hanging on a stuck service, I went back into rescue and masked everything — docker, containerd, Synology, tailscale, fail2ban, the coolify runner. Stripped the boot down to barely-vanilla Ubuntu. Rebooted. Still nothing.
What finally exposed the actual problem was journalctl --list-boots from the mounted disk. The persistent journal showed exactly three boot IDs: the original good one from May 15, the previous good one ending at the OOM crash on May 22 21:43, and… nothing else. None of my reboot attempts had reached systemd-journald.
This isn't a stuck service. The boot isn't even reaching systemd. Whatever's failing happens earlier — at the kernel, or below.
KVM finally reveals it
I'd refused to escalate to Hetzner for KVM because the box was "alive enough" to PXE-rescue. After hours of dead ends I caved and ordered a manual power-cycle — which Hetzner couldn't make work either, so they attached a free KVM console for three hours.
Within minutes I was watching the actual screen during boot. After BIOS POST, a five-second countdown, "Boot from first hard disk", and then:
Type 'exit' to get the menu back
iPXE>
It fell straight back to the iPXE shell. Which meant: the BIOS was set up with Network as Boot Option #1. The NIC's PXE ROM handed off to Hetzner's iPXE. iPXE's "boot from first hard disk" tried to chainload from sda — found no MBR boot code (because I'd erased it five days earlier converting sda to GPT) — and dumped me to the iPXE shell.
iPXE doesn't fall through to the next disk on its own. That's the part I'd missed in my "BIOS will fall through to NVMe" reasoning. Native BIOS would have. Hetzner's iPXE chainloader tries the first disk and gives up.
The system had only "worked" for five days because nothing had triggered the boot path in that time. The OOM crash was the first reboot since the conversion — and the first time iPXE got to fail.
The actual fix
From rescue: mount the disk, chroot, reinstall GRUB on the disk that's still in good shape (grub-install --target=i386-pc /dev/nvme0n1 works; sda/sdb fail because they're now GPT with no BIOS Boot Partition — a fix-it-later TODO). But GRUB on the NVMe isn't enough on its own — the BIOS needs to actually try the NVMe before the network. So: KVM into the boot menu (F11), pick the NVMe manually, watch Ubuntu boot. Then into BIOS Setup, change the order to NVMe first and disable LAN PXE Boot.
Save, exit, reboot. Boots clean. About four hours from OOM crash to "everything green again."
The lesson
The bomb I planted sat dormant for five days until an OOM forced the first reboot — and picked the time for me. That's the deep one: a change's blast radius can outlive the window in which you make the change.
Show the verification, the mass-masking and the BIOS order
Five days earlier: I checked that all three disks were bootable
$ dd if=/dev/nvme0n1 bs=512 count=1 2>/dev/null | strings | head
GRUB
$ dd if=/dev/sda bs=512 count=1 2>/dev/null | strings | head
GRUB # ← here too
$ dd if=/dev/sdb bs=512 count=1 2>/dev/null | strings | head
GRUB # ← and here
All three had GRUB stage 1 in their MBR — standard Hetzner RAID install, GRUB on every member so the system survives a single disk failure. I knew the conversion would remove it from sda/sdb. I just didn't know iPXE wouldn't fall through to the NVMe.
Mass-masking from rescue
systemctl --root=/mnt/root mask docker.service docker.socket containerd.service \
synology-active-backup-business-linux-service.service \
tailscaled.service fail2ban.service unattended-upgrades.service \
'actions.runner.pksorensen-agentic-live-www.coolify-runner.service'
echo 'blacklist synosnap' > /mnt/root/etc/modprobe.d/blacklist-synosnap.conf
sed -i '/\/mnt\/bulk/s/^/#/' /mnt/root/etc/fstab
All wasted work — the failure was below systemd, not in userspace. Afterwards I unmasked it all again.
The BIOS boot order, before and after
Boot Option #1: SAMSUNG MZVL21T0HCLR-00B00 (was: IBA CL Slot 00FE = network)
Boot Option #2: P0: HGST HUH721008ALE600
Boot Option #3: P1: HGST HUH721008ALE600
LAN PXE Boot Option ROM: Disabled (was: Enabled)
What I took away
- A change's blast radius can outlive the change window. I reasoned correctly that the GPT conversion would erase GRUB from sda/sdb. What I didn't account for was the chain of what happens when the BIOS is network-first and Hetzner's iPXE is the one trying to chainload — outside the system I was changing. Test your assumptions by actually rebooting — preferably right after a change, on a schedule you control, not when an OOM picks the time.
- Don't diagnose downstream when the problem is upstream. I spent hours on initramfs, services, kernel modules — all of which run after GRUB. The actual failure was that GRUB itself was never loaded.
journalctl --list-bootsshowing no entries after the crash would have ruled out userspace an hour earlier. - A remote graphical console is worth its weight in gold. I avoided escalating to KVM because the box was alive enough to PXE-rescue. That was the wrong frame: a system that can disk-rescue but can't disk-boot is exactly when KVM is most valuable. You need to see what the firmware is doing.
- GPT disks need a BIOS Boot Partition to be BIOS-bootable. A small (~1 MB) partition with the
bios_grubflag is enough. I should have created one on each HDD as part of the GPT conversion. (Genuine TODO.) - Verify the running kernel matches the boot default. The
*** System restart required ***notice means your running kernel and boot-time default have diverged. That's a landmine that only triggers on reboot — the worst possible time to discover an untested combination. Periodic deliberate reboots catch it before an OOM does.
The box has been stable since this recovery — and stayed stable for 22 days before the next disaster, which is a different story entirely.