I thought I'd been hacked

A confession: I run more infrastructure than I have any business running. One Hetzner dedicated box with about 30 services, half a dozen devcontainers, a private Docker registry, a FreeSWITCH stack, agent FTP ingest from my Sony camera, plus whatever I'm hacking on that week. Most of it works most of the time. When it doesn't, the failure modes are usually exotic.

This is the story of the incident that looked exotic — like "I've been pwned" exotic — and turned out to be the most boring possible diagnosis. It's also the story of how, while cleaning up, I discovered something completely unrelated wasting 13 TB of my storage, set out to fix it, did it carefully — and quietly planted a time bomb in my boot config that wouldn't detonate for another five days. That second part is the next post. This is the setup.

The symptom: the box was eating itself

I came back to the server after a couple of days away. SSH took forever to give me a prompt. Once in, htop showed a wall of memory pressure, load average climbing into the double digits, services flapping. Containers restarting at random. The box wasn't fully down, but it was the kind of unwell that makes a paranoid mind go straight to: someone's mining crypto on my server.

I'd been bitten by that exact thing earlier in the year — actual cryptominers, processes named /tmp/runnv/runnv chewing 14 GB of RAM each. The symptom pattern was familiar enough that I assumed the same diagnosis applied.

It didn't. But before getting to what it actually was, the methodology matters.

The 5-layer audit

When you suspect compromise, you don't poke and hope. You audit systematically. Five layers:

• Process layer. Anything running you can't identify? Suspicious paths (/tmp/*, /dev/shm/*, weirdly-named binaries)? ps auxf and look for anomalies.
• Network layer. Listening sockets you didn't put there? Outbound connections to weird IPs or pools? ss -tlnp and ss -tnp | grep ESTABLISHED.
• Persistence layer. Cron jobs, systemd timers, services, ~/.ssh/authorized_keys, /etc/sudoers.d/*. Anywhere an attacker survives a reboot.
• Filesystem layer. Recently modified files in system dirs: find /etc /usr/local /opt -mtime -7 -ls — narrow the window aggressively to whenever things were last fine.
• Container layer. Containers you didn't start? Image pulls you didn't initiate? docker ps -a and docker image ls --digests.

All five came back clean. No rogue processes. No unexpected listeners. No new SSH keys. No modified system binaries. No mystery containers.

Which left me with: the box is sick, but it's sick from legitimate workload, not malice.

The actual culprit, in three parts

One: a service crash-looping for three months. systemctl --failed and a glance at journalctl -u freeswitch-logger.service revealed something embarrassing: a service I'd written months ago and forgotten had restarted over 655,000 times. The unit file had a shell-escaping bug that meant the command never ran. systemd faithfully restarted it. Every restart spawned a process, allocated some memory, failed, exited, triggered another restart. Multiply by months. systemctl mask freeswitch-logger.service — the bleed stopped.

Two: fail2ban looping on its own config bug. fail2ban is supposed to be the defender against SSH brute-forcing. Mine had been failing to start cleanly for — again — three months, crash-looping. Root cause: a duplicate ignoreip line in /etc/fail2ban/jail.local. Fix: sed out the duplicate, restart, watch it come up clean for the first time in a season.

Three: a Docker landfill. The big one. docker ps -a | wc -l returned 256. Of those, 230 were stopped. docker volume ls | wc -l returned 2,173 — about half dangling from containers removed without -v.

How does a box accumulate 256 containers? Easy: I run aspire run to spin up my dev environment, and that orchestrator spawns a handful of containers each time (Next.js dev servers, .NET Aspire AppHost, some Postgres, some Redis). On Ctrl-C, Aspire stops the containers but doesn't always remove them. Multiply by every time I started Aspire over six months — and the box ends up holding hundreds of dead containers, each with its own writable layer and a clutch of attached volumes.

Cleanup reclaimed about 52 GB inside my main devcontainer alone, and on the order of 90 GB on the host overall. Load average dropped from "alarming" to "boring" within minutes.

The bonus discovery: 13 TB gone

While I was on the box anyway, I ran lsblk and df -h. What I saw stopped me: three disks — one ~954 GB NVMe and two 7.3 TB HDDs — all combined into a single RAID5 array yielding 1.8 TB usable. With RAID5 of three disks, the array is sized to the smallest disk. The NVMe being there capped the entire array's usable capacity. The other 6.3 TB on each HDD — about 13 TB total — was completely unaddressed.

This was Hetzner's default installimage layout from when I'd ordered the box ten months earlier. I'd accepted "automatic RAID setup" without thinking carefully about what they'd actually do — and they'd done something that "works" in the most pedantic sense (it's redundant! it's a RAID!) while wasting most of my capacity. I'd been on the box ten months without noticing.

Reclaiming the 13 TB — and the bomb I planted

The plan was simple in shape: convert sda and sdb from MBR to GPT in place with sgdisk -g, carve a new partition in the previously-unaddressed space, build a fresh RAID1 array md3, mount as /mnt/bulk, migrate the FTP photo-ingest data over.

Simple in shape, careful in execution. A full block-level backup to a Synology NAS over Tailscale first (~415 GB), verified restorable, then the disk work step by step with explicit confirmation between each destructive operation. RAID stayed healthy [UUU] throughout. The new /mnt/bulk came online clean. The migration went smoothly. I rebooted nothing during this work — the RAID was modified live; userspace never noticed.

And that's where I planted the bomb, without knowing it. The MBR→GPT conversion would destroy GRUB on sda and sdb. I reasoned that the NVMe (which I wasn't touching) would still boot, and that BIOS would fall through to it. That reasoning was almost right. The "almost" cost me a whole night five days later — but that's the next post.

The lesson

The "I've been hacked" suspicion turned out to be the most mundane diagnosis: a server quietly choking on its own accumulated state, with a couple of misconfigured services pouring on fuel. The actual fix was just housekeeping, done thoroughly. I went to bed with a clear conscience. Five days later the box died again — and the post-mortem traced the cause back to a decision I made during this exact cleanup.

$ docker system df
TYPE         TOTAL    ACTIVE   SIZE      RECLAIMABLE
Images       57       12       42.5GB    28.1GB (66%)
Containers   256      26       8.3GB     7.1GB (85%)
Volumes      2173     78       145.2GB   89.4GB (61%)
Build Cache  —        —        12.4GB    12.4GB (100%)

docker container prune -f                       # remove stopped containers
docker volume prune --filter "label!=keep" -f   # remove dangling volumes
docker image prune -a -f                        # remove unused images
docker builder prune -a -f                       # nuke the build cache

NAME        SIZE  MOUNTPOINT
nvme0n1     954G
├─md0         4G  [SWAP]
├─md1         1G  /boot
└─md2       1.8T  /
sda         7.3T
├─md0         …
├─md1         …
└─md2         …
sdb         7.3T
├─md0         …
├─md1         …
└─md2         …