498% committed: anatomy of a silent server death

My Hetzner dedicated box ran clean for 22 days. Then, around 00:40 last night, it went silent. No alerts, no SSH, no ping. No emails from Hetzner — they don't bother you when your box is alive enough not to trip their hardware watchdogs, but dead enough that nothing's responding from inside.

A manual hardware reset from the Robot panel brought it back this morning, fast — because we'd already nailed the BIOS boot order and GRUB after a previous adventure. The interesting question wasn't how to bring it back; it was what actually killed it.

Last time this happened I had to ask Hetzner to attach a console so I could see what the machine was doing. This time a simple reboot was enough — and then digging into it afterwards.

No OOM, no panic, no log

First instinct on any Linux death: journalctl -b -1. If the kernel OOM-killer had fired, there'd be unmistakable signatures — invoked oom-killer, Out of memory: Killed process, page allocation failures.

There were none.

No kernel panic. No allocation failures. Just three lonely lines from systemd-journald saying "under memory pressure, flushing caches" around 00:37, and then journald simply stopped writing at 00:40:58. The very last entries are mundane Docker health-check timeouts — the kind of thing that happens when something is congested but still alive. Then nothing.

That's not OOM. That's a hang.

What sysstat caught

Luckily, sysstat had been quietly sampling system state every 10 minutes for the entire 22-day boot — one file per day under /var/log/sysstat/. The smoking gun was in sa13:

Two columns matter.

%commit is the ratio of committed memory to physical RAM plus swap. Committed memory is what processes have asked for — via mmap, malloc, brk — regardless of whether they've actually touched it yet. Linux defaults to vm.overcommit_memory=0: "heuristic" overcommit. The kernel happily hands out far more virtual address space than physically exists, betting that most processes never use everything they reserve. (And usually they don't. Java heaps reserve gigabytes they'll never touch. Node's V8 does similar tricks.)

At 23:20 the commit crossed 100%. At 23:50 it was at 222%. By 00:40 it was at 498% — nearly 678 GB committed against 125 GB of physical RAM and 4 GB of swap.

Load average went from 2.28 to 5,369 in forty minutes. That's thousands of threads queued up, waiting on something they can't even be killed out of.

What actually happened

I'd started some heavy work in a devcontainer around 23:20 — a pks brain extract spawning hundreds of workers (that's the app-level story from the same night). But the precise what isn't the point. The pattern is.

The workload committed enormous amounts of memory. For a while that's fine — committed memory isn't real memory until it's touched. But as the workload actually ran, it began touching pages. The kernel had to back those commits with physical RAM. Free pages dwindled.

Then, swap. Every single swap slot was occupied with something the kernel had decided to evict earlier. There was no free swap to evict more pages into. So the kernel was stuck doing the swap-in/swap-out shuffle: read a page back from swap, write a different page out, over and over. Classic swap thrashing.

Disk I/O climbed. Processes that needed memory blocked on I/O. They went into D-state — uninterruptible sleep — and accumulated. Load average exploded. The kernel's OOM heuristic stayed quiet, because from its perspective reclaim was working: pages were moving between RAM and swap. That looks like success, not failure.

Eventually the I/O queue was so deep the system couldn't make forward progress on anything. Including writing journal entries. Including answering ping. Including the SSH daemon's accept loop.

The box wasn't out of memory in the OOM sense. It was deadlocked on its own swap.

The lesson — and the fix

The actual fix is unglamorous: limit how much memory each container can use. Docker has supported this via cgroups for years; I just hadn't been using it on a host where I run a lot of devcontainers.

Applied to all existing containers, live, no restart:

for c in $(docker ps -aq); do
  docker update --memory=64g --memory-swap=64g "$c"
done

Setting --memory-swap=64g (equal to --memory) disables swap usage for the container entirely — which is the right call here, since swap thrashing was the failure mode. If a container tries to exceed the ceiling, the OOM-killer fires against processes inside that container's cgroup only. The host stays sane.

This is the other half of the spawn narrow story: the app layer shouldn't carry tools it doesn't use — but even a well-behaved app, the host should survive. Per-container limits are that insurance.

sudo tee /usr/local/bin/docker-default-memcap.sh > /dev/null << 'EOF'
#!/bin/bash
CAP="${1:-64g}"
for cid in $(docker ps -aq 2>/dev/null); do
  lim=$(docker inspect -f '{{.HostConfig.Memory}}' "$cid" 2>/dev/null)
  if [ "$lim" = "0" ]; then
    name=$(docker inspect -f '{{.Name}}' "$cid" 2>/dev/null | sed 's|^/||')
    docker update --memory="$CAP" --memory-swap="$CAP" "$cid" > /dev/null 2>&1 \
      && logger -t docker-memcap "Applied $CAP cap to $name"
  fi
done
EOF
sudo chmod +x /usr/local/bin/docker-default-memcap.sh
(sudo crontab -l 2>/dev/null; echo "* * * * * /usr/local/bin/docker-default-memcap.sh 64g") | sudo crontab -

sudo tee /usr/local/bin/mem-monitor.sh > /dev/null << 'EOF'
#!/bin/bash
{
  echo "=== $(date -Iseconds) ==="
  free -h
  cat /proc/buddyinfo | grep -E "DMA32|Normal"
  ps -eo rss,user,comm --sort=-rss | head -11
  docker stats --no-stream --format '{{.Name}} {{.MemUsage}} ({{.MemPerc}})' | head -15
} >> /var/log/mem-monitor.log 2>&1
EOF
sudo chmod +x /usr/local/bin/mem-monitor.sh
(sudo crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/mem-monitor.sh") | sudo crontab -

sudo tee /etc/sysctl.d/99-debug-hung.conf > /dev/null << 'EOF'
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = 10
kernel.softlockup_all_cpu_backtrace = 1
EOF
sudo sysctl -p /etc/sysctl.d/99-debug-hung.conf