How I Debug a Full Linux Server Disk Without Making Things Worse

A disk-full alert is one of the few incidents where the obvious reaction actively makes things worse. rm -rf on the largest directory you can find will occasionally fix it and will occasionally take down the database.

What follows is the order I work in. It is deliberately slow at the start, because the first three commands almost always change what I do next.

Step 0: confirm what is actually full

“Disk full” can mean three different things, and they have different fixes.

# Space, per mounted filesystem
df -h

# Inodes, per mounted filesystem
df -i

If df -h shows a filesystem at 100%, you have a space problem. If df -h looks fine but writes still fail with No space left on device, check df -i — you have an inode problem, typically millions of tiny files (session files, cache entries, unrotated per-request logs). Deleting a few large files will not help at all; you need to delete many files.

The third case: the filesystem is fine, but the process is out of quota, or it is writing to a tmpfs that is sized independently of the disk. df -h lists tmpfs mounts too — read the mount point, not just the percentage.

Note which filesystem is full. On a server with separate /, /var and /home, chasing space on the wrong one wastes the first ten minutes.

Step 1: find the space, top-down

Start at the mount point that is actually full and descend one level at a time.

# One level under /, sorted, without crossing into other filesystems
du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20

The flags matter:

-x stays on one filesystem. Without it you walk into network mounts and /proc and waste minutes.
--max-depth=1 gives you a decision at each level instead of a 40,000-line dump.
2>/dev/null hides the permission noise that otherwise scrolls the useful output away.

Repeat into whichever directory dominates. Usually two or three iterations land you on the culprit.

If ncdu is already installed, it is a nicer interactive version of the same thing:

ncdu -x /var

Step 2: the one that catches people out — deleted-but-open files

This is the single most useful check on a full disk, and it is the one that is most often missed.

If a process has a file open and something deletes that file, the directory entry is gone but the blocks are not freed until the last file descriptor closes. du will not see the space. df still counts it. You end up staring at a filesystem that is “80% used” according to du and 100% according to df.

sudo lsof -nP +L1

+L1 lists open files whose link count is below 1 — exactly the deleted-but-held case. The SIZE/OFF column tells you how much space each one is holding.

Classic cause: a log file was rotated or deleted while the writing process still had it open, because the rotation config used create semantics without signalling the process, or because someone ran rm on a log by hand.

You reclaim this space by making the process close the descriptor. In rough order of preference:

# 1. Ask the service to reopen its log files (best — no downtime)
sudo systemctl reload nginx

# 2. Truncate through the still-open descriptor, if you know the PID and FD
#    (from lsof output: PID 1234, FD 3w)
sudo truncate -s 0 /proc/1234/fd/3

# 3. Restart the service (last resort — this is a real interruption)
sudo systemctl restart some-service

Option 2 frees the space immediately without restarting anything, but confirm what the descriptor is before you truncate it. A log file, yes. A database’s data file, absolutely not.

Step 3: check the usual suspects

With the top-down walk and the deleted-file check done, a handful of directories account for most real-world cases.

Journald. It is bounded by configuration, but the default bound can be generous.

journalctl --disk-usage

Package manager caches.

du -sh /var/cache/apt /var/cache/yum /var/cache/dnf 2>/dev/null

Container layers and volumes. On any host running Docker, this is the first place I look.

docker system df          # summary: images, containers, volumes, build cache
docker system df -v       # per-object detail

Build cache on a CI runner can reach tens of gigabytes without anyone noticing.

Application logs that escaped rotation.

sudo find /var/log -type f -size +100M -printf '%s\t%p\n' 2>/dev/null \
  | sort -rn | head -20

Old kernels, on Debian/Ubuntu hosts with a small /boot:

df -h /boot
dpkg --list | grep linux-image

Reclaiming space

Now, and only now, start deleting — in order of “how certain am I that nothing needs this”.

Safe, reversible, no service impact

# Vacuum the systemd journal down to a size or an age
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

# Package caches — re-downloadable by definition
sudo apt-get clean            # Debian/Ubuntu
sudo dnf clean all            # Fedora/RHEL

Docker, with attention to what the flags mean

# Dangling images and stopped containers only
docker image prune
docker container prune

# Build cache — safe, costs you a slower next build
docker builder prune

# Everything unused INCLUDING named volumes — read this twice
docker system prune -a --volumes

Logs

Prefer truncation over deletion for anything that is currently open:

# Empties the file, keeps the inode, keeps the writer happy
sudo truncate -s 0 /var/log/some-app/app.log

> file and truncate -s 0 both preserve the inode and its permissions. rm does not, and a recreated file often ends up with the wrong owner — so the service silently stops logging until someone notices a week later.

Old files, carefully

# Look first
sudo find /var/log -name '*.gz' -mtime +30 -print

# Then act on the same expression
sudo find /var/log -name '*.gz' -mtime +30 -delete

Always run the -print form first and read the list. The two-step habit costs ten seconds and has saved me more than once.

Then fix the cause

Space reclaimed is not an incident resolved. Three follow-ups, roughly in order of value:

Rotation that actually rotates. Check that the file you truncated has a working logrotate entry, and test it:

sudo logrotate -d /etc/logrotate.d/some-app   # dry run, prints decisions

If the app holds the file open, the config needs copytruncate, or a postrotate hook that signals the process. Rotation that silently does nothing is a very common root cause.

Bound the journal. In /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M

Alert before full, not at full. An alert at 100% gives you no time. Two thresholds work better than one:

A warning at 80% used.
A predictive alert on the trend — “this filesystem will be full within 4 hours at the current rate”. In Prometheus that is predict_linear over node_filesystem_avail_bytes.

And alert on inodes as well as bytes, because the file-count failure mode arrives with no warning at all from a bytes-only dashboard.

The short version

df -h and df -i. Know which one is the problem.
du -xh --max-depth=1 from the full mount point, descending.
lsof -nP +L1 for deleted-but-open files.
Reclaim in order: journal, caches, build artefacts, logs, old files.
Truncate rather than delete anything a process still has open.
Fix rotation and add a predictive alert, or you will be back.

The reason to work in this order is not thoroughness for its own sake. It is that the first three steps are free and reversible, and they routinely change which of the later steps is correct.