Linux Says No Space Left on Device? Check Inodes Before Resizing

Imagine a background worker that writes a small file for each job. It suddenly logs No space left on device. You run df -h, see gigabytes free, and start wondering whether the worker is looking at a different disk. That’s a good first question. The second is whether that disk still has room to describe a new file.

Before resizing, run df -h and df -i against an existing parent directory of the failed write. A filesystem can have available bytes and no free inodes. If the inode count is exhausted, look for a directory full of small files and stop the process creating them before deciding what to remove.

Ask the filesystem that received the write

Use the nearest existing parent of the path in the error. A service might write temporary files under /tmp and final files to a mounted data directory. Checking / by itself can miss either location. For a container, start in its mount view with the app’s actual path. If it’s a mounted volume or bind mount, identify the backing filesystem before deciding what to do on the host.

target=/var/lib/my-worker/jobs
findmnt --target "$target" --output TARGET,SOURCE,FSTYPE
df -h -- "$target"
df -i -- "$target"

Replace the example path with a directory that exists. findmnt --target reports the filesystem for the path, and df reports space or inodes for that filesystem. Compare Avail and IFree directly. A rounded percentage by itself can obscure how much room is left.

A directory full of tiny files can use very few bytes

An inode holds filesystem metadata for an object such as a file or directory. A queue that leaves a separate marker for every completed job can consume many inodes while the markers themselves use little space. On a filesystem such as ext4, inode density is chosen when the filesystem is made. Other filesystem types account for inodes differently, so check the filesystem type before drawing conclusions from df -i. The e2fsprogs manual describes ext4 inode density.

Count entries where the worker writes

Once IFree points to the problem, start with the application directory you already identified. This GNU du command counts inode usage rather than file sizes. The depth flag limits what is printed. The command still scans the tree, so start narrow on a busy system.

scope=/var/lib/my-worker
du --inodes -x --max-depth=1 -- "$scope" | sort -nr | head -15

The first line may be the total for the parent. Compare its child directories, then repeat for the largest one. -x avoids descending into directories on other filesystems. If the failing path is on a separate mount, run the scan inside that mount. Permission errors can make results incomplete, and hard links can share an inode, so a pathname count can differ from the inode count. du documents these options.

Fix what you can verify

• Inodes are exhausted and the producer is still running. Pause or throttle the producer if the service allows it. Identify files that your retention policy permits you to remove, such as expired cache entries. Use the application’s cleanup mechanism when available. Avoid sweeping deletes in upload, queue or database directories.
• Inodes are available but bytes are exhausted. Investigate what grew in bytes. If Docker logs are the cause, use Krova’s log retention guide.
• Both counts look healthy. Confirm that the command ran in the service’s mount context and checked the path where creation failed. Then inspect filesystem-specific limits, quotas or the writable backing layer before changing disk size.

After an authorized cleanup, run df -i again and retry one representative job with the usual service identity. A deleted pathname may leave its inode allocated while a process still has the file open, or while another hard link remains. If the job succeeds, compare inode usage after the next normal batch to see whether the producer is creating the same backlog. Linux documents when unlinking releases a file.

This guide was prepared with AI assistance using the linked documentation. The worker is hypothetical. The read-only commands were checked on a local fixture with free inodes. That check did not reproduce an exhausted filesystem.