When the server ran out of inodes

A while ago I couldn’t add any files to a Debian server I control. It was pretty odd, as there was plenty of free disk space available (df -h will show that).

After I deleted a number of web cache files, I could add new files again. But soon the problem would return, leaving me wondering what was wrong.

Then I ran df -i, which showed that the server had run out of inodes. From Wikipedia:

Each inode stores the attributes and disk block location(s) of the filesystem object’s data.

Now, where were all those files that ate my inodes? After much searching I found that /var/lib/php5 contained millions of files. It is where PHP stores its session files. Obviously there was something wrong with the server’s PHP garbage collection, as session files are supposed to be removed after a while.

Session removal is handled by a cron job in /etc/cron.d/php5, which runs every 30 minutes. It calls two scripts in /usr/lib/php5 : maxlifetime and sessionclean. I tried to run them from the command line, but they failed and reported a deprecated entry in one of the server’s php.ini files.

The entry (I can’t recall exactly which one it was) had become deprecated with the upgrade from PHP 5.3 to 5.4. I may or may not have read about it, but as PHP (Apache and cli) worked fine after the upgrade I probably just didn’t bother. My mistake.

Removing the files

I fixed the php.ini file, restarted Apache and started to remove the session files. That turned out to be easier said than done. Simply running rm /var/lib/php5/sess* would more or less bring the server to a halt. I stopped the cron job in /etc/cron.d/php5 to prevent an enormous CPU spike.

Here are a couple of commands I found useful for handling large numbers of files in loops. Use at own risk.

The session files have names like sess_51c6b3a6e5620d59c3c13cfdc739b1aa. To find out how many files there are with names starting with for example sess_5, run

n=0; for i in /var/lib/php5/sess_5*; do n=$((n+1)); done; echo $n

Multiply the output by 16 and you have an estimated total number of files. If there are millions of files in the directory, you may need to use a more specific file name pattern (like sess_51c). Actually, it’s smart to start with a more specific pattern and loosen it step by step to find out what the server can handle.

Show the modification dates of a set of files:

for i in /var/lib/php5/sess_5*; do echo $(date -r $i); done

To remove a set of files that are over one day old:

now=$(date +%s); time for i in /var/lib/php5/sess_5*; do if ((($(stat "$i" -c '%Z') + (86400 )) < $now)); then rm -fv $i; fi; done

-f makes it only remove files (you never know…), -v makes it verbose, so you can see what’s going on. time makes it show script execution duration when it finishes.

Faster, doesn’t care how old the files are, i.e. kills current sessions:

for i in /var/lib/php5/sess_5*; do rm -fv $i; done

It took me at least a day to get rid of the files and have the server operate normally again.