Linux “No space left on device” Error

symptom:

a web site is down, but the httpd (in my case lighttpd) service started okay. restarting the service still has the same problem.

touch test_file, get the “No space left on device” error

df –h, show a lot of free disk space

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              10G  2.8G  6.8G  29% /
/dev/sda1              99M   21M   74M  22% /boot
tmpfs                 125M     0  125M   0% /dev/shm

problem:

out of inodes

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1             655776  655776       0  100% /
/dev/sda1              26208      38   26170    1% /boot
tmpfs                  31871       1   31870    1% /dev/shm

fix:

find the files of a certain size (e.g. great than three blocks in this example) to locate these files (in my case is the ruby session files in /srv/www/lighttpd/rails/tmp/sessions, the file name is ruby_sess.xxxxxxxx)

find / –size +3 –print

but rm –f ruby_sess.* failed because “bash: /bin/rm: Argument list too long” (see http://en.kioskea.net/faq/1086-unable-to-delete-file-argument-list-too-long)

ls ruby_sess.* | xargs rm get “-bash: /usr/bin/ls: Argument list too long”

find . –type f –name ruby_sess.* | xargs rm get “-bash: /usr/bin/find: Argument list too long”

finally find . –name “ruby_sess.*” | xargs rm worked

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1             655776  122320  533456   19% /
/dev/sda1              26208      38   26170    1% /boot
tmpfs                  31871       1   31870    1% /dev/shm