One of my backup scripts started reporting errors recently about running out of space:
cp: cannot create regular file `filename in here': No space left on device
Running the command df was showing lots of free space.
However, a quick google on cp "no space left on device" turned up the suggestion to try:
df -i
This showed up the problem straight away - I had run out of inodes.
Once the problem is identified it is generally easy to resolve, so I figured out that there was a cron job that was running a wget task and not discarding the output, and so had saved 600,000+ files in their home directory.
However, when I tried to delete them I encountered another problem:
# rm filepattern*
bash: /bin/rm: Argument list too long
There were so many files that I couldn't use standard delete commands.
Another quick google turned up this gem:
I ran the following command to check that I was going to delete the correct files:
find . | grep filepattern
And then added the command to actually do the delete:
find . | grep filepattern | xargs rm
One final thing was to fix up the cron job that was causing the problem. Adding the parameter --delete-after to wget kept the directory nice and clean.
All fixed. One of the things this has reinforced for me is how important it is to have /home on a seperate partition. If this had not been the case then the problem would have taken longer to happen (due to larger filesystem) but would have been more destructive - as all systems and processes would not have been able to create new files.