cp "No space left on device" problem - Solved

Running the command df was showing lots of free space.

However, a quick google on cp "no space left on device" turned up the suggestion to try:

This showed up the problem straight away - I had run out of inodes.

Once the problem is identified it is generally easy to resolve, so I figured out that there was a cron job that was running a wget task and not discarding the output, and so had saved 600,000+ files in their home directory.

However, when I tried to delete them I encountered another problem:

There were so many files that I couldn't use standard delete commands.

Another quick google turned up this gem:

I ran the following command to check that I was going to delete the correct files:

One final thing was to fix up the cron job that was causing the problem. Adding the parameter --delete-after to wget kept the directory nice and clean.

All fixed. One of the things this has reinforced for me is how important it is to have /home on a seperate partition. If this had not been the case then the problem would have taken longer to happen (due to larger filesystem) but would have been more destructive - as all systems and processes would not have been able to create new files.