[SOLVED] Out of inodes and can't access /tmp!

fizix610 · 01-18-2010, 03:50 PM

Ok, I'm hoping somebody can help me out with this. Its kind of a strange issue:

My server started acting flaky this weekend and my Webmin interface was throwing strange errors. I finally tracked it down to the fact that I was out of inodes on my primary partition. I'm fairly certain that the /tmp folder has an outrageous number of files in it. I can't do an ls on the directory because the console just sits there forever after I issue the command. I also tried to do an rm -rf on the /tmp directory and it did the same thing. Any suggestions on how I can clear out this directory?

AleLinuxBSD · 01-18-2010, 03:58 PM

Probably you haven't sufficient space on your OS:
Maintaining free space in filesystems

You could access on your Pc using a livecd and remove the file unnecessary ...

chrism01 · 01-18-2010, 06:08 PM

Please add your distro & version to your info so we can advise more accurately.

fizix610 · 01-18-2010, 06:21 PM

Quote:

Originally Posted by chrism01

Please add your distro & version to your info so we can advise more accurately.

I'm running CentOS 5.4 and I have Webmin installed.

chrism01 · 01-18-2010, 06:34 PM

In that case, I'd use the install DVD. Boot from that and at the initial prompt (where it asks what you want to do) type in:

linux rescue

It'll then boot from the DVD, and mount (read-only ?) your HDD(s). You can then

mount -o remount,rw /tmp

or whatever partition it's on and delete files etc.

FragInHell · 01-18-2010, 08:53 PM

Also, if your using ext3 then the file system has problems over 32,000 files.

Therefore you might have to delete in small sections

e.g rm /tmp/a*
rm /tmp/b*

etc

df -i shows u your inode usage, the only way to increase inodes is to re-create a file system

mkfs -N is what you need or also look at -T option.

fizix610 · 01-19-2010, 08:29 AM

Quote:

Originally Posted by FragInHell

Also, if your using ext3 then the file system has problems over 32,000 files.

Therefore you might have to delete in small sections

e.g rm /tmp/a*
rm /tmp/b*

etc

df -i shows u your inode usage, the only way to increase inodes is to re-create a file system

mkfs -N is what you need or also look at -T option.

I booted from the install disc with the rescue option but I'm still having trouble. I tried deleting with rm -r [a-m]* while in the tmp directory that it mounted from my filesystem but after accessing the hard drive for about a half hour it didn't find any files to delete. I can't do an rm -r * on the directory because it gives me an error about having too many arguments (which I believe means there are too many files for it to delete). How else can I break up this delete process so there aren't too many files at a time? Keep in mind that I can't do an ls so I don't know the names or dates on the files.

FragInHell · 01-19-2010, 05:31 PM

Hi.

Ok so the problem is file system limits booting to a rescue disk is not going to help you here.

Even rm -r [a-m]* might be too large a number, try rm -r a* (you could of course script this to do each letter)

chrism01 · 01-19-2010, 05:41 PM

Yes, one letter at a time should do the trick. You may also have filenames starting with a number, so use the same trick there.

fizix610 · 01-20-2010, 08:10 AM

Quote:

Originally Posted by FragInHell

Hi.

Ok so the problem is file system limits booting to a rescue disk is not going to help you here.

Even rm -r [a-m]* might be too large a number, try rm -r a* (you could of course script this to do each letter)

Usually when the request matches too many files it gives me the "too many arguments" error but when I tried rm -r [a-m]* it gave me a "no such file or directory error" leading me to believe that there aren't any files in that directory that start with a through m. I'm going to take chrism01's advice and see if it matches any numbers.

cantab · 01-20-2010, 08:54 AM

Your expansion syntax is wrong; the first and last letters are separated by two dots, not a hyphen. You might try curly brackets instead, eg rm -r {a..m}*

fizix610 · 01-20-2010, 09:01 AM

Quote:

Originally Posted by cantab

Your expansion syntax is wrong; the first and last letters are separated by two dots, not a hyphen. You might try curly brackets instead, eg rm -r {a..m}*

That was worth a try; this is what I got though:

Code:

rm: cannot remove `/tmp/a*': No such file or directory
rm: cannot remove `/tmp/b*': No such file or directory
rm: cannot remove `/tmp/c*': No such file or directory
rm: cannot remove `/tmp/d*': No such file or directory
rm: cannot remove `/tmp/e*': No such file or directory
rm: cannot remove `/tmp/f*': No such file or directory
rm: cannot remove `/tmp/g*': No such file or directory
rm: cannot remove `/tmp/h*': No such file or directory
rm: cannot remove `/tmp/i*': No such file or directory
rm: cannot remove `/tmp/j*': No such file or directory
rm: cannot remove `/tmp/k*': No such file or directory
rm: cannot remove `/tmp/l*': No such file or directory
rm: cannot remove `/tmp/m*': No such file or directory

I proceeded through all numbers and letters and got the same results! How is this possible? I know there's something in the directory because it's using 78M.

chrism01 · 01-20-2010, 06:41 PM

Did you check UPPERCASE as well; *nix is case sensitive.
Also, it could be that the partition that /tmp is on is full; might not be that dir specifically.

fizix610 · 01-22-2010, 10:01 AM

OK, I let an ls run all night and all day while I was at work and finally got a directory listing. It turns out that I had thousands of CURLCOOKIE files in /tmp and those used up all my inodes. These must have been generated by my webcrawler script that was using CURL and would generate a cookie file for each site it connected to. I've been systematically deleting them using "rm CURLCOOKIE{a..e}*". Looks good so far. Thanks for your help!

chrism01 · 01-24-2010, 07:49 PM

You should also amend the curl script to clean up after itself ..