LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   RHEL4 root directory says full but sum of use is nowhere near full. Why? (http://www.linuxquestions.org/questions/linux-server-73/rhel4-root-directory-says-full-but-sum-of-use-is-nowhere-near-full-why-702250/)

SteveInTallyFL 02-04-2009 09:58 AM

RHEL4 root directory says full but sum of use is nowhere near full. Why?
 
1 Attachment(s)
df -ah says the mounted root (/) is full.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-root
1008M 1008M 0 100% /

I summed the du -sh of all directories under root (not in other mapped shares) and come up with less than 25%.
du -sh /etc
37M /etc

(Attached txt file with complete output of df -ah and du -sh results)

URGENTLY need to know how to find/identify where the space is consumed so I can housekeep it?

Thanks in advance for any advice.

ilikejam 02-04-2009 10:56 AM

Hi.

You might have files hiding underneath mount points. There is a way of finding out without umount'ing everything.

Share the / filesystem out over NFS to localhost (read-only, unless you like living on the edge), and mount it somewhere (/mnt maybe?). You can then run 'du -sh' at the mount point (i.e. cd /mnt; du -sh) and see where all the space is. The trick works because NFS exports shouldn't cross filesystem boundaries. Mount as NFS V3 to avoid some interestingness with NFS V4

Dave

SteveInTallyFL 02-04-2009 11:09 AM

Thanks for advice, Dave.

I am so new at administering my systems that I do not know the proper steps to follow to try your suggestion.

Would I create a directory in the existing /mnt directory? then then mount that as an NFS? Then copy the contents of / over?

I apologize for being so dense.

ilikejam 02-04-2009 11:27 AM

Follow this:
http://sysdigg.blogspot.com/2008/03/...-on-linux.html
but use
Code:

/ localhost(ro)
in /etc/exports instead of what he's got.

Then run
Code:

mount -f nfs localhost:/ /mnt
and you should have a read-only / filesystem mounted at /mnt, so you can find out where all the space has gone.

Dave

ilikejam 02-04-2009 11:28 AM

Follow this:
http://sysdigg.blogspot.com/2008/03/...-on-linux.html
but use
Code:

/ localhost(ro)
in /etc/exports instead of what he's got.

Then run
Code:

mount -t nfs localhost:/ /mnt
and you should have a read-only / filesystem mounted at /mnt, so you can find out where all the space has gone.

Dave

SteveInTallyFL 02-04-2009 11:37 AM

I may be totally boinked!

Cannot even update /etc/exports because the disk appears full so cannot commit the write.

It was a good idea, though.

Thanks for the pointers. Will add to my toolbox.

ilikejam 02-04-2009 03:28 PM

You could try removing some unused packages to make some space.

rpm -qa

will show the installed packages, and

rpm -qi <package name>

will give you some info on the package.

Also, cores usually aren't switched on, but you could see if any core files are lying around taking up space with:

find / -type f -name 'core*'

run the 'file' command on anything it finds to make sure it is actually a core file, though.

Dave

anomie 02-04-2009 03:43 PM

Quote:

Originally Posted by SteveInTallyFL
URGENTLY need to know how to find/identify where the space is consumed so I can housekeep it?

Post the results of:
Code:

# find / -xdev -size +5M -exec du -k {} \; | sort -nr | head -20
(Note the # prompt. You're going to need to be root.)

FragInHell 02-04-2009 04:27 PM

You Might have run out of Inodes. Even if the filesystem is full if there are no free inodes you cannot write to it.

df -i will tell the status of Inode Usage.

SteveInTallyFL 02-04-2009 09:43 PM

Thanks to all for responses.

Just rebooted server. When I did df -ah I found this:
Filesystem_______________Size___Used__Avail__Use%_Mounted on
/dev/mapper/vg00-root____1008M__212M___746M__23%_____/

I have no idea why the reboot (unmount / remount) would 'clean up' the issue. Can someone explain?

ilikejam 02-05-2009 04:33 AM

Hi again.

Such is the mysticism of an inode based filesystem. Basically what's (probably) happened is that an application had one or more files open, which were then rm'ed. On Unix, files which are open aren't actually removed from disk until all the filesystem links are gone and all the open file descriptors are dropped (incidentally, this is one of the reasons that you don't need to reboot Unix boxes when you're updating software - you can remove files from under a running application and it won't necessarily notice).

If there's rm'ed, but still open files on a filesystem, the space will be taken up (so df shows the disk as full), but the files aren't visible any more (so du can't find them to count the space). When you rebooted, all the open file handles were dropped, so the space was freed. The fact that you had processes filling up the / filesystem is worrying, though.

To see processes with the open, but unlinked files, run 'lsof' as root, and take a look at the output. Any processes with these files open should have something like 'anon_inode' listed.

Dave

robertjinx 02-05-2009 05:45 AM

Your system is doing this due to some application, this happened to me due to samba, but can be anything. Do a reboot and see what happens.


All times are GMT -5. The time now is 07:38 AM.