df -ah says the mounted root (/) is full.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-root
1008M 1008M 0 100% /

I summed the du -sh of all directories under root (not in other mapped shares) and come up with less than 25%.
du -sh /etc
37M /etc

(Attached txt file with complete output of df -ah and du -sh results)

URGENTLY need to know how to find/identify where the space is consumed so I can housekeep it?

Thanks in advance for any advice.

Attached Files

diag_info.txt (1.4 KB, 8 views)

Hi.

You might have files hiding underneath mount points. There is a way of finding out without umount'ing everything.

Share the / filesystem out over NFS to localhost (read-only, unless you like living on the edge), and mount it somewhere (/mnt maybe?). You can then run 'du -sh' at the mount point (i.e. cd /mnt; du -sh) and see where all the space is. The trick works because NFS exports shouldn't cross filesystem boundaries. Mount as NFS V3 to avoid some interestingness with NFS V4

Dave

Thanks for advice, Dave.

I am so new at administering my systems that I do not know the proper steps to follow to try your suggestion.

Would I create a directory in the existing /mnt directory? then then mount that as an NFS? Then copy the contents of / over?

I apologize for being so dense.

Follow this:
http://sysdigg.blogspot.com/2008/03/...-on-linux.html
but usein /etc/exports instead of what he's got.

Then runand you should have a read-only / filesystem mounted at /mnt, so you can find out where all the space has gone.

Dave

Follow this:
http://sysdigg.blogspot.com/2008/03/...-on-linux.html
but usein /etc/exports instead of what he's got.

Then runand you should have a read-only / filesystem mounted at /mnt, so you can find out where all the space has gone.

Dave

I may be totally boinked!

Cannot even update /etc/exports because the disk appears full so cannot commit the write.

It was a good idea, though.

Thanks for the pointers. Will add to my toolbox.

You could try removing some unused packages to make some space.

rpm -qa

will show the installed packages, and

rpm -qi <package name>

will give you some info on the package.

Also, cores usually aren't switched on, but you could see if any core files are lying around taking up space with:

find / -type f -name 'core*'

run the 'file' command on anything it finds to make sure it is actually a core file, though.

Dave

Post the results of:
(Note the # prompt. You're going to need to be root.)

You Might have run out of Inodes. Even if the filesystem is full if there are no free inodes you cannot write to it.

df -i will tell the status of Inode Usage.

Thanks to all for responses.

Just rebooted server. When I did df -ah I found this:
Filesystem_______________Size___Used__Avail__Use%_Mounted on
/dev/mapper/vg00-root____1008M__212M___746M__23%_____/

I have no idea why the reboot (unmount / remount) would 'clean up' the issue. Can someone explain?

Hi again.

Such is the mysticism of an inode based filesystem. Basically what's (probably) happened is that an application had one or more files open, which were then rm'ed. On Unix, files which are open aren't actually removed from disk until all the filesystem links are gone and all the open file descriptors are dropped (incidentally, this is one of the reasons that you don't need to reboot Unix boxes when you're updating software - you can remove files from under a running application and it won't necessarily notice).

If there's rm'ed, but still open files on a filesystem, the space will be taken up (so df shows the disk as full), but the files aren't visible any more (so du can't find them to count the space). When you rebooted, all the open file handles were dropped, so the space was freed. The fact that you had processes filling up the / filesystem is worrying, though.

To see processes with the open, but unlinked files, run 'lsof' as root, and take a look at the output. Any processes with these files open should have something like 'anon_inode' listed.

Dave

Your system is doing this due to some application, this happened to me due to samba, but can be anything. Do a reboot and see what happens.