LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 02-04-2009, 09:58 AM   #1
SteveInTallyFL
Member
 
Registered: May 2008
Location: Tallahassee, FL
Distribution: RHEL4, RHEL 5, OEL4, OEL5
Posts: 58

Rep: Reputation: 17
Unhappy RHEL4 root directory says full but sum of use is nowhere near full. Why?


df -ah says the mounted root (/) is full.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-root
1008M 1008M 0 100% /

I summed the du -sh of all directories under root (not in other mapped shares) and come up with less than 25%.
du -sh /etc
37M /etc

(Attached txt file with complete output of df -ah and du -sh results)

URGENTLY need to know how to find/identify where the space is consumed so I can housekeep it?

Thanks in advance for any advice.
Attached Files
File Type: txt diag_info.txt (1.4 KB, 8 views)
 
Old 02-04-2009, 10:56 AM   #2
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Hi.

You might have files hiding underneath mount points. There is a way of finding out without umount'ing everything.

Share the / filesystem out over NFS to localhost (read-only, unless you like living on the edge), and mount it somewhere (/mnt maybe?). You can then run 'du -sh' at the mount point (i.e. cd /mnt; du -sh) and see where all the space is. The trick works because NFS exports shouldn't cross filesystem boundaries. Mount as NFS V3 to avoid some interestingness with NFS V4

Dave
 
Old 02-04-2009, 11:09 AM   #3
SteveInTallyFL
Member
 
Registered: May 2008
Location: Tallahassee, FL
Distribution: RHEL4, RHEL 5, OEL4, OEL5
Posts: 58

Original Poster
Rep: Reputation: 17
Thanks for advice, Dave.

I am so new at administering my systems that I do not know the proper steps to follow to try your suggestion.

Would I create a directory in the existing /mnt directory? then then mount that as an NFS? Then copy the contents of / over?

I apologize for being so dense.
 
Old 02-04-2009, 11:27 AM   #4
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Follow this:
http://sysdigg.blogspot.com/2008/03/...-on-linux.html
but use
Code:
/ localhost(ro)
in /etc/exports instead of what he's got.

Then run
Code:
mount -f nfs localhost:/ /mnt
and you should have a read-only / filesystem mounted at /mnt, so you can find out where all the space has gone.

Dave
 
Old 02-04-2009, 11:28 AM   #5
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Follow this:
http://sysdigg.blogspot.com/2008/03/...-on-linux.html
but use
Code:
/ localhost(ro)
in /etc/exports instead of what he's got.

Then run
Code:
mount -t nfs localhost:/ /mnt
and you should have a read-only / filesystem mounted at /mnt, so you can find out where all the space has gone.

Dave
 
Old 02-04-2009, 11:37 AM   #6
SteveInTallyFL
Member
 
Registered: May 2008
Location: Tallahassee, FL
Distribution: RHEL4, RHEL 5, OEL4, OEL5
Posts: 58

Original Poster
Rep: Reputation: 17
I may be totally boinked!

Cannot even update /etc/exports because the disk appears full so cannot commit the write.

It was a good idea, though.

Thanks for the pointers. Will add to my toolbox.
 
Old 02-04-2009, 03:28 PM   #7
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
You could try removing some unused packages to make some space.

rpm -qa

will show the installed packages, and

rpm -qi <package name>

will give you some info on the package.

Also, cores usually aren't switched on, but you could see if any core files are lying around taking up space with:

find / -type f -name 'core*'

run the 'file' command on anything it finds to make sure it is actually a core file, though.

Dave
 
Old 02-04-2009, 03:43 PM   #8
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora, Lubuntu, FreeBSD
Posts: 3,930
Blog Entries: 5

Rep: Reputation: Disabled
Quote:
Originally Posted by SteveInTallyFL
URGENTLY need to know how to find/identify where the space is consumed so I can housekeep it?
Post the results of:
Code:
# find / -xdev -size +5M -exec du -k {} \; | sort -nr | head -20
(Note the # prompt. You're going to need to be root.)
 
Old 02-04-2009, 04:27 PM   #9
FragInHell
Member
 
Registered: Sep 2003
Location: Sydney Australia
Distribution: Redhat, Centos, Ubuntu
Posts: 282

Rep: Reputation: 45
You Might have run out of Inodes. Even if the filesystem is full if there are no free inodes you cannot write to it.

df -i will tell the status of Inode Usage.
 
Old 02-04-2009, 09:43 PM   #10
SteveInTallyFL
Member
 
Registered: May 2008
Location: Tallahassee, FL
Distribution: RHEL4, RHEL 5, OEL4, OEL5
Posts: 58

Original Poster
Rep: Reputation: 17
Thanks to all for responses.

Just rebooted server. When I did df -ah I found this:
Filesystem_______________Size___Used__Avail__Use%_Mounted on
/dev/mapper/vg00-root____1008M__212M___746M__23%_____/

I have no idea why the reboot (unmount / remount) would 'clean up' the issue. Can someone explain?

Last edited by SteveInTallyFL; 02-04-2009 at 09:47 PM. Reason: formatting
 
Old 02-05-2009, 04:33 AM   #11
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Hi again.

Such is the mysticism of an inode based filesystem. Basically what's (probably) happened is that an application had one or more files open, which were then rm'ed. On Unix, files which are open aren't actually removed from disk until all the filesystem links are gone and all the open file descriptors are dropped (incidentally, this is one of the reasons that you don't need to reboot Unix boxes when you're updating software - you can remove files from under a running application and it won't necessarily notice).

If there's rm'ed, but still open files on a filesystem, the space will be taken up (so df shows the disk as full), but the files aren't visible any more (so du can't find them to count the space). When you rebooted, all the open file handles were dropped, so the space was freed. The fact that you had processes filling up the / filesystem is worrying, though.

To see processes with the open, but unlinked files, run 'lsof' as root, and take a look at the output. Any processes with these files open should have something like 'anon_inode' listed.

Dave
 
Old 02-05-2009, 05:45 AM   #12
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 561

Rep: Reputation: 58
Your system is doing this due to some application, this happened to me due to samba, but can be anything. Do a reboot and see what happens.
 
  


Reply

Tags
file size limit, filesystem, rhel4


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
The / directory is full sunhui Linux - General 5 03-30-2007 12:11 PM
RHEL4: Proper full system restores hooliganism Red Hat 1 11-29-2006 08:52 AM
/root directory full kpachopoulos Linux - General 6 07-07-2005 12:01 AM
X not running, Root directory full. Serendipity0404 Linux - General 3 04-04-2005 05:00 PM
/tmp directory is full? Moses420ca Linux - Newbie 13 01-21-2005 07:48 PM


All times are GMT -5. The time now is 05:36 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration