LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-14-2013, 09:15 AM   #1
TheGoatreich
LQ Newbie
 
Registered: Aug 2013
Posts: 6

Rep: Reputation: Disabled
Free disk space disparity


Hey all, I wonder if someone can help me stop banging my head against a wall with disk space issues?

I have a Red Hat Enterprise 64-bit 6.2 server that is reporting the root partition is 100% used.

Code:
[root@servername /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_servername-lv_root
                       32G   31G     0 100% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/cciss/c0d0p1     485M   89M  371M  20% /boot
/dev/mapper/vg_servername-lv_home
                       18G  173M   17G   2% /home
//backupserver/backups$
                       69G   28G   41G  41% /mnt/backups
However disk usage is reporting that the root is using far less space than the 31GB df is reporting:

Code:
[root@servername /]# du -h --max-depth=1
7.7M    ./bin
14M     ./sbin
0       ./misc
2.3G    ./var
46M     ./tmp
26M     ./lib64
0       ./net
0       ./sys
4.0K    ./srv
152K    ./home
176K    ./dev
4.0K    ./media
308M    ./lib
4.6M    ./root
25M     ./etc
4.0K    ./opt
16K     ./lost+found
79M     ./boot
4.0K    ./cgroup
4.0K    ./selinux
2.6G    ./usr
du: cannot access `./proc/2643/task/2643/fd/4': No such file or directory
du: cannot access `./proc/2643/task/2643/fdinfo/4': No such file or directory
du: cannot access `./proc/2643/fd/4': No such file or directory
du: cannot access `./proc/2643/fdinfo/4': No such file or directory
0       ./proc
3.8G    ./mnt
9.1G    .
I've googled this like mad and have found a lot of people suggesting that the discrepancy is often deleted files that are still open, and say to run lsof | grep deleted to find out what process is using these files, but running that command on my server comes back with nothing.

Can anybody please help me to find what is using the extra space on my root partition?
 
Old 08-14-2013, 09:52 AM   #2
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 2,961

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
Another thing that could use space and be hidden from du would be files hidden under an active mount point. Do this:
Code:
mkdir /tmp/rootfs
mount --bind / /tmp/rootfs
Now you can look in /tmp/rootfs/boot, /tmp/rootfs/home, and any directories in /tmp/rootfs/mnt and see if anything is hidden there. You can also run "du /tmp/rootfs" to get a picture of the root filesystem without anything hidden by other mounted filesystems. If there is stuff that you want to get rid of, just remove it. Run "umount /tmp/rootfs" when you are done.
 
2 members found this post helpful.
Old 08-15-2013, 05:31 AM   #3
TheGoatreich
LQ Newbie
 
Registered: Aug 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
Thanks, that was really helpful.

Running du on /tmp/rootfs/mnt I could see that 25GB of data was being held with /tmp/rootfs/mnt/backups.
Ls on that directory showed me lots of files, that have been copied to the /mnt/backups samba partition, but for whatever reason have stayed within the tmp partition and not been cleared out.

The earliest file in the /tmp/rootfs/mnt directory was 27th July 2013, so this has been working with no issues for a year, until then. Very odd.

I've cleared the data in that directory, and unmounted the /tmp/rootfs partition, and now my df -h / is looking much more healthy.

I'll keep a close eye on that /tmp directory over the next few weeks to check it's not filling up again, and if it is I'll troubleshoot why.

Thanks again for your reply, I've been frustrated with this all week.
 
Old 08-15-2013, 02:50 PM   #4
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 2,961

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
Quote:
Originally Posted by TheGoatreich View Post
Running du on /tmp/rootfs/mnt I could see that 25GB of data was being held with /tmp/rootfs/mnt/backups.
Ls on that directory showed me lots of files, that have been copied to the /mnt/backups samba partition, but for whatever reason have stayed within the tmp partition and not been cleared out.
What probably happened was that a backup procedure was run while no drive was mounted on /mnt/backups. All the files created by that procedure just ended up stored under that directory in the root filesystem.

FWIW, I protect myself from having that happen by putting my backups into a subdirectory "rback" on my backup drive. My backup script refuses to run if it doesn't see a directory /mnt/backups/rback, and that directory would not be present unless my backup drive were mounted.
 
1 members found this post helpful.
Old 08-15-2013, 03:53 PM   #5
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 2,324
Blog Entries: 1

Rep: Reputation: 447Reputation: 447Reputation: 447Reputation: 447Reputation: 447
Other possibilities include:

Data that is in / itself, since du shows the space of the sub-directories. Until the very last hit, but unless you add up all the previous hits and subtract you could miss a large chunk of data on / without looking at / directly.

When /var/log/ gets archived via logrotate and therefor cron, you need to restart the syslog daemon to recover the space saved. Most times that's automagically done (in a perfect world). But sometimes things don't always go as planned.

/tmp is basically the new swap space. So applications that used /tmp and crashed do not clear out /tmp. They might clean up when started again, or might not. In days of old /tmp was cleared at boot time. So the microsoft methodology of rebooting could indeed fix some ills. This differs slightly from actual RAM that is automatically reclaimed when the parent application terminates.

A sometimes overlooked thing is that filesystems are a linked list of sorts. And is basically a bunch of chunks of data. If you have large chunks and small files you can waste a bit of storage potential there. Block size or nodes or whatever you want to call it these days. If you have 4k chunks and all 1k files, you can reach capacity while using just 25% of the storage.


In a crunch you can safely remove or move data from /usr/share/doc/ most of the time. Some distros give you some options to not install the -doc portion of packages. Others install everything. Not likely your issue, but if you're not using it, one source of potential real-estate when no other is available. And /usr/share/doc/ can be quite girthy if you install a decent percentage of the 20k+ potential packages these days.
 
1 members found this post helpful.
Old 08-16-2013, 04:17 AM   #6
TheGoatreich
LQ Newbie
 
Registered: Aug 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
Thanks all. I've learned a lot from this problem so it hasn't been all bad
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Free Up Disk Space prathamesh7478 Linux - Newbie 13 02-28-2013 09:02 PM
No free space on disk, but there is? Recirqie SUSE / openSUSE 4 02-22-2013 02:17 PM
partition space and disk space usage and free don't match nsic Linux - Newbie 3 02-22-2013 01:57 PM
Htop vs 'top' vs 'free' disparity schachwizard Ubuntu 2 09-22-2010 05:33 PM
Free Disk Space Not Available qwertyman19 Linux - General 5 04-24-2007 04:40 PM


All times are GMT -5. The time now is 04:59 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration