LinuxQuestions.org - Free disk space disparity

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Free disk space disparity (https://www.linuxquestions.org/questions/linux-newbie-8/free-disk-space-disparity-4175473345/)

Free disk space disparity

Hey all, I wonder if someone can help me stop banging my head against a wall with disk space issues?

I have a Red Hat Enterprise 64-bit 6.2 server that is reporting the root partition is 100% used.

Code:

[root@servername /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_servername-lv_root

                      32G  31G    0 100% /

tmpfs                7.8G    0  7.8G  0% /dev/shm

/dev/cciss/c0d0p1    485M  89M  371M  20% /boot

/dev/mapper/vg_servername-lv_home

                      18G  173M  17G  2% /home

//backupserver/backups$

                      69G  28G  41G  41% /mnt/backups

However disk usage is reporting that the root is using far less space than the 31GB df is reporting:

Code:

[root@servername /]# du -h --max-depth=1

7.7M    ./bin

14M    ./sbin

0      ./misc

2.3G    ./var

46M    ./tmp

26M    ./lib64

0      ./net

0      ./sys

4.0K    ./srv

152K    ./home

176K    ./dev

4.0K    ./media

308M    ./lib

4.6M    ./root

25M    ./etc

4.0K    ./opt

16K    ./lost+found

79M    ./boot

4.0K    ./cgroup

4.0K    ./selinux

2.6G    ./usr

du: cannot access `./proc/2643/task/2643/fd/4': No such file or directory

du: cannot access `./proc/2643/task/2643/fdinfo/4': No such file or directory

du: cannot access `./proc/2643/fd/4': No such file or directory

du: cannot access `./proc/2643/fdinfo/4': No such file or directory

0      ./proc

3.8G    ./mnt

9.1G    .

I've googled this like mad and have found a lot of people suggesting that the discrepancy is often deleted files that are still open, and say to run lsof | grep deleted to find out what process is using these files, but running that command on my server comes back with nothing.

Can anybody please help me to find what is using the extra space on my root partition?

Another thing that could use space and be hidden from du would be files hidden under an active mount point. Do this:

Code:

mkdir /tmp/rootfs

mount --bind / /tmp/rootfs

Now you can look in /tmp/rootfs/boot, /tmp/rootfs/home, and any directories in /tmp/rootfs/mnt and see if anything is hidden there. You can also run "du /tmp/rootfs" to get a picture of the root filesystem without anything hidden by other mounted filesystems. If there is stuff that you want to get rid of, just remove it. Run "umount /tmp/rootfs" when you are done.

Thanks, that was really helpful.

Running du on /tmp/rootfs/mnt I could see that 25GB of data was being held with /tmp/rootfs/mnt/backups.
Ls on that directory showed me lots of files, that have been copied to the /mnt/backups samba partition, but for whatever reason have stayed within the tmp partition and not been cleared out.

The earliest file in the /tmp/rootfs/mnt directory was 27th July 2013, so this has been working with no issues for a year, until then. Very odd.

I've cleared the data in that directory, and unmounted the /tmp/rootfs partition, and now my df -h / is looking much more healthy.

I'll keep a close eye on that /tmp directory over the next few weeks to check it's not filling up again, and if it is I'll troubleshoot why.

Thanks again for your reply, I've been frustrated with this all week.

Quote:

Originally Posted by TheGoatreich (Post 5009638)

Running du on /tmp/rootfs/mnt I could see that 25GB of data was being held with /tmp/rootfs/mnt/backups.
Ls on that directory showed me lots of files, that have been copied to the /mnt/backups samba partition, but for whatever reason have stayed within the tmp partition and not been cleared out.

What probably happened was that a backup procedure was run while no drive was mounted on /mnt/backups. All the files created by that procedure just ended up stored under that directory in the root filesystem.

FWIW, I protect myself from having that happen by putting my backups into a subdirectory "rback" on my backup drive. My backup script refuses to run if it doesn't see a directory /mnt/backups/rback, and that directory would not be present unless my backup drive were mounted.

Other possibilities include:

Data that is in / itself, since du shows the space of the sub-directories. Until the very last hit, but unless you add up all the previous hits and subtract you could miss a large chunk of data on / without looking at / directly.

When /var/log/ gets archived via logrotate and therefor cron, you need to restart the syslog daemon to recover the space saved. Most times that's automagically done (in a perfect world). But sometimes things don't always go as planned.

/tmp is basically the new swap space. So applications that used /tmp and crashed do not clear out /tmp. They might clean up when started again, or might not. In days of old /tmp was cleared at boot time. So the microsoft methodology of rebooting could indeed fix some ills. This differs slightly from actual RAM that is automatically reclaimed when the parent application terminates.

A sometimes overlooked thing is that filesystems are a linked list of sorts. And is basically a bunch of chunks of data. If you have large chunks and small files you can waste a bit of storage potential there. Block size or nodes or whatever you want to call it these days. If you have 4k chunks and all 1k files, you can reach capacity while using just 25% of the storage.

In a crunch you can safely remove or move data from /usr/share/doc/ most of the time. Some distros give you some options to not install the -doc portion of packages. Others install everything. Not likely your issue, but if you're not using it, one source of potential real-estate when no other is available. And /usr/share/doc/ can be quite girthy if you install a decent percentage of the 20k+ potential packages these days.

Thanks all. I've learned a lot from this problem so it hasn't been all bad :)