[SOLVED] du and fd commands give contradictory outputs

sburnay · 12-04-2015, 10:55 AM

Hey!

I have an Ubuntu Server 12.04 LTS (64bit version),

My filesystem is organized in partitions and one of them is used for logging purposes (/dev/sdb mounted on /var/log).

Somehow when I run df I get information that my log partition is at 100 of its 7GB capacity and by running du I find it is only using a few hundred kB, it makes no sense.

Even when I'm just listing the /var/log content with 'ls -lh' I do not find many files:

Code:

root@MYSERVER:/# sudo du --max-depth=1 -x -h /var/log/
4.0K    /var/log/news
12K     /var/log/mysql
76K     /var/log/apache2
16K     /var/log/lost+found
4.0K    /var/log/sysstat
4.0K    /var/log/apt
4.0K    /var/log/landscape
816K    /var/log/
root@MYSERVER:/# df -h | grep log
Filesystem                        Size  Used Avail Use Mounted on
/dev/sdb                          7.0G  6.7G   24M 100% /var/log

Thank you in advance for your help

smallpond · 12-04-2015, 11:51 AM

du only lists files which are in directories. Anonymous files are not listed. For example if a program has a file open which has been unlinked from any directory then it will not be counted.

sburnay · 12-04-2015, 03:37 PM

Quote:

Originally Posted by smallpond

du only lists files which are in directories. Anonymous files are not listed. For example if a program has a file open which has been unlinked from any directory then it will not be counted.

But I also tried 'ls -lh' and didn't find any big files nor a large collection of smaller files.

I really don't know what is taking so much space in my /var/log partition and I'm afraid it might affect the server's performance.

I even tried a reboot to see weather it would make a difference, but it remained the same

rknichols · 12-04-2015, 05:22 PM

I notice you used the "-x" (--one-file-system) option in your du command. Is there actually anything mounted anywhere under /var/log ? That seems unusual, bit if so, you could have files hidden under that active mount point. If not, then "fsck -f /dev/sdb" would be in order.

sburnay · 12-07-2015, 04:49 AM

Quote:

Originally Posted by rknichols

I notice you used the "-x" (--one-file-system) option in your du command. Is there actually anything mounted anywhere under /var/log ? That seems unusual, bit if so, you could have files hidden under that active mount point. If not, then "fsck -f /dev/sdb" would be in order.

Hey!

Thanks for your advice, but I have no mount points within /var/log.

How can I check for hidden files?

I've tried "fsck -f /dev/sdb" and it identified/corrected a few issues:

several "Free blocks count wrong (98036, counted=1769110)"
Orphaned inodes

The result from fsck:

Code:

/dev/sdb: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb: 44/458752 files (11.4 non-contiguous), 65898/1835008 blocks

The partition remains accusing 100

Yet, after another reboot, it went to 4% used which is the correct value

chrism01 · 12-07-2015, 06:50 PM

Quote:

Yet, after another reboot, it went to 4 used which is the correct value

This is because when a file is 'deleted' (aka unlinked), the entry disappears from the inode (dir), but the storage is only returned to the system when ALL processes that have it open have relinquished the file. If any process has the file open, it will continue to take up space.

df 'sees' the file space usage as still there, du doesn't.

Its a very common qn/soln and you'll see a lot of hits on google for it.
You can easily test it yourself by writing a small prog that writes to a test file and the 'delete' the file manually from another terminal.

MadeInGermany · 12-08-2015, 12:06 AM

Code:

lsof /var/log

shows the processes that have files opened there - in some cases even shows the "deleted" files.