[SOLVED] Different results in du and df?

Sayan Acharjee · 10-29-2010, 09:01 AM

Hi,
I was checking /opt and then found something unusual. Below are the results of du and df -h,

Code:

$sudo du -h /opt/|tail -3
1.1G    /opt/IBM
16K     /opt/lost+found
1.1G    /opt/

and

Code:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_root
                      2.0G  614M  1.3G  33% /
/dev/mapper/rootvg-lv_var
                      2.0G  112M  1.8G   6% /var
/dev/mapper/rootvg-lv_opt
                       16G   16G     0 100% /opt
/dev/mapper/rootvg-lv_home
                      4.9G  582M  4.1G  13% /home
/dev/mapper/rootvg-lv_usr
                      7.8G  2.0G  5.4G  28% /usr
/dev/sda1              99M   12M   82M  13% /boot
tmpfs                1014M     0 1014M   0% /dev/shm

While du shows that the /opt's size is only 1.1.G, df -h showing its 16G space is 100% utilized.

Any idea what is going on here? :O

MensaWater · 10-29-2010, 10:09 AM

This looks like the infamous "open file deletion" issue.

When you delete a file that is being held "open" by a process what actually happens is the file's name is deleted but not its inode (and its data). df sees what is happening at filesystem level whereas du sees what is happening at file/directory level. If the filename is gone du doesn't see it in the directory any longer. However since the inode is still in use df sees that the filesystem is still using the space.

You can try to run lsof against the filesystem and look for any reg file that has a large size but no name. This will show you the process that has it "open". If you stop then restart the process it will clear the inode. Failing that you can do a reboot which will stop all processes and should definitely clear any such open inodes.

By the way doing a "mv" of a file doesn't help because it keeps the same inode when moved within the filesystem. A "mv" to another filesystem would result in the same situation as the deletion did because such a "mv" actually copies then deletes as the inode is unique to the filesystem it is on.

The most common cause of this is deleting a log file to clear up space. Many log files are held "open" by their applications so deletion should only be done after the application is shutdown. You might want to look at logrotate (man logrotate) to setup automatic aging of logfiles if you have one that is consistently filling up your filesystem.

Sayan Acharjee · 10-30-2010, 11:51 AM

Quote:

Originally Posted by MensaWater

This looks like the infamous "open file deletion" issue.

When you delete a file that is being held "open" by a process what actually happens is the file's name is deleted but not its inode (and its data). df sees what is happening at filesystem level whereas du sees what is happening at file/directory level. If the filename is gone du doesn't see it in the directory any longer. However since the inode is still in use df sees that the filesystem is still using the space.

You can try to run lsof against the filesystem and look for any reg file that has a large size but no name. This will show you the process that has it "open". If you stop then restart the process it will clear the inode. Failing that you can do a reboot which will stop all processes and should definitely clear any such open inodes.

By the way doing a "mv" of a file doesn't help because it keeps the same inode when moved within the filesystem. A "mv" to another filesystem would result in the same situation as the deletion did because such a "mv" actually copies then deletes as the inode is unique to the filesystem it is on.

The most common cause of this is deleting a log file to clear up space. Many log files are held "open" by their applications so deletion should only be done after the application is shutdown. You might want to look at logrotate (man logrotate) to setup automatic aging of logfiles if you have one that is consistently filling up your filesystem.

Hi, I found out the problem and its exactly the same case, one file was deleted while it was used by another process. lsof confirmed that.
Thanks for that simple explanation.

nndhanasekaran · 02-29-2012, 09:53 PM

for example, /var mountpoint is showing different sizes try following command and restart the process, here mysql

/usr/sbin/lsof | grep var
...
...
mysqld 6184 mysql 1w REG 253,1 1841495 65594 /var/log/mysql/mysqld.log.1 (deleted)
mysqld 6184 mysql 2w REG 253,1 1841495 65594 /var/log/mysql/mysqld.log.1 (deleted)
mysqld 6184 mysql 4w REG 253,1 3139487983 65575 /var/log/mysql/mysql-slow.log.1 (deleted)
...
...
...