LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-29-2010, 08:01 AM   #1
Sayan Acharjee
Member
 
Registered: Feb 2010
Location: Chennai, India
Distribution: Manjaro
Posts: 616

Rep: Reputation: 64
Different results in du and df?


Hi,
I was checking /opt and then found something unusual. Below are the results of du and df -h,

Code:
$sudo du -h /opt/|tail -3
1.1G    /opt/IBM
16K     /opt/lost+found
1.1G    /opt/
and

Code:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_root
                      2.0G  614M  1.3G  33% /
/dev/mapper/rootvg-lv_var
                      2.0G  112M  1.8G   6% /var
/dev/mapper/rootvg-lv_opt
                       16G   16G     0 100% /opt
/dev/mapper/rootvg-lv_home
                      4.9G  582M  4.1G  13% /home
/dev/mapper/rootvg-lv_usr
                      7.8G  2.0G  5.4G  28% /usr
/dev/sda1              99M   12M   82M  13% /boot
tmpfs                1014M     0 1014M   0% /dev/shm

While du shows that the /opt's size is only 1.1.G, df -h showing its 16G space is 100% utilized.

Any idea what is going on here? :O
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 10-29-2010, 09:09 AM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,905
Blog Entries: 14

Rep: Reputation: 1142Reputation: 1142Reputation: 1142Reputation: 1142Reputation: 1142Reputation: 1142Reputation: 1142Reputation: 1142Reputation: 1142
This looks like the infamous "open file deletion" issue.

When you delete a file that is being held "open" by a process what actually happens is the file's name is deleted but not its inode (and its data). df sees what is happening at filesystem level whereas du sees what is happening at file/directory level. If the filename is gone du doesn't see it in the directory any longer. However since the inode is still in use df sees that the filesystem is still using the space.

You can try to run lsof against the filesystem and look for any reg file that has a large size but no name. This will show you the process that has it "open". If you stop then restart the process it will clear the inode. Failing that you can do a reboot which will stop all processes and should definitely clear any such open inodes.

By the way doing a "mv" of a file doesn't help because it keeps the same inode when moved within the filesystem. A "mv" to another filesystem would result in the same situation as the deletion did because such a "mv" actually copies then deletes as the inode is unique to the filesystem it is on.

The most common cause of this is deleting a log file to clear up space. Many log files are held "open" by their applications so deletion should only be done after the application is shutdown. You might want to look at logrotate (man logrotate) to setup automatic aging of logfiles if you have one that is consistently filling up your filesystem.

Last edited by MensaWater; 10-29-2010 at 09:11 AM.
 
2 members found this post helpful.
Old 10-30-2010, 10:51 AM   #3
Sayan Acharjee
Member
 
Registered: Feb 2010
Location: Chennai, India
Distribution: Manjaro
Posts: 616

Original Poster
Rep: Reputation: 64
Quote:
Originally Posted by MensaWater View Post
This looks like the infamous "open file deletion" issue.

When you delete a file that is being held "open" by a process what actually happens is the file's name is deleted but not its inode (and its data). df sees what is happening at filesystem level whereas du sees what is happening at file/directory level. If the filename is gone du doesn't see it in the directory any longer. However since the inode is still in use df sees that the filesystem is still using the space.

You can try to run lsof against the filesystem and look for any reg file that has a large size but no name. This will show you the process that has it "open". If you stop then restart the process it will clear the inode. Failing that you can do a reboot which will stop all processes and should definitely clear any such open inodes.

By the way doing a "mv" of a file doesn't help because it keeps the same inode when moved within the filesystem. A "mv" to another filesystem would result in the same situation as the deletion did because such a "mv" actually copies then deletes as the inode is unique to the filesystem it is on.

The most common cause of this is deleting a log file to clear up space. Many log files are held "open" by their applications so deletion should only be done after the application is shutdown. You might want to look at logrotate (man logrotate) to setup automatic aging of logfiles if you have one that is consistently filling up your filesystem.
Hi, I found out the problem and its exactly the same case, one file was deleted while it was used by another process. lsof confirmed that.
Thanks for that simple explanation.
 
Old 02-29-2012, 08:53 PM   #4
nndhanasekaran
LQ Newbie
 
Registered: Feb 2012
Posts: 1

Rep: Reputation: Disabled
example try : /usr/sbin/lsof | grep var

for example, /var mountpoint is showing different sizes try following command and restart the process, here mysql

/usr/sbin/lsof | grep var
...
...
mysqld 6184 mysql 1w REG 253,1 1841495 65594 /var/log/mysql/mysqld.log.1 (deleted)
mysqld 6184 mysql 2w REG 253,1 1841495 65594 /var/log/mysql/mysqld.log.1 (deleted)
mysqld 6184 mysql 4w REG 253,1 3139487983 65575 /var/log/mysql/mysql-slow.log.1 (deleted)
...
...
...

Last edited by nndhanasekaran; 02-29-2012 at 08:55 PM. Reason: for more info
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
ChkRootkit Results? mexbeachbum Linux - Security 3 07-14-2007 01:52 PM
Help with DD and emailing the results booyaadewd Linux - Newbie 6 09-22-2006 03:48 PM
different results with df / fdisk -l ? sneutrinono Linux - Hardware 1 11-12-2004 09:48 AM
Same C code different results Veteq Programming 17 07-24-2004 03:26 AM
nmap results djcomplex Linux - Software 3 03-20-2004 01:46 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 12:18 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration