LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 04-08-2013, 12:30 AM   #1
smilemukul
Member
 
Registered: Jun 2009
Distribution: Redhat,CentOS,Ubuntu,Puppet
Posts: 292

Rep: Reputation: 34
Difference in sizes of df & du


HI,

There is a scenario where there is a difference in sizes of df & du & where I had followed the steps as,

1. Checked lsof where there was deleted files
2. Executed, lsof | grep "deleted" | awk '{print $2}' | xargs kill -9 to kill the deleted processes in lsof.

but I wanted to know what will be the correct approch as a system admin to handle these type of issues in production environment where other application might also be linked with the processes of deleted files in lsof without reboot.

Any solution will be appreciated.
 
Old 04-08-2013, 02:10 AM   #2
sag47
Senior Member
 
Registered: Sep 2009
Location: Philly, PA
Distribution: Kubuntu x64, RHEL, Fedora Core, FreeBSD, Windows x64
Posts: 1,505
Blog Entries: 35

Rep: Reputation: 383Reputation: 383Reputation: 383Reputation: 383
It highly depends on what you're doing.

There's a list of open file descriptors for a process located in /proc/<PID>/fd. "ls -l" will tell you more information about the file descriptor (e.g. which file descriptor is linked to the deleted file). You could then truncate that file to clear the file and free up some space. For example, let's say your application only has one file open (other than the default stdin, stderr, and stdout). Normally your deleted open file would be open with file descriptor 3. So you would truncate that file (pretend your PID is 12345)
Code:
cd /proc/12345/fd
echo -n '' > ./3
Remember, before truncating any file descriptor associated with a process be sure that you're operating on the right one with "ls -l"!

As far as I know there's no good way to reestablish the hard link of the deleted file. If you need to recover the data from the deleted log you could tail -f the file descriptor
Code:
tail -c +0 -f /proc/12345/fd/3 > /path/filename
It will also get all subsequent writes to that file since the -f option is set for tail.

There is a github project which attempts to resolve the "restore a deleted hard link" problem called fdlink however I have not used that kernel module myself. I'd just recommend using the tail -f command and clearing the file descriptor every once in a while (because the file descriptor and the file will essentially be doubling the data produced on the HDD).

You can read more about the proc filesystem in the kernel documentation proc.txt.

See this behavior in action

You can play with this behavior using a python interactive prompt.
Code:
$ mkdir ~/sandbox
$ cd ~/sandbox
$ python
In interactive python mode,
Code:
f=open("test.log","a")
f.write("my log test 1\n")
f.close()
In bash, while python interactive check the contents of the log file.
Code:
$ cat test.log
my log test 1
$
Open the file again in python interactive,
Code:
f=open("test.log","a")
f.write("my log test 2\n")
In bash, delete test.log while python has it open. Check it out to see that it was deleted. Check the python process ID and then visit the open file descriptor (pretend lsof reported a PID of 12345).
Code:
$ rm test.log
$ lsof | grep deleted | grep test.log
$ cd /proc/12345/fd
$ ls -l
$ cat 3
my log test 1
$ echo -n '' > ./3
$ cat 3
$ tail -c +0 -f ./3 > ~/sandbox/test.log
Now in the python interactive prompt your last f.write() command hasn't been committed to disk. This will happen when you close the file of which the previous tail command will catch. In python interactive,
Code:
f.close()
You can now stop the tail -f command since the file has been closed for writing. Go ahead and cat your new test.log file. You should see the last written lines.
Code:
$ cat ~/sandbox/test.log
my log test 2
$
Kind of a neat little experiment.

SAM

Last edited by sag47; 04-08-2013 at 02:43 AM.
 
1 members found this post helpful.
Old 04-08-2013, 02:13 AM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Basically you'd use lsof & fuser to check what delete files are still in use, then look at those apps and decide what to do.
This is part of being a SysAdmin, there's no exact procedure, you have to use your initiative.
If you're not sure about an app, find out who who does know and talk to them.
Make notes for future ref.
Normally an app shouldn't take up so much space that you have to resort to this anyway.
Only do this as a last resort...
 
Old 04-08-2013, 05:39 AM   #4
smilemukul
Member
 
Registered: Jun 2009
Distribution: Redhat,CentOS,Ubuntu,Puppet
Posts: 292

Original Poster
Rep: Reputation: 34
Thanks for the info but how can I find the correct file descriptor no under /proc/12345/fd as there are number of numerical files in it.
 
Old 04-08-2013, 06:07 AM   #5
pan64
Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 5,146

Rep: Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364
I have SuSE 11 and after removing test.log I got another filename: (3 -> <dir>/.nfs0000000000f49c800002022e), and also it was not on the list of the deleted files (lsof | grep deleted). I assume those files are not really deleted, they are still in use. If you want you can use du on them or grep or whatever to identify anyone of them. Modifying those files by something like echo "" > ./3 or similar may cause problems (improper functionality?), I would rather kill that app.
 
Old 04-08-2013, 08:29 AM   #6
sag47
Senior Member
 
Registered: Sep 2009
Location: Philly, PA
Distribution: Kubuntu x64, RHEL, Fedora Core, FreeBSD, Windows x64
Posts: 1,505
Blog Entries: 35

Rep: Reputation: 383Reputation: 383Reputation: 383Reputation: 383
Quote:
Originally Posted by smilemukul View Post
Thanks for the info but how can I find the correct file descriptor no under /proc/12345/fd as there are number of numerical files in it.
Code:
cd /proc/12345/fd
ls -l
The long format listing (-l or minus ell) will present a list of links, one of them being to the deleted file which has been opened for writing.

Quote:
Originally Posted by pan64 View Post
I have SuSE 11 and after removing test.log I got another filename: (3 -> <dir>/.nfs0000000000f49c800002022e), and also it was not on the list of the deleted files (lsof | grep deleted). I assume those files are not really deleted, they are still in use. If you want you can use du on them or grep or whatever to identify anyone of them. Modifying those files by something like echo "" > ./3 or similar may cause problems (improper functionality?), I would rather kill that app.
Interesting, I've not experienced that behavior myself. I've only tested this on RHEL, Fedora, and Ubuntu. I'm not doubting your report of the strange behavior; just that I haven't personally seen it. What kernel are you using?

Here's what I see on my Ubuntu system.
Code:
sam@farcry:~/sandbox/deleted_hardlink$ lsof | grep deleted | grep test.log
python    25555        sam    3w      REG               0,21        12   16121975 /home/sam/sandbox/deleted_hardlink/test.log (deleted)
sam@farcry:~/sandbox/deleted_hardlink$ cd /proc/25555/fd
sam@farcry:/proc/25555/fd$ ls
0  1  2  3
sam@farcry:/proc/25555/fd$ ls -l
total 0
lrwx------ 1 sam sam 64 Apr  8 08:34 0 -> /dev/pts/9
lrwx------ 1 sam sam 64 Apr  8 08:34 1 -> /dev/pts/9
lrwx------ 1 sam sam 64 Apr  8 08:34 2 -> /dev/pts/9
l-wx------ 1 sam sam 64 Apr  8 08:34 3 -> /home/sam/sandbox/deleted_hardlink/test.log (deleted)
sam@farcry:/proc/25555/fd$
Here's systems and kernel versions I've tested the behavior with.
Code:
$ head -n1 /etc/issue
Red Hat Enterprise Linux Server release 6.1 (Santiago)
$ uname -rm
2.6.32-71.el6.x86_64 x86_64
$ python --version
Python 2.6.6

$ head -n1 /etc/issue
Ubuntu 12.04.2 LTS \n \l
$ uname -rm
3.2.0-38-generic x86_64
$ python --version
Python 2.7.3

$ cat /etc/issue
Fedora release 16 (Verne)
$ uname -rm
3.6.11-4.fc16.x86_64 x86_64
$ python --version
Python 2.7.3
I got the behavior I originally described in post #2 on all of those systems.

Last edited by sag47; 04-08-2013 at 08:45 AM.
 
Old 04-08-2013, 09:04 AM   #7
pan64
Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 5,146

Rep: Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364
Quote:
Originally Posted by sag47 View Post
What kernel are you using?
Linux hostname 2.6.32.45-0.3-xen #1 SMP 2011-08-22 10:12:58 +0200 x86_64 x86_64 x86_64 GNU/Linux

SUSE Linux Enterprise Desktop 11 SP1 (x86_64)

I think the reason is that my home is located on an nfs drive...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Why are fdisk & df reporting different sizes for encrypted disk? ItsTheSource Linux - Software 4 10-21-2012 11:03 PM
Partition sizes & reinstall of suse10.3 Ratty_cat Suse/Novell 5 11-01-2008 09:46 PM
define different physical sizes on Multiple Monitors of different sizes MasterC Linux - Desktop 2 03-18-2008 05:24 PM
Difference between Tomcat 1.5&1.6 and Apache 2.1 & 2.2 chnlinux Linux - Newbie 1 12-04-2007 10:44 PM
What's the difference between su - & su ? Thaidog Linux - Newbie 7 07-28-2005 04:58 PM


All times are GMT -5. The time now is 11:04 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration