Difference in sizes of df & du
HI,
There is a scenario where there is a difference in sizes of df & du & where I had followed the steps as, 1. Checked lsof where there was deleted files 2. Executed, lsof | grep "deleted" | awk '{print $2}' | xargs kill -9 to kill the deleted processes in lsof. but I wanted to know what will be the correct approch as a system admin to handle these type of issues in production environment where other application might also be linked with the processes of deleted files in lsof without reboot. Any solution will be appreciated. |
It highly depends on what you're doing.
There's a list of open file descriptors for a process located in /proc/<PID>/fd. "ls -l" will tell you more information about the file descriptor (e.g. which file descriptor is linked to the deleted file). You could then truncate that file to clear the file and free up some space. For example, let's say your application only has one file open (other than the default stdin, stderr, and stdout). Normally your deleted open file would be open with file descriptor 3. So you would truncate that file (pretend your PID is 12345) Code:
cd /proc/12345/fd As far as I know there's no good way to reestablish the hard link of the deleted file. If you need to recover the data from the deleted log you could tail -f the file descriptor Code:
tail -c +0 -f /proc/12345/fd/3 > /path/filename There is a github project which attempts to resolve the "restore a deleted hard link" problem called fdlink however I have not used that kernel module myself. I'd just recommend using the tail -f command and clearing the file descriptor every once in a while (because the file descriptor and the file will essentially be doubling the data produced on the HDD). You can read more about the proc filesystem in the kernel documentation proc.txt. See this behavior in action You can play with this behavior using a python interactive prompt. Code:
$ mkdir ~/sandbox Code:
f=open("test.log","a") Code:
$ cat test.log Code:
f=open("test.log","a") Code:
$ rm test.log Code:
f.close() Code:
$ cat ~/sandbox/test.log SAM |
Basically you'd use lsof & fuser to check what delete files are still in use, then look at those apps and decide what to do.
This is part of being a SysAdmin, there's no exact procedure, you have to use your initiative. If you're not sure about an app, find out who who does know and talk to them. Make notes for future ref. Normally an app shouldn't take up so much space that you have to resort to this anyway. Only do this as a last resort... |
Thanks for the info but how can I find the correct file descriptor no under /proc/12345/fd as there are number of numerical files in it.
|
I have SuSE 11 and after removing test.log I got another filename: (3 -> <dir>/.nfs0000000000f49c800002022e), and also it was not on the list of the deleted files (lsof | grep deleted). I assume those files are not really deleted, they are still in use. If you want you can use du on them or grep or whatever to identify anyone of them. Modifying those files by something like echo "" > ./3 or similar may cause problems (improper functionality?), I would rather kill that app.
|
Quote:
Code:
cd /proc/12345/fd Quote:
Here's what I see on my Ubuntu system. Code:
sam@farcry:~/sandbox/deleted_hardlink$ lsof | grep deleted | grep test.log Code:
$ head -n1 /etc/issue |
Quote:
SUSE Linux Enterprise Desktop 11 SP1 (x86_64) I think the reason is that my home is located on an nfs drive... |
All times are GMT -5. The time now is 07:16 AM. |