kcore question / root partition full.

dxangel · 08-26-2009, 10:09 AM

Before anyone jumps down my throat, i know Kcore is the kernel core and is simply a reference to memory and is not on the filesystem. apparently.

However -

Code:

 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     4.0G  4.0G  6.2M 100% /
/dev/cciss/c0d0p1      99M   14M   81M  14% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/cciss/c0d0p8     7.9G  263M  7.3G   4% /home
/dev/cciss/c0d0p5     7.9G   61M  7.5G   1% /opt
/dev/cciss/c0d0p10     95G   58G   32G  65% /space
/dev/cciss/c0d0p9    1012M   34M  927M   4% /tmp
/dev/cciss/c0d0p6     7.9G  1.9G  5.7G  25% /usr
/dev/cciss/c0d0p7     7.9G  165M  7.4G   3% /var

We also have nfs storage mounting to this machine that im not including here.

a closer look:

Code:

df -ha

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     4.0G  4.0G  6.2M 100% /
none                     0     0     0   -  /proc
none                     0     0     0   -  /sys
none                     0     0     0   -  /dev/pts
usbfs                    0     0     0   -  /proc/bus/usb
/dev/cciss/c0d0p1      99M   14M   81M  14% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/cciss/c0d0p8     7.9G  263M  7.3G   4% /home
/dev/cciss/c0d0p5     7.9G   61M  7.5G   1% /opt
/dev/cciss/c0d0p10     95G   58G   32G  65% /space
/dev/cciss/c0d0p9    1012M   34M  927M   4% /tmp
/dev/cciss/c0d0p6     7.9G  1.9G  5.7G  25% /usr
/dev/cciss/c0d0p7     7.9G  165M  7.4G   3% /var

Code:

du -ha /proc/kcore 4.8G    /proc/kcore

It is noticeably bigger than the root fs anyway.

Ive done many many finds, du etc etc and i cannot locate a 4GB file on the root partition ir anything that adds up to 4GB. As this machine mounts a lot ( as terabytes) of data, so random finds for files over 500M isnt appropriate.

I also went through each directory that isnt on the root partition manually and still couldnt find anything that got over 10M. Overall adding up the directories in the filesystem, minus /proc doesnt equal anywhere near 4gig, more like a couple hundred meg.

The only place where a 4gig file is referenced is in /proc. and that file isnt actually on the filesystem.

Reboot is out of the question

im stumped.

Suggestions?

centosboy · 08-26-2009, 10:16 AM

Quote:

Originally Posted by dxangel

Before anyone jumps down my throat, i know Kcore is the kernel core and is simply a reference to memory and is not on the filesystem. apparently.

However -

Code:

 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     4.0G  4.0G  6.2M 100% /
/dev/cciss/c0d0p1      99M   14M   81M  14% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/cciss/c0d0p8     7.9G  263M  7.3G   4% /home
/dev/cciss/c0d0p5     7.9G   61M  7.5G   1% /opt
/dev/cciss/c0d0p10     95G   58G   32G  65% /space
/dev/cciss/c0d0p9    1012M   34M  927M   4% /tmp
/dev/cciss/c0d0p6     7.9G  1.9G  5.7G  25% /usr
/dev/cciss/c0d0p7     7.9G  165M  7.4G   3% /var

We also have nfs storage mounting to this machine that im not including here.

a closer look:

Code:

df -ha

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     4.0G  4.0G  6.2M 100% /
none                     0     0     0   -  /proc
none                     0     0     0   -  /sys
none                     0     0     0   -  /dev/pts
usbfs                    0     0     0   -  /proc/bus/usb
/dev/cciss/c0d0p1      99M   14M   81M  14% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/cciss/c0d0p8     7.9G  263M  7.3G   4% /home
/dev/cciss/c0d0p5     7.9G   61M  7.5G   1% /opt
/dev/cciss/c0d0p10     95G   58G   32G  65% /space
/dev/cciss/c0d0p9    1012M   34M  927M   4% /tmp
/dev/cciss/c0d0p6     7.9G  1.9G  5.7G  25% /usr
/dev/cciss/c0d0p7     7.9G  165M  7.4G   3% /var

Code:

du -ha /proc/kcore 4.8G    /proc/kcore

It is noticeably bigger than the root fs anyway.

Ive done many many finds, du etc etc and i cannot locate a 4GB file on the root partition ir anything that adds up to 4GB. As this machine mounts a lot ( as terabytes) of data, so random finds for files over 500M isnt possible.

I also went through each directory that isnt on the root partition manually and still couldnt find anything that got over 10M. Overall adding up the directories in the filesystem, minus /proc doesnt equal anywhere near 4gig, more like a couple hundred meg.

The only place where a 4gig file is referenced is in /proc. and that file isnt actually on the filesystem.

Reboot is out of the question

im stumped.

Suggestions?

try this

Code:

find / -type f  -printf '%k %u  %p   \n' | perl -nae 'print if $F[0] > 50000'

This will search for any files that are bigger then 50000kb (roughly 50mb)
you can alway send the command output to a file for further analysis.

Code:

find  / -type f  -printf '%k %u  %p   \n' | perl -nae 'print if $F[0] > 50000' > out_file

centosboy · 08-26-2009, 10:21 AM

you can also run this command.

Code:

du / --max-depth 1 | sort -rn

dxangel · 08-26-2009, 10:24 AM

erm, like i said i have terabytes of data mounted to this machine, so a simple du or find will take a VERY long time to come back, if at all.

centosboy · 08-26-2009, 10:30 AM

Quote:

Originally Posted by dxangel

erm, like i said i have terabytes of data mounted to this machine, so a simple du or find will take a VERY long time to come back, if at all.

ok...start off with the /var directory as many log files are kept here

johnsfine · 08-26-2009, 10:59 AM

It takes a very careful read of this thread to even try to guess what question is being asked. I doubt most experts who might be helpful at finding the answer will guess the question.

I think the question is what is taking up all the space on
/dev/cciss/c0d0p3

The OP has listed all the subdirectories of / (including /proc) which are not located on /dev/cciss/c0d0p3. Other subdirectories of / are located on /dev/cciss/c0d0p3 and somewhere in there is some use of space that the OP doesn't understand (if I'm correctly guessing the question).

Edit:

Quote:

Originally Posted by centosboy

start off with the /var directory

Some content was lost when GrapefruiTgirl removed the flames, so please notice that /var is one of the directories listed by the OP as not being physically included in the filesystem that has the unexplained use of space.

dxangel · 08-26-2009, 11:05 AM

Quote:

Originally Posted by johnsfine

It takes a very careful read of this thread to even try to guess what question is being asked. I doubt most experts who might be helpful at finding the answer will guess the question.

I think the question is what is taking up all the space on
/dev/cciss/c0d0p3

The OP has listed all the subdirectories of / (including /proc) which are not located on /dev/cciss/c0d0p3. Other subdirectories of / are located on /dev/cciss/c0d0p3 and somewhere in there is some use of space that the OP doesn't understand (if I'm correctly guessing the question).

That is correct, yes. However everything that is on /dev/cciss/c0d0p3 does not add up to 4GB.

johnsfine · 08-26-2009, 11:16 AM

Quote:

Originally Posted by dxangel

However everything that is on /dev/cciss/c0d0p3 does not add up to 4GB.

I think df is telling you that what is actually on /dev/cciss/c0d0p3 does add up to 4GB and that is excluding anything (including /proc and /var etc.) that is mounted as subdirectories in or below /

I assume there is some tool that will go through every subdirectory from a specific path (initially /) and for each recursively add up all the used space below that subdirectory and will skip all soft links, all hard links and all directory entries to which other file systems have been mounted (in other words skip everything that is really somewhere else).

I don't happen to know what tool that is, but if you asked more nicely, I expect one of the experts would tell you. With such a tool, you ought to be able to figure out where that 4GB is hiding.

But I also don't know how you look for more obscure uses of disk space. For example, I think that if you open a file and then delete it, it will no longer appear in the directory it was in but it will still exist on disk, taking up space, until you close it. Experts may know how you find out whether a lot of space is tied up that way and/or know what other ways there are to tie up disk space without showing up in directories.

GrapefruiTgirl · 08-26-2009, 11:16 AM

Quote:

Originally Posted by dxangel

I now feel like my original post has been swamped by irrelevant replies :^( can this be deleted and i'll repost?

Please do not post the issue again elsewhere. Give me a few moments to tidy up the thread

UPDATE: erronious posts hidden at OP's request. Let's all have a breather now, and continue where we left off..

Sasha

dxangel · 08-26-2009, 11:23 AM

Quote:

Originally Posted by johnsfine

I think df is telling you that what is actually on /dev/cciss/c0d0p3 does add up to 4GB and that is excluding anything (including /proc and /var etc.) that is mounted as subdirectories in or below /

I assume there is some tool that will go through every subdirectory from a specific path (initially /) and for each recursively add up all the used space below that subdirectory and will skip all soft links, all hard links and all directory entries to which other file systems have been mounted (in other words skip everything that is really somewhere else).

I don't happen to know what tool that is, but if you asked more nicely, I expect one of the experts would tell you. With such a tool, you ought to be able to figure out where that 4GB is hiding.

But I also don't know how you look for more obscure uses of disk space. For example, I think that if you open a file and then delete it, it will no longer appear in the directory it was in but it will still exist on disk, taking up space, until you close it. Experts may know how you find out whether a lot of space is tied up that way and/or know what other ways there are to tie up disk space without showing up in directories.

Yes, I have examined the directories on /dev/cciss/c0d0p3 using du -h which examines and adds up all space used in the manner that you describe.
Ive also examined the system using lsof.

dxangel · 08-26-2009, 11:23 AM

thank you moderator!

dxangel · 08-26-2009, 12:01 PM

Im going to sum up this issue, in case my original post is too hard to understand.

the root partition is 6meg away from filling up.
the machine is partitioned out so things like /var /tmp are not on the same filesystem as / ( see OP )
the device is /dev/cciss/c0d0p3
du -h on the directories on c0d0p3
boot/ bin/ initrd/ media/ mnt/ lib
lib64/ misc/ root/ srv/ etc/ mnt/ sbin/

show no more than a couple of hundred meg used.

lsof shows nothing is writing to the device

There is also a kcore file of 4.8GB

Reboots are out of the question.

Im not going to sit here and beg for crumbs from 'experts' - i am seriously interested in what people might think it is. I am still working on the issue myself. IMHO i think this may be an issue with a process taking out a R/W file descriptor, then unlinking it.

Any other suggestions would be most welcome.

sarin · 08-26-2009, 12:35 PM

I don't know the real solution to your problem. But, as far as I can see, you are not running a find on it since it has lot of other file systems mounted on it. Why not run find with -xdev (-mount) option? Du has a similar option (-x). Hope this helps.

johnsfine · 08-26-2009, 01:04 PM

Quote:

Originally Posted by dxangel

lsof shows nothing is writing to the device

Sorry if these are stupid questions, but:

1) Did you use lsof as root? Much of the info is missing if you aren't root.

2) What does "writing" matter. Even opening a file for read can keep tie up the disk space of that file even if something else deletes the file. You want to know if large files or large numbers of files are open that would be physically located in that filesystem

dxangel · 08-26-2009, 01:33 PM

Yes i am running as root.

I mentioned that there are no files being written to / to make it clear the space taken up is static.