[SOLVED] huge difference between du and df

BCarey · 09-20-2016, 12:55 PM

Hi all,

I had a system (slack64 14.1) crash this morning and I couldn't start X on reboot. Turns out my / filesystem was full, as reported by df. So, trying to clear things up, I used du to find out where the bloat was. Unfortunately, du seemed to find a lot less used space than du reported.

To be more specific, "du -xsh " showed 22G used while "df -h" showed 28G used. Since the filesystem was on lvm I was able to increase the volume size to solve my immediate problem, but the discrepancy between du and df still exists. I have run fsck on the root filesystem, which reports all okay. I have rebooted to ensure that no running processes could be still claiming deleted files. Never-the-less I still have this non-trivial discrepancy. Any ideas about how to research further?

Thanks,
Brian

Richard Cranium · 09-20-2016, 01:05 PM

I assume that du does not account for the metadata associated with a file whereas df does.

BCarey · 09-20-2016, 01:09 PM

Quote:

Originally Posted by Richard Cranium

I assume that du does not account for the metadata associated with a file whereas df does.

Could that really account for a 6G (21-27%) difference?

smallpond · 09-20-2016, 01:22 PM

du reports allocated disk usage. df reports filesystem space available to a non-root user. df does not include reserved space as available.

phenixia2003 · 09-20-2016, 01:31 PM

Hello,

By default, 5% of disk space is reserved. 28-22=6 => 5%(X)=6G => X=120G

--
SeB

BCarey · 09-20-2016, 01:33 PM

Quote:

Originally Posted by phenixia2003

Hello,

By default, 5% of disk space is reserved. 28-22=6 => 5%(X)=6G => X=120G

--
SeB

But this partition was only 30G, not 120G, so 5% would only be 1.5G. Since I expanded the file system to 50G, I'm getting identical results from df and du. Reserved space would still only account for 2.5G, which still leaves a huge amount "unaccounted" for.

rknichols · 09-20-2016, 01:49 PM

That difference is almost always due to a huge deleted file still held open by some process, often a big log file. You can run (as root)

Code:

lsof | grep '(deleted)' | less

and scroll down looking for something huge.

If you have rebooted and still see the problem, then "fsck -f" on the filesystem is warranted.

And, now that I see you've already done that, about the only remaining possibility is files hidden under an active mount point. Try this

Code:

mkdir /tmp/tmpmnt
mount --bind / /tmp/tmpmnt
du /tmp/tmpmnt

phenixia2003 · 09-20-2016, 02:03 PM

Quote:

Originally Posted by BCarey

But this partition was only 30G, not 120G, so 5% would only be 1.5G. Since I expanded the file system to 50G, I'm getting identical results from df and du. Reserved space would still only account for 2.5G, which still leaves a huge amount "unaccounted" for.

I've had an issue like that, but without crash, and without lvm. I've found there was huge files in hidden folders and large files (~20M) in folder like /var/tmp/kdecache-seb with commands below :

Code:

$ du -h ~ | grep -E "^[[:digit:]]+[G]"
$ du -h /var/tmp | grep -E "^[[:digit:]]+[G]"
$ du -h /tmp | grep -E "^[[:digit:]]+[G]"

$ du -h ~ | grep -E "^[[:digit:]]{2}[M]"
$ du -h /var/tmp | grep -E "^[[:digit:]]{2}[M]"
$ du -h /tmp | grep -E "^[[:digit:]]{2}[M]"

--
SeB

BCarey · 09-20-2016, 02:05 PM

Quote:

Originally Posted by rknichols

And, now that I see you've already done that, about the only remaining possibility is files hidden under an active mount point. Try this

Code:

mkdir /tmp/tmpmnt
mount --bind / /tmp/tmpmnt
du /tmp/tmpmnt

This looks promising. When doing this du reported the 28G. So I will investigate that. Thank you very much.

Brian

BCarey · 09-21-2016, 10:37 AM

So indeed there were 6 Gigs of files hiding under one of the mounts. Thank you rknichols for pointing to the solution. Thank you also phenixia for your efforts on my behalf. Now both df and du report 22G used, not even a discrepancy d/t reserved space.

Brian

saurabhve · 08-23-2019, 12:10 AM

Quote:

Originally Posted by rknichols

That difference is almost always due to a huge deleted file still held open by some process, often a big log file. You can run (as root)

Code:

lsof | grep '(deleted)' | less

and scroll down looking for something huge.

If you have rebooted and still see the problem, then "fsck -f" on the filesystem is warranted.

And, now that I see you've already done that, about the only remaining possibility is files hidden under an active mount point. Try this

Code:

mkdir /tmp/tmpmnt
mount --bind / /tmp/tmpmnt
du /tmp/tmpmnt

binding the mount showed up that there was a local directory which had some used space and at some point of time this directory was mounted to some different disk partition, thus leaving behind the old data in the previous partition.

Richard Cranium · 08-23-2019, 01:41 AM

"Better late than never" isn't always true.