[SOLVED] Partition full (100%) even if sum of all files is less than 50%

lpallard · 02-24-2015, 05:53 PM

Hi,

So I have an encrypted /home partition (created with LVM and encryption during Centos 6.5 install) that ran out of space all of a sudden (or it was close to 100% for weeks and I never noticed...) but the problem is that the sum of all files and folders is less than 50% of the partition's capacity.

df -h

Code:

[root@workstation workstation-user]# df -H
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
                       53G  9.8G   41G  20% /
tmpfs                 4.2G  2.6M  4.2G   1% /dev/shm
/dev/sda1             500M   62M  412M  14% /boot
/dev/mapper/vg_workstation-lv_home
                      923G  876G   98M 100% /home

If I do a du -h --max-depth=1 on /home I get:

Code:

[root@workstation home]# du -h --max-depth=1 .
16K	./lost+found
816G	./workstation-user
816G	.

Then finally if I go in "workstation-user" and issue the same command:

Code:

[root@workstation workstation-user]# du -h --max-depth=1 .
4.0K	./.vnc
96K	./.gnome2
4.0K	./.themes
12K	./.dvdcss
168K	./.pulse
4.0K	./.gvfs
18M	./.eclipse
371G	./file-storage
16K	./.gnupg
36K	./.macromedia
8.0K	./.redhat
668K	./.gstreamer-0.10
68K	./.java
23M	./.mozilla
6.0G	./.local
980K	./.kde
16K	./.mplayer
16K	./.gnote
8.0K	./.cups
20K	./Desktop
4.0K	./.qt
12K	./.dbus
36K	./.sane
36K	./.pki
580K	./.adobe
70M	./.cache
68K	./.fontconfig
28K	./.designer
4.0K	./.nautilus
12K	./.abrt
5.8M	./.config
4.0K	./.gnome2_private
136K	./.nv
4.0K	./.icons
20G	./to-sort
8.0K	./.ssh
8.0K	./.emacs.d
852K	./.gconf
80K	./.gconfd
524K	./.gimp-2.6
152K	./.blender
816G	.

As you can see, under /home the "biggest" folder is "file-storage" and it is 371GB. There's still 552GB to go until the capacity of 923GB is reached... All other folders and files do not account to the missing 552GB (not even close), so whats going on?

The only thing I can think of, is the encryption... Would it be possible that somehow every encrypted GB takes another GB for the encryption? Even that, roughly adding everything "du" lists and doing X2 will yield to 796GB which is still far from 923GB..

Any insight appreciated!..
Thanks

jailbait · 02-24-2015, 07:25 PM

I think that whatever partition contains /home contains a lot more than just /home.

--------------------
Steve Stites

Daws · 02-24-2015, 07:48 PM

It could be open unlinked files. You can find them with

Code:

lsof | grep deleted

They normally only appear in tmp directories though.

replica9000 · 02-24-2015, 11:41 PM

Quote:

Originally Posted by lpallard

Hi,

So I have an encrypted /home partition (created with LVM and encryption during Centos 6.5 install) that ran out of space all of a sudden (or it was close to 100% for weeks and I never noticed...) but the problem is that the sum of all files and folders is less than 50% of the partition's capacity.

df -h

Code:

[root@workstation workstation-user]# df -H
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
                       53G  9.8G   41G  20% /
tmpfs                 4.2G  2.6M  4.2G   1% /dev/shm
/dev/sda1             500M   62M  412M  14% /boot
/dev/mapper/vg_workstation-lv_home
                      923G  876G   98M 100% /home

If I do a du -h --max-depth=1 on /home I get:

Code:

[root@workstation home]# du -h --max-depth=1 .
16K	./lost+found
816G	./workstation-user
816G	.

Then finally if I go in "workstation-user" and issue the same command:

Code:

[root@workstation workstation-user]# du -h --max-depth=1 .
4.0K	./.vnc
96K	./.gnome2
4.0K	./.themes
12K	./.dvdcss
168K	./.pulse
4.0K	./.gvfs
18M	./.eclipse
371G	./file-storage
16K	./.gnupg
36K	./.macromedia
8.0K	./.redhat
668K	./.gstreamer-0.10
68K	./.java
23M	./.mozilla
6.0G	./.local
980K	./.kde
16K	./.mplayer
16K	./.gnote
8.0K	./.cups
20K	./Desktop
4.0K	./.qt
12K	./.dbus
36K	./.sane
36K	./.pki
580K	./.adobe
70M	./.cache
68K	./.fontconfig
28K	./.designer
4.0K	./.nautilus
12K	./.abrt
5.8M	./.config
4.0K	./.gnome2_private
136K	./.nv
4.0K	./.icons
20G	./to-sort
8.0K	./.ssh
8.0K	./.emacs.d
852K	./.gconf
80K	./.gconfd
524K	./.gimp-2.6
152K	./.blender
816G	.

As you can see, under /home the "biggest" folder is "file-storage" and it is 371GB. There's still 552GB to go until the capacity of 923GB is reached... All other folders and files do not account to the missing 552GB (not even close), so whats going on?

The only thing I can think of, is the encryption... Would it be possible that somehow every encrypted GB takes another GB for the encryption? Even that, roughly adding everything "du" lists and doing X2 will yield to 796GB which is still far from 923GB..

Any insight appreciated!..
Thanks

I think files only grow about 8k-12k when encrypted.

lpallard · 02-25-2015, 06:12 AM

Quote:

Originally Posted by jailbait

I think that whatever partition contains /home contains a lot more than just /home.

--------------------
Steve Stites

What do you mean by "a lot more than just /home"? Symlinks to other partitions? /home is being used as /home (that is, personal files no system files).. A few weeks ago I was not using encryption and I had less than 400GB of files... Impossible that it grew to almost a terabyte in less than 3 weeks..

lsof | grep deleted yielded 3 entries under /tmp... Nothing under /home

As for the encrypted space.... Thats what I expected.. a marginal increase in disk usage but not 200%+..

Im baffled...

lpallard · 02-25-2015, 06:26 AM

Got it...

I ran a fsck -fv on the faulty filesystem, it didnt report any errors (not that I could see) but nevertheless upon rebooting now I get

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
53G 9.7G 41G 20% /
tmpfs 4.2G 148k 4.2G 1% /dev/shm
/dev/sda1 500M 62M 412M 14% /boot
/dev/mapper/vg_workstation-lv_home
923G 427G 450G 49% /home

Which is more normal to me....

I am using ext4 with Centos 6.6... First time in a long time using linux that I see this kind of glitch...

fatmac · 02-25-2015, 08:41 AM

I neither use LVM nor encription, but usually when you see 100% usage, (allowing for root's reserved space), & know there should be more space, it is a lack of inodes.

You probably have a lot of small files on your disk using up all the inodes, only way to overcome this situation is to have more inodes for this systems usage.

lpallard · 02-25-2015, 04:37 PM

Quote:

...& know there should be more space, it is a lack of inodes

Would fsck fix this?

Quote:

You probably have a lot of small files on your disk using up all the inodes, only way to overcome this situation is to have more inodes for this systems usage.

Exactly 588,534 files on that FS... Not much if you ask me..

Thats really strange

rknichols · 02-25-2015, 06:05 PM

Quote:

Originally Posted by lpallard

Got it...

I ran a fsck -fv on the faulty filesystem, it didnt report any errors (not that I could see) but nevertheless upon rebooting now I get

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
53G 9.7G 41G 20% /
tmpfs 4.2G 148k 4.2G 1% /dev/shm
/dev/sda1 500M 62M 412M 14% /boot
/dev/mapper/vg_workstation-lv_home
923G 427G 450G 49% /home

Which is more normal to me....

I am using ext4 with Centos 6.6... First time in a long time using linux that I see this kind of glitch...

That strongly suggests that there were indeed open, deleted files, and the the fsck was superfluous. "lsof | grep deleted" won't find all of them. "lsof | grep -i del" will find a lot more that show "DEL" in the "FD" column and do not show "(deleted)" after the name. "DEL" indicates deleted files that were mmap()-ed and the file descriptor closed, but are still "in use" because the mmap() is still active.

lpallard · 02-25-2015, 06:26 PM

Generally speaking, is it something to worry about? Why would this happen?

rknichols · 02-25-2015, 06:42 PM

Quote:

Originally Posted by lpallard

Generally speaking, is it something to worry about? Why would this happen?

It's only something to worry about if you run out of space. It happens because a program intentionally unlinked some mmap()-ed file, perhaps a temporary database, to ensure that it would be deleted automatically when the program terminated. Another place it happens a lot, though not generally under /home, is with libraries that get updated while some program is still using the old version. The old library has to stay around until it is no longer in use, and those libraries are almost always mmap()-ed and not held on an open file descriptor.

You would have to run that "lsof | grep -i del" while the problem was present to see exactly what was causing it.

lpallard · 02-25-2015, 07:06 PM

Quote:

You would have to run that "lsof | grep -i del" while the problem was present to see exactly what was causing it.

Allright! I guess Ill keep an eye on that and next time it happens, Ill run the command you suggested!

Thanks!!!

syg00 · 02-25-2015, 07:28 PM

Hmmm - how often do you reboot ?. A reboot should both run fsck if needed, and get rid of unlinked filed descriptors.
If you reboot regularly (FSVO "regularly") and the problem persisted, but went away after a fsck that reported no errors (and reboot) , that is odd.

anomie · 02-25-2015, 10:06 PM

What fatmac said. Check:

Code:

df -i

And, no, a fsck(8) does not fix that. You can either grow the filesystem, or you can dump(8), recreate the filesystem with a more suitable bytes-to-inode ratio, and restore(8).

lpallard · 02-26-2015, 05:46 AM

Quote:

Hmmm - how often do you reboot ?

About every week... I usually put the machine in suspend to RAM instead of shutting it down. Been doing this for years with the same hardware and between real reboots, I cannot pretend that the FS changed so much (added or removed thousands of files, etc).. Actually, a few weeks ago I reinstalled centos and restored everything from a backup.

df -i currently reports:

[workstation-user@workstation ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_workstation-lv_root
3276800 297867 2978933 10% /
tmpfs 1002657 11 1002646 1% /dev/shm
/dev/sda1 128016 87 127929 1% /boot
/dev/mapper/vg_workstation-lv_home
57229312 592891 56636421 2% /home