LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-24-2015, 05:53 PM   #1
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Rep: Reputation: Disabled
Partition full (100%) even if sum of all files is less than 50%


Hi,

So I have an encrypted /home partition (created with LVM and encryption during Centos 6.5 install) that ran out of space all of a sudden (or it was close to 100% for weeks and I never noticed...) but the problem is that the sum of all files and folders is less than 50% of the partition's capacity.

df -h
Code:
[root@workstation workstation-user]# df -H
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
                       53G  9.8G   41G  20% /
tmpfs                 4.2G  2.6M  4.2G   1% /dev/shm
/dev/sda1             500M   62M  412M  14% /boot
/dev/mapper/vg_workstation-lv_home
                      923G  876G   98M 100% /home
If I do a du -h --max-depth=1 on /home I get:
Code:
[root@workstation home]# du -h --max-depth=1 .
16K	./lost+found
816G	./workstation-user
816G	.
Then finally if I go in "workstation-user" and issue the same command:
Code:
[root@workstation workstation-user]# du -h --max-depth=1 .
4.0K	./.vnc
96K	./.gnome2
4.0K	./.themes
12K	./.dvdcss
168K	./.pulse
4.0K	./.gvfs
18M	./.eclipse
371G	./file-storage
16K	./.gnupg
36K	./.macromedia
8.0K	./.redhat
668K	./.gstreamer-0.10
68K	./.java
23M	./.mozilla
6.0G	./.local
980K	./.kde
16K	./.mplayer
16K	./.gnote
8.0K	./.cups
20K	./Desktop
4.0K	./.qt
12K	./.dbus
36K	./.sane
36K	./.pki
580K	./.adobe
70M	./.cache
68K	./.fontconfig
28K	./.designer
4.0K	./.nautilus
12K	./.abrt
5.8M	./.config
4.0K	./.gnome2_private
136K	./.nv
4.0K	./.icons
20G	./to-sort
8.0K	./.ssh
8.0K	./.emacs.d
852K	./.gconf
80K	./.gconfd
524K	./.gimp-2.6
152K	./.blender
816G	.
As you can see, under /home the "biggest" folder is "file-storage" and it is 371GB. There's still 552GB to go until the capacity of 923GB is reached... All other folders and files do not account to the missing 552GB (not even close), so whats going on?

The only thing I can think of, is the encryption... Would it be possible that somehow every encrypted GB takes another GB for the encryption? Even that, roughly adding everything "du" lists and doing X2 will yield to 796GB which is still far from 923GB..

Any insight appreciated!..
Thanks

Last edited by lpallard; 02-24-2015 at 05:54 PM.
 
Old 02-24-2015, 07:25 PM   #2
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Virginia, USA
Distribution: Debian 12
Posts: 8,337

Rep: Reputation: 548Reputation: 548Reputation: 548Reputation: 548Reputation: 548Reputation: 548
I think that whatever partition contains /home contains a lot more than just /home.

--------------------
Steve Stites
 
Old 02-24-2015, 07:48 PM   #3
Daws
Member
 
Registered: May 2006
Location: UK
Distribution: Debian
Posts: 447

Rep: Reputation: 39
It could be open unlinked files. You can find them with
Code:
lsof | grep deleted
They normally only appear in tmp directories though.
 
Old 02-24-2015, 11:41 PM   #4
replica9000
Senior Member
 
Registered: Jul 2006
Distribution: Debian Unstable
Posts: 1,124
Blog Entries: 2

Rep: Reputation: 260Reputation: 260Reputation: 260
Quote:
Originally Posted by lpallard View Post
Hi,

So I have an encrypted /home partition (created with LVM and encryption during Centos 6.5 install) that ran out of space all of a sudden (or it was close to 100% for weeks and I never noticed...) but the problem is that the sum of all files and folders is less than 50% of the partition's capacity.

df -h
Code:
[root@workstation workstation-user]# df -H
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
                       53G  9.8G   41G  20% /
tmpfs                 4.2G  2.6M  4.2G   1% /dev/shm
/dev/sda1             500M   62M  412M  14% /boot
/dev/mapper/vg_workstation-lv_home
                      923G  876G   98M 100% /home
If I do a du -h --max-depth=1 on /home I get:
Code:
[root@workstation home]# du -h --max-depth=1 .
16K	./lost+found
816G	./workstation-user
816G	.
Then finally if I go in "workstation-user" and issue the same command:
Code:
[root@workstation workstation-user]# du -h --max-depth=1 .
4.0K	./.vnc
96K	./.gnome2
4.0K	./.themes
12K	./.dvdcss
168K	./.pulse
4.0K	./.gvfs
18M	./.eclipse
371G	./file-storage
16K	./.gnupg
36K	./.macromedia
8.0K	./.redhat
668K	./.gstreamer-0.10
68K	./.java
23M	./.mozilla
6.0G	./.local
980K	./.kde
16K	./.mplayer
16K	./.gnote
8.0K	./.cups
20K	./Desktop
4.0K	./.qt
12K	./.dbus
36K	./.sane
36K	./.pki
580K	./.adobe
70M	./.cache
68K	./.fontconfig
28K	./.designer
4.0K	./.nautilus
12K	./.abrt
5.8M	./.config
4.0K	./.gnome2_private
136K	./.nv
4.0K	./.icons
20G	./to-sort
8.0K	./.ssh
8.0K	./.emacs.d
852K	./.gconf
80K	./.gconfd
524K	./.gimp-2.6
152K	./.blender
816G	.
As you can see, under /home the "biggest" folder is "file-storage" and it is 371GB. There's still 552GB to go until the capacity of 923GB is reached... All other folders and files do not account to the missing 552GB (not even close), so whats going on?

The only thing I can think of, is the encryption... Would it be possible that somehow every encrypted GB takes another GB for the encryption? Even that, roughly adding everything "du" lists and doing X2 will yield to 796GB which is still far from 923GB..

Any insight appreciated!..
Thanks
I think files only grow about 8k-12k when encrypted.
 
Old 02-25-2015, 06:12 AM   #5
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jailbait View Post
I think that whatever partition contains /home contains a lot more than just /home.

--------------------
Steve Stites
What do you mean by "a lot more than just /home"? Symlinks to other partitions? /home is being used as /home (that is, personal files no system files).. A few weeks ago I was not using encryption and I had less than 400GB of files... Impossible that it grew to almost a terabyte in less than 3 weeks..

lsof | grep deleted yielded 3 entries under /tmp... Nothing under /home

As for the encrypted space.... Thats what I expected.. a marginal increase in disk usage but not 200%+..

Im baffled...
 
Old 02-25-2015, 06:26 AM   #6
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Original Poster
Rep: Reputation: Disabled
Got it...

I ran a fsck -fv on the faulty filesystem, it didnt report any errors (not that I could see) but nevertheless upon rebooting now I get

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
53G 9.7G 41G 20% /
tmpfs 4.2G 148k 4.2G 1% /dev/shm
/dev/sda1 500M 62M 412M 14% /boot
/dev/mapper/vg_workstation-lv_home
923G 427G 450G 49% /home

Which is more normal to me....

I am using ext4 with Centos 6.6... First time in a long time using linux that I see this kind of glitch...
 
Old 02-25-2015, 08:41 AM   #7
fatmac
LQ Guru
 
Registered: Sep 2011
Location: Upper Hale, Surrey/Hants Border, UK
Distribution: Mainly Devuan, antiX, & Void, with Tiny Core, Fatdog, & BSD thrown in.
Posts: 5,482

Rep: Reputation: Disabled
I neither use LVM nor encription, but usually when you see 100% usage, (allowing for root's reserved space), & know there should be more space, it is a lack of inodes.

You probably have a lot of small files on your disk using up all the inodes, only way to overcome this situation is to have more inodes for this systems usage.
 
Old 02-25-2015, 04:37 PM   #8
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Original Poster
Rep: Reputation: Disabled
Quote:
...& know there should be more space, it is a lack of inodes
Would fsck fix this?

Quote:
You probably have a lot of small files on your disk using up all the inodes, only way to overcome this situation is to have more inodes for this systems usage.
Exactly 588,534 files on that FS... Not much if you ask me..

Thats really strange
 
Old 02-25-2015, 06:05 PM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,776

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
Quote:
Originally Posted by lpallard View Post
Got it...

I ran a fsck -fv on the faulty filesystem, it didnt report any errors (not that I could see) but nevertheless upon rebooting now I get

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_workstation-lv_root
53G 9.7G 41G 20% /
tmpfs 4.2G 148k 4.2G 1% /dev/shm
/dev/sda1 500M 62M 412M 14% /boot
/dev/mapper/vg_workstation-lv_home
923G 427G 450G 49% /home

Which is more normal to me....

I am using ext4 with Centos 6.6... First time in a long time using linux that I see this kind of glitch...
That strongly suggests that there were indeed open, deleted files, and the the fsck was superfluous. "lsof | grep deleted" won't find all of them. "lsof | grep -i del" will find a lot more that show "DEL" in the "FD" column and do not show "(deleted)" after the name. "DEL" indicates deleted files that were mmap()-ed and the file descriptor closed, but are still "in use" because the mmap() is still active.
 
Old 02-25-2015, 06:26 PM   #10
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Original Poster
Rep: Reputation: Disabled
Generally speaking, is it something to worry about? Why would this happen?
 
Old 02-25-2015, 06:42 PM   #11
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,776

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
Quote:
Originally Posted by lpallard View Post
Generally speaking, is it something to worry about? Why would this happen?
It's only something to worry about if you run out of space. It happens because a program intentionally unlinked some mmap()-ed file, perhaps a temporary database, to ensure that it would be deleted automatically when the program terminated. Another place it happens a lot, though not generally under /home, is with libraries that get updated while some program is still using the old version. The old library has to stay around until it is no longer in use, and those libraries are almost always mmap()-ed and not held on an open file descriptor.

You would have to run that "lsof | grep -i del" while the problem was present to see exactly what was causing it.
 
Old 02-25-2015, 07:06 PM   #12
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Original Poster
Rep: Reputation: Disabled
Quote:
You would have to run that "lsof | grep -i del" while the problem was present to see exactly what was causing it.
Allright! I guess Ill keep an eye on that and next time it happens, Ill run the command you suggested!

Thanks!!!
 
Old 02-25-2015, 07:28 PM   #13
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,123

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
Hmmm - how often do you reboot ?. A reboot should both run fsck if needed, and get rid of unlinked filed descriptors.
If you reboot regularly (FSVO "regularly") and the problem persisted, but went away after a fsck that reported no errors (and reboot) , that is odd.
 
Old 02-25-2015, 10:06 PM   #14
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora
Posts: 3,935
Blog Entries: 5

Rep: Reputation: Disabled
What fatmac said. Check:
Code:
df -i
And, no, a fsck(8) does not fix that. You can either grow the filesystem, or you can dump(8), recreate the filesystem with a more suitable bytes-to-inode ratio, and restore(8).
 
Old 02-26-2015, 05:46 AM   #15
lpallard
Senior Member
 
Registered: Nov 2008
Posts: 1,045

Original Poster
Rep: Reputation: Disabled
Quote:
Hmmm - how often do you reboot ?
About every week... I usually put the machine in suspend to RAM instead of shutting it down. Been doing this for years with the same hardware and between real reboots, I cannot pretend that the FS changed so much (added or removed thousands of files, etc).. Actually, a few weeks ago I reinstalled centos and restored everything from a backup.

df -i currently reports:

[workstation-user@workstation ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_workstation-lv_root
3276800 297867 2978933 10% /
tmpfs 1002657 11 1002646 1% /dev/shm
/dev/sda1 128016 87 127929 1% /boot
/dev/mapper/vg_workstation-lv_home
57229312 592891 56636421 2% /home
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Is possible? link-up between 100 Halt and 100 Full? poplinux Linux - Networking 3 10-07-2014 07:32 PM
RHEL4 root directory says full but sum of use is nowhere near full. Why? SteveInTallyFL Linux - Server 11 02-05-2009 05:45 AM
root partition full, not sure how; where are my files? lefty.crupps MEPIS 10 01-31-2006 05:21 PM
root partition shows 100% full in RH 7.2 dmcmurray5 Red Hat 2 05-18-2004 03:24 PM
partition help / 100% full JanusDay Slackware 7 02-23-2004 05:57 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:19 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration