LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-06-2017, 09:10 PM   #1
servaed
LQ Newbie
 
Registered: Apr 2017
Posts: 5

Rep: Reputation: Disabled
Why is my root filesystem full, I can't find anything that taking up the filesystem space


Hello there, I want to ask about my root filesystem, when I do 'df -h' it shows 92% usage

Code:
[root@prodbdaast1dr usr]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_prodbdaast1dr-lv_root
                      120G  105G  9.4G  92% /
tmpfs                 253G   76K  253G   1% /dev/shm
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1
                      477M   34M  418M   8% /boot
/dev/mapper/vg_prodbdaast1dr-lv_home
                       79G  3.0G   72G   4% /home
/dev/mapper/vg_prodbdaast1dr-lv_opt
                       79G  2.5G   73G   4% /opt
/dev/mapper/vg_prodbdaast1dr-lv_usr
                       79G  4.0G   71G   6% /usr
/dev/mapper/vg_prodbdaast1dr-lv_var
                       79G  446M   75G   1% /var
/dev/sda1             3.6T  209M  3.6T   1% /data/01
/dev/sdb1             3.6T  210M  3.6T   1% /data/02
/dev/sdc1             3.6T  209M  3.6T   1% /data/03
/dev/sdd1             3.6T  209M  3.6T   1% /data/04
/dev/sde1             3.6T   69M  3.6T   1% /data/05
/dev/sdf1             3.6T   69M  3.6T   1% /data/06
/dev/sdg1             3.6T   69M  3.6T   1% /data/07
/dev/sdh1             3.6T   69M  3.6T   1% /data/08
/dev/sdi1             3.6T   69M  3.6T   1% /data/09
/dev/sdj1             3.6T   69M  3.6T   1% /data/10
/dev/sdk1             3.6T   69M  3.6T   1% /data/11
/dev/sdl1             3.6T   69M  3.6T   1% /data/12
cm_processes          253G  4.8M  253G   1% /var/run/cloudera-scm-agent/process
And when i do 'du / | sort -rn | head', nothing is taking the space that big

Code:
[root@prodbdaast1dr usr]# du / | sort -rn | head
du: cannot access `/proc/49888/task/49888/fd/4': No such file or directory
du: cannot access `/proc/49888/task/49888/fdinfo/4': No such file or directory
du: cannot access `/proc/49888/fd/4': No such file or directory
du: cannot access `/proc/49888/fdinfo/4': No such file or directory
11426196        /
4117232 /usr
3064312 /home
3064052 /home/beehive
2504112 /opt
2140156 /usr/share
2005040 /opt/cloudera
2004880 /opt/cloudera/parcels
2004872 /opt/cloudera/parcels/CDH-5.7.5-1.cdh5.7.5.p0.3
1396272 /home/beehive/toolchain
How to know what is taking up my root filesystem space
 
Old 04-07-2017, 08:33 AM   #2
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,793

Rep: Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713
Running the following command may be illuminating:
Code:
sudo du -smx /* | sort -n
Please let me know what that tells you.
Also, if we could see the output of
Code:
sudo mount
please.

Last edited by wpeckham; 04-07-2017 at 08:34 AM.
 
Old 04-07-2017, 08:38 AM   #3
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,492

Rep: Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532
also please post: df -i
 
Old 04-07-2017, 10:07 AM   #4
cepheus11
Member
 
Registered: Nov 2010
Location: Germany
Distribution: Gentoo
Posts: 286

Rep: Reputation: 89
You have probably data hidden by mounting a different filesystem at the directory (/home, /usr, /var). Mount your / again with the "--bind"-option somewhere else and analyze it from there:

Code:
mkdir /tmp/rootbind 
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind
 
Old 04-07-2017, 11:04 AM   #5
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
inodes?

Code:
df -i /
Just sayin'
 
Old 04-07-2017, 04:55 PM   #6
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,137
Blog Entries: 1

Rep: Reputation: 873Reputation: 873Reputation: 873Reputation: 873Reputation: 873Reputation: 873Reputation: 873
# du -h --max-depth=1 /

And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.
 
Old 04-07-2017, 09:28 PM   #7
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,793

Rep: Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713
Quote:
Originally Posted by Shadow_7 View Post
# du -h --max-depth=1 /

And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.
That will only help if the hogs happen to be visible on the file system. As earlier mentioned, that may not be the case. There are several things that can result in hidden hogs (aside from camo paint spilled into the hog pen ) such as mounting over the hog folder. Also, the issue may not be space as such, there could be inode issues (although I do not suspect those in this case).

We need more data to determine which cases to pursue, so we await the feedback from the OP.
 
Old 04-08-2017, 01:24 AM   #8
TobyV
LQ Newbie
 
Registered: Apr 2017
Posts: 19

Rep: Reputation: Disabled
@ servaed

I'm not an admin, but I found something online that can cause a filesystem to fill up too.

Sometimes a process will continue to write to an unlinked file and fill up disk space quickly. Below is an excerpt from the website explaining it better. Click on the link below to learn how to find them. Again, this is just a suggestion for you to try and/or investigate.



Quote:
a. Finding an Unlinked Open File
=================================

A pesky variant of a file that is filling a file system is an
unlinked file to which some process is still writing. When a
process opens a file and then unlinks it, the file's resources
remain in use by the process, but the file's directory entries
are removed. Hence, even when you know the directory where the
file once resided, you can't detect it with ls.

This can be an administrative problem when the unlinked file is
large, and the process that holds it open continues to write to
it. Only when the process closes the file will its resources,
particularly disk space, be released.


Lsof can help you find unlinked files on local disks. It has an
option, +L, that will list the link counts of open files. That
helps because an unlinked file on a local disk has a zero link
count. Note: this is NOT true for NFS files, accessed from a
remote server.

You could use the option to list all files and look for a zero
link count in the NLINK column -- e.g.,
http://www.akadia.com/services/lsof_quickstart.txt

Last edited by TobyV; 04-08-2017 at 05:01 AM.
 
Old 04-08-2017, 06:24 AM   #9
dave@burn-it.co.uk
Member
 
Registered: Sep 2011
Distribution: Puppy
Posts: 601

Rep: Reputation: 172Reputation: 172
There is a limited number of possible entries in the directory of root.
It is often the directory that fills up not the disk space.
Just create folders and move the entries for a lot of files.
 
Old 04-08-2017, 10:17 AM   #10
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,547

Rep: Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082
Quote:
Originally Posted by dave@burn-it.co.uk View Post
There is a limited number of possible entries in the directory of root.
That is not true. It was true back in the days of FAT16, FAT12, and earlier. In FAT32 (and later) and all Linux filesystems, the root directory is an ordinary directory and can grow to a size limited only by the available disk space.
 
Old 04-08-2017, 10:31 AM   #11
Emerson
LQ Sage
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~arch
Posts: 7,231

Rep: Reputation: Disabled
Quote:
Originally Posted by Shadow_7 View Post
# du -h --max-depth=1 /

And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.
You may want to add -x to this command to avoid diving into other filesystems and remote shares.
 
Old 04-08-2017, 11:26 PM   #12
servaed
LQ Newbie
 
Registered: Apr 2017
Posts: 5

Original Poster
Rep: Reputation: Disabled
Hi All,

Thanks for your response, I will try every each of your comments on Monday and will inform the result later
 
Old 04-09-2017, 09:21 PM   #13
servaed
LQ Newbie
 
Registered: Apr 2017
Posts: 5

Original Poster
Rep: Reputation: Disabled
Hi All,

This is the result of some commands you guys told me, I think I found something when doing this mkdir, mount, and DU

mkdir /tmp/rootbind
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind


Code:
mkdir /tmp/rootbind 
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind

104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current/finalized
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current
4.0K    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/tmp
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953
104G    /tmp/rootbind/data/12/dfs/dn/current
104G    /tmp/rootbind/data/12/dfs/dn
104G    /tmp/rootbind/data/12/dfs
4.0K    /tmp/rootbind/data/12/impala/impalad/impala-scratch
8.0K    /tmp/rootbind/data/12/impala/impalad
12K     /tmp/rootbind/data/12/impala
104G    /tmp/rootbind/data/12
sudo du -smx /* | sort -n

Code:
[root@prodbdaast1dr ~]# sudo du -smx /* | sort -n
du: cannot access `/proc/49335/task/49335/fd/4': No such file or directory
du: cannot access `/proc/49335/task/49335/fdinfo/4': No such file or directory
du: cannot access `/proc/49335/fd/4': No such file or directory
du: cannot access `/proc/49335/fdinfo/4': No such file or directory
0       /misc
0       /net
0       /primary
0       /proc
0       /sys
1       /cgroup
1       /data
1       /dev
1       /lost+found
1       /media
1       /mnt
1       /selinux
1       /srv
3       /tmp
8       /bin
15      /sbin
27      /lib64
32      /boot
40      /etc
161     /lib
395     /var
457     /root
2446    /opt
2993    /home
4021    /usr
sudo mount

Code:
[root@prodbdaast1dr ~]# sudo mount
/dev/mapper/vg_prodbdaast1dr-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1 on /boot type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_home on /home type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_opt on /opt type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_usr on /usr type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_var on /var type ext4 (rw)
/dev/sda1 on /data/01 type ext4 (rw,noatime)
/dev/sdb1 on /data/02 type ext4 (rw,noatime)
/dev/sdc1 on /data/03 type ext4 (rw,noatime)
/dev/sdd1 on /data/04 type ext4 (rw,noatime)
/dev/sde1 on /data/05 type ext4 (rw,noatime)
/dev/sdf1 on /data/06 type ext4 (rw,noatime)
/dev/sdg1 on /data/07 type ext4 (rw,noatime)
/dev/sdh1 on /data/08 type ext4 (rw,noatime)
/dev/sdi1 on /data/09 type ext4 (rw,noatime)
/dev/sdj1 on /data/10 type ext4 (rw,noatime)
/dev/sdk1 on /data/11 type ext4 (rw,noatime)
/dev/sdl1 on /data/12 type ext4 (rw,noatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
cm_processes on /var/run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/blkio type cgroup (rw,blkio)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpuacct type cgroup (rw,cpuacct)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpu type cgroup (rw,cpu)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/memory type cgroup (rw,memory)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
df -i
Code:
[root@prodbdaast1dr ~]# df -i
Filesystem              Inodes  IUsed     IFree IUse% Mounted on
/dev/mapper/vg_prodbdaast1dr-lv_root
                       7970816  10288   7960528    1% /
tmpfs                 66149867      4  66149863    1% /dev/shm
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1
                        128016     39    127977    1% /boot
/dev/mapper/vg_prodbdaast1dr-lv_home
                       5242880  36883   5205997    1% /home
/dev/mapper/vg_prodbdaast1dr-lv_opt
                       5242880  41375   5201505    1% /opt
/dev/mapper/vg_prodbdaast1dr-lv_usr
                       5242880 104511   5138369    2% /usr
/dev/mapper/vg_prodbdaast1dr-lv_var
                       5242880   3874   5239006    1% /var
/dev/sda1            244195328   1401 244193927    1% /data/01
/dev/sdb1            244195328   1401 244193927    1% /data/02
/dev/sdc1            244195328   1399 244193929    1% /data/03
/dev/sdd1            244195328   1401 244193927    1% /data/04
/dev/sde1            244146176    114 244146062    1% /data/05
/dev/sdf1            244146176    114 244146062    1% /data/06
/dev/sdg1            244195328    114 244195214    1% /data/07
/dev/sdh1            244195328    114 244195214    1% /data/08
/dev/sdi1            244195328    114 244195214    1% /data/09
/dev/sdj1            244195328    114 244195214    1% /data/10
/dev/sdk1            244195328    114 244195214    1% /data/11
/dev/sdl1            244195328    116 244195212    1% /data/12
cm_processes          66149867    697  66149170    1% /var/run/cloudera-scm-agent/process
df -i /

Code:
root@prodbdaast1dr ~]# df -i /
Filesystem            Inodes IUsed   IFree IUse% Mounted on
/dev/mapper/vg_prodbdaast1dr-lv_root
                     7970816 10288 7960528    1% /
du -h --max-depth=1 /

Code:
[root@prodbdaast1dr ~]# du -h --max-depth=1 /
7.7M    /bin
15M     /sbin
0       /net
12K     /.dbus
2.4G    /opt
4.0K    /selinux
4.0G    /usr
4.0K    /srv
392K    /dev
569M    /data
27M     /lib64
4.0K    /mnt
8.0K    /cgroup
161M    /lib
3.0G    /home
du: cannot access `/proc/5775/task/6338/fd/251': No such file or directory
du: cannot access `/proc/50951/task/50951/fd/4': No such file or directory
du: cannot access `/proc/50951/task/50951/fdinfo/4': No such file or directory
du: cannot access `/proc/50951/fd/4': No such file or directory
du: cannot access `/proc/50951/fdinfo/4': No such file or directory
0       /proc
4.0K    /media
2.9M    /tmp
0       /misc
32M     /boot
404M    /var
457M    /root
40M     /etc
16K     /lost+found
0       /sys
11G     /

Last edited by servaed; 04-09-2017 at 09:35 PM.
 
Old 04-09-2017, 09:48 PM   #14
TobyV
LQ Newbie
 
Registered: Apr 2017
Posts: 19

Rep: Reputation: Disabled
@ servaed

Did you check my post #8?

I'm not sure if this is the cause, but it doesn't hurt to investigate. Commands like ls, du, and find willl not list unlinked files because they've been deleted, but some process is still writing to it or holding it open. Once you find and kill the process, it will release the file and restore the disk space and other sources.

Look for a zero in the NLINK column and deleted at the end of line. Also look under the SIZE/OFF column. It will be in bytes.
 
Old 04-10-2017, 07:52 AM   #15
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,547

Rep: Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082
Quote:
Originally Posted by servaed View Post
Code:
mkdir /tmp/rootbind 
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind

104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current/finalized
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current
4.0K    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/tmp
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953
104G    /tmp/rootbind/data/12/dfs/dn/current
104G    /tmp/rootbind/data/12/dfs/dn
104G    /tmp/rootbind/data/12/dfs
4.0K    /tmp/rootbind/data/12/impala/impalad/impala-scratch
8.0K    /tmp/rootbind/data/12/impala/impalad
12K     /tmp/rootbind/data/12/impala
104G    /tmp/rootbind/data/12
You appear to have 104GB hiding under the /data/12 mount point directory. You can look in /tmp/rootbind/data/12 and its subdirectories and see if there is anything there you need to keep, and either move it elsewhere or delete it.

It's a bit odd that the filesystem you have mounted on /data/12 only appears to be using about 69MB. How 104GB got stored in that directory while that filesystem was not mounted is a bit of a mystery.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
e2fsck taking a lot of space for use by the filesystem itself. stf92 Slackware 1 09-11-2016 09:31 PM
[SOLVED] /var filesystem is full but unable to find what is being used. kathirvel Linux - Server 10 01-25-2016 01:48 PM
[SOLVED] root filesystem full - what to do? Gustavo AzDbn Linux - Newbie 5 06-19-2014 09:40 PM
Root Filesystem (/) appears almost Full raheel_com88 Linux - Software 3 11-20-2012 10:09 AM
Creating a filesystem to use free space on an existing filesystem idsouza28 Linux - Newbie 7 06-05-2012 12:29 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration