Why is my root filesystem full, I can't find anything that taking up the filesystem space

servaed · 04-06-2017, 10:10 PM

Hello there, I want to ask about my root filesystem, when I do 'df -h' it shows 92% usage

Code:

[root@prodbdaast1dr usr]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_prodbdaast1dr-lv_root
                      120G  105G  9.4G  92% /
tmpfs                 253G   76K  253G   1% /dev/shm
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1
                      477M   34M  418M   8% /boot
/dev/mapper/vg_prodbdaast1dr-lv_home
                       79G  3.0G   72G   4% /home
/dev/mapper/vg_prodbdaast1dr-lv_opt
                       79G  2.5G   73G   4% /opt
/dev/mapper/vg_prodbdaast1dr-lv_usr
                       79G  4.0G   71G   6% /usr
/dev/mapper/vg_prodbdaast1dr-lv_var
                       79G  446M   75G   1% /var
/dev/sda1             3.6T  209M  3.6T   1% /data/01
/dev/sdb1             3.6T  210M  3.6T   1% /data/02
/dev/sdc1             3.6T  209M  3.6T   1% /data/03
/dev/sdd1             3.6T  209M  3.6T   1% /data/04
/dev/sde1             3.6T   69M  3.6T   1% /data/05
/dev/sdf1             3.6T   69M  3.6T   1% /data/06
/dev/sdg1             3.6T   69M  3.6T   1% /data/07
/dev/sdh1             3.6T   69M  3.6T   1% /data/08
/dev/sdi1             3.6T   69M  3.6T   1% /data/09
/dev/sdj1             3.6T   69M  3.6T   1% /data/10
/dev/sdk1             3.6T   69M  3.6T   1% /data/11
/dev/sdl1             3.6T   69M  3.6T   1% /data/12
cm_processes          253G  4.8M  253G   1% /var/run/cloudera-scm-agent/process

And when i do 'du / | sort -rn | head', nothing is taking the space that big

Code:

[root@prodbdaast1dr usr]# du / | sort -rn | head
du: cannot access `/proc/49888/task/49888/fd/4': No such file or directory
du: cannot access `/proc/49888/task/49888/fdinfo/4': No such file or directory
du: cannot access `/proc/49888/fd/4': No such file or directory
du: cannot access `/proc/49888/fdinfo/4': No such file or directory
11426196        /
4117232 /usr
3064312 /home
3064052 /home/beehive
2504112 /opt
2140156 /usr/share
2005040 /opt/cloudera
2004880 /opt/cloudera/parcels
2004872 /opt/cloudera/parcels/CDH-5.7.5-1.cdh5.7.5.p0.3
1396272 /home/beehive/toolchain

How to know what is taking up my root filesystem space

wpeckham · 04-07-2017, 09:33 AM

Running the following command may be illuminating:

Code:

sudo du -smx /* | sort -n

Please let me know what that tells you.
Also, if we could see the output of

Code:

sudo mount

please.

pan64 · 04-07-2017, 09:38 AM

also please post: df -i

cepheus11 · 04-07-2017, 11:07 AM

You have probably data hidden by mounting a different filesystem at the directory (/home, /usr, /var). Mount your / again with the "--bind"-option somewhere else and analyze it from there:

Code:

mkdir /tmp/rootbind 
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind

Habitual · 04-07-2017, 12:04 PM

inodes?

Code:

df -i /

Just sayin'

Shadow_7 · 04-07-2017, 05:55 PM

# du -h --max-depth=1 /

And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.

wpeckham · 04-07-2017, 10:28 PM

Quote:

Originally Posted by Shadow_7

# du -h --max-depth=1 /

And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.

That will only help if the hogs happen to be visible on the file system. As earlier mentioned, that may not be the case. There are several things that can result in hidden hogs (aside from camo paint spilled into the hog pen ) such as mounting over the hog folder. Also, the issue may not be space as such, there could be inode issues (although I do not suspect those in this case).

We need more data to determine which cases to pursue, so we await the feedback from the OP.

TobyV · 04-08-2017, 02:24 AM

@ servaed

I'm not an admin, but I found something online that can cause a filesystem to fill up too.

Sometimes a process will continue to write to an unlinked file and fill up disk space quickly. Below is an excerpt from the website explaining it better. Click on the link below to learn how to find them. Again, this is just a suggestion for you to try and/or investigate.

Quote:

a. Finding an Unlinked Open File
=================================

A pesky variant of a file that is filling a file system is an
unlinked file to which some process is still writing. When a
process opens a file and then unlinks it, the file's resources
remain in use by the process, but the file's directory entries
are removed. Hence, even when you know the directory where the
file once resided, you can't detect it with ls.

This can be an administrative problem when the unlinked file is
large, and the process that holds it open continues to write to
it. Only when the process closes the file will its resources,
particularly disk space, be released.

Lsof can help you find unlinked files on local disks. It has an
option, +L, that will list the link counts of open files. That
helps because an unlinked file on a local disk has a zero link
count. Note: this is NOT true for NFS files, accessed from a
remote server.

You could use the option to list all files and look for a zero
link count in the NLINK column -- e.g.,

http://www.akadia.com/services/lsof_quickstart.txt

dave@burn-it.co.uk · 04-08-2017, 07:24 AM

There is a limited number of possible entries in the directory of root.
It is often the directory that fills up not the disk space.
Just create folders and move the entries for a lot of files.

rknichols · 04-08-2017, 11:17 AM

Quote:

Originally Posted by dave@burn-it.co.uk

There is a limited number of possible entries in the directory of root.

That is not true. It was true back in the days of FAT16, FAT12, and earlier. In FAT32 (and later) and all Linux filesystems, the root directory is an ordinary directory and can grow to a size limited only by the available disk space.

Emerson · 04-08-2017, 11:31 AM

Quote:

Originally Posted by Shadow_7

# du -h --max-depth=1 /

And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.

You may want to add -x to this command to avoid diving into other filesystems and remote shares.

servaed · 04-09-2017, 12:26 AM

Hi All,

Thanks for your response, I will try every each of your comments on Monday and will inform the result later

servaed · 04-09-2017, 10:21 PM

Hi All,

This is the result of some commands you guys told me, I think I found something when doing this mkdir, mount, and DU

mkdir /tmp/rootbind
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind

Code:

mkdir /tmp/rootbind 
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind

104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current/finalized
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current
4.0K    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/tmp
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953
104G    /tmp/rootbind/data/12/dfs/dn/current
104G    /tmp/rootbind/data/12/dfs/dn
104G    /tmp/rootbind/data/12/dfs
4.0K    /tmp/rootbind/data/12/impala/impalad/impala-scratch
8.0K    /tmp/rootbind/data/12/impala/impalad
12K     /tmp/rootbind/data/12/impala
104G    /tmp/rootbind/data/12

sudo du -smx /* | sort -n

Code:

[root@prodbdaast1dr ~]# sudo du -smx /* | sort -n
du: cannot access `/proc/49335/task/49335/fd/4': No such file or directory
du: cannot access `/proc/49335/task/49335/fdinfo/4': No such file or directory
du: cannot access `/proc/49335/fd/4': No such file or directory
du: cannot access `/proc/49335/fdinfo/4': No such file or directory
0       /misc
0       /net
0       /primary
0       /proc
0       /sys
1       /cgroup
1       /data
1       /dev
1       /lost+found
1       /media
1       /mnt
1       /selinux
1       /srv
3       /tmp
8       /bin
15      /sbin
27      /lib64
32      /boot
40      /etc
161     /lib
395     /var
457     /root
2446    /opt
2993    /home
4021    /usr

sudo mount

Code:

[root@prodbdaast1dr ~]# sudo mount
/dev/mapper/vg_prodbdaast1dr-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1 on /boot type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_home on /home type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_opt on /opt type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_usr on /usr type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_var on /var type ext4 (rw)
/dev/sda1 on /data/01 type ext4 (rw,noatime)
/dev/sdb1 on /data/02 type ext4 (rw,noatime)
/dev/sdc1 on /data/03 type ext4 (rw,noatime)
/dev/sdd1 on /data/04 type ext4 (rw,noatime)
/dev/sde1 on /data/05 type ext4 (rw,noatime)
/dev/sdf1 on /data/06 type ext4 (rw,noatime)
/dev/sdg1 on /data/07 type ext4 (rw,noatime)
/dev/sdh1 on /data/08 type ext4 (rw,noatime)
/dev/sdi1 on /data/09 type ext4 (rw,noatime)
/dev/sdj1 on /data/10 type ext4 (rw,noatime)
/dev/sdk1 on /data/11 type ext4 (rw,noatime)
/dev/sdl1 on /data/12 type ext4 (rw,noatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
cm_processes on /var/run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/blkio type cgroup (rw,blkio)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpuacct type cgroup (rw,cpuacct)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpu type cgroup (rw,cpu)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/memory type cgroup (rw,memory)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

df -i

Code:

[root@prodbdaast1dr ~]# df -i
Filesystem              Inodes  IUsed     IFree IUse% Mounted on
/dev/mapper/vg_prodbdaast1dr-lv_root
                       7970816  10288   7960528    1% /
tmpfs                 66149867      4  66149863    1% /dev/shm
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1
                        128016     39    127977    1% /boot
/dev/mapper/vg_prodbdaast1dr-lv_home
                       5242880  36883   5205997    1% /home
/dev/mapper/vg_prodbdaast1dr-lv_opt
                       5242880  41375   5201505    1% /opt
/dev/mapper/vg_prodbdaast1dr-lv_usr
                       5242880 104511   5138369    2% /usr
/dev/mapper/vg_prodbdaast1dr-lv_var
                       5242880   3874   5239006    1% /var
/dev/sda1            244195328   1401 244193927    1% /data/01
/dev/sdb1            244195328   1401 244193927    1% /data/02
/dev/sdc1            244195328   1399 244193929    1% /data/03
/dev/sdd1            244195328   1401 244193927    1% /data/04
/dev/sde1            244146176    114 244146062    1% /data/05
/dev/sdf1            244146176    114 244146062    1% /data/06
/dev/sdg1            244195328    114 244195214    1% /data/07
/dev/sdh1            244195328    114 244195214    1% /data/08
/dev/sdi1            244195328    114 244195214    1% /data/09
/dev/sdj1            244195328    114 244195214    1% /data/10
/dev/sdk1            244195328    114 244195214    1% /data/11
/dev/sdl1            244195328    116 244195212    1% /data/12
cm_processes          66149867    697  66149170    1% /var/run/cloudera-scm-agent/process

df -i /

Code:

root@prodbdaast1dr ~]# df -i /
Filesystem            Inodes IUsed   IFree IUse% Mounted on
/dev/mapper/vg_prodbdaast1dr-lv_root
                     7970816 10288 7960528    1% /

du -h --max-depth=1 /

Code:

[root@prodbdaast1dr ~]# du -h --max-depth=1 /
7.7M    /bin
15M     /sbin
0       /net
12K     /.dbus
2.4G    /opt
4.0K    /selinux
4.0G    /usr
4.0K    /srv
392K    /dev
569M    /data
27M     /lib64
4.0K    /mnt
8.0K    /cgroup
161M    /lib
3.0G    /home
du: cannot access `/proc/5775/task/6338/fd/251': No such file or directory
du: cannot access `/proc/50951/task/50951/fd/4': No such file or directory
du: cannot access `/proc/50951/task/50951/fdinfo/4': No such file or directory
du: cannot access `/proc/50951/fd/4': No such file or directory
du: cannot access `/proc/50951/fdinfo/4': No such file or directory
0       /proc
4.0K    /media
2.9M    /tmp
0       /misc
32M     /boot
404M    /var
457M    /root
40M     /etc
16K     /lost+found
0       /sys
11G     /

TobyV · 04-09-2017, 10:48 PM

@ servaed

Did you check my post #8?

I'm not sure if this is the cause, but it doesn't hurt to investigate. Commands like ls, du, and find willl not list unlinked files because they've been deleted, but some process is still writing to it or holding it open. Once you find and kill the process, it will release the file and restore the disk space and other sources.

Look for a zero in the NLINK column and deleted at the end of line. Also look under the SIZE/OFF column. It will be in bytes.

rknichols · 04-10-2017, 08:52 AM

Quote:

Originally Posted by servaed

Code:

mkdir /tmp/rootbind 
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind

104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current/finalized
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/current
4.0K    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953/tmp
104G    /tmp/rootbind/data/12/dfs/dn/current/BP-855843374-10.254.200.3-1487663914953
104G    /tmp/rootbind/data/12/dfs/dn/current
104G    /tmp/rootbind/data/12/dfs/dn
104G    /tmp/rootbind/data/12/dfs
4.0K    /tmp/rootbind/data/12/impala/impalad/impala-scratch
8.0K    /tmp/rootbind/data/12/impala/impalad
12K     /tmp/rootbind/data/12/impala
104G    /tmp/rootbind/data/12

You appear to have 104GB hiding under the /data/12 mount point directory. You can look in /tmp/rootbind/data/12 and its subdirectories and see if there is anything there you need to keep, and either move it elsewhere or delete it.

It's a bit odd that the filesystem you have mounted on /data/12 only appears to be using about 69MB. How 104GB got stored in that directory while that filesystem was not mounted is a bit of a mystery.