SDA2 full

jim_cliff11 · 06-13-2021, 10:04 AM

Hi,

My OpenSuse 15.1 server running LAMP configuration and samba spat me an error this morning. On closer inspection

Code:

df -h

it seems SDA2 is full.

Code:

CliffServ:/ # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.9G  8.0K  7.9G   1% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           7.9G   50M  7.8G   1% /run
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2        41G   39G     0 100% /
/dev/sda2        41G   39G     0 100% /srv
/dev/sda2        41G   39G     0 100% /tmp
/dev/sda2        41G   39G     0 100% /opt
/dev/sda2        41G   39G     0 100% /root
/dev/sda2        41G   39G     0 100% /boot/grub2/x86_64-efi
/dev/sda2        41G   39G     0 100% /usr/local
/dev/sda2        41G   39G     0 100% /var
/dev/sda2        41G   39G     0 100% /boot/grub2/i386-pc
/dev/sdb1       1.4T  3.7G  1.4T   1% /home
/dev/sda3       1.8T  108G  1.7T   6% /big
/dev/sdc1       466G  108G  358G  24% /exthdd
tmpfs           1.6G     0  1.6G   0% /run/user/0

For the life of me I cannot find any major GB consuming files on SDA2. /exthdd is an external drive which rsync backs up to every night. /home and /big are on separate partitions. I've checked for 'snapshots', logs, /tmp, /var/tmp without success.

Code:

mount

gives me:

Code:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=8182920k,nr_inodes=2045730,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/@)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16984)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/srv)
/dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/tmp)
/dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/opt)
/dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/root)
/dev/sda2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/boot/grub2/x86_64-efi)
/dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/usr/local)
/dev/sda2 on /var type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/var)
/dev/sda2 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/i386-pc)
/dev/sdb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda3 on /big type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc1 on /exthdd type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1640148k,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

Code:

du -sh *

gives me:

Code:

CliffServ:/ # du -sh *
106G    big
2.1M    bin
86M     boot
8.0K    dev
23M     etc
108G    exthdd
2.3G    home
5.7M    info
712M    lib
11M     lib64
0       mnt
0       opt
du: cannot access 'proc/13119/task/13119/fd/4': No such file or directory
du: cannot access 'proc/13119/task/13119/fdinfo/4': No such file or directory
du: cannot access 'proc/13119/fd/4': No such file or directory
du: cannot access 'proc/13119/fdinfo/4': No such file or directory
0       proc
192K    root
50M     run
10M     sbin
0       selinux
37M     srv
0       sys
4.0K    tmp
4.8G    usr
1.1G    var

Can anyone shed any light on this for me?

Many thanks,
Jim

business_kid · 06-13-2021, 02:37 PM

It seems correct.

sda2 is 41G and full. /var or /tmp could have filled up. Medium term, organise yourself better. short term, fix it.

Code:

cd / 
[sudo] du -sh *

Delete/transfer as necessary. Look after the gigabytes, and the Megabytes will look after themselves

berndbausch · 06-14-2021, 12:35 AM

You may have deleted files that are still open. They are not caught by du, but

Code:

lsof | grep delete

should show them.

Also, df on BTRFS is not necessarily very reliable. Use the btrfs tool as well (sorry, I can't tell you which subcommand is appropriate).