[SOLVED] LVM issues

sgosnell · 08-01-2020, 09:37 PM

I'm not sure exactly where to post this, nor what my problem is. I have a desktop running Debian Stable, using LVM. Some time back I got a notice that my root drive was full. I had a spare 128GB SD card, so I formatted it and made it part of the root group. Very soon it was full. I have a hard time believing that I have 145GB on my root. I can update, most of the time, by running apt clean, but sometimes the drive is full before the update completes. I don't see anything that I would consider unusual returned from du or df. Something seems to be filling my root drive, and i suspect it's a problem with the LVM setup, but I'm not experienced enough with it to be sure. I've deleted logs, and running logrotate, and var/log isn't at all that large. Any help or suggestions would be appreciated.

berndbausch · 08-01-2020, 10:33 PM

Quote:

Originally Posted by sgosnell

I don't see anything that I would consider unusual returned from du or df.

Neither do I, or anybody else, because we can't see anything at all.

If you ask what could fill up your drive, all I can say is "tell me which directories contain a lot of data".

By the way, making an additional drive part of the root volume group doesn't increase the root filesystem.

This is the kind of information we need to help you:

Code:

du -s /*
df -h
pvs
vgs
lvs

There are also tools that generate a graphical disk usage report, but I am not familiar enough with them to make suggestions.

sgosnell · 08-01-2020, 10:57 PM

I just got back to my Linux box, been away so I had nothing to post.

Code:

root@chromebox:/# du -s /*
0	/bin
137716	/boot
0	/dev
11752	/etc
16069988	/home
0	/initrd.img
0	/initrd.img.old
0	/lib
0	/lib32
0	/lib64
0	/libx32
16	/lost+found
585266172	/media
4	/mnt
771664	/opt
du: cannot access '/proc/2518': No such file or directory
du: cannot access '/proc/2694/task/2694/fd/3': No such file or directory
du: cannot access '/proc/2694/task/2694/fdinfo/3': No such file or directory
du: cannot access '/proc/2694/fd/3': No such file or directory
du: cannot access '/proc/2694/fdinfo/3': No such file or directory
0	/proc
2804	/root
12960	/run
0	/sbin
4	/srv
0	/sys
164	/tmp
7231112	/usr
406844	/var
0	/vmlinuz
0	/vmlinuz.old
root@chromebox:/# df -h
Filesystem                      Size  Used Avail Use% Mounted on
udev                            1.9G     0  1.9G   0% /dev
tmpfs                           389M   13M  377M   4% /run
/dev/mapper/chromebox--vg-root  143G  137G  686M 100% /
tmpfs                           1.9G   72M  1.9G   4% /dev/shm
tmpfs                           5.0M  4.0K  5.0M   1% /run/lock
tmpfs                           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sdb1                       914G  187G  681G  22% /media/stan/ExtremeSSD
/dev/sdd1                       688G  187G  466G  29% /media/stan/USB3_HDD
/dev/mapper/chromebox--vg-home   86G   16G   66G  20% /home
/dev/sda1                       236M  137M   87M  62% /boot
/dev/sde1                       233G  186G   47G  80% /media/stan/passport
tmpfs                           389M   12K  389M   1% /run/user/1000
/dev/mapper/veracrypt1          2.0G   40M  1.8G   3% /media/veracrypt1
/dev/mapper/veracrypt2          1.1G   36M  925M   4% /media/veracrypt2
root@chromebox:/# pvs
  PV                     VG           Fmt  Attr PSize    PFree
  /dev/mapper/sda5_crypt chromebox-vg lvm2 a--   118.98g    0 
  /dev/sdc1              chromebox-vg lvm2 a--  <117.32g    0 
root@chromebox:/# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  chromebox-vg   2   3   0 wz--n- 236.30g    0 
root@chromebox:/# lvs
  LV     VG           Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home   chromebox-vg -wi-ao---- <87.11g                                                    
  root   chromebox-vg -wi-ao---- 145.25g                                                    
  swap_1 chromebox-vg -wi-ao----  <3.94g

I haven't used any of the graphical tools either. Perhaps it's time to.

sgosnell · 08-01-2020, 11:55 PM

I've gone through every top-level directory under /, and the total usage is under 10GB, with /usr being the largest at ~7GB. I have no idea why the system is reporting this:
/dev/mapper/chromebox--vg-root 143G 137G 686M 100% /
/home is on a separate partition, and a different volume group in the LVM, but even that wouldn't fill the root volume.

I don't have enough experience with LVM to tell where the issue is, nor to get the pertinent settings to show. I've never used LVM before, until a fresh install of Debian 10, and it seemed like a good idea at the time. I believe I need to revise the LVM settings, but I don't want to bork the entire system and have to do a reinstall of everything. I've looked at the LVM documentation, but I'm still unclear on a lot of it. I could use some pointers.

berndbausch · 08-02-2020, 12:10 AM

This has nothing to do with LVM. There are two common causes of "hidden" disk usage: Files that are deleted but still open, and directories that contain data but were mounted over.

Given the magnitude of your problem, I'd first check for the latter. I'd unmount the various /media and /home filesystems, then check if any data is stored under those mount points.

To get rid of deleted files, a reboot would be my first approach.

sgosnell · 08-02-2020, 09:43 AM

The /media mounts are external hard drives not in the LVM group. Removing them has no effect. /home was created and formatted at the same time as the rest of the system, and the entire SSD was formatted. I've rebooted many times since the disk was reported full. I just find it hard to believe that 140+GB of files actually exist.

sgosnell · 08-02-2020, 01:34 PM

Well, I found the problem. Somehow one of the /media mount points used by an external HDD apparently got mounted to the internal SSD. I still need to find that link and remove it, but deleting the entire /media/stan/USB3_HDD directory, with the external HDD disconnected, restored the missing space. I'm not sure how that happened, but I'll eventually track down the link, since I know what it is. Thanks for the suggestions.

berndbausch · 08-02-2020, 01:56 PM

Quote:

Originally Posted by sgosnell

The /media mounts are external hard drives not in the LVM group.

This has nothing to do with LVM.

Quote:

Removing them has no effect.

You mean /media is empty after unmounting them?

The lsof tool finds files that have been deleted: lsof -a +L1 /.

That's all I can currently offer.

Could it have anything to do with veracrypt (I am quite ignorant about this technology)?

berndbausch · 08-02-2020, 02:00 PM

Quote:

Originally Posted by sgosnell

Somehow one of the /media mount points used by an external HDD apparently got mounted to the internal SSD.

I would say that at some point, you mistakenly thought the USB drive was mounted and created gigabytes of data in that directory. These data were later hidden by mounting the drive.

sgosnell · 08-02-2020, 03:02 PM

Yes, the filling was likely done by rclone through a cron job. I was syncing two external drives with it.