CentOS 6 ran out of space, need to reclaim it

DannyBoyCentOS · 08-26-2019, 11:33 AM

Hello everyone,

I am having an issue here with CentOS release 6.6 (Final) that shows all of the space used up, but I can't tell where the space went.

Seemingly I am using up 100%, according to

Code:

df -h

Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_sugar-lv_root   50G   49G     0 100% /

So that's 49G out of 50G, fair enough, but when I try to see what exactly is being used (x to remove mounts):

Code:

du -xsh /*

1.8G    /usr
2.0G    /var

That's all, these are the 2 biggest directories, all others combined are under 1G.

Here is info on Disk /dev/mapper/vg_sugar-lv_root, it does show close to 50G (this is a virtual machine):

Code:

fdisk -l

Disk /dev/mapper/vg_sugar-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Node count is at 7%:

Code:

df -i

Filesystem                   Inodes  IUsed     IFree   IUse% Mounted on
/dev/mapper/vg_sugar-lv_root 3276800 200552    3076248    7% /

The system was rebooted several times, so any log files should've been deleted. I'd appreciate some tips in resolving this.

Code:

find / -xdev -type f -size +100M

returned only a single file. The problem is not in amount of large files, but that the space is hidden or used for something else, or not released.

Code:

 du -ahx / | sort -rh | head -20 
5.0G / 
2.0G /var 
1.8G /usr
1.1G /var/lib 
945M /var/lib/mysql 
909M /var/lib/mysql/sugarcrm 
759M /var/www 
745M /var/www/html 
707M /root 
672M /usr/lib64 
642M /var/www/html/sugar 
611M /usr/share 
480M /var/lib/mysql/sugarcrm/emails_text.MYD 
441M /lib 
392M /lib/modules 
228M /usr/lib

Firerat · 08-26-2019, 01:39 PM

snapshots?

maybe some script has been happily snapshotting your logical volumes every day

DannyBoyCentOS · 08-26-2019, 01:43 PM

Quote:

Originally Posted by Firerat

snapshots?

maybe some script has been happily snapshotting your logical volumes every day

Thanks for the suggestion, but how do I check for that? I mostly work with MS SQL, only occasional Linux work.

Firerat · 08-26-2019, 01:46 PM

Quote:

Originally Posted by DannyBoyCentOS

Thanks for the suggestion, but how do I check for that? I mostly work with MS SQL, only occasional Linux work.

I knew you were going to ask that

it has been a long time since I used LVM, but I'll have a look and see what I can find/recall

DannyBoyCentOS · 08-26-2019, 01:50 PM

Quote:

Originally Posted by Firerat

I knew you were going to ask that

it has been a long time since I used LVM, but I'll have a look and see what I can find/recall

Thanks! I'd appreciate it a bunch. Here is some more info:

Code:

fdisk -l

Disk /dev/vda: 90.2 GB, 90198507520 bytes
16 heads, 63 sectors/track, 174770 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000734ea

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3        1018      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2            1018      133153    66595840   8e  Linux LVM
Partition 2 does not end on cylinder boundary.

Disk /dev/mapper/vg_sugar-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_sugar-lv_swap: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_sugar-lv_home: 10.3 GB, 10276044800 bytes
255 heads, 63 sectors/track, 1249 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Firerat · 08-26-2019, 02:22 PM

I'm not really finding much

it seems that snapshots will create block devices in /dev/vg0
start by listing that

Code:

ls /dev/vg0/

note it may not be vg0,
if you are havving trouble locating the volume group in /dev

Code:

ls -l /dev/mapper/vg_sugar-lv_root

that should be a symlink , so you may get

Code:

/dev/mapper/vg_sugar-lv_root -> /dev/vg_sugar/lv_root

if you do see a whole bunch of things in /dev/vg0/ with what looks like dates or a range of numbers they are probably snapshots

it seems lvs is the best way to list snapshots

https://access.redhat.com/documentat...apshot_command

https://access.redhat.com/documentat...pshot_creation

start with

Code:

lvs

DannyBoyCentOS · 08-26-2019, 02:46 PM

Quote:

Originally Posted by Firerat

I'm not really finding much

it seems that snapshots will create block devices in /dev/vg0
start by listing that

Code:

ls /dev/vg0/

note it may not be vg0,
if you are havving trouble locating the volume group in /dev

Code:

ls -l /dev/mapper/vg_sugar-lv_root

that should be a symlink , so you may get

Code:

/dev/mapper/vg_sugar-lv_root -> /dev/vg_sugar/lv_root

if you do see a whole bunch of things in /dev/vg0/ with what looks like dates or a range of numbers they are probably snapshots

it seems lvs is the best way to list snapshots

https://access.redhat.com/documentat...apshot_command

https://access.redhat.com/documentat...pshot_creation

start with

Code:

lvs

/dev/vg0 didn't exist, here is what I have in /dev/:

Code:

ls /dev/
autofs           disk     hugepages  loop7               null    ram13   raw       systty  tty18  tty29  tty4   tty50  tty61    usbmon1  vcsa4
block            dm-0     hvc0       lp0                 nvram   ram14   root      tty     tty19  tty3   tty40  tty51  tty62    vcs      vcsa5
bsg              dm-1     input      lp1                 oldmem  ram15   rtc       tty0    tty2   tty30  tty41  tty52  tty63    vcs1     vcsa6
btrfs-control    dm-2     kmsg       lp2                 port    ram2    rtc0      tty1    tty20  tty31  tty42  tty53  tty7     vcs2     vda
bus              dvd      log        lp3                 ppp     ram3    scd0      tty10   tty21  tty32  tty43  tty54  tty8     vcs3     vda1
cdrom            fb       loop0      MAKEDEV             ptmx    ram4    sg0       tty11   tty22  tty33  tty44  tty55  tty9     vcs4     vda2
char             fb0      loop1      mapper              pts     ram5    shm       tty12   tty23  tty34  tty45  tty56  ttyS0    vcs5     vga_arbiter
console          fd       loop2      mcelog              ram0    ram6    snapshot  tty13   tty24  tty35  tty46  tty57  ttyS1    vcs6     vg_sugar
core             full     loop3      mem                 ram1    ram7    sr0       tty14   tty25  tty36  tty47  tty58  ttyS2    vcsa     zero
cpu              fuse     loop4      net                 ram10   ram8    stderr    tty15   tty26  tty37  tty48  tty59  ttyS3    vcsa1
cpu_dma_latency  hidraw0  loop5      network_latency     ram11   ram9    stdin     tty16   tty27  tty38  tty49  tty6   urandom  vcsa2
crash            hpet     loop6      network_throughput  ram12   random  stdout    tty17   tty28  tty39  tty5   tty60  usbmon0  vcsa3

Code:

ls -l /dev/mapper/vg_sugar-lv_root
lrwxrwxrwx 1 root root 7 Aug 26 11:08 /dev/mapper/vg_sugar-lv_root -> ../dm-0

I am not sure waht to make out of ../dm-0

Code:

lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg_sugar/lv_root
  LV Name                lv_root
  VG Name                vg_sugar
  LV UUID                GhYJ8y-2pGM-sSQ2-8hV9-uxAw-1O0a-U1O8DF
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/vg_sugar/lv_home
  LV Name                lv_home
  VG Name                vg_sugar
  LV UUID                wpHAH0-blhe-h3WW-kPee-KwNU-ENrY-vGFA3a
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                9.57 GiB
  Current LE             2450
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/vg_sugar/lv_swap
  LV Name                lv_swap
  VG Name                vg_sugar
  LV UUID                ugCsGI-JEiF-pIWN-uF3R-1DTb-wz8I-aOGN66
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                3.94 GiB
  Current LE             1008
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

While looking for solution, I've read that if another file system creates a partition in the OS space, that space will be used, but not reported. Do you happen to know how I can check for something like that?

I appreciate your help!

Firerat · 08-26-2019, 03:32 PM

your volume group is called vg_sugar ( very near the end of your /dev/ listing

Code:

ls /dev/vg_sugar

should show the Logical Volumes that belong in that VG

you should see lv_root lv_home and lv_swap

you may also see others , possibly ending with snap numbered simply or like a date

Code:

lvs # should list 
lvs -a # that may list more
## 
lvs -o name,lv_ancestors,lv_descendants vg_sugar

Quote:

While looking for solution, I've read that if another file system creates a partition in the OS space, that space will be used, but not reported. Do you happen to know how I can check for something like that?

not certain on that one
I do have a 'black hole' on my system, where I can't see space used by ZFS backed LXCs unless I'm root

I may have to fireup a VM, I never really did snapshots with lvm2, most of what I did with it was create LVs in various raid configs for VMs.

DannyBoyCentOS · 08-26-2019, 06:17 PM

Quote:

Originally Posted by Firerat

your volume group is called vg_sugar ( very near the end of your /dev/ listing

Code:

ls /dev/vg_sugar

should show the Logical Volumes that belong in that VG

you should see lv_root lv_home and lv_swap

you may also see others , possibly ending with snap numbered simply or like a date

Code:

lvs # should list 
lvs -a # that may list more
## 
lvs -o name,lv_ancestors,lv_descendants vg_sugar

not certain on that one
I do have a 'black hole' on my system, where I can't see space used by ZFS backed LXCs unless I'm root

I may have to fireup a VM, I never really did snapshots with lvm2, most of what I did with it was create LVs in various raid configs for VMs.

So with MySQL error 190826 11:00:55 [ERROR] Can't start server: can't create PID file: No space left on device,
and this config file:

The daemon is here: /var/lib/mysql

Code:

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
port=3306
bind-address=0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

How can I tell which partition is hosting the /var/run/ ?

Code:

ls /dev/vg_sugar
lv_home  lv_root  lv_swap

Code:

lvs #
  LV      VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg_sugar -wi-ao----  9.57g
  lv_root vg_sugar -wi-ao---- 50.00g
  lv_swap vg_sugar -wi-ao----  3.94g

Is /var/run/ on lv_home or lv_root ?

Firerat · 08-26-2019, 07:12 PM

production?

you may need to get someone onsite
a CRM being down is not ideal.

Code:

lvs -a

I think that one may give more, -a is for all not just the main ones

then again, it could be something else and not snapshots

location of /var/run ? that is in lv_root, it might even be tmpfs i.e. in memory

you only have two partitions / and /home
well, I say partitions they are actually logical volumes.

/var/run never has anything exciting in it, just tiny files which store the pids of running daemons

what is in /root

Code:

707M /root

we need some space, and root dosn't typically have files in it

also, what filesystem

Code:

df -T /

we might be able to grow the LV
add a new virtual disk to the VM, add that as a PE grow the VG and then the LV
but it has been a while since I have done that.

another option is to setup a new VM and transfer the data
what VM is it?
does it already have redundancy ?

Firerat · 08-26-2019, 07:19 PM

just had a thought

your du did you run as root?

Firerat · 08-26-2019, 07:31 PM

re-visting this

Quote:

Originally Posted by DannyBoyCentOS

While looking for solution, I've read that if another file system creates a partition in the OS space, that space will be used, but not reported. Do you happen to know how I can check for something like that?

remember I mentioned that I can't see the space consumed by lxc unless I use root

well, lxc usually lives in
/var/lib/lxc

Code:

ls /var/lib/lxc/*

if you get a bunch of Permission denied
use `du` as root

edit, might be lxd
could even be that you have some other containers , like docker I'm not sure where they live

DannyBoyCentOS · 08-26-2019, 08:00 PM

Quote:

Originally Posted by Firerat

just had a thought

your du did you run as root?

yes, I ran it as root.
I noticed some old backup files in /root and deleted 80 mbs, so /root now is 632M , but still no go -
190826 17:52:21 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190826 17:52:21 [ERROR] Can't start server: can't create PID file: No space left on device

df shows slightly more space, but still not enough

Code:

df -T /
Filesystem           Type 1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_sugar-lv_root
                     ext4  51475068 50158000         0 100% /

there is nothing resembling lx* in /var/lib/

Code:

lvs -a
  LV      VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg_sugar -wi-ao----  9.57g
  lv_root vg_sugar -wi-ao---- 50.00g
  lv_swap vg_sugar -wi-ao----  3.94g

Well, I am software developer for our company, I work with MS stuff 99% of the time. We do have outsource help, but they threw the towel in last Friday, and I am posting on Linux forums beseeching assistance. No, CRM being down is not fun, I am seeing some frowns around the office already.

I would not know how to extend that partition.

> what VM is it?
> does it already have redundancy ?
I think this is a ProxMox, and there is not even a backup.

Firerat · 08-26-2019, 09:16 PM

Quote:

Originally Posted by DannyBoyCentOS

there is nothing resembling lx* in /var/lib/
yeah, was a long shot

Quote:

Code:

lvs -a
  LV      VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg_sugar -wi-ao----  9.57g
  lv_root vg_sugar -wi-ao---- 50.00g
  lv_swap vg_sugar -wi-ao----  3.94g

yeah, I think we can give up on the idea that it is snapshots

Quote:

Well, I am software developer for our company, I work with MS stuff 99% of the time. We do have outsource help, but they threw the towel in last Friday, and I am posting on Linux forums beseeching assistance. No, CRM being down is not fun, I am seeing some frowns around the office already.

They quit? was it over this or something else?

Quote:

I would not know how to extend that partition.

it is actually quite straight forward, I noted ext4 was the filesystem

so, the trick is to add a vitual drive to the VM ( I'm hoping the host has enought space )
then we can partition it to match the first, add vdb2 as a PV ( Physical Volume ) extend the Volume Group with that new PV , and resize the LV

I'm pretty sure this is what I've used as refrence myself
https://www.systutorials.com/5621/ex...-lvm-in-linux/

Quote:

> what VM is it?
> does it already have redundancy ?
I think this is a ProxMox, and there is not even a backup.

ah, I've only use VMware , Vitualbox, qemu/kvm
these days I mostly use containers

Backup, need to think about that before the resize

how custom is the VM?
can you setup an new VM and transfer the SugarCRM data?

keep in mind that even if the resize is succesful the drive may just fill up again, do you know if it was "overtime" or "overnight"?

creating a new VM, migrating the data , firewalls, user accounts ( if any )
might actually be quicker

JJJCR · 08-26-2019, 10:52 PM

do you have any logging in place?

sometimes logging can quickly consume all space /var/log check for any culprit

or check out this link: https://www.cyberciti.biz/faq/linux-...using-find-du/

from link above: du -a /dir/ | sort -n -r | head -n 10

list top 10 largest files

to zero a file like: 480M /var/lib/mysql/sugarcrm/emails_text.MYD

but make sure you don't need the contents before doing below command.

echo "" > /var/lib/mysql/sugarcrm/emails_text.MYD

It will empty emails_text.MYD and size will be zero