CentOS 6 ran out of space, need to reclaim it

Firerat · 08-27-2019, 12:12 AM

the odd thing is, df reports full but du does not indcate anything near filling 50gb

I guess

Code:

du --apparent-size

could show something else

a recovery option
create new VitualDrive and attach to VM
partition

Create new VG
create new LVs
put ext4fs on the new LVs
create temp. mount points

Code:

mkdir /tmp/lv_root
mkdir /tmp/lv_newroot

mount -o ro /dev/mapper/vg_sugar-lv_root /tmp/lv_root
mount /dev/mapper/vg_newsugar-lv_newroot /tmp/lv_newroot

the tmp mount of the original shouldn't have all the generated files in /dev/
check contents of /tmp/lv_newroot/dev/
home/ should also be empty

Code:

cp -a tmp/lv_root/* /tmp/lv_newroot/
## add -v if you want to see progress
## you might want to use rsync instead of cp

need to edit /tmp/lv_newroot/etc/fstab
and change the original / mount to the lv_newroot

Code:

umount /tmp/lv_root

either run update-grub or manually edit grub.cfg
if it didn't take long might aswell repeat with lv_home, only no need for the temp mount ( we don't have dev or mounted fs to worry about )
again update the new fstab to point to the lv_newhome

unmount new LVs
reboot to the "new"

currently booting off the grub on the original vda
so install grub to the new VirtualDrive
update-grub or manual edit grub.config

the original VirtualDrive can then be removed from the VM
update grub again

still need to figure out what went wrong to prevent it from happening again

it may also be worth splitting up the volumes

/home
/var/log
/var/cache
/var/www
/var/lib/mysql/sugarcrm

easier to backup/migrate and monitor disk consumption

DannyBoyCentOS · 08-27-2019, 01:48 AM

Quote:

Originally Posted by JJJCR

do you have any logging in place?

sometimes logging can quickly consume all space /var/log check for any culprit

or check out this link: https://www.cyberciti.biz/faq/linux-...using-find-du/

from link above: du -a /dir/ | sort -n -r | head -n 10

list top 10 largest files

to zero a file like: 480M /var/lib/mysql/sugarcrm/emails_text.MYD

but make sure you don't need the contents before doing below command.

echo "" > /var/lib/mysql/sugarcrm/emails_text.MYD

It will empty emails_text.MYD and size will be zero

du /var/log
84744 /var/log

Thank you for this suggestion. Today I've removed 500Mb of old files and backups, and that wasn't enough to start MySQL. I like your idea of truncating /var/lib/mysql/sugarcrm/emails_text.MYD , and I will take a copy to \mnt\backup prior to doing that, but I am not sure it will release the needed space. I might try this tomorrow.

DannyBoyCentOS · 08-27-2019, 01:58 AM

Quote:

Originally Posted by Firerat

the odd thing is, df reports full but du does not indcate anything near filling 50gb

I guess

Code:

du --apparent-size

could show something else

a recovery option
create new VitualDrive and attach to VM
partition

Create new VG
create new LVs
put ext4fs on the new LVs
create temp. mount points

Code:

mkdir /tmp/lv_root
mkdir /tmp/lv_newroot

mount -o ro /dev/mapper/vg_sugar-lv_root /tmp/lv_root
mount /dev/mapper/vg_newsugar-lv_newroot /tmp/lv_newroot

the tmp mount of the original shouldn't have all the generated files in /dev/
check contents of /tmp/lv_newroot/dev/
home/ should also be empty

Code:

cp -a tmp/lv_root/* /tmp/lv_newroot/
## add -v if you want to see progress
## you might want to use rsync instead of cp

need to edit /tmp/lv_newroot/etc/fstab
and change the original / mount to the lv_newroot

Code:

umount /tmp/lv_root

either run update-grub or manually edit grub.cfg
if it didn't take long might aswell repeat with lv_home, only no need for the temp mount ( we don't have dev or mounted fs to worry about )
again update the new fstab to point to the lv_newhome

unmount new LVs
reboot to the "new"

currently booting off the grub on the original vda
so install grub to the new VirtualDrive
update-grub or manual edit grub.config

the original VirtualDrive can then be removed from the VM
update grub again

still need to figure out what went wrong to prevent it from happening again

it may also be worth splitting up the volumes

/home
/var/log
/var/cache
/var/www
/var/lib/mysql/sugarcrm

easier to backup/migrate and monitor disk consumption

Code:

du --apparent-size
339649  .

Oh, thanks for all those suggestions, but they are way, WAY over my head. I doubt I will be able to perform those steps. Fresh copy to a different mount sure sounds nice though! Thank you for your assistance, I'll try to look at it again tomorrow morning.

Firerat · 08-27-2019, 02:11 AM

Quote:

Originally Posted by DannyBoyCentOS

Oh, thanks for all those suggestions, but they are way, WAY over my head. I doubt I will be able to perform those steps. Fresh copy to a different mount sure sounds nice though! Thank you for your assistance, I'll try to look at it again tomorrow morning.

I'll clean up the instructions and include step by step commands

DannyBoyCentOS · 08-27-2019, 10:02 AM

Quote:

Originally Posted by Firerat

I'll clean up the instructions and include step by step commands

Hold up, no need - I found the missing space. Well, not myself, but it had something to do with /mnt/backup/. I will post a link to the solution soon! Thank you for your help!

https://www.unix.com/unix-for-beginn...reclaim-3.html

Mr. Firerat, thank you for your time and help. I am somewhat relieved I didn't have to do something as intricate as extending ext4 partition, I am not sure if there was enough space available.

Somebody created a local directory /mnt/backup, filled up with 45G worth of backups, and then a mount was created with the same name of /mnt/backup, and it was hiding the content of that directory. Once unmounted, /mnt/backup showed the 45G size. I've cleaned up 2 Gs, and MySQL is now running. I am going to delete some server images from 2016 later, after I get to the office

Thanks again, man!

Firerat · 08-27-2019, 12:11 PM

Quote:

Originally Posted by DannyBoyCentOS

Hold up, no need - I found the missing space. Well, not myself, but it had something to do with /mnt/backup/. I will post a link to the solution soon! Thank you for your help!

https://www.unix.com/unix-for-beginn...reclaim-3.html

Mr. Firerat, thank you for your time and help. I am somewhat relieved I didn't have to do something as intricate as extending ext4 partition, I am not sure if there was enough space available.

Somebody created a local directory /mnt/backup, filled up with 45G worth of backups, and then a mount was created with the same name of /mnt/backup, and it was hiding the content of that directory. Once unmounted, /mnt/backup showed the 45G size. I've cleaned up 2 Gs, and MySQL is now running. I am going to delete some server images from 2016 later, after I get to the office

Thanks again, man!

I now understand why ZFS refuses to mount over a populated dir by default

Glad everything is ok now

Note to self:

Code:

umount -a

that would have unmounted /mnt/backup, or at least said it was busy
then a du would show the disk use

I've had some fun with CentOS 6.10 ( I couldn't get 6.6 )

I wanted to double mount root so I could copy over the default /dev/ structure
But Cent refuses to mount it a second time
I've *almost* got it booting , not sure what I'm missing yet.

rknichols · 08-27-2019, 01:04 PM

Quote:

Originally Posted by Firerat

:I wanted to double mount root so I could copy over the default /dev/ structure
But Cent refuses to mount it a second time
I've *almost* got it booting , not sure what I'm missing yet.

You can look under an active mount point directory by using a bind mount:

Code:

mkdir /tmp/tmproot
mount --bind / /mnt/tmproot

Now you can look under /tmp/tmproot and see the root filesystem as it was before anything was mounted. But, you will find that /dev directory empty. The devtmpfs pseudo-filesystem gets mounted there during early boot, prior to the switch to the real root filesystem. If you are having device problems early in the boot sequence, that is an issue in the initramfs.

JJJCR · 08-28-2019, 01:56 AM

In addition to reply of Rknichols to Firerat, this link might help about mount --bind: https://support.rackspace.com/how-to...unts-in-linux/

Site says:

Quote:

Bind mounts in Linux® enable you to mount an already-mounted file system to another location within the file system. Generally, bind mounts are used when restricting the access of specified users to designated parts of a website by replicating the website’s directory into a jailed user’s home directory.

Configure a bind mount
This section provides steps for how to grant a user access to a directory by using bind mounting to bind the directory to that user’s home directory.

Configure a bind mount by using the following command:

mount --bind /path/to/domain /path/to/home/directory

Firerat · 08-28-2019, 05:47 AM

yeah, thanks guys

regards the bind mount, I wanted to see what was in /dev prior to any mounts

but as it turns out CentOS 6.10 doesn't seem to have anything in /dev

I'll maybe unpack the initrd to see what it is doing
but I'm in no great rush at the moment, might be something I return to some rainy afternoon

MadeInGermany · 08-28-2019, 06:43 AM

Quote:

Originally Posted by rknichols

You can look under an active mount point directory by using a bind mount:

Code:

mkdir /tmp/tmproot
mount --bind / /mnt/tmproot

Now you can look under /tmp/tmproot and see the root filesystem as it was before anything was mounted. But, you will find that /dev directory empty. The devtmpfs pseudo-filesystem gets mounted there during early boot, prior to the switch to the real root filesystem. If you are having device problems early in the boot sequence, that is an issue in the initramfs.

Sorry, that does not work.
The bind mount (and the lofs mount in Solaris) work as a reflector, or "alias mount".
That means the primary mount appears a second time under the bind mount, and all access is redirected to the primary mount. Therefore, any sub mounts are reflected on the bind mount.

Maybe there is a further option to turn off the reflector behavior?

But an NFS share will work. The NFS-mounted file system will not reflect the sub mounts, i.e. will show the "covered" files.

Firerat · 08-28-2019, 07:03 AM

Quote:

Originally Posted by MadeInGermany

Sorry, that does not work.
The bind mount (and the lofs mount in Solaris) work as a reflector, or "alias mount".
That means the primary mount appears a second time under the bind mount, and all access is redirected to the primary mount. Therefore, any sub mounts are reflected on the bind mount.

Maybe there is a further option to turn off the reflector behavior?

But an NFS share will work. The NFS-mounted file system will not reflect the sub mounts, i.e. will show the "covered" files.

in linux you can turn it on with -rbind

going offtopic now ,

thanks all but no need to worry OP is solved and I'm just trying to do somethng that wouldn't be too much of a problem had I not constrained myself.

rknichols · 08-28-2019, 08:19 AM

Quote:

Originally Posted by rknichols

You can look under an active mount point directory by using a bind mount:

Code:

mkdir /tmp/tmproot
mount --bind / /mnt/tmproot

Now you can look under /tmp/tmproot and see the root filesystem as it was before anything was mounted. But, you will find that /dev directory empty. The devtmpfs pseudo-filesystem gets mounted there during early boot, prior to the switch to the real root filesystem. If you are having device problems early in the boot sequence, that is an issue in the initramfs.

Quote:

Originally Posted by MadeInGermany

Sorry, that does not work.
The bind mount (and the lofs mount in Solaris) work as a reflector, or "alias mount".
That means the primary mount appears a second time under the bind mount, and all access is redirected to the primary mount. Therefore, any sub mounts are reflected on the bind mount.

Maybe there is a further option to turn off the reflector behavior?

It most certainly does work. I use it often. The option to include submounts is off by default. You need to use the "--rbind" option instead of "--bind" to include the submounts. Here is the excerpt from the manpage:

The bind mounts.
...
This call attaches only (part of) a single filesystem, not possible submounts. The entire file hierarchy including submounts is attached a second place using
mount --rbind olddir newdir
or shortoption
mount -R olddir newdir

MadeInGermany · 08-28-2019, 12:47 PM

You are right!
My test was wrong somehow.
I have repeated it, and now the sub mounts are not present in the bind mount.
And thanks for showing the --rbind option!