Adding a new LVM partition

NotionCommotion · 04-24-2014, 08:20 PM

Quote:

Originally Posted by rknichols

Since I was the one who suggested that, I guess I have to share in the blame, but I don't see what is wrong with it.

With the system running, put that line back in /etc/fstab, perhaps changing "barrier=0" to "barrier=0,noauto" to stop the automatic mounting during boot, and then see what messages you get from a simple

Code:

mount /var/lib/mysql

At least that way you can record the error messages.

No blame to you, just me and fstab. I did as you suggested, and below are the results.

Code:

[root@devserver ~]# mount /var/lib/mysql
mount: special device /dev/mapper/VolGroup/lv_mysql does not exist
[root@devserver ~]#

Quote:

Originally Posted by Firerat

Edit: then unmount!

I didn't. You don't mean before rebooting, right?

Quote:

Originally Posted by syg00

- whoever updated fstab is responsible, not fstab.

It was strictly fstabs fault, and not the innocent individual who updated it.

Quote:

Originally Posted by syg00

- why are you trying to disable write barriers in ext3 (which doesn't enable them by default) ?.

Quote:

Originally Posted by rknichols

Really? That is contrary to the mount command manpage, which states:

"The ext3 filesystem enables write barriers by default,"

and warns against turning them off.

To the OP: If it's the "barrier=0" causing a problem in /etc/fstab, you might try the alternative "nobarrier".

A while back, I came reasons why. I posted questions at multiple forums, but never received a suitable explanation.
http://forums.mysql.com/read.php?11,...554#msg-589554
https://www.centos.org/forums/viewto...orumpost186338
http://serverfault.com/questions/518...ntos-6x-not-5x
http://forums.devshed.com/php-develo...highlight=slow

suicidaleggroll · 04-24-2014, 09:16 PM

Should that be

Code:

/dev/mapper/VolGroup-lv_mysql

to match the others?

rknichols · 04-24-2014, 09:21 PM

Quote:

Originally Posted by NotionCommotion

No blame to you, just me and fstab. I did as you suggested, and below are the results.

Code:

[root@devserver ~]# mount /var/lib/mysql
mount: special device /dev/mapper/VolGroup/lv_mysql does not exist
[root@devserver ~]#

You mixed together two alternatives. Either of these will work

Code:

/dev/VolGroup/lv_mysql
             ^ 
/dev/mapper/VolGroup-lv_mysql
                    ^

NotionCommotion · 04-24-2014, 10:11 PM

Thanks suicidaleggroll and rknichols,

So close! Now what?

Code:

[root@devserver ~]# mount /var/lib/mysql
mount: wrong fs type, bad option, bad superblock on /dev/mapper/VolGroup-lv_mysql,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

[root@devserver ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sat Apr 19 05:57:56 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=12a081eb-285e-4599-95ab-db23b70280df /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_home  /home                  ext4    defaults        1 2
/dev/mapper/VolGroup-lv_mysql /var/lib/mysql         ext3    barrier=0;noauto       1 2
#/dev/VolGroup/lv_mysql        /var/lib/mysql          ext3    barrier=0;noauto       1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@devserver ~]#

rknichols · 04-25-2014, 09:08 AM

That is saying there is no ext3 filesystem on that LV.

Stop!

Let's review what you did. Does this agree with the sequence of your actions?

service mysqld stop
lvcreate -L 100G -n lv_mysql VolGroup
mv /var/lib/mysql /var/lib/mysql.bak
mkfs -t ext3 /dev/VolGroup/lv_mysql
mkdir /var/lib/mysql
mount -t ext3 -o barrier=0 /dev/VolGroup/lv_mysql /var/lib/mysql
cp -a /var/lib/mysql.bak/ /var/lib/mysql
lvextend -l +100%FREE /dev/VolGroup/mysql
resize2fs /dev/VolGroup/lv_mysql

Most importantly, was it "cp" or "mv" that you used in step 7? If you used "mv" your data is seriously at risk now, and we have to work very carefully.

What is the output from "file -s /dev/mapper/VolGroup-lv_mysql" ?
Does "ls -l /dev/mapper" reveal any possible mis-typings of the VG-LV name?

There will be a history of the LVM actions in the /etc/lvm/archive directory. Run

Code:

grep '^description' /etc/lvm/archive/VolGroup*.vg /etc/lvm/backup/VolGroup

and see if anything looks amiss.

I've asked the moderator to merge this thread with your previous one. This whole sequence should be kept together.

NotionCommotion · 04-25-2014, 12:10 PM

Quote:

Originally Posted by rknichols

Most importantly, was it "cp" or "mv" that you used in step 7? If you used "mv" your data is seriously at risk now, and we have to work very carefully.

Definitely mv and not copy. Note that I current do not have any data I care about, and am open to reinstalling MySQL. I am 99% sure I did exactly this:

Code:

service mysqld stop
mv /var/lib/mysql /var/lib/mysql.bak

mkdir var/lib/mysql

# Mount the new partition
mount -t ext3 -o barrier=0 /dev/VolGroup/lv_mysql /var/lib/mysql

mv /var/lib/mysql.bak/* /var/lib/mysql
service mysqld start

Quote:

Originally Posted by rknichols

What is the output from "file -s /dev/mapper/VolGroup-lv_mysql" ?

Code:

[root@devserver bidjunction]# file -s /dev/mapper/VolGroup-lv_mysql
/dev/mapper/VolGroup-lv_mysql: symbolic link to `../dm-3'
[root@devserver bidjunction]#

Quote:

Originally Posted by rknichols

Does "ls -l /dev/mapper" reveal any possible mis-typings of the VG-LV name?

Code:

[root@devserver bidjunction]# ls -l /dev/mapper
total 0
crw-rw----. 1 root root 10, 58 Apr 24 14:10 control
lrwxrwxrwx. 1 root root      7 Apr 24 14:10 VolGroup-lv_home -> ../dm-2
lrwxrwxrwx. 1 root root      7 Apr 24 14:10 VolGroup-lv_mysql -> ../dm-3
lrwxrwxrwx. 1 root root      7 Apr 24 14:10 VolGroup-lv_root -> ../dm-0
lrwxrwxrwx. 1 root root      7 Apr 24 14:10 VolGroup-lv_swap -> ../dm-1
[root@devserver bidjunction]#

Quote:

Originally Posted by rknichols

There will be a history of the LVM actions in the /etc/lvm/archive directory. Run....

Code:

[root@devserver bidjunction]# grep '^description' /etc/lvm/archive/VolGroup*.vg /etc/lvm/backup/VolGroup
/etc/lvm/archive/VolGroup_00000-1943002327.vg:description = "Created *before* executing '/sbin/vgs --noheadings -o name --config 'log{command_names=0 prefix=\"  \"}''"
/etc/lvm/archive/VolGroup_00001-663506972.vg:description = "Created *before* executing 'lvreduce -L 1000G /dev/VolGroup/lv_home'"
/etc/lvm/archive/VolGroup_00002-918014128.vg:description = "Created *before* executing 'lvcreate -L 100G -n lv_mysql VolGroup'"
/etc/lvm/backup/VolGroup:description = "Created *after* executing 'lvcreate -L 100G -n lv_mysql VolGroup'"
[root@devserver bidjunction]#

I also noticed I have a funny database in MySQL called "#mysql50#lost+found", and expect it is related. I can't seem to access it.

Code:

mysql> SHOW DATABASES;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| bidjunction         |
| gitlabhq_production |
| #mysql50#lost+found |
| mysql               |
| osticket            |
| performance_schema  |
+---------------------+
7 rows in set (0.00 sec)

mysql>

Thank you kindly for your help!

rknichols · 04-25-2014, 01:01 PM

Quote:

Originally Posted by NotionCommotion

Definitely mv and not copy. Note that I current do not have any data I care about, and am open to reinstalling MySQL. I am 99% sure I did exactly this:

Code:

service mysqld stop
mv /var/lib/mysql /var/lib/mysql.bak

mkdir var/lib/mysql

# Mount the new partition
mount -t ext3 -o barrier=0 /dev/VolGroup/lv_mysql /var/lib/mysql

mv /var/lib/mysql.bak/* /var/lib/mysql
service mysqld start

Code:

[root@devserver bidjunction]# file -s /dev/mapper/VolGroup-lv_mysql
/dev/mapper/VolGroup-lv_mysql: symbolic link to `../dm-3'
[root@devserver bidjunction]#

It is odd that the find command did not dereference the symbolic link, as it should by default. Try again with

Code:

file -s -L /dev/mapper/VolGroup-lv_mysql /dev/dm-3

It really looks like the ext3 filesystem never got created, in which case the mount command would have failed right there. That would mean your mv command would have moved the data right back into the newly created /var/lib/mysql directory. (I assume the missing "/" in your mkdir command above is just a typo.) With the new partition unmounted (Of course it's not mounted -- it wouldn't mount!), what do you get from

Code:

ls -A /var/lib/mysql /var/lib/mysql.bak

NotionCommotion · 04-25-2014, 01:19 PM

Quote:

Originally Posted by rknichols

Try again with

Code:

file -s -L /dev/mapper/VolGroup-lv_mysql /dev/dm-3

Code:

[root@devserver ~]# file -s -L /dev/mapper/VolGroup-lv_mysql /dev/dm-3
/dev/mapper/VolGroup-lv_mysql: Linux rev 1.0 ext3 filesystem data (needs journal recovery) (large files)
/dev/dm-3:                     Linux rev 1.0 ext3 filesystem data (needs journal recovery) (large files)
[root@devserver ~]#

Quote:

Originally Posted by rknichols

I assume the missing "/" in your mkdir command above is just a typo.

Yep. I was 99% sure but not 100%

Quote:

Originally Posted by rknichols

It really looks like the ext3 filesystem never got created, in which case the mount command would have failed right there. That would mean your mv command would have moved the data right back into the newly created /var/lib/mysql directory. ... With the new partition unmounted (Of course it's not mounted -- it wouldn't mount!), what do you get from

Code:

ls -A /var/lib/mysql /var/lib/mysql.bak

Code:

[root@devserver ~]# ls -A /var/lib/mysql /var/lib/mysql.bak
/var/lib/mysql:
bidjunction          ib_logfile0  mysql       performance_schema
gitlabhq_production  ib_logfile1  mysql.sock  website_tracker
ibdata1              lost+found   osticket

/var/lib/mysql.bak:
ibdata1  ib_logfile0  ib_logfile1  mysql  performance_schema  test
[root@devserver ~]#

Note that I believe I tried mounting/moving before turning off MySQL, and /var/lib/mysql was automatically created (confused me for a while). Don't know if that could have been part of the cause.

Thanks again

rknichols · 04-25-2014, 02:02 PM

You do have an ext3 filesystem now on /dev/mapper/VolGroup-lv_mysql, so it should mount. In fact, it might even be mounted right now -- hence the "needs journal recovery". See what "df /var/lib/mysql" shows as the device. If that non-empty directory is still part of lv_root, then you have files created on what should be a mount point directory. You will have to figure out whether it's those files or the ones in the lv_mysql device that are the ones you want to keep and get rid of the others.