LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-11-2016, 11:00 AM   #16
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192

You have damaged the xfs filesystem, and recovery attempts may fail. The fatal step was the resizing of the expanded root filesystem ("lvextend -r -l+100%FREE /dev/mapper/centos_sfvm03-root"). That has written some filesystem metadata into the new space, essentially scribbling garbage over parts of the old home filesystem.

Usually, the first step in recovery would be to undo the resizing of the root LV and filesystem, putting it back to its original 12800 extents. Unfortunately, your root filesystem appears to be xfs, and there is no way to shrink an xfs filesystem in place. The only way to do that is to back up the entire root filesystem elsewhere. I was going to suggest using pvmove to do that, but that would conflict with restoring the LVM configuration with vgcfgrestore and just make things even more complicated. So however you do it, make sure that the backup is stored somewhere not in the centos_sfvm03 volume group.

Once you have the root filesystem backed up, you can use vgcfgrestore to put the LVM configuration bck the way it was. Look in the most recent centos_sfvm03_*.vg files in /etc/lvm/archive (they are ASCII files) for one that contains the description, "Created *before* executing 'lvreduce -L 70G /dev/mapper/centos_sfvm03-home'", and run
Code:
vgcfgrestore -f /path/to/that/file centos_sfvm03
You can include the "--test" and "--verbose" options if you like to see what it will do.

Then, you can make a new xfs filesystem in /dev/mapper/centos_sfvm03-root and restore that root filesystem from its backup.

Finally, you can attempt your xfs_repair on /dev/mapper/centos_sfvm03-home. Due to the "scribbling" above, there is no guarantee that it will be successful, but that's your best shot. If it fails and you can't recover you data, ... well ..., running a server for "200-300 websites" with no backups leaves you vulnerable to that.
 
1 members found this post helpful.
Old 11-11-2016, 02:15 PM   #17
sahruk
LQ Newbie
 
Registered: Nov 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
Usually, the first step in recovery would be to undo the resizing of the root LV and filesystem, putting it back to its original 12800 extents. Unfortunately, your root filesystem appears to be xfs, and there is no way to shrink an xfs filesystem in place.

Hello rknichols, thanks so much for your detailed answer.
I'm not sure about type of file system of root. but now I tried
Code:
lvreduce -t -l 12800 /dev/mapper/centos_sfvm03-root
and got "size of logical volume centos-sfvm03/root changed from 77.51 GiB (19842 extents to 50.00 GiB (12800 extents)." message in test mode.

And I didn't get the part which you said I need to take backup from root to somewhere. What is outcome out of that? and if I could resize root, do I need to go with backup etc.?

On the other hand, the .vg file just before changes is /etc/lvm/archive/centos_sfvm03_00001-1621702387.vg there are 2 more files there, one very old, and one with 2 second after which the name is centos_sfvm03_00002-1249077736.vg so in
Code:
vgcfgrestore -f /path/to/that/file centos_sfvm03
you meant path to this file, isn't that?

Again thanks for your replies which will help me solving this issue
 
Old 11-11-2016, 05:27 PM   #18
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192
Quote:
Originally Posted by sahruk View Post
Hello rknichols, thanks so much for your detailed answer.
I'm not sure about type of file system of root. but now I tried
Code:
lvreduce -t -l 12800 /dev/mapper/centos_sfvm03-root
and got "size of logical volume centos-sfvm03/root changed from 77.51 GiB (19842 extents to 50.00 GiB (12800 extents)." message in test mode.
Do NOT do that! That would damage the root filesystem as well. Any time you reduce the size of an LVM logical volume, you must shrink the filesystem, either as part of the LVM operation ("lvreduce -r ..."), or else prior to the LVM operation. Otherwise, you are left with a broken filesystem that is larger than the volume that is holding it. Since there is no way to shrink an xfs filesystem, neither option is possible.

Quote:
And I didn't get the part which you said I need to take backup from root to somewhere. What is outcome out of that? and if I could resize root, do I need to go with backup etc.?
If (and that's a very big "if") you could resize root, you wouldn't need to do the backup. You can try it. Just run "lvreduce -r -l 12800 centos_sfvm03-root", but that is going to fail if root is an xfs filesystem. Do NOT leave out the "-r".

Quote:
On the other hand, the .vg file just before changes is /etc/lvm/archive/centos_sfvm03_00001-1621702387.vg there are 2 more files there, one very old, and one with 2 second after which the name is centos_sfvm03_00002-1249077736.vg so in
Code:
vgcfgrestore -f /path/to/that/file centos_sfvm03
you meant path to this file, isn't that?
Yes. There indeed should be a later file with the description "Created *before* executing 'lvextend -r -l+100%FREE /dev/mapper/centos_sfvm03-root'". That first file is the one you want. It will restore the volume group back to the way it was before you did any of this. That will give you a home filesystem that fits properly in its container. Now the issue is whatever unknown damage was done by the metadata written over that filesystem when you expanded the root filesystem. Running xfs_repair will either fix that or give you the bad news.

Last edited by rknichols; 11-11-2016 at 05:29 PM.
 
1 members found this post helpful.
Old 11-11-2016, 11:08 PM   #19
sahruk
LQ Newbie
 
Registered: Nov 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
If (and that's a very big "if") you could resize root, you wouldn't need to do the backup. You can try it. Just run "lvreduce -r -l 12800 centos_sfvm03-root", but that is going to fail if root is an xfs filesystem. Do NOT leave out the "-r".
Once again thanks for your reply.
Is there any way to find out type of partitions at the moment while home damaged?
 
Old 11-12-2016, 01:06 AM   #20
sahruk
LQ Newbie
 
Registered: Nov 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
I copied VM Disk to somewhere else to deal with data recovery. I used 2 different software for data recovery. Geeksnerd XFS data recovery, which found even old sized partition on hard and I tried to scan files, almost 100% are found recoverable, and second software is UFS Explorer which showing me disks with new partition but I can travel live over home partition and almost 100% are recoverable. Just curious how fast UFS Explorer could find files and folder structures lively while partition is damaged without even scanning! Just curious why it's not possible to heal partition while a software like UFS Explorer makes traveling lively possible?
 
Old 11-12-2016, 08:13 AM   #21
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192
Quote:
Originally Posted by sahruk View Post
Is there any way to find out type of partitions at the moment while home damaged?
Run "lsblk -f" to see the filesystem types on all existing devices. You can also run "file -sL /dev/mapper/centos_sfvm03-root".
 
Old 11-12-2016, 08:54 AM   #22
JeremyBoden
Senior Member
 
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,937

Rep: Reputation: 498Reputation: 498Reputation: 498Reputation: 498Reputation: 498
I'm wondering why you are using a non-resizable file system with LVM (one benefit of which is the simple resizing of volumes)?
 
Old 11-12-2016, 09:58 AM   #23
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (15 current), Slack15, Ubuntu studio, MX Linux, FreeBSD 13.1, WIn10
Posts: 10,261

Rep: Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207Reputation: 2207
Quote:
Originally Posted by JeremyBoden View Post
I'm wondering why you are using a non-resizable file system with LVM (one benefit of which is the simple resizing of volumes)?
his answer from an earlier post.
Quote:
..I'm just managing the server, not the first creator.
 
Old 11-12-2016, 11:00 AM   #24
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192
Quote:
Originally Posted by JeremyBoden View Post
I'm wondering why you are using a non-resizable file system with LVM (one benefit of which is the simple resizing of volumes)?
The default filesystem type for RHEL/CentOS 7 is xfs, and the default storage arrangement is LVM, so blame Red Hat. There is no problem with expanding an xfs filesystem. It's just shrinking that can't be done.

Last edited by rknichols; 11-12-2016 at 11:01 AM.
 
1 members found this post helpful.
Old 11-12-2016, 05:59 PM   #25
sahruk
LQ Newbie
 
Registered: Nov 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
The default filesystem type for RHEL/CentOS 7 is xfs, and the default storage arrangement is LVM, so blame Red Hat. There is no problem with expanding an xfs filesystem. It's just shrinking that can't be done.
Now that I'm almost done with recovering data, would you please instruct me how exactly should I expand?
Server is running as a virtual machine (VMWare) and I have ability to extend hard disk, this makes a new unformated raw partition. Shall I first create a physical partition? then add it to Volume Group or what?
 
Old 11-12-2016, 07:30 PM   #26
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192
Making a virtual disk larger typically gives you unallocated space, not an "unformatted raw partition." Use any partitioning tool to make a new partition in that space, add it to the existing LVM volume group, and then enlarge whichever LVs need more space. You can do a pvcreate on the new partition prior to the vgextend, but a pvcreate will be done automatically (with default parameters) if you don't. Use the "-r" (--resizefs) option on the lvextend command as you did before so that the filesystem gets enalrged as well.

If your added space is on a new virtual disk rather than enlarging the present one, then it is your choice whether to partition it or just use the entire unpartitioned device as an LVM physical volume. I always recommend making a partition first. That is so that someone doesn't come along later, see that a partitioning tool reports the drive as having no partition table, and conclude that the drive is unused.
 
1 members found this post helpful.
Old 11-13-2016, 06:54 AM   #27
sahruk
LQ Newbie
 
Registered: Nov 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Any one has any idea about issue below?
Code:
[root@sfvm08 mapper]# ls
centos_sfvm03-home  centos_sfvm03-root  centos_sfvm03-swap  control
[root@sfvm08 mapper]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/centos_sfvm03-root   78G   69G  9.1G  89% /
/dev/loop0                      2.2G   29M  2.1G   2% /tmp
home and swap doesn't show up in list above, but I already mounted it and have access to /home !!
 
Old 11-13-2016, 08:42 AM   #28
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192
Let's see the output from "lsblk" and "df /home".

Last edited by rknichols; 11-13-2016 at 08:47 AM.
 
Old 11-13-2016, 09:07 AM   #29
sahruk
LQ Newbie
 
Registered: Nov 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
Let's see the output from "lsblk" and "df /home".
Code:
[root@sfvm08 mapper]# lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0                      2:0    1     4K  0 disk
sda                      8:0    0   200G  0 disk
├─sda1                   8:1    0   500M  0 part
└─sda2                   8:2    0 149.5G  0 part
  ├─centos_sfvm03-root 253:0    0  77.5G  0 lvm  /
  ├─centos_sfvm03-swap 253:1    0     2G  0 lvm  [SWAP]
  └─centos_sfvm03-home 253:2    0    50G  0 lvm  /home
sr0                     11:0    1   603M  0 rom
loop0                    7:0    0   2.3G  0 loop /tmp
 
Old 11-13-2016, 09:33 AM   #30
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,716

Rep: Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192Reputation: 2192
I have no idea why /home isn't showing up in the df output. Swap devices aren't mounted like regular filesystems, and thus will never be reported by df.

Again, what happens when you run "df /home" specifically asking about that filesystem?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] after lvreduce reboot now can't read superblock scriptkiddy Linux - Newbie 12 11-11-2016 07:29 AM
can i mount my home partition read only? F,Dillon@NYC Linux - Newbie 3 07-20-2015 05:43 PM
Partition doesn't mount after resizing dim3dro1 Linux - General 5 11-25-2008 02:19 AM
mount: server:/home can't read superblock phil.d.g Linux - General 0 05-11-2006 04:03 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:11 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration