LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   home partition doesn't mount after resizing using lvreduce - can't read superblock (https://www.linuxquestions.org/questions/linux-newbie-8/home-partition-doesnt-mount-after-resizing-using-lvreduce-cant-read-superblock-4175593319/)

rknichols 11-11-2016 11:00 AM

You have damaged the xfs filesystem, and recovery attempts may fail. The fatal step was the resizing of the expanded root filesystem ("lvextend -r -l+100%FREE /dev/mapper/centos_sfvm03-root"). That has written some filesystem metadata into the new space, essentially scribbling garbage over parts of the old home filesystem.

Usually, the first step in recovery would be to undo the resizing of the root LV and filesystem, putting it back to its original 12800 extents. Unfortunately, your root filesystem appears to be xfs, and there is no way to shrink an xfs filesystem in place. The only way to do that is to back up the entire root filesystem elsewhere. I was going to suggest using pvmove to do that, but that would conflict with restoring the LVM configuration with vgcfgrestore and just make things even more complicated. So however you do it, make sure that the backup is stored somewhere not in the centos_sfvm03 volume group.

Once you have the root filesystem backed up, you can use vgcfgrestore to put the LVM configuration bck the way it was. Look in the most recent centos_sfvm03_*.vg files in /etc/lvm/archive (they are ASCII files) for one that contains the description, "Created *before* executing 'lvreduce -L 70G /dev/mapper/centos_sfvm03-home'", and run
Code:

vgcfgrestore -f /path/to/that/file centos_sfvm03
You can include the "--test" and "--verbose" options if you like to see what it will do.

Then, you can make a new xfs filesystem in /dev/mapper/centos_sfvm03-root and restore that root filesystem from its backup.

Finally, you can attempt your xfs_repair on /dev/mapper/centos_sfvm03-home. Due to the "scribbling" above, there is no guarantee that it will be successful, but that's your best shot. If it fails and you can't recover you data, ... well ..., running a server for "200-300 websites" with no backups leaves you vulnerable to that.

sahruk 11-11-2016 02:15 PM

Quote:

Originally Posted by rknichols (Post 5629423)
Usually, the first step in recovery would be to undo the resizing of the root LV and filesystem, putting it back to its original 12800 extents. Unfortunately, your root filesystem appears to be xfs, and there is no way to shrink an xfs filesystem in place.


Hello rknichols, thanks so much for your detailed answer.
I'm not sure about type of file system of root. but now I tried
Code:

lvreduce -t -l 12800 /dev/mapper/centos_sfvm03-root
and got "size of logical volume centos-sfvm03/root changed from 77.51 GiB (19842 extents to 50.00 GiB (12800 extents)." message in test mode.

And I didn't get the part which you said I need to take backup from root to somewhere. What is outcome out of that? and if I could resize root, do I need to go with backup etc.?

On the other hand, the .vg file just before changes is /etc/lvm/archive/centos_sfvm03_00001-1621702387.vg there are 2 more files there, one very old, and one with 2 second after which the name is centos_sfvm03_00002-1249077736.vg so in
Code:

vgcfgrestore -f /path/to/that/file centos_sfvm03
you meant path to this file, isn't that?

Again thanks for your replies which will help me solving this issue

rknichols 11-11-2016 05:27 PM

Quote:

Originally Posted by sahruk (Post 5629505)
Hello rknichols, thanks so much for your detailed answer.
I'm not sure about type of file system of root. but now I tried
Code:

lvreduce -t -l 12800 /dev/mapper/centos_sfvm03-root
and got "size of logical volume centos-sfvm03/root changed from 77.51 GiB (19842 extents to 50.00 GiB (12800 extents)." message in test mode.

Do NOT do that! That would damage the root filesystem as well. Any time you reduce the size of an LVM logical volume, you must shrink the filesystem, either as part of the LVM operation ("lvreduce -r ..."), or else prior to the LVM operation. Otherwise, you are left with a broken filesystem that is larger than the volume that is holding it. Since there is no way to shrink an xfs filesystem, neither option is possible.

Quote:

And I didn't get the part which you said I need to take backup from root to somewhere. What is outcome out of that? and if I could resize root, do I need to go with backup etc.?
If (and that's a very big "if") you could resize root, you wouldn't need to do the backup. You can try it. Just run "lvreduce -r -l 12800 centos_sfvm03-root", but that is going to fail if root is an xfs filesystem. Do NOT leave out the "-r".

Quote:

On the other hand, the .vg file just before changes is /etc/lvm/archive/centos_sfvm03_00001-1621702387.vg there are 2 more files there, one very old, and one with 2 second after which the name is centos_sfvm03_00002-1249077736.vg so in
Code:

vgcfgrestore -f /path/to/that/file centos_sfvm03
you meant path to this file, isn't that?
Yes. There indeed should be a later file with the description "Created *before* executing 'lvextend -r -l+100%FREE /dev/mapper/centos_sfvm03-root'". That first file is the one you want. It will restore the volume group back to the way it was before you did any of this. That will give you a home filesystem that fits properly in its container. Now the issue is whatever unknown damage was done by the metadata written over that filesystem when you expanded the root filesystem. Running xfs_repair will either fix that or give you the bad news.

sahruk 11-11-2016 11:08 PM

Quote:

Originally Posted by rknichols (Post 5629551)
If (and that's a very big "if") you could resize root, you wouldn't need to do the backup. You can try it. Just run "lvreduce -r -l 12800 centos_sfvm03-root", but that is going to fail if root is an xfs filesystem. Do NOT leave out the "-r".

Once again thanks for your reply.
Is there any way to find out type of partitions at the moment while home damaged?

sahruk 11-12-2016 01:06 AM

I copied VM Disk to somewhere else to deal with data recovery. I used 2 different software for data recovery. Geeksnerd XFS data recovery, which found even old sized partition on hard and I tried to scan files, almost 100% are found recoverable, and second software is UFS Explorer which showing me disks with new partition but I can travel live over home partition and almost 100% are recoverable. Just curious how fast UFS Explorer could find files and folder structures lively while partition is damaged without even scanning! Just curious why it's not possible to heal partition while a software like UFS Explorer makes traveling lively possible?

rknichols 11-12-2016 08:13 AM

Quote:

Originally Posted by sahruk (Post 5629620)
Is there any way to find out type of partitions at the moment while home damaged?

Run "lsblk -f" to see the filesystem types on all existing devices. You can also run "file -sL /dev/mapper/centos_sfvm03-root".

JeremyBoden 11-12-2016 08:54 AM

I'm wondering why you are using a non-resizable file system with LVM (one benefit of which is the simple resizing of volumes)?

BW-userx 11-12-2016 09:58 AM

Quote:

Originally Posted by JeremyBoden (Post 5629729)
I'm wondering why you are using a non-resizable file system with LVM (one benefit of which is the simple resizing of volumes)?

his answer from an earlier post.
Quote:

..I'm just managing the server, not the first creator.

rknichols 11-12-2016 11:00 AM

Quote:

Originally Posted by JeremyBoden (Post 5629729)
I'm wondering why you are using a non-resizable file system with LVM (one benefit of which is the simple resizing of volumes)?

The default filesystem type for RHEL/CentOS 7 is xfs, and the default storage arrangement is LVM, so blame Red Hat. There is no problem with expanding an xfs filesystem. It's just shrinking that can't be done.

sahruk 11-12-2016 05:59 PM

Quote:

Originally Posted by rknichols (Post 5629785)
The default filesystem type for RHEL/CentOS 7 is xfs, and the default storage arrangement is LVM, so blame Red Hat. There is no problem with expanding an xfs filesystem. It's just shrinking that can't be done.

Now that I'm almost done with recovering data, would you please instruct me how exactly should I expand?
Server is running as a virtual machine (VMWare) and I have ability to extend hard disk, this makes a new unformated raw partition. Shall I first create a physical partition? then add it to Volume Group or what?

rknichols 11-12-2016 07:30 PM

Making a virtual disk larger typically gives you unallocated space, not an "unformatted raw partition." Use any partitioning tool to make a new partition in that space, add it to the existing LVM volume group, and then enlarge whichever LVs need more space. You can do a pvcreate on the new partition prior to the vgextend, but a pvcreate will be done automatically (with default parameters) if you don't. Use the "-r" (--resizefs) option on the lvextend command as you did before so that the filesystem gets enalrged as well.

If your added space is on a new virtual disk rather than enlarging the present one, then it is your choice whether to partition it or just use the entire unpartitioned device as an LVM physical volume. I always recommend making a partition first. That is so that someone doesn't come along later, see that a partitioning tool reports the drive as having no partition table, and conclude that the drive is unused.

sahruk 11-13-2016 06:54 AM

Any one has any idea about issue below?
Code:

[root@sfvm08 mapper]# ls
centos_sfvm03-home  centos_sfvm03-root  centos_sfvm03-swap  control
[root@sfvm08 mapper]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/centos_sfvm03-root  78G  69G  9.1G  89% /
/dev/loop0                      2.2G  29M  2.1G  2% /tmp

home and swap doesn't show up in list above, but I already mounted it and have access to /home !!

rknichols 11-13-2016 08:42 AM

Let's see the output from "lsblk" and "df /home".

sahruk 11-13-2016 09:07 AM

Quote:

Originally Posted by rknichols (Post 5630117)
Let's see the output from "lsblk" and "df /home".

Code:

[root@sfvm08 mapper]# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                      2:0    1    4K  0 disk
sda                      8:0    0  200G  0 disk
├─sda1                  8:1    0  500M  0 part
└─sda2                  8:2    0 149.5G  0 part
  ├─centos_sfvm03-root 253:0    0  77.5G  0 lvm  /
  ├─centos_sfvm03-swap 253:1    0    2G  0 lvm  [SWAP]
  └─centos_sfvm03-home 253:2    0    50G  0 lvm  /home
sr0                    11:0    1  603M  0 rom
loop0                    7:0    0  2.3G  0 loop /tmp


rknichols 11-13-2016 09:33 AM

I have no idea why /home isn't showing up in the df output. Swap devices aren't mounted like regular filesystems, and thus will never be reported by df.

Again, what happens when you run "df /home" specifically asking about that filesystem?


All times are GMT -5. The time now is 04:25 PM.