Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I did a update on ubuntu server to latest versions (have done that before), but
this time the machine wouldn't boot and I had to connect a monitor to it.
The boot process was stuck - it couldn't mount LVM drive(s) and was waiting to either skip or manually mount.
Is there anything I can do, the disk(s) have been showing signs of
bad sectors or other problems for some time now, but usually some kind of fsck would keep the thing working.
Code:
lvdisplay
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
--- Logical volume ---
LV Path /dev/abram_vg/abram_disk
LV Name abram_disk
VG Name abram_vg
LV UUID y6gUb5-FWUt-y16S-e7J1-5p9O-geNW-00pWB9
LV Write Access read/write
LV Creation host, time ,
LV Status NOT available
LV Size 3.64 TiB
Current LE 953864
Segments 3
Allocation inherit
Read ahead sectors auto
Code:
fsck /dev/abram_vg/abram_disk
fsck from util-linux 2.20.1
e2fsck 1.42.8 (20-Jun-2013)
fsck.ext2: No such file or directory while trying to open /dev/abram_vg/abram_disk
Possibly non-existent device?
BTW - is there a way to configure ubuntu to continue booting no matte what, ignoring all the errors?
So I assume these are data drives? Without knowing your exact configuration it is a bit hard to say. If they are in the /etc/fstab file you can add the no auto option.
So it looks the the drive is failing and ignoring errors could lead to losing all of the data on the drive if it is not safely backed up. If the VG spans multiple drives then you could possibly lose everything.
1 disk that has: swap, main partition for ubuntu and one more for a place where I put programs. It's a small disk
of about 80 GB.
LVM has 3 I think, about 4 T together.
Code:
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=0fd1729b-ed72-4699-9803-8155ea4f67b0 / ext4 errors=remount-ro 0 1
# /opt was on /dev/sda5 during installation
UUID=4876b2ef-d9f9-4434-b1de-b30573bcb52d /opt ext4 defaults 0 2
# swap was on /dev/sda6 during installation
UUID=1e276083-e710-4240-a998-1e288500dbac none swap sw 0 0
#abram disk:
UUID=5ed42986-7f63-4437-8aa0-421cbae70b3b /opt/abramdisk ext3
The 3 disks that make LVM are used for storing movies, music and such for everybody on LAN. It won't be
a huge tragedy if we loose everything that's there...
I'd like to find a way to mount the LVM disk so that I can try to fix it with fsck - it helped before, but
I was never in a situation where I couldn't even mount the LVM and access it.
The system is booted and running, it's our router to the net.
During the boot process it stopped at the point when it tried to mount the LVM, waited for key press.
I know I can comet out the fstab part and it will ignore and not even try to mount that. Although, I
will probably still get A LOT of this error msgs:
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
PV /dev/sdd5 VG abram_vg lvm2 [1.36 TiB / 0 free]
PV /dev/sdb1 VG abram_vg lvm2 [931.51 GiB / 0 free]
PV unknown device VG abram_vg lvm2 [1.36 TiB / 0 free]
Total: 3 [3.64 TiB] / in use: 3 [3.64 TiB] / in no VG: 0 [0 ]
vgscan
Code:
Reading all physical volumes. This may take a while...
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
Found volume group "abram_vg" using metadata type lvm2
vgchange -a y
Code:
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
Refusing activation of partial LV abram_disk. Use --partial to override.
0 logical volume(s) in volume group "abram_vg" now active
The syslog message mentions "device error" and not "media error", which suggests that the problem is a lot worse than just a few bad sectors on the disk. Normally, I would suggest getting a new disk of the same or larger size (you're going to need that anyway) and letting ddrescue take its best shot at copying an image of the failing drive, but if this is a failure in the control electronics you won't get much, if anything, that way.
It might be best to simply unplug that disk and try to activate the VG with the "-P" (--partial) flag. Then you can see what files can be recovered without error from the remaining physical volumes. As for the failed drive, all of the usual "put it in the refrigerator," "try running it upside down," and "swap the electronics from an identical model" cards are on the table, though professional data recovery might be the best bet.
If you do manage to recover at least part of the data from that disk onto another drive, the manpage for vgcfgrestore has some suggestions for getting that incomplete image back into the volume group.
Well, I removed the broken disk (no more errors during boot up), but I don't think I can save any of the data from the 2 remaining discs.
I think I'll try to remove LVM and crate two new samba shares, one on each disk. We lost a lot of TV shows, movies and music, but nothing too essential I think...
It becomes an issue of, "How dead is that broken drive?" and how much time you might want to spend determining that and trying to recover files with photorec. Sorry it's come to that.
If that VG had just one LV spread over all three disks, that ~3TiB filesystem now has a 1.36TiB unrecoverable hole right in the middle of it. Recovering data from that might be a bit of a challenge, and should begin by making a copy of the ~2Tib that remains. Just inserting a new disk to fill that hole in the LV and running "fsck -y" to patch up the filesystem could well make things worse and preclude any recovery.
If there were any intact filesystems in that VG, that should have already been discovered when the partial VG was activated.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.