LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 04-04-2014, 05:35 PM   #1
dugave_111
LQ Newbie
 
Registered: Oct 2006
Posts: 24

Rep: Reputation: 0
LVM failure


Hi!

I did a update on ubuntu server to latest versions (have done that before), but
this time the machine wouldn't boot and I had to connect a monitor to it.
The boot process was stuck - it couldn't mount LVM drive(s) and was waiting to either skip or manually mount.


Is there anything I can do, the disk(s) have been showing signs of
bad sectors or other problems for some time now, but usually some kind of fsck would keep the thing working.

Code:
lvdisplay
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
  --- Logical volume ---
  LV Path                /dev/abram_vg/abram_disk
  LV Name                abram_disk
  VG Name                abram_vg
  LV UUID                y6gUb5-FWUt-y16S-e7J1-5p9O-geNW-00pWB9
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              NOT available
  LV Size                3.64 TiB
  Current LE             953864
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto

Code:
fsck /dev/abram_vg/abram_disk
fsck from util-linux 2.20.1
e2fsck 1.42.8 (20-Jun-2013)
fsck.ext2: No such file or directory while trying to open /dev/abram_vg/abram_disk
Possibly non-existent device?


BTW - is there a way to configure ubuntu to continue booting no matte what, ignoring all the errors?

Thanks!
 
Old 04-05-2014, 11:43 AM   #2
michaelk
Moderator
 
Registered: Aug 2002
Posts: 21,212

Rep: Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972
So I assume these are data drives? Without knowing your exact configuration it is a bit hard to say. If they are in the /etc/fstab file you can add the no auto option.

So it looks the the drive is failing and ignoring errors could lead to losing all of the data on the drive if it is not safely backed up. If the VG spans multiple drives then you could possibly lose everything.
 
Old 04-05-2014, 01:33 PM   #3
dugave_111
LQ Newbie
 
Registered: Oct 2006
Posts: 24

Original Poster
Rep: Reputation: 0
Hi, thanks for the reply!

This is my fstab:

1 disk that has: swap, main partition for ubuntu and one more for a place where I put programs. It's a small disk
of about 80 GB.

LVM has 3 I think, about 4 T together.

Code:
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=0fd1729b-ed72-4699-9803-8155ea4f67b0 /               ext4    errors=remount-ro 0       1
# /opt was on /dev/sda5 during installation
UUID=4876b2ef-d9f9-4434-b1de-b30573bcb52d /opt            ext4    defaults        0       2
# swap was on /dev/sda6 during installation
UUID=1e276083-e710-4240-a998-1e288500dbac none            swap    sw              0       0
#abram disk:
UUID=5ed42986-7f63-4437-8aa0-421cbae70b3b /opt/abramdisk  ext3

The 3 disks that make LVM are used for storing movies, music and such for everybody on LAN. It won't be
a huge tragedy if we loose everything that's there...

I'd like to find a way to mount the LVM disk so that I can try to fix it with fsck - it helped before, but
I was never in a situation where I couldn't even mount the LVM and access it.
 
Old 04-05-2014, 01:53 PM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 21,212

Rep: Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972Reputation: 3972
by commenting out abram VG in the fstab file the OS should boot without pausing.

Post the out of the commands
pvscan
vgscan
What happens when you try to activate all VGs:
vgchange -a y

If that works you can see if the drives are available using the following command:
lvscan

If that works you can try fsck.
 
Old 04-05-2014, 02:58 PM   #5
dugave_111
LQ Newbie
 
Registered: Oct 2006
Posts: 24

Original Poster
Rep: Reputation: 0
Hi!

The system is booted and running, it's our router to the net.

During the boot process it stopped at the point when it tried to mount the LVM, waited for key press.
I know I can comet out the fstab part and it will ignore and not even try to mount that. Although, I
will probably still get A LOT of this error msgs:


Code:
[ 3699.933397] ata3.01: failed command: READ DMA EXT
[ 3699.935519] ata3.01: cmd 25/00:08:80:7a:a8/00:00:ae:00:00/f0 tag 0 dma 4096 in
[ 3699.935519]          res 61/04:08:80:7a:a8/04:00:ae:00:00/f0 Emask 0x1 (device error)
[ 3699.939761] ata3.01: status: { DRDY DF ERR }
[ 3699.941898] ata3.01: error: { ABRT }
[ 3699.961199] ata3.00: configured for UDMA/133
[ 3699.976137] ata3.01: configured for UDMA/133 (device error ignored)
[ 3699.976151] ata3: EH complete
[ 3699.977196] ata3.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 3699.979313] ata3.01: BMDMA stat 0x25
[ 3699.981435] ata3.01: failed command: READ DMA EXT
It went for a few minutes like that.

pvscan
Code:
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
  PV /dev/sdd5        VG abram_vg   lvm2 [1.36 TiB / 0    free]
  PV /dev/sdb1        VG abram_vg   lvm2 [931.51 GiB / 0    free]
  PV unknown device   VG abram_vg   lvm2 [1.36 TiB / 0    free]
  Total: 3 [3.64 TiB] / in use: 3 [3.64 TiB] / in no VG: 0 [0   ]

vgscan
Code:
Reading all physical volumes.  This may take a while...
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
  Found volume group "abram_vg" using metadata type lvm2

vgchange -a y
Code:
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301819904: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1500301901824: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  Couldn't find device with uuid UO980D-zNOR-GeXx-d1gx-OneR-jFzR-dBBl0Q.
  Refusing activation of partial LV abram_disk. Use --partial to override.
  0 logical volume(s) in volume group "abram_vg" now active

Thanks!!
 
Old 04-05-2014, 04:45 PM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,535

Rep: Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077
The syslog message mentions "device error" and not "media error", which suggests that the problem is a lot worse than just a few bad sectors on the disk. Normally, I would suggest getting a new disk of the same or larger size (you're going to need that anyway) and letting ddrescue take its best shot at copying an image of the failing drive, but if this is a failure in the control electronics you won't get much, if anything, that way.

It might be best to simply unplug that disk and try to activate the VG with the "-P" (--partial) flag. Then you can see what files can be recovered without error from the remaining physical volumes. As for the failed drive, all of the usual "put it in the refrigerator," "try running it upside down," and "swap the electronics from an identical model" cards are on the table, though professional data recovery might be the best bet.

If you do manage to recover at least part of the data from that disk onto another drive, the manpage for vgcfgrestore has some suggestions for getting that incomplete image back into the volume group.
 
Old 04-06-2014, 10:09 AM   #7
dugave_111
LQ Newbie
 
Registered: Oct 2006
Posts: 24

Original Poster
Rep: Reputation: 0
Thanks again for the reply!

OK, I'll pull out the disk and report back what happens. Will try to activate LVM partially first.
 
Old 04-06-2014, 11:22 AM   #8
dugave_111
LQ Newbie
 
Registered: Oct 2006
Posts: 24

Original Poster
Rep: Reputation: 0
Well, I removed the broken disk (no more errors during boot up), but I don't think I can save any of the data from the 2 remaining discs.

I think I'll try to remove LVM and crate two new samba shares, one on each disk. We lost a lot of TV shows, movies and music, but nothing too essential I think...
 
Old 04-06-2014, 01:23 PM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,535

Rep: Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077
It becomes an issue of, "How dead is that broken drive?" and how much time you might want to spend determining that and trying to recover files with photorec. Sorry it's come to that.
 
Old 04-06-2014, 07:18 PM   #10
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,587

Rep: Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507Reputation: 3507
Have a look at this article - quite informative.
If you're lucky you should only lose that disks data - let's hope it wasn't the first disk in the vg.
 
Old 04-06-2014, 09:43 PM   #11
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,535

Rep: Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077Reputation: 2077
If that VG had just one LV spread over all three disks, that ~3TiB filesystem now has a 1.36TiB unrecoverable hole right in the middle of it. Recovering data from that might be a bit of a challenge, and should begin by making a copy of the ~2Tib that remains. Just inserting a new disk to fill that hole in the LV and running "fsck -y" to patch up the filesystem could well make things worse and preclude any recovery.

If there were any intact filesystems in that VG, that should have already been discovered when the partial VG was activated.
 
Old 04-09-2014, 10:03 AM   #12
voleg
Member
 
Registered: Oct 2013
Distribution: RedHat CentOS Fedora SuSE
Posts: 353

Rep: Reputation: 51
Probably bad SATA cable, as common problem ?
But you've already played with disks ...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Recover LVM VG after hardware failure megamojo Linux - Desktop 1 08-22-2008 11:42 PM
LVM, raid1 and scsi failure Ezplan Linux - Server 1 05-18-2007 01:59 AM
HD Failure and LVM, how much can I recover? vaaccess Linux - Hardware 10 01-12-2007 01:12 AM
HD Failure in LVM - What can I recover? vaaccess Linux - General 1 01-05-2007 01:22 PM
LVM Disk Failure R4z0r Linux - Software 3 09-19-2006 02:32 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 08:15 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration