Tampered with LVM on Fedora 9: need to restore configuration
 

I tampered with gnome-settings-lvm trying to add some space to my only hard drive.
LogVol02 contains my home directory which is encrypted with LUKS.
Please do not blame me for not remembering what I exactly did. I surely tried to shrink a logical volume to free some space to let my /home grow... then I had to reboot as my system hung.

At reboot:

Reading all physical volumes..
Found volume group "VolGroup00" using metadata type lvm2
3 logical volume(s) in volume group "VolGroup00" now active .......

Starting udev: [OK]
Setting hostname ... [OK]
Starting disk encryption:
Enter LUKS passphrase for /dev/mapper/VolGroup00-LogVol02


once I enter the correct passphrase i get the message "slot 0 unlocked" then all still OK


Setting up Logical VolumeManagement: 3 logical volume(s) in volume group "VolGroup00" now active OK

Starting disk encryption [OK]

Checking filesystems
/dev/VolGroup00/LogVol00: clean, 262148/6406144 files, 10344642/25600000 blocks
/dev/mapper/luks-VolGroup00-LogVol02: The filesystem size (according to the superblock) is 34865023 blocks
the physical size of the device is 5250943 blocks
Either the superblock or the partition table is likely to be corrupt!

/dev/mapper/luks-VolGroup00-LogVol02: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
/boot: clean, 43/50200 files, 28756/200780 blocks [FAILED]

*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance

.......... I enter my root password then I get:

mktemp: failed to create file via template '/tmp/elektraenv.d2XEM9sQM': Read-only file system
bash: $FILE: ambiguous redirect
bash: $FILE: ambiguous redirect
bash: $FILE: ambiguous redirect
bash: .: filename argument required
.: usage: . filename [arguments]
(Repair filesystem) 1 #

My first idea at this stage was to take a look at the LVM configuration, so I issue lvm and get to the lvm prompt

but when I try

lvm> pvscan (or any other lvm command) I always get:

Locking type 1 initialisation failed

What now?

> I have read somewhere that I should do something in GRUB to go to runlevel 1 (I have only a vague idea of what runlevels are and I thought that at this stage one would be in runlevel 1)

I have found archived backups of the lvm configuration under /etc/lvm/
but I am unable to issue vgconfrestore fue to the lockingn issue described above :-(

I have read the first few sectors of the hd partition containing the volume group /dev/sda2 but I am unsure if I may overwrite them with the correct information.........

> I tried to boot from the F9 netinst (aka rescue) CD but my computer seems to have a serious problem with USB Keyboard and no PS2 connector which results in being unable to select the CD boot options, so no way to start the rescue install)
?? would it be possible to have a bootable CD with the rescue install image _only_?
?? would any other tool like Hiren, Helix or other rescue tools be suitable?

It would be very nice to get my system back, as it has been running very smoothly for nearly one year, but it would be much more important to get my data back. Does anybody know if I have a chance?
would it make sense / be easier to connect the HD to another machine via USB externally? Would I then be able to access the data on the encrxpted /home directory?

Any idea is greatly appreciated. Thanks in advance for your help

Not that I think it would help much if "mktemp: failed to create file ..: Read-only file system" means the filesystem gets remounted readonly on error but would disabling SELinux from the GRUB commandline get you any further? And adding "--ignorelocking-failure" to any lvm command?

Thanks unSpawn for the ideas. Unfortunately the --ignorelocking option doesn't work. :(
I will try to figure out how to disable SELinux from the GRUB command line.

BTW, after comparing /ets/lvm/backup (information right AFTER the VG change) with the one contained in the backups under /etc/lvm/archive/VolGroup00_00001.vg, it appears to me that the only thing changed when I tampered is the 'extent_count' of my LogVol02...
so now I have 641 # 20,0312 Gigabytes while before I used to have 4256 # 133 Gigabytes.

If the difficulties to run lvm from the "recovery console" persist, could I correct the lvm information "hard coded" on the bits of my HD according to what stated above by means of a binary editor like gvim from within a live CD?

should I post the files/listings?

Thanks in advance for any thoughts

Please don't say "doesn't work" but please be precise like: "application doesn't know --ignorelocking-failure" or "same result shown after adding --ignorelocking-failure".


Add "selinux=0" to the kernel line?


Wrt editing stuff have a look at this, this and this. Note the solved problems address correcting metadata on the premise no physical changes were made.

Now from the information you provided I see you have one VG providing 3 LV's (/boot, /what? and /home). LogVol00 seems clean, LogVol01 you haven't posted about, and LogVol02 (/home) has problems. Because you run LUKS over LVM your stack looks like this: PV -> VG -> LV -> LUKS -> filesystem. As far as I understand, before reducing (lvreduce) an LV 'resize2fs' must be run to let the filesystem know its boundaries changed (and which extents to clean up) and then 'fdisk' to update the partition table (the other way around for lvextend). How LUKS behaves during resizing I don't know though the "slot 0 unlocked" suggests it behaved OK. Asserting there's no physical changes or other problems, it looks like this might (I say might) be correctable by updating the partition table with fdisk and running fsck to correct the superblock. But because your filesystem resides inside LUKS you'll have to find a way to have the LV online (which the system did OK), then access LUKS (again OK), but then access the filesystem but without it being fscked for consistency to be able to work on it. Once you find a way then maybe running 'testdisk' without making changes can show more info. And please feel free to verify if this is the right or wrong approach elsewhere like on the lvm users mailing list. After all it's your data, not mine.


* But before you do anything you must realise the source of your problem is not of a technical nature. You thought your system hung and then you thought rebooting would fix that. There is no system error in that. Maybe not nice to hear but vital to realise as your next task is to make a complete backup of your disk before proceding or risk being SOL on failure. Yours, that choice, is.


Sure. More info is better.

Sorry and thanks a million for stressing out this. I have found out that the option --ignorelocking indeed does work!

Thanks for the links. I will need some more time to take a thorough look at the posts. Obviously the "successful one" is most interesting for me ;-)

LogVol01, guessing from the size of 1.9375 Gigabytes and remembering vaguely how anaconda proceeds, must be /swap.

I have been trying to collect some information. Pity that I have no idea how to get any output to a file when using the "Repair filesystem" shell due to the read only filesystem... Do you know any magic trick here?

I booted into KDE using Knoppix live CD and that's what lvm could identify there:

lvm> lvscan
inactive '/dev/VolGroup00/LogVol00' [97.66 GB] inherit
inactive '/dev/VolGroup00/LogVol02' [20.03 GB] inherit
inactive '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

I know for sure that the site of LogVol02 cannot be correct. Any clue where "did Knoppix get" this informtion from?

Now:

root@Knoppix:/# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup00
PV Size 232.62 GB / not usable 0
Allocatable yes
PE Size (KByte) 32768
Total PE 7444
Free PE 3616
Allocated PE 3828
PV UUID 9OZrOh-DNCW-Di10-hryC-0980-kpc0-2mtJpB

and

lvm> vgdisplay
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 232.62 GB
PE Size 32.00 MB
Total PE 7444
Alloc PE / Size 3828 / 119.62 GB
Free PE / Size 3616 / 113.00 GB
VG UUID 89Q0vJ-KGKR-M81F-HIms-jWid-G73R-Kdx6W3

and now the most interesting part: in the first few bits of my HD I dumped via the dd command, I find under:

LogVol02 {
id = "mL35eh-alTv-xgOh-Vi32-xdVh-6mE8-Y9IxVi"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 4256

type = "striped"
stripe_count = 1 # linear

stripes = [
"pv0", 3125

[...]
# Generated by LVM2 version 2.02.33 (2008-01-31): Wed Jul 23 14:10:19 2008

then, a few lines further down:

LogVol02 {
id = "mL35eh-alTv-xgOh-Vi32-xdVh-6mE8-Y9IxVi"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 641

type = "striped"
stripe_count = 1 # linear

stripes = [
"pv0", 3125

[...]
# Generated by LVM2 version 2.02.33 (2008-01-31): Mon Jun 8 21:39:14 2009

Now, in my "(Repair filesystem)" shell, when I cat /etc/lvm/archive/VolGroup00_00001.vg, in the description header I can read:

description = "Created *after* executing '/sbin/lvreduce -f -l 641 /dev/VolGroup00/LogVol02'"

Am I correct in the assumption that this is the last command I "issued" with lvm?

I recall better and better that I was in a Gnome session, ran gnome-settings-lvm and then must have "reduced" LogVol02 in a silly attempt to create a LogVol03. The fact that I did not get any warninig or message asking to confirm/submit any change from gnome-settings.lvm (and did not really read the respective documentation) made me believe it would have been safe to shutdown the computer. Now I even remember, that the box was not "hung" this time, and I even shut down using the regular menue in Gnome.

BTW, this is what what I get when I run fdisk

root@Knoppix:/# fdisk /dev/sda

The number of cylinders for this disk is set to 30394.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 250.0 GB, 250000000000 bytes
255 heads, 63 sectors/track, 30394 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 25 200781 83 Linux
/dev/sda2 26 30394 243938992+ 8e Linux LVM

////////////

root@Knoppix:/# fdisk /dev/sda2
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklab el
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 30369.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sda2: 249.7 GB, 249793528320 bytes
255 heads, 63 sectors/track, 30369 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Command (m for help): q


Is there any further information I should try to ather to find out whther it would be safe and potentially successful to try to set the size of LogVol02 back to ehat it should be by using vgcfgrestore from the "Repair filesystem" shell using the --ignorelocking option?

Thanks so much again for all your input!!

Please do take your time.


Can you mount a floppy or USB stick? Might require mounting /proc and /sys beforehand.


LVM by default leaves one or more "signature" blocks of "meta information" on the disk so 'lvscan' scans the disk to find and read it. As you've shown by dd'ing the size is 'dd if=/dev/VolGroup00/(LV name) "of=/some/path/metadata" bs=512 count=384'.


Reading all /etc/lvm/archive/VolGroup00_.*.vg, a few lines below the "description =" line you'll find "seqno =". The highest sequence number for that VG should indicate the last performed action and as far as I know log filename numbering starts at _00001.


That only makes sense to see the size of your PV (which in your case spans one partition /dev/sda2). The "interpreted volume" (device-mapper) can be accessed from /dev/mapper/ or /dev/VolGroup00/. Try 'dmsetup -v status', 'dmsetup -v table' or 'sfdisk -l /dev/mapper/(LV name)'.


No, I don't think you can find out more. But do make a backup.

Onee again thanks a million for the tremendous amount of information you are giving to me unSpawn.

I have noticed that. The change I was reportingin my last post was the one with the highest count=5. The log file numbering indeed starts at _00000. N° _00001 is the latest one. So that should be the "right one", right?


I am going to try to learn how to interpret the outputs I get from dmsetup as well as how this interacts with lvm...


I will definately do a backup. Would you advise to 'dd' the whole sda HD or only the partition sda2 ?
Since I am not savy enough (yet) to mount anything under the (Repair filesystem) shell, would it also be OK to boot in a live CD and issue the 'dd' command from there on sda and the "unmounted-but-shown" USB HD?

many more thanks and cheers

Right.


The PV that holds your VG consists of only /dev/sda2. Because LVM doesn't keep meta data anywhere else, and because you're not about to mess with the other partitions, you could well backup just /dev/sda2 to a removable, yes. Not to curb your enthusiasm but I have to emphasise that for a LUKS-over-LVM setup I have no idea how LUKS is handling the resizing: it may work and it may not. That was one thing I forgot to post about in my "lvreduce" post: using cryptsetup to reduce the size of the LUKS encrypted area. Anyway, if you can't make it work at this time then at least you've got a backup to try later on. Do let us know how it's going, OK?

Hi again,

I have now backed-up my HD onto the external HD. As the HDs were of identical size, I did dd sda to of=sdb

Now I am a littel afraid to try vgcfgrestore on the local HD and I am thinking about experimenting on the backup - giving the "lvcfgrestore" approach a try on the EXternal HD.
I tried to boot from the USB HD on the same machine and on a laptop, but when I boot, but then

Any ideas please?

I feel I am getting a bit closer
 

Quick update: I have installed ubuntu 9.04 on a notebook, then mounted the USB HD, started gnome-settings-lvm, there I have selected my 'problematic' LogVol02 and reset the suspected extents_count=4256 to what the were set in the last
/etc/lvm/archive/VolGroup00_00001.vg
I could aready see in the graphical view that it 'fitted' leaving only 1 extent free.
I was offered to select a filesystem and whether I wanted to mount: answered ext3 and mount... after a while I could see my LogVol02 mounted on the desktop. Opened it and found only a "Lost and found" folder.
OK, must be due to the encryption.
I am also not sure if it was wrong to select ext3 as the filesystem...

So now I unmounted everything and tried:

Tried the passphrase 4 times, I am sure been using the correct one. Now: where is my key? ;-)

any idea very appreciated
cheers