A botched lvreduce command has caused *all* LVM commands to indefinitely hang, including vgcfgrestore.

I've searched everywhere and I can't find the location of the *active* lvm metadata, I know the backup and archive files are in /etc/lvm but I need the current version so I can replace it.

Thanks.

It is stored within the various headers within the LVM physical volume itself with some redundancy making it not amenable to manual editing. You should be able to edit one of the files in /etc/lvm/archive or /etc/lvm/backup to describe the way the configuration should be, and then use vgcfgrestore from that file. The only issue would be if the physical volume header were damaged, and that could be corrected by running pvcreate with the "-ff", "--uuid", and "--restorefile" options.

The ultimate weapon would be to use pvremove with the "-ff" option to wipe the PV header, then pvcreate with the "--uuid" and "--restorefile" options, followed by vgcfgrestore. I can't imagine a situation where that pvremove would be necessary. The above "pvcreate -ff ..." should always suffice.

I have a file in the /etc/lvm/archive folder that is exactly the right configuration (Just before I did the lvreduce) but I can't run vgcfgrestore because it just hangs (I've left it for over an hour).

I think the PV metadata is fine, the problem occured because I accidentally increased the LV to 16.1TB which is too large, I did lvreduce and it hung, after that no LVM commands are working, I can't do pvscan, pvremove or pvcreate, I even tried reinstalling lvm2 through apt-get and that hung too.

When you ran lvresize and/or lvreduce, did you include the "-r" (--resizefs) option? If so, and especially on the lvreduce, the implied fsck can take a very long time (many hours for a 16TB ext3 filesystem) and will keep the volume busy while it is running. Does top show any suspicious running processes? I don't know what sort of locks the various LVM commands might be using to keep from stepping on each other.

I don't want to divert this discussion, but when this is all over, you should file a bug - this should never happen. You should never be able to make a lv "too large" - parsing should catch that.

I take it you are doing this recovery from the affected system - any success running lvm commands from a liveCD ?.

I didn't use --resizefs, I tried running that separately after lvresize and it refused due the 16tb limit, which is why I ran the lvreduce. The lvreduce command hung for at least 20 minutes before I Ctrl-C'd it, I thought It'd be easier to just revert the metadata. I tried turning off the locking in lvm.conf but the commands still won't run.

I'll make sure I do that, I'll wait and see if I find a fix first though so I can add that too.

Running the commands from a liveCD is a great idea! I didn't think of that, I've tried Ubuntu on USB and DVD and Mint on USB and they all get stuck at boot up at the logo? I can switch to the other TTY's but they're just a flashing underscore, not a prompt. I've booted liveCD's on this machine before and it's never done this.