Can't activate any thin volume after pool repair

Hi.
I'm on Rhel 6.5. There was not enough space for pool metadata, neither free space to extend pool lv:

May  8 03:28:02 oktest-stage-db-1 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.

May  8 16:58:09 oktest-stage-db-1 lvm[896]: No longer monitoring thin vg_oktestdb1-pool-tpool.
May  8 16:58:09 oktest-stage-db-1 lvm[896]: Monitoring thin vg_oktestdb1-pool-tpool.

May  8 17:49:11 oktest-stage-db-1 kernel: device-mapper: thin: dm_thin_insert_block() failed
May  8 17:49:12 oktest-stage-db-1 kernel: device-mapper: block manager: recursive lock detected in pool metadata
May  8 17:49:12 oktest-stage-db-1 kernel: device-mapper: thin: dm_thin_insert_block() failed
May  8 17:49:12 oktest-stage-db-1 kernel: db2sysc[9152]: segfault at 0 ip 00007ffff2d6ecd0 sp 00007ffef640fc38 error 6 in libdb2e.so.1[7fffecce4000+a55b000]
May  8 17:49:13 oktest-stage-db-1 abrt[830]: Can't open 'core.2077': Permission denied

After hard reset tried to activate volume:

[root@oktest-stage-db-1 ~]# lvchange -ay vg_oktestdb1/lv_ibm
  Check of thin pool vg_oktestdb1/pool failed (status:1). Manual repair required (thin_dump --repair /dev/mapper/vg_oktestdb1-pool_tmeta)!

Tried to repair:

[root@oktest-stage-db-1 ~]# thin_dump --repair /dev/mapper/vg_oktestdb1-pool_tmeta
<superblock uuid="" time="23" transaction="34" data_block_size="512" nr_data_blocks="1808288">
</superblock>
[root@oktest-stage-db-1 ~]# lvconvert --repair vg_oktestdb1/pool
  Volume group "vg_oktestdb1" has insufficient free space (0 extents): 28 required.
  WARNING: If everything works, remove "vg_oktestdb1/pool_tmeta0".
  WARNING: Use pvmove command to move "vg_oktestdb1/pool_tmeta" on the best fitting PV.

The output of thin_dump --repair is too short! There should be thousands of lines.


Free some space:

[root@oktest-stage-db-1 ~]# swapoff /dev/mapper/vg_oktestdb1-lv_swap
[root@oktest-stage-db-1 ~]# lvreduce -l 90%LV vg_oktestdb1/lv_swap
  WARNING: Reducing active logical volume to 7.03 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lv_swap? [y/n]: y
  Reducing logical volume lv_swap to 7.03 GiB
  Logical volume lv_swap successfully resized
[root@oktest-stage-db-1 ~]# mkswap /dev/mapper/vg_oktestdb1-lv_swap

Retry repair and reboot:

[root@oktest-stage-db-1 ~]# lvconvert --repair vg_oktestdb1/pool
  Logical volume "lvol0" created
  Logical volume "lvol1" created
  WARNING: If everything works, remove "vg_oktestdb1/pool_tmeta1".
  WARNING: Use pvmove command to move "vg_oktestdb1/pool_tmeta" on the best fitting PV.
[root@oktest-stage-db-1 ~]# reboot

Trying to activate:

[root@oktest-stage-db-1 ~]# lvchange -ay vg_oktestdb1/lv_ibm
  device-mapper: reload ioctl on  failed: No data available

Dmesg shows:

device-mapper: thin: Data device (dm-3) discard unsupported: Disabling discard passdown.
device-mapper: table: 253:5: thin: Couldn't open thin internal device
device-mapper: ioctl: error adding target to table

Hi leniviy, Three years later I have the same problem.
Could you solve it?

Originally Posted by jotamolas Hi leniviy, Three years later I have the same problem. Could you solve it?

Sorry, no. Luckily this was a database host so I had backups. In the end we migrated to a scheme with btrfs and btrfs-sxbackup over ssh.

Just ran into the same issue. Should I understand from this that all my data is gone?

I filled up the metadata space and in the process of trying the repair ended-up in this situation where none of my THIN provisioned volumes can be mounted.

root@pve1:~# lvchange -ay pve/vm-100-disk-5
device-mapper: reload ioctl on failed: No data available

I guess I'm screwed?