I did finally get this problem resolved.
The root of the problem was that I used Webmin to reduce the logical volume size.
Webmin did not
reduce the file system size prior to reducing the logical volume size. Just a warning for others out there.
Here's how I resolved:
1. Created new Logical Volume (e.g. /dev/VolGroup00/newLV). I used the Gnome Logical Volume Manager, but lvcreate
would have worked as well.
2. Mounted new Logical Volume to a mount point
# mount /dev/VolGroup00/newLV /mnt
3. Check for open files on the old file system/logical volume and kill them
# lsof /old_fs_mountpoint
# kill <pids_from_previous_command>
4. Use cpio to perform block copy of entire file system from the old volume to the new volume. Using cpio will ensure that all
files and directories, even ones that begin with (.
) will be copied. I was fortunate that most of the data in my old volume was retained. I only had to restore one directory tree using a tape backup (see Step 7).
# cd /old_fs_mountpoint
# find -xdev | cpio -pvdam /mnt
5. Unmount volumes and remount new volume to old volume's mountpoint
# umount /old_fs_mountpoint
# umount /mnt
# mount /dev/VolGroup00/newLV /old_fs_mountpoint
6. Edit /etc/fstab
to make the new volume mount to the correct mountpoint by default, and to keep the old volume from mounting at all. Helpful website for understanding /etc/fstab
7. Restore any directories/files from archive if necessary
# cpio -idvmu /dir_to_be_restored/* < /dev/st0
Repeat for each directory to restore.
8. Run fsck to check the volume, and you may even want to reboot.
# umount /dev/VolGroup00/newLV
# fsck -f /dev/VolGroup00/newLV
9. Once you are confident that the new volume is working properly, you can delete the old volume.
Hope this helps someone else at some point!
Just remember that when you want to reduce the size of a volume, reduce the filesystem size first
And don't use Webmin to do it!