XFS filesystem spontaneously shrunk by 12TB

elimorris · 07-08-2010, 07:30 PM

Hi All,

I'm running a system where we had a 62 TB XFS filesystem running on top of a LVM2 logical volumes (which in turn is comprised of 5 PVs, each of which is a external 16 bay RAID). Well, this weekend there were some I/O errors on the filesystem. I rebooted and then ran xfs_repair. The filesystem looked ok, but then I noticed using df, that it was only reporting 51 TB!

I looked at pvdisplay and also noticed the entry for one of the PVs is as follows:

PV Name /dev/sdc1
pvdisplay VG Name vg1
pvdisplay PV Size 12.73 TB / not usable 12.00 TB
pvdisplay Allocatable yes (but full)
pvdisplay PE Size (KByte) 4096
pvdisplay Total PE 3337859
pvdisplay Free PE 0
pvdisplay Allocated PE 3337859
pvdisplay PV UUID sDWFWu-fJ5u-wITT-ikmU-6ytD-OdUf-pHnRd1

Please note where it says that 12.00 TB is 'not usable'. I don't know what precisely is meant by 'not usable'. I looked back and it looks like it's been this way before the problems this weekend, but it is very suspicious.

I really have no idea as to how to proceed. I've never seen anything like this before. lvdisplay says the volume is 61.85 TB. parted says the filesystem on top of this volume is 68.0 TB. xfs_info says the volume is 51 TB, which is what df -h is also saying.

Any help would be appreciated. Even if anyone has an idea of what to look at.

The system is Centos 5.x (not sure which version, but the kernel is 2.6.18-128.1.14.el5). Hardware is a Dell Poweredge 2950 with an LSI fusion SAS card, which is connected to a SecurStor ES RAID 16 bay unit w/ 4 additional JBODs.

Thanks very much,

Eli

rayfordj · 07-10-2010, 08:37 AM

Why it is showing 12TB not usable, I don't know, but typically "not usable" is a remainder of PE. That is, in your instance, it would be something less than 4M because the PE Size is 4096, so something like "not usable 3.77 MB" would be acceptable and/or expected.

I would first start with confirming the health of the disk(s) that are known to the OS as /dev/sdc and work your way up the layers.

Regarding the LVM stuff, you could start with /etc/lvm/backup/vg1 and /etc/lvm/archive/vg1_#####.vg (where ##### is the last/largest number, or most recently modified file). Additionally, some output from dmsetup might be helpful. Since I don't know the name of the 62TB LV on vg1 I will call it xfsLV. To identify the actual name as known to dmsetup you can run 'dmsetup ls' and find it in the list.

Code:

dmsetup deps vg1-xfsLV

dmsetup status vg1-xfsLV

dmsetup table vg1-xfsLV

If you post any of the command output or the contents of lvm archive or backup for vg1, please be sure to enclose them within CODE tags.