Hello everyone.

I've run into a problem and I can't find a solution to this anywhere, so I'm hoping someone here can help me out as I have no idea where to go from here....

Situation: I have a 4TB external drive that I've attached to my KVM host, and mapped to a virtual machine. When I try to rsync my data from another server to this one, I keep getting this error half way through the upload: "rsync ... failed: Structure needs cleaning (117)". When I reboot the machine and do a sudo fdisk -l, I get " GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted."

Attempted steps: using my google/bing resources, I've tried the following but without success:
- running fsck -t ext4 /dev/sda1 -y multiple attempts
-- using gdisk to delete and remake partition - format to ext4 - no success..... delete partition and format to ext2 - no success
-- tried using gparted - delete the partition, mkfs -ext4 OR mkfs -ext2 - no success
-- deleted the file that failed the RSYNC - still no success and still keep getting these errors.....

Am I missing something? Is there anything I can do to make this drive work?

OS - Ubuntu 14.04.5 LTS
KVM Hypervisor VM
external drive - Seagate 5tb external USB hard drive
Server: HP DL-360 Gen 5 - 2x dual core processors; 12 gb ram;

Thanks and hopefully someone has an idea on how to fix this issue so I can use my new external drive for my file storage / backups.

mikey (dakoris73@gmail.com)

Is this an USB external drive? Did it come packaged in the enclosure you're using? I can only think the drive might be too large for something, somewhere on the system. Older USB enclosures use an older version of the SCSI command set that is limited to 32-bits, or 2 TB drives.

MBR partitioning is limited to 2 TB disks and below. But you can cheat and just make multiple primary partitions each under 2 TB. Or you can use the gpt partition scheme with parted.

The BIOS of the VM might have a size limitation on disks. In any event, it sounds like rsync is trying to read or write an address it can't find because the disk geometry is not compatible with something.

You'll have to do some investigating. I hope this helps.

AwesomeMachine - Thank you for your response. Yes, this is an external USB drive and yes, it is in the original enclosure. The drive is only about 2 months old, and worked flawlessly on a previous ubuntu desktop server that is no longer usable. Since moving it to this rack server and adding as a storage device in KVM for my vm, I can't do any rsync to it as it causes these errors.

I thought about splitting the drive up into smaller partitions, but that won't work as well overall as this is a media drive and I've tried using the gpt partitioning with gparted and also with gdisk. Still not working......

not sure about how to update / change the bios of the VM, but could be something to look into.....

Any other suggestions / advice / considerations that anyone has to help me with getting this figured out is very much needed and appreciated.

Thanks.

Update: In searching further, I found a thread where there might be a bug in Ubuntu 14.04 and using rsync to go to external drives. So I created a whole new VM sing Ubuntu 16.04, did all the updates, then reformatted the external drive using the 16.04 server and gdisk. It looked like it was working, but then after moving about 2 TB of data to this drive using rsync from my old 14.04 server thats dying, I'm getting the same errors again.

I looked at the /var/logs/syslog and here's what I see here:

Aug 21 08:52:49 PlexVM kernel: [497329.621154] EXT4-fs error (device sda1): ext4_find_dest_de:1826: inode #119930946: block 959451233: comm mv: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=65569277, rec_len=27750, name_len=251

4397: comm rsync: pblk 959497179 bad header/extent: invalid magic - magic ffff, entries 65535, max 65535(0), depth 65535(0)
Aug 21 11:15:28 PlexVM kernel: [505888.329090] EXT4-fs error: 231 callbacks suppressed
Aug 21 11:15:28 PlexVM kernel: [505888.329120] EXT4-fs error (device sda1): ext4_find_extent:910: inode #119934397: comm rsync: pblk 959497179 bad header/extent: invalid magic - magic ffff, entries 65535, max 65535(0), depth 65535(0)
Aug 21 11:15:28 PlexVM kernel: [505888.353639] EXT4-fs error (device sda1): ext4_find_extent:910: inode #119934397: comm rsync: pblk 959497179 bad header/extent: invalid magic - magic ffff, entries 65535, max 65535(0), depth 65535(0)

I'm really hoping that someone can help me figure this out as I need to move off this data from my old server before it dies completely......

Is the host OS logging any errors related to that drive?

RKNichols - Here's all I can see from the Host machine's log at /var/log/syslog. Don't really see any errors for this drive on the HOST machine... it all seems to be on the vvirtual OS's...... particularly Ubuntu16.04 and Ubuntu14.04 server editions........ If this is NOT what I needed, please let me know what commands to run and I'll do so......


Aug 25 13:17:01 virPylit01 CRON[3055]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 25 13:52:22 virPylit01 kernel: [22317.656022] usb 1-4: reset high-speed USB device number 3 using ehci-pci
Aug 25 13:52:22 virPylit01 kernel: [22317.789630] scsi host5: uas
Aug 25 13:52:22 virPylit01 kernel: [22317.793862] scsi 5:0:0:0: Direct-Access Seagate Expansion Desk 9401 PQ: 0 ANSI: 6
Aug 25 13:52:22 virPylit01 kernel: [22317.795767] sd 5:0:0:0: Attached scsi generic sg1 type 0
Aug 25 13:52:22 virPylit01 kernel: [22317.800042] sd 5:0:0:0: [sda] 9767541167 512-byte logical blocks: (5.00 TB/4.55 TiB)
Aug 25 13:52:22 virPylit01 kernel: [22317.800049] sd 5:0:0:0: [sda] 4096-byte physical blocks
Aug 25 13:52:22 virPylit01 kernel: [22317.801419] sd 5:0:0:0: [sda] Write Protect is off
Aug 25 13:52:22 virPylit01 kernel: [22317.801425] sd 5:0:0:0: [sda] Mode Sense: 4f 00 00 00
Aug 25 13:52:22 virPylit01 kernel: [22317.802168] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 25 13:52:22 virPylit01 kernel: [22317.963163] audit: type=1400 audit(1503687142.330:16): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-b93412c9-8ad2-9803-0393-8d8a1c609b04" pid=3351 comm="apparmor_parser"
Aug 25 13:52:22 virPylit01 kernel: [22318.118580] sda: sda1
Aug 25 13:52:22 virPylit01 kernel: [22318.125164] sd 5:0:0:0: [sda] Attached SCSI disk

Also to note..... whenever I do the following command: sudo fdisk -l and for the external hdd in question, i get:

Disk /dev/sda: 5001.0 GB, 5000981077504 bytes
255 heads, 63 sectors/track, 608001 cylinders, total 9767541167 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 33553920 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sda1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.

I', now trying a full rebuild of this drive using the EXT2 file system as thats what was on it from the factory...... Not holding my breath though........

I don't see anything there that indicates an error. That misaligned partition will cause horrible write performance (less than 10% of what you should have), but shouldn't cause an error, though I suppose the constant misaligned writes might be triggering a fault in the drive's firmware.

Hello again..... Just wanted to send an update on whats been happening..... I did reformat the drive back to EXT2 and that went pretty good. Started to copy my data over with this physically attached to the dying server, and things were going good. got the data saved I wanted and moved this to the new server to start running. However, not even an hour onto the new server, I started seeing these errors again: "Structure needs cleaning" when I couldn't access my data. so I unmounted the drive, and have been running the following command now for the last 7 days: fsck.ext2 /dev/sda1 -y

Periodically, I would see this message come across the screen:

Too many illegal blocks in inode 142434763.
Clear inode? yes
Inode 142442876, i_size is 1072125075, should be 1073790976. Fix? yes
Inode 142442877, i_blocks is 1742792, should be 10000. Fix? yes
Restarting e2fsck from the beginning...
Pass 1: Checking inodes, blocks, and sizes


Is this drive bad or do I just continue to let it chug along and do its thing? Its been running steadily now for the last 7 days, and not sure how much longer I want it to run... Or even better yet, I don't know how long this is suppossed to even run.

Any help / thoughts / advice anyone can help me with would be greatly appreciated.

Thanks.

That fsck shouldn't take anywhere near that long.

Two things to do:

1. See if the drive is complaining about anything. Post the result of "smartctl -l error /dev/sda" and "smartctl -A /dev/sda". Wrap those in [CODE] ... [/CODE] tags, please, to preserve formatting.
2. Repartition the drive with the partition properly aligned (starting sector a multiple of 8) and try again.

Hello again.... Another update of what DIDN'T work again. I moved this drive to my Windows 10 box, and reformatted it yet again, but to NTFS file share, and it started copying my files over to this drive using my Win10 box. The data moved across successfully and initially I began to be hopeful that this might be the fix I've been looking for. However, this means that for this to work, I need to istall a Windows box, which defeats part of the purpose for doing this on my Ubuntu server which is learning Linux Administrations. after about a month of this running on my windows box and seemingly without issues, I did install a Win2012 server to move this drive to, but its on a VM being run by: UBUNTU Server 14.04 LTS KVM, but the drive wouldn't be "installed" on the Win2012K VM so I moved it to my media player VM that is also running on this same server, and wow... it seemed to start working as expected. UNTIL day 4 when I went to do an ls -lart to see abiyt what files are there, and thats when I get this error:

ls: reading directory parents_movies/: Input/output error
total 0

now, I know there's approx 1200+ folder's in this one folder, and my plex server is running and dishing out files as expexted. Jut can't see them to do a search on them or not. I've tried emailing Seagates Tech Support team, but they got nothing as they can't figure out why this isn't working like it should.

Any thoughts as to what my next step is or what else I can do?????????

The system log on the Ubuntu server should have some messages about the error. Disconnect the drive from all VMs and then on the server run "smartctl -l error /dev/sd{X}" and "smartctl -A /dev/sd{X}" (replacing "{X}" with the appropriate drive letter) and post the result. Please wrap that in [CODE] ... [/CODE] tags to preserve formatting. Hopefully the USB bridge chip in that external interface will let you do that, else it's going to be hard to get any info.

rknichols;
I typed in those commands as requested and here is what I have, which unfortunately isn't much to go with:

not really alot to go on so still trying to figure the next step to keep this drive usable......

mikey

You run smartctl on the overall drive (sda), not on a partition (sda1, sda2)

Ok... well I tried that and this is all I got for both commands: