LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices

Reply
 
Search this Thread
Old 05-02-2009, 01:16 PM   #1
taylorkh
Senior Member
 
Registered: Jul 2006
Location: North Carolina
Distribution: CentOS 6 on my desktop, Ubuntu 12.04 LTS on my server, Ubuntu 12.04 on my netbook and the wife's PC
Posts: 1,200

Rep: Reputation: 96
External USB hard drive - file corruption


Started out trickling down hard drives. 2 - 1 TB SATA drives into server, 2 - 320 GB SATA drives from server to PCs etc. Anyhow I ended up with some spare PATA (IDE) drives (IBM/Hitachi Deskstars and a Seagate Barracuda). Looking for something to use them for besides paper weights I got hold of a "PortaDrive" gizmo which connects to an IDE or SATA drive and converts it to USB. So I connected the Barracuda, duly wiped and reformatted to NTFS, to the device and then to the server (Ubuntu 8.04.2).

The drive came up as expected. I used Gnome to drag and drop 10 large (4.5 GB PGP disk image) files to the external drive to test speed. About half what I get SATA drive to drive so not too bad. I then unmounted the external drive and connected it to an XP machine. I ran a file compare over the network to monitor the external drive performance vs. network. Again, not bad except...

All 10 if the files showed minor errors when compared to the originals. The errors were too small to be found by MD5. I reconnected the drive to the Linux machine and confirmed the errors with cmp.

I have tried 3 different drives, a drive formatted ext3, large and mixed size files. In every case I get some file corruption. I have connected the drives by PATA cable directly to a machine and copied the same file - no errors. I have copied smaller selections of files to a USB 8 GB flash drive on the same machines. No errors.

So I guess my questions are...

Are USB connected hard just drives flaky by nature? (Perhaps it is the little connector gizmo, I have a different unit on the way).

Is there a way to copy files in Linux with on-the-fly verification? Something like DOW/Windows xcopy -v ?

I have looked at the options with cp and rsync and am not sure I find any to do the trick.

TIA,

Ken

p.s. Perhaps 40 - 160 GB drives are not good for anything but paper weights these days. With 1 TB drives are going for as low as $80US... My first 42 MB Seagate was $399 :-((
 
Old 05-02-2009, 01:58 PM   #2
Pearlseattle
Member
 
Registered: Aug 2007
Location: Switzerland
Distribution: Gentoo
Posts: 716

Rep: Reputation: 81
"The errors were too small to be found by MD5"
What do you mean with that? I know that md5 is being replaced by sha because of some problems with a too high number of possible collisions, but I am not aware that MD5 does a kind of approximation, ignoring some data.

USB HDDs work and are supposed to work the same way as sata, IDE, esata, scsi, etc. No loss of data is supposed to happen.

I therefore think your converter, or some USB cable you're using (e.g. not longer than 5m), is doing dirty tricks.

Greetings
 
Old 05-02-2009, 03:20 PM   #3
taylorkh
Senior Member
 
Registered: Jul 2006
Location: North Carolina
Distribution: CentOS 6 on my desktop, Ubuntu 12.04 LTS on my server, Ubuntu 12.04 on my netbook and the wife's PC
Posts: 1,200

Original Poster
Rep: Reputation: 96
Hi Pearlseattle,

I thought the MD5 result was quite interesting. Basically I performed an MD5 calculation on each of the 10 large files as they sat on the server. I then did an MD5 calculation on each of the files as they sat on the USB drive. All of the MD5 values matched file by file! When I had moved the USB drive to the XP machine I again ran the MD5 calculations and again the values matched the other 2 runs.

From what I have read about MD5 it is theoretically possible to defeat it - that is get the same MD5 number from two different files. However, that was supposed to be more theoretical than practical. That said I also recall something about the MD5 algorithm treating data from the beginning of the file with more significance than later data. Perhaps with a LARGE file containing seemingly random data, it is encrypted after all, MD5 just doesn't cope.

As to the size of the error... cmp found bad data and, it being binary, attempted to display the mismatch. I would guess that perhaps 200 or so characters appeared in the result.

I opened two matching files with PGP and did a compare of the data files within. Out of 25K+ small files, only 6 mismatched. Not sure how PGP handles corruption within an encrypted disk file but at least the entire disk image file was not lost.

The USB cable is hard wired to the converter and is at most 2 feet long. I have one of the external enclosure type converters coming in Tuesday if Fed Ex is on schedule and I will do some more testing. Perhaps I need to write my own file comparison routine to better quantify the differences. I once did that when I was FTPing 80,000+ AutoCad drawings to a Silicon Graphics box, renaming them, converting the to MicroStation then force feeding them into a drawing management system. MD5 would have been great for that application as they were small files. However, I did not know about MD5 then and so I rolled my own. I wonder where I put the source??? I guess I should look into sha and see if that will help.

Thanks again for the reply.

Ken
 
Old 05-06-2009, 12:23 PM   #4
taylorkh
Senior Member
 
Registered: Jul 2006
Location: North Carolina
Distribution: CentOS 6 on my desktop, Ubuntu 12.04 LTS on my server, Ubuntu 12.04 on my netbook and the wife's PC
Posts: 1,200

Original Poster
Rep: Reputation: 96
Update - it seems that the problem was the USB converter/adapter gizmo I was using. I procured an external drive box/adapter, installed a 160 GB Barracuda in it, connected it to various machines, copied and verified files. Works great!

Ken
 
Old 05-06-2009, 01:15 PM   #5
Pearlseattle
Member
 
Registered: Aug 2007
Location: Switzerland
Distribution: Gentoo
Posts: 716

Rep: Reputation: 81
Cool!
Thx for the information
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
file backup on rehat server to WD my book external usb hard drive justintime24 Linux - Server 2 05-09-2008 01:31 PM
usb external mini hard drive file system offbyte Linux - Hardware 1 11-04-2007 03:53 PM
External USB disk: repeated file system corruption Sven Sandberg Linux - Hardware 1 08-12-2007 06:02 AM
external enclosure for an internal hard drive vs external USB hard drive powah Linux - Hardware 1 03-10-2006 10:03 AM
Installing grub to external USB hard drive for later use as internal hard drive dhave Linux From Scratch 2 12-10-2005 09:48 AM


All times are GMT -5. The time now is 12:31 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration