LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
LinkBack Search this Thread
Old 09-25-2011, 04:38 AM   #1
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Rep: Reputation: Disabled
Question md5sum sha1sum are different after copy file/s by cp command, krusader or nautilus


Hi,

I have a problem when I made copy of big file/s. I have been trying to check their md5 and sha1 (md5sum -b file.vdi , sha1sum -b file.vbi) but copied file differs against the original file. Files are copied between ext4 partitions.

I have tried to copy file by cp command, krusader or nautilus - it has always different md5 or sha1.

Could anybody please explain what am I doing wrong and what should I do that copied file should have same content ?

thank you.
 
Old 09-25-2011, 04:54 AM   #2
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Slackware
Posts: 14,872
Blog Entries: 2

Rep: Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805
This behavior can be caused by hardware failure, I would recommend to check the health of the RAM (using Memtest86+) and the disk(s) (using the manufacturers diagnosis tool).
Something like that may also be caused by a not stable running overclock, does your system run within its specs?
 
Old 09-25-2011, 08:19 AM   #3
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Question

Hi,

Thanks for quick reaction. I have checked before many times memory by memetest86+ - it is ok according to this test.
I have many SATA III/II hard drives but in windows 7 this errors does not exist even if I made copy from different partition/s BUT between same disk/s even if partitions are ntfs. And this behaviour is only with files bigger that tens of gigabytes (vrtualbox vdi files).
When copy ISO files ( up to 4.7Gbytes) - it is correct with nautilus but it is not correct with krusader for same files, cp terminal command I did not check so many times md5 or sha1 so I cannot say for now.
I did no check all hard drives which I am actively using with manufacturer diagnostic tools yet because those disks are correct in windows. So , I am not assuming that only ext4 or btrf partitions have problems and ntfs partitions are correct on the same hard drives :-) . But I am going to try it some night/s.
According to those it seems to be only problem because of overclock.
I have maximus iv extreme R3, i7-2600k(3.4GHz) - OC:5.1GHz , memory OC:1600MHz , 9-9-8-24-1 - it should not be problem , graphic card is not OC.
Which is strange as well because win764bit works more or less properly - not with such problems (have another :-( ).

P.S. would it be possible that antivirus is doing this things (I have avg and clamav) ?
(Hope it is not some kind of malware )
 
Old 09-25-2011, 08:21 AM   #4
mreff555
Member
 
Registered: Sep 2011
Location: Philly
Distribution: Gentoo
Posts: 435

Rep: Reputation: Disabled
Quote:
Originally Posted by masuch View Post
Hi,

I have a problem when I made copy of big file/s. I have been trying to check their md5 and sha1 (md5sum -b file.vdi , sha1sum -b file.vbi) but copied file differs against the original file. Files are copied between ext4 partitions.

I have tried to copy file by cp command, krusader or nautilus - it has always different md5 or sha1.

Could anybody please explain what am I doing wrong and what should I do that copied file should have same content ?

thank you.
try using dd and see if you still have the same problem.

PHP Code:
dd if=[inputfileof=[outputfile
hopefully you don't have to do this for small files but dd is a better choice for large files.
 
Old 09-25-2011, 11:39 AM   #5
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Question

Quote:
Originally Posted by mreff555 View Post
try using dd and see if you still have the same problem.

PHP Code:
dd if=[inputfileof=[outputfile
hopefully you don't have to do this for small files but dd is a better choice for large files.

Unfortunately sha1 is different by using dd ... AND

I have done sha1 on the same file and ... result is different:

$ time sha1sum -b file-ubu1.vdi
daef63a16afaaf4ef7cf596f62c528d543597739 *file-ubu1.vdi

real 6m9.180s
user 1m45.150s
sys 0m10.450s
:-) 2.02 1.97 1.71 1/713 6384 pts/2 jobs=0 16:32:22 /home/_virtual.readANDwrite
u1@PCEUBU1 1:2006:7 $ time sha1sum -b file-ubu1.vdi
582acf68f271e44540c5bb5e94ed22694ebad1df *file-ubu1.vdi

real 6m6.380s
user 1m47.040s
sys 0m9.620s

Now I am completely confused. Does journaling of ext4 or file caching doing this ? Can anybody explain what I am doing wrong ?
thank you for any hint.
 
Old 09-25-2011, 11:47 AM   #6
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
I forgot to mentioned that I used:

dd if=/dir/to/source-file/file-ubu1.vdi of=/dir/to/dest-file/file-ubu1.vdi
and
dd bs=1G oflag=dsync,sync,nocache conv=fdatasync,fsync if=/dir/file of=/dir/file

file has 47 Gbytes.

sha1 is different.
 
Old 09-25-2011, 11:55 AM   #7
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Slackware
Posts: 14,872
Blog Entries: 2

Rep: Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805
Quote:
Originally Posted by masuch View Post
I have maximus iv extreme R3, i7-2600k(3.4GHz) - OC:5.1GHz , memory OC:1600MHz , 9-9-8-24-1 - it should not be problem , graphic card is not OC.
Which is strange as well because win764bit works more or less properly - not with such problems (have another :-( ).
You say that it should work and that Windows works more or less, so I assume that your overclock is not stable. Linux is not Windows, it may act different to such an unstable system.
Try it again without the overclock and see if that works. If it works your overclock is not stable and causing your issues.
 
Old 09-25-2011, 01:45 PM   #8
mreff555
Member
 
Registered: Sep 2011
Location: Philly
Distribution: Gentoo
Posts: 435

Rep: Reputation: Disabled
Quote:
Originally Posted by masuch View Post
I forgot to mentioned that I used:

dd if=/dir/to/source-file/file-ubu1.vdi of=/dir/to/dest-file/file-ubu1.vdi
and
dd bs=1G oflag=dsync,sync,nocache conv=fdatasync,fsync if=/dir/file of=/dir/file

file has 47 Gbytes.

sha1 is different.
wow, that's a big file. What transfer rate did dd default to with the first command? That must have taken a while.
 
Old 09-25-2011, 03:25 PM   #9
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by TobiSGD View Post
You say that it should work and that Windows works more or less, so I assume that your overclock is not stable. Linux is not Windows, it may act different to such an unstable system.
Try it again without the overclock and see if that works. If it works your overclock is not stable and causing your issues.
I setup BIOS to manufacturer settings and it did not helped. It behaves like following:

I have checked sha1sum of tens of .ISO files and .vdi files - NOT COPIED. It looks like files bigger than 3.8(3.9)Gbytes have always different sha1 result measured by sha1sum.
 
Old 09-25-2011, 03:27 PM   #10
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by mreff555 View Post
wow, that's a big file. What transfer rate did dd default to with the first command? That must have taken a while.
47090896896 bytes (47 GB) copied, 595.566 s, 79.1 MB/s
 
Old 09-25-2011, 03:28 PM   #11
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by mreff555 View Post
wow, that's a big file. What transfer rate did dd default to with the first command? That must have taken a while.
the best result:
# real 6m6.380s
# user 1m47.040s
# sys 0m9.620s
the worst result:
# real 10m52.592s
# user 1m55.320s
# sys 0m10.430s
 
Old 09-25-2011, 04:26 PM   #12
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Question

Quote:
Originally Posted by TobiSGD View Post
You say that it should work and that Windows works more or less, so I assume that your overclock is not stable. Linux is not Windows, it may act different to such an unstable system.
Try it again without the overclock and see if that works. If it works your overclock is not stable and causing your issues.
I have really fun with this:
I have tried to use groovy (version 1.7.4) source code taken from: http://groovy.codehaus.org/Calculati...or+large+files
and
I tried openssl sha1 file.vdi (version OpenSSL 0.9.8o 01 Jun 2010)

and guess what ?
It behaves exactly the same way (sha1sum or sha1deep) as I described:
for files bigger than ~4Gbytes it had different sha1.
I did not copy files - just repeat the same command on same file.
 
Old 09-27-2011, 05:49 AM   #13
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Question

Summary:

I do not have to copy file,
It have happened if I run twice or more times same check-sum command on the same file.
File has to be bigger than approximately 3.9 (3.8-4.2) Gigabytes.
Tested with sha1sum -b , md5sum -b , sha1deep , md5deep , openssl sha1 , groovy sha1 algorithm.
I made test on four EXT4 partitions - they are located on 4 different hard drives (2 external and 2 internal HDDs/USB flash drive) on one computer (other 3 computers does not have this issue).
NTFS partitions DO NOT have these check-sum problems measured on the same hard drive where is located ext4 partition as well (3 HDDs). ... (This I assume eliminates possibility of hardware error/s of hard drive/s)
On BTRFS partition it is not possible to check because of I/O problems:
md5sum -b ubu1104.vdi
# md5sum: ubu1104.vdi: Input/output error
openssl sha1 ubu1104.vdi
# Read Error in ubu1104.vdi
# 5210:error:0200B005:system library:fread:Input/output error:bss_file.c:202:
# 5210:error:20082002:BIO routines:FILE_READ:system lib:bss_file.c:203:
sha1sum -b ubu1104.vdi
# sha1sum: ubu1104.vdi: Input/output error

Run sha1sum from live-cd ubuntu 11.04 64bit have same results.
No overclocking.
Tested on linux kernel from 3.0.1 to 3.0.4 versions compiled by myself with ubuntu patches, config changes only core2 and mtune=native march=native
tested on (3.0.0-8 - 3.0.0-12 kernels compiled by ubuntu team)

I have no idea what causing this issue? Could it be some kind of malware ?

Please any help , advice I really much appreciate.
Thank you,
 
Old 09-27-2011, 09:50 AM   #14
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Slackware
Posts: 14,872
Blog Entries: 2

Rep: Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805Reputation: 3805
Check your harddisks with the manufacturer's diagnosis tools, also have a look for harddisk errors in the logs.
If it is not the disks and not the RAM and also happens with different versions of Linux I would assume that it is a hardware error, reseat all cables, maybe replace them, if that doesn't help I would think it is some serious failure, maybe motherboard or CPU.
 
Old 09-27-2011, 10:29 AM   #15
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 128

Original Poster
Rep: Reputation: Disabled
Question

Quote:
Originally Posted by TobiSGD View Post
Check your harddisks with the manufacturer's diagnosis tools, also have a look for harddisk errors in the logs.
If it is not the disks and not the RAM and also happens with different versions of Linux I would assume that it is a hardware error, reseat all cables, maybe replace them, if that doesn't help I would think it is some serious failure, maybe motherboard or CPU.
Thanks for advice (I already changed data cables before 2 times, power source it should not be - corsair AX1200) I am going to test it anyway ,

but how you can explain that it DID NOT happened on ntfs partition ?
but happened only on ext4 partitions ?
it is the same command , same hard disk , same file but located on ntfs partition.
how it could be hardware failure ?

this is what I do not understand ... .
 
  


Reply

Tags
copy, file, krusader, md5, nautilus


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
md5sum vs. cksum vs. sha1sum bichonfrise74 Linux - Newbie 3 07-22-2009 08:27 PM
encode text with md5sum or sha1sum GATTACA Programming 6 01-24-2008 07:24 AM
Help verifying the download using the included SHA1SUM file EtherGhost Fedora 2 06-02-2007 11:57 AM
krusader copy mode Padawan.AVT Linux - Software 0 04-01-2007 05:26 PM
md5sum .vs. SHA1SUM thelonius Linux - Software 2 07-18-2006 08:44 AM


All times are GMT -5. The time now is 04:42 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration