md5sum sha1sum are different after copy file/s by cp command, krusader or nautilus
 

Hi,

I have a problem when I made copy of big file/s. I have been trying to check their md5 and sha1 (md5sum -b file.vdi , sha1sum -b file.vbi) but copied file differs against the original file. Files are copied between ext4 partitions.

I have tried to copy file by cp command, krusader or nautilus - it has always different md5 or sha1.

Could anybody please explain what am I doing wrong and what should I do that copied file should have same content ?

thank you.

This behavior can be caused by hardware failure, I would recommend to check the health of the RAM (using Memtest86+) and the disk(s) (using the manufacturers diagnosis tool).
Something like that may also be caused by a not stable running overclock, does your system run within its specs?

Hi,

Thanks for quick reaction. I have checked before many times memory by memetest86+ - it is ok according to this test.
I have many SATA III/II hard drives but in windows 7 this errors does not exist even if I made copy from different partition/s BUT between same disk/s even if partitions are ntfs. And this behaviour is only with files bigger that tens of gigabytes (vrtualbox vdi files).
When copy ISO files ( up to 4.7Gbytes) - it is correct with nautilus but it is not correct with krusader for same files, cp terminal command I did not check so many times md5 or sha1 so I cannot say for now.
I did no check all hard drives which I am actively using with manufacturer diagnostic tools yet because those disks are correct in windows. So , I am not assuming that only ext4 or btrf partitions have problems and ntfs partitions are correct on the same hard drives :-) . But I am going to try it some night/s.
According to those it seems to be only problem because of overclock.
I have maximus iv extreme R3, i7-2600k(3.4GHz) - OC:5.1GHz , memory OC:1600MHz , 9-9-8-24-1 - it should not be problem , graphic card is not OC.
Which is strange as well because win764bit works more or less properly - not with such problems (have another :-( ).

P.S. would it be possible that antivirus is doing this things (I have avg and clamav) ?
(Hope it is not some kind of malware )

try using dd and see if you still have the same problem.

hopefully you don't have to do this for small files but dd is a better choice for large files.

Unfortunately sha1 is different by using dd ... AND

I have done sha1 on the same file and ... result is different:

$ time sha1sum -b file-ubu1.vdi
daef63a16afaaf4ef7cf596f62c528d543597739 *file-ubu1.vdi

real 6m9.180s
user 1m45.150s
sys 0m10.450s
:-) 2.02 1.97 1.71 1/713 6384 pts/2 jobs=0 16:32:22 /home/_virtual.readANDwrite
u1@PCEUBU1 1:2006:7 $ time sha1sum -b file-ubu1.vdi
582acf68f271e44540c5bb5e94ed22694ebad1df *file-ubu1.vdi

real 6m6.380s
user 1m47.040s
sys 0m9.620s

Now I am completely confused. Does journaling of ext4 or file caching doing this ? Can anybody explain what I am doing wrong ?
thank you for any hint.

I forgot to mentioned that I used:

dd if=/dir/to/source-file/file-ubu1.vdi of=/dir/to/dest-file/file-ubu1.vdi
and
dd bs=1G oflag=dsync,sync,nocache conv=fdatasync,fsync if=/dir/file of=/dir/file

file has 47 Gbytes.

sha1 is different.

You say that it should work and that Windows works more or less, so I assume that your overclock is not stable. Linux is not Windows, it may act different to such an unstable system.
Try it again without the overclock and see if that works. If it works your overclock is not stable and causing your issues.

wow, that's a big file. What transfer rate did dd default to with the first command? That must have taken a while.

I setup BIOS to manufacturer settings and it did not helped. It behaves like following:

I have checked sha1sum of tens of .ISO files and .vdi files - NOT COPIED. It looks like files bigger than 3.8(3.9)Gbytes have always different sha1 result measured by sha1sum.

47090896896 bytes (47 GB) copied, 595.566 s, 79.1 MB/s

the best result:
# real 6m6.380s
# user 1m47.040s
# sys 0m9.620s
the worst result:
# real 10m52.592s
# user 1m55.320s
# sys 0m10.430s

I have really fun with this:
I have tried to use groovy (version 1.7.4) source code taken from: http://groovy.codehaus.org/Calculati...or+large+files
and
I tried openssl sha1 file.vdi (version OpenSSL 0.9.8o 01 Jun 2010)

and guess what ?
It behaves exactly the same way (sha1sum or sha1deep) as I described:
for files bigger than ~4Gbytes it had different sha1.
I did not copy files - just repeat the same command on same file.

Summary:

I do not have to copy file,
It have happened if I run twice or more times same check-sum command on the same file.
File has to be bigger than approximately 3.9 (3.8-4.2) Gigabytes.
Tested with sha1sum -b , md5sum -b , sha1deep , md5deep , openssl sha1 , groovy sha1 algorithm.
I made test on four EXT4 partitions - they are located on 4 different hard drives (2 external and 2 internal HDDs/USB flash drive) on one computer (other 3 computers does not have this issue).
NTFS partitions DO NOT have these check-sum problems measured on the same hard drive where is located ext4 partition as well (3 HDDs). ... (This I assume eliminates possibility of hardware error/s of hard drive/s)
On BTRFS partition it is not possible to check because of I/O problems:
md5sum -b ubu1104.vdi
# md5sum: ubu1104.vdi: Input/output error
openssl sha1 ubu1104.vdi
# Read Error in ubu1104.vdi
# 5210:error:0200B005:system library:fread:Input/output error:bss_file.c:202:
# 5210:error:20082002:BIO routines:FILE_READ:system lib:bss_file.c:203:
sha1sum -b ubu1104.vdi
# sha1sum: ubu1104.vdi: Input/output error

Run sha1sum from live-cd ubuntu 11.04 64bit have same results.
No overclocking.
Tested on linux kernel from 3.0.1 to 3.0.4 versions compiled by myself with ubuntu patches, config changes only core2 and mtune=native march=native
tested on (3.0.0-8 - 3.0.0-12 kernels compiled by ubuntu team)

I have no idea what causing this issue? Could it be some kind of malware ?

Please any help , advice I really much appreciate.
Thank you,

Check your harddisks with the manufacturer's diagnosis tools, also have a look for harddisk errors in the logs.
If it is not the disks and not the RAM and also happens with different versions of Linux I would assume that it is a hardware error, reseat all cables, maybe replace them, if that doesn't help I would think it is some serious failure, maybe motherboard or CPU.

Thanks for advice (I already changed data cables before 2 times, power source it should not be - corsair AX1200) I am going to test it anyway ,

but how you can explain that it DID NOT happened on ntfs partition ?
but happened only on ext4 partitions ?
it is the same command , same hard disk , same file but located on ntfs partition.
how it could be hardware failure ?

this is what I do not understand ... .