Md5sum Erratic Behaviour
 

Hi all:
Thanks for reading this post and hope you can help me determine the cause of this erratic behaviour.

### Working on XP
I downloaded the ubuntu-8.04.1-alternate-386.iso image using azureus (torrent), checked the md5 hash using HashTab2 getting a matching hash. Burned .iso to CD using Nero and off to install the latest Ubuntu I went.

During the instalation the CD Integrity Check found the disck to be corrupt so aborted the install.

### Working on DEBIAN
Fired up my Etch install and:

A) Used aria2c to repair a copy of the iso and did 3 consecutive

$ md5sum aria2c-ubuntu-8.04.1-alternate-i386.iso

from /home/user/Lindows/Ubuntu/$ and md5sum produced 3 different sums as you can see below:

f32e51d3a554046cd9f8708d1755ff2d aria2c-ubuntu-8.04.1-alternate-i386.iso
8283e8a05babbe6a4c30df6305147040 aria2c-ubuntu-8.04.1-alternate-i386.iso
06a2ba757a734e698f3fd99df57fe8f2 aria2c-ubuntu-8.04.1-alternate-i386.iso

How Come I got 3 different checksums?

B) Used jigdo to repair another copy of the iso and did 3 consecutive

$ md5sum jigdo0-ubuntu-8.04.1-alternate-i386.iso

and got

d2750b9dfbac0a1a79ceef788a388537 jigdo0-ubuntu-8.04.1-alternate-i386.iso
e8e082bce2a77caa7caf4fb5e98623b0 jigdo0-ubuntu-8.04.1-alternate-i386.iso
7dedb56769ee4ef97749365af42c6120 jigdo0-ubuntu-8.04.1-alternate-i386.iso

Maybe the vfat partition, the '-' character, and/or long path could be an issue so moved the files to /home/user (ext3), croped the names to aria2c.iso and jigdo0.iso, and repeated the md5sums getting another 6 different checksums.

Tried using

$ cat filename | md5sum

with same results.No Luck

## Working Back on XP

Downloaded Md5Calc and WinMd5Sum and checked all four iso files (2 long names and 2 short names) on the fat32 partition getting on all 4 .iso file with all 3 softwares the same (and expected) checksum.

BBD21DED02C06B41C59485266833937A Windows HashTab2
BBD21DED02C06B41C59485266833937A Windows Md5Calc
BBD21DED02C06B41C59485266833937A WinMd5Sum

My Box:
Processor: Intel P4 3.20GHz L1: 16K, L2: 1024K
Ram: 2.0 Gb ( 2 x 1024 Gb)
Video: MSI/ATI PCI-E 128MB
Overclocking: None
MemTest: 2 Months old. OK

Disks: /sda: 80.0 GB, 80026361856 bytes
Device Boot Start End Id System
/sda1 * 3 4501 7 NTFS Wxp
/sda2 4502 7001 83 Linux Ubuntu
/sda4 7002 9729 f W95 Ext'd (LBA)
/sda5 7002 7101 82 Linux swap / Solaris Swap
/sda6 * 7102 8484 83 Linux Debian
/sda7 8485 9729 83 Linux Slack

/sdb: 250.0 GB, 250059350016 bytes
Device Boot Start End Id System
/sdb1 3 21161 7 NTFS MyData
/sdb2 * 21162 30401 c W95 FAT32 (LBA) Win/Linx

Media Drives:
/dev/hda' rwrw-- : 'LITE-ON' 'DVDRW SHW-160P6S' DVDRW
/dev/hdb' rwrw-- : 'LITE-ON' 'LTR-52327S' CDRW

fstab:
<file system> <mount point> <type> <options> <d p>
proc /proc proc defaults 0 0
/dev/sda5 none swap sw 0 0
/dev/sda6 / ext3 defaults,errors=remount-ro 0 1
/dev/hdb /media/cdrom0 udf,iso9660,user,noauto 0 0
/dev/fd0 /media/floppy0 rw,user,noauto 0 0

/dev/sdb2 /home/user/Lindows vfat,defaults,uid=1000,gid=1000
umask=022,iocharset=utf8 0 0

Any Ideas?

Only a guess, but maybe you're being hit by some kernel bug
that flips bits on the fly? I recall there being issues around
2.6.19 (don't tie me down on this) that would lead to data corruption
with certain combinations of chipset and large read/writes

Thanks Tinkster for your Input.

I am currently running Debian 2.6.18-6-686.1-22
and gcc version 4.1.2 20061115

Searching for info on kernel bit-flips, cause, remedies and such, found that:
"Memory errors are proportional to the amount of RAM in a computer as well as the duration of operation."

Since I have 2x1024 DDR2-533 modules and I seldom shutdown my box I thought that this might be the issue; but then again this not an issue when on XP?

Back to it

I think the explanation is each time you did a CD copy, either using aria2c or jigdo, the cd drive returned a different reading, and as result, 3 different copies. Of course, each on of then returned a different checksum.

You can double check this by comparing the copies itself, using the command "cmp".

A defective media can return reading errors at random positions and this can explain why you got 3 different copies.

Thanks Marozsa for your idea. I ran

# cmp -b -l aria2c.iso jigdo0.iso

and it returned NO differing bytes. From my initial post you can see that 3 md5sum runs on the aria2c.iso file gave 3 different sums and that 3 md5sum runs on the jigdo.iso file gave 3 different sums. Moreover the six sums are different from each other BUT here the two files are found to be similar when compared byte by byte.

###NOW

After re-checking aria2c.iso sum on windows I mounted the aria2c.iso (ubuntu) image on VMWare's, fired up a new virtual machine and:

Ran Ubuntu CD-Integrity-check ----> O.K
Ran the install process ----> O.K.

I think's is safe to say that the images are good.

I would really like to find out what is wrong with md5sum, the hardware, the kernel, the libs, or whatever. It really kills me not to know.

I have this issue on one SATA controller and not the other, so I wonder if you could tell me which controller you use?

This is indeed a mystery. Now, I'm curious about :)
Could you, please, try to use sha1sum instead ?

Marozsas:
Here are the results of 3 consecutive runs on the files.

$ sha1sum aria2c.iso
9b5d62f0b1d300610c7f7e0041493ec7ef6497aa aria2c.iso
3bf7ead29390b91d4b2a2830ae89cff14d5ab96d aria2c.iso
686e66f4a258ebdc526888dbdda8fd45c4d0fb2f aria2c.iso

$ sha1sum jigdo0.iso
88b2dd72c37c74d697408adf33df25e2c95265c5 jigdo0.iso
62e4d056a798058cbf527d08b5f949902dd2892a jigdo0.iso
7de1144b622c0770e58134a1e9928913aab91b4b jigdo0.iso

QuakeBoy02:

$ dmesg | grep SATA
ata1: SATA max UDMA/133 cmd 0xEC00 ctl 0xE882 bmdma 0xE400 irq 209
ata2: SATA max UDMA/133 cmd 0xE800 ctl 0xE482 bmdma 0xE408 irq 209

Does this answer your question? If not is there a command to find out the answer to your question from linux?

I googled "UDMA/133" it gave "Maxtor 120GB UDMA/133 5400RPM 2MB IDE HDD"
I can tell you that both my SATA HD are Western Digital 7200RPM and in the bios SATA drives are configured as IDE.

Holy crap ! :scratch:

You told us you have 2 disks, right ?
Could you copy the files to another disk anywhere and repeat the test with the checksum on the copied file ?

I am trying to figure out if it is a problem in the disk or not.
If the checksum is different even on the other disk, could be a problem on the controller/motherboard or in your linux.

If so, the next test would be boot this machine using a live CD, and repeating the test with checksum. If it still return different values at each run, looks like more a problem with your controller/motherbord.

I'm sorry but there is no other way, but pluging the disk on another motherboard and repeating the test.

If it return different checksums on another computer THAN we can be positive about the existence of ghosts, enchantment, true magic, gnomes, big foot, Yeti and similar things :)

Please post the output of "lspci" to list your controllers.

Were you running the md5sum while the iso was being updated or did you wait till all the fileops were done before running md5sum?

Marozsas:
Yes I have two disk and I've already tested on different disk and different filesystem
/sda6 * 7102 8484 83 Linux Debian
/sdb2 * 21162 30401 c W95 FAT32 (LBA) Win/Linx Data

and got the same results.

Working from XP
I copied the iso files to my Vista laptop via Flash_USB device. Checked the integrity of the files first on the USB and then on the Vista Laptop getting the expected and uptimum sums. Burned one iso to a CD on the laptop. Fired Ubuntu's CD integrity Check and found the CD to be sound.

I suspect my debian install and/or debian controler rather than the motherboard because the sums came all ok when ran from XP and the flash_USB and further confirmed on the laptop.
############################################################################################

Quakeboy02

See below lspci output. Note that even though both disk are SATA and both fromw WD, they are assigned different controllers; first HD has an IDE controler and the second HD a SATA cotroler.

00:00.0 Host bridge: Intel Corporation 915G/P/GV/GL/PL/910GL Express Memory Cont roller Hub (rev 04)
00:01.0 PCI bridge: Intel Corporation 915G/P/GV/GL/PL/910GL Express PCI Express Root Port (rev 04)
00:1c.0 PCI bridge: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) PCI Ex press Port 1 (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) US B UHCI #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) US B UHCI #2 (rev 03)
00:1d.2 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) US B UHCI #3 (rev 03)
00:1d.3 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) US B UHCI #4 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) US B2 EHCI Controller (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d3)
00:1f.0 ISA bridge: Intel Corporation 82801FB/FR (ICH6/ICH6R) LPC Interface Brid ge (rev 03)
00:1f.1 IDE interface: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) IDE Controller (rev 03)
00:1f.2 IDE interface: Intel Corporation 82801FB/FW (ICH6/ICH6W) SATA Controller (rev 03)
00:1f.3 SMBus: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) SMBus Contr oller (rev 03)
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)]
01:00.1 Display controller: ATI Technologies Inc RV370 [Radeon X300SE]
02:00.0 Multimedia audio controller: Creative Labs SB Audigy LS
03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller (rev 15)

I think that the SATA disk are configured as IDE on the BIOS. Do you think we've catched our gost? Is there a linux way to re-configre the disks to be either both sata or both IDE?
#############################################################################################

estabroo
If by fileops you mean downloading of the images yes they were finished.
Also the md5 and/or sha1 checks were ran after the previous check had finished.

Hi,

Like you I just had md5sum producing erratic results on my PC running Slackware 12.0 & 12.1. Small files md5 were OK.

It has turned out to be a memory problem. I have 2x1Gb sticks and running Memtest from UBCD (Ultimate Boot CD) I detected lots of errors around 2 locations between 1 and 2 GB. I removed 1 stick and reran Memtest - this turned out to be the faulty stick. Then ran with just the second stick - no errors.

I first spent some time running tests with smartctl, reiserfsck (I use Reiser3 filesystem) and badblocks without finding any problems.

Good luck

I had the same problem, turned out to be a bad stick of RAM. When I was doing small tasks, the problem went unnoticed because the affected area on the stick was around 900MB (1GB sticks). But obviously, as soon as the affected area was accessed, problems would be all over the place - although I didn't really get suspicious until my md5sums started to fail. I had had some warnings, though, like media files becoming corrupt for no apparent reason - but I didn't think any further then.

Thanks Geep995 & Jay73:

I also have two RAM sticks. I ran mem test from Ultimate boot CD for one cycle of tests getting positive results. I heard that it's best to leave the test for several cycles so I 'll leave memtest to run overnight and comment on the results tomorrow AM.