what would you think of this: have you ever experienced a 650 MB corrupted download (with MD5 sums not matching). what if someone told you that you could download something like 5 MB and fix your download.

would you like such an utility? if you mantain a repository, would you consider adding 5 MB for every 650 MB file out there?

feedbacks welcomed.

aj.

That's where P2P is good for, every "piece" of the "thing" you download is checked against corruption

Just to add another bad experience: Downloaded Red-Hat mmm 6? I don't remember, it was on year 1999 using dial-up... as you may guess my download was corrupted! And here where I live, you've to pay for local calls... it was like a month 6 hours per day.. not good :/

As you may guess, it would be nice to have this kind of utility

Unfortunatly there's no magic file since it's imposible to know which part is corrupted unless you check it enterily against the original. Maybe you could download "partial" checksums which could be compared against parts of the file, and if one fails, download that part (like p2p networks.. )

actually there is. it's a very interesting algorithm which is being used currently today especially on sound and data transmission.

i would know how to do this: i'm just asking if you think if this could be interesting

If what you say is true, then all you'd need to reconstruct a "broken" 650MB download is a 5MB file. What if I just pretended to download the 650MB, and then used your magic 5MB file to turn a completely empty file into the required 650MB one? All currently existing file compression algorithms would be redundant!

But this isn't true, and cannot be true.

So there's a catch.

What's the catch?

You can build a correct file if you have more that 1 corupt download but I cant see how a 5 mb file will know which part is corrupted.

i invented nothing new, i'm talking about the Reed-Solomon algorithm.

basically with this algorithm (used to create the famous PAR files) you can rebuid parts of the missing data within specific limits.

you cannot of course rebuilt 650 MB from 5MB. but, if you miss, for instance, 2 MB of the 650 MB file (and in bad MD5 sums is often much less than that, just a bunch of bytes), it is enough to have a 2MB PAR file to fill that empty.

and i confirm, this is true whatever 2 MB you are missing in the 650 MB file.

well, still noone answers my question would this be of interest? this IS possible. i just need to know if it does interest anyone.

So, I'll be the first to answer. As an end-user (currently my computer is not a repository for such big files), I'm interested.

However the last time I had a corrupted file was when downloading an old version of Mandrake via dial-up, some years ago. As the broadband gets more and more popular around the world, I think files up to 600-700MB are no longer a problem. But if you were able (if not yet) to implement this algorithm for files as big as DVD images, then I think your software will be a success.

thanks amwink,

the idea is to have ano size limit, or set it up quite high.

it is no easy task, i'm currently developing it. if someone is interested in contributing just email me, however this is not for the immediate tomorrow.

I'm now writing such prog as a part of one large program. I think, there must be pattern recognition, addtionally to partial checksums. (may be there'll appera one byte and shift all file - so all checksums'll be inequal. But pattern recognition will recognize this)

OK, I think I understand now.

So how much error-correcting code would need to be added to a 650MB download in order to make it correctible by RS?

There is large theory about that codes. AFAIK, RS code is used for protect bytes. (I don't know good)

it's up to human judgement.

i believe that if you perform a full download (which is then found corrupted) you might be missing only some bytes. in this case, a 2 MB file, for instance, is largely enough to cover all errors that the corrupted file has.

Nad0xFF: exactly what are you coding? i want to avoid reinventing the well and duplicating efforts.

That is part of large solution for my work - find different files very fast and make them "equal"

Acidjuice: You should learn a little about error correcting codes in general, there is very good articles which explain basics very good

any links welcomed.

Sorry, I saw it only on Russian, but now found IMHO good articles:
http://mathworld.wolfram.com/Error-CorrectingCode.html
http://guest.engelschall.com/~sb/hamming/
http://www.mdstud.chalmers.se/~md7sh...in/node21.html


----
http://www.eccpage.com/
----



http://en.wikipedia.org/wiki/Error-correcting_code