Incomplete bz2 download

Hacker X · 03-13-2007, 07:44 AM

Trying to download a darcs.bz2 file over a modem,
the download keeps stopping before it's done.

The file is supposed to be 1.6M, but it came out
334K and 436K on two different tries.

The starting point is here:
http://darcs.net/DarcsWiki/CategoryBinaries

Click "Linux v1.08"
=> http://evan.martin.googlepages.com/home

Click "darcs v1.08, built ..." and it starts to download,
but just gives up and exits early.
bunzip2 says the file is corrupt.

??

nx5000 · 03-13-2007, 07:54 AM

Here I managed to get the file.

Try a more robust downloader, maybe wget for command line:

wget http://evan.martin.googlepages.com/darcs.bz2

Junior Hacker · 03-13-2007, 07:54 AM

I would'nt trust anything with the name "martin" in it.

Hacker X · 03-13-2007, 09:02 AM

Quote:

Originally Posted by nx5000

Try a more robust downloader, maybe wget for command line

Holy heaps of halibut! 5.1 attempts and it finally finished.
WTF is going on there? I've never seen that before.

PS. Thanks, that worked great.

PPS. JH: Why not?

Junior Hacker · 03-13-2007, 08:38 PM

Just a cheap shot

Elbow in the side of the ribs type comment.
Actually there are many many Martins around, very popular name, have not found an extremely bad egg in the bunch yet.

Glad you got your download.

Hacker X · 10-27-2007, 07:12 PM

Quote:

Originally Posted by Hacker X

Originally Posted by nx5000
> Try a more robust downloader, maybe wget for command line
Thanks, that worked great.

Okay, here's a harder problem. This has happening a couple/few times over the past couple years. Downloading a file over the dialup connection, the download just stops, even with wget. I think wget always stops at the same spot for any given file as Firefox does. The process doesn't exit, it just sits there not doing anything visible. In this case, Firefox and wget both stopped just two SECONDS shy of completing another bz2 file

It's not necessarily the file format though. I think I once had the same problem with a pdf file.

matthewg42 · 10-27-2007, 07:27 PM

As far as firefox when downloading, or wget is concerned it is just a sequence of bytes - you are quite correct that it doesn't have anything to do with the file type.

I remember with horror the days of dialup - always dropping connections at the worst possible time. I found that the biggest two factors governing reliability of dialup service were phone line quality and service provider... it takes a while to find a combination which works. I had a line from comcast and no where I dialled to, the connection would drop periodically. Make sure you have call waiting turned off, and any other service which interjects in a connected call.

With wget you can use the --continue option to complete a partial download, and you can use the --tries option to specify a number of times to retry.

Hacker X · 10-28-2007, 04:01 AM

Quote:

Originally Posted by matthewg42

As far as firefox when downloading, or wget is concerned it is just a sequence of bytes - you are quite correct that it doesn't have anything to do with the file type.

I've heard, I have no recollection where, that the dialup modem has a certain bit sequence that it interprets as a command to terminate the transmission. The only filetype issue I can think of is that maybe some formats don't contain that special code, if it exists, while others do. That's just desperate speculation, though. I have no idea for sure.

Quote:

I remember with horror the days of dialup - always dropping connections at the worst possible time.

The connection is fine, except for this download (and the poky speed).

Great galloping gefiltefish[1], a miracle seems to have occurred. There's a .tgz version of the file I'm trying to download, so I tried that one with wget. Firefox didn't appear to do anything with that file. It would just sit there with the Firefox tab "Loading" for possibly upwards of an an hour or more, until my ISP cut off my connection, without even displaying the download popup window.

wget at least displayed its progress, which was much more reassuring. At the 99% point, it also stopped, so for a while there, all hope appeared to be lost. However, after sitting there for another half hour or so, it eventually retried. Here's the transcript:

Code:

dan[python]$ wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
--02:41:04--  http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
           => `Python-2.5.1.tgz'
Resolving www.python.org... 82.94.237.218
Connecting to www.python.org|82.94.237.218|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11,060,830 (11M) [application/x-tar]

99% [========================================================> ] 11,046,419    --.--K/s    ETA 00:04

03:35:57 (3.28 KB/s) - Read error at byte 11046419/11060830 (Connection timed out). Retrying.

--03:35:58--  http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
  (try: 2) => `Python-2.5.1.tgz'
Connecting to www.python.org|82.94.237.218|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 11,060,830 (11M), 14,411 (14K) remaining [application/x-tar]

100%[+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 11,060,830     4.93K/s

03:36:02 (4.92 KB/s) - `Python-2.5.1.tgz' saved [11060830/11060830]

wget wins again! (checksum is correct) So I guess this thread is now a matter of curiosity rather than panic. It would be nice for future reference to know where the culprit is, now that we know it applies to both .bz2 and .tgz formats.

--Hacker X (a.k.a. "Dan")
www.prairienet.org/~dsb/

[1] My 2nd gratuitous fish reference on this thread, apparently caused by wget's awesome performance.

matthewg42 · 10-28-2007, 04:19 AM

I would imagine that if the modem traffic data can contain a hangup sequence, the modem protocol would be designed to escape it somehow... just like it is possible in C programs to put a double quote in double quotes by prefixing it with a backslash, for example: "like \" this". I would expect this to be automatic.

If you are able to retrieve the file on re-try this doesn't seem likely to be the case.

There does seem to be /something/ fishy going on if it always happens at 99%. If you start to download a different file which is twice as large, does it happen after the same number of bytes, or at 99% of the new file too? Curious!

Mackerel magic!

Hacker X · 10-28-2007, 04:56 AM

Quote:

Originally Posted by matthewg42

I would imagine that if the modem traffic data can contain a hangup sequence, the modem protocol would be designed to escape it somehow... I would expect this to be automatic.

It certainly would be convenient.

Quote:

If you are able to retrieve the file on re-try this doesn't seem likely to be the case.
There does seem to be /something/ fishy going on if it always happens at 99%. If you start to download a different file which is twice as large, ...

This doesn't happen with just any random file. Historically, it's only happened a couple/few times out of scores of downloads over several years.

Quote:

... does it happen after the same number of bytes, or at 99% of the new file too? Curious!

It's 99% of the file in both cases, even though they differ by about 2MB (tgz 11MB, bz2 9MB). That might point away from file format or content, unless there's some content that bz2 can't compress any more than or differently from tgz and gets inserted at the end of the file in both formats. Other than that, it looks like some more insidious phenomenon. Maybe it's GvR's fault

nx5000 · 10-30-2007, 09:32 AM

Humm no clue really..

I would say one of these reasons:
There is a stupid (pleonasm) antivirus somewhere blocking the traffic,
Bad dial-up quality,
Bad modem/router.

Can you try these 2 files:
wget http://kernel.org/pub/linux/kernel/v...h-2.6.23.1.bz2
wget http://kernel.org/pub/linux/kernel/v...geLog-2.6.23.1
then try with replacing http by https and see if commands finish.

Also there is this eicar test files. It is not a virus but is designed to test your antivirus so it is triggered as a "test virus"
Try to download them here (http version should do):
http://www.eicar.org/anti_virus_test_file.htm

Hacker X · 10-31-2007, 05:54 AM

Quote:

Originally Posted by nx5000

I would say one of these reasons:
There is a stupid (pleonasm) antivirus somewhere blocking the traffic,
Bad dial-up quality,
Bad modem/router.

Or some kind of timeout issue? The files were ~10MB each, downloaded over a 56K modem. That takes at least about 1/2 an hour.

Quote:

Can you try these 2 files:
wget http://kernel.org/pub/linux/kernel/v...h-2.6.23.1.bz2
wget http://kernel.org/pub/linux/kernel/v...geLog-2.6.23.1

Worked fine:

Code:

Length: 1,412 (1.4K) [application/x-bzip2]
100%[=========================================================>] 1,412         --.--K/s
05:25:29 (21.18 KB/s) - `patch-2.6.23.1.bz2' saved [1412/1412]

Length: 756 [text/plain]
100%[=========================================================>] 756           --.--K/s
05:26:44 (10.30 MB/s) - `ChangeLog-2.6.23.1' saved [756/756]

Quote:

then try with replacing http by https and see if commands finish.

Excerpts:

Code:

...
Connecting to kernel.org|204.152.191.37|:443... connected.
ERROR: Certificate verification error for kernel.org: self signed certificate
ERROR: certificate common name `localhost.localdomain' doesn't match requested host name `kernel.org'.
...
Connecting to kernel.org|204.152.191.5|:443... connected.
ERROR: Certificate verification error for kernel.org: self signed certificate
ERROR: certificate common name `localhost.localdomain' doesn't match requested host name `kernel.org'.
...

Quote:

Also there is this eicar test files. It is not a virus but is designed to test your antivirus so it is triggered as a "test virus"
Try to download them here (http version should do):
http://www.eicar.org/anti_virus_test_file.htm

I've never knowingly installed any antivirus software. What am I supposed to do, just download the four files at the bottom of the page and look for warnings? And is it safe without my having installed any antivirus code?

nx5000 · 10-31-2007, 09:26 AM

The eicar stuff is a dummy program but its signature is known by antiviruses. It is a safe way to test an antivirus. I thought downloading these small files could show if there is an antivirus on the way.
But I guess it's a ppp problem. Eventhough I've worked on ppp I've never really understood everything to be honest

You should look at the /var/log/pppd.log <-- can't remember the exact logfile name
Change the IP adresses if you post them here.

For the unescape stuffs, it is handled by asyncmap (some caracters are translated), usually set to a000; but I guess you didn't touch the configuration.

This only happens for compressed files?

When I had problems with ppp, I usually went to comp.protocols.ppp groups, there are some very skilled persons overthere.

Hacker X · 10-31-2007, 11:16 AM

Quote:

Originally Posted by nx5000

You should look at the /var/log/pppd.log <-- can't remember the exact logfile name

I don't see anything like that, and the man page says pppd uses syslog. There doesn't seem to be much in `grep pppd /var/log/syslog*`, or anything about pppd in /etc/syslog.conf.

Quote:

This only happens for compressed files?

I don't think so. I think it happened to a pdf file last year sometime.

Quote:

When I had problems with ppp, I usually went to comp.protocols.ppp groups, there are some very skilled persons overthere.

Sounds good. I probably won't ask until/unless I have the problem again, but it helps to know where to ask. Thanks.