[BUG] lftp 4.4.10 too slow

zerouno · 10-23-2013, 07:34 AM

There is some bug on lftp 4.4.10 (slackware64-current).

The transfer rate is too slow when I use that version (sometime < 1kb), specially in function 'mirror' during the retrieve file listing (the single file transfer somtime obtains a good speed).

Is not the adsl problem.
I tried to roll-back to 4.4.9 and I obtain a fastest connection.

We (users of slacky.eu community) found the problem during download alienbob's multilib
# lftp -c 'open http://slackware.com/~alien/multilib/ ; mirror current'
about 45min to give directory content

the same command with lftp 4.4.9: about 2 seconds to get directory content.

drmozes · 10-23-2013, 08:34 AM

Quote:

Originally Posted by zerouno

There is some bug on lftp 4.4.10 (slackware64-current).

The transfer rate is too slow when I use that version (sometime < 1kb), specially in function 'mirror' during the retrieve file listing (the single file transfer somtime obtains a good speed).

Is not the adsl problem.
I tried to roll-back to 4.4.9 and I obtain a fastest connection.

We (users of slacky.eu community) found the problem during download alienbob's multilib
# lftp -c 'open http://slackware.com/~alien/multilib/ ; mirror current'
about 45min to give directory content

the same command with lftp 4.4.9: about 2 seconds to get directory content.

Not that I know if there is a bug here, but what I'd do is first issue 'deb 9' inside lftp to turn on max debugging.
It could be that the functionality has changed in the newer version, such that it's issuing different FTP commands than the previous one which is adding delay. I would be more descriptive, but it's been ages since I had to debug FTP so I can't recall - but if you do deb 9 on both versions and see a difference in the FTP commands being issued, it might allow you to revert to the previous behaviour.

zerouno · 10-23-2013, 08:40 AM

I will try to debug. However I tested lftp with http protocol only.

lftp -c 'open http://slackware.com/~alien/multilib/ ; mirror current'

TommyC7 · 10-23-2013, 12:08 PM

Have you tried using lftp for anything else other than alien's multilib packages?

If you could, can you also do:

Code:

echo "debug" >> ~/.lftprc

And then give us the output when you attempt to use lftp? It should help out a bit.

zerouno · 10-23-2013, 03:11 PM

es, I tried with other sites, with the same result.

The ftp protocol have not problems.

In debug mode I can notice the high latency (but the debug mode does not show time of request) and a large number of disconnections for timeout.
Now I've not the pc, so I can't post output, but that output isn't really significant becouse lftp 4.4.9 generate exactly the same (that is the http header of all requests)

willysr · 10-23-2013, 05:24 PM

i can confirm that this bug is reproduceable, even though i can't find any git commits that are relevant to this regression. I have tested to revert some of their commits, but still it has the same effect.

chrisretusn · 10-23-2013, 05:33 PM

Here is a debug output from lftp 4.4.10, while I was trying to mirror Alien Bob's openjdk build. Note the "**** Hit EOF while fetching headers". I've reverted back to 4.4.9.

Code:

---- Resolving host address...
---- 1 address found: 98.158.153.254                   
---- Connecting to taper.alienbase.nl (98.158.153.254) port 80
---- Sending request...                                                                   
---> HEAD /mirrors/people/alien/slackbuilds/openjdk/ HTTP/1.1
---> Host: taper.alienbase.nl
---> User-Agent: lftp/4.4.10
---> Accept: */*
---> Connection: keep-alive
---> 
<--- HTTP/1.1 200 OK                                                                                
<--- Date: Wed, 23 Oct 2013 01:37:34 GMT
<--- Server: Apache/2.2.24 (Unix) DAV/2 PHP/5.3.23
<--- Keep-Alive: timeout=5, max=100
<--- Connection: Keep-Alive
<--- Content-Type: text/html;charset=ISO-8859-1
<--- 
---- Sending request...
---> HEAD /mirrors/people/alien/slackbuilds/openjdk/build/ HTTP/1.1
---> Host: taper.alienbase.nl
---> User-Agent: lftp/4.4.10
---> Accept: */*
---> Connection: keep-alive
---> 
<--- HTTP/1.1 200 OK                  
<--- Date: Wed, 23 Oct 2013 01:37:34 GMT
<--- Server: Apache/2.2.24 (Unix) DAV/2 PHP/5.3.23
<--- Keep-Alive: timeout=5, max=99
<--- Connection: Keep-Alive
<--- Content-Type: text/html;charset=ISO-8859-1
<--- 
---- Closing HTTP connection
---- Connecting to taper.alienbase.nl (98.158.153.254) port 80
---- Sending request...                         
---> GET /mirrors/people/alien/slackbuilds/openjdk/build/ HTTP/1.1
---> Host: taper.alienbase.nl
---> User-Agent: lftp/4.4.10
---> Accept: */*
---> Pragma: no-cache
---> Cache-Control: no-cache
---> Connection: keep-alive
---> 
<--- HTTP/1.1 200 OK                                      
<--- Date: Wed, 23 Oct 2013 01:37:34 GMT
<--- Server: Apache/2.2.24 (Unix) DAV/2 PHP/5.3.23
<--- Content-Length: 3290
<--- Keep-Alive: timeout=5, max=100
<--- Connection: Keep-Alive
<--- Content-Type: text/html;charset=ISO-8859-1
<--- 
---- Receiving body...
---- Received all
---> HEAD /mirrors/people/alien/slackbuilds/openjdk/build/../ HTTP/1.1
---> Host: taper.alienbase.nl
---> User-Agent: lftp/4.4.10
---> Accept: */*
---> Connection: keep-alive
---> 
<--- HTTP/1.1 200 OK                                      
<--- Date: Wed, 23 Oct 2013 01:37:34 GMT
<--- Server: Apache/2.2.24 (Unix) DAV/2 PHP/5.3.23
<--- Keep-Alive: timeout=5, max=99
<--- Connection: Keep-Alive
<--- Content-Type: text/html;charset=ISO-8859-1
<--- 
**** Hit EOF while fetching headers                       
---- Closing HTTP connection
Getting files information (7%) [Delaying before reconnect: 22]

chess · 10-23-2013, 09:10 PM

Looks like Pat has reverted back to 4.4.9 as well in the ChangeLog.txt.

hba · 10-24-2013, 12:48 AM

I was getting the same message "Delaying before reconnect", but i solved it with echoing 'set net:reconnect-interval-base 0' to ~/.lftprc

https://github.com/nihilismus/bob-in...bob-sources.sh