LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-16-2017, 06:49 AM   #1
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Rep: Reputation: Disabled
Using lftp to connect to a Bitwise SSH server.


Hi,

I'm trying to use lftp with a script that would automatically log in an SSH server and download in multipart mode all the files that are situated on 2 given directories. The names and the number of the files is unknown. I'm running lftp on Windows, but since it's originally a Linux utility and uses the same syntax, I hope to find some useful advice here.

The server is a Bitwise SSH server.

Let's say that the server MyPC is foo.dynu.com, the port is 9999, the username MyUser and the password MyPass. The Server is configured to only accept the use of a key pair, logging with only the username and password is rejected.

If I use CrossFT to log in, the Bitwiser server log shows (note that the user name, password, IPs and ports have been changed to random values):

Code:
desc="Connection accepted.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Client version string received.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters clientVersion="SSH-2.0-JSCH-0.1.53"/>
  </event>
 
desc="Key exchange algorithms.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters kexAlg="diffie-hellman-group14-sha1" cipherAlgIn="aes128-ctr" cipherAlgOut="aes128-ctr" macAlgIn="hmac-md5" macAlgOut="hmac-md5" comprAlgIn="none" comprAlgOut="none"/>
  </event>
 
desc="User authentication failed with partial success.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="TestOnly" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
    <parameters failureReason="NoSignature"/>
    <help message="The public key supplied has been verified; requesting signature."/>
  </event>
 
desc="User authentication succeeded.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="WithSignature" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
  </event>
 
desc="Session channel opened.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
  </event>
 
desc="SFTP protocol version negotiated.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
    <parameters versionRequested="3" versionSelected="3"/>
  </event>
 
desc="Virtual filesystem: get file status.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
desc="Getting file status succeeded.">
      <parameters path="d:\caps" flags="0x8000002F" partialFailure="false"/>
    </sfs>
  </event>
I tried the following script with lftp:

Code:
debug 10
set sftp:connect-program "ssh -a -x -i D:/m1/.ssh/id_rsaNoPass.ppk"
open MyUser,MyPass@foo.dynu.com:9999
cd /d/caps
cat *
and I get on the terminal:

Code:
nt{lftp}% lftp test
D:/m1/soft/l/lftp/lftp-4.7.5.win64-openssl/bin/lftp.exe -f test
Password:
---- Resolving host address...
buffer: EOF on FD 5
---- 1 address found: 222.22.222.222
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
buffer: EOF on FD 5
<--- !!!T!!!!!!,BssLoginTimeout: user authentication timeout!!!!-?w%SNJ?G
**** Peer closed connection
---- Closing control socket
---- dns cache hit
---- attempt number 2 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
cd `/d/caps' [FEAT negotiation...]
with the following log:


Code:
desc="Connection accepted.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters disconnectReason="EofReceived"/>
    <help message="The client has disconnected the session by sending EOF."/>
  </event>
So, there are a few problems:

- lftp asks for a password. But the private key does not need a passphrase, and the user password is provided in the script. So, what is this password asked for?

- the server acknowledges the connection, so it's not a firewall problem. But then, it stalls, there is no key pair authentication.

What am I missing?
 
Old 04-16-2017, 12:15 PM   #2
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,800

Rep: Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725
Quote:
Originally Posted by Bob4K View Post
Hi,

I'm trying to use lftp with a script that would automatically log in an SSH server and download in multipart mode all the files that are situated on 2 given directories. The names and the number of the files is unknown. I'm running lftp on Windows, but since it's originally a Linux utility and uses the same syntax, I hope to find some useful advice here.

The server is a Bitwise SSH server.

Let's say that the server MyPC is foo.dynu.com, the port is 9999, the username MyUser and the password MyPass. The Server is configured to only accept the use of a key pair, logging with only the username and password is rejected.

If I use CrossFT to log in, the Bitwiser server log shows (note that the user name, password, IPs and ports have been changed to random values):

Code:
desc="Connection accepted.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Client version string received.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters clientVersion="SSH-2.0-JSCH-0.1.53"/>
  </event>
 
desc="Key exchange algorithms.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters kexAlg="diffie-hellman-group14-sha1" cipherAlgIn="aes128-ctr" cipherAlgOut="aes128-ctr" macAlgIn="hmac-md5" macAlgOut="hmac-md5" comprAlgIn="none" comprAlgOut="none"/>
  </event>
 
desc="User authentication failed with partial success.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="TestOnly" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
    <parameters failureReason="NoSignature"/>
    <help message="The public key supplied has been verified; requesting signature."/>
  </event>
 
desc="User authentication succeeded.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="WithSignature" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
  </event>
 
desc="Session channel opened.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
  </event>
 
desc="SFTP protocol version negotiated.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
    <parameters versionRequested="3" versionSelected="3"/>
  </event>
 
desc="Virtual filesystem: get file status.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
desc="Getting file status succeeded.">
      <parameters path="d:\caps" flags="0x8000002F" partialFailure="false"/>
    </sfs>
  </event>
I tried the following script with lftp:

Code:
debug 10
set sftp:connect-program "ssh -a -x -i D:/m1/.ssh/id_rsaNoPass.ppk"
open MyUser,MyPass@foo.dynu.com:9999
cd /d/caps
cat *
and I get on the terminal:

Code:
nt{lftp}% lftp test
D:/m1/soft/l/lftp/lftp-4.7.5.win64-openssl/bin/lftp.exe -f test
Password:
---- Resolving host address...
buffer: EOF on FD 5
---- 1 address found: 222.22.222.222
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
buffer: EOF on FD 5
<--- !!!T!!!!!!,BssLoginTimeout: user authentication timeout!!!!-?w%SNJ?G
**** Peer closed connection
---- Closing control socket
---- dns cache hit
---- attempt number 2 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
cd `/d/caps' [FEAT negotiation...]
with the following log:


Code:
desc="Connection accepted.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters disconnectReason="EofReceived"/>
    <help message="The client has disconnected the session by sending EOF."/>
  </event>
So, there are a few problems:

- lftp asks for a password. But the private key does not need a passphrase, and the user password is provided in the script. So, what is this password asked for?

- the server acknowledges the connection, so it's not a firewall problem. But then, it stalls, there is no key pair authentication.

What am I missing?
Not sure what you are missing, but when keys are involved I pass the logon name and password on the command line. It is critical to know if anyone else can log in and see the process list, as this exposes the name and password in the process list.

In these cases, by the way, sftp behavior and lftp behavior are quite different. Lftp may delay authentication on the remote server until it needs a folder or file list, or otherwise actually needs to transfer data. SFTP does the authentication up front.

Try this manually first, before turning it over to a script, so you get a chance to observe and troubleshoot during the dialog.
 
Old 04-16-2017, 12:26 PM   #3
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Original Poster
Rep: Reputation: Disabled
Thanks for the answer.

Unfortunately, this turns out to be Windows ssh problem.

My script had a few errors, it should be:

Code:
debug 10
set sftp:connect-program 'D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i d:/m1/.ssh/id_rsaNP'
open sftp://MyUser:dummy@foo.dynu.com:9999
It turned out that I needed the full path to shh.

But still, I get:

Code:
---- attempt number 1 (max_retries=1000)
---- Running connect program (D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i id_rsaNP -s -l MyUser -p 9999 rrg.dynu.co
m sftp)
---> sending a packet, length=5, type=1(INIT), id=0
<--- OpenSSH_7.4p1, OpenSSL 1.0.2j  26 Sep 2016
<--- debug1: Connecting to foo.dynu.com [222.22.222.222] port 9999.
<--- debug1: Connection established.
<--- Could not create directory '/home/m1/.ssh'.
<--- debug1: key_load_public: No such file or directory
So, ssh is unable to load "d:/m1/.ssh/id_rsaNP". I tried something like "/cygwin/d/m1/.ssh/id_rsaNP", but that does not work either.

This forum is likely not the right place to address this Windows/Cygwin port to lftp/ssh, any idea where I could find some help?
 
Old 04-16-2017, 02:08 PM   #4
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,800

Rep: Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725
I use primarily windows tools on windows (putty and winscp), and use cygwin for xwindows integration etc.
Everything I have done in cygwin has pretty much worked as if it were running under linux, but there are path restrictions and differences. I would look into why it thinks it should create that folder, and why it cannot.


Meanwhile, I might find another way to do the transfer. This is only ONE way, and there are many more tools than lftp!
 
1 members found this post helpful.
Old 04-16-2017, 02:09 PM   #5
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Original Poster
Rep: Reputation: Disabled
I tried an older version, 4.4.15 32 bits instead if the current (Windows) 4.4.17 64 bits, and it works nicely.

In case anybody is interested, here is a working script on Windows:

Code:
set sftp:connect-program 'D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i id_rsaNP'
open sftp://MyUser:dummy@foo.dynu.com:9999
cd /d/caps
set mirror:use-pget-n 8
mirror /d/caps /cygdrive/d/caps
 
Old 04-16-2017, 02:14 PM   #6
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by wpeckham View Post

Meanwhile, I might find another way to do the transfer. This is only ONE way, and there are many more tools than lftp!
Windows clients that can do multi-part transfers are not very good. The best is CrossFTP, who makes a bunch of small files, and then rejoin them, which is a waste fo time.

lftp creates a large file from the get go, and then fills it. That's the main reason I wanted to try it.
 
Old 04-16-2017, 02:17 PM   #7
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,800

Rep: Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725
Quote:
Originally Posted by Bob4K View Post
I tried an older version, 4.4.15 32 bits instead if the current (Windows) 4.4.17 64 bits, and it works nicely.

In case anybody is interested, here is a working script on Windows:

Code:
set sftp:connect-program 'D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i id_rsaNP'
open sftp://MyUser:dummy@foo.dynu.com:9999
cd /d/caps
set mirror:use-pget-n 8
mirror /d/caps /cygdrive/d/caps
Ahaa! Do you think that you tripped over a bit-width code issue? I had not heard of one, but this is interesting. I may try it myself to see if I can replicate the behavior.
meanwile, you might want to mark the thread solved so others will be able to find this solutions!
 
Old 04-16-2017, 02:26 PM   #8
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by wpeckham View Post
Ahaa! Do you think that you tripped over a bit-width code issue? I had not heard of one, but this is interesting. I may try it myself to see if I can replicate the behavior.
I tested the 64 bit 4.4.15 version, and it works too, so it's a bug in 4.7.7.

Quote:
meanwile, you might want to mark the thread solved so others will be able to find this solutions!
Done.

That said, CrossFTP appears to be better overall: I reach 16 MBs with it, while it peaks at 10 with lftp. So, despite a better way to handle multi-part, lftp takes more time to download large files.

Well, at least I had fun making it work...
 
Old 04-16-2017, 02:55 PM   #9
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Original Poster
Rep: Reputation: Disabled
Interesting find: if I download to an SSD rather than a regular HDD, the speed jumps to 21 MBs.

So, I'll be replacing CrossFTP with lftp, with more speed and the added bonus of a scripted automated download.
 
Old 04-17-2017, 06:33 AM   #10
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,800

Rep: Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725Reputation: 1725
Wonderful! I am not sure I ever saw anyone advance lftp for speed. It is a great tool for transfer automation, and handles MANY protocols, but it may not be as fast as a dedicated tool that does or checks less. I never tested performance for lftp.

Very glad you solved this!
 
Old 04-17-2017, 11:49 AM   #11
Bob4K
LQ Newbie
 
Registered: Apr 2017
Posts: 8

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by wpeckham View Post
Wonderful! I am not sure I ever saw anyone advance lftp for speed. It is a great tool for transfer automation, and handles MANY protocols, but it may not be as fast as a dedicated tool that does or checks less. I never tested performance for lftp.

Very glad you solved this!
Every SFTP client I tried on Windows runs at max at 1.5MBs (that's lftp speed) by connection, despite the fact that I have a Gb FTTH connection. Filezilla maxes out at .5MBs.

So, segmented downloads are the only way to get speed, and I found only a few clients that support that:

- Bitkinex (I think), but it's unusable on a high res monitor due to its use of tiny fonts.

- SmartFTP. But it uses an incredibly bloated UI, and does something to the Bitwise server that often leaves it on a loop at 100% cpu. Also, it requires to install a Microsoft patch on Windows 7 that prevents the Windows Resource Monitor to work.

- CrossFTP, which is ok but annoying in the way it creates multiple small files before joining them.

lftp with 32 connections works the best, but 32 simultaneous accesses is too much for a regular hard drive, so it needs an SSD.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] lftp - perform task only if connect successful fgcoppola Linux - Newbie 1 05-18-2014 03:21 PM
LFTP can't connect to Proftpd server (hangs) guzabi Linux - Server 10 12-28-2010 03:17 AM
lftp cant' connect tekmann33 Linux - Newbie 2 10-02-2007 02:45 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration