LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-08-2017, 12:28 PM   #1
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Rep: Reputation: Disabled
Expect is not working with mutiple return lines in shell script


Hi Guys,

i am new to shell scripting, please help me, i am facing below issue.

MY CODE is as below :-

#!/bin/bash
expect << EOF
set timeout 3
spawn sftp mnaf_ftp@nms.test.systor.st
expect "*#"
send "123456\r"
expect "*#"
send "put MNAF_number_status_2017-11-07_0001.zip\r"
expect "*#"
send "ls -l\r"
expect "*#"
send "pwd\r"
expect "*#"
send "bye\r"
expect "*#"
EOF

OUTPUT :-

[mnaf_ftp@sftp NMSTRA]$ sh test.sh
spawn sftp mnaf_ftp@nms.test.systor.st
123456
put MNAF_number_status_2017-11-07_0001.zip
password
Enter password for mnaf_ftp
Password:
password
Enter password for mnaf_ftp
Password:
password
Enter password for mnaf_ftp
Password:
mnaf_ftp@nms.test.systor.st's password:


Issue is when "sftp mnaf_ftp@nms.test.systor.st" is executed return value for this is below :-

password
Enter password for mnaf_ftp
Password:

with multiple lines so i guess expect command is not able to understand return value.

can someone help me to resolve this issue ?
 
Old 11-08-2017, 01:59 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 23,962

Rep: Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032Reputation: 7032
Please use CODE tags when posting code, to make things easier to read. And the best advice I can offer is (since you're using SFTP), is that you do an SSH keyswap, and then don't even NEED a password. So no need for expect/etc.
 
Old 11-08-2017, 02:02 PM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,693
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Welcome.

Following up on what TB0ne wrote, you're using SFTP so you can and should be using SSH keys for authentication instead of expect. They are secure and easier and will eliminate the perceived need for using expect which is difficult, somewhat insecure, and a bit fragile. With keys you can then also take advantage of the SFTP client's batch mode with the -b option.

Code:
cd ~/.ssh/
ssh-keygen -f systor.rsa -t rsa -b 2048 -C "amalshah120987 at mnaf"
ssh-copy-id -i systor.rsa mnaf_ftp@nms.test.systor.st
ssh-add systor.rsa
Then make a batch file.

Code:
cat << EOT > mnaf.sftp.batch
put MNAF_number_status_2017-11-07_0001.zip
ls -l
pwd
bye
EOT
Then use the batch file with the key.

Code:
sftp -i ~/.ssh/systor.rsa -b ~/.ssh/mnaf.sftp.batch mnaf_ftp@nms.test.systor.st
 
Old 11-08-2017, 05:20 PM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 21,634

Rep: Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189Reputation: 4189
Welcome to LinuxQuestions.

I agree with the above posts that using keys and a batch file would be the better method.

For your information expect is an extension of the tcl language and therefore uses different syntax then bash. You can start expect with the debug option i.e. -d so that you can see it how it is matching the "expect" pattern. Here is a hint to fixing your code.


Code:
#!/bin/bash
expect -d << EOF
set timeout 3
spawn sftp mnaf_ftp@nms.test.systor.st
expect "password:"
send "123456\r"
expect ">"
...(rest of code)
EOF
 
Old 11-09-2017, 12:23 AM   #5
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Issue with expect script from bash script.

Hi Guys,

Thank u so much TB0ne,Turbocapitalist & michaelk for your answer unfortunately i cannot use passwordless sftp because my file which i need to upload is not static.

above issue is just part of my task i can give you idea of my whole task so you can undersatnd and give proper solution.

My task is :

i need to download file from Server X which is part 1 (FTP) and same file i need to upload on server Y (SFTP) which is part 2, i am not facing any issue in part 1 and working fine but in part 2 i am facing difficulty which i described in my previous question for upload .

I tried another method to call a expect script from bash script which also having same issue i am providing code and its out put as below:

My Complete Task code is as below :

Code:
[mnaf_ftp@sftp NMSTRA]$ cat ftpupload.sh
#!/bin/bash
yr=`date +%Y --date="1 days ago"`
dd=`date +%d --date="1 days ago"`
mm=`date +%m --date="1 days ago"`
cd /home/mnaf_ftp/NMSTRA/
expect << EOF
set timeout 3
spawn ftp 172.19.2.165
expect "Name (172.19.2.165:mnaf_ftp):"
send "portal\r"
expect "Password:"
send "portal@123456\r"
expect "*#"
send "cd /Crestel_Team_Data/NMS_TRA/\r"
expect "*#"
send "binary\r"
expect "200 Type set to I."
send "get MNAF_number_status_${yr}-${mm}-${dd}_0001.zip\r"
expect "*#"
send "ls -l\r"
expect "*#"
send "pwd\r"
expect "*#"
send "bye\r"
expect "*#"
EOF

cd /home/mnaf_ftp/NMSTRA/
expect -d sftp_trn.exp MNAF_number_status_${yr}-${mm}-${dd}_0001.zip

exit;
Code for sftp_trn.exp script is as below :

Code:
[mnaf_ftp@sftp NMSTRA]$ cat sftp_trn.exp 
#!/usr/bin/expect -f

cd /home/mnaf_ftp/NMSTRA/

set file [lindex $argv 0];

spawn sftp mnaf_ftp@nms.test.systor.st:/ <<< $"put $file"

expect { 
  -re ".*word:" {
     exp_send "123456\r"
             }
       }
Out put for Overall script which debug mode for expect is as below :

Code:
[mnaf_ftp@sftp NMSTRA]$ sh ftpupload.sh
spawn ftp 172.19.2.165
Connected to 172.19.2.165 (172.19.2.165).
220 Microsoft FTP Service
Name (172.19.2.165:mnaf_ftp): portal
331 Password required for portal.
Password:
230 User portal logged in.
Remote system type is Windows_NT.
ftp> cd /Crestel_Team_Data/NMS_TRA/
250 CWD command successful.
ftp> binary
200 Type set to I.
ftp> get MNAF_number_status_2017-11-08_0001.zip
local: MNAF_number_status_2017-11-08_0001.zip remote: MNAF_number_status_2017-11-08_0001.zip
227 Entering Passive Mode (172,19,2,165,11,205).
125 Data connection already open; Transfer starting.
226 Transfer complete.
495 bytes received in 0.000623 secs (794.54 Kbytes/sec)
ftp> ls -l
227 Entering Passive Mode (172,19,2,165,11,206).
125 Data connection already open; Transfer starting.
11-09-17  12:21AM                  495 MNAF_number_status_2017-11-08_0001.zip
226 Transfer complete.
ftp> pwd
257 "/Crestel_Team_Data/NMS_TRA" is current directory.
ftp> bye
221  
expect version 5.45
argv[0] = expect  argv[1] = -d  argv[2] = sftp_trn.exp  argv[3] = MNAF_number_status_2017-11-08_0001.zip  
set argc 1
set argv0 "sftp_trn.exp"
set argv "MNAF_number_status_2017-11-08_0001.zip"
executing commands from command file sftp_trn.exp
spawn sftp mnaf_ftp@nms.test.systor.st:/ <<< $"put MNAF_number_status_2017-11-08_0001.zip"
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {23263}
Gate keeper glob pattern for '.*word:' is '*word:'. Activating booster.

expect: does "" (spawn_id exp6) match regular expression ".*word:"? Gate "*word:"? gate=no
usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
          [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
          [-o ssh_option] [-P port] [-R num_requests] [-S program]
          [-s subsystem | sftp_server] host
       sftp [user@]host[:file ...]
       sftp [user@]host[:dir[/]]
       sftp -b batchfile [user@]host

expect: does "usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]\r\n          [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]\r\n          [-o ssh_option] [-P port] [-R num_requests] [-S program]\r\n          [-s subsystem | sftp_server] host\r\n       sftp [user@]host[:file ...]\r\n       sftp [user@]host[:dir[/]]\r\n       sftp -b batchfile [user@]host\r\n" (spawn_id exp6) match regular expression ".*word:"? Gate "*word:"? gate=no
expect: read eof
expect: set expect_out(spawn_id) "exp6"
expect: set expect_out(buffer) "usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]\r\n          [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]\r\n          [-o ssh_option] [-P port] [-R num_requests] [-S program]\r\n          [-s subsystem | sftp_server] host\r\n       sftp [user@]host[:file ...]\r\n       sftp [user@]host[:dir[/]]\r\n       sftp -b batchfile [user@]host\r\n"
i think my pattern matching is not correct i am also giving you out put for normal SFTP Command which is generate by below command :

sftp mnaf_ftp@nms.test.systor.st:/ <<< $"put MNAF_number_status_2017-11-08_0001.zip"

Code:
[mnaf_ftp@sftp NMSTRA]$ sftp mnaf_ftp@nms.test.systor.st:/ <<< $"put MNAF_number_status_2017-11-08_0001.zip"
password
Enter password for mnaf_ftp
Password:
so pattern matching unable to identify properly or expect version issue not sure.

Appericiate if you can suggest solution for same. Thanks Guys again for help.

if you guys having any other good method to achieve this task other than this are welcome, Thanks again.
 
Old 11-09-2017, 12:32 AM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,693
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by amalshah120987 View Post
my file which i need to upload is not static.
The batch file has nothing to do with the actual authentication. You can use keys -- with a strong passphrase -- and still automate your task. The batch file can be dyamically generated in several ways. Here's one way:

Code:
#!/bin/sh

yr=$(date +%Y --date="1 days ago")
dd=$(date +%d --date="1 days ago")
mm=$(date +%m --date="1 days ago")

cd /tmp
cat << EOF > mnaf.sftp.batch
cd /Crestel_Team_Data/NMS_TRA/
get MNAF_number_status_${yr}-${mm}-${dd}_0001.zip
EOF

sftp -i ~/.ssh/systor.rsa -b ~/.ssh/mnaf.sftp.batch \
        mnaf_ftp@nms.test.systor.st
 
1 members found this post helpful.
Old 11-09-2017, 01:02 AM   #7
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
[code]
cd ~/.ssh/
ssh-keygen -f systor.rsa -t rsa -b 2048 -C "amalshah120987 at mnaf"
ssh-copy-id -i systor.rsa mnaf_ftp@nms.test.systor.st
ssh-add systor.rsa
When i run above command i am getting below reply : -


Code:
[mnaf_ftp@sftp .ssh]$ ssh-keygen -f systor.rsa -t rsa -b 2048 -C "amalshah120987 at mnaf"
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in systor.rsa.
Your public key has been saved in systor.rsa.pub.
The key fingerprint is:
SHA256:wXDGltQVHxNYYT6djQWRBf43ne5NPaBWDzP/EImsc/8 amalshah120987 at mnaf
The key's randomart image is:
+---[RSA 2048]----+
|      .o+o .o+@Oo|
|       =+ . .=.*o|
|       .o     *.o|
|         . . . +o|
|        S   o*oo+|
|           .o O.+|
|          oo. .=o|
|          .o ..o+|
|              .oE|
+----[SHA256]-----+
[mnaf_ftp@sftp .ssh]$ 
[mnaf_ftp@sftp .ssh]$ ssh-copy-id -i systor.rsa mnaf_ftp@nms.test.systor.st
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "systor.rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
password
Enter password for mnaf_ftp
Password:
exec request failed on channel 0
[mnaf_ftp@sftp .ssh]$ ssh-copy-id -i systor.rsa mnaf_ftp@nms.test.systor.st
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "systor.rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
password
Enter password for mnaf_ftp
Password:
exec request failed on channel 0
[mnaf_ftp@sftp .ssh]$ 
[mnaf_ftp@sftp .ssh]$ ssh-add systor.rsa
Could not open a connection to your authentication agent.
[mnaf_ftp@sftp .ssh]$
is it a problem ?
 
Old 11-09-2017, 01:15 AM   #8
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by michaelk View Post
Welcome to LinuxQuestions.

I agree with the above posts that using keys and a batch file would be the better method.

For your information expect is an extension of the tcl language and therefore uses different syntax then bash. You can start expect with the debug option i.e. -d so that you can see it how it is matching the "expect" pattern. Here is a hint to fixing your code.


Code:
#!/bin/bash
expect -d << EOF
set timeout 3
spawn sftp mnaf_ftp@nms.test.systor.st
expect "password:"
send "123456\r"
expect ">"
...(rest of code)
EOF
i Tried your code i put this code as below :

Code:
[mnaf_ftp@sftp NMSTRA]$ vim test.sh

#!/bin/bash
expect -d << EOF
set timeout 3
spawn sftp mnaf_ftp@nms.test.systor.st
expect "password:"
send "123456\r"
expect ">"
send "put MNAF_number_status_2017-11-08_0001.zip\r"
EOF
Out put of this code is as below

Code:
[mnaf_ftp@sftp NMSTRA]$ sh test.sh 
expect version 5.45
argv[0] = expect  argv[1] = -d  
set argc 0
set argv0 "expect"
set argv ""
executing commands from command file
spawn sftp mnaf_ftp@nms.test.systor.st
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {23453}

expect: does "" (spawn_id exp6) match glob pattern "password:"? no
password
Enter password for mnaf_ftp
Password:
expect: does "password\r\r\nEnter password for mnaf_ftp\r\r\nPassword:" (spawn_id exp6) match glob pattern "password:"? no
expect: timed out
send: sending "wPFtq2Gf\r" to { exp6 }

expect: does "password\r\r\nEnter password for mnaf_ftp\r\r\nPassword:" (spawn_id exp6) match glob pattern ">"? no


expect: does "password\r\r\nEnter password for mnaf_ftp\r\r\nPassword:\r\n" (spawn_id exp6) match glob pattern ">"? no
Connected to nms.test.systor.st.

expect: does "password\r\r\nEnter password for mnaf_ftp\r\r\nPassword:\r\nConnected to nms.test.systor.st.\r\n" (spawn_id exp6) match glob pattern ">"? no
sftp> 
expect: does "password\r\r\nEnter password for mnaf_ftp\r\r\nPassword:\r\nConnected to nms.test.systor.st.\r\nsftp> " (spawn_id exp6) match glob pattern ">"? yes
expect: set expect_out(0,string) ">"
expect: set expect_out(spawn_id) "exp6"
expect: set expect_out(buffer) "password\r\r\nEnter password for mnaf_ftp\r\r\nPassword:\r\nConnected to nms.test.systor.st.\r\nsftp>"
send: sending "put MNAF_number_status_2017-11-08_0001.zip\r" to { exp6 }
[mnaf_ftp@sftp NMSTRA]$
 
Old 11-09-2017, 02:11 AM   #9
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,693
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by amalshah120987 View Post
exec request failed on channel 0
It means that the keys are not getting copied.

Just to be sure, on the local machine which SSH client including version do you have? And on the remote host, which SSH server do you have, including version?

Code:
ssh -V

nc nms.test.systor.st 22
The first line will tell you the client version. The second line will tell you the server version. Just press enter to quit nc then after the remote host announces which version of SSH it has.
 
1 members found this post helpful.
Old 11-09-2017, 02:21 AM   #10
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
It means that the keys are not getting copied.

Just to be sure, on the local machine which SSH client including version do you have? And on the remote host, which SSH server do you have, including version?

Code:
ssh -V

nc nms.test.systor.st 22
The first line will tell you the client version. The second line will tell you the server version. Just press enter to quit nc then after the remote host announces which version of SSH it has.
Here is your output

Code:
[mnaf_ftp@sftp NMSTRA]$ ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
[mnaf_ftp@sftp NMSTRA]$ nc nms.test.systor.st 22
SSH-2.0-CrushFTPSSHD_5
 
Old 11-09-2017, 02:33 AM   #11
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,693
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by amalshah120987 View Post
Code:
SSH-2.0-CrushFTPSSHD_5

Ok. Your SSH client is up-to-date and familiar.

The server is neither familiar, nor is it up-to-date. Unfortunately version 5 is not even listed in their online documentation any more, so it is probably abandoned. If it can handle keys then the transfer of the keys may have to be done by hand and not with ssh-copy-id My suggestion would be to upgrade the SSH server first and then look at the options. If version 8 is not to your taste, then OpenSSH-server should be in the repository and installable. The CrushFTPSSHD online documentation, however, is still talking about using DSA keys which are nowadays known to be insecure...
 
1 members found this post helpful.
Old 11-09-2017, 02:45 AM   #12
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Fixed with java small jar file !!

Hi Guys,

Thanks for your effort and support task was urgent and need to be done in hours only, so i left my first love which is scripting and done the second part of upload with small java file and done coding.

part of code for you info as below :

Code:
[mnaf_ftp@sftp NMSTRA]$ cat ftpupload.sh 
#!/bin/bash
yr=`date +%Y --date="1 days ago"`
dd=`date +%d --date="1 days ago"`
mm=`date +%m --date="1 days ago"`
cd /home/mnaf_ftp/NMSTRA/
expect << EOF
set timeout 3
spawn ftp 172.19.2.165
expect "Name (172.19.2.165:mnaf_ftp):"
send "portal\r"
expect "Password:"
send "portal@crestel\r"
expect "*#"
send "cd /Crestel_Team_Data/NMS_TRA/\r"
expect "*#"
send "binary\r"
expect "200 Type set to I."
send "get MNAF_number_status_${yr}-${mm}-${dd}_0001.zip\r"
expect "*#"
send "ls -l\r"
expect "*#"
send "pwd\r"
expect "*#"
send "bye\r"
expect "*#"
EOF

cd /home/mnaf_ftp/NMSTRA/
java -jar UploadFile.jar MNAF_number_status_${yr}-${mm}-${dd}_0001.zip

mv MNAF_number_status_${yr}-${mm}-${dd}_0001.zip /home/mnaf_ftp/NMSTRA/2017/

exit;
Task is done issue resolved !! Thanks again Guys !!
 
Old 11-09-2017, 03:01 AM   #13
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,693
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Ok. Be sure to file a trouble ticket regarding the SSH server. It looks like it needs to be upgraded or replaced. Breaches are expensive to repair but rather cheap to avoid.
 
Old 11-09-2017, 03:02 AM   #14
amalshah120987
LQ Newbie
 
Registered: Nov 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Sure many Thanks !!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to return from shell 'read' command passed in expect script? legendmac Linux - Newbie 2 05-22-2015 01:15 PM
[SOLVED] Shell Script Check for 2 lines Return? khandu Programming 12 05-12-2014 10:34 AM
Expect script package return ShadowAce Programming 1 02-02-2012 10:09 AM
Can expect return control of a spawned process to a shell script? duffrecords Programming 4 04-28-2010 02:33 PM
return value of expect script .. shriyer Linux - Software 9 07-10-2009 04:06 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration