LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-30-2014, 03:39 AM   #1
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Rep: Reputation: Disabled
SFTP Bash Script


Hey Guys,

I'm still playing around with some FTP Scripts that I have been working on for some time.

I have needed to adapt my original script to use SFTP. However many other things have changed as a result.

I've got the script working using Expect to pass the password (keys aren't an option unfortunately) I do understand that Expect is not secure, and therefore negates the need for SFTP but unfortunately there's no other option I think.

As it stands the scripts work.. but I want to add some logging.

I've read lots about this, and I know that you can examine the return code of the SFTP Session, however to do this you have to run SFTP in batch mode? However, can you run SFTP in batch mode and still pass the password using Expect?

Code so far:

Code:
D=$(date +%d%m%y)
HOST=XX.XX.XX.XX
USER=ftpxfer
PASS=xxxxx
FTPLOG=/tmp/ftplogfile2
FILES=*$D*.csv.pgp
PORT="Port=10022"

#Change into the working directory
cd /sitsimp

#Download all *.csv.pgp files that contain today's date
/usr/bin/expect <<!
spawn sftp -o$PORT $USER@$HOST
expect "password:"
send "$PASS\r"
expect "sftp>"
send "mget $FILES \r"
send "bye \r"
!

for i in $FILES; do
        gpg --batch --passphrase-file /root/.gpgpass --output /storage/${i%.pgp} --decrypt $i
done
I've tried just adding:

Code:
spawn sftp -b -o$PORT $USER@$HOST > $FTPLOG
But unfortunately it does not work.

I've also tried adding some of the SFTP Commands into a batch file, however it does not work either, assuming this is to do with the session being run within expect?

Any help, as always is greatly appreciated.

Thanks
Jon
 
Old 06-30-2014, 09:26 AM   #2
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
UPDATE:

I think I have found a rough way of doing it.. it seems to work okay. However I now have another problem! I'm now running two FTP Sessions, one to capture the list of files and then another to download the files if they exist.

When I run the IF statement on the variable that holds the list of files it seems to return nothing even though the list contains a valid file:

Code:
#!/bin/bash

D=$(date +%d%m%y)
HOST=X.X.X.X
USER=ftpxfer
PASS=xxxxxx
FTPLOG=/tmp/ftplogfile2
FILES=*$D*.csv.pgp
PORT="Port=10022"
FTPLIST=/tmp/ftplist

cd /sitsimp

/usr/bin/expect <<! > $FTPLIST 
spawn sftp -o$PORT $USER@$HOST
expect "password:"
send "$PASS\r"
expect "sftp>"
send "ls\r"
send "bye\r"
expect eof
!

if grep "$FILES" $FTPLIST ; then

/usr/bin/expect <<! > $FTPLOG 
spawn sftp -o$PORT $USER@$HOST
expect "password:"
send "$PASS\r"
expect "sftp>"
send "mget $FILES \r"
send "bye \r"
expect eof
!

for i in $FILES; do 
	gpg --batch --passphrase-file /root/.gpgpass --output /storage/${i%.pgp} --decrypt $i
done

if fgrep "100%" $FTPLOG ; then
      	else	
       		 echo -e "File did not download, send email!"
fi
else
	echo "No file today"
fi
The bit that seems to have the problem is:

Code:
if "$FILES" $FTPLIST; then
It returns no file every time.

If I add:
Code:
cat $FTPLIST
I can see the list of files, so they are there to be searched.

I got over the original problem by using a similar method, by redirecting the whole session to a file and searching for 100% which seems to work, albeit not elegant.
 
Old 06-30-2014, 09:35 AM   #3
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,120

Rep: Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876
this is what i do for the mainframe which doesnt have an ssh server:
Code:
### download spool data ###
/usr/bin/ftp -n -v $server <<END_SCRIPT
quote USER $user
quote PASS $pass
quote site FILE=JES
prompt
get $job1
quit
END_SCRIPT
the ideal would always be to use keys (it would be far easier to script and much more secure).

Last edited by schneidz; 06-30-2014 at 09:36 AM.
 
Old 06-30-2014, 09:36 AM   #4
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
I've looked at this method before, but unfortunately in this instance I have to use SFTP.
 
Old 06-30-2014, 09:43 AM   #5
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,120

Rep: Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876
that sux. you can continue to try to hax something up but it mite be less time consuming and energy to schedule a meeting with your executive manager, development manager and automation manager and convince them that using keys will be easier and more secure (you will probably need to cook up some numbers on cost savings for programming/maintenance as well as in the event your company gets sued for privacy breaches).
 
Old 06-30-2014, 09:44 AM   #6
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
I agree, and I've tried the Keys route.. but unfortunately we're working with a 3rd party who are not prepared to negotiate on the methods used
 
Old 06-30-2014, 09:54 AM   #7
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,120

Rep: Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876
this:
Code:
 if "$FILES" $FTPLIST
mite need to be this:
Code:
if [ "`grep \"$FILES\" $FTPLIST`" ]
 
Old 06-30-2014, 10:29 AM   #8
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,136

Rep: Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336Reputation: 1336
Quote:
Originally Posted by jonnybinthemix View Post
Code:
FILES=*$D*.csv.pgp
...
if grep "$FILES" $FTPLIST ; then
grep takes a regexp, but you have put a glob pattern in $FILES, translate it to a regexp:
Code:
FILES=$D'.*\.csv\.pgp'
 
1 members found this post helpful.
Old 06-30-2014, 11:25 AM   #9
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora
Posts: 3,935
Blog Entries: 5

Rep: Reputation: Disabled
Quote:
Originally Posted by jonnybinthemix
Code:
if grep "$FILES" $FTPLIST
This test doesn't make sense. Your grep expression contains one or more files (or a literal, if there's no match).

What you really care about is whether $FTPLIST contains any results from ls, right?

Code:
if [ -s "$FTPLIST" ] ; then
http://www.tldp.org/LDP/abs/html/fto.html

--------------

edit: If the sftp session produces non- ls chatter, then my test won't work. You can instead grep for a proper regex, like others mentioned, e.g.:

Code:
grep -qE '\.csv\.pgp$' "$FTPLIST"

if [ $? -eq 0 ] ; then

        # There's a match, so grab the files next.
        #

Last edited by anomie; 06-30-2014 at 03:55 PM.
 
1 members found this post helpful.
Old 06-30-2014, 02:42 PM   #10
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,188

Rep: Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184
Quote:
Originally Posted by jonnybinthemix View Post
However, can you run SFTP in batch mode and still pass the password using Expect?
Yep, rather counterintuitively you want to set -oBatchMode=no in addition to -b. Quickly scanning your first post, I would just use sshpass instead of expect but each to their own. Another thing, you want to "get" your files. No need for "mget" with sftp, in fact on MacOS and Slackware sftp has no mget command, just use get with wildcards in the filename. I think you could probably do something like this to achieve what you desire:

Code:
export SSHPASS=xxxxx
echo ls | sshpass -e sftp -oBatchMode=no -P $PORTNUMBER -b- $USER@$HOST | tee $FTPLIST
echo "get $FILES" | sshpass -e sftp -oBatchMode=no -P $PORTNUMBER -b- $USER@$HOST | tee $FTPLOG

Last edited by ruario; 06-30-2014 at 03:05 PM. Reason: added in example to create list of files
 
Old 06-30-2014, 03:00 PM   #11
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,188

Rep: Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184Reputation: 1184
Your other option would be to use a different client that supports the sftp protocol and can be scripted, e.g. lftp or curl (when compiled with libssh2 present). In my experience curl is very slow when using sftp (though that might just be for uploads), so I would personally go for lftp. Unless speed is not an issue because curl does work nicely in all other aspects.
 
Old 07-03-2014, 05:45 AM   #12
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Thanks guys that really helped.

Adding the glob pattern did the trick perfectly

Thanks again for all help
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sftp connection and bash together leandrobrancher Linux - Server 2 12-14-2013 12:26 PM
SSH connection from BASH script stops further BASH script commands tardis1 Linux - Newbie 3 12-06-2010 08:56 AM
[SOLVED] SFTP file upload bash script issue. moodah Programming 1 10-07-2010 10:55 PM
Automating SFTP in Bash with password? arashi256 Linux - Newbie 4 07-14-2010 07:35 AM
Bash and sftp solar05 Linux - Newbie 5 07-01-2009 01:36 AM


All times are GMT -5. The time now is 07:11 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration