[SOLVED] SFTP Bash Script

jonnybinthemix · 06-30-2014, 03:39 AM

Hey Guys,

I'm still playing around with some FTP Scripts that I have been working on for some time.

I have needed to adapt my original script to use SFTP. However many other things have changed as a result.

I've got the script working using Expect to pass the password (keys aren't an option unfortunately) I do understand that Expect is not secure, and therefore negates the need for SFTP but unfortunately there's no other option I think.

As it stands the scripts work.. but I want to add some logging.

I've read lots about this, and I know that you can examine the return code of the SFTP Session, however to do this you have to run SFTP in batch mode? However, can you run SFTP in batch mode and still pass the password using Expect?

Code so far:

Code:

D=$(date +%d%m%y)
HOST=XX.XX.XX.XX
USER=ftpxfer
PASS=xxxxx
FTPLOG=/tmp/ftplogfile2
FILES=*$D*.csv.pgp
PORT="Port=10022"

#Change into the working directory
cd /sitsimp

#Download all *.csv.pgp files that contain today's date
/usr/bin/expect <<!
spawn sftp -o$PORT $USER@$HOST
expect "password:"
send "$PASS\r"
expect "sftp>"
send "mget $FILES \r"
send "bye \r"
!

for i in $FILES; do
        gpg --batch --passphrase-file /root/.gpgpass --output /storage/${i%.pgp} --decrypt $i
done

I've tried just adding:

Code:

spawn sftp -b -o$PORT $USER@$HOST > $FTPLOG

But unfortunately it does not work.

I've also tried adding some of the SFTP Commands into a batch file, however it does not work either, assuming this is to do with the session being run within expect?

Any help, as always is greatly appreciated.

Thanks
Jon

jonnybinthemix · 06-30-2014, 09:26 AM

UPDATE:

I think I have found a rough way of doing it.. it seems to work okay. However I now have another problem! I'm now running two FTP Sessions, one to capture the list of files and then another to download the files if they exist.

When I run the IF statement on the variable that holds the list of files it seems to return nothing even though the list contains a valid file:

Code:

#!/bin/bash

D=$(date +%d%m%y)
HOST=X.X.X.X
USER=ftpxfer
PASS=xxxxxx
FTPLOG=/tmp/ftplogfile2
FILES=*$D*.csv.pgp
PORT="Port=10022"
FTPLIST=/tmp/ftplist

cd /sitsimp

/usr/bin/expect <<! > $FTPLIST 
spawn sftp -o$PORT $USER@$HOST
expect "password:"
send "$PASS\r"
expect "sftp>"
send "ls\r"
send "bye\r"
expect eof
!

if grep "$FILES" $FTPLIST ; then

/usr/bin/expect <<! > $FTPLOG 
spawn sftp -o$PORT $USER@$HOST
expect "password:"
send "$PASS\r"
expect "sftp>"
send "mget $FILES \r"
send "bye \r"
expect eof
!

for i in $FILES; do 
	gpg --batch --passphrase-file /root/.gpgpass --output /storage/${i%.pgp} --decrypt $i
done

if fgrep "100%" $FTPLOG ; then
      	else	
       		 echo -e "File did not download, send email!"
fi
else
	echo "No file today"
fi

The bit that seems to have the problem is:

Code:

if "$FILES" $FTPLIST; then

It returns no file every time.

If I add:

Code:

cat $FTPLIST

I can see the list of files, so they are there to be searched.

I got over the original problem by using a similar method, by redirecting the whole session to a file and searching for 100% which seems to work, albeit not elegant.

schneidz · 06-30-2014, 09:35 AM

this is what i do for the mainframe which doesnt have an ssh server:

Code:

### download spool data ###
/usr/bin/ftp -n -v $server <<END_SCRIPT
quote USER $user
quote PASS $pass
quote site FILE=JES
prompt
get $job1
quit
END_SCRIPT

the ideal would always be to use keys (it would be far easier to script and much more secure).

jonnybinthemix · 06-30-2014, 09:36 AM

I've looked at this method before, but unfortunately in this instance I have to use SFTP.

schneidz · 06-30-2014, 09:43 AM

that sux. you can continue to try to hax something up but it mite be less time consuming and energy to schedule a meeting with your executive manager, development manager and automation manager and convince them that using keys will be easier and more secure (you will probably need to cook up some numbers on cost savings for programming/maintenance as well as in the event your company gets sued for privacy breaches).

jonnybinthemix · 06-30-2014, 09:44 AM

I agree, and I've tried the Keys route.. but unfortunately we're working with a 3rd party who are not prepared to negotiate on the methods used

schneidz · 06-30-2014, 09:54 AM

this:

Code:

 if "$FILES" $FTPLIST

mite need to be this:

Code:

if [ "`grep \"$FILES\" $FTPLIST`" ]

ntubski · 06-30-2014, 10:29 AM

Quote:

Originally Posted by jonnybinthemix

Code:

FILES=*$D*.csv.pgp
...
if grep "$FILES" $FTPLIST ; then

grep takes a regexp, but you have put a glob pattern in $FILES, translate it to a regexp:

Code:

FILES=$D'.*\.csv\.pgp'

anomie · 06-30-2014, 11:25 AM

Quote:

Originally Posted by jonnybinthemix

Code:

if grep "$FILES" $FTPLIST

This test doesn't make sense. Your grep expression contains one or more files (or a literal, if there's no match).

What you really care about is whether $FTPLIST contains any results from ls, right?

Code:

if [ -s "$FTPLIST" ] ; then

http://www.tldp.org/LDP/abs/html/fto.html

--------------

edit: If the sftp session produces non- ls chatter, then my test won't work. You can instead grep for a proper regex, like others mentioned, e.g.:

Code:

grep -qE '\.csv\.pgp$' "$FTPLIST"

if [ $? -eq 0 ] ; then

        # There's a match, so grab the files next.
        #

ruario · 06-30-2014, 02:42 PM

Quote:

Originally Posted by jonnybinthemix

However, can you run SFTP in batch mode and still pass the password using Expect?

Yep, rather counterintuitively you want to set -oBatchMode=no in addition to -b. Quickly scanning your first post, I would just use sshpass instead of expect but each to their own. Another thing, you want to "get" your files. No need for "mget" with sftp, in fact on MacOS and Slackware sftp has no mget command, just use get with wildcards in the filename. I think you could probably do something like this to achieve what you desire:

Code:

export SSHPASS=xxxxx
echo ls | sshpass -e sftp -oBatchMode=no -P $PORTNUMBER -b- $USER@$HOST | tee $FTPLIST
echo "get $FILES" | sshpass -e sftp -oBatchMode=no -P $PORTNUMBER -b- $USER@$HOST | tee $FTPLOG

ruario · 06-30-2014, 03:00 PM

Your other option would be to use a different client that supports the sftp protocol and can be scripted, e.g. lftp or curl (when compiled with libssh2 present). In my experience curl is very slow when using sftp (though that might just be for uploads), so I would personally go for lftp. Unless speed is not an issue because curl does work nicely in all other aspects.

jonnybinthemix · 07-03-2014, 05:45 AM

Thanks guys that really helped.

Adding the glob pattern did the trick perfectly

Thanks again for all help