Expect SFTP Rename

jonnybinthemix · 09-16-2014, 08:29 AM

Hey Guys,

Having a bit of trouble trying to get an Expect script to do something simple!

I need to rename a file after I've downloaded it, and although I've probably gone the long way round I've got an 'almost' working solution.

I download the files, and then do an ls and pass the contents to a variable, then find the files within that variable and run through each line in a loop... the problem is with the rename command:

Code:

/usr/bin/expect <<! > $FTPLIST
        set timeout -1
        spawn sftp -o$PORT $USER@$HOST
        expect "password:"
        send "$PASS\r"
        expect "sftp>"
        send "cd Output\r"
        expect "sftp>"
        send "ls -1 *$D*\r"
        send "bye\r"
        expect eof
!
awk '/MF_BAT/' $FTPLIST | while read i; do

/usr/bin/expect <<!
        set timeout -1
        spawn sftp -o$PORT $USER@$HOST
        expect "password:"
        send "$PASS\r"
        expect "sftp>"
        send "cd Output\r"
        expect "sftp>"
        send "rename $i PROCESSED_$i"
        expect "sftp>"
        send "bye\r"
        expect eof
!
done

The problem is with send "rename $i PROCESSED_$i"...

If I substitute the $i for the actual filename in txt it works fine, but as soon as I add the $i it tries to send the command immediately after the first $i and therefore the SFTP Session just moans that the rename command is unfinished.

Anyone have any ideas?

Thanks
Jon

schneidz · 09-16-2014, 09:25 AM

if the rename subcommand does what i think it does this would probably be simpler:

Code:

ssh user@host mv /whatever/floats /your/boat

jonnybinthemix · 09-16-2014, 09:27 AM

Unfortunately ssh is not an option. We have no ssh access to the server. The server belongs to someone else, and we have access only via SFTP.

I would have preferred not to use Expect for this at all, but unfortunately I've no other option really.

jonnybinthemix · 09-17-2014, 03:22 AM

Just a quick bump wondering if anyone has any ideas?

I don't understand why if I send the following command in my expect script:

Code:

send "rename file1 processed_file1"

It works and renames the file.

But when I do it with the loop:

Code:

awk '/MF_BAT/' $FTPLIST | while read i; do

/usr/bin/expect <<!
        set timeout -1
        spawn sftp -o$PORT $USER@$HOST
        expect "password:"
        send "$PASS\r"
        expect "sftp>"
        send "cd Output\r"
        expect "sftp>"
        send "rename $i PROCESSED_$i\r"
        expect "sftp>"
        send "bye\r"
        expect eof
!
done

And then run it, it outputs the following:

Code:

sftp> cd Output
sftp> rename MF_BAT_BB160914192302.csv.PGP
You must specify two paths after a rename command.
sftp>  PROCESSED_MF_BAT_BB160914192302.csv.PGP
Invalid command.
sftp> bye

So it picks up the filename from the loop... and adds it to the rename command.. and also it constructs the new filename... but it separates the commands into two!

If anyone has any ideas, it would be greatly appreciated.

Thanks
Jon

schneidz · 09-17-2014, 08:21 AM

maybe there is a \n character at the end of the file you are reading in. (did you by chance create the file on windows ?)

jonnybinthemix · 09-17-2014, 08:23 AM

Not sure what you mean by the file I'm reading in? There's a chance the file was created by a windows box, but I'm not calling in the file really, I'm just renaming it on the remote SFTP server.

schneidz · 09-17-2014, 08:59 AM

what is $FTPLIST ?

jonnybinthemix · 09-17-2014, 09:00 AM

It's a file I'm writing the output of the SFTP session to.

schneidz · 09-17-2014, 09:02 AM

output ?; you seem to be reading it in to your while loop via a pipe| ?

what is the output of:

Code:

cat -A $FTPLIST

jonnybinthemix · 09-17-2014, 09:03 AM

Sorry I've not been clear..

So, before the bit of the script I've posted, theres another SFTP session that logs onto the server, and carries out the command ls -1 *.JPEG.PGP

Because the output of the whole session is written to $FTPLIST, the next bit finds all filenames that start with MF_BAT.

Then I want to loop through those filenames and rename them

jpollard · 09-17-2014, 09:08 AM

We sort of need to know what the contents are - Something appears to be including a newline as part of a file name.

jonnybinthemix · 09-17-2014, 09:08 AM

the result is:

Code:

spawn sftp -oPort=22 Login@XX.XX.XX.XX^M$
Connecting to XX.XX.XX.XX...^M$
reverse mapping checking getaddrinfo for xxxx-xxxx.xxxxxx.xxx-xx.com [XX.XX.XX.XX] ^M^M$
Login@XX.XX.XX.XX's password: ^M$
sftp> cd Output^M$
sftp> ls -1 *160914*^M$
MF_BAT_BB160914192302.csv.PGP^M$
sftp> bye^M$

just realised I changed the ls command.. it's sending ls -1 *$D*, $D is just $(date -d 'yesterday' +%d%m%y) so it's listing files with yesterdays date in the name.

schneidz · 09-17-2014, 09:09 AM

is MF_BAT_BB160914192302.csv.PGP the only file you are having problems with ?

where is it in $FTPLIST (first line / last line ?)

schneidz · 09-17-2014, 09:12 AM

Quote:

Originally Posted by jonnybinthemix

the result is:

Code:

spawn sftp -oPort=22 Login@XX.XX.XX.XX^M$
Connecting to XX.XX.XX.XX...^M$
reverse mapping checking getaddrinfo for xxxx-xxxx.xxxxxx.xxx-xx.com [XX.XX.XX.XX] ^M^M$
Login@XX.XX.XX.XX's password: ^M$
sftp> cd Output^M$
sftp> ls -1 *160914*^M$
MF_BAT_BB160914192302.csv.PGP^M$
sftp> bye^M$

just realised I changed the ls command.. it's sending ls -1 *$D*, $D is just $(date -d 'yesterday' +%d%m%y) so it's listing files with yesterdays date in the name.

those ^M's usually indicate windows newlines... it is the equivelent of hitting the return key after the filename (MF_BAT_BB160914192302.csv.PGP-enter-key).

seems like you are sending \r in the previous iteration.

jonnybinthemix · 09-17-2014, 09:14 AM

That would make sense as the above is the output of this:

Code:

/usr/bin/expect <<! > $FTPLIST
        set timeout -1
        spawn sftp -o$PORT $USER@$HOST
        expect "password:"
        send "$PASS\r"
        expect "sftp>"
        send "cd Output\r"
        expect "sftp>"
        send "ls -1 *$D*\r"
        send "bye\r"
        expect eof
!

I tried omitting the \r on the rename line, but no difference.