Running sftp under Expect and transfers fail
 

Guys,
I have a Linux Server (Debian 2.6.18-6-686)from which I want to sftp files to an external (unknown other than some form of windows)server.

If I issue the commands interactively, I get something like this (copied from the screen directly and anonomized):

DEBIAN:/usr/autoftp/scripts# sftp UserName@www.xxx.yyy.zzz
Connecting to www.xxx.yyy.zzz...
UserName@www.xxx.yyy.zzz's password:
sftp> cd IN
sftp> lcd /usr/autoftp/xfr
sftp> put *
Uploading test.csv to /HOME/IN/test.csv
test.csv 100% 726KB 5.4KB/s 02:15
sftp> quit
DEBIAN:/usr/autoftp/scripts#

and the file transfers properly.

However, if I run an expect script that looks like this:

#!/usr/bin/expect
spawn sftp UserName@www.xxx.yyy.zzz
expect "password:"
sleep 2
send "password\r"
expect "sftp>"
send "cd IN\r"
expect "sftp>"
send "lcd /usr/autoftp/xfr\r"
expect "sftp>"
send "put *\r"
expect "sftp>"
send "exit\r"

I get the file transferring to about 60-70% then failing, with no error message. I tried running the sftp with the "-v" switch, but got no information about why the transfer failed. I looked in all the obvious places in /var/log and got nothing. This us utterly reproducible, but I've failed to find the reason...any clues?

And I just noticed the interactive work ends in "quit" while the script ends in "exit" but I don't *think* that will make a material difference, because the error is happening upstream of the difference.

(And yes, I know embedding passwords in a file is not clever, but humour me, I just want to get a script that runs so I can cron it....)

Thanks for any suggestions

You mentioned that the problem is reproducible but have you measured how long it takes for the transfer to fail? I'm betting it's less time than it takes for your manual transfer to complete.

Expect scripts have a timeout (ten seconds by default, if memory serves) that I suspect you will need to increase so that there's sufficient time for your transfers to complete before expect gives up waiting for the response from sftp when the transfer finishes. If you are confident that the transfers won't (ever) fail, you can set the timeout to infinity by including the statement:
near the top of your script or, even better, just before you actually begin the file get/put. Watch out, though, because your script could really hang is there's a connection problem between the two hosts. Better to set it to some reasonable value after running it a few times with a large value and getting the true elapsed time, then increasing it by some fudge factor to reduce the chance of false timeouts. I have some file transfers at work that used to have a timeout of 600 seconds (they're big files) but have actually had to be increased to 1800 seconds (the files have increased in size over time).

And... I don't think "exit" is a recognized sftp command. I've always used "quit".

Hope this helps...

--
Rick

Problem Solved!
 

Thanks Rick, that hit the nail right on the head...I've been using Linux and shell scripting for years, but never used expect before, hence the error, but as soon as I applied your suggested fix: Bingo, the xfer works correctly.

Many thanks for your swift help