Automate FTP session interaction with Expect script

phpguru · 11-20-2012, 05:58 PM

I need to write an expect script that connects to an FTP server on the command line. I need to traverse into a specific folder and download all the files in a certain directory and delete them when I'm done downloading them.

I can do this without expect using

Code:

ncftp -u username ftp.domain.com
cd downloads
get *.tgz

I am trying to solve two different problems.

The first is, I need to get a list of the files in the remote directory, and iterate over them, downloading each one and deleting it.

The second problem is, I need the script to not timeout. I figure once I have switched my stupid code below into a foreach, that's the perfect place to put exp_continue, but I am not sure how to code getting the file list and iterating on the files.

Where I'm stuck is this:

Code:

send "ls -a"

gives me a nice file list, but I have to send "ls -a" and store the resulting list in an array inside expect. How do you store a list of files returned from an FTP server in an array inside expect?

Here's what I have so far...

Code:

#!/usr/bin/expect
set timeout 300
# @TODO Change to a argv
cd /home/myusername/downloads
# Start ncftp
spawn ncftp -u $ftpusername $ftphost
expect "Password:"
# Send the password
send "$password\r"
# Get an FTP prompt
expect "*$ftpusername*"
# Change directories
send "cd downloads\r"
# Moved into directory
expect "*successfully changed*"
# Get an FTP prompt
expect "*downloads*"
# here's where I need a loop
send "ls -a\r"
send "get *.tgz\r"
expect "100%"
# exp_continue
# end loop
expect "*outgoing*"
send "bye"
puts "All done.\r"

firstfire · 11-21-2012, 12:59 AM

Hi.

Consider this example in which I save output of shell command ls -1 to a list:

Code:

#!/usr/bin/expect
spawn sh
expect "$ "
send "ls -1 /\n"
expect "$ "

set results [regexp -all -inline {[^\r\n]+} $expect_out(buffer)]

for {set i 1} {$i<[llength $results]-1} {incr i} {
	puts "+++[lindex $results $i]"
}

I use for loop instead of foreach to ignore first and last elements of $expect_out(buffer), which are "ls -1 /" and "$ ".

PS: The strange [regexp ...] here (instead of just [split ...]) is because expect (unlike tcl) insist on using '\r\n' as line endings.

phpguru · 11-21-2012, 01:42 AM

Quote:

Originally Posted by firstfire

Hi.
Consider this example in which I save output of shell command ls -1 to a list:

Code:

#!/usr/bin/expect
spawn sh
expect "$ "
send "ls -1 /\n"
expect "$ "

set results [regexp -all -inline {[^\r\n]+} $expect_out(buffer)]

for {set i 1} {$i<[llength $results]-1} {incr i} {
	puts "+++[lindex $results $i]"
}

I use for loop instead of foreach to ignore first and last elements of $expect_out(buffer), which are "ls -1 /" and "$ ".

So, this mess:

Code:

set results [regexp -all -inline {[^\r\n]+} $expect_out(buffer)]

is where the magic comes from? You have to catch and parse the output buffer? It's a helpful example, thanks.

I thought that when you

Code:

ls -l

in bash, you get a list (array) of filenames automagically. It doesn't work that way when listing an FTP server? Or within Expect? I'm confused about that.

Quote:

Originally Posted by firstfire

PS: The strange [regexp ...] here (instead of just [split ...]) is because expect (unlike tcl) insist on using '\r\n' as line endings.

Really? I find that odd. On what distro? You send just fine with "\r" at the end only, correct?

firstfire · 11-21-2012, 02:47 AM

Honestly, I'm not expert in expect or tcl and do not know why this happens, but if I print $expect_out(buffer) to stdout and look at it through hexdump, I see 0d 0a (\r\n) after each line. Probably this quote from man expect explains this behaviour:

Quote:

Newlines are usually converted to carriage return, linefeed sequences when output by the terminal
driver. Thus, if you want a pattern that explicitly matches the two lines, from, say,
printf("foo\nbar"), you should use the pattern "foo\r\nbar".

I spent an hour trying to split such response into separate lines. Now I understand why they wanted to call it sex instead of expect (see man expect):

Quote:

BUGS
It was really tempting to name the program "sex" (for either "Smart EXec" or "Send-EXpect"), but good
sense (or perhaps just Puritanism) prevailed.

BTW, as you can see on each my post (small icon on the lower left side of message), I use Ubuntu.

Anyway, my main point was that you can find your file listing in $expect_out(buffer).

phpguru · 11-21-2012, 10:01 AM

Quote:

Originally Posted by firstfire

Code:

set results [regexp -all -inline {[^\r\n]+} $expect_out(buffer)]
for {set i 1} {$i<[llength $results]-1} {incr i} {
	puts "+++[lindex $results $i]"
}

I tried pasting this into my script and run it, and it gives:

Code:

command returned bad code: -101
    while executing
"for {set i 1} {$i<[llength $results]-1} {incr i} {
	puts "+++[lindex $results $i]"
        exp_continue
}"

I took out exp_continue and it ran fine. I figured I would want to plug the downloading inside this for loop, and call exp_continue inside there.