I need to download files from a sftp site and remove from the site what has been successfully downloaded.
i have pvt/public key access and need an automated script the script ive been working on (which isnt working) looks like this


#connect to sftp, cd to out folder and download all #files to local folder
sftp netaxiz@sftp.site.com <<EOF
cd out
mget *.* /home/netaxiz/

quit
EOF
#put list of all downloaded files into filelist.txt
ls /home/netaxiz > filelist.txt

#connect again to SFTP and delete all files that are #in filelist.txt
sftp netaxiz@sftp.site.com <<EOF
cd out

#this is were things go wrong
for I in $(cat filelist.txt); do rm $I; done

EOF

any help with the finishing touches would be greatly appreciated.

thanks

Ok i have progressed to this which seems a little more promising

#connect to sftp, cd to out folder and download all files to local folder
sftp netaxiz@sftp.site.com <<EOF
cd out
mget *.* /home/netaxiz/

quit
EOF
#put list of all downloaded files into filelist.txt
ls /home/netaxiz > filelist.txt

#connect again to SFTP and delete all files that are in filelist.txt
sftp netaxiz@sftp.site.com <<EOF
cd out


EOF
cat /home/netaxiz/filelist.txt | while read F
do
rm $F
done;



except I now get these errors

rm: cannot remove ‘file1.jpg’: No such file or directory
rm: cannot remove ‘file2.jpg’: No such file or directory
rm: cannot remove ‘file3.jpg’: No such file or directory
rm: cannot remove ‘file4.jpg’: No such file or directory

file1.jpg - file4.jpg are the names of the files in the out folder of the sftp server

i didnt know you could do rm from within an sftp prompt (i would just use ssh for the removal of server files instead of haxing around with sftp).

you mite want to check into scp which is meant for automated machines whereas sftp prompts and waits for human input (which your script is hacking around).

on second thought i would just use sshfs to mount the remote filesystem then cp/cat/rm/mv/grep/... to my hearts content.

there are multiple ways to skin this cat.

Thanks Schneidz
your right, in fact rm is trying to remove files locally not on the sftp server
although rm is a recognised sftp command and will run interactively.
my problem is that I don't have shell access on the sftp server so scp or sshfs won't work

this is what i get:i think in your example above the eof is before the rm so it logs out before it gets to that step.

I managed to solve this with the following script, which does exactly what i needed
it downloads the files from the sftp site and deletes from the sftp site only what has been downloaded
it does this without the need of having a shell on the sftp site no scp or rsync access.


#connect to sftp and download all files to folder
sftp user@sftpsite.com <<EOF
mget * /home/download/
quit
EOF

#put list of all downloaded files into filelist.txt
ls /home/download > /home/docs/filelist.txt

# prefix rm command to each file in filelist.txt
sed -i -e 's_.*_rm "&_' /home/docs/filelist.txt

# put each file in filelist.txt into "" to be able to parse spaces
sed -i.bak 's/$/"/' /home/docs/filelist.txt

#prefix sftp connection information at begining of filelist.txt
sed -i '1s/^/sftp user@sftpsite.com <<EOF\n/' /home/docs/filelist.txt

#append quit and EOF to end of filelist.txt
cat /home/docs/filelist.txt ; echo "quit" >> /home/docs/filelist.txt
cat /home/docs/filelist.txt ; echo "EOF" >> /home/cgerada/sftp/filelist.txt

#make filelist.txt executable
chmod +x /home/docs/filelist.txt

#run filelist.txt as a script to remove all files already downloaded
/bin/bash /home/docs/filelist.txt

# move contents of locally downloaded folder to processed folder to be ready for next download
mv /home/download/* /home/processed/

1 members found this post helpful.

@netaxiz - Even though its been 5+ years, thank you! You solution is creative and works well.