LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-15-2014, 07:04 AM   #1
netaxiz
LQ Newbie
 
Registered: Mar 2014
Posts: 4

Rep: Reputation: Disabled
SFTP script to download and delete whats been downloaded


I need to download files from a sftp site and remove from the site what has been successfully downloaded.
i have pvt/public key access and need an automated script the script ive been working on (which isnt working) looks like this


#connect to sftp, cd to out folder and download all #files to local folder
sftp netaxiz@sftp.site.com <<EOF
cd out
mget *.* /home/netaxiz/

quit
EOF
#put list of all downloaded files into filelist.txt
ls /home/netaxiz > filelist.txt

#connect again to SFTP and delete all files that are #in filelist.txt
sftp netaxiz@sftp.site.com <<EOF
cd out

#this is were things go wrong
for I in $(cat filelist.txt); do rm $I; done

EOF

any help with the finishing touches would be greatly appreciated.

thanks
 
Old 03-15-2014, 09:46 AM   #2
netaxiz
LQ Newbie
 
Registered: Mar 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
Ok i have progressed to this which seems a little more promising

#connect to sftp, cd to out folder and download all files to local folder
sftp netaxiz@sftp.site.com <<EOF
cd out
mget *.* /home/netaxiz/

quit
EOF
#put list of all downloaded files into filelist.txt
ls /home/netaxiz > filelist.txt

#connect again to SFTP and delete all files that are in filelist.txt
sftp netaxiz@sftp.site.com <<EOF
cd out


EOF
cat /home/netaxiz/filelist.txt | while read F
do
rm $F
done;



except I now get these errors

rm: cannot remove ‘file1.jpg’: No such file or directory
rm: cannot remove ‘file2.jpg’: No such file or directory
rm: cannot remove ‘file3.jpg’: No such file or directory
rm: cannot remove ‘file4.jpg’: No such file or directory

file1.jpg - file4.jpg are the names of the files in the out folder of the sftp server

Last edited by netaxiz; 03-15-2014 at 01:22 PM.
 
Old 03-15-2014, 10:52 AM   #3
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,028

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
i didnt know you could do rm from within an sftp prompt (i would just use ssh for the removal of server files instead of haxing around with sftp).

you mite want to check into scp which is meant for automated machines whereas sftp prompts and waits for human input (which your script is hacking around).

on second thought i would just use sshfs to mount the remote filesystem then cp/cat/rm/mv/grep/... to my hearts content.

there are multiple ways to skin this cat.
 
Old 03-15-2014, 11:16 AM   #4
netaxiz
LQ Newbie
 
Registered: Mar 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thanks Schneidz
your right, in fact rm is trying to remove files locally not on the sftp server
although rm is a recognised sftp command and will run interactively.
my problem is that I don't have shell access on the sftp server so scp or sshfs won't work

Last edited by netaxiz; 03-15-2014 at 11:18 AM.
 
Old 03-15-2014, 01:37 PM   #5
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,028

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
this is what i get:
Code:
[schneidz@hyper ~]$ echo hello world >  netaxiz.txt
[schneidz@hyper ~]$ sftp localhost
schneidz@localhost's password: 
Connected to localhost.
sftp> cat netaxiz.txt 
Invalid command.
sftp> rm netaxiz.txt 
Removing /home/schneidz/netaxiz.txt
sftp>
i think in your example above the eof is before the rm so it logs out before it gets to that step.
 
Old 03-25-2014, 07:27 AM   #6
netaxiz
LQ Newbie
 
Registered: Mar 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
I managed to solve this with the following script, which does exactly what i needed
it downloads the files from the sftp site and deletes from the sftp site only what has been downloaded
it does this without the need of having a shell on the sftp site no scp or rsync access.


#connect to sftp and download all files to folder
sftp user@sftpsite.com <<EOF
mget * /home/download/
quit
EOF

#put list of all downloaded files into filelist.txt
ls /home/download > /home/docs/filelist.txt

# prefix rm command to each file in filelist.txt
sed -i -e 's_.*_rm "&_' /home/docs/filelist.txt

# put each file in filelist.txt into "" to be able to parse spaces
sed -i.bak 's/$/"/' /home/docs/filelist.txt

#prefix sftp connection information at begining of filelist.txt
sed -i '1s/^/sftp user@sftpsite.com <<EOF\n/' /home/docs/filelist.txt

#append quit and EOF to end of filelist.txt
cat /home/docs/filelist.txt ; echo "quit" >> /home/docs/filelist.txt
cat /home/docs/filelist.txt ; echo "EOF" >> /home/cgerada/sftp/filelist.txt

#make filelist.txt executable
chmod +x /home/docs/filelist.txt

#run filelist.txt as a script to remove all files already downloaded
/bin/bash /home/docs/filelist.txt

# move contents of locally downloaded folder to processed folder to be ready for next download
mv /home/download/* /home/processed/

Last edited by netaxiz; 03-25-2014 at 07:29 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
script auto delete files after downloaded slowerogue Linux - Newbie 1 05-03-2013 12:14 AM
[SOLVED] delete selected files from sftp dina3e Linux - Newbie 4 04-07-2010 03:43 AM
sftp user cannot delete Velocity123 Linux - General 1 10-20-2007 02:43 AM
Install and delete of downloaded software K22 Linux - General 1 09-23-2006 12:11 PM
Can I delete RPM Packages from where I downloaded them AFTER they are installed? Balarabay1 SUSE / openSUSE 3 05-27-2006 10:18 PM


All times are GMT -5. The time now is 04:30 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration